parallel computation in linux cluster

Posted on
No user picture. Veronica_echo Joined: 02/23/2018
Hi everyone,

I want to conduct several 10,000-simulations for an OpenMx model in Linux cluster. I have over 20 parameters in the model and for each parameter, I need 1000-bootstrap confidence interval to test the coverage. I've add lines in below screenshot, and there are 16 cores available for OpenMx. Even though, the simulation may take about one month at my best guess. Could someone kindly advise some other ways to speed up my simulations? Thanks in advance.

Replied on Wed, 03/28/2018 - 04:45
Picture of user. tbates Joined: 07/31/2009

It will be useful to ensure each run is optimally-coded: make sure things that can be reused are being reused, that big arrays are pre-built, not grown (which is slow), picking the fastest optimizer for your particular problem, good start values.

Other than that, you need more cores. Find a large processor farm - several leading universities have high-performance facilities with 2000 or more 8-32 core mini-computer level cores. Or setup an cloud service (Amazon and google both sell these). You can recruit as many machines as you like.

With setups like these, the task should run over-night or quicker.

Replied on Wed, 03/28/2018 - 16:18
Picture of user. AdminNeale Joined: 03/01/2013

Hi Veronica

We have had good experiences with cloud computing services, and are preparing a 'how-to' article for publication. Obviously, one would want to try out a small number of simulations locally first, then the same scale on the cloud, before burning money on a possibly errant set of analyses. If you have no such funds available for this type of activity, then the local cluster may be best. Possibly, relatively few threads (cores), but a larger number of separate jobs, would be more efficient than enabling a large number of threads. There is something of a law of diminishing returns with parallel computation, at least in my experience with OpenMx and multicore analyses.

Replied on Fri, 03/30/2018 - 17:48
No user picture. Veronica_echo Joined: 02/23/2018

In reply to by AdminNeale

Dr. Neale,

Thanks for sharing your experience and providing kind advice. I've divided my simulation to "small" jobs, and our system analyst may help me run those separated jobs on the cluster. Thanks again for the advice.