You are here

parallel computation in linux cluster

5 posts / 0 new
Last post
Veronica_echo's picture
Offline
Joined: 02/23/2018 - 01:57
parallel computation in linux cluster

Hi everyone,

I want to conduct several 10,000-simulations for an OpenMx model in Linux cluster. I have over 20 parameters in the model and for each parameter, I need 1000-bootstrap confidence interval to test the coverage. I've add lines in below screenshot, and there are 16 cores available for OpenMx. Even though, the simulation may take about one month at my best guess. Could someone kindly advise some other ways to speed up my simulations? Thanks in advance.

tbates's picture
Offline
Joined: 07/31/2009 - 14:25
effcient code + compute farm

It will be useful to ensure each run is optimally-coded: make sure things that can be reused are being reused, that big arrays are pre-built, not grown (which is slow), picking the fastest optimizer for your particular problem, good start values.

Other than that, you need more cores. Find a large processor farm - several leading universities have high-performance facilities with 2000 or more 8-32 core mini-computer level cores. Or setup an cloud service (Amazon and google both sell these). You can recruit as many machines as you like.

With setups like these, the task should run over-night or quicker.

Veronica_echo's picture
Offline
Joined: 02/23/2018 - 01:57
Thank you

Dr. Bates,

Thanks for your kindest advice!

AdminNeale's picture
Offline
Joined: 03/01/2013 - 14:09
Cloud is good!

Hi Veronica

We have had good experiences with cloud computing services, and are preparing a 'how-to' article for publication. Obviously, one would want to try out a small number of simulations locally first, then the same scale on the cloud, before burning money on a possibly errant set of analyses. If you have no such funds available for this type of activity, then the local cluster may be best. Possibly, relatively few threads (cores), but a larger number of separate jobs, would be more efficient than enabling a large number of threads. There is something of a law of diminishing returns with parallel computation, at least in my experience with OpenMx and multicore analyses.

Veronica_echo's picture
Offline
Joined: 02/23/2018 - 01:57
Thanks

Dr. Neale,

Thanks for sharing your experience and providing kind advice. I've divided my simulation to "small" jobs, and our system analyst may help me run those separated jobs on the cluster. Thanks again for the advice.