parallel computation in linux cluster
Posted on
Veronica_echo
Joined: 02/23/2018
Attachment | Size |
---|---|
Screen Shot 2018-03-27 at 10.52.10 PM.png | 98.47 KB |
Forums
Hi everyone,
I want to conduct several 10,000-simulations for an OpenMx model in Linux cluster. I have over 20 parameters in the model and for each parameter, I need 1000-bootstrap confidence interval to test the coverage. I've add lines in below screenshot, and there are 16 cores available for OpenMx. Even though, the simulation may take about one month at my best guess. Could someone kindly advise some other ways to speed up my simulations? Thanks in advance.
effcient code + compute farm
Other than that, you need more cores. Find a large processor farm - several leading universities have high-performance facilities with 2000 or more 8-32 core mini-computer level cores. Or setup an cloud service (Amazon and google both sell these). You can recruit as many machines as you like.
With setups like these, the task should run over-night or quicker.
Log in or register to post comments
In reply to effcient code + compute farm by tbates
Thank you
Thanks for your kindest advice!
Log in or register to post comments
Cloud is good!
We have had good experiences with cloud computing services, and are preparing a 'how-to' article for publication. Obviously, one would want to try out a small number of simulations locally first, then the same scale on the cloud, before burning money on a possibly errant set of analyses. If you have no such funds available for this type of activity, then the local cluster may be best. Possibly, relatively few threads (cores), but a larger number of separate jobs, would be more efficient than enabling a large number of threads. There is something of a law of diminishing returns with parallel computation, at least in my experience with OpenMx and multicore analyses.
Log in or register to post comments
In reply to Cloud is good! by AdminNeale
Thanks
Thanks for sharing your experience and providing kind advice. I've divided my simulation to "small" jobs, and our system analyst may help me run those separated jobs on the cluster. Thanks again for the advice.
Log in or register to post comments