You are here

mxTryHard() indication for when rety limit reached in loop

14 posts / 0 new
Last post
ZR's picture
ZR
Offline
Joined: 07/26/2017 - 09:34
mxTryHard() indication for when rety limit reached in loop

Hi, I wanted to know if there was a way to know when a model with mxTryHard results in a retry limit reached message other than the output on the screen. I am running a lot of models in a for loop and so checking manually which result in this is not easy, so I wondered if there was a way to tell from looking at the fitted model? I thought that looking at output status code might work but all models including those with retry limit reached gave a code of 0, so I'm not sure how I can tell.

Thanks!

AdminRobK's picture
Offline
Joined: 01/24/2014 - 12:15
not presently
Hi, I wanted to know if there was a way to know when a model with mxTryHard results in a retry limit reached message other than the output on the screen.

I'm afraid there's not. But, that might be a worthwhile future feature. What sort of "way to know" might be useful to you?

Would you mind giving a bit more context to your question? For instance, I assume you're not using the mxTryHardOrdinal() wrapper to mxTryHard(), right? Also for instance, if your concern is uncertainty about how many extra tries were used up, you could just pick some reasonable number of extra tries, use exhaustive=TRUE, and have no worries.

ZR's picture
ZR
Offline
Joined: 07/26/2017 - 09:34
Thank you Rob,

Thank you Rob,

So I am just using mxTryHard() alone within a for loop over 100 models (although I run this parallelised 42 times as I have 4,200 models to run). The data is binary so I am using a liability threshold model and also trying to estimate CIs, so I think for some models it might just be that the model is a little harder to estimate. I have not used extra tries as the model would probably take a very long time to run for some of my 42 jobs. With just the normal mxTryHard some of them do not finish within 100 hours on a cluster.

So increasing the number of tries probably is not ideal, but just knowing which ones give me solution found and which ones give me retry limit reached would be useful, so I know which outputs are ok and which are based on imperfect soutions. For my purpose this could just be a 0 or 1 output, but it's possible there might be something more widely useful. Otherwise the only thing I can think to do is use my error files from job submission and use bash to grep out where these are, but something built into OpenMx might be useful.

Thank you

AdminRobK's picture
Offline
Joined: 01/24/2014 - 12:15
bug?; "Max minutes"

Hmm. You should be able to detect models that appear not to have converged by checking modelFit$output$status$code, which will be zero if it converged, and a positive integer otherwise. But, you're saying that you're getting status codes of zero from models that used up all their extra tries? That's weird. That could be a bug. Can you provide a reproducible instance? Which argument values, if any, are you passing to mxTryHard()?

With just the normal mxTryHard some of them do not finish within 100 hours on a cluster.

There's an MxOption, "Max minutes", which, if set to a positive value, places a maximum on the number of minutes that are allowed to elapse during a call to mxRun() (which of course is called internally by mxTryHard()). The on-load default of "Max minutes" is 0, which does not set a maximum amount of time.

AdminRobK's picture
Offline
Joined: 01/24/2014 - 12:15
mxVersion?

BTW, what is your output from mxVersion()?

ZR's picture
ZR
Offline
Joined: 07/26/2017 - 09:34
Thanks Rob,

Thanks Rob,

It might be a version thing. I'm currently running the whole thing with the latest version as the one I was using was OpenMx_2.9.4-17, which I think may have been a branch or something from Git I used a while ago because I was trying something specifically in that version back then (I think it was after speaking to yourself about something else), but anyway I had not updated it as I've found in the past sometimes the results have changed with different versions and I've been working on similar things for a while. It might therefore be that it works ok with the latest version.

But yes I am getting all 0's from modelFit$output$status$code when I try this on those that report a solution was found and those that get to 11 and say retry limit reached.

That's useful to know about the max time option as well, so thank you.

ZR's picture
ZR
Offline
Joined: 07/26/2017 - 09:34
Actually I've just done a

Actually I've just done a quick check to see if it does give me a 1 in the newer version, but I just get 0's still with OpenMx version: 2.13.2.

The ouptut for the ones where I would expect 1 is:

Computing Hessian and/or standard errors and/or confidence intervals from imperfect sol
Retry limit reached; Best fit=12990.227 (started at 32898.98) (11 attempt(s): 11 valid, 0 errors)

It does say 0 errors, so I'm not sure if we would expect status code 0 or 1 from this?

AdminRobK's picture
Offline
Joined: 01/24/2014 - 12:15
mxTryHardOrdinal()?
It does say 0 errors, so I'm not sure if we would expect status code 0 or 1 from this?

The 0 errors just means that none of the attempts (calls to mxRun()) raised errors. A nonzero status code, by itself, only raises a warning and not an error.

The ouptut for the ones where I would expect 1 is:

Computing Hessian and/or standard errors and/or confidence intervals from imperfect sol
Retry limit reached; Best fit=12990.227 (started at 32898.98) (11 attempt(s): 11 valid, 0 errors)

It says the solution is imperfect, meaning that it did not have an "acceptable" status code. For mxTryHard(), the default is to consider any nonzero status code "unacceptable". It now occurs to me...you said you're analyzing a threshold trait, right? You should probably be using mxTryHardOrdinal(), or at least, not be using all-default argument values with mxTryHard(). mxTryHardOrdinal() is just a wrapper to mxTryHard() that passes the latter some argument values tailored toward analyses of threshold data. Most importantly, mxTryHardOrdinal() uses finetuneGradient=FALSE, since finetuneGradient=TRUE is actually counterproductive with threshold variables. Additionally, and more relevant to this thread, mxTryHardOrdinal() uses OKstatuscodes=c(0,1,5,6), meaning it considers status codes of 1, 5, and 6 to be "acceptable" (see here for an explanation of what each status code means). That's because it's nearly impossible to get status code 0 with some threshold analyses (e.g., the infamous "iloo's model"), due to the limited numerical accuracy of the algorithm that calculates the multivariate-normal probability integrals. To compensate for not relying on status codes, mxTryHardOrdinal() uses exhaustive=TRUE, meaning that it always exhausts its allotment of extra tries.

Two more questions: which optimizer are you using, and are you using any MxConstraints?

ZR's picture
ZR
Offline
Joined: 07/26/2017 - 09:34
Thank you Rob,

Thank you Rob,

Ok thanks, I'll retry with mxTryHardOrdinal().

If I've understood correctly though is it still unclear why I get status code 0 when I get the imperfect solution as it should be one of the unacceptable codes right?

Also to answer your other questions I am using CSOLNP as the optimiser and I'm not using mxConstraints. It's been a while since I first started coding this, but I seem to remember there was a reason I didn't use it.

AdminRobK's picture
Offline
Joined: 01/24/2014 - 12:15
minimal reproducible example?
Ok thanks, I'll retry with mxTryHardOrdinal().

OK. Bear in mind it uses exhaustive=TRUE. You may want to reduce the number of extra tries it uses if you find things are going too slowly.

If I've understood correctly though is it still unclear why I get status code 0 when I get the imperfect solution as it should be one of the unacceptable codes right?

Yes, that's right. If you could provide a minimal example that reproduces that behavior, we'd appreciate it.

Also to answer your other questions I am using CSOLNP as the optimiser and I'm not using mxConstraints. It's been a while since I first started coding this, but I seem to remember there was a reason I didn't use it.

OK, good. Published simulations have shown that CSOLNP is the best choice of the 3 main optimizers for unconstrained models using threshold data. CSOLNP also had a bug that caused it to sometimes enter an uninterruptible infinite loop, which was fixed in the most recent release. That might explain why some of your models ran so slowly with an older version of OpenMx. But, the bug usually (though not always) manifested itself in analyses involving MxConstraints.

ZR's picture
ZR
Offline
Joined: 07/26/2017 - 09:34
Hi Rob,

Hi Rob,

I've attached a script to simulate data and run the model. It's not quite as complex as the model I run (I have covariates in mine, but for simplicity I have excluded those in the simulated data). But I did still get one of the rety limit reached errors in my loop. I had to run it a couple of times to get this. For some reason on some runs I get them and on others I don't so it seems to be a bit inconsistent in that sense.

Let me know if anything I've done is unclear or if it doens't work for you.

So with the exhaustive=T can I still limit the number of Tries with extra tries? I'll try without first and see if it runs ok. Although the ones I submitted yesterday ran much more quickly, despite having retry limit reached for some of the models. So yes it might have been that bug you mentioned.

File attachments: 
ZR's picture
ZR
Offline
Joined: 07/26/2017 - 09:34
Hi Rob,

Hi Rob,

I've also tried this now with mxTryHardOrdinal(). I can see that it uses all the Tries, like below:

Beginning fit attempt 10 of at maximum 10 extra tries
Running twinACE with 5 parameters

Retry limit reached

Solution found

Final run, for Hessian and/or standard errors and/or confidence intervals

Running twinACE with 5 parameters
Warning messages generated from final run for Hessian/SEs/CIs

Solution found! Final fit=11930.744 (started at 29851.992) (11 attempt(s): 11 valid, 0 errors)

So from what you said before, this will now always give a retry limit reached output and then solution found? I'm assuming then the results from this might be ok to use? I guess now with this I can't see any differences between each model output because of the exhaustive=T default. Whereas before I could see which ones found a solution before this. Are these results therefore acceptable to use as they are?

Thanks!

AdminRobK's picture
Offline
Joined: 01/24/2014 - 12:15
printing to console

If it says "Solution found", then mxTryHard()has found a solution that satisfies what has been required of it per the argument values it has received. The mxTryHardOrdinal() wrapper passes mxTryHard() a default of OKstatuscodes=c(0,1,5,6). If you're uneasy about which status code your models have when being returned from mxTryHard(), you could add something like the following toward the end of your loop,

print(paste("status code ",twinACEFit$output$status$code,sep=""))
print(paste("done iteration ",i,sep=""))

, or even have blocks of code that condition on the status code.

And yes, with exhaustive=TRUE, you'll always see "Retry limit reached" in the printing.

ZR's picture
ZR
Offline
Joined: 07/26/2017 - 09:34
Thank you for all the

Thank you for all the suggestions!