bootstrap coverage probability and the bootstrap replication
Posted on

Forums
Hi everyone,
I am conducting a simulation study of a growth model and would like to evaluate the bootstrap CP of it. I kept the simulation replication as 1000 and set bootstrap replication as 1000 and 2000, respectively. The results seemed wiered, since the CPs of bootstrap 1000 (all of them were located between (0.93, 0.97)) were much better than those of bootstrap 2000 (some CPs were quite low, say 0.86). Any advice about this issue? Should I increase the bootstrap replication to a larger number, say 5000? Thank you in advance.
Increase bootstrap replications
Log in or register to post comments
In reply to Increase bootstrap replications by AdminRobK
Thanks, and one more question
Log in or register to post comments
In reply to Thanks, and one more question by Veronica_echo
some tips
If you're using
mxRun()
to initially (i.e., before any bootstrapping or jackknifing) run the model in each replication, you could replacemxRun()
withmxTryHard()
or one of its wrappers. If you do, you'll probably want to read the man page formxTryHard()
.Are you running your simulation in a 'for' loop? If so, you could instead run it in a 'while' loop, e.g.,
i <- 1
while(i <= 1000){
# do simulation stuff here;
# be sure to somewhere define boolean variable `modelRanWell` as TRUE if no errors or warnings, FALSE otherwise
if(modelRanWell){i <- i+1}
}
, which will keep it running until it gives you 1000 "effective" replications.
I don't know what sort of model you're using to generate and fit to data. But, the error message in your attachment makes it sound as though the model-expected covariance matrix was non-PD at the start values. Using
mxTryHard()
should help in cases where the start values are poor for the current dataset. You could also calculate empirical start values from the dataset in each replication, but I can't make any specific suggestions there without knowing more about the model you're fitting.None of the 3 main optimizers can get off the ground if the covariance matrix is non-PD at the start values, and only 14 warnings in 1000 replications is pretty good. It sounds as though CSOLNP, which is the on-load default optimizer, is working well for you. Using
mxTryHard()
, if you're not already, should really cut down on the number of errors and warnings you get.Log in or register to post comments
In reply to some tips by AdminRobK
Thanks for your kind and
Log in or register to post comments
In reply to Thanks for your kind and by Veronica_echo
sounds good
OK. I guess your script just
break
s out of the loop eventually, when some criterion is satisfied?I bet you could get really good start values via
lmer()
, from package 'lme4'.I'm a bit surprised that fitting a growth-curve model at its true parameter values, to data generated under that model, would lead to a non-PD covariance matrix.
Log in or register to post comments
In reply to sounds good by AdminRobK
Thanks for your kind and prompt reply.
My growth model has definition variables, that might be an explaination to non-PD issue? If so, could I have a smaller number of errors by decreasing the range of definition variables (current setting: scaled equally-spaced time and $dv\sim unif(t_{j}-0.45, t_{j}+0.45)$)?
When I use lme4::lmer() or nlme::nlme(), I guess I should use "reml" instead of "ml"?
Log in or register to post comments
In reply to Thanks for your kind and prompt reply. by Veronica_echo
lme4; definition variables
"ml" is actually closer to what OpenMx does than "reml", but it shouldn't matter much for your purposes, since you're just trying to get start values.
I'm not sure, though I doubt it.
Log in or register to post comments