CIs are NA but their status is OK - is this OK?
IvanVoronin
Joined: 08/18/2013
Hello OpenMx community,
I am trying to compute confidence intervals for the parameters that I define as matrix algebra
summary(model) gives the following (only CIs showing):
confidence intervals:
lbound estimate ubound note
ACE.Variance_components[1,1] NA 6.601569e-01 NA !!!
ACE.Variance_components[2,1] NA 3.638004e-01 NA !!!
ACE.Variance_components[3,1] NA 7.779408e-01 NA !!!
ACE.Variance_components[4,1] NA 3.401663e-01 NA !!!
ACE.Variance_components[5,1] NA 3.728521e-01 NA !!!
ACE.Variance_components[6,1] NA 5.272372e-01 NA !!!
ACE.Variance_components[7,1] NA 6.268351e-01 NA !!!
ACE.Variance_components[8,1] NA 4.262247e-01 NA !!!
summary(model, verbose = TRUE) gives the following output (also truncated):
CI details:
parameter side value fit diagnostic statusCode method ACE.mean[1,1]
1 ACE.Variance_components[1,1] lower 0.6308384 342452.3 active box constraint OK neale-miller-1997 -0.005694432
2 ACE.Variance_components[1,1] upper 0.6893814 342452.3 active box constraint OK neale-miller-1997 -0.005786365
3 ACE.Variance_components[2,1] lower 0.3225724 342452.3 active box constraint OK neale-miller-1997 -0.005760040
4 ACE.Variance_components[2,1] upper 0.4051682 342452.3 active box constraint OK neale-miller-1997 -0.005729072
5 ACE.Variance_components[3,1] lower 0.7592378 342452.3 active box constraint OK neale-miller-1997 -0.005746466
6 ACE.Variance_components[3,1] upper 0.7939006 342452.3 active box constraint OK neale-miller-1997 -0.005738854
7 ACE.Variance_components[4,1] lower 0.3079076 342452.3 active box constraint OK neale-miller-1997 -0.005746881
8 ACE.Variance_components[4,1] upper 0.3735943 342452.3 active box constraint OK neale-miller-1997 -0.005738552
9 ACE.Variance_components[5,1] lower 0.3407635 342452.3 active box constraint OK neale-miller-1997 -0.005762424
10 ACE.Variance_components[5,1] upper 0.3928472 349204.8 alpha level not reached OK neale-miller-1997 -0.005742996
My questions are:
1) Can I use the intervals that have status OK or OK/green?
2) Is there some documentation that explains the status codes of CIs?
Thank you,
Ivan
optimizer status codes & confidence-limit diagnostics
You probably shouldn't. While it's true that the optimizer's status code is "OK", the diagnostic is telling you that there are active box constraints at the solutions of the confidence-limit searches, meaning that the confidence intervals may be wrong because they are too narrow. That's why the limits are reported as `NA` in the vanilla `summary()` output. Getting rid of unnecessary lbounds in your `mxMatrix()` calls might resolve the issue. Also, if you're not currently using SLSQP as the optimizer, try switching to it.
Yes. The optimizer status codes you're seeing in the verbose `summary()` output have the same meaning as they always do.
See the man page for `as.statusCode()`.
Log in or register to post comments
In reply to optimizer status codes & confidence-limit diagnostics by AdminRobK
---
Thank you, this is very helpful.
Do SE's suffer from this in the same way?
Log in or register to post comments
In reply to --- by IvanVoronin
SEs
No. SEs are a function of the second partial derivatives of the loglikelihood at the ML solution, and do not require additional searches by the optimizer. However, confidence intervals calculated from SEs do not have the attractive theoretical properties that profile-likelihood confidence intervals do. Specifically, profile-likelihood confidence intervals (1) respect constraints on the parameter space, (2) are not necessarily symmetric around the point estimate, and (3) are invariant under transformation of the parameter.
If you're going to form confidence intervals from the SEs, you should at least get robust SEs first, from
imxRobustSE().Log in or register to post comments
In reply to SEs by AdminRobK
...
Thank you again for the very helpful response. At the end I changed the model to relax the constraints and the problem with CIs disappeared
Log in or register to post comments
In reply to ... by IvanVoronin
You're welcome. Glad to hear
You're welcome. Glad to hear it worked out for you.
Log in or register to post comments
In reply to optimizer status codes & confidence-limit diagnostics by AdminRobK
Status green should be ok IMO
Perhaps the CIs are ok. Rob and I have different philosophies regarding whether a bounded confidence interval may be regarded as ok or not. If 0 is a lower bound for a parameter’s value, and its maximum likelihood estimate is not 3.84 chi-squared units away from zero then Rob considers this not to be a valid lower CI. My perspective is that 95% of estimates will fall within the zero to upper CI range; it just so happens that this lower CI is also, e.g., the 90% CI, the 70% CI - depending on how statistically close to zero the MLE is. An adjustment to the upper CI is necessary; this was described in Pritikin et al 2017 and should be semi-automatic in OpenMx.
Using standard errors to generate a confidence interval can lead to peculiar CIs in these circumstances - the CI of a variance may be less than zero, for example. By and large, I prefer to use unbounded parameter estimation, because the statistical properties are known and valid. The difficulty arises with interpretability, but in some respects this is information about the suitability of the model for the data being analyzed. Such nonsensical results might simply be due to sampling variation. If a variance component is actually zero, then its estimate, on average, should be zero. This average will not be found if zero is a lower bound, in which case the distribution of fit statistics will be a 50:50 mixture of 0 and chi-squared with 1df. The situation gets much messier in the multivariate case, see Wu & Neale. To avoid this statistical challenge, I prefer to use unbounded estimation to avoid, e.g., the Cholesky Problems described by the great Greg Carey.
Log in or register to post comments
In reply to Status green should be ok IMO by AdminNeale
yes, but
Yes, but, the adjustment is only implemented for the case where the reference quantity is a free parameter (i.e., not an algebra) which has a box constraint that is active at one of its confidence-limit solutions. For the case that the reference quantity is an algebra, or that there are active box constraints on multiple/other free parameters, the adjustment is not implemented (because the math is still not worked out).
Log in or register to post comments