CIs are NA but their status is OK - is this OK?

8 posts / 0 new
Offline
Joined: 08/18/2013 - 15:33
CIs are NA but their status is OK - is this OK?

Hello OpenMx community,

I am trying to compute confidence intervals for the parameters that I define as matrix algebra
summary(model) gives the following (only CIs showing):

confidence intervals:
lbound     estimate ubound note
ACE.Variance_components[1,1]        NA 6.601569e-01     NA  !!!
ACE.Variance_components[2,1]        NA 3.638004e-01     NA  !!!
ACE.Variance_components[3,1]        NA 7.779408e-01     NA  !!!
ACE.Variance_components[4,1]        NA 3.401663e-01     NA  !!!
ACE.Variance_components[5,1]        NA 3.728521e-01     NA  !!!
ACE.Variance_components[6,1]        NA 5.272372e-01     NA  !!!
ACE.Variance_components[7,1]        NA 6.268351e-01     NA  !!!
ACE.Variance_components[8,1]        NA 4.262247e-01     NA  !!!

summary(model, verbose = TRUE) gives the following output (also truncated):

CI details:
parameter  side     value      fit              diagnostic statusCode            method ACE.mean[1,1]
1  ACE.Variance_components[1,1] lower 0.6308384 342452.3   active box constraint         OK neale-miller-1997  -0.005694432
2  ACE.Variance_components[1,1] upper 0.6893814 342452.3   active box constraint         OK neale-miller-1997  -0.005786365
3  ACE.Variance_components[2,1] lower 0.3225724 342452.3   active box constraint         OK neale-miller-1997  -0.005760040
4  ACE.Variance_components[2,1] upper 0.4051682 342452.3   active box constraint         OK neale-miller-1997  -0.005729072
5  ACE.Variance_components[3,1] lower 0.7592378 342452.3   active box constraint         OK neale-miller-1997  -0.005746466
6  ACE.Variance_components[3,1] upper 0.7939006 342452.3   active box constraint         OK neale-miller-1997  -0.005738854
7  ACE.Variance_components[4,1] lower 0.3079076 342452.3   active box constraint         OK neale-miller-1997  -0.005746881
8  ACE.Variance_components[4,1] upper 0.3735943 342452.3   active box constraint         OK neale-miller-1997  -0.005738552
9  ACE.Variance_components[5,1] lower 0.3407635 342452.3   active box constraint         OK neale-miller-1997  -0.005762424
10 ACE.Variance_components[5,1] upper 0.3928472 349204.8 alpha level not reached         OK neale-miller-1997  -0.005742996

My questions are:
1) Can I use the intervals that have status OK or OK/green?
2) Is there some documentation that explains the status codes of CIs?

Thank you,
Ivan

Offline
Joined: 01/24/2014 - 12:15
optimizer status codes & confidence-limit diagnostics
My questions are:
1) Can I use the intervals that have status OK or OK/green?

You probably shouldn't. While it's true that the optimizer's status code is "OK", the diagnostic is telling you that there are active box constraints at the solutions of the confidence-limit searches, meaning that the confidence intervals may be wrong because they are too narrow. That's why the limits are reported as NA in the vanilla summary() output. Getting rid of unnecessary lbounds in your mxMatrix() calls might resolve the issue. Also, if you're not currently using SLSQP as the optimizer, try switching to it.

2) Is there some documentation that explains the status codes of CIs?

Yes. The optimizer status codes you're seeing in the verbose summary() output have the same meaning as they always do.
See the man page for as.statusCode().

Offline
Joined: 08/18/2013 - 15:33
---

Thank you, this is very helpful.
Do SE's suffer from this in the same way?

Offline
Joined: 01/24/2014 - 12:15
SEs
Do SE's suffer from this in the same way?

No. SEs are a function of the second partial derivatives of the loglikelihood at the ML solution, and do not require additional searches by the optimizer. However, confidence intervals calculated from SEs do not have the attractive theoretical properties that profile-likelihood confidence intervals do. Specifically, profile-likelihood confidence intervals (1) respect constraints on the parameter space, (2) are not necessarily symmetric around the point estimate, and (3) are invariant under transformation of the parameter.

If you're going to form confidence intervals from the SEs, you should at least get robust SEs first, from imxRobustSE().

Offline
Joined: 08/18/2013 - 15:33
...

Thank you again for the very helpful response. At the end I changed the model to relax the constraints and the problem with CIs disappeared

Offline
Joined: 01/24/2014 - 12:15

You're welcome. Glad to hear it worked out for you.

Offline
Joined: 03/01/2013 - 14:09
Status green should be ok IMO

Perhaps the CIs are ok. Rob and I have different philosophies regarding whether a bounded confidence interval may be regarded as ok or not. If 0 is a lower bound for a parameter’s value, and its maximum likelihood estimate is not 3.84 chi-squared units away from zero then Rob considers this not to be a valid lower CI. My perspective is that 95% of estimates will fall within the zero to upper CI range; it just so happens that this lower CI is also, e.g., the 90% CI, the 70% CI - depending on how statistically close to zero the MLE is. An adjustment to the upper CI is necessary; this was described in Pritikin et al 2017 and should be semi-automatic in OpenMx.

Using standard errors to generate a confidence interval can lead to peculiar CIs in these circumstances - the CI of a variance may be less than zero, for example. By and large, I prefer to use unbounded parameter estimation, because the statistical properties are known and valid. The difficulty arises with interpretability, but in some respects this is information about the suitability of the model for the data being analyzed. Such nonsensical results might simply be due to sampling variation. If a variance component is actually zero, then its estimate, on average, should be zero. This average will not be found if zero is a lower bound, in which case the distribution of fit statistics will be a 50:50 mixture of 0 and chi-squared with 1df. The situation gets much messier in the multivariate case, see Wu & Neale. To avoid this statistical challenge, I prefer to use unbounded estimation to avoid, e.g., the Cholesky Problems described by the great Greg Carey.

Offline
Joined: 01/24/2014 - 12:15
yes, but
An adjustment to the upper CI is necessary; this was described in Pritikin et al 2017 and should be semi-automatic in OpenMx.

Yes, but, the adjustment is only implemented for the case where the reference quantity is a free parameter (i.e., not an algebra) which has a box constraint that is active at one of its confidence-limit solutions. For the case that the reference quantity is an algebra, or that there are active box constraints on multiple/other free parameters, the adjustment is not implemented (because the math is still not worked out).