Attachment | Size |
---|---|

dat100NormalNA198.dat | 14.23 KB |

TestCI2.pdf | 101.62 KB |

While running simulations with the development version of OpenMx, I ran into an interesting error:

Error in runHelper(model, frontendStart, intervals, silent, suppressWarnings, :

NLOPT unrecognized error -1; please report to developers

It is somewhat difficult to replicate and it took some time to put together the attached example. As you may see in the attached, I read in the same dataset and fit the same model 10 times, obtaining likelihood-based CIs each time for a single parameter. The error only shows up for some replications.

Thought you all should know!

In the meantime, I have no problem rolling back to a more stable version, provided that the boundAdj argument is available for mxCI (even if in this particular example it's not necessary)

Thank you!

Nice to see that you are trying boundAdj. This option is only available with the latest code.

I just pushed a similar simulation,

inst/models/enormous/wu-neale-2012-lgc.R

This is how I recommend you structure your simulations. Use set.seed(result[rx,'rep']) inside the replications loop to ensure that every replication can be replicated. That way you can point to a specific replication that triggers NLOPT unrecognized error.

As you suspected, NLOPT unrecognized error -1 is only reported by SLSQP. You will not see this error with other optimizers. However, if you are interested in the boundary adjusted CIs then you will need to use SLSQP.

Hm, looking at your script, I think you need to add lbound=0 for the latent factor covariance. But wait, this parameter cannot be less than 0 so the boundary adjustment is not needed. I will email you my draft manuscript to explain better. I suspect you can avoid optimizer failures by estimating the cholesky factor of the correlation matrix. inst/models/passing/bivCorM.R is an example of how to do this.

Oh, perhaps the covariance can be negative so the boundary adjustment is needed.

I see, thank you!

I put this together quickly, and so didn't notice that perhaps I could also put bounds on the factor correlations (-1 to 1 or some such if that's possible).

So, this is a known problem for SLSQP? Do we know how frequently (and under what conditions) this occurs? and why?

I also noticed that the OpenMx version number seems odd to me, even though I got the latest version from GitHub and compiled from source yesterday in order to put this together: 2.6.8.198 [GIT v2.6.8-198-gacf5469]

I thought the most stable release was 2.6.9?

I will have a follow-up question regarding implementation of the underlying algorithm for likelihood-based CIs - in particular whether Wu & Neale is always used (as opposed to the Neale & Miller algorithm), and where to poke around in the source code for its actual implementation. For the most part, I can "trick" OpenMx into giving me what I want for a recent manuscript, but my manual implementation of Wu & Neale fails for trivial cases even if I use Pek & Wu code as a model. But, I still have to debug to ensure I haven't done something incorrectly. Perhaps that issue is best placed in a separate thread anyway.

Regarding the bounds on factor correlations, note that Wu & Neale will only be attempted if there is a lower bound or an upper bound, but not both.

Regarding the version, 2.6.8 and 2.6.9 are almost exactly the same. It outputs the incorrect version because I forgot to rebase on top of 2.6.9 when we did the release. 2.6.8.X is the latest.

"So, this is a known problem for SLSQP? Do we know how frequently (and under what conditions) this occurs? and why?" -- Yes, it is a known problem. It seems to occur when the parameter vector is close to boundary of the feasible set.

The implementation is in ComputeGD.cpp starting around line 310. You can get details about which algorithm was used by using summary(model, verbose=TRUE) after the CIs are run. I provide an example in the draft manuscript I sent over.

Wait, so if one asks for CIs, using mxCI for example, and there are no actual boundaries on the parameter(s) of interest, Neale & Miller is used? Wu and Neale is not the default?

Yes. Also, the algorithm used is reported in summary(model, verbose=TRUE)

Hi,

I think OpenMx has moved from Neale and Miller (1994) to Wu and Neale (2012). By Wu and Neale (2012) I don't mean the algorithm for a bounded parameter, but the algorithm for a parameter without any bound. The help file for the function mxComputeConfidenceInterval cites Pek and Wu (2015) rather than Wu and Neale (2012). The help file for the function mxCI says "The likelihood-based confidence intervals returned using MxCI are obtained by increasing or decreasing the value of each parameter until the -2 log likelihood of the model increases by an amount corresponding to the requested interval", which is a description of Wu and Neale algorithm.

To be specific, when looking for the CI for a parameter "a" without bounds, does the program:

1) minimize (wrt "a") the function:

F_L= (F+3.84- F_a)^2+a, where F is the -2F_{ML}, and

F_U= (F+3.84- F_a)^2-a,

or

2) maximize and minimize the parameter "a" subject to the constraint that

F_a-F <3.84 ?

You're asking whether OpenMx uses an equality or inequality constraint? It depends on the optimizer. SLSQP prefers an inequality constraint. The other optimizers use a penalty function that is added to the fit.

Hi, thanks for attending to my question. I appreciate it a lot.

My question concerns the two different algorithms that could be used for calculating a CI for a parameter without any artificial bound. They do differ in whether an inequality constraint is imposed, but they are fundamentally different algorithms and minimize different functions. Algorithm 1 (#1 in my last post) was proposed by Neale and Miller (1994), it minimizes the function

(F+3.84- F_a)^2 +/- a

This algorithm does not find the CI as defined by profile likelihood CI. This was acknowledged in their paper.

Algorithm 2 (#2 in my last post) was proposed by Wu and Neale and later reviewed by Pek and Wu. Although in that paper we tackled parameters with bounds, but the algorithm was proposed for parameters with no bound. It maximize and minimize the parameter "a" subject to the constraint that F_a-F <3.84. This gives the profile likelihood CI as it is defined.

When you said "SLSQP prefers an inequality constraint", did you mean Algorithm 2 (b/c algorithm 1 has no constraint at all)? When you said "the other optimizers use a penalty function that is added to the fit", did you mean Algorithm 1?

By the way, SLSQP is the default optimizer. Am I right?

As you have described them, algorithm 1 & 2 both arrive at the same MLE. If one gives the profile likelihood CI then the other one does too.

> When you said "SLSQP prefers an inequality constraint", did you mean Algorithm 2 (b/c algorithm 1 has no constraint at all)? When you said "the other optimizers use a penalty function that is added to the fit", did you mean Algorithm 1?

This is clarified in Pritikin, Rappaport, & Neale (in-press). See Equations 1-4.

> By the way, SLSQP is the default optimizer. Am I right?

It was, but CSOLNP is the default in OpenMx v2.7.4

Thanks for clarification. But the two algorithms technically do not give the same result. Neale and Miller (1994) commented right below their Eq. 1 that their algorithm has in a bias (the last term of Eq. 1). Wu and Neale algorithm does not have this problem.

May I know why SLSQP is no longer the default?

> But the two algorithms technically do not give the same result. Neale and Miller (1994) commented right below their Eq. 1 that their algorithm has in a bias (the last term of Eq. 1).

I think it is not the form of the constraint that matters but whether the constraint is encoded as a penalty function (which will have bias) or as a separate constraint (to a constraint aware optimizer).

> May I know why SLSQP is no longer the default?

Some believe that CSOLNP performs better in most circumstances.

You are absolutely right. With CSOLNP, would the CI be found with the penalty function or as a separate constraint?

CSOLNP and NPSOL both by default use the "penalty" formulation (I believe--Joshua will correct me if I'm wrong). Only SLSQP by default uses the "separate constraint" formulation, since it is unambiguously the best of the three gradient-descent optimizers at handling nonlinear constraints.

Thank you to you and Josh for picking up this discussion. I appreciate it.

I revised your code and it works. See below. The key is to set bounds of parameters to restrict the search of the CI algorithm. Note that the bounds I mentioned here has nothing to do with bounds of the parameters discussed in Wu and Neale (2012). Their introduction is just to make the search successful.

The reason that your version did not work out is because you did not specify appropriate bounds for the parameters. The search for CI is very delicate and prone to failure. You need to bound the search within some reasonable range. In your model, the parameter cov is now bound between -0.4 and 0.9; two loadings are bound to be positive to avoid sign change. As long as the search does not touch the artificial bounds, it is OK. If one of them is touched, set a slightly more lenient bound.

Here is the revised code.