Hello all,
I am fitting some variance known models (akin to meta-analysis). To handle models with possible covariates, I use a RAM model. The results based on 0.5.2 are comparable to those based on other statistical packages. After upgraded to 0.9.0, the results look odd.
The followings are an example. The -2LL of the 0.5.2 and 0.9.0 versions are 27.79916 and 29.02565, respectively. Any suggestions are highly appreciated. Thanks.
Regards,
Mike
yi <- c(-0.264,-0.230,0.166,0.173,0.225,0.291,0.309,0.435,0.476,0.617,0.651,0.718,0.740,0.745,0.758,0.922,0.938,0.962,1.522,1.844)
vi <- c(0.086,0.106,0.055,0.084,0.071,0.078,0.051,0.093,0.149,0.095,0.110,0.054,0.081,0.084,0.087,0.103,0.113,0.083,0.100,0.141)
my.df <- cbind(yi,vi)
test <- mxModel("test", mxData(observed=my.df, type="raw"),
mxMatrix("Zero", ncol=1, nrow=1, free=FALSE, values=0, name="A"),
mxMatrix("Full", ncol=1, nrow=1, free=FALSE, values=0, labels="data.vi", name="V"),
mxMatrix("Full", ncol=1, nrow=1, free=TRUE, values=0.1, lbound=0.0000001, name="Tau"),
mxMatrix("Iden", ncol=1, nrow=1, name="F", dimnames=list(c("yi"), c("yi"))),
mxMatrix("Full", ncol=1, nrow=1, free=T, values=0, name="M"),
mxAlgebra(V+Tau, name="S"),
mxRAMObjective("A", "S", "F", "M")
)
summary(mxRun(test))
################## Version 0.9.0
Running test
data:
$test.data
yi vi
Min. :-0.2640 Min. :0.05100
1st Qu.: 0.2745 1st Qu.:0.08025
Median : 0.6340 Median :0.08650
Mean : 0.5999 Mean :0.09120
3rd Qu.: 0.7990 3rd Qu.:0.10375
Max. : 1.8440 Max. :0.14900
The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).
free parameters:
name matrix row col Estimate Std.Error
1 Tau 1 1 0.1000 NaN
2 M 1 1 0.5999 9.115587e-10
observed statistics: 20
estimated parameters: 2
degrees of freedom: 18
-2 log likelihood: 29.02565
saturated -2 log likelihood: NA
number of observations: 20
chi-square: NA
p: NA
AIC (Mx): -6.974355
BIC (Mx): -12.44877
adjusted BIC:
RMSEA: NA
timestamp: 2010-09-11 09:48:48
frontend time: 0.3790069 secs
backend time: 0.003436089 secs
independent submodels time: 0.0001730919 secs
wall clock time: 0.3826160 secs
cpu time: 0.3826160 secs
openmx version number: 0.9.0-1417
Warning message:
In model 'test' NPSOL returned a non-zero status code 1. The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).
################## Version 0.5.2
Running test
data:
$test.data
yi vi
Min. :-0.2640 Min. :0.05100
1st Qu.: 0.2745 1st Qu.:0.08025
Median : 0.6340 Median :0.08650
Mean : 0.5999 Mean :0.09120
3rd Qu.: 0.7990 3rd Qu.:0.10375
Max. : 1.8440 Max. :0.14900
The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).
free parameters:
name matrix row col Estimate Std.Error
1 Tau 1 1 0.1315197 0.07353606
2 M 1 1 0.5790348 0.10510040
observed statistics: 20
estimated parameters: 2
degrees of freedom: 18
-2 log likelihood: 27.79916
saturated -2 log likelihood: NA
number of observations: 20
chi-square: NA
p: NA
AIC (Mx): -8.200837
BIC (Mx): -13.06201
adjusted BIC:
RMSEA: NA
timestamp: 2010-09-11 09:53:24
frontend time: 0.3790886 secs
backend time: 0.01191211 secs
independent submodels time: 0.0001611710 secs
wall clock time: 0.3911619 secs
cpu time: 0.3911619 secs
openmx version number: 0.5.2-1376
Warning message:
In model 'test' NPSOL returned a non-zero status code 1. The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).
RAM and raw data is evaluated as a FIML model. We had a bug in the -2 LL calculation that was corrected in 0.9.0. This bug affected all the 0.5.x releases (unless my memory serves me wrong). Could someone run this model with 0.3.3 and report the results? I'm not making a claim as to which value is correct, only that a third datapoint will help. Old releases can be downloaded at openmx.psych.virginia.edu/packages.
Thanks for the prompt reply. Attached are the output for 0.3.3. They are the same as those in 0.5.2. I have also run the analyses (but with a different model parametrization) in Mx 1.70a and Mplus 6. They are similar to those in 0.5.2.
Mx:
Parameter Estimate Std Error Lower 95% Upper 95%
Mplus 6:
MODEL RESULTS
Two-Tailed
Estimate S.E. Est./S.E. P-Value
Means
U 0.579 0.107 5.406 0.000
Variances
U 0.132 0.078 1.689 0.091
################## Version 0.3.3
Running test
data:
$test.data
yi vi
Min. :-0.2640 Min. :0.05100
1st Qu.: 0.2745 1st Qu.:0.08025
Median : 0.6340 Median :0.08650
Mean : 0.5999 Mean :0.09120
3rd Qu.: 0.7990 3rd Qu.:0.10375
Max. : 1.8440 Max. :0.14900
The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).
free parameters:
name matrix row col Estimate Std.Error
1 Tau 1 1 0.1315197 0.07353606
2 M 1 1 0.5790348 0.10510040
observed statistics: 20
estimated parameters: 2
degrees of freedom: 18
-2 log likelihood: 27.79916
saturated -2 log likelihood: NA
number of observations: 20
chi-square: NA
p: NA
AIC (Mx): -8.200837
BIC (Mx): -13.06201
adjusted BIC:
RMSEA: NA
timestamp: 2010-09-11 11:40:24
frontend time: 0.3829899 secs
backend time: 0.01260304 secs
independent submodels time: 9.202957e-05 secs
wall clock time: 0.3956850 secs
cpu time: 0.3956850 secs
openmx version number: 0.3.3-1264
Warning message:
In model 'test' NPSOL returned a non-zero status code 1. The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).
Thanks for the input, Mike! Any chance we can add your script to our test suite so that we can catch any future discrepancies?
Hmmm. Seems like a problem concerning definition variables, but there is a definition variable example in the test suite which still runs fine. However, that example does not use a continuous definition variable.
First, thanks for running your code with the new version. Finding problems like these really helps the project.
I've attached full output for runs under 0.5.2 and 0.9.0. While I can't identify the problem, I find the difference in numbers of iterations across the two versions interesting. The successful 0.5.2 run goes for 9 major iterations, while 0.9.0 runs for three. This is interesting because it means that 0.9.0 (a) got stuck, as indicated by the non-invertible hessian at iteration 3 and the invalid standard errors, but (b) successfully ran through two complete iterations before hitting the problem. Its unfortunate that we get the same NPSOL message for both runs, though the hessian/SE differences should indicate a problem to users who look at them.
Ryne
Thanks for catching this so quickly.
These discrepancies resulted from a bug in the latest optimization for FIML RAM models with definition variables.
The latest source revision, r1419, contains the fix.
Would you mind if we add your script to the test suite to make sure we catch any similar bugs that crop up in the future?
This has been fixed in OpenMx 0.9.1. Thanks for spotting the bug so quickly.
Thanks a lot for all the responses and fixes. The latest version (r1420) works fine!
Please feel free to use the script. The original data were adopted from Joop Hox's "Multilevel analysis: Techniques and Applications" at http://www.ats.ucla.edu/stat/hlm/examples/ma_hox/chapter8.htm
By the way, I built OpenMx r1420 from source. When I ran the analysis, it still indicates "0.9.0-1417." It would be better to reflect the correct versions being used.
When OpenMx is compiled, the version number is hard coded into the Makefile and the DESCRIPTION file. I've thought about using the command-line program 'svnversion' to extract the subversion revision number. It would be easier to place this number into the Makefile, and more difficult to place this number into the DESCRIPTION file. The only schemes we have come up with are way too complicated for the problem. So we just tell OpenMx developers to realize they are compiling from source and ignore the version number.
Thanks. It is not a big issue.