You are here

Versions 0.9.0 vs. 0.5.2

11 posts / 0 new
Last post
Mike Cheung's picture
Offline
Joined: 10/08/2009 - 22:37
Versions 0.9.0 vs. 0.5.2

Hello all,

I am fitting some variance known models (akin to meta-analysis). To handle models with possible covariates, I use a RAM model. The results based on 0.5.2 are comparable to those based on other statistical packages. After upgraded to 0.9.0, the results look odd.

The followings are an example. The -2LL of the 0.5.2 and 0.9.0 versions are 27.79916 and 29.02565, respectively. Any suggestions are highly appreciated. Thanks.

Regards,
Mike

yi <- c(-0.264,-0.230,0.166,0.173,0.225,0.291,0.309,0.435,0.476,0.617,0.651,0.718,0.740,0.745,0.758,0.922,0.938,0.962,1.522,1.844)
vi <- c(0.086,0.106,0.055,0.084,0.071,0.078,0.051,0.093,0.149,0.095,0.110,0.054,0.081,0.084,0.087,0.103,0.113,0.083,0.100,0.141)
my.df <- cbind(yi,vi)
test <- mxModel("test", mxData(observed=my.df, type="raw"),
mxMatrix("Zero", ncol=1, nrow=1, free=FALSE, values=0, name="A"),
mxMatrix("Full", ncol=1, nrow=1, free=FALSE, values=0, labels="data.vi", name="V"),
mxMatrix("Full", ncol=1, nrow=1, free=TRUE, values=0.1, lbound=0.0000001, name="Tau"),
mxMatrix("Iden", ncol=1, nrow=1, name="F", dimnames=list(c("yi"), c("yi"))),
mxMatrix("Full", ncol=1, nrow=1, free=T, values=0, name="M"),
mxAlgebra(V+Tau, name="S"),
mxRAMObjective("A", "S", "F", "M")
)
summary(mxRun(test))

################## Version 0.9.0

Running test

data:

$test.data

yi vi

Min. :-0.2640 Min. :0.05100

1st Qu.: 0.2745 1st Qu.:0.08025

Median : 0.6340 Median :0.08650

Mean : 0.5999 Mean :0.09120

3rd Qu.: 0.7990 3rd Qu.:0.10375

Max. : 1.8440 Max. :0.14900

The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).

free parameters:

name matrix row col Estimate Std.Error

1 Tau 1 1 0.1000 NaN

2 M 1 1 0.5999 9.115587e-10

observed statistics: 20

estimated parameters: 2

degrees of freedom: 18

-2 log likelihood: 29.02565

saturated -2 log likelihood: NA

number of observations: 20

chi-square: NA

p: NA

AIC (Mx): -6.974355

BIC (Mx): -12.44877

adjusted BIC:

RMSEA: NA

timestamp: 2010-09-11 09:48:48

frontend time: 0.3790069 secs

backend time: 0.003436089 secs

independent submodels time: 0.0001730919 secs

wall clock time: 0.3826160 secs

cpu time: 0.3826160 secs

openmx version number: 0.9.0-1417

Warning message:

In model 'test' NPSOL returned a non-zero status code 1. The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).

################## Version 0.5.2

Running test

data:

$test.data

yi vi

Min. :-0.2640 Min. :0.05100

1st Qu.: 0.2745 1st Qu.:0.08025

Median : 0.6340 Median :0.08650

Mean : 0.5999 Mean :0.09120

3rd Qu.: 0.7990 3rd Qu.:0.10375

Max. : 1.8440 Max. :0.14900

The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).

free parameters:

name matrix row col Estimate Std.Error

1 Tau 1 1 0.1315197 0.07353606

2 M 1 1 0.5790348 0.10510040

observed statistics: 20

estimated parameters: 2

degrees of freedom: 18

-2 log likelihood: 27.79916

saturated -2 log likelihood: NA

number of observations: 20

chi-square: NA

p: NA

AIC (Mx): -8.200837

BIC (Mx): -13.06201

adjusted BIC:

RMSEA: NA

timestamp: 2010-09-11 09:53:24

frontend time: 0.3790886 secs

backend time: 0.01191211 secs

independent submodels time: 0.0001611710 secs

wall clock time: 0.3911619 secs

cpu time: 0.3911619 secs

openmx version number: 0.5.2-1376

Warning message:

In model 'test' NPSOL returned a non-zero status code 1. The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).

mspiegel's picture
Offline
Joined: 07/31/2009 - 15:24
RAM and raw data is evaluated

RAM and raw data is evaluated as a FIML model. We had a bug in the -2 LL calculation that was corrected in 0.9.0. This bug affected all the 0.5.x releases (unless my memory serves me wrong). Could someone run this model with 0.3.3 and report the results? I'm not making a claim as to which value is correct, only that a third datapoint will help. Old releases can be downloaded at openmx.psych.virginia.edu/packages.

Mike Cheung's picture
Offline
Joined: 10/08/2009 - 22:37
Thanks for the prompt reply.

Thanks for the prompt reply. Attached are the output for 0.3.3. They are the same as those in 0.5.2. I have also run the analyses (but with a different model parametrization) in Mx 1.70a and Mplus 6. They are similar to those in 0.5.2.

Mx:
Parameter Estimate Std Error Lower 95% Upper 95%


  2     0.579035E+00  0.105101E+00  0.373036E+00  0.785034E+00
  1     0.131520E+00  0.737999E-01 -0.131281E-01  0.276168E+00

Mplus 6:
MODEL RESULTS
Two-Tailed
Estimate S.E. Est./S.E. P-Value
Means
U 0.579 0.107 5.406 0.000
Variances
U 0.132 0.078 1.689 0.091

################## Version 0.3.3

Running test

data:

$test.data

yi vi

Min. :-0.2640 Min. :0.05100

1st Qu.: 0.2745 1st Qu.:0.08025

Median : 0.6340 Median :0.08650

Mean : 0.5999 Mean :0.09120

3rd Qu.: 0.7990 3rd Qu.:0.10375

Max. : 1.8440 Max. :0.14900

The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).

free parameters:

name matrix row col Estimate Std.Error

1 Tau 1 1 0.1315197 0.07353606

2 M 1 1 0.5790348 0.10510040

observed statistics: 20

estimated parameters: 2

degrees of freedom: 18

-2 log likelihood: 27.79916

saturated -2 log likelihood: NA

number of observations: 20

chi-square: NA

p: NA

AIC (Mx): -8.200837

BIC (Mx): -13.06201

adjusted BIC:

RMSEA: NA

timestamp: 2010-09-11 11:40:24

frontend time: 0.3829899 secs

backend time: 0.01260304 secs

independent submodels time: 9.202957e-05 secs

wall clock time: 0.3956850 secs

cpu time: 0.3956850 secs

openmx version number: 0.3.3-1264

Warning message:

In model 'test' NPSOL returned a non-zero status code 1. The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).

Steve's picture
Offline
Joined: 07/30/2009 - 14:03
Thanks for the input, Mike!

Thanks for the input, Mike! Any chance we can add your script to our test suite so that we can catch any future discrepancies?

neale's picture
Offline
Joined: 07/31/2009 - 15:14
Hmmm. Seems like a problem

Hmmm. Seems like a problem concerning definition variables, but there is a definition variable example in the test suite which still runs fine. However, that example does not use a continuous definition variable.

Ryne's picture
Offline
Joined: 07/31/2009 - 15:12
First, thanks for running

First, thanks for running your code with the new version. Finding problems like these really helps the project.

I've attached full output for runs under 0.5.2 and 0.9.0. While I can't identify the problem, I find the difference in numbers of iterations across the two versions interesting. The successful 0.5.2 run goes for 9 major iterations, while 0.9.0 runs for three. This is interesting because it means that 0.9.0 (a) got stuck, as indicated by the non-invertible hessian at iteration 3 and the invalid standard errors, but (b) successfully ran through two complete iterations before hitting the problem. Its unfortunate that we get the same NPSOL message for both runs, though the hessian/SE differences should indicate a problem to users who look at them.

Ryne

tbrick's picture
Offline
Joined: 07/31/2009 - 15:10
Thanks for catching this so

Thanks for catching this so quickly.

These discrepancies resulted from a bug in the latest optimization for FIML RAM models with definition variables.

The latest source revision, r1419, contains the fix.

Would you mind if we add your script to the test suite to make sure we catch any similar bugs that crop up in the future?

mspiegel's picture
Offline
Joined: 07/31/2009 - 15:24
This has been fixed in OpenMx

This has been fixed in OpenMx 0.9.1. Thanks for spotting the bug so quickly.

Mike Cheung's picture
Offline
Joined: 10/08/2009 - 22:37
Thanks a lot for all the

Thanks a lot for all the responses and fixes. The latest version (r1420) works fine!

Please feel free to use the script. The original data were adopted from Joop Hox's "Multilevel analysis: Techniques and Applications" at http://www.ats.ucla.edu/stat/hlm/examples/ma_hox/chapter8.htm

By the way, I built OpenMx r1420 from source. When I ran the analysis, it still indicates "0.9.0-1417." It would be better to reflect the correct versions being used.

mspiegel's picture
Offline
Joined: 07/31/2009 - 15:24
When OpenMx is compiled, the

When OpenMx is compiled, the version number is hard coded into the Makefile and the DESCRIPTION file. I've thought about using the command-line program 'svnversion' to extract the subversion revision number. It would be easier to place this number into the Makefile, and more difficult to place this number into the DESCRIPTION file. The only schemes we have come up with are way too complicated for the problem. So we just tell OpenMx developers to realize they are compiling from source and ignore the version number.

Mike Cheung's picture
Offline
Joined: 10/08/2009 - 22:37
Thanks. It is not a big

Thanks. It is not a big issue.