You are here

Change in type="cor" behavior?

15 posts / 0 new
Last post
bwiernik's picture
Offline
Joined: 01/30/2014 - 19:39
Change in type="cor" behavior?

In OpenMx v. 2.11.5, the fit statistics output is the same whether type="cov" or type="cor". In a previous version of OpenMx, I recall that the manifest variable variances would be subtracted from the number of degrees of freedom and observed statistics, and the fit statistics were calculated using the reduced degrees of freedom. Now, the number of observed statistics number is reduced, but the degrees of freedom and fit statistics are not similarly adjusted.

Also, previously the warning about standard errors and fit statistics for type="cor" being potentially incorrect with the Steiger (1980) citation was printed when summary.MxModel() was called as when mxRun() was called. Now the warning only prints with mxRun()

For example, using the HS.ability.data example in help(mxModel), the output for type="cov" is:

Model Statistics: 
               |  Parameters  |  Degrees of Freedom  |  Fit (-2lnL units)
       Model:             21                     24              10970.90
   Saturated:             45                      0              10928.42
Independence:              9                     36              11882.01
Number of observations/statistics: 301/45
 
chi-square:  χ² ( df=24 ) = 42.48505,  p = 0.01138111
Information Criteria: 
      |  df Penalty  |  Parameters Penalty  |  Sample-Size Adjusted
AIC:      -5.514954               84.48505                 87.79687
BIC:     -94.485600              162.33436                 95.73430
CFI: 0.9798547 
TLI: 0.9697821   (also known as NNFI) 
RMSEA:  0.05058496  [95% CI (0.01638508, 0.07931089)]
Prob(RMSEA <= 0.05): 0.4530891

The output for type="cor" is:

Model Statistics: 
               |  Parameters  |  Degrees of Freedom  |  Fit (-2lnL units)
       Model:             21                     15              1788.898
   Saturated:             45                     -9              1746.413
Independence:              9                     27              2700.000
Number of observations/statistics: 301/36
 
chi-square:  χ² ( df=24 ) = 42.48505,  p = 0.01138111
Information Criteria: 
      |  df Penalty  |  Parameters Penalty  |  Sample-Size Adjusted
AIC:       12.48505               84.48505                 87.79687
BIC:      -43.12161              162.33436                 95.73430
CFI: 0.9798547 
TLI: 0.9697821   (also known as NNFI) 
RMSEA:  0.05058496  [95% CI (0.01638508, 0.07931089)]
Prob(RMSEA <= 0.05): 0.4530891

I'm not sure which version this behavior changed in.

I'm not sure when the change occurred. Was this an intentional change? If so, was it determined that the unadjusted df/fit statistics are more likely to be correct (the problems with doing covariance structure analyses with correlations acknowledged)?

jpritikin's picture
Offline
Joined: 05/24/2012 - 00:35
mistake suspected

I'm pretty sure this is unintentional and indicates a bug.

jpritikin's picture
Offline
Joined: 05/24/2012 - 00:35
v2.9.9

I'm searching back. There has been no change in behavior since v2.9.9 (2018-05-11).

jpritikin's picture
Offline
Joined: 05/24/2012 - 00:35
more info needed

It's hard to test earlier version of OpenMx. Can we narrow this bug down? Is the problem in summary()?

bwiernik's picture
Offline
Joined: 01/30/2014 - 19:39
I would expect it to be in

I would expect it to be in summary() as that is where the fit indices are computed and the Model statistics table constructed.

bwiernik's picture
Offline
Joined: 01/30/2014 - 19:39
Did a change occur in OpenMx:

Did a change occur in OpenMx:::computeOptimizationStatistics()? That is where the DoF are calculated.

jpritikin's picture
Offline
Joined: 05/24/2012 - 00:35
what exactly is wrong?

I installed v2.6.9 (2016-07-28). Cov has observed statistics: 45, degrees of freedom: 24. Cor has observed statistics: 36, degrees of freedom: 15. These look the same. However, I do notice differences in the chi-squared and Information Criteria fit stats,

For cov:

chi-square:  X2 ( df=24 ) = 88.7134,  p = 2.351412e-09
Information Criteria: 
      |  df Penalty  |  Parameters Penalty  |  Sample-Size Adjusted
AIC:       40.71340               130.7134                       NA
BIC:      -48.25725               208.5627                 141.9627

For cor:

chi-square:  X2 ( df=24 ) = 88.7134,  p = 2.351412e-09
Information Criteria: 
      |  df Penalty  |  Parameters Penalty  |  Sample-Size Adjusted
AIC:      58.713401               130.7134                       NA
BIC:       3.106747               208.5627                 141.9627

There are lots of differences here compared to current output. What specifically do you think is incorrect?

tbates's picture
Offline
Joined: 07/31/2009 - 14:25
See Steiger (1980)...

When I run a RAM model with type = "cor" data , I get warning message:

In FUN(X[[i]], ...) :
OpenMx does not yet correctly handle mxData(type='cor') standard errors and fit statistics.
See Steiger (1980), "Tests for 
comparing elements of a correlation matrix".

This is with the current OpenMx: 2.12.2.267 [GIT v2.12.2-267-g3cbee07]

tbates's picture
Offline
Joined: 07/31/2009 - 14:25
update in the works

An upcoming version of OpenMx will constrain the variances of the manifest portion of the S matrix to 1 if type="cor" in RAM models. Secondly, we'll likely stop allowing cor in non-RAM models, with a note to the user to assert type = cov and add constraints where they need to be based on their matrix usage, or warn people that they need to take care of this.

AdminRobK's picture
Offline
Joined: 01/24/2014 - 12:15
tricky

Automating the constraint on the diagonal of the model-expected covariance/correlation matrix is going to be trickier than I thought when I volunteered to tackle this bug. A model of type="RAM" has no MxMatrix or MxAlgebra for its model-expected matrix. The constraint would have to exist solely in the backend. We do have a precedent for creating new constraints in the backend: we do that to implement the constrained representation of the confidence-limit problem. But, those constraints don't persist to be returned to the frontend, and they don't do things like adjust the degrees-of-freedom of the model. Maybe we should just deprecate type="cor"?

bwiernik's picture
Offline
Joined: 01/30/2014 - 19:39
Analyses with correlation

Analyses with correlation matrices are really common, and returning to the previous behavior where the sole change for type="cor" is to display a warning about standard errors and adjusting the degrees of freedom and associated fit indices would be much preferable I think to users just analyzing correlation matrices as covariance matrices without any warning or adjustment.

Rather than doing the diagonal constraint implicitly, would adding a new mxCorrelationConstraint() function that adds an explicit constraint as part of the model be easier? Then, type="cor" could error out if that constraint were not added?

AdminRobK's picture
Offline
Joined: 01/24/2014 - 12:15
version?
and returning to the previous behavior where the sole change for type="cor" is to display a warning about standard errors and adjusting the degrees of freedom and associated fit indices would be much preferable I think to users just analyzing correlation matrices as covariance matrices without any warning or adjustment.

It does warn, though. I just tested it in version 2.12.2. I notice in your OP you said you were running v2.11.5. Have you updated since then?

bwiernik's picture
Offline
Joined: 01/30/2014 - 19:39
Sorry I wasn't clear there.

Sorry I wasn't clear there. The current version does warn during estimation (though not during summary() which would also be useful I think), but does not adjust df or any fit statistics.

My comment about "without any warning or adjustment" is regarding the proposed option of deprecating type="cor". The consequence of doing that, I think, is that users would just analyze the correlation matrix as a covariance, but then not even with the warning.

Mike Cheung's picture
Offline
Joined: 10/08/2009 - 22:37
The create.vechsR() function

The create.vechsR() function in the metaSEM package can be used to create a model-implied covariance matrix ensuring that the diagonals are always ones. It does not use the mxConstraint() function in OpenMx. It treats the error variances of the dependent variables as computed parameters rather than as free parameters.

It was written to fit a correlation structure with weighted least squares in the context of meta-analysis. But it can be modified for maximum likelihood. Please see the attached examples.

File attachments: 
AdminRobK's picture
Offline
Joined: 01/24/2014 - 12:15
OLD behavior, for reference

I still have an installation of R v2.15.3, running OpenMx v1.3, on my 32-bit Windows 7 laptop. When I analyze the Holzinger-Swineford correlation matrix with type="cov", I get this:

observed statistics:  45 
estimated parameters:  23 
degrees of freedom:  22 
-2 log likelihood:  2050.228 
saturated -2 log likelihood:  1714.86 
number of observations:  301 
chi-square:  335.3673 
p:  7.71944e-58 
Information Criteria: 
     df Penalty Parameters Penalty Sample-Size Adjusted
AIC:   291.3673           381.3673                   NA
BIC:   209.8109           466.6309             393.6879
CFI: 0.6698406 
TLI: 0.4597392 
RMSEA:  0.2175366 

When I analyze the correlation matrix with type="cor", I get this (along with status BLUE):

observed statistics:  36 
estimated parameters:  23 
degrees of freedom:  13 
-2 log likelihood:  2050.228 
saturated -2 log likelihood:  1714.86 
number of observations:  301 
chi-square:  335.3673 
p:  9.242367e-64 
Information Criteria: 
     df Penalty Parameters Penalty Sample-Size Adjusted
AIC:         NA                 NA                   NA
BIC:         NA                 NA                   NA
CFI: 0.6698406 
TLI: 0.4597392 
RMSEA:  0.2175366 

So, if we go back THIS far, OpenMx didn't handle correlation matrices correctly, and didn't warn about it, either.