Change in type="cor" behavior?
bwiernik
Joined: 01/30/2014
In OpenMx v. 2.11.5, the fit statistics output is the same whether type="cov" or type="cor". In a previous version of OpenMx, I recall that the manifest variable variances would be subtracted from the number of degrees of freedom and observed statistics, and the fit statistics were calculated using the reduced degrees of freedom. Now, the number of observed statistics number is reduced, but the degrees of freedom and fit statistics are not similarly adjusted.
Also, previously the warning about standard errors and fit statistics for type="cor" being potentially incorrect with the Steiger (1980) citation was printed when summary.MxModel() was called as when mxRun() was called. Now the warning only prints with mxRun()
For example, using the HS.ability.data example in help(mxModel), the output for type="cov" is:
| Parameters | Degrees of Freedom | Fit (-2lnL units)
Model: 21 24 10970.90
Saturated: 45 0 10928.42
Independence: 9 36 11882.01
Number of observations/statistics: 301/45
chi-square: χ² ( df=24 ) = 42.48505, p = 0.01138111
Information Criteria:
| df Penalty | Parameters Penalty | Sample-Size Adjusted
AIC: -5.514954 84.48505 87.79687
BIC: -94.485600 162.33436 95.73430
CFI: 0.9798547
TLI: 0.9697821 (also known as NNFI)
RMSEA: 0.05058496 [95% CI (0.01638508, 0.07931089)]
Prob(RMSEA <= 0.05): 0.4530891
The output for type="cor" is:
| Parameters | Degrees of Freedom | Fit (-2lnL units)
Model: 21 15 1788.898
Saturated: 45 -9 1746.413
Independence: 9 27 2700.000
Number of observations/statistics: 301/36
chi-square: χ² ( df=24 ) = 42.48505, p = 0.01138111
Information Criteria:
| df Penalty | Parameters Penalty | Sample-Size Adjusted
AIC: 12.48505 84.48505 87.79687
BIC: -43.12161 162.33436 95.73430
CFI: 0.9798547
TLI: 0.9697821 (also known as NNFI)
RMSEA: 0.05058496 [95% CI (0.01638508, 0.07931089)]
Prob(RMSEA <= 0.05): 0.4530891
I'm not sure which version this behavior changed in.
I'm not sure when the change occurred. Was this an intentional change? If so, was it determined that the unadjusted df/fit statistics are more likely to be correct (the problems with doing covariance structure analyses with correlations acknowledged)?
mistake suspected
I'm pretty sure this is unintentional and indicates a bug.
Log in or register to post comments
v2.9.9
I'm searching back. There has been no change in behavior since v2.9.9 (2018-05-11).
Log in or register to post comments
In reply to v2.9.9 by jpritikin
more info needed
It's hard to test earlier version of OpenMx. Can we narrow this bug down? Is the problem in summary()?
Log in or register to post comments
In reply to more info needed by jpritikin
I would expect it to be in
I would expect it to be in summary() as that is where the fit indices are computed and the Model statistics table constructed.
Log in or register to post comments
In reply to I would expect it to be in by bwiernik
Did a change occur in OpenMx:
Did a change occur in OpenMx:::computeOptimizationStatistics()? That is where the DoF are calculated.
Log in or register to post comments
In reply to I would expect it to be in by bwiernik
what exactly is wrong?
I installed v2.6.9 (2016-07-28). Cov has observed statistics: 45, degrees of freedom: 24. Cor has observed statistics: 36, degrees of freedom: 15. These look the same. However, I do notice differences in the chi-squared and Information Criteria fit stats,
For cov:
chi-square: X2 ( df=24 ) = 88.7134, p = 2.351412e-09 Information Criteria: | df Penalty | Parameters Penalty | Sample-Size Adjusted AIC: 40.71340 130.7134 NA BIC: -48.25725 208.5627 141.9627For cor:
chi-square: X2 ( df=24 ) = 88.7134, p = 2.351412e-09 Information Criteria: | df Penalty | Parameters Penalty | Sample-Size Adjusted AIC: 58.713401 130.7134 NA BIC: 3.106747 208.5627 141.9627There are lots of differences here compared to current output. What specifically do you think is incorrect?
Log in or register to post comments
See Steiger (1980)...
When I run a RAM model with type = "cor" data , I get warning message:
In FUN(X[[i]], ...) :
OpenMx does not yet correctly handle mxData(type='cor') standard errors and fit statistics.
See Steiger (1980), "Tests for
comparing elements of a correlation matrix".
This is with the current OpenMx: 2.12.2.267 [GIT v2.12.2-267-g3cbee07]
Log in or register to post comments
update in the works
An upcoming version of OpenMx will constrain the variances of the manifest portion of the `S` matrix to 1 if `type="cor"` in RAM models. Secondly, we'll likely stop allowing cor in non-RAM models, with a note to the user to assert `type = cov` and add constraints where they need to be based on their matrix usage, or warn people that they need to take care of this.
Log in or register to post comments
In reply to update in the works by tbates
tricky
Automating the constraint on the diagonal of the model-expected covariance/correlation matrix is going to be trickier than I thought when I volunteered to tackle this bug. A model of `type="RAM"` has no MxMatrix or MxAlgebra for its model-expected matrix. The constraint would have to exist solely in the backend. We do have a precedent for creating new constraints in the backend: we do that to implement the constrained representation of the confidence-limit problem. But, those constraints don't persist to be returned to the frontend, and they don't do things like adjust the degrees-of-freedom of the model. Maybe we should just deprecate `type="cor"`?
Log in or register to post comments
Analyses with correlation
Analyses with correlation matrices are really common, and returning to the previous behavior where the sole change for type="cor" is to display a warning about standard errors and adjusting the degrees of freedom and associated fit indices would be much preferable I think to users just analyzing correlation matrices as covariance matrices without any warning or adjustment.
Rather than doing the diagonal constraint implicitly, would adding a new mxCorrelationConstraint() function that adds an explicit constraint as part of the model be easier? Then, type="cor" could error out if that constraint were not added?
Log in or register to post comments
In reply to Analyses with correlation by bwiernik
version?
It _does_ warn, though. I just tested it in version 2.12.2. I notice in your OP you said you were running v2.11.5. Have you updated since then?
Log in or register to post comments
In reply to version? by AdminRobK
Sorry I wasn't clear there.
Sorry I wasn't clear there. The current version does warn during estimation (though not during summary() which would also be useful I think), but does not adjust df or any fit statistics.
My comment about "without any warning or adjustment" is regarding the proposed option of deprecating type="cor". The consequence of doing that, I think, is that users would just analyze the correlation matrix as a covariance, but then not even with the warning.
Log in or register to post comments
The create.vechsR() function
The create.vechsR() function in the metaSEM package can be used to create a model-implied covariance matrix ensuring that the diagonals are always ones. It does not use the mxConstraint() function in OpenMx. It treats the error variances of the dependent variables as computed parameters rather than as free parameters.
It was written to fit a correlation structure with weighted least squares in the context of meta-analysis. But it can be modified for maximum likelihood. Please see the attached examples.
Log in or register to post comments
OLD behavior, for reference
I still have an installation of R v2.15.3, running OpenMx v1.3, on my 32-bit Windows 7 laptop. When I analyze the Holzinger-Swineford correlation matrix with `type="cov"`, I get this:
observed statistics: 45
estimated parameters: 23
degrees of freedom: 22
-2 log likelihood: 2050.228
saturated -2 log likelihood: 1714.86
number of observations: 301
chi-square: 335.3673
p: 7.71944e-58
Information Criteria:
df Penalty Parameters Penalty Sample-Size Adjusted
AIC: 291.3673 381.3673 NA
BIC: 209.8109 466.6309 393.6879
CFI: 0.6698406
TLI: 0.4597392
RMSEA: 0.2175366
When I analyze the correlation matrix with `type="cor"`, I get this (along with status BLUE):
observed statistics: 36
estimated parameters: 23
degrees of freedom: 13
-2 log likelihood: 2050.228
saturated -2 log likelihood: 1714.86
number of observations: 301
chi-square: 335.3673
p: 9.242367e-64
Information Criteria:
df Penalty Parameters Penalty Sample-Size Adjusted
AIC: NA NA NA
BIC: NA NA NA
CFI: 0.6698406
TLI: 0.4597392
RMSEA: 0.2175366
So, if we go back THIS far, OpenMx didn't handle correlation matrices correctly, and didn't warn about it, either.
Log in or register to post comments