I am fitting ACE and ADE families of models and comparing them to the fully saturated model. I am getting results like this:
observed statistics: 2965
estimated parameters: 4
degrees of freedom: 2961
-2 log likelihood: 848.4525
saturated -2 log likelihood: 844.858
number of observations: 2266
chi-square: 3.594549
p: 1
I am a bit confused by the interpretation of the chi-squared statistic and its degrees of freedom. Before, I have seen output from classic Mx where the df for the chi-square statistic would be three for an ACE model, because it had six observed statistics (i.e. four variances and two covariances from the MZ and DZ covariance matrices) and three estimated parameters (not estimating means).
Here, the df are given as 2961 and all of the p-values for the different models round to 1, because of the very high df. This makes the chi-squared test virtually useless for assessing model fit. Am I missing something here? Is there anything else I could do?
Thankyou
Karin
Printer-friendly version
So the df for the chi-squared should really be the difference between the number of parameters of the saturated model and the number of parameters of the fitted model. If OpenMx is not using that number, then the p-value is incorrect.
mxCompare() might be used to check this out. I suspect that you have 8 parameters in the saturated models (2 x 3 covariance parameters and 2 x 1 mean parameters), so the df for the reported chi-sq should be 4. Indeed it looks like the value of 1 is wrong and this is a bug.
Thanks for the recommendation for mxCompare. That seems to work:
1 univTwinSat 10 844.8580 2955 -5065.142 NA NA NA
2 univTwinSat univACE 4 848.4525 2961 -5073.547 3.594549 6 0.7313508
Looks like my df is 6....
Thankyou
Karin
As far as I can see this is still a problem with the summary() output...
Hi Charles,
Thanks for pointing this out. The degrees of freedom reporting was a bit confusing. The degrees of freedom reported in summary() are basically "number of observed statistics" minus "number of estimated parameters". These are not the same as the degrees of freedom for the Chi-square test, which are "number of estimated parameters in the saturated model" minus "number of estimated parameters in the fitted model", or equivalently "model degrees of freedom" minus "saturated degrees of freedom". In future versions of OpenMx, summary() will report the degrees of freedom used for the Chi-square test along side the Chi-square value to avoid this confusion.
Great, looks better... but I think number of observed statistics is also coming up wrong, which is throwing things off. I was just running a model for 30 subjects with 2 variables at each time point and 6 time points, and observed statistics is outputting as 360... when as far as I can see this should be (6*2)^2 + 6 ? Am I thinking about this wrong?
Hi Charles
You're not thinking about it wrong, so much as OpenMx is thinking about it differently. With raw data, OpenMx counts every observation in the dataset as a statistic. The rationale for this is that it is possible to add numerous individual-level parameters to a model, moderators of paths for example, which would drive the degrees of freedom negative, despite there being plenty of information to estimate the parameters of the model. Note also that if all the observations in a column (i.e., for a variable) were missing, the traditional way of computing df would also be incorrect. Yet the model being fitted might again be perfectly reasonable to fit.
Note also that there are models other than those for means and covariances that can be specified with OpenMx, and the metric of degrees of freedom in terms of the number of observed data points is a good one to hold constant across covariance structure and other types of statistical model, since this will always be "sensical".
Ok, sure. Then this is fine, but the chi sq df are not. I've been adding a $SaturatedLikelihood value to the fit myself, but I can't see where the df of 149 that I get for the chi sq test are taken from? Can I set this somehow? Everything comes out fine with mxCompare...
mxCompare is the preferred way to compare models, even in the null hypothesis significance testing way in which you compare the saturated model to your candidate model as is done by the Chi-Squared test for goodness of fit.
If you want summary to work for this purpose, first check out the help page. Unfortunately, this help page is too hard to get to. Second, give summary your saturated model. Alternatively, give summary the saturated likelihood and the saturated degrees of freedom. In code this looks like
HTH!
Thanks. Yes, the summary help is troublesome, I was just reading the code to try understanding... There is a slot in the fit object that works for SaturatedLikelihood, but when I tried SaturatedDoF this didn't seem to help. It would be convenient if I could add such a parameter to the fit object, is this possible and I'm just doing it wrong, or not implemented? Not a big deal, just a small nicety :)
In general we don't support or recommend modifying the slots of OpenMx objects directly. Sometimes it is possible, but not other times and in either case it is liable to be broken in the future. Sorry!
This makes sense :) cheers