Model Fit Wish list
OpenMx, Mx and other software return multiple fit indices, but a large number exist. This page is a wish-list of indices, how OpenMx or umx computes them.
✓ = returned by OpenMx Summary. ✓(umx) means available via the umx::umxFitIndices function.
- ✓ χ² non-significant?
- ✓ RMSEA < .05?
- ✓ RMSEA 90% CI lower bound < .01 and upper < .10 ?
- ✓ (umx) Is the SRMR < .08 ?
- ✓ CFI > .95 ?
- ✓ TLI
- ✓ Is p-close non-significant ? [tests the null hypothesis that RMSEA in the population is < .05]
- ✓ Parameter estimates make sense/are significant
- All correlation residuals < .10 (can check via helper?)
- All standardized residuals < 1.96 ? [less important in large samples]
- These two are suspect with missing data (if the data are not missing completely at random) FIML estimates of covariances etc. may not match their sample counterparts and residuals could exceed arbitrarily set thresholds for deviation. In addition stats with raw data may be costly to obtain because it requires fitting a model with as many parameters as there are means and covariances. Most studies have at least some missing data and discarding or imputing these increase risk of biased results compared to FIML. It may be worth the loss of certain measures of fit to use FIML.
- Does the quantile plot of standardized residuals look OK (do the standardized residuals fall along a diagonal line)?
- R has great graphical capabilities, and if the missing data caveat does not apply, it would be a one- or two-liner to generate the QQ plot.
- Are indirect effects statistically significant? [test with bootstrap method]
- likelihood-based confidence interval on the function of parameters of interest - this would be very easy to request and faster than bootstrap (though that can usually be done quite easily as well).
- Do we have sufficient statistical power for the test of the close-fit hypothesis and the test of the not-close-fit hypothesis? [generate script with Preacher’s web site]
- Can we argue against equivalent and near-equivalent models?
Model comparison
OpenMx uses mxCompare() to compare models.
- Significant difference in chi-square?
- ✓ Model with lowest AIC
- ✓ Model with lowest BIC
- Model with lowest DIC?
Valuable discussion on fit indices, their vicissitudes, and therefore reasons OpenMx doesn't report many of these are here:
https://openmx.ssri.psu.edu/thread/765#comment-6757
https://openmx.ssri.psu.edu/node/1301
https://openmx.ssri.psu.edu/thread/2204#comment-4780
https://openmx.ssri.psu.edu/thread/697
Printer-friendly version