You are here

RMSEA in multiple-group analysis

5 posts / 0 new
Last post
Mike Cheung's picture
Offline
Joined: 10/08/2009 - 22:37
RMSEA in multiple-group analysis
AttachmentSize
PDF icon RMSEA.pdf199.85 KB

Hi,

It seems that OpenMx uses the same formula of RMSEA for both single- and multiple-group analyses. According to Steiger (1998), the RMSEA should be adjusted by a factor of sqrt(K) where K is the no. of groups.

The attached PDF includes the test suggested by Steiger (1998, p. 417):
1. Construct two identical arbitrary data sets (random numbers will suffice).
2. Test one sample with a simple model, for example, a single factor model, and record the RMSEA value.
3. Construct a two-sample model, where each group is tested with the same model used in Step 2, but with no parameters constrained to be equal across populations.
4. The two-sample RMSEA reported by the software should of course be identical to the one-sample value, because two (in theory) completely independent samples have yielded identical fit to two independent versions of the same model. If it is not, check whether multiplying the value by sqrt(2) yields an identical value. If it does, then it is highly likely that the software is generating incorrect values by using the single-sample formula inappropriately.
5. If the test in Step 4 confirms that an error is present, multiply all point estimates and interval estimates of the RMSEA by sqrt(2) to obtain correct values.

Steiger, J. H. (1998). A note on multiple sample extensions of the RMSEA fit index. Structural Equation Modeling: A Multidisciplinary Journal, 5(4), 411–419. http://doi.org/10.1080/10705519809540115

Best,
Mike

tbates's picture
Offline
Joined: 07/31/2009 - 14:25
We're looking at this now

Hi Mike,
Thanks for the clear retro and citations.

We're looking at this now: Should get a response and fix if necessary promptly.

mhunter's picture
Offline
Joined: 07/31/2009 - 15:26
I disagree

Thanks for the test, Mike!

The article is behind a pay-wall that my university doesn't have access to, so I have not yet read the article. With that said, I disagree with the logic of your 4th point:

The two-sample RMSEA reported by the software should of course be identical to the one-sample value, because two (in theory) completely independent samples have yielded identical fit to two independent versions of the same model.

I say this with due reservations about disagreeing with Steiger about anything. I do not think it follows that if sample 1 has RMSEA x, and sample 2 has RMSEA x on the same model that the combined RMSEA for the two models and the two samples should also be x. That's not how the RMSEA formula works. This scenario doubles the (1) sample size, (2) Chi-Squared, and (3) degrees of freedom. When you double all of these things, RMSEA changes by a multiplicative factor of 1/sqrt(2): the precise correction factor. This might seem weird, but it is a consequence of the sample size dependence of RMSEA. The same model, with the same Chi-squared and the same degrees of freedom, will give different RMSEA values if there are 500 independent observations or if there are 1000 independent observations.

This argument is equivalent to saying RMSEA does not depend on sample size, which is false. The formula has sample size in it. In fact, if you thought that doubling the sample size while keeping the Chi-Squared and degrees of freedom constant should produce the same RMSEA, then the correction would be exactly what Steiger suggests.

It is possible Steiger said all this in the article and has convincing arguments against these points, but for now I am not convinced the change is correct.

Mike Cheung's picture
Offline
Joined: 10/08/2009 - 22:37
Thanks, Michael and

Thanks, Michael and Timothy.

I am not arguing for or against this definition. Both lavaan and Mplus use it. I haven't checked other packages yet. I think that it is probably the case. It is fine to use a different definition. But it is important to remind the users about this difference.

mhunter's picture
Offline
Joined: 07/31/2009 - 15:26
No change planned

Hi Mike,

Thanks again for pointing this out to us! It has certainly spurred a lot of discussion at our development team meetings. Currently, we do not plan on making this adjustment to multigroup RMSEA values. The issue is now raised in the help page for the summary method (?mxSummary) and the Steiger paper is cited there, but we say that we do not make the adjustment.

Best,
Mike Hunter