Attachment | Size |
---|---|
Approach 1 - R Code.txt [6] | 5.42 KB |
Approach 1 - Results.txt [7] | 244.54 KB |
Approach 2 - R Code.txt [8] | 6.04 KB |
Approach 2 - Results.txt [9] | 258.94 KB |
Approach 3 Model 1 - R Code.txt [10] | 2.79 KB |
Approach 3 Model 1 - Results.txt [11] | 105.77 KB |
Approach 3 Model 2 - R Code.txt [12] | 3 KB |
Approach 3 Model 2 - Results.txt [13] | 159.69 KB |
Approach 3 Model 3 - R Code.txt [14] | 3.05 KB |
Approach 3 Model 3 - Results.txt [15] | 195.13 KB |
Dear Mike and Colleagues,
Greetings! I hope you are well.
I am working on a TSSEM project where the study aims to compare three competing models and pick the “best” model. I have attached a file titled Figure Model 1,2,&3.png to illustrate the three models. I am writing this post to run through my analyses and ask a few questions regarding the same. It would be of immense help if you could please throw some light.
My understanding is that these three models are non-nested. From my reading, I understood that for non-nested models, the AIC value would be the only goodness-of-fit index to ascertain the “best” model. Besides, I figured out that AICs aren’t comparable if non-nested models contain different constructs—it appears that to bring the AICs to the same scale, non-nested models need to draw from the same correlation matrix AND the constructs NOT present in a model need to be freely correlated. I am unsure how to set up TSSEM for non-nested models with different constructs such that the AICs are comparable.
Based on my readings and suggestions in this forum, I found two TSSEM approaches (named approach 1 and 2 and described below) that I believe set up my models such that the models draw from the same correlation matrix and the constructs NOT present in a model are freely correlated. Besides, I describe another approach, named approach 3. I have attached the R code and results for all approaches to this post.
Approach 1: In this approach, in stage 1, I estimated ONE BIG pooled correlation matrix using random-effects TSSEM. In stage 2, I have set up all three models where the constructs NOT present in a model are freely correlated with every other variable. This code is adapted from your example in this link: https://openmx.ssri.psu.edu/sites/default/files/Test.pdf.
Approach 2: In this approach, in stage 1, I estimated ONE BIG pooled correlation matrix using random-effects TSSEM. But I haven’t freely correlated the constructs as I did in approach 1. Instead, I used the select variables approach, where the three correlation matrices, one for each model, are constructed from the BIG pooled correlation matrix. This code is adapted from your example in this link: https://openmx.ssri.psu.edu/sites/default/files/select_variables.pdf.
Approach 3: uses the standard TSSEM approach where each model has its stage-1 pooled correlation matrix estimated ONLY using the constructs present in the corresponding model to be estimated in stage-2.
Questions:
(1) From a TSSEM setup standpoint, I am not sure which one of these approaches renders the AICs of my non-nested models with different variables comparable and consequently helps me ascertain the “best” model? If not, could you please provide some guidance?
(2) Interestingly, the results for all three models in approach 1 are identical to approach 2. Does it mean that these approaches are theoretically the same such that approach 1 explicitly models the covariance among variables, and approach 2 implicitly does so by virtue of drawing from the same pooled correlation matrix?
(3) In case approaches 1 or 2 let me compare my AICs: from a reporting standpoint, would it be appropriate to report that ONE BIG stage-1 pooled correlation matrix and then report the three models in my manuscript? I haven’t seen many TSSEM papers like this. Could you point me to TSSEM citations with similar reporting?
(4) If you notice approach 3 results, the N (total sample size) and K (number of studies) for models 1 and 2 are different from that of model 3. But, in approaches 1 and 2, I chose a universal N value for stage-2 estimation by following the code examples. Is there a way to code the three models in approaches 1 and 2 such that I choose different Ns and Ks?
(5) Do the different N and k values render the stage-2 AICs of models 1 and 2 incomparable with that of stage-2 AICs of model 3? If yes, is there another way to ascertain the “best” model? Probably, perform TSSEM only with common studies across models such that N and k values are the same?
(6) The goodness-of-fit indices are identical for models 2 and 3 (in approaches 1 and 2). I am afraid I have set up my models incorrectly. Do my model setups in stage 2 look congruent to the model pictures?
(7) We have many (about 30) matrices that are non-positive definite. From my readings, it is recommended that we remove those studies. However, there seem to be potential alternatives. It appears that there are various ways to correct such correlation matrices. Some of these approaches sound tedious, but it also appears one approach is to set the negative eigenvalues to 0. I am unsure if that’s a good idea or worth the time in TSSEM analyses. Here are some resources I saw on this:
• http://www.deltaquants.com/manipulating-correlation-matrices
• https://www.r-bloggers.com/2012/10/fixing-non-positive-definite-correlation-matrices-using-r-2/#:~:text=When%20a%20correlation%20or%20covariance,to%20noise%20in%20the%20data
Note: Our effect sizes were corrected for reliability.
(8) In one of your papers (Cheung & Hong, 2017), you used AIC to ascertain the “best” model. Could I also use BIC, given the limitation of AIC for large datasets?
Cheung, M. W. L., & Hong, R. Y. (2017). Applications of meta-analytic structural equation modelling in health psychology: Examples, issues, and recommendations. Health Psychology Review, 11(3), 265-279.
I apologize there are a lot of questions. I think it’s crucial to run this by you and get your thoughts and answers before we move ahead and are confident in our analysis. I hereby humbly request your help. Please let me know if there is additional clarification needed.
Thank you so much for your time and kind consideration of my request.
Best Regards,
Srikanth Parameswaran