You are here

Heterogeneity indices and constraints on the between-studies variance-covariance matrix

6 posts / 0 new
Last post
forscher's picture
Offline
Joined: 03/05/2013 - 09:31
Heterogeneity indices and constraints on the between-studies variance-covariance matrix

Hi all, hi Mike,

I am asking a question in reference to this project (https://osf.io/awz2p/). I am revising a paper based on the project in response to reviewer comments.

For most analyses in this project, we fit multivariate meta-analytic models, estimating 11 quantities across all studies. Of course, not every study contributes an estimate of each quantity, so the between-studies variance-covariance matrix is constrained such that all variances are equal and all covariances are equal. This ensures that the model is identifiable.

In the first submission, we reported I2 for each meta-analytic quantity as reported by summary.meta(). One of our reviewers asked us to report tau2 as well as an absolute measure of heterogeneity. This seems reasonable.

However, over the course of thinking about heterogeneity, it occurs to me that it might not make sense to report separate heterogeneity indices for each of the 11 estimated meta-analytic quantities -- after all, all the between-studies variances are constrained to be equal.

Does it make sense to report separate heterogeneity indices given that the between-studies variances are constrained to be equal? If not, how would I obtain a single heterogeneity estimate? From Cheung (2008; https://www.statmodel.com/download/MCheung.pdf), I'm guessing that I can use these formulas:

H2 = Q/Qdf
I2 = (H2-1)/H2

Assuming that mod is a model fit using meta() in R, I believe this would be:

H2 <- summary(mod)$Q.stat$Q/summary(mod)$Q.stat$Q.df
I2 <- (H2-1)/H2

(P.S.: Mike, your advice on this and other projects over the years has been exceptionally helpful! You'll see that you are acknowledged in the author notes of this project and your work is referenced extensively throughout the paper)

Mike Cheung's picture
Offline
Joined: 10/08/2009 - 22:37
Hi Patrick,

Hi Patrick,

If I understand it correctly, the 11 heterogeneity variances are constrained equally. That said, the estimated I2 indices can be different because the within variances may not be the same. The argument "I2=I2q" calculates the heterogeneity index using the Q statistic. The other two options are harmonic mean ("I2=I2hm") and arithmetic mean ("I2=I2am"). The following is an example.

library(metaSEM)
RE <- matrix(c("0.1*Tau2", "0*Cov", "0*Cov", "0.1*Tau2"), ncol=2, nrow=2)
bcg <- meta(y=cbind(ln_Odd_V, ln_Odd_NV), data=BCG, v=cbind(v_ln_Odd_V, cov_V_NV, v_ln_Odd_NV), RE.constraints = RE, I2=c("I2q", "I2hm", "I2am"))
summary(bcg)

Best,
Mike

forscher's picture
Offline
Joined: 03/05/2013 - 09:31
Follow-up questions

Hi Mike,

Thanks for responding. I have a few follow-up questions.

(1) Different I2 estimators. Is there any reason to prefer one of the three I2 estimators ("I2q", "I2hm", "I2am") in this special case?

(2) An absolute measure of heterogeneity. One of our reviewers wanted us to provide an absolute measure of heterogeneity across the 11 meta-analytic effect sizes. Because Tau2 is constrained to be equal, is that possible in our situation?

(3) The meaning of I2 under equality constraints. It seems to be that, because I2 is a function of both Tau2 and the "typical" within-studies variance, the only information that I2 conveys when Tau2 is constrained to be equal across the 11 effect size estimates is about how the "typical" within-studies variance is different across the 11 estimates (large I2 -> small "typical" within-studies variance). Given this, does it even make sense to obtain separate I2 estimates for each of the 11 effect size estimates?

Mike Cheung's picture
Offline
Joined: 10/08/2009 - 22:37
Hi Patrick,

Hi Patrick,

(1) "I2q" is the most popular choice. The following section in my book will give you some ideas and references.

https://books.google.com.sg/books?id=Pw7QBwAAQBAJ&pg=PA92&lpg=PA92&dq=4.3.3+Quantifying+the+degree+of+the+heterogeneity+of+effect+sizes&source=bl&ots=zClT5nmES-&sig=-R_RBdub7pCibrOPqSwfHqyZtdY&hl=en&sa=X&ved=0ahUKEwixhemuoqzQAhWLKY8KHUluCzYQ6AEIGzAA#v=onepage&q=4.3.3%20Quantifying%20the%20degree%20of%20the%20heterogeneity%20of%20effect%20sizes&f=false

(2) No, they are all the same by definition.

(3) If the constraints on the 11 estimates are correct, I2 and the "typical" within-studies variance are just one-to-one transformation. Although it is still correct to interpret the I2, they may not carry much information.

Best,
Mike

forscher's picture
Offline
Joined: 03/05/2013 - 09:31
Overall I2 estimate

Thank you -- this is what I suspected (and thanks for the book reference -- I don't know very much about the different I2 estimators).

I am coming to the conclusion that an overall I2 estimate, collapsed across the 11 effect sizes, will be more informative than the individual I2 estimates. Assuming that "mod" is a model fit with meta(), the following will give me an overall I2 estimate, correct?

H2 <- summary(mod)$Q.stat$Q/summary(mod)$Q.stat$Q.df
I2 <- (H2-1)/H2

Mike Cheung's picture
Offline
Joined: 10/08/2009 - 22:37
Hi Patrick,

Hi Patrick,

My previous R code will report 11 I2 indices. If you want to calculate one single I2 index, you may refer to the references in the following link https://books.google.com.sg/books?id=Pw7QBwAAQBAJ&pg=PA132&lpg=PA132&dq=...

Best,
Mike