You are here

Multiple testing

6 posts / 0 new
Last post
JBosdriesz's picture
Offline
Joined: 03/07/2017 - 09:59
Multiple testing

One of my Phd students recently received reviewer comments for a submitted paper in which she presented 9 univariate ACE models on separate outcome measures. One of the reviewers asked: “Please indicate how correction for multiple comparisons was handled for the genetic modeling. Given that several brain regions were being analyzed, what statistical threshold was used?”
First of all, is any such correction really needed for 9 outcomes?
Second, how would we go about performing such a correction in openmx? What output I'm used to seeing reported in papers often includes the p-value only for selection of the best fitting model, and the path loadings obtained from the model, but no p-values for these path loadings.
Thanks.

AdminRobK's picture
Offline
Joined: 01/24/2014 - 12:15
Let me make sure I understand

Let me make sure I understand the situation. Your student reported results of 9 monophenotype ACE models, for some metric on 9 brain regions, correct? How many different models did she fit for each phenotype? If she fit more than one model per trait, how did she select the "best" model?

AdminNeale's picture
Offline
Joined: 03/01/2013 - 14:09
How was it decided which models to fit?

To further RobK's comment, was there a fixed set of models to be fit for each phenotype, or did the number and type of models vary depending on what was found from the modeling itself?

Possibly, false discovery rate correction could be used. On the whole though, with genetic model estimates I don't think that these procedures are very helpful. By analogy, it is as though one has decided that 20,000 feet is a significantly high part of planet earth, so one draws a map that has a few contours around the Himalayas and the Andes, but is blank white otherwise. For the purposes of global navigation, the map is largely useless. It answers the question of whether there are significantly high bits of the Earth, but whether there are significantly highly heritable regions is rarely the question in structural MRI studies. For practical purposes, it is typically better to have a global map (recognizing its errors and all) than a map of where the big mountains are.

JBosdriesz's picture
Offline
Joined: 03/07/2017 - 09:59
She compared monophenotype

She compared monophenotype ACE vs AE vs CE vs E for each of the 9 brain regions, selecting the 'best' model based on log likelihood and AIC.

AdminRobK's picture
Offline
Joined: 01/24/2014 - 12:15
suggestions

A reviewer can understandably become uneasy if a manuscript reports a large number of hypothesis tests without any Type I error correction. I was wondering if the manuscript presents some results as statistical hypothesis tests when doing so isn't necessary or even accurate. Assuming you are only going to report the results of a single "best" model for each brain region, I would suggest the following. First, for each region, compare the AICs of all four candidate models to one another once all have been fitted (and not in a pairwise manner), and select the one with the smallest AIC. Do not frame this model selection in terms of statistical "significance." Model-selection by AIC is not a statistical hypothesis test in any sense; instead, it approximates cross-validation, for a particular loss metric and given certain assumptions. Second, from each best model, report confidence intervals only for parameters of substantive interest to the study. You could acknowledge in the text that the confidence intervals will only have marginal coverage probability of 95%, whereas their joint coverage probability would be lower than that.

rahyabmosahver's picture
Offline
Joined: 02/28/2018 - 16:03
I have two questions:

I have two questions:
1. Are you saying there is a way to estimate the models entirely without specifying the N or SE?

  1. Does your function also speak to Michael's point about using the correlation of the correlations in the model?
Log in or register to post comments