nep-sog New Economics Papers
on Sociology of Economics
Issue of 2020‒02‒03
one paper chosen by
Jonas Holmström
Axventure AB

  1. Estimating publication bias in meta-analyses: A meta-meta-analysis across disciplines and journal tiers By Mathur, Maya B; VanderWeele, Tyler

  1. By: Mathur, Maya B; VanderWeele, Tyler
    Abstract: Selective publication and reporting in individual papers compromise the scientific record, but are meta-analyses as compromised as their constituent studies? We systematically sampled 63 moderately large meta-analyses (at least 40 studies per meta-analysis) in PLOS One, top medical journals, top psychology journals, and Metalab, an online, open-data database of devel- opmental psychology meta-analyses. We empirically estimated publication bias in each. Across all meta-analyses, “statistically significant” results in the expected direction were only 1.20 times more likely to be published than “nonsignificant” results or those in the unexpected direction (95%CI: [0.94, 1.53]), with a confidence interval substantially overlapping the null. Comparable estimates were 0.82 for meta-analyses in PLOS One, 1.23 for top medical journals, 1.54 for top psychology journals, and 4.68 for Metalab. We estimated that for 87% of meta-analyses, the amount of publication bias that would be required to attenuate the point estimate to the null exceeded the amount of publication estimated to be actually present in the vast majority of meta-analyses from the relevant scientific discipline (exceeding the 95th percentile of publication bias). Study-level measures (“statistical significance” with a point estimate in the expected direction and point estimate size) did not indicate more publication bias in higher-tier versus lower-tier journals, nor in the earliest studies published on a topic versus later studies. Overall, the mere act of performing a meta-analysis with a large number of studies (at least 40) and that includes non-headline results may largely mitigate publication bias in meta-analyses, suggesting optimism about the validity of meta-analytic results.
    Date: 2019–12–18

This nep-sog issue is ©2020 by Jonas Holmström. It is provided as is without any express or implied warranty. It may be freely redistributed in whole or in part for any purpose. If distributed in part, please include this notice.
General information on the NEP project can be found at For comments please write to the director of NEP, Marco Novarese at <>. Put “NEP” in the subject, otherwise your mail may be rejected.
NEP’s infrastructure is sponsored by the School of Economics and Finance of Massey University in New Zealand.