|
on Sociology of Economics |
| By: | Kevin Boudreau |
| Abstract: | Scientific evaluation and peer review govern the allocation of resources and certification of knowledge in science, yet have been subjected to limited causal investigation. This chapter synthesizes randomized experiments embedded in live peer-evaluation systems at journals, conferences, and funding agencies, restricting attention to published studies. I organize this evidence using a Q–A–R–S framework that decomposes peer review into attributes of submissions (Q), authors (A), reviewers (R), and evaluation systems (S), and interpret outcomes through a view of the core problem of scientific evaluation as assessing new knowledge using the existing stock of knowledge. The chapter treats experimental design choices as objects of analysis, assessing what existing interventions can—and cannot—identify given their designs and settings, the institutional constraints they face, and opportunities for higher-leverage experimentation. I show that randomized experimentation embedded in peer review spans the full Q–A–R–S space, albeit sparsely, and yields uneven but informative insights across different margins. Based on the full body of evidence, I advance several novel claims: (1) system interventions often affect participant behavior with little impact on core evaluative judgments; (2) core evaluations are most clearly shaped by who reviews and their expertise; and (3) peer review functions more reliably as a “filter” of poor submissions than as a fine-grained “ranker” of acceptable submissions. Overall, the evidence points to a functioning institution operating under binding epistemic and organizational constraints, rather than to systemic failure. I identify channels for improving the speed, cost, and reliability of scientific evaluation institutions. Substantial scope remains to redesign embedded experiments to increase inferential power, generalizability, and cumulative insight, while reducing disruption and more tightly linking to institutional innovation and policy changes. |
| JEL: | A14 C93 D02 D83 G18 I23 O31 O38 |
| Date: | 2026–02 |
| URL: | https://d.repec.org/n?u=RePEc:nbr:nberwo:34811 |
| By: | Okalanwa, Solomon; Okalanwa, Stella; Taudes, Alfred |
| Date: | 2026 |
| URL: | https://d.repec.org/n?u=RePEc:wiw:wus051:81476891 |
| By: | Briggs, Ryan C.; Mellon, Jonathan; Arel-Bundock, Vincent |
| Abstract: | Publication practices in the social sciences act as a filter that favors statistically significant results over null findings. While the problem of selection on significance (SoS) is well-known in theory, it has been difficult to measure its scope empirically, and it has been challenging to determine how selection varies across contexts. In this article, we use large language models to extract granular and validated data on about 100, 000 articles published in over 150 political science journals from 2010 to 2024. We show that fewer than 2% of articles that rely on statistical methods report null-only findings in their abstracts, while over 90% of papers highlight significant results. To put these findings in perspective, we develop and calibrate a simple model of publication bias. Across a range of plausible assumptions, we find that statistically significant results are estimated to be one to two orders of magnitude more likely to enter the published record than null results. Leveraging metadata extracted from individual articles, we show that the pattern of strong SoS holds across subfields, journals, methods, and time periods. However, a few factors such as pre-registration and randomized experiments correlate with greater acceptance of null results. We conclude by discussing implications for the field and the potential of our new dataset for investigating other questions about political science. |
| Date: | 2026 |
| URL: | https://d.repec.org/n?u=RePEc:zbw:i4rdps:281 |
| By: | Alexander C. Furnas; Ruixue Jia; Margaret E. Roberts; Dashun Wang |
| Abstract: | This study provides evidence that geopolitical considerations systematically shape funding evaluations of international collaboration proposals. We examine this dynamic in the consequential context of U.S.–China collaboration. Across two large-scale randomized experiments with U.S. policymakers and U.S.-based scientists, we find substantial and consistent penalties for proposals involving China-based collaborators. Policymakers express much greater unconditional support for proposals with Germany-based collaborators than for otherwise identical proposals with China-based collaborators (68% vs. 28%). Crucially, this penalty is not confined to policymakers: scientists themselves exhibit a sizeable 18 percentage-point gap (48% vs. 30%), despite professional expectations of merit-based evaluation. Much of the difference reflects a shift from unconditional to conditional approval rather than outright rejection. These penalties are remarkably consistent across scientific fields and respondent characteristics, with little evidence of heterogeneity, indicating that they reflect geopolitical rather than domain-specific concerns. Overall, the findings suggest that geopolitics influences gatekeeping judgments in government funding, with broad implications for peer review, scientific norms, and the future of international collaboration in an era of intensifying geopolitical competition. |
| JEL: | C93 F52 H40 O31 O38 |
| Date: | 2026–02 |
| URL: | https://d.repec.org/n?u=RePEc:nbr:nberwo:34789 |