|
on Sociology of Economics |
| By: | Zinovyeva, Natalia (University of Warwick); Makany, Milan (Erasmus University Rotterdam) |
| Abstract: | Science disproportionately relies on top researchers to evaluate the work of others, potentially diverting their scarce time from research, mentoring, and other service. We study this trade-off in Italy’s national academic qualification system, where evaluators are randomly assigned to committees. Committees with better-published evaluators select candidates with stronger subsequent citation and career outcomes; they also place greater weight on publication impact rather than quantity. These effects partly reflect spillovers within committees: evaluators assigned to serve alongside more accomplished colleagues exert more effort and adjust their criteria. Assignment to a committee, however, reduces evaluators’ subsequent publication output and PhD supervision, especially among top researchers. Research losses are especially large in the Social Sciences and Humanities, and among those working in small teams. More productive researchers are also disproportionately less likely to volunteer for service again. |
| Keywords: | expert evaluation, committees, academic promotions, decision quality, peer effects, opportunity costs JEL codes: I23, D71, D83, M51, J45 |
| Date: | 2026 |
| URL: | https://d.repec.org/n?u=RePEc:wrk:warwec:1622 |
| By: | Dilger, Alexander |
| Abstract: | Das akademische Publikationssystem ist sehr vielfältig und ausdifferenziert. Der Umgang mit daraus gewinnbaren Informationen erscheint allerdings zu eingeschränkt, vor allem bei Berufungsverfahren mit der Konzentration auf eine oder wenige Publikationen in absoluten TopZeitschriften. Dabei handelt es sich um Spitzenleistungen, doch es gibt noch viele andere wissenschaftliche Leistungen, die insgesamt wichtiger sein können, insbesondere über die Dauer von wissenschaftlichen Karrieren. |
| Abstract: | The academic publishing system is highly diverse and differentiated. However, the way the information derived from it is utilised appears too limited, particularly in appointment procedures that focus on just one or a few publications in top-tier journals. While these represent outstanding achievements, there are many other scholarly contributions that may be more significant overall, especially when viewed across the span of academic careers. |
| Keywords: | Journal, Nachwuchs, Publikation, Qualität, Wissenschaft |
| JEL: | A11 D81 D83 I23 J24 M51 |
| Date: | 2026 |
| URL: | https://d.repec.org/n?u=RePEc:zbw:umiodp:341987 |
| By: | de Gendre, Alexandra; Salamanca, Nicolás |
| Abstract: | Brodeur, Carrell, Figlio and Lusher (2023) present the first descriptive evidence on p-hacking and publication bias throughout the publication process in economics. Using data from the Journal of Human Resources, they show that 1) p-values in initial submissions present humping around the 10 and 5% significance thresholds, 2) the distribution of desk rejections shows more humping than that of papers sent for review, 3) favorable reviewer recommendations are positively related to statistical significance and 4) overall, the p-value distribution of initial and final versions of accepted drafts appear similar. We first conduct a mostly successful computational reproduction of those results. Next, we conduct a series of robustness replication exercises and show that, while the authors' main results are robust to random removals of small amounts of data and to bandwidth choices, they are sensitive to removing papers that contribute many estimates, to some key modelling choices in density discontinuity tests, and to some conceptually equivalent variations of their tests. Last, we complement the authors' results by providing statistical tests of their main hypotheses. Our results support the claims that 1) the distribution of p-values of deskrejected papers displays greater bunching than that of papers sent for review, and 2) the distributions of p-values of initial submissions and final drafts are similar. Overall, our findings lend support to the authors' main claims, suggesting that after excluding desk-rejected papers, the peer review process does not seem to affect p-hacking or publication bias. |
| Date: | 2026 |
| URL: | https://d.repec.org/n?u=RePEc:zbw:i4rdps:305 |
| By: | Brodeur, Abel; Carrell, Scott; Figlio, David; Lusher, Lester |
| Abstract: | We are grateful to Alexandra de Gendre and Nicolás Salamanca for their careful reproduction, robustness analyses, and extensions of our article, "Unpacking p-Hacking and Publication Bias." Their comment exemplifies the type of scientific engagement that strengthens research and advances knowledge. We are particularly pleased that they were able not only to reproduce many of our findings, but also to probe their robustness, develop additional statistical tests, and propose new methodological approaches. Such efforts are especially valuable given the unusual data constraints surrounding our study. Because the underlying editorial data are highly confidential, substantial restrictions governed data access and sharing. Nevertheless, we worked to make the project as transparent and reproducible as possible, and we are delighted to see that these efforts enabled a thoughtful and productive independent assessment. We believe the revised comment makes a meaningful contribution to the literature on publication bias, p-hacking, peer review, and research transparency. We clarify a minor issue regarding sample sizes, highlight points of agreement between the comment and our original article, and discuss how this exchange illustrates the value of reproduction and critical scrutiny in empirical economics. |
| Date: | 2026 |
| URL: | https://d.repec.org/n?u=RePEc:zbw:i4rdps:306 |
| By: | OECD |
| Abstract: | Research assessment frameworks play a central role in shaping the priorities, direction, and culture of scientific research. Yet there are growing concerns over their misalignment with evolving policy priorities, public expectations, and new demands from science. Over-reliance on narrow performance measures has generated perverse incentives and undesirable behaviours. At the same time, these measures tend to undervalue critical research practices and outputs, such as collaboration, openness, societal engagement, and support to policy making. This paper provides a system-level overview of research assessment, highlighting key tensions, mapping the main actors and drivers, and identifying a set of common reform principles through a comparative review of the literature. It makes the case for developing new assessment frameworks that are better aligned with the evolving expectations and demands placed on science. |
| Keywords: | Open science, Research assessment, Research evaluation frameworks, Science policy |
| JEL: | I23 O30 O38 D02 |
| Date: | 2026–04–29 |
| URL: | https://d.repec.org/n?u=RePEc:oec:stiaaa:2026/07-en |
| By: | Sievertsen, Hans Henrik; Smith, Sarah |
| Abstract: | This paper reports results from a survey experiment comparing the effect of (the same) opinions expressed by male versus female experts. Members of the public were asked for their opinions on topical issues and shown the opinion of either a male or a female economist, all professors at leading US universities. We find, first, that experts can persuade members of the public – the opinions of individual expert economists have an effect on public opinion – and, second, that the opinions expressed by female economists are more persuasive than the same opinions expressed by male economists. |
| Date: | 2025–07 |
| URL: | https://d.repec.org/n?u=RePEc:cpr:ceprdp:20484 |
| By: | Garau, Alessio |
| Abstract: | Can a language model improve how economists classify their own papers? Only 15% of four million IDEAS/RePEc records carry usable JEL codes, and similar papers often receive different ones. I use a large language model (LLM) to solve this problem and assign three-digit codes from titles and abstracts, evaluating it on 69, 503 coded articles published from 1991 to 2023 in the top 100 economics journals. Two tests do not assume that author codes provide the correct classification. Across semantic neighbors identified by a separate embedding model, model codes are 1.8 times as consistent as author codes. Holding codes per paper fixed, a blind check finds that 83% of model codes fit official American Economic Association (AEA) guidelines, compared with 67% of author codes. The classifier expands coverage, and the evaluation framework applies whenever human labels are incomplete or noisy. |
| Keywords: | JEL codes; field classification; generative artificial intelligence; large language models; text as data; semantic similarity |
| JEL: | A14 C45 C81 |
| Date: | 2026–07 |
| URL: | https://d.repec.org/n?u=RePEc:pra:mprapa:130163 |