| Abstract: |
This paper examines how estimates of AI use in scientific writing can be
biased when evaluation methods ignore contextual differences across countries
and fields. Using large-scale data on journal publications from Dimensions, we
construct AI-likeness benchmarks based on differences between human-written
and LLM-rephrased abstracts. We show that a pooled benchmark may confound
pre-existing stylistic variation with AI-generated text, producing substantial
distortions across country-field groups even in pre-LLM publications. In
contrast, country-field-specific benchmarks attenuate such distortions and
provide a more credible baseline for comparison. Applying these methods to
publications in 2025 reveals that the pooled benchmark systematically
overestimates AI use in certain countries and fields while underestimating it
in others. These findings highlight the importance of context-aware
measurement for accurate and equitable evaluation of AI use in science. |