On “AI-generated norms” and Anglocentrism: a peer review

I saw a thing fly by on PsyArxiv and must write about it. Warning: snark detected. This is a new paper by Green, Kong, Brysbaert, and Keogh with the following abstract:

“This paper revisits the Age of Acquisition (AoA) norms of Kuperman et al. (2012). Three studies were conducted. Study 1 reports a crowdsourcing ‘megastudy’ obtaining 790,024 estimates from participants providing the age they could first read and write 11,074 early acquired words from Kuperman et al. (2012). The study aimed to differentiate between oral language receptive AoA and print AoA. The results correlate well with the original estimates and offer slightly higher AoA values for print knowledge. They are released as useful supplements to the original norms. Study 2 explored the potential of Large Language Models (LLMs), specifically GPT-4o, to replicate these new crowdsourced AoA norms. The findings indicated a strong correlation between AI-generated estimates and human judgments, showing the utility of AI in generating AoA estimates. The results confirm that AI is a valuable resource of norms for psycholinguistic and educational research, of particular value for under-resourced languages and researchers with limited resources. Based on the successful application of AI in Study 2, Study 3 extended the method to the entire set of words in the English Crowdsourcing Project (ECP), producing AI-generated AoA estimates for approximately 62,000 English words. This provides a substantial database of AoA norms that are found to correlate very highly with human-generated estimates (r =.86) and perform well in accounting for word processing times in regression analyses. The AI generated results have some important limitations, including overestimating the vocabulary acquired at some ages.”

I am not a fan of this method myself and I think, like the authors, that it has some important limitations. I have noted some of these limitations in work with Bill Thompson, where we warn against taking any ratings and especially imputed ones at face value and show that they amplify biases and artefacts of rating instructions (Dingemanse & Thompson 2020). For more arguments, see also Cassandra Jacobs’ piece ‘Why ChatGPT is bad for open psycholinguistics‘. And for more a general discussion of the risks of poor reproducibility of anything coming out of proprietary language models, see Liesenfeld et al. 2023. I think it would be useful for studies like this to engage with such existing critiques, but this is not my main target here.

“Under-resourced languages”

My main point of critique is very simple. There is a bold claim in this abstract, and in the discussion of the paper, which I think is utterly irresponsible and a slap in the face of any psycholinguists and cognitive scientists doing serious cross-linguistic work (e.g., Malt & Majid 2013; Blasi et al. 2022). It is this bit: “The results confirm that AI is a valuable resource … of particular value for under-resourced languages and researchers with limited resources.”

This piqued my interest when I first saw it. I could see why people might want to cite a paper like this for a point like that. Nice to have justification of wanting to use AI to make our life easier. Serving under-resourced languages sounds like a noble goal, and here readers are being told it can be done cheaply, too. Sounds like a game changer. Might this be one of the rare studies looking beyond English, and trying to cross-validate its method or materials on “under-resourced languages”? But how could this even work, given that under-resourced languages are, well, under-resourced?

I searched in vain for support for the claim, or indeed any form of critical discussion of it. The only place where it recurs is in the discussion, where the paper states without evidence: “Complementing human judgements through AI may be particularly valuable to supplement norms for languages that do not yet have the wealth of resources available for English.” (p. 4)

I find it hard to be gentle here because frankly this is an irresponsible and reckless claim. It is gratuitous AI boosterism that actively undermines the interests of communities speaking under-resourced languages, the scientists working with them, and in fact anyone with a spark of scientific curiosity. I am calling it out in public because this is a preprint posted in public, implying the authors are open to feedback, and the ms may be revised. Here goes (with apologies for medium levels of snark).

Take any language, for instance English

Let’s start by figuring out what the paper does and doesn’t do. You wouldn’t know it from title, abstract, discussion or conclusion, but this work deals exclusively with English (it would benefit from following the Bender rule of identifying the language you work on). In particular, it uses GPT4, a transformer-based model trained on unfathomable amounts of English data (and maybe some other languages, see below) to generate psycholinguistic ratings for 62,000 English words; then it shows that such ratings, for English words we can cross-check, show reasonable correlations with crowdsourced judgements by English speakers.

Now let’s think about what it means to claim that any of this would be useful, no, “particularly valuable”, for under-resourced languages and for researchers with limited resources. It means we need to believe that the high-dimensional distributional semantics derived from countless documents in English is going to give us some kind of unique insight into psycholinguistic properties of vocabulary in other languages. Not just closely related varieties of English; not just other Germanic languages; no, any language at all, in fact especially lower-resourced ones.

Where do I even start. This is literally proposing that everyone, especially researchers with “limited resources”, can rely on a bullshit generator that will blurt out numbers for machine-translated bits of vocabulary, where those numbers ultimately are based just on humongous amounts of data for a single WEIRD language. Never mind looking at languages on their own terms, never mind thinking carefully about linguistic diversity, never mind designing experimental or observational studies — here, just try on these rosy English-tinted glasses and everything will look all prim and proper, promise.

We know that English words and concepts cannot be assumed to naturally or unproblematically map onto the vocabulary of other languages (Malt & Majid 2013), yet this ms is advocating to do just that. We know under-resourced languages typically are precisely the ones that are most different from the high-resourced ones (Joshi et al. 2021), yet this ms blithely ignores that. We know that over-reliance on English hinders progress in cognitive science (Blasi et al. 2022), yet this ms proposes to turn up that reliance several notches. We know under-resourced languages are typically those that suffer most from systematic inequalities in language technology (Blasi et al. 2021), and this ms just tells us drink the AI Koolaid and forget everything. We know that painstakingly collected and carefully calibrated inventories like the MB-CDI reveal intriguing and important cross-linguistic differences in early vocabulary (Wordbank), and this ms just proposes that English-based AI sludge is all we need.

Tech hype cycle

All this is not even taking into account any of the limitations of “AI”-generated ratings the ms itself notes. Ratings are distorted in unpredictable ways for some bits of vocabulary, we don’t quite know which bits. Age of acquisition is overestimated for some bits, we can’t predict which bits exactly. Caution is needed even when taking this English-derived data and using it for follow-up work on English.

A moment’s thought will show that these limitations and uncertainties must only become larger as we move further away from known territory. We are by definition on much more shaky ground for any languages and datasets that are less well-resourced than English. It follows we should probably be extra cautious about generalizing to contexts where we don’t have the same opportunities for cross-validation against existing measures or for crowd-sourcing new data. Yet strikingly, all talk of limitations goes out the window when it comes to “under-resourced” languages and scientists with “limited resources”. Now suddenly all we need to think about is the opportunities.

This —the shoving under the carpet of limitations, and the wide-eyed focusing on opportunities— is of course a well known mechanism of the tech hype cycle. I am disappointed to see it in an academic paper that purports to make a scientific contribution. I would have liked to see serious thought put into actually empowering scientists to do meaningful work.

If this preprint gets revised, I hope we’ll see an updated version that omits the gratuitous AI boosting, the misguided claims about “under-resourced languages”, and the cynical siren song to scientists with “limited resources”. It might even engage in some of the critique of GPT-derived ratings others have provided. Despite the snark, I offer this post as a constructive contribution and I hope some of the sources below will be appreciated by readers.

Reviewed preprint

• Green, Clarence, Anthony Pak Hin Professor KONG, Marc Brysbaert, and Kathleen Keogh. 2025. ‘Crowdsourced and AI-Generated Age of Acquisition (AoA) Norms for Vocabulary in Print: Extending the Kuperman et al. (2012) Norms’. OSF. doi:10.31234/osf.io/698mw_v1.

References cited

• Bender, Emily M. 2019. ‘The #BenderRule: On Naming the Languages We Study and Why It Matters’. The Gradient, September. https://thegradient.pub/the-benderrule-on-naming-the-languages-we-study-and-why-it-matters/.
• Blasi, Damian, Antonios Anastasopoulos, and Graham Neubig. 2022. ‘Systematic Inequalities in Language Technology Performance across the World’s Languages’. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5486–5505. Dublin, Ireland: Association for Computational Linguistics. https://aclanthology.org/2022.acl-long.376.
• Blasi, Damián E., Joseph Henrich, Evangelia Adamou, David Kemmerer, and Asifa Majid. 2022. ‘Over-Reliance on English Hinders Cognitive Science’. Trends in Cognitive Sciences, October. doi:10.1016/j.tics.2022.09.015.
• Dingemanse, Mark, and Bill Thompson. 2020. ‘Playful Iconicity: Structural Markedness Underlies the Relation between Funniness and Iconicity’. Language and Cognition 12 (1): 203–24. doi:10.1017/langcog.2019.49.
• Joshi, Pratik, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. ‘The State and Fate of Linguistic Diversity and Inclusion in the NLP World’. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 6282–93. Online: Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.560.
• Liesenfeld, Andreas, Alianda Lopez, and Mark Dingemanse. 2023. ‘Opening up ChatGPT: Tracking Openness, Transparency, and Accountability in Instruction-Tuned Text Generators’. In CUI ’23: Proceedings of the 5th International Conference on Conversational User Interfaces. doi:10.1145/3571884.3604316.
• Liesenfeld, Andreas, and Mark Dingemanse. 2024. ‘Rethinking Open Source Generative AI: Open-Washing and the EU AI Act’. In The 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’24), 1774–87. Rio de Janeiro, Brazil: ACM. doi:10.1145/3630106.3659005.
• Malt, Barbara C., and Asifa Majid. 2013. ‘How Thought Is Mapped into Words’. Wiley Interdisciplinary Reviews: Cognitive Science, n/a-n/a. doi:10.1002/wcs.1251.
  

Afterthought: “But GPT4 is multilingual”

Ah yes, GPT4 “boasts strong multilingual support” as you can read in countless online sources, most of them GPT4-generated by the looks of it. For legal reasons OpenAI will never be transparent about the pre-training data —likely the biggest intellectual property theft of the past millenium— but it is safe to assume that the amount of English training data is multiple orders of magnitude larger than the few next “high resource” languages. Important to note, even OpenAI claims support only for some 30-odd languages, all of them by definition not low resource. Anyway, since when are we ready to take OpenAI’s word for anything? They’re less transparent than even the least transparent competitors, as we’ve shown in our work on LLM openness and reproducibility (Liesenfeld & Dingemanse 2024).

Putting on our linguists’ hat for a moment, what do we even know about GPT4 performance in those 30-odd languages? Not much more than what OpenAI itself has claimed, and what we see there doesn’t inspire confidence. OpenAI’s performance claims are based on the MMLU multiple choice benchmark, one of the least trustworthy benchmarks even according to experts. Worse, it is based on machine-translations of that already not-great benchmark. Even a quick look at the few examples given of this translation by OpenAI reveals oddities like the word bluish in English auto-translated into “Welsh” as glas-ish, a weird chimera of a Welsh root and an English suffix where there’s a perfectly serviceable Welsh word glasaidd. As I said, doesn’t inspire confidence. But hey if OpenAI tells us it does well (like in the bar exam) we can definitely believe it.