Exploring Supervised Approaches to the Detection of Anthropomorphic Language in the Reporting of NLP Venues (2025)

Abstract: We investigate the prevalence of anthropomorphic language in the reporting of AI technology, focussed on NLP and LLMs. We undertake a corpus annotation focussing on one year of ACL long-paper abstracts and news articles from the same period. We find that 74% of ACL abstracts and 88% of news articles contain some form of anthropomorphic description of AI technology. Further, we train a regression classifier based on BERT, demonstrating that we can automatically label abstracts for their degree of anthropomorphism based on our corpus. We conclude by applying this labelling process to abstracts available in the entire history of the ACL Anthology and reporting on diachronic and inter-venue findings, showing that the degree of anthropomorphism is increasing at all examined venues over time.

## introduction in an age of ubiquitous ai agents powered by nlp technologies, science communicators must be careful in their choice of words. overly technical language can stymie the readability, citability and impact of scientific endeavour. in addition to this, authors are regularly instructed by funders and editors to write for non-technical audiences (typically as a lay summary in addition to a technical abstract). in an effort to improve communication authors turn to more familiar language and in particular to the rhetorical device of anthropomorphism, defined by merriam webster 1 as : an interpretation of what is not human or personal in terms of human or personal characteristics 1 https://www.merriam-webster.com/dictionary/ anthropomorphism anthropomorphism is a useful literary tool that uses known human experiences as characteristic metaphors, with many terms such as 'machine learning' or 'chain-of-thought' being widely accepted beyond academic borders. within the field of nlp, common anthropomorphisms are generally understood to refer to the technical contributions they indicate. however, they also give potential for incorrect assumptions to be made about the capacities of llms. recent criticism of anthropomorphised reporting has highlighted the responsibility of ai practitioners to faithfully describe their techniques @xcite . take for example, the following sentences extracted from an online news source (1) and an acl main conference paper (2) in 2022 (emphasis from the author, sources available on request): note that in both these examples there is some inanimate agent upon which human capability is inferred: a robot capable of deceit in (1) and the reinforcement learning algorithm capable of inquiry in (2). the use of anthropomorphism has the potential to misrepresent the capabilities of nlp technologies and in this work we seek to better understand this phenomenon. we examine the use of anthropomorphisms in the nlp literature and in news text. to achieve this goal, we present a new corpus 2 of annotated anthropomorphisms from scientific literature and news sources (section 3), giving insights on the types of anthropomorphisms present. we further train a model to predict the degree of anthropomorphism in a scientific abstract (section 4.1) and apply this model to the acl anthology (section 4.2). we report our results and provide discussion points in section 5. ## related work several prior efforts have sought to identify examples of texts featuring the anthropomorphisation of ai systems @xcite @xcite . @xcite argues that the term artificial intelligence is itself an anthropomorphisation, indicating that the agent possessing the inferred quality of 'ai' has attained a human characteristic. work to better quantify cases of ai anthropomorphisation has led to categorisations of phrases referring to ai into ambiguous and explicit anthropomorphism @xcite depending on the authors intent. later work to measure the degree to which anthropomorphisation is prevalent focussed on unsupervised automated scoring of model descriptions identifying whether a llm was more likely to replace the name of a system with 'it' (indicating non-anthropomorphic language) or 'he/she' (indicating anthropomorphised language) @xcite . anthropomorphic language in ai is not limited to the models, but is also applied to the tasks they complete, such as 'reading comprehension' or 'sentiment analysis' @xcite . anthropomorphised language is often a factor in the misrepresentation of ai abilities @xcite . misrepresentation leads to misunderstanding and misapplication of ai tools which leads to confusion amongst ai scholars, developers and the general public @xcite . further, in a recent study, @xcite analysed user trust when interacting with anthropomorphised and deanthropomorphised descriptions of ai systems, finding that the presence of anthropomorphic terminology alone did not influence user trust. various audiences who may produce and/or consume anthropomorphised descriptions of ai systems have been considered in the literature. firstly, we may consider scientists in the nlp and ai community, who are actively working on the latest models and have typically been engaged with the technology for a number of years prior to the recent increase in ai technologies. these scholars are prone to ai anthropomorphisation with a recent study showing that 32 out of 81 examined papers (39.5%) concerning language modelling technology exhibited some form of anthropomorphisation in the abstract @xcite . secondly, journalists reporting on ai for the general public are also responsible for anthropomorphisation with a growing body of evidence to demonstrate that public news reporting is more anthropomorphic than science communication of the same topics @xcite . finally, the general public possess lay knowledge of ai systems and may prefer anthropomorphised descriptions in some cases @xcite . science communicators must work to ensure that descriptions are not harmful in misrepresenting the abilities of ai systems to the general public @xcite . prior to our work, no systematic annotation of anthropomorphism in the nlp literature has been undertaken. our work complements existing studies by providing a large-scale annotation of anthropomorphism and evidence-based nlp methodologies to longitudinally detect anthropomorphic language. ## corpus development in this section, we present the first ever annotated corpus of anthropomorphic language used to describe llms. we begin by describing the corpus collection and annotation procedure and then present an analysis of the annotations within the corpus. the acl anthology @xcite is freely available online in xml format foot_1 . we selected all abstracts of long papers presented at the 60th meeting of the association of computational linguists (acl 2022) from the anthology. this gave us 601 abstracts, comprising of 3,584 sentences. to contrast with scientific writing, we also selected news articles for annotation. to collect the news articles, we harvested public-facing rss feeds from bbc news, the new york times and the register for article titles containing keywords (artificial intelligence, ai, a.i., chatgpt, large language model, llm) related to llms. we collected articles for a two month period (may-june 2023) executing the searches every day and remov-ing duplicates. we obtained 49 articles that were suitable for annotation, extracting the plain text. ## anthology study in this section we train a classifier to predict s d for unseen abstracts using the acl abstracts in our annotated corpus. we subsequently apply this classifier to the entire acl anthology, reporting our findings. as our model is designed for acl abstracts, we do not consider claim detection as we previously noted (see section 3.1) that most abstracts are made up entirely of claims. instead we predict anthropomorphism score based on the entire abstract. ## discussion of results the analysis of anthropomorphism over time is presented in figure 2 , which shows bubble plots and trend lines for each venue on a separate subfigure. we have additionally included aggregated figures for the bubble plots and trend lines in appendix b. figure 2 shows the average predicted anthropomorphism score in each year for every venue. we do not have information on every venue for each year with most venues starting in 2016 and 2017. two notable exceptions to this are lrec and the 'other' category (covering workshops and smaller conferences) which start in 2004 and 1991 respectively. the y-axis in figure 2 shows that the average predicted anthropomorphism score is generally low (0.02-0.15). we found that most documents were predicted to have a low anthropomorphism score, which is reflective of the data distribution of the gold labels (reflecting the annotation of acl 2022). however, in most venues there existed a long tail of abstracts with higher than average anthropomorphism scores. the averages that are presented against the y-axis are reflective of an aggregation of all documents within a venue for that year. in light of this, we do not claim findings on the anthropomorphism levels of specific abstracts, but only on the aggregated values between years and venues. we also do not make claims about the raw average predicted anthropomorphism level of any one venue or year, but instead rely on comparing data points. we expect that even given some shift in scores due to model variability, the broad trends exposed by this research remain consistent. we observe in figure 2 that the degree of anthropomorphism in scientific writing as found in the acl anthology has increased in the period 1991-2023. we can further see that the trend towards increased anthropomorphism is present for every venue that we analysed. this is particularly clear in figure 4 , where each trend line is clearly increasing. figure 2 demonstrates some clear trends in terms of the predicted degree of anthropomorphism between venues. lrec is notably at a lower degree of anthropomorphism than other venues throughout the entire reporting period for which we have abstracts available. lrec typically publishes resource descriptions and evaluations, so the tendency to anthropomorphise these may be lower. acl, naacl and emnlp exhibit the highest average predicted anthropomorphism across the period. we note that for the period 2017-2023 (where we have data for all venues) the average predictions for the 'other' category is in between all analysed venues representing a mixture of the type of reporting found across the acl anthology. this helps to contextualise the earlier period where we only have data for 'other' and lrec, indicating that other is a generally reliable measure of the average anthropomorphism in a given year. ## conclusion we present the first large-scale annotated corpus of anthropomorphisms related to large language models. we use this corpus to demonstrate that anthropomorphism is present in both scientific abstracts and news articles. we have also developed a new model for predicting the degree of anthropomorphism in a scientific abstract and demonstrated that the use of anthropomorphism is a growing trend in the acl anthology. we expect to extend our corpus in future work, making use of our existing model to bootstrap future annotations and extending the data sources beyond the anthology. we will also investigate other tasks related to anthropomorphic language such as sentence classification and style transfer. we are not advocating for the abolition or prohibition of anthropomorphic terminology. there are clearly many valid cases where this language is widely accepted and understood to refer to underlying technological advances (indeed artificial intelligence is an anthropomorphism which has been in use since the 1950s). instead, we hope that this work provokes timely discussion amongst science communicators, both in the press and the nlp community. we must take stock of our own usage of anthropomorphised language to describe our technology and consider the impact of this on the public mind.