An Annotation Protocol for Diachronic Evaluation of Semantic Drift in Disability Sources (2025)

Abstract: Annotating terms referring to aspects of disability in historical texts is crucial for understanding how societies in different periods conceptualized and treated disability. Such annotations help modern readers grasp the evolving language, cultural attitudes, and social structures surrounding disability, shedding light on both marginalization and inclusion throughout history. This is important as evolving societal attitudes can influence the perpetuation of harmful language that reinforces stereotypes and discrimination. However, this task presents significant challenges. Terminology often reflects outdated, offensive, or ambiguous concepts that require sensitive interpretation. Meaning of terms may have shifted over time, making it difficult to align historical terms with contemporary understandings of disability. Additionally, contextual nuances and the lack of standardized language in historical records demand careful scholarly judgment to avoid anachronism or misrepresentation.

## introduction language constantly evolves and adapts to speakers' communicative needs and socio-cultural changes; understanding these shifts is crucial for grasping the dynamic nature of language and its intricate relationship with social and cultural phenomena. the semantics of words of a language shift due to influences from social practices, events, and political circumstances @xcite @xcite . the functioning and disability of individuals, 1 such as 1 who disability classification standards. in natural language processing (nlp), the task of semantic shift detection (ssd) focuses on detecting, interpreting, and assessing potential changes in the meaning of words over time @xcite . the international workshops on semantic evaluation (semeval) @xcite and ever evolving nlp (evonlp foot_0 ) have proposed various tasks and models. in the semantic web, ontology evolution @xcite studies how and why ontologies and knowledge graphs change over time; various works have proposed models based on heuristics @xcite and machine learning models for semantic change in biomedicine @xcite and generalised domains @xcite , with some studies looking into the impact of seman-tic change on reasoning and hierarchies @xcite . as explained in previous works @xcite , changes in language semantics over time can influence what is considered offensive. however, to the best of our knowledge no existing work facilitates resources for semantic change over large time spans (as these changes can be slow), considering both textual and semantic representations, and addressing discriminatory and harmful language in disability. in this paper, we propose an annotation protocol for the analysis and evaluation of semantic change in the disability domain, which is built on two rounds of iteration. our approach involves designing an annotation framework to capture both the descriptive and offensive nuances of historically relevant disability-related terms, accounting for their evolving connotations across different historical and social contexts. this includes structured guidelines for annotators to assess the perceived offensiveness, descriptive intent, and type of disability referenced in each instance. we present the quantitative and qualitative analyses on annotation disagreement that highlight the importance of capturing the nuanced and subjective nature of disability-related discourse, and discuss the four main challenges in annotating disability-related discourse over time. the annotation data and guidelines have been made available 3 to promote further research in this direction. ## background and related work there are several previous studies directed towards the evolution of disability terminology across various mediums, including media representations, scholarly publications, and broader social discourse @xcite @xcite . importantly, these studies show the changing landscape of disability discourse, its impact on societal perceptions and attitudes, and the dynamic nature of language and its role in shaping perceptions of disability within diverse contexts @xcite . a number of research projects have addressed the issues of bias and representation in historical texts, developing several resources that focus on the language and portrayal of disability @xcite ; de-bias project consortium, 2025). these initiatives aim to highlight and mitigate the marginalization of disabled individuals in historical records by providing analytical frameworks and lexical resources that bring attention to the social and cultural contexts in which disability-related terms were used in the past and how they should be used today. within the research area of semantic shift detection, benchmark datasets and text corpora capable of supporting the analysis of word meaning change over time have been developed @xcite for an overview and @xcite for a discussion of this task in the context of semantic change research). the semeval 2020 dataset @xcite ) contains a multilingual set of annotated sentences from english, german, latin, and swedish historical texts; other gold standard datasets exist @xcite . these datasets were all annotated by human experts, which ensures a high level of accuracy and contextual understanding, particularly important when dealing with nuanced and historically contingent language, but it is also a time-consuming and labor-intensive process. @xcite present a dataset of historical british newspapers from the 19th century where the contexts of a number of terms related to vehicles were annotated with their meaning via voluntary crowdsourcing, leveraging the scalable, collective effort of non-expert contributors. while existing annotated datasets from semantic change detection research constitute a promising avenue for studying semantic change and improving the understanding of historical language use, the existing resources solely utilize corpora amassed from general domains. as a result, they often overlook specialized areas such as disability discourse, where terminology carries distinct social and cultural significance that requires focused analysis. on the other hand, previous studies on the language of disabilities have not looked specifically at the challenges of corpus annotation in historical texts. our study addresses both these gaps by focussing on an annotation protocol specifically tailored to the annotation of disability terms whose semantics has changed in historical texts. in addition to the semantic change literature, our work also intersects with annotation challenges explored in socially sensitive domains. similar challenges have been discussed in the hate speech detection literature, where offensiveness and inflammatory intent often vary by context, speaker identity, and target community @xcite . recent work has introduced graded offensiveness scales, soft-labeling approaches, and community-informed annotation schemes to better reflect the subjective and socially contingent nature of such language @xcite . our annotation protocol draws on these developments by adopting a five-point offensiveness scale and encouraging annotators to consider both historical context and social intent when evaluating terms. ## data sources for designing the annotation protocol for measuring the semantic change in the disability domain, we selected texts for annotation from gale's history of disabilities: disabilities in society, seventeenth to twentieth century foot_1 , a collection of monographs, manuscripts, and ephemera documenting disability history (17th-20th centuries) through personal memoirs, accounts of care and rehabilitation, advocacy efforts, and policies impacting individuals with disabilities, thus examining society's evolving perceptions of disability. additionally, we collected an initial list of terms used to refer to disabilities from wikipedia foot_2 and the disability at stanford project. foot_3 ## annotation protocol the purpose of the annotation is to trace the evolution of selected terms related to disabilities over time in historical texts. we conducted two annotation rounds to assess the quality of the sources and refine the annotation protocol. the pilot round was carried out by a team of five annotators working in digital humanities and natural language processing and from career levels ranging from doctoral students to senior lecturers. the aim of this pilot was to assess the quality of the source texts for the annotation task at hand. the annotation protocol was built and refined based on the feedback given by participants in the pilot. in the first version of the protocol, each annotation line displayed a focus sentence with the disability term (one of the selected terms) in bold, along with the sentence before and after it for context. annotators were tasked to choose from a drop-down menu whether the term was 'derogatory', 'not derogatory', 'not referring to a disability', or 'unclear due to illegible ocr'-a necessary option given the limitations of historical documents. if the term did refer to a disability, annotators also indicated whether it referred to a 'mental' or 'physical' disability. this distinction was important for understanding how different types of impairments were perceived and treated historically, as societal attitudes and institutional responses often varied between mental and physical disabilities. feedback from the pilot annotation round revealed several important insights and challenges that guided the updates to the following round of the protocol. annotators noted, for example, that demented often appeared in medical texts to classify individuals deemed "mentally insane" by historical standards. though medically framed at the time, the term would now be seen as stigmatizing. similarly, downie was sometimes used as a personal name rather than a reference to down syndrome, and in certain cases, it appeared in affectionate or familiar contexts-underscoring the importance of contextual interpretation. the term cripple also prompted discussion among annotators. while it was sometimes used descriptively in medical contexts, it often appeared in passages reflecting harsh or dehumanizing attitudes. these examples highlighted the limitations of a binary classification (derogatory vs. not derogatory), which could not capture the nuance of tone and intent. annotators also found the mental vs. physical distinction for disability types too narrow, noting that many instances involved cognitive or sensory disabilities (e.g., blindness, deafness) that fell outside these categories. based on this feedback from the pilot, we modified the protocol to better account for the historical and contextual subtleties encountered in the data. again, each annotation line presents a focus sentence with the disability term highlighted, preceded by the sentence before it and the sentence after. the annotation consists now in choosing from the dropdown menu the best category to which the term can be assigned according to the following dimensions. the first decision annotators make is to determine whether the term is used as part of a 'formal diagnosis' or within 'common language'. this distinction helps clarify whether the term is functioning within an institutionalized medical discourse or in more casual, everyday speech. next, annotators assess whether the term is used with a 'descriptive' or 'offensive' intent. to capture varying degrees of offensiveness and contextual appropriateness, we implemented a graded scale, allowing annotators to position the term along a five-point scale: this graded scale was introduced to replace the earlier binary classification of 'derogatory' vs. 'not derogatory', which proved inadequate in capturing the nuances of language and intent found in historical texts. with a more granular approach we acknowledge that offensiveness exists on a spectrum and is deeply influenced by context, authorial intent, and audience perception-particularly in diachronic corpora. further, if the term in context refers to a disability, annotators are asked to mark the 'type of disability' it pertains to. annotators can select from cognitive, sensory, and/or physical categories. this refinement allows us to better track how different forms of disability were represented and discussed over time, and how terminology may have shifted in relation to different kinds of impairments. finally, in an optional comment field, annotators can explain their decision or provide additional observations. these qualitative notes are crucial for later analysis of annotation disagreements and for understanding the reasoning processes behind individual annotations. ## annotation process in the pilot annotation round, we examined four terms (henceforth referred to as "keywords"): abnormal, cripple, demented, and downie. these were chosen for their historical relevance to disability and their shifting meanings and acceptability over time. the selection balanced terms referring to physical disabilities (cripple, downie) and cognitive or mental ones (abnormal, demented) to explore varied linguistic representations. abnormal, derived from latin abnormis ("irregular"), was commonly used in 19th-and early 20thcentury clinical texts to describe physical or mental deviations from a perceived norm. though often descriptive, the term has accumulated negative connotations, reinforcing ideas of deviance and stigma. cripple once served as a general descriptor for individuals with physical disabilities, especially mobility impairments. while historically common in both medical and everyday language, it is now widely viewed as offensive due to its reductive and dehumanizing implications. some activists have attempted to reclaim the term in recent years to subvert its derogatory implications (wikipedia contributors, 2025). demented, from latin demens ("out of one's mind"), was used in medical contexts to describe cognitive and psychiatric impairments. though originally clinical, it has since acquired derogatory connotations and is often used pejoratively in modern speech. downie, a colloquial term sometimes aimed at individuals with down syndrome, appeared in both derogatory and affectionate contexts. however, its frequent use as a personal surname made annotation difficult due to ambiguity and low interannotator agreement. in the first round of annotation, for each keyword, we selected three textual excerpts from monographs and one from manuscripts through advanced search throughout the gale's history of disabilities collection (as described in §3). this approach aimed to capture both institutional and personal uses of the terms while accounting for sources' distributions. in the subsequent annotation round, we excluded downie from the dataset due to its ambiguity. most occurrences were personal surnames unrelated to disability, resulting in non-relevant instances and inconsistent annotator agreement. additionally, the limited context in some documents made it difficult to determine whether the term was used derogatorily or descriptively. as a result, we selected the word blind for further analysis. the term blind has a long history, originating from old english meaning "sightless" or "obscured" (oxford university press, 2024a). historically, blind was commonly used to describe individuals with significant visual impairments. although originally a neutral descrip-tor, modern disability discourse has raised concerns about its use, particularly in metaphorical contexts where it can perpetuate negative stereotypes (e.g., "blind to the truth"). in disability advocacy, there is increasing emphasis on person-first language (e.g., "person who is blind") or identity-first language (e.g., "blind person"), depending on individual and community preferences. for this second round, we aimed to curate a larger annotation corpus for a more detailed analysis. for each of the four keywords, we first identified 15 monographs and 10 manuscripts from the collection through advanced keyword search. from these, a list of 40 sentences were randomly selected for each keyword (along with the previous and next sentences for context), resulting in a curated annotation corpus of 120 textual excerpts in total. the annotation workshop comprised 12 annotators from research teams within the authors' university. one annotator had a background in linguistics and all others had background in computer science. the levels of experience ranged from early career researchers (doctoral students, postdocs) to senior lecturers. during the workshop, participants were first introduced to the annotation protocol and guidelines. then, they worked in small groups of three to annotate the selected sentences along the dimensions discussed in §4 following a structured approach foot_4 . ## analysis of annotations in this section, we analyse the results of the annotation process described in §5. specifically, we present a quantitative analysis regarding annotators' agreement in §6.1. in addition, we present a qualitative analysis discussing the challenges and some of the interesting cases that were observed during the annotation process in §6.2 foot_5 . ## observations and conclusions the annotation disagreements described in §6 reflect the inherent subjectivity in interpreting historical texts that contain socially charged language. annotators brought divergent perspectives on the historical role of terminology, the socio-political context of the sentences, and the contemporary implications of stigmatizing language. these divergences align with observations in prior research that annotation of socio-psychological constructs often entails subjective and multidimensional judgments @xcite . the annotation guidelines provided to annotators did not fully account for these interpretive differences. future annotation tasks involving socially sensitive language would benefit from clearer operational definitions, explicit guidance on balancing historical and modern interpretations, and perhaps more granular label schemes. another key challenge, also noted in hate speech annotation literature, is the variation in perceived offensiveness based on the background of the annotators and their relationship to the communities referenced in the texts @xcite . this is especially relevant for disability discourse, where community preferences around person-first versus identity-first language and perceptions of terms as outdated or offensive can differ widely. while our annotators were trained to reflect on historical and social context, future annotation efforts would benefit from including individuals with lived experience of disability or from adopting participatory annotation approaches that foreground community perspectives. additionally, methods that embrace annotation disagreement such as soft labeling @xcite may better reflect the inherent subjectivity of such tasks than traditional majority vote approaches. other annotation disagreement challenges, such as different readings of a sentence's tone, remain outside the capabilities of textual representations and we consider them much harder to address through annotation protocols alone. the findings from this analysis suggest several implications for the development of annotation schemes in the context of socio-political constructs and sensitive domains such as disability discourse. first, annotation tasks involving sociopsychological or politically charged constructs should acknowledge that disagreements are not necessarily indicative of noise, but may instead reflect valid differences in perspective that offer richer interpretive possibilities @xcite . second, annotation protocols might benefit from incorporating structured reflection or justification fields, prompting annotators to explicitly state the reasoning behind their choices. finally, our study highlights the need for methodological innovations in annotation aggregation. majority voting may obscure valuable minority perspectives that offer critical insights into the data. alternative approaches such as adjudication by discussion or perspectivist approaches @xcite may be better suited to capturing the complexities inherent in the annotation of multidimensional socio-linguistic phenomena. our analysis shows the deeply subjective nature of such annotation tasks. where social and ethical considerations intersect with linguistic analysis, disagreements may be inevitable and even desirable, provided they are systematically analysed and leveraged.