Sense-specific Historical Word Usage Generation (2025)

Abstract: Large-scale sense-annotated corpora are important for a range of tasks but are hard to come by. Dictionaries that record and describe the vocabulary of a language often offer a small set of real-world example sentences for each sense of a word. However, on their own, these sentences are too few to be used as diachronic sense-annotated corpora. We propose a targeted strategy for training and evaluating generative models producing historically and semantically accurate word usages given any word, sense definition, and year triple. Our results demonstrate that fine-tuned models can generate usages with the same properties as real-world example sentences from a reference dictionary. Thus the generated usages will be suitable for training and testing computational models where large-scale sense-annotated corpora are needed but currently unavailable.

## introduction language is essential to almost every aspect of human life and is often a crucial remnant of previous societies. it is through previous writings that we can learn how societies were built and have since evolved. but as our societies evolve, so does our language. great effort has been made to support historical linguistics, for example, by using computational approaches to model how words change their meaning over time, a phenomenon called semantic change. however, these efforts have been severely hindered by a lack of diachronic sense-annotated corpora, primarily because sense-annotation of historical texts requires significant time, expertise, and effort and is thus extremely costly @xcite . our contribution: we have brought together the generative abilities of large language models (llms) and the vast resources of a high-quality, large-scale dictionary to train and evaluate generative models using the real-world example sentences provided with each sense in the oxford english dictionary (oed). specifically, we: these resources open up the possibility of developing models for a range of different tasks, ranging from diachronic word sense disambiguation to semantic change detection. using generative models, we can produce sentences in which a word is used in a semantically and historically accurate way. given a word and a sense definition, these models can generate temporally accurate sentences for each specified time period. in we show that fine-tuned llms (1) can generate sentences that have the same properties as the original example sentences from the oed with a time error of (on average) 50 years; and (2) can be used to simulate synthetically large-scale linguistic phenomena such as semantic change, providing accurate and diverse historical word usages. we provide a suite of tools for testing (a) context variability, (b) temporal accuracy, and (c) semantic accuracy of generated historical usages. we have also released a new human-annotated dataset of semantic relatedness between generated historical sentences and sense definitions. 1 the annotations show empirical evidence of a correlation between year usage 1980 he put the phone back in the cradle and turned toward the kitchen. @xcite we made the telephone call from a public telephone booth that had phones, instruments and paper on top of each other in a jumble. 2000 i was carrying my cell phone so that i could hear the signals if there were any. 2010 you can buy a low-priced 3g phone (the phone's the thing, not the service) for less than £60. it's no phone you'd be proud to have on your desk, but it'll handle text messaging, e-mail and internet-browsing. the hierarchical organization of senses in the oed and human perceptions of the closeness between the word senses. ## motivation large diachronic sense-annotated corpora are needed to develop models for longitudinal studies. such corpora could be obtained through dictionaries that store implicitly sense-annotated word usages via example sentences attached to each sense. however, these usages are sparse, even in large dictionaries. for example, in the oed each word sense is accompanied by approximately four sentences, which are spread across an average of four distinct years and sometimes even fewer. for example, the corruption sense of the word graft appears only in a single year. when explicitly sense-annotated datasets are available, they are often small and synchronic @xcite . to simulate diachronic datasets, they can be divided into batches, but they then lack historically accurate language. to retain a historically accurate linguistic style, efforts have been made to derive sense information implicitly by clustering similarity judgments between pairs of usages of a word @xcite , as these are cheaper to produce than sense annotation. however, these are also small in scale and typically only cover two time periods. to train and evaluate models for longitudinal studies, for example, to detect semantic change over time, we need sense-specific and temporally changing word usages on a large scale, that is, with sufficient representation of each sense across different historical contexts. example sentences or small-scale sense-annotated data are insufficient for this purpose. however, using fine-tuned llms, we can generate the desired number of sentences that match the historical context and specific senses of words for training and test models. we can thus ensure that the generated output has a continuous and representative temporal distribution of any and all word senses across all (for the sense's valid) time periods. ## data in this work, we used the oxford english dictionary @xcite because it is the most comprehensive and authoritative record of the english language. it includes over 273,000 words with detailed etymologies and (frequency) bands. 2 for each word, there is a set of entries that contain a group of senses that are semantically related (see figure 1 ). multiple entries for a word reflect either different parts of speech or homonymic senses. for each sense, a set of example sentences offer the reader a sense of how a word can and has been used. a sense entry is introduced when there is evidence of the sense in writing, and a sense is considered rare or outdated when it is no longer in everyday use. we collected words, senses, and examples using the oed research api (oed api) from the year 1700 onward. the dataset was split into two distinct partitions, one for training and the other for testing. the estimation of word polysemy was based on the number of main senses reported in the oed, appearing on the second level or above in the hierarchy (in yellow in figure 1 ). the lemmas were classified into three categories: (1) those train test words lemmas 81,220 2,768 part-of-speech 11 4 senses senses 301,395 13,762 avg. definition length 128.01 146.70 avg. number of usages per sense 3.95 6.52 usages usages 1,191,851 89,759 avg. text length 91.22 90.29 with at least one new sense introduced after 1800, (2) those with a disappeared sense, and (3) all remaining lemmas. stratified sampling was used to ensure a uniform distribution across the various classes of word frequencies, polysemy, and introduction/extension of senses. the training and test data were entirely distinct from one another; the lemmas that appear in the test data and their respective senses were removed from the training data. ## generation of usages the primary focus of our work is a proof-ofconcept using an open source model, for which we chose llama. we fine-tuned the pretrained llama 3 8b model using the oed dataset, producing two versions: llama 3-janus and llama 3-janus-pos. 3 janus was trained without pos data, while janus (pos) incorporated pos tags. then we tested fine-tuned models against the instructed models. specifically, we used the instructed llama 3 8b instruct, and llama 3 70b instruct, consisting, respectively, of 8 and 70 billion parameters (ai@meta, 2024). we also undertook a comparison to gpt as a commercial tool, working with gpt-3.5 and gpt-4o as the most efficient and effective models provided by openai (openai). ## evaluating generated usages in this section, we present the evaluation of the usages generated by janus, janus (pos), llama 3 8b instruct, llama 3 70b instruct, gpt-3.5, gpt-4o, and few shot gpt-4o. this evaluation is based on the oed test set, as introduced in section 3. for each entry in the oed test set, we added a triple with a sampled year (w, d i , y) to t . using each model m, we then generated ten usages for all triples in t , resulting in u (w,d i ,y) containing a total of 137,620 generated usages for each model. for each m, we evaluated the set of generated usages u (w,d i ,y) focusing on: validation: to validate our results (and obtain a baseline for λ, τ and σ), we compared the generated usages with the original usages included in the oed test set. furthermore, we validated context variability and temporal accuracy, using a control dataset without sense annotations comprising usages for each (w, * , y) ∈ t . the control set thus consisted of historical word usages u (w, * ,y) extracted from the corpus of late modern english texts @xcite (1710-1920) and the clean corpus of historical american english @xcite (1800-2010). ## (exact sense) to lay sticks between (timber boards) in order to facilitate the circulation of air during seasoning. frequently with specifying the sharp-pointed object. to be reluctant or unwilling (to do something); to hesitate, to scruple. chiefly in negative constructions (e.g., he did not stick to) 3 (peer sense) the final score for each λ @xmath0, bleu, cosine, bscore} over the set of usages was calculated as the average of the individual usage pair (u, v) scores, given by 1 for the control dataset, which had no information on word sense, we aggregated at the lemma level, i.e., u (w, * ,y) . for the oed test set, which does not have enough usages for each year considered, we aggregated all years and evaluated the diversity of usages for the specific sense, i.e., u (w,d i , * ) . when comparing the control and the generated usages, we should therefore bear in mind that both validation datasets naturally exhibit a greater variety of contexts than the generated uses. the usages of the control dataset show a greater variety along the sense dimension, whereas the usages of the oed test set show a greater variety along the temporal dimension. ## application of generated usages we aim to verify whether automatically generated word usages can be used effectively for training and evaluating models in downstream applications. specifically, we choose the lexical semantic change detection (lscd), where the aim is to determine how much a word has changed over time based on usages from different time periods. the lscd pipeline typically involves, for each word, sampling of usages from two different time periods. next, these usages are compared pairwise (either from within each time period, or across periods) and assigned a score using the durel scale (1-4, for this study, we focus on the first step, the pairwise comparison of usages, commonly known as graded word-in-context (wic) task. we first evaluate using the standard english lscd dataset, the dwug en dataset @xcite , which consists of 46 words with annotated usage graphs with sentences sampled from time periods @xcite next, for the same 46 words, we take example sentences from the oed, wordnet @xcite and semcor @xcite and construct wic datasets. these are constructed such that example sentences from the same sense are labeled as 1 and from difference senses are labeled as 0. the example sentences from oed are taken from two historical periods @xcite , while wordnet+semcor have only modern sentences. using these datasets, we want @xcite to know if janus generated, synthetic usages can help improve model performance on the graded word-in-context task. to fine-tune the models, we generate janus usages for the 46 words of dwug en using definitions from both wordnet (wn) and the oxford english dictionary (oed). since wordnet is a synchronic dataset, we generate examples for the year 2000. for the oed, we generate usages corresponding to the years 1800 and 2000, as well as a combined set (oed all) that includes examples from both time periods. for each word-definition pair in these datasets, we generate 100 usages. for model training, we adopt a siamese encoder approach following @xcite . the target word is highlighted using special tokens (<@xmath1. each usage pair is encoded with roberta-large, and the centroid of the subword vectors is used as the final usage representation. the model is trained with a contrastive loss to ensure that vectors of word usages with the same meaning are closer in space, while those with different meanings are farther apart. the results indicate a preference for the generated sentences. these previous studies relied on a smaller subset of the oed dictionary without temporal information @xcite . our work goes beyond previous work by generating example sentences over long timespans while taking the temporal language into account. for this reason, we created a new dataset using the oed api that preserves the year from which the example sentences stem. our dataset differs from previous ones in both size (it is significantly larger) and oed version (using a later version). unfortunately, the absence of sense identifiers in the previous dataset makes direct comparison challenging. our work is the first to assess both temporal accuracy and context diversity-two fundamental aspects in lscd applications. furthermore, unlike previous studies, our evaluation of semantic accuracy extends beyond ensuring that the sentence aligns with the provided definition. we also focus on ensuring that the sentence is unambiguous and does not express other meanings. therefore, our method penalizes cases where the usage does not distinctly reflect the intended meaning. for example, the term meat used to refer broadly to food but has narrowed to mean flesh. the oed provides clear examples of this shift, such as ''first take all the meat out of the lobster,'' where meat clearly refers to flesh. however, in broader contexts like ''he had meat and drink enough,'' the meaning of meat becomes more ambiguous. by accounting for the word's other meanings, we make our evaluation more robust and better suited for testing the creation of semantic change datasets. resources for lscd while semantic change can be modeled on any temporal dataset, prior to 2020 evaluation relied on small datasets of words with attested meaning changes in lexicographic resources @xcite @xcite . semeval-2020 task 1 @xcite marks the first systematic approach to evaluation grounded in the texts themselves using the durel framework @xcite . despite this progress with follow-ups in several languages, two key limitations remain: (1) explicit word sense labels, and (2) longitudinal datasets covering multiple time periods. the former can be addressed by splitting synchronic sense-annotated corpora (schlechtweg and schulte im walde, 2020), but inherits the limitation of both the small size and the synchronic nature of the corpus. the latter is addressed using diachronic corpora and the introduction of synthetic changes by altering word frequencies or contexts using a replacement schema @xcite @xcite . although this produces a naturally changing linguistic style, the sentences are not natural (e.g., the chair was purring loudly). generative models for lscd when modeling semantic change, llms are usually used in one of two ways @xcite : (1) as computational annotators, where instruct-based models have been used to predict the annotation between sentence pairs @xcite @xcite ; or (2) to generate definitions for each usage of a word, either using prompting or fine-tuning @xcite . despite these advances, there is a notable gap in the literature regarding the ability of these models to augment data for historical texts. although llms have access to vast amounts of data, historical data is underrepresented compared to modern data, as the majority of their training data is sourced from the web. this paper shows that llms specifically fine-tuned to model the temporal dimension can generate text that is temporally accurate to a substantial degree. ## conclusion in this paper, we have demonstrated significant progress in the generation of sense and time-specific text using llms. we have shown that llama, when fine-tuned, can generate historically accurate text and that higher temperature settings improve stylistic authenticity at the expense of semantic accuracy-a consequence of the long-tail distribution problem. we have successfully developed a model capable of generating historical texts with properties that closely mirror example sentences found in the oed. qualitative investigation of the generated sentences shows that in the context of a target word w, the model uses temporally accurate clues such as word choices (e.g., thee, thou, you); and correct senses of the context words (e.g., gay used as happy, awful as impressive). our contributions include the development of a time classifier, a sense classifier, and a robust methodology that can be utilized in future research and adapted for new models of historical text generation. additionally, we provide a human-annotated dataset of word usages paired with the corresponding sense definitions, a valuable resource for further studies. our empirical evidence confirms a strong alignment between the durel scale and the hierarchical structure of the oed, reinforcing the validity of our approach. janus opens new avenues for the study of lexical semantic change by enabling model training and testing with accurate historical sense-annotated data over extensive time periods, where previously, no such data existed. our findings demonstrate that the temporal dimension influences both model performance and annotation outcomes. future research should consider these factors by more thoroughly investigating the role of temperature settings in models and by employing experienced annotators. the ambiguity of the oed labels, influenced by the lumpers and splitters issue, presents intriguing avenues for research. moreover, while our work currently focuses on english, there is potential to extend it to other languages by testing janus's zero-shot capabilities or by fine-tuning on additional languages.