Non-Parametric Word Sense Disambiguation for Historical Languages (2022)

Abstract: Recent approaches to Word Sense Disambiguation (WSD) have profited from the enhanced contextualized word representations coming from contemporary Large Language Models (LLMs). This advancement is accompanied by a renewed interest in WSD applications in Humanities research, where the lack of suitable, specific WSD-annotated resources is a hurdle in developing ad-hoc WSD systems. Because they can exploit sentential context, LLMs are particularly suited for disambiguation tasks. Still, the application of LLMs is often limited to linear classifiers trained on top of the LLM architecture. In this paper, we follow recent developments in non-parametric learning and show how LLMs can be efficiently fine-tuned to achieve strong few-shot performance on WSD for historical languages (English and Dutch, date range: 1450-1950). We test our hypothesis using (i) a large, general evaluation set taken from large lexical databases, and (ii) a small real-world scenario involving an ad-hoc WSD task. Moreover, this paper marks the release of GysBERT, a LLM for historical Dutch.

## introduction & related work a common task in natural language processing (nlp) applications is the disambiguation of a particular target word in a given context. due to a variety of reasons (see e.g. @xcite , word forms may be semantically extended to a range of different meanings or word senses (e.g. rat 'animal' > 'informer, snitch'). automated disambiguationi.e. the mapping of an ambiguous word form to its intended underlying word sense-is a task that can help in many different types of information extraction and text mining tasks. in recent years, wsd approaches have shifted towards contextualized embeddings extracted from large language models (llm) like bert @xcite . these token-based embeddings incorporate substantial semantic information from the target lexical item and the sentential context that surrounds it. for this reason, llms are particularly well suited to disambiguation tasks and have, in fact, already been shown to indirectly capture word senses and perform competitively @xcite . more recently, llms also obtained state-of-the-art performance using semantic networks @xcite and gloss information @xcite @xcite . these recent advancements in wsd by means of llms are also interesting for humanities scholars, as large-scale corpus-based studies have steadily become more standard practice over the last decades. traditionally, data annotation in humanities research is done manually, but with evergrowing size of corpora, such manual wsd has become decreasingly feasible. at the same time, researchers are also increasingly interested in disambiguating word senses without relying on intuitive judgment, which is ultimately subjective. this desire for a more data-driven approach to wsd is particularly prominent in research involving historical language, for which researchers are unable to solicit native speaker interpretations and expert annotators are rare. there is, in short, a growing interest in and need for automated wsd in humanities research, and researchers have turned to nlp techniques to meet their needs. as examples, we find projects aiming to trace the history of concepts over time (e.g. @xcite , automatically detect (different types of) lexical-semantic (e.g. @xcite and grammatical change (e.g. @xcite or to quantify the evolution of the senses of a word (e.g. @xcite . because of their temporal dimension, such projects require nlp architectures that can process word senses in settings which are complicated by an evolving grammar and lexicon @xcite , shifting spelling conventions, and noise introduced by ocr errors @xcite . yet, applications of llms on historical corpora tend to be either limited to arithmetic operations on the vector space of frozen contextual embeddings, or restricted to fine-tuning a linear layer to perform disambiguation to a pre-determined number of senses @xcite @xcite . in this paper, we explore the capabilities of large language models (llms) á la bert @xcite for wsd on historical text. in doing so, we investigate to what extent (i) fine-tuning can improve wsd over arithmetic operations on plain frozen embeddings, and (ii) how much annotated data is needed in order to obtain gains over a baseline. our focus lies on examining the data efficiency of llm-based approaches for wsd, because annotated resources for wsd are generally scarce and costly to generate -a problem that is exacerbated with historical languages, where rare expert historical knowledge is required to produce annotated resources. to attain these goals, we take inspiration from recent metric-based non-parametric approaches @xcite @xcite . these aim to optimize a model on a set of learning tasks (e.g. the disambiguation of a given ambiguous word) so that the model can quickly adapt to perform well on similar future tasks (e.g. disambiguating sentences of an ambiguous new word on the basis of a small annotated set of word senses). more specifically, we deploy a non-parametric approach to wsd fine-tuning that does not rely on additional task-specific parameters and that achieves surprisingly strong performance on out-of-domain lemmas. we argue that these results are promising if we aim to extend the scope of applications of wsd models in the humanities. ## method 2.1 architecture the present approach deploys a parameter-free architecture which is heavily inspired by both the matching networks @xcite and the prototypical networks @xcite frameworks. in this non-parametric approach, we fine-tune a given llm using episodic training. in this type of training, each batch constitutes a training episode which is designed in order to match the experimental conditions expected at inference time. in the case of wsd, each episode consists of a number of randomly sampled sentences-a 'support set'-, exemplifying different word senses of a given lemma, as well as a second set of randomly sampled sentences-a 'query set'-, for which a word sense prediction needs to be made. 2 sentences in the query set are used in order to obtain a contextualized word embedding for the target word (i.e. the word representing the lemma to be disambiguated). the support set is used in order to compute abstract word sense representations for each of the word senses a lemma may have. 3 these abstract sense embeddings can be computed in multiple ways, but for simplicity we have chosen a centroid approach. first, the contextualized embeddings of the target lemma in the support sentences are extracted, and, then, averaged in order to get a single representation per word sense. 4 more formally, let e(s i ) and e(q j ) denote the contextualized embeddings of the target lemma in the i th support sentence and the j th query sentence in the current training episode. and let r(s j ) denote the word sense of the j th sentence in the support set. then, the representation for the k th word sense r k in a given training episode is computed as follows: the objective of the approach is to maximize the probability of the true word sense given by the following equation. ## datasets the datasets underlying the present study come from the oxford english dictionary (henceforth: oed oxford university press) and the "woordenboek der nederlandsche taal" (dictionary of the dutch language, henceforth: wnt instituut voor de nederlandse taal). both resources consist of large historical lexicons, where each lemma is categorized into a hierarchy of word senses. each word sense is given a definition, and is exemplified by a set of sentences spanning a certain time window. to construct a suitable corpus for testing our wsd approaches, we sampled 1,000 words according to frequency from each language foot_3 and searched for them in the corresponding resource. the collected data for each language is described in on this data set, we produced a 10% split of lemmas, which were used to evaluate models on unseen lemmas. we refer to this as the "held-out set". finally, for each lemma, we produced a 50% split of quotations, following the original distribution of word senses.