Finetuning L atin BERT for Word Sense Disambiguation on the Thesaurus Linguae Latinae (2022)

Abstract: The Thesaurus Linguae Latinae (TLL) is a comprehensive monolingual dictionary that records contextualized meanings and usages of Latin words in antique sources at an unprecedented scale. We created a new dataset based on a subset of sense representations in the TLL, with which we finetuned the Latin-BERT neural language model (Bamman and Burns, 2020) on a supervised Word Sense Disambiguation task. We observe that the contextualized BERT representations finetuned on TLL data score better than static embeddings used in a bidirectional LSTM classifier on the same dataset, and that our per-lemma BERT models achieve higher and more robust performance than reported by Bamman and Burns (2020) based on data from a bilingual Latin dictionary. We demonstrate the differences in sense organizational principles between these two lexical resources, and report about our dataset construction and improved evaluation methodology.

## introduction in the field of natural language processing (nlp), there is a growing amount of languages for which contextualized representation models are created. for latin, a pretrained bert model (cf. @xcite was published by @xcite , which they finetuned for four classical nlp tasks, among others for word sense disambiguation (wsd). wsd, an area of computational semantics, has been approached in nlp by several machine learning setups (for an overview cf. @xcite , and recent works (e.g. @xcite have also targeted the use of neural models and architectures in combination with lexical knowledge bases and encyclopaedic resources. wsd is typically cast as supervised classification, where the learning task consists of predicting the appropriate sense label for one or more focus tokens in their context unit, e.g. within a sentence. based on the application end task, sense labels can be defined in a variety of ways, e.g. aiming to distinguish coarse or fine granularity of senses, binary or multiple sense distinctions, etc. creating labeled data for a supervised wsd application is nontrivial. large, sense-annotated benchmark datasets are scarce, especially in languages other than english. a promising resource to be utilized for latin wsd could be the latin wordnet foot_0 ; for its evaluation and references cf. @xcite . seeking proxy resources and methods to leverage wsd resources is important, since it is expensive to manually produce a sense labeled corpus from scratch that captures contextual information for several senses of a word. therefore, our study aims to contribute insights into methods that use dictionaries for automatically assigning sense labels. bamman and burns (2020) (henceforth: b&b) constructed wsd data for bert, a transformerbased language model, by taking the textual examples (i.e., quotes from antique sources) inventorized for a particular headword (aka 'lemma') in the bilingual latin dictionary of lewis and short (1879): to each quote snippet in the first two sense groups of each lemma, they assigned its sense category (i.e., i or ii) as gold standard label. inspired by this sense inventory creation (i.e., bi-nary class labeling) method of b&b, we requested data for the same lemmas that b&b presented, from a currently proprietary resource: the thesaurus linguae latinae foot_1 (tll) foot_2 . the tll is a comprehensive monolingual latin dictionary that aims to record all meanings of all ancient latin words, citing all (or a representative sample) of its seen attestations. the tll is vast: it is estimated to comprise cca. 53k-56k entries as of now foot_3 , so it likely holds a major part of the quotes that occur in l&s and thus in the b&b wsd dataset. the prospect of comparing wsd performance across datasets constructed from two dictionaries -one bilingual, another monolingual -was intriguing in several scholarly respects, a.o. for gaining quantitative insights into dictionary structuring practices, or even for attempting to validate sense structuring in an empirical way. after inspecting the data, we realized that a direct comparison of machine learning performance based on data constructed from the tll resp. from l&s would be methodologically flawed: our aims in the current contribution were thus: the paper is structured as follows: first, a short exploratory analysis is given for the b&b resp. the tll data in terms of the original resources and their construction principles. afterwards we report on the finetuning experiments and we summarize the study with a conclusion section. 2 exploratory data analysis ## wsd experiments finetuning bert is a technique that takes its pretrained language model and explicitly trains it for the wsd task, i.e., in our case on latin quote -typically on the subsentential level -that are labeled to have class i or class ii, as assigned based on the tll sense inventory. this yields a classification model that can distinguish exactly two meanings for the token that designates the focus lemma. this is certainly a simplified wsd setup, nevertheless helpful for pilot studies to assess the power of newly constructed data for disambiguating between two major senses (or usage contexts) of words. the finetuning task is in contrast with what already took place in the first phase of creating lexical representations, the so-called pretraining. there the task was that bert's latin language model learns as many senses of a word as possible. ## summary and conclusion our study aimed to confirm the impact of latin bert @xcite and to point out an important new latin wsd resource. we constructed a large dataset from the tll that holds quotes labeled with the first two highest-level senses of a headword. these likely incorporate senses that the b&b dataset did not include. we experimentally validated that the nested dictionary structure of the tll is able to deliver wsd data for finetuning contextual representations in a transformer architecture. the wsd models yielded a large improvement above the static embeddings baseline, when evaluated on held-out data from our new, tll-based dataset. we plan to scale up this study and to release a benchmark dataset and trained models for latin wsd in future work.