Predicate Sense Disambiguation for UMR Annotation of L atin: Challenges and Insights (2024)

Abstract: This paper explores the possibility to exploit different Pretrained Language Models (PLMs) to assist in a manual annotation task consisting in assigning the appropriate sense to verbal predicates in a Latin text. Indeed, this represents a crucial step when annotating data according to the Uniform Meaning Representation (UMR) framework, designed to annotate the semantic content of a text in a cross-linguistic perspective. We approach the study as a Word Sense Disambiguation task, with the primary goal of assessing the feasibility of leveraging available resources for Latin to streamline the labor-intensive annotation process. Our methodology revolves around the exploitation of contextual embeddings to compute token similarity, under the assumption that predicates sharing a similar sense would also share their context of occurrence. We discuss our findings, emphasizing applicability and limitations of this approach in the context of Latin, for which the limited amount of available resources poses additional challenges.

## introduction word sense disambiguation (wsd), i.e. the task of identifying the correct sense of a word in a specific instance or sentence, poses non-trivial challenges especially in the context of languages where resources are relatively scarce. this is the case of latin, whose few existing resources confront the inherent complexity of the task and often resort to a binary approach revolving around the assumption that the several senses of a word can be reduced to two primary senses. this inevitably leads to resources that are overly coarse-grained. while such simplifications serve as valuable starting points for future experiments, their granularity may not universally cater to the diverse research needs. the present work originates from the needs of a distinct project, which focuses on the annotation of latin data according to the uniform meaning representation framework (umr) (van gysel et al., 2021). the text to be annotated is de coniuratione catilinae 'conspiracy of catiline' by sallust. the umr framework is designed to annotate the semantic content of a text, and was developed with cross-linguistic scope in mind. it is primarily based on abstract meaning representation (amr) (banarescu et al., 2013), and aims at extending it to other languages -in particular to morphologically complex, possibly low-resource languages -in a cross-lingual and typological perspective. in amr and umr graphs, nodes represent semantic concepts. if word senses are available, semantic concepts are defined as word senses; participant roles associated to each predicate (e.g., arg0, arg1) are included in the graph if realized in the sentence. for instance, the predicate utimur in the sentence corporis servitio magis utimur 'of the body we rather employ the service' corresponds to the semantic concept utor-03, i.e. the sense "put into service; make work or employ for a particular purpose or for its inherent or natural purpose" to which arg0 (first person plural, not overtly realized) and arg1 (servitio) are associated. within the whole annotation process, manual selection of the correct sense constitutes a time-consuming and demanding sub-task. we thus aim to investigate whether the existing resources allow to develop a strategy to expedite this process, by deriving annotation suggestions for unannotated predicates based on already manually annotated ones. the paper is structured as follows. section 2 presents an overview of related work, while section 3 discusses latin vallex as the main linguistic resource that has been exploited, as well as the limitations it presents. section 4 describes the methodology designed for the task, while its outcomes are evaluated in section 5. section 6 highlights some conclusive remarks and possible future research directions. ## related work the exploration of wsd tasks for classical languages, and notably latin, has recently gained attention, especially from a diachronic perspective with regard to lexical semantic change (beelen et al., 2021; mcgillivray, 2021; mcgillivray et al., 2022, 2023a; marongiu and mcgillivray, 2023). however, the granularity of available resources remains a significant obstacle to successful wsd, as discussed by navigli (2006) and mcgillivray et al. (2023b). in the context of introducing the latin bert model, bamman and burns (2020) discuss a wsd task framed as a binary classification task, where only the first two major senses are selected for each headword and, thus, the sense to be predicted has to be chosen out of two possible candidates only. building on their work, lendvai and wick (2022) create a new dataset based on a subset of sense representations from the thesaurus linguae latinae, 1 and use it to fine-tune latin bert on a supervised wsd task. despite achieving more robust performances, the task remains configured as binary classification, retaining only the first two sense groups for each lemma. pivoting a low-resource language to a highresource one via parallel corpora has been observed to be a valid strategy to obtain wsd annotations in the under-resourced language (pasini et al., 2021). as the issue of data scarcity applies to latin as well, ghinassi et al. (2024) extend such approach to historical languages, leveraging parallel corpora to pivot latin to english. propagating wsd annotations from english to latin then helps tackle the challenge represented by the lack of large sense annotated corpora. the need for automated wsd has been observed, particularly for historical languages, in light of the increasing size of corpora to annotate and of the subjectivity involved in the intuitive judgment required by sense disambiguation, even more so when native speakers cannot be exploited, as noted by manjavacas arevalo and fonteyn (2022). however, efforts to expedite the annotation process do represent a more general need. for instance, in the context of expanding an event-type ontology straková et al. (2023) try to exploit finetuned llms to generate annotation suggestions that could expedite the manual annotation process of verbs to be included in the ontology. despite not working with a historical language -as their focus 1 https://tll.degruyter.com/about . furthermore, scarlini et al. (2020) experiment with developing a semi-supervised approach foot_0 to obtain sense embeddings for lexical meanings within a lexical knowledge base like wordnet. although their approach does not include latin and thus cannot be leveraged in our work, it interestingly builds upon the semantic information already carried by contextual word embeddings. in general -as it provides a comprehensive lexical inventory for the identification of the different word senses -wordnet is a crucial resource for wsd. the current latin wordnet foot_1 (wn) (franzini et al. 2019; mambrini et al. 2021) is the outcome of an ongoing and substantial revision of the original latinwordnet (minozzi, 2010) as initiated within the multiwordnet project (pianta et al., 2002). in wordnet, diverse senses of a polysemic word are assigned to distinct synsets. within the lila knowledge base (passarotti et al., 2020), these wn synsets are mapped with valency frames of the valency lexicon latin vallex foot_2 , thanks to the shared lexical entries between the two resources. as a result, the latin vallex contains not only valency frames but also synset definitions associated to them. ## in between latin vallex and wordnet let us delve deeper into the examination of the linguistic resources exploited, and notably latin vallex. @xcite nonetheless, speaking of vallex implies speaking of wordnet as well, as the two resources are interlinked in lila (section 2). for each lemma, vallex contains information about the synset definition (taken from wordnet) and the valency frame associated to it. a closer look at the entries immediately reveals how some synsets are semantically close. in many cases, their strikingly similar definitions are not justified by diverging valency frames. among the many examples, two senses of porto, both with frame act (actor), pat (patient), are defined respectively as definition synset_id have on one's person v#00047745 have with oneself; have on one's person v#02717102 three very similar entries are associated to augeo, all with the same valency frame act, pat: definition synset_id make strong or stronger v#00220869 make stronger v#00222472 make more intense, stronger, or more marked v#00227165 the examples just mentioned represent instances of extremely high similarity of synset definitions. although not infrequent, such cases are not the majority. metior can serve as a less extreme example, yet still informative about vallex/wn granularity; see a list of its 9 synsets, all with frame act, pat: ## methodology in response to the aforementioned need of deriving annotation suggestions for verbal senses, we develop a predicate sense disambiguation (henceforth psd) workflow leveraging contextual embeddings. @xcite as the core of the approach, we try to assess the similarity 7 between the verbal tokens in the target text and those in the reference corpus, with the goal of disambiguating the token sense by virtue of its contextual surroundings. reference and target corpus 8 are defined based on text paragraphs (reference: par. 1-30 + par. 41-61; target: par. 31-40). the workflow consists of the following steps: extracting of verbal tokens. we collect a list of all verbal tokens by extracting them from our source text, i.e., sallust's de coniuratione catilinae annotated in the xml-based format prague markup language (pml). @xcite the pml files of the treebank are organized by annotation layers and linked to each other through stand-off annotation; we exploit the morphological (lemmatization and morphological tagging) and the tectogrammatical (semantic and pragmatic annotation) layers in combination. we retrieve all verbs by extracting nodes with a valency frame and the required pos. @xcite the extracted verbs are split according to the reference/target corpus partition, @xcite and are then manually annotated by a single annotator. storing annotated synsets. for each of the extracted tokens in the reference corpus, we store the synset definition that was manually assigned to it. three cases can occur: i) most verbs receive a synset from the latin wn/vallex, as linked in the lila knowledge base. for instance, dominor in lubidinem dominandi 'lust of dominion' is assigned the synset v#02442106 "be master; reign or rule". ii) when no appropriate synset can be found in the resource, a new one is defined. the definition of the new synset can consist either of an existing wn synset which was not yet assigned to the verb, or of a new definition modeled on a dictionary entry for the verb. e.g., for vivo there is no entry in wn; to its occurrence in alii alio more viventes 'living with different customs' we assign a new frame with synset v#02614387 "lead a certain kind @xcite measured in terms of cosine similarity. @xcite since we are not training any model, we decided not to call them training and test. @xcite the text is available at https://itreebank. marginalia.it/view/download.php as part of the latin dependency treebank (ldt). @xcite based on the guidelines of the prague dependency treebank, whose annotation the ldt replicates, valency mainly applies to verbs, yet not exclusively. see https://ufal.mff. cuni.cz/pdt2.0/doc/manuals/en/t-layer/html/ . @xcite the respective sizes of reference and target corpus are: i) tokens: 13,297 and 1,775 tokens; ii) extracted predicate tokens: 1,787 and 259. the division approximately conforms to a 9:1 ratio, while preserving the paragraph structure of the original work. of life; live in a certain style". iii) some tokens lack assigned synsets, as they can be treated as umr abstract predicates; 12 for instance, the verb sum 'to be' can be treated e.g. as identity-91, belong-91, have-mod[ification]-91. we proceed to exclude such tokens from the corpus. computing and comparing embeddings. for each verbal token in its respective sentence, both in reference and target corpus, embeddings are computed exploiting the flair library. @xcite we then compute cosine similarity to compare embeddings, and more precisely to quantify the degree of similarity between each target token and each reference token. similarity scores are then sorted in descending order, so that we can extract the five closest tokens (those with the highest scores -even if the scores are generally low). the synsets of these tokens are 12 umr features 9 types of abstract predicates, used to represent predication of properties, possession, location. they are identified by special labels serving as artificial lemmas and have their own roleset. for example, identity-91 has an arg1 role for the theme, and arg2 for the equated referent. further constraining candidate tokens. additionally, we retrieve all the tokens that are extracted as candidates before the first one with the same lemma as the target token 14 is found, i.e. those tokens with higher similarity score than the first one with constrained lemma. as preliminary results did not appear very promising, we decide to apply this additional lemma-based constraint on the candidate extraction. specifically, we henceforth select as candidates only those tokens which share the lemma with the target token. the samelemma requirement is merely an artificial constraint intended to facilitate the task, as in a real-case scenario it is possible to derive a correct synset even when the lemma differs. for instance, the synset v#00406243 "make ready or suitable or equip in advance for a particular purpose or for some use, event, etc." is shared by pario, instituo, and facio among other verbs. in theory, such tokens that share synsets should be retrievable aside from whether they share the same lemma or not. yet, the necessity of defining a simplified scenario through the imposition of a lemma constraint becomes ap-parent from the initial results of the experiments. output. as a result, the output file provides all retrieved information about each token: five annotation suggestions; i.e. the most plausible synsets; the number of incorrect guesses before suggesting a token with the same lemma; foot_6 the list of lemmas retrieved before a correct one was found. ## evaluation in this section we present and discuss a comparison between outputs yielded by different plms (subsection 4.1), with respect to various criteria. additionally, we manually evaluate a subset of the target corpus so as to complement the evaluation metrics with a qualitative analysis. ## conclusions the complexity of the task has been apparent from the beginning, and is confirmed by observations from related studies. bamman and burns (2020) already discuss comparable challenges, emphasizing the inherent difficulty of the wsd task and the lack of suitable resources for latin -an observation also echoed by keersmaekers et al. (2023). in light of such complexity, our study was never truly conceived as a solution to a specific task, but rather as a qualitative assessment of the available resources as well as of the results they can lead to. therefore, our main objective revolved around a thorough examination of the task, its objectives, and challenges, with the intention of critically analyzing and identifying realistic possibilities within the constraints of the available resources. one of the key questions concerned whether we can actually exploit available resources: in particular, can latin vallex represent a suitable resource for psd? at its present stage, its exploitation for psd does not appear to be feasible; its fine-grained granularity definitely presents challenges for this specific task. nevertheless, adopting a binary classification approach, as suggested by previous works (bamman and burns, 2020; lendvai and wick, 2022), may not offer a satisfactory solution either. as an illustrative example, the verb postulo demonstrates the need for at least three distinct frames, even under a coarse-grained granularity: i) 'to ask, demand, require' (act, addr, pat); ii) 'arraign before a court, to prosecute, accuse' [juridical] (act, pat, reg); iii) 'to contain, measure' [of things] (act, pat). currently, latin vallex/wn provides nine frames for postulo. the granularity of latin vallex and the simplicity of a binary classification demand a thoughtful exploration of alternative strategies to address such challenges. a possibility could be represented by sense clustering, as described e.g. by navigli (2006) and martelli et al. (2022). additionally, an important limitation of the study arises from the decision not to fine-tune plms, whose performances would most probably be enhanced through fine-tuning. however, fine-tuning requires training data, and the annotated dataset currently at our disposal is of limited size. the quantitative results, as illustrated in figure 1 , clearly highlight the substantial impact of the limited amount of available data on results. therefore, what can be also inferred from the present study is the need for a larger reference corpus, to be obtained by enlarging the existing dataset with additional data. an envisioned extension to the presented workflow involves the computation of sentence embeddings for definitions. without constraining either on same lemma or on same synset, and thus handling even oov cases, cosine similarity could be leveraged to identify the most probable synset by comparing all the synset definitions associated to the target token against the synset definition of the extracted candidates, to find the most similar one(s). in other words, embeddings for the synset definition of retrieved candidates could be generated, as well as for the list of synset definitions as available in vallex/wn for the lemma under scrutiny. we could then select candidate synset definitions by computing cosine similarity between all synsets associated in vallex/wn to the target lemma and synsets of the extracted candidate tokens in the reference corpus, in order to be able to deal not only with synsets shared by verbs with different lemma, but also with synsets that do not occur in the reference corpus. however, we expect the issues encountered so far (to name one, the dataset size) to pose similar challenges even in this furtherdefined setting.