UA lberta at S em E val-2021 Task 2: Determining Sense Synonymy via Translations (2021)

Abstract: We describe the University of Alberta systems for the SemEval-2021 Word-in-Context (WiC) disambiguation task. We explore the use of translation information for deciding whether two different tokens of the same word correspond to the same sense of the word. Our focus is on developing principled theoretical approaches which are grounded in linguistic phenomena, leading to more explainable models. We show that translations from multiple languages can be leveraged to improve the accuracy on the WiC task.

## introduction this paper describes the university of alberta systems for semeval-2021 task 2: multilingual and cross-lingual word-in-context disambiguation @xcite . we focus on the monolingual (english) variant of the task, which is the same as the original wic task @xcite ). an instance of the wic task consists of two sentences that share a focus word in common; the word may be inflected differently in each sentence (e.g. "they had searched his flat a few days before" and "the production of lithium from salt flats") but will share the same lemma and part of speech. a wic task system must decide, given such a pair of sentences, whether the focus tokens have the same meaning in both sentences. systems are compared in terms of their accuracy, the percentage of test instances correctly identified as true (same meaning) or false (different meaning). the dataset includes training, development, and testing splits; as our methods are unsupervised, we do not use the training data. the goal of this paper is an exploration of the use of translation information for the wic task. the intuition underlying our work is that distinctions in meaning tend to be reflected in distinctions in translation. we have previously presented methods leveraging translation information to improve word sense disambiguation @xcite , and most frequent sense detection @xcite , and have demonstrated that word senses which share translations are, in general, semantically related @xcite . we have also presented theoretical formalizations of lexico-semantic phenomena which view synonymy and translation as two aspects of semantic equivalence @xcite . our team additionally presented a method based on translation information @xcite for the semeval-2020 task 2 on predicting multilingual and cross-lingual lexical entailment @xcite . in this task, we investigate whether translation can be used to detect semantic equivalence in context, just as in the aforementioned prior task we investigated whether translation can be used to detect lexical entailment between word types. our focus is on developing principled theoretical approaches which are grounded in linguistic phenomena, leading to more explainable models. our more complex methods depend upon a mapping between word senses and translations, as different senses of a word often translate differently. we obtain such a mapping from babelnet @xcite , which combines information from princeton wordnet @xcite , multilingual lexical resources, and translations produced by mt models. wordnet is comprised of synonym sets, or synsets, which babelnet enriches with translations. each of the resulting multi-lingual synsets, or multi-synsets, contain lexicalizations of a single concept in various languages, allowing the translations of a given sense of a word to be identified. we treat babelnet as an imperfect implementation of a universal multi-wordnet with the theoretical properties described by @xcite . our results can be interpreted as a proof-ofconcept for the use of contextual translations as indicators of semantic similarity. we show that the methods that we develop for the wic task can leverage translations to improve over baselines, especially when multiple target languages are considered. while it is not our objective to compete with state-of-the-art supervised methods, we consider this to be a positive result, and a strong lead for future work on contextual semantic analysis. this paper is structured as follows: section 2 provides an overview of relevant prior literature. section 3 discusses the theoretical model underlying our work. section 4 outlines our methods. section 5 describes our experiments and results. ## related work methods for wic task can be roughly divided into two paradigms: contextualized-embedding-based systems, and word sense disambiguation-based systems. pilehvar and camacho-collados (2018) introduce the wic dataset as a benchmark for evaluating context sensitive word representations. @xcite achieve improvements by combining similarity scores from different types of contextual word and sentence embeddings. @xcite propose a method to enhance contextual representations by leveraging other pre-trained contextual or static embeddings. another approach to wic task is to employ a word sense disambiguation (wsd) system to tag the target words with senses from a pre-defined sense inventory and subsequently make a decision based on the predicted synsets of the target words. @xcite use the lmms sense embeddings @xcite to disambiguate the target words. a simple approach of checking if the disambiguated senses are equal lead to competitive performance in the semdeep-5 wic challenge @xcite . sensembert @xcite and ares @xcite embeddings, when used as features in a bert-based model, also achieve competitive results on the wic task. our methods combine elements of both paradigms. we employ contextual embeddings in our proposed translation-based methods. however, we take the embeddings of the translations of the target words instead of the target words themselves. similarly to wsd based approaches, our methods also analyze the common synsets of the focus tokens and their translations, with the goal of identifying a probable shared synset. the most similar prior work to our approach is that of pessutto et al. (2020) at the graded word similarity task (armendariz et al., 2020) of semeval 2020, who propose a translation-based approach to evaluate the contextual similarity of a pair of words. they hypothesize that leveraging similarity information from more languages would allow greater accuracy. we follow a similar intuition in our work. ## theoretical solution we first present a theoretical solution, which provides the foundation for the development of our actual methods described in section 4. we assume that the two source sentences s 1 and s 2 in each instance of the wic task can be translated into any natural language as sentences t 1 and t 2 . furthermore, we assume that the literal lexical translations t 1 and t 2 of the focus word s can be identified in t 1 and t 2 , respectively. for example, in figure 1 , the focus word s in the english sentences s 1 and s 2 is the noun differential, and word alignment identifies écart and différentiel as t 1 and t 2 . note that the two translations may have the same pos and lemma, a scenario we denote as t 1 = t 2 . ## methods in this section we describe four methods based on the theoretical ideas in section 3. all four methods rely on identifying lexical translations of the focus word in both source sentences. if the lexical translations cannot be recovered from the translated sentences for any of the target languages, all methods use the same backoff approach, which is to return false for that test instance. ## experiments in this section, we describe the application of our methods to the english development and test sets. we begin by specifying various implementation details. next, we describe our development experiments, including results and error analysis. finally, we present our results on the test set. while our method is, in theory, applicable to any language, and even to cross-lingual subtasks, we focus exclusively on the english monolingual substask due to time and resource constraints. ## conclusion overall, our results provide a solid proof-ofconcept for the utility of multilingual translation for the wic task. while not competitive with state-ofthe-art supervised methods, our results empirically verify the hypothesis that translations convey semantic information, and that this phenomenon has applications in lexical semantics. the ident and sub methods consistently benefit from translation into multiple languages, and this result generalizes to unseen test data.