S zeged AI at S em E val-2021 Task 2: Zero-shot Approach for Multilingual and Cross-lingual Word-in-Context Disambiguation (2021)

Abstract: In this paper, we introduce our system that we participated with at the multilingual and cross-lingual word-in-context disambiguation SemEval 2021 shared task. In our experiments, we investigated the possibility of using an all-words fine-grained word sense disambiguation system trained purely on sense-annotated data in English and draw predictions on the semantic equivalence of words in context based on the similarity of the ranked lists of the (English) WordNet synsets returned for the target words decisions had to be made for. We overcame the multi,-and cross-lingual aspects of the shared task by applying a multilingual transformer for encoding the texts written in either Arabic, English, French, Russian and Chinese. While our results lag behind top scoring submissions, it has the benefit that it not only provides a binary flag whether two words in their context have the same meaning, but also provides a more tangible output in the form of a ranked list of (English) WordNet synsets irrespective of the language of the input texts. As our framework is designed to be as generic as possible, it can be applied as a baseline for basically any language (supported by the multilingual transformed architecture employed) even in the absence of any additional form of language specific training data.

## introduction a major obstacle in solving word sense disambiguation (wsd) problems in a supervised manner is the scarcity of annotated training corpora. as the construction of high quality sense-annotated training data can be extremely labor-intensive and difficult @xcite , the word-in-context (wic) disambiguation task was recently proposed by @xcite as a surrogate for the traditional wsd problem. while in the traditional fine-grained wsd setting, the aim is to assign a precise and often nuanced meaning to a word in its context according to some sense inventory, wic is framed as a binary classification problem, where the task is to decide whether two target words originating from a pair of input sentences have the same meaning. this kind of binary decision can also be made in the absence of a nuanced sense inventory, making the annotation process less demanding and also more suitable across languages @xcite . in this paper, we analyze the utilization of multilingual transformer-based language models for performing both multi-lingual and cross-lingual wic in the zero-shot setting, by employing nothing but english sense annotated training data and utilizing the model predictions in a transductive model that is capable of performing zero-shot wsd and wic disambiguation for any language that is supported by the multilingual transformer encoder model that gets employed. @xcite showed that a simple, nearest neighbor approach relying on contextual word embeddings can achieve impressive wsd results in english. in our follow-up work @xcite , we demonstrated, how sparse contextualized word representations can be exploited for obtaining significant improvements over the lmms approach introduced by @xcite . our shared task participation was focused on comparing the two techniques in a zero-shot multilingual and cross-lingual wic evaluation setting. ## system overview at the core of our multi,-and cross-lingual wic systems, we employed fine-grained wsd systems, originally intended to solely handle english texts. the two models that we employed were the lmms @xcite and the s-lmms @xcite approaches. we dub the latter solution as s-lmms, highlighting its resemblance to the lmms approach and the fact that it operates with sparse contextualized word representations. both lmms and s-lmms requires sense-labeled training data for constructing their respective fine-grained wsd models. we provide a brief overview of the two approaches and encourage readers interested in more details to read the original papers @xcite introducing them. lmms and s-lmms both has in common, that they encode the inputs with a transformer model (bert-large). lmms constructs a prototype vector for each english synset based on the bertencoded vectors of the sense-annotated training data and the actual contents of the english wordnet glosses. for a given token in its context, lmms takes its bert-encoded contextualized vector and finds the nearest synset prototype for determining its sense. the way s-lmms differs from lmms is that it additionally incorporates a sparsity inducing dictionary learning step, which turns the contextualized word representations into a sparse format, i.e., to such vectors that contain a high fraction (> 90%) of zero coefficients. additionally, the methodology for creating the synset prototype vectors has substantial differences between the two approaches, as lmms uses the actual contextualized embeddings pertaining to a certain synset as prototypes, whereas s-lmms distills a vectorial representation to each synset based on an information theoretic measure. the important technical change that we performed over the previously described fine-grained wsd models, so that they can be employed in the cross-lingual setting, is that we replaced the bert-large encoders that the lmms and s-lmms models use by default to the xlm-roberta-large @xcite architecture. we shall refer to the variants of lmms and s-lmms that were obtained by relying on xlm-roberta as an encoder as opposed to bert-large as mlmms and ms-lmms, owing to the multilingual nature of xlm-roberta. we used the transformers library @xcite for obtaining the contextualized multilingual embeddings for our experiments. when performing fine-grained wsd in english, one can simply restrict the scope of predicting the most likely synset for some word to those that are deemed viable for a given word in wordnet. addi-tionally, one can also filter the synsets over which the prediction is performed, based on the part-ofspeech category of a word in question. with these heuristics, it is possible to reduce the number of synsets that a word can belong to a few dozens of synsets even for the most ambiguous cases. in order to test a solution that is as generic as possible, we did not integrate any of these heuristics into our framework, meaning that our models returned a ranked list over all the 117,659 english wordnet synsets to any word from some sentence. this way, our solution can also work basically any language (supported by the multilingual transformer employed), even in the absence of a multilingual sense-inventory resource such as babelnet @xcite and also when we have no access to the part-of-speech information, nor to a part-of-speech tagger for some language. these design choices ensures that we are able to handle a much wider range of languages as if we decided otherwise. to this end, we regard our approach a particularly good fit being used as a baseline for wsd related evaluations involving low-resource languages. as mentioned previously, our * lmms models assigned a ranked list of 117,659 english synsets to every target word irrespective of the language of the sentence it was written in. since the ranking of the synsets for a given word was performed over all the synsets of wordnet, it would be too restrictive to expect that words with identical meaning should be assigned the exact same most likely english synset. to this end, we measured the similarity for a pair of ranked lists that a model returned for a pair of words in their contexts and decided about the semantic equivalence of the two words based on that similarity score. as the similarity scores calculated for the ranked lists of synsets that fit those pairs of words that have the same meaning are expected to be higher on average, we decided to determine a threshold for the similarity scores of the ranked lists above which we predicted the two words to have the same meaning, and to have a different meaning otherwise. we experimented with three strategies for measuring the similarity of two ranked synset lists for a pair of words. let s 1 and s 2 refer to the ranked lists of wordnet synsets assigned to two words. as the bottom of the ranking is arguably not as meaningful as its top-ranked elements, we decided to formulate s (100) 1 and s (100) 2 . these ranked lists differed from s 1 and s 2 in that they contained their top 100-ranked elements, respectively. 1 since we only focus on the highest ranked synsets from s 1 and s 2 , it is almost sure that certain element from s (100) 1 are not included in s (100) 2 , and vice versa. as such, the usage of standard rank correlation scores would be inconvenient for measuring the similarity between ranked lists s (100) 1 and s (100) 2 . one motivation behind the introduction of ranking-biased overlap (rbo) @xcite was particularly this, i.e. to provide such a distance metric that is capable of operating between non-conjoint rankings. rbo is an overlap-based metric, that can operate over such rankings when the ranked elements themselves are not totally identical. to this end one of our metric for measuring the similarity between s (100) 1 and s (100) 2 was based on the rbo metric. our other approach for measuring the similarity of ranked lists s (100) 1 and s (100) 2 was to simply take their jaccard similarity, i.e. the fraction of the size of their intersection and the elements in their union. as a third approach, we calculated the harmonic mean of the mean reciprocal rank (mrr) of the highest ranked synset from s instead of using some supervised approach, we determined a threshold for the similarity score for a pair of ranked synset lists s (100) 1 and s (100) 2 , above which we predicted that the words they got assigned to had identical meaning. we determined this threshold in a transductive manner, without using any of the labeled training or development set sentence pairs at all. for the cross-lingual evaluation it would have been impossible at the first place, as no annotated pairs of sentences were released during the shared task. we used expectation maximization for determining the similarity threshold above which we predicted a pair of words to have the same meaning. that is, we took all the similarity scores that we calculated for a certain test set based on the s (100) 1 and s (100) 2 ranked synset lists, and fitted a gaussian mixture model over the similarity scores. that way, we managed to fit a gaussian distribution for 1 experiments with different thresholds (10, 25, 50, 250 and 500) also provided similar results that we omit for brevity. the similarity scores of pairs of words with identical and different meanings. we identified the fitted gaussian distribution with the higher expected value to be the one that corresponds to the distribution of similarity scores for those words that have identical meaning. as expectation maximization algorithms are prone to find local optima, we initialized each model 100 times and chose the one which resulted in the best log-likelihood score. our decisions for a particular test sample was then based on the density functions on the similarity scores of the two classes determined by the best fitting model. ## experiments we tested our approach on both the multilingual and the cross-lingual subtasks of the shared task @xcite . the multilingual test sets consisted of sentence pairs that were written in the same language (either arabic, english, french, russian or chinese), whereas, an input was comprised of an english and a non-english (either arabic, french, russian or chinese) sentence for the cross-lingual scenario. the fine-grained wsd model that we built our system on was trained over english senseannotated training data. we used two sources of training signal, the semcor dataset as well as the princeton wordnet gloss corpus (wngc), which has been shown to improve fine-grained wsd results @xcite . unless stated otherwise, we used these three sources of sense-annotated training data for obtaining our * lmms models. foot_0 ## conclusions in this paper, we introduced our cross,-and multilingual wic framework that we approached from an all-words fine-grained word sense disambiguation perspective. as such, our model not only provides a yes or no answer for a pair of words in their contexts, but also provides a more tangible explanation for it in the form of the similarity between the ranked lists of english wordnet synsets assigned to the target words. during the design of our approach, we made such choices that would make our framework conveniently applicable to new languages without the need for any training data. although the results of our framework lags behind the top performing systems, due to of its convenient applicability to new languages and the fact that practically no additional training data is required for applying it to new and possibly low-resourced languages, we think it can provide an easy to use baseline in further wic-related research efforts.