S em E val-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation ( MCL - W i C ) (2021)

Abstract: In this paper, we introduce the first SemEval task on Multilingual and Cross-Lingual Word-in-Context disambiguation (MCL-WiC). This task allows the largely under-investigated inherent ability of systems to discriminate between word senses within and across languages to be evaluated, dropping the requirement of a fixed sense inventory. Framed as a binary classification, our task is divided into two parts. In the multilingual sub-task, participating systems are required to determine whether two target words, each occurring in a different context within the same language, express the same meaning or not. Instead, in the cross-lingual part, systems are asked to perform the task in a cross-lingual scenario, in which the two target words and their corresponding contexts are provided in two different languages. We illustrate our task, as well as the construction of our manually-created dataset including five languages, namely Arabic, Chinese, English, French and Russian, and the results of the participating systems. Datasets and results are available at: https://github.com/SapienzaNLP/mcl-wic .

## introduction during recent decades, the field of natural language processing (nlp) has witnessed the development of an increasing number of neural approaches to representing words and their meanings. word embeddings encode a target word type with one single vector based on co-occurrence information. however, word embeddings conflate different meanings of a single target word into the same representation, thus they fail to capture the polysemous nature of words. to address this limitation, more sophisticated representations such as multi-prototype and contextualized embeddings have been put forward. multi-prototype embeddings concentrate on the semantics which underlie a target word by clustering occurrences based on their context similarities @xcite . in an effort to exploit the knowledge derived from lexical-knowledge bases, @xcite introduced a new approach which allows sense representations to be linked to a predefined sense inventory. more recently, contextualized embeddings were proposed. these representations are obtained by means of neural language modeling, e.g. using lstms @xcite or the transformer architecture @xcite , and are capable of representing words based on the context in which they occur. contextualized representations have also been used to obtain effective sense embeddings @xcite @xcite . although virtually all the above approaches can be evaluated in downstream applications, the inherent ability of the various embeddings to capture meaning distinctions still remains largely underinvestigated. while word sense disambiguation (wsd), i.e. the task of determining the meaning of a word in a given context @xcite , has long explored the aforementioned ability, the task does not make it easy to test approaches that are not explicitly linked to existing sense inventories, such as wordnet @xcite and babelnet @xcite . this has two major drawbacks. first, sense inventories are not always available, especially for rare languages. second, such requirement limits the evaluation of word and sense representations which are not bound to a sense inventory. to tackle this limitation, some benchmarks have recently been proposed. the cosimlex dataset @xcite and the related semeval-2020 task 3 @xcite focus on evaluating the similarity of word pairs which occur in the same context. more recently, the word-in-context (wic) task @xcite , included in the su-perglue benchmark for natural language understanding (nlu) systems @xcite and its multilingual extension xl-wic @xcite , require systems to determine whether a word occurring in two different sentences is used with the same meaning, without relying on a predefined sense inventory. for instance, given the following sentence pair: despite the steps forward made in this promising research direction, existing benchmarks suffer from the following shortcomings: i) they are mostly automatically retrieved; ii) they do not enable cross-lingual evaluation scenarios in which systems are tested in different languages at the same time; iii) they do not cover all open-class parts of speech. in order to address the aforementioned drawbacks, we propose the first semeval task on multilingual and cross-lingual word-in-context disambiguation (mcl-wic) and present the first entirely manually-annotated dataset for the task. importantly, mcl-wic enables new cross-lingual evaluation scenarios covering all open-class parts of speech, as well as a wide range of domains and genres. the dataset is available in five european and non-european languages, i.e. arabic (ar), chinese (zh), english (en), french (fr) and russian (ru). ## related work several different tasks have been put forward which go beyond traditional wsd and drop the requirement of fixed sense inventories. among the first alternatives we cite monolingual and cross-lingual lexical substitution @xcite . word-in-context similarity has also been proposed as a way to capture the dynamic nature of word meanings: the stanford contextual word similarities (scws) dataset, proposed by @xcite , contains human judgements on pairs of words in context. along these same lines, armendariz et al. introduced cosimlex, a dataset designed to evaluate the ability of models to capture word similarity judgements provided by humans. mcl-wic sub-task dataset train dev test multilingual ar-ar -500 500 en-en 4000 500 500 fr-fr -500 500 ru-ru -500 500 zh-zh -500 500 cross-lingual en-ar --500 en-fr --500 en-ru --500 en-zh --500 ## the multilingual and cross-lingual word-in-context task in this section, we present our semeval task and describe a new dataset called multilingual and cross-lingual word-in-context (mcl-wic). the task is divided into a multilingual and a crosslingual sub-task, each containing different datasets divided according to language combination. each dataset instance is focused on a given lexeme 1 and is composed of a unique id, a target lemma, its part of speech, two sentential contexts in which the target lemma occurs, and positional indices for retrieving the target words in each sentence. in in the following, we introduce the multilingual and cross-lingual sub-tasks. then, we describe the data sources, the selection of the target lexemes and sentence pairs and, finally, the annotation process. ## linguistic issues in this section, we describe interesting languagespecific issues which required additional guidelines. due to space limits, we focus on languages which do not use the latin alphabet, i.e. arabic, chinese and russian, illustrating only the most significant issues encountered. arabic from a wsd perspective, compared to other languages, written arabic poses bigger challenges due to the omission of vocalization, which increases the degree of semantic ambiguity. in fact, the vocalization, expressed by diacritics placed above or below consonants, contributes significantly to determining the right interpretation and thus the meaning of words. for instance, the unvocalized word form b-r-d could be interpreted as bard ("cold"), burd ("garment") or barad ("hail"). of course, in arabic, polysemy also affects vocalized words, which can have multiple meanings, e.g. ummiyy means "maternal", but also "illiterate". for the purposes of mcl-wic, we chose to keep the sentences as they are found in unpc and wikipedia, i.e. unvocalized in the vast majority of cases, while -instead -providing the target lemmas in the vocalized form. this was done in order to avoid lexical ambiguity deriving from lemmas which share the same word form but are vocalized in a different way. furthermore, this choice facilitated the selection and annotation of sentence pairs in which a given target lemma occurs. chinese since chinese does not adopt an alphabet, the semantic ambiguity that can be found in english homographs is basically lost. in chinese, if two unrelated words are pronounced in the same way, such as "plane" (the airplane) and "plane" (the surface), they are not usually written in the same way. by way of illustration, 沉默, meaning "silent; to be silent" and 沉没, "to sink", are both pronounced as chénmò, but, because they are written with different characters, they cannot be considered ambiguous words. analogously, some characters have an extremely high semantic ambiguity themselves, but since they appear most frequently in polysyllabic words, their ambiguity is lost. for example, the character guǒ 果 has at least two meanings, "fruit" and "result", but this character almost never stands as a word on its own in contemporary chinese. in the current lexicon most of the chinese words are composed of two or more characters; when it appears in actual texts, guǒ is al-most always connected to other characters, and the word thus formed is no longer semantically ambiguous. finally, similarly to the cross-lingual sub-task, some ambiguity had to be discarded in translation, as in the case of chinese classifiers which have a marked potential for semantic ambiguity. for example, dào 道 is, among others, the classifier for long and narrow objects, as in yī dào hé 一道河, a river (one+classifier+river), or for doors, walls and similar objects with an entry and an exit, as in yī dào mén 一道门, a door (one+classifier+door). however, since classifiers are virtually absent in european languages, they could not be applied in the cross-lingual sub-task and were discarded. russian a noteworthy issue encountered by russian annotators concerned the verbal aspects which can be viewed as one of the most challenging features of the russian language especially for l2learners foot_9 with no slavic background. in russian, a verb can be perfective, imperfective or both. normally, a perfective verb has one or more imperfective counterparts and vice versa. broadly speaking, perfective verbs are typically used to express nonrepetitive actions completed in the past, or actions which will certainly be carried out in the future, and also in general for past or future actions for which the speaker intends to emphasize the result that was or will be achieved. conversely, imperfective verbs are used to express actions which are incomplete, habitual, in progress, or actions for which the speaker does not stress the result to be attained. in mcl-wic, given a verbal target lexeme, we decided to choose sentences in which the target words occurring in the selected sentences and the target lemma shared the same aspect. in fact, in russian, although pairs of perfective and imperfective verbs such as delat , sdelat (to do) or spraxivat , sposit (to ask) show a high degree of morphological relatedness, they tend to be considered as distinct lemmas. another interesting issue regards participles. in some cases, annotators raised issues concerning the part of speech of participles occurring as target words in the selected sentences. in fact, russian participles derive from verbs, but are declined and can behave as adjectives. since the target lexemes and the corresponding occurrences must share the same part of speech, we decided to discard sentences in which the part of speech of the target words could not be determined unequivocally. ## participating systems this section is devoted to the participating systems. first, we briefly describe the rules of the competition. subsequently, we provide an overview of the data and approaches used by participants. then, we focus on some of the best-scoring systems and provide a breakdown of the techniques adopted. we report the three best-performing teams for each sub-task and language combination in tables 6 and 7 . all results are publicly available on the official mcl-wic page on github foot_10 . for each winning team, we show only the best performance in the corresponding category. ## baselines following raganato et al. (2020), we used a baseline transformer-based binary classifier. thus, first, given a sentence pair, a dense representation is obtained for each target occurrence. as indicated in @xcite , in the case that a target occurrence is split into multiple sub-tokens, the first sub-token is selected. the resulting representations are then given as input to a binary classifier implemented following @xcite . we selected the adam optimizer (kingma and ba, 2015) with learning rate and weight decay equal to 1e-5 and 0, respectively, and trained for 10 epochs. we experimented with two different contextualized embedding models: bert (base-multilingualcased) and xlm-roberta (base). as for the data, in contrast to most participants, we made use of the data provided for the task only. we used en-en as training and development data for english. as for other language combinations, we trained on en-en and validated both on en-en or and on the other language multilingual development data. ## results and discussion in this section, we discuss the results achieved in our competition. overall, the mcl-wic dataset allows systems to attain high performances, in the 85-93% accuracy range. this leads us to hypothesize that, in general, systems were able to develop a good ability in capturing sense distinctions without relying on a fixed sense inventory. when compared to the proposed baselines, we observe that best-performing systems were able to achieve an absolute improvement of up to 27.1 points over the corresponding baselines (e.g. on en-ar, cf. tables 7 and 8 ). both our baselines and the systems developed by participants confirm that, in this task, xlm-roberta outperforms bert in most language combinations. the highest score was obtained in en-en, with the best system achieving 93.3% accuracy. note that our baselines were also able to attain good performances in en-en, i.e. 84.0% using bert and 86.6% with xlm-roberta, without benefiting from additional training and development data. interestingly, chinese was the language which achieved the second-best results, both in zh-zh and en-zh, attaining on average results which were considerably higher. instead, arabic seems to have been the most difficult language for participants, especially in ar-ar. a reason for this result, deserving further exploration, could lie in morpho-semantic features inherent in arabic, which we briefly outlined in section 4. zero-shot approaches differ in the performances achieved by participants in the two sub-tasks: in the cross-lingual sub-task participants were able to achieve slightly better performances than those in the multilingual setting, most probably thanks to the presence of english in both the training and the test data, and, more in general, to the availability of english wic-style datasets which could be used to enrich the already provided data. with the exception of chinese, instead, on the multilingual sub-task we observe a performance drop between 1.6 and 4.3%. finally, we note that performance boosts were observed across the board when using data augmentation, especially by swapping the two sentences within a pair or by coupling the second sentences of two pairs sharing the same first sentence and the same meaning. another consistent performance increase, observed both in the multilingual and in the cross-lingual sub-task, was obtained when adding a signal on both sides of the target occurrences. ## conclusions in this paper, we described the semeval-2021 task 2 and introduced multilingual and cross-lingual word-in-context (mcl-wic), the first entirely manually-curated wic-style dataset in five european and non-european languages, namely arabic, chinese, english, french and russian. mcl-wic allows the inherent ability of systems to discriminate between word senses within the same language to be tested, and also, interestingly, within crosslingual scenarios in which a system is evaluated in two languages at the same time, namely english and one of the remaining mcl-wic languages. while current word-in-context datasets focus primarily on single tokens, as a suggestion for future work we would like to further explore the integration of multi-word expressions and idiomatic phrases into a word-in-context task. this would allow us to investigate the intrinsic ability of a system to correctly discriminate the semantics of such linguistic constructs, especially those whose meaning is not compositional, i.e. it cannot be derived by combining the meaning of each of their individual components.