AM 2i C o: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples (2021)

Abstract: Capturing word meaning in context and distinguishing between correspondences and variations across languages is key to building successful multilingual and cross-lingual text representation models. However, existing multilingual evaluation datasets that evaluate lexical semantics “in-context” have various limitations. In particular, 1) their language coverage is restricted to high-resource languages and skewed in favor of only a few language families and areas, 2) a design that makes the task solvable via superficial cues, which results in artificially inflated (and sometimes super-human) performances of pretrained encoders, and 3) no support for cross-lingual evaluation. In order to address these gaps, we present AM2iCo (Adversarial and Multilingual Meaning in Context), a wide-coverage cross-lingual and multilingual evaluation set; it aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts for 14 language pairs. We conduct a series of experiments in a wide range of setups and demonstrate the challenging nature of AM2iCo. The results reveal that current SotA pretrained encoders substantially lag behind human performance, and the largest gaps are observed for low-resource languages and languages dissimilar to English.

## introduction pretrained language models (lms) such as bert @xcite and xlm-r @xcite offer a natural way to distinguish different word meanings in context without performing explicit sense disambiguation. this property of "meaning contextualization" is typically evaluated either via standard entity linking @xcite and word sense disambiguation (wsd) tasks @xcite @xcite or, recently, via the word-in-context (wic) evaluation paradigm @xcite . although monolingual evaluation in english is still predominant, a need has been recognized to construct similar resources for other languages to support cross-lingual evaluation and model diagnostics. this includes multilingual and crosslingual wsd benchmarks @xcite @xcite , cross-lingual entity linking @xcite @xcite and, most recently, multilingual wic (termed xl-wic) spanning 12 languages @xcite . this most recent wic evaluation approach is particularly attractive as 1) it bypasses the dependence on modeling predefined ontologies (entity linking) and explicit sense inventories (wsd), and 2) it is framed as a simple binary classification task: for a target word w appearing in two different contexts c 1 and c 2 , the system must decide whether w conveys the same meaning in both contexts, or not. however, the current wic evaluation still allows ample room for improvement: 1) current language coverage is limited, and biased towards resourcerich indo-european languages; 2) coverage of lexical concepts, due to their paucity in languagespecific wordnets, is also limited; 3) xl-wic is a monolingual resource available in different languages, i.e., it does not support cross-lingual assessments. further, 4) the current wic datasets offer low human upper bounds and inflated (even superhuman) performance for some languages. 1 this is due to superficial cues where 5) many examples in the current wic datasets can be resolved relying either on the target word alone without any context or on the context alone, which eludes evaluation honing in on the interplay between target words and their corresponding contexts. in order to address these limitations and provide a more comprehensive evaluation framework, we present am 2 ico (adversarial and multilingual meaning in context), a novel multilingual and cross-lingual wic task and resource. it covers a typologically diverse set of 15 languages, see based on wikipedia in lieu of wordnet, am 2 ico covers a wider set of ambiguous words which especially complements wic on the long tail of entity names and adds challenges in generalization for a larger vocabulary than a restricted set of common words. more importantly, the use of wikipedia enables wic evaluation on low-resource languages (e.g., basque, georgian, bengali, kazakh). we also improve the wic resource design; it now 1) includes adversarial examples and careful data extraction procedures to prevent the models from backing off to superficial clues, 2) results in a more challenging benchmark with truer and much wider gaps between current sota pretrained encoders and human capability (see §2.3), and 3) enables crosslingual evaluation and analysis. the ample and diverse data in am 2 ico enables a wide spectrum of experiments and analyses in different scenarios. we evaluate sota pretrained encoders, multilingual bert and xlm-r, both off-the-shelf using a metric-based approach (i.e., without any task adaptation) and after task-specific fine-tuning. with fine-tuned models, we investigate zero-shot cross-lingual transfer as well as transfer from multiple source languages. in general, our results across these diverse scenarios firmly indicate a large gap between human and system performance across the board, which is even more prominent when dealing with resource-poor languages and languages dissimilar to english, holding promise to guide modeling improvements in the future. in the hope that am 2 ico will be a challenging and valuable diagnostic and evaluation asset for future work in multilingual and cross-lingual representation learning, we release the data along with the full guidelines at https://github.com/ cambridgeltl/am2ico . ## am ico: cross-lingual word-in-context evaluation task definition. am 2 ico is a standard binary classification task on pairs of word in context instances. each pair consists of a target word with its context in english and a target word with its context in a target language. formally, each dataset of am 2 ico spans a set of n examples xi , @xmath0, . . . , n for a language pair. each example xi is in fact a pair of items @xmath1,src , x i,trg ), where the item x i,src is provided in the source language l src and the item x i,trg is in the target language l trg . the item x i,src in turn is another pair x i,@xmath2,src , c i,src ); it contains a target word w i,src from l src and its (wider) context c i,src (also in l src ) in which that word appears, see final resource. the full am 2 ico resource comprises datasets for 14 language pairs, where english is paired with 14 target languages. for brevity, in the rest of the paper we refer to the dataset of each language pair simply with the l trg language code (e.g., zh instead of en-zh); languages and codes are provided in as illustrative examples, we show a positive pair (label 't') and a negative pair (label 'f') from the zh am 2 ico dataset in in what follows we describe the creation of am 2 ico. we also demonstrate the benefits of am 2 ico and its challenging nature. ## experimental setup we now establish a series of baselines on am 2 ico to measure the gap between current sota models and human performance. pretrained encoders. multilingual contextualized representations e @xmath3, @xcite and xlm-r foot_5 @xcite , available in the huggingface repository @xcite . classification. given two contextualized representations e i,src and e i,trg for a pair of target words, two setups to make prediction are considered: the first, metric-based, is a non-parametric setup. in particular, we follow pilehvar and @xcite and score the distance δ between the representations via cosine similarity. a threshold t from the development set is set via grid search across 0.02 intervals in the [0, 1] interval. therefore, if δ(e i,src , e i,trg ) ≥ t the pair is classified as negative, and positive otherwise. on the other hand, the fine-tuning setup is parametric: following @xcite , we train a logistic regression classifier that takes the concatenation of the contextualized representations [e i,src ⊕ e i,trg ] as input. 7 the entire model (both the encoder and the classifier) is then fine-tuned to minimize the cross-entropy loss of the training set examples with adam (kingma and ba, 2015). we perform grid search for the learning rate in [5e-6, 1e-5, 3e-5], and train for 20 epochs selecting the checkpoint with the best performance on the dev set. cross-lingual transfer. in addition to supervised learning, we also carry out cross-lingual transfer experiments where data splits may belong to different language pairs. the goal is transferring knowledge from a source language pair ℓ s to a target language pair ℓ t . to simulate different scenarios of data paucity, in the fine-tuning setup we consider: 1) zero-shot transfer, where train and development sets belong to ℓ s and the test set to ℓ t ; 2) zero-shot + tld foot_7 transfer, which is similar except for the dev set given in ℓ t ; 3) on top of zero-shot + tld, we provide a small amount of training examples in ℓ t , which we denote as few-shot transfer; 4) finally, in the joint multilingual setup, we train a single model on the concatenation of the train sets for all language pairs and select the hyper-parameters on the development set of ℓ t . ## results and discussion metric-based vs fine-tuning. we report the results for the supervised learning setting (where all data splits belong to the same language pair) in ta- ## related work cross-lingual evaluation of word meaning in context. going beyond readily available sense inventories required for wsd-style evaluations, the comprehensive benchmarks for evaluating word meaning in context cross-lingually are still few and far between. xl-wic @xcite extends the original english wic framework of @xcite to 12 other languages, but supports only monolingual evaluation, and suffers from issues such as small gaps between human and system performance. the semeval-2021 shared task mcl-wic does focus on crosslingual wic, but covers only five high-resource languages from three language families (english, french, chinese, arabic, russian). both xl-wic and mcl-wic mainly focus on common words and do not include less frequent concepts (e.g., named entities). further, their language coverage and data availability are heavily skewed towards indo-european languages. there are several other 'non-wic' datasets designed to evaluate cross-lingual context-aware lexical representations. bilingual contextual word similarity (bcws) @xcite challenges a model to predict graded similarity of crosslingual word pairs given sentential context, one in each language. in the bilingual token-level sense retrieval (btsr) task @xcite , given a query word in a source language context, a system must retrieve a meaning-equivalent target language word within a target language context. foot_8 however, both bcws and btsr are again very restricted in terms of language coverage: bcws covers only one language pair (en-zh), while btsr contains two pairs (en-zh/es). further, they provide only test data: as such, they can merely be used as general intrinsic probes for pretrained models, but cannot support fine-tuning experiments and cannot fully expose the relevance of information available in pretrained models for downstream applications. this is problematic as intrinsic tasks in general do not necessarily correlate well with downstream performance @xcite . am 2 ico vs. entity linking. our work is related to the entity linking (el) task @xcite @xcite similarly to how the original wic (based on wordnet knowledge) is related to wsd. el systems must map entities in context to a predefined knowledge base (kb). while wsd relies on the wordnet sense inventory, the el task focuses on kbs such as wikipedia and dbpedia. when each entity mention is mapped to a unique wiki page, this procedure is termed wikification @xcite . the cross-lingual wikification task @xcite grounds multilingual mentions to english wikipedia pages. similar to wsd, el evaluation is tied to a specific kb. it thus faces similar limitations of wsd in terms of restricting meanings and their distinctions to those predefined in the inventory. in comparison, am 2 ico leverages wikipedia only as a convenient resource for extracting the examples, similar to how the original wic work leverages wordnet. am 2 ico itself is then framed on natural text, without requiring the modeling of the kbs. also, in comparison with el, am 2 ico provides higher data quality and a more challenging evaluation of complex word-context interactions, achieved by a carefully designed data extraction and filtering procedure. ## conclusion we presented am 2 ico, a large-scale and challenging multilingual benchmark for evaluating word meaning in context (wic) across languages. am 2 ico is constructed by leveraging multilingual wikipedias, and subsequently validated by humans. it covers 15 typologically diverse languages and a vocabulary substantially larger than all previous wic datasets. as such, it provides more comprehensive and reliable quality estimates for multilingual encoders. moreover, am 2 ico includes adversarial examples: resolving such examples requires genuine lexical understanding, as opposed to relying on spurious correlations from partial input. finally, am 2 ico offers the possibility to perform cross-lingual evaluation, pairing context between different languages. moreover, we explored the impact of language relatedness on model performance by transferring knowledge from multiple source languages. we established a series of baselines on am 2 ico, based on sota multilingual models, revealing that the task is far from being 'solved' even with abundant training data. all models struggle especially when transferring to distant and resourcelean target languages. we hope that am 2 ico will guide and foster further research on effective representation learning across different languages.