XL - W i C : A Multilingual Benchmark for Evaluating Semantic Contextualization (2020)

Abstract: The ability to correctly model distinct meanings of a word is crucial for the effectiveness of semantic representation techniques. However, most existing evaluation benchmarks for assessing this criterion are tied to sense inventories (usually WordNet), restricting their usage to a small subset of knowledge-based representation techniques. The Word-in-Context dataset (WiC) addresses the dependence on sense inventories by reformulating the standard disambiguation task as a binary classification problem; but, it is limited to the English language. We put forward a large multilingual benchmark, XL-WiC, featuring gold standards in 12 new languages from varied language families and with different degrees of resource availability, opening room for evaluation scenarios such as zero-shot cross-lingual transfer. We perform a series of experiments to determine the reliability of the datasets and to set performance baselines for several recent contextualized multilingual models. Experimental results show that even when no tagged instances are available for a target language, models trained solely on the English data can attain competitive performance in the task of distinguishing different meanings of a word, even for distant languages. XL-WiC is available at https://pilehvar.github.io/xlwic/ .

## introduction one of the desirable properties of contextualized models, such as bert @xcite and its derivatives, lies in their ability to associate dynamic representations to words, i.e., embeddings that can change depending on the context. this provides the basis for the model to distinguish different meanings (senses) of words without the need to resort to an explicit sense disambiguation step. the conventional evaluation framework for this property has authors marked with a star ( ) contributed equally. pilehvar and camacho-collados (2019) addressed this dependence on sense inventories by reformulating the wsd task as a simple binary classification problem: given a target word w in two different contexts, c 1 and c 2 , the task is to identify if the same meaning (sense) of w was intended in both c 1 and c 2 , or not. the task was framed as a dataset, called word-in-context (wic), which is also a part of the widely-used superglue benchmark @xcite . despite allowing a significantly wider range of models for direct wsd evaluations, wic is limited to the english language only, preventing the evaluation of models in other languages and in cross-lingual settings. in this paper, we present a new evaluation benchmark, called xl-wic , that extends the wic dataset to 12 new languages from different families and with different degrees of resource availability: bulgarian (bg), chinese (zh), croatian (hr), danish (da), dutch (nl), estonian (et), farsi (fa), french (fr), german (de), italian (it), japanese (ja) and korean (ko). with over 80k instances, our benchmark can serve as a reliable evaluation framework for contextualized models in a wide range of heterogeneous languages. xl-wic can also serve as a suitable testbed for cross-lingual experimentation in settings such as zero-shot or few-shot transfer across languages. as an additional contribution, we tested several pretrained multilingual models on xl-wic, showing that they are generally effective in transferring sense distinction knowledge from english to other languages in the zero-shot setting. however, with more training data at hand for target languages, monolingual approaches gain ground, outperforming their multilingual counterparts by a large margin. ## xl-wic: the benchmark in this section, we describe the procedure we followed to construct the xl-wic benchmark. our framework is based on the original wic dataset, which we extend to multiple languages. ## experimental setup for our experiments, we implemented a simple, yet effective, baseline based on a transformer-based text encoder @xcite and a logistic regression classifier, following @xcite . the model takes as input the two contexts and first tokenizes them, splitting the input words into subtokens. the encoded representations of the target words are concatenated and fed to the logistic classifier. for those cases where the target word was split by the tokenizer into multiple sub-tokens, we followed @xcite and considered the representation of its first sub-token. as regards the text encoder, we carried out the experiments with three different multilingual models, i.e., the multilingual version of bert (devlin et al., 2019) (mbert) and the base and large versions of xlm-roberta @xcite (xlmr-base and xlmr-large, respectively). in the monolingual setting, we used the following language-specific models: bert-de 8 , 7 even if not part of xl-wic, we also compiled and validated a small italian wordnet dataset to compare it with its wiktionary counterpart. camembert-large @xcite foot_7 , bertit foot_8 , and parsbert foot_9 @xcite , respectively, for german, french, italian, and farsi. as for all the other languages covered by the word-net datasets, i.e., bulgarian, chinese, croatian, danish, dutch, estonian, japanese and korean, we used the pre-trained models made available by turkunlp. 12 we refer to each language-specific model as l-bert. in all experiments we trained the baselines to minimize the binary cross-entropy loss between their prediction and the gold label with the adam (kingma and ba, 2015) optimizer. training is carried out for 10 epochs with the learning rate fixed to 1e -5 and weight decay set to 0. as for tuning, results are reported for the best training checkpoint (among the 10 epochs) according to the performance on the development set. ## results in this section, we report the results for the configurations discussed in the previous section on the xl-wic benchmark. we organize the experiments into two parts, based on the test dataset: wordnet (section 5.1) and wiktionary (section 5.2). ## analysis in this section, we delve into the performance of the models on xl-wic and analyze relevant aspects about their behaviour. ## conclusions in this paper we have introduced xl-wic, a large benchmark for evaluating context-sensitive models. xl-wic comprises datasets for a heterogeneous set of 13 languages, including the original english data in wic @xcite , providing an evaluation framework not only for contextualized models in those languages, but also for experimentation in a cross-lingual transfer setting. our evaluations show that, even though cur16 bigger datasets are supersets of smaller ones. rent language models are effective performers in the zero-shot cross-lingual setting (where no instances in the target language are provided), there is still room for improvement, especially for far languages such as japanese or korean. as for future work, we plan to investigate using languages other than english for training (e.g., our larger french and german training sets) in our cross-lingual transfer experiments, since english may not always be the optimal source language (anastasopoulos and neubig, 2020). finally, while in our comparative analysis we have focused on a quantitative evaluation for all languages, an additional error analysis per language would be beneficial in revealing the weaknesses and limitations of cross-lingual models.