Massively Multilingual Lexical Specialization of Multilingual Transformers (2023)

Abstract: While pretrained language models (PLMs) primarily serve as general-purpose text encoders that can be fine-tuned for a wide variety of downstream tasks, recent work has shown that they can also be rewired to produce high-quality word representations (i.e., static word embeddings) and yield good performance in type-level lexical tasks. While existing work primarily focused on the lexical specialization of monolingual PLMs with immense quantities of monolingual constraints, in this work we expose massively multilingual transformers (MMTs, e.g., mBERT or XLM-R) to multilingual lexical knowledge at scale, leveraging BabelNet as the readily available rich source of multilingual and cross-lingual type-level lexical knowledge. Concretely, we use BabelNet’s multilingual synsets to create synonym pairs (or synonym-gloss pairs) across 50 languages and then subject the MMTs (mBERT and XLM-R) to a lexical specialization procedure guided by a contrastive objective. We show that such massively multilingual lexical specialization brings substantial gains in two standard cross-lingual lexical tasks, bilingual lexicon induction and cross-lingual word similarity, as well as in cross-lingual sentence retrieval. Crucially, we observe gains for languages unseen in specialization, indicating that multilingual lexical specialization enables generalization to languages with no lexical constraints. In a series of subsequent controlled experiments, we show that the number of specialization constraints plays a much greater role than the set of languages from which they originate.

## introduction massively multilingual transfomers (mmts) such as mbert @xcite and xlm-r @xcite , among others, have been the primary vehicle of cross-lingual nlp transfer, offering state-of-the-art performance for many tasks and target languages in various zero-shot and fewshot transfer scenarios @xcite @xcite @xcite @xcite . much less work, however, investigated their capabilities as multilingual typelevel word encoders @xcite . recent work, focusing primarily on monolingual plms, demonstrated that they can be turned into effective type-level lexical encoders using lexical constraints @xcite , i.e., a process commonly referred to as lexical specialization. existing work, however, investigated lexical specialization in monolingual or bilingual settings only, namely specializing plms for a single language or a pair of languages using either english monolingual or noisy translated bilingual lexical constraints @xcite . this is not only computationally inefficient, since one specialization needs to be executed for each language or language pair, but also does not tap into the wealth of multilingual knowledge of mmt's simultaneous pretraining on many (100+) languages, as well as large amounts of manually curated knowledge from massively multilingual knowledge bases like babelnet @xcite . in this work, in contrast, we investigate massively multilingual lexical specialization of mmts, i.e., the potential benefits and limitations of a single lexical specialization procedure in multiple languages. to this end, we tap into babelnet as the readily available massively multilingual lexicosemantic resource. concretely, we release a dataset of synonym pairs or synonym-gloss pairs that cover 50 languages (representing 14 different language families) obtained from babelnet's multilingual synsets and leverage them as positive instances in a contrastive specialization training procedure. our evaluation on two multilingual lexical tasks -bilingual lexicon induction (bli) and cross-lingual semantic word similarity (xlsim) -as well as on the task of cross-lingual sentence retrieval demonstrate the effectiveness of the multilingual lexical special-ization when compared against vanilla mmts. we complement our evaluation with diagnostic experiments aimed at studying properties of the multilingual lexical constraints that might drive the downstream lexical performance of the specialized models, namely the choice of the languages and the number of constraints. for this, we perform experiments where we control mmt specialization for (i) the linguistic diversity of the language represented in the specialization dataset and (ii) the size of the specialization dataset, i.e., the number of synonym pairs from babelnet used in contrastive training. the results of our diagnostic experiments suggest that, counterintuitively, the typological diversity of the languages used in specialization (i.e., the specialization languages) has barely any effect in defining the downstream performance. these findings for multilingual specialization for (typelevel) lexical tasks contrast the observations for higher-level tasks, requiring the modeling of sentence or sentence-pair semantics, in which both multi-source specialization/fine-tuning on diverse languages @xcite and linguistic proximity between training and evaluation languages @xcite have been shown to strongly affect the transfer performance. at the same time, we find that the alignment performance quickly saturates with few constraints: this corroborates the rewiring hypothesis of @xcite , here in a massively multilingual setting. to encourage further research on this topic, we release our code and all our datasets of lexical constraints and synonym-gloss pairs at https://github.com/umanlp/babelbert . ## related work external lexical knowledge from lexico-semantic resources (e.g., wordnet, conceptnet) has been extensively leveraged for improving distributional representations of words -a process commonly referred to as semantic specialization. earlier work on semantic specialization of word embeddings can roughly be divided into (i) joint specialization approaches @xcite @xcite @xcite , which integrate the external lexical constraints in the word embedding (pre)training and (ii) retrofitting or postprocessing techniques that post-hoc modify the pretrained embeddings, conforming them to external lexical constraints @xcite @xcite @xcite . external lexical knowledge has also been used to enrich monolingual plms by coupling masked language modeling with an auxiliary pretraining objective using lexical relations from wordnet @xcite . however, these approaches are all aimed at enriching the transformer with additional knowledge to be exploited in various downstream tasks, rather than producing better type-level word representations. several paradigms for obtaining semantically improved multilingual representation spaces have been proposed for static word embeddings: (i) align-and-specialize @xcite starts from mutually unaligned monolingual embedding spaces and both aligns them and semantically specializes for semantic similarity (as opposed to other types of semantic relatedness) by means of both multilingual (i.e., monolingual constraints for multiple languages) and cross-lingual lexical constraints; (ii) in cross-lingual specialization transfer @xcite the embedding space of a target language is first projected into the (unspecialized) space of the highresource language (typically english) using a limited amount of cross-lingual source-target lexical alignments and then specialized with the specialization model trained using abundant monolingual lexical constraints of the resource-rich source language; (iii) constraint transfer @xcite noisily translates the (abundant) source language constraints into the target languages and uses those to specialize the monolingual embedding spaces of the target languages. more recently, @xcite showed that pretrained transformers encode a wealth of lexical knowledge in their parameters and that it is possible to obtain from them type-level (i.e., static, out-ofcontext) word representations that outperform representations obtained with word embedding models like fasttext @xcite . in a subsequent approach dubbed lexfit @xcite , they specialize monolingual bert models for a range of languages using english monolingual constraints from wordnet and roget's thesaurus, and automatically translated constraints @xcite for languages other than english. finally, once they obtain static word representations for each language, they induce a bilingual space in a standard fashion, by means of an orthogonal pro-jection @xcite . the effectiveness of lexfit stems from the fact that language-specific plms tend to produce better representations for a language than the mmts (e.g., mbert) due to the "curse of multilinguality" @xcite . this, however, limits its applicability to a few languages with existing monolingual plms, crucially excluding low-resource ones. the approach is also computationally expensive as it entails (i) a separate specialization process for each language (plus a noisy translation of english constraints to the target language), and (ii) a bilingual alignment for each language pair. additionally, it does not exploit cross-lingual lexical constraints in specialization, merely in post-hoc alignment. in this work, we propose a single massively multilingual lexical specialization approach (dubbed babelbert) to be applied to pretrained mmts. to this end, we propose to obtain the lexical constraints from ba-belnet, a massively multilingual lexico-semantic resource @xcite . babelbert has several advantages: (i) we leverage multilingual (i.e., monolingual from multiple languages) as well as cross-lingual lexical constraints and (ii) perform a single multilingual specialization procedure, instead of specializing separately for each language or language pair; (iii) the multilingual specialization on top of mmts removes the need for language-specific transformers and additionally allows for specialization effects to propagate to unseen languages, i.e., languages without readily available lexical constraints. ## multilingual lexical specialization we first describe how we create lexical constraints from babelnet in §3.1 and then how we leverage them in a contrastive learning procedure in §3.2. ## experimental setup multilingual lexical constraints. we focus on the 50 diverse languages from the popular xtreme-r benchmark @xcite which we report with language codes for brevity: af, ar, az, bg, bn, de, el, en, es, et, eu, fa, fi, fr, gu, he, hi, ht, hu, id, it, ja, jv, ka, kk, ko, lt, ml, mr, ms, my, nl, pa, pl, pt, qu, ro, ru, sw, ta, te, th, tl, tr, uk, ur, vi, wo, yo, zh. the sample covers 14 language families (afro-asiatic, austro-asiatic, austronesian, dravidian, indo-european, japonic, kartvelian, kra-dai, niger-congo, sino-tibetan, turkic, uralic, creole, and quechuan) and additionally contains basque and korean as two language isolates. we collect the constraints from babelnet with the procedure described in §3.1. as seed words, we select the top-n english most frequent words (@xmath0, 000, filtering for stopwords using nltk @xcite ) and retain only words that belong to the top-k (@xmath1, 000) words in the frequency list of a language. the total training set consists of 761,273 lexical constraints: we provide additional statistics of the dataset in appendix a and a few examples in evaluation tasks. we evaluate on two standard cross-lingual word-level tasks, bilingual lexicon induction (bli) and cross-lingual word similarity (xlsim). we couple this with an evaluation on unsupervised cross-lingual sentence retrieval. for the word-level tasks (xlsim and bli): a) we make sure not to include in the training set of lexical constraints from babelnet any word that appears in the test sets; b) we take the mean-pooling of the embeddings of the subwords from the bestperforming layer (see task 1: bilingual lexicon induction. bli tests the quality of a multilingual (bilingual) representation space by means of type-level word alignment between languages. for a given query word from a source language, words from the vocabulary of a target language are ranked based on their similarity with the query. the position of the correct translation of the query in the target language ranking reflects the quality of the type-level word alignment between the languages. we evaluate on two well-established benchmarks: g-bli @xcite covers 28 language pairs between 8 languages (de, en, fi, fr, hr, it, ru, tr), whereas panlexbli @xcite spans 15 diverse languages (bg, ca, eo, et, eu, fi, he, hu, id, ka, ko, lt, no, th, tr) for a total of 210 language pairs. for both datasets, we evaluate on the test portions consisting of 2,000 word pairs per language pair and report the performance in terms of mean reciprocal rank (mrr) as recommended by @xcite . following @xcite , the vocabularies used for retrieval cover the top 100k most frequent words from the respective fasttext wikipedia vectors @xcite . task 2: cross-lingual word similarity. xl-sim measures the correlation between the similarities of cross-lingual word pairs obtained based on their representations in the multilingual (bilingual) representation space and similarity scores assigned by human annotators. we evaluate the performance on 66 language pairs between 12 languages (zh, cy, en, et, fi, fr, he, pl, ru, es, sw, yue) from the multisimlex dataset @xcite and use spearman's ρ between the cosine similarities between words' embeddings and the corresponding human-assigned similarity scores. task 3: cross-lingual sentence retrieval. for cross-lingual sentence retrieval, we use the tatoeba dataset (artetxe and schwenk, 2019) which comprises 112 languages, where each language has 1,000 sentences paired with their translations in english. we obtain the sentence embedding by mean-pooling the representations of all of its subword tokens at the output of the best-performing transformer layer. we straightforwardly compute the similarities between sentences as the cosine of the angle between their embeddings. we compare each query sentence to its nearest neighbour and compute accuracy as our evaluation measure. training details. we experiment using two different mmts: multilingual bert (mbert) @xcite and xlm-r @xcite . 5 in all experiments, we set the temperature of the infonce loss to τ = 0.07. to account for the very skewed distribution of constraints across language pairs (cf. figure 3 ), following @xcite , we sample batch constraints from the multinomial distribution {q i,j } over language pairs (l i , l j ) as follows: hyperparameters and training details for the fully fine-tuned models, we search for the optimal learning rate @xmath2, 5e -6, 1e -6} and the batch size n @xmath3, 64}. for adapter-based models, we additionally try @xmath4.e., we set the bottleneck size of the adapters to 48). every experiment ## results and discussion we present our main results in overall, the results indicate that multilingual lexical fine-tuning improves the performance of both mmts (mbert and xlm-r) on all tasks. all three specialization variants (babel-ad/ft/gl) yield similar performance on all three tasks for mbert. the same is, however, not the case for xlm-r, where full fine-tuning (babel-ft and babel-gl) leads to substantially better performance across the board compared to adapter-based training (babel-ad). training with sense-level information in the form of synset glosses (babel-gl) seems particularly beneficial for cross-lingual sentence retrieval on tatoeba (ttb) -we assume that this is because, much like the rest of the sentences in tatoeba, the glosses in specialization training provide sentential context to word representations. interestingly, despite the fact that vanilla mbert produces substantially higher quality word representations than vanilla xlm-r (e.g., 14.5 vs. 8.5 on g-bli or 10.3 vs. 5.9 on xlsim), our multilingual lexical specialization tilts the results in favour of xlm-r, with comparable bli results between the two and much better xlm-r performance on xlsim (9-point gap) and . this suggests that xlm-r actually contains richer multilingual lexical knowledge than mbert, which is, however, buried deeper in the model (this is corroborated by the fact that for xlm-r we generally get better lexical representations from lower transformer layers, see the appendix): once uncovered by means of our multilingual lexical specialization, this richer lexical information of xlm-r surfaces. figure 1 shows the five language pairs for each evaluation task/dataset for which we observe the largest performance improvement with babel-ft mbert. in pl-bli, we find two languages, norwegian and catalan, for which no constraint was seen during training: this indicates that the benefits of massively multilingual lexical specialization propagate to unseen languages. what we find particularly encouraging is the fact that tatoeba pairs with the largest gains include some low-resource languages and dialects (e.g., wu and yu chinese, tamil, and interlingue), some of them not even present in mmts in pretraining. ## conclusion in this work, we presented a multilingual lexical specialization approach that leverages the massively multilingual lexical knowledge available in babelnet. differently from prior approaches, which perform monolingual or bilingual specialization procedures, we subject mmts to a single training regime with lexical constraints from 50 languages and report substantial gains over the unspecialized baseline (i.e., mmt not subject to lexical specialization). we perform both type-level lexical specialization, i.e. with words fed in isolation to the transformer, and sense-level lexical specialization, by accompanying each word with a gloss. we perform a series of additional experiments to study the driving factors of lexical specialization: in one experiment, we keep the training set size fixed while diversifying the languages in the training set. we observe that this does not seem to have a significant impact on the overall performance. in a subsequent experiment, we again use constraints encompassing 50 languages, but limit the number of constraints per language pair: we find that more constraints help the model perform better, however, few samples are necessary to reach peak performance. our results support the rewiring hypothesis of @xcite that lexical specialization resurfaces the existing lexical knowledge stored in mmts, rather than injecting it. such extraction seems to be independent of the training languages and quickly saturates with few constraints. limitations. while we try to perform multilingual lexical specialization on a set of typologically diverse languages, we are still restricting our analysis to a small fraction of all the languages of the world. in addition to this, our analysis investigates only two mmts -albeit arguably the two most widely used. due to hardware limitations, we experimented with xlm-r base: the results we report may be substantially different for xlm-r large (or other larger mmts like mt5), which possibly encodes more lexical knowledge.