Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders (2023)

Abstract: Pretrained multilingual language models (LMs) can be successfully transformed into multilingual sentence encoders (SEs; e.g., LaBSE, xMPNet) via additional fine-tuning or model distillation with parallel data. However, it remains unclear how to best leverage them to represent sub-sentence lexical items (i.e., words and phrases) in cross-lingual lexical tasks. In this work, we probe SEs for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs. We also devise a simple yet efficient method for exposing the cross-lingual lexical knowledge by means of additional fine-tuning through inexpensive contrastive learning that requires only a small amount of word translation pairs. Using bilingual lexical induction (BLI), cross-lingual lexical semantic similarity, and cross-lingual entity linking as lexical probing tasks, we report substantial gains on standard benchmarks (e.g., +10 Precision@1 points in BLI). The results indicate that the SEs such as LaBSE can be ‘rewired’ into effective cross-lingual lexical encoders via the contrastive learning procedure, and that it is possible to expose more cross-lingual lexical knowledge compared to using them as off-the-shelf SEs. This way, we also provide an effective tool for harnessing ‘covert’ multilingual lexical knowledge hidden in multilingual sentence encoders.

## introduction transfer learning with pretrained language models (lms) such as bert @xcite and roberta @xcite offers unmatched performance in many nlp tasks @xcite . however, despite the wealth of semantic knowledge stored in the pretrained lms @xcite , they do not produce coherent and effective sentence representations when used off-the-shelf @xcite . to this effect, further specializa-tion for sentence-level semantics -not unlike the standard task fine-tuning -is needed @xcite @xcite . lms get transformed into sentence encoders (ses) via dual-encoder frameworks that leverage contrastive learning objectives @xcite , in supervised (i.e., leveraging labeled external data such as nli or sentence similarity annotations) @xcite @xcite or, more recently, fully unsupervised finetuning @xcite setups. following the procedures from monolingual setups, another line of research has been transforming multilingual lms into multilingual ses @xcite , which enable effective sentence matching and ranking in multiple languages as well as cross-lingually @xcite . the transformation is typically done by coupling 1) lm objectives on monolingual data available in multiple languages with 2) cross-lingual objectives such as translation language modeling (tlm) @xcite ) and/or cross-lingual contrastive ranking @xcite . such multilingual ses consume a large number of parallel sentences for the latter objectives. consequently, they outperform multilingual off-the-shelf lms in cross-lingual sentence similarity and ranking applications @xcite . however, as we show in this work, such multilingual ses may still lag behind traditional static cross-lingual word embeddings (clwes) when encoding sub-sentence lexical items (e.g., words or phrases) @xcite for cross-lingual lexical tasks (e.g., bli). in this work, we probe multilingual ses for cross-lingual lexical knowledge, relying on standard semantic similarity tasks in cross-lingual setups as our 'lexical probes' @xcite . we demonstrate that, due to their fine-tuning on multilingual and parallel data, they indeed store a wealth of such knowledge, much more than what 'meets the eye' when they are used 'off the shelf'. however, this lexical knowledge needs to be exposed from the original multilingual ses, (again) through additional fine-tuning. in other words, we show that multilingual ses can be 'rewired' into effective cross-lingual lexical encoders, as illustrated in figure 1 . this rewiring is again done via a quick and inexpensive contrastive learning procedure: with merely 1k-5k word translation pairs, we successfully convert multilingual ses into state-ofthe-art bilingual lexical encoders for any language pair (present in a specific dataset). 1 we probe the original lms and ses as well as demonstrate the usefulness of the proposed contrastive procedure for 'exposing' cross-lingual lexical knowledge on three standard lexical crosslingual tasks using standard evaluation data and protocols: bli, cross-lingual lexical semantic similarity (xlsim), and cross-lingual entity linking (xl-el). we show that the 'exposure' procedure is highly effective for both vanilla multilingual lms (mbert and xlm-r) and multilingual ses (labse and xmpnet): e.g., we observe ≈+10 precision@1 points gains on standard bli benchmarks @xcite . multilingual ses offer substantially better cross-lingual lexical performance than vanilla lms, both before and after being subjected to contrastive cross-lingual lexical fine-tuning (see figure 1 ). this indicates that it is possible to expose more cross-lingual lexical knowledge from multilingual ses than from their vanilla lm counterparts, likely owing to their additional exposure to parallel data. finally, inspired by @xcite , we validate that word vectors produced by cross-lingual lexical encoders (i.e., after the contrastive cross-lingual lexical 'exposure') can be effectively interpolated with static clwes @xcite and offer even stronger performance in cross-lingual lexical tasks. encouragingly, our cross-lingual lexical specialization of multilingual ses (as well as the further interpolation with static clwes), yields particularly massive performance gains for pairs of low-resource languages, as demonstrated on the low-resource bli benchmark @xcite . 1 note that this is a typical requirement of standard mapping-based approaches for learning static cross-lingual word embeddings which excel in the bli task @xcite @xcite . @xcite . a contrastive crosslingual lexical fine-tuning procedure (③) (requiring an external bilingual dictionary) can be applied on both ① and ②, yielding a fine-tuned cross-lingual lexical encoder (④). at inference, a word/phrase is encoded by the lexical encoder (⑤). in addition, its encoding can be interpolated with the corresponding static (cross-lingual) word embedding (⑥), producing the final embedding of the word/phrase (⑦). before the interpolation, static clwes must be mapped into the vector space of the lexical encoder (④): to this end, we learn the standard orthogonal (procrustes) projection matrix. the work combines two research threads, being inspired by the work on probing monolingual plms for lexical knowledge @xcite , as well as on interpreting representations in multilingual plms @xcite @xcite @xcite . previous work also tried to prompt multilingual lms for word translations via masked natural lan-guage templates @xcite and extract type-level word embeddings from lms (i) directly without context @xcite or (ii) by averaging contextual embeddings over a large auxiliary corpus in the target language @xcite . this existing body of work 1) demonstrated that even sophisticated templates and extraction strategies cannot outperform cross-lingual word embedding spaces (e.g., induced from monolingual fasttext vectors) in cross-lingual lexical tasks such as bli @xcite and 2) did not attempt to expose crosslingual lexical knowledge from multilingual ses and compare it against the (same type of) knowledge extracted from vanilla multilingual lms. ## experimental setup multilingual sentence encoders. we probe two widely used multilingual ses: 1) languageagnostic bert sentence embedding (labse) @xcite which adapts pretrained multilingual bert (mbert) (devlin et al., 2019) into a multilingual se; 2) multilingual xmp-net is a distillation-based adaptation (reimers and gurevych, 2020) of xlm-r (conneau et al., 2020) as the student model into a multilingual se, based on the monolingual english mpnet encoder (song et al., 2020) as the teacher model. labse is the current state-of-the-art multilingual se and supports 109 languages, while xmpnet is the best-performing multilingual se in the sentence-bert repository (reimers and gurevych, 2019): for further technical details regarding the models in our comparison, we refer to the original papers. along with labse and xmpnet as ses, we experiment with the original multilingual lms -mbert and xlm-r -using the the same training and evaluation protocols (see figure 1 and §2 ), aiming to quantify: (i) the extent to which cross-lingual lexical knowledge can be exposed from lms that have not been specialized for sentence-level semantics, as well as (ii) the increase in quality of lexical knowledge brought about with sentencelevel specialisation (i.e., when multilingual lms get transformed into multilingual ses). evaluation tasks. we evaluate on the standard and diverse cross-lingual lexical semantic tasks treated as 'cross-lingual lexical probes'. in other words, we fine-tune the models to steer them towards becoming better lexical encoders and then we check how well they fare across a set of representative (intrinsic) lexical tasks which could be seen as such 'lexical probes'. task 1: bilingual lexicon induction (bli), a standard task to assess the "semantic quality" of static cross-lingual word embeddings (clwes) @xcite , allows us to 1) directly assess the extent to which cross-lingual word translation knowledge can be exposed from multilingual lms and ses and 2) immediately test the ability to transform multilingual sentence encoders into bilingual lexical encoders. we run a series of bli evaluations on two standard bli benchmarks. 1) gt-bli @xcite , constructed semiautomatically from google translate, comprises 28 language pairs with a good balance of typologically similar and distant languages (croatian: hr, english: en, finnish: fi, french: fr, german: de, italian: it, russian: ru, turkish: tr). 2) panlex-bli @xcite focuses on bli evaluation for lower-resource languages, deriving training and test data from panlex @xcite . we evaluate on 10 pairs comprising the following five typologically and etymologically diverse languages: bulgarian (bg), catalan (ca), estonian (et), hebrew (he), and georgian (ka). standard bli setups and data are adopted: 5k training word pairs are used as seed dictionary d, and another 2k pairs as test data. note that d is used to (i) contrastively fine-tune multilingual encoders ( §2.1), (ii) learn the (baseline) static vecmap clwe space, as well as to (iii) learn the projection between the static clwe space and the representation spaces of multilingual encoders required to obtain the interpolated representations ( §2.2). the evaluation metric is standard precision@1 (p@1). 6 for panlex-bli, we also run experiments using smaller d, spanning 1k pairs. task 2: cross-lingual lexical semantic similarity (xlsim) tests the extent to which lexical representations can capture the (human perception of) fine-grained semantic similarity of words across languages. we use the comprehensive xlsim benchmark multi-simlex @xcite , which comprises cross-lingual datasets of 2k-4k scored word and phrase pairs over 66 language pairs. we evaluate on a subset of language pairs shared with the gt-bli dataset: en, fi, ru, fr. the evaluation metric is the standard spearman's rank correlation between the average of gold human-elicited xlsim scores for word pairs and the cosine similarity between their respective word representations. to avoid any test data leakage, we remove all xlsim test pairs from the bilingual dictionary d prior to fine-tuning and clwe mapping. task 3: cross-lingual entity linking (xel) is a standard task in knowledge base (kb) construction @xcite , where the goal is to link an entity mention in any language to a corresponding entity in an english kb or in a languageagnostic kb. 7 we evaluate on the cross-lingual biomedical entity linking (xl-bel) benchmark of @xcite : it requires the model to link an entity mention to entries in umls (bodenreider, 2004), a language-agnostic medical knowledge base. we largely follow the xl-bel experimental setup of @xcite and probe the encoders first without any additional task-specific fine-tuning on umls data, and then with subsequent umls fine-tuning (i) only on the en umls data, (ii) on all the umls data in 10 languages of the xl-bel dataset. 8 due to a large number of experiments, we again focus on the subset of languages in xl-bel shared with gt-bli: en, de, fi, ru, tr. ## results and discussion bilingual lexicon induction (bli). the behavior of ses, on the other hand, is much more favorable. labse, for example, surpasses baseline vecmap performance with interpolation alone, even without the contrastive lexical finetuning. when contrastively fine-tuned (and then interpolated with vecmap) both labse and xmp-net surpass the baseline vecmap performance by a much wider margin (+6.4 and +5.2, respectively). this implies that contrastive fine-tuning exposes more of the high quality cross-lingual lexical knowledge from multilingual ses. both (i) contrastive cross-lingual lexical learning (+cl) and (ii) interpolation with vecmap consistently improve the performance for all four encoders: we reach peak scores by combining contrastive fine-tuning and interpolation with static clwes (+cl (λ)). contrastive fine-tuning crucially contributes to the overall performance: compared to interpolation alone (nocl (λ)), +cl (λ) brings an average gain of over 6 bli points. in ## conclusion and future work we investigated strategies to probe and expose cross-lingual lexical knowledge from pretrained models, including multilingual language models (lms) and multilingual sentence encoders (ses). based on an extensive probing experiments on a suite of cross-lingual lexical tasks, we verified that multilingual ses (e.g., labse, xmpnet) are superior to multilingual lms (mbert, xlm-r) in terms of stored cross-lingual lexical knowledge. we empirically validated that the ses store more lexical knowledge than 'what meets the eye' when they are used off-the-shelf, but this knowledge must be exposed from them. to this end, we proposed new methods to further fine-tune their representations based on contrastive learning to 'rewire' the models' parameters and transform them from lms and ses into more effective cross-lingual lexical encoders. these lexical encoders yield gains for all underlying models, and are especially significant for resource-poor languages and in lowdata learning regimes. while this work focused on two widely used state-of-the-art multilingual ses, the contrastive framework is versatile and modelindependent and can be applied on top of other multilingual ses in future work. we will also investigate other more sophisticated contrastive learning this work has been partially supported by the erc poc grant multiconvai (no. 957356) and a huawei research donation to the university of cambridge. ivan vulić is supported by a personal royal society university research fellowship (no 221137; 2022-2027). fangyu liu is supported by grace & thomas c.h. chan cambridge scholarship. the work of goran glavaš has been supported by the multi2convai project of mwk baden-württemberg. the work of edoardo maria ponti's work has been supported by the facebook cifar ai chair program during his affiliation with mila -quebec ai institute. nigel collier kindly acknowledges grant-in-aid funding from esrc (grant number es/t012277/1). we would like to thank yaoyiran li for interesting insights and discussions. we also thank the three anonymous reviewers for their extremely substantial and in-depth reviews with a large number of helpful suggestions.