On Bilingual Lexicon Induction with Large Language Models (2023)

Abstract: Bilingual Lexicon Induction (BLI) is a core task in multilingual NLP that still, to a large extent, relies on calculating cross-lingual word representations. Inspired by the global paradigm shift in NLP towards Large Language Models (LLMs), we examine the potential of the latest generation of LLMs for the development of bilingual lexicons. We ask the following research question: Is it possible to prompt and fine-tune multilingual LLMs (mLLMs) for BLI, and how does this approach compare against and complement current BLI approaches? To this end, we systematically study 1) zero-shot prompting for unsupervised BLI and 2) few-shot in-context prompting with a set of seed translation pairs, both without any LLM fine-tuning, as well as 3) standard BLI-oriented fine-tuning of smaller LLMs. We experiment with 18 open-source text-to-text mLLMs of different sizes (from 0.3B to 13B parameters) on two standard BLI benchmarks covering a range of typologically diverse languages. Our work is the first to demonstrate strong BLI capabilities of text-to-text mLLMs. The results reveal that few-shot prompting with in-context examples from nearest neighbours achieves the best performance, establishing new state-of-the-art BLI scores for many language pairs. We also conduct a series of in-depth analyses and ablation studies, providing more insights on BLI with (m)LLMs, also along with their limitations.

## introduction and motivation bilingual lexicon induction (bli), also known as word translation, is a fundamental research topic in multilingual nlp that aims to bridge the lexical gap between languages @xcite . it has a wide range of applications such as machine translation @xcite @xcite and cross-lingual transfer learning, especially for low-resource languages @xcite @xcite . over the past decade, state-of-the-art (sota) bli approaches have been predominantly supported by learning a cross-lingual word embedding (clwe) space, with which bli is tackled via nearest neighbour retrieval @xcite @xcite @xcite . meanwhile, autoregressive text-to-text large language models (llms) have emerged as the cornerstone of cutting-edge nlp research @xcite @xcite . for example, multilingual llms (mllms) have shown (sentence-level) machine translation capabilities @xcite ), although they have not been pretrained for machine translation in a supervised manner. motivated by the recent remarkable success of (m)llms, in this work we investigate 1) the potential of prompting and fine-tuning of mllms for bli and 2) how their capabilities compare against and complement current bli approaches. we focus on how to expose word-level bilingual knowledge and elicit word translations from multilingual llms. to our best knowledge, we are the first to leverage autoregressive mllms for bli. 1 we systematically study zero-shot and fewshot prompting for bli with off-the-shelf encoder-decoder and decoder-only autoregressive mllms @xcite @xcite , respectively. in the few-shot scenario, we propose to incorporate in-context examples from nearest neighbours into the prompts to boost the bli performance. in order to guide the mllms' generation, we hand-craft 'mask-fillingstyle' and 'gpt-style' templates catering to the characteristics of different llms and conduct extensive template search for bli. in addition to pro-viding a complete and effective pipeline for bli via prompting off-the-shelf mllms, we also investigate bli-oriented fine-tuning with the llms' own pretraining objectives, aiming at specialising mllms into 'few-shot word translators'. we conduct extensive experiments on two standard bli benchmarks, xling @xcite and panlex-bli @xcite , investigating the word translation capabilities of off-theshelf mllms (we adopt 18 models from 5 llm families) in various bli setups. our comprehensive comparisons between mllms confirm, as expected, that 1) different llm families display varying word translation capabilities and 2) stronger bli performance tends to be associated with larger model sizes. to demonstrate the effectiveness of our prompt-based approach, we benchmark our method against two sota clwe-based baselines. notably, our approach with llama 13b outperforms the clwe-based sota on the xling dataset by a considerable margin, establishing new sota results on many language pairs in all bli setups. meanwhile, we also identify two limitations of bli with mllms: 1) they are less competitive on the panlex-bli benchmark for lower-resource languages; 2) clwe-based approaches usually support more languages than mllms. finally, we run a series of insightful ablations and discuss the usefulness of bli-oriented fine-tuning. in short, our work validates the bli capabilities of mllms and proposes new methodology for bli. we hope that the combination of our comprehensive analyses and discussions, including on limitations, will pave the way for the development of stronger bli systems in the future. our code is publicly available at github.com/cambridgeltl/prompt4bli. more recently, researchers have attempted bli by leveraging encoder-only multilingual masked language models (mmlms) such as mbert @xcite and xlm-r @xcite whose neural architecture consists of only transformer encoders @xcite . @xcite prompt mbert with templates where the target word is replaced with a '<@xmath0, and the language modelling head of mbert outputs a subword token to fill the mask. this method is theoretically flawed because it cannot address the cases where the target word comprises two or more subword tokens. therefore, @xcite only evaluate bli on a small set of 'toy' examples rather than standard bli datasets. in terms of performance, this method lags far behind traditional bli approaches. a more successful way of leveraging mmlms is to extract decontextualised word representations from them @xcite . the strongest clwes for bli so far are learned via a two-stage contrastive approach combining both static (e.g., fasttext) and mmlmextracted features @xcite . 2 text-to-text llms. autoregressive llms have established new state-of-the-art results on many nlp tasks. the prominent model groups include 1) encoder-decoder llms such as bart @xcite and @xcite ; 2) ope-nai's decoder-only gpt series such as @xcite , gpt-3 @xcite , and instructgpt @xcite ; 3) other gptlike llms with specific improvements such as chinchilla @xcite , palm @xcite , and llama llm series @xcite . our work adopts five families of open-source text-to-text multilingual llms for bli, including mt5 @xcite , mt0 @xcite , xglm @xcite , mgpt @xcite , and llama @xcite . we introduce each of these in more detail in §3.1. unlike the encoder-only mlms, text-to-text llms are theoretically capable of generating words consisting of arbitrary numbers of subword tokens. we assume a set d s of n available seed translation pairs, constituting the so-called seed dictionary, which are used as the training set. depending on the number of training pairs, the task is usually referred to as supervised bli (typically, @xmath1, semi-supervised bli (e.g., 0 < @xmath2, and unsupervised bli (@xmath3. for convenience, we also refer to the unsupervised setup as zero-shot bli (@xmath4, corresponding to how we prompt mllms for bli (we describe zero-shot and few-shot prompts for bli later in §3). a test set d t , where d s ∩ d @xmath5, is used for evaluation. in some cases, a source word may have more than one ground-truth translation (i.e., there exist two or more word pairs in a bli dictionary that share the same source word). following previous work @xcite @xcite , we consider a prediction correct as long as it is any of the ground-truth translations. the bli scores are reported based on the standard precision@k (p@k) bli measure, where k denotes the length of the ranked list. ## experimental setup training and evaluation data. our experiments adopt two standard and publicly available bli datasets, also used in a body of very recent bli research @xcite @xcite @xcite . 1) xling @xcite provides bli dictionaries covering 8 languages and 56 bli directions. among these 8 languages, 5 are supported by all of our mllms: english (en), french (fr), german (de), italian (it), and russian (ru). therefore, §5 mainly focuses on and reports results on all the 20 = 5 × 4 bli directions for the 5 languages. 10 for each language pair, xling provides a test set d t of 2k translation pairs. it also provides training sets d s of 5k and 1k translation pairs, where the former is the superset of the latter. for brevity, we denote the cases |d s | = 5k, |d s | = 1k, and |d s | = 0 as the 5k setup, 1k setup, and unsupervised setup, respectively. foot_10 2) panlex-bli @xcite offers bli lexicons spanning 15 lower-resource languages and all 210 bli directions. we select three languages that are supported by most of our mllms: bulgarian (bg), catalan (ca), and hungarian (hu). the test set size of panlex-bli is also 2k; under the lower-resource assumption, we only focus on unsupervised and 1k bli setups. ## conclusion this paper presents the first study on bilingual lexicon induction (bli) with multilingual text-to-text large language models (mllms). we develop the methodology to prompt mllms for bli, conduct extensive template search, and systematically experiment with 5 representative mllm families (18 models) on a variety of zero-shot and few-shot bli tasks. relying on off-the-shelf mllms, our experiments on the standard xling dataset offer strong performance in all bli setups, where our proposed few-shot prompting with in-context examples from nearest neighbours outperforms the strongest clwe-based sota by a considerable margin. however, our study also points out that prompting-based methods still need to be successfully extended to lower-resource languages. finally, we conduct a series of in-depth analyses covering variants of our few-shot prompting and preliminary investigations on bli-oriented finetuning. our key findings and comprehensive analyses may pave the way for the development of stronger mllm-based bli systems in the future.