Self-Augmented In-Context Learning for Unsupervised Word Translation (2024)

Abstract: Recent work has shown that, while large language models (LLMs) demonstrate strong word translation or bilingual lexicon induction (BLI) capabilities in few-shot setups, they still cannot match the performance of ‘traditional’ mapping-based approaches in the unsupervised scenario where no seed translation pairs are available, especially for lower-resource languages. To address this challenge with LLMs, we propose self-augmented in-context learning (SAIL) for unsupervised BLI: starting from a zero-shot prompt, SAIL iteratively induces a set of high-confidence word translation pairs for in-context learning (ICL) from an LLM, which it then reapplies to the same LLM in the ICL fashion. Our method shows substantial gains over zero-shot prompting of LLMs on two established BLI benchmarks spanning a wide range of language pairs, also outperforming mapping-based baselines across the board. In addition to achieving state-of-the-art unsupervised BLI performance, we also conduct comprehensive analyses on SAIL and discuss its limitations.

## introduction and motivation the task of word translation (wt), also known as bilingual lexicon induction (bli), aims to automatically induce lexica of words with the same or similar meaning in different languages, thus bridging the lexical gap between languages. even in the era of large language models (llms), bli still has wide applications in machine translation and cross-lingual transfer learning @xcite @xcite @xcite . a particular bli setup, termed (fully) unsupervised bli, is especially compelling because it is not only more technically challenging but is also used as a pivotal component towards unsupervised machine translation @xcite @xcite . until recently, bli approaches have predominantly relied on learning cross-lingual word embedding (clwe) mappings: these are known as mapping-based approaches and are developed based on static or decontextualised word embeddings (wes) @xcite @xcite . meanwhile, autoregressive llms have become the cornerstone of modern nlp techniques @xcite @xcite with success in many real-world tasks @xcite @xcite . given this trend, recent bli research has also started to shift towards exploring llms. @xcite first show that prompting llms with gold-standard wt pairs as in-context examples (few-shot in-context learning: icl) outperforms all existing bli approaches in the supervised and semi-supervised bli setups (where typically 1k∼5k gold-standard wt pairs are available for training or icl), while zero-shot prompting still falls behind traditional mapping-based approaches in the fully unsupervised bli setup, especially for lower-resource languages. in this work, we thus aim at improving unsupervised bli with llms. to this end, we analyze the limitations of zero-shot prompting and propose a novel self-augmented in-context learning (sail) method for unsupervised bli with llms. the key idea is to first retrieve a set of high-confidence wt pairs by zero-shot prompting llms, then iteratively refine the high-confidence dictionary and finally use the gradually refined bilingual lexicon for bli inference in an icl fashion ( §2). our extensive experiments show that sail establishes new state-of-the-art unsupervised bli performance on two standard bli benchmarks. we also conduct thorough analyses on our approach, providing further insights into its inner workings ( §3- §4). our code is publicly available at https: //github.com/cambridgeltl/sail-bli . ## methodology unsupervised bli: task preliminaries. we assume a pair of two languages: a source language l x with its vocabulary x and a target language l y with vocabulary y. in a typical, standard bli setup the vocabulary of each language contains the most frequent 200, 000 word types in the language @xcite . given a source word w @xmath0, the unsupervised bli task then aims to infer its translation in l y , without any word-level parallel data (i.e., seed translation pairs from a lexicon) available for training or icl. 1 zero-shot prompting. @xcite have proposed to prompt autoregressive llms for the bli task, where the input word w x is embedded into a predefined text template. we adopt the pool of templates provided by @xcite and conduct template search for each llm on a randomly chosen language pair. as an example, the zero-shot template for llama-2 7b is as follows: foot_1 'the l x word w x in l y is:', where l x , l y , and w x are placeholders for the source language, target language, and the query word in the source language (e.g., l @xmath1, w @xmath2, l @xmath3. the deterministic beam search (with beam size of n as a hyper-parameter) is adopted to generate n output text pieces in the final beam, ranked by their sequence scores. 3 for each of the n outputs, the first word in the generated output following the input sequence is extracted as a candidate answer. after filtering out those candidate answers not in y, the candidate l y word with the highest associated sequence score is returned as the final word translation prediction. ## conclusion we proposed self-augmented in-context learning (sail) to improve unsupervised bli with llms. the key idea is to iteratively retrieve a set of highconfidence word translation pairs by prompting llms and then leverage the retrieved pairs as incontext examples for unsupervised bli. our experiments on two standard bli benchmarks showed that the proposed sail method substantially outperforms established mapping-based and zero-shot bli baselines. we also conducted a series of indepth analyses on the high-confidence dictionary, key hyper-parameters, and the back-translation mechanism, and we additionally show that our sail approach with llama-2 13b can even outperform zero-shot prompting with the state-of-the-art gpt-4 model.