Topic-guided Example Selection for Domain Adaptation in LLM -based Machine Translation (2024)

Abstract: Current machine translation (MT) systems perform well in the domains on which they were trained, but adaptation to unseen domains remains a challenge. Rather than fine-tuning on domain data or modifying the architecture for training, an alternative approach exploits large language models (LLMs), which are performant across NLP tasks especially when presented with in-context examples. We focus on adapting a pre-trained LLM to a domain at inference through in-context example selection. For MT, examples are usually randomly selected from a development set. Some more recent methods though select using the more intuitive basis of test source similarity. We employ topic models to select examples based on abstract semantic relationships below the level of a domain. We test the relevance of these statistical models and use them to select informative examples even for out-of-domain inputs, experimenting on 7 diverse domains and 11 language pairs of differing resourcedness. Our method outperforms baselines on challenging multilingual out-of-domain tests, though it does not match performance with strong baselines for the in-language setting. We find that adding few-shot examples and related keywords consistently improves translation quality, that example diversity must be balanced with source similarity, and that our pipeline is overly restrictive for example selection when a targeted development set is available.

## introduction adaptation of neural machine translation (mt) models to unseen domains remains a difficult problem because it requires handling out-of-distribution data at inference @xcite . large language models (llms) offer an alternative method to the standard approach of fine-tuning an mt model or selected layers therein @xcite . openly available models such as llama-2 @xcite and explicitly multilingual models including bloom @xcite and xglm @xcite perform well crosslingually in classification and generation tasks, including many-to-many translation despite lacking explicit mt training. however, regardless of the choice of llm, some domains and vocabulary will remain under-exposed or unseen, especially for low-resource languages. additionally, the optimal use of llms at inference to enhance translation quality remains underexplored. domain adaptation of llm-based translation is therefore an open and persistent challenge. translation with llms requires prompting to elicit outputs in the desired language and domain, either via a zero-shot instruction or more effectively with in-context examples @xcite . in this work, we address the problem of domain adaptation at inference by exploring in-context example selection. selecting lexically, semantically or grammatically relevant translation examples for prompting llms is arguably more important when translating out-of-domain texts, to help fill gaps in domain vocabulary or demonstrate different styles. many works select examples randomly from a development set @xcite @xcite . however, other strategies have been developed. while some works show example diversity helps task performance @xcite , intuitively we expect that examples showing translations of words in or related to the test source will improve output quality. prior work has selected relevant examples based on n-gram overlap @xcite , feature matching @xcite or embedding similarity to the test source @xcite . here we test a method that exploits more abstract semantic relationships that are also more fine-grained than domain categories. for this we use a topic modelling pipeline @xcite that predicts a topic for a source sentence, and selects examples from this topic for translating in the given domain. the motivation is two-fold: first, we aim to test the continuing relevance of these simple models to complement llms for mt; and second, we aim to study the importance of semantic similarity for domain adaptation at a more abstract granularity than prior work, providing an alternative method for example selection. in practice we test llama-2-13b, a state-of-theart llm, on mt in varied domains including medical, legal, educational, religious, and entertainment texts. we test across several high and low-resource languages from and into english: french, german, czech, romanian, finnish, lithuanian, and tamil. we compare two uses of topic models for domain adaptation: topic-guided few-shot example selection and adding topic keywords. we test these against random baselines, information retrieval and embedding similarity-based selection, as well as simply adding domain labels. our standard method uses multilingual topic models to select examples or keywords from seen domains across all tested languages. we show that our topic-guided method is robust to unseen domains and outperforms strong baselines in this setting, but is too restrictive to achieve competitive results against baselines for simpler in-language tests, suggesting a trade-off between similarity and diversity of examples. recent work explores using pre-trained llms as a form of unsupervised transfer learning @xcite . many llms are competent in multilingual translation despite lacking explicit mt training @xcite @xcite , though llms often struggle in low-resource settings @xcite . for zero-shot translation, prompt design is key, with prior work improving translation with instructions @xcite , dictionary hints @xcite , chained bilingual dictionary entries @xcite , or chain-of-thought prompting to predict keywords, topics, and relevant examples @xcite . incontext learning, i.e. providing few-shot task examples, is effective for llm prompting @xcite , and various aspects of examples have been shown to impact translation quality: vilar et al. ( 2023 ) find example quality outweighs domain provenance or source similarity, while @xcite show semantic similarity correlates with improved performance, and @xcite show example diversity helps task performance more generally. prior example selection methods include using n-gram-based bm25 retrieval plus a reranking model @xcite , a regression model with manually defined features to score retrieved prompts @xcite , training a dense retrieval model @xcite , or selecting based on proximity to the test source in a pre-trained llm's embedding space @xcite . in this work we intend to achieve similar results using an alternative topic-guided selection method, permitting more abstract semantic relationships than n-gram overlap or embedding similarity. topic models are statistical tools that model latent semantic structure in texts @xcite , and while not state-of-the-art, these methods remain relevant for neural nlp. prior work has integrated topic models into neural mt architectures to improve translation performance @xcite , or fused external topic knowledge to improve domain robustness @xcite . @xcite study in-domain training data selection methods using unsupervised clustering methods based on pre- ## experiments data and preprocessing we select several diverse high and low-resource languages: czech (cs), german (de), english (en), french (fr), finnish (fi), lithuanian (lt), romanian (ro), and tamil (ta), both into and out of english. these languages vary from group 3 to 5 in joshi et al.'s (2020) taxonomy of language resourcedness. we test on 7 domains with data in most languages: medical european medicines agency texts (emea), transcribed ted talks @xcite , localisation files for kde4 software, educational video transcripts from qcri @xcite (qed), quran translations (tanzil), eu legal texts (jrc), and transcripts of tv and films from opensubtitles 2 @xcite . all data was 2 www.opensubtitles.org obtained from opus @xcite , and we release our data splits, topic models, and code to aid reproducibility and future research. 3 our preprocessing involves removing newlines and sentences over 175 tokens with moses scripts @xcite ); foot_1 removing sentences with over 50% punctuation; correct language identification using fasttext @xcite ; foot_2 and sentence-level deduplication. for each domainlanguage pair, our development and test sets consist of 5000 and 500 sentences respectively (n.b. tanzil tamil-english has only 4800 dev set sentences). development sets are used to train topic models and as sources for example selection. models we used the huggingface @xcite implementation of llama-2-13b, foot_3 with greedy decoding up to 256 tokens. this model is mainly english with substantial multilingual capabilities, and is a state-of-the-art open-source llm. we note however that the training data is not published, so our experiments are potentially at risk of data contamination. our results therefore can only be considered in the context of this specific model. while llama-2 is not as performant on translation tasks as significantly larger models such as gpt-3.5 @xcite , we chose llama-2 for our experiments because it outperforms similarly-sized, explicitly multilingual llms including xglm and bloomz models @xcite . llama-2 is also more robust to translation prompt perturbations than bloom models @xcite . further, llama-2's permissive licence and open-source weights are a significant benefit against api-only models, leading to substantial research interest such as llama-2-based translation models including alma @xcite . our translation research on llama-2 is therefore robustly motivated. we implement our topic modelling pipeline with bertopic (grootendorst, 2022). 7 our topic models are trained on parallel development sets; we focus on multilingual seen domain models, and also test in-language all-domain models. the multilingual setting is challenging and tests generalisation: without full domain coverage, methods must compensate, perhaps template prompt base l1: [source sentence] = l2: verbose given the following source text in l1: [source sentence], a good l2 translation is: label domain: [domain description] \n l1: [source sentence] = l2: keywords-10 related keywords: [keyword list] \n l1: [source sentence] = l2: fewshot (n) l1: [example source] = l2: [example target] \n l1: [source sentence] = l2: by using related examples from non-target languages. we expect the abstract cross-lingual semantic relationships identified by topic models to show robustness across domains. all models use embeddings from the 100-language paraphrase-multilingual-minilm-l12-v2 model from sentencetransformers @xcite , with umap dimensionality reduction @xcite and hdbscan clustering @xcite . our standard topic model has 500 topics trained on multilingual seen domains. further details and hyperparameters are available in appendix c. ## conclusion our method offers a lightweight, robust solution for when no parallel data is available for a new domain. however if suitable (in-domain and inlanguage) development data is available then information retrieval and embedding similarity-based methods are more performant, simpler solutions. in future work, we intend to assess the transferability of our method to llm-based translation more generally by testing across various llms, including more explicitly multilingual models. with this work we show an example of how statistical models can complement the performance of llama-2, an english-centric llm, in translation tasks to and from english.