L ex M atcher: Dictionary-centric Data Curation for LLM -based Machine Translation (2024)

Abstract: The fine-tuning of open-source large language models (LLMs) for machine translation has recently received considerable attention, marking a shift towards data-centric research from traditional neural machine translation. However, the area of data collection for instruction fine-tuning in machine translation remains relatively underexplored. In this paper, we present LexMatcher, a simple yet effective method for data curation,the design of which is driven by the coverage of senses found in bilingual dictionaries. The construction process comprises data retrieval from an existing corpus and data augmentation that supplements the infrequent senses of polysemous words. Utilizing LLaMA2 as our base model, our method outperforms the established baselines on the WMT2022 test sets and also exhibits remarkable performance in tasks related to word sense disambiguation and specialized terminology translation. Our method is also applicable to other pre-trained models, and complements the method of continual pre-training using monolingual data, demonstrating the effectiveness of LexMatcher in enhancing LLM-based machine translation.

## introduction the emergence of large language models (llms) @xcite @xcite ) has brought about new opportunities for machine translation, and improving the translation performance of smaller-sized llms (7b or 13b) has attracted a lot of attention @xcite @xcite . unlike traditional neural machine translation (nmt) which relies heavily on abundant parallel data @xcite @xcite , llms have demonstrated less dependency * corresponding author on vast amounts of supervised data to achieve competitive performance. similar to other tasks by llms @xcite , the quality of fine-tuning data plays a more crucial role in nmt @xcite . current work primarily focuses on constructing fine-tuning data by leveraging human-written development sets, and creating refined instruction data for special purposes such as contrastive translation pairs and interactive translation @xcite . despite these advancements, these methods do not fully exploit the potentially valuable information embedded within the existing large parallel corpus. in fact, it has been demonstrated that fine-tuning llms with extensive parallel data can impair their inherent translation capabilities @xcite . furthermore, the quality of data distributions has been emphasized to have a more significant impact on the model performance than quantity alone @xcite , with more uniform data distributions contributing to improved generalization for unseen compositions @xcite . motivated by the above observations, we investigate a principled method, lexmatcher, for curating supervised fine-tuning data for llm-based translation. the objective is to collect a small yet carefully selected dataset that follows a proper distribution for maximizing translation quality. to this end, we leverage a bilingual dictionary as a pivotal resource to ensure comprehensive coverage of word or phrase senses in bilingual contexts. the construction of the dataset involves two steps: data retrieval and data augmentation. in the data retrieval step, we traverse commonly-used corpora (e.g., wmt training data) and identify sentence pairs that are guided by the coverage of dictionary senses. inevitably, however, there may be uncovered senses of polysemous words, representing long-tail knowledge essential for accurate translation. in the data augmentation step, we employ a commercial llm (e.g., chatgpt) to generate precise and concise sentence pairs that contain the uncovered senses. finally, we fine-tune llms using a combination of the retrieved and generated data. we conduct extensive experiments on six language directions including zh⇔en, en⇔de, and en⇔ru. by employing lexmatcher, we extract 0.1% of the wmt data, totaling 1 million samples across all six language directions. results of finetuned llms on the test sets show the superiority of our method over the baselines in both standard and zero-shot settings. the fine-tuned models also achieve comparable or better performance in terminology translation and translation disambiguation compared to the dedicated or commercial systems. further analyses of different data collection methods and composition generalization underscore the significance of high-quality data distributions, and the experiment of fine-tuning alma indicates the complementarity between our method and the monolingual data curation. the code, data, and models are available at https://github.com/arieslm/lexmatcher-mt.git. llms for mt. the usage of llm-based mt is significantly different from the conventional nmt. llms, particularly large ones like gpt-4, serve as interfaces that can perform translation with simple translation instructions or in-context learning (icl) @xcite @xcite . for icl, the influence of data selection methods on model performance is not significantly noticeable @xcite @xcite . finetuning smaller-sized llms such as llama @xcite for translation has garnered increasing attention @xcite , which has the potential to achieve an improved trade-off between quality and efficiency. tim @xcite constructs translation pairs for comparison and introduces an additional preference loss. bayling @xcite automatically generates interactive translation instructions. mao and yu (2024) construct an additional crosslingual discrimination task using word alignment for low-resource languages. @xcite finetune llms using more than 300 million parallel instances while @xcite indicate that such strategy could potentially impair the translation capabilities of llms. instead, they propose a twostage process that involves further post-training llms using a substantial amount of mixed monolingual data, followed by a subsequent step of finetuning with human-written parallel data. in line with the above efforts, we also aim to improve the open-source llms. the difference is that we propose specific parallel data collection methods, following the principle of achieving uniform coverage of semantic units in the dictionary. moreover, our approach achieves a better balance between efficiency and performance, and we can obtain a high-quality translation model using fewer computational resources compared to continual pretraining. bilingual dictionary for nmt. bilingual dictionaries have been employed to enhance translation quality, particularly for rare words or domainspecific entities. one approach involves augmenting the training data with pseudo-parallel sentences generated based on the dictionary. for example, @xcite al. (2020 ) enhance the parallel corpus with the help of paired entities extracted from multilingual knowledge graphs. hu et al. (2022) propose denoising entity pretraining for nmt using monolingual data and paired entities. these methods do not consult bilingual dictionaries for translation candidates during the inference stage. another approach involves leveraging bilingual alignments corpus data retrieval data augmentation missing senses of polysemous words instruction: given a word pair in english and chinese and its definition, generate sentence pairs that can accurately reflecting the meaning. example: …… rank lemmatize as lexical constraints @xcite @xcite . for llms, bilingual dictionaries have been used as a part of prompts @xcite for the llms of more than 100b. in contrast, we aim to improve llms' fine-tuning performance on translation tasks. the dictionaries serve as a pivot for data collection and can also be added in prompts when needed. ## method the overview of lexmatcher is illustrated in figure 1, which takes data retrieval ( §3.1) and data augmentation ( §3.2) steps for curating a compact parallel dataset for instruction fine-tuning. ## instruction fine-tuning llm for mt instruction fine-tuning has become standard practice in llm-based translation @xcite @xcite . our instructionfollowing data is constructed based on @xmath0. generally, each instance comprises an "instruction" c describing the task the model should perform (e.g., "translate the sentences from english to chinese."), an "input" x indicating the source sentence, and a corresponding output y indicating the answer to the instruction, i.e., the target sentence. the llms are optimized by minimizing the negative log-likelihood of the output y: we use two kinds of translation instructions: 1) general translation instructions mainly used to indicate translation directions (e.g., "translate the following sentence to english"), and 2) constrained translation instructions that specify word translations from a given dictionary or based on specific user requirements. (e.g., 'translate the following sentence to english using the given reference translations.") for the latter, we randomly sample a 2 gpt-3.5-turbo-0314 ## analysis 6.1 effect of k the maximal number of bilingual contexts of each matched sense is influenced by k. we show the performance of varying ks across different model sizes on the wmt22 test sets (figure 3 ). regard- ## conclusion in this paper, we presented lexmatcher, a dictionary-centric data curation method for supervised fine-tuning smaller-sized llms to better translation models. we use the bilingual dictionary as the pivot and try to collect limited parallel sentence pairs to cover the senses uniformly. experiments and analyses validate the effectiveness of lexmatcher from multiple perspectives including zero-shot translation, disambiguation, and terminology translation. one potential avenue for future research involves extending lexmatcher to lowresource scenarios, where the utilization of monolingual data is crucial for achieving satisfactory translation performance.