Mitigating the Language Mismatch and Repetition Issues in LLM -based Machine Translation via Model Editing (2024)

Abstract: Large Language Models (LLMs) have recently revolutionized the NLP field, while they still fall short in some specific down-stream tasks. In the work, we focus on utilizing LLMs to perform machine translation, where we observe that two patterns of errors frequently occur and drastically affect the translation quality: language mismatch and repetition. The work sets out to explore the potential for mitigating these two issues by leveraging model editing methods, e.g., by locating Feed-Forward Network (FFN) neurons or something that are responsible for the errors and deactivating them in the inference time.We find that directly applying such methods either limited effect on the targeted errors or has significant negative side-effect on the general translation quality, indicating that the located components may also be crucial for ensuring machine translation with LLMs on the rails.To this end, we propose to refine the located components by fetching the intersection of the locating results under different language settings, filtering out the aforementioned information that is irrelevant to targeted errors. The experiment results empirically demonstrate that our methods can effectively reduce the language mismatch and repetition ratios and meanwhile enhance or keep the general translation quality in most cases.

## introduction pre-trained large language models (llms) are natural machine translators with in-context learning @xcite @xcite @xcite , while they still fall behind specialized machine translation (mt) systems like nllb @xcite . previous studies utilize in-context learning @xcite (icl), instruction tuning @xcite and post-editing methods @xcite @xcite to improve the translation quality. one further question is: are there any specific issues that were ignored in previous studies hindering the llm-based machine translation from further development? in this work, we identify two issues in the llm-based machine translation: language mismatch and repetition (as shown in figure 1 ). we check the occurrence of these errors and find that: (1) they are common errors in the whole translation set (e.g., in the @xmath0, language mismatch occurs in over 40% cases with zero-shot prompting); (2) they are severe errors for machine translation systems (e.g., repetition errors usually lead to an over 50% bleu decrease compared with a standard generation). nonetheless, the inherent reason for these errors still remains unclear, let alone patching them. in recent research works on model editing @xcite @xcite , they typically leverage analyzing tools like causal mediation analysis @xcite , integrated gradient attribution @xcite to locate important component units (e.g., feed-forword network (ffn) neurons, attention heads and stuff) that are highly responsible for specific behavior patterns of llms, and then precisely control these behaviors by manipulating the located components (e.g., amplifying or suppressing the activation values of neurons). inspired by these works, we ask a research question: can we leverage model editing methods to mitigate aforementioned language mismatch and repetition issues? to explore the potential of model editing on mitigating these errors, we set out to adapt two widely-used model editing techniques, function vectors @xcite (fv) and knowledge neurons @xcite (kn), to mt scenarios in an aim to locate error-relevant component units inside llms. however, our empirical results show that directly adapting fvs and kns either has limited effect on the targeted errors or has significant side-effect on the general translation quality, which indicates that the located component units may be not only responsible for targeted error patterns but also crucial for ensuring machine translation with llms on the rails and hence directly manipulate them could result in affecting the general translation behavior. we then aim to filter out the error-irrelevant components from the located results. a possible hypothesis is that the location for the important errorrelevant modules is supposed to be independent of translation language settings. after comparing the locating results under the different translation language settings (@xmath1, @xmath2, @xmath3, we do observe that a proportion of located component units are shared across different language settings, which valid that the error-related components are highly corresponding to the mt rather than individual languages. grounded on this observation, we propose to refine the located components by fetching the intersection of the locating results under different language settings. the empirical results across different language settings demonstrate that the modified methods can effectively reduce the language mismatch and repetition ratios and meanwhile keep or enhance the general translation quality in most cases. our main contributions are three-fold: additionally, the performance of our methods could sometimes be comparable with traditional methods that adapt llms to mt tasks (e.g., 5-shot icl @xcite , lora @xcite and full-finetuning @xcite ) without additional requirements like long-context prompting or fine-tuning. besides, the proposed methods are compatible with the above techniques for further improvements. ## preliminary in this section, we detail the data preparation process, including the data source, prompt template, and dataset construction. additionally, we provide information about the model, the evaluation metrics used to support the ensuing experiments and the model editing methods used in this work. ## language mismatch and repetition in our initial experiments, we observe that llmbased machine translation struggles with the following two types of common errors. one is language mismatch, referring to the language of the translation result is not the target language. for example, in the @xmath4, the target language is chinese while the language of generated sentence is still english. another is repetition, referring to a substring is generated repeatedly until the end of the generation. to evaluate these errors, we additionally introduce two metrics: language mismatch ratio (lmr) (the percentage of cases occurring the language mismatch error) and repetition ratio (rr) (the percentage of cases occurring the repetition error). language mismatch and repetition error are common and crucial after detecting these errors, we first try to provide a quantitive analysis by analyzing the ratio of language mismatch and repetition error in zero-shot and one-shot. for detailed language settings, we consider @xmath5, @xmath6, @xmath7, and @xmath8. we utilize the d test and d exps as the test set and prompt examplar source, respectively. we choose lmr and rr to represent the ratio of language mismatch and repetition in a setting (e.g. @xmath9. for translation quality evaluation, we choose the bleu @xcite as the metrics since these errors can easily be detected on the word level with a sharp decrease on bleu or human check. we observe that language mismatch is frequent in zero-shot and seldom in one-shot. repetition error cases in one-shot are without language mismatch but combined with language mismatch in zero-shot. based on our observation, we do experiments and analysis in zero-shot for the language mismatch and in one-shot for the repetition error. to explore the relation between the above errors and translation quality, we split the translation results into four sets to evaluate the bleu performance after error detection. the four sets include two error sets: language mismatch set and repetition error set, one regular set (where instances without both errors), and one origin set that includes all cases. the results of in this section, we run all experiments by using the lang prompt @xcite @xcite as the default prompt template. currently, we notice that other prompt templates are used in llm-based mt research @xcite @xcite @xcite . to comprehensively explore these errors, we test other prompt templates with the same data and find these errors again. the only difference is the concrete ratio. this extension experiment further demonstrates the conclusion that language mismatch and repetition error are common and crucial. (detailed experimental setting and results can be found in appendix c) 5 can we mitigate language mismatch and repetition via model editing? in this section, we aim to investigate the potential for leveraging model editing methods @xcite @xcite to precisely mitigate the aforementioned two severe issues in mt: language mismatch and repetition. we mainly focus on two widely-used model editing methods: function vectors (fv) @xcite and knowledge neurons (kn) @xcite , for both of them are representative (i.e., causal mediation analysis @xcite bution @xcite for kn) and influential @xcite @xcite . in the following paragraphs, we adapt the idea of fv (corresponding to machine translation vectors) and kn (corresponding to machine translation neurons and repetition neurons) to mt scenarios, with an aim to both enhance the llms' understanding (for both language mismatch and repetition errors) and ability on mt by handling these errors. ## modifications to fv and kn in mt scenarios in section, we mainly discuss our modifications (section 6.1) to fv and kn methods (section 5) to release their potential for better mitigating the language mismatch errors, repetition errors and hopefully improving the general translation quality. besides, we present systematical evaluation results for the modified editing methods and baselines in section 6.2 to facilitate a deeper understanding of llm-based mt. ## conclusion in the work, we find that two types of errors, language mismatch and repetition, occur frequently when performing machine translation with llms, bringing severe negative effects on the translation quality. we investigate the potentials of leveraging model editing methods to mitigate these issues and find that directly adopting function vectors and knowledge neurons may either have limited improvement on the identified errors or bring noteworthy negative effect on the general machine translation quality (e.g., bleu score), which indicates that the located attention heads and ffn neurons might be too coarse to only affect the error ratios without hurting the translation quality. to this end, we propose to refine the located attention heads and neurons by fetching the intersection of the locating results under different language settings. our em-5 https://github.com/weichuanw/ llm-based-mt-via-model-editing pirical results suggest that the modified function vectors and knowledge neurons methods (mtv-i-d and rpn-i) can effectively reduce the aforementioned two types of errors and generally bring a positive influence evaluated with the translation quality metrics in most settings, indicating that there indeed exist a small set of modules that are highly responsible for the language mismatch and the repetition errors.