Aligning Translation-Specific Understanding to General Understanding in Large Language Models (2024)

Abstract: Large Language models (LLMs) have exhibited remarkable abilities in understanding complex texts, offering a promising path towards human-like translation performance. However, this study reveals the misalignment between the translation-specific understanding and the general understanding inside LLMs. This understanding misalignment leads to LLMs mistakenly or literally translating some complicated concepts that they accurately comprehend in the general scenarios (e.g., QA). To align the translation-specific understanding to the general one, we propose a novel translation process, DUAT (Difficult words Understanding Aligned Translation), explicitly incorporating the general understanding on the complicated content incurring inconsistent understandings to guide the translation. Specifically, DUAT performs cross-lingual interpretation for the difficult-to-translate words and enhances the translation with the generated interpretations. Furthermore, we reframe the external tools to improve DUAT in detecting difficult words and generating helpful interpretations. We conduct experiments on the self-constructed benchmark Challenge-WMT, consisting of samples that are prone to mistranslation. Human evaluation results on high-resource and low-resource language pairs indicate that DUAT significantly facilitates the understanding alignment, which improves the translation quality (up to +3.85 COMET) and reduces translation literalness by -25% ∼ -51%.

## introduction recently, large language models (llms) have demonstrated remarkable language understanding and generation, paving the way for a higher level of performance in machine translation (zhao et al., 2023; openai, 2023; jiang et al., 2023; workshop, 1 the dataset is available at: challengewmt means corresponding author. ## approach: duat in this section, we first introduce our translation framework duat ( §3.1). specifically, duat consists of three components: difficult word detection ( §3.2), cross-lingual interpretation ( §3.3), and interpretation quality control ( §3.4). to make the llm follow the procedure of each component as expected, we adopt the in-context learning strategy and design an automatic method for constructing demonstrations of duat ( §3.5). ## testbed: challenge-wmt to better analyze the understanding misalignment problem of llms, we propose the benchmark challenge-wmt, which contains challenge sentences that are prone to mistranslation. this benchmark is constructed by collecting samples that multiple sota systems translate poorly from multiyear wmt testsets of six language pairs (chinese (zh), estonian (et), and icelandic (is) to/from english (en)). additionally, we believe that this dataset could promote future research in understanding the limitations of existing mt systems. we select three sota mt systems: google translate, chatgpt, and nllb @xcite . due to the poor performance of nllb in the zh⇔en translation, we additionally train a zh↔en translation model based on deltalm @xcite on the parallel corpus from opus foot_1 . next, all of the system translations are scored with comet metric, and the ρ of samples with the lowest score for each system are extracted as its difficult samples set. we vary the value of ρ across different language pairs to ensure an appropriate scale for each difficult sample set. finally, the intersection of all systems' difficult sample sets is taken as the challenge-wmt testbed. challenge-wmtcomprises around 600+ sentence pairs for each language pair, which is illustrated in tab.6. we equally split this dataset into the validation set and the test set. we report the translation performance measured by comet on challenge-wmt and the complete wmt set in fig. 8 , which shows that the performance decreases dramatically on challenge-wmt (84.5⇒73.6 averagely). next, we conduct a multi-aspect comparison for challenge-wmt and the complete set in appendix c, and find that the samples of challenge-wmt have higher perplexity (214⇒252 averagely). this result indicates that sentences in challenge-wmt are more complex. ## analysis 6.1 human evaluation to quantitatively analyze the understanding misalignment problem, we employ one senior human translator for each language direction to assess generalization failures and translation literalness. specifically, in each direction, we randomly sample 100 sentences foot_2 from challenge-wmt and ask the senior translator to annotate the mistranslated words and phrases (i.e., mistranslation) and score the literalness (1 to 5 score) of the translations generated by the strong baseline (gpt-3.5-turbo with icl). next, we ask the llm to generate interpretations of the mistranslated content. these interpretations are presented to the translator, who judges whether they contain the correct understanding of the content. if the interpretation is accurate, the content is annotated as a generalization failure. finally, the translator is asked to annotate the generalization failures of duatand score the literalness of the translations produced by duat. analysis of generalization failures. as the results shown in tab.4, generalization failures ac- analysis of translation literalness. as the results shown in tab.4, the baseline translations are highly biased towards literal translation. duatsignificantly reduces the bias towards literal translation, indicating that the process of interpreting the difficult words first and then translating aligns better with sense-for-sense translation. ## conclusion in this work, we propose a novel translation process, duat, to take the first step in resolving the misalignment between the translation-specific un-derstanding and the general understanding. furthermore, we utilize the token-level qe tool to enhance the detection of difficult words and the sentencelevel qe tool to remove harmful interpretations. human evaluation results on high-resource and lowresource language pairs indicate that duat significantly facilitates the understanding alignment, which improves the translation quality and alleviates translation literalness.