Exploiting Target Language Data for Neural Machine Translation Beyond Back Translation (2024)

Abstract: Neural Machine Translation (NMT) encounters challenges when translating in new domains and low-resource languages. To address these issues, researchers have proposed methods to integrate additional knowledge into NMT, such as translation memories (TMs). However, finding TMs that closely match the input sentence remains challenging, particularly in specific domains. On the other hand, monolingual data is widely accessible in most languages, and back-translation is seen as a promising approach for utilizing target language data. Nevertheless, it still necessitates additional training. In this paper, we introduce Pseudo- k NN-MT, a variant of k -nearest neighbor machine translation ( k NN-MT) that utilizes target language data by constructing a pseudo datastore. Furthermore, we investigate the utility of large language models (LLMs) for the k NN component. Experimental results demonstrate that our approach exhibits strong domain adaptation capability in both high-resource and low-resource machine translation. Notably, LLMs are found to be beneficial for robust NMT systems.

## introduction neural machine translation (nmt) has witnessed significant progress with the adoption of deep learning techniques @xcite , particularly the transformer model @xcite . despite these advancements, challenges still exist in translating uncommon words and adapting nmt systems to different domains @xcite . to tackle these challenges, researchers have proposed various methods to incorporate external knowledge into nmt. one such approach involves imposing constraints from terminology dictionaries @xcite ), * corresponding author. or the incorporating fuzzy matches retrieved from translation memories (tms) @xcite @xcite @xcite . these methods enhance nmt systems by leveraging bilingual knowledge. however, due to the limitations in the scale and domain coverage of bilingual data, it is highly challenging to find sentences that closely match the input sentence, especially in specific domains or for low-resource languages. one natural idea is to utilize the vast amount of monolingual data, which can provide a pool of highly relevant sentences in terms of meaning. as a promising method, back translation @xcite has been proven to be helpful for utilizing monolingual data in nmt systems. however, it requires additional training, including training a reverse nmt model and retraining an nmt model with the augmented training data. in this paper, we introduce pseudo-knn-mt, a training-free approach that leverages target language data for translation. specifically, given an input sentence, we retrieve its top-k similar target sentences using a cross-lingual retriever. our primary objective is to effectively utilize these retrieved sentences. initially, we pair the retrievals with the input sentence to create pseudo sentence pairs, then perform k-nearest neighbor machine translation (knn-mt) following @xcite . additionally, llms are known for their strong text compression capabilities of mapping texts into the representation space @xcite , and their training paradigm endows them with good generalization, which might be effective in handling low-frequency patterns. besides, llms also demonstrate strong translation capabilities @xcite @xcite . therefore, we take a step further to investigate the potential of utilizing llms for the knn component, where the datastore and the context representation vectors are derived from llms rather than the nmt model itself. furthermore, we explore the integration of llms with nmt without relying on target retrieval, focusing on leveraging the translation ability of llms and enhancing translation fluency. experimental results on multi-domain test sets demonstrate that our approach improves the translation results with a great margin, achieving an average improvement of 4.51 sacrebleu points. in low resource mt scenarios, our method's performance is comparable to or even superior to back-translation in certain domains. ## background in this section, we provide background information on knn-mt and cross-lingual retrieval. ## methodology in this section, we will introduce retrieving similar sentences from target dataset ( §3.1), as well as the proposed method of pseudo knn-mt ( §3.2) and the large language model integration ( §3.3 and §3.4) in detail. ## experiments in this section, we will introduce our experiments, including the main experiment, llm integration, low-resource machine translation and a comprehensive analysis from various perspectives. ## related works as a mature and widely known method, knn-mt @xcite has various variants. zheng et al. (2021) propose adaptive knn-mt, which can dynamically select k to avoid noisy neighbors. deguchi et al. (2023) introduce subset knn-mt, which speeds up inference by retrieving from a small subset based on source similarity. we also leverage subset retrieval while relying cross language similarity. wang et al. (2022) introduces cluster-based knn-mt, which adopts a compact network to prune feature datastore extremely. martins et al. (2022) introduces chuck-based knn-mt, which changes retrieve granularity from single tokens to chunks. dai et al. (2023) introduces a fast knn-mt method, which combines subset knn-mt and distance-aware λ together. liu et al. (2023) introduced knn-tl, which explores the combination of transfer learning method and knn-mt in low-resource scenarios. zhu et al. (2023b) introduces ink, a training framework that refines the representation space of an nmt model according to the extracted knn knowledge to avoid the high inference cost of the knn-mt. additionally, wang et al. (2023) explores how non-parametric knn-mt method can improve machine translation models at the fine-tuning stage. cao et al. (2023) introduces a method to address the gap between the upstream nmt model and downstream domains datastores, making knn-mt more suitable for downstream tasks by reconstructing datastore. ## conclusion in this paper, we propose pseudo-knn-mt to exploit target language data to nmt. experimental result show its strong domain adaptation capability on both high-resource and low-resource mt scenarios, validating the effectiveness of incorporating target monolingual data in the knn-mt. within this method, we employ a cross-lingual retrieval model to retrieve similar sentences from the target language dataset and pair them with the input sentences to construct pseudo-bilingual data, which is then used to build a key-value datastore. we also explore methods of incorporating llms to nmt from various perspectives and find that llms are beneficial for robust nmt systems.