Dynamic- FACT : A Dynamic Framework for Adaptive Context-Aware Translation (2023)

Abstract: “Document-level neural machine translation (NMT) has garnered considerable attention sincethe emergence of various context-aware NMT models. However, these static NMT models aretrained on fixed parallel datasets, thus lacking awareness of the target document during infer-ence. In order to alleviate this limitation, we propose a dynamic adapter-translator frameworkfor context-aware NMT, which adapts the trained NMT model to the input document prior totranslation. Specifically, the document adapter reconstructs the scrambled portion of the originaldocument from a deliberately corrupted version, thereby reducing the performance disparity be-tween training and inference. To achieve this, we employ an adaptation process in both the train-ing and inference stages. Our experimental results on document-level translation benchmarksdemonstrate significant enhancements in translation performance, underscoring the necessity ofdynamic adaptation for context-aware translation and the efficacy of our methodologies. Introduction”

## introduction numerous recent studies have introduced a variety of context-aware models aiming to effectively harness document-level context either from the source side @xcite @xcite @xcite , target side @xcite @xcite , or both @xcite @xcite @xcite . in the prevailing practice, a context-aware model remains fixed after training and is then employed for every testing document. nonetheless, this approach presents a potential challenge, as the model is required to encapsulate all translation knowledge, particularly from diverse domains, within a predefined set of parameters. accomplishing this task within the confines of reality poses a formidable undertaking. the "one sentence one model" approach for sentence-level nmt, as proposed by @xcite , aims to familiarize the model with each sentence in the test dataset by fine-tuning the nmt model for every testing sentence. however, acquiring suitable fine-tuning sentences for a given testing sentence proves to be highly time-consuming, as they require meticulous extraction from the bilingual training data through similarity search. this presents a significant challenge when attempting to replicate their methodology by seeking similar documents from the bilingual document-level training data. moreover, this approach assesses sentence similarity solely based on the levenshtein distance, thereby disregarding the document-level context of these sentences extracted from distinct documents. to address the potential challenge of employing a fixed, trained model for all testing documents, we propose the "one document one model" approach in this paper. this alternative approach aims to achieve the objective by introducing the document adapter. unlike other methods, the adapter relies solely on the input document itself and does not require additional input forms. its primary function is to reconstruct the original document from a deliberately corrupted version, thereby enabling the model to familiarize itself with the task of document-level translation. notably, this approach differs from previous methods where the input and output are in different languages, as opposed to the same language. following adaptation, this modified model is utilized to translate the document. figure 1 : the figure presented in this section depicts the adapter-translator architecture designed for context-aware neural machine translation. in this architecture, the encoder(s) are shared between the adapter denoted as ϕ and the translator denoted as ψ. it is important to note that the translator and adapter constitute two distinct stages within the same model, rather than being treated as separate models. both the adapter model and the nmt model employed in our study are context-aware and utilize shared encoder(s), while each having its dedicated decoder. in this paper, we present a training methodology that aims to adapt a pre-trained nmt model to a specific document through a process of alternating document reconstruction and document translation for each document batch. this approach is employed during both the training and inference stages. to evaluate the effectiveness of our proposed approach, we conducted experiments on three english-to-german document-level translation tasks. the results reveal significant enhancements in translation performance, providing strong evidence for the necessity of employing a one document one model approach and the efficacy of our proposed methodology. overall, we make the following contributions. ## adapter-translator architecture the adapter-translator architecture entails an iterative procedure involving an adaptation process denoted as ϕ and a translation process denoted as ψ. figure 1 presents a visual representation of the proposed architecture. the translator ψ, which is a context-aware nmt model, comprises context-aware encoder(s) and a decoder specific to translation. 1 the adapter shares the encoder(s) with the translator while possessing a decoder specifically designed for adaptation. given a source document x , the corpus processing script generates a deliberately corrupted version x of the document. this corrupted version is then utilized to optimize the adapter in order to reconstruct the scrambled segments of the original document x . as the encoder(s) are shared between the adapter and the translator, the capability to capture context during document adaptation can also be harnessed during the document translation process. the translation component of this architecture resembles that of other document-level translation models. ## application to various document-level nmt model to evaluate the effectiveness of our proposed framework in context-aware nmt, we select the following five representative nmt models: all of these models utilize a context encoder to encode global or local contexts, thereby improving document-level translation performance. to apply our proposed adapter-translator architecture to these models, we introduce an adapter decoder. we conduct experiments on english-to-german (@xmath0: talks, news, and speeches. additionally, we evaluate our proposed framework for the chinese-to-english translation task. we obtained the three document-level translation datasets from @xcite . 3 for the sourceside english sentences, we segmented them using the corresponding bpe model trained on the training data. meanwhile, for the target-side german sentences, we used the bpe model with 25k operations trained on the corresponding target-side data. model settings. for all translation models, we have set the hidden size to 512 and the filter size to 2048. the number of heads in the multi-head attention mechanism is 8, and the dropout rate is 0.1. during the training phase, we train the models for 100k steps using four a100 gpus, with a batch size of 40960 tokens. we employ the adam optimizer (kingma and ba, 2015) with β 1 = 0.9, β 2 = 0.98, and a learning rate of 1, incorporating a warm-up step of 16k. as for the fine-tuning stage, we fine-tune the models for 40k steps on a single a100 gpu, with a batch size of 40960 tokens, a learning rate of 0.3, and a warm-up step of 4k. during the inference phase, we set the beam size to 5. ## analysis and discussion in this section, we employ the chinese-to-english translation task as a representative to offer additional evidence for the efficacy of our proposed framework. in addition to reporting s-bleu, we also present case-insensitive document-level bleu (d-bleu) scores. ## related work local context has been extensively investigated in neural machine translation (nmt) models, including both rnn-based rnnsearch and transformer-based models @xcite ). an early attempt in rnn-based nmt was the concatenation method proposed by @xcite . subsequently, the adoption of multiple encoders emerged as a promising direction in both rnnsearch and transformer-based nmt models @xcite @xcite @xcite @xcite . cache/memorybased approaches @xcite @xcite also fall under this category, as they utilize a cache to store word/translation information from previous sentences. an alternative approach in document-level nmt treats the entire document as a unified translation unit and dynamically extracts pertinent global knowledge for each sentence within the document. this global context can be derived either from the source side @xcite @xcite or the target side @xcite . moreover, several endeavors have been undertaken to enhance the performance of document-level translation through the utilization of monolingual document data. for instance, in order to improve translation coherence within a document, @xcite propose docrepair, which is trained on monolingual target language document corpora to address inconsistencies in sentence-level translations. similarly, @xcite train a document-level language model to re-evaluate sentence-level translations. in contrast, dowmunt(2019) harness monolingual source language document corpora to investigate multi-task training using the bert-objective on the encoder. ## conclusion to enhance the alignment between the trained context-aware nmt model and each input document, we present in this paper an adapter-translator framework, designed to facilitate the model's familiarity with a document prior to translation. our modification to the nmt model involves incorporating an adapter encoder, which reconstructs the intentionally corrupted portions of the original document. empirical findings from chinese-to-english translation tasks and various english-to-german translation tasks demonstrate the considerable performance improvement achieved by our approach compared to several robust baseline models.