Domain-Specific Text Generation for Machine Translation (2022)

Abstract: Preservation of domain knowledge from the source to target is crucial in any translation workflow. It is common in the translation industry to receive highly-specialized projects, where there is hardly any parallel in-domain data. In such scenarios where there is insufficient in-domain data to fine-tune Machine Translation (MT) models, producing translations that are consistent with the relevant context is challenging. In this work, we propose leveraging state-of-the-art pretrained language models (LMs) for domain-specific data augmentation for MT, simulating the domain characteristics of either (a) a small bilingual dataset, or (b) the monolingual source text to be translated. Combining this idea with back-translation, we can generate huge amounts of synthetic bilingual in-domain data for both use cases. For our investigation, we used the state-of-the-art MT architecture, Transformer. We employed mixed fine-tuning to train models that significantly improve translation of in-domain texts. More specifically, our proposed methods achieved improvements of approximately 5-6 BLEU and 2-3 BLEU, respectively, on Arabic-to-English and English-to-Arabic language pairs. Furthermore, the outcome of human evaluation corroborates the automatic evaluation results.

## introduction neural machine translation (nmt) has the ability to produce good quality translations in terms of both fluency and adequacy @xcite . nevertheless, nmt still faces some challenges when it comes to translation of out-of-domain texts @xcite . domain adaptation of mt systems on in-domain parallel texts has been an active area of research to handle this situation. among popular contributions to the domain adaptation research, @xcite proposed to adapt an already existing nmt system to a new domain, with further training on the in-domain data only. in an effort to avoid overfitting on the in-domain data, @xcite employed the mixed fine-tuning approach, resuming training the baseline nmt model on a mix of in-domain and out-of-domain data. other researchers suggested adding domain tags to either the source or target sentences of the in-domain data, to inform the nmt model about the domain during training and decoding @xcite @xcite . in this sense, several research works on domain adaptation assume the availability of indomain data. however, in-domain data scarcity is common in translation settings, due to the lack of specialized datasets and terminology, or inconsistency and inaccuracy of available indomain translations. to tackle this problem, researchers have proposed diverse approaches, such as utilizing large monolingual datasets through selecting instances related to a given test ## related work in recent years, several pre-trained large lms have been made available to the research community, covering a wide range of linguistic tasks. among the state-of-the-art lms are gpt-2 @xcite , bert @xcite , roberta @xcite , xlnet @xcite , gpt-3 @xcite , electra @xcite , deberta @xcite , t5 @xcite , gopher @xcite , gpt-j @xcite , gpt-neox @xcite , palm @xcite , chinchilla @xcite , elmforest @xcite , mt-nlg @xcite , and opt @xcite . some of these models are multilingual, such as bloom (bigscience, 2022), alexatm @xcite and mgpt @xcite . using lms for specialized domains has been explored by previous works for diverse tasks. researchers explored the possibility to retrieve factual knowledge from lms in various domains @xcite . similarly, @xcite developed large-scale models of foundational scientific knowledge that can be effectively used to perform a wide range of in-domain and out-of-domain tasks. lms have been used in unsupervised nmt @xcite @xcite . large-scale pre-trained lms have also been employed in a variety of mt tasks, to improve the robustness of mt models or their ability to work on domain texts @xcite @xcite . recently, @xcite aimed at addressing the lack of training data for new application domains for data-to-text generation. they automatically augmented the data available for training by (a) generating new text samples by replacing specific values with alternative ones from the same category, (b) generating new text samples using gpt-2, and (c) proposing an automatic method for pairing the new text samples with data samples. their approach boosted the performance of a standard seq2seq model by over 5 bleu points. @xcite investigated the use of gpt-2 for source-side data augmentation to improve the robustness of a generic pre-trained nmt model. they first fine-tuned the pre-trained model, bert-fused @xcite , on authentic bilingual data. then, they augmented the english source with data generated by gpt-2. thereafter, they forward-translated the source-side english monolingual data with the fine-tuned version of bert-fused. finally, they fine-tuned the model on a combination of the authentic and synthetic data. while the reported results showed reasonable improvement (approx. 2.0 bleu points) for the english-to-japanese language direction, insignificant improvement (avg. 0.3 bleu) was achieved for both english-to-german and english-to-chinese language directions. the authors concluded that the result could be due to the relatively small amount of the original english-to-japanese data compared to the other two language directions. we conjecture that more factors might have led to this result, including using forward-translation (rather than back-translation) of a huge amount of data, due to the noise it introduces for the decoder @xcite . in our current work, we try to be more specific about the task description, focussing on domain adaptation in the absence of enough in-domain data; utilizing back-translation as an effective data augmentation technique @xcite ; and giving more attention to data distribution through applying approaches like mixed fine-tuning and oversampling @xcite . back-translation @xcite @xcite ) corresponds to the scenario where target-side monolingual data is translated using an mt system to give corresponding synthetic source sentences, the idea being that it is particularly beneficial for the mt decoder to see well-formed sentences @xcite . back-translation has become a popular strategy among mt researchers, especially in low-resource scenarios @xcite . @xcite performed a systematic study, which showed that forward-translation might lead to some improvements in translation quality, but not nearly as much as back-translation. @xcite concluded that forward-translation is more sensitive to the quality of the system used to produce synthetic data. compared to backtranslation, biases and errors in synthetic data are intuitively more problematic in forwardtranslation, since they directly affect the gold labels. the authors also reported that human evaluators favoured their back-translation systems over forward-translation systems, mostly in terms of fluency, while adequacy was largely the same across all of them, especially on the original translation direction. in their analysis, @xcite showed that sampling or noisy synthetic data gives a much stronger training signal than data generated by beam or greedy search. @xcite proposed a simpler alternative to noising techniques, consisting of tagging back-translated source sentences with an extra token. @xcite empirically showed that the quality of the back-translation system matters for synthetic corpus creation, and that nmt performance can be improved by iterative back-translation in both high-resource and low-resource scenarios. when it comes to fine-tuning strategies for mt domain adaptation, researchers demonstrated that applying the right data distribution can significantly mitigate catastrophic forgetting of strong baselines in domain adaptation settings. @xcite proposed the mixed fine-tuning method, whose training procedure is as follows: (a) train an nmt model on outof-domain data until convergence, and (b) resume training the nmt model from the first step on a mix of in-domain and out-of-domain data (by oversampling the in-domain data) until convergence. according to the authors, mixed fine-tuning can address the overfitting problem of regular fine-tuning. in addition, mixed fine-tuning does not worsen the quality of out-of-domain translations, while regular fine-tuning does. similarly, @xcite studied the problem in an adaptation setting where the goal is to preserve the existing system quality while incorporating data for domains that were not the focus of the original mt system. they found that they could improve over the performance trade-off offered by elastic weight consolidation @xcite with a relatively simple data mixing strategy. ## methods in this work, we investigate two scenarios of in-domain data scarcity, and propose approaches to leverage pre-trained lms for domain-specific data generation for mt training. ## results in this section, we elaborate on our automatic and human evaluations and discuss the results. as ## conclusion in this paper, we propose two simple methods to utilize pre-trained language models for domainspecific data augmentation for nmt systems. we report significant improvements, supported by both automatic and human evaluation. the proposed techniques enable the generation of large amounts of data, simulating the characteristics of the specialized text to be translated, and facilitating the domain adaptation process. for the arabic-to-english language direction, human evaluation demonstrates that setup 2 is on par with setup 1 even though in the former we did not have any authentic bilingual indomain data (cf. section 3). nevertheless, the english-to-arabic model in setup 2 has lower performance compared to the setup 1 model, although both setups outperform the baseline on the in-domain test set. we believe this might be due to the quality of synthetic data generated for arabic, which is an interesting aspect to explore further. in the future, we would like to investigate utilizing terminology for domain-specific data generation, and experiment with employing the same proposed approaches for low-resource languages and multilingual settings.