Scheduled Multi-task Learning for Neural Chat Translation (2022)

Abstract: Neural Chat Translation (NCT) aims to translate conversational text into different languages. Existing methods mainly focus on modeling the bilingual dialogue characteristics (e.g., coherence) to improve chat translation via multi-task learning on small-scale chat translation data. Although the NCT models have achieved impressive success, it is still far from satisfactory due to insufficient chat translation data and simple joint training manners. To address the above issues, we propose a scheduled multi-task learning framework for NCT. Specifically, we devise a three-stage training framework to incorporate the large-scale in-domain chat translation data into training by adding a second pre-training stage between the original pre-training and fine-tuning stages. Further, we investigate where and how to schedule the dialogue-related auxiliary tasks in multiple training stages to effectively enhance the main chat translation task. Extensive experiments on four language directions (English-Chinese and English-German) verify the effectiveness and superiority of the proposed approach. Additionally, we will make the large-scale in-domain paired bilingual dialogue dataset publicly available for the research community.

## introduction a cross-lingual conversation involves speakers in different languages (e.g., one speaking in chinese and another in english), where a chat translator can be applied to help them communicate in their native languages. the chat translator bilaterally converts the language of bilingual conversational text, e.g. from chinese to english and vice versa @xcite @xcite . generally, since the bilingual dialogue corpus is scarce, researchers @xcite @xcite resort to making use of the large-scale general-domain data through the pretraining-then-fine-tuning paradigm as done in many context-aware neural machine translation models @xcite @xcite @xcite @xcite @xcite , having made significant progress. however, conventional pre-training on large-scale generaldomain data usually learns general language patterns, which is also aimless for capturing the useful dialogue context to chat translation, and fine-tuning usually suffers from insufficient supervised data (about 10k bilingual dialogues). some studies @xcite @xcite @xcite have shown that learning domain-specific patterns by additional pre-training is beneficial to the models. to this end, we firstly construct the large-scale in-domain chat translation data 2 . and to incorporate it for learning domain-specific patterns, we then propose a three-stage training framework via adding a second pre-training stage between general pre-training and fine-tuning, as shown in fig. 1 . to further improve the chat translation performance through modeling dialogue characteristics (e.g., coherence), inspired by previous studies @xcite @xcite , we incorporate several dialogue-related auxiliary tasks to our three-stage training framework. unfortunately, we find that simply introducing all auxiliary tasks in the conventional multi-task learning manner does not obtain significant cumulative benefits as we expect. it indicates that the simple joint training manner may limit the potential of these auxiliary tasks, which inspires us to investigate where and how to make these auxiliary tasks work better for the main nct task. to address the above issues, we present a scheduled multi-task learning framework (sml) for nct, as shown in fig. 1 . firstly, we propose a three-stage training framework to introduce our constructed in-domain chat translation data for learning domain-specific patterns. secondly, to make the most of auxiliary tasks for the main nct task, where: we analyze in which stage these auxiliary tasks work well and find that they are different strokes for different folks. therefore, to fully exert their advantages for enhancing the main nct task, how: we design a gradient-based strategy to dynamically schedule them at each training step in the last two training stages, which can be seen as a fine-grained joint training manner. in this way, the nct model is effectively enhanced to capture both domain-specific patterns and dialogue-related characteristics (e.g., coherence) in conversation, which thus can generate better translation results. we validate our sml framework on two datasets: bmeld @xcite (en↔zh) and bcon-trast @xcite ) (en↔de). experimental results show that our model gains consistent improvements on four translation tasks in terms of both bleu @xcite and ter @xcite scores, demonstrating its effectiveness and generalizability. human evaluation further suggests that our model can produce more coherent and fluent translations compared to the previous related methods. our contributions are summarized as follows: we introduce the conventional multi-task learning framework @xcite for nct, which includes four parts: problem formalization ( § 2.1), the nct model ( § 2.2), existing three auxiliary tasks ( § 2.3), and training objective ( § 2.4). ## scheduled multi-task learning for nct in this section, we introduce the proposed scheduled multi-task learning (sml) framework, including three stages: general pre-training, indomain pre-training, and in-domain fine-tuning, as shown in fig. 1 . specifically, we firstly describe the process of in-domain pre-training ( § 3.1) and then present some findings of conventional multi-task learning ( § 3.2), which inspire us to investigate the scheduled multi-task learning ( § 3.3). finally, we elaborate on the process of training and inference ( § 3.4). ## conclusion this paper proposes a scheduled multi-task learning framework armed with an additional in-domain pre-training stage and a gradient-based scheduled multi-task learning strategy. experiments on en↔zh and en↔de demonstrate that our framework significantly improves translation quality on both bleu and ter metrics, showing its effectiveness and generalizability. human evaluation further verifies that our model yields better translations in terms of coherence and fluency. furthermore, we contribute two large-scale in-domain paired bilingual dialogue datasets to the research community. // www.zimutiantang.com/ and then manually post-edited them according to the dialogue history by native chinese speakers who are postgraduate students majoring in english. finally, following @xcite , they assume 50% speakers as chinese speakers to keep data balance for @xmath0. for the chinese, we follow them to segment the sentence using stanford corenlp toolkit 14 .