BJTU - W e C hat’s Systems for the WMT 22 Chat Translation Task (2022)

Abstract: This paper introduces the joint submission of the Beijing Jiaotong University and WeChat AI to the WMT’22 chat translation task for English-German. Based on the Transformer, we apply several effective variants. In our experiments, we apply the pre-training-then-fine-tuning paradigm. In the first pre-training stage, we employ data filtering and synthetic data generation (i.e., back-translation, forward-translation, and knowledge distillation). In the second fine-tuning stage, we investigate speaker-aware in-domain data generation, speaker adaptation, prompt-based context modeling, target denoising fine-tuning, and boosted self-COMET-based model ensemble. Our systems achieve 81.0 and 94.6 COMET scores on English-German and German-English, respectively. The COMET scores of English-German and German-English are the highest among all submissions.

## introduction we participate in the wmt 2022 shared task on chat translation in two language directions, @xmath0. in this year's chat translation task, we apply the two-stage training strategy. in the first stage, we investigate model architecture and data augmentation. in the second stage, we mainly focus on exploiting speaker-aware in-domain data augmentation, speaker adaptation, prompt-based context modeling, target denoising fine-tuning @xcite , and model ensemble strategies. this task aims to build machine translation systems to translate conversational text and thus supports fluent communication between an agent speaking in en-glish and a customer speaking in a different language (e.g., german), which is different from the first pre-training stage @xcite @xcite @xcite @xcite . therefore, we mainly pay attention to the second fine-tuning stage. in the first pre-training stage, we follow previous work @xcite @xcite and utilize several effective transformer variants. specifically, we combine the multi-head-attention @xcite , average attention transformer @xcite , and talking-heads attention @xcite , which have shown significant model performance and diversity. for data augmentation, we employ the backtranslation method to use the target-side monolingual data and apply the forward-translation to leverage the source-side monolingual data. to fully utilize the source-side of bilingual data, we use the sequence-level knowledge distillation method @xcite . in the second fine-tuning stage, for speakeraware in-domain data augmentation, based on the bcontrast @xcite dataset of the wmt20 chat translation task, we firstly adapt our pre-trained model to each speaker by using the speaker tag as a pseudo token and then apply it to the taskmaster-1 @xcite corpus to generate the speaker-aware in-domain data. for speaker adaptation, we follow previous work @xcite to prepend the corresponding speaker tag to each utterance on both the source and the target side to get a speakeraware dataset. for prompt-based context modeling, we exploit the prompt learning to incorporate the bilingual context and then apply the target denoising fine-tuning method @xcite to train our model. for the model ensemble, inspired by @xcite , we select high-potential can-didate models from two aspects, namely model performance (comet scores) and model diversity (self-comet scores among all candidate models). based on this, we design a search algorithm to gradually select the current best model of the model candidate pool for the final model ensemble. ## model architectures in this section, we describe the model architectures we used in two translation directions, where we mainly follow the previous state-of-the-art models @xcite . we also refer readers to read the paper for details. ## system overview in this section, we describe our system used in the wmt 2022 chat translation shared task, which includes two parts, namely general pre-training and in-domain fine-tuning. the pre-training part includes data filtering and synthetic data generation. the in-domain fine-tuning consists of speakeraware in-domain data generation, speaker adaptation, prompt-based context modeling, the target denoising fine-tuning @xcite , and boosted self-comet-based model ensemble. ## conclusions we investigate the pre-training-then-fine-tuning paradigm to build chat translation systems, which are some effective transformer-based architectures. our systems are also built on several popular data augmentation methods such as back-translation, forward-translation, and knowledge distillation. in the fine-tuning, we enhance our system by speaker-aware in-domain data generation, speaker adaptation, prompt-based context modeling, target denoising fine-tuning @xcite , and boosted self-comet-based model ensemble. our systems achieve 0.810 and 0.946 comet @xcite scores on @xmath1, respectively. these comet scores are the highest among all submissions.