HW - TSC ’s Submissions to the WMT 22 Word-Level Auto Completion Task (2022)

Abstract: This paper presents the submissions of Huawei Translation Services Center (HW-TSC) to WMT 2022 Word-Level AutoCompletion Task. We propose an end-to-end autoregressive model with bi-context based on Transformer to solve current task. The model uses a mixture of subword and character encoding units to realize the joint encoding of human input, the context of the target side and the decoded sequence, which ensures full utilization of information. We uses one model to solve four types of data structures in the task. During training, we try using a machine translation model as the pre-trained model and fine-tune it for the task. We also add BERT-style MLM data at the fine-tuning stage to improve model performance. We participate in zh \rightarrow en, en \rightarrow de, and de \rightarrow en directions and win the first place in all the three tracks. Particularly, we outperform the second place by more than 5% in terms of accuracy on the zh \rightarrow en and en \rightarrow de tracks. The result is buttressed by human evaluations as well, demonstrating the effectiveness of our model.

## introduction in recent years, machine translation quality has improved significantly with advances in model architecture @xcite @xcite , bilingual data availability, as well as data augmentation strategies @xcite @xcite @xcite @xcite . in scenarios where machine translation is used to facilitate understanding, machine translation outputs can basically satisfy audience's demands. however, in areas where translation quality is crucial (such as translating product manual, patent description, etc.), post-editing is required. techniques to improve post-editing efficiency are meaningful and necessary. researches @xcite @xcite in this regard falls into this category of computer-aided translation (cat). word-level auto completion, as a new task in wmt22, fall into this category as well. this task aims at auto-completing a target word given a source sentence, translation context, and a humantyped character sequence, so as to improve postediting efficiency. @xcite define the task in detail, offer comprehensive analysis and provide a baseline system. for this task, we choose the subword-level modeling strategy @xcite . comparing with word-level modeling, this strategy enables the usage of pre-trained models from other mainstream tasks, and solves the out-of-vocabulary (oov) issues at the same time. as human-typed input is just several characters of the target word, the input is not suitable for subword segmentation. we use character-level encoding instead. our final submission is an autoregressive model with bi-context, ensuring mixed encoding of characters and subwords. in view of the possible discontinuity between the context and human-typed inputs in the target-side text, we use tags to wrap the inputs, and then encode jointly with the context, in conjunction with the autoregressively decoded pre-token sequence. the joint coding maximizes the usage of information without introducing additional rnn @xcite or vocabulary reduction modules. during training, we use a standard machine translation model as the pre-trained model, and fine-tune it for this task. we then add bert-style mask language model (mlm) @xcite data in the fine-tuning stage to enhance the language model capabilities of the decoder, thereby improving the overall model performance. the inference mechanism is different from that of traditional nmt. in general, the entire decoder sequence must be used for encoding each token, @xmath0. context @xmath1. however, given the model's parallel ability and the short decoding sequence (the number of subwords in a word), this issue is not very serious for this task and this strategy is applicable for practical use. figure 1 shows the input representation of our model's decoder. we submit results for three language directions. all of them achieve the highest accuracy. our @xmath2, and get a good lead in human evaluation, demonstrating the effectiveness of our strategy. ## data processing @xmath3.0 (about 15.9m) and en↔de data comes from wmt14 (about 4.5m). the task allows the use of additional monolingual data, but we add no additional data on the zh-en track given the amount of bilingual data available. an additional 24m monolingual data is used for the en-de track, and the data comes from the wmt news task as well. we generate synthetic parallel data by sampling bt @xcite for the @xmath4. the specific reasons will be given later. we follow basic strategies to cleanse the data, including: deduplication, garbled character filtering, xml conversion, and fast-align @xcite , etc. the data sizes before and after data processing are shown in as for subword, we employ sentencpiece on @xmath5, and set the vocabulary size to 36k. on @xmath6, we use the bpe algorithm, and set the vocabulary size to 32k. ## system design in this chapter, we introduce the model structure, training strategy, inference strategy and corresponding data generation strategy used for this task. our model is based on the encoder-decoder architecture of the standard transformer. ## experiment during the experiment, we first build a baseline based on the mt model in order to measure our model's performance more accurately. after that, in the training process, we adopt several strategies to improve the model performance, that is, finetuning a mt model and introducing bert-style mlm data. validation and debugging of these strategies are done on the @xmath7. we use the finally determined strategy to train models for other tracks. ## result and analysis due to time restriction, we only conduct detailed comparison experiments on the @xmath8. en↔de tracks simply follows the final strategy we apply to the @xmath9. the results are shown in first of all, the performance of the mt baseline we trained is very poor, indicating that the mt task is not well adapted to the current task. so we give up the idea of using the mt results to enhance the model performance. after our constructed wlac data is added, the model performance improves by nearly 12 points, indicating the effectiveness of our strategy. but it is worth noting that the @xmath10. we assume that the quality of the st data is not good. in addition, the target-side of wlac data is confusing, resulting in training failure. so for the @xmath11, we directly generate bt data based on beam search to avoid the issue. after mlm data is added, we again obverse a significant improvement. the accuracy on the @xmath12.19%. the @xmath13, which was not converged at the previous stage, gains an accuracy of 81.79%. the results support our assumption that mlm data can enhance the language model ability of the decoder, while avoiding noise interference from the source-side text. in the end, model averaging leads to improvements on all three tracks, and the improvement is more significant than ensemble. ensemble leads to significant improvement on the @xmath14. we assume this is because the @xmath15. ## conclusion in this paper, we detail our team's participation in the wmt22 word-level autocomplete task. we first analyze the input and output, as well as challenges in this task. we notice that the modeling granularity of human-typed input and context information are different. therefore, we propose modeling human-typed input at character-level and modeling context information at subword-level, explicitly distinguishing and jointly encoding the two, thereby maximizing information usage in the encoding stage. at the same time, there is a semantic discontinuity between context and human-typed input. we add tags to differentiate the two. finally we propose an autoregressive model with bi-context to process four types of data at the same time. during training, we use an nmt model as the pre-trained model and fine-tune it for this task. bert-style mlm data is also introduced to improve the model performance, and at the same time to solve the single-direction decoding issue of the self-attention model. in the end, our models are well adapted to the task and gain safe leads in both automatic and human evaluations.