I llinois J apanese \leftrightarrow E nglish N ews T ranslation for WMT 2021 (2021)

Abstract: This system paper describes an end-to-end NMT pipeline for the Japanese \leftrightarrow English news translation task as submitted to WMT 2021, where we explore the efficacy of techniques such as tokenizing with language-independent and language-dependent tokenizers, normalizing by orthographic conversion, creating a politeness-and-formality-aware model by implementing a tagger, back-translation, model ensembling, and n-best reranking. We use parallel corpora provided by WMT 2021 organizers for training, and development and test data from WMT 2020 for evaluation of different experiment models. The preprocessed corpora are trained with a Transformer neural network model. We found that combining various techniques described herein, such as language-independent BPE tokenization, incorporating politeness and formality tags, model ensembling, n-best reranking, and back-translation produced the best translation models relative to other experiment systems.

## introduction despite recent advances in machine translation made possible by neural networks with attention mechanism @xcite , the japanese-english pair remains a challenging language pair for machine translation systems to handle. challenges posed by this language pair are multifaceted, starting from seemingly trivial differences in orthographic representations to deep structural divergence in syntax. this paper describes an end-to-end neural machine translation system and related experiments dedicated to the news translation shared task where the target language pair is japanese ↔ english, as part of a submission to the sixth conference on machine translation -wmt 2021. in our experiments, we explored the efficacy of techniques such as tokenizing with language-independent and languagedependent tokenizers, normalizing by orthographic conversion, creating a politeness-andformality-aware model by implementing a tagger, back-translation, model ensembling, and n-best reranking. we found that normalizing the text by orthographic conversion did not improve over the baseline but controlling for politeness and formality levels of the text increased bleu by 1.2 points for the @xmath0, and other techniques such as backtranslation, model ensembling, n-best reranking also produced improvements. the paper gives a detailed review of prior work, with a particular focus on wmt 2020 submissions, and then proceeds to describe our data, model architecture, experiments, results, and discussion of their implications. ## prior work in this section, techniques and development in neural machine translation will be reviewed with a focus on the techniques and implementation most recently used for the japanese-english language pair. general techniques deployed across papers submitted to wmt 2020 are bitext data filtering, back-translation, fine tuning with in-domain data, knowledge distillation, rule-based reranking, transfer learning, co-reference processing, hyperparameter search, segmenting by subword units, bpe dropout, model ensembling, pre-training with monolingual data, experimenting with different word segmentation methods, context word embedding, domain adaptation, using related languages in joint training, domain tagging, reranking using backward and forward scores, and dual conditional cross-entropy filtering @xcite . in subsequent subsections, representative methods and techniques will be described and the impacts of these methods presented, in so far as they are applicable to the japanese-english pair. ## data our system was trained, developed, and tested fully on data provided by the wmt 2021 organizers, making it a constrained submission. details of the raw parallel corpora prior to substantial filtering foot_0 used in our baseline and experiment models can be viewed in we used the wmt 2020 development and test sets to compare various experiment models against the baseline: 1998 sentences in the development set in both directions, 1000 test sentences for the @xmath1, and 993 sentences for the @xmath2. from the raw datasets, we applied data filtering to remove noisy data based on two main criteria, alignment confidence and language identification. an alignment score is available for both jparacrawl and wikimatrix corpora; we chose 0.6 and 1.0 as the threshold for alignment confidence in jparacrawl and wikimatrix respectively. we used fasttext @xcite and its pre-trained language identification model to identify the language of our text sentence-by-sentence, and then we filtered sentence pairs where the language identification confidence score is less than 0.8. we also applied on-the-fly filtering of sentences longer than 100 tokens during training. according to @xcite , the kftt corpus contained instances of having japanese names followed by its english equivalent in parentheses, which caused their model to append english names after the japanese name in the translation output, for example キ ャ シ デ ィ • ス テ イ (cassidy stay). to avoid this, we filtered out english translations of names in japanese source text, specifically wikimatrix and kftt, so that any english names in parentheses following its japanese equivalent were removed. for english, we normalized punctuation and remove non-printing characters using the moses scripts @xcite . the amount of parallel training data after filtering was 12.7 m for training our submission models. ## model architecture we trained the parallel corpora using the transformer base and transformer big settings as described in @xcite , presented in ## conclusion we produced several models to tackle the task of translating japanese to english and english to japanese. namely, we have used bpe, employed a politeness and formality tagger, and during decoding, utilized model ensembling and n-best reranking. normalizing by orthographic conversion did not produce improvement compared to the baseline, but the other techniques have all proven to be effective and thus have been employed in our final submissions. we also found that for both @xmath3, adding back-translated data improved the results. this may be explained by the fact that there is very little parallel data in the news domain, and adding synthetic data from alternative in-domain sources helped tune the model. while improvement in the bleu score is modest for @xmath4, we expect the results to improve further if we increase the amount of back-translated data. we also showed that employing a tagger to introduce more contextual cues related to politeness and formality to our translation system is an effective technique. differences in formality and politeness levels present are issues often encountered when using training data in languages with rich honorifics. thus the technique employed in this paper could be extended to other languages such as korean.