NAIST - NICT - TIT WMT 22 General MT Task Submission (2022)

Abstract: In this paper, we describe our NAIST-NICT-TIT submission to the WMT22 general machine translation task. We participated in this task for the English ↔ Japanese language pair. Our system is characterized as an ensemble of Transformer big models, k-nearest-neighbor machine translation (kNN-MT) (Khandelwal et al., 2021), and reranking.In our translation system, we construct the datastore for kNN-MT from back-translated monolingual data and integrate kNN-MT into the ensemble model. We designed a reranking system to select a translation from the n-best translation candidates generated by the translation system. We also use a context-aware model to improve the document-level consistency of the translation.

## introduction we participated in the wmt22 general machine translation task in two language directions, english-to-japanese (en-ja) and japaneseto-english (ja-en). we built our system on an ensemble of transformer big models, k-nearestneighbor machine translation (knn-mt) @xcite , and reranking. figure 1 shows an overview of our system. our translation system is a combination of knn-mt and an ensemble of four transformer big models. we train each of the transformer nmt models using a different random seed, and pick one model as knn-mt. a notable point about our system is that we construct the datastore for knn-mt from back-translated monolingual data rather than reusing training data. we found that using back-translated data improves translation performance compared with using a parallel training corpus for the datastore. our reranker is designed to select the translation from the n-best translation candidates generated by the translation system. the reranker computes the weighted sum of each translation candidate across multiple models and selects the translation candidate with the highest score. we used k-best batch mira @xcite to select the weights for the model scores that maximize the bleu score of the development set. ## corpora and preprocessing for the training data, we used all the provided bilingual parallel data: jparacrawl v3 @xcite , news commentary v16 @xcite , wiki titles v3, wikimatrix, japanese-english subtitle corpus @xcite , the kyoto free translation task corpus, and ted talks. as the english translation of the japanese-english subtitle corpus is only available in lowercase, we trained a moses truecaser @xcite using the other corpora to add capitalization into the subtitle corpus. after truecasing, the first letter of each sentence was capitalized using detruecasing to produce sentence-cased english text to match the casing in the other corpora. we cleaned the data by removing duplicate lines and applying language filtering. as much of the training data was crawled from the internet, we used fasttext @xcite to predict the language of each sentence and removed the sentences that were not predicted to be the correct language. this has the effect of reducing the noise of the dataset by removing sentences with garbage tokens. we tokenized the text into subword units using a joint vocabulary size of 64,000, a character coverage of 99.98%, and byte fallback using sentencepiece @xcite . after subword segmentation, all sentences shorter than 1 token or longer than 250 tokens were removed. we also removed all sentences where the number of tokens in one language was more than double the number of tokens in the translation, i.e the ratio of tokens between the source and target @xmath0.0. after filtering, 27,784,519 sentence pairs remained for training. ## reranker our reranker selects one of the n-best translation candidates from the translation system. similar to other rerankers, it computes the weighted sum of multiple model scores for each translation candidate and selects the candidate with the maximum score. we used the average log-likelihoods of the words in each document as the model scores: the weights of the model scores were trained to maximize the bleu score of the development set. we used k-best batch mira @xcite to optimize the weights. ## conclusions in this paper, we described our submission of the joint team of naist, nict, and tit (naist-nict-tit) to the wmt22 general mt task. we participated in this task for the en ↔ ja translation. our system is built on an ensemble of transformer big models, knn-mt with using monolingual data, and k-best batch mira reranker. we would like to investigate each method and further improve translation performance.