NAIST - NICT WMT ’23 General MT Task Submission (2023)

Abstract: In this paper, we describe our NAIST-NICT submission to the WMT’23 English ↔ Japanese general machine translation task. Our system generates diverse translation candidates and reranks them using a two-stage reranking system to find the best translation. First, we generated 50 candidates each from 18 translation methods using a variety of techniques to increase the diversity of the translation candidates. We trained seven models per language direction using various combinations of hyperparameters. From these models we used various decoding algorithms, ensembling the models, and using kNN-MT (Khandelwal et al., 2021). We processed the 900 translation candidates through a two-stage reranking system to find the most promising candidate. In the first step, we compared 50 candidates from each translation method using DrNMT (Lee et al., 2021) and returned the candidate with the best score. We ranked the final 18 candidates using COMET-MBR (Fernandes et al., 2022) and returned the best score as the system output. We found that generating diverse translation candidates improved translation quality using the well-designed reranker model.

## introduction we participated in the wmt'23 general machine translation task for english-to-japanese (en-ja) and japanese-to-english (ja-en) translation. our team aimed to improve translation performance using only the provided parallel data. our system generates diverse translation candidates and reranks them using a two-stage reranking system to find the best translation. figure 1 shows an overview of our system. we trained 7 transformer @xcite nmt models per language direction using various combinations of hyperparameters. the translation generator consists of 9 instances: 7 mt models, the ensemble model, and a knn-mt @xcite figure 1 : overview of our system. "e18-d4" denotes "18-layer encoder and 4-layer decoder", and "do" and "ado" denote "dropout" and "dropout after applying attention softmax", respectively. @xcite system that interpolates tokens from retrieved examples using the ensemble model. the generator generates the 50-best translations each from two decoding methods: beam search and topp sampling. this combination allows the generator to find diverse translation candidates. next, the 900 candidates (9 generators × 2 decoding methods × 50 best) are passed to our two-stage reranker to find the best translation. the first step of reranking uses drnmt @xcite to rerank the 50-best translation candidates to select the 1-best translation from each of the 18 generator and decoding method combinations. drnmt is trained to maximize the bleu @xcite score, whereas we use the second step reranking to find the highest comet @xcite score expectation from the remaining candidates. the 18 candidates from the first step are reranked using comet-mbr @xcite to select the best translation that is returned by the system. our experiments show that our two-stage reranker outperforms the bleu, chrf, and comet scores by drnmt alone, and the bleu and chrf scores by comet-mbr alone in both en-ja and ja-en translation tasks on wmttest2022 @xcite . ## preprocessing for the training data, we used the provided bilingual parallel data, which included jparacrawl v3 @xcite , news commentary v18.1, wiki titles v3, wikimatrix, the japanese-english subtitle corpus @xcite , the kyoto free translation task corpus @xcite , and the web inventory of transcribed and translated talks @xcite . we did not backtranslate the monolingual data due to resource constraints for training mt models and a reranker model. as the english translation of the japanese-english subtitle corpus was only available in lowercase, we trained a moses truecaser @xcite using the other corpora to add capitalization to the subtitle corpus. after truecasing, the first letter of each sentence was capitalized using detruecasing to produce sentence-case english text that matched the casing in the other corpora. we cleaned the data by removing duplicate lines and applying language filtering. because much of the training data were crawled from the internet, we used fasttext @xcite to predict the language of each sentence and removed sentences that were not predicted to be in the correct language. this helped to reduce noise in the dataset by removing sentences with garbage tokens. we tokenized text into subword units using sentencepiece @xcite . since our system generates many candidates using multiple models, we preliminary measured the generation speed and selected the number of vocabulary with the fastest decoding. our initial experiments demonstrated that when the target language was japanese, a vocabulary size of 32k resulted in fewer tokens needing to be generated, which increased the translation speed. however, when the target language was english, a vocabulary size of 16k was faster than an english vocabulary of 32k. therefore, we trained separate dictionaries #sentence pairs no filter 33,875,242 + deduplicate 29,940,444 ++ language filter 29,279,161 +++ length filter 27,880,378 for english and japanese, with the english-side dictionary containing nearly 16k tokens and the japanese-side containing nearly 32k tokens. the character coverage of the tokenizers also varied between languages. we trained the english tokenizer with 100% character coverage, whereas character coverage for japanese was 99.98%. after subword segmentation, we removed all sentences shorter than one token or longer than 250 tokens. we also removed all sentences in which the number of tokens in one language was more than double the number of tokens in the translation, i.e., the ratio of tokens between the source and target @xmath0.0. the number of sentence pairs before/after preprocessing is shown in ## experimental results we evaluated the translation performance of our system on wmttest2022 @xcite . we measured the bleu and chrf scores using sacrebleu, and the comet score using unbabel/wmt22-comet-da. generator were trained using fairseq @xcite . we used knn-seq 3 @xcite for knn-mt generation built on top of fairseq. the first stage of our reranker, drnmt, was also built using fairseq, whereas comet-mbr was built using comet @xcite . in addition, we investigated which hypothesis was selected as the system output in drnmt+comet-mbr. figure 3 shows the percentages of counts selected as the system output. in the figure, when the system output comes from multiple hypotheses, i.e., duplicated hypotheses are selected, each hypothesis is counted as selected. the results show that the hypotheses generated by beam search of the ensemble and ensemble+knn-mt models were selected as the system outputs roughly 40% in en-ja and 50% in ja-en. thus, half of the system outputs were not selected from hypotheses generated from the ensemble model using beam search. therefore, it can be said that "drnmt+comet-mbr" outperformed "drnmt" by selecting from the hypotheses generated by various generators and various decoding methods. ## conclusion in this paper, we described our submission as a joint team of naist and nict (naist-nict) to the wmt'23 general mt task. we participated in this task in the en-ja and ja-en translation directions. we built our system using a diverse translation generator and two-stage reranker. in future work, we will investigate qualitatively how translation diversity contributes to translation quality.