P he MT : A Phenomenon-wise Dataset for Machine Translation Robustness on User-Generated Contents (2020)

Abstract: Neural Machine Translation (NMT) has shown drastic improvement in its quality when translating clean input, such as text from the news domain. However, existing studies suggest that NMT still struggles with certain kinds of input with considerable noise, such as User-Generated Contents (UGC) on the Internet. To make better use of NMT for cross-cultural communication, one of the most promising directions is to develop a model that correctly handles these expressions. Though its importance has been recognized, it is still not clear as to what creates the great gap in performance between the translation of clean input and that of UGC. To answer the question, we present a new dataset, PheMT, for evaluating the robustness of MT systems against specific linguistic phenomena in Japanese-English translation. Our experiments with the created dataset revealed that not only our in-house models but even widely used off-the-shelf systems are greatly disturbed by the presence of certain phenomena.

## introduction the advancement of neural machine translation (nmt) has brought great improvement in translation quality when translating clean input, such as text from the news domain @xcite , and it was recently claimed that nmt has even achieved human parity in certain language pairs @xcite . despite its remarkable advancements, the applicability of nmt over user-generated contents (ugc), such as social media text, still remains limited @xcite . since ugc are prevailing in our real-life communication, it is undoubtedly one of the challenges we need to overcome to make mt systems invaluable for promoting cross-cultural communication. recently, with the increasing interest in handling ugc, a shared task was organized to measure how well mt systems adapt to those texts @xcite . however, the way in which they evaluate systems is just giving an overall score to a dataset, which is the same as traditional mt evaluation (figure 1a ). the overall score does not provide precise information for understanding what leads to the huge performance gap between the translation of clean input and that of ugc. to find a clue for improving the performance of mt systems on ugc, we need a solid basis for more detailed error analysis. as a first step towards a more refined evaluation of mt systems on ugc, we present a new dataset, phemt: phenomenon-wise dataset for machine translation robustness, designed for phenomenonwise evaluation in japanese-english translation (figure 1b ). more specifically, we create a set of datasets, each of which provides a focused evaluation on one of four linguistic phenomena commonly seen on ugc, i.e., proper noun, abbreviated noun, colloquial expression and variant. by focusing locally on a specific part of a sentence presenting one of the above phenomena, we directly measure the ability of mt systems to handle the phenomenon with the help of translation accuracy. moreover, based on the idea of contrastive datasets, we normalize targeted expressions to its canonical form in the dictionary. we feed both original and normalized versions of a source sentence to obtain the difference of arbitrary metrics as our robustness measure. using our dataset, we analyze the strengths and weaknesses of current nmt systems from the point of available training data size and the way of tokenization. we reveal that some of the phenomena are severely problematic even to widely used, strong off-the-shelf systems. ## translation models we prepared five in-house models with different size of training data and preprocessing methods for our experiments. the smaller model (small) was trained on the data offered in the wmt 2019 shared task for machine translation robustness, namely ted talks, kftt (kyoto free translation task), and jesc (japanese-english subtitle corpus). the mtnt dataset was also available in the task, but we didn't include any of the sentence pairs to train our models. we replaced emojis and emoticons with placeholders following a previous study by @xcite . in addition, we replaced possible usernames with regular expressions. we offered this model to see whether or not the phenomena would become less problematic with increasing amount of training data. for the other four models, we additionally used jparacrawl v2.0 @xcite , one of the largest parallel corpora available in japanese-english. the larger model (large), is only different in the size of training data from the small. we applied byte-pair-encoding (bpe) models @xcite with a joint vocabulary of 32,000 for these models using the sentencepiece toolkit @xcite . the character-based model (char) is different from the two models in the way of segmentation. the model translates a sequence of characters in the source language into another sequence of characters in the target language @xcite . @xcite pointed out that character-based models are more robust to noisy text than bpe-based models. we revisit the issue of segmentation to see if the model is also good at handling ugc. we shared the vocabulary between two languages in this setting as well to expect the model to capture copying behavior. for the pronunciation-based model (pron), we applied a unique preprocessing method to the source (japanese) sentence. more specifically, we first applied the mecab toolkit @xcite , a japanese morphological analyzer, with naist-jdic for the dictionary to obtain the pronunciation of each morpheme in the sentences. we can transliterate any words in japanese by using phonetic symbols such as hiragana and katakana characters. since the mecab toolkit outputs the pronunciation in katakana characters by default, we simply concatenated them to create a fully pronunciation-based corpus. we prepared this model with the expectation to improve the robustness against variant expressions. more specifically, we aimed at absorbing the orthographic variations caused by hiragana-katakana confusion, which is a part of variant. also, previous study suggests that phonetic information is highly useful to resolve homophone noise @xcite . finally, we prepared the concatenated model (cat), trained on a joined corpus for the large and the pron. 5 in this setting, we converted the transliterated part into hiragana characters and applied the same bpe model as used in the large to the whole corpus. we expect the model to learn robust representations by forcing it to produce the same translation from the original source sentence and its transliterated counterpart. we used transformer-base architecture @xcite implemented in the fairseq toolkit @xcite and hyperparameters proposed by @xcite for all models. the size of the training data was 3.9 m for the small, 14.0 m for the large, char and pron, and 28.0 m for the cat. in addition to the in-house models, we also investigated the impact of the phenomena on two widely used mt systems, namely, google translate foot_5 and deepl translator. 7,8 these systems are expected to be more robust against ugc because they are by nature exposed to user input. by conducting experiments on such systems, we reveal the presence of phenomena with impending needs for improvement, and also confirm the usefulness of normalization as one of the tricks users can do. ## phenomenon-wise evaluation we provide an overview of the current state of nmt by evaluating the performance of both in-house models and off-the-shelf systems on the proposed phenomenon-wise dataset. we fed both the original and normalized sentences to the models and measured the difference of single reference bleu between them. since the only difference between the two sentences is the presence of the corresponding phenomenon, our dataset ensures that a phenomenon degrades the models more significantly if there is a ## correlation with human evaluation to demonstrate a potential use case of our phenomenon-wise dataset, we conducted an additional experiment, where we reassessed the systems submitted to the wmt 2019 robustness shared task in a phenomenon-wise manner. we downloaded five official submissions for the blind test foot_9 portion of the mtnt dataset. 11 we then extracted the intersection between the blind test data and our phenomenonwise dataset, obtaining 136 sentences for proper noun, 48 for abbreviated noun, 21 for colloquial expression, and 11 for variant. the task organizer also provides the results of human judgment of all submissions for each sentence @xcite , where three human raters were instructed to rate each translation on a scale from 1 (completely incorrect) to 100 (accurate). for each of the five submissions, we averaged all the human ratings for each sentence in the phenomenon subset and investigated the correlation between the averaged human ratings and the phenomenon-wise accuracy. from the results in figure 4 , we could see that the accuracy for our proper noun and abbreviated noun dataset strongly correlated to the human judgment scores with @xmath0.9. this is worth surprising because we have no access to the whole sentences but only to the targeted expressions in our accuracy-based method. the result suggests that the two phenomena are key factors for humans in evaluating overall translation quality. one reason might be that humans can easily tell whether the translated sentences include these nouns or not. this implies that undertranslation of words for these two phenomena could bring a more serious impact on human judgment. we believe that the accuracy can be used as a strong signal for estimating human judgment scores, when combined with traditional evaluation metrics such as bleu. ## conclusion we proposed a novel dataset designed for phenomenon-wise evaluation in japanese-english translation. in this research, we focused on four linguistic phenomena commonly seen on user-generated contents, namely proper noun, abbreviated noun, colloquial expression, and variant. using our dataset, we analyzed how current mt systems are negatively affected by the presence of the phenomena. the result showed that variant is one of the phenomena that significantly degrade the model's performance including widely used, strong off-the-shelf systems. this implies that collecting massive training corpora is not a sufficient condition to handle these peculiar inputs, and we need special treatment against them to further improve mt systems. we also analyzed the correlation between human judgment and translation accuracy scores from our dataset by using official submissions from the wmt 2019 shared task. from the experiments, we confirmed that the accuracy-based scores from our dataset strongly correlated with human judgment, showing its potential to reduce the cost of the evaluation. we made our dataset publicly available for further development in mt systems foot_11 . as future work, we would like to consider new model architectures or data preprocessing methods to improve performance against specific phenomena using our dataset. score definition 5 translations that conveys the meaning completely and fluent as target language sentence 4 translations that does not show any lack of information, but highly translationese (verbatim) 3 translations that has locally untranslated / mistranslated parts, but accep