Overview of the 7th Workshop on A sian Translation (2020)

Abstract: This paper presents the results of the shared tasks from the 7th workshop on Asian translation (WAT2020). For the WAT2020, 20 teams participated in the shared tasks and 14 teams submitted their translation results for the human evaluation. We also received 12 research paper submissions out of which 7 were accepted. About 500 translation results were submitted to the automatic evaluation server, and selected submissions were manually evaluated.

## introduction the workshop on asian translation (wat) is an open evaluation campaign focusing on asian lan guages. following the success of the previous workshops wat2014wat2019 @xcite , wat2020 brings together machine transla tion researchers and users to try, evaluate, share and discuss brandnew ideas for machine transla tion. we have been working toward practical use of machine translation among all asian countries. for the 7th wat, we included the following new tasks: all the tasks are explained in section 2. wat is a unique workshop on asian language translation with the following characteristics: ## participants ## baseline systems human evaluations of most of wat tasks were conducted as pairwise comparisons between the translation results for a specific baseline system and translation results for each participant's sys tem. that is, the specific baseline system was the standard for human evaluation. at wat 2020, we adopted a neural machine translation (nmt) with attention mechanism as a baseline system. the nmt baseline systems consisted of pub licly available software, and the procedures for building the systems and for translating using the 22 https://github.com/tsuruoka-lab/bsd systems were published on the wat web page. 23 we also have smt baseline systems for the tasks that started at wat 2017 or before 2017. the base line systems are shown in tables 16 , 17 , and 18. smt baseline systems are described in the wat 2017 overview paper @xcite . the commercial rbmt systems and the online transla tion systems were operated by the organizers. we note that these rbmt companies and online trans lation companies did not submit themselves. be cause our objective is not to compare commercial rbmt systems or online translation systems from companies that did not themselves participate, the system ids of these systems are anonymous in this paper. ## human evaluation in wat2020, we conducted 2 kinds of human eval uations: pairwise evaluation (only for jaen multi modal translation task, section 6.1) and jpo ad equacy evaluation (other than hien multimodal translation task, section 6.2) and a pairwise varia tion of direct assessment (section 6.4) for the hien multimodal task. ## evaluation results in this section, the evaluation results for wat2020 are reported from several perspectives. some of the results for both automatic and human evalu ations are also accessible at the wat2020 web site. 48 the detailed automatic evaluation results are shown in appendix a. the detailed jpo adequacy evaluation results for the selected submissions are shown in ## findings 8.1 aspec task there is only one team (kyotou+ecnu) who par ticipated aspec task this year. kyotou+ecnu team participated japanese ↔ chinese translation subtasks. they achieved the stateoftheart auto matic evaluation scores, however, the human eval uation scores are below those of last year's (see fig ure 8 and 9 . strictly speaking, the human evalua tion scores of this year and last year are not directly comparable because the evaluators might be differ ent. they trained a lot of different nmt models which exploit 1) different training data (outof domain external data, back/forwardtranslation of japaneseside of aspecje), 2) different s2s frameworks (lstm, convs2s, transformer and ligntconv) and 3) different model capacities (dif ferent hyperparameter settings). they also tried to use mbart. among all the models, the ones which exploit data augmentation by back/forward translation performed the best. they also tried to combine various nmt models trained above. the bleu score improves a lot (about 0.5 to 1.5 points) by adding first 2 or 3 models, however, the impact is getting smaller (about 0.1 to 0.2 points improve ment) after that. from the results, we can say that using exter nal resources and combining various models both improves the automatic evaluation scores because of the generalization effect or improvement of flu ency, but they might have a bad effect on lexical choice of technical term translations which directly affect the human evaluation scores. ## conclusion and future perspective this paper summarizes the shared tasks of wat2020. we had 14 participants worldwide who submitted their translation results for the human evaluation, and collected a large number of use ful submissions for improving the current machine translation systems by analyzing the submissions and identifying the issues. for the next wat workshop, we will include several new datasets and new languages (arabic and additional indic languages). we are also plan ning to provide documentlevel test sets for some translation tasks because sentencelevel translation quality is almost saturated for some tasks. we also plan to have a new shared task named "restricted translation" task where we will investigate trans lation with restricted target vocabularies.