Simultaneous Translation and Paraphrase for Language Education (2020)

Abstract: We present the task of Simultaneous Translation and Paraphrasing for Language Education (STAPLE). Given a prompt in one language, the goal is to generate a diverse set of correct translations that language learners are likely to produce. This is motivated by the need to create and maintain large, high-quality sets of acceptable translations for exercises in a language-learning application, and synthesizes work spanning machine translation, MT evaluation, automatic paraphrasing, and language education technology. We developed a novel corpus with unique properties for five languages (Hungarian, Japanese, Korean, Portuguese, and Vietnamese), and report on the results of a shared task challenge which attracted 20 teams to solve the task. In our meta-analysis, we focus on three aspects of the resulting systems: external training corpus selection, model architecture and training decisions, and decoding and filtering strategies. We find that strong systems start with a large amount of generic training data, and then fine-tune with in-domain data, sampled according to our provided learner response frequencies.

## introduction machine translation systems are typically trained to produce a single output, but in certain cases, it is desirable to have many possible translations of a given input text. for example, duolingo-the world's largest language-learning platform-uses translation-based exercises for some of its lessons. for any given translation prompt there may be hundreds or thousands of valid responses, so we use a set of human-curated translations in order to grade learner responses. the manual process of maintaining these sets is laborious, and we believe it can be improved with the aid of rich multi-output translation and paraphrase systems. ## results a total of 20 teams participated during the dev phase, 13 teams during the test phase, and 11 teams submitted system description papers. of the teams with system descriptions, three of them (jbrem, sweagraw, jindra.helcl) participated in all five language tracks. one team (rakchada) submitted to two tracks, and the remaining teams only submitted to a single track, with japanese and portuguese being the most popular. official weighted f1 results are shown in ## meta-analyses in this section, we analyze different facets of the various approaches taken, in an effort to understand which design choices were most impactful on final results. we identified three major areas of variance: use of external training corpora ( §4.1), model architecture and training procedures ( §4.2), and decoding and filtering strategies ( §4.3). ## conclusion and future work we have presented the staple task, described a new and unique corpus for studying it, and reported on the results of a shared task challenge designed to explore this new domain. the task successfully drew participation from dozens of research teams from all over the world, synthesizing work in machine translation, mt evaluation, and automatic paraphrasing to name a few. we learned that a pipeline of strong machine translation followed by fine-tuning on learnerweighted staple data produces strong results. while the data for this task are geared toward language learners (and are therefore simpler than more commonly-studied domains such as newswire), it is our hope that the staple task provides a blueprint for ongoing interdisciplinary work in this vein. all task data, including dev and test labels, will remain available at: https://doi.org/10.7910/dvn/38ojr6