Growing Together: Modeling Human Language Learning With n-Best Multi-Checkpoint Machine Translation (2020)

Abstract: We describe our submission to the 2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE). We view MT models at various training stages (i.e., checkpoints) as human learners at different levels. Hence, we employ an ensemble of multi-checkpoints from the same model to generate translation sequences with various levels of fluency. From each checkpoint, for our best model, we sample n-Best sequences (n=10) with a beam width =100. We achieve an 37.57 macro F1 with a 6 checkpoint model ensemble on the official shared task test data, outperforming a baseline Amazon translation system of 21.30 macro F1 and ultimately demonstrating the utility of our intuitive method.

## introduction machine translation (mt) systems are usually trained to output a single translation. however, many possible translations of a given input text can be acceptable. this situation is common in online language learning applications such as duolingo, 1 babbel 2 , and busuu. 3 in applications of this type, learning happens via translation-based activities while evaluation is performed by comparing learners' responses to a large set of human acceptable translations. figure 1 shows an example of a typical situation extracted from the duolingo application. the main set up of the 2020 duolingo shared task on simultaneous translation and paraphrase for language education (staple 2020) @xcite is such that one starts with a set of english sentences (prompts) and then generates highcoverage sets of plausible translations in the five 1 https://www.duolingo.com/ 2 https://www.babbel.com/ 3 https://www.busuu.com/ figure 1 : translations proposed by english language learners at various levels of fluency, from diverse backgrounds. our multi-checkpoint ensemble models mimic learner fluency. 4 target languages: portuguese, hungarian, japanese, korean, and vietnamese. for instance, if we want to translate the english (en) sentence "is my explanation clear?" to portuguese (pt), all the translated portuguese sentences illustrated in the remainder of the paper is organized as follows: section 2 is a brief overview of related work. in section 3, we describe the data we use for both training and fine-tuning our models. section 4 presents the proposed mt system. section 5 describes our different methods. we discuss our results in section 6, and conclude in section 7. ## related work we focus our related work overview on the task of paraphrase generation and its intersection with machine translation. paraphrasing is the task of expressing the same textual units (e.g. sentence) with alternative forms using different words while keeping the original meaning intact. 5 over the last few years, mt has been the dominant approach for paraphrase generation. for instance, @xcite ; @xcite use multiple translations of the same text to train a paraphrase system. similarly, @xcite use an mt phrase table to mapping an english sentences to various non-english sentences. english sentence is my explanation clear? -minha explicac ¸ão está clara? accepted -minha explicac ¸ão é clara? portuguese -a minha explicac ¸ão é clara? translations -está clara minha explicac ¸ão? -minha explanac ¸ão está clara? -é clara minha explicac ¸ão? english sentence you look so pretty! -você está tão linda! accepted -você está tão bonita! portuguese -você está muito linda! translations -você está muito bonita! -você parece tão linda! -você parece tão bonita! more recently, advances in neural machine translation (nmt) have spurred interest in paraphrase generation @xcite @xcite . for example, @xcite employ a stacked residual lstm network to learn a sequence-to-sequence model on paraphrase data. a parpahrase model with adversarial training is presented by @xcite . @xcite ; @xcite propose a translation-based paraphrasing system, which is based on ntm to translate one side of a parallel corpus. paraphrase generation with pivot nmt is used by @xcite . ## models in this section, we first describe the architecture of our models. we then explain the different ways we train the models on various subsets of the data. ## model deployment methods in order to enhance the 1-to-n en-pt translation, we propose three methods based on the previously discussed mt models (see section 4). these methods are n-best prediction, multi-checkpoint translation, and paraphrasing. ## evaluation in order to evaluate our methods, we carry out a number of experiments. first, we consider performance of each proposed method on the official training and development datasets of sta-ple @xcite . our models were ultimately evaluated on the shared task test data. we now describe staple evaluation metrics and baselines as provided by organizers, before report-ing on our results on training, development, and test. ## conclusion in this work, we described our contribution to the 2020 duolingo shared task on simultaneous translation and paraphrase for language education (staple) @xcite . our system targeted the english-portuguese sub-task. our models effectively make use of an approach based on n-best prediction and multi-checkpoint translation. our use of the opus dataset for training proved quite successful. in addition, based on our results, our intuitive deployment of a multi-checkpoint ensemble coupled with n-best decoded translations seem to mirror leaner proficiency. as future work, we plan to explore other methods on new language pairs.