The JHU Submission to the 2020 D uolingo Shared Task on Simultaneous Translation and Paraphrase for Language Education (2020)

Abstract: This paper presents the Johns Hopkins University submission to the 2020 Duolingo Shared Task on Simultaneous Translation and Paraphrase for Language Education (STAPLE). We participated in all five language tasks, placing first in each. Our approach involved a language-agnostic pipeline of three components: (1) building strong machine translation systems on general-domain data, (2) fine-tuning on Duolingo-provided data, and (3) generating n-best lists which are then filtered with various score-based techniques. In addi- tion to the language-agnostic pipeline, we attempted a number of linguistically-motivated approaches, with, unfortunately, little success. We also find that improving BLEU performance of the beam-search generated translation does not necessarily improve on the task metric—weighted macro F1 of an n-best list.

## introduction the duolingo 2020 staple shared task @xcite focuses on generating a comprehensive set of translations for a given sentence, translating from english into hungarian, japanese, korean, portuguese, and vietnamese. the formulation of this task ( §2) differs from the conventional machine translation setup: instead of the n-gram match (bleu) against a single reference, sentence-level exact match is computed between a list of proposed candidates and a weighted list of references (as in figure 1 ). the set of references is drawn from duolingo's language-teaching app. any auxiliary data is allowed for building systems, including existing very-large parallel corpora for translation. our approach begins with strong mt systems ( §3) which are fine-tuned on duolingo-provided data ( §4). we then generate large n-best lists, from which we select our final candidate list ( §5). our 0.015 0.001 0.001 0.049 can i walk there? in addition to our system description, we perform additional analysis ( §7). we find that stronger bleu performance of the beam-search generated translation is not indicative of improvements on the task metric-weighted macro f1 of a set of hypotheses-and suggest this should encourage further research on how to train nmt models when n-best lists are needed ( §7.1). we perform detailed analysis on our output ( §7.2), which led to additional development on english-portuguese ( §8.1). we also present additional linguistically-informed methods which we experimented with but which ultimately did not improve performance ( §8). for each language pair, we created an internal split of the duolingo-provided training data: 100 training prompts for use in validating the mt system (jhu-valid), another 100 intended for model selection (jhu-dev), foot_0 and a 300-prompt test set for candidate selection (jhu-test). the remaining data (jhu-train) was used for training the mt models. ## fine-tuning after training general-domain machine translation models, we fine-tune them on the duolingo data. 5 the duolingo data pairs single prompts with up to hundreds of weighted translations; we turned this into bitext in three ways: ## candidate generation and selection from the models trained on general-domain data ( §3) and refined on in-domain data ( §4), we generate 1,000-best translations. for each translation, fairseq provides word-level and length-normalized log-probability scores, which all serve as grist for the next stage of our pipeline: candidate selection. 5 training on the duolingo data directly was less effective. 6 a better method might be to train using the weights to weight the sentences in training as available in marian @xcite but that was not available in fairseq, so we improvised. ## results we present results of our different methods on our internal development set in §6.1 and present our official evaluation performance in §6.2. ## analysis we perform qualitative and quantitative analyses of our output, which informed our own work and will motivate future work. ## what didn't work we explored additional methods both for selecting candidates from an n-best lists and for generating additional candidates based on an n-best list. while they did not improve performance and were not included in our final submission, we discuss the methods and the analyses learned from them. ## conclusion our approach was general, borrowing from best practices in machine translation. we built large, general-domain mt systems that were then finetuned on in-domain data. we then followed an "overgenerate and filter" approach that made effective use of the scores from the systems to find a per-prompt truncation of large n-best lists produced from these systems. these techniques performed very well, ranking first in all five language pairs. we expect that further refinement and exploration of standard mt techniques-as well as techniques that we were unsuccessful with ( §8)would bring further improvements that would accrue generally across languages. at the same time, the duolingo shared task is distinct from machine translation in subtle but important ways: presenting simpler, shorter sentences and a 0-1 objective. while we were not able to get additional gains from linguistic insights, we don't see these failures as conclusive indictments of those techniques, but instead as invitations to look deeper.