Generating Diverse Translations via Weighted Fine-tuning and Hypotheses Filtering for the D uolingo STAPLE Task (2020)

Abstract: This paper describes the University of Maryland’s submission to the Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE). Unlike the standard machine translation task, STAPLE requires generating a set of outputs for a given input sequence, aiming to cover the space of translations produced by language learners. We adapt neural machine translation models to this requirement by (a) generating n-best translation hypotheses from a model fine-tuned on learner translations, oversampled to reflect the distribution of learner responses, and (b) filtering hypotheses using a feature-rich binary classifier that directly optimizes a close approximation of the official evaluation metric. Combination of systems that use these two strategies achieves F1 scores of 53.9% and 52.5% on Vietnamese and Portuguese, respectively ranking 2nd and 4th on the leaderboard.

## introduction while machine translation (mt) typically produces a single output for each input, scoring and generation for second language learning applications might benefit from systems whose outputs better capture the diversity of translations produced by language learners. the duolingo simultaneous translation and paraphrase for language education (staple) shared task @xcite provides a framework for developing and testing such systems, grounded in real translations produced by english learners into five native languages (portuguese, vietnamese, hungarian, japanese, korean). in this task, given an english sentence prompt, systems are asked to produce a set of translations for that prompt, and are scored based on how well their outputs cover human-curated acceptable translations, weighted by the likelihood that an english learner would respond with each translation ( prompt is my explanation clear? output minha explicac ¸ão está clara? 0.267 minha explicac ¸ão é clara? 0.161 a minha explicac ¸ão está clara? 0.111 a minha explicac ¸ão é clara? 0.088 minha explanac ¸ão está clara? 0.057 está clara minha explicac ¸ão? 0.044 minha explanac ¸ão é clara? 0.039 while the multiple translations can be viewed as paraphrases, we propose to address the staple task primarily as a mt task to better understand the strengths and weaknesses of neural mt architectures for generating multiple learner-relevant translations. given a transformer model for the language pair of interest, we use beam search to generate n-best translation candidates. however, since n-best lists are known to lack diversity, we propose to generate hypotheses that better match the requirements of the staple task via: controlled experiments and analysis show the benefits of both strategies. our final submission which includes both techniques achieves f1 scores of 53.9% and 52.5% for en-vi and en-pt respec-tively, reaching a rank of 2 nd and 4 th on the leaderboard, only 2 points below the top scoring system. for completeness, we also submitted systems for the remaining language pairs using frequency-aware n-best lists: our system ranked 2 nd for japanese and 3 rd for korean and hungarian. ## background unlike in the staple task, recent attempts at generating multiple translations for a single source have targeted output variability along specific stylistic dimensions @xcite @xcite or produce diverse outputs without a specific use case @xcite . the techniques used can be divided in three categories: (a) constrain the decoding process to generate diverse candidates @xcite @xcite ; (b) optimize via a diversity promoting loss function @xcite ; (c) expose the model to different translation candidates with side-constraints @xcite @xcite @xcite or without @xcite . since it is unclear what dimensions of variations are captured in the staple translation, we focus instead on improving n-best lists generated by a standard neural mt model. source texts with multiple references have mostly been used to evaluate rather than train mt systems @xcite @xcite . evaluation sets with 4 or 5 references have been converted to singlereference training samples @xcite to improve mt training, but reference translations vary in arbitrary ways and often exhibit poor diversity, mostly limited to translationese effects. the staple data presents an opportunity to explore multiple translations generated in a more comprehensive fashion. ## 5 evaluation we evaluate the lowercased detokenized output of the systems on our internal test dataset using: weighted macro f1 this is the official scoring metric which quantifies how the set of system outputs covers the human-curated acceptable translations, weighted by the lrf of each translation. it is defined as the harmonic mean of unweighted precision (p) and weighted recall (wr) calculated for each prompt e i , and averaged over all the prompts in the corpus. specifically, using the same notation as introduced in § 3.1, for each translation t i generated by the mt model, we have: the weighted macro f1 (wmf1) is then given by: ## submitted systems we tested our systems on the official blind development set to select the best performing models for final evaluation on the test set. for portuguese and vietnamese, our official submissions include frequency-aware hypothesis generation and hypothesis filtering: en-vi c2: frequency-aware (10-best) + frequencyaware (filtered 50-best) we did not build hypothesis filtering models for the other languages, and submitted systems based only on unfiltered models: en-ja frequency-aware (50-best) + unweighted (50best) ## conclusion we proposed two strategies to obtain multiple outputs that mimic translations by produced by language learners from a standard neural mt model. our experiments showed that (1) finetuning mt models using all reference translations and their weight yields more diverse n-best hypotheses that better reflect learner preferences, and (2) filtering these n-best lists using a feature-rich classifier trained to maximize an approximation of the staple evaluation metric yields further improvements. combinations of systems that use these two strategies approach the top scoring submission in the official evaluation. while these results suggest that some degree of output diversity can be achieved with little change to core neural mt models, oracle scores obtained with unfiltered n-best lists indicate that better modeling the space of learner translations might benefit both candidate generation and the filtering model in future work.