Exploring Model Consensus to Generate Translation Paraphrases (2020)

Abstract: This paper describes our submission to the 2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE). This task focuses on improving the ability of neural MT systems to generate diverse translations. Our submission explores various methods, including N-best translation, Monte Carlo dropout, Diverse Beam Search, Mixture of Experts, Ensembling, and Lexical Substitution. Our main submission is based on the integration of multiple translations from multiple methods using Consensus Voting. Experiments show that the proposed approach achieves a considerable degree of diversity without introducing noisy translations. Our final submission achieves a 0.5510 weighted F1 score on the blind test set for the English-Portuguese track.

## introduction machine translation (mt) systems are typically used to produce a single output for a given source sentence, whereas in human translation the same source sentence can often be translated in various different ways while still preserving its meaning. in the 2020 duolingo shared task on simultaneous translation and paraphrase for language education (staple) @xcite , participating mt systems are evaluated using multiple reference translations to measure their ability to generate diverse, yet high quality translations. for that, a new dataset with multiple human translations for each source sentence is provided. these human translations were produced by language learners as part of a translation exercise on the duolingo platform foot_1 where they were asked to translate sentences from the language they were learning (e.g. english) to their native language. each translation in the dataset is assigned a weight based on the learner response frequency. in this paper, we experiment with various methods to improve the diversity of translations, while preserving their quality. we show that simply by generating n-best translations with larger beam size, we can already achieve a considerable degree of diversity. our final submission is based on the integration of multiple translations from various methods, namely n-best translation, monte carlo dropout, mixture of experts, ensembling, and lexical substitution, through a consensus voting mechanism. it achieves 0.5510 weighted f1 score on the official blind test set. this paper is structured as follows: section 2 describes the methods we used in our experiments. section 3 introduces the experimental settings, including data preparation, model hyperparameters, and the evaluation procedure. section 4 describes the results and analysis. section 5 presents our three official submissions to staple blind test set. finally, section 6 summarises our submission to the shared task and our contributions. ## methods in what follows we describe the methods used in our experiments, including n-best translation, monte carlo dropout, diverse beam search, mixture of experts, ensembling and lexical substitution. we combine all of these methods except the diverse beam search in our official submissions through a consensus voting mechanism. details about the submissions can be found in section 5. ## official submissions our best submission achieves the second position in the english-portuguese track, with only 0.0006 weighted f1 score behind the winning submission. the official result on staple test set is shown in ## conclusions this paper describes our submissions to the sta-ple shared task for english-portuguese translation. we showed that simply generating n-best translations already achieves a considerable degree of diversity and quality. we experimented with various methods to improve the diversity in the mt output, including n-best translation, mc dropout, diverse beam search, mixture of experts, ensembling, consensus voting, and lexical substitution. we showed the benefits and drawbacks of these methods in generating diverse, high quality translations. our systems combining these methods further improve over the n-best translation and achieve 0.5510 weighted f1 score on staple blind test set, which is only 0.0006 behind the winning submission.