Expand and Filter: CUNI and LMU Systems for the WNGT 2020 D uolingo Shared Task (2020)

Abstract: We present our submission to the Simultaneous Translation And Paraphrase for Language Education (STAPLE) challenge. We used a standard Transformer model for translation, with a crosslingual classifier predicting correct translations on the output n-best list. To increase the diversity of the outputs, we used additional data to train the translation model, and we trained a paraphrasing model based on the Levenshtein Transformer architecture to generate further synonymous translations. The paraphrasing results were again filtered using our classifier. While the use of additional data and our classifier filter were able to improve results, the paraphrasing model produced too many invalid outputs to further improve the output quality. Our model without the paraphrasing component finished in the middle of the field for the shared task, improving over the best baseline by a margin of 10-22 % weighted F1 absolute.

## introduction the usual goal of machine translation (mt) is to generate a single correct translation of a source sentence. neural machine translation (nmt; @xcite models a conditional distribution over possible target sentences given a source sentence, and uses beamsearch decoding as a heuristic to get one or more translations. however, the number of possible correct translations is often vast in comparison @xcite . the staple challenge @xcite ) poses the problem of mt slightly differently. here, the goal is to generate as many correct translations as possible. knowing many correct translations can be useful e.g. for automatic scoring in tools for language education, such as duolingo. 1 on one hand, the learners should be guided to use the more common formulations, on the other hand, they should not be penalized for providing a correct but unusual answer. we present a pipeline of two systems in our submission to the staple challenge. as the first step, we use a standard nmt model trained with additional, carefully filtered data. the nmt output n-best lists are filtered using a classifier. second, we use a levenshtein transformer model @xcite to generate paraphrases of the outputs of the first model. again, the outputs of the levenshtein transformer are filtered using another classifier. the paper is structured as follows. our training datasets are described in detail in section 2. we describe the two models in sections 3 and 4 respectively. we conduct experiments with all five target languages in the challenge, i.e. hungarian, japanese, korean, vietnamese and portuguese. the source language is english in all setups. the experiment settings are shown in section 5. the results (section 6) show that the levenshtein transformer paraphrase generator cannot easily improve on the filtered nmt output n-best list. ## paraphrasing model as an additional step to increase the number of valid translations produced, we train a target- the model is based on the levenshtein transformer (levt; @xcite , which is a sequence generation model based on the transformer @xcite architecture. instead of leftto-right autoregressive generation, levt generates sequences in an arbitrary order using two basic operations -insertion and deletion. using an initial sequence as a starting point, levt is able to perform sequence refinement. ## results the results on the blind development and test data are shown in experiments with training data filtering showed that a smaller training set with better selected sentences leads to better trasnslation quality than using a larger general domain corpus (see despite our extensive efforts, the paraphrasing model did not bring substantial improvements over the translation model. increase in recall typically came with lower precision and lower weighted f1 score. even with the specific classifier training and its overall accuracy over 99%, the levt output was too noisy to be precisely filtered. the only improvement was achieved for portuguese, where the final paraphrasing and filtering setting resulted in slightly higher precision and similar recall. however, the improvement was mostly influenced by the second round of filtering (on the paraphrasing output). therefore, we did not include the paraphrasing output in our primary submission. we suppose that the inefficiency of the levt model is caused by a mismatch between model training criteria and its application: the individual policies are trained separately, but need to complement each other to achieve good results. moreover, the loss computed independently for each paraphrase may prevent the model from learning to generate multiple paraphrases. the shortage of useful paraphrases may be also caused by the lack of additional training data. ## conclusions we presented our submission to the 2020 sta-ple translation and paraphrasing shared task. our model is based on the transformer architecture, used additional carefully selected training data, a xlm-r-based classifier to filter mt output beams, and an optional paraphrasing component based on the levenshtein transformer approach. the mt improving the paraphrasing model could be an interesting direction of future work. the amount of incorrect output could be reduced by better ac-counting for deletion and insertion policy interplay. however, computing the loss independently for each paraphrase may still hinder the ability of the model to generate multiple paraphrases for a single sentence. it may be thus necessary to rethink the training objective and tie it together with the inference process.