Simultaneous paraphrasing and translation by fine-tuning Transformer models (2020)

Abstract: This paper describes the third place submission to the shared task on simultaneous translation and paraphrasing for language education at the 4th workshop on Neural Generation and Translation (WNGT) for ACL 2020. The final system leverages pre-trained translation models and uses a Transformer architecture combined with an oversampling strategy to achieve a competitive performance. This system significantly outperforms the baseline on Hungarian (27% absolute improvement in Weighted Macro F1 score) and Portuguese (33% absolute improvement) languages.

## introduction this paper describes the third place submission to the shared task @xcite on simultaneous translation and paraphrasing for language education at the 4th workshop on neural generation and translation (wngt) for acl 2020. the shared task involves generating multiple translations for a given source text in english and a target language. the five target languages in the task are hungarian (hu), portuguese (pt), japanese (ja), @xcite and @xcite . we competed in the hungarian and portuguese tracks. a goal of the shared task, hosted by duolingo, is to enable development of automated grading processes and curation systems for language learners' responses. a high-coverage and precise multi-output translation and paraphrasing system would vastly help such automated efforts. for the task, participants were provided with hand-crafted and field-tested sets of several possible translations for each english sentence. each of these translations were also ranked and weighted according to actual learner response frequency and these weights were provided as additional features. along with these, translations from aws were provided as a baseline and additional data. the challenges associated with the shared task are two-fold: i) translating from english to target languages and ii) producing multiple valid translations (paraphrases) while balancing precision with the coverage. we conduct several experiments to address these two challenges and develop a simple system that leverages pre-trained transformer @xcite models and a wide beam search strategy. furthermore, we leverage the provided translation scores and experiment with multiple training distribution strategies to develop a simple oversampling strategy that produces improvements over the vanilla method of using one translation one time. ## related work paraphrasing and machine translation are wellstudied research areas in general but there's not much research specifically in the context of multi-output translation systems, especially for low resource languages. @xcite train a transformer-based neural machine translation model for hungarian-english and portugese-english translation. however, their goal was to assess the benefits of multilingual modeling by clustering languages and is different from that of a multi-output translation system. for english-portuguese, @xcite build a phrasebased machine translation system to translate biomedical texts. for multilingual parahrasing, @xcite release a database consisting of paraphrases for several languages, including hungarian and portuguese, at lexical, phrasal and syntactic level. @xcite build a zero-shot multilingual paraphrase generation model to show mixed results. however, their end goal was to generate paraphrases in the same language (english) as opposed to our shared task which requires generating paraphrases in a different language. we describe dataset statistics and evaluation metrics in this section. ## system design we now describe the final submitted system design in detail. we have experimented with several other variants and describe these in a later section 5. ## other modeling variants we experimented with different modeling alternatives for the shared task. we describe them in this section. the results of these variations are listed in ## results & discussion ## summary we describe the system for our submission to the shared task on simultaneous translation and paraphrasing for language education at the 4th workshop on neural generation and translation (wngt) for acl 2020. the final submitted system leverages pre-trained translation models, with transformer architecture, and an oversampling strategy to achieve competitive performance. for future, it'd be interesting to see if initializing the model with latest state-of-the-art sequence-tosequence pre-trained models such as @xcite and @xcite and finetuning could help boost performance. it would also be a promising direction to explore the benefit of using cross-lingual models such as xlm-roberta @xcite . one way to use them would be to initialize the encoder part of the architecture with pre-trained representations. given the shared representations, it might be interesting to see if concatenating several language pairs' train datasets and training a joint model produces additional benefits.