POSTECH Submission on D uolingo Shared Task (2020)

Abstract: In this paper, we propose a transfer learning based simultaneous translation model by extending BART. We pre-trained BART with Korean Wikipedia and a Korean news dataset, and fine-tuned with an additional web-crawled parallel corpus and the 2020 Duolingo official training dataset. In our experiments on the 2020 Duolingo test dataset, our submission achieves 0.312 in weighted macro F1 score, and ranks second among the submitted En-Ko systems.

## introduction simultaneous translation and paraphrase for language education (staple) is the task of automatically producing multiple translations from a single source sentence @xcite . because staple can be regarded as a mixture of the machine translation (mt) and paraphrasing problem, mt and paraphrasing techniques play an important role in this task. unlike in a typical mt task, systems are demanded to generate high-coverage sets on a sentence-level, as opposed to word-level. subsequently, systems require a deeper linguistic understanding of the target language to generate accurate target sentences. recent nlp studies have alleviated this problem by transfer learning @xcite from pre-trained language models. @xcite proposed a generative pre-trained language model (gpt), which trains a transformer decoder with large-scale monolingual data, to achieve significantly improved performance in nine out of the twelve datasets. despite these improvements, gpt shows a limited ability to model bidirectional context due to using the classical generative model-ing approach. on the other hand, @xcite proposed bidirectional encoder representations from transformers (bert), trained for the reconstruction of natural language from sentences containing masked tokens, in order to obtain deeper representations for natural language. by training on an enormous amount of training data, they achieved state-of-the-art results on eleven nlp tasks. to take advantage of both pre-trained generative models and pre-trained bidirectional encoders, @xcite introduced a denoising autoencoder for pre-training sequence-to-sequence models called bart. bart aims to learn linguistic knowledge in the process of first corrupting the text using various noise functions and then restoring it, and showed state-of-the-art performance in various tasks. given this background, we expected that using a transfer-learning-based approach could resolve two difficulties of the en-ko track of staple: data insufficiency and multiple sentence generation. unlike recent mt models which used over 4.5 million sentence pair for training data, the staple official dataset includes only 2500 en-ko source sentences. with such small data, we predicted that recent nmt models would not be able to learn translation knowledge effectively. also, we speculated that paraphrasing requires a deep understanding of the language. based on this prediction, a welltrained language model and a generative model for target language were needed to achieve this task's objectives. with these considerations, we concluded that bart, a sequence-to-sequence generative model pre-trained on a large amount of data, is most suitable for staple and thus propose a transferlearning-based simultaneous translation model by extending bart. our model added a randomly initialized source-side encoder in place of the embedding layer of bart pre-trained by korean monolingual data and predicts translation weights with an additional feed-forward network using hidden vectors generated by the pre-trained decoder. the remainder of the paper is organized as follows: section 2 describes our proposed method. section 3 summarizes the experimental procedure and results, and section 4 gives the conclusion. ## method we adopt bart to the staple problem, which takes source sentence to generate multiple target sentences. our model consists of a pre-trained autoencoder with the source-side encoder that proposed in @xcite and a feed-forward network to predict translation weights (figure 1 ). in the following subsections, we describe our methods in detail. ## conclusion in this paper, we present postech's submissions to the 2020 duolingo shared task. we propose a transfer-learning based simultaneous translation model by extending bart. the proposed model is first pre-trained by reconstructing large corrupted text using text infilling and sentence shuffling, and then fine-tuned with an additional parallel corpus and the official training dataset with a newly added randomly initialized encoder in place of the embedding layer. it has an additional feed-forward network to predict translation weight trained on the official dataset. finally, our model outperforms the baseline by a large margin and ranks second out of the submitted systems.