Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020)

Abstract: Self-supervised neural machine translation (SSNMT) jointly learns to identify and select suitable training data from comparable (rather than parallel) corpora and to translate, in a way that the two tasks support each other in a virtuous circle. In this study, we provide an in-depth analysis of the sampling choices the SSNMT model makes during training. We show how, without it having been told to do so, the model self-selects samples of increasing (i) complexity and (ii) task-relevance in combination with (iii) performing a denoising curriculum. We observe that the dynamics of the mutual-supervision signals of both system internal representation types are vital for the extraction and translation performance. We show that in terms of the Gunning-Fog Readability index, SSNMT starts extracting and learning from Wikipedia data suitable for high school students and quickly moves towards content suitable for first year undergraduate students.

## introduction human learners, when faced with a new task, generally focus on simple examples before applying what they learned to more complex instances. this approach to learning based on sampling from a curriculum of increasing complexity has also been shown to be beneficial for machines and is referred to as curriculum learning (cl) @xcite . previous research on curriculum learning has focused on selecting the best distribution of data, i.e. order, difficulty and closeness to the final task, to train a system. in such a setting, data is externally prepared for the system to ease the learning task. in our work, we follow a complementary approach: we design a system that selects by itself the data to train on, and we analyse the selected distribution of data, order, difficulty and closeness to the final task, without imposing it beforehand. our method resembles self-paced learning (spl) @xcite , in that it uses the emerging model hypothesis to select samples online that fit into its space as opposed to most curriculum learning approaches that rely on judgements by the target hypothesis, i.e. an external teacher @xcite to design the curriculum. we focus on machine translation (mt), in particular, self-supervised machine translation (ssnmt) @xcite , which exploits the internal representations of an emergent neural machine translation (nmt) system to select useful data for training, where each selection decision is dependent on the current state of the model. self-supervised learning @xcite involves a primary task, for which labelled data is not available, and an auxiliary task that enables the primary task to be learned by exploiting supervisory signals within the data. in ssnmt, both tasks, data extraction and learning mt, enable and enhance each other. this and the mutual supervision of the two system internal representations lead to a selfinduced curriculum, which is the subject of our investigation. in section 2 we describe related work on cl, focusing on mt. section 3 introduces the main aspects of self-supervised neural machine translation. here, we analyse the performance of both the primary and the auxiliary tasks. this is followed by a detailed study of the self-induced curriculum in section 4 where we analyse the characteristics of the distribution of training data obtained in the auxiliary task of the system. we conclude and present ideas for further work in section 5. ## related work machine translation has experienced major improvements in translation quality due to the introduction of neural architectures @xcite @xcite . however, these rely on the availability of large amounts of parallel data. to overcome the need for labelled data, unsupervised neural machine translation (usnmt) @xcite @xcite focuses on the exploitation of very large amounts of monolingual sentences by combining denoising autoencoders with back-translation and multilingual encoders. further combining these with phrase tables from statistical machine translation leads to impressive results @xcite @xcite . usnmt can be combined with pre-trained language models (lms) @xcite @xcite . @xcite train a very large lm on billions of monolingual sentences which allows them to perform nmt in a few-shot setting. self-supervised nmt (ssnmt) @xcite is an alternative approach focusing on comparable, rather than parallel data. the internal representations of an emergent nmt system are used to identify useful sentence pairs in comparable documents. selection depends on the current state of the model, resembling a type of self-paced learning @xcite . data selection in ssnmt is directly related to curriculum learning, the idea of presenting training samples in a meaningful order to benefit learning, e.g. in the form of faster convergence or improved performance @xcite . inspired by human learners, @xcite argues that a neural network's optimization can be accelerated by providing samples in order of increasing complexity. while sample difficulty is an intuitive measure on which to base a learning schedule, curricula may focus on other metrics such as task-relevance or noise. to date, curriculum learning in nmt has had a strong focus on the relevance of training samples to a given translation task, e.g. in domain adaptation. @xcite train on increasingly relevant samples while gradually excluding irrelevant ones. they observed an increase in bleu over a static nmt baseline and a significant speed-up in training as the data size is incrementally reduced. @xcite adapt an nmt model to a domain by introducing increasingly domain-distant (difficult) samples. this seemingly contradictory behavior of benefiting from both increasingly difficult (domain-distant) and easy (domain-relevant) samples has been analyzed by @xcite , showing that the initial phases of training benefit from easy samples with respect to a hypothetical competent model (target hypothesis), while also being boosted @xcite by samples that are difficult with respect to the current state of the model @xcite . in @xcite , both domain-relevance and denoising are combined into a single curriculum. the denoising curriculum for nmt proposed by @xcite is related to our approach in that they also use online data selection to build the curriculum based on the current state of the model. however, the noise scores for the dataset at each training step depend on fine-tuning the model on a small selection of clean data, which comes with a high computational cost. to alleviate this cost, @xcite use reinforcement learning on the pre-scored noisy corpus to jointly learn the denoising curriculum with nmt. in section 3.2 we show that our model exploits its self-supervised nature to perform denoising by selecting parallel pairs with increasing accuracy, without the need of additional noise metrics. difficulty-based curricula for nmt that take into account sentence length and vocabulary frequency have been shown to improve translation quality when samples are presented in increasing complexity @xcite . @xcite link the introduction of difficult samples with the nmt models' competence. other difficulty-orderings have been explored extensively in @xcite , showing that they, too, can speed-up training without a loss in translation performance. ssnmt jointly learns to find and extract similar sentence pairs from comparable data and to translate. the extractions can be compared to those obtained by parallel data mining systems where strictly parallel sentences are expected. beating early feature-based approaches, sentence representations obtained from nmt systems or tailored architectures are achieving a new state-of-the-art in parallel sentence extraction and filtering @xcite @xcite @xcite . using a highly multilingual sentence encoder, @xcite scored wikipedia sentence pairs across various language combinations (wikimatrix). due to its multi-lingual aspect and the close similarity with the raw wikipedia data we use, we also use scored wikimatrix data for one of the comparisons (section 3.2). ## self-supervised neural machine translation (ssnmt) ssnmt is a joint data selection and training framework for machine translation, introduced in @xcite . ssnmt enables learning nmt from comparable rather than parallel data, where comparable data is a collection of multilingual topicaligned documents. ssnmt is based on a bidirectional nmt system {l1, l2}→{l2, l1} where the engine learns to translate simultaneously from a language l1 into another language l2 and vice-versa with a single encoder and a single decoder. this is important in the self-supervised architecture because it represents the two languages in the same semantic space. in principle, the input data to train the system is a monolingual corpus of sentences in l1 and a monolingual corpus of sentences in l2 and the system learns to find and select similar sentence pairs. in order to speed-up training, we use a comparable corpus such as wikipedia, where we can safely assume that there are comparable (similar) and parallel sentence pairs in related documents d l1 , d l2 . given a document pair d l1 , d l2 , the ssnmt system encodes each sentence of each document into two fixed-length vectors c w and c h ## summary and conclusions this paper explores self-supervised nmt systems which jointly learn the mt model and how to find its supervision signal in comparable data; i.e. how to identify and select similar sentences. this association makes the system naturally and internally evolve its own curriculum without it having been externally enforced. we observe that the dynamics of mutual-supervision of both system internal representations, c w and c h , is imperative to the high recall and precision parallel data extraction of ssnmt. their combination for data selection over time instantiates a denoising curriculum in that the percentage of non-matching pairs, i.e. nontranslations, decreases from 18% to 2%, with an especially fast descent at the beginning of training. even if the quality of extraction increases over time, lower-similarity sentence pairs used at the beginning of training are still relevant for the development of the translation engine. we analyze the translation quality of a supervised nmt system trained on the epoch-wise data extracted by ss-nmt and observe a continuous increase in bleu. analogously, we also analyze the similarity scores of extracted sentences and observe that they also increase over time. as extracted pairs are increasingly similar, and precise, the extraction itself instantiates a secondary curriculum of growing task-relevance, where the task at hand is nmt learning with parallel sentences. a tertiary curriculum of increased sample complexity is observed via an analysis of the extracted data's gunning fog indices. here, the system starts with sentences suitable for initial high school students and quickly moves towards content suitable for first year undergraduate students: an overachiever indeed as the norm over the complete wp is end of high school level. lastly, by estimating the perplexity with an external lm trained on wp, we observe a steep decrease in perplexity at the beginning of training with fast convergence. this indicates that the extracted data quickly starts to resemble the underlying distribution of all wp data, with a larger amount of outliers at the beginning. these outliers can be accounted for by the importance of homographs at that point. this raises the question of how ssnmt will perform on really distant languages (less homographs) or when using smaller bpe sizes (more homographs), which is something that we will examine in our future work.