Self-Supervised Learning for Pairwise Data Refinement (2020)

Abstract: Pairwise data automatically constructed from weakly supervised signals has been widely used for training deep learning models. Pairwise datasets such as parallel texts can have uneven quality levels overall, but usually contain data subsets that are more useful as learning examples. We present two methods to refine data that are aimed to obtain that kind of subsets in a self-supervised way. Our methods are based on iteratively training dual-encoder models to compute similarity scores. We evaluate our methods on de-noising parallel texts and training neural machine translation models. We find that: (i) The self-supervised refinement achieves most machine translation gains in the first iteration, but following iterations further improve its intrinsic evaluation. (ii) Machine translations can improve the de-noising performance when combined with selection steps. (iii) Our methods are able to reach the performance of a supervised method. Being entirely self-supervised, our methods are well-suited to handle pairwise data without the need of prior knowledge or human annotations.

## introduction deep learning models are widely adopted and have demonstrated their usefulness in many areas and applications. despite their diversity, one common characteristic of these models is the large number of parameters that need to be adjusted during training (some recent models that have billions of parameters include t5 @xcite and gpt-2 @xcite ). this leads to the need of collecting large amounts of training examples. pairwise data, that captures the relationship in two modalities, is used to train deep learning models such as neural machine translation (nmt) @xcite , question answering @xcite , image captioning @xcite , etc. to train this kind of models, large-scale data can often be obtained from weak signals like text cooccurrence @xcite or dictionary n-gram matching @xcite . for example, in the machine translation community, the large amount of multilingual text available on the internet has naturally led to the idea of using internet data to train nmt models @xcite . this approach has proven advantageous but it has the drawback that data mined this way is intrinsically noisy @xcite . despite the poor quality, usually this kind of data contains a helpful subset that can be recovered through a process of data cleaning or refinement. data cleaning could be implemented with linguistic knowledge such as its script, vocabulary, syntax, etc. alternatively, a model can be trained on "clean" or "trusted" pairs that are verified through manual annotation. both options can be highly effective, but the former is limited in scope and error-prone, while the latter can be costly due to the number of required annotated examples. in this paper we introduce two self-supervised methods to obtain data subsets from noisy pairwise data that can be helpful to train dual-encoder (d-e) and neural machine translation (nmt) models. as noisy pairwise data, in our experiments we use parallel texts mined from the internet. our methods do not require external knowledge (e.g. syntactic rules), language-dependent heuristics (e.g. script verification) or synthetic positive or negative training examples. by eliminating the need of annotations, our methods directly address the data labelling bottleneck. our methods employ d-e models @xcite to learn a shared embedded space from the co-located text in the sentence pairs mined from the internet. following @xcite we use the embedding distance in the learned space as a measure of cross-lingual similarity between sentences. our hypothesis is that if higher scores are associated with cross-lingual similarity, pairs with higher scores will be closer to be actual translations of each other and, in that case, may be part of the data subset useful to train the models. in our experiments, our methods show effective refining parallel texts mined from the internet. much of the gains in the downstream evaluation are achieved in the first iteration of the method, but later iterations keep improving the d-e models. despite being self-supervised, our methods show competitive performance when compared against a de-noising method that uses supervision. ## related work one line of the research that directly relates to our work is corpus filtering for training nmt models. below we classify the related work into two categories depending on the amount of supervision needed (e.g. high quality parallel texts). ## dual-encoder model dual-encoder (d-e) models have demonstrated to be an effective learning framework applied to both supervised @xcite and unsupervised tasks @xcite . a multi-task d-e model consists of two encoders and a combination function for each of the tasks. in the context of the d-e framework, the selection of bilingual text can be interpreted as a ranking problem where, with y i as the true target of source sentence x i , p (y i |x i ) is ranked above all the other target candidates in y. p (y i |x i ) can be expressed as a log-linear model but, for practical reasons, we approximate the full set of target candidates y with a sample @xcite . when training in a batch, p (y i |x i ) can be approximated as: based on the results of @xcite with additive margin softmax @xcite , we modify our loss function to include margin m: when using the dot product as similarity function φ, a single matrix multiplication can be used to efficiently compute scores for all the examples in the batch. when set to learn from clean crosslingual paired texts, a d-e model can be used to learn strong cross-lingual embeddings for bitext retrieval as shown in @xcite and @xcite . the challenge is to learn similar embeddings when training d-e models on noisy data. ## experimental setup machine translation model to assess if we can recover useful subsets from noisy data, we train transformer-big @xcite nmt models using data refined with our methods. to train the models, we split the source and target texts into pieces using bilingual sentence piece models @xcite ) that were trained with the paracrawl v1.0 data only. we train for a maximum of 200k steps using @xcite and pick the best checkpoint according to the performance on a validation set. the models are trained on google's cloud tpu v3 with batch size 3072. in all our experiments, the configuration of the nmt models is kept the same with the only difference being the training data. ## conclusions we introduced two self-supervised methods to refine pairwise data aimed at selecting useful subsets from noisy data. in our experiments we used parallel texts mined from the internet as example of the weakly constructed pairwise data to refine. our methods do not require linguistic knowledge or human annotated data. they use iterative selection of the data to train two kinds of models. our first method is based on self-boosting dualencoder models iteratively. we applied this method to denoise data to train nmt models. our second method integrates the nmt models into the iterative process to generate translations that, after a selection step, are used to train the dual-encoder models. our results show that most of the gains in terms of bleu score can be achieved in the first iteration of our methods, but later iterations keep improving the performance of the dual-encoder models in the bucc evaluation. in our experiments, using translated text in combination with a selection step helped to improve the de-noising capabilities of the dual-encoder models. we observed that selection is effective to break the model-data memorization cycle. one characteristic that our self-supervised methods do not seem to capture well is an indication of the language identity. if we use language identification on the denoised data as a post-processing step, the performance, in terms of bleu scores, turns very competitive against supervised targeted efforts tailored to remove noise from the dataset. these results encourage us to pursue future lines of work that include using cross-attention in the pairwise data to better capture the relationship in the pairs. also, specific to parallel sentences mined from the internet, we would like to explore ways to include language identification in the models. on the other hand, it seems natural to leverage the self-supervision characteristics of our methods and apply them to language pairs where noisy internet data may be available but annotated data is not. lastly, we are interested in expanding our methods to other pairwise data such as text-image pairs.