Coping with Noisy Training Data Labels in Paraphrase Detection (2021)

Abstract: We present new state-of-the-art benchmarks for paraphrase detection on all six languages in the Opusparcus sentential paraphrase corpus: English, Finnish, French, German, Russian, and Swedish. We reach these baselines by fine-tuning BERT. The best results are achieved on smaller and cleaner subsets of the training sets than was observed in previous research. Additionally, we study a translation-based approach that is competitive for the languages with more limited and noisier training data.

## introduction paraphrase detection is a task where a model is trained to recognize from a collection of phrases, whether two phrases carry approximately the same meaning. paraphrase detection provides a natural way to study how well models operate on semantic abstractions by recognizing similar meanings from syntactically and lexically different sequence pairs. models that are capable of operating on such a semantic level are useful in a number of natural language processing tasks, such as information retrieval or machine translation, by providing semantically coherent alternative queries or translation hypotheses for a sequence of words. in this paper, we study paraphrase detection trained on noisy data in six european languages. we train paraphrase detection models by finetuning bert @xcite and explore the effect of noisy labels in the training data. we reach new paraphrase detection benchmarks on all six languages in the opusparcus @xcite sentential paraphrase corpus, a collection of paraphrase pairs extracted from the opensubtitles2016 @xcite movie and television subtitle collection. the new benchmarks are obtained with considerably smaller and cleaner subsets of the training set than was observed in previous research. by exploring different proportions of noisy labels in the training data, we identify a level at which fine-tuned bert models still appear to be robust to the noisy labels. we also present a competitive translation-based approach that is especially useful for languages with more limited and noisier paraphrase data in our experimental setup. opusparcus contains training, development and test sets in english (en), finnish (fi), french (fr), german (de), russian (ru), and swedish (sv). for each language, the development and test sets consist of a few thousand sentence pairs that have been annotated manually. the annotators have indicated on a four-grade scale to which extent two proposed sentences mean the same thing. the training sets are considerably larger, comprising millions of sentence pairs, but lack manual annotations. however, the training data have been ordered in such a way that the sentence pairs that are most likely to be true paraphrases are placed first and the pairs of sentences that are least likely to carry the same meaning are placed last. this ranking is produced automatically, according to metrics based on point-wise mutual information. consequently, when training a paraphrase detection model on these training sets, one needs to decide how much of the available training data to use, weighing the benefits of a smaller and cleaner set versus a larger but noisier set. paraphrase detection based on opusparcus has previously been studied by @xcite , who train two different neural network models to represent the phrases and apply a cosine distancebased metric to detect the paraphrases. the authors observe that the networks are rather robust to noise and they can benefit from more data, albeit increase in data size results in increased amounts of noise within the training data. whereas @xcite use neural networks trained from scratch, we approach paraphrase detection by fine-tuning language-specific bert models for sequence classification. further previous explorations of opusparcus include the fine-tuning of pre-trained bert models for the purpose of word similarity assessment in english and finnish @xcite as well as german text summarization @xcite . pragst et al. (2020) compare different classifiers in different paraphrasing-related tasks on different corpora, including opusparcus. fabre et al. (2021) investigate automatic paraphrase generation on opusparcus as well as other datasets. none of these previous works, however, establish new benchmarks for paraphrase detection on the opusparcus test sets. an alternative to using large neural language models, such as bert, is to leverage the capability of machine translation systems instead. wieting et al. (2017) and iyyer et al. (2018 ) create paraphrases using massive backtranslation; from a parallel corpus of two languages (bitext), paraphrases in the first language are obtained by automatically translating the corresponding sentence in the second language back to the first language. in our proposed translation-based approach for paraphrase detection, we do not need bitexts. we simply use existing machine translation models to translate our paraphrase candidates in one language (for instance, english) to another language (such as spanish) and see whether the two source sentences (in english) have common translations in the target language (spanish), in which case they are considered to be paraphrases. however, we need to take into consideration that machine translation is not perfect and produces noisy results. we discuss our methods in detail in section 2. the results of our experiments are presented in section 3, and we conclude in section 4. ## experiments we evaluate the performance of language-specific bert models on paraphrase detection based on differently sized subsets consisting of different proportions of noise sampled from the opusparcus training data. we also evaluate our translationbased model on the paraphrase detection task. ## results the performance of our fine-tuned bert models with respect to different amounts of data and noise are presented in figure 1 . the results show that while the baseline models by @xcite benefit from adding more and noisier data, this is not the case with bert to the same extent. we notice three types of behaviour of bert on different languages. with finnish and swedish, the languages with most limited training data, bert obtains high results with a small and clean training subset, and adding more noisy data is only detrimental to the model. with the higher-resource languages, french, german and russian, we notice an initial increase in performance when adding more data. the models peak at 1 million positive examples and start to deteriorate when more noisy data is added. the results of the highest-resource between potential paraphrases via another language. the final test results are presented in ## conclusion we have presented a new state-of-the-art benchmark based on bert for six opusparcus languages. our experiments show that when fine-tuning bert for paraphrase detection, we need only a small portion of the number of training examples that are required for training a recurrent neural network from scratch. in fact, it is detrimental to extend the amount of training data too far, as this causes the proportion of noise to go up. we find that bert is considerably less robust to noise than the neural network models presented by @xcite . for some languages (english, finnish and russian) bert appears to be robust with up to 20 % of noisy labels in the training set, whereas the experiments on french, german and swedish indicate lower tolerance to noise in the data. previous research on bert @xcite has claimed that even a considerable amount of noise in labels does not result in large performance degredations. our findings are that we reach that point at a noise level of 20 % maximum. in addition, we have shown that competitive results for paraphrase detection can be reached with a rather simple translation-based approach. in three languages, we were able to outperform the previous recurrent neural network-based benchmarks, and for one language (finnish) we reached results that are almost on par with results obtained by finetuning bert. this suggests that a translation-based model can be useful in situations where enough high-quality paraphrase data for fine-tuning bert is lacking, or when an appropriate bert model does not exist. in future work we intend to explore means of further mitigating the effects of noisy training data. we could also study why certain target languages work better than others in the translation-based approach. future work could also include an explicit study of the opusparcus training set in order to identify possible other features in addition to noisy labels that affect the model performance when increasing training data. due to high computational resources needed for training large models, we limit the number of positive examples to five million. future work could include training a bigger english model to detect when a similar performance drop to the other languages occurs in english. furthermore, we are interested in studying paraphrases in scenarios, where meaning is highly context-dependent.