Striking a Balance: Alleviating Inconsistency in Pre-trained Models for Symmetric Classification Tasks (2022)

Abstract: While fine-tuning pre-trained models for downstream classification is the conventional paradigm in NLP, often task-specific nuances may not get captured in the resultant models. Specifically, for tasks that take two inputs and require the output to be invariant of the order of the inputs, inconsistency is often observed in the predicted labels or confidence scores. We highlight this model shortcoming and apply a consistency loss function to alleviate inconsistency in symmetric classification. Our results show an improved consistency in predictions for three paraphrase detection datasets without a significant drop in the accuracy scores. We examine the classification performance of six datasets (both symmetric and non-symmetric) to showcase the strengths and limitations of our approach.

## introduction symmetric classification tasks involve two inputs and require that the model output should be independent of the order in which the two input texts are given. in other words, the output of the classifier should be the same and the confidence score must not be significantly different, if the inputs x and y are instead supplied as y and x. paraphrase detection, multi-lingual semantic similarity are examples of symmetric classification tasks. although attention-based @xcite pre-trained language models have led to significant performance gains in multiple text classification tasks, they demonstrate a peculiar erratic behaviour on symmetric classification: inconsistency. an example foot_0 of inconsistency for paraphrase detection is shown in figure 1 . additional examples can be found in the appendix ( a revolutionary government or a provisional government has been declared several times in the philippines by insurgent groups . drop-in fine-tuning objective, based on either the kullback-leibler (kl) or jensen-shannon (js) divergence (or any f -divergence @xcite ), to the cross-entropy loss for symmetric tasks. we refer to this as the consistency loss. the main contributions of this paper are: (a) highlight inconsistency issues in symmetric classification tasks, (b) describe a consistency loss function to alleviate inconsistency, and (c) demonstrate the applicability and limitations of the loss function via qualitative and quantitative analyses on tasks from the glue benchmark. additionally, to drive future research, we have made the data and code public 2 . note: the problem of inconsistency can be attributed in part to the positional embedding. however, it has been shown that eliminating positional embedding results in a poor performance of the model @xcite . ## results our experiments address three questions: q1. what are the shortcomings of the current objective function for symmetric classification tasks? (section 1, section 5.2) q2. does adding the consistency loss alleviate the inconsistency problem? (section 5.1) q3. can consistency-based fine-tuning improve other downstream tasks? (section 5.1) ## conclusion in this paper, we proposed an additional objective: consistency loss between l2r and r2l predictions so as to alleviate the problem of input ordersensitive inconsistency in the case of symmetric classification tasks. for three symmetric classification tasks, our proposed solution, bert-withconsistency-loss, results in an improved consistency in terms of pearson's correlation and mse. as expected, consistency loss results in a drop in the performance of non-symmetric classification tasks such as qnli and rte. surprisingly, using kl divergence results in marginally higher consistency than the js counterpart. we leave this analysis for future work. our qualitative analysis shows that all error types, including change in phrases or addition/deletion of details are reduced when the consistency loss is incorporated. while consistency loss ensures that the predicted labels are the same even if the order of inputs is swapped, it can be adapted in the future to ensure expected outputs for anti-symmetric classification tasks (where p(l2r) = 1 -p(r2l)) like next and previous sentence prediction, where reordering the inputs must result in an opposite predicted label. in addition, the proposed method can be applied to evaluate paraphrase generation models @xcite as well. in order to validate that paraphrasing models are indeed generating semantically similar outputs, bert-with-consistency-loss can be used to either evaluate and filter out incorrect generations or be used as an objective to train learned metrics like bleurt @xcite .