GAPX: Generalized Autoregressive Paraphrase-Identification X
Abstract
Paraphrase Identification is a fundamental task in Natural Language Processing.
While much progress has been made in the field, the performance of many state-ofthe-art models often suffer from distribution shift during inference time.
We verify that a major source of this performance drop comes from biases introduced by negative examples.
To overcome these biases, we propose in this paper to train two separate models, one that only utilizes the positive pairs and the other the negative pairs.
This enables us the option of deciding how much to utilize the negative model, for which we introduce a perplexity based out-of-distribution metric that we show can effectively and automatically determine how much weight it should be given during inference.
We support our findings with strong empirical results
Introduction
Paraphrases are sentences or phrases that convey the same meaning using different wording, and is fundamental to the understanding of languages [7].
Paraphrase Identification is a well-studied task of identifying if a given pair of sentences has the same meaning [51, 47, 56, 57, 31], and has many important downstream applications such as machine translation [61, 44, 40, 27], and question-answering [11, 35].
What causes stool color to change to yellow?
What can cause stool to come out as little balls?
QQP similar topic
Emma Townshend is represented by David Godwin at
DGA Associates .
David David Godwin is represented at DGA Associates by Emma Townshend .
PAWS same bag-of-word
But it changed the name and IDN and there was nothing to deal with.
WMT poor machine translation
That amber alert was getting annoying
Why do I get amber alerts tho
PIT similar topics
The company then changed its company ID number, and that was it.
biases during training. Recently, researchers have observed that neural network architectures trained on different datasets could achieve state-of-the-art performances for the task of paraphrase identification [52, 16, 50].
While these advances are encouraging for the research community, it has however been observed that these models can be especially fragile in the face of distribution shift [45].
In other words, when a model trained on a source dataset D s is tested on another dataset, D t , collected and annotated independently, and with a distribution shift, the classification accuracy drops significantly [62].
This paper presents our findings and observations that negative pairs (i.e., non-paraphrase pairs) in the training set, as opposed to the positive pairs, do not generalize well to out-of-distribution test pairs.
Intuitively, negative pairs only represent a limited perspective of how the meanings of sentences can be different (and indeed it is practically infeasible to represent every possible perspective).
We conjecture that negative pairs are so specific to the dataset that they adversely encourage the model to learn biased representations.
We show this observation in Figure 1.
Quora Question Pair (QQP)extracts its negative pairs from similar topics.
Paraphrase Adversarials from Word Scrambling (PAWS)
Zhang et al.
[62] generate negative pairs primarily from word swapping.
World Machine
Translation Metrics Task 2017 (WMT) [8] considers negative examples as poor machine translations.
We therefore hypothesize that biases introduced by the different ways negative pairs are mined are major causes of the poor generalizability of paraphrase identification models.
Based on this observation, we would like to be able to control the reliance on negative pairs for out-of-distribution prediction.
In order to achieve this, we propose to explicitly train two separate models for the positive and negative pairs (we will refer to them as the positive and negative models respectively), which will give us the option to choose when to use the negative model.
It is well known that just training on positive pairs alone can lead to a degenerate solution [21, 59, 42, 13], e.g., a constant function network would still produce a perfect training loss.
To prevent this, we propose a novel generative framework where we use an autoregressive transformer [39, 49], specifically BART [30].
Given two sentences, we condition the prediction of the next token in the second sentence on the first sentence and the previous tokens.
In a Bayesian sense, this would mean that the next token predicted has a higher probability of being a positive/negative pair to the first sentence for the positive and negative model respectively.
This learning strategy has no degenerate solutions even when we are training the positive and negative models separately.
We call our proposed approach GAP to stand for Generalized Autoregressive Paraphrase-Identification.
One potential pitfall of GAP is that it ignores the “interplay” between positive and negative pairs that would otherwise be learned if they are utilized in training together.
This is especially important when the test pairs are in-distribution.
To overcome, we utilize an extra discriminative model, trained with both positive and negative pairs, to capture the interplay.
We call this extension GAPX (pronounced as “Gaps”) to capture the e X tra discriminative model used.
For all practical purposes, the weights we placed on the positive, negative and/or discriminative model in GAP and GAPX need to be determined automatically during inference.
For in-distribution pairs, we desire to use them all, while for out-of-distribution pairs, we hope to rely on the positive model much more heavily.
This obviously leads to a question of how to determine whether a given pair is in or out of distribution [9, 14, 43, 17, 20].
During testing, our method ensembles the positive model, the negative model, and the discriminative model based on the degree of the similarity of the test pair to the training pairs, and found that this works well for our purpose.
We measure this degree of similarity with probability cumulative density function (cdf) in terms of perplexity [25], and show that it is superior to other measures.
In summary, our contributions are as follow:
We report new research insights, supported by empirical results, that the negative pairs of a dataset could potentially introduce biases that will prevent a paraphrase identification model from generalizing to out-of-distribution pairs.
To overcome, we propose a novel autoregressive modeling approach to train both a positive and a negative model, and ensemble them automatically during inference.
Further, we observe that the interplay between positive and negative pairs are important for in-distribution inference, for which we add a discriminative model.
We then introduce a new perplexity based approach to determine whether a given pair is out-of-distribution to achieve auto ensembling.
We support our proposal with strong empirical results.
Compared to state-of-the-art transformers in out-of-distribution performance, our model achieves an average of 11.5% and 8.4% improvement in terms of macro F1 and accuracy respectively over 7 different out- of-distribution scenarios.
Our method is especially robust to paraphrase adversarials like
PAWS, while keeping comparable performance for in-distribution prediction.
Related Works 2.1 Distribution Shift and Debiasing Models in NLP
The issue of dataset bias and model debiasing has been widely studied in a lot of field in NLP such as Natural Language Inference [23, 3] and Question Answering [36, 2, 4].
Notable work by [22, 5, 10, 15, 48] utilize ensembling to reduce models’ reliance on dataset bias.
These models share the same paradigm where they break down a given sample x into signal feature x s and biased feature x b , in the hope of preventing their model from relying on x b , which has been shown to be the limiting factor preventing the model from generalizing to out-of-distribution samples [5].
Here, a separate model is first either trained on x b or on datasets with known biases [10, 15, 22], or acquired from models known to have limited generalization capability.
Then they train their main model with a regularization term that encourages the main model to produce predictions that deviate from that of the “biased model”.
However, this type of approach has shown limited success [45] in debiasing paraphrase identification models.
In contrast, our method is based on our observation that negative pairs limit the generalization of paraphrase identification models.
2.2 Out-of-distribution Detection
Another line of work relevant to this paper is the task of detecting out-of-distribution samples [9, 14, 43, 17, 20, 55].
Researchers have proposed methods to detect anomaly samples by examining the softmax scores [32] or energy scores [33, 60] produced by discriminative models, while others take a more probabilistic approach to estimate the probability density [14, 25, 66, 1] or reconstruction error [38].
In this paper, we introduce a novel perplexity based out-of-distribution detection method that we show empirically to work well for our purpose.
Specifically, during inference, an out-of-distribution score is utilized to weigh the contributions from the positive and negative models: the higher the confidence that the sample is out-of-distribution, the lesser the negative model’s contribution.
2.3 Text Generation Metrics
Finally, we would like to note the difference between our work and autoregressive methods that have been explored [58, 46] for evaluating text generation.
Our work differs as follows: 1) Paraphrase identification seeks to assign a label of paraphrase or not while text generation metrics seeks to assign a score to measure the similarity of sentences; 2) Current text generation metrics either cannot be trained to fit to a specifc distribution [61, 58, 63] or are limited to the i.i.d.
setting [44, 41] of the training distribution.
In contrast, our method not only significantly improves out-of-distribution performances but is also competitive with state-of-the-art paraphrase identification methods for in-distribution predictions.
Methodology
We observe that negative pairs in paraphrase identification constitute the main source of bias.
To overcome this, we propose the following training paradigms to learn a significantly less biased paraphrase identification model.
We employ an autoregressive conditional sentence generators with transformer architecture as the backbone of our model.
Specifically, we train a positive and negative model to estimate the distribution of positive and negative pairs in a dataset respectively.
During testing, the two models are ensembled based on how likely the input pair is out of distribution.
This section provides details on our method.
3.1 Separation of Dependence on Positive and Negative Pairs
Let S be the space of all sentences, X =
( s 1 , s 2 ) be the random variable representing a sample pair from S , and Y the random variable representing the labels, with Y =
1 indicating that s 1 and s 2 are paraphrases and
otherwise when Y = 0 .
We seek to separate the dependence between the distribution of positive and negative pairs, motivated by the observation of the presence of bias in the negative
[Figure] pairs. To begin, we model the distribution of sentences by splitting the sentence s 2 of length n into the autoregressive product of individual words, where w ( i ) 2 denotes the i th word in s 2 .
By applying
Bayesian Inference Law, we have:
[Equation]
Subtracting the logarithm for Y = 1 and Y = 0 , we get:
[Equation]
In this way, we break the probability inference into 3 terms resulting in Eqn. 2: (1) (log P ( Y = 1) − log P
( Y = 0)) , which should just be a constant; (2) ( (cid:80)
[Equation]
log P
[Equation]
) , which depends only on the distribution of positive pairs; (3) ( (cid:80) ni [-=1 log P ( w ( i ) 
-][Equation]
P [Equation] w [Equation]
i
[Equation]
Y
[Equation]
[Equation]
) , which depends only on the distribution of negative pairs.
We define the score of confidence as follows:
[Equation]
In the above, we are now left with two terms, the first representing the positive model and the second the negative model.
If we were to train the two terms together, the effects of the negative pairs in the resulting model can never be removed during inference, which we have observed to be a major source of bias.
To avoid this, we propose to train the first term and the second term separately, and then subsequently ensemble them based on the degree that a given pair is out of distribution.
We train each model on top of the pretrained autoregressive transformer described in [30] known as BART.
Given s 1 and s 2 , we feed s 1 into the encoder as the condition, shift s 2 to the right by one-token, and feed shifted s 2 to the decoder.
While the decoder proceeds autoregressively, we record the next-word probability distribution.
We calculate the cross entropy between the next-word probability distribution and the target token in s 2 to update the model parameters.
Note that here the Bayesian formulation has been similarly raised in some of the previous work like Moore and Lewis [37], but to the best of our knowledge, we’re the first to propose this Bayesian formulation to control the reliance on different components of the model.
3.2 Ensembling
To combine the positive and negative model, if we know a priori D t is in the same distribution as D s , we can directly substitute the prediction of the positive and negative model into Eqn. 3.
We will refer to this as the In-distribution Predictor (IDP).
If we have reason to believe that there is a significant distribution shift between D s and D t (e.g., different sources of corpus and different dataset collection procedure), we observe empirically that we should only utilize the positive model and disregard the negative model due to the bias it introduces.
We will refer to this as the Out-of-distribution Predictor (OODP).
3.2.1 Automatic Ensembling
However, in most cases, we have little or no knowledge of the testing distribution, in which case we need to automatically decide how important the negative model is by detecting how much a test pair is in the same distribution as the training set.
We adopt a weighted interpolation between a constant and the negative model in addition to the positive model as follow:
[Equation]
where λ ( s 1 , s 2 ) is a weight parameter depending on s 1 and s 2 , and C is a constant that achieves a regularization effect.
See Appendix for ablations on how C can be set.
P ( s 2 | s 1 , Y = 1) and
P ( s 2 | s 1 , Y = 0) are the same terms in Eqn.3.
To automatically assign λ ( s 1 , s 2 ) for different sentence pairs, we measure an out-of-distribution score for ( s 1 , s 2 ) with regard to the training distribution.
Specifically, we use the same set of training data, comprising both positive and negative pairs, from
D s , on which we train another autoregressive model, which we will refer to as the distribution model.
The distribution model is trained by feeding an empty string into the encoder and the concatenated sand s 2 into the decoder, with the training goal of predicting the next token.
We measure the perplexity of each sentence pair ( s 1 , s 2 ) using the distribution model based on the following formula, w i being the i th token of the concatenated ( s 1 , s 2 ) of length n :
[Equation]
We then fit a Weibull distribution to the perplexity of a held-back set of validation data, so that it can better model the right-skewed property of the distribution.
We derive the exponential parameter a , the shape parameter c , the location parameter loc , and the scale parameter scale .
During testing, λ ( s 1 , s 2 ) can now be determined as:
[Equation]
For the final prediction, we predict the sentence pair to be paraphrase if S ( s 1 , s 2 ) ≥ 0 and nonparaphrase otherwise.
This forms what we referred to earlier as GAP (Generalized Autoregressive Paraphrase-Identification).
3.2.2 Capturing Interplay Between Positive and Negative Pairs
In practice, training a positive and negative model separately disregards the interplay between the positive and negative pairs, which could be important when the test pairs are in-distribution.
To capture such interplay, we utilize both positive and negative pairs to train a discriminative model for sequence classification.
Specifically, we first define a thresholding function based on the value of λ :
[Equation]
We then ensemble the discriminative model and GAP using the value of τ ( λ ) :
[Equation]
where P ( Y = 1 | s 1 , s 2 ) can be estimated by any discriminative model, and M is a sufficiently large constant.
Note that this definition is essentially the same as trusting the discriminative model when we do not have statistical evidence that the pair is out-of-distribution (p-test < 10%) while trusting the GAP model otherwise.
For the final prediction, we predict the sentence pair to be paraphrase if
S ∗ ( s 1 , s 2 ) ≥ 0 and non-paraphrase otherwise.
This defines GAPX (Generalized Autoregressive
Paraphrase Identfication X), for which we set M to be sufficiently large ( > 1000 ), so that when comparing the model confidence for different pairs, the score of the discriminative model will be prioritized.
Experiments
Our experiments are designed to (1) verify that the task of paraphrase identification suffers from biases in the datasets that is the main obstacle to generalization in this field of study, (2) test the accuracy of our perplexity based out-of-distribution detection method, and (3) test that balancing the utilization of the negative model can help outperform the state-of-the-art in the face of distribution shift, without losing in the in-distribution scenarios.
4.1 Datasets
We compare our method against the other state-of-the-art methods on different combinations of the following datasets: • Quora Question Pair (QQP) consists of over 400,000 lines of potential question duplicate pairs. Since the original sampling method returns an imbalanced dataset, the authors attempt to balance it with additional negative pairs collected from similar topics to make them harder. Note that to scale QQP down to approximately the same size of PAWS and PIT (see below), we take the first 10k training pairs and 2k testing pairs from the train and test split by Wang et al.
• World Machine Translation Metrics Task 2017 (WMT) [8] contains in total 3793 manual ratings of machine translations from 7 languages to English. Each rating result contains a source sentence in the source language, a reference sentence in English (ground truth translation), a machine translated sentence in English, and a manual rating of the quality of translations. We take the ground-truth reference sentence and the machine translated sentence as the sentence pair. Sentences with a higher quality score ( > 0 ) are labeled as paraphrases, while those with lower quality scores ( ≤ 0 ) are labeled otherwise.
The resulting paraphase identification dataset is balanced. Note that this dataset is significantly smaller than other datasets, so we only use it as a test set. • Paraphrase and Semantic Similarity in Twitter (PIT) [53, 54] contains 18762 sentence pairs automatically extracted from a similar distribution of topics as QQP.
Annotators manually assigned integer scores from 0 to 5 to each sentence pair, representing the degree of similarity between the sentence pair. To make it a paraphrase identification dataset, we label sentence pairs with low scores (0, 1) as non-paraphrases and those with high scores (4,5) as paraphrases. The original dataset is unbalanced, so we randomly sample a maximum balanced subset of the dataset. The original test set processed in this way shrinks to only 350, and is not comparable to the other datasets.
Hence, we use the original development data of size 1896 as the test set while keeping original test set of size 350 for development. The training set contains 5332 sentence pairs. • Paraphrase Adversarials from Word Scrambling (PAWS) [62] contains 49,401 sentence pairs, each of which is constructed from the same bag-of-word to make the evaluation more challenging. Most of the negative pairs are generated by word swapping while positive pairs are supplemented by back translation.
This dataset contains paraphrase pairs that are the adversarial counterparts of standard paraphrase datasets such as QQP and PIT. With these datasets, we perform experiments where different models (Sec. 4.2) are trained on one dataset and evaluated on another in order to observe whether their performance hold in the face of distribution shift.
4.2 Benchmarks
We benchmark the paraphrase identification task with these methods: • BOW [62] represents two input sentences with bag of words.
The bag of words representation of each input sentence is passed through a fully-connected network and cosine similarity between of the final layer is used to compute the classification output.
• BiLSTM [28] passes each of the two input sentences through a bidirectional LSTM network. The output state of the two sentences are then concatenated together and passed through a fully-connected network to get the classification output. • BERT [16] is representative of the state-of-the-art transformer methods for text classification.
We finetune the pretrained model "bert-base-uncased" in a standard way. We concatenate the sentence pair separated by a [SEP] token and take the [CLS] token as aggregate representation for the sentence pair. The embedding for the [CLS] token is then fed into an output layer for classification. • BART [30] is the original transformer model that we build on by finetuning the pretrained model "bart-base-uncased".
We concatenate the sentence pair separated by a 
 token, feeding it both into the encoder and the decoder. We use the 
 token at the end of the sentence pair for aggregate representation so that it can attend to decoder states from the complete input. The embedding for 
 is then fed into an output layer for classification. • RoBERTa [34] shares the same transformer architecture with BERT, but uses a more robust pretraining strategy, and as a result performs better than BERT in many NLP tasks [34].
For our experiments, we employ RoBERTa in the same way as BERT. • IDP (In-distribution Predictor) is our model for in-distribution prediction if we know a priori that the testing pairs come from the same distribution as the training pairs.
It combines the positive and negative models as given in Eqn. 3. • OODP (Out-of-distribution Predictor) is our model for out-of-distribution prediction if we know a priori that the testing and training pairs are not in the same distribution.
It only makes use of the positive model. We expect our OODP to have better generalizability than our IDP, because of its reduced reliance on negative examples. • GAP (Generalized Autoregressive Paraphrase-Identification) is our method that utilizes the perplexity based out-of-distribution detector to automatically control the reliance on the negative model, using Eqn. 4.
This setting is different from IDP and OODP in that we do not have a priori knowledge of the test distribution. • GAPX (Generalized Autoregressive Paraphrase-Identification X) ensembles GAP with RoBERTa described above, because we found RoBERTa to be the strongest baseline for in-distribution prediction. The intention is to capture via RoBERTa the interplay between positive and negative pairs. As depicted in Eqn. 8, when we do not have significant evidence that the given pair is out of distribution (p-test > 10%), we trust the prediction given by the discriminative model (RoBERTa).
Otherwise, we trust GAP. Following the previous literature in sentence matching [52, 16], we mainly use macro F1 score (F1) and accuracy score (ACC) to evaluate the models.
Results based on Area-under-curve of the Receiver
Operating Characteristic Curve (AUROC), a common metrics used to evaluate out-of-distribution metrics, are also provided in Appendix. A.
4.3 Measuring Distribution Shift
To situate our experiments properly, we note that different datasets does not equate to different distributions.
It is thus important for us to be able to measure the distribution gap between datasets, and shed light on the models that perform the best when transferring between datasets with high distribution gap.
Metrics that is per sample based such as our perplexity measure are not suitable for measuring at dataset level.
To this end, we take a look at the Reverse Classification Accuracy* (RCA*) metric that has been proposed to predict the drop of model performance [18] and model selection [19, 64].
Here, given D s and D t , we first train a model M 1 from the training set of D s .
We then take a certain amount of samples from the target distribution D t (1000 in our experiments), and use M 1 to relabel them.
The relabeled pairs are then utilized to train a new model M 2 .
We measure the performance (in terms of AUC or ACC) of M 2 on a held-out test set from D s .
As a control, we train another model M 3 following the same procedure except the relabeled data comes from D s .
We denote the performance drop from M 3 to M 2 as the RCA* score indicative of the distribution gap.
To calibrate RCA*, for each dataset, we randomly selected 1000 held-out pairs and measure its shift from the dataset itself (which we expect to characterize an in-distribution RCA*).
Afterrepetitions of measurements, we get a probability distribution of RCA* scores for each distribution in itself.
Calibration results are reported in Appendix. 9.
All the distributions share a mean of aroundand a standard deviation around 2.
By a p-test of 10%, it is unlikely that datasets with a RCA* greater than 4.0 would belong to the same distribution.
Based on this observation, most of the transfers between different datasets are likely out-of-distribution, except for transfers from QQP to PIT and vice versa, which have a RCA* score of 2.8 and 3.4 respectively.
This is probably because QQP and
PIT are curated in a similar fashion, where they extract sentences with similar topics from social platforms, while all other datasets adopt different strategies to collect their data.
Finally, it is important to note that although RCA* provides a good estimate of the distribution shift at the dataset level, it’s utility does not easily extend to Eqn. 4 and 8 as opposed to per sample out-of-distribution metrics.
RCA* assumes availability of the entire test set, while in real world, we are much more likely to get a single or small batches of test pairs at a time, all of which could even be from different distributions.
4.4 Implementation details
In practice, for testing, it helps to average the cross entropy by the length of the sentence, and average the cross entropy of generating s 2 from s 1 and generating s 1 from s 2 .
To optimize conditional sentence generators, we use Adam optimizer with learning rate 2e-5.
We adopt cross entropy loss for each word logit.
All experiments are run on Nvidia
Ti with 11 GB memory.
[Figure] [Figure]
4.5 Main Results
Bias in Negative Pairs To understand whether negative pairs are major sources of bias, we plot the Macro F1 and ACC score against RCA* in Fig. 3(a) and Fig. 4(a), comparing the performances of OODP, IDP
[Figure]
and BART, all finetuned from the same pretrained checkpoint.
The x-axis is plotted in ascending order of RCA* between pairs of datasets given in Table 1 and 2.
There are three pairs
[Table]
[Table]
that have RCA* score of 0 (Table 2), for which we average the performance in the plots.
Both the
F1 and ACC plots share a similar pattern.
In the in-distribution region, where the RCA* score is less than 4% (Fig. 9), BART and IDP achieves similarly high performances of 79.7% and 75.6% F on average respectively.
The 4.1% gap in F1 is potentially due to the fact that IDP trains a negative model and positive model separately, neglecting the interaction between positive and negative pairs.
Comparatively, the performance of OODP is significantly inferior to the other two, with only 60.4% average F1.
This changes in the out-of-distribution region, where the RCA* score is now greater than 4%.
OODP turns out to be the leading model over BART and IDP.
With increasing degree of distribution shift, we observe that both BART and IDP are especially fragile and their performance drop significantly.
When the RCA* is greater than 20, both models’ F1 drop to as low as around 60%, which can be hardly useful in practice.
In contrast, OODP maintains an advantage in F1 of as much as 10-20% throughout the out-of-distribution region.
Since the only difference between OODP and IDP is that OODP transfers only the positive model, it confirms our hypothesis that the negative model does not generalize as well as the positive model.
Importance of the Interplay between Positive and Negative Pairs To understand the necessity of capturing the interplay for test pairs that are in distribution, we compare the performances of
GAP, GAPX, and BART in Fig. 3(a) and Fig. 4(a).
GAP only ensembles the positive and negative model trained separately, so does not contain any interplay information.
On the other hand, both
BART, trained with positive and negative pairs together, and GAPX (Sec. 3.2.2) capture interplay information.
As shown in the plot, the performance of GAP in the in-distribution region is not directly comparable to BART with a gap of 10.9% in macro F1.
In contrast, GAPX’s in-distribution result has a much smaller margin of 1.6% macro F1 compared to BART, yet, by automatically weighing the contribution of the discriminative model, GAPX also closely matches the performance of GAP in the out-of-distribution region with only a 2.3% loss in macro F1.
Effectiveness of Perplexity-based Ensembling We also substitute PP ( s 1 , s 2 ) with other state-ofthe-art out-of-distribution metrics used for estimating the probability in Eqn. 6 for GAPX.
Specifically, we compare our perplexity metric with Maximum Softmax Probability (SOFTMAX) [24, 26, 6],
Energy Score (ENERGY)
[33], Mahalanobis Distance (MAHA) [29], and COSINE [65].
See
Appendix for more details.
The results are given in Fig. 4(b) and Fig. 3(b).
F1 and ACC are all similar in the in-distribution region.
However, in the out-of-distribution region, SOFTMAX, MAHA, and COSINE start to perform poorly.
ENERGY turned out to be the most robust but is still obviously not matching our perplexity metric, often by a large margin.
Generalization We implemented one of the most popular methods for domain generalization in paraphrase identification, Expert Product [15], as a potential strong baseline for handling dataset bias.
As described in Sec.
4.2, we train a BERT classifier with only the first sentence s 1 as the biased model for BERT+EP.
We report the results in Fig. 4(c) and Fig. 5(c), together with BERT,
BART and RoBERTa.
In addition, we also provide performance of traditional methods like BoW [62] and ESIM [12] in Table 1 and Table 2.
BERT, BART, and RoBERTa all produce similar results on all combinations of datasets.
We observe that their performances are consistently better than traditional methods like BoW and ESIM, showing that pretraining and finetuning can indeed improve the generalizability of classifiers for paraphrase identification.
However, their performances in out-ofdistribution setting are still far from their in-distribution performances, with accuracy below 65% in most of the cases (around 20% drop).
GAPX maintains an absolute margin of around 10% in terms of ACC and an absolute margin of 7-20% in terms of F1 in the out-of-distribution region.
The best transformer-based models in the out-of-distribution region is BART, with an average of 56.2% in F1 and 59.8% in ACC, while GAPX maintains an average of 67.7% in F1 and 68.2% in ACC, with an absolute gain of 11.5% in F1 and 8.4% in ACC.
What is also encouraging is that GAPX’s performance is close to that of OODP, which is promising that we do not need a priori information on the domain gap between the source and target.
Lastly, BERT+EP fails to provide much gain, which we conjecture is due to the difficulty of “finding” the right bias model or features.
Conclusion
We have shown that negative samples introduce bias that prevent the generalization of paraphrase identification models.
To overcome, we present a novel paradigm to train separate models for the distribution of positive and negative samples independently, and utilize a perplexity based out-ofdistribution detection to ensemble them automatically.
Experiments show that our method achieves an average of 11.5% gain of F1 and 8.4% gain of ACC in various different out-of-distribution scenarios over other state-of-the-art methods.
5.1 Limitations
Our methodology is specifically designed for only “verification” problems, where the samples come in pairs.
Scenarios that involve other types of bias will require non-trivial turn-key formulations to explicitly model the source of bias (much like the negative model).  Acknowledgements This work is sponsored by Meta AI.
We would also like to thank Prof. Yoav Artzi for a helpful discussion in the early stage of this project.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] .
See our listed contributions in the introduction.
(b) Did you describe the limitations of your work?
[Yes] .
See the end of our conclusion, Sec. 5.1.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] .
This is a traditional Natural Language Processing research topic.
We do not see any conceivable negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] .
See Sec. 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] .
See Section.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] .
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] .
See Sec.4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
The baseline models in NLP that we are comparing to do not report error bars in their paper.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] .
See Sec. 4.4. 4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] .
See Appendix 4.
(b) Did you mention the license of the assets?
[N/A] .
Those are all public datasets.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] .
Those are all public datasets, allowing researching purpose.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] .
Those data does not contain personal information 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
We include full results of AUROC scores here.
[Table]
[Table]
[Figure]
We also conducted ablations on several variants of GAPX. • GAPX(neg-log) replaces perplexity with the neg-log likelihood of the concatenated ( s 1 , s 2 ) as the measure. Specifically, GAPX(neg-log) modifies the Eqn.
5 and Eqn.
6 such that:
[Equation]
s
[Equation]
[Equation]
( s 1 , s 2 ) , Weibull ( a, c, loc, scale )) .
• GAPX(w/ IDP) ensembles GAP with IDP.
• GAPX(w/ BART) explores the option of ensembling GAP with BART as described in Appendix.4.2.
These variants of GAPX mostly perform similarly to the GAPX model proposed in the main paper, which ensembles GAP with RoBERTa.
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Table]
terms of macro F1/accuracy (ACC)
[Table]
[Table]
GAPX(cosine) 82.6/82.
GAPX(softmax) 82.2/82. GAPX(maha) GAPX(energy) 82.8/83. C Additional Discussions C.1 Ablations on C , Eqn.
[Table]
To set C in Equation.
4, in the case of 0 samples, we roughly estimate an integer from 1-5 for the interpolation constant C . For adversarial distributions like PAWS, we expect a lower perplexity because both sentences share the same bag-of-word, so we set C = 1 . For standard distributions like QQP and WMT, we expect a modest perplexity, so we set C = 3 . For informal distributions like PIT, where sentences do not strictly follow syntax and grammar, we expect a higher perplexity, so we set C = 5 .
In other cases, with validation data, we determine the best constant C based on the validation data (if there are multiple constants with the same results on the validation data, we take the smallest one).
As shown in Table 3 (WMT results are not included here due to the lack of validation data), although using 0 or 10 samples achieve worse performances than using 100 samples, meaning the best constant threshold is not found, the results appear to be stable overall (on average 2% fluctuations in macro F1 and 1% fluctuations in accuracy).
If a small amount of validation data is accessible, the performance of GAPX can be further improved.
C.2 Interpreting the Results
B i d i r ec ti ona l E n c od e r
A u t o r e g r e ss i ve D ec od e r
Is
India a soft nation in terms of its foreign relationsrelations ？
Can
I register my marriage to a foreign national in relationsIndia ？ 4.
relations0.
s s
Pr
Figure 6, Figure 7, and Figure 8 shows examples of how our autoregressive paraphrase identification models work.
For OODP, our model will output a log of conditional probability for each word in sgiven s 1 and all the previous words in s 2 , namely:
[Equation]
For IDP, we can use the log of the quotient of the conditional probability for each word given by the positive model and the negative model as an indicator which words contribute the most to the prediction result:
[Equation]
For GAP, we can use the the score defined in Eqn.. 4 split on each word, namely:
[Equation]
B i d i r ec ti ona l E n c od e r
A u t o r e g r e ss i ve D ec od e r
Is
India a soft nation in terms of its foreign relationsrelations ？
Can
I register my marriage to a foreign national in relationsIndia ？ 1. relations-0.
- s s
Pr
B i d i r ec ti ona l E n c od e r
A u t o r e g r e ss i ve D ec od e r
[Figure]
B i d i r ec ti ona l E n c od e r
A u t o r e g r e ss i ve D ec od e r
Is
India a soft nation in terms of its foreign relationsrelations ？
Can
I register my marriage to a foreign national in relationsIndia ？ 1.
relations-2.
s s
Pr
B i d i r ec ti ona l E n c od e r
A u t o r e g r e ss i ve D ec od e r
[Figure]
[Figure]
In all three models, higher scores represent a higher chance of being non-paraphrases.
For IDP and
GAP, the threshold is 0 while for OODP the threshold is 3.
All three models predict this sentence pair to be non-paraphrases, attending to slightly different key words.
The top 3 words with the highest scores in OODP and GAP are ’register’, ’marriage’, and ’national’.
All of them represent the words that are unlikely to occur in a paraphrase of the original sentence.
The top 3 words with the highest scores in IDP are ’I’, ’marriage’, and ’a’.
Its reliance on the word ’a’ might be due to the error propagation of the autoregressive decoding.
C.3 Implementation of out-of-distribution metrics
To compare our perplexity metric with different off-the-shelf out-of-distribution metrics (Fig. 4(b) and 5(b)), we first train a RoBERTa model as described in Sec.
Both MAHA and COSINE need in-distribution validation data, for which we use half of the development data provided in each dataset.
We use the other half of the development data to estimate the Weibull distribution of the metrics.
We rely on the implementation of SOFTMAX, ENERGY, MAHA, and COSINE from Zhou and Chen [65].
The metrics are calculated as follow:
SOFTMAX .
We use the maximum class probability 1 − max j =0 , 1 pj among
classes (paraphrases and non-paraphrases) in the final softmax layer.
ENERGY .
We use the following formula to calculate energy score:
[Equation]
where w j is the weight of the j th class in the softmax layer, and h is the input to the softmax layer (of the concatenated ( s 1 , s 2 ) input).
MAHA .
We use the input representation h of the penultimate layer of the model
and fit a Gaussian distribution to each class in the in-distribution development data D val = { ( x i ,
[Equation] [Equation] = [Equation]
Then, the MAHA distance is calculated as:
[Equation]
where Σ + is the pseudo-inverse of Σ . 4.
COSINE .
We use the maximum cosine similarity of h (of the concatenated ( s 1 , s 2 ) input) to samples in the validation dataset:
[Equation]
C.4 RCA Implementation Details
To instantiate a measurement of RCA* scores.
We use the RoBERTa model described in Section 4.
as our classifier.
To measure the distribution shift from a source distribution D s to a target distribution
D t .
We use a training set of D s , a test set of D s , and a development set of D t .
All the measurements of RCA* scores in this paper fix the size of the development to be 1000, and use the test set of D s to be the entire test set of the original dataset.
[Figure]
To calibrate the RCA* scores, we measure the distribution shift of one dataset to itself.
We first train a binary classifier RoBERTa M 1 (of paraphrases and non-paraphrase) with the training set of D s for 3 epoches with an Adam Optimizer of learning rate 2e-5 (training with multiple runs with different random seeds to select the best model on a development set of D s , different from the development set of D t ).
Then we apply M 1 to the development set of D t to relabel those data.
We take the relabeled data to retrain a classifier M 2 and apply M 2 to the test set of D s .
We measure the performance drop from M 1 to M 2 on the test set of D s as the RCA score.
Note here that we make small changes to the originally proposed RCA score where the performances are measured in terms of ACC scores.
We found that AUROC scores are in practice more stable for measuring RCA scores, so we use AUROC scores instead.
To get the final RCA* score, we use the equation:
[Equation]
where RCA ∗ ( D s , D t ) is the RCA* score from D s to D t , while RCA ( D s , D t ) is the RCA score from D s to D t .
Under this definition, the RCA* score of one dataset to itself is defined to be 0.
To repeat the measurements of distribution shift from D s to itself, at each time
[Table]
we hold-backrandom pairs from D s , and measure the RCA.
We include plots of the distribution of the RCA* in
Fig.
9 as well as the corresponding raw RCA* data below:
[Table]
PAWS: -3.0, -0.0092, -0.46, -5.6, -1.7, 2.7, 0.5, 2.4, 3.6, 0.034, -4.1, -3.2, 3.6, 8.2, 0.71, -0.079, 2.9, 5.4, 1.7, -5.2, 3.2, 2.8, -3.0, 2.0, -2.2, -1.7, -0.2, 4.1, -0.22, 3.9, 1.1e+01, 1e+01, 1.2, 0.0074, -0.91, -2.2, 3.9, -2.9, -2.7, -2.8, -4.2, -4.1, -1.0, -3.1, 2.6, -0.83, 0.74, 1.7, 0.56, 2.2, 0.87, -0.5, 0.83, -5.5, 0.4, -3.0, 1.1, -3.1, 0.92, -1.6, 4.1, -3.1, -0.99, 0.79, -0.15, -1.3, 5.2, 0.18, -5.7, 3.4, 2.9, 3.2, -4.0, -2.4, -4.9, -0.51, -1.8, -2.7, -3.7, 6.6, 0.65, 2.5, 1.5, -2.3, 3.1, 1.9, 1.2, -2.1, 1.7, 0.95, 2.1, -2.7, 5.9, -4.0, 1.8, -1.8, -0.2, 4.6, -1.3, -1.