Evaluating Paraphrastic Robustness in Textual Entailment Models (2023)

Abstract: We present PaRTE, a collection of 1,126 pairs of Recognizing Textual Entailment (RTE) examples to evaluate whether models are robust to paraphrasing. We posit that if RTE models understand language, their predictions should be consistent across inputs that share the same meaning. We use the evaluation set to determine if RTE models’ predictions change when examples are paraphrased. In our experiments, contemporary models change their predictions on 8-16% of paraphrased examples, indicating that there is still room for improvement.

## introduction recognizing textual entailment (rte), the task of predicting whether one sentence (hypothesis) would likely be implied by another @xcite , is central to natural language understanding (nlu; @xcite , as this task captures "all manners of linguistic phenomena and broad variability of semantic expression" @xcite . if an rte model has a sufficiently high capacity for reliable, robust inference necessary for full nlu @xcite , then the model's predictions should be consistent across paraphrased examples. we introduce p art e, a test set to evaluate how reliable and robust models are to paraphrases ( while this may not be a sufficient test to determine whether rte models fully understand language, as there are many semantic phenomena that rte models should capture @xcite , it is necessary that any nlu system be robust to paraphrases. ## related work with the vast adoption of human language technology (hlt), systems must understand when different expressions convey the same meaning (paraphrase) and support the same inferences (entailment). paraphrasing and entailment are closely connected as the former is a special case of the latter where two sentences entail each other @xcite @xcite . para-phrasing has been used to improve rte predictions @xcite and rte has been used for paraphrase identification @xcite and generation @xcite . furthermore, both phenomena are key to nlu (androutsopoulos and malakasiotis, 2010) and work such as @xcite ; @xcite have explored rewriting rte examples to create more robust models. we follow a long tradition of evaluating linguistic phenomena captured in rte models @xcite . recent tests focus on evaluating how well contemporary rte models capture phenomena such as monotonicity @xcite , verb veridicality @xcite , presuppositions @xcite implicatures @xcite , basic logic @xcite , figurative language @xcite , and others @xcite @xcite . unlike many of those works that evaluate models' accuracy on examples that target specific phenomena, we use a contrastive approach @xcite to determine whether rte models' predictions change when examples are paraphrased. ## p art e to explore whether these rte models are robust to paraphrases, we create p art e, a modified version of the pascal rte1-3 challenges @xcite @xcite . p art e contains 1,126 examples of an original unmodified rte sentence-pair grouped with a sentence-pair with a modified premise, hypothesis, or both. we use the examples in rte1-3 to create our test set, as opposed to other rte datasets due to its long-standing history. ## experimental setup we explore models built upon three different classes of sentence encoders: bag of words (bow), lstms, and transformers. our bow model represents premises and hypotheses as an average of their tokens' 300 dimensional glove embeddings @xcite . the concatenation of these representations is fed to an mlp with two hidden layers. for the bilstm model, we represent tokens with glove embeddings, extract sentence representations using max-pooling, and pass concatenated sentence representations to an mlp with two hidden layers. our transformer-based models are pre-trained bert @xcite and roberta @xcite encoders with an mlp attached to the final layer. additionally, we use gpt-3 in a zero-shot setting where we ask it to label the relationship between a premise and hypothesis. 5 the rte training sets do not contain enough examples to train deep learning models with a large number of parameters. we follow the common practice of training models on mnli and using our test set to evaluate how well they capture a specific phenomenon related to nlu. during testing, we map the mnli 'contradiction' and 'neutral' labels to the 'not-entailed' label in rte, following common practice @xcite @xcite . ## results entailed vs not-entailed examples rte analyses often differentiate how models perform on entailed vs not entailed examples @xcite . in figure 3 , we do not see meaningful differences in how models' predictions change on paraphrased examples based on the gold label. this might suggest that our dataset does not contain statistical irregularities based on the rte labels. change in model predictions (%) 0 5 10 15 20 25 gpt-3 roberta bert bilstm bow figure 4: percentage of examples where models' predictions change when the original prediction was correct (left blue bar) or incorrect (right red bar). correct vs not-correct predictions figure 4 shows that the transformer models' predictions is more likely to change when it's prediction on an original example was incorrect (right red bars) compared to when the prediction for an original example was correct (left blue bars). for example, when roberta's prediction for an original rte example was correct, the model changed its prediction on just 5.5% of the corresponding paraphrased examples. when roberta's predictions for an original rte example were incorrect, roberta's predictions changed for 20.88% corresponding paraphrased examples. analyzing differences in models' confidences assigned to predictions might provide more insight @xcite . we leave this for future work. source task rte1-3 examples originated from multiple domains and downstream tasks, e.g. question-answering @xcite , information extraction @xcite , and summarization @xcite . this enables researchers to evaluate how rte models perform on examples that contain different aspects of open domain inference necessary for the task @xcite . figure 5 reports the changes in models' predictions across the different sources of examples. we do not see consistent trends across the original data sources. ## conclusion we introduced p art e, a high-quality evaluation set of rte examples paired with paraphrased rte examples. we use our evaluation set to determine whether rte models are robust to paraphrased examples. our experiments indicate that while these models predictions are usually consistent when rte examples are paraphrased, there is still room for improvement as models remain sensitive to changes in input @xcite @xcite . we hope that researchers will use p art e to evaluate how well their nlu systems perform on paraphrased data.