It Is Not Easy To Detect Paraphrases: Analysing Semantic Similarity With Antonyms and Negation Using the New S em A nto N eg Benchmark (2022)

Abstract: We investigate to what extent a hundred publicly available, popular neural language models capture meaning systematically. Sentence embeddings obtained from pretrained or fine-tuned language models can be used to perform particular tasks, such as paraphrase detection, semantic textual similarity assessment or natural language inference. Common to all of these tasks is that paraphrastic sentences, that is, sentences that carry (nearly) the same meaning, should have (nearly) the same embeddings regardless of surface form. We demonstrate that performance varies greatly across different language models when a specific type of meaning-preserving transformation is applied: two sentences should be identified as paraphrastic if one of them contains a negated antonym in relation to the other one, such as “I am not guilty” versus “I am innocent”.We introduce and release SemAntoNeg, a new test suite containing 3152 entries for probing paraphrasticity in sentences incorporating negation and antonyms. Among other things, we show that language models fine-tuned for natural language inference outperform other types of models, especially the ones fine-tuned to produce general-purpose sentence embeddings, on the test suite. Furthermore, we show that most models designed explicitly for paraphrasing are rather mediocre in our task.

## introduction large pretrained language language models have pushed nlp forward in many sub-fields, and their ability to embed essential linguistic properties makes them applicable across a wide range of tasks. however, it is still an open question how well they cope with systematic compositionality and to what level of abstraction they reflect the actual meaning behind a given sentence. this work is in line with other experiments based on dedicated test suites that study specific linguis-tic phenomena in connection with neural representation models. in particular, we publish a novel benchmark called semantoneg foot_0 , that tests the ability of language models to properly represent expressions that contain antonymy and negation, embedded into a paraphrase task. our test suite consists of contrastive examples where the task is to select the correct paraphrase for a given input sentence among three candidate expressions that include combinations of negated sentences and antonym substitutions. to provide a simple example, a semantic opposite to an input sentence i am guilty would be i am not guilty, where the opposite meaning is invoked by an insertion of a negation marker. similarly, instead of inserting the negation marker, substituting the adjective to its antonym inverses the meaning of the sentence: i am innocent. to maintain paraphrasticity with respect to the original sentence, performing both of the operations is necessary, resulting in: i am not innocent. thus, for a model to succeed in the semantoneg test suite, the models need to understand that insertion or deletion of a negation accompanied with antonym substitution produces a sentence that is semantically equivalent to the original sentence, and the sentence embeddings should represent this relationship. using this benchmark, we study the following questions: in order to answer those questions, we systematically evaluate a large number of publicly available pretrained language models on the novel test suite. we notice a large amount of variation between different models, mostly depending on the fine-tuning objective and data, but, in some cases, also between models fine-tuned for the same task. selecting an appropriate model becomes a challenge considering the sea of releases available on public model hubs. simply selecting among the most popular ones might be a poor strategy, as we can see in model accuracy all-minilm-l6-v2 1.9 paraphrase-minilm-l6-v2 43.3 bert-base-nli-mean-tokens 83.3 all-mpnet-base-v2 31.9 distiluse-base-multilingual-cased-v2 1.5 all-minilm-l12-v2 8.2 multi-qa-mpnet-base-dot-v1 17.5 paraphrase-multilingual-minilm-l12-v2 61.8 paraphrase-mpnet-base-v2 76.1 distilbert-multilingual-nli-stsb-quora-ranking 47.1 our analysis below provides new insights into the semantic abstraction abilities with respect to antonyms and negation and gives additional guidance for the selection of an appropriate model for tasks that require proper inference with such constructions. ## description of the task the objective of our test suite is to test to what extent sentence representation models succeed in distinguishing sentences similar in meaning when the change in semantics is realised only by a substitution of a distributionally similar @xcite word token (antonym in this case) or by an insertion or deletion of a negation marker. performing either of the operations results in a sentence that conveys the semantic opposite of the original sentence while maintaining a high lexical overlap between both sentences. we cast the benchmark in terms of a paraphrase detection task: a model is confronted with three alternatives of potential reformulations of an input sentence and only one of them is a proper equivalent on a semantic level. the candidates are designed to include negations and antonyms of adjec-tives to create the specific challenge of the test suite. more details about the data sets and its creation are given in section 3 below. the idea is not to fine-tune any model for this particular task (because any model would quickly overfit to such regular constructions) but rather to test independently trained sentence embedding models that can be used to measure semantic distance to make the decision. as such, it is in line with other natural language understanding benchmarks such as senteval @xcite and glue @xcite but it represents a dedicated linguistic probing task rather than a general-purpose evaluation framework. the work is motivated by previous research that identified deficiencies in the popular sentence representation benchmarks. for instance, existing paraphrase detection data sets (e.g., qqp foot_1 ) lack examples that are characterised by a high lexical overlap without paraphrastic meaning @xcite . classification models can simply learn to measure lexical overlap to make proper decisions and, therefore, @xcite generate a more difficult test suite of paraphrases and non-paraphrases with a high bag-of-words overlap by word scrambling. the same problem has been observed in natural language inference, where contradicting sentences typically exhibit low lexical overlap. word permutations have been proposed @xcite to generate difficult cases that require better knowledge of compositionality. in our test suite, we go beyond the creation of more challenging distractors (e.g., better negative examples with high vocabulary overlaps) by introducing more challenging positive candidates that are explicitly different from the source by adding antonyms from a lexical resource. in connection with negation, the model is now forced to disregard surface features (such as matching tokens) and to properly understand negated messages to make the correct decision. thus, we test not only for semantic similarity but also evaluate the ability to understand the relationship between antonyms as well as the effect of negation on meaning representations. before discussing the experimental setup (section 4) and the results of our practical experiments (section 5), we will present details of the data set and how it was created in the section below. ## data creation we created the test suite in a semi-automatic way. first, we downloaded 1.5 million sentence pairs from the english training set of the opusparcus paraphrase corpus @xcite . next, we removed sentence pairs where the length of either of the sentences was less than four tokens. furthermore, we retained only sentence pairs, where either one, but not both, of the sentences contains a negation (e.g., i'm innocent. -i'm not guilty.) after this filtering process, our data consisted of approximately 7500 sentence pairs. at this point we realised that, even though many of the sentence pairs were meaningful for our experiments, our filtered set also contained pairs that were paraphrastic but did not include antonymous relations, such as aren't you cold? -are you cold? therefore, we proceeded with a second round of filtering, where we pos-tagged the sentences using nltk @xcite and retained sentences containing explicit negation (not) as an adverb (rb) as well as adjectives (jj). finally, we removed duplicate sentences. this process resulted in 1317 sentences, all of which included an explicit negation marker and an adjective. for each of the 1317 sentences, we generated three hypotheses from which the model is supposed to choose the one that conveys the same meaning as the input sentence. first, we queried an antonym for the adjective in the input sentence from the wordnet electrical lexical database @xcite to get a an opposing sentence. to obtain the second contradicting sentence, we deleted the negator from the input sentence. finally, we substituted the adjective from the second contradicting sentence to its antonym to obtain the paraphrase of the input sentence. examples of entries in the test suite are provided in we reviewed the resulting test suite manually to ensure its good quality. the antonym substitution procedure introduced some grammatical errors to the data, such as wrong agreement of articles (a evil idea) or a question tag not agreeing with the main clause (you're not serious, aren't you?). we corrected such phrases manually. the sentences also included some examples where the automatically retrieved antonyms were not considered to carry opposite meanings, such as are you hungry? -are you not thirsty? we removed such examples from the final test suite. eventually, we obtained 788 examples, from which we permuted all possible input sentences to result in 3152 test examples, containing 209 unique adjectives, which constitute the semantoneg test suite. ## experimental setup to compare sentence representations derived from different transformer-based pretrained language models, we ran 114 of the 120 pretrained sentence-bert @xcite models that are publicly available in the hugging face transformers library @xcite . 3 we provide the full list of the models we tested accompanied with the accuracy they acquired on the semantoneg test suite in appendix a. we embed the sentences in our test suite using the different language models. to create the embeddings we apply the same pooling strategy used in training the original sentence-transformers. we then evaluate each model by its ability to produce embeddings such that the input sentence and its true paraphrase are closest to each other in the vector space. to compare embeddings we use cosine similarity. basically, we have run a systematic loop over all available models but we are especially interested in specific groups of models, which we will discuss next. first of all, we want to see the performance of general-purpose models before looking at dedicated paraphrase models. finally, we discuss other task-specific models before we present a detailed error analysis and conclude with prospects for future work. ## results we analyse the results of our experiments by grouping the language models into four different categories, determined by the fine-tuning objective of the models: general-purpose embeddings, as well as embeddings specialised for paraphrasing, natural language inference and semantic textual similarity. figure 1 shows average accuracies and variances for these four types of models. individual results of all models are provided in appendix a. since there are three possible choices in our task, a random selection yields a baseline level of 33.3% accuracy. we observe models that diverge clearly from the baseline level, either positively or negatively. this means that many tested models make good or bad choices systematically. we return to these findings in the error analysis in section 6. the following sections summarise the analyses of the separate model groups. ## error analysis we have analysed misclassified examples from a set of different models. naturally, the errors the best-performing models make differ from the ones made by the worst-performing models. we have studied an intersection of the examples that were misclassified by all of the best models (listed in another frequently shared incorrectly predicted antonym pair includes the words same and other in the definite form, for instance: it is not the same. -it is the other. here, the definite form the other could be the cause of incorrect predictions, whereas a more suitable opposing relation could be expressed using for example some as an indefinite article as in some other. the antonym pair the same -the other comprises 17 of the remaining 35 incorrectly predicted examples shared by the best performing models. a bar chart of the best paraphrase model and the four paraphrase models that obtain accuracy below random choice is presented in figure 2 . the results indicate that the poor performance of the worst paraphrase models is explained by them systematically preferring the sentence with the highest lexical overlap disregarding the negation completely, which is reflected by a high proportion of label 1 in the figure. the trend seems similar for the other models whose accuracies are below random choice. compared to the models that perform well, the poorly performing models seem more prone to associating lexical similarity with semantic similarity, leading them to predict the sentence with the opposite meaning with respect to the input sentence. ## limitations the test suite comes with some limitations that we find important to discuss. we acknowledge that the proposed task is not difficult to solve using a simple rule-based model and that it is rather easy to overfit a neural network-based model to the data. therefore, the kind of data the test suite instantiates is supposed to be used for evaluating models by probing sentence representations for the certain kinds of linguistic phenomena exclusively. the test suite includes overrepresentation of certain frequent adjectives. the adjective good occurs more than 270 times in the input sentence, whereas some rarer adjectives, such as opaque only occur twice. adding more unique adjectives to make the data even more representative and balancing the data is left for future work. another caveat of the test suite is that for now it only probes the sentence representations for a set of negated antonyms that belong to the adjective class. as some of the other word classes also include words that have a related opposing concept (e.g., accept -reject in the verb class), and it would be equally important to assess how language models understand the relation of the words in other word classes. additionally, the test suite consists only of one certain negation pattern: not + adjective. adding examples with more variable negation patterns would require an adapted filtering method to extract the sentences from a paraphrase corpus or more manual work to ensure high-quality paraphrases of the sort (e.g., i walked -i didn't stand still.) augmenting the test suite with test examples of more varied negation patterns, as well as antonymous tokens from different word classes is left for future work. ## related work previous work has focused on understanding negation, on the one hand, and antonymy, on the other. @xcite show that pretrained language models (transformer-xl @xcite , elmo @xcite and bert @xcite in this case) are poor at recognising the difference between a sentence in affirmative or negative form when they are queried with a negative cloze test, and are prone to predict the same token regardless of the polarity of the sentence. ettinger (2020) study how bert understands negation with similar minimal pairs to our test suite. however, they probe bert for predicting one word token in a sentence pair where one sentence is a negated version of the other (e.g., most smokers find that quitting is very __. -most smokers find that quitting isn't very __.) bert does not seem to be highly robust for the transformation, and it does not seem that negation alone suffices to prime the model for systematically predicting the opposite of the prediction in the affirmative sentence @xcite . @xcite show that nli models are not robust to negation by analysing models on a new benchmark designed to assess how models understand negation. in a similar spirit, assessing a multilingual language model fine-tuned for nli on a test suite of minimal pairs of label-changing and label-preserving negations, hartmann et al. (2021) find that multilingual language models are not fully aware of the effects that a negation marker can have on sentence semantics. furthermore, nli models' difficulty to represent negations reliably has been traced to training data, suggesting that models trained on snli @xcite or mnli @xcite do not properly learn to reason with expressions that include negation @xcite . @xcite analyse different pretraining objectives for predicting textual entailment on various function word probing tasks, one of which assesses models' understanding of negation in a similar manner to our test suite. they find that the natural language inference models outperform other pretraining objectives in representing negation, mostly owing to nli models' capability to represent explicit negation. however, analysis of examples that were difficult for a state-of-the-art nli model has suggested that antonymy and negation are challenging phenomena to represent reliably, as models do not recognise antonymous relations as semantically opposing and may associate explicit negation to contradiction in neutral or entailed examples @xcite . in addition to analysing language models' understanding of natural language in textual entailment, representation of antonyms has been studied for instance by comparing the mapping of negated adjectives in vector space @xcite . bert has also been adapted to perform a cloze problem for predicting antonyms in context @xcite . additionally, previous research has focused on learning vector-based representations of word semantics that can model the relationship between distributionally similar but semantically opposing words better (e.g., @xcite . @xcite study how lan-guage models understand semantic compositionality with respect to contrasting meanings but focus on transformations of (negative) polarity items. ## conclusions we have presented a novel test suite, semantoneg, designed to probe pretrained language models for the understanding of the relationship between negation and antonymy. contradicting examples in the test suite are close to the input by design and lead to a challenging benchmark. in order to succeed on our test suite, a model needs to recognise the semantic opposites invoked either by antonym substitution or by an insertion or a deletion of a negation marker. equally, the model needs understanding of semantic compositionality to understand how the operations affect semantics of the sentence when performed together. we have evaluated publicly available pretrained sentence representation models and reported results that display a large amount of variation when assessed on the new test suite. surprisingly, dedicated paraphrase models are not among the best performing models and deliver rather poor results in many cases, whereas fine-tuning to natural language inference seems very beneficial for the task. general-purpose models are overall not very good at recognising our examples either, except for recent very large multi-task models such as t5-xl. our findings highlight that models that fare well in established natural language understanding benchmarks may still have crucial deficiencies in representing certain, rather typical, linguistic constructions and may produce critical mistakes. as a result, more structured test suites are necessary for assessing how the pretrained models understand language. this paper provides another contribution in that direction. there are various avenues in future work we would like to explore. first of all, we need to further test the scaling effects when moving to very large language models such as gpt-3 @xcite . the t5 results already indicate that size matters but it is too early to draw general conclusions. furthermore, we plan to investigate prompting as an alternative to vector similarity. however, prompt engineering is a challenging task in itself and we will need to explore the influence of prompts on results we can expect. finally, we would also like to move to other languages and potentially crosslingual setups.