A survey on Recognizing Textual Entailment as an NLP Evaluation (2020)

Abstract: Recognizing Textual Entailment (RTE) was proposed as a unified evaluation framework to compare semantic understanding of different NLP systems. In this survey paper, we provide an overview of different approaches for evaluating and understanding the reasoning capabilities of NLP systems. We then focus our discussion on RTE by highlighting prominent RTE datasets as well as advances in RTE dataset that focus on specific linguistic phenomena that can be used to evaluate NLP systems on a fine-grained level. We conclude by arguing that when evaluating NLP systems, the community should utilize newly introduced RTE datasets that focus on specific linguistic phenomena.

## introduction as nlp technologies are more widely adopted, how to evaluate nlp systems and how to determine whether one model understands language or generates text better than another is an increasingly important question. recognizing textual entailment ( @xcite , the task of determining whether the meaning of one sentence can likely be inferred from another was introduced to answer this question. we begin this survey by discussing different approaches over the past thirty years for evaluating and comparing nlp systems. next, we will discuss how rte was introduced as a specific answer to this broad question of how to best evaluate nlp systems. this will include a broad discussion of efforts in the past three decades to build rte datasets and use rte to evaluate nlp models. we will then highlight recent rte datasets that focus on specific semantic phenomena and conclude by arguing that they should be utilized for evaluating the reasoning capabilities of downstream nlp systems. ## evaluating nlp systems the question of how best to evaluate nlp systems is an open problem intriguing the community for decades. a 1988 workshop on the evaluation of nlp systems explored key questions for evaluation. these included questions related to valid measures of "black-box" performance, linguistic theories that are relevant to developing test suites, reasonable expectations for robustness, and measuring progress in the field @xcite . the large number of acl workshops focused on evaluations in nlp demonstrate the lack of consensus on how to properly evaluate nlp systems. some workshops focused on: 1) evaluations in general @xcite ; 2) different nlp tasks, e.g. machine translation @xcite and summarization @xcite ; or 3) contemporary nlp approaches that rely on vector space representations @xcite @xcite . in the quest to develop an ideal evaluation framework for nlp systems, researchers proposed multiple evaluation methods, e.g. eagles @xcite , tsnlp @xcite , fracas @xcite , sense-val @xcite , clef @xcite , and others. these approaches are often divided along multiple dimensions. here, we will survey approaches along two dimensions: 1) intrinsic vs. extrinsic evaluations; 2) general purpose vs task specific evaluations. foot_2 ## recognizing textual entailment nlp systems cannot be held responsible for knowledge of what goes on in the world but no nlp system can claim to "understand" language if it can't cope with textual inferences. @xcite recognizing and coping with inferences is key to understanding human language. while nlp systems might be trained to perform different tasks, such as translating, answering questions, or extracting information from text, most nlp systems require understanding and making inferences from text. therefore, rte was introduced as a framework to evaluate nlp systems. rooted in linguistics, rte is the task of determining whether the meaning of one sentence can likely be inferred from another. unlike the strict definition of entailment in linguistics that "sentence a entails sentence b if in all models in which the interpretation of a is true, also the interpretation of b is true" (janssen, 2011), rte relies on a fuzzier notion of entailment. for example, annotation guidelines for an rte dataset foot_3 stated that in principle, the hypothesis must be fully entailed by the text. judgment would be false if the hypothesis includes parts that cannot be inferred from the text. however, cases in which inference is very probable (but not completely certain) are still judged as true. @xcite starting with fracas, we will discuss influential work that introduced and argued for rte as an evaluation framework. fracas over a span of two years (december 1993 -january 1996), @xcite developed fracas as "an inference test suite for evaluating the inferential competence of different nlp systems and semantic theories". created manually by many linguists and funded by fp3-lre, 6 fracas is a "semantic test suite" that covers a range of semantic phenomena categorized into 9 classes. these are generalized quantifiers, plurals, anaphora, ellipsis, adjectives, comparatives, temporal reference, verbs, and attitudes. based on the descriptions in §2, we would classify fracas as an intrinsic evaluation and a general purpose test suite. examples in fracas contain a premise paired with a hypothesis. premises are at least one sentence, though sometimes they contain multiple sentences, and most hypotheses are written in the form of a question and the answers are either yes, no, or don't know. maccartney (2009) (specifically chapter 7.8.1) converted the hypotheses from questions into declarative statements. 7 pascal rte challenges with a similar broad goal as fracas, the pascal recognizing textual entailment challenges began as a "generic evaluation framework" to compare the inference capabilities of models designed to perform different tasks, based on the intuition "that major inferences, as needed by multiple applications, can indeed be cast in terms of textual entailment" @xcite . unlike fracas's goal of determining whether a model performs distinct types of reasoning, the pascal rte challenges primarily focused on using this framework to evaluate models for distinct, real-world downstream tasks. thus, the examples in the pascal rte datasets were extracted from downstream tasks. the process was referred to as recasting in the thesis by @xcite . nlu problems were reframed under the rte framework and candidate sentence pairs were extracted from existing nlp datasets and then labeled under variations of the rte definition (including the quote above @xcite ). 8 for example, the rte1 data came from 7 tasks: comparable documents, reading comprehension, question answering, information extraction, machine translation, information retrieval, and paraphrase acquisition. 9 starting with @xcite , there have been eight iterations of the rte challenge, with the most recent being @xcite . snli and mnli the most popular recent rte datasets, stanford natural language inference (snli; @xcite and its successor multi-nli @xcite , each contain over half a million examples and enabled re-searchers to apply data-hungry deep learning methods to rte. unlike the rte datasets, these two datasets were created by eliciting hypotheses from humans. crowd-source workers were tasked with writing one sentence each that is entailed, neutral, and contradicted by a caption extracted from the flickr30k corpus @xcite . next, the label for each premise-hypothesis pair in the development and test sets were verified by multiple crowd-source workers and the majority-vote label was assigned for each example. ## revisiting rte as an nlp evaluation there has been little evidence to suggest [that rte models] capture the type of compositional or world knowledge tested by datasets like the fracas test suite. @xcite as large scale rte datasets, like snli and mnli, rapidly surged in popularity, some researchers critiqued the datasets' ability to test the inferential capabilities of nlp models. a high accuracy on these datasets does not indicate which types of reasoning rte models perform or capture. as noted by @xcite , "researchers compete on which system achieves the highest score on 11 @xcite explored rte-1 and blake (2007) analyzed rte-2 and rte-3. a test set, but this itself does not lead to an understanding of which linguistic properties are better captured by a quantitatively superior system." in other words, the single accuracy metric on these challenges indicates how well a model can recognize whether one sentence likely follows from another, but it does not illuminate how well nlp models capture different semantic phenomena that are important for general nlu. this issue was pointed out regarding the earlier pascal rte datasets. in her thesis that presented "a test suite for adjectival inference developed as a resource for the evaluation of computational systems handling natural language inference." @xcite blamed "the difficulty of defining the linguistic phenomena which are responsible for inference" as the reason why previous rte resources "concentrated on the creation of applications coping with textual entailment" rather than "resources for the evaluation of such applications." as current studies began exploring what linguistic phenomena are captured by neural nlp models and auxiliary diagnostic classifiers became a common tool to evaluate sentence representations in nlp systems, ( §2.3), the community saw an interest in developing rte datasets that can provide insight into what type of linguistic phenomena are captured by neural, deep learning models. in turn, the community is answering @xcite plea to the community to test "more kinds of inference" than in previous rte challenge sets. here, we will highlight recent efforts in creating datasets that demonstrate how the community has started answering chatzikyriakidis et al.'s call. we group these different datasets based on how they were created. ## recommendations these efforts resulted in a consistent format and framework for testing how well contemporary, deep learning nlp systems capture a wide-range of linguistic phenomena. however, so far, most of these datasets that target specific linguistic phenomena have been used to solely evaluate how well rte models capture a wide range of phenomena, as opposed to evaluating how well systems trained for more applied nlp tasks capture these phenomena. since rte was introduced as a framework to evaluate how well nlp models cope with inferences, these newly created datasets have not been used to their full potential. a limited number of studies used some of these datasets to evaluate how well models trained for other tasks capture these phenomena. @xcite evaluated how well a bilstm encoder trained as part of a neural machine translation system capture phenomena like semantic proto-roles, paraphrastic inference, and anaphora resolution. @xcite used their rte datasets focused on function words to evaluate different encoders trained for tasks like ccg parsing, image-caption matching, predicting discourse markers, and others. those studies relied on the use of auxiliary classifiers as a common probing technique to evaluate sentence representations. as the community's interest in analyzing deep learning systems increases, demonstrated by the recent work relying on @xcite and improving upon @xcite @xcite the popular auxiliary 14 https://gluebenchmark.com/diagnostics classifier-based diagnostic technique, we call on the community to leverage the increasing number of rte datasets focused on different semantic phenomena ( another recent line of work uses rte to evaluate the output of text generation systems. for example, @xcite explore "whether textual entailment predictions can be used to detect errors" in abstractive summarization systems and if errors "can be reduced by reranking alternative predicted summaries" with a textual entailment system trained on snli. while @xcite results demonstrated that current models might not be accurate enough to rank generated summaries, @xcite demonstrate that contemporary transformer models trained on the adversarial nli dataset @xcite ) "achieve significantly higher accuracy and have the potential of selecting a coherent summary." therefore, we are encouraged that researchers might be able to use many of these new rte datasets focused on specific phenomena to evaluate the coherency of machine generated text based on multiple linguistic phenomena that are integral to entailment and nlu. this approach can help researchers use the rte datasets to evaluate a wider class of models, specifically non-neural models, unlike the auxiliary classifier or probing methods previously discussed. the overwhelming majority, if not all, of these rte datasets targeting specific phenomena rely on categorical rte labels, following the common for-mat of the task. however, as @xcite recently illustrated, categorical rte labels do not capture the subjective nature of the task. instead, they argue for scalar rte labels that indicate how likely a hypothesis could be inferred by a premise. @xcite similarly lament how labels are currently used in rte datasets. pavlick and kwiatkowski demonstrate that a single label aggregated from multiple annotations for one rte example minimizes the "type of uncertainty present in [valid] human disagreements." instead, they argue that a "representation should be evaluated in terms of its ability to predict the full distribution of human inferences (e.g., by reporting crossentropy against a distribution of human ratings), rather than to predict a single aggregate score (e.g., by reporting accuracy against a discrete majority label or correlation with a mean score)." future rte datasets targeting specific phenomena that contain scalar rte labels from multiple annotators (following chen et al.'s and pavlick and kwiatkowski's recommendations) can provide more insight into contemporary nlp models. ## conclusion with the current zeitgeist of nlp research where researchers are interested in analyzing state-of-theart deep learning models, now is a prime time to revisit rte as a method to evaluate the inference capabilities of nlp models. in this survey, we discussed recent advances in rte datasets that focus on specific linguistic phenomena that are integral for determining whether one sentence is likely inferred by another. since rte was primarily motivated as an evaluation framework, we began this survey with a broad overview of prior approaches for evaluating nlp systems. this included the distinctions between instrinsic vs extrinsic evaluations and general purpose vs task specific evaluations. we discussed foundational rte datasets that greatly impacted the nlp community and included critiques of why they do not fulfill the promise of rte as an evaluation framework. we highlighted recent efforts to create rte datasets that focus on specific linguistic phenomena. by using these datasets to evaluate sentence representations from neural models or rank generated text from nlp systems, researchers can help fulfil the promise of rte as unified evaluation framework. ultimately, this will help us determine how well models understand language on a fine-grained level.