P rop S egm E nt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition (2023)

Abstract: The widely studied task of Natural Language Inference (NLI) requires a system to recognize whether one piece of text is textually entailed by another, i.e. whether the entirety of its meaning can be inferred from the other. In current NLI datasets and models, textual entailment relations are typically defined on the sentence- or paragraph-level. However, even a simple sentence often contains multiple propositions, i.e. distinct units of meaning conveyed by the sentence. As these propositions can carry different truth values in the context of a given premise, we argue for the need to recognize the textual entailment relation of each proposition in a sentence individually. We propose PropSegmEnt, a corpus of over 45K propositions annotated by expert human raters. Our dataset structure resembles the tasks of (1) segmenting sentences within a document to the set of propositions, and (2) classifying the entailment relation of each proposition with respect to a different yet topically-aligned document, i.e. documents describing the same event or entity. We establish strong baselines for the segmentation and entailment tasks. Through case studies on summary hallucination detection and document-level NLI, we demonstrate that our conceptual framework is potentially useful for understanding and explaining the compositionality of NLI labels.

## introduction natural language inference (nli), or recognizing textual entailment (rte), is the task of determining whether the meaning of one text expression can be inferred from another @xcite . given two pieces of text (p, h), we say the premise p entails the hypothesis h if the entirety of h's meaning can be most likely inferred true after a human reads p . if some units of meaning in h are contradicted by, or cannot be determined ## motivations & design challenges our study concerns the challenges of applying nli/rte task formulations and systems in realworld downstream applications and settings. as textual entailment describes the relation between the meanings of two text expressions, one natural type of downstream use cases for nli systems is to identify alignments and discrepancies between the semantic content presented in different documents/sources @xcite @xcite . our study is motivated by the task of comparing the content of topically-related documents, e.g. news documents covering the same event @xcite , or wikipedia pages from different languages for similar entities @xcite . as existing nli datasets typically define the textual entailment relations at the sentence or paragraph level @xcite , nli systems trained on such resources can only recognize whether or not the entirety of a hypothesis sentence/paragraph is entailed by a premise. however, we estimate that, in these two domains, around 2 8875 90% of the sentences that convey any informational propositions contain more than one proposition (figure 1 ). in the presence of multiple propositions, partial entailment @xcite describes the phenomenon where only a subset of propositions in the hypothesis is entailed by the premise. partial entailment is 3× more common than full-sentence entailment. in our corpus, we observe that, given two topically related documents from news or wikipedia, 46% of sentences in one document have at least some information supported by the other document (figure 2 ). but 74% of such sentences are partially entailed, with only some propositions supported by the other document. in this sense, a sentence-level nli model can only detect a quarter of sentences that have meaningful entailment relations. in applications that seek a full understanding of cross-document semantic links, there is thus 4× headroom, a significant blind spot for sentence-level nli models. as we observe that most natural sentences are compositional, i.e. contain more than one proposition, we argue for the need to decompose and recognize textual entailment relation at the more granular level of propositions. in other words, instead of assessing the entire hypothesis as one unit in the context of a premise, we propose to evaluate the truth value of each proposition individually, and aggregate for the truth value of the hypothesis. current predicate-argument based methods often fail to extract all propositions in a sentence. the linguistic notion of a proposition refers to a single, contextualized unit of meaning conveyed in a sentence. in the nlp community, propositions are usually represented by the predicate-argument structure of a sentence. for example, resources like framenet @xcite , propbank @xcite , nombank @xcite , among others, represent a proposition by a predicate (verbal, nominal, etc.), with arguments filling its thematic proto-roles. such resources facilitate the development of srl systems @xcite for proposition extraction, with a closed, predefined set of proto-roles. to increase the coverage of propositions extracted, openie formulations @xcite @xcite were proposed to forgo the limits on fixed semantic roles and account for both explicit and implicit predicates. however, we observe that openie systems often fail to account for the complete set of propositions in a sentence. in many cases, e.g. the andy warhol's hometown example in furthermore, @xcite and fatahi @xcite observe that neural ope-nie systems tend to extract long arguments that could potentially be decomposed into more compact propositions. for textual entailment, we argue for the need to extract the complete set of propositions in their most compact form, due to the fact that their truth value could vary individually. to illustrate the difference between openie and our approach, we offer a list of example propositions from our proposed propsegment dataset, and compared them to extractions from rule-based and neural openie systems, in appendix d. ## propsegment dataset wikipedia documents, with human annotated propositions and entailment labels. ## analysis and discussion we exemplify the utilities of our propositional segmentation and entailment framework, which we refer to as propnli, through the lens of two downstream use cases, e.g. summary hallucination detection ( § 6.1), and document-level nli w/ variablelength hypotheses ( § 6.2). ## conclusion in this paper, we presented propsegment, the first large-scale dataset for studying propositionlevel segmentation and entailment. we demonstrate that segmenting a text expression into propositions, i.e. atomic units of meanings, and assessing their truth values would provide a finer-grained characterization of the textual entailment relation between two pieces of text. beyond nli/rte tasks, we hypothesize that proposition-level segmentation might be helpful in similar ways for other text classification tasks as well. we hope that propseg-ment will serve as a starting point, and pave a path for research forward along the line.