W i CE : Real-World Entailment for Claims in W ikipedia (2023)

Abstract: Textual entailment models are increasingly applied in settings like fact-checking, presupposition verification in question answering, or summary evaluation. However, these represent a significant domain shift from existing entailment datasets, and models underperform as a result. We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim, and a minimal subset of evidence sentences that support each subclaim. To support this, we propose an automatic claim decomposition strategy using GPT-3.5 which we show is also effective at improving entailment models’ performance on multiple datasets at test time. Finally, we show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.

## introduction textual entailment @xcite and natural language inference @xcite @xcite are longstanding problems in nlp that take many forms. the snli dataset has a stated purpose to use nli "as a tool for the evaluation of domain-general approaches to semantic representation" @xcite . however, this is far from how nli is used today, e.g., to validate qa system outputs @xcite , evaluate generated summaries @xcite or understand knowledge-grounded dialog @xcite @xcite . there are some major gaps when applying modern entailment systems to these tasks. first is the fact that many nli datasets target short premises, often single sentences, such as vitaminc (schuster 1 our data is available at: https://github.com/ ryokamoi/wice ## the wice dataset we aim to annotate claims that are: (1) naturallyoccurring; we extract claims from wikipedia and its cited documents, where the noise in citations gives realistic negative examples, (2) in-context with surrounding text; this mirrors real use cases where claims occur in discourse context, and (3) fine-grained; we break down complex claims into multiple subclaims with claim-split and provide entailment judgments for both granularities. we also provide token-level annotation for nonsupported tokens. the main altar houses a 17th-century fresco. the fresco is of figures interacting with the framed 13th-century icon of the madonna. the icon of the madonna was painted by mario balassi in 1638. ## experiments on wice we have three main questions for our dataset. ## conclusion we collect wice, a new nli dataset constructed from wikipedia. by comparing sentences in wikipedia against cited evidence documents, we find a rich set of real-world entailment phenomena distinct from those in prior nli datasets. we also show that decomposing complex claims into subclaims can be a valuable pre-processing step for both annotation and entailment prediction.