Improving Evidence Retrieval with Claim-Evidence Entailment (2021)

Abstract: Claim verification is challenging because it requires first to find textual evidence and then apply claim-evidence entailment to verify a claim. Previous works evaluate the entailment step based on the retrieved evidence, whereas we hypothesize that the entailment prediction can provide useful signals for evidence retrieval, in the sense that if a sentence supports or refutes a claim, the sentence must be relevant. We propose a novel model that uses the entailment score to express the relevancy. Our experiments verify that leveraging entailment prediction improves ranking multiple pieces of evidence.

## introduction claim verification verifies the credibility of a textual claim by inferring relevant and reliable textual evidence. an example in this space is fever, which regards wiki pages as potential evidence and creates claims by crowdsourcing @xcite . they propose a three-step pipeline: (i) document-level evidence retrieval; (ii) sentence-level evidence retrieval; (iii) claim-evidence entailment. some works follow the pipeline and propose new models to improve claim verification @xcite @xcite , while other works combine the second and the third step and leverage all possible sentences for claim verification @xcite . we refer to the former as the pipeline framework and the latter as the multi-task framework. the pipeline framework restricts a few sentences, and therefore it may not cover relevant evidence. the multi-task framework includes all possible sentences, where irrelevant ones may bring overwhelming noise and hurt claim verification. we argue that previous works focus on improving individual components but neglect to examine how those components connect. for example, will the entailment improve the retrieval? we hypothesize that claim-evidence entailment can provide useful signals for evidence retrieval: if a sentence supports or refutes a claim, the sentence must be relevant. as in to our knowledge, this is the first work that uses the entailment prediction to measure relevancy. our experiment demonstrates that a reliable entailment prediction improves evidence retrieval. this is beyond previous works that merely share lowlevel text encoder and train the two steps together. ## method we adopt two base models, the decomposable attention (da) model @xcite and the enhanced sequential inference (esi) model @xcite , to encode claim-evidence pairs. both models are designed for textual entailment @xcite @xcite . although there are more methods in the area of textual entailment @xcite @xcite @xcite @xcite , we prefer the da model and the esi model because they have been widely applied for sentence retrieval and claim-evidence entailment @xcite @xcite . ## experiments the focus of the experiments is to understand if the entailment score can benefit the retrieval. we conducted experiments on the fever dataset @xcite 1 . fever contained 80,035 support claims, 29,775 refute claims, and 35,639 note-noughinfo claims for training. the shared task of fever released 6,666 support claims, 6,666 refute claims, and 6,666 notenoughinfo claims for validation, and held another blind test set of 6,666 support claims, 6,666 refute claims, and 6,666 notenoughinfo claims. we considered two scenarios in our experiment, and we describe them as follows: ec: short for entailment comparison. we explored the claim-evidence entailment by augmenting gold evidence with irrelevant sentences. we varied the irrelevant sentences so that we maintained the recall of the evidence retrieval. this scenario emphasizes the importance of evidence retrieval. ## conclusion in this work, we show that leveraging the entailment prediction can improve evidence retrieval when the entailment step produces a reliable result. in the future, we will adopt pre-trained models, e.g., bert @xcite , for our experiments. we expect improvement because bert shows competitive results on the textual entailment tasks @xcite . ethical consideration this work conducts experiments on benchmark datasets that have been extensively studied in the literature. although the datasets used in the work was manually annotated, there is no identity characteristics. also, we use rnn-based models with only a few layers, which are more eco-friendly compared to transformer based models.