Exploring Factual Entailment with NLI : A News Media Study (2024)

Abstract: We explore the relationship between factuality and Natural Language Inference (NLI) by introducing FactRel – a novel annotation scheme that models factual rather than textual entailment, and use it to annotate a dataset of naturally occurring sentences from news articles. Our analysis shows that 84% of factually supporting pairs and 63% of factually undermining pairs do not amount to NLI entailment or contradiction, respectively, suggesting that factual relationships are more apt for analyzing media discourse. We experiment with models for pairwise classification on the new dataset, and find that in some cases, generating synthetic data with GPT-4 on the basis of the annotated dataset can improve performance. Surprisingly, few-shot learning with GPT-4 yields strong results on par with medium LMs (DeBERTa) trained on the labelled dataset. We hypothesize that these results indicate the fundamental dependence of this task on both world knowledge and advanced reasoning abilities.

## introduction in recent years, the concept of factuality in news media has garnered increasing attention. studies increasingly examine the relation between factsas presented in news coverage -and phenomena such as political polarization, misinformation and fake news @xcite @xcite . as a result, the ability to model factual relations between claims becomes increasingly important. this has led to a line of work on automated fact-checking, which involves textual pipelines for detecting and evaluating factual claims @xcite . in automatic fact-checking, fact verification is predominantly addressed via the natural language inference (nli) task, also known as recognizing textual entailment (rte) @xcite @xcite , which has been used for decades for evaluating natural language understanding capabilities @xcite . nli is traditionally formulated as a categorical classification task between a premise p and a hypothesis h, where p can either contradict, entail or be neutral with respect to h. large nli datasets such as @xcite have become highly popular, leading nli to be adapted to various uses such as zero-shot classification @xcite and semantic similarity @xcite . in fact verification, nli is used to evaluate the relations between a candidate fact and trusted pieces of evidence @xcite . however, the adequacy of nli for analyzing factual relationships in news media is hindered by two primary reasons, relating to the nature of the task as well as to the characteristics of commonly used nli datasets. first, large nli datasets such snli and mnli define the pairwise relationship in terms of necessity of meaning @xcite . thus, in mnli an entailment is defined to be the case whereby a hypothesis "is necessarily true or appropriate whenever the premise is true", and similarly a contradiction is when the hypothesis "is necessarily false or inappropriate whenever the premise is true" @xcite . however, these types of relationships may be too restrictive for the analysis of media discourse, where explicit contradictions and entailments are likely to be rare, as such discourse tends take place in the margins of plausibility. secondly, texts in popular nli datasets considerably differ from news texts. while sentences in nli datasets tend to be short, simple, highly generic and convey a single idea or statement, media sentences tend to be longer, more complex, more specific and convey multiple pieces of information. a common feature of nli datasets such as rte, snli and mnli is that while premises are naturally occurring texts, the hypotheses are specifically written to correspond to the categories @xcite . while this method is effective in generating large amounts of data, constructed hypotheses are likely to express a simple relationship to the premise and thus not resemble pairs of naturally occurring sentences. additionally, @xcite notes that these datasets feature strictly logical relationships and stresses the need for datasets capturing other sorts of inferential relationships. in this work, we set out to examine the relationship between nli and textual factuality. for this purpose, we have developed a novel annotation scheme that expresses factual rather than textual entailment, encoding each pair of sentences with the relation of factual support, factual undermining, or neither. we have annotated a new dataset of naturally occurring sentence pairs from news media using both our factual entailment scheme and nli, enabling a comparison of the schemes on news media. we also check the ability of recent generative llms (gpt-4) to generate such pairs correctly. we end with a set of experiments that demonstrate the ability to learn the factual entailment task using fine-tuned models as well as generative llms, and draw conclusions regarding the task's relation to real world knowledge in comparison to nli. overall, we analyze differences between nli and factual entailment in their scope, relevance to news text and dependence on world knowledge, and show potential for new ways to model factual relations. ## factual entailment for the purpose of exploring the relationship between factual relations and textual entailment, we have developed factrel, a novel annotation scheme encoding the factual entailment between pairs of sentences. similarly to nli, factrel is a 3-category pairwise classification task. given a premise p and a hypothesis h, p can either factually support h (support), factually undermine h (undermin-ing), or be factually neutral w.r.t h (neutral). p is said to factually support h when p being true would make h more plausible or likely to be true, compared to a situation in which the truth value of p is unknown. p is said to factually undermine h when p being true would make h less plausible or likely to be true, compared to a situation in which the truth value of p is unknown. finally, p is said to be factually neutral w.r.t to p when p's truth has no bearing on the plausibility of h, and the likelihood of h would not change if p was known to be either true or false. while this example is factually neutral, and its nli label is neutral as well. ## experiments we tackle the task of factual entailment with several types and sizes of language models. baseline model. as a simple baseline, we embed the premise and hypothesis using the uae-large-v1 encoder @xcite and calculate the cosine similarity between them, on which we train a decision tree with a max depth of 10. zero shot and few shot (no training). we use two models in a zero-shot setting. first, we utilize a state-of-the-art nli model trained on many nli datasets @xcite . the nli model, based on deberta v3 large @xcite , was used as if the nli categories are equivalent to fac-trel categories (e.g., contradiction equals undermining). second, we utilize gpt-4 in a zero-shot setting provided only with a description of the task and the categories. we additionaly use gpt-4 in a 3-shot setting, adding three example pairs, one for each category. trained models. we fine-tune several encoder models: roberta-base @xcite , deberta v3 large @xcite , and de-berta v3 sota nli checkpoint @xcite . training variants included training with class weights and utilizing focal loss. we also fine-tune gpt-3.5 using openai's api with the recommended settings. all the models were tested using two types of training sets -the core training set, and the augmented set with gpt-4 synthetic pairs added. full technical details of the training setup are laid out in appendix b. macro-f1 results on the validation set for the baseline model, the stock nli model and the top performing models are reported in ## conclusion in this paper we explored the relationship between nli and factual relations. for this purpose, we designed a new annotation scheme for factual entailment, factrel; examined it in comparison to nli on a sample of annotated pairs from news coverage; and examined the performance of various models on the task. we have shown that factual entailment relations are significantly more common in news articles in comparison to semantic entailment, thus underlining the shortcomings of nli when applied to naturally occurring text. we have also shown that gpt-4 performs better in a few-shot setting than smaller models trained on the entire training set. moreover, gpt-4's performance even in a zero-shot setting is competitive with other models. the success of these llms, even with significantly less data, can give us insight on the challenge involved in the factrel task and how it differs from nli. nli is a fundamentally semantic task, as determining whether p entails or contradicts h hinges on understanding the meaning of the words and concepts employed in both. thus, if p semantically entails h, then h itself must be included either explicitly or implicitly in p itself. the relations are therefore to be found in the meaning of the words. modelling factual relationships, on the other hand, also requires a significant amount of background knowledge on the referents of the words, a detailed world model, and nuanced reasoning abilities. thus, in order to identify that the premise "twitter has locked trump's account for 12 hours, and required the removal of the tweets" supports the hypothesis "facebook locked trump's account for 24 hours following two policy violations", it is required to not only understand the words and concepts, but to also be able to infer why a social network might lock one's account, and why such actions on two social networks are likely to cooccur. it is thus hypothesized that llms that have broad world knowledge, and especially those that excel at reasoning such as gpt-4, are well placed for this task, and their world knowledge and rea-soning capabilities can compensate for decreased exposure to training data. finally, the addition of synthesized data improves performance of the top medium size lm, showing that data synthesis can be successfully employed on this task. however, this improvement is not consistent for all configurations.