C laim D iff: Comparing and Contrasting Claims on Contentious Issues (2023)

Abstract: With the growing importance of detecting misinformation, many studies have focused on verifying factual claims by retrieving evidence. However, canonical fact verification tasks do not apply to catching subtle differences in factually consistent claims, which might still bias the readers, especially on contentious political or economic issues. Our underlying assumption is that among the trusted sources, one’s argument is not necessarily more true than the other, requiring comparison rather than verification. In this study, we propose ClaimDIff, a novel dataset that primarily focuses on comparing the nuance between claim pairs. In ClaimDiff, we provide human-labeled 2,941 claim pairs from 268 news articles. We observe that while humans are capable of detecting the nuances between claims, strong baselines struggle to detect them, showing over a 19% absolute gap with the humans. We hope this initial study could help readers to gain an unbiased grasp of contentious issues through machine-aided comparison.

## introduction with an ever-increasing amount of textual information on the web, many researchers have focused on detecting misinformation from diverse sources, such as fake news @xcite and rumor tweets @xcite . in particular, fact verification has become a popular task due to its utility and the availability of reliable datasets such as fever @xcite . for contentious issues, however, fact verification alone is not sufficient; comparing and contrasting claims from the opposing sides are necessary to gain an unbiased understanding of the issue. figure 1 presents two articles reporting on the treatment of long covid -long-term physical and mental symptoms that can occur after covid-19 infection. although both articles are published in the same week, their perspectives on long covid are quite different; article a downplays the risk of long covid stating that it has become less common in the recent covid variants, while article b expresses concerns about the increase in the number of people suffering from long covid. in this way, even articles from trusted sources may provide biased views about an issue. in this paper, we present claimdiff, a novel dataset consisting of 2,941 claim pairs extracted from 268 news articles on 134 contentious issues. 1 claimdiff comes in two variations-claimdiff-s and claimdiff-w-each consisting of labels targeting a different relation: determining whether a claim strengthens and weakens another claim, respectively. for instance, the two claims in figure 1 weaken each other, providing inconsistent perspectives surrounding the undisputed factual information -the human body has a natural ability to heal from long covid. more specifically, claim a argues that medical treatments should be geared toward supporting the natural ability to heal, while claim b calls for more active interventions for fast recovery. claimdiff focuses on recognizing such relations between two claims on an issue; this is distinguished from existing tasks such as fact verification -verifying the veracity of a claim using evidence text @xcite -and stance detection -identifying the stance of a claim toward a topic of interest @xcite . we also demonstrate the efficacy of claimdiff on two tasks it supports -relation classification and rationale extraction -and an extended ap-… similarly, the uk office of national statistics reported that for double-vaccinated participants, self-reported long covid was less common after infections compatible with the omicron ba.1 variant than the delta variant. … it provides additional evidence that only a small proportion of people infected with an omicron variant will experience long covid. … [but taking a holistic view of health and bolstering the body's natural ability to heal itself could be an important way for those struggling with long covid to combat their symptoms.] ## claimdiff in this section, we formally define the tasks supported by claimdiff, describes how the dataset is constructed, and provide the statistics and analysis of the resulting dataset. ## task 2 -rationale extraction we perform rationale extraction over "strengthening" and "weakening" pairs, which have positive labels for claimdiff-s and claimdiff-w. because of the low appearance of "weakening" pairs, training individual models for each task is challenging. therefore, unlike relation classification, we train a single model to extract rationales from both "strengthening" and "weakening" pairs. we experiment with extractive and generative models. the evaluation is also conducted on combined sets of claimdiff-s and claimdiff-w. ## extension: document-level claimdiff suppose we want to compare the articles with opposing views on contentious issues. for instance, there are two articles about a topic, "will gas prices come down soon or stay high?". one forecasts the increase in gas prices, while the other supports the prices have already reached a peak. a single stance label on the relation between the articles (i.e., whether one article supports or opposes the other) might not be enough to understand the complex relations of claims in the articles. claimdiff can be applied to provide a granular-level comparison between two articles. document-level claimdiff enables to provide information about which arguments of the first article strengthen or weaken the views of the other. we provide a live demo for document-level extension with robertalarge model. 11 as an example, the demo result of the above topic is shown in appendix g. in order for the real-world document-level comparison scenario, we evaluate our baseline models on the test-doc dataset, which follows the realworld label distribution. specifically, the testdoc dataset includes all non-overlapping sentence pairs, which were originally filtered out for the test dataset construction as described in section 3.2. the results are shown in although the fine-grained comparison is helpful for understanding contentious issues, looking over the whole article pair is costly. future work includes providing summarized statistics from finegrained comparisons. for example, the ratio of strengthening / weakening pairs can represent how much the two articles oppose each other. we can further extend claimdiff to compare more than two articles with summarized results, and even compare between different presses. ## conclusion this paper presents claimdiff , a new benchmark dataset of 2.9k annotation to compare claims in news articles on contentious issues. unlike the previous fact verification, claimdiff focues on comparing the nuance between claim pairs from trusted sources, whether one claim strengthens or weakens the other. we experiment with pre-trained language models in finetuning, parameter-efficient finetuning, and zero-shot approaches. the results show a significant room for improvement with over 19% absolute gap between human and model performance. we further suggest document-level claimdiff as a real-world application and show its potential by presenting the baseline performance on the test-doc dataset that follows the real-world distribution. we hope this initial study could pave the way for providing an analysis tool for article readers to obtain an unbiased understanding of contentious issues.