Exploring Unsupervised Semantic Similarity Methods for Claim Verification in Health Care News Articles (2023)

Abstract: In the 21st century, the proliferation of fake information has emerged as a significant threat to society. Particularly, healthcare medical reporters face challenges when verifying claims related to treatment effects, side effects, and risks mentioned in news articles, relying on scientific publications for accuracy. The accurate communication of scientific information in news articles has long been a crucial concern in the scientific community, as the dissemination of misinformation can have dire consequences in the healthcare domain. Healthcare medical reporters would greatly benefit from efficient methods to retrieve evidence from scientific publications supporting specific claims. This paper delves into the application of unsupervised semantic similarity models to facilitate claim verification for medical reporters, thereby expediting the process. We explore unsupervised multilingual evidence retrieval techniques aimed at reducing the time required to obtain evidence from scientific studies. Instead of employing content classification, we propose an approach that retrieves relevant evidence from scientific publications for claim verification within the healthcare domain. Given a claim and a set of scientific publications, our system generates a list of the most similar paragraphs containing supporting evidence. Furthermore, we evaluate the performance of state-of-the-art unsupervised semantic similarity methods in this task. As the claim and evidence are present in a cross-lingual space, we find that the XML-RoBERTa model exhibits high accuracy in achieving our objective. Through this research, we contribute to enhancing the efficiency and reliability of claim verification for healthcare medical reporters, enabling them to accurately source evidence from scientific publications in a timely manner.

## introduction the rise of misinformation has been greatly amplified by the advent of social media, primarily due to its increased dissemination and influence. one prominent manifestation of this issue is vaccine hesitancy, which has had significant societal repercussions. to illustrate this, a web-based survey @xcite was conducted in june 2021 among 600 adults in florida, revealing substantial exposure to covid-19 vaccine misinformation among participants. approximately 73% reported encountering misinformation in the past six months. an overview of current fake news research is given by @xcite . through the convergence of computational and social science research, they delve into the significance and trajectory of enhancing "digital media literacy" in diverse contexts of news generation and consumption. detecting misinformation has emerged as a critical challenge due to the rapid dissemination of news and the potentially severe consequences associated with false information. however, only a limited number of approaches have been developed to address the dynamic, versatile, and fast-spreading nature of fake news editorials. this challenge becomes even more pronounced in the healthcare domain, where the availability of training data is scarce, and pre-trained models may not be readily applicable. while supervised models rely on manually annotated training data, an unsupervised evidence retrieval and verification approach proves more suitable for quick response and works effectively with low-resource languages and domains. the german healthnewsreview project mediendoktor.de at tu dortmund university evaluates the quality of medical reporting in german-speaking countries. in this paper, we aim to develop a semiautomated tool that will support journalists in their daily work by evaluating the quality of their ongoing reporting and, also, by finding scientific claims in research papers and journalistic articles with a team of highly renowned medical reporters. the medical reporters evaluate the quality of medical reporting in german-speaking countries and assess the quality of print, radio, online, and tv contributions by applying a catalog of criteria in a journalistic peer review process as explained by @xcite . the detailed criteria have been developed following the example of international research projects such as healthnewsreview.org in the usa as discussed by @xcite . the detailed evaluations are published on the website medien-doktor.de, along with advice on scientific reporting, media analyses, and blog posts on selected topics. target groups are not only journalists, but also communication officers at research institutions, teachers, and lay citizens interested in improving their media and scientific literacy. in some newsrooms (among others german press agency, wdr, zdf), these criteria of medien-doktor have already been taught as a possible standard for early-career reporters. nevertheless, many non-specialized newsrooms still lack quality standards in science and medical reporting, particularly among regional media. in contrast to large national media with well-established science sections, regional newspapers often lack editors with scientific backgrounds. as analyses of evaluated articles have shown over the past years, the quality of medical reporting in local journalism usually lags behind the standards of national media. nevertheless, especially in the german media landscape regional media still contribute significantly to opinion-forming and decision-making in wide circles of the population, while at the same time suffering the most from the loss of advertising income and structural upheaval in the time of changing habits of media usage. we, therefore, propose here the first steps towards quality-assuring tools that will help regional but also other media with their daily health reporting by economizing editorial resources. as an initial step towards developing a semiautomated tool, we focus on the "positive effects" criteria from the criteria catalog, which assesses how the potential benefits of therapies, tests, products, or procedures are presented. journalists need to find evidence supporting claims made in scientific publications, a manual and time-consuming process. this presents a major challenge for healthcare reporters, as they rely on scientific publications to verify claims in news articles. healthcare news reporting is further complicated by the fact that journalists often need to translate highly technical language into layperson-friendly terms, as they disseminate scholarly information to audiences outside the research community, including the general public and policymakers. the public relies on the media to learn about new scientific findings, and media portrayals of science significantly influence people's trust in science and their subsequent actions. however, there is a risk of inadvertently spreading misinformation in this process. in this paper, we leverage recent advancements in natural language processing, specifically the transformer architecture, to develop a semanticaware multilingual transformer-based architecture for unsupervised evidence retrieval in healthcare claims. we propose an evidence retrieval approach instead of treating the issue as a simple classification task, thus aiding journalists by providing a list of supporting evidence and reducing their manual workload. we present an architecture that assists factchecking journalists in verifying the veracity of claims by contextually comparing them against evidence found in scientific publications. this paper addresses the following challenges: both these challenges are demonstrated by an example showcased in fig, 1 . the state-of-the-art methods for misinformation detection deal with claim verification in news articles and involve supervised methods, e.g., @xcite . a good survey is @xcite . most authors treat evidence retrieval and claim verification as a single task referred to as factual verification, e.g., @xcite . to overcome the main challenge of supervised approaches, i.e., the time and labor-intensive construction of reliably annotated datasets to train supervised models, some groups explore the potential of unsupervised models for misinformation detection, e.g., @xcite . independent of the modeling approach, the reliability of a source plays an important role in evidence retrieval and the verification process. some work has been done to explicitly compute the reliability of a source, e.g., @xcite . in this section, we will briefly present representative results for each category. the authors of @xcite present a connected system consisting of three homogeneous neural semantic matching models that conduct document retrieval, sentence selection, and claim verification jointly for fact extraction and verification. in @xcite , the authors break down the process into three modules: potentially relevant documents are gathered based on key phrases in the claim, then sentences relevant to the claim are extracted as evidence from these documents, and finally, the classifier discards any evidence deemed irrelevant and uses the remaining to classify the claim's veracity. an approach in which the evidence is gathered automatically for each claim is proposed in @xcite . the approach extracts supporting evidence from the web articles and then selects appropriate text to be treated as evidence sets. a pre-trained model is used to summarize these evidence sets and then these extracted summaries are used as supporting evidence to aid the classification task. the approach collects evidence and prunes to top-krelated news items based on semantic similarity via bertscore. in @xcite the authors proposed integrating credibility assessment as a part of the fact-checking task. the model first strengthens the interaction between claims and relevant articles to discover key evidence fragments, and then incorporates source features of articles and mitigates the interference of extreme semantics to explore more credible evidence discussing the questionable parts of claims. in @xcite , authors worked on the problem of automatically identifying trustworthy information and sources from multiple conflicting data sources. the authors propose to model the conflict resolution problem on data of heterogeneous types using a general optimization framework called crh that integrates the truth-finding process on various data types seamlessly. they model the problem using an optimization framework where truths and source reliability are defined as two sets of unknown variables. the objective is to minimize the overall weighted deviation between the truths and the multi-source observations where each source is weighted by its reliability. in @xcite , authors designed a general framework for the veracity problem and invent an algorithm, called truthflnder, which utilizes the relationships between websites and their information, i.e., a website is trustworthy if it provides many pieces of true information, and a piece of information is likely to be true if it is provided by many trustworthy websites. an iterative method is used to infer the trustworthiness of websites and the correctness of information from each other. in @xcite , the authors follow an unsupervised approach by leveraging a bayesian network model to capture the conditional dependencies among the truths of news, the users' opinions, and the users' credibility and proposed an efficient collapsed gibbs sampling approach to infer the truths of news and the users' credibility without any labeled data. in @xcite , authors propose a novel reputation model to quantify the newly defined source reliability, which will be accumulated as the longterm source quality. they propose a reputationbased truth discovery model, where initial weights are assigned based on source reputations. in @xcite , the authors presented a study on predicting the factuality of reporting and bias of news media. the models use a rich set of features derived from the content of the articles from the target news medium, its wikipedia page, its twitter account, and information about the web traffic it attracted. in @xcite the authors analyzed the effect of different factors like language, topics, and perspectives on the credibility rating of articles in a news community. these factors and their mutual interactions are the features of a novel model for jointly capturing the credibility of news articles, the trustworthiness of news sources, and the expertise of users. most of the state-of-the-art methods make use of supervised-based models. this will be a challenge when we don't have annotated data to train large models. in this work, we explored unsupervised based semantic models since in our use case, we have only 20 manually annotated articles. instead of tackling the problem of claim verification as a classification problem, we propose supporting the journalists with a list of evidence from scientific journals for that given claim. the unsupervised approaches in the literature depend on the characteristics of the news source and the features extracted from the article. these approaches do not consider the semantics and context of the text in the article. ## dataset this paper investigates the correlation between claims made in health news articles and the supporting evidence found in scientific publications. the claims and evidence were manually annotated by medical reporters due to the labor-intensive nature of this task. our dataset comprises 20 meticulously annotated articles from prominent german news sources, including focus online, berliner zeitung, bild, and welt. these healthcare news articles encompass a range of topics, such as the positive effects of different treatments/medications, including vaccines for covid-19, and the relationship between aspirin and the coronavirus. to substantiate these claims, medical reporters typically refer to scientific publications published in esteemed journals like nature, pubmed, and lancet. however, our particular use case presents a multilingual challenge as the news articles are in german, while the scientific studies are in english. the annotated claims consist of a collection of sentences, and correspondingly, the evidence paragraphs in the scientific publications are annotated by the journalists. as part of this ongoing project, we are curating this dataset, which will be made available for future publication. ## background the assessment of text similarity has garnered significant attention from researchers in the fields of natural language processing and information retrieval. this longstanding problem is inherently complex, leading to the development of diverse approaches aimed at capturing a wide range of characteristics. the evaluation of semantic similarity can be categorized into two primary methods: sentence-embedding-based approaches and wordalignment-based approaches. ## proposed approach in this section, we describe the architecture of our semantic matching component as illustrated in fig. 2 . given a claim in a healthcare news article, we need to find paragraphs in the scientific publication where the evidence for the claim are present. the news article for our use case is in german and the scientific publications are in english, introducing a cross-lingual aspect to the problem. a claim annotated by the medical reporters is a set of sentences in which the positive effects of a medicine or a therapy are explained. as a first step, we extract all the paragraphs from the scientific publication and preprocess the text. preprocessing in the case of the models trained on english corpus, we translate the claim in english using deepl translate 1 . the semantic match component takes a claim and the union of the paragraph set from the 1 https://www.deepl.com/en/translator scientific publication as inputs and outputs a subset of paragraphs. evidence paragraph selection can also be formulated as semantic matching between each paragraph and the claim to select the most plausible evidence set. the selection is done via these steps: our proposed pipeline generates a collection of the top k most similar paragraphs, which serve as evidence in the context of fake news detection. unlike traditional approaches that treat fake news as a classification problem, our pipeline introduces an evidence retrieval approach. this approach effectively assists journalists in locating relevant supporting evidence, thereby reducing the need for manual search efforts. the advantages of our system include: (i) its unsupervised nature, allowing it to adapt to concept drifts without relying on labeled data, and (ii) empowering users with decision-making capabilities while minimizing manual workload. ## evaluation and discussion in this paper, we employed manual annotation by medical reporters to annotate the evidence for claims in healthcare news articles, which were predominantly in german. to bridge the language gap, we utilized deepl, a translation tool, to translate german claims into english. for the experiments, we explored both monolingual and multilingual semantic similarity models. the monolingual models utilized deepl translations, while the multilingual models, such as xml-roberta, enabled us to handle both english and german texts. to measure the semantic distance between sentences, we developed a component that searches for semantically similar evidence in scientific studies once a new claim is received. this component employs transformer models to generate representation embeddings for each claim and the paragraphs in the scientific publications. by calculating the similarity distance, we identify the most similar evidence, aiming to provide users with semantically related evidence. during the evaluation, we considered partial evidence matches within the extracted "k"-nearest neighbors as valid matches. this approach supports journalists in finding additional evidence to supplement partial matches. we evaluated various methods, including word alignment-based approaches and sentence embedding methods, for unsupervised evidence retrieval. the models extracted the "k"-nearest neighbors that exhibited the highest similarity to the given paraphrased claim. from our results presented in in terms of transformer-based models, semantic similarity using xml-roberta embeddings performed the best. additionally, cross-lingual models outperformed monolingual models, highlighting the benefits of leveraging multilingual capabilities in our approach. ## conclusion and future work in this paper, we tackled the challenge of claim verification by employing evidence retrieval techniques from scientific studies. our approach involved developing a semantic matching method capable of retrieving the most similar evidence from a given scientific report. through the evaluation of various semantic similarity methods, including text representations and word alignment techniques, we demonstrated the effectiveness of using a multilingual model like xml-roberta to calculate semantic similarity and identify relevant paragraphs containing the evidence. by approaching this as an evidence retrieval rather than a classification problem, our proposed approach aims to support medical reporters and journalists in efficiently locating supporting evidence for paraphrased claims, thereby reducing the need for manual searching. moving forward, we will focus on retrieving the most relevant scientific papers from a pool of documents that encompass the supporting evidence for a given claim in healthcare news articles. additionally, we recognize the need for improvement in the k-nearest neighbors within our models. to achieve this, we plan to target specific sections within scientific publications, such as the results or conclusion section, where there is a higher probability of finding pertinent evidence. these advancements will further enhance the efficacy and precision of our evidence retrieval approach, paving the way for more accurate claim verification.