Querying Across Genres for Medical Claims in News (2020)

Abstract: We present a query-based biomedical information retrieval task across two vastly different genres – newswire and research literature – where the goal is to find the research publication that supports the primary claim made in a health-related news article. For this task, we present a new dataset of 5,034 claims from news paired with research abstracts. Our approach consists of two steps: (i) selecting the most relevant candidates from a collection of 222k research abstracts, and (ii) re-ranking this list. We compare the classical IR approach using BM25 with more recent transformer-based models. Our results show that cross-genre medical IR is a viable task, but incorporating domain-specific knowledge is crucial.

## introduction in recent years, the general population has increasingly sought out online sources for medical information @xcite . among the various types of sources, they mostly rely on online news articles, which often serve to disseminate medical findings from research studies @xcite . it is, however, important to identify the source of a medical claim, especially during times of pervasive misinformation and during a pandemic, when people may not be able to visit a healthcare professional. when reporting a medical study, many news articles cite the original study either by embedding hyperlinks or explicitly showing a citation, thus providing the reader with critical markers of credibility @xcite . not all articles do this, however. here, we present our work on finding scientific research publications that support the primary claims being made in a health-related news article. we design it as crossgenre query-based (or ad hoc) information retrieval (ir): given a medical claim made in a news article, retrieve the research publication supporting it. when scientific research makes its way out of conferences and journals into news meant for general consumption, the information is presented in a drastically different language. the general audience is often poorly equipped for specialist language comprehension, to the extent that changing domain-specific language to one meant for a general audience has been treated as a discipline by itself @xcite . so this change is necessary on one hand, but on the other hand, it also increases the difficulty of ir, especially so in token-based methods such as bm25 @xcite . in this work, we present a dataset (sec. 2) of claims made in medical news articles, where each claim is associated with at least one peer-reviewed research publication supporting it. for each claim, we present an ir task in sec. 3 -search for the corresponding publication from a large corpus of medical research literature. the task itself is divided into two stages: (i) retrieve a candidate list of 500 abstracts from a large corpus, and (ii) rerank them to obtain the correct publication. after discussing our findings, we present an overview of related research in sec. 4 before concluding. ## dataset over a period of 18 months (oct 2018 -march 2020), we collect 72,028 news articles from the rss feeds of several medical news websites and also from the health category of popular general news websites. to ensure that only articles citing peer-reviewed scientific publications are retained, we check every document for hyperlinks to domains listed by wikipedia as medical journals foot_0 and the list of top scientific publications on alexa, foot_1 leaving 17,712 articles (24.6%) in our collection. further, many articles were aggregations of disparate medical studies. we discard these using a combination of heuristics and manual verification, and retain only those articles that report on a single study or on a series of research studies that closely relate to each other. for articles retained after this step, the headline reflects the focal claim or finding of the cited research. this was observed by three independent readers who were given a random sample of 371 articles (7.4% of the dataset). all three agreed that for each one of these 371 articles, the headline did, indeed, present the main research finding. since some articles cite using embedded hyperlinks, while others offer a reference section at the end of the article, we are able to collect the abstracts of the cited research. our final dataset foot_2 consists of tuples of the form (h, {a i }), where h is the headline from a news article, and a i are the abstracts of the research publications cited by that article. the publication titles are retained as well. there are 5,034 headlines and 4,566 abstracts (since some research publications are cited by multiple news articles). fig. 1 shows the distribution of the news headlines over the top ten news domains in our collection. since not all research is open-access, we restrict ourselves to collecting the abstracts instead of the entire publication. we believe this does not prove to be a hindrance to the task, since it is reasonable to assume that the primary findings of a research study are mentioned in the abstract. we collect these abstracts through pubmed. 4 further, to mimic the realistic scenario where a human reader or fact-checker needs to retrieve the correct publication (i.e., the research actually upholding the claim being made in a news article) from a vast collection, we also add 217,665 spurious abstracts from the biomedical research literature. we collect these abstracts from the non-commercial use openaccess subset of pubmed central, foot_4 to serve as the negative samples in our ir task. ## experiments our task is formulated in two stages, similar to other recent ad hoc ir @xcite @xcite ) -a token-based first step to obtain a candidate list, and then the final ranking by a transformer @xcite . in spite of recent advances, the transformerbased models are large, and using them to compare each query with each document is computationally expensive even for a small corpus. thus, the two-stage approach remains a prudent choice. ## related work modern ad hoc ir systems are largely built upon bag-of-words representations, using termweighting techniques like bm25 @xcite or its variants @xcite . @xcite used such a variation for querybased news retrieval, which focuses on specific regions in an article. they use the headlines as queries and formulate the task as retrieving the corresponding article. such headline-content pairs from newswire have similarly been used in neural ir models as well @xcite . neural models have also recently been used in biomedical ir tasks, due to the availability of large datasets. @xcite introduce a deep learning model to retrieve biomedical research literature. further, deep neural architectures have been coupled with external knowledge bases @xcite , where research documents are retrieved as part of a precision medicine task. in this body of work, the query is either an in-domain keyword, or structured information. as such, they cannot be readily used where the query may be expressed using complex linguistic structures found in the newswire. example 1b in given the success of bert and its successors in natural language inference tasks, ad hoc ir systems have used them for claim verification @xcite @xcite . applications of such models to binary classification for query-based passage re-ranking suggest that contextual information can be valuable when re-ranking an initial list of possibly relevant documents retrieved by bm25 model @xcite . these approaches are not readily suitable for cross-genre ir, but they motivated some of our technical choices. for instance, our use of pointwise (instead of pairwise) loss was based on the discussion in @xcite regarding ir tasks with bert-style models. fact-checking is a critical component in fight-ing misinformation, but medical misinformation is known to be nuanced. for example, instead of outright false claims, statements are known to undergo exaggeration. in this general context of thwarting medical misinformation, there is some notable work that, while being distinct from the ir task discussed here, complements our research. for instance, @xcite studied the exaggeration of medical claims in the news vis-à-vis the original findings in research publications. ## conclusion in contrast to recent research in ad hoc neural ir, which require large amounts of training data (mitra and craswell, 2018), we present a system that combines term-weighting techniques and neural models across two distinct linguistic genres. we also provide a novel dataset of medical newswire queries linked to research literature. our results show that while neural models excel at re-ranking a small number of documents when pre-trained contextual embeddings are tuned on domain-specific data, classical token-based approaches remain difficult to beat in a cross-genre retrieval scenario when the search space is larger. our data collection process also reveals that even in a domain as critically important as medical news, only a small fraction of news articles (24.6%) include a complete citation and a link to the original research. thus, the presented task has utility in medical factchecking, identifying health-related misinformation, and assessing some empirically verifiable aspects of health news reporting.