An Empirical Assessment of the Qualitative Aspects of Misinformation in Health News (2021)

Abstract: The explosion of online health news articles runs the risk of the proliferation of low-quality information. Within the existing work on fact-checking, however, relatively little attention has been paid to medical news. We present a health news classification task to determine whether medical news articles satisfy a set of review criteria deemed important by medical experts and health care journalists. We present a dataset of 1,119 health news paired with systematic reviews. The review criteria consist of six elements that are essential to the accuracy of medical news. We then present experiments comparing the classical token-based approach with the more recent transformer-based models. Our results show that detecting qualitative lapses is a challenging task with direct ramifications in misinformation, but is an important direction to pursue beyond assigning True or False labels to short claims.

## introduction in recent years, health information-seeking behavior (hisb) -which refers to the ways in which individuals seek information about their health, risks, illnesses, and health-protective behaviors @xcite has become increasingly reliant on online news articles @xcite @xcite . some studies also posit that with increasing involvement of the news media in health-related discussions, and direct-to-consumer campaigns by pharmaceutical companies, people are turning to the internet as their first source of health information, instead of healthcare practitioners @xcite . this behavior is primarily driven by the users' need to gain knowledge @xcite about some form of intervention (e.g., drugs, nutrition, diagnostic and screening tests, dietary recommendations, psychotherapy). furthermore, and perhaps counter-intuitively, information seekers seldom spend a lot of time on health news headline: experts warn coronavirus is 'as dangerous as ebola' in shocking new study. source: www.express.co.uk/life-style/health/1275700/ebolaelderly-patients-coronavirus-experts-study-research-deathfigures published: apr 30, 2020 accessed: march 21, 2021 cause of misinformation comparing numbers from two different contexts: (1) the hospital fatality rate of covid-19, and (2) the overall case fatality rate of ebola. in stark contrast to hisb, there is also growing lack of trust in the accuracy of health information provided on the internet @xcite . this is perhaps to be expected, given how widespread health-related misinformation has become. for instance, in surveys where expert panels have judged the accuracy of health news articles, nearly half were found to be inaccurate @xcite . health-related misinformation, however, is rarely a binary distinction between true and fake news. in medical news, multiple aspects of an intervention are typically presented, and a loss of nuance or incomplete understanding of the process of medical research can lead to various types of qualitative failures, exacerbating misinformation in this domain. recently, news articles citing leading medical journals have suffered because of this. given how these issues are specific to medical misinformation, and how intricately the accuracy of medical facts is intertwined with the quality of health care journalism, the imperative to move beyond a binary classification of true and fake becomes clear. to this end, a set of specific principles and criteria have been proposed by scientists and journalists, based largely on the acclaimed work by @xcite and the statement of principles by the association of health care journalists @xcite . we present a dataset (sec. 2) specifically tailored for health news, and labeled according to a set of domain-specific criteria by a multi-disciplinary team of journalists and health care professionals. the detailed data annotation was carried out from 2006 to 2018 @xcite . for each criterion, we present a classification task to determine whether or not a given news article satisfies it (sec. 3), and discuss the results. finally, we present relevant prior work (sec. 4) before concluding. ## dataset our data is collected from health news review @xcite 1 , which contains systematic re-1 www.healthnewsreview.org/ news headline: virtual reality to help detect early risk of alzheimer's source: www.theguardian.com/society/2018/dec/16/alzheim ers-dementia-cure-virtual-reality-navigation-skills published: dec 16, 2018 accessed: april 26, 2021 criterion labeled "not applicable": (2) does the story adequately quantify the benefits of the treatment/test/product/procedure? going beyond the reviews themselves, we then collect the news articles being reviewed from the original news sites. however, nearly 30% of those pages have ceased to exist. further, some articles could not be retrieved due to paywalls. multiple prominent news organizations are featured in this data, with fig. 1 showing the distribution over these organizations (for brevity, we show the top ten entities, with the tenth being "others"). our final dataset comprises 1,119 articles (740 news stories and 379 pr news releases) along with their criteria-driven reviews. these are maintained as (n, {c i }) tuples, where n is the news article, and c i are the review results for each criteria. since criteria 6 and 10 are slightly different for news stories and pr releases, we remove these from our empirical experiments. further, we also remove criteria 5 and 9, since these require highly topic-specific medical knowledge. we do this in order to have our approach reflect the extent of medical knowledge available to the lay reader, who is unlikely to fully comprehend the specialized language of medical research publications @xcite . ## experiments we approach the problem as a series of supervised classification tasks, where the performance is evaluated separately for each review criterion. moreover, since the reviewers use the not applicable label based on additional topic-specific medical knowledge, we discard the (n, {c i }) tuples where c i carries this label. this eliminates approximately 2.35% of the total number of tuples in our dataset, and paves the way for a binary classification task where each article is deemed satisfactory or not for the criterion c i . the numbers of remaining news for each criterion are as shown in below in all experiments, we use 70% of the data for training. the rest is used as the test set. as a simple baseline, we use the zero rule (also called zeror or 0-r), which uses the base rate and classifies according to the prior, always predicting the majority class. we then experiment with the classical representation using tf-idf feature encoding, as well as the state-of-the-art transformer-based models. in both approaches, we use 5-fold cross-validation during training to select the best hyperparameters for each model. these are described next. ## related work for many years now, concerns have been raised about medical misinformation in the coverage by news media @xcite . moynihan et al. studied 207 news stories about the benefits and risks of three medications to prevent major diseases, and found that 40% of the news did not report benefits quantitatively while only 47% mentioned potential harms. various tasks and approaches have been formulated @xcite ) for factchecking information. multiple datasets have also been put forth. ferreira and vlachos @xcite released a collection of 300 claims with corresponding news. this dataset was later significantly enlarged in the fake news challenge @xcite . at a similarly large scale, wang introduced a dataset comprising 12.8k manually labeled statements from politifact.com and treated it as a text classification task. a large body of work, however, has dealt with fact-checking of short claims, both for fact-checking @xcite as well as for identifying what to check @xcite . furthermore, a vast majority of prior work was on political news, while medical misinformation remained relatively neglected until its impact was underscored by the covid-19 pandemic (e.g., hossain et al. (2020); serrano et al. (2020) , among others). this body of work, however, continues to assign true/false labels or binary stance labels to short claims. in contrast, our work analyzes long articles and identifies whether or not they satisfy various qualitative criteria specifically important to medical news, as determined by journalists and health care professionals. ## conclusion we present a first empirical analysis of qualitative reviews of medical news, since the traditional true/fake dichotomy does not adequately capture the nuanced world of medical misinformation. to this end, we collect a dataset of medical news along with their detailed reviews based on multiple criteria. the novelty of this work lies in highlighting the importance of a deeper review and analysis of medical news to understand misinformation in this domain. for example, misinformation may easily be caused by the use of sensational language, or disease-mongering, or not disclosing a conflict of interest (all of which are criteria used in this work). our results show that this is a challenging task. the data reveals that for most of the criteria, less than half of the news articles are satisfactory. the commonly perceived notion of reputation notwithstanding, several articles from well-known sources (such as the ones shown in fig. 1 ) also fall short of these qualitative benchmarks set by domain experts. this presents a clear data-driven picture of how the qualitative aspects of misinformation defy our expectations. we have presented a first step in this direction, and our hope is that this work leads to collaborative creation of similar datasets at larger scale by computer scientists and journalists, and in multiple domains even outside of health care.