Explainable Assessment of Healthcare Articles with QA (2022)

Abstract: The healthcare domain suffers from the spread of poor quality articles on the Internet. While manual efforts exist to check the quality of online healthcare articles, they are not sufficient to assess all those in circulation. Such quality assessment can be automated as a text classification task, however, explanations for the labels are necessary for the users to trust the model predictions. While current explainable systems tackle explanation generation as summarization, we propose a new approach based on question answering (QA) that allows us to generate explanations for multiple criteria using a single model. We show that this QA-based approach is competitive with the current state-of-the-art, and complements summarization-based models for explainable quality assessment. We also introduce a human evaluation protocol more appropriate than automatic metrics for the evaluation of explanation generation models.

## introduction the internet has become an important source of medical advice. according to @xcite , in 2017, 74.4% of the us population first looked for health-related information on the internet, while only 13.3% of the population first asked a physician or healthcare provider. however, poor quality reporting, including misinformation, cherry-picking, exaggerations, etc., is often present online and can be a severe threat to public health. recent events, such as the covid-19 pandemic, demonstrate the necessity of developing quality assessment systems for healthcare reports to limit these harms. fortunately, websites such as healthnewsreview foot_0 critically analyze medical articles to identify poor quality reporting and improve the public discourse about healthcare. however, the manual review of medical news is a time-consuming task that would story #1511 criterion 1: does the article adequately discuss the costs of the intervention? answer: not satisfactory explanation: there was no discussion of cost as there was in the competing ap story. criterion 2: does the article adequately quantify the benefits of the treatment/test/product/procedure? answer: satisfactory explanation: the story adequately quantified the benefits seen in the study that led to fda approval. criterion 3: ... assessing the quality of news articles has been the focus of numerous studies that tackle it as a text classification task @xcite @xcite . however, explanations for the predictions only recently started receiving attention, despite being necessary to convince the readers of such assessments. for instance, @xcite have built on the evaluation work conducted by the healthnewsreview website (see beyond quality assessment, previous works have formulated textual explanation generation for classification as summarization @xcite . however such approaches suffer from a number of shortcomings when applied to the assessment of an article based on multiple criteria. as these approaches always output a single summary for a given input text, separate models must be trained to generate explanations for each classification label and evaluation criterion (e.g. reliability of sources, lack of information, etc.), as in the example given in in this work, we develop an explainable quality assessment system for health news reports, and we evaluate it on the fakehealth corpus @xcite . it differs from previous work as its explanation generation model is based on question answering (qa), which takes into consideration the definition of each evaluation criterion in the form of a question (see we compare our approach for explanation generation against a summarization-based system inspired from kotonya and toni (2020). our results show that both approaches are complementary and perform better in different cases. more specifically, summarization-based systems are more appropriate when relevant information is explicitly given in articles, while qa-based systems perform better when relevant information is missing. finally, evaluating generated explanations is not an easy task as we should consider both the structure and the sense of texts. previous works used automatic metrics for the evaluation of explanations, which are known to be insufficient for abstractive text generation. @xcite precisely insisted that assessing the readability and quality of a generated text requires human annotators as no automatic metric can achieve good performance on this task. likewise, @xcite have recently highlighted that automatic evaluation protocols, usually relying on rouge scores, correlate weakly with human judgement and fail to evaluate critical features, such as factual consistency. for this reason, we propose a new human evaluation protocol to assess the fluency, consistency, and factual correctness of the explanations, and we show that automatic metrics are not appropriate for this task. ## methodology our system starts with classifying articles according to ten evaluation criteria, then generates explanations using qa, taking into account the predicted classification labels. the purpose of the text classification step is to determine whether an article is satisfactory with respect to different evaluation criteria. we consider different options from the literature: logistic regression for its simplicity, bert-based classification which is commonplace but truncates texts to 512 tokens, and a longformerbased encoder model @xcite , which is able to deal with long input texts like those of our study. both bert and longformer-based classifiers are pre-trained for a large classification task on a biomedical dataset, pubmed foot_1 , then fine-tuned on the fakehealth dataset. in line with @xcite 's recommendation, we use a classification objective for longformer classifier, that places a global attention mask on a [cls] token. this token aggregates the representation of the whole text at the beginning of the input text as shown in the second stage of the pipeline generates abstractive explanations for the previously predicted classes. as the qa approach takes into account the classes and the questions posed by criteria, we only need to train a single model, handling all criteria and classes. following soni and roberts (2020), we have chosen to work with a longformerbased encoder-decoder that we first train on the open-domain dataset squad v2.0 @xcite , and then fine-tune on fakehealth. because gold explanations in the fakehealth dataset are abstractive, our model learns to write complete explanations despite the pre-training step on squad whose explanations are spans of phrases. even though we always use the same ten questions (shown in following recent previous work on explainable factchecking in healthcare by kotonya and toni (2020), we implement a baseline for the explanation generation task, based on summarization. because such a system does not take into account the criteria definitions in its input, it cannot combine all criteria together as it would always produce the same explanation for all criteria. therefore, this approach requires training independent models for each class within a criterion, which results in 30 models (10 criteria × 3 classes) in the case of the fakehealth dataset. we use here a summarization objective for the longformer model, that applies a global attention mask to the very first token of input texts (see ## human evaluation of explanations unlike previous works that assess generated text with automatic metrics, we design a human evaluation protocol that assesses four aspects of explanations: their fluency, consistency, factual correctness, and whether they are indicative of the label that they are supposed to explain. an explanation is considered fluent if it sounds natural, and consistent if it does not contradict itself, include repetitions, or information that is not mentioned in the article. the factual correctness criterion looks for incorrect facts, contradictions with respect to the article, or hallucinations. finally, generated explanations should allow a human judge to infer correctly the label they are meant to explain. we conducted two pilot studies in order to assess the quality of our guidelines. as reported in the final guidelines used for the evaluation in section 5 are fully detailed in appendix b. in ## data we evaluate our qa and summarization-based models on the fakehealth corpus of health news articles, released by @xcite . each article in the dataset was evaluated by at least two experts, according to ten criteria that assess diverse aspects such as "the overclaiming, missing of information, reliability of sources and conflict of interests" @xcite . @xcite found zero to a minor positive correlation between the criteria, which justifies the relevance of all of them. these criteria are reported in for each criterion, articles are annotated with one of three labels, not satisfactory, satisfactory, and not applicable, and a textual explanation that justifies the assigned label, as shown in ## conclusion and discussion in this work, we propose a new qa-based approach to generate explanations for quality assessment systems. this approach allows us to build a single model, able to generate explanations for different criteria and classes, by taking into account the questions related to criteria. we have shown that the qa-based system is competitive with the summarization-based one, and that they are complementary. notably, the qa-based approach is more appropriate when the relevant information is not explicitly given in articles or for small classes. as for the classification task, longformer-based models perform best thanks to their ability to deal with long input texts. finally, we have highlighted that automatic metrics, such as rouge, correlate very weakly with human judgment when it comes to evaluating explanation generation models. this paper could serve as a starting point to explore the use of qa models for explainable article assessment. enyan dai, yiwei sun, and suhang wang. 2020. ginger cannot cure cancer: battling fake health news with a comprehensive data repository. corr, abs/2002.00837. liana ermakova, jean valère cossu, and josiane mothe. 2019. a survey on evaluation of summarization methods. information processing management, 56(5):1794-1814. neema kotonya and francesca toni. 2020. explainable automated fact-checking for public health claims. wojciech kryscinski, nitish shirish keskar, bryan mc-cann, caiming xiong, and richard socher. 2019. neural text summarization: a critical evaluation. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 540-551, hong kong, china. association for computational linguistics. wojciech kryscinski, bryan mccann, caiming xiong, and richard socher. 2020. evaluating the factual consistency of abstractive text summarization. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 9332-9346, online. association for computational linguistics. chin-yew lin. 2004. rouge: a package for automatic evaluation of summaries. in text summarization branches out, pages 74-81, barcelona, spain. association for computational linguistics. annie louis and ani nenkova. 2013. what makes writing great? first experiments on article quality prediction in the science journalism domain. transactions of the association for computational linguistics, 1:341-352. inderjeet mani. 2002. summarization evaluation: an overview. kishore papineni, salim roukos, todd ward, and wei-jing zhu. 2002. bleu: a method for automatic evaluation of machine translation. in proceedings of the 40th annual meeting of the association for computational linguistics, pages 311-318, philadelphia, pennsylvania, usa. association for computational linguistics. pranav rajpurkar, robin jia, and percy liang. 2018. know what you don't know: unanswerable questions for squad. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pages 784-789, melbourne, australia. association for computational linguistics. lila j. finney rutten, kelly d. blake, alexandra j. greenberg-worisek, summer v. allen, richard p. moser, and bradford w. hesse. 2019. online health information seeking among us adults: measuring progress toward a healthy people 2020 objective. public health reports, 134(6):617-625. pmid: 31513756. sarvesh soni and kirk roberts. 2020. evaluation of dataset selection for pre-training and fine-tuning trans-former language models for clinical question answering. in proceedings of the 12th language resources and evaluation conference, pages 5532-5538, marseille, france. european language resources association. dustin wright and isabelle augenstein. 2021. semisupervised exaggeration detection of health science press releases.