A Simple Three-Step Approach for the Automatic Detection of Exaggerated Statements in Health Science News (2021)

Abstract: There is a huge difference between a scientific journal reporting ‘wine consumption might be correlated to cancer’, and a media outlet publishing ‘wine causes cancer’ citing the journal’s results. The above example is a typical case of a scientific statement being exaggerated as an outcome of the rising problem of media manipulation. Given a pair of statements (say one from the source journal article and the other from the news article covering the results published in the journal), is it possible to ascertain with some confidence whether one is an exaggerated version of the other? This paper presents a surprisingly simple yet rational three-step approach that performs best for this task. We solve the task by breaking it into three sub-tasks as follows – (a) given a statement from a scientific paper or press release, we first extract relation phrases (e.g., ‘causes’ versus ‘might be correlated to’) connecting the dependent (e.g., ‘cancer’) and the independent (‘wine’) variable, (b) classify the strength of the relationship phrase extracted and (c) compare the strengths of the relation phrases extracted from the statements to identify whether one statement contains an exaggerated version of the other, and to what extent. Through rigorous experiments, we demonstrate that our simple approach by far outperforms baseline models that compare state-of-the-art embedding of the statement pairs through a binary classifier or recast the problem as a textual entailment task, which appears to be a very natural choice in this settings.

## introduction exaggerations in health news can have tremendous adverse effects on the lifestyle of the common masses who feed themselves mostly on such news instead of the source scientific publication. this problem is challenging as it involves many intrinsic complexities that need to be addressed. first, while encountering a scientific claim, we need to identify the true fact related to the claim from the knowledge base (in most cases the source journal article). for instance, if the press release/ news report states that 'chocolate causes acne' then this claim needs to be compared to the scientific study that actually recruits human subjects and does experiments to study connections between chocolate consumption and acne vulgaris. in fact, the study reported in @xcite proves that chocolate consumption is 'not related' to acne. in this paper, we propose a very simple threestep method foot_0 that given a pair of statements, e.g., 'chocolate causes acne' taken from the press release and 'chocolate consumption is not related to acne' taken from the source journal, can identify if the former is an exaggerated version of the latter. note that the problem that we aim to solve requires the pair of statements to be compared as inputs to produce the desired output (exaggerated or not). however, toward the end of the paper, we outline a simple heuristic that dissolves this constraint for the considered dataset that the pair of statements that we compare from the whole article (press release or source journal) needs to be known to us a priori. we note that comparing state-of-the-art embedding of the two statements using binary classifiers does not give much advantage; even adapting the problem as a textual entailment task (a natural adaptation since one statement is an exaggerated version of the other) surprisingly does not bring much additional benefits. this paper therefore puts forward a simple three step approach, breaking the task down to three rational steps: (1) given a statement from a scientific paper or press release, extract relation phrases (e.g., 'causes' versus 'correlated to') connecting the dependent (e.g., 'acne') and the independent (e.g., 'chocolate') variable, (2) classify the strength of the relation phrase and (3) compare the strengths of the relation phrases extracted from the statements to identify whether one sentence contains an exaggerated version of the other, and to what extent. our approach is operationalised on the data released by @xcite . the exact statements containing the relation phrases within an article or the source journal are annotated in the dataset allowing us to effectively train our models. the main results of our paper are, ## related work media manipulation is a set of related techniques in which the manipulator attempts to create an image or argument that favors particular interests @xcite . media manipulation and fake news got huge attention from the research community in the current decade. hundreds of studies got published in this domain which makes it impossible to cite them all. the surveys of different methods and datasets published in these domain can be found in parikh and atrey (2018); zhou and zafarani (2018); sharma et al. (2019); bondielli and marcelloni (2019); oshikawa et al. (2018); haciyakupoglu et al. (2018); zhou and zafarani (2020); @xcite . previous study @xcite suggests that press releases are a major source of exaggeration. identification of relationship between entities and associated strength in scientific articles and comparing them with that of news reports and press releases is the key to exaggeration detection. in literature, there are many techniques present separately to identify entities along with their relationships @xcite and strength associated in a relationship @xcite . the authors in @xcite propose a framework to identify claims from tweet corpus related to major events. in @xcite the authors build autonomous message classifier that filters relevant and trustworthy information from twitter. in @xcite the authors develop a knowledge extraction and knowledge discovery system that extracts causal knowledge from textual databases. present study: none of the studies mentioned above tackle the exaggeration detection problem. in a recent study @xcite analyze the same dataset as we use in this paper and attempt to identify exaggeration. they also assume that the pair of statements are available as inputs for the detection task. however, the biggest drawback of this work is that the authors additionally assume that the relationship phrase in the input statement is also known which makes the task significantly simpler. in fact, the authors also do not use these relationship phrases as important signals but treat each statement as a whole as bag-of-words. we, on the other hand, identify the relation phrases automatically in the first place and use it as an important signal for the next two sub-tasks (strength classification and exaggeration detection) which is the most important contribution of our work. we also compare this work with our method and show that we considerably outperform them. ## methodology as discussed earlier, our approach has three steps. they are (i) relation phrase labeling, (ii) strength classification, and (iii) exaggeration identification. all of the three modules are connected in a sequence as shown in figure 1 . the relation phrase labeling module takes a selected statement and labels the relation phrase present in it. next, the strength classification module takes a relation phrase as input and predicts its strength level. finally, in the exaggeration identification module, the strengths of source statement and target statement are compared, and it calculates the exaggeration or underplay level of the target statement with respect to the source statement. the individual modules are described in details in the subsequent subsections. ## results in this section, we present the results from the three phases -relation phrase labeling, strength classification and exaggeration identification one after the other. finally we outline a scheme to spot the exact location of main claims in the news articles and press releases that particularly works for this dataset. this enabled us to fully automate the process based on this scheme and measure its performance. ## discussion in this section we present the findings of our error analysis and results correspond to the special sub cases where simple heuristics is taken into account to automate the whole process. ## conclusion in this work, we proposed a simple yet explainable three step approach that automatically identifies whether a given statement typically from a press release or a news article is exaggerated in comparison to the source statement present in the journal. our first step adapts the recently proposed bert technology and models the relationship extraction problem as a sequence labeling task. this beats other sequence labeling and syntax driven approaches by a large margin. the relationship phrase extracted is encoded as a bag of words and fed to standard classifiers to obtain the strength of the phrase. ground-truth labels of relationship phrases and those obtained from our model achieve similar performance. finally, once the strength of the relationship phrase is available, a pair of statements can be easily compared. this method of exaggeration identification beats standard baselines that directly feed the two individual bert embedding of the statement pair into binary classifiers or use the bert textual entailment framework. in future, one can check if this kind of approaches work for other tasks where whole task can be divided into explainable subtasks.