Semi-Supervised Exaggeration Detection of Health Science Press Releases (2021)

Abstract: Public trust in science depends on honest and factual communication of scientific papers. However, recent studies have demonstrated a tendency of news media to misrepresent scientific papers by exaggerating their findings. Given this, we present a formalization of and study into the problem of exaggeration detection in science communication. While there are an abundance of scientific papers and popular media articles written about them, very rarely do the articles include a direct link to the original paper, making data collection challenging, and necessitating the need for few-shot learning. We address this by curating a set of labeled press release/abstract pairs from existing expert annotated studies on exaggeration in press releases of scientific papers suitable for benchmarking the performance of machine learning models on the task. Using limited data from this and previous studies on exaggeration detection in science, we introduce MT-PET, a multi-task version of Pattern Exploiting Training (PET), which leverages knowledge from complementary cloze-style QA tasks to improve few-shot learning. We demonstrate that MT-PET outperforms PET and supervised learning both when data is limited, as well as when there is an abundance of data for the main task.

## introduction factual and honest science communication is important for maintaining public trust in science @xcite , and the "dominant link between academia and the media" are press releases about scientific articles @xcite . however, multiple studies have demonstrated that press releases have a significant tendency to sensationalize their associated scientific articles (sumner et al., 2014; bratton et al., 2019; woloshin et al., 2009; woloshin and 1 the code and data are available online at https://github.com/copenlu/ scientific-exaggeration-detection figure 1 : scientific exaggeration detection is the problem of identifying when a news article reporting on a scientific finding has exaggerated the claims made in the original paper. in this work, we are concerned with predicting exaggeration of the main finding of a scientific abstract as reported by a press release. @xcite . in this paper, we explore how natural language processing can help identify exaggerations of scientific papers in press releases. while @xcite and @xcite performed manual analyses to understand the prevalence of exaggeration in press releases of scientific papers from a variety of sources, recent work has attempted to expand this using methods from nlp @xcite @xcite . these works focus on the problem of automatically detecting the difference in the strength of causal claims made in scientific articles and press releases. they accomplish this by first building datasets of main claims taken from pubmed abstracts and (unrelated) press releases from eurekalert foot_0 labeled for their strength. with this, they train machine learning models to predict claim strength, and analyze unlabelled data using these models. this marks an important first step toward the goal of automatically identifying exaggerated scientific claims in science reporting. however, existing work has only partially at-tempted to address this task using nlp. particularly, there exists no standard benchmark data for the exaggeration detection task with paired press releases and abstracts i.e. where the data consist of tuples of the form (press release, abstract) and the press release is written about the paired scientific paper. collecting paired data labeled for exaggeration is critical for understanding how well any solution performs on the task, but is challenging and expensive as it requires domain expertise @xcite . the focus of this work is then to curate a standard set of benchmark data for the task of scientific exaggeration detection, provide a more realistic task formulation of the problem, and develop methods effective for solving it using limited labeled data. to this end, we present mt-pet, a multi-task implementation of pattern exploiting training (pet, schick and schütze (2020a,b)) for detecting exaggeration in health science press releases. we test our method by curating a benchmark test set of data from the expert annotated data of sumner et al. (2014) and bratton et al. (2019), which we release to help advance research on scientific exaggeration detection. contributions in sum, we introduce: • a new, more realistic task formulation for scientific exaggeration detection. • a curated set of benchmark data for testing methods for scientific exaggeration detection consisting of 563 press release/abstract pairs. • mt-pet, a multi-task extension of pet which beats strong baselines on scientific exaggeration detection. ## problem formulation we first provide a formal definition of the problem of scientific exaggeration detection, which guides the approach described in §3. we start with a set of document pairs {(t, s) ∈ d}, where s is a source document (e.g. a scientific paper abstract) and t is a document written about the source document s (e.g. a press release for the paper). the goal is to predict a label @xmath0, 1, 2} for a given document pair (t, s), where 0 implies the target document undersells source document, 1 implies the target document accurately reflects the source document, and 2 implies the target document exaggerates the source document. two realizations of this formulation are investigated in this work. the first (defined as t1) is an inference task consisting of labeled document pairs used to learn to predict l directly. in other words, we are given training data of the form (t, s, l) and can directly train a model to predict l from both t and s. the second (defined as t2) is as a classification task consisting of a training set of documents @xmath1, and a classifier is trained to predict the claim strength l of sentences from these documents. in other words, we don't require paired documents (t, s) at train time. at test time, these classifiers are then applied to document pairs (t, s) and the predicted claim strengths (l s , l t ) are compared to get the final label l. previous work has used this formulation to estimate the prevalence of correlation to causation exaggeration in press releases @xcite , but have not evaluated this on paired labeled instances. following previous work @xcite , we simplify the problem by focusing on detecting when the main finding of a paper is exaggerated. the first step is then to identify the main finding from s, and the sentence describing the main finding in s from t. in our semi-supervised approach, we do this as an intermediate step to acquire unlabeled data, but for all labeled training and test data, we assume the sentences are already identified and evaluate on the sentence-level exaggeration detection task. ## approach one of the primary challenges for scientific exaggeration detection is a lack of labeled training data. given this, we develop a semi-supervised approach for few-shot exaggeration detection based on pattern exploiting training (pet, @xcite ). our method, multi-task pet (mt-pet, see figure 2 ), improves on pet by using multiple complementary cloze-style qa tasks derived from different source tasks during training. we first describe pet, followed by mt-pet. ## data collection one of the main contributions of this work is a curated benchmark dataset for scientific exaggeration detection. labeled datasets exist for the related task of claim strength detection in scientific abstracts and press releases @xcite , but these data are from press releases and abstracts which are unrelated (i.e. the given press releases are not written about the given abstracts), making them unsuitable for benchmarking exaggeration detection. given this, we curate a dataset of paired sentences from abstracts and associated press releases, labeled by experts for exaggeration based on their claim strength. we then collect a large set of unlabeled press release/abstract pairs useful for semi-supervised learning. ## experiments our experiments are focused on the following primary research questions: we experiment with the following model variants: we perform two evaluations in this setup: one with t1 as the main task and one with t2. for t1, we use the 100 expert annotated instances with paired press release and abstract sentences labeled for exaggeration (200 sentences total). for t2, we use 100 sentences from the press data from @xcite and 100 sentences from the abstract data in @xcite strength. we use roberta base @xcite from the huggingface transformers library @xcite as the main model, and set α m to be 1, and α @xmath2, |dm| |da| ). all methods are evaluated using macro-f1 score, and results are reported as the average performance over 5 random seeds. ## conclusion in this work, we present a formalization of and investigation into the problem of scientific exaggeration detection. as data for this task is limited, we develop a gold test set for the problem and propose mt-pet, a semi-supervised approach based on pet, to solve it with limited training data. we find that mt-pet helps in the more difficult cases of identifying and differentiating direct causal claims from weaker claims, and that the most performant approach involves classifying and comparing the individual claim strength of statements from the source and target documents. the code and data for our experiments can be found online foot_4 . future work should focus on building more resources e.g. datasets for exploring scientific exaggeration detec-tion, including data from multiple domains beyond health science. finally, it would be interesting to explore how mt-pet works on combinations of more general nlp tasks, such as question answering and natural language inference or part-of-speech tagging and named entity recognition.