Understanding Fine-grained Distortions in Reports of Scientific Findings (2024)

Abstract: Distorted science communication harms individuals and society as it can lead to unhealthy behavior change and decrease trust in scientific institutions. Given the rapidly increasing volume of science communication in recent years, a fine-grained understanding of how findings from scientific publications are reported to the general public, and methods to detect distortions from the original work automatically, are crucial. Prior work focused on individual aspects of distortions or worked with unpaired data. In this work, we make three foundational contributions towards addressing this problem: (1) annotating 1,600 instances of scientific findings from academic papers paired with corresponding findings as reported in news articles and tweets wrt. four characteristics: causality, certainty, generality and sensationalism; (2) establishing baselines for automatically detecting these characteristics; and (3) analyzing the prevalence of changes in these characteristics in both human-annotated and large-scale unlabeled data. Our results show that scientific findings frequently undergo subtle distortions when reported. Tweets distort findings more often than science news reports. Detecting fine-grained distortions automatically poses a challenging task. In our experiments, fine-tuned task-specific models consistently outperform few-shot LLM prompting.

## introduction lay people, i.e., non-experts with limited experience or knowledge of a specific domain, rely on effective science communication to learn about scientific research. in order to make scientific information understandable to a lay audience, science communicators must first simplify the highly technical language of science @xcite . in do-paper finding:adolescents whose fathers engaged in weight conversations were significantly more likely to engage in dieting and uwcb. reported finding: parents, stop talking to your kids about their weight! they're saying that this could cause eating disorders ing so, authors may knowingly or unknowingly distort the information conveyed by the original scientific publication to achieve specific rhetorical goals @xcite @xcite @xcite . for example, simplifying findings for a non-expert audience requires balancing accuracy, accessibility and comprehensibility @xcite which can lead to information being omitted purposefully. at the same time, the way that science is communicated to the public is crucial as it influences people's behavior and trust in science @xcite @xcite . consider fig. 1 . the paper finding describes a correlation between "weight conversations" and "dieting and uwcb" 1 while clearly stating to whom the finding applies. in the reported finding those constraints are omitted, it generalizes from "fathers" to all "parents" and "adolescents" to "kids." further, the reported finding states a causal relationship between "talking to kids about their weight" and "eating disorders." finally, the reported finding expresses high certainty in the correlational relationship found, while the reported finding speculates that a causal relationship "could" exist, thus communicating lower certainty. while subtle, the reported finding presents a different picture than that of the original paper, highlighting the need for a fine-grained comparison between paper and reported findings. previous work has been limited to either a subset of the distortions studied in this work @xcite @xcite or detecting general information change without fine-grained labels @xcite . we improve on this by making three core contributions: we collect an expert-annotated dataset of 1,600 scientific findings paired across scientific papers and their reported findings in news media and twitter 2 (c1). the data is annotated with fine-grained distortion labels, i.e., causal claim strength, level of certainty, level of generality and sensationalism. using this labeled data, we train and analyze the performance of benchmark models on the task of fine-grained distortion detection (c2). finally, using the data from c1 and models from c2, we perform a large scale analysis of the prevalence and types of distortions present in both our expert labeled data and a large-scale automatically labeled dataset of 1,655,570 paper findings, 422,626 news findings and 356,275 tweets (c3). we answer the following research questions: rq1 how are scientific findings changed when reported to lay audiences? rq2 how reliably can we detect distortions automatically? for rq1, we find that scientific findings undergo fine-grained changes when they are reported even when their overall content is well-aligned. this is consistent across scientific disciplines. we find that 54 % of findings are reported with a changed causal relation and 60 % of findings are reported with a changed level of certainty. in 49 % of the paired findings, the reported finding is more general than the paper finding, and reported findings are typically more sensational compared to paper findings. across all change dimensions, findings reported in tweets are more susceptible to mis-reporting compared to reports in science news. with respect to rq2, we find that detecting fine-grained distortions automatically poses a challenging task. in our experiments, fine-tuning task-specific models consistently outperform few-shot llm prompting. our best models achieve macro f 1 scores of 0.58, 0.56, 2 twitter is now called x. 0.57 and pearson correlation of 0.61 for predicting causality, certainty, generalizing and sensationalism, respectively. ## dimensions of information changes we consider four dimensions found to be notable in the science of science literature @xcite @xcite ) that characterize scientific findings and may undergo change when reported: causality, i.e., the type of causal relation (or its absence) described in finding; certainty, i.e., the level of confidence or certainty that is expressed wrt. a finding; generality, i.e., the level of generalization or specificity of a finding compared to its reporting; sensationalism, i.e., the extent to which a finding is presented in a way to elicit an emotional reaction by using urgent or exaggerated language and descriptions. the divergence between those dimensions in the paper and the reported findings allows us to estimate how accurate a reporting is and which properties may be distorted. more specifically, we consider a reported finding to be mis-reported if the label for a given characteristic changes from the paper finding to the corresponding reported finding. we build a dataset of 1,600 findings from four scientific disciplines (medicine, psychology, biology and computer science). findings are paired between scientific paper and news and scientific paper and tweet, giving 800 pairs total. in an annotation conducted with crowdworkers, we label each instance/pair with regards to the change type. annotators rate certainty and sensationalism levels, identify causality relations and check for generalizations in both versions of the finding. fig. 1 shows an example. ## understanding distorted science communication at large to gauge critical information changes more broadly, we use a large scale set of paper findings, news findings, and tweets and analyze the prevalence of distortions in science communication at large. data. the initial dataset is collected by pairing scientific papers from the s2orc dataset @xcite with news articles and tweets using altmetric 16 , an aggregator of mentions of scientific papers online. we automatically identify the result descriptions and select for each news and tweet finding the paper finding with the highest information matching score @xcite above 4. sec. a.3.1 describes the filtering process in detail. this gives us a set of 35,150 findings paired between papers and news and 72,032 findings paired between papers and tweets. using the best performing models from sec.4, we estimate critical distortions wrt. causality, certainty and sensationalism. 17 our goal is to understand which type of reports -news vs. tweets-are more susceptible to mis-reporting. 16 https://www.altmetric.com/ 17 we exclude generalization from this analysis because of the varied classification performance across target classes. results. overall, we find that science communication on twitter is more frequently affected by mis-reporting. tweets show pronounced critical changes in causality (fig. 10b , appendix a.3), while the vast majority of findings reported in science news accurately report the causal relation from the original finding (fig. 10a , appendix a. 3). science communicators on twitter frequently overstate findings' certainty. fig. 11 in appendix a.3 shows changes in certainty levels. notably, the vast majority of findings reported in tweets exhibit an increased level of certainty. this effect is less pronounced in the findings reported in science news. most notably, reported scientific findings are presented with heightened levels in sensationalism, both in tweets and science news. the density plot visualizing sensationalism scores in fig. 4 shows a large shift of towards increased sensationalism for reported findings compared to their counterparts in the original papers. analysis. to validate the robustness of these re-sults, we annotate 108 instances of the unlabeled data for the properties of causality and certainty. sec. a.3.3 provides details on the annotation. evaluating the predicted distortions against the human labels, we find that the results in this setting are robust (macrof 1 of .65 and .59 causal distortions and certainty) and on par with the classifiers' performance on the original test set. notably, given the relatively high precision for predicting the "certain" class (0.75) and the volume of reported findings which are predicted as such, these results provide further evidence that reports frequently overstate findings' certainty (fig. 11 ). for sensationalism we validate the results by analyzing if the difference in distributions that we observe in fig. 4 is in fact significant. we see a large statistically significant effect (fig. 12 ), indicating that different sources are potentially perceived as more or less sensational. ## conclusion given both the societal impact and growing volume of scientific information online, it is crucial to understand how this information is presented to the public. in this work, we lay a foundation for performing large scale analysis and automatic detection of several types of distortions in the reporting of scientific findings. we contribute the first dataset of multiple fine-grained distortions in science communication, allowing us to study how scientific findings are changed when they are reported to lay audiences. we show both that findings frequently undergo subtle distortions when reported, and that detecting these distortions automatically poses a challenging problem. we find that fine-tuning custom models consistently outperforms llm prompting, presumably because the models are not able to effectively leverage the annotation instructions and examples we provide as prompts. using our best baseline models, we study the prevalence of distortions in science communication at large, observing that scientific findings are potentially frequently subject to distortions in terms of causality, level of certainty, and how sensationally they are presented.