Measuring Correlation-to-Causation Exaggeration in Press Releases (2020)

Abstract: Press releases have an increasingly strong influence on media coverage of health research; however, they have been found to contain seriously exaggerated claims that can misinform the public and undermine public trust in science. In this study we propose an NLP approach to identify exaggerated causal claims made in health press releases that report on observational studies, which are designed to establish correlational findings, but are often exaggerated as causal. We developed a new corpus and trained models that can identify causal claims in the main statements in a press release. By comparing the claims made in a press release with the corresponding claims in the original research paper, we found that 22% of press releases made exaggerated causal claims from correlational findings in observational studies. Furthermore, universities exaggerated more often than journal publishers by a ratio of 1.5 to 1. Encouragingly, the exaggeration rate has slightly decreased over the past 10 years, despite the increase of the total number of press releases. More research is needed to understand the cause of the decreasing pattern.

## introduction recent studies have found that press releases issued by research institutions are a major source of exaggeration in science communication, which is later spread to mainstream media @xcite @xcite . exaggeration in press releases threatens to undermine public trust in science, which is a foundation for the scientific research enterprise. the influence of press releases on science communication has increased since the 1980s. with the growing competition for reputation and funding, research institutions are increasingly using press releases as a public relations tool for research promotion @xcite . at the same time, independent journalism faces financial challenges and staff shortage @xcite . newspapers, especially small newspapers, rely heavily on press release material to write science news stories @xcite @xcite . prior research also shows that high quality press releases seem to improve the quality of associated news stories @xcite . since press releases are now the dominant link between academia and news media, the information quality of press releases plays a critical role in communicating science research to the public. the problem of exaggeration in press releases is particularly severe in health research. for example, a manual analysis of a sample of uk press releases in health research found that over a third of them contained exaggerated advice, causal claims from correlational findings, or inference of animal studies to humans @xcite . the errors and inaccuracies in reporting health research findings may also misinform the public about their conditions, diagnosis, and treatments. on the other hand, medicine and health are the dominant topic in science news articles. research outside the health domain was much less covered in science news @xcite . therefore, studying exaggeration in the health domain is particularly important for both science communication and public health. researchers and media watchdogs such as healthnewsreview have been conducting manual content analysis to estimate and monitor press release quality @xcite @xcite . due to the large number of press releases and scarce funding, this labor-intensive approach is difficult to maintain. in fact, healthnewsreview stopped operation at the end of 2018 after a nearly 13-year run. manual content analysis is also inadequate for answering important research questions that require large-scale, real-time analyses. for example, has the exaggeration problem worsened or improved over the years? do some institutions exaggerate more than others? seeking answers to these research questions requires developing a computational approach to automatically analyze exaggeration in press releases. in this study we propose an nlp approach for automatically detecting exaggerated causal claims, one of the most common types of exaggeration, by comparing claims in press releases to the corresponding claims in the original research papers. an exaggerated causal claim is defined as a causal statement in a press release with a correlational counterpart in the corresponding research paper. our study consists of several subtasks. first, we developed a corpus with paper-press pairs. press releases were downloaded from eurekalert! foot_0 , a nonprofit news-release distribution platform operated by the american association for the advancement of science as a resource for journalists and the public. doi links were obtained from eurekalert! and sciencedaily to associate the press releases with the original research papers in pubmed. the corpus includes 64,177 press releases with references to 62,317 pubmed-indexed journal papers. we then trained a classification model to identify research papers on observational studies; they are designed to establish correlational findings, but are often exaggerated as causal @xcite @xcite . focusing on research papers with structured abstracts, in which the sentences in the conclusion subsection were used as main statements in research papers, we identified 15,962 observational studies. the headline and first two sentences of a press release are considered as its main statements, according to the "inverted pyramid" structure commonly used by news stories. we then paired the conclusion subsections in the research papers with the main statements in the corresponding press releases for exaggeration analysis. some research papers were associated with multiple press releases, resulting in 16,431 paper-press pairs dated from 2008 to october 2020 in the final dataset. second, we developed bert-based sentence classifiers to categorize the main statements in press releases and research papers respectively. the sentences were classified by their strength as either "direct causal", "conditional causal", "correlational", or "not claim". third, by applying the sentence classifiers to the 16,431 paper-press pairs, we were then able to identify the press releases that used direct causal claims to describe correlational findings in observational studies. then we used the identified exaggeration cases to answer the following questions: (1) what is the trend of correlation-to-causation exaggeration from 2008 to 2020? (2) did university press releases make more exaggerated claims than journal press releases? ## related work institutional press releases are an important presentation of science research to the public @xcite . a press release serves the dual purpose of science communication and public relations @xcite . now that research and innovation have become essential drivers of economic growth and international competitiveness, research institutions are facing growing competition for research funding and talents, and are increasingly emphasizing the commercialization of university research. such pressure results in tension between the goal of responsible science reporting and the goal of marketing when issuing press releases @xcite . a survey study on uk science press officers found that they were aware of such tension, and were trying to balance and achieve both sensation and accuracy in a delicate way @xcite . in addition, prior studies have found an increasing influence of press release on the content of science news since the 1980s @xcite . actually, press releases have become the dominant link between academia and news media @xcite . a number of studies have found that journalists rely heavily on press release materials to write science news stories, instead of acting as watchdogs by conducting independent investigations @xcite @xcite @xcite . for example, over one third of u.s health news stories were largely or solely based on press releases @xcite . under-staffed smaller newspapers even used press releases verbatim @xcite . this trend is likely to continue as science news reporting positions are cut and journalists are rushed for time @xcite . journalists have long been criticized for pursuing sensational stories that overstate research results @xcite . however, recent studies have found that press releases may contain more exaggerations than news stories. according to @xcite , 58%-86% of news stories derived from the exaggerated press releases contained similar exaggerations, compared with the exaggeration rate of 10%-18% in news articles when the press releases were not exaggerated. in the context of the high institutional expectation for generating "hype" and the shrinking of journalism as the watchdog, it has been argued that press releases have become a major source of exaggeration in science communication, e.g. @xcite @xcite . furthermore, university press releases were found to contain more exaggerations than journal press releases, probably because the university press officers face more pressure to generate expectations and hype @xcite . prior studies on exaggeration in press releases have been limited to manual content analysis, which have provided valuable insights for understanding the types of exaggeration and estimating the problem severity at certain times. for example, @xcite is the first study on estimating the exaggeration rate in press releases by manually checking a sample of press releases from 20 uk univerities in 2011. however, the manual content analysis method falls short for answering important research questions that require large-scale, real-time analyses, such as longitudinal trend of exaggeration rate over time. this creates a need to develop a computational approach that can consistently measure exaggeration in a large number of press releases, and further study its trend and difference by factors such as the source of press releases (e.g. academic institutions vs. journal publishers). although computational modeling of exaggeration in press releases is a new research problem, it is closely related to the field of nlp applications for information credibility assessment, especially in the health domain, given the strong need for curbing misinformation in recent years. the task of credibility assessment has often been modeled as a text categorization problem. a common way to categorize credibility is true/fake binary prediction, e.g. @xcite . however, many online documents are not completely true or fake. instead, multiple claims can be made in one document, and claims can be inaccurate in different ways. therefore, a more precise way to assess credibility is faceted evaluation guided by established credibility frameworks with multiple quality criteria. some frameworks were designed for rating online health information, such as the hon principles and discern. for example, the discern instrument includes 16 questions, such as "are the aims clear?" and "is it balanced and unbiased?" @xcite . in comparison, healthnewsreview.org designed 10 criteria specifically for news stories. these criteria include exaggeration, missing information, etc. expert ratings based on these criteria have been used to train nlp models. for example, an automated system was developed to assess health websites' conformity to the hon principles @xcite . autodiscern is an nlp tool designed to automatically assign discern quality ratings to webpages using hierarchical encoder attention-based neural network @xcite . nlp models were also developed to automate ratings based on the ten healthnewsreview criteria @xcite . these methods rely on human expert ratings to train credibility prediction models. in comparison, our method links press releases to the original research papers for fact checking, and we focus on exaggeration, a specific criterion used by healthnewsreview, due to its significant impact on the quality of press releases @xcite . our work is also built upon prior studies on identifying causal relations, e.g. @xcite . a causal relation can be broadly defined as any type of cause-effect relation between events, for example event a is the consequence of event b @xcite . it is commonly realized by causal indicators such as "because", "as a result of", "as a consequence of" @xcite . this broad definition is also adopted in the nlp tasks semeval-2007 @xcite and semeval-2010 @xcite . however, a causal relation presented in a research finding sentence is in a more specific linguistic format, i.e. a relation between an independent variable and a dependent variable @xcite , such as "smoking causes cancer". to identify causal claims in research findings, @xcite developed a corpus of claims with different strength levels and trained a bert-based model to distinguish them. compared to research papers, press releases are a different genre. @xcite found that genre differences may result in more challenge for identifying causal claims in news articles. based on these prior studies, we decided to construct a new corpus for identifying causal claims in press releases, which are more similar to news stories, and to develop prediction models based on bert. ## sentence-level claim type classification due to the significant difference in writing styles between academic papers and press releases, we chose to train separate prediction models, one for each genre. for academic papers, we adopted the classification model developed by @xcite that has achieved a performance of 0.88 macro-f1 and 0.90 accuracy. for press releases, we trained a new prediction model using the press release corpus that we constructed as described above. ## conclusion and future work in this study we examined patterns of correlation-to-causation exaggeration in press releases. we developed a new corpus with 16,431 pairs of observational study papers and associated press releases, and a prediction model for identifying cases of correlation-to-causation exaggeration in press releases. applying the prediction model to the new corpus, we found that among all observational study papers that made correlational claims only, 22% of the associated press releases made exaggerated causal claims. furthermore, universities made more exaggerated claims than journal publishers by a 1.5 to 1 ratio. the good news is the exaggeration rate has slightly decreased in the past ten years, despite the increase in the total number of press releases. more research is needed to understand the cause of the decreasing pattern. this study has a few limitations which we plan to improve in future work: first, the current exaggeration detection method relies on the structured abstract of a research paper to identify its main claims from the conclusion subsection. this restriction limits our data set size since only about one third of papers have a structured abstract. in future work we will design an algorithm that can automatically identify the conclusion sentences from unstructured abstracts. second, the current approach uses the doi links provided by eurekalert! and sciencedaily to link research papers and press releases; however, these manually-added links cover less than 50% of eurekalert! press releases. in future work we would like to develop a content-based matching system to link press releases to scientific literature. third, the current approach matches the main claims in research papers and press releases at article level, by utilizing the inverted-pyramid structure in press release. this approach works well for press releases, but might not be suitable for identifying exaggeration in other genres. for example, inverted-pyramid structure is not common in online discussion forums (e.g. reddit @xcite ) and social media posts. some news articles may also cite findings from multiple scientific studies. in future work we would like to develop a sentence-level approach for pairing claims in research papers and their paraphrased versions in news and social media.