Modeling Information Change in Science Communication with Semantically Matched Paraphrases (2022)

Abstract: Whether the media faithfully communicate scientific information has long been a core issue to the science community. Automatically identifying paraphrased scientific findings could enable large-scale tracking and analysis of information changes in the science communication process, but this requires systems to understand the similarity between scientific information across multiple domains. To this end, we present the SCIENTIFIC PARAPHRASE AND INFORMATION CHANGE DATASET (SPICED), the first paraphrase dataset of scientific findings annotated for degree of information change. SPICED contains 6,000 scientific finding pairs extracted from news stories, social media discussions, and full texts of original papers. We demonstrate that SPICED poses a challenging task and that models trained on SPICED improve downstream performance on evidence retrieval for fact checking of real-world scientific claims. Finally, we show that models trained on SPICED can reveal large-scale trends in the degrees to which people and organizations faithfully communicate new scientific findings. Data, code, and pre-trained models are available at http://www.copenlu.com/publication/2022_emnlp_wright/.

## introduction science communication disseminates scholarly information to audiences outside the research community, such as the public and policymakers @xcite . this process usually involves translating highly technical language to nontechnical, less-formal language that is engaging and easily understandable for lay people @xcite . the public relies on the media to learn about new scientific findings, and media portrayals of science affect people's trust in science while at the same time influencing their future actions ## related work the analysis of scientific communication directly relates to fact checking, scientific language analysis, and semantic textual similarity. we briefly highlight our connections to these. fact checking automatic fact checking is concerned with verifying whether or not a given claim is true, and has been studied extensively in multiple domains @xcite including science @xcite @xcite . fact checking focuses on a specific type of information change, namely veracity. additionally, the task generally assumes access to pre-existing knowledge resources, such as wikipedia or pubmed, from which evidence can be retrieved that either supports or refutes a given claim. our task is concerned with a more general type of information change beyond categorical falsehood and is a required task to complete prior to performing any kind of fact check. scientific language analysis automating tasks beneficial for understanding changes in scientific information between the published literature and media is a growing area of research @xcite @xcite @xcite @xcite @xcite . the three tasks most related to our work are understanding writing strategies for science communication @xcite , detecting changes in certainty @xcite , and detecting changes in causal claim strength i.e. exaggeration @xcite . however, studying these requires access to paired scientific findings. to be able to do so at scale will require the ability to pair such findings automatically. ## spiced we introduce spiced, a new large-scale dataset of scientific findings paired with how they are commu-nicated in news and social media. communicating scientific findings is known to have a broad impact on public attitudes @xcite and to influence behavior, e.g., the way vaccines are framed in the media has an effect on vaccine uptake @xcite . building upon prior work in nlp @xcite @xcite , we define a scientific finding as a statement that describes a particular research output of a scientific study, which could be a result, conclusion, product, etc. this general definition holds across fields; for example, many findings from medicine and psychology report on effects on some dependent variable via manipulation of an independent variable, while in computer science many findings are related to new systems, algorithms, or methods. following, we describe how the pairs of scientific findings were selected and annotated. ## scientific information change models we now use spiced to evaluate models for estimating the ims of finding pairs in two settings: zero-shot transfer and supervised fine-tuning. ## application: zero-shot evidence retrieval for scientific fact checking accurately measuring the similarity of scientific findings written in different domains enables a wide range of downstream analyses and tasks. as a first task, we consider evidence retrieval for scientific fact checking of real-world scientific claims. in general, automatic fact checking consists of retrieving relevant evidence for a given claim and predicting if that evidence supports or refutes the claim. we test the ability of models trained on spiced to perform the evidence retrieval task in a zeroshot setting. in this, we use the models as is, with no further fine-tuning on any evidence retrieval data. we consider two fact checking datasets: covert @xcite is a dataset of scientific claims sourced from twitter, mostly in the domain of biomedicine. we use the 300 claims and the 717 unique evidence sentences in the corpus in our experiment. covid-fact @xcite is a semi-automatically curated dataset of claims related to covid-19 sourced from reddit. the corpus contains 4,086 claims with 3,219 unique evidence sentences. ## application: modeling information change in science communication whether the media faithfully communicate scientific information has long been a core question to the science community (national academies of sciences, engineering, and medicine, 2017). our dataset and models allow us to conduct a largescale analysis to study information change in science communication. here, we focus on three research questions: rq1-2 focus on the holistic information change captured in ims, while rq3 focuses on what types of information might be changing.