Writing Strategies for Science Communication: Data and Computational Analysis (2020)

Abstract: Communicating complex scientific ideas without misleading or overwhelming the public is challenging. While science communication guides exist, they rarely offer empirical evidence for how their strategies are used in practice. Writing strategies that can be automatically recognized could greatly support science communication efforts by enabling tools to detect and suggest strategies for writers. We compile a set of writing strategies drawn from a wide range of prescriptive sources and develop an annotation scheme allowing humans to recognize them. We collect a corpus of 128k science writing documents in English and annotate a subset of this corpus. We use the annotations to train transformer-based classifiers and measure the strategies’ use in the larger corpus. We find that the use of strategies, such as storytelling and emphasizing the most important findings, varies significantly across publications with different reader audiences.

## introduction communicating scientific discoveries to a general audience of readers is difficult. a researcher or writer interested in doing so is faced with the challenging task of translating complex scientific ideas in an engaging manner without misleading or overwhelming their audience. there are many guides to science communication (e.g., @xcite ), but they rarely offer empirical evidence for how their advice is used, or proven effective, in practice. the potential science communicator is then confronted with the additional hurdle of understanding how to implement these guidelines in their writing. effective science communication requires understanding the unique needs and expectations of different audiences and stakeholders in science @xcite . we envision natural language processing technologies that help science writers communicate more effectively. these technologies might automatically classify common strategies in a writer's own text, support writers to adapt language to specific readers, or guide readers through personalized article recommendations. as a first step, we compile a set of strategies from a wide range of prescriptive science writing sources in english and develop an annotation scheme allowing humans to recognize these strategies in texts about science. we introduce a new corpus of 128k university press releases, science blogs, and science magazines and annotate a subset of 337 texts. we use the annotations to train transformer-based classifiers to explore the communicative goals of science writing by analyzing variations in the strategies' use across several scientific communication forums. our paper is the first computational analysis of writing strategies driven by science communication theory. we find that most strategies are prevalent throughout our corpus and that publication venues with varying audiences use strategies differently. for example, press releases emphasize the impacts of science more than magazine articles, which instead tell more stories about the science. we also find that higher quality newspaper articles, as rated by expert journalists, use more storytelling and analogies than lower quality articles. ## defining science communication writing strategies the goal of general science communication is to increase public awareness, enjoyment, interest, and understanding about science @xcite . based on the idea of compositionality in discourse theory (bender and lascarides, 2019), we can think of the communicative intent of science writing as being made up of smaller communication goals rep-resented in particular passages of an article @xcite . our computational approach builds on this theoretical assumption by annotating sentences and letting an article inherit the attributes we find in its sentences. past work on science communication has taken a similar view @xcite by using syntactic relations to characterize an article's communicative goals, allowing them to emerge inductively rather than from a theory of science communication. our complementary approach starts with science communication guides to construct theory-driven communicative goals (referred to as "writing strategies" and consisting of lexical to multi-sentence features), and explores their use in a diverse range of science communication text. to define our writing strategies, we categorized and grouped advice from style guides for science communicators. these guides were a mix of online resources, books, and academic articles (see two authors open-coded @xcite the suggestions from each guide by assigning each piece of advice in a resource a code that represented its highlevel strategy (such as "avoid jargon"). the authors then looked at other resources to see whether the same advice appeared there. each new piece of advice was added with a new code. after coding all resources, the authors grouped the codes into a set of 10 suggested writing strategies. appendix a.1 provides additional details on the coding and categorization. the strategies are as follows (examples of each are given in lede a few sentences at the beginning of an article, called a lede (spelled "lede" for easier differentiation with its homograph "lead"), that draws a reader in and makes them want to read more. main sentences describing the main findings being reported by the original paper in order to not overwhelm the reader with details. impact writing about the real world impact of the science or findings being reported in order to excite readers. this can include future technolo-gies, breakthroughs the findings might enable, or their societal implications. explanation explanations about scientific subjects to improve reader understanding. this could be explaining a certain topic or word, or what researchers did in a study and what the findings mean. analogy analogies or metaphors used as a way to explain concepts or make ideas in the article more relatable. story stories to engage readers and make the reported science more interesting. this can include short story snippets, or coming back to an underlying story throughout an article. ## dataset in order to study the use of these strategies by science writers and build classifiers for automatic identification, we collected a corpus of 128k documents from a variety of science communication sources. we focused on four major types of u.s.based venues, representing a broad spectrum of science communication for different audiences: blog sites, popular science magazines, university press releases, and scientific journal magazines. past work has shown that blog sites usually write to scientifically literate and engaged readers @xcite , while university press releases often write to other science journalists @xcite . we selected popular science magazines since they target a more general audience, and scientific journal magazines as they often write to those involved in research, though not necessarily in the same domain @xcite . the choices of website or publication we collected from each venue category were based either on previous work covering those categories (e.g., blog posts; @xcite or as a convenience sample based on what was widely available. one note is that while past work has used the blogs sites we selected as sources for high quality science blogs (sciencedaily.com and phys.org), these sites also source a large portion of their content from press releases, often only changing headlines and lede sentences. we scraped articles from each of these sources for all of 2016-2019 using the wayback machine, foot_1 resulting in 137,828 articles. appendix a.2 provides more details on site selection. ## hypotheses our annotated corpus allowed us to begin to explore how strategies relate to the communicative goals of different science communication venues. to do this, we introduce hypotheses informed by existing literature. hypotheses h1-h4 are based on our expectations for how strategies can differentiate science writing venues in our corpus based on their underlying goals. for these hypotheses, we evaluate strategy use across our corpus. hypotheses h5 and h6 focus on how strategies might relate to other important issues in science communication. they build on past research in science communication exploring article quality @xcite and sensationalism @xcite . these issues are introduced with their own annotated datasets, and since we have no strategy annotations for these datasets, we report only on the aggregated predictions of our classifiers. h1: lede is used once or twice within an article, but consistently across our entire corpus. because the lede strategy is well adopted as a common strategy in general journalism @xcite , and lede sentences are only used at the beginning of an article, we expect low but consistent use of lede across the corpus. h2: press releases use higher impact than other venues. one goal of press releases is to encourage other science writers to pick up a story. because a key component to selecting stories is impactful findings @xcite , we expect that press releases will emphasize this more. h3: magazines use lower jargon, higher ac-tive and present and higher story than other venues. magazines target a broader readership compared to other venues, making it likely they use these strategies that are common in prescriptive guides for general good writing @xcite to relate to a wider audience. ## strategy classification we used our collected corpus and annotations to automate recognition of writing strategies and to evaluate our hypotheses. we describe our methods for classifying strategies with rules and with human annotations (sections 5.1 and 5.2). we then discuss methods for using these classifiers to estimate the use of strategies in our corpus and overall classifier performance (section 5.3). ## evaluating strategy applications evaluating our classifier output against goldstandard human annotations, as reported in section 5.3, establishes the validity of our classifiers. we next turn to our hypotheses introduced in section 4 to illustrate how we can use the strategies, classifiers, and corpus to explore the communicative goals of science writing. we introduce each hypothesis and report on its results separately. h1: lede is used once or twice within an article, but consistently across our entire corpus. figure 2a plots the estimated number of lede sentences per article across each site. supporting h1, the majority of sites peak at either 0 or 1 lede sentences, with all sites tapering off quickly after that. theatlantic.com does have a higher number of predicted lede sentences (with 20% of articles containing more than 2 sentences). this might be due to theatlantic.com articles being longer (since they are full magazine articles) and therefore using more text to entice readers. h2: press releases use higher impact than other venues. we find support for h2: press release sites like news.harvard.edu, rochester.edu, and news.stanford.edu have larger modes than other sites for impact sentences in figure 2c . for example, close to 15% of articles in rochester.edu have over 5 estimated impact sentences, compared to 7 or 8% of articles in scientificamerican.com or theatlantic.com having that same number. this is especially striking because scientificamerican.com and theatlantic.com generally have much longer texts, since they are full magazines, compared to press releases. ## related work we highlight additional areas of research relevant to our work beyond those already discussed. ## future work we annotated writing strategies at the sentence level, but some strategies, such as story and analogy, might be better annotated at the fragment level to account for longer or shorter use of strategies. future work can explore more finegrained analysis of these strategies (e.g., with metaphor detectors; @xcite . we also hope to build on these findings by exploring how effective the strategies are at engaging different readers. ## conclusion in this paper we compile writing strategies from theory and practical advice, collect a large corpus and annotate a subset of it to measure strategies' use. we observe how strategies covary with intended audience. for example, blog sites, which target researchers, use more jargon and focus on the main findings of a paper, while magazine articles, which target a much broader audience of readers, tell more stories and use more active voice. our findings also suggest that science newspaper articles judged by experts to have higher quality use more metaphorical language and tell more stories. we expect that our strategy formulations, classifiers, annotations and dataset will enable nlppowered tools to support effective science communication for different audiences. online article communicating with the public from aaas @xcite online article explaining tech to non-techies @xcite online article tips for communicating scientific research to non experts (scientifica) online article what does research say about effective communication about science? (maynard and scheufele, 2020) school handled press separately), or the majority of press was unrelated to research output. as