Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communications (2021)

Abstract: Certainty and uncertainty are fundamental to science communication. Hedges have widely been used as proxies for uncertainty. However, certainty is a complex construct, with authors expressing not only the degree but the type and aspects of uncertainty in order to give the reader a certain impression of what is known. Here, we introduce a new study of certainty that models both the level and the aspects of certainty in scientific findings. Using a new dataset of 2167 annotated scientific findings, we demonstrate that hedges alone account for only a partial explanation of certainty. We show that both the overall certainty and individual aspects can be predicted with pre-trained language models, providing a more complete picture of the author’s intended communication. Downstream analyses on 431K scientific findings from news and scientific abstracts demonstrate that modeling sentence-level and aspect-level certainty is meaningful for areas like science communication. Both the model and datasets used in this paper are released at https://blablablab.si.umich.edu/projects/certainty/ .

## introduction expressing certainty about what is known is a necessary characteristic of scientific work as science involves producing knowledge about what was previously unknown @xcite . given the natural aversion to uncertainty, existing studies have found that presenting uncertainty in science communications influences people's perception of scientific findings and trust in science @xcite @xcite . therefore, understanding how journalists and scientists communicate certainty and uncertainty is critical for understanding the current ecosystem of science journalism and further provides better guidance for uncertainty communication @xcite . arctic sea ice is definitely declining [probability ] at a rate of 13.3 [number] percent per decade [extent] and we believe that [psychological] the society may need to take action [suggestion] to control global warming. arctic sea ice is declining [probability ] … at a rate of 13.3 [number] percent per decade [extent] and we believe that [psychological] the society may need to take action [suggestion] to control global warming. arctic sea ice could be declining [probability ] … at a rate of 13.3 [number] percent per decade [extent] and we believe that [psychological] the society may need to take action [suggestion] to control global warming. arctic sea ice might be declining [probability ] at a rate of 13.3 [number ] percent per decade [extent ] and we believe that [framing ] the society may need to take action [suggestion ] to control global warming. uncertain certain s e n te n c e -l e v e l c e rt a in ty figure 1 : certainty is a multi-dimensional construct. the certainty of a scientific finding can be perceived holistically at the sentence level from its description. however, scientific findings may involve multiple aspects that may each be described as certain (↑ ↑) or uncertain (↓ ↓) (aspect-level certainty). multiple studies in linguistics, nlp, and the science of science literature have examined how certainty is expressed. these studies have modeled certainty in multiple ways, including epistemic modality @xcite , semantic uncertainty @xcite , verbal uncertainty @xcite , factuality @xcite , and hedging @xcite . in practice, most uses of uncertainty rely on hedging as a coarse characterization of the overall uncertainty @xcite . however, as suggested by @xcite , certainty itself is a complex construct and has to be modeled from multiple dimensions. the complex and subjective nature of certainty makes annotation challenging, often resulting in moderate-to-low annotator agreement @xcite , and motivating a better and more practical way to model and annotate certainty in text. we propose to study certainty from the commonly-used sentence-level and, in parallel, introduce a new dimension of aspect-level certainty, providing a fine-grained description of how certainty is communicated in text. this approach is analogous to work in sentiment analysis that models both holistic @xcite , and aspect-level valence (schouten and frasincar, 2015)-and their interactions. based on existing categorizations of certainty, we compile six aspects of scientific findings including: number, extent, probability, framing, condition, and suggestion. following carefully designed annotation guidelines and after extensive annotator training, we introduce a new annotated corpus of 2200 scientific findings equally, sampled from news and scientific abstracts for both sentencelevel and aspect-level certainty, and attained reliable inter-annotator agreements. analysis with this dataset suggests that the number of hedges can only partially explain the variance of the overall sentence-level of certainty (pearson's @xmath0.55). therefore, to better model certainty in scientific findings, we fine-tuned a scibert @xcite for the two tasks and it achieves 0.63 pearson's r for sentence-level certainty and an average of 0.66 binary-f1 for aspect-level certainty. our paper offers the following three contributions. first, we provide the first dataset of scientific findings annotated with both sentence-level and aspect-level certainty and fine-tune neural language models to predict certainty in scientific findings. second, using our best-performing model, we infer the sentence-level and aspect-level certainty for 431k scientific findings in news and abstracts and show that the sentence-level certainty of findings in abstracts is associated with journal impact factor and team size. regression analysis reveals that low-impact journals and large teams often present scientific findings with higher sentencelevel certainty. third, using 6586 findings from abstracts paired with their description in news, we find that news reports a finding with lower certainty than its corresponding description in the abstract. fine-grained regression analysis over aspect-level certainty further reveals that journalists describe some key aspects with less certainty. despite some studies suggest that news reports tend to describe uncertain findings as more certain (e.g., @xcite , our study indicates that journalists may actually play down the certainty when reporting scientific findings. ## modeling certainty communicating certainty involves qualifying parts of a statement to indicate what is known and what the speaker's beliefs are. multiple approaches have proposed models for this property of language. the majority of these studies have focused on the degree or level of certainty @xcite . within this approach, hedges have been considered effective proxies for uncertainty @xcite across multiple fields including linguistics @xcite , economics @xcite , and psychology @xcite , where the frequent use of hedges indicates more uncertainty and absence indicates certainty. other work has proposed examining which aspects of language contribute to the perception of uncertainty. for example, @xcite proposed a four dimension model of certainty, including perspective, focus, timeline, and level, and each dimension contains several sub-categories. following, we synthesize multiple approaches for measuring sentence-level and aspect-level certainty and proposed a representative categorization. sentence-level certainty prior research has assumed that the level of certainty for a finding is presented, perceived, and further analyzed within one or several sentences @xcite @xcite . this aggregate perception represents a unified perception of various information expressed in the given piece of text and is the primary judgment of certainty along a continuum from uncertain to certain @xcite . this perception of this overall level of certainty is known to influence people's following actions in many contexts @xcite . therefore, in modeling certainty, we include a sentence-level estimate of the certainty for a scientific finding's description. aspect-level certainty in describing a finding, authors may contextualize which aspects are more or less certain. many categorizations have been proposed to model the complex nature of these descriptions of certainty and uncertainty including types of uncertainty @xcite , sources of uncertainty @xcite , issues of uncertainty @xcite , forms of uncertainty @xcite , focus and perspective of certainty @xcite . while prior work focuses on various aspects of certainty, their categorizations are often domain-specific @xcite or overly limited in scope to cover common aspects appearing in scientific findings @xcite . here, we synthesize prior approaches and propose six repre-sentative aspects of scientific findings which could involve certainty or uncertainty, with a goal of creating a comprehensive scheme that captures most of what is seen in knowledge-intensive corpora. number refers to certainty towards specific quantities. for example, "approximately 250 individuals participated in this study" is uncertain towards number. numerical information is vitally important in science communication as it is found to be the best way to promote scientific understanding in situations like climate change @xcite and health @xcite . accordingly, the imprecision of numbers or the inaccuracy of calculations are usually considered as a form of uncertainty @xcite . how to effectively communicate numerical information in scientific findings has been identified as one of the major challenges of science communication @xcite . identifying certainty regarding number in scientific findings could help to understand how journalists and scientists communicate certainty about numbers and inspire better ways to communicate this information. extent refers to certainty about the proportion/ratio of properties that make up an object/event or the extent of a change. for example, "this bridge is mainly composed of agate" and "we observe a moderate increase of suicide in winter" involves uncertainty towards extent. extent can be described with numbers in certain situations. for example, "the average sea level across the world increased by approximately 30%" expresses extent via a number. however, unlike numbers that focus on specific quantities, extent focuses on the components of an object, substance, or the extent of a change/effect. previously, extent was not explicitly proposed as a source of uncertainty, although some studies have brought up similar ideas. for example, @xcite considers "uncertainty about how much [the] impacts matter" as a form of uncertainty and @xcite propose uncertainty about the strength or validity of evidence about risks, which may not be described with specific quantities. existing studies suggest that journalists may misreport the extent of scientific findings to which it is supported by evidence @xcite , motivating its inclusion here. probability refers to certainty about the probability that something will occur, has occurred, or is associated with another factor. for example, "this medicine could possibly cure cancer" and "a is possibly associated with b" involves uncertainty about probability. probability has been widely recognized as one major source of uncertainty @xcite @xcite and how to communicate probabilities effectively has long been an important question in science communication @xcite . condition refers to the situation where something depends on a specific condition, and the condition involves certainty or uncertainty @xcite . scientific findings are often qualified by specific conditions under which the result is valid, which may themselves be certain or uncertain @xcite . for example, "cancer could be cured if the medicine can be made shelfstable" is uncertain regarding condition. framing refers to the certainty about how scientists or journalists themselves frame or interpret the scientific finding. for example, "we suspect a has effects on b" involves the uncertainty from the authors while "we conclude that a has effects on b" frames the finding with conviction. this aspect is related to expressions about epistemic uncertainty(the speaker having or lacking knowledge @xcite and psychological uncertainty in the psychology literature @xcite . in the news, journalists actively add their interpretations about the original information @xcite and different framing may further affect people's perception of the overall certainty of the presented information @xcite . therefore, identifying the certainty about framing could help us to better understand how journalist's framing affects people's perceptions of scientific findings. suggestion refers to certainty or uncertainty about the implications or future actions for the public or science community. scientific findings do not only describe facts, but can also communicate practical implications for people's daily life @xcite . for example, "patients probably need more medicine to cure this disease" involves uncertainty regarding future actions. uncertainty about suggestion was previously identified as dynamic uncertainty in @xcite . a single scientific finding may include multiple aspects with their own certainties. for example, "the vaccine is effective for 76% of chances" is uncertain regarding probability while is certain about the specific number. similarly, "the sci-entists need to do more research to understand the effect of a on b" indicates the uncertainty about probability but is certain about suggestion. ## data to study certainty, we construct a dataset of scientific findings reported in news and research articles. news data comes from altmetrics, which tracks mentions of scientific articles in news outlets. we restrict our analysis to u.s.-based outlets where we could retrieve the full text of the article and where the doi for the scientific article was recorded in the microsoft academic graph (mag), which provides metadata on the article (e.g., authors, abstract, and publication venue). supplemental material §a contains additional details on preprocessing steps. a total of 128,942 news/article pairs were collected, spanning 273 different news outlets and 57,807 different scientific articles. for scientific articles, we extract the findings from the abstract reported in the mag using the abstract parser developed by @xcite , which labels sentences as background, method, introduction, result, and conclusion. we use sentences labeled as result or conclusion in our analysis. for news, we adopt a heuristic approach and identify all sentences containing a discoveryrelated keyword (e.g., find, conclude). we retain the subjective clause after the verb as the finding. examples of findings produced by each method, as well as additional details, are reported in supplemental material §c. this process finally leads to 608,694 unique scientific findings from abstracts and 106,612 unique scientific findings from news reports. among the 128,942 news-paper pairs, 52,406 have both identified findings from news and paper abstracts. ## annotating certainty we annotate scientific findings for both their sentence-level certainty and the presence and certainty of each of the six aspects. given subjectivity in perception, the same expression of certainty may evoke different perceptions @xcite . prior work in annotating uncertainty has generally reported low to moderate inter-annotator agreement @xcite ; for example, rubin (2007) reports cohen's κ=0.41 when annotating news certainty with a five-level likert scale. to mitigate these challenges, we carefully designed the annotation procedures which are described in this section. annotation setup annotators were recruited from a us university and received initial one-hour training. all annotators are fluent in english and have extensive experience in reading scientific news and research articles. annotators who attained high iaa with our gold standard were retained and then went through four additional rounds of pilot annotation and discussions (2 rounds for sentence-level and 2 for aspect-level) to build consensus. all annotators were paid $15/hr for training and annotation. annotation was performed in three phases. in the first phase, the initial data was sampled in a way to be more balanced across levels of certainty. markers of certainty are not equally distributed throughout scientific communication @xcite ; for example, 87% of the data labeled in henriksson and velupillai (2010) were found to be very certain. therefore, in the initial data, we sample 1000 findings equally from news and paper abstracts where 50% scientific findings containing no hedges, 35% with one hedge, and 15% with 2+ hedges. the hedge words are collected from @xcite . the annotators were first asked to rate how certain they perceived the finding on a six-point likert scale as the sentence-level certainty. aspect-based ratings were performed in a separate round so as not to potentially bias annotators towards focusing their sentence-level judgments on the basis of aspects. for each aspect, annotators were asked to assess whether that aspect was present and, if so, whether the language for the aspect was certain or uncertain. for instances that are clearly not scientific findings, the annotators were instructed to label it as bad-text. each finding was rated by at least two annotators for sentence-level and, due to increased variance observed during training, three annotators for aspect-level. in this phase, 2349 sentence-level and 3209 aspect-level annotations were collected for 1000 findings. annotators had a high agreement in both sentence-level and aspect-level tasks. for sentence-level, annotators attained a krippendorff's α=0.67, which is substantially higher than iaa for the closest comparable task (rubin, 2007, cohen's κ=0.41). 1 the final sentence-level rating is computed as the average across all annotators' scores. for aspect-level cer- in the second phase, given the label imbalance in both sentence-level and aspect-level certainty, we sample additional data for annotation based on model predictions using different sampling strategies for sentence-level and aspect-level certainties. for sentence-level certainty, we fine-tuned a roberta-base classification model and predicted all the extracted findings with sentence-level certainty. we further sampled 400 findings with low model confidence equally from news and abstracts for the second-phase annotation. for aspect-level certainty, we fine-tuned a scibert classification model and up-sampled 600 findings for less frequent aspects, including suggestion, extent, condition, and framing. given that the first and second phase data do not reflect the distribution of certainty in a natural sample, we further randomly sampled 200 findings from all the extracted findings as to the third phase. given that the annotators were capable of doing reliable annotations after the first phase, annotators independently annotated 1200 findings for the second and third phases. ## predicting certainty in this section, we build models to predict sentenceand aspect-level certainty in scientific findings to support downstream analyses of certainty. we test two linear baseline models and two deep-learning models based on neural language models. as linear baselines, we include a model using bag-of-words (bow) features and another based on the frequency of each hedging word. for neural models, we use @xcite , and roberta model @xcite as the base models and fine-tune them over our annotated dataset. for both sentence-level and aspect-level certainty, the data labeled in phase 1 and phase 2 are split 8:1:1 into training, validation, and test. to better reflect the expected performance generalization, the test is made from the random set annotated in phase 3 and the 10% test partition of phases 1 and 2. for all the models, we also report their performance on the random test set to demonstrate their performance over the natural samples. supplemental section §b describes additional training details. ## certainty in science communications certainty is a core aspect of science communication @xcite and presenting certainty in different forms (i.e., aspects) may further affect people's perception and future action about a series of issues including climate change @xcite and the covid-19 vaccine @xcite . our models and dataset enable us to study how journalists and scientists present certainty in science communications. here, we focus on the following five research questions. rq1: are findings in science news more certain than those in paper abstracts? rq2: do journalists and scientists differ in their use of aspect-level certainty? rq3: does aspect-level certainty in abstract findings affect the sentence-level certainty in news findings? rq4: does journal impact factor affect the certainty of scientific findings and how they are covered in news reports? rq5: does team size affect the certainty of scientific findings and how they are covered in news reports? rq1-rq3 focus on changes to the description of certainty in the science communication process. while studies have found that news reports tend to describe uncertain findings as more certain @xcite , some studies suggest that news articles may also add more uncertainty to science finding in some cases @xcite . our model and dataset allow us to study (1) if certainty is changed, (2) if so, what aspects are changed, and (3) what drives the change. rq4-rq5 examine external factors that may affect how journalists and scientists present certainty. we focus on (4) the prestige/quality of the journal, asking whether lower-or higher-quality journals differ in how certain their findings are, and (5) in the era of team science @xcite , whether team size influences how certain the authors describe the results, given known differences in the nature of team science research outputs @xcite @xcite . data and method for rq1 and rq2, to control the effects of the content of the finding, we propose a method to match the same scientific finding in news and paper abstracts. for each extracted finding in a paper abstract, we identify the paraphrased findings in the corresponding news article reporting on that paper. we first remove all punctuation and stop words and then stem all the words in each sentence. next, we calculate the overlap and jaccard similarity between each pair of findings in the news and abstract. we manually evaluated the matched findings and set word @xmath1.3 as the threshold. based on the findings from 52,406 news-paper pairs, we identify 6,586 unique finding pairs from news and abstracts. we manually annotated 70 matched finding pairs, and 63 (90%) refer to the same science finding, indicating high precision of our matching process. supplemental material §f shows a random sample of the matched finding pairs. we construct separate regressions predicting the sentence-level (rq1) and each aspect-level (rq2) certainty in findings with the source of the finding (i.e., news or abstract). we further control the fields, author and affiliation ranking, journal impacts, finding length, and flesch reading ease score. for rq3, we construct a regression predicting the overall sentence-level certainty in news findings with the aspect-level certainty in the corresponding finding from the paper abstract. except for all the ivs above, we further control the news outlet and the sentence-level certainty of the finding in the abstract. for rq4 and rq5, we construct a regression predicting the sentence-level certainty in 265,758 findings presented in 55,178 paper abstracts using journal impacts factors and the number of authors. recognizing its limitations (kurmis, 2003), we use journal impact factor as a proxy for the quality of science based on prior use @xcite . we include controls the field of the research, author, and affiliation ranking extracted from the microsoft academic graph @xcite , finding length and flesch reading ease score to remove the potential confounds. to test the connection between certainty in news findings and external factors, we construct another regression to predict the level of uncertainty in 72,013 findings presented in 27,000 news articles. besides all the ivs regarding abstract and authors, we also control the outlet to remove potential confounds. rq1: are findings in news more certain than those in paper abstracts? the regression analysis (details in supplemental