Automatic Explanation Generation For Climate Science Claims (2022)

Abstract: Climate change is an existential threat to humanity, the proliferation of unsubstantiated claims relating to climate science is manipulating public perception, motivating the need for fact-checking in climate science. In this work, we draw on recent work that uses retrieval-augmented generation for veracity prediction and explanation generation, in framing explanation generation as a query-focused multi-document summarization task. We adapt PRIMERA to the climate science domain by adding additional global attention on claims. Through automatic evaluation and qualitative analysis, we demonstrate that our method is effective at generating explanations.

## introduction the rapid dissemination of misinformation and disinformation through social media is a pressing issue, especially in the domain of climate science @xcite where climate change has become one of the biggest challenges to humankind. claims such as 97% consensus on human-caused global warming has been disproven seed scepticism, discredit climate science, and manipulate public perception and interpretation. to alleviate the influence of such potentially false claims, experts have increasingly engaged in science communication, including investigating such claims based on scientific evidence through websites such as climatefeedback.org and skepticalscience.com. this paper concerns the use of external knowledge to semi-automate the process of claim verification, as an assistive technology for contributors to such websites. inspired by recent work on retrieval-augmented generation @xcite and explainable fact-checking @xcite , we aim to (semi-)automate the process of claim veracity classification along with explanation generation. our work draws on previous work on generating explanations in the climate science domain @xcite text label c: sea-level rise is not accelerating. refu e1: climate-change driven accelerated sealevel rise detected in the altimeter era. ## related work fact checking is the task of assessing whether a textual claim is true, based on a corpus or knowledge base. conventionally, the task is performed manually by human experts @xcite . however, manual efforts do not easily scale @xcite , leading to increasing attention in automatic fact checking (wang, 2017; alhindi et al., 2018; @xcite @xcite . debunking simply by assigning a false label to the claim is not persuasive, and can even reinforce mistaken beliefs @xcite . as such, it is necessary for automated fact-checking methods to provide explanations to support model predictions. separately, there has been recent work on extracting parameterized knowledge from large language models @xcite , as well as augmenting them using external knowledge sources through retrieval augmentation @xcite @xcite . here, a claim or question is used to retrieve documents, which are fed into the generator as additional inputs, as a means of extending and domainadapting large language models without additional pre-training. there has also been recent work on the applications of nlp to the domain of climate science. bhatia et al. (2021b) explored automatic classification of neutralization techniques in discourse relating to climate change/science. diggelmann et al. (2020) introduced climate-fever as a novel dataset for veracity prediction. the closest work to our own is that on explanation generation by bhatia et al. (2021a), which is based on fusion in decoder @xcite , a sequence-to-sequence model that takes as input the claim and passages sourced through retrieval augmentation @xcite . unlike prior work, we first approach the task via multi-document summarization @xcite @xcite , with a focus on the claim; as such, our approach can be interpreted as query-focused summarization. specifically, we adopt primera @xcite , a state-of-the-art pre-trained encoderdecoder model for multi-document summarization. ## data there are two key data components in our task: (1) an external knowledge source from which we retrieve documents; and (2) paired claim-explanation data, to serve as the input (claim) and output (explanation). for the external knowledge source, we use climate science-related abstracts from pubmed and reports from the intergovernmental panel on climate change ("ipcc"). ipcc reports are written by a mix of scientists, experts, and policy makers and provide scientific, technical, and socio-economic knowledge on climate change and options to mitigate its impacts. we sample climate science-related publications using mesh descriptors. climate-fever @xcite contains 1,535 claims relating to climate change. see in our framework, the claim serves as the input for us to query the knowledge source to retrieve related documents, and the evidence constitutes the explanation that we want to generate as output. ## method in figure 1 , we present an overview of our method, which is made up of two components: (1) a document retriever; and (2) a generator. given a claim c i , the retriever retrieves k passages {p 1 , p 2 , ..., p k |c i } from the knowledge source, based on which the generator generates a veracity label y i along with explanation e i . 3 the generator is an encoder-decoder model which jointly processes the retrieved passages and claim in the form of an abstractive summarization model. we adopt binary passage retriever (bpr) (yamada et al., 2021) as the retriever. bpr is a memory efficient version of dense passage retriever @xcite . it first uses two independent bert @xcite encoders to encode question and passages into continuous embeddings and then incorporates a hashing layer to reduce computational cost for similarity calculation. bpr is trained with a multi-task objective over two tasks: effective candidate generation based on binary codes and accurate reranking based on continuous vectors. we use the official release of bpr 4 which was pre-trained on natural questions @xcite without fine-tuning, and consider each claim as the query to retrieve top-k relevant passages from our knowledge source. for the generator, we adopt primera @xcite to generate explanations, where the input is the claim concatenated with the top-k retrieved passages. primera is designed for multi-document summarization with entity pyramid masking, a novel pre-training strategy to select and aggregate salient information from multiple documents. primera uses longformer (beltagy et al., 2020) as its encoder, and replaces standard full self-attention with sparse self-attention, i.e. it features a combination of local attention (selfattention between tokens in a narrow context window) and global attention (selected tokens that attends to all other words). we structure the input by @xmath0, and concatenating them with the claim with @xmath1. moreover, we prepend claims and passages with the special @xmath2:> @xmath3:> tokens respectively (to provide explicit indication of their functions). by default, primera assigns global attention only @xmath4. we extend this idea by adding extra global attention to the claim words and the two special prefix tokens (<claim:> @xmath5:>). this is to better focus the model on the claim. we also perform veracity prediction by generating veracity labels together with explanations, following @xcite . that is, the output takes the form of veracity:[lab]\nexplanation: @xcite , where [lab] is the veracity label and [exp] is the generated explanation. ## experiments as our baseline, we compare against bhatia et al. (2021a) who use a retrieval-augmented generation framework to jointly perform veracity prediction and explanation generation using fusion in decoder @xcite and model it as question answering task. note that in their approach a claim is concatenated with each passage and these claim-passage pairs are encoded separately -so as to reduce the computational overhead due to full self-attention -before they are fed to the decoder. our approach, on the other hand, frames the task as query-focused multi-document summarization, and the use of primera means we can use the concatenated claim and all passages as input due claim: about 60% of the warming observed from 1970 to 2000 was very likely caused by the above natural 60-year climatic cycle during its warming phase. label: refutes gen: in the scientific literature, there is an overwhelming consensus that global surface temperatures have increased in recent decades and that the trend is caused mainly by human-induced emissions of greenhouse gases. ref: it is extremely likely (95-100% probability) that human influence was the dominant cause of global warming between 1951-2010. claim: that humans are causing the rise in atmospheric co2 is confirmed by multiple isotopic analyses. label: supports gen: human activity since the industrial revolution has increased the amount of greenhouse gases in the atmosphere, leading to increased radiative forcing from co2, methane, tropospheric ozone, cfcs, and nitrous oxide. ref: while co2 absorption and release is always happening as a result of natural processes, the recent rise in co2 levels in the atmosphere is known to be mainly due to human (anthropogenic) activity. to its sparse attention mechanism. to clarify, the main difference between our model and @xcite lies in the generator, as both models use bpr as the retriever. in terms of evaluation metrics we use rouge @xcite and bertscore @xcite for assessing generation quality, and accuracy for veracity prediction. ## conclusion in this work, we tackle the problem of claim veracity prediction and explanation generation in the domain of climate change. we use pubmed and ipcc reports as a knowledge source, and frame explanation generation as a query-focused summarization task and use primera as our generation model. quantitative and qualitative analyses demonstrate that our proposed model improves the quality of generated explanations, and that additional global attention on the claim tokens is helpful. a @xmath8: 97 % consensus on human -ca used global warming has been dispro @xmath9: since the mid -19 th century , human activities have increased greenhouse gases such as carbon dioxide , methane , and nit rous oxide in the earth 's atmosphere that resulted in increased average temperature . the effects of rising temperature include soil degradation , loss of productivity of agricultural land , desert ification , loss of biodiversity , degradation of ecosystems , reduced fresh -water resources , acid ification of the oceans , and the disruption and depletion of strat osp heric ozone . all these have an impact on human health , causing non -commun icable diseases such as injuries during natural disasters , malnutrition during famine , and increased mortality during heat waves due to complications in chronically ill patients . direct exposure @xmath10. 6 degrees c since 1975 , earth now appears to be warming due to a variety of clim atic effects , most notably the casc ading effects of greenhouse gas emissions resulting from human activities . there remains , however , no universal agreement on how rapidly , region ally , or asymm etr ically the planet will warm or on the true impact of global warming on natural disasters and public health outcomes . most reports to date of the public health impact of global warming have been anecdotal and retrospective in design and have focused on the increase in heat -stroke @xmath11. 6 degrees c during the 20 th century , but as z w iers and weaver discuss in their perspective , the warming was not continuous . two distinct periods of warming , from 1910 to 1945 and since 1976 , were separated by a period of very gradual cooling . the authors highlight the work by st ott et al ., who have performed the most comprehensive simulation of 20 th century climate to date . the agreement between observed and simulated temperature variations strongly suggests that forcing from anthrop ogenic activities , moder ated by variations in solar and volcanic forcing , has been the main driver @xmath12. compar isons of observations with simulations from an energy balance climate model indicate that as much as 41 to 64 % of pre anthrop ogenic ( pre -18 50 ) dec adal -scale temperature variations was due to changes in solar irrad iance and volcan ism . removal of the forced response from reconstructed temperature time series yields residual s that show similar variability to those of control runs of coupled models , thereby lending support to @xmath13. understanding this period and the subsequent slowdown of warming is key to dis ent angling the relationship between dec adal variability and the response to human influences in the present and future climate . this review discusses the observed changes during the e tc w and hypotheses for the underlying causes and mechanisms . attribution studies estimate that about a half ( 40 -54 %; @xmath14. 8 ) of the global warming from 1901 to 1950 @xmath15: visualization of attention weights on model input a.1 analyzing attention weights attention weights can provide insights into what the model focuses on during learning, and how it affects generation. we visualize attention strength on tokens in our model input in figure 2. darker shades indicate higher weights on corresponding words. we analyse the (summed) cross-attention weights on the input words at the final decoding step, and observe that our model tends to: (1) produce strong attention on the claim words @xmath16; and (2) focus on relevant words in the passages.