Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models (2024)

Abstract: The use of Natural Language Processing (NLP) for helping decision-makers with Climate Change action has recently been highlighted as a use case aligning with a broader drive towards NLP technologies for social good. In this context, Aspect-Based Summarization (ABS) systems that extract and summarize relevant information are particularly useful as they provide stakeholders with a convenient way of finding relevant information in expert-curated reports. In this work, we release a new dataset for ABS of Climate Change reports and we employ different Large Language Models (LLMs) and so-called Small Language Models (SLMs) to tackle this problem in an unsupervised way. Considering the problem at hand, we also show how SLMs are not significantly worse for the problem while leading to reduced carbon footprint; we do so by applying for the first time an existing framework considering both energy efficiency and task performance to the evaluation of zero-shot generative models for ABS. Overall, our results show that modern language models, both big and small, can effectively tackle ABS for Climate Change reports but more research is needed when we frame the problem as a Retrieval Augmented Generation (RAG) problem and our work and dataset will help foster efforts in this direction.

## introduction climate change reports are critical for policymakers and researchers in tackling climatic challenges and, as such, summarization of such reports is a task in line with recent work advocating for ways in which natural language processing (nlp) can help climate scientists and policy-makers and make a positive impact @xcite . when dealing with such information-dense documents, however, simple summarization might be too limiting, as the end user might need a summary with respect to a specific topic. summarizing a text with respect to a specific aspect or topic is known as aspect-based summarization (abs) and it has a long history in nlp @xcite . recently, the landscape of nlp has seen a revolution happening in the form of large language models (llms), which are capable of performing many tasks without training, therefore being particularly useful in under-resourced domains like the one of climate change reports @xcite . these models, however, comprise billions of parameters, and, as such, their carbon footprint is one of the main factors leading to criticisms of their use @xcite , while relatively smaller llms, labeled as small language models (slms), have started gaining traction in the literature @xcite . in this work, then, we show that llms and slms can be successfully applied to the task of abs in the context of climate change reports. the main questions informing our work are: q1: can llms and slms successfully perform abs of climate change reports and how do they compare to other unsupervised alternatives? q2: are slms comparable in performance to larger llms for our task? q3: how do our models' performance deteriorate in the absence of ground truth paragraphs to summarize? our main contributions then are: 1) we evaluate llms and slms in the context of abs for climate change reports with ground truth paragraphs and within a rag setting, and we introduce a new dataset for this task. ## data for the purpose of this work, we have collected and released the sumipcc dataset, comprising 140 topic-annotated summaries and relative paragraphs from climate change reports. we used two reports from the authoritative intergovernmental panel on climate change (ipcc) as a data source. the reports we used are the synthesis reports ar5 (ipcc, 2014b) and ar6 (ipcc, 2023b) for two separate years, 2014 and 2023, which collected the contributions of different working groups on a variety of topics related to climate change and linked policies. the two reports were chosen among the ipcc synthesis report collections as they both include accompanying publications named summary for policy-makers (ipcc, 2014a, 2023a), which include short summaries related to specific topics and referring to paragraphs in the respective synthesis reports. each summary includes the main highlights with regard to a specific topic as discussed in the report and it might refer to multiple paragraphs in the original report, in case the specific topic is treated in different parts of the report. on occasions, we observed summaries that were too broad in scope, referring to many different long paragraphs, but comprising just a few lines on a broad topic: we filtered out these cases. the final result is a dataset comprising 140 paragraphsummary pairs with associated topics, which we manually annotated to be as precise as possible. paragraphs and section headers from the summary for policy-makers could also have been used to annotate the summaries, but they were ambiguous as they grouped different summaries; they are also included as features in the dataset, even though we don't explore their use in this work. as we will see, however, there are a number of summaries sharing the same topic but in different contexts and future work might include additional information to better disambiguate these cases, especially in the rag context. ## experiments 6.1 slms evaluation when comparing slms with the llms baselines, we can observe some striking results in that the chatgpt baseline appears to be the bestperforming system overall, even more so than the superior gpt4 baseline. this apparently counterintuitive result can, however, be explained by three factors: first, as the metric we use is based on chatgpt itself it might show a bias in favor of the model, as observed in previous studies (panickssery et al., 2024), second, the reliability of the metric in the context of high-quality summaries is generally lower @xcite , and third, chatgpt is not significantly better than gpt4 in any evaluation dimension. these points also apply to most slms. more recent and relatively more powerful slms like llama 3, in fact, appear to be worse than other models like chatgpt itself, but ultimately the difference is statistically insignificant, rather indicating that most slms and llms perform similarly in our context. slms, then, can be as effective as larger llms for our task (q2). moreover, figure 1 shows how the energy consumption shows a weak, but relevant correlation with llms performance on each dimension. a key driver of this correlation is the poor performance of qwen 0.5b, suggesting that there is a threshold under which model size severely impacts the capacity of slms to perform this task. the updated ranking of models in figure 2 using the carburacy technique, however, shows how on certain occasions, notably that of qwen 1.8b, very small slms can perform similarly to larger ones. the re-ranking confirms once more that most slms perform similarly, and that are generally better than very small llms (qwen 0.5b) and then the extractive baseline. it follows, that chatgpt and gpt4 are actually the worst models when considering the efficiency/effectiveness trade-off because the increase in energy consumption is not justified by a relevant increase in the models' performance. ## conclusion and future directions in this work, we have investigated the use of llms and slms for abs in the context of climate change reports, showing how the task can be virtually solved by such models when considering ground truth paragraphs. apart from the task itself, which has a variety of uses in policy-making and education, our aim was also that of evaluating whether smaller, more efficient llms (i.e. slms) can lead to comparable results to bigger one in a task in which llms are extremely capable. the results indeed confirmed that slms are a valid alternative to bigger llms, especially in the easier scenario in which ground truth paragraphs were provided. when we turned to the rag scenario, instead, it could be seen that the task at hand became more challenging, while the difference in the models we used appeared to be more significant. also in this case, however, the smallest model performed comparably with the best-performing one and, even though this might be due to various things not reflecting a more general equivalence, the evidence suggests, at least, that smaller models can be a valid alternative also in more challenging cases. finally, we release our dataset and this can lead to many interesting research directions. specifically, future research could explore the rag setting further by incorporating more fine-grained information during retrieval (e.g. section and/or paragraph titles, which are included in the dataset) and finetune slms on the small available data to test the ability of such models to learn from small data. we leave these directions open for future research. 130 our work deals with the use of slms for abs and has shown that they often perform similarly to larger llms in our context. given the specific domain of application (i.e. climate change reports), however, we are limited to a small size dataset, which in turn increases results' variability. another limitation of our work involve the evaluation metric, which includes a number of problems such as having around 80% agreement with human judgement, as shown in appendix a: this value is relatively high for summarization metrics, but it is still low enough to represent a significant limitation in terms of how much we can trust the metric itself in certain cases. other evaluation limitations include the fact that our metric has been shown to correlate less with human judgement when dealing with high-performing systems (which is our case in the first experiment using ground truth paragraphs) and the already noticed fact that the metric appears to be biased towards certain llms (i.e. chatgpt). finally, there is initial evidence that the aspects we have evaluated for each sample in our dataset might be too broad. future research might consider using the additional features we provided in the released dataset in order to better define the aspect on which the summarization models should focus.