The Accuracy, Robustness, and Readability of LLM -Generated Sustainability-Related Word Definitions (2025)

Abstract: A common language with shared standard definitions is essential for effective climate conversations. However, there is concern that LLMs may misrepresent and/or diversify climate-related terms. We compare 305 official IPCC glossary definitions with those generated by OpenAI’s GPT-4o-mini and investigate their adherence, robustness, and readability using a combination of SBERT sentence embeddings and statistical measures. The LLM definitions received average adherence and robustness scores of 0.58 \pm 0.15 and 0.96 \pm 0.02 , respectively. Both sustainability-related terminologies remain challenging to read, with model-generated definitions varying mainly among words with multiple or ambiguous definitions. Thus, the results highlight the potential of LLMs to support environmental discourse while emphasizing the need to align model outputs with established terminology for clarity and consistency.

## introduction large language models (llms) have proven effective in a range of tasks, such as analyzing climate-related texts @xcite and explaining sustainability reports @xcite . however, as citizens and politicians turn to llms for information and inspiration, there is concern that these probabilistic models fail to consistently convey the specificity and accuracy required to discuss climate change. for example, agreeing to a standard set of definitions is essential to achieve common ground in the climate @xcite . however, streamlining language around climate is already challenging. for instance, julian @xcite showed that among 114 different definitions for "circular economy," most failed to convey all nuances of the concept. thus, this can lead to inconsistencies in research and policy-making. to address this issue, the interdisciplinary panel on climate change (ipcc) and the united nations (un) maintain the online glossaries ipcc glossary @xcite , and unterm (un, 2024a). although llms have access to these repositories during training, they are not constrained to them during inference. therefore, llms could further diversify and confuse these terms. as more people rely on llms, it is of special interest to study how llm-generated explanations adhere to the official definitions, how robust the completions are, and what lessons we should keep in mind when using these models at ever higher levels of climate discourse. motivated by this, we analyze the adherence, robustness, and readability of word definitions generated by one closed-source and two open-source models compared to official ipcc definitions. studies show that sustainability literature can be complex to read @xcite . this complexity challenges the accessibility and transparency of sustainability debates and reporting. studies spanning the sustainability to medical domains use llms to simplify these texts and make them interactive @xcite . ## methodology we present a framework for assessing the adherence and robustness of llm sustainability word definitions. specifically, given a term, we let an llm generate five definitions for each of the five prompt templates (25 completions per term). then, we use sbert sentence embeddings to compute the sentence similarity between the official and generated definitions (adherence), as well as the similarity between the generated definitions for a given term and prompt template (robustness). thus, we define adherence and robustness for each term as follows: ## discussion the adherence scores suggest that all llms generally capture the core semantic meanings of official definitions. intriguingly, all llms achieved similar average adherence scores and had many common outlier terms. this similarity may be due to the models being trained using similar methods and on roughly the same training data. notably, the adherence score did not significantly improve when we explicitly prompted the model for ipcc definitions. these results imply that providing a climate context may not automatically align language models for a given terminology group. the models do not have perfect recall of definitions; instead, they operate based on probability distributions. therefore, it is advisable to include the exact definitions in the prompts or llm systems to ensure they are readily available for reference. regarding robustness, the five prompt templates tested did not result in significant variations in generated model definitions. however, there was a notable variability among several terms. as anticipated, the terms with lower robustness scores tend to have multiple meanings, such as "projection", "exposure", and "equity". for instance, "equity" displayed many definitions, reflecting its complex and multi-faceted meanings. this ambiguity aligns with discussions in recent sustainability reports, such as the un's 2024 emissions gap report, which dedicates an entire section to discuss different equity models @xcite . thus, the robustness score can help target terms needing further standardization. however, we must also note that robustness is very dependent on the temperature settings of the models. in this paper, we use the default temperature for the models. however, model parameters play a significant role in the consistency and variability of model outputs. these variations could impact how the model presents terms to different users across time. in terms of readability, both the ipcc and model definitions scored poorly across both readability metrics. this finding is consistent with previous studies, which suggest that sustainability texts are inaccessible to most readers. notably, all model completions consistently received more complex readability scores than the already ## conclusion this study provides a comprehensive framework for assessing the adherence, robustness, and readability of llm-generated definitions of sustainability terms compared to official glossaries. while the llms capture the semantic meaning of most terms, there is significant variation, particu-larly for terms with multiple meanings or ambiguous definitions. in addition, ipcc and model definitions show low readability, highlighting the need for further work to simplify sustainability-related language without sacrificing accuracy. moreover, the case studies show the difficulty in retrieving official definitions even using explicit prompting, indicating the need to include official definitions directly in the prompt. these findings highlight the potential of llms to support the environmental conversation but also underscore the importance of carefully aligning model outputs with established terminology to ensure clarity and consistency.