C li ME : Evaluating Multimodal Climate Discourse on Social Media and the Climate Alignment Quotient ( CAQ ) (2025)

Abstract: The rise of Large Language Models (LLMs) has raised questions about their ability to understand climate-related contexts. Though climate change dominates social media, analyzing its multimodal expressions is understudied, and current tools have failed to determine whether LLMs amplify credible solutions or spread unsubstantiated claims. To address this, we introduce CliME (Climate Change Multimodal Evaluation), a first-of-its-kind multimodal dataset, comprising 2579 Twitter and Reddit posts. The benchmark features a diverse collection of humorous memes and skeptical posts, capturing how these formats distill complex issues into viral narratives that shape public opinion and policy discussions. To systematically evaluate LLM performance, we present the Climate Alignment Quotient (CAQ), a novel metric comprising five distinct dimensions: Articulation, Evidence, Resonance, Transition, and Specificity. Additionally, we propose three analytical lenses: Actionability, Criticality, and Justice, to guide the assessment of LLM-generated climate discourse using CAQ. Our findings, based on the CAQ metric, indicate that while most evaluated LLMs perform relatively well in Criticality and Justice, they consistently underperform on the Actionability axis. Among the models evaluated, Claude 3.7 Sonnet achieves the highest overall performance. We publicly release our code and dataset to foster further research in this domain.

## introduction climate change has emerged as one of the most pressing challenges of our time, not only in scientific and policy circles but also in the public imagination @xcite @xcite . @xcite found that youtube had the * these authors contributed equally to this work. text: it hits so hard. description: the image and caption convey a powerful message about the urgency of addressing climate change. the visual metaphor of a train colliding with a bus represents the collision between efforts to curb climate change and the reality of the climate crisis. … the caption "it hits so hard" emphasizes the impact of the climate crisis, … most significant positive effect on climate awareness in latin america, followed by instagram and twitter, demonstrating the role of these platforms in disseminating climate-related information . social media platforms like twitter (now x) and reddit have emerged as prime spaces for climate discourse, shaping public opinion, mobilizing activism, and influencing policy. viral campaigns such as #fridaysforfuture, #showyourstripes and #earthhour illustrate the power of social networks in transforming grassroots efforts into global movements. however, these online platforms propagate misinformation and polarization, vastly associated with skepticism, contrarianism, and denial @xcite . in 2023 alone, posts denying climate change on x tripled compared to previous years, highlighting the persistence of climate denial narratives online @xcite . the rise of multimodal content in this domain further complicates this landscape. a flooded city image paired with "climate policies harm the economy" spreads faster than peer-reviewed data, exploiting visual-emotional resonance (o'neill, 2020). the rising hype of llms show their outstanding capabilities at text generation but their proficiency in grounding responses in visual-textual context in climate communication, remains unproven. early studies reveal concerning trends: when prompted with climate-skeptical posts, models like gpt-4 often default to vague, non-committal language. for instance, a study by newsguard found that gpt-4 @xcite was more likely than its predecessor gpt-3.5 @xcite to generate misinformation when prompted with false narratives, including those about climate change, without sufficient disclaimers or corrections @xcite . this mirrors the "false balance" tactic often used to undermine scientific consensus, where both sides of an issue are presented as equally valid despite overwhelming evidence favoring one side. in addition, automated fact-checking systems for climate-related content, such as climate-fever @xcite face challenges while evaluating complex claims. to solve this, recent llm based frameworks like climinator @xcite integrates multiple scientific viewpoints in the verification process. however, these approaches often assumes that llms contain comprehensive understanding of climate science from all relevant perspectives. as a result, there remains a gap in evaluating how well llms can assess climate-related contents and generate informed suggestions alongside their factchecking verdicts. to address these challenges and assess whether llms can interpret and generate credible climate communication, we introduce clime, a benchmark comprising 2579 pairs of posts scraped from reddit and twitter, with most of the data focused on memes, infographics, and skeptic content (see fig. 1 ). we then generate descriptions of these images and texts using deepseek janus pro @xcite , as it has demonstrated proven capability to understand memes and skeptic contents @xcite , followed by human annotation (see section 3.2). these descriptors serve as the basis for evaluating llms' capacity to comprehend and address climate change, guided by our proposed three key lenses: (i) actionability, (ii) criticality, and (iii) justice (see section 4.1). to augment the existing climate-fact-checking pipelines, with the measurement of the llm responses generated through these lenses, we introduce the climate alignment quotient (caq), a novel metric that quantifies gaps across five critical axes: (i) articulation, (ii) evidence, (iii) resonance, (iv) transition, and (v) specificity (see section 4.2), thereby determining the extent to which llms capture intrinsic climate knowledge. figure 2 illustrates the entire workflow. in summary, our contributions are as follows: ## clime dataset to understand how large language models (llms) engage with and comprehend climate discourse, specifically in addressing climate change, we introduce the first-of-its-kind multimodal benchmark, clime * (climate change multimodal evaluation) dataset. comprising 2,579 data points, sourced and filtered from real-world twitter and reddit posts, primarily memes, skeptics, and infographics, clime shifts the focus toward evaluating models based on their ability to generate credible, actionable, and equitable climate-related communication with multimodal (image-text) contexts. ## axioms 5 dimensions cohen kappa: 92% ## conclusion in this work, we present clime, a novel multimodal dataset curated from social media, primarily featuring memes and skeptics, and introduce the climate alignment quotient (caq) as a comprehensive metric to evaluate climate discourse generated by llms. by benchmarking five state-of-theart models, we analyzed their outputs across three lenses: actionability, criticality, and justice. our experiments reveal that while all models consistently capture climate-related context, significant variations exist in generating actionable and critically evaluative messages. the gap analysis uncovers subtle imbalances among these dimensions, indicating areas for improvement in the models' outputs. in today's landscape, misinformation and polarized narratives on climate change contribute to social harm and undermine effective action; our caq framework offers a promising approach to understanding the strengths and weaknesses of llms. by fostering balanced climate communication, our work aims to prevent the spread of misleading information and support informed public dialogue, ultimately guiding policy-making for social good. future efforts will focus on refining the caq metric and expanding the clime dataset to include a broader spectrum of multimodal content, thereby empowering both llms and vlms to contribute more effectively to a sustainable future.