Post-hoc Study of Climate Microtargeting on Social Media Ads with LLM s: Thematic Insights and Fairness Evaluation (2025)

Abstract: Climate change communication on social media increasingly employs microtargeting strategies to effectively reach and influence specific demographic groups. This study presents a *post-hoc* analysis of microtargeting practices within climate campaigns by leveraging large language models (LLMs) to examine Meta (previously known as Facebook) advertisements. Our analysis focuses on two key aspects: **demographic targeting** and **fairness**. We evaluate the ability of LLMs to accurately predict the intended demographic targets, such as gender and age group. Furthermore, we instruct the LLMs to generate explanations for their classifications, providing transparent reasoning behind each decision. These explanations reveal the specific thematic elements used to engage different demographic segments, highlighting distinct strategies tailored to various audiences. Our findings show that ***young adults*** are primarily targeted through messages emphasizing *activism and environmental consciousness*, while **women** are engaged through themes related to *caregiving roles and social advocacy*. Additionally, we conduct a comprehensive fairness analysis to uncover biases in model predictions. We assess disparities in accuracy and error rates across demographic groups using established fairness metrics such as Demographic Parity, Equal Opportunity, and Predictive Equality. Our findings indicate that while LLMs perform well overall, certain biases exist, particularly in the classification of **male** audiences. The analysis of thematic explanations uncovers recurring patterns in messaging strategies tailored to various demographic groups, while the fairness analysis underscores the need for more inclusive targeting methods. This study provides a valuable framework for future research aimed at enhancing transparency, accountability, and inclusivity in social media-driven climate campaigns.

## introduction climate change represents one of the most pressing global challenges of the 21st century, necessitating widespread public awareness and engagement to drive meaningful environmental action @xcite . as traditional media channels evolve, social media has emerged as a pivotal platform for climate communication, enabling organizations, activists, and policymakers to disseminate information, mobilize support, and influence public discourse on environmental issues @xcite @xcite . the interactive and targeted nature of social media advertising allows for the customization of messages to resonate with specific audiences, thereby enhancing the effectiveness of communication strategies aimed at fostering climate awareness and behavioral change @xcite @xcite . in recent years, the utilization of microtargeting strategies in social media campaigns has gained significant traction @xcite @xcite . microtargeting involves the precise tailoring of messages to distinct demographic segments based on factors such as age, gender, location, and interests @xcite @xcite . this approach leverages vast amounts of user data to craft personalized content that is more likely to engage and persuade individual users. in the context of climate communication, microtargeting can enhance the relevance and impact of messages, potentially leading to greater public engagement and support for environmental initiatives. however, the sophistication of these strategies also raises critical questions about the transparency, ethical implications, and overall effectiveness of targeted climate messaging @xcite . figure 1 illustrates the targeted climate advertisements on social media, with a specific focus on demographic targeting. the first ad source targets young adults aged 18-24, with a message encouraging action against climate change and inviting them to become the climate initiative (tci) 1 ) ambassadors. the second ad source is tailored to a female audience with a message emphasizing the importance of reducing carbon pollution from home electricity and making a pledge for their children's future. the third ad source targets a male audience, focusing on the benefits of using clean fuel like ohio propane for heating rural homes and fueling appliances. despite the growing prevalence of microtargeting in climate campaigns, there remains a limited understanding of the specific techniques and linguistic patterns employed to engage different demographic groups. traditional method @xcite @xcite @xcite often falls short in capturing the nuanced and context-dependent nature of targeted communication. this gap highlights the need for advanced analytical tools that can dissect and interpret the complex language and strategies used in microtargeted climate advertisements. large language models (llms) @xcite , with their robust natural language processing (nlp) capabilities, offer a promising solution to this challenge. in this paper, we investigate whether the newly emerging paradigm in nlp-zero-shot prompting of llms @xcite and the practice of providing explanations of answers are better equipped to address those challenges. explanations are fundamental to human learning @xcite , as they underscore task principles that facilitate broad generalizations @xcite . consider the example text ( "the turbines that provide clean wind energy also create new habitats for fish, keeping the ecosystem healthy." ) from figure 2 (a) for the gender identifica-1 https://theclimateinitiative.org/ tion task. an explanation can elaborate on a brief answer (e.g., male) by connecting it to the broader reasoning process necessary to solve the problem (e.g., "emphasizing fish habitats may appeal more to men interested in fishing and environmental conservation."). thus, explanations enhance understanding by demonstrating how task principles connect questions to their answers. while llms offer powerful capabilities for analyzing and generating text, their widespread use has also highlighted significant challenges related to fairness @xcite @xcite and bias @xcite @xcite @xcite . research has shown that llms, like other machine learning (ml) models, can inherit and even amplify biases present in the data on which they are trained @xcite . these biases can manifest in various forms, such as differential accuracy across demographic groups @xcite @xcite , harmful stereotypes @xcite @xcite , and discriminatory language patterns @xcite . the implications are particularly concerning in high-stakes applications such as healthcare, finance, and social networks, where biased outcomes can perpetuate inequities and undermine trust in artificial intelligence (ai) systems @xcite . addressing these issues requires fairness evaluations using metrics like demographic parity, equal opportunity, and predictive equality to ensure that models perform equitably across all user groups @xcite . in the context of this study, examining the fairness of llms' predictions in demographic targeting is crucial for understanding the microtargeted climate messaging. in this study, we conduct a post-hoc analysis of climate microtargeting practices on social media by leveraging the power of llms (openai's o1preview model foot_0 ). post-hoc analysis, typically performed after the main experiment or event, allows us to retrospectively evaluate how effective these campaigns are in targeting specific demographics. building upon data from previous research by islam et al. (2023b); islam and goldwasser (2024a), we investigate the ability of llms to accurately detect targeted messaging based on specific demographic variables, including gender and age group. additionally, llms provide explanations for their classification decisions, offering insights into the thematic and linguistic elements used to engage different audiences. furthermore, we conduct a comprehensive fairness analysis to identify potential biases in model predictions. our exploration leads to the following research questions (rq), which are crucial for assessing the potential of llms to understand microtargeting patterns and provide deeper and more nuanced insights: the main idea of this paper is to utilize llms as a tool to analyze real-world targeting practices, measuring biases and fairness in actual campaigns. we explore how llms can be used post-hoc as third-party tools to analyze patterns in targeted communication, especially when internal platform logic is not transparent. while platforms have white-box access, external stakeholders (researchers, auditors, policymakers) do not. our method offers an explainable approach to reverseengineer targeting practices and uncover potential bias or messaging disparities. the implications of this research are multifaceted. from an academic perspective, it contributes to the burgeoning field of computational social science (css) by showcasing the application of advanced language models in dissecting complex communication strategies. practically, the findings offer valuable insights for policymakers, environmental organizations, and social media platforms seeking to enhance the transparency, accountability, and inclusivity of their climate communication efforts. by illuminating the specific methods used to tailor messages to different demographics and by highlighting the need for fairer and more inclusive targeting methods, this study lays the groundwork for future investigations into the role of ai in enhancing the efficacy and ethical standards of digital climate advocacy. ## related work the intersection of microtargeting, social media, and climate communication has drawn considerable interest in recent years, with studies examin-ing how digital platforms influence public opinion and engagement @xcite @xcite . the growing capabilities of social media platforms to deliver personalized messages based on user data have sparked a significant body of research into computational advertising @xcite @xcite , demographic targeting @xcite @xcite @xcite , and the broader implications of these practices for societal discourse. microtargeting on social media platforms such as meta has been widely studied in the context of political campaigns @xcite @xcite @xcite and public health messaging @xcite @xcite , revealing both the potential benefits and ethical concerns associated with this practice. in the realm of climate communication, microtargeting can be a powerful tool for enhancing message relevance and impact by tailoring content to the specific values and interests of diverse demographic groups @xcite . however, the effectiveness and ethical implications of such targeted messaging remain underexplored, particularly in terms of how different demographic groups engage and whether biases are present in targeting strategies. recent advances in llms have demonstrated their capability for in-context learning (icl), significantly enhancing their ability to perform tasks traditionally handled by humans @xcite @xcite @xcite @xcite @xcite . this progress suggests a strong potential for effectively applying llms to our specific task. in the realms of qualitative research (qr) and nlp, innovative methods are being explored to integrate llms into thematic analysis (ta) @xcite . researchers have proposed various frameworks, including an llms-inthe-loop approach @xcite , integrating gpt-3 with expertdesigned codebooks @xcite , and developing collaborative interfaces that utilize llms for code generation and support in decision-making processes @xcite . other recent work has shown that llms can benefit from examples that decompose the reasoning process (can be seen as an explanation), leading to an answer @xcite . despite the impressive capabilities of llms, there are concerns about fairness, accountability, and transparency in llms' predictions, which have been highlighted in recent literature @xcite @xcite @xcite , emphasizing the need for a rigorous evaluation of biases and disparities in model performance across different groups. in this paper, we leverage llms to identify targeted demographics and provide explanations of demographic targeting regarding climate-related advertisements on meta. besides, we develop a new set of themes and aspects based on those explanations, specifically tailored for analyzing messaging. additionally, we extend previous research by not only focusing on the accuracy of these predictions but also conducting a comprehensive fairness analysis to identify and address potential biases in the model's performance. ## dataset we investigate the climate campaigns case study for this work. we work on the corpus of 21372 ads @xcite originally released by @xcite . this dataset includes climate-related english ads on meta from the us, spanning from january 2021 to january 2022. each ad includes the following attributes: ad id, title, ad description, ad body, funding entity, spend, impressions, and distribution of impressions broken down by gender (male, female, unknown), age (seven groups), and location down to the state level in the usa. additional details about the dataset can be found in the original publications. for this work, we consider two demographic indicators, i.e., gender and age group. we consider two gender categories, i.e., male vs female. regarding age group, we consider four age group categories, i.e., young adults whose age range is 18-24, early working age group (25-44), late working age group (45-64), and senior citizens (65+). however, the meta ad library api doesn't provide explicit targeting information beyond ad impressions (views). note that this paper infers targeting from exclusive impression distributions, which may differ from advertiser intent. in this work, we focus only on ads viewed exclusively by either males or females; ads viewed by both genders were excluded. therefore, if an ad is viewed solely by males or solely by females, we consider it a targeted ad. the same logic is applied to age groups. we do not include any ads that overlap across age categories, such as young adults, early workingage individuals, late working-age individuals, and senior citizens. by following this methodology, we narrow down the dataset to 227 ads. among them, 59 ads target only females and 47 ads target only males. however, we find 25 ads target only young adults, 82 ads target only the early working age group, 8 ads target only the late working age group, and 6 ads target only senior citizens. ## experimental setup the identification of the targeted demographic (with explanation) in a text involves the following steps: gender prediction with explanation: for a given text t, the task involves identifying the targeted gender and explaining the rationale behind its selection. age group prediction with explanation: subsequently, the task requires predicting the targeted age group and providing an explanation for the specific choice. to identify the targeted demographics, we use the most recently foot_1 released by openai, o1-preview model foot_2 . this is a new, large language model that uses reinforcement learning (rl) and chain of thought (cot) techniques for complex reasoning, allowing it to think through a detailed internal process before responding to users. we use the openai playground api to run o1-preview by keeping the default parameters. the prompt template for the demographic prediction task with an explanation using llms can be found in fig. 3 in app. a.1. fig. 2 shows the example prompts for gender prediction. fig. 4 in app. a.1 shows the example prompts for age group prediction from the climate campaign dataset. ## thematic insights of explanations as llms provide explanations to provide the reasoning behind their prediction, we use those explanations to understand thematic insights to answer rq2. in this analysis, we only include the correct predictions and their explanations. using only correct predictions is not arbitrary-it is supported by methodological norms @xcite @xcite to ensure pattern coherence. ## fairness and bias analysis in this section, we present a comprehensive fairness analysis of the model for gender and age group classifications to answer rq3. we evaluate the models using established fairness metrics such as 8 https://maartengr.github.io/bertopic/#common demographic parity, equal opportunity, and predictive equality to assess their performance across different groups. by analyzing confusion matrices and classification reports, we identify any disparities in prediction accuracy and error rates between groups. our analysis aims to identify biases and investigate the underlying reasons for any observed biases. the insights gained from this analysis are critical for guiding future research in developing fair and inclusive algorithms. ## conclusion this study shows that llms can function as practical, third-party auditors of microtargeted climate ads-identifying whom messages are meant to reach, explaining why, and surfacing equity concerns that are otherwise opaque. on our climate-ad corpus, the llms accurately infer demographic targets overall (88.55% accuracy), with especially strong performance for gender (94.92% for femaletargeted ads; 85.10% for male-targeted ads) and more variable results across age groups. beyond prediction, model-generated explanations enable a compact, reusable taxonomy of themes used to engage different audiences, providing interpretable signals that researchers and practitioners can inspect and act on. besides, the fairness analysis conducted in our study underscores the importance of evaluating and addressing the biases. disparities in prediction accuracy and error rates highlight the need for more inclusive and equitable targeting methods. although we show our analysis on the climate campaigns dataset, our approach can easily be adapted to any dataset. it is designed to be scalable without any modifications. code and dataset are available foot_6 .