Predicting Narratives of Climate Obstruction in Social Media Advertising (2024)

Abstract: Social media advertising offers a platform for fossil fuel value chain companies and their agents to reinforce their narratives, often emphasizing economic, labor market, and energy security benefits to promote oil and gas policy and products. Whether such narratives can be detected automatically and the extent to which the cost of human annotation can be reduced is our research question. We introduce a task of classifying narratives into seven categories, based on existing definitions and data.Experiments showed that RoBERTa-large outperforms other methods, while GPT-4 Turbo can serve as a viable annotator for the task, thereby reducing human annotation costs. Our findings and insights provide guidance to automate climate-related ad analysis and lead to more scalable ad scrutiny.

## introduction advertising has allowed firms to construct narratives that align with their commercial interests and sway public discourse. this is true in the context of climate change, where strategies akin to tobacco industry propaganda are employed to shape public perception by redirecting responsibility away from corporations @xcite . in the domain of social media, entities, including fossil fuel corporations, utilize the platform to bolster existing beliefs about the significance of fossil fuels @xcite . for example, some advertisements (or ads) claim the indispensability of fossil fuels for jobs and the economy, and promote the idea that they are "clean." we refer to such narratives that obstruct progress against climate change as climate obstructive narratives. the scale of the public relations effort and advertising of climate obstructive narratives 1 ). entity information is blacked out. identifying ads that contain climate obstructive narratives poses significant challenges in terms of efficiency and scale. this task usually relies on human expertise from academics or non-profit organizations (npos), due to the unique nature of the domain and the nuanced presentation of the ads. for instance, to accurately label about 1,500 ads, an npo required the expertise of five subject matter experts @xcite , resulting in an estimated total of 120 hours spent. in this context, natural language processing (nlp) may potentially offer a viable alternative. this paper proposes a multi-label classification task with seven classes to identify climate obstructive narratives. our dataset was constructed based on the definitions and annotations of @xcite , which includes facebook ads by fossil fuel entities. an example of the 'patriotic energy mix' type is shown in figure 1 , where this type suggests that the production of domestic oil and gas reserves benefits the country. we utilize pre-trained language models, such as bert @xcite , and large language models (llms), such as gpt-4 turbo @xcite in our experiments. a comparable study on this task can be found in the work of @xcite , while our approach differs in several ways. we utilize a high-quality annotated and relatively large dataset, benchmark various baseline models on it, and provide insights toward scalable ad scrutiny. the experimental results show that robertalarge @xcite yields the best f-score, while gpt-4 turbo performs close to robertabase. we also found that when gpt-4 turbo is used to annotate training data for fine-tuning, only about 30 annotation examples are required to outperform roberta-large. given the need for experts to create the training data, the gpt-4 turbo utilization is attractive even when considering the tradeoff between prediction cost and performance. given that recent research indicates that social media platforms are not adequately addressing the dissemination of misleading information @xcite , it is important to scale scrutiny of social media ads efficiently. our study suggests that even with limited human resources, llms can be used to assist in monitoring climate obstructive ads. we release the code on github ( https://github.com/climatenlp/climate-obstruction-narratives ). ## background interdisciplinary study of climate change and nlp has gained attention in recent years. major attempts have been made in this area to detect claims related to climate change. for example, datasets and models have been proposed to detect environment claims @xcite or net-zero claims @xcite . @xcite address theme classification of facebook ads from fossil fuel entities. there are similarities to our study, although their work uses a smaller human-annotated dataset and does not use multi-label classification. the labels 'patriotism', 'pragmatism', 'economy_pro', and 'climatesolution' found within the work align with some labels identified by @xcite , though it appears that the respective studies were conducted independently. some labels proposed in the work of @xcite are more fine-grained, and the results described in this paper provide a more detailed discussion. technically, our study differs from the work of @xcite in that we benchmark various baseline models and few-shot learning on larger datasets based on high-quality annotations. @xcite use sentence-bert @xcite to classify ads and achieved an accuracy of 38.4%. we include different finetuned models and llms in our experiments and best models achieve f-scores around 70%. we also use our experimental results to discuss guidelines for automated ad scrutiny. in summary, our study provides more reliable and detailed discussions compared to the work of @xcite . from the view of technical classification of nlp, our work could be contextualized within climate change debate analysis (stede and patz, 2021), argument mining @xcite , discourse analysis, and persuasion and propaganda technique analysis. @xcite analyzed the global warming controversy using bert, and there is similarity with our study in that it examines stances on climate change issues. in the propaganda typology @xcite , 'flag-waving' is somewhat sim-ilar to 'patriotic energy mix' in our dataset. our focus is, however, on the specific domain of the oil and gas sector. although the oil and gas sector may seem limiting, it is important to recognise that the domain is deceptively large, extending well beyond the primary operations of oil and gas companies to an extensive value chain. this value chain comprises refiners, pipeline operators, and manufacturers of secondary products @xcite , all of which play integral roles in the industrial overall impact on both the economy and the environment. furthermore, the industrial efforts to shape public discourse and policy are amplified through a network of agents, including political action committees (pacs), trade associations, and lobbyists @xcite . our study is focused on ads that are affected by these value chains. research in other domains, such as health and politics, has also utilized nlp methods to detect misinformation @xcite . for instance, nlp methods have been applied to identify false statements and health-related misinformation @xcite , demonstrating the versatility of these techniques across different areas of study and highlighting the critical role of accurate information in maintaining public trust and safety (hirlekar and kumar, 2020). ## the climate obstruction data we built a dataset tailored for text classification based on the original data of @xcite . the original dataset was compiled to include ads related to climate change that were run in the united states between january 1, 2020, and january 1, 2021 by utilizing the facebook ad library api. the restriction to a limited timeframe in our study serves an important purpose, allowing us to target and analyze specific patterns in climate disinformation during a defined period. given the rapid evolution of trends within this sphere, label definitions may need to be reconsidered as the dataset is expanded to a broader timeframe. the dataset primarily focused on the top ten fossil fuel companies, the top five industry associations representing the oil and gas sector, and ten advocacy groups with significant spending and connections to the fossil fuel industry. typology: ## experiments because each ad can be associated with multiple labels, we define our task as multi-label classification for the text included in the ads. conceptually, given input ad text x , the output is a set of labels y ⊂ {ca, cb, ga, gc, p a, p b, sa}. an empty set is allowed for text that does not correspond to any of the labels. although ads can contain images and videos, this study does not consider them. we use standard f-scores to evaluate the classification performance. models: we are motivated to compare different models that are simple and well-known, but strong baselines. to this end, we use a conventional approach using pre-trained language model finetuning and llms with prompting. bert @xcite and roberta 2022 )) prompting (gpt3.5-trb (cot)). we use dspy @xcite and vllm @xcite to implement the llm experiments. we also investigate conventional automated training data labeling with @xcite . we use gpt4-trb to label our training data, resulting in 'silver' training data. then, we finetune roberta-large on this data, referring to this model as roberta-gpt4-trb-label (rogl). we investigate a low-resource scenario by fine-tuning rogl on sampled human labeled training data. for implementation and hyperparameter details, refer to appendix a.2. ## discussion which method is reliable in replicating the nuanced understanding of the ads? as we showed, fine-tuned roberta-large performed best. there is no simple way to compare; however, given the inter-annotator agreement score of the dataset is 0.78, we can see that the classification performance of roberta-large is close to expert performance. however, we found that roberta runs the risk of overfitting the training data; gpt4-trb can produce more intuitive output but our prompts cannot reproduce the subtle nuances contained in the biases of the annotation. if the test data domain does not change significantly, roberta seems to be the with no associated labels. we refer to such cases as 'no label'; roberta is trained with binary cross-entropy loss and assigns 'no label' only if the output probability of all labels are less than 0.5. gpt-4 trb is also supported to output 'x' (the same as 'no label') if there is no label to assign. which method is better for practical applications? our methods are useful for identifying trends in corporate advertising and detecting their stance on climate policy. on the other hand, the appropriate method can be selected depending on the use case. if there is sufficient training data, roberta is effective, otherwise gpt4-trb can be used. gpt4-trb would be better suited for smallscale analyses because of throughput and cost issues, making it difficult to process the analysis efficiently. if one can provide training data of about 30 samples, rogl is accurate with higher throughput. we believe that rogl is sufficiently practical for ad analysis. ## conclusion research indicates that social media platforms are not adequately addressing the dissemination of misleading information @xcite . while we acknowledge the limitations, our study introduced methods that are both time-efficient and scalable for analyzing social media ads, offering a valuable way for npos and academic researchers aiming to undertake extensive evaluations. importantly, the applicability of this approach extends beyond the environmental sector, holding promise for other areas impacted by disinformation, including natural ecosystems, biodiversity, and food security. in future studies, we will assess the accuracy of these methods against more recent ads which may display new climate narrative trends.