A Few Hypocrites: Few-Shot Learning and Subtype Definitions for Detecting Hypocrisy Accusations in Online Climate Change Debates (2024)

Abstract: The climate crisis is a salient issue in online discussions, and hypocrisy accusations are a central rhetorical element in these debates. However, for large-scale text analysis, hypocrisy accusation detection is an understudied tool, most often defined as a smaller subtask of fallacious argument detection. In this paper, we define hypocrisy accusation detection as an independent task in NLP, and identify different relevant subtypes of hypocrisy accusations. Our Climate Hypocrisy Accusation Corpus (CHAC) consists of 420 Reddit climate debate comments, expert-annotated into two different types of hypocrisy accusations: personal versus political hypocrisy. We evaluate few-shot in-context learning with 6 shots and 3 instruction-tuned Large Language Models (LLMs) for detecting hypocrisy accusations in this dataset. Results indicate that the GPT-4o and Llama-3 models in particular show promise in detecting hypocrisy accusations (F1 reaching 0.68, while previous work shows F1 of 0.44). However, context matters for a complex semantic concept such as hypocrisy accusations, and we find models struggle especially at identifying political hypocrisy accusations compared to personal moral hypocrisy. Our study contributes new insights in hypocrisy detection and climate change discourse, and is a stepping stone for large-scale analysis of hypocrisy accusation in online climate debates.

## introduction perhaps no accusation is more commonly lobbied in political discourse as that of hypocrisy @xcite . allegations of hypocrisy, defined by the oxford english dictionary as the assumption "of a false appearance of virtue or goodness" @xcite and understood in practice as an incongruity between behavior and publicly expressed beliefs @xcite , are so ubiquitous as to lead hannah arendt to describe politics itself as a "never ending fight to ferret out hypocrites" @xcite . making such an accusation of one's rival is effective, as hypocrisy is widely and deeply loathed, with research showing that perceived hypocrisy negatively affects voters' opinions of politicians above and beyond other underlying scandals @xcite @xcite . it may also seem, in a polarized political landscape, like the only rhetorical tool available. when political opponents lack shared standards, moral persuasion becomes near-impossible, and what is left is undermining one's opponent with "the revelation that. . . [he or she] is not living up to his own professed ideal" @xcite . while discourse on political hypocrisy dates centuries back, it has only intensified online, buoyed by social media's amplification of polarization @xcite and valorization of authenticity @xcite . an arena of online discourse in which the hypocrisy charge is especially pertinent is that of debates around climate change, where hypocrisy accusations have been shown to be central to increasingly polarized, cross-ideological online interactions @xcite . an example of such an online hypocrisy accusation is a reddit commenter describing cop26, an international political climate conference in 2020, as "the biggest hypocrisy in the world" [because the politicians are] "arriving in their private jets." hypocrisy accusations have been found to drive polarization in online climate debates @xcite , facilitating not cross-camp deliberation but rather segregation into opposing ideological camps @xcite and affective polarization @xcite . being able to analyze (online) debates about climate policy, focusing on hypocrisy accusations can help us better measure the use of hypocrisy in such debates. detecting accusations of hypocrisy is understudied in computational text analysis and natural language processing (nlp). usually, hypocrisy is only included as part of broader logical fallacy detection tasks, as in @xcite . however, hypocrisy accusations are a distinct phenomenon: linguistically, they often contain contrastive conjunctions between clauses to highlight inconsistencies. semantically, their identification relies on context to understand contradictions. earlier work (see section 2) reports good performance on detecting logical fallacies in general, but it is noticeable in the literature that hypocrisy accusation are a difficult and sometimes neglected phenomenon in state-of-the-art nlp. @xcite report relatively low performance of 0.43 in macro-f1 for detecting whataboutism, a hypocrisyrelated fallacy -and only 0.21 in macro-f1 for all fallacies in climate debates. recently, @xcite report a performance of roberta-xlm multilingual detection of the whataboutism concept with an f1 of 0.06. however, this concept consists of only 0.05% of their fallacy dataset. due to a lack of attention to the specific hypocrisy accusation construct, there is a scarcity of data annotated for hypocrisy detection as well as little research of nuances in this construct, such as different types of hypocrisy accusations. yet hypocrisy allegations are varied, and we believe an account of different kinds will afford a better understanding of the online debate and the performance of models detecting such accusations. our contributions to the literature are: (1) we, unlike previous work, analyze hypocrisy accusation detection as an individual and nuanced task particularly relevant for analyzing online climate discourse; this paper is as organized as follows: section 2 describes previous research on measuring social science constructs using llms, including hypocrisy detection. section 3 introduces our dataset: the different types of hypocrisy, our annotation process, and corpus statistics. we describe our experiments to evaluate the capabilities of different instruction-tuned models on this dataset in section 4, and present results of these experiments in section 5. we then reflect on our results and the complexity of our task (section 6), and conclude our work in section 7. ## background large language models (llms) are models trained on predicting sequences of text. these models are trained in human preferences and instructions, and can perform in-context learning: with a natural language prompt or instruction, these models are able to do tasks such as classifying text examples on a new construct @xcite . ## data we present a dataset based on the english-language reddit european sustainability initiatives corpus released by @xcite . this corpus consists of 2,073 sustainability discussions from between 2017 and 2022 on the reddit.com subcommunities @xcite called europe, europeanunion, and europes, with 46,285 comments. nearly half (922) of these discussions have at least one comment. we focus on the comments in this dataset, as they constitute active discourse between users on the identified discussion topics, which are relevant for hypocrisy accusations. this means that our unit of analysis is a comment, which can contain a single or multiple sentences. ## experimental approach our experiments are a first attempt at using the climate hypocrisy accusations corpus to measure the capabilities of different currently popular and high-performing llms in detecting hypocrisy accusations. we use few-shot prompting, also known as in-context learning. @xcite . previous research has shown that for complex social constructs, few-shot out performs zero-shot (alhindi et al., 2023). ## discussion our results provide more insight into the complexities of the hypocrisy accusation as a construct. specifically, we find different model strengths (see section 6.1) and specifics of different types of hypocrisy accusations (section 6.2). ## conclusions hypocrisy accusations are central to increasingly polarized, cross-ideological online interactions. despite recent research on detecting argument fallacies, hypocrisy accusations remain underresearched and are often a small sub-class in argument fallacy datasets. we define hypocrisy accusation detection as an individual nlp task and create an annotation scheme where we identify subclasses of hypocrisy accusation: personal moral hypocrisy versus political hypocrisy. we present a dataset, the climate hypocrisy accusation corpus (chac), consisting of 420 reddit comments, annotated by six experts. using our dataset, we compare three different instruction-tuned models (gpt-4o , gpt-3.5, and llama-3) in a six-shot setting for detecting hypocrisy accusations. the different models have different strengths, but overall perform with a macro f1 class of around 0.80, and show that llama, as a more open model than the gptfamily and one not requiring payment, can perform on par for hypocrisy accusation detection with the less open gpt model that requires payment. llms are capable of detecting accusations with a binary distinction, but we identify room for improvement when it comes to the different accusation types. models are somewhat worse at detecting political hypocrisy than personal moral hypocrisy, which could have implications for social science research.