Paying Attention to Deflections: Mining Pragmatic Nuances for Whataboutism Detection in Online Discourse (2024)

Abstract: Whataboutism, a potent tool for disrupting narratives and sowing distrust, remains under-explored in quantitative NLP research. Moreover, past work has not distinguished its use as a strategy for misinformation and propaganda from its use as a tool for pragmatic and semantic framing. We introduce new datasets from Twitter/X and YouTube, revealing overlaps as well as distinctions between whataboutism, propaganda, and the tu quoque fallacy. Furthermore, drawing on recent work in linguistic semantics, we differentiate the ‘what about’ lexical construct from whataboutism. Our experiments bring to light unique challenges in its accurate detection, prompting the introduction of a novel method using attention weights for negative sample mining. We report significant improvements of 4% and 10% over previous state-of-the-art methods in our Twitter and YouTube collections, respectively.

## introduction whataboutism is the practice of deflecting criticism or avoiding an unfavorable issue by raising a different, more favorable matter, or by making a counteraccusation. since its first use in 1974 @xcite , it has emerged as a common variation of the classical fallacy known as tu quoque (lit. "you also") -attacking the opponent's behavior or action for being inconsistent with their argument, thereby discrediting them. despite significant work devoted to misinformation and propaganda, the detection of whataboutism has largely relied on the ease of tracking "what about" phrases. this, however, is informed by popular notions of the phenomenon, leading to a naïve linguistic treatment and subsequently, a neglect of the unique challenges to its detection. many "what about" phrases do not, in fact, signal propagandist use (fig. 1 ). whataboutism drives information disorder foot_2 by derailing proper argumentation, instead of the relatively conspicuous act of misleading by directly lying. as epistemologist @xcite ) argues, information disorder comprises two equally odious functions: (a) creating false beliefs, and (b) preventing the creation of belief in truth. empirical efforts target the former through models for fact-checking and fake news detection @xcite . whataboutism, on the other hand, is a prevalent and potent contributor to the latter as well. given its deleterious effects on discourse and social cohesion, it is thus critical for scalable propaganda detection models to be informed by a deeper understanding of whataboutism so that valid argumentation is not conflated with propaganda. figure 2 : syntactically and semantically similar (or even identical) responses may exhibit extreme pragmatic flexibility: being instances of whataboutism @xcite or not @xcite depending on the discursive context. furthermore, the latter category may include valid argumentation tools such as suggestion or challenge, instead of propaganda. this work provides just such a scrutiny: with two new datasets ( §3), we describe the first rigorous framework for the analysis and detection of whataboutism, distinguishing valid argumentation from propagandist attempts to derail a narrative. we find that instances of whataboutism, redirection, and tu quoque fallacies do not coincide in contemporary social media. contrary to suppositions of prior work on propaganda detection, our analysis reveals that whataboutism serves purposes other than propaganda. further, we demonstrate that a habitual application of transfer learning or contrastive learning are remarkably poor at detecting whataboutism. 4 to overcome this, we describe a novel method to mine negative samples using attention weights instead of contrast using cosine similarity ( §4). with this new formulation of (dis) similarity achieving superior results ( §5), our work holds implications for understanding complex discourse surrounding misinformation and propaganda, and modeling natural language pragmatics. whether "what about" is explicit or not, it can suggest a solution, challenge a statement, add specification to prior discussion, or laterally redirect the qud @xcite . this extreme pragmatic flexibility makes it difficult to distinguish valid argumentation from propagandist use. the problem is further compounded by the fact that the pragmatic distinctions coexist with lexical and semantic similarity. fig. 2 illustrates how the same text may serve as (i) harmful deflection via accusation or diversion, which are characteristics of propaganda, and (ii) accommodation or "centering" of attention. 5 thus, strategies relying solely on syntax or semantics may not differentiate these opposing discourse maneuvers. youtube comments have not received much attention in empirical studies. our first collection comprises youtube comments due to a balanced user distribution across demographics @xcite and a less regulated environment, fostering diverse opinions @xcite . targeting socially divisive topics prone to whataboutism, we formulate search queries for six such topics on youtube. from the top five most viewed videos per query, we collect english comments along with the title, transcript, the number of up-votes, and the publisher information. this corpus, tq + yt , comprises 1,642 labeled comments from 17 videos across 6 topics, sorted by up-votes. similar to tq + yt , our twitter dataset tq + tw focuses on socially relevant topics that tend to elicit strong emotional responses. we gather english tweets and their replies 8 such topics. the original tweet offers the pragmatic context for identifying whataboutism in responses. while six topics overlap with tq + yt , the other two contain fewer tweets, so that our corpus can be used to analyze model performance on topics with limited data. for each topic, we gather tweets with significant engagement, filtering out threads with less than 200 messages. within each thread, we exclude messages that do not directly respond to the original tweet, lack opinions or discursive content (e.g., "wow"), are socially too inappropriate, consist solely of emojis/emotions, or contain images/videos. in tq + tw , each datum comprises a tweet-reply pair, and the collection consists of 1,202 messages. we use stratified partitioning to divide both datasets into training, validation, and testing sections, ensuring an even distribution of comments across topics in each. within each topic, comments were split into 80% training, 5% validation, and 15% testing, maintaining the class distribution throughout. these stratified segments are then combined across topics to form the final training, validation, and test sets. the distribution of labeled data across these sets, as well as the class-wise distributions, are described in appendix a. we enlist three annotators with native fluency in english who independently assign binary labels to each comment. 7 to attain an understanding of the broader sociopolitical discourse pertaining to each topic, annotators dedicate considerable time to carefully review the entire youtube video or read the complete twitter conversation before labeling each instance. this meticulous process is expected to ensure diligent annotation and achieve a high level of data fidelity, despite limiting the size of the datasets. the final label is determined by majority vote, with inter-annotator agreements measured by fleiss' kappa @xcite at κ = 0.65 (tq + yt ) and κ = 0.75 (tq + tw ). foot_7 following recent recommendations @xcite , we include two comprehensive transparency artifacts: (a) the data statement (appendix a) and (b) the annotation guide/codebook, included in our data repository. ## experiments we start with two sets of experiments before describing our novel approach: (i) transfer-learning techniques akin to recent empirical research on propaganda detection, and (ii) possible enhancements to these baselines by incorporating conventional measures of semantic (dis-) similarity. ## results and analysis baseline models: the baseline models perform reasonably well on twitter, but not on tq + yt : bert achieves higher precision but lower recall, while roberta shows the opposite behavior. it is thus evident that models adept at propaganda detection have room for improvement in identifying whataboutism, especially given their tendency to assign the majority label nw to test instances. semantic similarity models: when applying the standard transfer-learning technique (i.e., finetuning on task-specific labeled data) to language (a) transfer-learning from pretrained language models. mining negatives with attention (mina): even while employing contextual embeddings, the inability of the above models to effectively detect whataboutism suggests a limitation in capturing the kind of pragmatic nuances illustrated earlier (fig. 1 , 2 ). mina, however, is designed to leverage cross-attention scores across comments within a topic to model pragmatic context. to demonstrate its efficacy, we assess the worst-performing models from previous categories, focusing on f 1 for the target class. on tq + yt , f 1 of roberta irony increases by 22%, rising from 0.3 to 0.52, while sbert shows an exceptional improvement of 51%, from 0.07 to 0.58: a 10% boost over the next best result. 15 when evaluated on tq + tw , sbert with mina outperforms other models, showing a 6% im15 to compare, the semeval-2020 task on propaganda detection @xcite saw the best performance in detecting whataboutism, straw man arguments, and red herrings, achieve @xmath0.269. (these three closely-related fallacies were merged into a single group due to insufficient data. ablation experiments: to determine the optimal configuration for mina, we conduct a series of experiments varying hyperparameters d (number of encoder layers), h (number of attention heads), d (number of comments for context size), and m (language model for input embeddings) (see the models are trained end-to-end with the configuration described earlier ( §4). each experiment varies one hyperparameter while keeping others constant. due to random selection of comments in context tuples, each ablation undergoes 50 runs, with mean f 1 and variance reported. the best hyperparameter configuration and its best run informs the results in the variance across all 50 trials for selecting the embeddings was very low, at σ 2 = 0.0089. a comparison of mining strategies: we compare mina's negative sample mining with two common strategies in recent contrastive mining literature: cosine similarity-based mining (e.g., @xcite and random sampling @xcite . challenges, negative results, and insights: in exploring several models and techniques to detect whataboutism, the path to mina, which uses negative sampling inspired by cross-attention, was marked by several unexpected outcomes. first, we experimented with the advanced pre-trained encoder deberta @xcite on the collection of youtube comments, but its performance was unexpectedly inferior to bert, achieving an f 1 score of 0.455 compared to bert's 0.48. although modern large language models (llms) like llama-2-7b, llama-2-13b @xcite , and gpt-3.5-turbo @xcite have shown impressive capabilities in various tasks, they did not yield satisfactory results in detecting whataboutism. the limitations we observe align with those noted by @xcite in detecting fallacious argumentation. on twitter comments, for instance, the highest f 1 score was 0.65, 4% lower than the bert baseline ( our experiments included incorporating video transcripts as contextual information, for which we leveraged longformer @xcite . contrary to initial expectations, these yielded lower precision and recall across all models. analyzing the errors revealed that comments from other users are a better predictor than the contextual information derived from the video. this observation, plus our thematic analysis ( §3.1) and the success of mina strongly indicate that while the video remains important in setting the topic, detecting whataboutism in a comment often depends more on the broader social discourse surrounding the topic. lastly, we decided against batch-contrastive learning due to two potential concerns raised by @xcite and @xcite : (i) large batch sizes are vital for it to be effective, and (ii) it typically involves augmentation, which introduces noise that hinders pragmatic understanding. further, it would still use cosine similarity on its own, which we have questioned in §4.2. nlp tasks have traditionally favored utilizing pretrained models like bert @xcite and roberta @xcite , fine-tuning them for specific tasks. @xcite accordingly approached propaganda detection, creating a corpus of news articles later used in shared tasks @xcite , where these pretrained models achieved top results @xcite . while related, our research differs substantially. they target propaganda detection, treating whataboutism only as propagandist language while the limited dataset size precludes deeper analysis. @xcite nevertheless note the difficulty in identifying whataboutism. since identifying the precise textual span of propaganda is laborious and error-prone, we look to @xcite , who label informal fallacies in social media comments. however, they continue to treat whataboutism purely as propaganda, while also turning to transfer learning through fine-tuning the models (i.e., the approach we use as baselines). framing and persuasion: unlike earlier studies, @xcite view whataboutism as strategic framing and persuasion rather than outright propaganda, thus offering a taxonomy that aligns more closely with our study. however, instances of whataboutism are sparse in their multilingual corpus (only 25 in english), limiting its utility in our work. additionally, fine-tuned transformer models or ensembles, as utilized by top-performing participants, yielded remarkably low f 1 scores for whataboutism and other related categories (see @xcite . the results of this shared task are noteworthy, yet they are confounded by ambiguity @xcite and low inter-annotator agreement. addressing these challenges is crucial for an accurate interpretation or analysis of the findings. ## conclusion our study underscores areas where state-of-theart propaganda detection models often fail to distinguish propagandist use of whataboutism from valid argumentation tools or figurative language use like irony, revealing how their suitability for this task may be improved. addressing this within the pragma-dialectic framework, we illustrate the need to develop models capable of understanding pragmatic variations. we find traditional similarity measures ineffective for this purpose. crossattention proves valuable instead. we thus propose a novel methodology, mina, for mining negative samples based on cross-attention, which achieves superior results in whataboutism detection and suggests its utility in other tasks that require grounding in ambient social discourse. this study is based on our contribution of two annotated datasets that can facilitate further research in related areas.