Detecting Online Community Practices with Large Language Models: A Case Study of Pro- U krainian Publics on T witter (2024)

Abstract: Communities on social media display distinct patterns of linguistic expression and behaviour, collectively referred to as practices. These practices can be traced in textual exchanges, and reflect the intentions, knowledge, values, and norms of users and communities. This paper introduces a comprehensive methodological workflow for computational identification of such practices within social media texts. By focusing on supporters of Ukraine during the Russia-Ukraine war in (1) the activist collective NAFO and (2) the Eurovision Twitter community, we present a gold-standard data set capturing their unique practices. Using this corpus, we perform practice prediction experiments with both open-source baseline models and OpenAI’s large language models (LLMs). Our results demonstrate that closed-source models, especially GPT-4, achieve superior performance, particularly with prompts that incorporate salient features of practices, or utilize Chain-of-Thought prompting. This study provides a detailed error analysis and offers valuable insights into improving the precision of practice identification, thereby supporting context-sensitive moderation and advancing the understanding of online community dynamics.

## introduction online communities on platforms like twitter 2 display distinctive and sustained patterns of behaviour and action, often referred to as practices @xcite @xcite , that are directed towards a goal and shaped by the socio-political context and affordances of digital platforms. practices are significant because they reflect the values and beliefs of communities that engage in them @xcite . for instance, consider knowledge figure 1 : analytical schema for identifying practices adopted from @xcite . images represent communities analysed in this study -a) fans of the eurovision song contest known for their active use of twitter, and b) nafo, recognized for its efforts in debunking russian propaganda on twitter, characterized by avatars featuring shiba inu dogs. identifying practices in texts of online communities at scale addresses this need. it is also a necessary step towards practice mapping, which involves using computational and qualitative methods to investigate communities' patterns of language use and non-language-centred actions, such as sharing of urls or interactions with other users @xcite . while there exist frameworks for identification of practices through ethnographic, survey, or discourse-analytical approaches @xcite @xcite , inferring them at scale in voluminous and ever-evolving digital trace data is a complex task. nuanced identification of practices requires expertise in a community's vernacular, values, and the contexts they operate within -a challenge preventing from easy crowd-sourcing of such task and producing goldstandard data sets of practices for a community of interest large enough to support fine-tuning approaches. in addition, there is no consensus on how to best represent such expertise in a form of an in-context learning prompt in a way that could help the model "locate" @xcite the task of identifying instances based on a complex sociological notion. in this work, in both codebook preparation and construction of chain-of-thought (cot) incontext learning prompts @xcite , we build upon an analytical schema for qualitative identification of practices in discourses of professional communities which separates practice identification into several steps -first examining the utterance, then identifying its meaning, followed by inferring the intention and action behind it (gherardi, 2012) (see figure 1 ). we examine two online communities that support ukraine following russia's 2022 full-scale invasion through distinct forms of sustained action -nafo, who engage in crowdfunding for the ukrainian defence and debunking russian propaganda through humorous or offensive posts, and fans of the eurovision song contest, whose active twitter community engaged with ukraine's performers calling attention to the war during the 2022 and 2023 competitions. to account for the difference in the social meaning between the two communities, we experiment with injecting salient features from our annotation codebook into prompts. to sum up, in this work, we propose a novel framework for using texts of social media posts to identify practices in online communities and present: ## background speech acts and social meaning the view of linguistic utterances as accomplishing an action is captured in the notion of a speech, or an "illocutionary", act @xcite ) -a performance of an action following a set of rules that ensure that the interlocutor understands the intention behind the utterance. this idea has informed numerous studies detecting intention and action in texts @xcite @xcite and performing goaloriented dialogue modelling @xcite @xcite . works identifying speech acts in social media data have achieved this goal through transformerbased classifiers @xcite trained on expert-led semi-automated lexicons @xcite . these studies aimed to develop a generic classification approach, disregarding variation in online speech acts resulting from the authors' belonging to online communities or topical publics @xcite . in contrast, our paper disaggregates online data sets prior to classification to ensure social meaning is preserved when identifying action in online communities. we understand social meaning as variation in communicative acts spawning from the level of social practice -or distinct patterns of speech and action established in a specific community @xcite . similar to distinct groups of teenagers in a usa high school studied by @xcite , online communities act through language in ways that are established through their practices. how they express themselves using the affordances of social media platforms is conditioned by both the goals their practices are directed towards @xcite and the properties of digital environments where their practices unfold @xcite . computational linguists have previously examined social meaning @xcite @xcite . however, only a handful of studies @xcite considered how variation in language use reflects and produces norms and values. our operationalisation of practice as a unit of analysis for text classifiers ensures community values and goals are captured in the output of such classifiers. importantly, by adopting the notion of practice -a pattern of action common among users sharing similar values and goals, this approach avoids the misconception (bruns, 2019) of online communities as segregated homophilous "echo chambers" where members share an opinion on a topic @xcite . the two communities examined in this study have a common interest in russia's invasion of ukraine and take the side of the invaded country, but they achieve their goal of supporting ukraine in distinct ways. as evident from practices like arguing or shitposting (see section 5 for details), they are also aware of the opposing side and actively engage with them. ## practice prediction predicting practices automatically and with high quality would open new possibilities for under-standing online action and its implications. extending beyond semantic meaning @xcite , results of practice prediction can provide insights for better regulation of online activities. however, human annotation of large amounts of text data and its quality control is costly and time-consuming @xcite . in the case of the proposed methodological workflow, a high level of familiarity with the contextual and vernacular specificity of the community under investigation is also crucial for correct identification of practices, complicating the potential crowd-sourcing of the annotation task. ## discussion despite this potential, our results demonstrate that predicting a patterned intention and action behind online utterances with a limited number of samples is a difficult task for pre-trained large language models. in addition, even the best-performing setting (pd+cot+mpe prompt) fails to successfully predict a number of practices most closely aligned with our overarching research interest in communities' unique expression of support towards ukraine. we examine confusion matrices (figure 2 ) and identify two categories of interest for each case study where the pd+cot+mpe prompting does not result in satisfactory performance. for nafo, these are two of the most distinctive practices through which the collective combats russian propaganda: shitposting, or use of humorous or offensive posts to derail online discussions, and arguing, or debating opponents with logic and facts. for the esc case study, we select expressing solidarity and community imagining foot_4 due to their relevance for understanding how russia's war on ukraine altered practices of fans and their sense of belonging to the european community. to seek insights that could improve practice prediction task results, we examine 450 false positive and 185 false negative tweets for these categories and identify prominent causes of errors. humour and sarcasm as observed in previous studies @xcite , humour and sarcasm presented challenges for the model with the pd+cot+mpe prompt. this was especially relevant for nafo's practice of shitposting which largely relies on jokes. like in the example below, 53.85% of false negatives for this practice were likely due to the model's failure to identify humour or sarcasm. you used to blame ukraine's leaders, but look at you backpedaling. i can't see you riding a bike! overlapping practices according to our analysis, co-occurrence of multiple practices in one post was the most frequent cause of misclassification, accounting for 24.41% of errors overall. we observed it most prominently in false negative samples for eurovision's expressing solidarity practice (41.46% of misclassified samples). as in the example below, expressions of solidarity towards ukraine co-occurred with speaking on behalf of the user's national community, expressing emotions, or engaging in audiencing (live com-mentary). amazingly done to ukraine from the uk! you deserve to win. we're excited for you! #esc2022 #westand-withukraine. we acknowledge that errors of this type may be inevitable, as studies indicate that even when investigated qualitatively, practices do not have easily identifiable boundaries and there exist overlaps between them @xcite . due to this, human coders in our study also experienced difficulties with assigning one practice label per post. one potential avenue for the resolution of this issue could be treating practice prediction as a multi-label classification problem. misidentified stance of the author as elaborated in section 3, one of the categories in our task involved the identification of tweets by users supporting russia and tweets unrelated to the war. we observed that pd+cot+mpe prompt was not always effective in identifying a pro-russian stance in tweets. we attributed 24.52% of false positive samples for nafo's shitposting category to instances where the collective's adversaries deployed offensive language or logic to attack nafo. while the expected label in this scenario was not applicable, the model would classify such tweets as shitposting or arguing. you keep changing the subject -you are not good at this nafo thing, fatty. as a potential future solution to this issue, studies may introduce an upstream task of stance detection prior to classification of practices of a subset of users of interest, or incorporate this subtask in a form of a step in a cot prompt @xcite . ## conclusion this paper proposes a systematic and scalable approach to associating the use of language in online texts with user practices as sustained patterns of behaviour shaped by sociopolitical and platform contexts. it provides a first empirically-driven systematic overview of practices on social media during russia's war on ukraine and presents a methodological workflow that can be applied by a wider range of studies aiming at identifying intention and action in communities of users. the study advances our understanding of the potential of llms to make associations between utterances and online community practices. we demonstrate that even with a limited amount of gold-standard data, openai's models, specifically gpt-4, are promising tools worth exploring. in addition, we show that representing the task of practice identification as a series of steps, and adding salient features as well as prioritisation and exclusion criteria to prompts, improves the performance of openai's models. despite these promising results, these models still struggle with identifying sarcastic and humorous utterances as well as the stance of the speaker in addition to the practice(s) they engage in. to address this, future studies may benefit from exploring approaches where identification of stance or sarcasm is treated as a separate task from practice prediction. our error analysis also confirmed claims made in theoretical literature around the overlap between practices of communities. to address this challenge, approaching practice prediction as a multi-label problem should be tested. our hope is that computational linguistics and nlp communities continue to explore the practice prediction problem, enabling social scientists through insights and tools for scalable and efficient identification of user practices as manifested through language and beyond.