A (More) Realistic Evaluation Setup for Generalisation of Community Models on Malicious Content Detection (2024)

Abstract: Community models for malicious content detection, which take into account the context from a social graph alongside the content itself, have shown remarkable performance on benchmark datasets. Yet, misinformation and hate speech continue to propagate on social media networks. This mismatch can be partially attributed to the limitations of current evaluation setups that neglect the rapid evolution of online content and the underlying social graph. In this paper, we propose a novel evaluation setup for model generalisation based on our few-shot subgraph sampling approach. This setup tests for generalisation through few labelled examples in local explorations of a larger graph, emulating more realistic application settings. We show this to be a challenging inductive setup, wherein strong performance on the training graph is not indicative of performance on unseen tasks, domains, or graph structures. Lastly, we show that graph meta-learners trained with our proposed few-shot subgraph sampling outperform standard community models in the inductive setup.

## introduction the combination of connectivity and anonymity offered by social media inadvertently also provides the perfect channel for wide-spread dissemination of malicious content @xcite @xcite . by malicious content, we consider any form of content detrimental to society, and focus on two specific forms: misinformation and hate speech. mitigating the effect of malicious content requires content moderation, but this is a labour-intensive process that exacts an immense psychological toll on moderators @xcite . consequently, automated detection of malicious content has seen increased academic interest @xcite and industry adoption. community models for malicious content detection are models that operate on social graphs, i.e., graphs of content and users. they 1) classify content nodes as malicious or not, 2) incorporate information from interacting users in the graph when doing so, and 3) leverage emergent network properties like homophily to boost detection performance @xcite . for community modelling on large-scale heterogeneous online communities, graph neural networks (gnns) are the architecture of choice @xcite . while community models for malicious content detection perform very well on benchmark datasets @xcite , social media platforms continue to grapple with such content. @xcite find that high accuracy in malicious content detection is not indicative of trustworthiness in general, as predictions often rely on dataset-specific features. models also become outdated quickly as online content and communities evolve @xcite . @xcite find detection models to be brittle to changes in domain or publication date, a finding that @xcite corroborate specifically for community models. finally, @xcite conclude that "we @xcite no graph benchmark data for fake news detection in the graph learning community" (p. 22), making any claims of state-of-the-art performance difficult to verify. evidently, there exists a mismatch in the performance of community models on research datasets and in more realistic application settings. research datasets are static; they capture a view of the social graph weeks or months after relevant content has been introduced and spread. current evaluation practices designed on static graphs are effectively transductive @xcite , i.e., they implicitly assume that no new content or users will be introduced into the social graph, which leads to performance scores that obscure the discussed deficiencies. in realistic settings, new users and content nodes are constantly added to the social graph, and the topic or style of malicious content often radically changes. this is a property inherent to online content and communities @xcite . thus, a successful community model should be able to rapidly adapt to domain shifts in content. since labelling is prohibitively expensive, adaptation should occur in a few-shot manner. furthermore, the community of interacting users also evolves. initially, only a few users take note of some content, but as it gains traction, more and more users interact. to prevent harm from malicious content, detection must occur before wide-spread dissemination. this requires adaptation from a limited exploration of the social graph. therefore, inductive evaluation is needed. in this paper, we seek to more realistically assess the generalisation capabilities of community models for malicious content detection, specifically making the following contributions: ## datasets & tasks we use three widely-adopted social graph datasets to train and evaluate community models. social graph datasets are difficult to collect and degrade as users or content are moderated out. the first dataset, gossipcop, is used for pre-training. the other two datasets, coaid and twitterhatespeech, are reserved for evaluating generalisation to unseen graphs. twitterhatespeech differs in task, domain, and relational schema from the other datasets. document nodes are tweets generated by twitter users during a few seed events. @xcite identified prolific hate speech tweeters, and include their followers and followees in the graph. they manually labelled all tweets as racist, sexist or innocuous (none). especially racist (0.3%) users are over-represented, leading to few diverse regions in the graph. user-document connections indicate authorship (as opposed to tweet/retweet interactions). users are also connected to other users. ## methodology a social graph, @xmath0, e), consists of a set of nodes v and a set of edges e indicating which nodes are incident to each other. a node's r-radius neighbourhood n r (v) contains all other nodes that can be reached by paths of length r (also called 'hops', a series of incident nodes), and always includes v. all datasets considered require modelling two node-types: documents (v docs. ) and users (v (users) ). hence, the social graph g is heterogeneous. document nodes v contain exogenous features x v (i.e. the content representation), and have target labels y @xmath1. documents are only connected to users (∀@xmath2, @xmath3, whereas users may while budget: figure 1 : support subgraph generation. left: collect the r-radius neighbourhood of an anchor user. middle: sub-sample using random walks from document nodes until reaching a maximum node count. right: unmask document nodes inversely proportional to the number of subgraphs they appear in. colours correspond to classes. also be connected with other users based on their interactions or relations. users are not labelled, and have no initial representation. ## conclusion this paper proposes a more realistic evaluation setup for community models on malicious content detection. we highlight several properties of evolving social graphs that are especially neglected: expensive labelling, limited context, and emerging content and users. experiments verified our motivation, with performance on a single, static dataset in a transductive manner bearing little resemblance to performance during few-shot inductive generalisation. our proposed few-shot subgraph sampling approach presented in section 4.2 is tailored to social media graphs and allows generalisation of community models to new networks, domains, and tasks. while standard community models performed poorly, incorporating our sampling procedure in graph meta-learners aided generalisation. particularly promising are models with prototypical initialisation. ultimately, our results suggest that malicious content detection using community models is not 'solved', despite some models achieving near perfect evaluation scores. current evaluation proce- an open problem, warranting further investigation, is the application of meta-learning to class imbalanced datasets. common to malicious content detection, class imbalance imposes a severe penalty on meta-learners that reset their classification heads.