Leveraging Community and Author Context to Explain the Performance and Bias of Text-Based Deception Detection Models (2021)

Abstract: Deceptive news posts shared in online communities can be detected with NLP models, and much recent research has focused on the development of such models. In this work, we use characteristics of online communities and authors — the context of how and where content is posted — to explain the performance of a neural network deception detection model and identify sub-populations who are disproportionately affected by model accuracy or failure. We examine who is posting the content, and where the content is posted to. We find that while author characteristics are better predictors of deceptive content than community characteristics, both characteristics are strongly correlated with model performance. Traditional performance metrics such as F1 score may fail to capture poor model performance on isolated sub-populations such as specific authors, and as such, more nuanced evaluation of deception detection models is critical.

## introduction the spread of deceptive news content in online communities significantly erodes public trust in the media @xcite . most social media users use these platforms as a means to consume news -71% of twitter users and 62% of reddit users -and in general, 55% of americans get news from online communities such as facebook, twitter, and reddit @xcite . the scale and speed with which new content is submitted to social media platforms are two key factors that increase the difficulty of how to respond to the spread of misinformation or deceptive news content online, and the appeal of automated or semiautomated defenses or interventions. natural language processing (nlp) models that identify deceptive content offer a path towards fortifying online communities, and a significant body of work ( § 2) has produced countless such models for deception detection tasks @xcite @xcite @xcite . however, evaluation of model performance is typically done in aggregate, across multiple communities, using traditional performance measurements like micro and macro f1-scores. we argue that it is critical to understand model behavior at a finer granularity, and we evaluate nuanced behavior and failure in the context of the populations that may be affected by predictive outcomes. in this work, we seek to characterize and explain deception detection model performance and biases using the context of social media postswho posted the content and what community it was posted to. to do so, we compute hundreds of community and author characteristics using information from two fact checking sources. for a given post, community characteristics detail where a post was submitted to, e.g., how many links to satirical news sources were submitted to the community this post was submitted to? author characteristics detail who submitted a post, e.g., how many links to satirical news sources has the author recently submitted? our nuanced evaluation leverages these author and community characteristics to highlight differences in behavior within varying communities or sub-populations, to determine whether the model is reliable in general, or if model failures disproportionately impact sub-populations. we make use of data from reddit, a popular social news aggregation platform. reddit is widely used for research @xcite due to its large size and public content @xcite , and is ideally suited for studying author and community characteristics due to its explicit segmentation into many diverse communities, called "subreddits", with different sizes, topics, and userbases. 1 we use post context (community and author characteristics) and content (text features) to address two research questions focused around (1) who posts deceptive news links and (2) where they post differ: we find that author characteristics are a stronger predictor of high model performance, with the model we evaluate performing especially well on authors who have a history of submitting low factual or deceptive content. we also find that the model performs especially well on posts that are highly accepted by the community, as measured by the community's votes on those posts. to our knowledge, we are the first to present a fine-grained evaluation of deception detection model performance in the context of author and community characteristics. ## related work in the last several years, users have seen a tremendous increase in the amount of misinformation, disinformation, and falsified news in circulation on social media platforms. this seemingly ubiquitous digital deception is in part due to the ease of information dissemination and access on these platforms. many researchers have focused on different areas of detecting deceptive online content. glenski and weninger (2018); @xcite examine the behaviors and activities of malicious users and bots on different social media platforms. while others have worked to develop systems to identify fraudulent posts at varying degrees of deception such as broadly classifying suspicious and non-suspicious news @xcite to further separating into finer-grained deceptive classes (e.g., propaganda, hoax) @xcite . common amongst recent detection methods is the mixed use of machine learning approaches, e.g., random forest and state-of-the-art deep learning models, e.g., hierarchical propagation networks @xcite . of the most prevalent are convolutional neural networks (cnns) @xcite @xcite , long short term memory (lstm) neural networks @xcite @xcite @xcite , and other variants with attention mechanisms @xcite . designing the right model architec-ture for a task can be very subjective and laborsome. therefore, we implement the binary classification lstm model from @xcite which reported an f1 score of 0.73 when distinguishing deceptive news from credible. as artificial intelligence or machine learning models are developed or investigated as potential responses to the issue of misinformation and digital deception online, it is key to understand how models treat the individuals and groups who are impacted by the predictions or recommendations of the models or automated systems. for example, the european union's gdpr directly addresses the "right of citizens to receive an explanation for algorithmic decisions" @xcite that requires an explanation to be available for individuals impacted by a model decision. domains outside of deception detection have shown clear evidence of disproportionate biases against certain sub-populations of impacted individuals, e.g., predictive policing @xcite , recidivism prediction @xcite , and hate speech and abusive language identification online @xcite @xcite . the realm of deception detection is another clear area where disparate performance across communities or certain user groups may have significant negative downstream effects both online and offline. in this work, we seek to go beyond traditional, aggregate performance metrics to consider the differing behavior and outcomes of automated deception detection within and across communities and user characteristics. ## deception detection model in this work, we focus on a binary classification task to identify posts which link to deceptive or credible news sources. we evaluate an existing, lstm-based model architecture previously published by @xcite that relies only on text and lexical features. as such, we refer to this model as the "contentonly model." ## community & author characteristics to evaluate fine-grained model performance and biases, we first quantify the context in which posts are submitted, using community and author characteristics. ## evaluation methodology before our evaluation of model performance across different community or author characteristics and settings, we examine the overall performance of the model on aggregate, using macro f1 score, and the variance of performance within communities. a model with strong aggregate performance may have significant variability within subcommunities, especially those which are underrepresented. we also consider the variability of individual predictive outcomes, such as the confidence of predictions, across each class (deceptive and credible news) to examine the differences in model behavior across classes overall. we aim to discover if the model treats all posts, communities, and authors equally, or if there are differences in performance for certain groups that would bias the negative impacts of model error. ## results although the contentonly model achieves an overall f1 score on the test set of 0.79, we see that the model performs much better on content from some communities than others (see figure 2 ). figure 1 presents the communities within the test set, partitioned by levels of model performance using the quartiles for the f1 scores. we find that 20% of the communities represented in our test set have @xmath0.40, despite an overall test set f1 of almost 0.8. in the following subsections, we examine how the model's performance can be explained by community and author characteristics, post popularity, and community acceptance as we seek to understand why the model performs far better on content from some communities than others. ## discussion and conclusions in summary, we quantify the context of deceptive and credible posts by computing community and author characteristics and use these characteristics, to explain and characterize the performance of an lstm-based model for deception detection, examining performance variance across communities or users to identify characteristics of sub-populations where the model disproportionately underperforms. we find that in general, sub-population characteristics are more strongly correlated with aggregate performance, and that, for both communities and authors, the model is more effective at identifying deceptive posts (higher f1 and precision) when the author/community has a greater tendency to submit or receive posts linked to deceptive, low factual, and right biased news sources. in contrast, a greater tendency to submit or receive posts linked to high factual and center biased content are correlated with weaker f1 and precision -the model is more likely to fail when identifying posts submitted to communities or users that engage with more trustworthy news sources. we also investigate the impact that community acceptance has on model performance, using community-normalized scores to quantify acceptance. we find that, for posts with low to moderator community acceptance, correlations with the model's confidence that a post belongs to its groundtruth annotation class are small, but for posts that are strongly accepted by the community they are submitted to, acceptance is strongly correlated with increased model confidence for deceptive content, but only moderately correlated with decreased model confidence for credible content. it is important to consider what kinds of failures are most impactful given the specific application of a model. for example, if considering a deception detection model for use as an intervention strategy, it may be more important for a model to have greater reliability when identifying content that gains widespread community acceptance or popularity as we find our contentonly model does -this is an impor-tant direction of evaluation for researchers in the deception detection domain to consider. we encourage nlp researchers working in the deception detection space to look beyond overall test-set performance metrics such as f1 score. although many models achieve high overall f1 score, the performance of these models varies dramatically from community to community. decisions about model design and training should not be made without considering the intended application of the model. for example, a model tasked with flagging posts for human review may be optimized with a very different precision-recall tradeoff than a model tasked with automatically taking entire enforcement actions, such as removing content.