Evaluating Deception Detection Model Robustness To Linguistic Variation (2021)

Abstract: With the increasing use of machine-learning driven algorithmic judgements, it is critical to develop models that are robust to evolving or manipulated inputs. We propose an extensive analysis of model robustness against linguistic variation in the setting of deceptive news detection, an important task in the context of misinformation spread online. We consider two prediction tasks and compare three state-of-the-art embeddings to highlight consistent trends in model performance, high confidence misclassifications, and high impact failures. By measuring the effectiveness of adversarial defense strategies and evaluating model susceptibility to adversarial attacks using character- and word-perturbed text, we find that character or mixed ensemble models are the most effective defenses and that character perturbation-based attack tactics are more successful.

## introduction over two-thirds of us adults get their news from social media, but over half (57%) "expect the news they see on social media to be largely inaccurate" @xcite . a 2020 reuters institute global news survey found a similar trend with 56% of respondents concerned with misinformation in online news @xcite . there are online and offline impacts from the spread of misinformation or deceptive news stories within online communities. however, the rate at which new content is submitted to social media platforms is a significant obstacle for approaches that require manual identification, annotation, or intervention. in recent efforts, evaluation has focused on aggregate performance metrics on test sets often collected from social media platforms like @xcite @xcite but these platforms are not representative in regards to user demographics or topics of discussion. further, aggregate performance metrics are not sufficient to provide insight on generalizable performance. when we consider the identification of deceptive news online -where humans often disagree on or challenge the judgements of others @xcite @xcite -we need more rigorous evaluations of model decisions, with a focus on expected performance across varied or manipulated inputs. our work examining reliability of performance when faced with linguistic variations is a step towards comprehensively understanding model robustness that may highlight inequalities in cases of failure. although machine learning models are often leveraged for their ability to tackle rapid response at scale, it is critical to understand nuanced model biases and the significant downstream consequences of model decisions on users. a known gap exists in our understanding of underlying machine learning decision-making processes, particularly with deep learning "black-box" models. the use of traditional, aggregate metrics for model performance, such as accuracy or f1 score, are not sufficient in pursuit of this understanding. we argue that evaluations need to explicitly measure the extent to which model performance is affected by data with a varied topic distribution. evaluations highlighting when models are correct, which examples can provide explanations, and clarification or reasoning for why a user should trust a given model are well-aligned with recent themes in research on machine learning interpretability, trust, fairness, accountability, and reliability @xcite @xcite . in this paper, we perform an adversarial model evaluation across two multimodal deception prediction tasks to identify which defensive strategies are most successful across a variety of attacks. our main contribution is a framework of analysis for model robustness across variations in linguistic sig-nals and representations that may be encountered in real-world applications of digital deception models (e.g., natural linguistic differences, evolving tactics from deceptive adversaries to evade detection). in particular, we present evaluations on the susceptibility of widely used text embeddings to naive adversarial attacks, which types of text perturbations lead to the most high-confident errors, and to what extent our findings are task specific. the perturbed text emulates real examples of linguistic variations, e.g. non-native speakers, spelling mistakes, or shortened online speech. our evaluations reveal how models react to perturbed text which we argue is a likely occurrence when deployed in a real-world setting. ## related work with the increasing concern for the impact of misinformation and deceptive news content online, many studies have explored or developed models that detect such news. recent efforts focus on identifying a spectrum of deception: from binary classification of content as suspicious and trustworthy @xcite to a more fine-grained separation within deceptive classes (e.g., propaganda, hoax, satire) @xcite . additional work has explored the behavior of malicious users and bots @xcite @xcite and spread patterns of misinformation or rumors @xcite to aid in classification tasks. strong evidence suggests that enriched features such as images, temporal and structural attributes, and linguistic features boost model performance over dependence on textual characteristics alone @xcite @xcite . the need for effective, trustworthy, and interpretable detection models is a vital concern and must be an essential requirement for models where decisions or recommendations can significantly affect end users. a variety of deep learning architectures applied to deception detection tasks include convolutional neural networks (cnns) @xcite @xcite , long short-term memory (lstm) models @xcite @xcite , and lstm variants with attention mechanisms @xcite . architecture and other aspects of neural network design typically depend on the classification task and require specialized hyperparameter tuning. in order to provide a fair comparison of model evaluations across tasks and for the purpose of consistency across experiments, we implement a multimodal lstm model similar to recent work. our approach allows for more accurate comparisons of factors related to adversarial susceptibility across classification tasks. developing novel state-of-the-art models for deception detection or comparing multiple architectures is beyond the scope of this paper. although popularly used across many domains, deep learning systems can be extremely brittle when evaluated on examples outside of the training data distribution @xcite @xcite . @xcite have shown that small perturbations in input data can cause highly probable misclassifications. further research demonstrates additional attacks that make neural networks more susceptible to adversaries such as locally trained dnns to crafted adversarial inputs @xcite and gradient-based attacks @xcite . to counteract these offensive strategies, proposed methods of defense include augmented training data with adversarial examples @xcite , training a separate model to distinguish genuine data from malicious data @xcite , and implementing a defensive distillation mechanism to increase a model's resiliency to data poisoning @xcite . however, as defense strategies are created, new attacks are continually developed to circumvent them @xcite . while there is a focus on image perturbations and related attacks, textual data is similarly vulnerable to such strategies @xcite @xcite ). the susceptibility of deception detection models to text-based adversarial attacks as well as the effectiveness of defense strategies have not been extensively evaluated. ## methodology in this section, we introduce our detection tasks, models, and evaluation methods. we randomly perturb words or characters with their nearest neighbors to mimic a low-effort adversarial attack (e.g., replacing words with synonyms) as opposed to methods that assume an adversary has technical expertise or require sophisticated augmentations (e.g., gradient-based algorithms). we argue that robustness against these low-effort attacks is a necessary first step towards trustworthy models; these attacks are reflective of natural or unintentional variations (e.g., misspellings, non-native speaker discussions) as well as sophisticated strategies. ## experimental results in this section, we detail our results when evaluating different combinations of adversarial defenses and attacks. in order to produce a holistic evaluation of model susceptibility, we examine defenses and attacks separately. although we consider the same model behavior, each position can highly impact the interpretation of the findings and key takeaways. we also want to understand model misclassifications, including those with high model confidence and those that can have a greater negative effect in practice which we accomplish with our high confidence and high impact analyses.