  Deceptive news posts shared in online communities can be detected with NLP
models, and much recent research has focused on the development of such models.
In this work, we use characteristics of online communities and authors -- the
context of how and where content is posted -- to explain the performance of a
neural network deception detection model and identify sub-populations who are
disproportionately affected by model accuracy or failure. We examine who is
posting the content, and where the content is posted to. We find that while
author characteristics are better predictors of deceptive content than
community characteristics, both characteristics are strongly correlated with
model performance. Traditional performance metrics such as F1 score may fail to
capture poor model performance on isolated sub-populations such as specific
authors, and as such, more nuanced evaluation of deception detection models is
critical.
