Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimation (2021)

Abstract: Content moderation is often performed by a collaboration between humans and machine learning models. However, it is not well understood how to design the collaborative process so as to maximize the combined moderator-model system performance. This work presents a rigorous study of this problem, focusing on an approach that incorporates model uncertainty into the collaborative process. First, we introduce principled metrics to describe the performance of the collaborative system under capacity constraints on the human moderator, quantifying how efficiently the combined system utilizes human decisions. Using these metrics, we conduct a large benchmark study evaluating the performance of state-of-the-art uncertainty models under different collaborative review strategies. We find that an uncertainty-based strategy consistently outperforms the widely used strategy based on toxicity scores, and moreover that the choice of review strategy drastically changes the overall system performance. Our results demonstrate the importance of rigorous metrics for understanding and developing effective moderator-model systems for content moderation, as well as the utility of uncertainty estimation in this domain.

## introduction maintaining civil discussions online is a persistent challenge for online platforms. due to the sheer scale of user-generated text, modern content moderation systems often employ machine learning algorithms to automatically classify user comments based on their toxicity, with the goal of flagging a collection of likely policy-violating content for human experts to review @xcite . however, modern deep learning models have been shown to suffer from reliability and robustness issues, especially in the face of the rich and complex sociolinguistic phenomena in real-world online conversations. examples include possibly generating confidently wrong predictions based on spurious lexical features @xcite , or exhibiting undesired biases toward particular social subgroups @xcite . this has raised questions about how current toxicity detection models will perform in realistic online environments, as well as the potential consequences for moderation systems @xcite . in this work, we study an approach to address these questions by incorporating model uncertainty into the collaborative model-moderator system's decision-making process. the intuition is that by using uncertainty as a signal for the likelihood of model error, we can improve the efficiency and performance of the collaborative moderation system by prioritizing the least confident examples from the model for human review. despite a plethora of uncertainty methods in the literature, there has been limited work studying their effectiveness in improving the performance of human-ai collaborative systems with respect to application-specific metrics and criteria @xcite @xcite . this is especially important for the content moderation task: real-world practice has unique challenges and constraints, including label imbalance, distributional shift, and limited resources of human experts; how these factors impact the collaborative system's effectiveness is not well understood. in this work, we lay the foundation for the study of the uncertainty-aware collaborative content moderation problem. we first (1) propose rigorous met-rics oracle-model collaborative accuracy (oc-acc) and auc (oc-auc) to measure the performance of the overall collaborative system under capacity constraints on a simulated human moderator. we also propose review efficiency, a intrinsic metric to measure a model's ability to improve the collaboration efficiency by selecting examples that need further review. then, (2) we introduce a challenging data benchmark, collaborative toxicity moderation in the wild (cotomod), for evaluating the effectiveness of a collaborative toxic comment moderation system. cotomod emulates the realistic train-deployment environment of a moderation system, in which the deployment environment contains richer linguistic phenomena and a more diverse range of topics than the training data, such that effective collaboration is crucial for good system performance @xcite . finally, (3) we present a large benchmark study to evaluate the performance of five classic and state-of-the-art uncertainty approaches on cotomod under two different moderation review approaches (based on the uncertainty score and on the toxicity score, respectively). we find that both the model's predictive and uncertainty quality contribute to the performance of the final system, and that the uncertainty-based review strategy outperforms the toxicity strategy across a variety of models and range of human review capacities. ## related work our collaborative metrics draw on the idea of classification with a reject option, or learning with abstention @xcite @xcite . in this classification scenario, the model has the option to reject an example instead of predicting its label. the challenge in connecting learning with abstention to oc-acc or oc-auc is to account for how many examples have already been rejected. specifically, the difficulty is that the metrics we present are all dataset-level metrics, i.e. the "reject" option is not at the level of individual examples, but rather a set capacity over the entire dataset. moreover, this means oc-acc and oc-auc can be compared directly with traditional accuracy or auc measures. this difference in focus enables us to consider human time as the limiting resource in the overall model-moderator system's performance. one key point for our work is that the best model (in isolation) may not yield the best performance in collaboration with a human @xcite . our work demonstrates this for a case where the collaboration procedure is decided over the full dataset rather than per example: because of this, @xcite 's expected team utility does not easily generalize to our setting. in particular, the user chooses which classifier predictions to accept after receiving all of them rather than per example. robustness to distribution shift has been applied to toxicity classification in other works @xcite , emphasizing the connection between fairness and robustness. our work focuses on how these methods connect to the human review process, and how uncertainty can lead to better decision-making for a model collaborating with a human. along these lines, @xcite analyzed how uncertainty affects optimal decisions in a medical context, though again at the level of individual examples rather than over the dataset. ## background: uncertainty quantification for deep toxicity classification types of uncertainty consider modeling a toxicity dataset @xmath0, x i } n @xmath1. here the x i are example comments, y i ∼ p * (y|x i ) are toxicity labels drawn from a data generating process p * (e.g., the human annotation process), and w are the parameters of the deep neural network. there are two distinct types of uncertainty in this modeling process: data uncertainty and model uncertainty @xcite . data uncertainty arises from the stochastic variability inherent in the data generating process p * . for example, the toxicity label y i for a comment can vary between 0 and 1 depending on raters' different understandings of the comment or of the annotation guidelines. on the other hand, model uncertainty arises from the model's lack of knowledge about the world, commonly caused by insufficient coverage of the training data. for example, at evaluation time, the toxicity classifier may encounter neologisms or misspellings that did not appear in the training data, making it more likely to make a mistake @xcite . while the model uncertainty can be reduced by training on more data, the data uncertainty is inherent to the data generating process and is irreducible. estimating uncertainty a model that quantifies its uncertainty well should properly capture both the data and the model uncertainties. to this end, a learned deep classifier f w (x) describes the data uncertainty via its predictive probability, e.g.: this distribution over w leads to a distribution over the predictive probabilities p(y|x, w ). as a result, at inference time, the model can sample model weights {w m } m @xmath2, and then compute the posterior sample of predictive probabilities {p(y|x, w m )} m @xmath3. this allows the model to express its model uncertainty through the variance of the posterior distribution var p(y|x, w ) . section 5 surveys popular probabilistic deep learning methods. in practice, it is convenient to compute a single uncertainty score capturing both types of uncertainty. to this end, we can first compute the marginalized predictive probability: p(y|x) = p(y|x, w )p(w ) dw which captures both types of uncertainty by marginalizing the data uncertainty p(y|x, w ) over the model uncertainty p(w ). we can thus quantify the overall uncertainty of the model by computing the predictive variance of this binary distribution: evaluating uncertainty quality a common approach to evaluate a model's uncertainty quality is to measure its calibration performance, i.e., whether the model's predictive uncertainty is indicative of the predictive error @xcite . as we shall see in experiments, traditional calibration metrics like the brier score @xcite do not correlate well with the model performance in collaborative prediction. one notable reason is that the collaborative systems use uncertainty as a ranking score (to identify possibly wrong predictions), while metrics like brier score only measure the uncertainty's ranking performance indirectly. ## the collaborative content moderation task online content moderation is a collaborative process, performed by humans working in conjunction with machine learning models. for example, the model can select a set of likely policy-violating posts for further review by human moderators. in this work, we consider a setting where a neural model interacts with an "oracle" human moderator with limited capacity in moderating online comments. given a large number of examples {x i } n @xmath4, the model first generates the predictive probability p(y|x i ) and review score u(x i ) for each example. then, the model sends a pre-specified number of these examples to human moderators according to the rankings of the review score u(x i ), and relies on its prediction p(y|x i ) for the rest of the examples. in this work, we make the simplifying assumption that the human experts act like an oracle, correctly labeling all comments sent by the model. ## benchmark experiments we first examine the prediction and calibration performance of the uncertainty models alone (section 6.1). for prediction, we compute the predictive accuracy (acc) and the predictive auc (both au-roc and auprc). for uncertainty, we compute the brier score (i.e., the mean squared error between true labels and predictive probabilities, a standard uncertainty metric), and also the calibration auprc (section 3). we then evaluate the models' collaboration performance under both the uncertainty-and the toxicity-based review strategies (section 6.2). for each model-strategy combination, we measure the model's collaboration ability by computing review efficiency, and evaluate the performance of the overall collaborative system using oracle-model collaborative auroc (oc-auroc). we evaluate all collaborative metrics over a range of human moderator review ca-pacities, with their review fractions (i.e., fraction of total examples the model sends to the moderator for further review) ranging over {0.001, 0.005, 0.01, 0.02, 0.05, 0.1, 0.15, 0.20}. results on further uncertainty and collaboration metrics (calibration auroc, oc-acc, oc-auprc, etc.) are in appendix d. ## conclusion in this work, we presented the problem of collaborative content moderation, and introduced co-tomod, a challenging benchmark for evaluating the practical effectiveness of collaborative (modelmoderator) content moderation systems. we proposed principled metrics to quantify how effectively a machine learning model and human (e.g. a moderator) can collaborate. these include oracle-model collaborative accuracy (oc-acc) and auc (oc-auc), which measure analogues of the usual accuracy or auc for interacting human-ai sys-tems subject to limited human review capacity. we also proposed review efficiency, which quantifies how effectively a model utilizes human decisions. these metrics are distinct from classic measures of predictive performance or uncertainty calibration, and enable us to evaluate the performance of the full collaborative system as a function of human attention, as well as to understand how efficiently the collaborative system utilizes human decision-making. moreover, though we focused here on measuring the combined system's performance through metrics analogous to accuracy and auc, it is trivial to extend these to other classic metrics like precision and recall. using these new metrics, we evaluated the performance of a variety of models on the collaborative content moderation task. we considered two canonical strategies for collaborative review: one based on the toxicity scores, and a new one using model uncertainty. we found that the uncertaintybased review strategy outperforms the toxicity strategy across a variety of models and range of human review capacities, yielding @xmath5.01 and ∼ 0.05 absolute increases in the collaborative system's auroc and auprc, respectively. this merits further study and consideration of this strategy's use in content moderation. the interaction between the data distribution and best review strategy demonstrated by the crossover between the two strategies' performance out-ofdomain) emphasizes the implicit trade-off between false positives and false negatives in the two review strategies: because toxicity is rare, prioritizing comments for review in order of toxicity reduces the false positive rate while potentially increasing the false negative rate. by comparison, the uncertaintybased review strategy treats false positives and negatives more evenly. further study is needed to clarify this interaction. our work shows that the choice of review strategy drastically changes the collaborative system performance: evaluating and striving to optimize only the model yields much smaller improvements than changing the review strategy, and misses major opportunities to improve the overall system. though the results presented in the current paper are encouraging, there remain important challenges for uncertainty modeling in the domain of toxic content moderation. in particular, dataset bias remains a significant issue: statistical correlation between the annotated toxicity labels and various surface-level cues may lead models to learn to overly rely on e.g. lexical or dialectal patterns @xcite . this could cause the model to produce high-confidence mispredictions for comments containing these cues (e.g., reclaimed words or counter-speech), resulting in a degradation in calibration performance in the deployment environment (cf. there exist several important directions for fu-ture work. one key direction is to develop better review strategies than the ones discussed here: though the uncertainty-based strategy outperforms the toxicity-based one, there may be room for further improvement. furthermore, constraints on the moderation process may necessitate different review strategies: for example, if content can only be removed with moderator approval, we could experiment with a hybrid strategy which sends a mixture of high toxicity and high uncertainty content for human review. a second direction is to study how these methods perform with real moderators: the experiments in this work are computational and there may exist further challenges in practice. for example, the difficulty of rating a comment can depend on the text itself in unexpected ways. finally, a linked question is how to communicate uncertainty and different review strategies to moderators: simpler communicable strategies may be preferable to more complex ones with better theoretical performance.