Learning to love diligent trolls: Accounting for rater effects in the dialogue safety task (2023)

Abstract: Chatbots have the risk of generating offensive utterances, which must be avoided. Post-deployment, one way for a chatbot to continuously improve is to source utterance/label pairs from feedback by live users. However, among users are trolls, who provide training examples with incorrect labels. To de-troll training data, previous work removed training examples that have high user-aggregated cross-validation (CV) error. However, CV is expensive; and in a coordinated attack, CV may be overwhelmed by trolls in number and in consistency among themselves. In the present work, I address both limitations by proposing a solution inspired by methodology in automated essay scoring (AES): have multiple users rate each utterance, then perform latent class analysis (LCA) to infer correct labels. As it does not require GPU computations, LCA is inexpensive. In experiments, I found that the AES-like solution can infer training labels with high accuracy when trolls are consistent, even when trolls are the majority.

## introduction for chatbots, it is not enough to learn to generate coherent utterances-many coherent utterances are undesirable, such as offensive remarks. postdeployment, to have the chatbot continuously improve its judgment, one way is to source training examples from feedback by live organic users @xcite . though more realistic than crowdworkers, among organic users are trolls, whose erroneous feedback fosters bad behavior @xcite . how can the chatbot learn continuously from user interactions in a way that is robust to trolls? we thus endeavor to de-troll these labels, as a preprocessing step @xcite . once labels are corrected (i.e. y * = y), the problem reduces to familiar supervised learning. toward de-trolling the training set, a twointervention solution was recently offered by @xcite . intervention 1 has each user provide multiple (x, y * ) pairs, so examples are nested within users. intervention 2 computes the "untrustworthiness" of each training example via (3fold) cross-validation (cv). a training example is removed if it exceeds some threshold of disagreement with the crowd. the intuition is that trolls end up having a bad track record against the crowd but occasionally, by chance, provide examples worth keeping. the (x, y * ) pairs remaining are then suitable for training. 1 there are two issues with this cv-based @xcite solution. first, as it involves cv on large language model (llm) computations, it is expensive. second, cv relies upon an abundance of helpers @xcite , good-faith users who end up in agreement with each other. thus, this approach can be overwhelmed in the event of a coordinated attack where trolls have both strength in numbers and are consistent among themselves. a promise of circumventing both issues lies in methodology for automated essay scoring (aes) @xcite @xcite . given an input essay x, an aes engine (which may incorporate an llm) must output a score y, a numeric or ordinal rating of the essay's quality. the burden of annotating a training set is divided among multiple human graders, which incurs rater effects-graders vary on how strict or permissive they are, which adds variance to automated scoring @xcite . analogously, the aes solution has two interventions @xcite . intervention 1 induces redundancy by randomly assigning multiple graders (users) to each training essay (utterance), yielding an inter-rater matrix z as in in the present work, my contribution is twofold. first, i establish a connection between the chatbot robust learning (e.g. @xcite and the aes (e.g. @xcite literatures. second, i conduct an examination of an aes-like solution to de-trolling the training set. the main result is that this solution thrives on consistency, even among trolls: when trolls are consistent among themselves and are majority of the users, inferred labels are accurate; but curiously, when trolls are majority but make only random noise, the aes-like solution is not as accurate. ## related work in open dialogue models, the capacity to improve past pre-training is desired. @xcite found that challenging crowdworkers to "break" a pre-trained classifier with adversarial inputs in follow-up rounds of training was helpful toward making the classifier robust. @xcite considered various methods to improve chatbots in deployment from different forms of human feedback, including free-form text. an emerging innovation is architectures that are self-monitoring, being both generator and classifier @xcite . but as such architectures assume that training examples have correct labels, they are solutions for supervised learning, not detrolling. addressing the problem takes more than finding a state-of-the-art classifier. the threat posed by trolls falls under the umbrella of robust learning @xcite . whereas the cv-based procedure @xcite is one way to do robust learning, another is expectation-maximization (em) algorithms @xcite , of which lca fitting (linzer and lewis, 2011) is an instance. in statistics, em is a local optimization technique for fitting models involving latent variables, such as unobserved class labels. an elementary instance of em is the familiar k-means clustering algorithm @xcite . more sophisticated instances of em are applicable to problems with untrustworthy training examples and rater effects, such as crowdworkers' tagging of images and documents @xcite . the use of aes in large-scale educational assessment dates back decades @xcite . the aes literature goes deeper-as essays may have been written in response to different prompts, some not appearing in training, and some assessments score essays in multiple criteria @xcite . but in the present work, the most basic aes setup suffices for the analogy-every annotation y * (the class label or score) is a commonsense holistic evaluation of the safety or toxicity of x (the utterance or essay). while the present article is focused on toxicity or safety @xcite , there are other desiderata for chatbot behavior, such as factual correctness and staying on-topic @xcite . ## from ratings to predicted classes to motivate lca, consider accordingly, there are two clusters: those rated "0" by grader 3 (i.e. essays 1 and 2); and those rated "0" by graders 1 and 2 (i.e. essays 3 to 5). realistic instances of z are not as clear-cut, so it takes a statistical model to apply such intuitions at scale. the lca model is depicted in figure 1 . despite the name, lca deals with clusters rather than classes. according to this model, each utterance belongs to one of two clusters, a and b, with a probability distribution (the prior). within each cluster, the distribution of the observed label (the likelihood) is user-specific: each user has some probability of labeling a cluster a utterance as safe, and some probability of labeling a cluster b utterance as safe. 3 an em algorithm (linzer and lewis, 2011), provided z, estimates all parameters (in both prior and likelihood). the intuition is that users' tendencies can be quantified due to utterances shared between them. provided these estimates, for each utterance, the posterior distribution can be computed, which is the answer to the question, "given the observed pattern of labels (i.e. row of z), which cluster is more probable?" each utterance is then assigned to the highest-probability cluster. note that lca fitting is possible even amid missing values in z, but some redundancy (from intervention 1) is required (see limitations section for details). however, clusters are not yet classes. in the dialogue safety task, trolls and helpers behave in exist, and altogether, the aes-like solution consists of (intervention 1) introducing redundancy, then (intervention 2) statistically leveraging it via lca with the safe-as-majority assumption (lca+sm). the end result is corrected training pairs (x, ỹ). the de-trolling solution is inexpensive because it is embeddings-free; and it is robust because user tendencies are accounted for, without taking for granted that the consensus is correct. unlike the cv-based solution @xcite , no untrustworthiness scores need be calculated, as user tendencies are already accounted for. no examples are removed from training-the intuition is that a user that reliably gives incorrect answers is perfectly good data once their labels are flipped. ## discussion how much data should z have for lca to be reliable? while the proportion of missing values was while the cv-based @xcite solution's issues were not directly experimentally validated, there is reason to believe that it would have been at a disadvantage against the aes-like solution. in terms of expense, cv would involve training a large language model, even at the de-trolling phase. in terms of robustness, cv can be thought of as simulating "checking each other's work" or tapping into a consensus in a way that mv directly does, so the cv-based solution would have been likewise vulnerable to a false consensus. scenarios where the aes-like solution faltered had little consensus, which would have been unfavorable to the cv-based solution as well. unlike @xcite , my experiments did not manipulate the prevalence of adversarial vs. nonadversarial utterances sampled. in the dataset @xcite , half of the utterances were adversarial. these prevalences would have affected the supervised learning phase but not the de-trolling phase, as mv and lca+sm are embeddings-free. it turns out that experiment 2 was a purely statistical exercise, though one relevant to the de-trolling problem. after all, producing z and evaluating imputation accuracy did not reference any utterance content, so no real data was needed. consequently, the results for imputation accuracy are widely generalizable in the sense that they are irrespective of the choice of neural classifier or the intrinsic difficulty of the utterances being classified. in addition, experiment 2 can be expediently adapted to different settings (e.g. unsafe_prevalence and number of training utterances) for researchers to mess around with. but for specific applications involving a specific dataset and neural classifier, downstream accuracy becomes of interest. ## conclusion the aes-like solution is inexpensive, as it does not require gpu computation. in the event of a coordinated attack where trolls are majority and consistent among themselves, the aes-like solution holds up well, bearing the wisdom that "diligent" trolls may be leveraged toward better performance. but when there is little consensus among users, there is little to work with, so de-trolling accuracy drops.