Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions (2025)

Abstract: For datasets to accurately represent diverse opinions in a population, they must preserve variation in data labels while filtering out spam or low-quality responses. How can we balance annotator reliability and representation? We empirically evaluate how a range of heuristics for annotator filtering affect the preservation of variation on subjective tasks. We find that these methods, designed for contexts in which variation from a single ground-truth label is considered noise, often remove annotators who disagree instead of spam annotators, introducing suboptimal tradeoffs between accuracy and label diversity. We find that conservative settings for annotator removal (<5%) are best, after which all tested methods increase the mean absolute error from the true average label. We analyze performance on synthetic spam to observe that these methods often assume spam annotators are less random than real spammers tend to be: most spammers are distributionally indistinguishable from real annotators, and the minority that are distinguishable tend to give fixed answers, not random ones. Thus, tasks requiring the preservation of variation reverse the intuition of existing spam filtering methods: spammers tend to be less random than non-spammers, so metrics that assume variation is spam fare worse. These results highlight the need for spam removal methods that account for label diversity.

## introduction because spam responses are common on crowdsourcing sites, researchers need reliable ways to filter out low-quality data. many of these methods aim to find annotators with unusual labeling behavior. however, a growing body of work has found that information from annotators with minority opinions can be a valuable source of information, since this disagreement helps to understand * equal contribution; order determined by coin flip. variability in the opinions of a population, identify cases where some annotators may be betteror worse-informed, or reveal ambiguity in the task. how can we preserve the opinions of annotators who disagree, while still removing spam annotations? we examine the effects of applying several common methods for discounting spam annotators based on their labeling behavior. despite the existence of spam removal methods that use attention checks or metadata (e.g., time spent on task), filtering based on labeling behavior remains common practice @xcite ; thus, weaknesses in these methods risk affecting a wide range of common machine learning tasks. we test three of these methods-mace @xcite , crowdtruth @xcite , and inter-annotator agreement metrics-on relatively subjective tasks and analyze effects on variability in the filtered data. we find that, although many methods are nearindistinguishable in terms of their accuracy at classifying spam annotators, some are far more likely to remove non-spam annotators who disagree. furthermore, we find that under most tested methods, removing more annotators degrades the variety of opinions expressed, without improving accuracy at removing spam annotators; thus, these methods seem most effective only when a very low number of annotators are removed. we also find that assumptions about the distribution of spam annotations can hinder the effectiveness of these methods. we examine performance on synthetic distributions of spam annotations to analyze whether these methods effectively remove spam annotations, or simply remove annotations farther from the mean. performance on synthetic spam indicates that most methods perform far better for random spam (e.g., randomly clicking answers) than fixed spam (e.g., always answering "no"). yet true spammer behavior exhibits the opposite trend: most spammers are distributionally similar to high-quality annotators; the minority that can be reliably identified tends to have fixed spamming behavior. as a result, methods that perform poorly on fixed spam tend to also perform poorly on real spam. our results indicate that spam detection for subjective problems flips model assumptions: spam annotators are often less random than non-spam ones. thus, attempts to remove spam can backfire by instead removing annotators with minority opinions who are not spammers. as a result, existing methods work best when only low percentages of annotators are removed based on their labeling behavior. when over-filtering for spam, these methods risk distorting the distributions of labels. ## related work methods for spammer removal impact the variation in resulting disaggregated datasets, as discussed in work on spammer detection and aggregation methods and studies underscoring the role of subjectivity and variation in annotation. ## datasets we selected two datasets for the basis of our experiments: dices 350 @xcite and @xcite 's survey of amazon mechanical turk workers. we present each dataset's statistics and discuss our dataset selection process below. dices-350 dices-350 @xcite , a harmful language dataset, consists of 43,050 annotations on a 3-point scale across 350 items, each of which was labeled by every participant. 123 annotators participated, of whom 19 annotators were labeled as spam (15% of annotators). @xcite 's survey, we used 16 questions on a 7-point scale, each of which was answered by every participant, for a total of 3,312 annotations. 207 annotators participated, of whom 40 were labeled as spam (19% of annotators). ## methods for spammer detection we study a number of established methods and baselines to calculate scores of annotator reliability. to perform spammer detection, we rank annotators using the respective reliability score and identify the k lowest-scoring annotators as spammers for a given value of k. ## results we applied mace, crowdtruth, the cohen's kappa filter, and a random baseline on both datasets, as the threshold for number of annotators removed increases. for studies in spammer detection and gold label aggregation (see section 2) the primary metric to optimize is downstream classification performance, often based on synthetic spam annotations, whereas we focus on tasks where preserving labeling variation is key. we measured the change in standard deviation, entropy, and accuracy of spammer detection for the dices-350 and mturk datasets, as well as the kl-divergence and mean absolute error of the filtered labels from the labels of non-spammers.