Weak Ensemble Learning from Multiple Annotators for Subjective Text Classification (2025)

Abstract: With the rise of online platforms, moderating harmful or offensive user-generated content has become increasingly critical. As manual moderation is infeasible at scale, machine learning models are widely used to support this process. However, subjective tasks, such as offensive language detection, often suffer from annotator disagreement, resulting in noisy supervision that hinders training and evaluation. We propose Weak Ensemble Learning (WEL), a novel framework that explicitly models annotator disagreement by constructing and aggregating weak predictors derived from diverse annotator perspectives. WEL enables robust learning from subjective and inconsistent labels without requiring annotator metadata. Experiments on four benchmark datasets show that WEL outperforms strong baselines across multiple metrics, demonstrating its effectiveness and flexibility across domains and annotation conditions.

## introduction harmful information, such as offensive and abusive language, has been known as one of the main threats on social media platforms. typically, the moderation of online harmful information is conducted manually. with an increasing amount of information, manual moderation is expensive and insufficient. there is a growing demand for developing a natural language processing (nlp) tool to support the detection and mitigation of harmful content on online platforms. addressing this challenge requires high-quality annotated data to train accurate and reliable machine learning models. in recent years, social media has become a popular source for data collection, and crowdsourcing has emerged as a widely used solution for largescale data annotation. however, concerns have been raised about the reliability of crowdworkers, particularly in complex linguistic tasks where annotators often lack domain-specific training @xcite . furthermore, the incentive structures of crowdsourcing platforms can encourage rapid completion of tasks with careful judgment, potentially compromising label quality @xcite @xcite . a particularly challenging issue in this context is human label variation @xcite , which arises when annotators assign different labels to the same instance. this is especially common in subjective tasks such as emotion detection @xcite and offensive language detection @xcite , where annotation involves personal interpretation, contextual nuance, and cultural perspective. unlike objective tasks with clearly defined ground truth, subjective annotations inherently invite disagreement. such variation introduces noise into training data, complicates evaluation, and challenges the assumption of a single "correct" label @xcite . understanding and modelling this variability is critical for developing nlp systems that are more robust, interpretable, and aligned with the diversity of human judgement. previously, several methods have been proposed to address this issue by estimating and incorporating annotator reliability into the modelling process @xcite @xcite . these approaches typically assign higher weights to labels provided by more consistent or trustworthy annotators, aiming to reduce the influence of noisy or unreliable inputs on the final model. however, their effectiveness is often limited by the composition of the annotator pool. they require sufficient diversity among annotators to model reliability accurately and may risk overfitting when such diversity is lacking or when the model overrelies on a small subset of annotators @xcite . we aim to develop a method applicable to more general multi-annotation settings. specifically, the proposed approach is designed to function effec-1 87 tively when annotators are shared across the entire dataset or when there is a heterogeneous distribution of annotator workload (i.e., some annotators contribute more than others). while prior approaches often rely on a single loss function, such as cross-entropy (ce) @xcite , to train classification models, this may be insufficient for subjective tasks where both hard and soft supervisory signals are informative. in such settings, different loss components capture complementary aspects of learning: ce supports probabilistic calibration, f1 loss promotes classification accuracy on hard labels, and distributional losses like mean absolute error or manhattan distance (md) @xcite help align predictions with the soft label distributions reflecting annotator disagreement. by jointly optimising these objectives, we can balance predictive accuracy with nuanced representation of label uncertainty, leading to more robust and interpretable models. our contributions can be summarised as follows: ## related work subjective nlp tasks such as offensive language detection, hate speech classification and emotion analysis often suffer from high variability in human annotations. annotators may interpret linguistic cues differently based on their personal, cultural or contextual backgrounds @xcite . this subjectivity introduces label noise and inconsistency, making it challenging to define a single ground truth (i.e., a hard label). in particular, datasets annotated via crowdsourcing tend to reflect these disagreements, raising questions about how best to represent and learn from multiple perspectives @xcite . a common approach to address label disagreement is to replace hard labels with soft targets, usually probability distributions over classes derived from annotator votes, and train models using probabilistic loss functions. the most prevalent is the cross-entropy loss, which treats soft distributions as targets, encouraging models to reflect label uncertainty rather than force a single decision @xcite . more recent methods have proposed alternative loss formulations, such as kullback-leibler divergence, expected calibration error @xcite and manhattan distance @xcite . these techniques aim to improve robustness to noisy or subjective labels by preserving the signal in disagreement rather than collapsing it through majority vote. ensemble methods have also been explored as a way to leverage annotator disagreement rather than suppress it. instead of aggregating labels before training, several works train separate models for each annotator and combine their predictions during inference @xcite @xcite . this strategy captures the full range of annotator perspectives and has shown promise in capturing subjective variation in tasks like emotion classification and hate speech detection. however, these models may suffer from scalability issues, especially when the number of annotators is large or unbalanced. other work has approached the problem from a probabilistic modelling perspective, estimating annotator reliability as a latent variable during training @xcite . these approaches often combine annotator-specific models with global learning signals, aiming to balance personalised and consensus-based predictions. in addition, instance weighting has been used as a practical solution to reduce the influence of unreliable or biased supervision. for instance, @xcite apply instance reweighting to mitigate demographic bias in toxicity detection, while @xcite introduce dynamic instance weighting to adapt to concept drift in evolving datasets. @xcite and @xcite proposed to compute individual annotator ratings and combine this information to better capture the subjectivity inherent. these methods adjust the learning signal based on 2 example-level characteristics, enabling models to better generalise under noisy or imbalanced conditions. our work unifies ensemble-based disagreement modelling. we extend ensemble methods that capture annotator disagreement by randomly sampling weak predictors to simulate diverse viewpoints, and by embedding annotator-specific models whose ensemble selection is learned end-to-end rather than relying on a fixed set as in @xcite . in contrast to probabilistic reliability estimation techniques that depend on annotator metadata @xcite , our framework requires no such information, broadening its applicability. at the ensemble level, we adapt instanceweighting strategies to emphasise predictor utility and mitigate dataset bias @xcite . drawing on label distribution modelling, our loss function blends soft and hard supervision to achieve both nuanced learning and interpretability @xcite . these elements yield a scalable and flexible approach for managing noisy subjective annotations. ## methods given a dataset annotated by multiple annotators, the goal is to learn a predictive function that accounts for the variability and potential noise introduced by differing annotator judgments. let @xmath0, y (1) i , . . . , y to address this challenge, we propose a threestage method, named weak ensemble learning (wel), designed to learn from multiple annotators while accounting for disagreement and annotator variability. first, we construct a set of weak predictors by employing a random sampling and a topranked annotators selection strategies (section 3.1). second, we aggregate the outputs of weak predictors using a weighted ensemble, where the weights will be tuned to balance contributions in the next stage, enabling the model to leverage diverse anno-algorithm 1 weak ensemble learning (wel) ## experiments to evaluate the performance of our method across diverse domains and text genres, we use @xcite , and md-agreement @xcite . each dataset includes multiple annotations per instance, with at least two annotators per instance sample. to maintain the generalisability of our approach, we exclude domain information and annotator metadata during training. all models are trained solely on input text and its associated labels. summary statistics are provided in ## conclusions in this paper, we introduced weak ensemble learning (wel), a flexible framework for subjective text classification that learns from multiple annotations by constructing diverse weak predictors and jointly optimising their contributions. we explored two variants: wel-random, which captures annotator variation through random label sampling, and wel-topann, which models the most frequent annotators individually. experiments on four datasets showed that wel consistently outperforms baselines, with wel-random excelling in hard-label classification and wel-topann offering advantages in distributional alignment when annotator consistency is high. future work will integrate annotator profiles and reliability estimates into a unified neural architecture to improve performance and efficiency, and extend wel to larger annotator pools and multilingual contexts.