Calibration as a Proxy for Fairness and Efficiency in a Perspectivist Ensemble Approach to Irony Detection (2025)

Abstract: Identifying subjective phenomena, such as irony in language, poses unique challenges, as these tasks involve subjective interpretation shaped by both cultural and individual perspectives. Unlike conventional models that rely on aggregated annotations, perspectivist approaches aim to capture the diversity of viewpoints by leveraging the knowledge of specific annotator groups, promoting fairness and representativeness. However, such models often incur substantial computational costs, particularly when fine-tuning large-scale pre-trained language models. We also observe that the fine-tuning process can negatively impact fairness, producing certain perspective models that are underrepresented and have limited influence on the outcome. To address these, we explore two complementary strategies: (i) the adoption of traditional machine learning algorithms—such as Support Vector Machines, Random Forests, and XGBoost—as lightweight alternatives; and (ii) the application of calibration techniques to reduce imbalances in inference generation across perspectives. Our results demonstrate up to 12× faster processing with no statistically significant drop in accuracy. Notably, calibration significantly enhances fairness, reducing inter-group bias and leading to more balanced predictions across diverse social perspectives.

## introduction in subjective tasks, such as hate speech or irony detection, (text) classification depends on cultural knowledge and the individual impact of the speech on each individual @xcite . an inherent characteristic of this type of problem is label disagreement (e.g., hate vs. non-hate or ironic vs. non-ironic) @xcite . therefore, modeling the individuality of perception, reflected in the labels, can provide valuable information for the task of automatic hate speech detection (classification). traditional classification methods aggregate multiple annotations through strategies such as choosing the majority class and discarding minority or less representative views @xcite . the proposal of perspectivism @xcite is to preserve multiple annotations to capture different views, promoting fairness between the models @xcite . by training independent models per cultural group, each reflecting specific interpretations, the cultural diversity in the data is considered. a desirable consequence of the perspectivist approach is the mitigation of biases against historically marginalized groups, such as lgbtq+, black, and religious minorities, among others @xcite . in particular, in @xcite , a perspectivist method is proposed, combining (or ensembling) models fine-tuned by each perspective, whose results indicate promising combinations. despite the good effectiveness of the results, fine-tuning multiple language models imposes high computational demands. in this context, this work has two central objectives. the first objective is to enhance the efficiency of the perspectivist approach proposed by @xcite -hereinafter referred to as confidence-based ensemble (cem) -through integration with traditional machine learning models, aiming to maintain effectiveness while reducing computational cost. the second is to improve the fairness between perspectivist models through calibration techniques. in the base method, it was observed that some perspectives showed low representativeness (low confidence in predictions), which limits or makes their contribution to the final label unfeasible, compromising the fair principle of perspectivism. we hypothesize that this effect results from miscalibration. in a properly calibrated classification model, the posteriori probability estimated by the classifier should present a higher correspondence with the empirical frequency of hits. thus, a calibration step was incorporated to increase the reliability of the methods. the experimental results demonstrate that combining the cem with traditional models reduces execution time by up to 12 times without a statistical loss in effectiveness. we also demonstrate, by means of a sustainability metric that integrates effectiveness and carbon footprint, that a reduction of over 34% is achievable using traditional models (e.g., logistic regression) as the classifier. finally, calibration promotes a greater balance in the contribution of the different perspectives in the final result, generating fairer models from a perspectivism viewpoint. indeed, this approach significantly improves the alignment between each perspective's contribution, shifting the contribution distribution closer to its actual perspective´s representations in the training data and reducing unfair imbalances introduced by miscalibrated probabilities. compared to the original method, the calibrated approach achieved a relative improvement of approximately 39% in fairness, yielding more balanced outcomes across diverse social and linguistic groups while preserving competitive performance. these findings highlight the value of calibration as a key mechanism for ensuring equity in perspectivist modeling. the rest of the paper is organized as follows. section 2 covers related work. section 3 details the proposed approach. section 4 presents the experimental protocol and discusses the experimental results. section 5 concludes the paper. ## related work in subjective nlp tasks -such as detecting hate speech, irony, sentiment, and abusive language -obtaining multiple rater annotations is often necessary due to the inherent ambiguity and variability of human judgment @xcite . in traditional approaches, disagreements between annotators are frequently treated as noise @xcite , with the final label determined by a majority vote scheme that disregards the perspectives of potentially affected minority groups @xcite . in contrast, the perspectivist approach advocates valuing this diversity by explicitly modeling individual variations rooted in demographic and cultural characteristics @xcite . this paradigm has gained prominence amid growing demands for fair, inclusive, and bias-aware nlp models @xcite @xcite . several recent studies have operationalized this concept in practice. in @xcite , for example, the authors divided the training data into distinct subsets aligned with specific social or demographic groups (e.g., male and female annotators), fine-tuning a dedicated language model for each group to capture their characteristic patterns. the individual outputs were then combined through a confidence-based ensemble method, yielding a final prediction. similarly, @xcite proposed an approach that explicitly incorporates the target group of an ironic statement by leveraging a dual-module architecture: gpt-2 to identify the group at which the statement is aimed, and roberta to estimate the annotators' scores, with both models adjusted for the specific classification task. meanwhile, @xcite introduced a technique that captures individual annotators' patterns by concatenating texts associated with the same annotator and including this information alongside the input for the language model, thereby embedding the annotators' belief profiles within the prediction process. in machine learning, model bias can lead to unfairness and discrimination against specific groups @xcite . calibration approaches ensure that the balances of positive predictions align with the proportions of positive examples in the training set @xcite . see kheya et al. (2024) for a survey on methods to reduce the bias. platt scaling, for example, is a widely used calibration technique that adjusts a model's output scores into well-calibrated probabilities using a logistic regression model, thereby promoting fairer outcomes @xcite . in recent work, ? integrates ensemble-based uncertainty estimation with calibration constraints using a multi-objective loss function to address fairness and calibration jointly. taken together, these works underscore the growing focus in nlp on recognizing, preserving, and leveraging the richness of diverse human perspectives, yielding advances in both the fairness and reliability of models applied to highly subjective and context-dependent tasks. that said, to the best of our knowledge, no prior study has examined the impact of calibration in perspectivism or its influence on the accurate and fair representation of social dimensions in the resulting models. ## proposed approach in this section, we describe how traditional machine learning techniques can be effectively combined with a perspectivist approach to improve computational efficiency. in addition, we present the incorporation of a calibration step designed to promote greater fairness across perspectives, ensuring that their contributions to the final prediction align more closely with their distribution in the training data. figure 1a illustrates the perspectivist approach (cem) introduced in casola et al. (2023 note that the confidence score is computed using the difference between the output probability of the model. in place of the fine-tuned language models used in the original method. since these algorithms require fixed-length numerical inputs, a pre-trained language model (in this case, roberta) is used as an encoder exclusively to extract features (i.e., using the average from the last four layers), leveraging its ability to encode complex syntactic and semantic patterns into dense vector representations. the generated embeddings are then used as inputs for training traditional classifiers, allowing the approach to leverage a rich data representation while significantly reducing computational overhead. all subsequent steps remain identical to those defined in the baseline approach, preserving the overall structure of the pipeline while making it more computationally efficient and broadly applicable 1 . the calibration procedure employed in this study is based on platt scaling, a widely adopted post-processing technique that leverages logistic regression to recalibrate the output scores or probabilities generated by base classifiers @xcite . this method addresses the common issue of miscalibrated probability estimates in machine learning models, where raw output scores (often referred to as logits) do not correspond well to true class membership likelihoods. we also explore the isotonic calibration approach, which is a nonparametric method that avoids making assumptions about the form of the relationship between the model's scores and the true probabilities @xcite . however, as noted by ojeda et al. (2023), a significant drawback of isotonic calibration is its propensity to overfit, which necessitates a larger calibration set to mitigate this risk. concretely, platt scaling involves fitting a parametric sigmoid function to the scores produced by the classifier on a validation set, distinct from the training data used to build the model. the function is defined as: in the context of perspectivist models, where multiple classifiers trained on distinct annotator 1 test data undergoes the same encoding process using the zero-shot language model, ensuring representation consistency and that there is no influence of the training data on the generation of the test representation and vice-versa. calibration introduces an additional step (step 2.c) that adjusts the probabilities generated by each prediction model to ensure they are on comparable scales. this adjustment prevents any single uncalibrated perspective from disproportionately dominating the label assignment process. in figure 1b , calibration uses the probabilities derived from inference on the validation set (indicated by the green arrow). importantly, step 2.c is orthogonal to the model type and can be applied regardless of whether the underlying classifier is a language model or a traditional machine learning algorithm. this flexibility allows calibration to enhance the reliability and fairness of the ensemble predictions without altering the base classifiers. ## experiments we report the experimental results corresponding to the two primary research objectives: (1) to quantify the computational efficiency gains achieved by combining zero-shot roberta for tokenization with traditional machine learning classifiers instead of the roberta finetuning process and (2) to evaluate the effects of calibration on enhancing fairness within the perspectivist framework. experiments were conducted on a computing environment comprising an amd 2990wx processor (64 threads, 3 ghz), a geforce rtx 2080 gpu (8 gb), and 128 gb of ram. the source code supporting this work will be made publicly available in the repository at https://...[to be released upon acceptance]. we begin by describing the experimental protocol, including details of the dataset, the evaluation metrics utilized, and our novel metric designed to quantify fairness in perspectivist classification scenarios. ## conclusions we propose integrating traditional classification methods as a way to simultaneously foster greater fairness and improved computational efficiency within recent perspectivist approaches. our results demonstrate that, due to miscalibrated probabilities, the method introduced by @xcite tends to produce biased outcomes, under-representing certain perspectives and, as a result, falling short of its central objective of promoting inclusivity and equity. to mitigate this limitation, we incorporated a calibration step as an orthogonal layer, allowing the model to more accurately align its final prediction distribution with the actual representation of each group in the training data. this adjustment not only improves balance across perspectives, yielding a fairer and more representative outcome, but also achieves competitive levels of effectiveness when compared with state-of-the-art approaches. in this way, the proposed method advances the state of the art by reconciling the often competing demands of efficiency, performance, and fairness. looking ahead, we intend to investigate the application of our framework to other perspectivist datasets, exploring a broader range of social, linguistic, and cultural contexts. also, more recent language models, including llama and its variants, as well as modern bert-based architectures. we want to dig deeper into the reasons why certain perspectives seem to dominate the ensemble's decision, not reflecting their ideal contributions. we will also evaluate other supervised stacking techniques @xcite as a means to further optimize effectiveness while reducing computational overhead and improving fairness even further, thereby supporting the design of more equitable and resource-efficient nlp systems.