Exploring the Cost-Effectiveness of Perspective Taking in Crowdsourcing Subjective Assessment: A Case Study of Toxicity Detection (2025)

Abstract: Crowdsourcing has been increasingly utilized to gather subjective assessment, such as evaluating the toxicity of texts. Since there doesnot exist a single “ground truth” answer for subjective annotations, obtaining annotations to accurately reflect the opinions of differentsubgroups becomes a key objective for these subjective assessment tasks. Traditionally, this objective is accomplished by directly soliciting a large number of annotations from each subgroup, which can be costly especially when annotators of certain subgroups are hard to access. In this paper, using toxicity evaluation as an example, we explore the feasibility of using perspective taking—that is, asking annotators to take the point of views of a certain subgroup and estimate opinions within that subgroup—as a way to achieve this objective cost-efficiently. Our results show that compared to the baseline approach of directly soliciting annotations from the target subgroup, perspective taking could lead to better estimates of the subgroup-level opinion when annotations from the target subgroup is costly while the budget is limited. Moreover, prompting annotators to take the perspectives of contrasting subgroups simultaneously can further improve the quality of the estimates. Finally, we find that aggregating multiple perspective-taking annotations while soliciting a small number of annotations directly from the target subgroup for calibration leads to the highest-quality estimates under limited budget.

## introduction crowdsourcing has become a ubiquitous paradigm for obtaining annotated data from people to enhance machine intelligence in a scalable and costeffective manner. recent research has shown that some crowdsourcing annotation tasks, like toxicity and political stance evaluation @xcite @xcite , are fundamentally subjective-annotations on these tasks will be influenced by annotators' identities, preferences, and personal opinions. for these subjective annotation tasks, a universal "ground truth" annotation does not exist by definition, since each annotator can establish their own "ground truth" for a task based on their subjective interpretations of the task. as such, a key objective for subjective annotation tasks is to accurately assess the distribution of annotations within different subgroups of annotators with varying characteristics. for instance, in the context of toxicity evaluation, a natural goal for collecting crowdsourced annotations is to understand the subgroup-level opinion, such as the fraction of people in different subgroups who will consider the text as toxic. traditionally, to achieve this goal, one can poll opinions directly within a subgroup by asking annotators in this subgroup to report their own annotation to the task (e.g., "do you think this comment is toxic?"; we refer to this as the "direct annotation"). however, this approach suffers from two major limitations. first, to get a reasonably precise estimation of the subgroup-level opinion for a target subgroup, one may need to solicit a large number of direct annotations from annotators of this subgroup, which requires a high annotation budget. moreover, annotators of the target subgroup can sometimes be difficult to get access to (e.g., when the target subgroup is under-represented in the annotator population or overloaded); thus, soliciting annotations from such subgroup may trigger additional administrative cost (e.g., for identifying annotators in the target subgroup). given these limitations, one may naturally ask if crowdsourcing subjective assessment can be conducted more cost-effectively. in this paper, we investigate a perspective-taking annotation approach for crowdsourcing subjective tasks-instead of providing their own annotations to a task, annotators are asked to actively take the view points of people from a target subgroup and then estimate the opinion statistics for that subgroup. this approach has the potential to increase the cost-effectiveness of crowdsourcing subjective assessment, because it involves direct estimation of the subgroup-level opinion (hence relax the requirement for a large number of annotations), and the annotation could be obtained from anyone regardless of their group identity (hence may decrease the cost for identifying hard-to-access annotators). to explore whether the perspective-taking annotation approach can indeed improve the costefficiency of crowdsourcing subjective assessment, we conduct a case study on the toxicity evaluation tasks. participants of our study were recruited from prolific to evaluate the toxicity of online comments that are targeted at males or females, and we consider female and male annotators as the two subgroups of interests. to establish the values of the subgroup-level opinion statistics of interestswhich are the female/male toxicity rate of comments (i.e., the percentage of females/males who consider a comment as toxic)-we first conducted a pilot study. in the pilot study, we adopted the direct annotation approach and asked at least 50 female (male) annotators to provide their own toxicity annotations on each comment in order to compute the female (male) toxicity rate for the comment. then, in the formal experiment, participants were recruited to take the perspective of a target subgroup and directly estimate the toxicity rate for that subgroup. results of our study show that when the cost of soliciting annotations from different subgroups is the same, the perspective-taking annotation approach often results in worse estimates of the subgroup-level opinion than the traditional, direct annotation approach, as measured by the mean squared errors (mse) of the estimates. this is because compared to the direct annotation approach, perspective taking often leads to estimates with higher bias despite the variance is reduced. however, when the cost of soliciting annotations from the target subgroup becomes higher, using the (cheap) perspective-taking annotations from annotators outside of the target subgroup could lead to higher-quality estimate of the subgroup-level opinion than the traditional approach, especially when the annotation budget is limited and annotators take contrasting perspectives simultaneously. finally, we show that with a limited annotation budget, the highest-quality estimates of the subgrouplevel opinion of a target subgroup can be obtained by aggregating multiple (cheap) perspective-taking annotations from annotators outside of the target subgroup, while using a small number of (expensive) direct annotations from annotators within the target subgroup for calibration foot_0 . ## related work early research and practice of crowdsourcing often view annotation tasks as objective, for which there exists the notion of "truth" in gold standard annotation. as such, the deviation of an annotation from the gold standard is often interpreted as a systematic error. this "error" can be caused by the lack of skills of the annotators, which inspires research on improving the quality of crowdsourced annotation by assigning the tasks to workers with suitable skills @xcite , or aggregating annotations from multiple workers while taking their skills into consideration @xcite @xcite . more recently, it has been recognized that such error can also reflect the "bias" resulted from the annotator's own cognitive bias @xcite or the task designs @xcite . as a result, a variety of methods have been proposed to mitigate biases in crowdsourcing tasks. for example, @xcite proposed to mitigate biases by raising annotators' awareness of biases. different task designs were explored to encourage deeper deliberation from annotators and mitigate their biases @xcite @xcite @xcite . novel algorithms were designed to take worker bias into account during the label aggregation process @xcite . in contrast, the most recent research has advocated for the view that some crowdsourcing annotation tasks are inherently subjective. for example, studies showed that annotators' demographics, identities, and beliefs impact the way they determine their annotations in hate speech detection tasks @xcite @xcite . @xcite also found that for fine-grained sentiment analysis, annotators' demographics have a significant impact on their annotations. therefore, it is believed that for subjective annotation tasks, there is a need to embrace the diverse human interpretations and capture the broad spectrum of opinions and perspectives @xcite . towards this goal, @xcite highlighted the need for transparent documentation of crowdsourced dataset, including recording who the annotators are for crowdsourced annotations. @xcite developed ml algorithms to model individual annotators and visualize the annotation disagreement within a group of annotators. in this study, we focus on understanding how to estimate the subgroup-level opinions for crowdsourced subjective annotation tasks in a more costefficient way. to this end, we explore the feasibility of a novel annotation approach, i.e., engage annotators in perspective taking in their annotation tasks. perspective taking is the act of perceiving and comprehending a situation by taking on the viewpoint of another person's psychological experience (i.e. thoughts, feelings, and attitudes) @xcite ; it may allow the perspectivetaker to better represent others' states and focus more on expressive cues that communicate information about the feelings of others @xcite . in our study, unlike the typical perspective taking approach where people are asked to take the perspective of a single person, we prompt annotators to take the perspective of a subgroup. moreover, while previous research proposed the "vicarious annotation" method, asking annotators to directly estimate the annotation for a subgroup @xcite , the perspective taking annotation method explored in this study has a key difference-the vicarious annotation method assumes a subgroup as a collection of homogeneous individuals who share the same thinking and solicits a single annotation for a subgroup, while the perspective-taking annotation approach acknowledges that even for people within the same subgroup, their opinions may differ (for subjective judgements), so what is being solicited is the annotation statistics within the subgroup. while in this work, we choose to use toxicity evaluation as an example to examine the effectiveness of the perspective-taking annotation approach in estimating subgroup-level opinion, the approach itself is general and can be applied to understanding subgroup-level opinions for other subjective assessment tasks. finally, another concept that is related to perspective taking is "common sense." according to whiting and watts (2024), "common sense" refers to those claims that almost everyone agrees (or disagrees), and everyone knows almost everyone else agrees (or disagrees). in this sense, we note that a claim about the annotation statistics in a subgroup (e.g., "50-55% of people in the subgroup will consider this comment as toxic") is commensensical is neither a sufficient nor a necessary condition for the perspective-taking annotations to be effective for estimating subgroup annotation statistics. this is because (a) it is possible that everyone agreed on a commonsensical claim that is incorrect (i.e., commonsensical does not imply effective perspective-taking), and (b) it is also possible that each individual's perspective-taking annotation has independently random errors, thus they do not agree on any claim yet accurate estimates of subgroup annotation statistics can be produced using their perspective-taking annotations due to the wisdom of the crowd phenomenon @xcite (i.e., effective perspective-taking does not imply commonsensical). ## study design to explore if the perspective-taking annotation approach can help estimate the subgroup-level opinions for subjective annotation tasks cost-effectively, we conduct a human-subject study on the tasks of toxicity evaluation. ## results a total of 546 participants participated in our pilot study and passed the attention check. among these participants, 274 were female annotators and 272 were male annotators. as a result, we collected a total of 13,080 direct annotations in the pilot study. in addition, 258 participants participated in our formal experiment (see the average number of annotations we collected from female and male annotators, respectively, on each comment, for both the pilot study and the formal experiment. we used the data collected from the pilot study to compute the female/male toxicity rate for each comment and treated them as our ground-truth subgroup-level opinion. we then analyzed the data obtained from the formal experiment to understand whether leveraging perspective taking improves the cost-effectiveness of crowdsourced subjective assessment under varying annotation budgets. specifically, in the context of toxicity evaluation, forming an estimate of the subgroup-level opinion (e.g., subgroup-level toxicity rate) for a target subgroup x requires the construction of an estimator. traditionally, the baseline estimator can be obtained by soliciting n direct annotations from annotators of subgroup x (i.e., they each answer "is this comment toxic?") and then computing the fraction of annotators among them who rate the comment as toxic, with each annotation costing c x . on the other hand, the perspective-taking-based estimator can be obtained by soliciting n perspectivetaking annotations from any annotator (i.e., they answer "what percentage of people in subgroup x will rate this comment as toxic?") and averaging their reported fractions, with each annotation from annotators of subgroup g costing c g . for simplicity, we assume that the cost for soliciting annotations from any annotator outside of the target subgroup x (i.e., c x ) is the same, and c @xmath0. note that as long as an annotation is solicited from the target subgroup x, it costs c x regardless of whether the annotation is a direct annotation or perspective-taking annotation. figure 1 compares the quality of the two estimators when the annotation budget varies (i.e., when n varies), both for the case when annotations from all subgroups are equally costly (β = 1, figure 1a-1c ) and when annotations from the target subgroup are more costly than those from the non-target subgroup (β = 10, figure 1d-1f ). we make the following important observations: perspective-taking is worse than direct anno-tation approach in estimating subgroup-level opinions when annotations from all subgroups are equally costly, mainly because it leads to higher bias. figure 1a -1c presents the comparisons when the annotation cost for annotators from the target subgroup is the same as other subgroups (i.e., β = 1). we find that the subgroup-level opinion estimates derived from perspective-taking annotations always exhibit a higher level of mse than the estimates derived from direct annotations, regardless of the level of annotation budget (figure 1a ). a closer look suggests that while the perspective-taking-based estimator consistently shows a lower level of variance than the baseline estimator (figure 1c ), it suffers from a much higher level of bias (figure 1b ) foot_2 , which is the main contributor to its high mse. interestingly, we also find that aligning the perspective that an annotator takes with their own group identity, tends to decrease the bias and the mse of the perspective-taking-based estimator (see the comparison between the blue and green curves in figure 1a-1b ). perspective-taking can lead to higher-quality estimates than direct annotation approach when annotations from the target subgroups are costly while the annotation budget is limited. figure 1d -1f show the comparisons when soliciting annotations from the target subgroup is much more costly. in this scenario, we find that obtaining perspectivetaking annotations from annotators outside of the target subgroup shows a degree of advantage over the direct annotation approach, when the annotation budget is limited. for example, as shown in figure 1d , when the annotation budget @xmath1, instead of soliciting a very small number of costly (e.g., c @xmath2, one may obtain a more accurate estimate (i.e., an estimate with lower mse) of the subgroup-level opinion by collecting cheap (e.g., c @xmath3. prompting annotators to take contrasting perspectives simultaneously can further improve the quality of the estimates. we compare the bias and mse of the perspective-taking-based estimator when non-target subgroup annotators were asked to only take the perspective of the target subgroup, versus when they were asked to take the perspectives of multiple subgroups simultaneously (see the comparison between the green and brown curve in figure 1a and figure 1b ). we find that taking multiple perspectives simultaneously results in a less biased estimate of the subgroup-level opinion for the target subgroup and the decrease in bias also leads to a decreased level of mse. as such, when annotations from the target subgroup is costly, by asking annotators outside of the target subgroup to evaluate the perspectives of multiple subgroups simultaneously, the resulting perspective-takingbased estimator of subgroup-level opinion may exhibit advantage over the baseline estimator for a wider range of annotation budget. ## post-hoc processing: label aggregation so far, the perspective-taking-based estimator we constructed is simple-we just took an average of the n perspective-taking annotations obtained to get an estimate of the opinion for the target subgroup. however, research on crowdsourcing label aggregation @xcite suggests that we may further improve the quality of the estimates by cleverly combining multiple perspective-taking annotations together and inferring the ground-truth value for the subgroup-level opinion. in particular, the high bias presented in perspective-taking annotations suggests that it may be helpful to supplement the perspective-taking annotations with a small number of direct annotations from the target subgroup to "calibrate" them. thus, in this section, we explore if we can design post-hoc label aggregation algorithms to improve the quality of the subgroup-level opinion estimates by combining some potentially costly direct annotations from the target subgroup with many more cheap perspective-taking annotations produced by annotators outside of the target subgroup. ## conclusions in this paper, we introduce a novel approach of leveraging perspective taking to characterize the subgroup-level opinion for subjective annotation tasks. we conduct an experimental study, using toxicity evaluation tasks as an example, to explore the cost-effectiveness of this approach. results of our experiment show that compared to the baseline approach of directly polling annotators of a target subgroup to estimate the opinions within that subgroup, estimation obtained from perspectivetaking annotations generally exhibits lower variance but higher bias. as such, the perspectivetaking-based estimator of the subgroup-level opinion only shows lower mean squared error than the traditional, direct-annotation-based estimator when soliciting annotations from the target subgroup is costly yet the annotation budget is limited. however, we find a approach to further improve the costeffectiveness of the perspective-taking annotation approach by using the expectation-maximization algorithm to aggregate multiple cheap perspectivetaking annotations while using a small number of costly direct annotations from the target subgroup for calibration.