Annotating the Annotators: Analysis, Insights and Modelling from an Annotation Campaign on Persuasion Techniques Detection (2025)

Abstract: Persuasion (or propaganda) techniques detection is a relatively novel task in Natural Language Processing (NLP). While there have already been a number of annotation campaigns, they have been based on heuristic guidelines, which have never been thoroughly discussed. Here, we present the first systematic analysis of a complex annotation task -detecting 22 persuasion techniques in memes-, for which we provided continuous expert oversight. The presence of an expert allowed us to critically analyze specific aspects of the annotation process. Among our findings, we show that inter-annotator agreement alone inadequately assessed annotation correctness. We thus define and track different error types, revealing that expert feedback shows varying effectiveness across error categories. This pattern suggests that distinct mechanisms underlie different kinds of misannotations. Based on our findings, we advocate for an expert oversight in annotation tasks and periodic quality audits. As an attempt to reduce the costs for this, we introduce a probabilistic model for optimizing intervention scheduling.

## introduction annotated corpora are critical in natural language processing (nlp) and supervised machine learning, for training and evaluation @xcite . the quality of these annotations is thus crucial @xcite . however, datasets often exhibit label variance @xcite , due to multiple factors: multiple valid categorizations @xcite , task complexity @xcite , the concentration and the proficiency required to detect and to understand the phenomenon @xcite , annotators' susceptibility to cognitive biases, such as overconfidence, confirmation and availability bias, anchoring and halo effect @xcite . poor annotations can substantially impact machine learning algorithms' training and evaluation, especially on benchmark datasets @xcite . while many studies used multi-annotation protocols and inter-annotator agreement for reliability assessment @xcite , @xcite highlighted that this does not evaluate annotation correctness, failing to determine whether annotators consistently misunderstood label applications. @xcite called for sound theoretical definitions and effective methodological strategies to evaluate and intervene in annotation. @xcite addressed the issue by introducing a post-hoc measure for assessing the annotation correctness after task completion. yet, this does not accommodate in-itinere assessment, aimed at evaluating and enhancing label correctness during the annotation process through "decision hygiene" interventions on annotators @xcite . in this work, we analyze the annotation campaign of @xcite aiming to get valuable insights and ultimately to improve the annotation protocol. the task is a multimodal and multilabel one that requires annotating 22 persuasion techniques in memes (see section 3 for further details and figure 1 for an example of a meme). this presents three main challenges: (i) labels require theoretical expertise and extensive training, (ii) the task's cognitive demands are high due to its multimodal and multilabel nature with 22 techniques, and (iii) accurate span detection requires precise text boundary identification. our contributions are as follows: (i) we present the first study with continuous expert oversight, for the persuasion techniques detection task, that monitor the process, check the annotations, and provide feedback to the team; (ii) we examine the inter-annotator agreement as an indicator of annotation correctness, demonstrating its limitations; @xcite we analyze the impact of the feedback sessions on an-figure 1 : example of a meme @xcite ; the technique thought-terminating cliché is present in the text, while causal oversimplification can be detected by analyzing both the textual and the visual content. ## related work ground truth in machine learning (ml) and nlp has been extensively debated. @xcite argued that while some tasks can have objective ground truth (e.g, identifying verb types), other are inherently subjective or allow multiple valid interpretations, configuring disagreement as a potential resource. building on this perspective, basile (2020) and @xcite demonstrated how preserving annotation variance can enhance models' flexibility and effectiveness, e.g., for inherently ambiguous phenomena such as irony and hate speech. several frameworks have been proposed to address disagreement evaluation. prabhakaran et al. (2024) @xcite and jiang and de marneffe (2022) developed taxonomies to analyze disagreement patterns and their underlying causes in subjective tasks. expert involvement represents another common approach to ensuring annotation quality. sánchez-montero et al. (2025) compared expert and non-expert annotations of spanish metaphors, while chau et al. (2020) explored expert participation in keyphrase extraction, focusing on cost optimization. most closely related to our work, stefanovitch and piskorski (2023) analyzed inter-annotator agreement in a large-scale multilingual campaign for annotating persuasion techniques in news articles. their focus was primarily on measuring the agreement across languages and on developing new metrics for this purpose. overall, these approaches, primarily focused on post-hoc analysis, providing useful retrospective insights about misannotation reasons and mechanisms. our work differs by introducing expert supervision during the annotation process itself, moving beyond measuring annotators' agreement to assess the annotation correctness: by integrating expert oversight while the annotation campaign is ongoing, this enables continuous quality improvement through targeted feedback sessions. we can see in the top row (corresponding to sections 3.1 and 3.3) replicates the setup in @xcite , while the bottom one details how the expert supervised the campaign, training and giving feedback to the annotators. each phase is described in the following sub-sections. the annotations were outsourced to two companies (at1 and at2) with prior experience in similar tasks. both the annotators and the consolidator were fairly compensated foot_1 professionals who possessed c1/c2 levels of english proficiency and held at least a bachelor's degree. the two teams at1 and at2 analyzed a total of 9,000 and 1,000 memes, respectively. the expert checked 1,360 memes annotated by at1 and 301 by at2. for this study, we focus our analysis exclusively on the subset of memes that were annotated by both a team and the expert. ## research questions and methodology we systematized the objectives of this paper in the following research questions: rq1 is the interannotator agreement(iaa) a useful indicator of annotation correctness? rq2 do the revisions of the expert allow us to observe characteristics about the annotation quality that, otherwise, would go unnoticed? rq3 do the follow-up feedback sessions help to improve the annotators' performance? to investigate rq1, we analyzed the agreement using at1's longitudinal dataset, which provided sufficient data for temporal analysis. we calculated four types of cohen's kappa coefficients for each session: consolidated vs. expert labels: we compared the consolidated annotations to the expert labels to track the annotation quality over time. annotator average vs. expert: between each annotator and the expert labels, then averaging them to assess the pre-consolidation quality. annotator average vs. consolidated: we compared the labels by each annotator and the consolidated labels, then averaged the scores to measure the annotation changes during consolidation. inter-annotator agreement: for each annotator pair per session based on their overlapping entries, then averaging these pairwise agreements to obtain a single score per session. to answer rq2 and rq3, we defined and tracked the occurrences of errors from a taxonomy of possible errors. first, we identified and removed the correct annotations, defined as cases where the same technique was identified in both consolidated annotations and the expert review, by finding matching techniques between the annotators and the reviewers in both txt and img steps. second, we analyzed the remaining misannotations as follows: substituted annotation (sub): when both the consolidated annotation and the expert identified a technique in the same text span (75% overlap), but disagreed on which specific technique was present (e.g., loaded language vs. name calling), requiring replacement. substitutions were not considered for the annotations in the image step, since no overlap is possible. after handling text span overlaps through sub errors, subsequent error analysis focused solely on technique presence/absence, treating the task as a multilabel classification problem. incorrect annotations (inc) occurred when a technique present in the consolidated annotation was deemed absent by the expert. these were identified by comparing techniques present in annotators' labels but absent in the expert's. missed annotations (mis) represented cases where the expert identified a technique that was absent in the consolidated annotation. these were detected by finding techniques present in the expert's labels but absent in the annotator's. ## toward efficient revision scheduling via probabilistic modeling of error rates our analysis demonstrates that expert supervision can improve annotation quality, particularly for incorrect annotations (section 5.3). however, expert involvement is resource-intensive, raising the question of how to optimize revision and intervention timing while maintaining good quality. we propose a probabilistic framework for adaptive scheduling based on annotators' evolving error rates. the key insight is that as annotators learn from expert feedback, their error probability should decreases. let us start by formalizing the annotation setting (we will inevitably simplify it to keep the model manageable). in the first phase, each annotator analyzes @xmath0, then curates them with a consolidator, resulting in a final error probability @xmath1, 1]. an erroneously identified item implies a cost @xmath2. the cost could be interpreted in broader terms than economical: for the model to be sound, it needs a disincentive to make an error; otherwise, there would be no gain in having the expert supervise the annotations. the number of undetected errors, say x, is distributed according to a binomial distribution with parameters n and p. after the n sentences, additional @xmath3, this time also checked by an expert, who receives a compensation @xmath4. the expert identifies and corrects all errors in the m annotations. the number of errors found, say z, are trivially distributed as an independent binomial distribution with parameters m and p. the z errors are shown and discussed with the annotators and the consolidator, allowing them to improve their ability to perform the task, reducing the error rate p. we model such update with the function we assume the interest rate is @xmath5, 1) 5 , and we call t (p) the average total cost incurred in this improvement process. the average total cost satisfies the following recurrent equation: figure 7 shows the values of t (p) and n * (p) as a function of the error rate p. note that the values on the y-axis are not expressed in terms of examples: in the model, we assumed that the errors have a binomial distribution, and, since our problem is multiclass multilabel, for each example, we take multiple decisions, and therefore we can make multiple errors as a result. one example therefore corresponds to multiple units on the yaxis. we chose for an example to correspond to the average number of the gold labels plus one (to account for all non-gold-labels wrongly annotated) units, but any other sensible choice would have led to a similarly shaped plot. if the annotators have an error @xmath6.8 then the expert should intervene after n * (0.8) = 294 decisions (roughly corresponding to 80.08 examples in our corpus) with an estimated cost t (0.8) = 5167.23. here, we consider the values of the parameters a, b, c, d as indicated in the caption of figure 7 , and we assume that p would not change. in practice, p would actually change after the intervention by the expert annotator, therefore resulting in new (lower) values of n * (p) and t (p). in our experimentation, the expert annotator intervened at fixed regular times for a cost of 5,469. by applying the model and thus intervening at different time intervals according to n * (p), the estimated cost t (p) would be 4,716. we would like to note that this comparison should by no means be interpreted literally. to have a fair comparison, we should have repeated the annotation campaign in the two different settings, and eq. ( 1 ) should be a good model of the learning curve of the annotators. while it is infeasible to redo the annotation campaign, in figure 8 , we plot the actual values of p computed on the data we collected. the overall decreasing trend in error rates aligns with our proposed model of annotators' learning through expert feedback, as formalized in eq. ( 1 ). however, notable spikes in error rates occur at interventions 7 and 11, corresponding to the peaks previously identified in figure 5a and figure 5b , respectively. as discussed in section 5.3, these temporary increases in errors stem from different underlying causes. if the expert intervention strategy could be optimized to promptly identify these errors (the plot is currently based on the actual interventions that were made at regular intervals), the learning curve would likely show a smoother decrease in error rates, more closely matching our theoretical model in eq. ( 1 ). ## conclusion and future work we examined dataset annotation quality through a case study of persuasion techniques in memes @xcite . our work extends the ongoing discussion about annotation error detection in machine learning @xcite @xcite , emphasizing the need to move beyond inter-annotator agreement to assess annotations correctness. the task of persuasive memes detection underscores the importance of such evaluations, particularly when computational models address misinformation that influences public opinion @xcite , making dataset integrity both a technical requirement and a matter of political and social accountability @xcite . a key contribution of our work is the introduction of a weak perspectivist-based annotation protocol @xcite (allowing annotators to merge their individual assessments during consolidation discussion) integrated with expert supervision during the annotation campaign, enabling continuous correctness management through systematic revisions and feedback. we addressed three research questions (section 4). our analysis of rq1 shows that interannotator agreement does not correlate with annotation correctness, challenging its use as a sole quality metric. however, discussions among annotators significantly improved annotation correctness, likely due to enhanced awareness of edge cases and collective reasoning strategies. results from rq2 revealed distinct error patterns, while through rq3 we showed how expert supervision effectively addressed incorrect annotations through targeted feedback, missed annotations remained more resistant to improvement across both annotation teams, suggesting systematic cognitive limitations rather than team-specific issues. additionally, expert supervision proved effective for improving annotation quality, but remains resource-intensive. to address this constraint, we introduced a probabilistic model that optimizes the frequency of expert interventions while maintaining annotation quality. as future work we plan to extend the model to differentiate between incorrect and missed annotations and customize it to individual annotator characteristics, such as learning curves. finally, we plan to integrate our framework into annotation platforms to enable real-time quality monitoring and automated intervention strategies.