Insights into developing analytical categorization schemes: three problem types related to annotation agreement (2025)

Abstract: Coding themes, frames, opinions and other attributes are widely used in the social sciences and doing that is also a base for building supervised text classifiers. Coding content needs a lot of resources, and lately this process has been utilized particularly in the training set annotation for machine learning models. Although the objectivity of coding is not always the purpose of coding, it helps in building the machine learning model, if the codings are uniformly done. Usually machine learning models are built by first defining annotation scheme, which contains definitions of categories and instructions for coding. It is known that multiple aspects affect to the annotation results, such as, the domain of annotation, number of annotators, and number of categories in annotation. In this article, we present few more problems that we show to be related with the annotation results in our case study. Those are negated presence of a category, low proportional presence of relevant content and implicit presence of a category. These problems should be resolved in all schemes on the level of scheme definition. To extract our problem categories, we focus on a media research case of extensive data on both the process as well as the results.

## introduction the coding of content features such as themes, frames, opinions and so on are widely used in the social sciences. in essence, the purpose of coding is to turn unstructured (qualitative) data such as text into structured data (the codes and their appearances) on which inferences can be made @xcite . the purpose of this study is to present few more characteristics, that has been already known, of the texts that cause difficulties in human-made coding, in other words, annotation. doing coding by hand is a resource-intensive process, particularly at scale @xcite . thus, within the computational social sciences, there have been many efforts to enable algo-rithms to do the coding for us @xcite . here, two distinct approaches appear, targeting different styles or phases of coding. the first is to use an unsupervised approach, such as clustering or topic modelling for an initial coding of the data, used to get an initial at-scale understanding of it @xcite @xcite . the second aligns with the more consolidated, theory-or hypothesis-informed codes which form a basis for inference. traditionally, these codes are often created in a second pass of coding based on information gained from the initial codes, or alternatively may already be defined at the beginning of research that employs a ready theory or hypothesis. here, the dominant mode of operation in computational social sciences currently is to train a classifier based on manual training data. while not completely obviating the need to do manual coding, the core idea here is that only a small portion of the data overall needs to be coded, and the classifier will handle propagating the codes to the rest of the material. this in turn is argued to lead to the possibility of using much larger unstructured datasets as research material, which is also argued to both broaden as well as solidify the inferences that can be made based on them. producing training data annotations for building text classifiers has been studied previously from various perspectives of the coding or annotation process: instructions given to the annotators @xcite , the number of tasks given at the same time @xcite , and the text difficulty of the classified texts @xcite . finnerty et al @xcite found that simplicity and the amount of tasks affects the agreement between annotators. weber et al @xcite found that text difficulty, measured by lexical diversity measure type-token ratio, predicts intercoder reliability so that increasing the diffi-culty lowers the reliability. budak et al @xcite found that training of the annotators improves the annotation results while using codebook not. also various other features have been shown to affect to the annotation performance, for example, annotation domain, number of annotators and number of annotation categories @xcite . detecting annotation errors in general, not only for full texts, has been studied for various tasks, such as slot filling @xcite , and partof-speech tagging @xcite ). annotation error detection can be divided to two areas: detecting them based on statistical measures and based on grammatical comparison @xcite . statistical error detection relies on finding anomalies from the annotations, such as rare local tag patterns in linguistic annotations @xcite . examples of grammatical comparison are pattern matching to detect invalid bigrams from a pos tag sequence @xcite , and utilizing different layers of linguistic information to find inconsistencies between the layers, such as pos tags and syntactic tree, and using them to correct erroneous pos tags @xcite . in the annotation error detection approaches above, ground truth has been known. for example in part-of-speech tagging, it is possible to define the correct tags, where in more complex tasks, identifying annotation errors is difficult as the correct class cannot be determined. there are also ways to take into account disagreement between the annotators in building the classification model, which is useful especially when the ground truth labels are not known. it can be done by training the classifiers using annotation distributions @xcite , adjusting the label distribution based on removing noise @xcite , or with jury learning, where annotators are chosen to compose a jury, that can be formed based on external attributes, such as to have balance between different genders @xcite . in current practice, the application and validation of machine-learned classifiers for coding usually happen in distinct stages @xcite . first, to evaluate both the coding scheme as well as annotation quality, typically a measure of interannotator agreement such as @xcite or in the case of multi-class and possibly multi-label settings, @xcite is used. if this is good enough, the process moves to training the classifier and evaluating it. most often this is done using the standard computer science evaluation metrics of precision and recall, as well as their harmonic mean, the f1-score. in a multi-class setting, performance measures across classes are also often summarised into a single number using either micro-or macro-f1 measures, although both have drawbacks @xcite . finally, once the classifier accuracy is deemed good enough, the most common approach is to just use the numbers produced by the classifier as is, although ways have also been designed to factor in classifier biases @xcite . a glaring problem in the workflow described above is that the evaluation of all of coding scheme, annotation quality and classifier accuracy happen in isolation from each other, and most often, none of the uncertainty identified at each stage is carried forward @xcite @xcite . in the field of qualitative methods, it has been even questioned, how realistic or desirable end result complete objectivity of annotation is (o'connor and joffe, 2020), and that is a problem for training machine learning classifiers. in practice, this can lead to the final data used for inference widely diverging from what its users expect. however, in this paper, we will not be discussing how to explicitly link these stages. instead, we will focus on trying to lessen the uncertainty in the first place. we start from the first stage of the process, the definition of the coding categories, and add three more features in the texts, in addition to those that have been already known, that lead towards increasing fuzziness and uncertainty, worse inter-annotator agreement and analytical usefulness. to extract our three problem categories, we focus on a research case of extensive data on both the process as well as the results. ## case description the study is based on experiences and annotation scores from an annotation project that sought to categorize finnish news media texts concerning alcohol policy. the aim of the annotation project is to develop a training dataset for a supervised classifier that detects categories related to finnish alcohol policy discussion to be used in a study of finnish political journalism. while the results of the aforementioned study will be published in due course, this article discusses issues related to the annotation process and, more specifically, provides a detailed analysis on how the annotation disagreement is distributed across articles of the data. the main goal is to give insight on how the annotation scheme performed in the context of annotation process. the data for this article are the inter-annotator scores of each annotated article. the annotation scheme is based on an earlier finnish study on representations of alcohol policy in finnish news media. this scheme was iteratively and reactively developed further to enhance the initially unsatisfactory inter-annotator scores. the iterative developments in the categorisation scheme included modifications both in the category level, such as dropping some categories from the original scheme, and excluding articles related to foreign issues. all changes to the annotation scheme were done in order to bring the annotation closer to the media studies aims of the study. the main motivation for the changes was to make the classification suit better the research interests and to improve the inter-annotator scores by dropping categories that were intuitively deemed as tricky for annotators. thus, the annotators were finally instructed to categorize the articles in three categories: the base dataset of the annotation project included in total 33,902 articles from four finnish news media: helsingin sanomat, yle, iltalehti and stt. the base dataset included all articles mentioning the finnish lexeme for alcohol, "alkoholi" and all of its word forms. the analysis presented in this article are based on three annotation tests ( on the first annotation round ( ## ruling out annotator effects after noticing lower inter-annotator scores in the annotation project than in the preceding annotation tests, possible annotator effects were studied in detail. we found one outlier annotator (annotator 4 in figure 1 ), whose performance in the pairwise agreement plot was visibly different than the performance of other annotators. the underlying reason for this was one annotator who failed to annotate stt's news, although submitted all annotations in the annotation user interface as empty annotations. after the removal of stt annotations of the annotator 4, there was still a disagreement in the annotations, which can be seen in figure 2 . the remaining potential sources of error were therefore the classification framework and the article texts. 5 surfacing effects of how phenomena appear in the data 5.1 extracting problem categories through close reading after excluding clear outlier annotations, we were interested in investigating how choices made in developing our annotation scheme and the features of the articles contributed to disagreements among annotators. through iterative close reading of both the articles and annotations of each three annotation rounds, we identified three properties of articles in relation to categories in the annotation scheme that tentatively seemed to be connected to high disagreement scores. all three properties relate to how the category-relevant content is positioned in the articles. we will discuss the properties in what follows. ## effects of the problem types after tentatively identifying the three factors above contributing to disagreement , we sought to measure how much they could explain the disagreement between annotators. to answer this question, we annotated altogether 150 articles, consisting of 100 from the final project and 50 from the second annotation test, using the three problem categories above. annotation was done using a binary classification and each article was annotated whether it had a proportional presence of some category, whether it had an implicit presence of a category and whether it had a negated presence of a category. in addition, we calculated the overall properties of the articles and student annotations for each article: ## discussion in this article, we have identified few problem types that we believe pervade many annotation schemes: low proportional presence of a category, implicit presence of a category and negated presence of a category. we have shown that in our case study, presence of these problem types in articles lead to lower levels of agreement. based on our empirical results, we have a few recommendations for projects that involve creating annotation schemes. first, it is important to make explicit choices with regard to wanting to extract "a clearly present category" vs "a category that appears in any measure" (majority vs minority theme e.g. still problematic, because leaves option for "appears in a side sentence" to be declared majority theme if nothing else appears). second, no annotation principle or category definition should depend on the presence or absence of any other category. designing categories should be done so that it should be possible to annotate each category in isolation. third, after a round of inter-annotator-agreement, the focus should be on disagreements, but not in isolation but as a whole. the following question should be asked: what unites the articles with disagreements? how does this contrast with commonalities in the articles/categories without disagreements? we argue that with focusing on disagreements before finishing the annotation scheme, the quality of the final scheme would be better.