The Incremental Process of Building an Annotation Scheme for Clinical Narratives in P ortuguese: the Contribution of Human Variation Analysis (2025)

Abstract: The development of a robust annotation scheme and corresponding guidelines is crucial for producing annotated datasets that advance both linguistic and computational research. This paper presents a case study that outlines a methodology for designing an annotation scheme and its guidelines, specifically aimed at representing morphosyntactic and semantic information regarding temporal features, as well as medical information in medical reports written in Portuguese. We detail a multi-step process that includes reviewing existing frameworks, conducting an annotation experiment to determine the optimal approach, and designing a model based on these findings. We validated the approach through a pilot experiment where we assessed the reliability and applicability of the annotation scheme and guidelines. In this experiment, two annotators independently annotated a patient’s medical report consisting of six documents using the proposed model, while a curator established the ground truth. The analysis of inter-annotator agreement and the annotation results enabled the identification of sources of human variation and provided insights for further refinement of the annotation scheme and guidelines.

## introduction manual annotation is a cornerstone of both linguistic research and natural language processing (nlp) (cf. e.g., @xcite @xcite , enabling the research of linguistic phenomena and providing "gold labels" for training and assessing models in multiple nlp tasks @xcite @xcite . in addition to supporting data-driven approaches, manual annotation contributes to formalizing linguistic theories by offering a structured framework for empirical validation @xcite . developing a comprehensive annotation scheme is critical to ensure that the annotation is systematic, consistent, interoperable, and comprehensive. a welldesigned scheme enables the accurate representation of complex linguistic phenomena grounded in theory while maintaining practical applicability for annotators @xcite . when the data pertains to highly specialized subject matter, such as medical discourse, or involves the intersection of distinct domains, such as linguistics and medicine, the demands on scheme design increase substantially. in such cases, the annotation scheme and corresponding guidelines must be particularly precise and detailed to ensure accurate interpretation. this complexity challenges scheme designers and places additional cognitive and interpretive burdens on annotators @xcite . among the additional challenges in annotating clinical narratives is the significant heterogeneity of the content and writing styles of medical reports, which vary not only across healthcare institutions @xcite , but also between different departments or services within the same hospital. these texts are often written in a free and spontaneous manner, reflecting an inherent diversity of topics and concepts specific to the medical domain. moreover, clinical texts differ substantially from nonclinical texts due to the highly technical and specialized nature of the field, as well as the frequent use of abbreviations, which significantly increases the complexity of their processing @xcite . additionally, biomedical terminology is inherently complex, and it is common for certain terms to have different meanings depending on the context in which they are used. this further underscores the need for clear and context-sensitive annotation guidelines @xcite . a critical aspect of the annotation process is the assessment of both the effectiveness of the annotation scheme and the annotators' understanding of the guidelines. successful annotation depends on the clarity, coherence, and comprehensiveness of the documentation, as well as the annotators' training and familiarity with the scheme @xcite . well-developed guidelines -featuring explicit definitions and illustrative examples -are essential for achieving reliable and accurate annotations @xcite . the validation of annotation schemes typically involves a combination of pilot studies, iterative guideline refinement, and qualitative analyses of problematic cases. the annotation process generally entails collecting judgments from multiple annotators for each data instance, a practice widely recognized for enhancing annotation quality @xcite . a commonly used metric to assess the quality of the annotation is inter-annotator agreement (iaa), which provides a quantitative assessment of annotation consistency @xcite . high iaa scores suggest clear and effective guidelines, whereas low agreement may stem from a variety of causes @xcite @xcite , often revealing ambiguities or conceptual difficulties that require further attention. analyzing sources of annotation disagreement is determinant in improving annotation frameworks, providing valuable information on areas where guidelines may need clarification or extension @xcite . although human variation in clinical annotation is natural, it is generally undesirable because, for example, the annotation can be used to develop information extraction algorithms for clinical research, where data must be unambiguous. therefore, ambiguity must be eliminated, and disagreement in the annotation should be minimal or ideally nonexistent. nevertheless, analyzing such variation in earlier stages of the annotation process can serve as a valuable diagnostic tool, revealing limitations or ambiguities in the current annotation design and accompanying guidelines. observing patterns of annotator disagreement helps refine the guidelines and ultimately contributes to reducing annotation errors @xcite . the primary objective of this paper is to describe a methodology to develop and validate an annotation scheme. we focus specifically on strategies aimed at minimizing human variation throughout the annotation process. to this end, we present a case study involving the design of an annotation scheme for medical reports written in european portuguese. our main contributions are as follows: the paper is structured as follows. section 2 reviews related work. section 3 presents the case study, beginning with a description of the annotation scheme (3.1), followed by the results of the evaluation and a discussion (3.2) of how the findings informed improvements to the scheme and guidelines (3.2.2). the paper concludes with final remarks and directions for future work (4). ## related work the development and validation of annotation schemes is a labor-intensive and demanding task. yet, it is essential for both linguistic research and nlp applications. over the past four decades, annotation strategies have evolved significantly. since the early 1990s, when annotation became central to training machine learning models and practices were mostly improvised @xcite , there has been substantial progress toward systematizing and formalizing annotation methodologies. a considerable body of work has focused on establishing principled standards for creating and validating annotation schemes. for example, graham and van der meer (2015) propose a seven-step annotation process. this process begins with selecting and preparing data, followed by formulating labels and attributes grounded in linguistic theory, and drafting the annotation scheme and accompanying guidelines. subsequent steps include piloting the scheme on a sample dataset, evaluating the outcomes through iaa, and revising the scheme and guidelines if needed. the process concludes with large-scale annotation, periodic evaluations, and, finally, model training. a comparable approach is presented by @xcite through the matter annotation cycle (model, annotate, train, test, evaluate, revise), which emphasizes the iterative nature of annotation development. a key component of this cycle is the mama loop (model-annotate-model-annotate), whereby annotation schemes are continually tested and refined. designing a robust annotation scheme is inherently complex and critical for producing highquality annotated datasets. as emphasized by @xcite , this process should be multi-phased, collaborative, and supported by appropriate tools. additionally, the complexity of annotation tends to increase with the level of linguistic detail involved @xcite . once the scheme is designed, it is necessary to rigorously evaluate the annotation scheme and its guidelines. among various evaluation approaches, iaa agreement remains one of the most widely adopted and recognized. @xcite points out that iaa is not just a measure of reliability; it is also a tool for refining annotation schemes and understanding how annotators interpret them. @xcite conceptualize iaa as an indicator of annotation "trustworthiness". commonly used metrics for measuring iaa include cohen's kappa @xcite ), krippendorff's alpha @xcite , and simple percentage agreement. @xcite introduce anveshan (annotation variance estimation), a framework designed to evaluate patterns of annotator agreement and disagreement. this framework includes iaa agreement analysis and outlier detection based on annotation values. however, reporting iaa results alone is often insufficient. additional contextual information is necessary for meaningful interpretation. bayerl and paul (2024) advocate for including essential metadata to ensure transparent assessment of agreement, such as annotator expertise (e.g., novices, domain experts, scheme developers). furthermore, bayerl and paul (2024) identify factors that can influence iaa agreement such as the annotation domain, the number of categories in the annotation scheme, the number and expertise of annotators, the training provided to annotators, the purpose of the annotation task, and the specific agreement metrics used. from a different perspective, @xcite challenge the idea of a singular "correct" annotation. they identify three primary sources of disagreement -annotator-related, data-driven, and context-dependent -and argue for embracing disagreement within evaluation frameworks, promoting the use of multiple annotations and adaptive metrics. analyzing the sources of annotator disagreement can be a productive strategy for improving annotation schemes and guidelines. @xcite and @xcite demonstrate that such analysis can lead to greater clarity in annotation instructions and scheme structure. likewise, @xcite advocate for breaking down annotation tasks into distinct layers to effectively isolate and address sources of disagreement. @xcite highlight the value of "iterative enhancement" -a process that involves identifying errors to accelerate annotation and improve its quality. this iterative process often results in enhanced guidelines and refined annotation schemes. @xcite discuss five different sources of problems in annotations: ambiguities and variations in the data, uncertainty among the annotators, errors, and biases. according to the authors, failing to address these issues can have undesirable consequences for different phases of the annotation process, while resolving them can yield more robust scientific results. while the majority of the reviewed studies emphasize important aspects to consider in the development and validation of annotation schemes, they rarely provide a detailed, step-by-step account of the entire annotation process. in contrast, our work aims to fill this gap by offering a comprehensive framework for structuring the annotation workflow. specifically, we highlight the critical role of analyzing human variation as a means to iteratively refine both the annotation scheme and the accompanying guidelines. ## a case study in this section, we present the methodology developed to design and validate our annotation scheme, as outlined in figure 1 . the proposed approach is structured into four distinct phases, each comprising multiple steps that guide the annotation process from conception to evaluation. to illustrate the practical application of our methodology, we conduct a case study in which we implement and assess an annotation scheme tailored to extract both grammatical and medical information embedded within clinical narratives. the source material includes admission reports, discharge summaries, and general clinical notes. this annotation scheme serves as the foundation for constructing an annotated corpus of medical records written in european portuguese, specifically from patients diagnosed with acute myeloid leukemia (aml), a relatively understudied condition, being the extraction of structured data from clinical narratives essential to support and facilitate research efforts. additionally, the proposed annotation scheme and the resulting annotated dataset will enable a detailed investigation of the semantic characteristics of medical records, particularly for temporal features. subsection 3.1 details the methodology employed in the development of the annotation scheme, while subsection 3.2 discusses the procedures used to validate the scheme. ## final remarks in this work, our main goal was to describe the incremental process of developing and validating an annotation scheme, along with its corresponding guidelines, capable of integrating both linguistic and medical domain information in an interdocument annotation. the results of the annotation and curation phases enabled improvements to both the scheme and the guidelines through an iterative refinement process. developing an annotation scheme requires ongoing efforts toward improvement. with that in mind, we intend to further explore issues related to the identification of grammatical features and to develop a question-answer system that facilitates the selection of domain-specific labels, even for annotators without prior knowledge of the field.