Enhanced Evaluative Language Annotation through Refined Theoretical Framework and Workflow (2025)

Abstract: As precursor work in preparation for an international standard ISO/PWI 24617-16 Language resource management – Semantic annotation – Part 16: Evaluative language , we aim to test and enhance the reliability of the annotation of subjective evaluation based on Appraisal Theory. We describe a comprehensive three-phase workflow tested on COVID-19 media reports to achieve reliable agreement through progressive training and quality control. Our methodology addresses some of the key challenges through the refinement of targeted guideline refinements and the development of interactive clarification tools, alongside a custom platform that enables the pre-classification of six evaluative categories, systematic annotation review, and organized documentation. We report empirical results that demonstrate substantial improvements from the initial moderate agreement to a strong final consensus. Our research offers both theoretical refinements addressing persistent classification challenges in evaluation and practical solutions for the implementation of the annotation workflow, proposing a replicable methodology for the achievement of reliable annotation consistency in the annotation of evaluative language.

## introduction annotating evaluative language has become increasingly important in the present era of artificial intelligence, particularly given the need for high-quality language resources for training large language models and understanding human emotions and personal stance. in response to this need, we are working on an international standard (iso/pwi 24617-16 language resource management -semantic annotation -part 16: ative language, to be adopted by the international organization for standardization. to ensure a sound practical application of the standard, we apply appraisal theory (at) as a foundational framework of analysis, fully described in @xcite . while at has proven influential across diverse research contexts, its practical implementation through manual annotation reveals significant methodological challenges. research addressing annotation methodology remains scarce, with studies rarely reporting inter-coder agreement measures or addressing reliability issues @xcite . similarly, the development of annotation workflow has received limited attention. while @xcite proposed a stepwise approach to appraisal annotation, the methodological guidance remains insufficient for addressing the complex practical challenges encountered in the annotation of evaluative language, which is fundamentally subjective. the limited focus on annotation methodologies has created a significant gap between theoretical significance and operational reliability, affecting the advancement of scientific interpretative approaches to the understanding of expressions of human stance and attitude. we aim to address these methodological gaps through the development of annotation workflows, refinement of guideline procedures, and tool-facilitated quality control mechanisms. we describe a comprehensive approach that combines refined theoretical grounding with practical solutions for the achievement of reliable inter-annotator agreement based on the annotation of a corpus of authentic texts sampled from media reports. in what follows, we describe a methodological framework that encompasses a multi-stage training protocol, problem identification through pilot annotations, and targeted intervention strategies that reduce annotator disagreements. 76 2 annotation framework and methodology ## conclusion this article has described a precursor study in preparation for an international standard (iso/pwi 24617-16 language resource management -semantic annotation -part 16) for the annotation of evaluative language, aiming at providing an empirical basis for the setting of the standard. the study addressed a critical gap in at research by developing a systematic annotation workflow that achieved reliable inter-annotator agreement for fine-grained analysis of evaluative language that is inherently subjective and ambiguous. through progressive training protocols and targeted guideline refinements, the study yielded substantial improvements in reliability and consistency, progressing from a moderate agreement (κ = 0.360) in an initial pilot study to a strong consistency (κ = 0.794) in the final assessment. the research reported in this article has contributed to methodological issues relating to the application of at in three key areas: a replicable workflow to ensure the reliable annotation of evaluative language, the refined theoretical guideline to address persistent classification challenges found particularly in judgementvs-appreciation distinction, and a tool-facilitated approach to ensure targeted corrections and quality control. several limitations warrant further research. the study focused on news discourse within the covid-19 domain, potentially limiting generalizability to other text types and evaluative contexts. the two-annotator design, while sufficient for establishing methodological principles, would benefit from an expansion to multiple annotators for broader validations. additionally, the theoretical refinements require further testing across diverse discourse domains to confirm their general applicability. future research should also explore human-ai collaborative annotation approaches, where our refined guidelines and training protocols could inform ai model development, while ai-assisted pre-annotation could potentially enhance human annotator efficiency and consistency in evaluative language annotation.