Beyond the Data: The Impact of Annotation Inconsistencies in UD Treebanks on Typological Universals and Complexity Assessment (2025)

Abstract: This study explores the impact of annotation inconsistencies in Universal Dependencies (UD) treebanks on typological research in computational linguistics. UD provides a standardized framework for cross-linguistic annotation, facilitating large-scale empirical studies on linguistic diversity and universals. However, despite rigorous guidelines, annotation inconsistencies persist across treebanks. The objective of this paper is to assess how these inconsistencies affect typological universals, linguistic descriptions, and complexity metrics. We analyze systematic annotation errors in multiple UD treebanks, focusing on morphological features. Case studies on Spanish and Dutch demonstrate how differing annotation decisions within the same language create contradictory typological profiles. We classify the errors into two main categories: overgeneration errors (features incorrectly annotated, since do not actually exist in a language) and data omission errors (inconsistent or incomplete annotation of features that do exist). Our results show that these inconsistencies significantly distort typological analyses, leading to false generalizations and miscalculations of linguistic complexity. We propose methodological safeguards for typological research using UD data. Our findings highlight the need for methodological improvements to ensure more reliable cross-linguistic generalizations in computational typology.

## introduction multilingual corpora with consistent annotation schemes have become invaluable resources for typological research in computational linguistics @xcite . among these, universal dependencies (ud) @xcite stands out as one of the most comprehensive collections of consistently annotated treebanks across diverse languages. the standardized annotation framework of ud has enabled researchers to conduct large-scale cross-linguistic comparisons and formulate typological universals based on empirical data rather than theoretical assumptions @xcite . this development has significantly advanced our understanding of linguistic diversity and universals. however, the promise of consistent crosslinguistic annotation faces substantial challenges in practice. despite rigorous guidelines and quality control measures, inconsistencies and errors in annotation persist across different treebanks, even within the same language. these inconsistencies, while perhaps minor when considering individual treebanks in isolation, can have significant implications when aggregated for typological studies, potentially leading to incorrect characterizations of languages and flawed formulations of linguistic universals. our research identified several systematic annotation errors across multiple ud treebanks that directly impact typological characterizations based on ud data. this paper examines how these annotation inconsistencies affect the formulation of typological universals, description of languages or information regarding linguistic complexity @xcite with a particular focus on morphological features such as gender, number, and verbal mode/tense. we establish a correlation between the concepts of linguistic complexity and linguistic universals. we understand the concept of complexity in terms of the difficulty of learning one language from another (second language acquisition); we interpret universals as structures/categories present in all languages. from this standpoint, we establish an inversely proportional relationship between the two concepts: the greater the degree of shared characteristics between two languages, the less challenging it will be to learn one from the other. in essence, the higher the universality of a language, the lower its complexity level when learned as a second language. given the interrelationship between typo-logical universals and complexity, and considering that typological universals are calculated from treebanks, eliminating inconsistencies in ud treebanks is crucial for accurately calculating linguistic complexity, as these inconsistencies can distort typological profiles, affecting the relationship between universals and complexity, particularly when measuring the ease of learning languages based on their shared features. we exemplify our research with cases from spanish and dutch treebanks to demonstrate how annotation decisions in one treebank can differ substantially from another for the same language, creating contradictory typological profiles. furthermore, we explore how conversion processes from legacy annotation schemes to ud can introduce systematic biases if not carefully supervised. the research questions guiding this investigation are: (1) how do annotation inconsistencies in ud treebanks affect typological characterizations of languages? (2) what methodological safeguards can researchers implement to account for these biases when conducting typological studies using ud data? by addressing these questions, we aim to strengthen the foundation of computational typology while acknowledging the inherent challenges in creating truly consistent cross-linguistic annotation schemes. rather than diminishing the value of resources like ud, our goal is to enhance their utility by promoting awareness of potential biases and suggesting practical approaches to mitigate their effects on typological research. ## theoretical framework ud @xcite has established itself as a standard framework for syntactic annotation across languages @xcite @xcite , with its primary goal being to capture linguistic universals while accommodating language-specific phenomena. the standardized annotation schema enables cross-linguistic comparison and facilitates typological research on an unprecedented scale in corpora @xcite . however, the application of a universal schema to typologically diverse languages inevitably creates tension between universal applicability and language-specific accuracy. the challenges of cross-linguistic annotation have been documented in the literature @xcite @xcite @xcite . these challenges include the difficulty of establishing truly universal categories, the problem of forcing language-specific phenomena into universal frameworks, and the lack of correspondence between ud anotation guidelines and classical linguistic claims or theories. while ud has made significant progress in addressing these issues through detailed guidelines and collaborative development, other authors have proposed alternative proposals in order to enhance these detected problems @xcite . in particular, morphological features present unique challenges for cross-linguistic annotation. features such as gender or number vary significantly across languages, both in terms of their existence and their manifestation. ud addresses this variability through a feature inventory that distinguishes universal from language-specific features. even if the annotation scheme is adaptable enough, the problems still arise due to annotators (or annotating) action. in this case we do not find as much error analysis as in the case of the revision of the annotation scheme from a theoretical perspective @xcite . the only frequent review is a specific review of problems inherent to certain languages, without being general or extendable. ## typology of annotation errors based on our analysis of ud treebanks, we propose a typology of annotation errors that affects typological generalizations. these errors can be classified into two broad categories: annotators may apply features appropriate in one context and these are (probably automatically) propagated to con-texts where they are linguistically unmotivated. ## methodology our investigation of annotation inconsistencies in ud treebanks follows a systematic methodology designed to identify, categorize, and assess the impact of annotation errors on typological generalizations. this section describes our data selection, query methods, and analytical approach. we analyzed all available treebanks from ud (version 2.15) quering information contained in greenberg (1963) universals. ## case studies this section presents detailed analyses of specific annotation inconsistencies identified in our investigation and their implications for typological research. we focus on three representative cases that illustrate both overgeneration and data omission errors across different morphological features. ## discussion our investigation into annotation inconsistencies in ud treebanks reveals several important implications for typological research. this section examines the broader significance of our findings and proposes approaches to mitigate the impact of annotation errors on typological generalizations. ## conclusion this study has identified and analyzed systematic annotation inconsistencies in ud treebanks that affect typological generalizations, with a focus on exemplifying it by morphological features in spanish and dutch. our investigation revealed two primary categories of errors: overgeneration, where features are incorrectly applied to elements that should not have them, and data omission, where features are inconsistently or incompletely annotated. these errors have direct implications for the validity of typological universals derived from ud data.