The revision of linguistic annotation in the U niversal D ependencies framework: a look at the annotators’ behavior (2025)

Abstract: This paper presents strategies to revise an automatically annotated corpus according to the Universal Dependencies framework and discusses the learned lessons, mainly regarding the annotators’ behavior. The revision strategies are not relying on examples from any specific language and, because they are languageindependent, can be adopted in any language and corpus annotation initiative.

## introduction the construction of annotated datasets is a challenging task, especially for low-resource languages. in order to take advantage of the experience of highresource languages, projects in other languages have adopted successful annotation models, "skipping" the steps of instantiating a theory (i.e., the linguistic model to be used) and creating tag sets, which are steps discussed by @xcite @xcite reutilizing annotation models is important, but is also key to have information on how to design an annotation task. it has become clear to the scientific community that sharing the know-how to building annotated corpora can encourage other research groups to undertake their own annotation projects. for this reason, over the last two decades, discussion on the corpus annotation process has been gaining prominence in the natural language processing (nlp) scene. seminal works laid the foundations of "annotation science" @xcite @xcite . the availability of new technologies has brought new possibilities, such as crowdsourcing the annotation @xcite and using llms as annotators @xcite @xcite . in addition, annotation has expanded its purposes, as shown by the case of perspectivism @xcite , which takes into account annotation disagreements. however, perspectivism hardly applies to the traditional prescriptive paradigm, which is the case of the annotation discussed here (see @xcite for a comparison between prescriptive and descriptive annotation paradigms). depending on the annotation model, different annotation formats and standards are adopted. for the universal dependencies (ud) framework @xcite -the focus of this paper -the conll-u format is the standard. this format is an evolution of conll-x @xcite and was developed to annotate datasets used in the shared tasks of 2017 and 2018 @xcite . to get an idea of the scope of the ud, its current version (may, 2025) has 319 treebanks and 179 languages, representing a valuable resource for training multilingual models and developing crosslanguage studies. thanks to this resource, several multilingual parsers have been trained, such as udpipe 2 @xcite , udify @xcite and stanza @xcite , which makes it possible to start a new annotation project by automatically pre-annotating the corpus and posteriorly manually revising it, which is another well established annotation method. the revision of a pre-annotated corpus is significantly different from annotating from scratch. correcting an entire corpus in order to improve the performance in some nlp task is a big challenge. it is not evident which sentences contain errors or how many errors there are. in particular, when the tool used for pre-annotation already has good accuracy, the annotators need to be very good judges in order to analyze the sentences, identify errors and propose corrections. in the particular case of conll-u, annotators have to deal with dozens of labels and a multilayered annotation. drawing on five years of experience with annotation, this paper presents adopted (language agnostic) annotation strategies and discusses the lessons learned -mainly those regarding annotator behavior -for a corpus of news texts in portuguese, following the ud framework. we believe that the fundamental lessons can provide insights for similar projects in other languages, and, for this reason, we have purposely not presented any examples in portuguese, and, where we considered important to provide an example, we have given it in english to increase its usefulness. basically, we decided to adopt a "divide-andconquer" strategy, which consisted of revising linguistic layers (in some of the 10 conll-u columns) separately and sequentially, as the information of one layer benefits from the corrections made in the others. this strategy allowed us to learn during the process and inspired us to develop resources to improve consistency, a fundamental requirement for building a gold standard corpus. this paper is organized as follows. in section 2, we comment on our project and on the reasons that led us to choose the ud annotation. section 3 presents our approach to annotation revision and the strategies developed to iteratively combine the best of human annotation skills with the best of computational power, doing our best to ensure consistency and to save time. section 4 comments on related work, and section 5 draws some conclusions and presents insights for future work. ## the porttinari project the aim of the porttinari @xcite project is to annotate corpora from different genres according to ud, with a view to train robust and multigenre parsers in portuguese that benefit downstream applications. the idea of choosing language-dependent theories, instantiating them, and creating our own annotation model was soon discarded, as this would limit the future use of our parsers in multilingual tasks. the reasons that led us to choose the ud "universal" annotation model were: in what follows, we describe and comment on the main steps of the annotation effort carried out on our initial corpus, called porttinari-base, composed of news texts, containing 168,080 tokens and 8,418 sentences. ## the annotation strategy: divide-and-conquer differentiating among 17 upos and 37 deprel labels is a complex task, even for experienced linguists. for this reason, we divided the revision task into four steps, based on conll-u columns: although we did not anticipate a cyclical nature, any decision that affected the entire corpus was followed by a punctual revision of the already annotated sentences, in order to maintain consistency. the remaining columns of conll-u were not revised: columns 1, 2, and 10 (id, form, and misc) were only changed when we corrected segmentation and tokenization problems; column 5 (xpos) was left blank because we had no need to use another pos tag set; column 9 (deps) was left blank because multilingual parsers were not (and are not at the time of writing this paper) prepared to simultaneously annotate enhanced dependencies. in the following, we comment on lessons learned during each of the four revision steps. ## related work the lack of a parser was a barrier for low-resource languages to start annotation for the morphosyntactic and syntactic layers. however, with datasets and multilingual models, the barrier is no longer the lack of a parser, but the lack of resources and systematic procedures to efficiently revise the preannotated corpus. in recent years, various proposals have been put forward to save effort in human revision. the following are some of them. @xcite adopt crowd-sourced lay annotators to annotate pos tags, putting the target word in bold, one context token on the left and one on the right, and presenting multiple choice questions, abridging the process of annotating from scratch. they used majority voting to decide disagreements. the model trained on the resulting data achieved slightly less than an expert in the task (82.6% and 86.8%, respectively). using a lexicon, they performed a new task, only restricting the labels available for a given token, achieving 83.7%. @xcite examined the morphological capabilities of chatgpt in 4 languages (english, german, turkish and tamil) and found that in none of them did llm achieve human-level performance in the proposed tasks, nor did it match the state-of-the-art models. freitas and de souza (2024) used two different models to annotate the corpus (udpipe 2 and stanza) and performed a human revision of all cases of disagreement between the two automatic annotations, adopting the heuristic that the agree-ment of the systems would be indicative of the correct annotation. machado and ruiz (2024) evaluated 3 llms in pos tag assignment using ud tag set in texts written in brazilian portuguese and showed that the best performance was achieved by chatgpt-3, with 90% of accuracy. none of them, however, covers the complete revision of the corpus. ## final remarks porttinari-base was launched in 2023 @xcite and has been used to train a state-of-the-art parser @xcite , reaching over 96% of accuracy. we have been using this parser to preannotate corpora of new genres within the larger multi-genre project porttinari. the divide-and-conquer strategy was very successful: the expected cascade effect was achieved, leading to an increasing reduction in errors. we hypothesize that, just as one annotation layer benefits greatly from improvements in another layer, small improvements in the performance of a tagger or parser can significantly impact the performance of downstream applications. for the interested reader, all the resources and tools that we mentioned are freely available on the poetisa project website: https://sites. google.com/icmc.usp.br/poetisa