Variety delights (sometimes) - Annotation differences in morphologically annotated corpora (2025)

Abstract: The goal of annotation standards is to ensure consistency across different corpora and languages. But do they succeed? In our paper we experiment with morphologically annotated Hungarian corpora of different sizes (ELTE DH gold standard corpus, NYTK-NerKor, and Szeged Treebank) to assess their compatibility as a merged training corpus for morphological analysis and disambiguation. Our results show that combining any two corpora not only failed to improve the results of the trained tagger but even degraded them due the inconsistent annotations. Further analysis of the annotation differences among the corpora revealed inconsistencies of several sources: different theoretical approach, lack of consensus, and tagset conversion issues.

## introduction annotation standards such as universal dependencies (ud) @xcite are intended to facilitate consistent annotation across corpora and languages. linguistic annotation is time-consuming; therefore, combining different corpora that share the same annotation scheme could be an effective strategy to increase corpus size. in our research, we explored this possibility with morphologically annotated corpora in hungarian. training a text processing tool with several different hungarian corpora has previously been proven to be an effective method for the recognition of named entities @xcite . our assumption was that a larger training corpus would increase the performance of a lemmatizer and morphological analyzer tool as well. however, linguistic annotation is a complex task and different theoretical approaches may allow subjectivity even within a well-defined annotation scheme. therefore, it is highly questionable whether the corpora that are expected to be compatible are indeed so; and if not, whether it is possible to ensure a higher level of compatibility without manually re-annotating one of them. in this paper we examine the compatibility of three morphologically annotated hungarian corpora by using them as training data for postagging tools. in section 3 we present the corpora, their tagsets, and the tagger tools in detail. the section also describes our experiment setup: each corpora was split into train, dev, and test subsets which we used in different combinations for training and testing. our results presented in section 4 showed that pairing different corpora lowered the performance in each case. to analyze the differences in the tagsets and annotation schemes of the corpora, we performed further training and testing experiments where we used one corpus for training and another for testing (section 5). the error analysis of these revealed inconsistencies of several sources: a different theoretical approach, lack of consensus, and tagset conversion issues. our findings contribute to the standardization of annotation schemes for hungarian, including the revision of the ud guidelines. we also detected some issues in the corpora and the ud-conversion tool that we used that need to be addressed in the future. ## related work the issue of combining different corpora was previously addressed by @xcite in the evaluation of udpipe version 1.1. they trained the pipeline on a wide range of languages where multiple ud corpora were available. the tagger and parser models were trained both on the individual corpora and on combinations of different corpora. generally, they found that the models achieved better results when only one corpus was used for training, combining different corpora de-graded performance. they also conducted more detailed experiments for smaller corpora with the goal of examining the possibility to enrich limited training data from other corpora. the paper shows the results in those cases only where the enrichment of the training corpus resulted in better performance in dependency annotation. this means a total of 12 corpora in ancient greek, czech, english, french, italian, latin, slovenian, and swedish languages. extending the original datasets from other corpora improved the performance of pos tagging in 6 cases, morphological feature identification in 4, and lemmatization in 7 cases. thus, increasing corpus size from other sources did not work in every case, not even for small corpora. the authors explain this with the inconsistencies in the annotations of the different corpora ("the universal dependencies are yet not so universal as everyone would like"). @xcite examine the discrepancies in annotations of ud corpora, focusing primarily on english and french treebanks, as these are among the most extensively represented languages. to detect differences between the corpora, they used the method of @xcite , which states that if two identical sequences are annotated differently, then one of the sequences is likely to be inconsistent. according to @xcite , inconsistencies may naturally occur within a corpus as well, but in all the cases examined, the ratio of conflicting annotations was higher between different corpora than within one. the authors conducted another experiment to characterize differences between corpora. in this, they trained a binary classifier to decide which of the two corpora a sentence belongs to. the intuitive assumption is that the higher the error rate of this classifier is, the more similar the two corpora are. the classifier was trained on words, pos tags, and word + pos tag pairs. the most successful classification was achieved with the last combination, which suggests that varying annotations of identical words (or sequences of words) characterize the corpora well, indicating that the differences between the annotations of different corpora are systematic. it can thus be said that the discrepancies in annotation schemes among different corpora of the same language are a known issue that affects multiple languages. ## corpora and tools used for our experiments, we used three manually annotated hungarian corpora of different sizes. the largest among them is the szeged treebank @xcite , which is currently used as the training corpus for huspacy @xcite . its total size is 1 362 505 tokens. the bulk of the original annotations @xcite was automatically converted to the universal dependencies standard foot_0 . on a small part of the corpus foot_1 (42 032 tokens), the converted ud annotations were manually checked and corrected; this is the only subset openly available in the ud treebank repository @xcite . the second largest corpus we used is nytk-nerkor foot_2 @xcite , which contains a total of 1 017 340 tokens, while the smallest elte dh gold standard corpus (k. molnár and dömötör, 2023) foot_3 consists of 496 060 tokens. both corpora were annotated with the same methodology. they used the emtsv @xcite text processing pipeline for pre-processing, and its output was manually corrected by human annotators. the rule-based morphological analyzer module @xcite of the pipeline assigns all possible morphological and morphosyntactic analyses to each word of the input text. the annotations are linked to each morpheme of the word (example 1). the pos tagger module, purepos @xcite disambiguates the analyses suggested by the analyzer module and provides the lemma and the morphological tag of the word (example 2). the emtsv tag is a simplified combination of the em-morph tags of each morpheme of the word. this means that the emtsv tags are not merely pos-tags. they also contain all the morphosyntactic information that is represented in the morphological features in universal dependencies. the emtsv tagset can be converted automatically to ud; both nerkor and the elte dh corpus used the emmorph2ud2 @xcite converting tool to add the ud annotation layer. the ud tags were not manually checked in either of the corpora, but nerkor did apply some dictionary-and rule-based corrections in cases where their scheme differed from the ud guidelines foot_4 . the elte dh corpus did not change the output of the ud conversion tool (as it is supposed to be unambigous). in summary, all three corpora have ud morphological annotations and two of them also contain emtsv tags, meaning the three corpora could potentially be merged to form a substantially larger and more comprehensive training dataset for morphological analyzers and pos-tagging tools. all three corpora are genre heterogeneous, containing overlapping and unique text types. combining the corpora thus achieves not only a larger size but also greater genre diversity. the genres found in the corpora are summarized in for testing the compatibility of the corpora, we trained the lemmatizer and morphological analyzer modules of huspacy and purepos on each. hus-pacy is a project that provides hungarian models for spacy, the latter of which does not officially support the language. similarly to spacy, it uses ud pos tags and morphological features. pure-pos is an hmm-based automatic morphological annotation tool optimized for the emtsv tagset with the option of pre-analysis using the rule-based em-morph @xcite module. for the train-dev-test split of the corpora, we used the division of huspacy's original training data (derived from the szeged treebank). the cutting ensured that each subcorpus is represented in the train, dev, and test sets with the same proportion, and that each set contained complete sentences only. first, the corpora were used separately for training and testing, then we attempted to combine them in pairs. all models were trained for at most 50 epochs. for huspacy, we disabled all components aside from the senter, tagger, morphologizer and lemmatizer modules. due to inconsistencies in the huspacy dependencies, we were unable to retrain the transformer-based models and only report results for the hu_core_news_lg foot_5 model. for context, these results can be compared with the numbers achieved by the public spacy @xcite models for other languages. the results of a total of 82 models in 24 languages are available on the official website. 7 the average performance of the models in pos tagging, morphological features identification, and lemmatization is shown in ## summary in summary, the consistency of annotations proved to be more crucial than corpus size in training morphological analyzers. the results obtained from the combination of different corpora demonstrated that even small discrepancies in the annotation schemes can pose significant challenges to the tagging tools. the annotation differences of the corpora are from several sources. in some cases they are deliberate like the different handling of multiword proper names in elte dh and nerkor. annotations may also differ due to the lack of consensus regarding a phenomenon or category, which is the case with auxiliaries in hungarian. in other cases the cause of difference was the fact that one of the corpora over-simplified (or complicated) a tag or simply made mistakes. an example for the former is the different annotations of participles in elte dh and nerkor, and for the latter we can mention the overuse of adv in szeged treebank, mostly at the expense of conjuncts. our research also revealed some issues with the emtsv-ud converter tool. for future work we plan to extend the list of subordinate conjuncts and add the missing cases. as we got good results with training with the corpora separately, the question arises whether compatibility of different corpora is really that essential. in our opinion, having detailed guidelines is crucial for an international standard like universal dependencies. the fact that this is still missing for hungarian presents an ongoing challenge for the hungarian nlp community. fixing the issues revealed in our research, such as the obsolete features in szeged treebank and the annotation of participles in elte dh, is also an important future work. however, emtsv is an inherently languagespecific annotation scheme for hungarian, which makes the emmorph analysis and the emtsv tag layer a suitable way for the corpora to retain their unique character.