Evaluating a U niversal D ependencies Conversion Pipeline for I celandic (2023)

Abstract: We describe the evaluation and development of a rule-based treebank conversion tool, UDConverter, which converts treebanks from the constituency-based PPCHE annotation scheme to the dependency-based Universal Dependencies (UD) scheme. The tool has already been used in the production of three UD treebanks, although no formal evaluation of the tool has been carried out as of yet. By manually correcting new output files from the converter and comparing them to the raw output, we measured the labeled attachment score (LAS) and unlabeled attachment score (UAS) of the converted texts. We obtain an LAS of 82.87 and a UAS of 87.91. In comparison to other tools, UDConverter currently provides the best results in automatic UD treebank creation for Icelandic.

## introduction the universal dependencies (ud) project is a multilingual project, consisting of dependency treebanks in 138 languages @xcite . udconverter is a tool which converts a phrase structure treebank to a ud treebank @xcite , and has been used for creating three ud corpora. originally configured for icelandic, the converter can be extended to convert treebanks in languages other than icelandic, as has been done for a faroese treebank @xcite , but it has not been thoroughly evaluated until now. without such evaluation, the benefit of using the converter is uncertain. therefore, we manually corrected a portion of a treebank created with the converted ud treebank and evaluate the conversion by comparing the converted sentences' output to the manually corrected ones. the evaluation is used to guide further development of udconverter, resulting in an improved conversion pipeline. the paper is structured as follows. section 2 discusses relevant resources, including ud corpora and methods of creating them. section 3 describes the evaluation setup used while section 4 discusses the results, including initial results before the converter was improved. we compare the converter's accuracy scores to the accuracy of three ud parsers in section 5 and finally, we conclude in section 6. ## background udconverter is a python module for converting bracket-parsed treebanks in the format of the penn parsed corpora of historical english (ppche) to the universal dependencies framework @xcite . it was created in order to convert the icelandic parsed historical corpus (icepahc) @xcite to the ud conll-u format and has been used for creating three ud corpora, ud_icelandic-icepahc, ud_icelandic-modern and ud_faroese-farpahc, all included in version 2.11 of universal dependencies @xcite . the converter takes an original icepahc-format tree and converts it to a ud tree, displayed in the conll-u format. as discussed in @xcite , the converter can be extended to convert treebanks in other languages than icelandic, as long as the input treebanks are in a format similar to the icepahc one. the converter's output generally adheres to ud annotation guidelines but no formal evaluation of the converter has been carried out until now. the ud corpora which were created by using udconverter were all converted from pre-existing constituency treebanks. these treebanks were manually annotated according to the ppche annotation scheme @xcite , which uses labeled bracketing in the same way as the penn treebank @xcite . this icepahc annotation scheme was used as a basis for the rule sets of udconverter. ud_icelandic-modern was converted from 21stcentury additions to icepahc, consisting of modern icelandic texts @xcite . it contains genres not previously found in the original icepahc @xcite , extracted from the icelandic gigaword corpus @xcite . two ud parsers have recently been released for icelandic @xcite , both of which utilize information from a pre-trained bertlike model, in this case an electra model that was pre-trained on icelandic texts @xcite . one of the models was trained with diaparser @xcite , an extended version of the biaffine parser @xcite , which uses contextualized embeddings, as well as attentions, from a transformer model as its input features. the other one was trained with combo @xcite , which accepts pre-trained embeddings from a transformer, as well as character and lemma embeddings, in addition to part-of-speech tags, as its input features. both parsers were trained on two icelandic ud corpora, ud_icelandic-icepahc and ud_icelandic-modern. ## evaluation in order to evaluate udconverter, we set up a testing experiment where output conll-u files from the converter were manually evaluated and corrected per strict annotation guidelines. these were then compared to the original raw output files. as these files contain identical texts, this enabled a one-to-one comparison, with the manually corrected files serving as a gold standard. in our evaluation, we focused on measuring the accuracy of the conversion when it comes to heads and dependency relations. for this project, we chose to source sentences for manual correction from the ud_icelandic-modern corpus, discussed in section 2, which then became the test set. in total, 651 sentences of the corpus, 15,140 tokens in total, were manually corrected, out of 80,395 tokens overall. two annotators with a background in linguistics worked on the manual correction. sentences were corrected to adhere to annotation rules used in the icelandic parallel universal dependencies (pud) corpus @xcite , which is the only icelandic ud corpus which was created manually. the corpus was used as a guideline when udconverter was developed. the annotators worked on separate sentences, and therefore information on inter-annotator agreement is not available. it would be beneficial to have information on the agreement, but the annotators discussed any uncertainties and came to joint conclusions. we used a labeled attachment score (las) to evaluate the converter, evaluating conll-u output based on how many tokens have been assigned both the correct syntactic head and the correct dependency relation @xcite . this simple accuracy score corresponds to a labeled f 1 score of syntactic relations. similar to this score is the unlabeled attachment score (uas), which evaluates the number of correct heads but does not take the dependency relations into account. ## results our results show that the converter achieves an las of 82.87 and a uas of 87.91. our results indicate that the overall error rate of the conversion is not affected by sentence length, with the relationship between sentence length and total errors per sentence being more or less linear. if sentence length is a rough indicator of syntactic complexity, this means that the converter handles complex syntactic structures just as well as simpler ones. this is expected, as the converter works off of a fixed rule set for a given language, which looks at the already annotated phrase structure of the input sentences. ## comparison all three parsers, which are the only available icelandic ud parsers, are trained using output from the converter, which presents some limitations when comparing them to the converter. the parsers learn from the training data, and can never produce results which are as accurate as the data itself. comparing the parsers' output to the converter's output is therefore not an equal comparison, but it does give an idea about their accuracy. furthermore, accuracy scores for udconverter are possibly higher than if they were obtained from development data, as discussed above. current scores show that using udconverter to create ud corpora will deliver the most accurate results, as the highest accuracy score for the three parsers is 11.41 points less than the converter's accuracy. however, each method has its advantages and drawbacks, as a converter requires a treebank which is annotated in the appropriate format, while parsers can create a corpus from plain text. ## conclusion we have described the evaluation of a rule-based conversion tool, udconverter, which converts treebanks in the phrase-structured ppche format to the dependency-based ud format. converted texts were manually corrected and used as testing data. we focused on the accuracy of dependency heads and dependency relations to achieve labeled and unlabeled accuracy scores (las, uas), which serve as f1 scores in our evaluation. our results show that udconverter achieves an las of 82.87 and a uas of 87.91. we compared these accuracy results to accuracy scores of three different icelandic ud parsers, our udpipe model along with diaparser and combo-parser, which showed that using udconverter most accurately delivers an icelandic ud corpus. culture. we would like to thank the anonymous reviewers for their contribution.