T urku Enhanced Parser Pipeline: From Raw Text to Enhanced Graphs in the IWPT 2020 Shared Task (2020)

Abstract: We present the approach of the TurkuNLP group to the IWPT 2020 shared task on Multilingual Parsing into Enhanced Universal Dependencies. The task involves 28 treebanks in 17 different languages and requires parsers to generate graph structures extending on the basic dependency trees. Our approach combines language-specific BERT models, the UDify parser, neural sequence-to-sequence lemmatization and a graph transformation approach encoding the enhanced structure into a dependency tree. Our submission averaged 84.5% ELAS, ranking first in the shared task. We make all methods and resources developed for this study freely available under open licenses from https://turkunlp.org .

## introduction the universal dependencies 1 (ud) effort @xcite ) seeks to create crosslinguistically consistent dependency annotation and has to date produced more than 150 treebanks in 90 languages. ud is a broad and open community effort with more than 300 contributors @xcite , and the resources they have created have been instrumental in driving progress in dependency parsing in recent years, also serving as the basis of widely attended conll shared tasks on multilingual parsing in 2017 and 2018 @xcite . while ud resources, the conll shared tasks, and recent advances in deep learning-based parsing technology @xcite @xcite have contributed substantially to accurate dependency parsing using a consistent syntactic representation for a wide range of human languages, these efforts have focused almost exclusively on the basic ud dependency trees. ud defines also an enhanced graph representation, which allows more detailed representation of the sentence. common types of enhancements include null nodes for elided predicates, propagation of conjuncts for making connections between words more explicit, and augmentation of modifier labels with prepositional or case-marking information. the ability to produce enhanced ud graphs from raw text, previously explored by e.g. @xcite , @xcite , and @xcite , would represent a further advance over existing tools. the iwpt 2020 shared task on multilingual parsing into enhanced universal dependendies foot_0 @xcite is the first shared task evaluation targeting the enhanced ud graph. the task was organized using data from 28 ud treebanks covering 17 languages, representing baltic, finnic, germanic, romance, semitic, slavic, and southern dravidian languages. we participated in the iwpt shared task with our parsing pipeline consisting of components for segmentation, part-of-speech and morphological tagging, lemmatization, dependency parsing, and enhanced dependency graph analysis. our approach builds on custom pre-trained deep language models @xcite , a deep neural network-based parser @xcite , a character-level sequence-to-sequence lemmatizer @xcite , and a custom graph transformation approach encoding an enhanced dependency graph in a labeled tree structure. the parsing pipeline is fully language agnostic, and therefore trainable with any ud treebank. our submission to iwpt achieved an average enhanced labeled attachment score (elas) of 84.5%, the best performance among the 35 evaluated submissions from ten participating groups with an approximately 2% point margin to the second-best submission. ## shared task data the shared task data invoves 28 ud treebanks for 17 languages, representing the subset of treebanks for which enhanced dependencies are available. the enhanced dependencies fall into five types: gapping, propagation of conjuncts, controlled and raised subjects, relative clause antecedents, and case information. however, not all treebanks have all of these types. while the training data is divided according to individual treebanks, test data is divided on language level through pooling of the individual treebank test sets, without any direct possibility to identify which test set sentence originates from which source treebank. we note that this is a departure from previous ud parsing shared tasks, where the treebank distinction was preserved also in the test data. the training and development data range from less than 10,000 words for tamil to over a million for czech. ## system overview we next introduce our system and our approach to predicting enhanced dependencies. ## language models we apply transfer learning using pre-trained bert models, using multilingual bert foot_1 (mbert) as a starting point. based on recent studies introducing language-specific bert models @xcite @xcite , we anticipated that parsing performance could be substantially improved by replacing the multilingual model with dedicated language-specific ones. to identify or create a model that would improve on performance with mbert for every treebank in the shared task, we adopted a three-stage approach: 1) use previously released models, 2) pre-train a new model on wikipedia data, and 3) continue pre-training on texts from a web crawl. ## results for our final submission, we trained a model for each language using the largest treebank (in terms of token count) for the language in the shared task data release. all segmentation, tagging, parsing, and lemmatization models are thus monolingual and trained using only a single treebank. each udify model is fine-tuned for 160 epochs using a number of warm-up steps foot_8 roughly equal to a single pass over the training dataset. for each language the fine-tuning is based on a custom pre-trained bert model selected as detailed in section 4.4. lemmatization models do not require any external resources, and all hyperparameters follow the values used in @xcite . the primary evaluation metric in the shared task is elas (labeled attachment score on enhanced dependencies), which calculates f-score over the set of enhanced dependencies in the system output and gold standard. foot_9 for the four languages where our system did not achieve the highest elas results, the differences to the highest-performing submission are small (0.3-0.4% points) for dutch and french, and 1.8% points for english. however, there is a more than 6% point difference to the top result for tamil, the language with the smallest treebank in the shared task. this difference indicates a tradeoff of our approach in training monolingual models: languages with particularly limited resources do not gain support from annotations in other languages as they would in multilingual training. ## discussion cross-lingual compatibility is a major goal of the ud effort and the ability to train multilingual models where lower-resourced languages can benefit from data in higher-resourced languages a clearly desirable aim in language modeling. while our approach -which trains monolingual models and uses language-specific pre-trained models -can be seen as running counter to these goals, we do nevertheless share them. our choice to train separate models for each language for the shared task is based in part in awareness of remaining compatibility issues in ud treebanks, even within languages. we hope contrasting results for joint and languagespecific models for this shared task will help identify and resolve some of these challenges. regarding multilingual language models, we note that in aiming to cover more than 100 languages without a corresponding increase in model and vocabulary size, mbert faces multiple challenges in its capacity, and the model training does not fully balance lower-and higher-resourced languages. while we here found language-specific models to outperform a specific mbert model, highly multilingual models addressing these challenges might well be competitive with language-specific ones, and the creation of such models would greatly benefit practical parsing efforts targeting a large number of languages. to study the impact of the language-specific language models in our shared task results, we reproduce our pipeline using exactly same configurations except for replacing all language-specific bert models with the multilingual mbert. in this experiment, all languages are using the same multilingual language model as a starting point, later individually fine-tuned for each language while training the language-specific parsing models. when comparing these models to the official submissions of all 10 teams, the average elas is approximately 1.7% points below our own primary submission (∼11% increase in error), but still slighty above the second best submission by approximately 0.2% points. this means that, our pipeline would have reached the highest average elas score among the official submissions also without the languagespecific bert models, but only with a very thin margin to the next best team. ## conclusions we have presented the approach of the turkunlp group to the iwpt 2020 shared task on multilingual parsing into enhanced universal dependencies. our approach is based on deep transfer learning with language-specific models, the stateof-the-art udify neural parsing pipeline, sequenceto-sequence lemmatization, and a graph transformation approach to predicting enhanced dependency graphs. our submission to the shared task achieved the highest performance for the primary evaluation metric (elas) both on average as well as for 13 out of the 17 languages involved in the task, also achieving the highest average performance for most other evaluation metrics. all of the methods and resources developed for this study are made freely available under open licenses from https://turkunlp.org .