TGIF : Tree-Graph Integrated-Format Parser for Enhanced UD with Two-Stage Generic- to Individual-Language Finetuning (2021)

Abstract: We present our contribution to the IWPT 2021 shared task on parsing into enhanced Universal Dependencies. Our main system component is a hybrid tree-graph parser that integrates (a) predictions of spanning trees for the enhanced graphs with (b) additional graph edges not present in the spanning trees. We also adopt a finetuning strategy where we first train a language-generic parser on the concatenation of data from all available languages, and then, in a second step, finetune on each individual language separately. Additionally, we develop our own complete set of pre-processing modules relevant to the shared task, including tokenization, sentence segmentation, and multiword token expansion, based on pre-trained XLM-R models and our own pre-training of character-level language models. Our submission reaches a macro-average ELAS of 89.24 on the test set. It ranks top among all teams, with a margin of more than 2 absolute ELAS over the next best-performing submission, and best score on 16 out of 17 languages.

## introduction the universal dependencies (ud; @xcite initiative aims to provide cross-linguistically consistent annotations for dependency-based syntactic analysis, and includes a large collection of treebanks (202 for 114 languages in ud 2.8). progress on the ud parsing problem has been steady @xcite , but existing approaches mostly focus on parsing into basic ud trees, where bilexical dependency relations among surface words must form single-rooted trees. while these trees indeed contain rich syntactic information, the adherence to tree representations can be insufficient for certain constructions including coordination, gapping, relative clauses, and argument sharing through control and raising @xcite . the iwpt 2020 @xcite and 2021 @xcite shared tasks focus on parsing into enhanced ud format, where the representation is connected graphs, rather than rooted trees. the extension from trees to graphs allows direct treatment of a wider range of syntactic phenomena, but it also poses a research challenge: how to design parsers suitable for such enhanced ud graphs. to address this setting, we propose to use a treegraph hybrid parser leveraging the following key observation: since an enhanced ud graph must be connected, it must contain a spanning tree as a subgraph. these spanning trees may differ from basic ud trees, but still allow us to use existing techniques developed for dependency parsing, including applying algorithms for finding maximum spanning trees to serve as accurate global decoders. any additional dependency relations in the enhanced graphs not appearing in the spanning trees are then predicted on a per-edge basis. we find that this tree-graph hybrid approach results in more accurate predictions compared to a dependency graph parser that is combined with postprocessing steps to fix any graph connectivity issues. besides the enhanced graphs, the shared task setting poses two additional challenges. firstly, the evaluation is on 17 languages from 4 language families, and not all the languages have large collections of annotated data: the lowest-resource language, tamil, contains merely 400 training sentencesmore than two magnitudes smaller than what is available for czech. to facilitate knowledge sharing between high-resource and low-resource languages, we develop a two-stage finetuning strategy: we first train a language-generic model on the concatenation of all available training treebanks from all languages provided by the shared task, and then finetune on each language individually. secondly, the shared task demands parsing from raw text. this requires accurate text processing pipelines including modules for tokenization, sentence splitting, and multi-word token expansion, in addition to enhanced ud parsing. we build our own models for all these components; notably, we pre-train character-level masked language models on wikipedia data, leading to improvements on tokenization, the first component in the text processing pipeline. our multi-word token expanders combine the strengths of pre-trained learning-based models and rule-based approaches, and achieve robust results, especially on low-resource languages. our system submission integrates the aforementioned solutions to the three main challenges given by the shared task, and ranks top among all submissions, with a macro-average eulas of 90.16 and elas of 89.24. our system gives the best evaluation scores on all languages except for arabic, and has large margins (more than 5 absolute elas) over the second-best systems on tamil and lithuanian, which are among languages with the smallest training treebanks. ## tgif: two-stage generic-to individual-language finetuning in addition to the tree-graph integration approach, our system submission also features a two-stage finetuning strategy. we first train a languagegeneric model on the concatenation of all available training treebanks in the shared task data regardless of their source languages, and then finetune on each individual language in a second step. this two-stage finetuning strategy is designed to encourage knowledge sharing across different languages, especially from high-resource languages to lower-resource ones. in our experiment results as reported in ## pre-tgif: pre-training grants improvements full-stack inspired by the recent success of pre-trained language models on a wide range of nlp tasks @xcite @xcite , inter alia), we build our own text processing pipeline based on pre-trained language models. due to limited time and resources, we only focus on components relevant to the shared task, which include tokenization, sentence splitting, and multiword token (mwt) expansion. ## official evaluation the shared task performs evaluation on ud treebanks that have enhanced ud annotations across 17 languages: arabic @xcite , bulgarian @xcite @xcite , dutch (van der @xcite , english @xcite , estonian @xcite , finnish @xcite , french @xcite , italian @xcite , latvian @xcite , lithuanian @xcite , polish @xcite , russian @xcite , slovak @xcite , swedish @xcite , tamil @xcite , ukrainian @xcite , and multilingual parallel treebanks @xcite . figure 2 visualizes the "delta elas" between our submission and the best result other than ours on a per-language basis, plotted against the training data size for each language. our system sees larger improvements on lower-resource languages, where we have more than 5-point leads on tamil and lithuanian, two languages among those with the smallest number of training sentences. ## closing remarks our submission to the iwpt 2021 shared task combines three main techniques: (1) tree-graph integrated-format parsing (@xmath0, and (3) preprocessing pipelines powered by language model pre-training. each of the above contributes to our system performance positively, 12 and by combining all three techniques, our system achieves the best elas results on 16 out of 17 languages, as well as top macro-average across all languages, among all system submissions. additionally, our system shows more relative strengths on lowerresource languages. due to time and resource constraints, our system adopts the same set of techniques across all languages and we train a single set of models for our primary submission. we leave it to future work to explore language-specific methods and/or model combination and ensemble techniques to further enhance model accuracies.