K øpsala: Transition-Based Graph Parsing via Efficient Training and Effective Encoding (2020)

Abstract: We present Køpsala, the Copenhagen-Uppsala system for the Enhanced Universal Dependencies Shared Task at IWPT 2020. Our system is a pipeline consisting of off-the-shelf models for everything but enhanced graph parsing, and for the latter, a transition-based graph parser adapted from Che et al. (2019). We train a single enhanced parser model per language, using gold sentence splitting and tokenization for training, and rely only on tokenized surface forms and multilingual BERT for encoding. While a bug introduced just before submission resulted in a severe drop in precision, its post-submission fix would bring us to 4th place in the official ranking, according to average ELAS. Our parser demonstrates that a unified pipeline is effective for both Meaning Representation Parsing and Enhanced Universal Dependencies.

## introduction the iwpt 2020 shared task on parsing into enhanced universal dependencies @xcite involves sentence segmentation, tokenization, lemmatization, part-of-speech tagging, morphological analysis, basic dependency parsing, and finally (for the first time) enhanced dependency parsing. the enhancements encode case information, elided predicates, and shared arguments due to conjunction, control, raising and relative clauses (see figures 1 and 2 ). in universal dependencies v2 (ud; @xcite , enhanced dependencies (ed) are a separate dependency graph than the basic dependency tree (bd). however, ed is almost a super-set of bd, 1 and so most previous approaches @xcite have attempted to recover ed from bd using languagespecific rules. on the other hand, hershcovich we were made to feel very welcome . arcs above the sentence are also in bd. @xcite experimented with tupa, a transitionbased directed acyclic graph (dag) parser originally designed for parsing ucca @xcite , for supervised ed parsing. they converted ed to ucca-like graphs and did not use pre-trained contextualized embeddings, yielding sub-optimal results. taking a similar approach, we adapt a transition-based graph parser @xcite designed for meaning representation parsing @xcite , but parse ed directly and use bert embeddings @xcite . the main contribution of our work is a transition system supporting the graph structures exhibited by ed, including null nodes (meaning this is not a strictly bilexical formalism), cycles and non-crossing graphs ( §3.1), as figure 4 demonstrates for the sentence from figure 2 . we parse ed completely separately from bd, demonstrating the applicability of a full graph parser, starting from only segmented and tokenized text, to ed. our code is available at https://github.com/ coastalcph/koepsala-parser . ## preprocessing as the focus of this shared task is ed parsing, we rely on existing systems for preprocessing. here, we consider two off-the-shelf pipelines: stanza deze is de modernste en grootste hal van belgië , en null de enige die voldoet aan de olympische normen . the dashed edges are deleted in ed, and the edges below the sentence are added. note the cycle null ↔ voldoet. @xcite foot_0 and udpipe 1.2 @xcite ), foot_1 both of which have models pre-trained on ud v2.5 treebanks. we experiment with either pipeline during prediction to process the raw text files for the dev and test sets, eventually selecting udpipe for our primary submission. this process entails sentence segmentation, tokenization, lemmatization, part-of-speech tagging, morphological feature tagging, and bd parsing. 4 for training our ed parser ( §3), however, we use gold inputs for simplicity. we use the conllu python package foot_3 to read conll-u files. ## transition-based enhanced dependency parser our system is a transition-based graph parser, based on the hit-scir system @xcite , which achieved the highest average score across frameworks (amr, eds, ucca, dm and psd) in the conll 2019 shared task on meaning representation parsing (mrp; @xcite . it is written in the allennlp @xcite framework. for training efficiently, it employs stack lstms @xcite , batching operations across sentences. for better encoding, hit-scir fine-tuned bert @xcite while training the parser. a transition-based parser operates by manipulating a buffer (originally containing the input words provided by the preprocessor, see §2) and a stack (originally containing the root, i.e., word at index 0), to incrementally create the output dependency graph. at each point in the parsing process, a tran-sition is selected out of a pre-defined set of possible transitions. a classifier is trained to predict the best transition to apply at each step, by mimicking an oracle during training (see §3.1). hit-scir used a different transition system per framework (amr, eds, ucca; and one system for dm and psd), according to the graph properties of each and based on existing framework-specific parsers @xcite @xcite . we construct a transition system for ed using subsets of transitions from two of the hit-scir systems: their system for dm and psd, as well as their system for ucca, with some further adaptations specific to ed graphs. ## results though we observed scant differences between the two preprocessing pipelines, it was udpipe that produced fewer validation errors. as such, we adopted it as the main preprocessor for our official submission. it is apparent in when comparing against the organizer-provided baselines, we see a strong improvement in using our system over both b2 and b3 systems. this is encouraging, as it demonstrates the benefit of parsing enhanced dependency graphs directly, rather than relying on predicted trees to accurately relay the enhanced structure (b1) or employing a heuristic-driven post-processor to derive it (b2). furthermore, though the organizers consider b1 as an indirect upper bound due to the gold-standard tokenization and dependency structure employed therein, we can nonetheless observe an advantage in using our parser for some languages. these are arabic (+2.16 elas), finnish (+3.32), italian (+4.46), and ukranian (+0.86). again, this is promising, given that our parser does not rely on any tree structure in order to parse graphs. ## conclusion in this paper, we have described the iwpt 2020 shared task submission by the copenhagen-uppsala team, consisting of graphs predicted by a transition-based neural dependency graph parser with pre-trained multilingual contextualized embeddings. while not ranked among the top submission according to the official scores, the parser architecture proved effective for the type of dependency graphs exhibited by ed, and after fixing a critial bug we found the scores to improve dramatically and agree with the scores we had observed during development. we expect that with more resources for bert fine-tuning, hyperparameter tuning, languagespecific pre-trained representations and careful preand post-processing, our parser will be a competitive system in this task. however, our contribution is a transition system that can directly handle ed, in a unified transition-based parsing system.