HUJI - KU at MRP 2020: Two Transition-based Neural Parsers (2020)

Abstract: This paper describes the HUJI-KU system submission to the shared task on CrossFramework Meaning Representation Parsing (MRP) at the 2020 Conference for Computational Language Learning (CoNLL), employing TUPA and the HIT-SCIR parser, which were, respectively, the baseline system and winning system in the 2019 MRP shared task. Both are transition-based parsers using BERT contextualized embeddings. We generalized TUPA to support the newly-added MRP frameworks and languages, and experimented with multitask learning with the HIT-SCIR parser. We reached 4th place in both the crossframework and cross-lingual tracks.

## introduction the conll 2020 mrp shared task @xcite combines five frameworks for graph-based meaning representation: eds, ptg, ucca, amr and drg. it further includes evaluations in english, czech, german and chinese. while eds, ucca and amr participated in the 2019 mrp shared task @xcite , which focused only on english, ptg and drg are newlyadded frameworks to the mrp uniform format. for this shared task, we extended tupa @xcite , which was adapted as the baseline system in the 2019 mrp shared task @xcite , to support the two new frameworks and the different languages. in order to add this support, only minimal changes were needed, demonstrating tupa's strength in parsing a wide array of representations. tupa is a general transition-based parser for directed acyclic graphs (dags), originally designed for parsing ucca @xcite . it was previously used as the baseline system in semeval 2019 task 1 @xcite , and generalized to support other frameworks @xcite . we also experimented with the hit-scir parser @xcite . this was the parser with the highest average score across frameworks in the 2019 mrp shared task, and has also since been applied to other frameworks @xcite . ## tupa-mrp tupa @xcite is a transitionbased parser supporting general dag parsing. the parser state is composed of a buffer b of tokens and nodes to be processed, a stack s of nodes currently being processed, and an incrementally constructed graph g. the input to the parser is a sequence of tokens: w 1 , . . . , w n . a classifier is trained using an oracle to select the next transition based on features encoding the parser's current state, where the training objective is to maximize the sum of log-likelihoods of all gold transitions at each step. the mrp variant @xcite supports node and edge labels, as well as node properties and edge attributes. the code is publicly available. 1 ## hit-scir parser the hit-scir parser @xcite is a transition-based parser, which extended previous parsers by employing stack lstm @xcite to allow computing homogeneous operation within a batch efficiently, and by adopting and fine-tuning bert @xcite embedding for effectively encoding contextual information. the parser is implemented in the allennlp framework @xcite . it supports parsing dm, psd, ucca, eds and amr, all included in the 2019 mrp shared task. the official dataset would be pre-processed for system input and postprocessed for output. mrp 2019 parser to support the 2020 data for english eds (for the cross-framework track) and german ucca (for the cross-lingual track). we also explored the possibilities of employing multitask learning with the parser ( §5). a repository containing our modified version of the parser is publicly available. 5 ## results validation vs. evaluation scores. the validation scores of 5 out of the 9 parsers is lower than their evaluation score: cf ptg by 0.01 f1 points, cf amr by 0.08, cf drg by 0.11, cl amr by 0.01 and cl drg by 0.1. we hypothesize it is due to the randomness in the evaluation metric: the mrp scorer uses a search algorithm to find a correspondence relation between the goldstandard and system graphs that maximizes tuple overlap. this search algorithm runs for a limited number of iterations. in order to decrease its running time, we used a lower limit on its parameters (10 random restarts, 5,000 iterations) than the default (20 random restarts, 50,000 iterations), which may have affected the accuracy of our validation score and potentially our system performance. cf vs. cl tracks. surprisingly, the cl track scores are mostly on-par with the cf tack ones, even though the cl parsers were often trained on significantly less examples. while the cf ucca training dataset contains 6,872 examples and the cl ucca contains only 3,713, both parsers gained similar scores. similarly, the cf drg dataset contains 6,606 example, while the the cl drg contains only 1,575. tupa trained only on the 1,575 examples gained a similar score to the cf one, while training on less then a fourth of the examples. the cf ptg dataset contains 42,024 examples. and while the cl ptg contains a lower, however similar, amount (39,560), it got a higher score (0.07 f1 point in validation, and 0.04 in evaluation). and while the cl amr dataset is only a third of the cf amr datsaet (16,529 and 57,885 examples respectively), both parser gained the same validation score. however, the evaluation score of the cf amr is higher by 0.07 f1 points. this could be possibly attributed to our mrp scorer low iteration limit. ## multitask cross-framework parsing in addition to training separate models per framework and language, we also experiment with training multitask cross-framework parsers, using a neural architecture with parameter sharing @xcite @xcite @xcite . we use the hit-scir parser as a basis, with different variations of shared architecture on top of it. for our experiments we choose the ucca and eds frameworks. the code is pub- ## conclusion we have presented tupa-mrp and a modified hit-scir parser, which constitute the huji-ku submission in the conll 2020 shared task on cross-framework meaning representation. tupa is a general transition-based dag parser with a uniform transition system, which is easily adaptable for multiple frameworks. we used it for parsing in both the cross-framework and the crosslingual tracks, adapting it for the newly introduced frameworks, ptg and drg. hit-scir is a transition-based parser with framework-specific transition systems, which we adapted for this year's shared task and used for english eds and ucca parsing in the cross-framework track. the hit-scir parser was additionally used in experimenting on multitask learning, with negative results for that approach. future work will tackle the mrp task with more modern transition-based-like parser architectures, such as pointer networks @xcite , which have so far only been applied to bilexical framworks, i.e., flavor-0 sdp @xcite .