MRP 2020: The Second Shared Task on Cross-Framework and Cross-Lingual Meaning Representation Parsing (2020)

Abstract: The 2020 Shared Task at the Conference for Computational Language Learning (CoNLL) was devoted to Meaning Representation Parsing (MRP) across frameworks and languages. Extending a similar setup from the previous year, five distinct approaches to the representation of sentence meaning in the form of directed graphs were represented in the English training and evaluation data for the task, packaged in a uniform graph abstraction and serialization; for four of these representation frameworks, additional training and evaluation data was provided for one additional language per framework. The task received submissions from eight teams, of which two do not participate in the official ranking because they arrived after the closing deadline or made use of additional training data. All technical information regarding the task, including system submissions, official results, and links to supporting resources and software are available from the task web site at: http://mrp.nlpl.eu

## background and motivation the 2020 conference on computational language learning (conll) hosts a shared task (or 'system bake-off') on cross-framework meaning representation parsing (mrp 2020), which is a revised and extended re-run of a similar conll shared task in the preceding year. the goal of these tasks is to advance data-driven parsing into graph-structured representations of sentence meaning. for the first time, the mrp task series combines formally and linguistically different approaches to meaning rep-resentation in graph form in a uniform training and evaluation setup. key differences in the 2020 edition of the task include the addition of a graph-based encoding of discourse representation structures (dubbed drg); a generalization of prague tectogrammatical graphs (to include more information from the original annotations); and a separate cross-lingual track, introducing one extra language (beyond english) for four of the frameworks involved. 1 participants were invited to develop parsing systems that support five distinct semantic graph frameworks in four languages (see §3 below)all encoding core predicate-argument structure, among other things-in the same implementation. ideally, these parsers predict sentence-level meaning representations in all frameworks in parallel. architectures utilizing complementary knowledge sources (e.g. via parameter sharing) were encouraged, though not required. learning from multiple flavors of meaning representation in tandem has hardly been explored (with notable exceptions, e.g. the parsers of @xcite @xcite . the task design aims to reduce frameworkspecific 'balkanization' in the field of meaning representation parsing. its contributions include 1 to reduce the threshold to participation, two of the target frameworks represented in mrp 2019 are not in focus this year, viz. the purely bi-lexical delph-in mrs bi-lexical dependencies and prague semantic dependencies (psd). these graphs largely overlap with the corresponding (but richer) frameworks in 2020, eds and ptg, respectively, and the original bi-lexical semantic dependency graphs remain independently available @xcite . ## definitions: graphs and flavors reflecting different traditions and communities, there is wide variation in how individual meaning representation frameworks think (and talk) about semantic graphs, down to the level of visual conventions used in rendering graph structures. increased terminological uniformity and guidance in how to navigate this rich and diverse landscape are among the desirable side-effects of the mrp task series. the following paragraphs provide semi-formal definitions of core graph-theoretic concepts that can be meaningfully applied across the range of frameworks represented in the shared task. basic terminology semantic graphs (across different frameworks) can be viewed as directed graphs or digraphs. a semantic digraph is a triple (t, n, e) where n is a set of nodes and e ⊆ n × n is a set of edges. the inand outdegree of a node count the number of edges arriving at or leaving from the node, respectively. in contrast to the unique root node in trees, graphs can have multiple (structural) roots, which we define as nodes with in-degree zero. the majority of semantic graphs are structurally multi-rooted. thus, we distinguish one or several nodes in each graph as top nodes, t ⊆ n ; the top(s) correspond(s) to the most central semantic entities in the graph, usually the main predication(s). in a tree, every node except the root has indegree one. in semantic graphs, nodes can have in-degree two or higher (indicating shared arguments), which constitutes a reentrancy in the graph. in contrast to trees, general digraphs may contain cycles, i.e. a directed path leading from a node to itself. another central property of trees is that they are connected, meaning that there exists an undirected path between any pair of nodes. in contrast, semantic graphs need not generally be connected. finally, in some semantic graph frameworks there is a (total) linear order on the nodes, typically (though not necessarily) induced by the surface order of corresponding tokens. such graphs are conventionally called bi-lexical dependencies (probably deriving from a notion of lexicalization articulated by @xcite and formally consti-tute ordered graphs. a natural way to visualize a bi-lexical dependency graph is to draw its edges as semicircles in the halfplane above the sentence. an ordered graph is called noncrossing if in such a drawing, the semicircles intersect only at their endpoints (this property is a natural generalization of projectivity as it is known from dependency trees). a natural generalization of the noncrossing property, where one is allowed to also use the halfplane below the sentence for drawing edges is a property called pagenumber two. @xcite provide additional definitions and a quantitative summary of various formal graph properties across frameworks. ## meaning representation frameworks the shared task combines five distinct frameworks for graph-based meaning representation, each with its specific formal and linguistic assumptions. this section reviews the frameworks and presents english example graphs for sentence #20209013 from the venerable wall street journal (wsj) corpus from the penn treebank (ptb; @xcite : the example exhibits some interesting linguistic complexity, including what is called a tough adjective (impossible), a scopal adverb (almost), a tripartite coordinate structure, and apposition. the example graphs in figures 1 through 4 are prewhere unanchored nodes for unexpressed material beyond the surface string can be postulated @xcite . whether or not these nodes occupy a well-defined position in the otherwise total order of basic ud nodes remains an open question, but either way the presence of unanchored nodes will take enhanced ud graphs beyond the bi-lexical flavor (0) graphs in our terminology. ## task setup the following paragraphs summarize the 'logistics' of the mrp 2020 shared task. except for the addition of the new cross-lingual track, the overall task setup mirrored that of the 2019 predecessor; please see @xcite for additional background. cross-framework track the english training, validation, and evaluation data are summarized in as a common point of reference, the training data includes a sample of 89 wsj sentences annotated in all five frameworks (twenty for drg); for all frameworks but drg, the evaluation data further includes parallel annotations over the same random selection of 100 sentences from the novel the little prince (by antoine de saint-exupéry) as used in mrp 2019, dubbed lpps. these parallel subsets of the gold-standard data are available for public download from the task site (see §9 below). cross-lingual track for four of the frameworks (excluding eds), gold-standard training and evaluation data has been compiled in other languages than english: mandarin chinese for amr, czech for ptg, and german for ucca and drg. for ucca and in particular drg, however, available data is comparatively limited, as summarized in additional resources for reasons of comparability and fairness, the shared task constrained which additional data or pre-trained models (e.g. corpora, word embeddings, language models, lexica, or other annotations) can be legitimately used besides the resources distributed by the task organizers-such that all participants should in principle have access to the same range of data. however, to keep such constraints to the minimum required, a 'white-list' of legitimate resources was compiled from nominations by participants (with a cut-off date eight weeks before the end of the eval- companion syntactic parses at a technical level, training (and evaluation) data were distributed in two formats, (a) as sequences of 'raw' sentence strings and (b) in pre-tokenized, partof-speech-tagged, lemmatized, and syntactically parsed form. for the latter, premium-quality morpho-syntactic dependency analyses were provided to participants, called the mrp 2020 companion parses. these parses were obtained using a prerelease of the 'future' udpipe architecture @xcite , trained on available gold-standard ud 2.x treebanks, for english augmented with conversions from ptb-style annotations in the wsj and ontonotes corpora @xcite , using the ud-style corenlp 4.0 tokenizer @xcite and jack-knifing where appropriate (to avoid overlap with the texts underlying the mrp semantic graphs). rules of participation while the various meaning representation frameworks and graph banks represented in the shared task inevitably present considerable linguistic variation, all mrp 2020 data was repackaged in a uniform and normalized abstract representation with a common serialization, the same json lines format as used in the previous year @xcite . because some of the semantic graph banks involved in the shared task had originally been released by the linguistic data consortium (ldc), the training data was made available to task participants by the ldc under no-cost evaluation licenses. all task data (including system submissions and evaluation results) is being prepared for general release through the ldc, while subsets that are copyright-free will also become available for direct, open-source download. the shared task was first announced in march 2020, the initial release of the cross-framework training data became available in late april, and the evaluation period ran between july 27 and august 10, 2020; during this period, teams obtained the unannotated input strings for the evaluation data and had available a little more than two weeks to prepare and submit parser outputs. submission of semantic graphs for evaluation was through the ## evaluation following the previous edition of the shared task, the official mrp metric for the task is the microaverage f 1 score across frameworks over all tuple types that encode 'atoms' of information in mrp graphs. the cross-framework metric uniformly evaluates graphs of different flavors, regardless of a specific framework exhibiting (a) labeled or unlabeled nodes or edges, (b) nodes with or without anchors, and (c) nodes and edges with optional properties and attributes, respectively (see the mrp metric generalizes earlier frameworkspecific metrics @xcite @xcite in terms of decomposing each graph into sets of typed tuples, as indicated in figure 6 . to quantify graph similarity in terms of tuple overlap, a correspondence relation between the nodes of the goldstandard and system graphs must be determined. adapting a search procedure for the np-hard maximum common edge subgraph (mces) isomorphism problem, the mrp scorer will search for the node-to-node correspondence that maximizes the intersection of tuples between two graphs, where node identifiers (m and n in figure 6 ) act like variables that can be equated across the gold-standard and system graphs. 10 this means that during evaluation all information in the mrp graphs is con- cross-framework cross-lingual teams amr drg eds ptg ucca amr drg ptg ucca reference hitachi ozaki et al. (2020) úfal samuel and straka (2020) hit-scir dou et al. (2020) huji-ku arviv et al. (2020) iscas tju-blcu jbnu na and min (2020) úfal samuel and straka (2020) erg oepen and flickinger (2019) ## submissions and results six teams submitted parser outputs to the shared task within the official evaluation period. in addition, we received two submissions after the submission deadline, which we mark as 'unofficial'. we further include results from an additional 'reference' system by one of the task co-organizers, namely eds outputs from the grammar-based erg parser @xcite . the main task results are summarized in results show that the hitachi and úfal sub-missions share the first place for both tracks, and together rank first or second for almost all the individual frameworks (save for ucca parsing in the cross-lingual track, where hitachi ranks third). hit-scir further ranks second for ucca parsing in both tracks. interestingly, rankings in the perframework track are similar across frameworks, which may indicate some similarity in the parsing problem exhibited by different linguistic schemes, despite differences in structure and content. per-framework scores using the official mrp metric are given in ## overview of approaches compared with systems from mrp 2019, there has been a fairly clear shift in approaches for participating systems this year, resulting in significant improvements in performance. the improvements for some of the frameworks are fairly substantial. for example, the hitachi system, one of the two winning systems, achieves a score of 0.82 f 1 in amr parsing, in comparison to 0.73 f 1 achieved by the top amr parser in mrp 2019. this reflects an improvement of over eight points, reflecting a number of innovations from the participants this year, as well as contemporaneous developments outside the shared task (see §8). broadly speaking, top performers at mrp 2020 have all adopted a system architecture that is based on an encoder-decoder framework in which the input sentence is encoded into contextualized token embeddings that are used as input to the decoder. the system vary in the decoding strategies. the hitachi system adopts a transformer-based encoder-decoder architecture. the system uses the standard transformer encoder in which selfattention and position embeddings are used to compute the contextualized token embeddings. in its decoder, this system has a number of innovations, however. first of all, the system rewrites the meaning representation graphs into a reversible plain graph notation (pgn), and enhances pgn with a number of pseudo-nodes that indicate the end of node prediction, the end of label prediction, etc. these correspond well with parsing actions commonly found in transition-based systems. in this sense, the systems combines the strengths of graphbased parsing on the encoder side resulting from self attention with efficiency of transition-based parsing on the decoder side. another innovation is the use of a 'hierarchical' decoding process in which the model first predicts a mode, and then predicts the next action conditioned on the mode. for example, if the mode is g(raph), the decoder predicts a meta node, and if the mode is s(urface), the decoder predicts the node label of a specific concept. this allows a fair competition among actions that are similar in nature. the perin system computes contextualized token embeddings with xlm-r @xcite on the encoder side, and then on the decoder side, uses separate attention heads to predict the node labels, identify anchors for nodes, and predict edges between nodes, as well as edge labels. because the label set for nodes is typically very large, rather than predicting the node labels directly, the perin system reduces the search space by predicting 'relative rules' that can be used to map surface token strings to node labels in meaning representation graphs, an idea that is similar to the use of factored concept labels in @xcite . another innovation of the perin system is that it is trained with a permutation-invariant loss function that returns the same value independently of how the nodes in the graph are ordered. this captures the unordered nature of nodes in (most of the mrp 2020) meaning representation graphs and prevents situations in which the model is penalized for generating the correct nodes in an order that is different from that in the training data. the hit-scir and jbnu systems adopt the iterative inference framework first proposed by @xcite for flavor (2) meaning representation graphs that do not enforce strict correspondences between tokens in the input sentence and the concepts in meaning representation graphs. the iterative inference framework is also based on an encoder-decoder architecture. the encoder takes the sentence as input and computes contextualized token embeddings that are used as text memory by a decoder that iteratively predicts the next node given the text memory and a predicted parent node in the partially constructed graph memory at the previous time step, and then identifies the parent node for the newly predicted node from the partially constructed graph. while the hit-scir system essentially uses the cai and lam (2020) architecture with little modification, the jbnu system attempts to extend the work of @xcite by using a shared state to make both predictions but did not observe substantial improvements. transition-based systems, which had achieved strong performance in the 2019 shared task, are also represented in the competition this year. the hit-scir team uses a transition-based system to parse flavor (1) meaning representations where there is a stricter correspondence between tokens in the input sentence and concepts in the meaning representation graph. the hit-scir transition-based system is essentially the overall top performing system they developed for mrp 2019. it uses stack lstm to compute transition states in the parsing process, and the parsing actions are tailored to specific meaning representation frameworks. in the training process, the system fine-tunes bert contextualized encodings. the huji-ku system also extends an entry in the 2019 mrp shared task (originally called tupa) to parse additional frameworks and handle meaning representation parsing in a multilingual setting. tupa is a transition-based system that supports general dag parsing. tupa applies separate constraints tailored to each meaning representation framework. when parsing cross-framework meaning representations for english, the system is trained with a bert-large-cased pretrained encoder, and when parsing cross-lingual meaning representations, it is trained with multilingual bert. ## on the state of the art mrp 2019 @xcite yielded parsers for five frameworks in a uniform format, of which eds, ucca, and amr are represented in mrp 2020 again. submissions included transition-, factorization-, and composition-based systems, and gold-standard target structures in 2019 were solely for english. comparability is limited by the fact that two of the 2020 frameworks (ptg and drg) are new, training and (in particular) evaluation sets for the others have been updated since mrp 2019, and additional validation sets was introduced. however, the lpps evaluation sub-corpus (le petite eds ucca amr p r f p r f p r f 2019 .92 .93 .93 .84 .82 .83 .74 .72 .73 2020 .97 .97 .97 .86 .80 .83 .78 .79 .79 prince) is identical between the two years for eds, ucca, and amr. this allows a comparison on nearly equal grounds: as for eds, the strongest results were obtained in the mrp 2019 official competition by suda-alibaba @xcite . however, in the post-evaluation stage, they were outperformed by the peking system @xcite . both used factorization-based parsing with pre-trained contextualized language model embeddings (which has consistently proved to be very effective for other frameworks too). these parsers even approached the performance of the carefully designed grammarbased erg parser @xcite . english ptg has not been comprehensively addressed by parsers prior to mrp 2020, but a bilexical framework called psd is a subset of ptg. it was included in the sdp shared tasks @xcite as well as in mrp 2019, and has been addressed by numerous parsers since @xcite @xcite . @xcite established the state of the art in supervised psd using a second-order factorization-based parser, and fernández-gonzález and gómez-rodríguez (2020) matched it using a stack-pointer parser. czech ptg, in its original form as published in the prague dependency treebank @xcite , has been used in several version of the tectomt machine translation system @xcite ; however, parsing results have not been published separately. a (lossy) conversion has been included in the conll 2009 shared task on semantic role labeling @xcite , but the differences in task design are and conversion make empirical comparison impossible. ucca parsing has been dominated by transitionbased methods @xcite @xcite . however, both english and german ucca parsing featured in a semeval shared task @xcite , where the best system, a composition-based parser @xcite , treated the task as constituency tree parsing with the recovery of remote edges as a postprocess-ing task. prior to mrp 2019, @xcite parsed amr using a joint probabilistic model with latent alignments, avoiding cascading errors due to alignment inaccuracies and outperforming previous approaches. @xcite recently improved the latent alignment parser using stochastic softmax. @xcite trained a composition-based parser on five frameworks including amr and eds, using the apply-modify algebra, on which the third-ranked saarland submission to mrp 2019 was based @xcite . they employed multi-task training with all tackled semantic frameworks and ud, establishing the state of the art on all graph banks but amr 2017. since then, a new state-of-the-art has been established for english amr, using sequenceto-sequence transduction @xcite and iterative inference with graph encoding @xcite . @xcite improved sequence-to-sequence parsing for amr by using pre-trained encoders, reaching similar performance to @xcite . @xcite introduced a stack-transformer to enhance transitionbased amr parsing (ballesteros and al-onaizan, 2017), and @xcite improved it further, using a trained parser for mining oracle actions and combining it with amr-to-text generation to outperform the state of the-art. @xcite parsed chinese amr with a transition-based system. for cross-lingual amr parsing, @xcite trained an amr parser similar to the approach of @xcite , using cross-lingual transfer learning, outperforming the transition-based cross-lingual amr parser of @xcite on german, spanish, italian, and chinese. drg is a novel graph representation format for drs that was specially designed for mrp 2020 to make it structurally as close as possible to other frameworks @xcite . however, several semantic parsers exist for drs, which employ different encodings. @xcite used a drg format that dominantly labels edges compared to nodes. @xcite process drss in a clausal form, sets of triples and quadruples. the latter format is more common among drs parsers, as it was officially used by the shared task on drs parsing @xcite . the shared task gave rise to several drs parsers: evang (2019 2019 ), among which the best results (f 1 = 0.85) were achieved by the word-level sequence-to-sequence model with tranformer @xcite . note that the drs shared task used f 1 calculated based on the drs clausal forms, which is not comparable to mrp f 1 over drgs. similarly to english drg, german drg has not been used for semantic parsing prior to the shared task due to the new drg format. moreover, semantic parsing with german drg is novel in the sense that its drs counterpart is also new. in german drg, concepts are grounded in english wordnet 3.0 @xcite senses assuming that synsets are language-neutral. the mismatch between german tokens and english lemmas of senses must be expected to add additional complexity to german drg parsing. direct comparison to non-mrp results is impossible: we are using a new version of amrbank. gold-standard tokenization is not provided for any of the frameworks. we use the mrp scorer. however, general trends appear consistent with recent developments. pretrained embeddings and crosslingual transfer help; but multi-task learning less so. there is yet progress to be made in sharing information between parsers for different frameworks and making better use of their overlap. ## reflections and outlook the mrp series of shared tasks has contributed to general availability of accurate data-driven parsers for a broad range of different frameworks, with performance levels ranging between 0.76 mrp f 1 (english ucca) and 0.94 f 1 (english eds). parsing accuracies in the cross-lingual track present comparable levels of performance, despite limited training data in the case of ucca and drg. furthermore, the evaluation sets for most of the frameworks comprise different text types and subject matters-offering some hope of robustness to domain variation. we expect that these parsers will enable follow-up experimentation on the utility of explicit meaning representation in downstream tasks like, for example, relation extraction, argumentation mining, summarization, or text generation. maybe equally importantly, the mrp task design capitalizes on uniformity of representations and evaluation, enabling resource creators and parser developers to more closely (inter)relate representations and parsing approaches across a diverse range of semantic graph frameworks. this facilitates both quantitative contrastive studies (e.g. the 'postmortem' analysis by @xcite , which observes that top-performing mrp 2019 parsers have complementary strengths and weaknesses) but also more linguistic, qualitative comparison. general availability of parallel gold-standard annotations over the same text samples-drawing from the wsj and lpps corpora-enables side-by-side comparison of linguistic design choices in the different frameworks. this is an area of investigation that we hope will see increased interest in the aftermath of the mrp task series, to go well beyond the impressionistic observations from §3 and ideally lead to contrastive refinement across linguistic schools and traditions. despite uniformity in packaging and evaluation, cumulative overall complexity and inherent diversity of the frameworks deemed participation in the shared task a formidable challenge. of the sixteen teams who participated in mrp 2019, only four teams (predominantly strong performers from before) decided to submit parser outputs in 2020. the two 'newcomer' teams, by comparison, only made partial submissions in the cross-lingual track and ended up not competing for top ranks overall. similar trends of 'competitive self-selection' and declining participant groups for consecutive instances have been observed with earlier conll shared task and similar benchmarking series. on the upside, with the possible exception of english amr (where there has been much contemporaneous progress recently), the mrp 2020 empirical results present a strong state-of-the-art benchmark for meaning representation parsing. on the more foundational question of the relevance of explicit, discrete representations of sentence meaning, the past several years of breakthrough neural advances have been comparatively insensitive to syntactico-semantic structure. in our view, these developments have at least in part been reflective of the stark lack of general techniques for the encoding of hierarchical structure in end-to-end neural architectures. increased adoption of graph convolutional networks @xcite and other hierarchical modeling techniques suggest new opportunities for the exploration of both structurally informed end-to-end archictures or e.g. multi-task learning setups. beyond such ultimately performance-driven research, explicit encoding of syntactico-semantic structure in our view further bears promise in terms of model interpretability and safe-guarding against 'neural meltdown' (e.g. discarding something as foundational as negation or inadvertently altering a date expression in summarization or translation). in a similar vain, meaning representations are being successfully applied in evaluation, e.g. to quantify system output vs. gold standard similarity beyond surface n-grams @xcite . all technical information regarding the mrp 2019 shared task, including system submissions, detailed official results, and links to supporting resources and software are available from the task web site at: http://mrp.nlpl.eu