Zero-shot cross-lingual Meaning Representation Transfer: Annotation of H ungarian using the P rague Functional Generative Description (2021)

Abstract: In this paper, we present the results of our experiments concerning the zero-shot cross-lingual performance of the PERIN sentence-to-graph semantic parser. We applied the PTG model trained using the PERIN parser on a 740k-token Czech newspaper corpus to Hungarian. We evaluated the performance of the parser using the official evaluation tool of the MRP 2020 shared task. The gold standard Hungarian annotation was created by manual correction of the output of the parser following the annotation manual of the tectogrammatical level of the Prague Dependency Treebank. An English model trained on a larger one-million-token English newspaper corpus is also available, however, we found that the Czech model performed significantly better on Hungarian input due to the fact that Hungarian is typologically more similar to Czech than to English. We have found that zero-shot transfer of the PTG meaning representation across typologically not-too-distant languages using a neural parser model based on a multilingual contextual language model followed by a manual correction by linguist experts seems to be a viable scenario.

## introduction two workshops on cross-framework meaning representation parsing (mrp) hosted by the 2019 and 2020 editions of the conference on computational natural language learning (conll), featured two editions of a shared task, where implementations of parsers turning raw text into different flavors of meaning representation graphs competed with each other. the first mrp 2019 task @xcite ) involved only english as the object language, and 5 frameworks of meaning representation were featured: dm, psd, eds, ucca and amr. two of these frameworks, dm (delph-in mrs bi-lexical dependencies, @xcite , and psd (prague semantic dependencies @xcite are simple bi-lexical dependency graphs generated from core predicateargument structure of a rich syntactic-semantic annotation based on general theories of grammar. the underlying linguistic theory is head-driven phrase structure grammar @xcite with minimal recursion semantics @xcite for dm and prague functional generative description @xcite for psd. in dm and psd, the nodes are surface word forms. the other three frameworks feature more complex graphs that contain nodes not in one-toone relation to input words. elementary dependency structures (eds) are based on english resource grammar @xcite aka english resource semantics (ers) @xcite ) annotation foot_0 that was turned into a variablefree semantic dependency graph consisting of labeled graph nodes representing logical predications and edges representing labeled argument positions. the conversion from ers to eds discards information on semantic scope. the nodes are anchored to spans of the input string. universal conceptual cognitive annotation (ucca, @xcite is an abstract annotation featuring only purely semantic categories and structure. the foundational layer of ucca (featured in the shared task) consists of a very basic set of semantic categories like process, argument, state, "adverb" (modifier) etc., that are used as labels on edges linking unlabeled nodes representing semantic units and surface word forms. finally, abstract meaning representation (amr, @xcite features graphs comparable to eds, but with more abstract predication labels due to application of lexical decomposition and normalization towards verbal senses e.g., representing 'similar' as the verbal sense 'resemble'. in contrast to eds (and the rest), amr nodes are not explicitly anchored to spans of the surface form. 2 the graphs in the mrp shared tasks were presented in a json-lines-based uniform graph interchange format, and participants were asked to design and train systems that predict sentence-level meaning representations in all frameworks in parallel to foster transfer and multi-task learning. in the mrp 2020 shared task @xcite , dm was dropped, as eds is a richer representation derived from the same resource. psd was also replaced by a richer meaning representation, prague tectogrammatical graphs (ptg, zeman and hajic, 2020), derived from the same prague functional generative description (fgd), but retaining more of the original tectogrammatical annotation than psd. eds, ucca, and amr were retained, and discourse representation graphs (drg, @xcite were added as a new meaning representation. drg is a graph encoding of discourse representation structures (drs), the meaning representations at the core of discourse representation theory (drt; @xcite . this model handles many challenging semantic phenomena from quantifiers to presupposition accommodation, and discourse structure. in addition to english data, the mrp 2020 task covered new languages, one for each of four of the five covered formalisms (except eds): czech for ptg, german for ucca and drg, and chinese for amr. training and evaluation data of the mrp shared task was only available to shared task participants distributed to them by the linguistic data consortium (ldc), since part of the data is based on ldc-owned material, the wsj part of the penn treebank (ptb). the shared task site states that, upon completion of each competition, subsets of task data that are copyright-free (including system submissions and evaluation results) will be made available for public, open-source download. however, unfortunately, we have not found a public release of the data. however, one of the top performing systems at mrp 2020, the perin parser @xcite , which was developed at the institute of formal and applied linguistics (úfal), faculty of mathematics and physics, charles university, prague, was made available at the úfal github repo foot_2 including pretrained models for the perin submission to the shared task. there is also a link to an interactive demo on google colab. this facilitated testing the models on various inputs. positive subjective impressions of the performance of the english and especially the czech ptg model of the parser on hungarian input prompted us to perform the experiment described here evaluating the zero-shot cross-lingual performance of the model. of the models available, we selected ptg, because • the categories/concepts it operates with looked immediately familiar, • the annotation it generated seemed reasonable and detailed, • the non-english model covers czech, a language sharing many typological features with hungarian (rich morphology, relatively free word order, pro drop etc.), • the model was trained on a sizable 740k-token corpus, • a rather detailed 1255-page annotation manual @xcite of the underlying prague tectogrammatical annotation is available in english, and • performance of the parsers (also of perin, in particular) on the czech ptg data reported in the mrp 2020 task results was relatively high. concerning the other formalisms featured in the mrp 2020 task, we had the following impressions, further motivating our model selection: ## related work although the mrp 2020 shared task featured a "cross-lingual" track, it only meant in practice that parsers were trained and tested on data in more than one language for meaning representations that had such annotation available. transfer from one language to another was not tested there. the prague dependency annotation scheme has been ported to languages other than czech or english, examples including the slovak dependency treebank @xcite , the paws treebank (including polish and russian in addition to english and czech, @xcite , and the prague arabic dependency treebank @xcite . however, all these syntactic annotations have been created manually. at the same time, there is a significant body of research literature describing work concerning crosslingual transfer using deep-neural-network-based models. multilingual pre-training of contextual language models like multilingual bert @xcite and xlm-roberta @xcite facilitated this kind of knowledge transfer. these models have been used to train massively multilingual syntactic dependency parsers (kondratyuk and straka, 2019), zero-shot named entity recognizers @xcite etc, with even specific multilingual benchmarks prepared for testing the cross-lingual generalization capability of models on various tasks such as sentence-pair classification, structured prediction (pos tagging, ner), question answering, natural language inference and sentence retrieval @xcite . in this paper, we examine a zero-shot approach to meaning representation transfer, which belongs to the structured prediction problem class. few studies address this topic, because evaluation requires tedious manual work. ## method the approach we took was the following. we had a 150-sentence hungarian corpus annotated by the perin czech ptg model. we turned this into a gold standard hungarian annotation by manually correcting the output of the parser following the annotation manual of the tectogrammatical level of the prague dependency treebank @xcite . members of the annotation team had solid training in theoretical and computational linguistics and cognitive science encompassing both dependency syntax and formal semantics. however, we had to understand and learn details of the annotation scheme during the process, which required substantial effort. fortunately, examples in the annotation guidelines have english translation. however, only a few illustrative examples have a full tree representation. we had to reiterate and rediscuss our solutions several times to converge on an annotation that we considered consistent with what is described in the pdt annotation guidelines. 50-sentence folds were annotated, discussed and reannotated several times, as our understanding of the annotation scheme evolved during the process. access to pcedt would have been very helpful, however, only the czech part of treebank is available on line, foot_5 so we could not take a look at the english translations, or efficiently search for specific constructions not being speakers of czech. when doing the manual annotation correction, we had to refrain from making modifications to the annotation scheme if we 'disliked' the way a specific phenomenon is handled (or ignored) in the original scheme. we also tried to refrain from interpreting dubious situations 'the way we would have made it', we tried to figure out instead, how úfal experts would do it. we assumed that if the parser more-or-less consistently generates some sort of sufficiently sensible annotation for a specific construction, it reflects a deliberate annotation pattern in the training data. the serialized json lines graph representations output by the parser cannot be edited themselves, although they can be visualized using mtool, foot_6 the swiss army knife for graph-based meaning representation. we thus created a converter from the json lines graph representations to conll-u foot_7 (using anchors to project the data) and vice versa, and we edited the graphs in the conll-u format. we used mtool to visualize our gold standard solutions while we edited the annotations and also to evaluate the zero-shot output of the parser against the edited gold standard version. based on the graph configurations, mtool creates potential node-to-node mappings between the two graphs, so the fact that the ordering of nodes is changed during conversions is not a problem from the point of view of evaluation and visualization. ## results in addition to visualization, we also used mtool to evaluate the zero-shot output of both the czech and the english ptg models against the edited gold standard version of the test corpus. english ptg has less node features than the czech model, and also the edge labels generally contain no subfunctor annotation. the english model also uses different patterns to generate node labels (lemmata), so the performance of the models would not be comparable without applying some sort of normalization to the annotations before comparison. the normalization we performed included a) replacement of node labels (lemmata) by the sequence of tokens anchored to the node (with the exception of unanchored tokens, which retained their labels), and b) removal of subfunctor annotation from edge labels (except for subtypes of coref and bridging relations, as these also have subfunctors in the english annotation). performance of the normalized output of the models as returned by mtool is compared in below we discuss specific details of the performance of the czech model on hungarian input. ## conclusion in this paper we presented our experiment investigating the zero-shot cross-lingual performance of neural parser models based on the pdt/ptg meaning representation formalism. the model yields reasonable performance, and it can be feasibly applied in a semi-automatic annotation scenario. the specific language pairs were czech-hungarian vs. english-hungarian. the former model performs better because the source and the target language share more typological characteristics like rich morphology, free word order, pro drop etc. even though they belong to different language fam-ilies. moreover, the pdt/ptg annotation scheme utilizing a rich set of dependency relations as edge labels seems to perform much better than, e.g., eds where edge labels are completely abstract. fortes of the pdt/ptg model: