SGL : Speaking the Graph Languages of Semantic Parsing via Multilingual Translation (2021)

Abstract: Graph-based semantic parsing aims to represent textual meaning through directed graphs. As one of the most promising general-purpose meaning representations, these structures and their parsing have gained a significant interest momentum during recent years, with several diverse formalisms being proposed. Yet, owing to this very heterogeneity, most of the research effort has focused mainly on solutions specific to a given formalism. In this work, instead, we reframe semantic parsing towards multiple formalisms as Multilingual Neural Machine Translation (MNMT), and propose SGL, a many-to-many seq2seq architecture trained with an MNMT objective. Backed by several experiments, we show that this framework is indeed effective once the learning procedure is enhanced with large parallel corpora coming from Machine Translation: we report competitive performances on AMR and UCCA parsing, especially once paired with pre-trained architectures. Furthermore, we find that models trained under this configuration scale remarkably well to tasks such as cross-lingual AMR parsing: SGL outperforms all its competitors by a large margin without even explicitly seeing non-English to AMR examples at training time and, once these examples are included as well, sets an unprecedented state of the art in this task. We release our code and our models for research purposes at https://github.com/SapienzaNLP/sgl .

## introduction being able to associate natural language text with well-defined and machine-actionable meaning representations, i.e. the task of semantic parsing (sp), is one of the holy grails in natural language processing (nlp) and understanding @xcite . considering how a breakthrough in this direction would empower nlp systems to explictly make sense of natural language, the evergrowing interest semantic parsing has been receiving really comes as no surprise. graph-based formalisms such as abstract meaning representation @xcite , elementary dependency structures @xcite , prague tectogrammatical graphs @xcite , universal conceptual cognitive annotation @xcite , inter alia, are emerging as the de facto standard for general-purpose meaning representations and have shown potential in machine translation @xcite , text summarization @xcite , human-robot interaction @xcite , and as evaluation metrics @xcite . these formalisms propose encoding meaning through directed graphs, however, each of them builds upon different linguistic assumptions, aims to target different objectives and, at a more practical level, assigns different functions to nodes and edges. for instance, while amr uses nodes to encode concepts and edges to express the semantic relations between them, ucca proposes using text tokens as terminal nodes and building graphs on top of them. as a result of this heterogeneous landscape, often referred to as framework-specific balkanization @xcite , graph-based semantic parsing has seen a proliferation of framework-specific solutions. however, approaches capable of competitively scaling across formalisms represent a natural desideratum, and recent works have started to explore this direction, examining the usage of multi-task learning in different architectures @xcite , or casting different formalisms under a unified framework where models can be trained to perform graph transduction @xcite . nevertheless, despite achieving promising results, research in this direction has been hindered by the general lack of training data that afflicts semantic parsing. indeed, due to the inherent complexity of this task, annotated corpora are still scarce and prohibitively expensive to expand. in this work, we set ourselves to address these issues and, in particular, we propose speak the graph languages (sgl), a many-to-many seq2seq architecture which we show to competitively scale across formalisms and across languages. 1 the key idea is to train a seq2seq model with a multilingual neural machine translation (mnmt) objective, where, given an input text and an identifier denoting the desired output formalism, a single shared model has to learn to translate towards the corresponding linearized graph. we use amr and ucca as our cases in point to show the effectiveness of this framework. in particular, we show that, once the learning procedure also considers large parallel corpora coming from machine translation (mt), this configuration becomes an effective approach for framework-independent parsing via a single model. even more interestingly, this model scales surprisingly well to cross-lingual parsing and is capable of navigating through translation paths like @xmath0, 2 which it has never seen during training. the contributions of this work are therefore as follows: we release our code and our models for research purposes. our work is mainly concerned with semantic parsing in ucca and amr, considering also the cross-1 by across languages, we mean that the model is capable of performing cross-lingual semantic parsing as defined for amr by @xcite . unless otherwise specified, we will follow this perspective throughout the paper. ## speak the graph languages (sgl) in this section, we describe sgl, our proposed approach to graph-based semantic parsing. we first explain the graph linearizations we employ for amr and ucca, along with their delinearizations ( §3.1). we then describe the seq2seq modelling approach we use ( §3.2) and, finally, we present our multilingual framework ( §3.3). ## experimental setup we assess the effectiveness of our proposed approach by evaluating its performance on all translation paths where the target language is a graph formalism, the only exception being @xmath1, with x any language but english. this choice is motivated by the fact that, differently from amr where cross-lingual amr aims to produce englishbased meaning representations (damonte and cohen, 2018), ucca builds graphs on top of its tokens which are, consequently, inherently in the same language as the input text @xcite ; we leave exploring this direction to future work. ## results we now report the results sgl achieves focusing on the following translation paths: i) @xmath2.1); ii) @xmath3, with x any language among chinese, german, italian and spanish ( §5.2); iii) @xmath4.3). ## analysis: is mt the one helping? although the performances of cross mt are remarkable, mbart st achieves competitive results on cross-lingual parsing and fares even better on english parsing. while mbart st admittedly features a massive amount of pre-training, this pre-training is over monolingual corpora and, as such, the model has never seen any parallel data. we therefore wonder to what extent the parallel nature of the additional mt data we use is crucial for cross mt . to answer this question, we treat our mt corpora as monolingual data by sampling, for each instance, either the source or target side and converting the translation task into a denoising one: given an instance @xmath5, we sample either en or it with equal probability, denoting the result by z, and convert the instance into g(z) → z, where g is a noising function that corrupts the input text. we follow @xcite and choose a noising function that masks 35% of the words by random sampling a span length from a poisson distribution (λ = 3.5). applying this noisification scheme to the mt data, we train a model identical to cross mt and denote it by cross n mt . as shown in ## conclusion in this work, we presented sgl, a novel framing of semantic parsing towards multiple formalisms as multilingual neural machine translation. that is to say, given a sentence and the desired output formalism, a many-to-many neural model has to learn to translate from the input sentence to the corresponding linearized graph. within this framework, we show that we can address the paucity of annotated data that afflicts semantic parsing effectively by performing the learning procedure jointly on large parallel corpora coming from mt, and leveraging the power of pre-trained transformer language models. using amr and ucca as our cases in point, we report competitive performances on their parsing, especially once pre-trained models enter the picture. furthermore, we find that the benefit mt data provide goes beyond merely improving englishcentric parsing, yielding astonishing performances on cross-lingual amr parsing as well, and allowing sgl to outperform all existing approaches by a large margin. most interestingly, differently from all previous literature, this result is attained without ever explicitly seeing at training time the translation paths the model is tested upon. once we use annotation projection and include these data as well, performances rise even further, attaining unprecedented results. as future work, thanks to the nimbleness with which we can add new languages, we plan to assess the scalability of this framework as more formalisms are taken into account. mbart we use the mbart-large-cc25 model provided by the huggingface's transformers library @xcite and follow the specifications given in the original paper for the training configuration: we use 2500 warm-up steps, 3e -5 maximum learning rate, adam as the optimizer and 0.3 dropout. we used 8000 effective token batch size and carried out training on a cloud-provided nvidia geforce 3090, using semantic parsing accuracy as the early stopping criterion with 25k steps of patience; training lasted slightly less than 2 days for mbart st and around 4 days for mbart mt . as we fine-tune the original model, the number of parameters is unchanged (but for the two additional vectors for ucca and amr in the embedding matrix), with both mbart st and mbart mt featuring 610m trainable parameters.