Frustratingly Simple but Surprisingly Strong: Using Language-Independent Features for Zero-shot Cross-lingual Semantic Parsing (2021)

Abstract: The availability of corpora has led to significant advances in training semantic parsers in English. Unfortunately, for languages other than English, annotated data is limited and so is the performance of the developed parsers. Recently, pretrained multilingual models have been proven useful for zero-shot cross-lingual transfer in many NLP tasks. What else does it require to apply a parser trained in English to other languages for zero-shot cross-lingual semantic parsing? Will simple language-independent features help? To this end, we experiment with six Discourse Representation Structure (DRS) semantic parsers in English, and generalize them to Italian, German and Dutch, where there are only a small number of manually annotated parses available. Extensive experiments show that despite its simplicity, adding Universal Dependency (UD) relations and Universal POS tags (UPOS) as model-agnostic features achieves surprisingly strong improvement on all parsers.

## introduction semantic parsing is the task of transducing natural language to meaning representations, which in turn can be expressed through many different semantic formalisms including discourse representation theory (drt) @xcite , abstract meaning representation (amr) @xcite , and so on. however, manually annotating meaning representations in a new language is a painstaking process which explains why there are only a few datasets available for different formalisms in languages other than english. for instance, in the case of drt, even in the latest release of the parallel meaning bank (pmb) (v3.0, †work done while federico fancellu was at the university of edinburgh. @xcite , there is no gold training data available for italian and dutch and the few annotated sentences are used as development and test data. how can one train a parser when training data is unavailable? this work answers this question by looking at what it is required for zeroshot cross-lingual semantic parsing -learning a semantic parser for english and testing it on other languages. prior research on cross-lingual semantic parsing leveraged machine translation techniques to map the semantics from a language to another (damonte and cohen, 2018). however, these methods require parallel corpora to extract automatic alignments which are often noisy or not available at all. other work exploited parameter-shared models, which are based on language-independent representations. in particular, cross-lingual word embeddings and pretrained multilingual models @xcite have been used in various cross-lingual nlp tasks, including semantic parsing @xcite . however, pretrained multilingual models are computationally expensive while cross-lingual word embeddings have inferior performance. to this end, we propose to add simple language-independent features in zero-shot crosslingual semantic parsing, which are lightweight extensions to all models instead of designing new architectures with high time and space complexity. specifically, we explore cross-lingual syntactic features, universal dependencies (ud) and universal pos tags (upos) @xcite , which have been widely annotated in 90 languages. since semantic parsers need to understand the syntax of sentences well, and ud relations and upos have been shown to be strong indicators of semantic roles @xcite , we hypothesize that using ud and upos as simple languageindependent and model-agnostic features can boost the performance of zero-shot cross-lingual semantic parsing in all models and languages. to test our hypothesis, we focus on the pmb, where sentences in english, german, italian and dutch are annotated with their meaning representations. the annotations in the pmb are based on discourse representation theory (drt, @xcite . figure 1 shows an annotation example usign drt for the sentence "ich bin schläfrig. (i am sleepy.)". a discourse representation structure (drs) is a nested structure with unary and binary predicates representing semantic roles, alongside logic operators (e.g. ¬) and discourse relations (e.g. continuation). in this example one can see that although a parser could understand the coarse meaning of the whole sentence without ud, it struggles with understanding some lexical-level meaning. however, it successfully identifies "schläfrig" as the event, based on its ud relation tag "root". we carry out our experiments on 6 drs parsers to test different architectures (lstm vs. transformers) as well as different decoding strategies (sequential vs. coarse-to-fine). whereas the original parsers utilize a sequential neural encoder with monolingual representations, we experiment with cross-lingual representations (i.e. cross-lingual word embeddings and a multilingual pretrained encoder), and language-independent features (i.e. upos, ud relations and structures). we also use tree-based encoders to replace the sequential encoder, in order to assess whether modelling syntax is beneficial. results show that adding ud relations and upos as features, despite its frustrating simplicity, leads to surprisingly strong zero-shot crosslingual semantic parsers, even when ud are the only input used during encoding. also, ud further boost the performance of strong pretrained multilingual models. surprisingly, small non-pretrained well-designed coarse-to-fine decoding models with ud relations and upos even outperform large pretrained multilingual models in some languages. a drs is usually represented in a nested 'box' structure. however, the systems we reference in this paper all use a linearization of this box representation. to avoid any confusion, we only show examples of linearized drs throughout the paper. ## related work previous work have explored two main methods for cross-lingual semantic understanding. one method requires parallel corpora to extract alignments between source and target languages using machine translation @xcite @xcite , often followed by projection of semantic representations @xcite . the other method is to use parameter-shared models based on cross-lingual representations such as cross-lingual word embeddings @xcite @xcite @xcite , pretrained multilingual models @xcite @xcite , and universal pos tags @xcite . recently, @xcite ; @xcite ; @xcite conducted supervised german drs parsing with pretrained multilingual models, but they did not explore zero-shot cross-lingual semantic parsing. besides, although ud was proven useful in other cross-lingual tasks @xcite , it has been under-explored in cross-lingual semantic parsing. ## conclusion this work proposes to use simple languageindependent features for the task of zero-shot crosslingual semantic parsing. we show that simple ud and upos features can significantly improve the performance of cross-lingual semantic parsers based on coarse-to-fine decoding techniques or pretrained multilingual models. in the future, we plan to use such features for other semantic formalisms (e.g. amr) and other languages (e.g. chinese).