Multitask Learning for Cross-Lingual Transfer of Broad-coverage Semantic Dependencies (2020)

Abstract: We describe a method for developing broad-coverage semantic dependency parsers for languages for which no semantically annotated resource is available. We leverage a multitask learning framework coupled with annotation projection. We use syntactic parsing as the auxiliary task in our multitask setup. Our annotation projection experiments from English to Czech show that our multitask setup yields 3.1% (4.2%) improvement in labeled F1-score on in-domain (out-of-domain) test set compared to a single-task baseline.

## introduction broad-coverage semantic dependency parsing (sdp) 1 was first introduced in the semeval shared task @xcite and aims to provide semantic analysis of sentences by capturing semantic relations between all content-bearing words in a sentence. the rich graph structure introduced by sdp allows the model to cover a wide range of semantic phenomena such as negation, comparatives, possessives and various types of modifications that have not been previously analyzed in other models such as semantic role labeling @xcite . despite all advantages provided by sdp, resources with annotated semantic dependencies are limited to the three languages released in the se-meval shared tasks @xcite @xcite namely english, czech and chinese. this data scarcity motivates us to use well-known and traditionally used transfer methods such as annotation projection for building sdp models for languages without semantically annotated data. in annotation projection, we assume that we have access to sentence-aligned corpora that can be used for transferring semantic annotations from a richresource source language to the target language 1 we use broad-coverage semantic dependencies and semantic dependencies interchangeably throughout this paper. top pat-arg twhen top pat-arg twhen figure 1: projecting sdp annotations from an english to a czech sentence. semantic dependencies of the english sentence (top) are projected using alignments (dashed lines in the middle) to obtain projected semantic dependencies (bottom) for the target sentence. motivated by the large amount of similarities between syntactic and semantic dependencies, we further propose a simple but effective multitask learning framework to leverage supervised syntactic parse information and improve the representation learning capability in the intermediate layers of our semantic parser. our multitask learning approach, despite its simplicity, yields significant improvements in the performance of the vanilla semantic dependency parser built using annotation projection. we conducted annotation projection experiments from english to czech. our experiments show that our multitask setup yields 3.1% and 4.2% improvement in the labeled f1 results on in-domain and out-of-domain evaluation sets respectively. furthermore, we explore the efficacy of contextualized word representations, bert @xcite and elmo @xcite as features in our annotation projection model and find a marginal gain by using those contextual features. to the best of our knowledge, this work is the first study to develop an enhanced semantic depen-dency parser through multitasking in the absence of annotated data. ## related work after the semeval shared tasks on broad-coverage semantic dependency parsing @xcite @xcite , there have been many studies to build supervised sdp models @xcite @xcite @xcite @xcite , however, all efforts were restricted to the three languages released through semeval shared tasks. there have been extensive number of studies that use annotation projection to cure data scarcity in different tasks such as part-of-speech tagging @xcite , syntactic parsing @xcite , semantic role labeling @xcite and semantic parsing @xcite . nevertheless, none of the previous works, to the best of our knowledge, looked into using annotation projection for building sdp models for languages without semantically annotated data. motivated by the fact that different semantic representations or formalisms cover different aspects of sentence-level semantics, there has been a line of studies to apply multitask learning over different semantic annotations @xcite @xcite or target cross-framework meaning representation @xcite . these studies use the shared semantic information across different representations to enhance the sdp model for a given language, however, none of them addressed the case that no semantically annotated data is available for a language. this paper is the first work that aims to build an sdp model based on cross-lingual transfer without any annotation in the target language of interest. ## the parsing model for an input sentence @xmath0, • • • , x n with n words, the goal of a semantic dependency parsing model is to learn binary dependency decisions y i,@xmath1, 1} for every head index 0 ≤ @xmath2, where x 0 is the dummy root token. for every head-dependent pair (i, j), such that y i,@xmath3, the parser finds a label l i,j from a set of predefined semantic dependency labels l. in most cases, the parsing decision is decomposed in two steps: unlabeled dependency parsing, and labeling each dependency edge. the only constraint here is that the final semantic graph should be acyclic. we use the standard model of @xcite for which the parsing model is based on a simple head selection algorithm. this model learns dependency edge scores s edge (i, j) for all possible head-dependent pairs (i, j). the final parsing decision is a sign function: similarly, the parser learns a labeling function s label l (i, j) for every pair that y i,@xmath4: our parsing model uses a deep neural model in which the first layer is the embedding layer that consists of word, part-of-speech tag, and character representations. the second layer consists of deep bidirectional lstms @xcite ) that construct recurrent representations r i for every word. the third layer uses four singlelayer feed-forward neural networks (fnn) as attention mechanisms for head and dependent binary decisions and label assignments. the final layer uses a bilinear function to score the fnn outputs. for training the model, the sigmoid cross-entropy function is used for the edges, and the softmax cross-entropy function is used for the labels. the two losses are interpolated to calculate the final loss value with a coefficient 0 < λ < 1. ## projecting semantic dependencies for a source sentence @xmath5, • • • , x m with m words, and a target sentence @xmath6, • • • , x n with n words, we obtain one-to-one alignments by running an unsupervised word alignment algorithm on both directions. we use the intersected alignments @xmath7, • • • , a m such that 0 ≤ a @xmath8. for every source dependency relation y i,@xmath9, 1} where a i , a @xmath10, we project the dependency edge and label to the target sentence y a i ,a @xmath11,j and l a i ,a @xmath12,j (if @xmath13. we then train a supervised parsing model on the projected dependencies. these projected dependencies are usually partial and contain some noise that are caused by different reasons such as translation shifts and alignment errors. ## multitask learning with syntax modeling auxiliary tasks in a multitask learning framework allows the main task to benefit from structural or statistical similarities found in one or more auxiliary tasks to improve the model learned for a target task @xcite . given the large amount of (labeled and unlabeled) correlations existing among syntactic and semantic dependencies, we consider syntactic dependency parsing as the auxiliary task for semantic dependency parsing. in order to find out the best parameter sharing structure, we try the following parameter sharing variations: 1) sharing embedding and recurrent layers, 2) sharing embedding and recurrent layers with an additional task-specific recurrent layer, 3) sharing all three layers, but with an additional task-specific recurrent layer, and 4) sharing all intermediate layers. figure 2 shows the first case for which only the first two layers are shared between the two tasks. the overall loss value for the multitask model is computed by interpolating semantic and syntactic losses using an interpolation coefficient ω which is tuned on the development data. we use projected semantic dependencies and syntactic dependency parses generated using a supervised parser to train the multitask model. thus the training data for the target language has projected semantic annotations plus fully parsed syntactic trees. ## experiments and results we consider english as the source language and czech as the target language. we use the semeval 2015 @xcite in-domain and out-ofdomain test sets to evaluate our models. since the psd (prague semantic dependencies) annotation is available for both english and czech, we use that throughout our experiments. we use giza++ @xcite with its default configuration to obtain intersected word alignments on the europarl parallel corpus @xcite . the training data used in our projection experiments is drawn from europarl which contains text from the political domain. the in-domain czech test set provided by the semeval 2015 contains translated texts from corresponding sections of wsj in the newswire domain, whereas the out-of-domain evaluation set for czech (also provided by semeval 2015) is drawn from prague dependency treebank 3.0 @xcite which mainly contains text from journals and scientific articles, thus considered of a fairly differ-ent domain compared to europarl (political). we explore efficacy of multitasking in our annotation projection model by comparing the multitask results with the single-task baseline model that does not use any multitasking. the training corpus of czech with projected annotations contains 612k sentences but due to computational limitations, we train all models on a sample of 80k sentences foot_0 randomly selected from original projections. in order to simulate a fully unsupervised approach, we use 5% of the projected data as the held-out data during training. parsing parameters we use the structural skipgram model of @xcite for english word embeddings and run word2vec @xcite on wikipedia text to acquire the word vectors for czech. we use udpipe (straka and straková, 2017) pretrained models v1.2.0 (trained on the universal dependencies v2.0) to produce automatic part-of-speech tags. we train the biaffine dependency parser of @xcite on the universal dependencies corpus v2.0 @xcite to generate supervised syntactic parses in our multitask learning experiments. all modules are implemented using the dynet library @xcite . we mainly use the hyper-parameters used in @xcite except that we use a character bilstm without any linear transformation layers. we use word and part-of-speech vectors of size 100, with 3-layer lstms of size 600, and feed-forward layers of size 600. we use a dropout of probability of 0.2 for words and partof-speech tags, and 0.25 for the recurrent and unlabeled feed-forward layers, and 0.33 for the labeled feed-forward layers. the interpolation constants λ and ω are set to 0.025 and 0.975 respectively to prioritize the semantic task as our main task in the multitask framework. we use the adam optimizer (kingma and ba, 2014) with a learning rate of 0.001 on minibatches of approximately thousand tokens. we also concatenate the contextual vectors to the input layer as additional features to the parser. we use the pretrained elmo embeddings @xcite of size 1024 from @xcite . their model is trained on the set of 20-million-words data randomly sampled from the raw texts released by the conll 2018 shared task for czech and uses the same model and hyper-parameters as @xcite . we use the pretrained multilingual bert models @xcite of size 768 from xiao (2018) with 12 layers and 12 heads. due to computational limitations, we only use the pretrained bert models in the input layer without finetuning. ## conclusion we have described a semantic dependency parsing model based on annotation projection that do not use any annotated semantic data in the target language. we enhance the target semantic model by incorporating syntax in a multitask learning framework. we demonstrate that our multitask model outperforms the single-task model on both in-domain and out-of-domain test sets on the czech language.