Project-then-Transfer: Effective Two-stage Cross-lingual Transfer for Semantic Dependency Parsing (2021)

Abstract: This paper describes the first report on cross-lingual transfer for semantic dependency parsing. We present the insight that there are twodifferent kinds of cross-linguality, namely sur-face level and mantic level, and try to cap-ture both kinds of cross-linguality by combin-ing annotation projection and model transferof pre-trained language models. Our exper-iments showed that the performance of our graph-based semantic dependency parser almost achieved the approximated upper bound.

## introduction cross-lingual dependency parsing attracted much attention for its powerful representational capability in grammatical and semantic lexical relations @xcite @xcite @xcite @xcite . several remarkable contributions have been made in syntactic dependency parsing, especially on universal dependencies (ud; @xcite . for example, @xcite showed that a single multilingually fine-tuned neural model utilizing a pre-trained language model could successfully parse 75 languages in ud with comparable performances to state-of-the-art parsers. however, cross-lingual semantic dependency parsing @xcite @xcite , which is totally different dependency structure from syntactic dependencies (shown in figure 1 ), has not been explored as far as we know. a reason for this is the lack of parallel graphbanks that cover many languages with consistent annotation policies. one exception is prague semantic dependencies (psd; @xcite , which is a treebank of bi-lexical semantic graphs and contains over 30,000 pairs of parallel annotated sentences from the wall street journal in english and czech. a technique is impossible to apply to crops considering these circumstances, we propose to train semantic dependency parsers by capturing commonalities across languages as a remedy for the absence of massive multilingual graphbanks. our work draws on the intuition that cross-linguality exists in both superficial level and semantic level. accordingly, we leverage a two-stage fashion involving treebank-based transfer and model-based transfer. treebank-based transfer, often called annotation projection, is a method of projecting source language annotations to a target language by using a mapping function such as word alignment. the annotation projection has been reported as a promising approach under truly low-resource settings for ud parsing @xcite . however, annotation projection often suffers from noise in word alignment @xcite . for the model-based transfer, several studies on transferring contextualized word vectors have reported that it improves the parsing performance @xcite . our experiments on psd graphbank indicate that the optimal performance can be achieved by incorporating the two-stage transfer. surprisingly, we observed improvement even when the projected treebank was erroneous. furthermore, the twostage transfer method achieved almost upper-bound performance, which was approximated by evaluating the cross-linguality of psd annotation through the projection. we also provide detailed analyses from both perspectives of cross-linguality. ## transfer strategies to enable cross-lingual semantic parsing, we focused on two different types of cross-linguality, namely cross-linguality on surface and crosslinguality in semantics. cross-linguality on surface lies on our hypothesis of typological correspondences among most of languages. for example, annotation projection, which is a treebank-based transfer, assumes cross-linguality on surface and projects source language annotations to a target language. on the other hand, cross-linguality in semantics is based on the assumption that lexical-, phrase-, or sentence-level meaning correspondences may exist among most of languages. recently, multilingual bert @xcite directly captures cross-linguality in semantics beyond lexicons by large-scaled language model training on parallel corpora. though both annotation projection and multilingual pre-trained model can handle either crosslinguality on surface or in semantics, we argue that they could not utilize both cross-linguality in effective way. hence, we propose two-stage transfer which incorporates both methods, to capture the two kinds of cross-linguality as possible. we firstly introduce the two transfer methods for applying them to psd graphbank, and then we explain our two-stage transfer method; project-then-transfer. ## conclusion this paper described transfer methods for crosslingual semantic dependency parsing. we showed that both cross-linguality on surface and in semantics were necessary to improve the performance. consequently, we achieved almost the upper bound performance approximated by the annotation projection. the results encouraged us to develop crosslingual semantic dependency parser for many languages. we will further conduct explore these models, and evaluations on cross-linguality across languages broadly. model parameters fast align -iteration: 10 -num. of trials: 1 -average runtime: less than 1min. graph-udify -batch size: 32 -learning rate: 1e-3 -activation: relu -beta: [0.99, 0.99] -tag representation dim: 128 -tag loss w: 0.1 -num. of trials: 32 for each -average runtime: 12 hrs. -num. of params.: 225695576 transition-bert -batch size: 4 -num. of trials: 1 -average runtime: 48 hrs. -num. of params.: 201555396