Multi-Source Cross-Lingual Constituency Parsing (2021)

Abstract: Pretrained multilingual language models have become a key part of cross-lingual transfer for many natural language processing tasks, even those without bilingual information. This work further investigates the cross-lingual transfer ability of these models for constituency parsing and focuses on multi-source transfer. Addressing structure and label set diversity problems, we propose the integration of typological features into the parsing model and treebank normalization. We trained the model on eight languages with diverse structures and use transfer parsing for an additional six low-resource languages. The experimental results show that the treebank normalization is essential for cross-lingual transfer performance and the typological features introduce further improvement. As a result, our approach improves the baseline F1 of multi-source transfer by 5 on average.

## introduction recent pretrained multilingual language models have become a key step in cross-lingual transfer for many natural language processing tasks such as name entity recognition, part-of-speech tagging, natural language inference, and dependency parsing @xcite . these models are desirable in research on cross-lingual transfer because bilingual information is not required. cross-lingual transfer is when a trained model for a source language is applied to a target (unseen) language. there are two transfer scenarios, singlesource and multi-source transfer. for single-source transfer, each time, the model is trained on only one source language. in this scenario, multiple models are available for cross-lingual transfer in practice. additional model selection is necessary for singlesource transfer because cross-lingual transfer relies on language isomorphism. for multi-source transfer, to leverage all existing resources, treebanks of multiple languages are combined to train a multilingual parser that can be later used for any unseen language. in this work, we study the multi-source transfer for sophisticated structure prediction, i.e., constituency parsing. our work will serve as a benchmark for cross-lingual constituency parsing using pretrained multilingual language model. for constituency parsing, training a multilingual parser has two main issues that must be considered. first, the source languages can produce diverse word orders-for instance, different subjectverb-object or noun-adjective orders. these language properties can be simply identified using existing typology databases, e.g., the world atlas of language structures (wals) or syntactic structures of the world's languages (sswl). it is intuitive that these language properties can be used to guide a multilingual parser to share corresponding model parameters among similar languages @xcite @xcite . for crosslingual transfer, the typological features could hurt performance @xcite , and an effective integration technique is required @xcite . inspired by this, we investigate the usefulness of typological features for cross-lingual constituency parsing and propose a training strategy to generalize the cross-lingual capability of the model using smooth sampling and random dropout. the second issue is that even though constituency structure is universal, the design of a label set is language specific. for dependency structures, this problem has inspired the creation of the universal dependency project @xcite . the syntactic label sets of constituency structure vary across languages-for instance, very few labels are shared and even labels for the same syntactic category may be different across languages. this increases the complexity of multi-source transfer. therefore, we propose normalization of the constituency treebanks to universalize the multilingual parsing model. the contributions of this paper are summarized as follows: 1) typological feature integration for model generalization on unseen languages (section 4.1), and 2) treebank normalization is proposed to reduce the complexity of cross-lingual structural prediction (section 4.2). ## related work in multi-source transfer, task-specific knowledge of multiple source languages is combined and jointly transferred to an unseen or zero-shot language. this combination can be categorized according to three levels @xcite , that is, the level of treebanks @xcite @xcite , model parameters @xcite , or parse outputs @xcite . this work focuses on treebank level, that is, treebank concatenation and, unlike previous studies, we study a more sophisticated structure, constituency treebanks, which simultaneously contain diverse syntactic labels across multiple source languages. typological features are a valuable resource for multi-source transfer where source languages have diverse structures, and they have been used specifically for sharing the parameters of non-neural @xcite @xcite and neural @xcite @xcite models. following the same motivation, we also investigate the usefulness of typological features for a multilingual constituent parser and propose a training strategy that generalizes the model for zero-shot languages. specifically, we integrate typological features into the self-attentive constituency parser @xcite . our work is similar to that of @xcite who investigated the multilingualism of the selfattentive constituency parser @xcite using the pretrained multilingual language model. however, our work differs from theirs such that we focus on zero-shot performance. in addition, we propose to normalize the concatenated treebanks and integrate typological features for better zero-shot performance. we also extend the sampling technique that @xcite use by constraining the minimum size of each treebank. ## the self-attentive parser the basis of our model (fig. 1b ) follows the selfattention based encoder-decoder architecture of @xcite . specifically, the encoder consists of word embedding and self-attention layers to produce the contextual presentation for each word. at the decoder side, all possible spans are extracted and each span (i, j) is represented by a hidden vector v i,j that is constructed by subtracting the representations associated with the start and end of the span. then, each span (i, j) is assigned a labeling score s(i, j, •) by an mlp span classifier as at test time, the optimal structure can be obtained using a cky-style inference algorithm. for training, the model is optimized using a maxmargin objective function, the details of which can be found in @xcite . in addition, the parser's hyperparameters are unchanged from @xcite . to perform cross-lingual parsing, an external pretrained multilingual language model must be used and simply take the place of the word embedding layer. because the model is trained on sub-words, only the last sub-word unit of the corresponding token is used to represent a word. in this experiment, we use a recent multilingual language model, i.e. xlm-roberta-large @xcite . ## conclusion we demonstrated the strong ability of recent pretrained multilingual language models for crosslingual constituency parsing. this result will serve as a new benchmark for future cross-lingual constituency parsing. moreover, we found that our treebank normalization is crucial when training multilingual treebanks with diverse label sets. in addition, our typological feature integration with dropout and smooth sampling generalizes and improves the model for zero-shot languages. because we integrated typological features into the span classifier using a simple concatenation approach, more advanced techniques-for instance, a parameter generator @xcite -with our dropout and smooth sampling could be studied in the future. additionally, our parser could make the applications that leverage structures possible for a wide range of languages without additional treebanks. for example, pseudo constituency structures that our parser generate could be used to apply the recurrent neural network grammar @xcite or the syntax-based neural ma-chine translation @xcite for many non-english languages. however, since the pseudo structures could be noisy or irrelevant to the model, selective or soft integration techniques should be considered @xcite .