A little perturbation makes a difference: Treebank augmentation by perturbation improves transfer parsing (2019)

Abstract: We present an approach for cross-lingual transfer of dependency parser so that the parser trained on a single source language can more effectively cater to diverse target languages. In this work, we show that the cross-lingual performance of the parsers can be enhanced by over-generating the source language treebank. For this, the source language treebank is augmented with its perturbed version in which controlled perturbation is introduced in the parse trees by stochastically reordering the positions of the dependents with respect to their heads while keeping the structure of the parse trees unchanged. This enables the parser to capture diverse syntactic patterns in addition to those that are found in the source language. The resulting parser is found to more effectively parse target languages with different syntactic structures. With English as the source language, our system shows an average improvement of 6.7% and 7.7% in terms of UAS and LAS over 29 target languages compared to the baseline single source parser trained using unperturbed source language treebank. This also results in significant improvement over the transfer parser proposed by (CITATION) that involves an “order-free” parser algorithm.

## introduction cross-lingual dependency parsing involves training a dependency parser using a treebank in one language (source language) and applying it to parse sentences in another language (target language). this can be used to develop parsers for languages for which no treebank is available. the syntactic similarity between the source and the target languages typically plays an important role in the success of a cross-lingual transfer parser @xcite @xcite . a major challenge in transfer parsing is to bridge the difference in the syntax of the source and the target languages. for example, the object usually occurs after the corresponding verb in english while the verb normally occurs at the final position in a clause in japanese. in order to achieve better performance of the transfer parsers, researchers have worked on the selection of syntactically similar source languages for a given target language @xcite @xcite . attempts have also been made towards improving the performance of the transferred parsers for a given source-target language pair by reducing the syntactic gaps between them. this is done by transforming the source language parse trees @xcite @xcite @xcite using knowledge of the typological properties of the target language. however, these approaches are target language specific and may not give satisfactory results for multiple languages. recent work by @xcite proposed an "order-free" parser model that comprises of a transformer-based encoder and a graph-based decoder. they show that the self-attention mechanism of the transformer with direction independent position encoding used in their model gives rise to improved performance for transfer between distant pair of languages compared to a standard parser model that uses an rnn based encoder and stack pointer-based decoder. in this paper, we propose a different approach for enhancing the performance of a target language independent transfer parser based on a single source language by augmenting the treebank of the source language without using any target language information. for this, we add sentences obtained by rearranging the original sentences in the treebank while keeping the parse tree of the sentence fixed. this can be construed as generating a more general treebank which may contain sentences not conforming syntactically to the source language. specifically, we introduce controlled perturbation in the relative ordering of the head-dependent pairs in the source language parse trees. we stochastically alter the order of some of the headdependent pairs in the source language sentences while keeping the head-dependent relations in the parse trees intact. this perturbation reduces the dependency of the parser on the word order in the training sentences and makes it more robust towards the variation in syntax. we show that a stack-pointer network-based parser model @xcite trained using this treebank results in improvement of the performance of the transfer parser over a baseline parsers trained on an unperturbed treebank. this parser also significantly outperforms the "orderfree" parser model proposed by @xcite model by 3.8% uas and 4.2% las. we also show that our target language independent approach gives a competitive performance with that of a target language specific transformation approach @xcite . ## related work initial work on model transfer involved training delexicalized models @xcite using only language independent non-lexical features such as pos tags in the source language treebanks. several approaches for model transfer that incorporate lexical features in the transfer models have been reported in the literature. these include use of cross-lingual word clustering @xcite , dictionary-based mapping of distributed word embeddings and projection-based bi-lingual word representations @xcite @xcite . søgaard (2011) proposed an approach for selecting training instances from source language by ranking them in terms of similarity with the target language sentences in terms of pos tag perplexity. @xcite ; @xcite ; @xcite presented a multilingual algorithm for dependency parsing that selectively learns the aspects (some features listed in world atlas of language structures (wals) @xcite ) of the source languages relevant to the target language and ties the model parameters accordingly. another approach for improving the performance of cross-lingual transfer parsers is by transforming the source language parse trees to match the syntax of the target language. @xcite improves performance of the transfer parsers by transforming the source language parse trees based on the knowledge of the target language syntax derived from wals. @xcite also proposed a similar source language treebank transformation method in which knowledge of the syntax of a target language is derived from small number annotated target language parse trees. @xcite generated synthetic treebanks by altering the word order of the source language treebanks using knowledge of the distribution of the noun and verb dependents of other real-world languages from their respective treebanks. @xcite proposed an approach for learning an optimized permutation parameter using the given source language treebank and a gold pos tag annotated corpus in the target language. this parameter set is then applied to permute the source language parse trees to approximately match the syntax of the target language. these methods are however target language specific and may not perform well for other languages. @xcite have shown that training a parser model using scrambled parse trees of sentences of one domain improves performance of the parser over a parser model trained using the original treebank on test sentences of another domain. they scrambled the parse trees of sentences from newswire data and tested on conversational data. the scrambled treebank consisted either of all possible permutations of a subset of the parse trees in the original treebank, or, a fixed number of permutations of all the parse trees, where the permuted parse trees with the lowest perplexity assigned by a language model are selected. @xcite proposed a parser algorithm that improves the quality of transfer parser independent of the target language. they have compared the performance of combinations of different encoder-decoder architectures. they consider a bidirectional lstm based encoder (ordersensitive) and a transformer-based encoder (orderfree), and, two types of decoders, stack-pointer based (order-sensitive) and a biaffine graph-based (order-free) and have shown that overall best cross-lingual performance of a parser across several target languages can be achieved using the combination of transformer-based encoder and graph-based decoder model. this system is expected to be agnostic to the word order of the source sentence and thus work effectively for a variety of target languages. multi-source transfer @xcite parsing approaches combine treebanks of multiple source languages to train cross-lingual transfer parsing models. ## data and parser model data we carried out our experiments using treebanks of 29 languages from the ud v2.2 treebanks. we used the language-independent ud upos tags and dependency relations. we have used the acronyms of the language names in the rest of the paper. the full names of the languages are listed in appendix a.1. ## experiments and results we carried out the experiments corresponding to the different perturbation approaches under the following settings. swappert given a sentence of length n , for n/k perturbations, we have carried out separate experiments with @xmath0. the stackpointer-based parser model @xcite (rs) was trained for this perturbation. ptspertrs this refers to the stack-pointerbased parser model @xcite (rs) parser model trained using a source language treebank augmented with its versions perturbed by ptspert. we experimented with different perturbation probability values (@xmath1.1, 0.2, 0.3, 0.4, 0.5}). stattrans this refers to the stack-pointerbased parser model @xcite (rs) parser model trained using source language treebank transformed using statistical knowledge of target language syntax derived from samples of 20 target language parse trees @xcite . for each target language, we randomly sampled 20 parse trees from combined training and development sets. we trained separate models specific to each target language. ## conclusion in this paper propose an approach for introducing perturbation in the source language treebank to improve single source target language independent cross-lingual transfer parsing. we show that this approach indeed helps to improve the performance of the transferred parsers over models trained using only source language treebanks.