PPT : Parsimonious Parser Transfer for Unsupervised Cross-Lingual Adaptation (2021)

Abstract: Cross-lingual transfer is a leading technique for parsing low-resource languages in the absence of explicit supervision. Simple ‘direct transfer’ of a learned model based on a multilingual input encoding has provided a strong benchmark. This paper presents a method for unsupervised cross-lingual transfer that improves over direct transfer systems by using their output as implicit supervision as part of self-training on unlabelled text in the target language. The method assumes minimal resources and provides maximal flexibility by (a) accepting any pre-trained arc-factored dependency parser; (b) assuming no access to source language data; (c) supporting both projective and non-projective parsing; and (d) supporting multi-source transfer. With English as the source language, we show significant improvements over state-of-the-art transfer models on both distant and nearby languages, despite our conceptually simpler approach. We provide analyses of the choice of source languages for multi-source transfer, and the advantage of non-projective parsing. Our code is available online.

## introduction recent progress in natural language processing (nlp) has been largely driven by increasing amounts and size of labelled datasets. the majority of the world's languages, however, are lowresource, with little to no labelled data available @xcite . predicting linguistic labels, such as syntactic dependencies, underlies many downstream nlp applications, and the most effective systems rely on labelled data. their lack hinders the access to nlp technology in many languages. one solution is cross-lingual model transfer, which adapts models trained on highresource languages to low-resource ones. this paper presents a flexible framework for cross-lingual transfer of syntactic dependency parsers which can leverage any pre-trained arc-factored dependency parser, and assumes no access to labelled target language data. one straightforward method of cross-lingual parsing is direct transfer. it works by training a parser on the source language labelled data and subsequently using it to parse the target language directly. direct transfer is attractive as it does not require labelled target language data, rendering the approach fully unsupervised. 2 recent work has shown that it is possible to outperform direct transfer if unlabelled data, either in the target lan-guage or a different auxiliary language, is available @xcite @xcite . here, we focus on the former setting and present flexible methods that can adapt a pre-trained parser given unlabelled target data. despite their success in outperforming direct transfer by leveraging unlabelled data, current approaches have several drawbacks. first, they are limited to generative and projective parsers. however, discriminative parsers have proven more effective, and non-projectivity is a prevalent phenomenon across the world's languages @xcite . second, prior methods are restricted to single-source transfer, however, transfer from multiple source languages has been shown to lead to superior results @xcite @xcite . third, they assume access to the source language data, which may not be possible because of privacy or legal reasons. in such source-free transfer, only a pre-trained source parser may be provided. we address the three shortcomings with an alternative method for unsupervised target language adaptation (section 2). our method uses high probability edge predictions of the source parser as a supervision signal in a self-training algorithm, thus enabling unsupervised training on the target language data. the method is feasible for discriminative and non-projective parsing, as well as multi-source and source-free transfer. building on a framework introduced in @xcite , this paper for the first time demonstrates their effectiveness in the context of state-of-the-art neural dependency parsers, and their generalizability across parsing frameworks. using english as the source language, we evaluate on eight distant and ten nearby languages @xcite . the singlesource transfer variant (section 2.1) outperforms previous methods by up to 11 % uas, averaged over nearby languages. extending the approach to multi-source transfer (section 2.2) gives further gains of 2 % uas and closes the performance gap against the state of the art on distant languages. in short, our contributions are: ## supervision via transfer in our scenario of unsupervised cross-lingual parsing, we assume the availability of a pre-trained source parser, and unlabelled text in the target language. thus, we aim to leverage this data such that our cross-lingual transfer parsing method out-performs direct transfer. one straightforward method is self-training where we use the predictions from the source parser as supervision to train the target parser. this method may yield decent performance as direct transfer is fairly good to begin with. however, we may be able to do better if we also consider a set of parse trees that have high probability under the source parser (cf. fig. 1 for illustration). if we assume that the source parser can produce a set of possible trees instead, then it is natural to use all of these trees as supervision signal for training. inspired by @xcite , we formalise the method as follows. given an unlabelled dataset {x i } n @xmath0, the training loss can be expressed as we consider an arc-factored neural dependency parser where the score of a tree is defined as the sum of the scores of its arcs, and the arc scoring function is parameterised by a neural network. the probability of a tree is then proportional to its score. ## related work cross-lingual dependency parsing has been extensively studied in nlp. the approaches can be grouped into two main categories. on the one hand, there are approaches that operate on the data level. examples of this category include annotation projection, which aims to project dependency trees from a source language to a target language @xcite @xcite ; and source treebank reordering, which manipulates the source language treebank to obtain another treebank whose statistics approximately match those of the target language @xcite . both methods have no restriction on the type of parsers as they are only concerned with the data. transferring from multiple source languages with annotation projection is also feasible @xcite . despite their effectiveness, these data-level methods may require access to the source language data, hence are unusable when it is inaccessible due to privacy or legal reasons. in such source-free transfer, only a model pre-trained on the source language data is available. by leveraging parallel data, annotation projection is indeed feasible without ac-cess to the source language data. that said, parallel data is limited for low-resource languages or may have a poor domain match. additionally, these methods involve training the parser from scratch for every new target language, which may be prohibitive. on the other hand, there are methods that operate on the model level. a typical approach is direct transfer (aka., zero-shot transfer) which trains a parser on source language data, and then directly uses it to parse a target language. this approach is enabled by the shared input representation between the source and target language such as pos tags @xcite or cross-lingual embeddings @xcite . direct transfer supports source-free transfer and only requires training a parser once on the source language data. in other words, direct transfer is unsupervised as far as target language resources. previous work has shown that unsupervised target language adaptation outperforms direct transfer. recent work by @xcite used a neural lexicalised dependency model with valence (dmv) @xcite as the source parser and fine-tuned it in an unsupervised manner on the unlabelled target language data. this adaptation method allows for source-free transfer and performs especially well on distant target languages. a different approach is proposed by @xcite , who gathered target language corpus statistics to derive constraints to guide inference using the source parser. thus, this technique also allows for source-free transfer. a different method is proposed by @xcite who explored the use of unlabelled data from an auxiliary language, which can be different from the target language. they employed adversarial training to learn language-agnostic representations. unlike the others, this method can be extended to support multisource transfer. an older method is introduced by @xcite , who leveraged ambiguityaware training to achieve unsupervised target language adaptation. their method is usable for both source-free and multi-source transfer. however, to the best of our knowledge, its use for neural dependency parsing has not been investigated. our work extends theirs by employing it for the said purpose. the methods of both @xcite and @xcite have several limitations. the method of @xcite requires the parser to be generative and projective. their generative parser is quite impoverished with an accuracy that is 21 points lower than a state-of-the-art discriminative arc-factored parser on english. thus, their choice of generative parser may constrain its potential performance. furthermore, their method performs substantially worse than direct transfer on nearby target languages. because of the availability of resources such as universal dependency treebanks @xcite , it is likely that a target language has a closely related high-resource language which can serve as the source language. therefore, performing well on nearby languages is more desirable pragmatically. on top of that, it is unclear how to employ this method for multisource transfer. the adversarial training method of @xcite does not suffer from the aforementioned limitations but is unusable for sourcefree transfer. that is, it assumes access to the source language data, which may not always be feasible due to privacy or legal reasons. ## conclusions this paper presents a set of effective, flexible, and conceptually simple methods for unsupervised cross-lingual dependency parsing, which can leverage the power of state-of-the-art pre-trained neural network parsers. our methods improve over direct transfer and strong recent unsupervised transfer models, by using source parser uncertainty for implicit supervision, leveraging only unlabelled data in the target language. our experiments show that the methods are effective for both single-source and multi-source transfer, free from the limitations of recent transfer models, and perform well for non-projective parsing. our analysis shows that the effectiveness of the multi-source transfer method is attributable to its ability to leverage diverse syntactic signals from source parsers from different languages. our findings motivate future research into advanced methods for generating informative sets of candidate trees given one or more source parsers.