Unsupervised Cross-Lingual Part-of-Speech Tagging for Truly Low-Resource Scenarios (2020)

Abstract: We describe a fully unsupervised cross-lingual transfer approach for part-of-speech (POS) tagging under a truly low resource scenario. We assume access to parallel translations between the target language and one or more source languages for which POS taggers are available. We use the Bible as parallel data in our experiments: small size, out-of-domain and covering many diverse languages. Our approach innovates in three ways: 1) a robust approach of selecting training instances via cross-lingual annotation projection that exploits best practices of unsupervised type and token constraints, word-alignment confidence and density of projected POS, 2) a Bi-LSTM architecture that uses contextualized word embeddings, affix embeddings and hierarchical Brown clusters, and 3) an evaluation on 12 diverse languages in terms of language family and morphological typology. In spite of the use of limited and out-of-domain parallel data, our experiments demonstrate significant improvements in accuracy over previous work. In addition, we show that using multi-source information, either via projection or output combination, improves the performance for most target languages.

## introduction majority of world's languages do not have annotated datasets even for the most simple nlp tasks such as part-of-speech (pos) tagging. however, efforts in documenting low-resource languages often contain translations, usually of religious text, into other high-resource languages. one such parallel corpus is the bible @xcite : 484 languages have a complete bible translation, while 2551 have a part of the bible translated. our goal is to learn pos taggers for a diverse set of target languages in a truly low-resource scenario, where only a limited and possibly out-of-domain set of translations into one or more high-resource languages is available (e.g., the bible), together with supervised pos taggers for the high-resource source language(s). unsupervised cross-lingual pos tagging via annotation projection has a long research history @xcite @xcite @xcite @xcite . in contrast to our work, these approaches either use large and/or in-domain parallel data or rely on a large number of source languages for projection. however, since projection could suffer from bad translation, alignment mistakes or wrong assumptions, a key consideration for all these approaches is how to obtain high-quality training instances for the target language (i.e., sentences with accurate pos tags projected from the source-language(s)). coupling token and type constraints @xcite @xcite , word-alignment confidence @xcite , multi-source projection @xcite and coverage (percentage of tokens covered by multisource projection) @xcite have shown to lead to training instances of better quality. however, only one or two of these have been usually employed. our first contribution is a robust approach for selecting training instances via cross-lingual annotation projection that exploits and expands all these best practices: coupling type and token constraints obtained in an unsupervised way, wordalignment confidence together with the density of the projected pos, and (optionally) multi-source projection (sub-section 2.1). our second contribution is a bilstm @xcite neural architecture that uses pre-trained contextualized word embeddings, affix embeddings and hierarchical brown clusters @xcite . as contextualized embeddings, we show gains by exploiting the mul-tilingual xml-r model @xcite , while affix embeddings are particularly useful for morphologically-rich languages, and word clusters have been shown to be useful for non-neural pos tagging @xcite @xcite . moreover, in addition to the single-source setups, we propose an approach that utilizes multiple source languages by combining the outputs of single-source taggers via weighted voting at the token level (sub-section 2.2). our third contribution is an extensive experimental setup, with 12 diverse target languages in terms of language family and morphological typology and six high-resource source languages (section 3). while projecting from a single source language can be efficient, we show that using multiple sources, either via projection or output combination, further improves the tagging accuracy for most target languages. our experiments, using limited and outof-domain parallel data, demonstrate significant improvements over previous work (both unsupervised and semi-supervised), even when comparing our single-source setups to other multi-source ones. we also investigate how much gold data is needed to develop supervised taggers comparable to our best unsupervised models. in addition, we show that cross-lingual annotation projection generalizes across languages of different typologies better than the zero-shot model-transfer approach by @xcite . finally, our tagging scripts and models are made publicly available foot_0 . ## approach our goal is to induce a neural pos tagger for a target language of interest without any direct supervision. instead, we rely on parallel translations between the target and one or more source languages for which pos taggers are accessible. this section describes our approach: 1) cross-lingual annotation projection via word alignments to prepare the training instances of the target language, and 2) neural pos tagging for the target language. ## related work unsupervised pos tagging through annotation projection was first proposed by @xcite , where they transferred pos tags from english to french and chinese. the work was then extended by @xcite , where they combined the outputs of single-source taggers based on different source languages. the multilingual setups were then further explored by @xcite and @xcite . in efforts to increase the coverage of the projected data, @xcite proposed graphbased label propagation to expand the projected tags on the target side, while @xcite and @xcite applied self-training and revision, where they performed the projection and training in iterations. on another side, @xcite and @xcite organized the projection process through the use of token and type constraints, which we adapt in our approach. semi-supervised setups have been explored by either restricting the type constraints through the use of a pos dictionary @xcite or by adding additional signals in training, either by using a pos dictionary @xcite or by combining manual and projected annotations @xcite . in contrast, our system is fully unsupervised, where we show that the robust construction of the training data can surpass the use of external resources. while most prior work does tagging for several target languages, and so is our work, some research focuses on specific language pairs such as projecting from german to hittite @xcite and from russian to ukrainian @xcite . ## conclusion and future work we presented a fully unsupervised cross-lingual pos tagger that does annotation projection by utilizing translation from one or more source languages into the target one. we showed that despite the use of limited and out-of-domain parallel data, our models outperform the state-of-the-art systems. we also showed that the robust selection of training instances and the rich word representation in our neural architecture are more efficient than utilizing some labeled data or external linguistic resources. in the future, we plan to enhance the system for handling morphologically complex languages trough unsupervised morphological segmentation. one approach is to perform the alignment and projection on the stem and morpheme levels. in addition, stem and morpheme information can be utilized as additional signals in training.