Graph-Based Multilingual Label Propagation for Low-Resource Part-of-Speech Tagging (2022)

Abstract: Part-of-Speech (POS) tagging is an important component of the NLP pipeline, but many low-resource languages lack labeled data for training. An established method for training a POS tagger in such a scenario is to create a labeled training set by transferring from high-resource languages. In this paper, we propose a novel method for transferring labels from multiple high-resource source to low-resource target languages. We formalize POS tag projection as graph-based label propagation. Given translations of a sentence in multiple languages, we create a graph with words as nodes and alignment links as edges by aligning words for all language pairs. We then propagate node labels from source to target using a Graph Neural Network augmented with transformer layers. We show that our propagation creates training sets that allow us to train POS taggers for a diverse set of languages. When combined with enhanced contextualized embeddings, our method achieves a new state-of-the-art for unsupervised POS tagging of low-resource languages.

## introduction in many applications, part-of-speech (pos) tagging is an important part of the nlp pipeline. in recent years, high-accuracy pos taggers have been developed owing to advances in machine learning methods that combine pretraining on large unlabeled corpora and supervised fine-tuning on wellcurated annotated datasets. this methodology only applies to a handful of high-resource (hr) languages for which the necessary training data exists, leaving behind the majority of low-resource (lr) languages. when training resources are scarce, an established method for training pos taggers is to automatically generate the training data via crosslingual transfer @xcite @xcite ., * equal contribution. actions noun w n speak verb louder adv than prep words noun taten noun sagen verb mehr adv als prep worte noun eylemler ? sözlerden ? daha ? yüksek ? sesle ? konuşur ? ‫ﮐﻧد‬ ‫ﻣﯽ‬ ? ‫ﺻﺣﺑت‬ ? ‫ﻋﻣل‬ ? ‫از‬ ? ‫رﺳﺎﺗر‬ ? ‫ﮐﻼم‬ ? figure 1: the sentence "actions speak louder than words" in english and its translations to persian, german, and turkish, aligned at the word level. the pos tags for high-resource english and german are known. we use a gnn to exploit this graph structure and compute pos tags for low-resource persian and turkish. 2020). typically, pos annotations are projected through alignment links from the hr source to the lr target of a word aligned parallel corpus. in this paper, we propose glp (graph label propagation), a novel method for transferring labels simultaneously from multiple high-resource source languages to multiple low-resource target languages. we formalize pos tag projection as graph-based label propagation. given translations of a sentence in multiple languages, we create a graph with words as nodes and alignment links as edges by aligning words for all language pairs. we then propagate pos labels associated with source language nodes to target language nodes, using a label propagation model that is formalized as a graph neural network (gnn) @xcite . nodes are represented by a diverse set of features that describe both linguistic properties and graph structural information. in a second step, we additionally employ self-learning to obtain reliable training instances in the target languages. our approach is based on multiparallel corpora, meaning that the translation of each sentence is available in more than two languages. we exploit the parallel bible corpus (pbc) of @xcite , foot_0 a multiparallel corpus covering more than 1000 languages, many of which are extremely low-resource, by which we mean that only a tiny amount of unlabeled data is available or that no language technologies exist for them at all @xcite . we evaluate our method on a diverse set of lowresource languages from multiple language families, including four languages not covered by pretrained language models (plms). we train pos tagging models for these languages and evaluate them against references from the universal dependencies corpus @xcite . we compare the results of our method against multiple stateof-the-art (sota) cross-lingual unsupervised and semisupervised pos taggers employing different approaches like annotation projection and zero-shot transfer. our experiments highlight the benefits of our new transfer and self-learning methods; crucially, they show that reasonably accurate pos taggers can be bootstrapped without any annotated data for a diverse set of low-resource languages, establishing a new sota for high-resource-to-lowresource cross-lingual pos transfer. we also assess the quality of the projected annotations with respect to "silver" references and perform an ablation study. to summarize, our contributions are: foot_1 ## method we now introduce our graph label propagation (glp) method, which formalizes the problem of annotation projection as graph-based label propagation. we first describe the graph structure, then the features associated with each node, and finally the architecture of our model. ## experimental setup we evaluate pos tagging performance on the universal dependencies (ud) (zeman et al., 2019) test sets. as ud and pbc tokenizations differ, we further adopt the following rule: if a pbc token corresponds to a sequence of several ud tokens, we replace the sequence with the original word, tagged with the tag of the ud token in the sequence that is highest in the dependency tree (cf. @xcite ). pos tagger model. ## results we evaluate glp on 17 test languages from different families, resource availabilities, and scripts, on universal dependencies v2.10, the latest version (see details in appendix a.2). our results are in prior work has used older versions of ud . we now compare against each baseline, evaluating on the relevant version of ud in each case. ## conclusion and future work we presented glp, a novel method for transferring labels from high-resource source to low-resource target languages, based on a formalization of annotation projection as graph-based label propagation. we exploited the parallel bible corpus and showed that reasonably accurate pos taggers can be bootstrapped from projected labels. since we do not use pbc-specific or language-specific features, glp is in principle applicable to the more than 1000 languages of pbc and to any other multiparallel corpus. one direction for the future is to employ a similar model to transfer higher-level structures such as dependency trees. since our method works with graph structures, one might be able to project dependency trees effectively. we could also extend our projection method to other tagging tasks like named entity recognition -although this requires using other parallel corpora to mitigate the domain shift problem of such a task. another line for future work is to study the best combinations of source languages to transfer to any target language.