Unsupervised Stem-based Cross-lingual Part-of-Speech Tagging for Morphologically Rich Low-Resource Languages
Abstract
Unsupervised cross-lingual projection for partof-speech (POS) tagging relies on the use of parallel data to project POS tags from a source language for which a POS tagger is available onto a target language across word-level alignments.
The projected tags then form the basis for learning a POS model for the target language.
However, languages with rich morphology often yield sparse word alignments because words corresponding to the same citation form do not align well.
We hypothesize that for morphologically complex languages, it is more efﬁcient to use the stem rather than the word as the core unit of abstraction.
Our contributions are: 1) we propose an unsupervised stem-based cross-lingual approach for POS tagging for low-resource languages of rich morphology; 2) we further investigate morpheme-level alignment and projection; and 3) we examine whether the use of linguistic priors for morphological segmentation improves POS tagging.
We conduct experiments using six source languages and eight morphologically complex target languages of diverse typologies.
Our results show that the stem-based approach improves the POS models for all the target languages, with an average relative error reduction of 10.3% in accuracy per target language, and outperforms the wordbased approach that operates on three-times more data for about two thirds of the language pairs we consider.
Moreover, we show that morpheme-level alignment and projection and the use of linguistic priors for morphological segmentation further improve POS tagging.
Introduction
Low-resource languages lack annotated data even for basic syntactic information such as parts of speech (POS).
To address this problem, two main unsupervised approaches have been adopted: zeroshot model transfer (Pires et al., 2019) and crosslingual POS tagging via alignment and projection (Yarowsky et al., 2001;
Fossum and Abney, 2005;
Das and Petrov, 2011; Duong et al., 2013; Täckström et al., 2013; Agi´c et al., 2015, 2016; Buys and Botha, 2016; Eskander et al., 2020b).
Eskander et al.
(2020b) show that the alignment and projection approach is less sensitive to the morphological dissimilarities between the source and target languages than zero-shot model transfer.
In annotation projection, the word structure in the source and target languages impacts the quality of the alignment and projection phases, and hence affects the overall performance of the ultimate POS model.
This becomes a concern for languages with rich word structure where afﬁxation is common as they usually suffer from sparse alignment models that often fail to align words corresponding to the same citation form due to the extensive paradigms and translation inconsistencies.
Sparse alignment hinders the ability of a system to project the tags properly and results in null tags on the target side.
These null tags then reduce the number of qualifying training examples and impact the POS model by introducing non-continuous labeled sequences.
Adding to these practical issues, the concept of word as a unit of structure has long been questioned in language sciences (Marantz, 2001).
We hypothesize that using the stem as the core unit of abstraction results in better POS models for low-resource languages of rich morphology.
Our contribution is three-fold.
Unsupervised stem-based cross-lingual approach for POS tagging for morphologically complex low-resource languages , where we use the stem as the core unit of abstraction.
In order to adapt a fully-unsupervised approach, we use a stateof-the-art unsupervised morphological segmenter,
MorphAGram (Eskander et al., 2016, 2020a), to derive the stems and morphemes.
We follow the setup of Eskander et al.
(2020b)
using the Bible as the only source of parallel data in order to emulate a low-resource scenario.
We experiment with the same six source languages, namely English,
Spanish, French, German, Russian and Arabic, but choose six morphologically complex target languages, namely Amharic, Basque, Finnish, Indonesian, Telugu and Turkish and add two new target languages, namely Georgian and Kazakh, where we contribute a small POS-annotated dataset for the former.
We show that the stem-based approach outperforms the word-based one in 43 language pairs out of 48, with an average relative error reduction of 10.3% in accuracy per target language, up to 21.0% in the case of Kazakh.
We also show that the stem-based approach outperforms the word-based approach which operates on three-times more data for about two thirds of the experimental pairs.
Morpheme-level alignment and projection , which allows for abstracting away from how the morphemes are combined in the source and target languages.
We test the setup with Arabic as the source language and show improvements for seven out of the eight target languages.
Using linguistic priors in morphological segmentation , which results in better segmentation models towards better alignment and projection.
Using
Georgian as a case study, we show that the use of linguistic priors, in the form of a set of afﬁxes provided by an expert in the target language, improves the ultimate POS models.
Finally, we make our code publicly available to encourage further research 1 .
Approach
We perform fully unsupervised cross-lingual POS tagging via alignment and projection.
We follow the main architecture presented by Eskander et al.
(2020b) (Section 2.1).
A primary difference is that we harness unsupervised morphological segmentation to use the stems as the core unit of abstraction for both alignment and projection (Section 2.2).
In addition, we experiment with morpheme-level alignment and projection (Section2.3) and examine the use of linguistic priors towards better morphological segmentation and POS tagging (Section 2.4).
This allows for less sparse alignment models and denser projections, which in turn produces larger POS training data of a better quality
[Table]
[Table]
[Figure]
2. Word-based Alignment and Projection
Figure 1a shows the pipeline presented by Eskander et al.
(2020b).
The only input to the process is a parallel text between the target language and a source one for which a POS tagger is accessible.
First, the parallel text gets white-space tokenized and is used to train two word-alignment models (sourceto-target and target-to-source) using GIZA++ (Och and Ney, 2003). The alignment models are then applied to align the source and target sides on the word level.
Next, the source side is tagged for POS using an off-the-shelf tagger (e.g., Stanza (Qi et al., 2020)).
The source tags are then projected onto the target across the word-based alignments, where only those bidirectional alignments whose conﬁdence is above a particular threshold are considered in order to eliminate one-to-many and many-to-one alignments and those alignments of low conﬁdence.
[Table]
[Figure]
The projected tags for each token represent what are called token constraints, while the tag distribution of each word type across the whole target side forms type constraints.
The token and type constraints are then coupled by nullifying those tag assignments whose type-level probabilities are below some threshold.
The target text, along with its projected tags, then constitutes the training data for a neural POS tagger, where only the top scoring sentences, in terms of tag-assignment density and alignment conﬁdence, are considered.
The neural tagger is a bidirectional long shortterm memory (BiLSTM) model (Hochreiter and
Schmidhuber, 1997) that uses a custom softmax activation to handle the null tags.
It uses word embeddings, both randomly initialized and the contextual multilingual embeddings XLM-R (Conneau et al., 2019); preﬁx and sufﬁx n-gram character embeddings, where n is in { 1 , 2 , 3 , 4 } ; and hierarchical
Brown-cluster (Brown et al., 1992) embeddings.
The architecture can beneﬁt from parallel data of multiple source languages, where either the projections from multiple source languages ( Mul _ proj ) or the decoded outputs that are based on multiple single-source models ( Mul _ out ) can be combined through maximum-voting mechanisms.
2. Stem-Based Alignment and Projection
While the architecture by Eskander et al. (2020b) yields the state-of-the-art results for unsupervised
POS tagging when evaluated on 12 languages of diverse typologies, the complexity of word structure in the source and target languages has a direct impact on the quality of both alignment and projection.
Rich word structure where afﬁxation is common increases the ratio of word types to word tokens, which in turn results in sparse alignment models and incomplete projections that form null tags on the target side.
Null tags result in a score that is too low for the underlying sentence to qualify as a training example and introduce missing information for the training of the POS model, which negatively impacts the overall quality of POS tagging.
An example is shown in Figure 2a, where Arabic and Amharic are the source and target languages, respectively.
The example corresponds to verse
MAT 15:35 , “He commanded the multitude to sit down on the ground” , where the word-alignment models are trained on the New Testament.
As shown, the two Arabic-Amharic pairs { M È¥m , (cid:138)wm(cid:20)(cid:152)(cid:0) } ( and the people , the people ) and { € ÈÈ , r(cid:155)(cid:2)(cid:143) } ( he commanded , then he commanded ) are not aligned, resulting in null tags.
The sparse wordalignment models are simply unable to properly align words that correspond to the same citation form because of the extensive paradigms, which, along with translation inconsistencies, leads to the loss of the one-to-one correspondence between word structures across parallel texts (examples are shown in Table 1).
Using the stem instead of the word as the core unit of abstraction is more productive; the stem is usually shared by all the members of a paradigm, which reduces misalignment.
Figure 2b shows that stemming the Arabic and
Amharic texts yields complete one-to-one alignments and projections, which in turn eliminates [Table] the word-based null assignments and assigns each word on the Amharic side a valid POS
Figure 1b illustrates our overall stem-based approach.
We ﬁrst stem the source and target sides and train two stem-level alignment models, one in each direction.
Next, we assign the stems of the source side the POS tags of their corresponding words, which are then projected onto the target stems through the stem-level alignments.
We then apply the token and type constraints on the labeled stems on the target side.
However, since we train the ultimate POS model on the word level, we replace each target stem by its corresponding word and assign that word the stem-based projected POS tag.
The rest of the pipeline for sentence selection and training the POS model are the same as in the word-based architecture described in Section 2.1.
We assume that the source language is a highresource one for which an off-the-shelf stemmer is accessible.
On the other hand, for the target languages, we use MorphAGram 2 (Eskander et al., 2020a) to train an unsupervised morphological segmentation model using the target side of the parallel text.
MorphAGram is a stateof-the-art framework for unsupervised morphological segmentation based on Adaptor Grammars (AGs) (Johnson et al., 2007), nonparametric Bayesian models that generalize Probabilistic
Context Free Grammars (PCFGs).
We run MorphAGram in a cascaded setup of two learning rounds.
In the ﬁrst round, we train a segmentation model using a language-independent highprecision grammar (PrStSu2a+SM 3 ) to obtain a list of morphemes.
We then seed these morphemes into the best performing language-independent grammar (PrStSu+SM) for the second round of learning as described by Eskander et al.
(2016, 2020a).
Both
PrStSu2a+SM and PrStSu+SM grammars model the word as a sequence of preﬁxes, a stem and sufﬁxes, where the afﬁxes are recursively deﬁned in order to model multiple consecutive items.
2. Morpheme-Based Alignment and Projection
Next, we perform morpheme-based alignment and projection in a similar fashion as in the stem-based approach (Section 2.2).
This approach abstracts away from whether the morphemes in the source and target languages are free-standing or not.
On the source side, each morpheme receives a separate POS tag using an off-the-shelf POS tagger.
These tags are then projected onto the target morphemes through bidirectional morpheme-level alignments.
We obtain the target morphemes using
MorphAGram, where the output of the PrStSu+SM grammar yields preﬁxes, a stem, and sufﬁxes for each word.
However, since we train the POS model on the word level, we replace each sequence of morphemes on the target side by its corresponding word and assign that word the POS tag of the representative morpheme.
We deﬁne the representative morpheme either as the morpheme whose POS tag ranks the highest among those of the other morphemes 4 ( RANK ) or as the stem morpheme ( STEM ).
Using Linguistic Priors for Segmentation
We hypothesize that better detection of stems yields more robust alignment and projection towards improved POS tagging.
Accordingly, instead of conducting morphological segmentation on the target side in a fully unsupervised manner, we follow Eskander et al.
(2021) by seeding afﬁx morphemes into the grammar tree prior to training the segmentation model textcolorbluein a minimally supervised fashion; these afﬁxes are generated manually by an expert in the target language.
With Georgian as a case study, we examine this setup in the stembased approach using the PrStSu+SM grammar.
2. Featurizing Segmented Data
In this setup, we utilize the unsupervised morphological-segmentation model that is trained on the target side of the parallel text to produce stem, complex-preﬁx and complex-sufﬁx features, and leverage these features as part of the neural
POS model.
For training, we use these features as randomly initialized embeddings that we concatenate with the existing word, afﬁx and words-cluster embeddings prior to applying the BiLSTM encoding layer.
Experiments and Evaluation 3. Languages and Data
We conduct our experiments on six source languages and eight target ones 5 , for a total oflanguage pairs.
We use the same source languages used by Eskander et al.
(2020b), namely
English, Spanish, French, German, Russian, and Arabic 6 , and experiment with eight typologically diverse target languages: six morphologically rich languages that are largely agglutinative, namely
Basque, Finnish, Georgian, Kazakh, Telugu, and Turkish; morphologically rich Amharic, where many morphological alterations rely on consonantal roots; and less morphologically rich Indonesian.
We conduct the experiments in a truly lowresource scenario, where we use the New Testament as the source of our parallel data (unless noted otherwise): limited in size and out-of-domain with respect to the evaluation sets.
We use the Multilingual Parallel Bible Corpus 7 (Christodouloupoulos and Steedman, 2015) as the source of data for all the languages, except for Georgian and Kazakh 8 .
3. Experimental Settings
For the tagging of the source languages, we use the same off-the-shelf taggers as in Eskander et al.
(2020b):
Stanza 9 (Qi et al., 2020) for English, Spanish, French, German and Russian and
MADAMIRA (Pasha et al., 2014) for Arabic.
We also use MADAMIRA for Arabic morphological segmentation.
For the stemming of the other source languages, we use the Snowball Stemmer (Porter, 2001) as part of NLTK 10 (Bird and Loper, 2004).
On the other hand, we use MorphAGram(Eskander et al., 2020a) to train and apply the morphological-segmentation models for the target languages as described in Section 2.2.
We follow Eskander et al.
(2020b)
by using the same thresholds for alignment and projection, along with the same neural hyperparameters of the
POS tagger.
We also evaluate our models in terms of POS accuracy on the same Universal Dependencies (UD) v2.5 (Zeman et al., 2019) test datasets.
However, since the UD project does not currently contain Georgian datasets, we developed a small POS dataset for Georgian (100 sentences) following the UD-tagging scheme.
The sentences are taken from the Modern Georgian and Political texts sub-corpora of the Georgian National Corpus 12 , and they are hand-tagged and carefully revised by a linguist who specializes in and speaks Georgian as a second language 13 .
Finally, all the results are averaged over three runs.
3. System Performance
Table 2 reports the POS accuracy for the baseline word-based approach and our stem-based approach for all the 48 target-source language pairs using the
New Testament as the source of parallel data.
In addition, we report the results for the two multilingual setups Mul _ out and Mul _ proj per target language.
The stem-based approach outperforms the word-based one in 43 language pairs and all the multilingual setups except Mul _ proj in the case of Indonesian, which stands out in our language sample as the least complex in terms of morphology.
The biggest improvement in the stem-based approach is achieved in the cases of Russian →
Turkish, Russian → Kazakh, Spanish → Kazakh and Arabic → Georgian, with relative error reductions of 33.8%, 30.2%, 28.6% and 27.2%, respectively.
When averaging across the sources (including the multilingual ones), Kazakh, Telugu and
Turkish experience the highest relative error reductions of 21.0%, 12.1% and 12.1%, respectively.
On the other hand, Russian and Arabic yield the highest relative error reductions of 16.3% and 15.6%, respectively, when averaging across the target languages, which is in line with the morphological complexity of the two languages. Eskander et al.
(2020b) show that related languages transfer best across each other. This results in efﬁcient word-based baselines for related language pairs, which in turn limits the corresponding gains in the stem-based approach. On the other [Table]
[Table] stem-based morpheme-based approaches when projecting from Arabic using the New Testament as the source of parallel data. The best result per target language is in bold . All the morpheme-based improvements are statistically signiﬁcant for p - value < 0 .
hand, a low word-based baseline makes room for improvement when operating on the stem level. For instance, both Georgian and Telugu witness the highest stem-based gains when transferring from
Arabic, their lowest performing source language in the word-based approach. For more information about the correlation between language relatedness and cross-lingual learning, see Eskander (2021).
Next, we evaluate the morpheme-based approach (Section 2.3). Table 3 reports the performance of the word-based, stem-based and morpheme-based approaches using Arabic as the source language since it is more morphologically complex than the other sources.
The morpheme-based approach results in dense training instances as both alignment and projection are performed in a more ﬁne-grained level compared to the word-based and stem-based approaches. It therefore improves POS tagging for all the target languages except Amharic, where
Telugu beneﬁts the most with relative error reductions of 23.9% and 15.3% over the stem-based approach using the RANK and STEM mechanisms, respectively.
The difference in the performance of the RANK and STEM mechanisms is only statistically signiﬁcant for p - value < 0 .
01 in the cases of Amharic and Basque, where the STEM mechanism yields better performance, and in the case of
Telugu, where the RANK mechanism is superior.
Finally, we hypothesize that the quality of morphological segmentation highly affects the efﬁciency of the morpheme-based setups, which explains the variation in the performance across the different target languages and mechanisms.
As mentioned earlier, we use Georgian as a case study to examine the impact of using linguistic priors for morphological segmentation on the quality of POS tagging (Section 2.4).
The results are listed [Table] in Table 4.
The use of linguistic priors improves the stem-based approach except when projecting from Arabic.
The lack of improvement in the case of Arabic can be explained by over-segmentation that produces incorrect POS tags for the common conjunction da ( and ).
The characters da also correspond to a verbal preﬁx that is manually seeded as a prior.
This seeding causes erroneous projections labeling da as a verb or an adverb when projecting from Arabic.
Finally, we experiment with the use of the stem and afﬁx information as training features in the
POS neural model (Section 2.5).
However, most of the improvements due to the use of these features are not statistically signiﬁcant (See Appendix A for full results) since such features are surpassed by the preﬁx and sufﬁx n-gram character embeddings.
3. Analysis of the Stem-Based Approach
Upon alignment and projection, the highest scoring target sentences are selected as training examples, where sentence score is deﬁned as the harmonic mean of the percentage of tokens with projected tags and the average alignment probability of those tokens.
The ﬁne-grained stem-level alignments allow for better alignment conﬁdence and more dense sentences, which in turn increases sentence scores and the number of training examples, and hence reduces the number of out-of-vocabulary words (OOVs).
Table 5 lists the average number of training examples, average relative increase in the number of training examples, average relative increase in sentence scores and average relative decrease in the number of OOVs for each target language in the stem-based approach with respect to the word-based one.
We witness improvements in the examined aspects for each target language, which explains the considerable improvements in the stem-based approach.
Next, we examine the average relative error reduction in the detection of open-class tags (nouns, verbs and adjectives) in the stem-based approach as compared to the word-based one per target language (Table 6) and per source language (Table 7).
Kazakh beneﬁts the most from the stem-based approach at the detection of nouns and adjectives, while Amharic receives the highest gains for verbs.
On the other hand, projecting from Russian in the stem space achieves the highest gains for nouns, while the stem-based projection from Arabic yields the highest gains for both verbs and adjectives.
Finally, Figure 3 illustrates the absolute improvements in POS tagging when applying the stembased approach using the New-Testament as the source of parallel data compared to the word-based approach using the entire Bible as the source of parallel data (three-times more data).
As illustrated, the stem-based approach achieves better performance in about two thirds of the language pairs with an average absolute gain of 1.7%.
This means using the stem as the core unit of abstraction compensates for the lack of adequate parallel data.
Related Work
The line of work most closely related to ours is unsupervised cross-lingual POS tagging via alignment and projection, which was ﬁrst introduced by Yarowsky et al.
They applied noisereduction techniques to improve the alignments and used the resulting transition and emission probabilities to deﬁne an HMM POS tagger.
Exploiting multiple source languages via maximum voting was then explored by Fossum and
Abney (2005), by voting among the outputs of different single-source models, and by Agi´c et al. (2015), by projecting the annotations from multiple languages before training the POS tagger.
In order to increase the size of the training data,
Das and Petrov (2011) proposed graph-based label propagation, while Duong et al. ; Agi´c et al. [Table]
[Table] [Table] (2015) applied self-training and revision.
On another hand, Täckström et al.
and Buys and
Botha (2016) investigated the use of token and type constraints to reject projections of low conﬁdence.
Eskander et al.
(2020b) derived a cross-lingual
POS-tagging pipeline that utilizes the best practices in alignment and projection.
In addition, they examined the use of pretrained multilingual contextual embeddings, along with afﬁx embeddings and Brown clusters, within a rich neural architecture, which achieves the state-of-the-art results for unsupervised POS tagging.
We follow their approach by presenting stem-based alignment and projection for morphologically complex low-resource languages.
Regarding unsupervised morphological segmentation, several generative and discriminative frameworks have been developed over the last two decades. The two most notable frameworks are: 1) Morfessor (Creutz and Lagus, 2007; Grönroos et al., 2014), a commonly-used HMM framework that utilizes the MDL principle to segment into morphemes of a hierarchical structure; and 2) MorphAGram (Eskander et al., 2020a), a segmentation framework that is based on Adaptor Grammars (AGs) (Johnson et al., 2007), Bayesian models that utilize Probabilistic Context Free Grammars (PCFGs).
We use MorphAGram to train morphological-segmentation models as it achieves the state-of-the-art performance and allows for deriving afﬁx and stem information (as opposed to a sequence of unlabeled morphemes).
Conclusion and Future Work
We presented a fully unsupervised stem-based approach for cross-lingual POS tagging via alignment and projection, where we use the stem as the core unit of abstraction to abstract away from complex afﬁxation.
Our experiments using six source languages and eight morphologically rich target languages in low-resource setups show improvements over the word-based approach in 43 language pairs out of 48, with an average relative error reduction of 10.3% in accuracy per target language.
In addition, we examined morpheme-based alignment and projection and the use of linguistic priors in morphological segmentation, which further improve
POS tagging.
In the future, we plan to study the role of morphological typology in cross-lingual learning. This allows for deriving disciplined guidelines for the selection of an appropriate source language that transfers well to the target language of interest. Acknowledgements This research is based upon work supported by the
Intelligence Advanced Research Projects Activity (IARPA), (contract #FA8650-17-C-9117) and the
National Science Foundation (awards # and #1941733). The views and conclusions herein are those of the authors and should not be interpreted as necessarily representing ofﬁcial policies, expressed or implied, of ODNI, IARPA, NSF or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.  Ethical Considerations
The Georgian annotations were done by a linguist with appropriate compensation after educating them about the research purpose and the annotation process.
We claim ownership of the Georgian dataset for open distribution as the text is taken from the Modern Georgian and Political texts subcorpora of the Georgian National Corpus, which is an open-sourced work that allows for modiﬁcations, derived work and redistribution.
The quality of the annotations was examined manually and empirically.
The source code and the data will be released open-source.
Finally, the limitations of the work lay within the reported performance.
There should be no potential risks given these stated limitations.
A Appendix: Segmentation Information as Training Features
Table 2 shows the POS tagging results (accuracy) when using the stem and afﬁx information as training features in the neural POS model, as described in Subsection 2.5.
[Table]
B Appendix: Hardware
We use a Google-Cloud virtual instance of2.00GHz cores and 240GB of RAM to run all of our experiments.
The training rate is nearly 2, sentences per hour.