A survey of part-of-speech tagging approaches applied to K’iche’ (2021)

Abstract: We study the performance of several popular neural part-of-speech taggers from the Universal Dependencies ecosystem on Mayan languages using a small corpus of 1435 annotated K’iche’ sentences consisting of approximately 10,000 tokens, with encouraging results: F_1 scores 93%+ on lemmatisation, part-of-speech and morphological feature assignment. The high performance motivates a cross-language part-of-speech tagging study, where K’iche’-trained models are evaluated on two other Mayan languages, Kaqchikel and Uspanteko: performance on Kaqchikel is good, 63-85%, and on Uspanteko modest, 60-71%. Supporting experiments lead us to conclude the relative diversity of morphological features as a plausible explanation for the limiting factors in cross-language tagging performance, providing some direction for future sentence annotation and collection work to support these and other Mayan languages.

## introduction this paper presents a survey of approaches to partof-speech tagging for k'iche', a mayan language spoken principally in guatemala. the mayan languages are a group of related languages spoken throughout mesoamerica. k'iche' belongs to the eastern branch, which contains 14 other languages, including kaqchikel in the quichean subgroup and uspanteko which belongs to its own subgroup. part-of-speech tagging has wide usage in corpus and computational linguistics and natural language processing, and is often considered part of a toolkit for basic natural language processing. in the definition of part-of-speech tagging we subsume the tasks of determining the part of speech, morphological analysis and lemmatisation. that is, given a sentence such as in (1) part-of-speech tagging would return both the sequence of part-ofspeech tags @xcite but also the lemmata @xcite and the set of feature value pairs for each of the forms. 1 (1) kinqʼojomaj k-∅-in-qʼojomaj imp-b3sg-a1sg-play le le the qʼojom. qʼojom. marimba. a brief reading guide: prior work, on mayan and other languages of the americas and on crosslanguage part-of-speech tagging, is reviewed in section 2. our experimental design including the mathematical model used for analysing performance are given section 3. universal dependencies annotation for k'iche' and the systems tested are described in section 4, and results are presented and analysed in section 5. @xcite explore morphological segmentation and analysis for the purpose of generating interlinearly glossed texts. they work with uspanteko, a language of the greater quichean branch, and the closest language to k'iche' we were able to identify with published studies of computational morphology. they explore several different systems: inducing morphology from parallel texts, an unsupervised segmentation+clustering strategy, and an interactive training strategy with a linguist. ## prior work in @xcite , a set of preliminary annotation conventions for mayan languages in general, and k'iche' in particular, are proposed. a maximum-entropy part-of-speech tagger is presented in @xcite for q'anjob'al, which, like k'iche', is a mayan language of guatemala. they work with a custom selection of 60 tags, and trained on an annotated corpus of 4100 words (no lemmatisation is performed). in contrast to the systems we will study, @xcite perform feature engineering and end up with f 1 scores between 63% and 78%, depending on the features chosen. there is much work on part-of-speech tagging for languages of the americas outside of the mayan family: statistical lemmatisation and part-of-speech tagging systems are described by @xcite and a finite-state morphological analyser by @xcite for shipibo-konibo, a panoan language of the amazonian region of peru. in @xcite and @xcite , respectively, finite-state morphology and support vector machine-based tagging+parsing systems are described for quechua. the latter uses a corpus that comprises 2k sentences. cross-language part-of-speech tagging through parallel corpora, sometimes called annotation projection, is well-studied; in mayan languages, @xcite use a parallel corpus as a bridge to a higher-resourced language for which a part-ofspeech tagger already exists. in the absence of such a corpus, so-called "zeroshot" methods are created from other (presumably higher-resourced) languages and applied to the target language. the main balance to strike is between specificity of resources (how closely-related are the other languages) and quantity of resources (how much linguistic data is accessible). udify of @xcite is an example of preferring the latter: a deep neural architecture is trained on all of the universal dependencies treebanks. the former strategy can be seen in @xcite , where in addition to annotation projection, authors attempt zero-shot tagging of ukrainian with a model trained on russian. ## methodology we used a corpus of k'iche' foot_1 annotated with partof-speech tags and morphological features @xcite . the corpus consisted of 1,435 sentences comprising approximately 10,000 tokens from a variety of text types and was annotated according to the guidelines of the universal dependencies (ud) project @xcite . an example of a sentence from the corpus can be seen in we studied the performance of several popular part-of-speech taggers within the universal dependencies ecosystem; these are reviewed in section 4. performance was computed as f 1 scores for lemmatisation, universal part-of-speech (upos), and universal morphological features (ufeats). we performed 10-fold cross validation to obtain mean and standard deviation of f 1 . we also recorded training time and model size to compare the resource consumption of the models in the training process. we selected the best-performing system and performed a convergence study (see section 5.3 for results). we decimated the training data of one of the test-train splits from the cross-validation, and plotted the performance of models trained on the decimations. we make the following assumption about the performance: additional training data provides exponentially decreasing performance improvement. under this assumption, we obtain the formula: here f 1 (n) is the performance of a model trained on n tokens, f 1 (∞) is the asymptotic performance, and ∆f 1 is the gap between f 1 (∞) (estimated maximum performance) and f 1 (0) (zero-shot performance). the parameter k is the characteristic number of tokens; each additional k tokens of training data causes the gap ∆f 1 = f 1 (∞) -f 1 (n) to shrink by a factor of 1/e ≈ 36%. this can be used to estimate the training data n required to meet a given performance target f target 1 we fit this curve against our convergence data and estimate peak performance and characteristic number. error propagation is used with the error in parameter estimation to compute the error bands in the graph: here x runs over the parameters of f 1 (n): f 1 (∞), ∆f 1 and k. we also studied the best-performer in crosslanguage tagging on the related kaqchikel and uspanteko languages. the 10 models trained in cross-validation were all evaluated on small part-ofspeech-tagged corpora of 157 (kaqchikel) and 160 (uspanteko) sentences. for results and overviews of the languages, see section 6. ## systems we tested morphological analysis on three systems designed for universal dependencies treebanks: udpipe @xcite , udpipe 2 @xcite , and udify @xcite . of these, only udpipe had a working tokeniser. for other taggers we trained, we trained the ud-pipe tokeniser and other tagger together. we thus present combined tokeniser-tagger systems. udpipe @xcite is a languageindependent trainable tokeniser, lemmatiser, pos tagger, and dependency parser designed to train on and produce universal dependencies-format treebanks. it uses gated linear units for tokenisation, averaged perceptrons for part-of-speech tagging, and a neural network classifier for dependency parsing. it is the least resource-hungry model in our study by an order of magnitude or more, and we trained it from-scratch using the k'iche' corpus in section 3. udpipe 2 @xcite is a python prototype for a tensorflow-based deep neural network postagger, lemmatiser, and dependency parser. it won high rankings in the conll 2018 shared task on multilingual parsing @xcite , taking first place by one metric. deep neural methods have achieved impressive performance results in recent years, but take considerable computational resources to train. we used udpipe 2 without pretrained embeddings, and trained it from-scratch using the k'iche' corpus in section 3. udify @xcite is a allennlp-based multilingual model using bert pretrained embeddings and trained on the combined universal dependencies treebank collection; we fine-tuned this pretrained model on our k'iche' data. this was our most resource-intensive model, even though we only fine-tuned on k'iche'; our initialisation was the udify-distributed bert+ud model. ## cross-language tagging there are around 32 mayan languages spoken in mesoamerica, in the countries of guatemala, mexico, honduras, el salvador and belize. given the impressive performance of the best-performing system on k'iche' data, we decided to test it on two related languages spoken in guatemala: kaqchikel and uspanteko. udify is also reported as being suited to zero-shot inference, so we include two udify-based models: fine-tuned on k'iche' (referred to as "udify-ft") and the original udify model (simply "udify"). ## concluding remarks we used an annotated corpus of 1435 part-ofspeech tagged k'iche' sentences to to survey a number of neural part-of-speech tagging systems from that ecosystem. we found the best performance was generally with udpipe 2, a deep neural system inte-grating lemmatisation, part-of-speech and morphological feature assignment. our udpipe 2-trained system achieved f 1 of 93% or better on all tasks, very encouraging results for a relatively small corpus. convergence studies showed that on corpora of similar morphological composition even better performance is attainable, but to close the gap to within 1% of projected optimal performance requires roughly half again the amount of training data. the high performance on k'iche' led us to experiment using our model to perform cross-language tagging on the related languages of kaqchikel and uspanteko. performance on the more closelyrelated language, kaqchikel, was still respectable, with f 1 ranging from 63 to 85% on the tasks; on uspanteko performance we observed more modest performance 60-71%. the k'iche' fine-tuned udify model does show noticably better performance, but possibly not worth the energy expenditure. our results after disabling our cross-language tagger by withholding some labels during training imply that cross-language performance could be improved by annotating more data with similar features to the kaqchikel and uspanteko evaluation corpora, and suggest that cross-language tagging is a path forward to greater availability of part-ofspeech annotation for mayan languages.