To POS Tag or Not to POS Tag: The Impact of POS Tags on Morphological Learning in Low-Resource Settings (2021)

Abstract: Part-of-Speech (POS) tags are routinely included as features in many NLP tasks. However, the importance and usefulness of POS tags needs to be examined as NLP expands to low-resource languages because linguists who provide many annotated resources do not place priority on early identification and tagging of POS. This paper describes an empirical study about the effect that POS tags have on two computational morphological tasks with the Transformer architecture. Each task is tested twice on identical data except for the presence/absence of POS tags, using published data in ten high- to low-resource languages or unpublished linguistic field data in five low-resource languages. We find that the presence or absence of POS tags does not have a significant bearing on performance. In joint segmentation and glossing, the largest average difference is an .09 improvement in F1-scores by removing POS tags. In reinflection, the greatest average difference is 1.2% in accuracy for published data and 5% for unpublished and noisy field data.

## introduction parts of speech (pos), also known as word classes or lexical categories, communicate information about a word, its morphological structure and inflectional paradigm, and its potential grammatical role in a clause. pos tagging is a well-studied problem in nlp. it is one of the first tasks undertaken for a new data set and a pos tagger is often one of the first nlp resources built for lowresource languages @xcite @xcite @xcite @xcite . although this priority on early pos tagging may be simply due to the relative ease of building a pos tagger, it seems to reflect an assumption that pos tags simplify or improve other nlp tasks @xcite . as far as we are aware, this assumption has not been methodically tested. this paper examines the impact of pos tags on morphological learning, an important area for low-resource languages, many of which are more morphologically complex than english, mandarin, or other large-resource languages. morphological learning can help reduce the out-of-vocabulary problem in morphologically complex languages, especially in low-resource settings. morphological learning also holds high priority in documentary and descriptive linguistics as a necessary foundation for further descriptive work. we focus on two related tasks that involve morphological learning: joint morpheme segmentation/glossing and morphological reinflection. joint segmentation and glossing segments a word into its component morphemes and glosses the segments. reinflection generates unseen inflected word forms from morphological features based on a language's inflectional patterns. since lexical categories (pos) are identified partly by morphological structure, it seems reasonable to assume the reverse -that knowing a word's part of speech makes it easier for a model to analyze its morphological structure. for example, knowing that a word is a noun in english makes it extremely unlikely that a final substring (e)n could be a participial affix (e.g. "oven" -noun; cf. "driven" -verb). on the other hand, pos tags may be providing redundant information when, for example, an affix that marks a morphosyntactic feature is identical across all categories where that feature appears (e.g. the russian morpheme /-i/ 'pl' is identical for for plural nouns and plural verb agreement). however, these hypotheses must be tested before claiming either one. the impact of (not) having pos tags has perhaps not been examined closely in part because it seems safe to assume that pos tags or a pos tagger will be available. however, as nlp expands its reach to new languages, pos tags may not be readily available. in fact, the lexical categories present in the language may not even be described yet when data becomes available. in documentary and descriptive linguistics, the description and tagging of lexical categories takes a relatively low priority compared to its place in nlp (cf. @xcite 's workflow). yet interlinear glossed texts (igt) are often the largest available annotated resource for a low-resource language-and sometimes the only available resource. the impact of pos tags on computational morphology may hold implications for linguistic theory as well. the nature of lexical categories @xcite , the criteria for identifying them @xcite , and even their very reality as a universal property of language @xcite are not entirely settled among linguists. if the morphological structure of unseen words can be analyzed and generated without reference to lexical categories, then perhaps such categories should not be considered an inherent property of the lexicon @xcite . this paper describes experiments that were run on corpora differing only in the presence or absence of pos tags. the results, which are generalized in figures 1 and 2 , indicate that pos tags do not have significant impact on computational morphological learning. section 2 presents related work in lexical categories, pos-tagging, segmentation and glossing, and (re)inflection. sections 3 and 4 describe the corpora and the nlp architecture used. the segmentation and glossing task and results are presented in section 5. the reinflection task and results are presented in section 6. implications of both experiments are discussed in section 7. ## related work work on pos tagging has led to the development of several related resources in nlp and linguistics including numerous methods for automatic tagging (e.g. @xcite ; @xcite ) as well as tag sets. the most popular tag set for english was developed by the penn treebank project @xcite . a universal pos tag set was proposed by @xcite and has been widely adopted. it closely follows traditional linguistic conventions for common lexical categories as can be seen by comparing to the leipzig glossing rules @xcite which also has recommended tags for less common categories. many nlp models have been applied to segmentation and glossing of low-resource languages but they often tackle just one of the two tasks, e.g. segmentation only @xcite @xcite @xcite . automatic morpheme segmentation was introduced by @xcite and much earlier segmentation research implemented unsupervised learning @xcite @xcite . published linguistic descriptive data is used as training data usually after some preprocessing. glossing-only experiments make the assumption that data is already segmented into morphemes. for example, mcmillan-major (2020) trained a conditional random field (crf) model to produce a gloss line for several high-resource languages and three low-resource languages. the low-resource language data came from interlinearized data that was polished for publication. @xcite and some other experiments such as @xcite use information from lines of interlinearized texts such as translation and pos tags. computational approaches to morphological inflection or reinflection have been developed by durrett and denero (2013); nicolai et al. (2015); liu and mao (2016); cotterell et al. (2017); kann and schütze (2016) ; @xcite , etc. some of the work was developed as part of the sigmorphon shared tasks. 1 our work partly replicates the conll-sigmorphon reinflection shared tasks @xcite @xcite . sequence-to-sequence neural network models have been very successful at handling the morphological (re)inflection task, even in low-resource conditions with model improvement designed to tackle the situation @xcite @xcite @xcite @xcite . the transformer @xcite is the model architecture which produces the current state-of-the-art performance on this task @xcite @xcite . therefore, we use the transformer for all the experiments in this paper. this paper is an expansion of a section in @xcite . the experimental setup and sigmorphon languages are the same as that work, but it does not look at what happens when pos tags are available in the field data. we expanded the re-inflection task to field corpora. we also ran the sigmorphon experiments 5 times instead of one time. the addition of the segmentation and glossing was inspired by @xcite . ## data we use published data in ten languages and unpublished data in five low-resource languages. the published and unpublished data is used for the mor- ## models for simple comparisons, we chose a single neural model architecture for both tasks. the tasks were trained with the transformer @xcite , the current state-of-the-art neural model architecture for morphological tasks @xcite . we used the implementation of the transformer model in the fairseq toolkit @xcite foot_6 with characterlevel transduction @xcite for morphology learning in low-resource settings. following @xcite , we employ @xmath0, each with 4 self-attention heads. the embedding size for the encoder and decoder is 256, and the hidden layer size is 1024. we use a dropout rate of 0.3 for encoding and beam search with a width of 5 at decoding time. the adam algorithm (kingma and ba, 2014) (β 1 = 0.9, β 2 = 0.98) is used to optimize the cross entropy loss with label smoothing @xcite of 0.1. all models have been trained on an nvidia ## pos for segmentation and glossing the first study asks whether pos tags makes a significant impact on automated morpheme segmenting and glossing. the experiment tests and compares two models on data that is identical except for the presence/lack of pos tags. we chose morpheme segmentation and glossing because it is a high-priority and early step in documenting and describing new languages. segmenting words into morphemes and glossing (strictly translating) them is usually the first task undertaken after new data has been transcribed. therefore, it is important to study how to provide and improve automated assistance for field linguists. automatic systems could greatly benefit the analysis of endangered languages and combat the "annotation bottleneck" caused by current manual methods @xcite @xcite . although adding pos tagging as a high-priority task would add to that bottleneck, if the tags have a significant and positive impact on automating segmentation and glossing, then linguists may receive long-term benefits from the addition to their workflow. therefore, we explore the impact of pos tags at very low-resource settings and the impact of pos tags when a new field project takes time to tag some, but not all, tokens. this is also why we chose noisy field corpora, rather than published, polished corpora which are not like the data that linguists typically work with. we are interested in how pos tags influence segmentation and glossing in the earliest work with a new language. ## pos for reinflection the second study asks whether pos tags make a significant impact on learning inflectional patterns and generating unseen inflected forms. we chose the morphological re-inflection task because it is easy to reproduce and to compare with the original sigmorphon shared task. eliciting and analyzing a language's inflectional patterns is a recommended next step after morpheme segmentation and glossing @xcite . the inflectional pattern of a lexeme or a lexical category is also known as a morphological paradigm. learning morphological paradigms can be viewed in terms of filling in, or generating, the missing forms of a paradigm table by generalizing over inflectional patterns @xcite @xcite @xcite @xcite . the experiments in this section partly replicates the conll-sigmorphon 2018 shared task 1 of morphological reinflection. reinflection consists of generating unknown inflected forms, given a related inflected form f (ℓ, t γ 1 ) and a target morphological feature vector t γ 2 . thus, it corresponds to learning the mapping f : σ * × @xmath1. the goal is then to produce the inflected form f (ℓ, t γ 2 ). an inflected form is generated when the model is given a related inflected form and the target morphological features (which are essentially glosses of affixes) of the inflected form to be generated. in previous work, pos tags have been included by default as part of the morphological features. that is, they have been assumed to be helpful and to be available. ## discussion the number of language we used is not large but a few general observations can be made. for both tasks the impact made by the presence or absence of pos tags is minimal. still, the best results with a small corpus are achieved when either all or no tokens are pos-tagged, at least for segmentation and glossing. this suggests that having a completely tagged corpus is better than an incompletely tagged corpus, so perhaps limited annotation time might be better spent on more segmentation and glossing. the size or specificity of the tag set may make a difference in the impact of pos tags. when comparing the tag sets in the conll-sigmorphon 2018 shared task data and the igt from fieldwork, the difference in the number of lexical categories is significant. the conll-sigmorphon 2018 shared task data sets have at most three: noun (n), verb (v), and adjective (adj). the igt corpora have larger tag sets; for example, they may have tags for both finite verb form (vf) and non-finite forms (vnf). the smallest igt tag set has six categories (manipuri). that is twice as many pos tags as the sigmorphon languages, but still much smaller than the other corpora, which have over 20 unique tags. however, the difference in results cannot be definitely attributed to tag set size. the igt tag sets are larger because the goal of descriptive work is to discover fine-grained categories, whereas the unimorph data use more general categories which are common for language learning material or general dictionaries. similar fine-grained distinctions appear in the penn treebank tag set and are presumably useful for nlp tasks. future work could re-tag igt with more general categories to test how the size and specificity of pos tags on small corpora impact these tasks. this could be fruitful area of research because it might help us predict the usefulness of another linguistic category: the category of morphemes. morpheme-level categories are similar to pos tags but tagged for individual morphemes. interestingly, morpheme categories generally take higher priority than word-level tags in documentary and descriptive linguists and are therefore more often available in field data. consistency of annotation may be significant. it is likely that the pos tags in the unimorph data were added carefully and correctly, but the field data were likely tagged as the lexical categories were being discovered and described. the differences in results between the two data sets may be due to these factors, but the differences are not huge. so it seems possible that the effect of pos tags may be similar no matter how the pos tags are added. a different approach to pos-tagging, such as training with context might affect results. this possibility points to many future useful experiments. we believe there may be many unresolved issues related to the way the pos tags were added or which pos tags were used. one auxiliary task would be to project pos tags from the target language of the translated sentences that are usually available in igt even before morpheme segmentation and glosses. also, metrics for annotation quality could be devised so that its impact is better understood. linguists need to know as they start annotation how best to perform their earliest analysis and annotation so that they gain optimal benefit from automated help later. finally, although a consistent impact by pos tags cannot be seen on morphological learning across all corpora, some corpora did show a more or less consistent impact from the presence or absence of pos tags. sometimes better results were achieved by removing pos tags, sometimes by adding them. reinflection in adyghe and the "clean" version of lezgi data tend to improve when pos tags are removed while persian, russian, and the noisy version of natügu generally have more accurate results when pos tags are available. in segmentation and glossing, alas and lamkang show in some settings nearly .1 points difference when pos tags are added and removed, respectively. with these trends, a more interesting question for these corpora becomes "when are pos tags helpful?" and this should be explored further. ## conclusion we conclude that the presence or absence of pos tags does not have a significant impact on two morphological learning tasks: segmentation and glossing, or reinflection. no clear advantage is gained or lost from pos-tagging on low-resource data. in segmentation and glossing, the greatest average difference is a loss of .09 f 1 -score when a large pos tag set is added to a small field corpus. in reinflection, the overall tendency, though slight, is that accuracy decreases when pos tags are added. the greatest average difference is 1.2 percentage points of accuracy for published data, 2.2 points for unpublished "clean" data, and 5 points for unpublished noisy data. we hypothesize that pos tags do not have a significant impact on these tasks because the information provided by pos tags is implicitly learned. these are, of course, not the only two tasks where pos tags could be leveraged for low-resource languages so we cannot make a definitive statement regarding the impact of pos tags in other nlp tasks with low-resource languages, particularly ones that more syntactic or semantic in nature. further methodical research needs to be done in order to produce a definitive analysis. however, it does bring into question whether the development of pos taggers and pos tagging should be prioritized less. future work should explore how other tasks are impacted by pos tags. the results might influence workflow priorities for documentary and descriptive linguists who want to receive benefit from, or give it to, nlp. when a sophisticated pos tag set and pos taggers are available for a language, leveraging pos tags is trivial. however, as nlp expands into a broader range of languages, the usefulness of pos tags may become an important question because documentary and descriptive linguistics does not currently place a high priority on lexical categories. discovering a language's lexical categories requires a detailed understanding of the language's syntax-something linguists do not always possess in the early stages of describing a new language.