What data should I include in my POS tagging training set? (2025)

Abstract: Building an NLP training set for understudied languages, including Indigenous and endangered languages, often faces challenges due to varying degrees of resource limitations in the speaker communities. What are some reasonable approaches for training set construction in these cases? We address this question with POS tagging as the test case. Although many might consider POS tagging “a solved problem”, it remains a crucial task for descriptive linguistics and language documentation and requires laborious manual annotation. Drawing data from 12 language families, we compare in-context learning, active learning (AL), and random sampling. Our results suggest: (1) for communities whose language data can be ethically shared with an API, using only 1,000 randomly sampled tokens as prompt examples, the proprietary GPT-4.1-mini can deliver desirable performance (F1>0.83) on par with that from a training set of thousands of tokens in AL iterations; (2) in cases where communities prefer not to share data, 4,500-5,500 tokens selected from AL can yield reasonable results at a pace statistically significantly faster than random sampling, evidenced by growth curve modeling.

## introduction when exploring a new language and domain for a variety of computational linguistics and natural language processing (nlp) tasks, we typically need to create a new training set for the task of interest. how should one create a new training set in order to derive more generalizable model performance on new unseen data in the wild? in particular, how much training data is necessary @xcite , and perhaps more importantly, what kind of data should be included in the training set? an important consideration when addressing these questions is resource availability. the notion of resource availability, from our perspective, entails: (1) the amount of financial support for de-veloping computational techniques; (2) the number of sources from which data can be collected; (3) the number and scope of manual annotations that can be acquired. as such, resource availability exists on a continuum with the abundant resources of the "big languages" that are the focus of most research @xcite at one end and the severe resource limitation as in indigenous and endangered languages at the other @xcite . now consider the other end of the resource continuum represented by indigenous and endangered languages, for which the resources described above for languages such as english are, in most cases, unattainable. because the focus on computational research has thus far been on languages with large speaker populations, there has consistently been much less financial support for and attention to language technology development for indigenous and endangered languages @xcite . given their extremely small speaker populations, the number of sources for data collection for these languages is limited. text data often comes from restricted domains such as the bible @xcite or grammar books @xcite , the content of which can be biased towards stilted or formal versions of the language that are detached from the reality of the language as it is spoken. speech data is often derived from linguistic fieldwork @xcite carried out over decades using variable equipment and elicitation techniques. the limited availability of l1 speakers presents challenges for digitization from written texts and manual transcription of audio data. it would be impossible for the training set for an indigenous or endangered language to ever equal that of a language like english. in addition, the annotations of the training data might vary depending on the linguistic expertise of the l1 speakers. these limitations complicate the question of training set creation, where great care is needed to perform resource allocation in a more thoughtful and efficient way. here we ask: what are some reasonable approaches to use when building a training set with limited annotation sources? our goal is to inform new training set construction, particularly for underrepresented languages using existing datasets. this means the task to select as the test case should: (1) be valuable for language documentation; (2) have data available in a wide range of languages with different typological characteristics. with these considerations, we focus on part-of-speech (pos) tagging, taking advantage of the universal dependencies (ud) project @xcite . we compare in-context learning with large language model (llm), uncertainty sampling (lewis, 1995) from the active learning framework @xcite , and random sampling @xcite . for in-context learning, we leverage data from 60 languages (one treebank per language) across 12 language families from ud v2.14 @xcite . 1 ; for active learning and random sampling, we expand to 112 treebanks spanning 60 languages of the same language families. the goal of pos tagging is to automatically assign each token in a given sentence a tag identifying its part-of-speech category. ## related work in-context learning there is a growing number of studies, mostly using in-context learning, to probe llms for their cross-linguistic capabilities in a variety of tasks, including but not limited to native language identification @xcite , machine translation @xcite , and word sense disambiguation @xcite . among these studies, some attend to lowresource or underrepresented languages specifically @xcite , with work ranging from interlinear glossing @xcite and grammar creation (spencer and kongborrirak, 2025) for endangered languages, to natural language inference for indigenous languages of the americas @xcite . only a few experiments have looked at pos tagging with in-context learning in particular. machado and ruiz (2024) compared three llms for pos tagging of portuguese: gpt-3, llama-7b, and maritaca (pretrained on portuguese specifically). their prompt included 10 sentence examples and each llm was asked to generate inference for 1,000 sentences. the results showed while gpt-3 achieved f1 scores above 0.8, the performance of llama-7b (∼0.56) and maritaca (∼0.42) was much worse. adelani et al. (2024) used zero-shot prompting with gpt-4 for 12 lowresource languages from brazil and two from africa, and found that this approach outperformed zero-shot cross-linguistic transfer learning with alternative pretrained lms @xcite . our study goes beyond the scope of prior work using llms via experimenting with few-shot incontext learning for 60 languages from 12 language families (60 treebanks). one might be inclined to believe that an llm would undoubtedly perform well for pos tagging given that the task is comparatively simple; on the other hand, it is not unreasonable to suspect the opposite, given that llm performance for low-resource languages is quite variable (e.g., @xcite ; @xcite . these assumptions necessitate quantifying the performance of llms for pos tagging to empirically inform training set construction. active learning the active learning (al) framework has long been proposed as a method for informative data selection when resources for acquiring manual annotations are limited @xcite @xcite @xcite . the goal is to derive models that can reach a certain level of performance from comparatively less training data. the general process of al is as follows. we have a designated test set and a training data pool to draw training samples. we first build an initial training set of a certain size from the training data pool, and train a model on this initial training set. we then apply the trained model to the residual data from the training data pool; the model generates a score (e.g., a confidence score) for each data point in the residual data that reflects how confident the model is in its prediction for that particular data point. then a number of x data points for which the model is the least certain about will be added to the initial training set with its correct output label. we build a model using this expanded training set to start the next iteration of al; this process continues iteratively with the value of x fixed in every cycle. in real-world scenarios, the initial training set and the additional data selected in each cycle of al are mostly annotated by human annotators then passed on to the next iteration. here we perform computational simulation of this process in controlled experimental settings, using existing annotations provided by ud. while al is promising, current research on this topic, including work focusing on pos tagging, faces shortcomings. first, the majority of al-related studies still attend to individual languages, most of which are english. large-scale cross-linguistic investigations are still rare, with a few exceptions that have focused on the task of morphological inflection @xcite . while some studies have looked into al for pos tagging, they are again constrained by the language samples in their individual experiments (e.g., only english in @xcite , eight indo-european languages in @xcite , and six lowresource languages in @xcite ). second, the analyses of results from al in the literature tend to take an "eyeballing" approach. these analyses rely on raw numbers and visualizations of the learning curves to draw conclusions about whether and to what extent al is helpful. our work addresses these gaps. we investigate al for pos tagging across 60 languages; the different treebanks for the same language are deliberately studied individually to conserve potential impact from different domains on the observations. in addition, unlike previous research that tends to lack statistical validation, we show how results of different sampling methods (al vs. random sampling) can be compared systematically using growth curve modeling (panik, 2014), originated from literature on biological analysis @xcite . ## discussion & future directions with existing datasets for 60 languages from 12 language families, our study makes the following recommendations for building pos tagging training sets for new languages that possibly face limitations in available annotation efforts. in scenarios where it is safe, ethical, and compatible with speaker community values to expose language data through an api, in-context learning with gpt-4.1-mini using a small randomly sampled training set of 1,000 tokens in the llm prompt can deliver desirable results that are mostly better than those derived from al and random sampling (at least for the languages investigated). in our experiments, api calls per treebank cost no more than $4 us dollars; this amount, together with the expense for obtaining manual annotations for 1,000 tokens, would be more economical than acquiring manual annotations for thousands of tokens for al. there are, however, many indigenous and endangered speech communities for whom data sovereignty is a sensitive issue @xcite . for these cases, al for pos tagging with a statistical model that runs locally can be effective. in our al experiments, the learning curves largely grow rapidly until reaching a training size of 4,500-5,500 tokens, after which growth diminishes and begins to approach an asymptote. our statistical model further reveals that across the languages tested, while al and random sampling might reach comparable maximum f1 scores eventually, models from al iterations arrive at the upper asymptote faster. this means that for resource-constrained scenarios with limited bandwidth for manual annotations, al is the better choice. we hope that our research methodology can be adopted in future work on al and dataset design more broadly. we see a number of fruitful directions in our future work. first, we would like to explore diversity sampling @xcite , an alternative sam-pling method that has been incorporated into al in previous literature @xcite . second, we hope to learn more about the practical impact of our findings. for instance, while our results might suggest certain recommendations for building datasets, we do not know the relationship between improvements in pos tagging f1 for these languages and downstream tasks that rely on pos tags. in addition, because our work only simulates the process of al, we do not yet fully understand the impact of putting al into action on the required effort of real annotators (see also @xcite ). finally, we would like to carry out similar experiments with other nlp tasks, especially those that are helpful in the contexts of documenting endangered languages, such as automatic speech recognition (prud'hommeaux et al., 2021) and morphological segmentation @xcite . ## limitations while we believe our work to be thorough and our choices well justified, we do acknowledge some potential limitations. first, we only experiment with a fixed size of 1,000 tokens as prompt examples for in-context learning; it is possible that even a smaller training sets might yield comparable performance. second, our prompt examples are randomly sampled; future work can consider more informed data selection strategies for in-context learning or finetuning with llm. finally, the ud project, while offering a wide range of language selections, is (as of now) heavily weighted toward ie languages, which are not necessarily representative linguistically of the world's languages at large, or indigenous and endangered languages in particular. we choose ud for its relatively consistent annotation standards across languages, as well as the fact that it is open-access. additionally, languages from ud with comparatively fewer resources can provide some idea of how well the approach would fare with endangered languages. since an endangered language is often tied closely to its respective community's identity and cultural heritage, work involving unpublished data from an endangered language is best done in close partnership with and for the direct benefit of its community. here, we focus on the methods that would enable such work. as more datasets annotated with (ud-style) pos tag information become publicly available, we hope our methods would be directly applicable to those cases. you are a helpful assistant who is an expert part of speech tagger that works with the universal dependencies part-of-speech tagset. the user will give you a sentence in irish to be tagged, with one token per line, where each line contains the token's index and the token. you must provide the tagged sentence in the following format: 1 token_1 tag_1 2 token_2 important: the sentence that the user provides has already been tokenized, and each sequence of characters separated by whitespace is a token. do not further split the tokens, and do not join tokens. also, do not change anything in the sentence provided. please provide only the tagged sentence, and nothing else. no explanation, no alternatives, only the tagger output in the format specified above. sentence: 1 bhí 2 an 3 geata 4 dúnta 5 .