Reducing Confusion in Active Learning for Part-Of-Speech Tagging (2021)

Abstract: Active learning (AL) uses a data selection algorithm to select useful training samples to minimize annotation cost. This is now an essential tool for building low-resource syntactic analyzers such as part-of-speech (POS) taggers. Existing AL heuristics are generally designed on the principle of selecting uncertain yet representative training instances, where annotating these instances may reduce a large number of errors. However, in an empirical study across six typologically diverse languages (German, Swedish, Galician, North Sami, Persian, and Ukrainian), we found the surprising result that even in an oracle scenario where we know the true uncertainty of predictions, these current heuristics are far from optimal. Based on this analysis, we pose the problem of AL as selecting instances that maximally reduce the confusion between particular pairs of output tags. Extensive experimentation on the aforementioned languages shows that our proposed AL strategy outperforms other AL strategies by a significant margin. We also present auxiliary results demonstrating the importance of proper calibration of models, which we ensure through cross-view training, and analysis demonstrating how our proposed strategy selects examples that more closely follow the oracle data distribution. The code is publicly released here.1

## introduction part-of-speech (pos) tagging is a crucial step for language understanding, both being used in automatic language understanding applications such as named entity recognition (ner; @xcite and question answering (qa; @xcite , but also being used in manual lan-guage understanding by linguists who are attempting to answer linguistic questions or document less-resourced languages @xcite . much prior work @xcite on developing high-quality pos taggers uses neural network methods, which rely on the availability of large amounts of labelled data. however, such resources are not readily available for the majority of the world's 7000 languages @xcite . furthermore, manually annotating large amounts of text with trained experts is an expensive and time-consuming task, even more so when linguists/annotators might not be native speakers of the language. active learning @xcite is a family of methods that aim to train effective models with less human effort and cost by selecting such a subset of data that maximizes the end model performance. although many methods have been proposed for al in sequence labeling @xcite @xcite , through an empirical study across six typologically diverse languages we show that within the same task setup these methods perform inconsistently. furthermore, even in an oracle scenario, where we have access to the true labels during data selection, existing methods are far from optimal. we posit that the primary reason for this inconsistent performance is that while existing methods consider uncertainty in predictions, they do not consider the direction of the uncertainty with respect to the output labels. for instance, in figure 1 we consider the german token ''die,'' which may be either a pronoun (pro) or @xcite . according to the initial model (iteration 0), ''die'' was labeled as pro majority of the time, but a significant amount of probability mass was also assigned to other output tags (other) for many examples. based on this, existing al algorithms that select uncertain tokens will likely select ''die'' because it is frequent and 1 transactions of the association for computational linguistics, vol. 9, pp. 1-16, 2021. https://doi.org/10.1162/tacl a 00350 action editor: yuji matsumoto. submission batch: 3/2020; revision batch: 6/2020; published 02/2021. c 2021 association for computational linguistics. distributed under a cc-by 4.0 license. figure 1 : illustration of selecting representative token-tag combinations to reduce confusion between the output tags on the german token ''die'' in an idealized scenario where we know true model confusion. we evaluate our proposed al method by running simulation experiments on six typologically diverse languages, namely, german, swedish, galician, north sami, persian, and ukrainian, improving upon models seeded with crosslingual transfer from related languages @xcite . in addition, we conduct human annotation experiments on griko, an endangered language that truly lacks significant resources. our contributions are as follows: informativeness represents the ability of the selected data to reduce the model uncertainty on its predictions, and representativeness measures how well the selected data represent the entire unlabeled data. al is an iterative process and is typically implemented in a batched fashion for neural models @xcite . in a given iteration, a batch of data is selected using some heuristic on which the end model is trained until convergence. this trained model is then used to select the next batch for annotation, and so forth. in this work we focus on token-level al methods, which require annotation of individual tokens in context, rather than full sequence annotation, which is more time consuming. given an unlabeled pool of sequences @xmath0, x 2 , next, this entropy is aggregated over all token occurrences across d to get an uncertainty score s uns (z) for each word type z: ## confusion-reducing active learning to address the limitations of the existing methods, we propose a confusion-reducing active learning (cral) strategy, which aims at reducing the confusion between the output tags. in order to combine both ''informativeness'' and ''representativeness'', we follow a two-step algorithm: again selecting the top b types having the highest score (given by b-arg max) gives us the most confusing word types (x init ). for each token, we also store the output tag that had the second highest probability which we refer to as the ''most confusing output tag'' for a particular x i, t : for each word type z, we aggregate the frequency of the most confusing output tag across all token occurrences: and compute the output tag with the highest frequency as the most confusing output tag for type z: for each of the top b most confusing word types, we retrieve its most confusing output tag, resulting in type-tag pairs given by because the true labels are unknown, there is no certain way of knowing which tokens have the ''most confusing output tag'' as the true label. therefore, each token representation algorithm 1 confusion-reducing al 1: d ← unlabeled set of sequences 2: z ← vocabulary 3: j ← output tag-set 4: b ← active learning batch size 5: p θ (y i, @xmath1: p i, t, j := p θ (y i, @xmath2: for x @xmath3: finally, the token instance that is closest to the centroid of this weighted token set becomes the most representative instance for annotation. going forward, we also refer to the most representative token instance as the centroid for simplicity. 4 this process is repeated for each of the word-types z k , resulting in the to-label set x label . this is illustrated in steps 14-19 in algorithm 1. during the annotation process, the selected representative tokens of each selected confusing word type are presented in context similar to @xcite and @xcite . 4 sener and savarese (2018) describe why choosing the centroid is a good approximation of representativeness. they pose al as a core-set selection problem where a core set is the subset of data on which the model if trained closely matches the performance of the model trained on the entire dataset. they show that finding the core set is equivalent to choosing b center points such that the largest distance between a data point and its nearest center is minimized. we take inspiration from this result in using the centroid to be the most representative instance. ## model and training regimen now that we have a method to select data for annotation, we present our pos tagger in section §4.1, followed by the training algorithm in section §4.2. ## simulation experiments in this section, we describe the simulation experiments used for evaluating our method. under this setting, we use the provided training data as our unlabeled pool and simulate annotations by using the gold labels for each al method. datasets: for the simulation experiments, we test on six typologically diverse languages: german, swedish, north sami, persian, ukrainian, and galician. we use data from the universal dependencies (ud) v2.3 @xcite @xcite treebanks with the target language transfer languages (treebank) german (de-gsd) english (en-ewt) + dutch (nl-alpino) swedish (sv-lines) norwegian (no-nynorsk) + danish (da-ddt) north sami (sme-giella) finnish (fi-ftb) persian (fa-seraji) urdu (ur-udtb) + russian (ru-gsd) galician (gl-treegal) spanish (es-gsd) + portuguese (pt-gsd) ukrainian (uk-iu) russian (ru-gsd) griko greek (el-gdt) + italian (it-postwita) baselines: as described in section §2, we compare our proposed method (cral) with uncertainty sampling (uns) and query-by-commitee (qbc). we also compare with a random baseline (rand) that selects tokens randomly from the unlabeled data d. for qbc, we use the following committee of models @xmath4, θ bwd , θ full }, where θ i are the cvt views ( §4.2). we do not include the θ fut and θ pst as they are much weaker in comparison to the other views. 6 for cral, uns, and rand, we use the full model view. ## human annotation experiment in this section, we apply our proposed approach on griko, an endangered language spoken by around 20,000 people in southern italy, in the grecìa salentina area southeast of lecce. the only available online griko corpus, referred to as uoi @xcite the ia agr. column reports the inter-annotator agreement against the expert linguist for the first iteration. wd is the wasserstein distance between the selected tokens and the test distribution. pos model. 9 to further improve the model, we fine-tune on the uoi corpus, which consists of 360 labeled sentences. we evaluate the al performance on the 10 gold-labelled stories from @xcite , of which the first two stories, comprising 143 labeled sentences, are used as the validation set and the remaining 800 labeled sentences form the test set. we use the unannotated stories as our unlabeled pool. we compare cral with uns and qbc, conducting three al iterations for each method, where each iteration selects roughly 50 tokens for annotation. the annotations are provided by two linguists familiar with modern greek and somewhat familiar with griko. to familiarize the linguists with the annotation interface, a practice session was conducted in modern greek. in the interface, tokens that need to be annotated are highlighted and presented with their surrounding context. the linguist then simply selects the appropriate pos tag for each highlighted token. because we do not have gold annotations for these experiments, we also obtained annotations from a third linguist who is more familiar with griko grammar. results: for linguist-1, we observe a decrease in performance in iteration-3. one possible reason for this decrease is attributed to linguist-1's poor annotation quality, which is also reflected in their low inter-annotator agreement scores. we observe a slight decrease for other linguists, which we hypothesize is due to domain mismatch between the annotated data and the test data. in fact, the test set stories and the unlabeled ones originate from different time periods spanning a century, which can lead to slight differences in orthography and usage. for instance, after three al iterations, the token ''i'' had been annotated as conj twice and det once, whereas in the test data all instances of ''i'' are annotated as det. similar to the simulation experiments, we compute the confusion score for all linguists in figure 9 . we find that, unlike in the simulation experiments, a model trained with uns causes less damage on the existing annotations as compared to cral. however, we note that the model performance from the uns annotations is much lower than cral to begin with. we also compute the inter-annotator agreement at iteration-1 with the expert (linguist-3) ( similar to the simulation experiments, we report the wasserstein distance (wd) for all linguists in low-resource pos tagging: several crosslingual transfer techniques have been used for improving low-resource pos tagging. @xcite and @xcite train a joint neural model on related highresource languages and find it be very effective on low-resource languages. the main advantage of these methods is that they do not require any parallel text or dictionaries. @xcite , @xcite , and @xcite use annotation projection methods to project pos annotations from one language to another. however, annotation projection methods use parallel text, which often might not be of good quality for low-resource languages. ## conclusion we have presented a novel active learning method for low-resource pos tagging that works by reducing confusion between output tags. using simulation experiments across six typologically diverse languages, we show that our confusionreducing strategy achieves higher accuracy than existing methods. further, we test our approach under a true setting of active learning where we ask linguists to document pos information for an endangered language, griko. despite being unfamiliar with the language, our proposed method achieves performance gains over the other methods in most iterations. for our next steps, we plan to explore the possibility of adapting our proposed method for complete morphological analysis, which poses an even harder challenge for al data selection due to the complexity of the task.