Diversity-Aware Batch Active Learning for Dependency Parsing (2021)

Abstract: While the predictive performance of modern statistical dependency parsers relies heavily on the availability of expensive expert-annotated treebank data, not all annotations contribute equally to the training of the parsers. In this paper, we attempt to reduce the number of labeled examples needed to train a strong dependency parser using batch active learning (AL). In particular, we investigate whether enforcing diversity in the sampled batches, using determinantal point processes (DPPs), can improve over their diversity-agnostic counterparts. Simulation experiments on an English newswire corpus show that selecting diverse batches with DPPs is superior to strong selection strategies that do not enforce batch diversity, especially during the initial stages of the learning process. Additionally, our diversity-aware strategy is robust under a corpus duplication setting, where diversity-agnostic sampling strategies exhibit significant degradation.

## introduction though critical to parser training, data annotations for dependency parsing are both expensive and time-consuming to obtain. syntactic analysis requires linguistic expertise and even after extensive training, data annotation can still be burdensome. the penn treebank project @xcite reports that after two months of training, the annotators average 750 tokens per hour on the bracketing task; the prague dependency treebank @xcite cost over @xmath05 per sentence). these high annotation costs present a significant challenge to developing accurate dependency parsers for under-resourced languages and domains. active learning (al; @xcite is a promising technique to reduce the annotation effort required to train a strong dependency parser by intel- * work done during an internship at bloomberg l.p. ligently selecting samples to annotate such that the return of each annotator hour is as high as possible. popular selection strategies, such as uncertainty sampling, associate each instance with a quality measure based on the uncertainty or confidence level of the current parser, and higher-quality instances are selected for annotation. we focus on batch mode al, since it is generally more efficient for annotators to label in bulk. while early work in al for parsing @xcite @xcite cautions against using individually-computed quality measures in the batch setting, more recent work demonstrates empirical success (e.g., @xcite without explicitly handling intra-batch diversity. in this paper, we explore whether a diversity-aware approach can improve the state of the art in al for dependency parsing. specifically, we consider samples drawn from determinantal point processes (dpps) as a query strategy to select batches of high-quality, yet dissimilar instances @xcite . in this paper, we (1) propose a diversity-aware batch al query strategy for dependency parsing compatible with existing selection strategies, (2) empirically study three al strategies with and without diversity factors, and (3) find that diversityaware selection strategies are superior to their diversity-agnostic counterparts, especially during the early stages of the learning process, in simulation experiments on an english newswire corpus. this is critical in low-budget al settings, which we further confirm in a corpus duplication setting. foot_0 2 active learning for dependency parsing ## selecting diverse samples near-duplicate examples are common in real-world data @xcite , but they provide overlapping utility to model training. in the extreme case, with a diversity-agnostic strategy for active learning, identical examples will be selected/excluded at the same time @xcite . to address this issue and to best utilize the annotation budget, it is important to consider diversity. we adapt @xcite to explicitly model diversity using determinantal point processes (dpps). ## experiments and results dataset we use the revised english news text treebank foot_2 @xcite converted to universal dependencies 2.0 using the conversion tool included in stanford parser @xcite version 4.0.0. we use sections 02-21 for training, 22 for development and 23 for test. ## related work modeling diversity in batch-mode al @xcite has recently attracted attention in the machine learning community. @xcite introduce a bayesian batch-mode selection strategy by estimating the mutual information between a set of samples and the model parameters. @xcite present a diversity-inducing sampling method using gradient embeddings. most related to our work, @xcite first apply dpps to batch-mode al. building on their approach, we flesh out a dpp treatment for al for a structured prediction task, dependency parsing. previously, @xcite consider named entity recognition but they report negative results for a diversityinducing variant of their sampling method. due to the high annotation cost, al is a popular technique for parsing and parse selection @xcite . recent advances focus on reducing full-sentence annotations to a subset of tokens within a sentence @xcite @xcite @xcite . we show that al for parsing can further benefit from diversity-aware sampling strategies. dpps have previously been successfully applied to the tasks of extractive text summarization @xcite and modeling phoneme inventories @xcite . in this work, we show that dpps also provide a useful framework for understanding and modeling quality and diversity in active learning for nlp tasks. ## conclusion we show that compared with their diversityagnostic counterparts, diversity-aware sampling strategies not only lead to higher data efficiency, but are also more robust under corpus duplication settings. our work invites future research into methods, utility and success conditions for modeling diversity in active learning for nlp tasks.