Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Training (2023)

Abstract: In this work we propose a pragmatic method that reduces the annotation cost for structured label spaces using active learning. Our approach leverages partial annotation, which reduces labeling costs for structured outputs by selecting only the most informative sub-structures for annotation. We also utilize self-training to incorporate the current model’s automatic predictions as pseudo-labels for un-annotated sub-structures. A key challenge in effectively combining partial annotation with self-training to reduce annotation cost is determining which sub-structures to select to label. To address this challenge, we adopt an error estimator to adaptively decide the partial selection ratio according to the current model’s capability. In evaluations spanning four structured prediction tasks, we show that our combination of partial annotation and self-training using an adaptive selection ratio reduces annotation cost over strong full annotation baselines under a fair comparison scheme that takes reading time into consideration.

## introduction structured prediction @xcite is a fundamental problem in nlp, wherein the label space consists of complex structured outputs with groups of interdependent variables. it covers a wide range of nlp tasks, including sequence labeling, syntactic parsing and information extraction (ie). modern structured predictors are developed in a data-driven way, by training statistical models with suitable annotated data. recent developments in neural models and especially pre-trained language models @xcite @xcite have greatly improved system performance on these tasks. nevertheless, the success of these models still relies on the availability of sufficient manually annotated data, which is often expensive and time-consuming to obtain. he saw the man with a backpack to mitigate such data bottlenecks, active learning (al), which allows the model to select the most informative data instances to annotate, has been demonstrated to achieve good model accuracy while requiring fewer labels @xcite . when applying al to structured prediction, one natural strategy is to perform full annotation (fa) for the output structures, for example, annotating a full sequence of labels or a full syntax tree. due to its simplicity, fa has been widely adopted in al approaches for structured prediction tasks @xcite @xcite . nevertheless, a structured object can usually be decomposed into smaller sub-structures having nonuniform difficulty and informativeness. for example, as shown in figure 1 , in a dependency tree, edges such as functional relations are relatively easy to learn, requiring fewer manual annotations, while prepositional attachment links may be more informative and thus more worthwhile to annotate. the non-uniform distribution of informative substructures naturally suggests al with partial annotation (pa), where the annotation budget can be preserved by only choosing a portion of informative sub-structures to annotate rather than laboriously labeling entire sentence structures. this idea has been explored in previous work, covering typical structured prediction tasks such as sequence labeling @xcite @xcite algorithm 1 al procedure. input: seed dataset l0, dev dataset d, unlabeled pool u, total budget t, batch selection size b, annotation strategy. output: final labeled dataset l, trained model m. s ← sentence-query(m, u) # sentence selection 5: if @xmath0: r ← auto-ratio(s, d) # decide adaptive ratio 7: u ← u -s; l ← l ∪ s; t ← tb 11: m ← train(l, u) # final model training 12: return l, m and dependency parsing @xcite @xcite . our work follows this direction and investigates the central question in al with pa of how to decide which sub-structures to select. most previous work uses a pre-defined fixed selection criterion, such as a threshold or ratio, which may be hard to decide in practice. in this work, we adopt a performance predictor to estimate the error rate of the queried instances and decide the ratio of partial selection accordingly. in this way, our approach can automatically and adaptively adjust the amount of partial selection throughout the al process. another interesting question for al is how to better leverage unlabeled data. in this work, we investigate a simple semi-supervised method, self-training @xcite , which adopts the model's automatic predictions on the unlabeled data as extra training signals. self-training naturally complements al in the typical pool-based setting where we assume access to a pool of unlabeled data @xcite . it is particularly compatible with pa-based al since the un-selected substructures are typically also highly-confident under the current model and likely to be predicted correctly without requiring additional annotation. we revisit this idea from previous work @xcite and investigate its applicability with modern neural models and our adaptive partial selection approach. we perform a comprehensive empirical investigation on the effectiveness of different al strategies for typical structured prediction tasks. we perform fair comparisons that account for the hidden cost of reading time by keeping the context size the same for all the strategies in each al cy-cle. with evaluations on four benchmark tasks for structured prediction (named entity recognition, dependency parsing, event extraction, and relation extraction), we show that pa can obtain roughly the same benefits as fa with the same reading cost but less sub-structure labeling cost, leading to better data efficiency. we also demonstrate that the adaptive partial selection scheme and self-training play crucial and complementary roles. ## conclusion in this work, we investigate better al strategies for structured prediction in nlp, adopting a performance estimator to automatically decide suitable ratios for partial sub-structure selection and utilizing self-training to make better use of the available unlabeled data pool. with comprehensive experiments on various tasks, we show that the combination of pa and self-training can be more dataefficient than strong full al baselines.