A Confidence-based Acquisition Model for Self-supervised Active Learning and Label Correction (2025)

Abstract: Supervised neural approaches are hindered by their dependence on large, meticulously annotated datasets, a requirement that is particularly cumbersome for sequential tasks. The quality of annotations tends to deteriorate with the transition from expert-based to crowd-sourced labeling. To address these challenges, we present CAMEL (Confidence-based Acquisition Model for Efficient self-supervised active Learning), a pool-based active learning framework tailored to sequential multi-output problems. CAMEL possesses two core features: (1) it requires expert annotators to label only a fraction of a chosen sequence, and (2) it facilitates self-supervision for the remainder of the sequence. By deploying a label correction mechanism, CAMEL can also be utilized for data cleaning. We evaluate CAMEL on two sequential tasks, with a special emphasis on dialogue belief tracking, a task plagued by the constraints of limited and noisy datasets. Our experiments demonstrate that CAMEL significantly outperforms the baselines in terms of efficiency. Furthermore, the data corrections suggested by our method contribute to an overall improvement in the quality of the resulting datasets.1

## introduction supervised training of deep neural networks requires large amounts of accurately annotated data @xcite @xcite . a particularly challenging scenario arises when training for sequential multi-output tasks. in this case, the neural network is required to generate multiple predictions simultaneously, one for each output category, at every time step throughout an input sequence. consequently, the labeling effort increases rapidly, becoming impractical as the demand for precise and consistent labeling across each time step and output category intensifies. therefore, a heavy dependence on human-generated labels poses significant limitations on the scalability of such systems. a prominent example of a sequential multi-output label task for which this bottleneck is evident is dialogue belief tracking. a dialogue belief tracker is one of the core components of a dialogue system, tasked with inferring the goal of the user at every turn @xcite . current state-of-the-art trackers are based on deep neural network models @xcite @xcite . these models outperform traditional bayesian network-based belief trackers @xcite . however, neural belief trackers are greatly hindered by the lack of adequate training data. real-world conversations, even those pertaining to a specific task-oriented domain, are extremely diverse. they encompass a broad spectrum of user objectives, natural language variations, and the overall dynamic nature of human conversation. while there are many sources for dialogue data, such as logs of call centers or virtual personal assistants, labeled dialogue data is scarce and several orders of magnitude smaller than, say, data for speech recognition @xcite or translation @xcite . although zero-shot trackers do not require large amounts of labeled data, they typically underperform compared to supervised models that are trained on accurately labeled datasets @xcite . one of the largest available labeled datasets for task-oriented dialogues is multiwoz, which is a multi-domain dialogue dataset annotated via crowdsourced annotators. the challenges in achieving consistent and precise human annotations are apparent in all versions of multiwoz ( @xcite @xcite @xcite . despite manual corrections in the most recent edition, model performance has plateaued, not due to limitations in the models, but as a result of data inconsistencies @xcite . addressing the omnipresent issue of unreliable labels, as evident in the multiwoz dataset, is a common problem that affects the quality and reliability of supervised learning systems. in order to mitigate these issues and enhance the robustness of model training, we propose a novel methodology. in this work, we present camel, a pool-based semi-supervised active learning approach for sequential multi-output tasks. given an underlying supervised learning model that can estimate confidence in its predictions, camel substantially reduces the required labeling effort. camel comprises: we first apply camel within an idealized setting for machine translation, a generative language modeling task. camel achieves impressive results, matching the performance of a model trained on the full dataset while utilizing less than 60% of the expert-provided labels. subsequently, we apply camel to the dialogue belief tracking task. notably, we achieve 95% of a tracker's full-training dataset performance using merely 16% of the expert-provided labels. additionally, we propose an adaptation of the meta-post-hoc model approach @xcite , tailored for cost-efficient active learning. we demonstrate that camel, utilizing uncertainty estimates from this cost-effective method, exhibits similar performance compared to using uncertainty estimates from a significantly more computationally expensive ensemble of models. on top of this framework, we develop a method for automatically detecting and correcting inaccuracies of human labels in datasets. we illustrate that these corrections boost performance of distinct tracking models, overcoming the limitations imposed by labeling inconsistencies. having demonstrated its efficacy in machine translation and dialogue belief tracking, our framework holds potential for broad applicability across various sequential multi-output tasks, such as object tracking, pose detection, and language modeling. ## camel: confidence-based acquisition model for efficient self-supervised active learning in this section, we introduce our pool-based active learning approach, named camel, to address sequential multi-output classification problems. let us consider a classification problem with input features x, and output y. according to @xcite , such a problem can be cast as a multi-output classification problem if the output consists of multiple label categories that need to be predicted simultaneously. specifically, for a problem with m categories, the output is represented as @xmath0, y 2 , . . . , y m , where each y m , @xmath1, m] can be binary or multivariate. furthermore, this problem is characterized as a sequential classification problem if the output is dependent on a sequence of prior inputs. for a sequence with t time-steps, the input-output pairs can be represented as (x 1 , y 1 ), (x 2 , y 2 ), . . . , (x t , y t ) , where y @xmath2, y 2 t , . . . , y m t represents the output labels at time step @xmath3, t ]. in a conventional setting, for an unlabeled data sequence x @xmath4, . . . , x t i , an annotator would typically be required to provide labels, y m t , for each label category m at every time step t, which is considerably expensive. ## conclusion we propose camel, a novel active learning approach that integrates self-supervision, with the goal of minimizing the reliance on labeled data in addressing sequential multi-output labeling problems. initially, we applied camel to a generative language modeling task in an idealized setting, specifically focusing on machine translation. subsequently, in a more realistic setting focused on the dialogue belief tracking task, we demonstrated that our approach significantly outperforms baseline methods in terms of robustness and data efficiency. additionally, we introduce a methodology for automated dataset correction. our experiments confirm that our label correction method enhances the overall quality of a dataset. we demonstrate that camell (with label validation) is capable of producing high-quality datasets with a fraction of the human annotation required, through online label correction, thereby highlighting the importance of the label validation component for this task. finally, it is important to note that while many presented experiments used ensembles to establish comparisons, we have also provided a mechanism for confidence estimation and active learning that does not utilize ensembles and thus is more environmentally friendly. we believe that this work has far-reaching implications. firstly, it underscores the indispensable role of uncertainty estimation in learning models. secondly, the versatility of camel opens up possibilities for its application across diverse sequential multi-output labeling problems, such as entity-relation extraction or weather forecasting. thirdly, it demonstrates that, in principle, dataset deficiencies can be addressed via data-driven approaches, circumventing the need for extensive manual or rule-based curation. this is particularly pertinent considering the prevailing belief that undesirable outcomes produced by nlp models are inherently linked to the training datasets and cannot be rectified algorithmically @xcite . looking ahead, we anticipate that refining the process of generating noisy datasets could result in a model capable of not only identifying label noise but also filtering out biases, false premises, and misinformation.