Adaptive Open-Set Active Learning with Distance-Based Out-of-Distribution Detection for Robust Task-Oriented Dialog System (2024)

Abstract: The advancements in time-efficient data collection techniques such as active learning (AL) has become salient for user intent classification performance in task-oriented dialog systems (TODS). In realistic settings, however, traditional AL techniques often fail to efficiently select targeted in-distribution (IND) data when encountering newly acquired out-of-distribution (OOD) user intents in the unlabeled pool. In this paper, we introduce a novel AL framework viz., AOSAL for TODS that combines a distance-based OOD detector using adaptive false positive rate threshold with an informativeness measure (e.g., entropy) to strategically select informative IND data points in the unlabeled pool. Specifically, we utilize the adaptive OOD detector to classify and filter out OOD samples from the unlabeled pool, then prioritize the acquisition of classified IND instances based on their informativeness scores. To validate our approach, we conduct experiments that display our framework’s flexibility and performance over multiple distance-based approaches and informativeness measures against deep AL baselines on benchmark text datasets. The results suggest that our AOSAL approach consistently outperforms the baselines on IND classification and OOD detection, advancing knowledge on improving robustness of task-oriented dialog systems.

## introduction recent advances in time-efficient data collection techniques such active learning (al) @xcite show the promise of significantly improving the performance of task-oriented dialog system (tods) for tasks related to user intent classification @xcite . the time-efficient al techniques not only improve the model accuracy of the tods, but also help reduce the annotation budget of human annotators when querying the most informative samples that accelerate training performance. in real-world applications, however, existing al methods often struggle to select in-distribution (ind) data from unlabeled pools containing outof-distribution (ood) user intents, leading to inefficiencies in the learning process. figure 1 illustrates the challenge of employing standard al frameworks in a tods application, where an unlabeled dataset of collected user intents are noisy due to instances that are ood. typical queries using e.g., uncertainty @xcite and diversitybased methods @xcite ) are prone to selecting a high number of ood instances, which in turn can waste the annotation budget of the human annotator. consequently, this can lead to low classification performance, and more concretely, incorrect dialog responses if there are insufficient amount of ind samples selected for training, as shown in the example scenario in figure 2 . previous works have investigated robust al frameworks in the context of open-set recognition @xcite . the work in @xcite develops a progressive active learn- help me book a table for two at joe's diner for 7:00pm. tods intent classification (ind) intent classification (ood) sentiment analysis (ood) hello! how can i assist you today? i would like to buy two tickets for the harry potter movie at 7 pm tonight. i've heard great things about joe's diner. i can't wait to try their food! user user user tods sure thing! i have reserved a table for two at joe's diner for 7:00 pm. you got it! i booked two tickets for the harry potter movie at 7:00 pm. tods sure thing! i have reserved a table for you at joe's diner. tods restaurant booking domain ing framework that implements an ood detector for filtering ood instances in the unlabeled pool. although other works have proposed similar methods related to open-set active learning @xcite @xcite , these works are mainly applied in the computer vision space, and are unsuited for nlp tasks in tods. in addition, while al frameworks in the nlp space such as cal @xcite and coun-teral @xcite address ood generalization challenges, they are not practical to open-set al where the unlabeled dataset contains a mixture of ind and ood samples. given the emergence and applicability of tods in various application domains e.g., healthcare, banking, there presents a need to develop robust al strategies that avoid ood instances while also acquiring informative ind instances that improve model training. in this paper, we present a novel adaptive openset al framework viz., "aosal" for tods that combines an adaptive distance-based ood detector with informative sampling measures (e.g., uncertainty, diversity) to effectively acquire ind samples in the unlabeled pool. our ood detector features a normalized score function that classifies unlabeled samples based on their distance to each class in the ind labeled space. we enable our ood detector to be sensitive to distribution shifts by employing an adaptive threshold that is controlled by using a predetermined false positive rate (fpr) over the ood detection performance. based on the prioritization of classified pseudo-ind samples, we then leverage sampling measures for selecting the most informative instances for annotation. in addition, we demonstrate the flexibility of our aosal approach to multiple distance-based functions @xcite and informative measures @xcite . we perform experiments to validate our aosal framework over four nlp benchmark related to intent classification @xcite , and sentiment analysis @xcite , comparing its performance against several deep al baselines that are based on uncertainty, diversity, and hybrid-based approaches. experimental results suggest that our aosal approach consistently outperforms the baselines on metrics such ind classification accuracy, and percentage of acquired ind/ood samples. the remainder of the paper is organized as follows: section 2 describes related work. in section 3, we detail the aosal methodology. in section 4, we detail the experimental setup and provide the main results against al baselines. an analysis to test the robustness of aosal is presented in section 5. section 6 discusses the limitations of our approach, and finally, section 7 concludes our work. ## methodology in this section, we describe the problem formulation for open-set al and then detail the overview and components of our proposed aosal framework. ## experiments and results in this section, we provide our experimental setup for open-set al on benchmark nlp datasets and provide the main results of our aosal approach against baseline al datasets. ## conclusion in this paper, we presented aosal which is an al framework that aims to improve the efficiency and effectiveness of tods. aosal combines a distance-based ood technique with an adaptive fpr threshold and an informativeness measure based on uncertainty and diversity. this integration enables aosal to improve the classification of ind and ood samples and thus focuses primarily on the most useful ind examples from an unlabeled data pool for training. the experimental analysis we have conducted shows that aosal is highly effective for dealing with complex datasets in comparison to traditional active learning techniques. these real-world applications have demonstrated the practical usefulness and effectiveness of the framework in enhancing not only the robustness but also the accuracy of intent classification in tods by the aosal framework's ability to selectively acquire high-value ind training samples. in future work, one can investigate advanced data augmentation and synthetic data approaches to facilitate training in data-deficient scenarios and design adaptive hyperparameter optimization of the system's responsiveness to data variability.