Intent-calibrated Self-training for Answer Selection in Open-domain Dialogues (2023)

Abstract: Answer selection in open-domain dialogues aims to select an accurate answer from candidates. The recent success of answer selection models hinges on training with large amounts of labeled data. However, collecting large-scale labeled data is labor-intensive and time-consuming. In this paper, we introduce the predicted intent labels to calibrate answer labels in a self-training paradigm. Specifically, we propose intent-calibrated self-training (ICAST) to improve the quality of pseudo answer labels through the intent-calibrated answer selection paradigm, in which we employ pseudo intent labels to help improve pseudo answer labels. We carry out extensive experiments on two benchmark datasets with open-domain dialogues. The experimental results show that ICAST outperforms baselines consistently with 1%, 5%, and 10% labeled data. Specifically, it improves 2.06% and 1.00% of F1 score on the two datasets, compared with the strongest baseline with only 5% labeled data.

## introduction open-domain dialogue systems (odss) interact with users by dialogues in open-ended domains @xcite . the responses in ods can be divided into different types, such as answer, gratitude, greeting, and junk @xcite . in this paper, we focus on the selection of answers, which aims to identify the correct answer from a pool of candidates given a dialogue context. typically, there are two main branches of approaches to produce answers, i.e., generation-based methods and selection-based methods @xcite . the former generate a response token by token; and the latter select a response from a pool of candidates. currently, pure generation methods such as chat-gpt still face challenges: (1) they may generate incorrect content. (2) they cannot generate timely answers. thus, it still needs selection-based methods to improve the correctness and timeliness of generation-based method. figure 1 illustrates our idea by comparing the answer selection paradigms of (a) context-aware methods, (b) intent-awaremethods, and (c) intentcalibrated methods. context-aware methods (see figure 1(a) ) capture the context of the ongoing dialogue for understanding users' information needs to select the most relevant responses from answer candidates @xcite . unlike task-oriented dialogue systems, it is much more challenging for odss to infer users' information needs due to their open-ended goals @xcite . to this end, user intents, i.e., a taxonomy of utterances, are introduced to guide the informationseeking process @xcite @xcite . if the intent of the previous (oq) is not satisfied by the (pa) provided by a system, then the users' next intent is more likely to be information request (ir). for example, if the user asks: ''can you send me a website, so i can read more information?'', the user's intent is ir. if the system does not consider the intent label ir, then it may provide an answer which does not satisfy the user's request. intent-aware methods (see figure 1(b) ) adopt intents as an extra input to better understand users' information needs in an utterance @xcite . however, they require sufficient human-annotated intent labels for training, the construction of which is time-consuming and labor-intensive. self-training has been widely used to mitigate label scarcity problem @xcite @xcite . but it is still underexplored for answer selection in odss. the principle of self-training is to iteratively learn a model by assigning pseudo-labels for large-scaled unlabeled data to extend the training set @xcite . the teacher-student self-training framework has been widely used in much recent work, where the teacher generates pseudo-labels and the student makes predictions @xcite @xcite . however, noisy pseudo labels incur error propagation across iterations, so the key challenge is how to assure both quality and quantity of pseudo labels @xcite . in this paper, we introduce an intent-calibrated answer selection paradigm, as in figure 1(c ). it first conducts both context-aware and intentaware answer selection to predict pseudo intent and answer labels, and then it selects high-quality intent labels to calibrate final answer labels. to be more specific, we develop an intent-calibrated self-training (icast) algorithm based on the teacher-student self-training and intent-calibrated answer selection paradigm. the core procedure is: first, we train a teacher model on the labeled data and predict pseudo intent labels for the unlabeled data. second, we select high-quality intent labels by estimating intent confidence gain and then add selected intents to the input of the answer selection model. the intent confidence gain measures how much information a candidate intent label can bring to the model. third, we re-train a student model on both the labeled and pseudo-labeled data. intuitively, icast synthesizes pseudo intent and answer labels and integrates them into teacher-student selftraining, which can assure synthetic answer quality by high-quality intents. we conduct experiments on two datasets: ms-dialog foot_0 @xcite and mantis foot_1 @xcite . the experimental results show that icast outperforms the state-of-the-art baseline by 2.51%/0.63% of f1 score on the msdia-log/mantis dataset, with 1% labeled data. the results demonstrate the effectiveness of icast which selects accurate answers with incorporating high-quality predicted intent labels. ## related work in this section, we summarize related work in terms of three categories, i.e., traditional answer selection models, intent-aware answer selection models, and self-training for data argumentation. ## reproducibility to facilitate reproducibility of the results reported in this paper, the code and data used are available at https://github.com/dengwentao99 /icast .