Developing Classifiers for Affirmative and Negative User Responses with Limited Target Domain Data for Dialogue System Development Tools (2025)

Abstract: We aim to develop a library for classifying affirmative and negative user responses, intended for integration into a dialogue system development toolkit. Such a library is expected to highly perform even with minimal annotated target domain data, addressing the practical challenge of preparing large datasets for each target domain. This short paper compares several approaches under conditions where little or no annotated data is available in the target domain. One approach involves fine-tuning a pre-trained BERT model, while the other utilizes a GPT API for zero-shot or few-shot learning. Since these approaches differ in execution speed, development effort, and execution costs, in addition to performance, the results serve as a basis for discussing an appropriate configuration suited to specific requirements. Additionally, we have released the training data and the fine-tuned BERT model for Japanese affirmative/negative classification.

## introduction in dialogue systems, classifying whether a user's response to a system's question is affirmative or negative is a crucial and fundamental task. one reason for this is that the dialogue flow needs to be switched on the basis of the classification result (figure 1 ). although conducting dialogues using large language models (llms) has recently become possible, current systems may still struggle with progressing dialogues as intended by the system developers. the classification is also essential for deciding whether the system should retain the information, such as user preferences or factual knowledge, included in the system's question (figure 2 ). users do not always respond to yes/no questions with simple expressions such as 'yes' or 'that's right.' these types of responses are known as indirect answers @xcite . furthermore, have you ever been to kyoto? yes/no question (affirmative or negative) what places did you visit? where do you want to go in kyoto? if the response is affirmative: if the response is negative: do you want talk about history? yes/no question in which century was kyoto thriving the most? affirmative negative figure 2 : example of acquiring user preferences on the basis of affirmative/negative classification. the user utterance shown in figure 2 , for example, is generally a question, but in a specific domain, it should be regarded as affirmative because it demonstrates interest in the topic. such domain-specific classification also needs to be considered. therefore, simple rule-based classification has its limitations, and utilizing machine learning, including llms, is a promising approach. however, machine learning-based methods require training data. collecting sufficient dialogue data and annotating it with correct labels is not practical for dialogue system developers who combine ready-for-use modules in toolkits. they prefer to minimize costs while still achieving a highperformance classifier. we present experimental results on developing the affirmative/negative classifier when little or no annotated data is available in the target domain. the goal of this work is not to pursue higher performance but to discuss which configuration would be most appropriate for use in dialogue system development toolkits. we assume a situation where only a few dozen examples can be prepared by developers and used as target domain data. several methods can be employed: fine-tuning a pretrained bert model @xcite with data from a different domain and the small amount of target domain data, and using a gpt api with fewshot learning by providing the target domain data as few-shot samples. pre-trained models have become useful for several tasks, such as in extracting entity-value pairs for state tracking @xcite . in our experiments, we used dialogue data collected from real users during the finals of the dialogue robot competition 2023 @xcite as the test set. we incrementally added target domain data for fine-tuning or as few-shot samples and evaluated the classification performance. on the basis of the results, we discuss an appropriate configuration of the classifier, considering not only performance but also execution speed, development effort, and execution cost. the contributions of this paper are as follows. several methods have been developed for classifying whether user responses are affirmative or negative. asao et al. (2020) implemented a classifier using bert, and watanabe et al. ( 2023 ) developed the models using bert and gpt. such studies assumed large amounts of annotated data in the target domain. several corpora containing indirect answers have also been collected and made publicly available @xcite @xcite . in contrast, our goal is to provide an easily accessible classifier that can be integrated into dialogue system development toolkits, such as rasa open source @xcite and dialbb @xcite . this paper shares the results of approaches that aimed at reducing the required effort by eliminating the need to collect and annotate large datasets. classifying whether an utterance is affirmative or negative can be considered a part of dialogue act classification @xcite , which has been addressed in various studies @xcite @xcite . however, there are cases where an affirmative/negative classification cannot be made solely on the basis of the dialogue act. often, there are expressions specific to the target domain, as shown in the example in figure 2 , and to situations where the preceding system utterance is a yes/no question. phenomena such as indirect answers @xcite have been theoretically examined from a linguistic perspective. @xcite provided a taxonomy of responses to questions, while @xcite analyzed responses to polar questions across 14 languages. studies on dialogue management have also considered such responses @xcite . ## experiment several methods were compared with minimal or no use of target domain data. the less annotated data required, the lower the cost for developers to build dialogue systems. ## error analysis and discussion as shown in figure 3 , particularly while less target domain data were available, (b2) performed better than (b1). this was likely due to incorporating general patterns of affirmation and negation into the model through fine-tuning with the hazumi data. (l1) performed even better with less data, presumably due to the extensive training data and the gpt model structure. we examined the incorrect classification results in (b0) to investigate why the absence of target domain data results in low accuracy. a common pattern involved users responding to a system's questions with questions, as shown in figure 4 . although users implicitly responded without 'yes' or 'no,' it appears that the different domain data, hazumi, had too few instances of such patterns. these patterns also frequently resulted in errors in (l0). more detailed analysis with examples is provided in appendix d. various surrounding factors in system development and operational circumstances @xcite should be considered when implementing and operating practical dialogue systems. there are several differences between pre-trained bert models and gpt apis beyond just accuracy, such as execution speed, development effort, and execution costs. pre-trained bert models are available for free, whereas gpt apis require usage fees, making budget considerations necessary, and also require network connections, which can sometimes cause response delays. on the other hand, fine-tuning bert can be time-consuming, and the resulting models are large, requiring considerable disk space and memory. in contrast, gpt apis do not necessarily require training and impose less burden on the local machine. therefore, the best approach should be chosen on the basis of the specific conditions. ## concluding remarks this paper presented an approach to developing an affirmative/negative response classifier using a small amount of target domain data. we have released the annotations for affirmative/negative classification on the hazumi datasets, along with the classifier based on the bert model, fine-tuned with this data. the experimental results may be specific to the particular dataset used. the performance depends on the model used, as well as the content of the data used for fine-tuning and few-shot learning. nevertheless, we believe that the results and discussion could serve as a useful reference for developing dialogue systems, especially when working with little or no target domain data. future work includes conducting experiments using other datasets and models, and extending our approach to languages other than japanese. additionally, we will not only focus on binary affirmative/negative classification but also address the classification of agreement and disagreement (gokcen and de marneffe, 2015), which is crucial for dialogue management and knowledge acquisition @xcite .