Investigating person-specific errors in chat-oriented dialogue systems (2022)

Abstract: Creating chatbots to behave like real people is important in terms of believability. Errors in general chatbots and chatbots that follow a rough persona have been studied, but those in chatbots that behave like real people have not been thoroughly investigated. We collected a large amount of user interactions of a generation-based chatbot trained from large-scale dialogue data of a specific character, i.e., target person, and analyzed errors related to that person. We found that person-specific errors can be divided into two types: errors in attributes and those in relations, each of which can be divided into two levels: self and other. The correspondence with an existing taxonomy of errors was also investigated, and person-specific errors that should be addressed in the future were clarified.

## introduction creating chatbots to behave like real people is important in terms of believability @xcite . errors in general chatbots @xcite and chatbots that follow a rough persona @xcite @xcite @xcite have been studied, but those in chatbots that behave like real people have not been thoroughly investigated. we analyzed dialogue data between a chatbot that imitates a certain person and users to identify "errors related to the target person" (hereafter referred to as person-specific errors). we collected a large amount of dialogue data between users and the latest generation-based chatbot trained with a large amount of dialogue data of the target person and analyzed the errors. the results indicate that person-specific errors can be divided into two types: errors in attributes and those in relations, each of which can be divided into two levels: self * work carried out during internship at ntt. and other. the correspondence with the existing taxonomy of errors was also investigated, and errors that should be addressed in the future were clarified. ## dialogue data collection we used a chatbot that imitates a specific person. by making the chatbot available to the public, we collected dialogue data from a large number of users. ## error analysis to extract system utterances causing personspecific errors from the data, we collected four types of information: dialogue breakdown labels, comments on the reasons for the breakdown @xcite , flags indicating whether the comments were about the person in question, and error types in chat-oriented dialogue systems @xcite . we first collected the dialogue breakdown labels and comments on their reasons. if the comments contained keywords related to kurisu, we considered the system utterances with those comments as indicating person-specific errors and extracted the comments for analysis. we also annotated system utterances with the error types in chat-oriented dialogue systems for investigating the correspondence between the existing taxonomy of errors and person-specific errors. ## person-specific error analysis we automatically clustered the error comments related to the target person and investigated the characteristics the person-specific errors. ## summary and future work we analyzed dialogue data between a chatbot that imitates a specific person and users to identify person-specific errors that have not been considered thoroughly before. we found that personspecific errors can be divided into four types: self-recognition error, self-relation error, otherrecognition error, and other-relation error, which are useful as a guideline for constructing chatbots that are based on specific people. future work includes the application of unlikelihood training @xcite or a classifier to estimate the identity of a speaker @xcite for suppressing person-specific errors. we focused on one specific person in this paper; thus, it will also be important to consider the generality of the results.