Applying Item Response Theory to Task-oriented Dialogue Systems for Accurately Determining User’s Task Success Ability (2023)

Abstract: While task-oriented dialogue systems have improved, not all users can fully accomplish their tasks. Users with limited knowledge about the system may experience dialogue breakdowns or fail to achieve their tasks because they do not know how to interact with the system. For addressing this issue, it would be desirable to construct a system that can estimate the user’s task success ability and adapt to that ability. In this study, we propose a method that estimates this ability by applying item response theory (IRT), commonly used in education for estimating examinee abilities, to task-oriented dialogue systems. Through experiments predicting the probability of a correct answer to each slot by using the estimated task success ability, we found that the proposed method significantly outperformed baselines.

## introduction although task-oriented dialogue systems have improved, not all users can accomplish their tasks @xcite . even in dialogue systems built using large language models such as ope-nai's chatgpt 1 , the system's performance is not always satisfactory @xcite . in particular, users with limited knowledge about the system may experience dialogue breakdowns or fail to achieve their tasks because they do not know how to communicate with the system. one solution would be for the system to estimate the user's task success ability and then engage in dialogue in accordance with that ability, for example, by changing the expressions in utterances or interaction strategies. we therefore propose a method (shown in figure 1 ) that estimates the user's task success ability by utilizing item response theory (irt) @xcite , which is commonly used in the field of education. specifically, we first collect dialogues between the system and users, where each user is presented with a unique dialogue goal and must engage in dialogue on the basis of that goal. next, considering correctly filling in each designated slot as a problem, we estimate the item characteristics of the slots by using irt. finally, we let a new user engage in a dialogue on the basis of a given dialogue goal, and the user's task success ability is estimated by using the item characteristics of the filled or unfilled slots. our experimental results showed that the proposed method significantly outperformed the baselines in accurately predicting the probabilities of correct answers to slots. in addition, our analysis of the item characteristics of slots in the multiwoz dataset @xcite gave further insights into how the dialogue goals should be determined for predicting task success abilities. the contributions of this paper are as follows. ## item response theory we first explain item response theory (irt), which is a measurement theory that quantifies examinees' abilities on tests @xcite . in traditional tests, the total score of the correctly answered questions represents the examinee's score. however, in such tests, it is necessary to predetermine the score of each problem, but the predetermined scores may not always represent the examinees' ability. in tests that utilize irt, the relationship between the examinee's abilities θ and the probabilities of correct answers to questions prob is calculated for each question using a large amount of user response data. the relationship is described by item characteristics such as discrimination a, difficulty b, and guessing c, as shown in the following equation. discrimination represents the extent to which a question distinguishes between examinees of different abilities. difficulty indicates an item's difficulty level. guessing represents the probability of a chance guess resulting in a correct response for an examinee with no ability. in multiple-choice questions, the reciprocal number of choices can be used to estimate the guessing parameter. on the basis of the item characteristics, the ability at which the examinee's response patterns are most likely to occur is estimated. to illustrate the effect of item characteristics, figure 2 provides examples of item characteristic curves that represent the characteristics of each particular question, where the horizontal axis of each curve represents the examinee's ability value θ, and the vertical axis represents the probability prob of a correct answer to the item. generally, the item characteristic curve shows that the probability of a correct answer is small when the ability is small, increases around the medium ability value, and reaches a high probability for large ability values. it forms an upward-sloping curve. in the figure, items a and b differ only in their discrimination parameters, items a and c differ only in their difficulty parameters, and items a and d differ only in their guessing parameters. ## proposed method in our method, we first collect dialogues between the system and users. next, we calculate the correctness of each slot by comparing the dialogue goal and the belief state at the end of the dialogue. we use irt to estimate item characteristics (difficulty, discrimination, and guessing for each slot) by means of marginal maximum likelihood estimation @xcite . here, marginal maximum likelihood estimation is a method that estimates only the item characteristics (users' abilities are not estimated) by assuming a standard normal distribution as the distribution of the users' ability. it is known to provide stable results even with an increased number of users. in task-oriented dialogue systems, the dialogue goal includes the content of the slots that the user should convey to the system (inform goals) and the slots that the user should ask about (request goals). we regard each dialogue as a single test and consider whether each slot is filled in correctly as the problem of irt. for an inform goal slot, it is considered correct if the user can appropriately convey their slot values to the system. let v and b[d][s] denote the value of the goal and the belief state at the end of the dialogue for a domain d and slot s. the correctness @xmath0, 1} is defined as follows. for a request goal slot, it is considered correct if the user can appropriately obtain the information from the system. let s and s[d] denote the slot of the goal and the set of slots of the domain d for which the system has conveyed information in the dialogue. the correctness @xmath1, 1} is defined as follows. having estimated the item characteristics of slots, we finally let the user whose task success ability we want to estimate engage in a dialogue for a given dialogue goal, judge whether each slot is correctly filled, and estimate the task success ability by expected a posteriori estimation based on bayesian statistics @xcite . we can calculate the probabilities of correct answers to the slots by using equation (1) with the estimated task success ability and item characteristics. ## experiment we collected dialogue data and estimated users' task success abilities using irt. we then evaluated the accuracy of estimating the probabilities of correct answers to slots utilizing the users' estimated task success abilities. assuming that the capability to fill slots correctly corresponds to the ability to complete dialogue tasks, if the proposed method can accurately estimate the probability of a correct answer to each slot, we can say that the method can accurately estimate the user's task success ability. we also analyzed the estimated item characteristics. ## conclusion and future work we proposed a method for estimating users' task success abilities with task-oriented dialogue systems utilizing item response theory. experiments on predicting the probability of a correct answer for each slot showed that the proposed method significantly outperformed the baselines. various challenges need to be addressed in future work, such as the dependence among slots; to this end, we want to explore methods that consider multiple slots as a single problem. we also want to estimate the task success ability using deep learningbased irt methods that may achieve higher accuracy @xcite . additionally, we aim to investigate methods for estimating task success abilities more quickly, that is, using less than a single dialogue. finally, we want to construct dialogue systems that can adapt their behavior on the basis of the users' estimated task success abilities. speaker utterance ... user is there a train leaving after 18:15 from broxbourne to cambridge on sunday? system we can book you for the train leaving at 18:15. would you like to take the train on sunday? would you like me to book a train to cambridge for you? user i need tickets for two. ... (a) dialogue with estimated task success ability of 0.195 speaker utterance ... user i want train service system when would you like to leave by? is there a time you would like to get there by? on what day will you be traveling? user 11.30 system sorry i do not understand, can you paraphrase? ... (b) dialogue with estimated task success ability of -1.116 figure 5: example dialogues from pipeline system with estimated task success abilities.