Item Response Theory for Efficient Human Evaluation of Chatbots (2020)

Abstract: Conversational agent quality is currently assessed using human evaluation, and often requires an exorbitant number of comparisons to achieve statistical significance. In this paper, we introduce Item Response Theory (IRT) for chatbot evaluation, using a paired comparison in which annotators judge which system responds better to the next turn of a conversation. IRT is widely used in educational testing for simultaneously assessing the ability of test takers and the quality of test questions. It is similarly well suited for chatbot evaluation since it allows the assessment of both models and the prompts used to evaluate them. We use IRT to efficiently assess chatbots, and show that different examples from the evaluation set are better suited for comparing high-quality (nearer to human performance) than low-quality systems. Finally, we use IRT to reduce the number of evaluation examples assessed by human annotators while retaining discriminative power.

## introduction one of the main problems in conversation dialog modeling is evaluation. unlike in machine translation and task-driven dialog, automated metrics for non-task driven open-domain generative conversational models (chatbots) seem not to correlate well with human judgments @xcite @xcite . while the creation of new automatic metrics is an extremely active area of research @xcite @xcite @xcite , human annotations are currently the gold standard for assessing model improvements. prior work mainly uses straightforward approaches, such as a two-sided ttest or binomial tests (e.g., @xcite @xcite ) where w sysa i is 1 if annotator i rated system a better and 0 otherwise, and similarly for system b. "-" indicates a tie vote. @xcite ), or pairwise bootstrap test (e.g. @xcite ). these methods do not assess or incorporate the effectiveness of prompts (conversational chunks used for evaluation). given that human evaluation is necessary, it is desirable to discriminate the performance of two different systems with minimal cost. in this paper, we present the use of item response theory (irt) @xcite ) to compare chatbot models using a head-to-head paired experimental (a/b test) design (e.g. irt is a latent variable bayesian model, with relative chatbot model quality (or student ability) being latent variables that probabilistically produce observable responses (one chatbot response to a prompt being judged as better than another, or a student answering a question correctly or wrong). irt is widely used in psychometric studies @xcite , and for paired comparison in psychological studies @xcite . however, it is almost entirely ignored in natural language processing (nlp), with the exception of @xcite ; @xcite ; @xcite ; @xcite ; @xcite . recent work has criticized the statistical methodology used in nlp and called for use of better statistical methods @xcite . here, we present irt as a powerful method for statistical assessment of model improvements. irt not only assesses the relative quality between two systems, but also assesses the usefulness of a prompt in comparing systems. we show that irt can filter and choose a subset of prompts from the evaluation set efficiently, i.e. with little loss in statistical power (figure 2 ), and that irt finds different prompts to be useful for assessing high quality vs. low quality chatbots. ## related work the structure of our chatbot evaluation is a comparison of two chatbots responses to each prompt. this form of head-to-head pairwise block (multiple evaluations shown to one annotator) comparison dates back at least to @xcite . subsequently, the bradley-terry (bt) model has become the most common model for pairwise block comparison experiments @xcite . @xcite describes further extensions and application of the bt model to machine translation. extended bt models can correct for dependent categorical object covariates (correlated examples) as well as subject covariates (annotator ratings) @xcite . as dras (2015) points out, the bt model and irt are similar in formulation, but irt additionally estimates the difficulty of each item using a latent variable bayesian model. fixed effect bt models @xcite or bootstrapping @xcite could be used to compare chatbots, but irt's ability to assess prompts is more attractive for this task where every annotation has a non-trivial cost. an alternative straightforward approach to assess usefulness (validity) of a prompt is item-total correlation (itc; @xcite ). however, itc does not take the student's ability into account. in general, irt is preferred over itc due to the more expressive formulation. itc is mostly used for survey analysis instead of testing. however, as a sanity check, we find that indeed prompts extremely low in discriminative power (according to irt) also have a low item-total correlation. there is surprisingly little work on improving statistical significance testing or prompt selection in chatbot evaluation. while this is less true for machine translation, only two prior works have used irt for model assessment @xcite . our work applies irt in a similar fashion as @xcite , but to chatbot evaluation instead of machine translation system evaluation. we differ from @xcite and @xcite as follows: 1. we do pairwise comparison instead of requiring baselines -this allows for improved prompt selection as models improve. their method is focused on wmt (batch/competition) settings whereas our work focuses on perpetual evaluation. 2. we aggregate annotators -which creates much more stable predictions (their graded mean is 1-baseline, 2tie, 3-win) whereas ours ranges from @xcite . 3. we explicitly assume independence of prompts and account for their correlation and thus do not overstate significance. 4. we use irt to reduce the total number of comparisons; @xcite suggest this for future work. irt has also been applied in nlp for dataset filtering @xcite . @xcite uses irt to efficiently subsample training data based on the difficulty. we differ from @xcite on prompt selection: 1. we select individual prompts based on evaluations using the discriminative ability of the prompt-not just the item difficulty. 2. we use model win rank instead of item difficulty for selecting prompts for "better" models. both of these yield more informative prompts. @xcite use a bayesian approach for testing for significance in interactive evaluation; however, the correlation between items is not taken into account. as in @xcite , irt allows us to directly compare distributions; however, the correlation between the prompts still needs to be accounted for in order not to overstate significance. machine translation much effort has been placed in machine translation for correlating human annotator judgements with automatic metrics; however, @xcite showed that automatic machine translation evaluation methods do not correlate with human judgments of opendomain conversational agents. this may be due to the fact that in machine translation there is a one-to-one semantic equivalence between reference and system output, whereas this is not true in the chatbot setting. nonetheless, relevant prior work on assessing human evaluation in machine translation is relevant to chatbot evaluation. in machine translation, shared tasks offer standard evaluation sets and workshops, which have yielded standardized results @xcite . since 2015, the workshop on machine translation (wmt) uses trueskill @xcite for model ranking. trueskill can also be applied to chatbot evaluation. @xcite used it to efficiently pair machine translation systems and compared them using random subsets of data. they show that their non-parametric method is empirically superior in accuracy to @xcite . however, this comparison is limited since the non-parametric might focus only on one axis of difference similar to stochastic gradient descent. returning to our student analogy, in an example of students taking the sat (an english and a math test), the trueskill method might focus on only the math portion to discriminate between students, whereas, irt would use both portions. trueskill does not select examples using item utility. @xcite and hopkins and may (2013) applied irt to machine translation. irt is more important in chatbot evaluation than in machine translation as human evaluation is rarely reported in machine translation papers (e.g. @xcite ), but is rarely omitted in chatbot comparison (e.g. liu et al. (2016 2020 )). comparison of conversational generative agents using next utterance generation is in many ways similar to the evaluation of machine translation (mt); however, differentiating between chatbot models is uniquely challenging; many more responses than translations are plausible. automated evaluation of mt is vastly better than of chatbots @xcite . the higher costs of human evaluation strongly encourage the use of more powerful statistical models such as irt. ## chatbot evaluation recently researchers tend to evaluate their methodological improvements relative to a sequence-to-sequence (seq2seq) baseline @xcite , as proposed for utterance generation by @xcite ; @xcite ; @xcite as well to compare against each other. while crowd-sourcing experiments are relatively cheap, the lack of automatic metrics means that every change in model architecture requires new evaluations. our goal is efficient and cost-effective model assessment. ideally, chatbots would be interactively evaluated, but due to the high cost, next utterance simulation is used as a surrogate. although next utterance generation is a more artificial task, @xcite observed a pearson correlation of 0.6 between conversation-level and utterance-level ratings. human judgments are often inconsistent for non-task driven chatbots, since there is no clear objective, which leads to low inter-annotator agreement (iaa) @xcite . however, @xcite point out that even with low iaa we can still find statistical significance. there are further tensions between local coherence assessments using standard evaluation sets and human interactive evaluation. these issues are exacerbated for non task-driven dialog systems, as there is rarely a single "correct" response, leading to more local minima. thus, there is a need to obtain the maximum possible statistical power at the minimal possible cost. @xcite found that relative rankings yield more discriminative results than absolute assessments when evaluating natural language generation. recent work of @xcite introduce both human-bot as well as self-chat for interactive evaluation and show that this is more effective than conversation-level likert scales. ## irt for chatbot evaluation we pose chatbot human evaluation as an item response theory (irt) problem, similar to the approach of @xcite . again, throughout this section we consider the analogy of student ∼ a/b chatbot comparison and question ∼ prompt. in the context of educational testing, we are seeking to find the ability of a student and the effectiveness of exam questions (e.g. sat exam) which in our setting is the comparative difference in pairs of chatbots. as seen in we first introduce the graded mean formulation of irt required to estimate the relative assessment of chatbots and the discriminative power of the prompts. subsequently, we describe the exact problem formulation in our setting. ## experimental details while human evaluation remains the gold standard for dialog research, the design of human evaluation experiments is far from standard. we restrict our analysis to designs where the annotator is shown a prompt and two possible responses and 2 if the number of annotators is variable, then we scale u i j to a fixed range which here we set to [-3, 3]. ## results we used irt to compare multiple neural models for their relative strength. furthermore, we also included human baselines in our model comparison. finally, we assessed the discriminative quality of the hand-crafted prompts from @xcite . ## conclusion we present a new method for incorporating irt into chatbot evaluation and show that we can use irt to adaptively and optimally weight prompts from the evaluation sets, eliminating less informative prompts. one of the strengths of our method is that prompt discriminative ability and difficulty are re-estimated as new evaluations are added. one can thus start with a larger evaluation set, such as a subset from the cornell movie database @xcite and continue refining the subset of the evaluation set. we showed that our method is effective with the ncm evaluation set. applying it to the cornell movie database evaluation set of @xcite , we found that we could reduce from 1000 to 150 prompts with negligible loss of accuracy. when evaluating a new model, one would start with a comparison, say against a human baseline on a large set of prompts, then against a similarly ranked model using an appropriate subset of prompts. after each evaluation, the accuracy of all comparisons will increase. irt can also be used to adapt evaluation sets as chatbot models improve in performance, reducing annotation costs. while our main exposition addresses single turn prompts for chatbot evaluation, our irt model comparison method generalizes to many natural language generation tasks, including machine translation and text simplification. it also generalizes to multi-turn prompts, point-wise evaluation, pairwise conversational evaluation (e.g. acute-eval @xcite ), and interactive evaluations such as those of @xcite .