Ditch the Gold Standard: Re-evaluating Conversational Question Answering (2022)

Abstract: Conversational question answering aims to provide natural-language answers to users in information-seeking conversations. Existing conversational QA benchmarks compare models with pre-collected human-human conversations, using ground-truth answers provided in conversational history. It remains unclear whether we can rely on this static evaluation for model development and whether current systems can well generalize to real-world human-machine conversations. In this work, we conduct the first large-scale human evaluation of state-of-the-art conversational QA systems, where human evaluators converse with models and judge the correctness of their answers. We find that the distribution of human machine conversations differs drastically from that of human-human conversations, and there is a disagreement between human and gold-history evaluation in terms of model ranking. We further investigate how to improve automatic evaluations, and propose a question rewriting mechanism based on predicted history, which better correlates with human judgments. Finally, we analyze the impact of various modeling strategies and discuss future directions towards building better conversational question answering systems.

## introduction conversational question answering aims to build machines to answer questions in conversations and has the promise to revolutionize the way humans interact with machines for information seeking. with recent development of large-scale datasets @xcite @xcite , rapid progress has been made in better modeling of conversational qa systems. current conversational qa datasets are collected by crowdsourcing human-human conversations, where the questioner asks questions about a specific topic, and the answerer provides answers based on an evidence passage and the conversational history. when evaluating conversational qa systems, a set of held-out conversations are used for asking models questions in turn. since the evaluation builds on pre-collected conversations, the gold history of the conversation is always provided, regardless of models' actual predictions (figure 1(b) ). although current systems achieve near-human f1 scores on this static evaluation, it is questionable whether this can faithfully reflect models' true performance in real-world applications. to what extent do humanmachine conversations deviate from human-human conversations? what will happen if models have no access to ground-truth answers in a conversation? to answer these questions and better understand the performance of conversational qa systems, we carry out the first large-scale human evaluation with four state-of-the-art models on the quac dataset @xcite by having human evaluators converse with the models and judge the correctness of their answers. we collected 1,446 human-machine conversations in total, with 15,059 question-answer pairs. through careful analysis, we notice a significant distribution shift from human-human conversations and identify a clear inconsistency of model performance between current evaluation protocol and human judgements. this finding motivates us to improve automatic evaluation such that it is better aligned with human evaluation. @xcite ; @xcite identify a similar issue in gold-history evaluation and propose to use models' own predictions for automatic evaluation. however, predictedhistory evaluation poses another challenge: since all the questions have been collected beforehand, using predicted history will invalidate some of the questions because of changes in the conversational history (see figure 1 (c) for an example). ## disagreements between human and gold-history evaluation we now compare the results from our human evaluation and gold-history (automatic) evaluation. note that the two sets of numbers are not directly comparable: (1) the human evaluation reports accuracy, while the automatic evaluation reports f1 scores; figure 2 shows different trends between human evaluation and gold-history evaluation (auto-gold). current standard evaluation cannot reflect model performance in human-machine conversations: (1) human evaluation and auto-gold rank bert and graphflow differently; especially, graphflow performs much better in automatic evaluation, but worse in human evaluation. (2) the gap between ham and excord is significant (f1 of 65.4% vs 67.7%) in the automatic evaluation but the two models perform similarly in human evaluation (accuracy of 87.8% vs 87.9%). ## strategies for automatic evaluation the inconsistency between human evaluation and gold-history evaluation suggests that we need better ways to evaluate and develop our conversational qa models. when being deployed in realistic scenarios, the models would never have access to the ground truth (gold answers) in previous turns and are only exposed to the conversational history and the passage. intuitively, we can simply replace gold answers by the predicted answers of models and we name this as predicted-history evaluation (auto-pred). formally, the model makes predictions based on the questions and its own answers: this evaluation has been suggested by several recent works @xcite , which reported a significant performance drop using predicted history. we observe the same performance degradation, shown in may become unnatural or invalid when the history is changed to (q * 1 , a 1 , ..., q * i-1 , a i-1 ). ## automatic vs human evaluation in this section, we compare human evaluation with all the automatic evaluations we have introduced: gold-history (auto-gold), predicted-history (auto-pred), and our proposed auto-rewrite and auto-replace evaluations. we first explain the metrics we use in the comparison ( §6.1) and then discuss the findings ( §6.2 and §6.3). ## towards better conversational qa with insights drawn from human evaluation and comparison with automatic evaluations, we discuss the impact of different modeling strategies, as well as future directions towards building better conversational question answering systems. ## conclusion in this work, we carry out the first large-scale human evaluation on conversational qa systems. we show that current standard automatic evaluation with gold history cannot reflect models' performance in human evaluation, and that humanmachine conversations have a large distribution shift from static conversational qa datasets of human-human conversations. to tackle these problems, we propose to use predicted history with rewriting invalid questions for evaluation, which reduces the gap between automatic evaluations and real-world human evaluation. based on the insights from the human evaluation results, we also nalyze current conversational qa systems and identify promising directions for future development.