Interview Evaluation: A Novel Approach for Automatic Evaluation of Conversational Question Answering Models (2023)

Abstract: Conversational Question Answering (CQA) aims to provide natural language answers to users in information-seeking dialogues. Existing CQA benchmarks often evaluate models using pre-collected human-human conversations. However, replacing the model-predicted dialogue history with ground truth compromises the naturalness and sustainability of CQA evaluation. While previous studies proposed using predicted history and rewriting techniques to address unresolved coreferences and incoherencies, this approach renders the question self-contained from the conversation. In this paper, we propose a novel automatic evaluation approach, interview evaluation. Specifically, ChatGPT acts as the interviewer (Q agent) with a set of carefully designed prompts, and the CQA model under test serves as the interviewee (A agent). During the interview evaluation, questions are dynamically generated by the Q agent to guide the A agent in predicting the correct answer through an interactive process. We evaluated four different models on QuAC and two models on CoQA in our experiments. The experiment results demonstrate that our interview evaluation has advantages over previous CQA evaluation approaches, particularly in terms of naturalness and coherence. The source code is made publicly available.

## introduction conversational question answering (cqa) expects machines to answer questions in conversation given the evidence passage. existing cqa datasets, such as quac @xcite , coqa @xcite , and doqa @xcite , are derived from human-human conversations, where the questioner poses questions to gain knowledge on a specific topic, and the answerer responds based on the evidence passage and the conversation history. while significant progress has been made in modeling cqa systems in recent years, the evaluation of cqa tasks has received less attention and remains flawed. the prior cqa evaluation asks model questions in turn based on the held-out human-human conversations due to the high cost of involving humans during the evaluation process. however, this automatic evaluation method utilizes golden answers from the conversation history, disregarding the actual predictions made by the model. it has been noted by @xcite that it deviates from real-world scenarios. they suggest using the models' own predictions for automatic evaluation. on the other hand, @xcite highlights that using predicted history can introduce issues such as unresolved coreference and incoherence and detect invalid questions by employing a coreference resolution model @xcite and subsequently rewrite these questions by substituting incorrect mentions with correct ones. although the evaluation based on prediction combined with a rewriting mechanism mitigates the gap with real-world scenarios and has been proven to align better with human evaluation, the substitution in rewriting makes the conversational question self-contained, which destroys the coherence of the conversation. furthermore, clarification is a significant feature in dialogues @xcite , where questioners often reformulate their questions if they are not satisfied with the previous answers received. building upon this observation, we propose a novel evaluation method called interview evaluation, which takes into account the dynamic nature of conversations. during an interview, interviewers often rely on a set of pre-prepared questions to assess interviewees. however, they also have the flexibility to generate new questions based on the interviewee's answers and provide prompts to guide them toward the correct response. similarly, our interview evaluation for cqa simulates this human interview process. it involves a q agent that acts as the interviewer and an a agent representing the cqa model as the interviewee. this setup allows us to dynamically generate questions and prompts, mimicking the interactive nature of real interviews. specifically, the q agent consists of a selector and a generator. the selector sequentially chooses the next question from the question set, assuming the previous question was answered correctly. otherwise, the generator iteratively generates new questions to prompt the cqa model until a question is marked as successful or unsuccessful. we summarize our contributions as follows: ## analysis and discussion adaptability and flexibility for interview evaluation. one notable characteristic of interview evaluation is its ability to generate prediction-aware questions and can adapt to different evaluated cqa models. to investigate this aspect, we analyze the number of overlapping questions generated by chatgpt across different models on quac. specifically, we focused on prompts originating from the same human question and selectively collected conversation cells that contained these prompts for analysis across all four models. our analysis resulted in a dataset of 408 examples with a conversation cell length of at least 2, and 37 examples with a conversation cell length of at least 3. we considered questions to be overlapping if they were an exact match in terms of their text. to visualize the degree of overlap, we utilized a venn diagram, which is presented in figure 5 . the diagram demonstrates that questions generated by chatgpt for different models exhibit a low degree of overlap, and this degree decreases as the number of prompts increases. in conclusion, the generated questions used in the evaluation process are diverse and effectively reflect the predictions of the evaluated models. this diversity and alignment with model predictions represent a key advantage of the interview evaluation approach. golden history vs. generated history. we conducted a comparison between two modes of interview evaluation: based on golden history and based on generated history. the number of prompts required to successfully answer a question for four cqa models on quac is reported in figure 6 . the results indicate that all models perform better on human-generated questions compared to the new questions generated by chatgpt. it is evident that the questions requiring chatgpt-generated prompts are more challenging for the models, as they have been answered incorrectly before. notably, we observed that ham performs worse than the other three models on turn 0 in the generatedhistory interview evaluation. however, ham per-forms better when answering new questions generated by chatgpt, benefiting from its history attention mechanism. furthermore, we analyze the pfr performance and compare it with the results of human evaluation, as depicted in figure 7 . the results demonstrate that the performance rankings of the golden-history interview evaluation and human evaluation align with each other. however, the performance rankings of the generated-history interview evaluation and human evaluation are not consistent. compared to human evaluation, interview evaluation introduces a q agent that is aware of model predictions and gives prompts. therefore, the generatedhistory interview evaluation expands the dialogue history to make history full of conflict. consequently, the generated interview evaluation scenarios are more intelligent and complex to better expose the limitations of the model such as ham. to gain a better understanding of the two evaluation modes, we analyze their agreement with human evaluation in it is important to note that the high consistency between human evaluation and golden-history interview evaluations does not imply that the interview evaluation based on golden history is superior. the reason for this high consistency is that, both of them utilize the golden history, which means the cqa model does not have to account for its previous performance in the conversation. as we previously analyzed, the interview evaluation based on generated history provides a better assessment of the model's ability to continuously understand the conversation and the interactive ability, which are not adequately captured by existing cqa evaluation methods. ## conclusion this paper presents the concept of interview evaluation as a novel approach for assessing cqa models. our interview evaluation framework employs a question-generation agent (q agent) to dynamically generate questions based on the model's predictions, facilitating a more realistic and interactive evaluation scenario. we introduce three metrics specifically designed for interview evaluation to assess models from different perspectives. through our analysis, we compared two modes of interview evaluation: golden history and generated history. we also examined the performance of these modes with automatic evaluations and human evaluation. however, it is important to acknowledge that the interview evaluation has its limitations. in some cases, it overlooks correct predictions when the model's output partially aligns with the correct answer. besides, it is difficult to evaluate the diversity and creativity of answers provided by the generative model. future research should focus on optimizing the answer-matching module and the question generator of the q agent to handle such cases more effectively.