Auto- ACE : An Automatic Answer Correctness Evaluation Method for Conversational Question Answering (2024)

Abstract: Conversational question answering aims to respond to questions based on relevant contexts and previous question-answer history. Existing studies typically use ground-truth answers in history, leading to the inconsistency between the training and inference phases. However, in real-world scenarios, progress in question answering can only be made using predicted answers. Since not all predicted answers are correct, indiscriminately using all predicted answers for training introduces noise into the model. To tackle these challenges, we propose an automatic answer correctness evaluation method named **Auto-ACE**. Specifically, we first construct an Att-BERT model which employs attention weight to the BERT model, so as to bridge the relation between the current question and the question-answer pair in history. Furthermore, to reduce the interference of the irrelevant information in the predicted answer, A-Scorer, an answer scorer is designed to evaluate the confidence of the predicted answer. We conduct a series of experiments on QuAC and CoQA datasets, and the results demonstrate the effectiveness and practicality of our proposed Auto-ACE framework.

## introduction conversational question answering (convqa) involves responding to a sequence of questions within a conversation, while considering the relevant context provided @xcite @xcite . different from the traditional extractive question-answering tasks which conduct one-turn dialog, as shown in figure 1 , convqa is expected to resolve such implicit information from the conversational history in a multi-turn way. with the rise of virtual assistants and chatbots, convqa has recently garnered increased interest. hence, numerous works have been conducted for further study. @xcite proposed a conversational question answering system specifically designed for the search-oriented conversational ai (scai) shared task, and provided a detailed analysis of its question rewriting module. @xcite introduced a positional history answer embedding method to encode conversation history with positional information using bert @xcite . they also designed a history attention mechanism (ham) for each question-answer pair and utilized multi-task learning to predict the final answer. nevertheless, despite their successes, these works on convqa rely on the ground-truth answer, overlooking the fact that real-world progress can only be achieved using predicted answers. existing researchers found a way to tackle this limitation by using the predicted label @xcite . this method can partially trade off the balance between training and inference. however, if the predicted answer is incorrect, it will introduce noisy samples into the model, thereby affecting performance. for example, as shown in figure 1 , {q 2 , a 2 , q 1 , a 1 } are input as the conversation history of q 3 into the question answering (qa) model to perform inference for a 3 . figure 1( @xcite indicates that ground-truth answers are used for inference, which significantly differs from the real-world inference scenarios. figure 1(b) indicates the use of all predicted answers for inference. although this way is more practical, it introduces noise into the model to some extent when the predicted answers are incorrect. therefore, we propose an answer scorer model that can automatically assign attention weights to predicted answers and incorporate them into the qa model's inference phase. the most similar work to ours is the work of @xcite , which requires an initial round of training and prediction to obtain the predicted answers along with their confidences and uncertainties before the official training. this additional training and prediction step increases the overall training time and computational resource consumption. in this paper, we propose an automatic answer correctness evaluation method named auto-ace, which comprises an att-bert and an a-scorer method to maximize the use of effective information from the predicted answers. to be more specific, we first construct an att-bert model which employs attention weight to the bert model, so as to bridge the relation between the current question and the question-answer pair in history. furthermore, to reduce the disturbance of the irrelevant content in the predicted answer, a-scorer, an answer scorer is designed to evaluate the confidence of the predicted answer. during the training phase, att-bert and a-scorer are trained, while in the inference period, a-scorer evaluates each questionanswering pair in the history to obtain the correctness of the predicted answer. numerous experiments on quac and coqa datasets demonstrate the effectiveness and practicality of our proposed auto-ace framework foot_0 . the main contributions of our work can be summarized as follows: ## methodology this section begins with a concise introduction to the convqa task, then we describe how the proposed auto-ace framework can bridge the gap between training and real-world inference scenarios by incorporating predicted answers into model training, as demonstrated in figure 2 . in addition, we discuss the calculation of attention weights and the overall training pipeline. ## conclusion in this paper, we introduce an automatic answer correctness evaluation method named auto-ace for convqa task, which can balance the inconsistency between training and inference. the proposed auto-ace method consists of two primary components including att-bert and a-scorer. the att-bert effectively bridges the current question with historical q&a pairs using attention mechanisms, enabling the model to focus on more relevant content. furthermore, the a-scorer is designed to evaluate the confidence of predicted answers and is applied to the att-bert as the confidencebased attention. experiments conducted on quac and coqa datasets demonstrate that our proposed auto-ace method significantly improves the performance and reliability of other baseline models.