Towards a more Robust Evaluation for Conversational Question Answering (2021)

Abstract: With the explosion of chatbot applications, Conversational Question Answering (CQA) has generated a lot of interest in recent years. Among proposals, reading comprehension models which take advantage of the conversation history (previous QA) seem to answer better than those which only consider the current question. Nevertheless, we note that the CQA evaluation protocol has a major limitation. In particular, models are allowed, at each turn of the conversation, to access the ground truth answers of the previous turns. Not only does this severely prevent their applications in fully autonomous chatbots, it also leads to unsuspected biases in their behavior. In this paper, we highlight this effect and propose new tools for evaluation and training in order to guard against the noted issues. The new results that we bring come to reinforce methods of the current state of the art.

## introduction the ability to automatically answer questions from a set of raw text paragraphs has long been coveted by computer scientists @xcite . for applications in search engines, one could consider an isolated task where a user formulates a single question @xcite . but recently, with usage in conversational agents (e.g. chatbots), a more contextualized variant referred to as conversational question answering (cqa) has attracted a great deal of attention @xcite . cqa differs from traditional (extractive) question answering @xcite because question-answer (qa) pairs are not single but come in sequences within conversations. therefore, models can use previous turns as context to extract the answer of the current question @xcite @xcite . in some cases, the history is even crucial to disambiguate pronouns in the question. similarly to other nlp tasks, the state-of-theart approaches for cqa are variants of the transformer encoder @xcite , a deep neural network with several self-attention layers that produce contextualized representations of the "tokens" (words, subwords) that compose a text. for instance, models like bert @xcite @xcite obtain a more than decent performance on cqa datasets like quac @xcite or coqa @xcite . however, they miss the context to fully understand the questions. proposals have been made to integrate the history in several manners: using a recursive strategy @xcite , appending previous qas to the current question as input @xcite , and contextualizing the questionparagraph pair with respect to the history. we can mention in particular bert-hae and bert-phae @xcite which improve bert in a simple yet efficient way by encoding, in addition to segment and position, the fact that parts of the paragraph's words belonged to previous answers. ## motivation and main contributions our objective here is not to propose yet another model to try to obtain the best predictive score on cqa leaderboards. instead, we focus our thinking around the current evaluation/training protocols with regards to the possible application cases. the starting point of our reflection is that currently, when evaluated on cqa datasets, models like bert-hae use the ground-truth answers of previous turns as context to answer the current question. this limits the scope of applicability to only a "semi-automatic" bot that would require a human providing supervision at each turn. we also show how it biases the selection of models towards those with an undesired filter behavior. to make approaches from the literature usable in more difficult/realistic scenarios like standalone chatbots (in which they can only access the previous questions and their predictions of the answers), we make the following contributions: (1) we implement new evaluation tools to first highlight the current unnoticed and undesirable behavior: in groundtruth free conditions, cqa approaches can become even less accurate than baselines like bert which do not exploit the history at all. (2) we develop the analog training protocol to make approaches robust to the observed issues. in particular, this gives back state-of-the-art models the strength to outperform the baseline but this time in a scenario that connects better to real-world conversational agents. our work comes with an implementation of conversational qa tools, based on the most widely used transformers library @xcite . ## conversational question answering conversational question answering (cqa) is a natural language processing task related to machine comprehension (mc) @xcite . mc has grown significantly over the last decade, particularly thanks to ( 1 ) large scale datasets such as squad @xcite or natural questions @xcite , (2) the improvement of representation learning models @xcite , (3) powerful mechanisms such as attention @xcite , and (4) the emergence of several related topics like multi-lingual modeling @xcite or conversational question answering @xcite . in cqa, questions are grouped in conversations and often require the context, i.e. previous qa turns, to be fully understandable. quac @xcite and coqa @xcite are two examples of cqa datasets. they were both generated by humans (a "student" and a "teacher") through conversations where the student asks a series of questions, complementary or not, on a given paragraph and the teacher answers them. in this paper, we focus on quac (question answering in context) which is more recent and described as more challenging than coqa @xcite . it contains 14k conversations and around 100k question-paragraph pairs, split into a training set (11,567 conversations / 83,568 questions), a validation set (1,000 conversations / 7,354 questions) and a test set. it evaluates models with several metrics, the main one being the f1-score @xcite . models proposed for quac are similar to those developed for squad (e.g. bidaf @xcite or bert @xcite ) but they additionally integrate the history. a popular example is bert-hae @xcite . it uses bert's architecture but modifies the input embedding layer to add a novel component: the history answer embedding (hae). as usual, the input questionparagraph pair is tokenized and marked with positions and segments. then, an additional history answer marker is added to indicate whether the tokens belonged to answers of previous questions or not, and the resulting embedding is simply added to the other embedding vectors (token, position, segment) before the self-attention blocks. bert-hae was enhanced, in a later publication @xcite by bert-phae (positional hae) which additionally encodes the turn position of the answers in the history. although very promising, we note that bert-hae and bert-phae, as well as other state of the art models for quac, access the ground-truth answers of previous turns during evaluation. therefore, reported results only reflect the performance within a reduced scope of applicability. in the following, we detail this limitation and propose to complement the current protocol in order to improve both evaluation and training. ## a more robust protocol consider a standalone chatbot that successively answers questions from documents. at each turn, it cannot know for sure the ground truth (gt) answers of the previous turns except if the user or another human provides supervision. this could happen in scenarios where the role of the algorithm is only to provide answer suggestions (semi-automatic) to a human agent (e.g. in customer support). however, applications often seek bots where the questionanswer loop is automated (standalone). here we investigate this second setting. we start by reproducing the literature results on the semi-automatic scenario, then we exhibit the limits and propose solutions for our target scenario. ## https://github.com/prdwb/bert_hae to get better insights of the impact of a filtering behavior in practice, we run three experiments. experiment 1: the negative impact of filtering due to overlap we first compute the best reachable f1-score (that we refer to as best) as if we had a model that always predicts the expected answer. then we compute "best w/ post filtering" with the same predictions except that we post-filter all tokens that belong the 6 previous turns' answers, except for the "cannot answer" tokens (reserved for unanswerable questions). best f1 score is 95.6 foot_0 while "best f1 w/ post filtering" is lower but very close: 92.7 ( experiment 2: global impact of a post filtering on the models after 6 turns, sometimes almost half of the paragraph tokens belong to the history of answers. even if experiment 1 suggests a negative impact of filtering due to overlap, the positive impact on our baselines (due to the significant reduction of the number of candidate answers) could counterbalance. we therefore re-evaluate the models trained in section 4.1, but this time we apply a post processing of their predictions: the start/end logits of tokens that belong to the answers of previous turns are set to -∞, except for the "cannot answer" tokens. this forces previous answers to be excluded from the final predicted span text. this simple strategy to integrate the history in bert allows an improvement to an f1-score of 56.9 ( ## conclusion the work presented in this paper comes to complement the current training and evaluation protocols for cqa. it allows (1) highlighting unnoticed and undesirable behavior in existing approaches from the literature and ( 2 ) more robustness for their application in autonomous chatbots. we hope that this will encourage additional proposals in the same direction. several improvements could be made in the future. first, because without teacher forcing the history is now predicted and not fixed, we could explore the impact of updating the model by backpropagating an answer's error through all previous turns and not only the current one. this would be analog to backpropagation through time. second, we could augment the current cqa datasets or propose new ones to prevent the biases we observed: for example, quac could have conversations including wrong answers, since this occurs in reallife, so that models could be properly trained for. the associated turns would of course only be used as a part of histories. finally, we should perform user tests to evaluate the robustness of models in real-life, because when a model's answer is wrong, we expect it to impact the next user's question(s). and this cannot be taken into account with the current protocol since the datasets are static.