Consistency Training by Synthetic Question Generation for Conversational Question Answering (2024)

Abstract: Efficiently modeling historical information is a critical component in addressing user queries within a conversational question-answering (QA) context, as historical context plays a vital role in clarifying the user’s questions. However, irrelevant history induces noise in the reasoning process, especially for those questions with a considerable historical context. In our novel model-agnostic approach, referred to as **CoTaH** (**Co**nsistency-**T**rained **a**ugmented **H**istory), we augment the historical information with synthetic questions and subsequently employ consistency training to train a model that utilizes both real and augmented historical data to implicitly make the reasoning robust to irrelevant history. To the best of our knowledge, this is the first instance of research using synthetic question generation as a form of data augmentation to model conversational QA settings. By citing a common modeling error prevalent in previous research, we introduce a new baseline and compare our model’s performance against it, demonstrating an improvement in results, particularly in later turns of the conversation, when dealing with questions that include a large historical context.

## introduction humans often seek data through an informationseeking process in which users engage in multiple interactions with machines to acquire information about a particular concept. a prominent example of this phenomenon is the introduction of chat-gpt @xcite . conversational question-answering (cqa) systems address user questions within the context of information-seeking interactions. in cqa, unlike conventional question answering, questions are interconnected, relying on previous questions and their corresponding answers (history) to be fully understood without ambiguities. @xcite showed that filtering irrelevant history can boost the model's accuracy. how-ever, it utilizes the gold answers of history instead of the predicted ones, like many previous methods. this setting deviates from the real-world scenario, where models have to rely on their own predictions for previous questions to answer the current question. our work aligns with the framework of addressing irrelevant history. however, unlike @xcite , our method abstains from utilizing the gold answers of history. moreover, unlike @xcite , which requires an iterative process to select relevant history, we utilize only one transformer @xcite during prediction, resulting in reduced time and memory. we augment the history of questions in the training set with synthetic questions. our underlying idea is to maintain the model's consistency in its reasoning, whether utilizing the original historical data or the augmented version. baselines like bert-hae @xcite , ham @xcite , and graphflow @xcite leverage the gold answers of history in their modeling. @xcite conducted a re-implementation of @xcite conducted a re-implementation of ham and graphflow using predicted history answers, which resulted in a significant performance decrease. as a result, in this paper, we employ the base transformer of our method as the baseline, as its performance surpasses the re-implementation of the mentioned methods. our method results in a 1.8% upgrade in overall f1 score compared to this baseline, causing a significant improvement in the scores of questions in the later turns (questions with large historical context). furthermore, our method introduces a substantial improvement in detecting unanswerable questions compared to the introduced baseline. ## related works the task of cqa has been introduced to extend question answering to a conversational setting. coqa @xcite and quac @xcite have been proposed as two extractive datasets in the cqa task. bert-hae @xcite employs a manually defined embedding layer to annotate tokens from previous answers within the document, and @xcite extends this approach by introducing an ordering to these annotations. graphflow @xcite utilizes a graph made out of document tokens to tackle the problem. flowqa @xcite utilizes multiple blocks of flow and context integration to facilitate the transfer of information between the context, the question, and the history. excord @xcite uses consistency regularization @xcite to regularize the training by leveraging re-written questions. @xcite introduces the idea of irrelevant history and its effect on degrading performance, proposing a policy network to select the relevant history before reasoning. however, the mentioned models employ the gold answers from history in their modeling. this approach deviates from real-world scenarios, where systems should rely on their previous predictions to answer current questions @xcite . @xcite re-implements bert-hae and ham, and @xcite re-implements ham, graphflow, and ex-cord using the model's predictions, reporting a sharp decrease in performance. flowqa experiences a performance drop from 64.6% to 59.0% on the development set when gold answers in history are not used @xcite . ## problem definition to model a cqa setting, at dialog turn k, a model receives a question (q k ), a document containing the answer (d), and the history of the question (h k ), which is represented as a set of tuples, such as we seek to make the reasoning robust to irrelevant history implicitly by augmenting the dataset. to this end, for question q k , we augment its history by injecting some synthetic questions. let h k be the augmented history. the intuition is that irrespective of whether the reasoning is performed with h k or h k , the result should be the same. in other words: to achieve this goal, we establish a two-stage pipeline. our pipeline consists of a history augmentation module, whose goal is to augment the history and a question-answering module, whose objective is to consistently train a qa network so that the reasoning is consistent. the overall architecture of our model is depicted in figure 1 . ## setup we utilize the quac dataset @xcite , to conduct our experiments on, and data splitting is described in a. we utilize bert @xcite as our base model to conduct experiments following the previous research. for question generation, we adopt bart-large @xcite . following @xcite , we use f1, heq-q, and heq-d as our evaluation metrics. f1 measures the overlap between a gold k and a pred k . heq-q and heq-d are the ratio of questions and dialogs, for which the model performs better than human @xcite . we run multiple experiments to choose the best set of hyperparameters, resulting in setting @xmath0, λ = 2.0, and τ = 6. in appendix c, the process of choosing all hyperparameters and their analysis is described. for all of our models, we concatenate the question with history questions, feeding them to the network. more details on reproducibility are presented in appendix e. ## conclusions in this paper, we introduced a novel model-agnostic method to make the reasoning of conversational question-answering models robust to irrelevant history. we coped with this issue by augmenting the history and training the model with consistency training. in our experiments, we didn't follow the wrong modeling of past research in using the gold answers of history. we examined our method with bert which exhibited a 1.8% performance boost compared to the baseline model. it was demonstrated that this improvement is primarily attributed to the enhancement of the model's performance on questions with a substantial historical context, suggesting that our method has been successful in making the reasoning robust for these questions.