Generating Extractive Answers: Gated Recurrent Memory Reader for Conversational Question Answering (2023)

Abstract: Conversational question answering (CQA) is a more complicated task than traditional single-turn machine reading comprehension (MRC). Different from large language models (LLMs) like ChatGPT, the models of CQA need to extract answers from given contents to answer follow-up questions according to conversation history. In this paper, we propose a novel architecture, i.e., Gated Recurrent Memory Reader (GRMR), which integrates traditional extractive MRC models into a generalized sequence-to-sequence framework. After the passage is encoded, the decoder will generate the extractive answers turn by turn. Different from previous models that concatenate the previous questions and answers as context superficially and redundantly, our model can use less storage space and consider historical memory deeply and selectively. Experiments on the Conversational Question Answering (CoQA) dataset show that our model achieves comparable results to most models with the least space occupancy.

## introduction recently, large language models (llms) like @xcite and @xcite have revolutionized the question-answering and conversation domains, pushing them to new heights. different from chatgpt-style conversations, the task of conversational question answering (cqa) requires models to answer follow-up questions using extracted answers based on given passages and conversation history. it can be regarded as an expansion of traditional singleturn machine reading comprehension (mrc) to multi-turn conversations. however, the followup questions usually have more complicated phenomena, such as co-reference, ellipsis and so on. it is necessary to consider historical memory in a conversation. to enable machines to answer such questions, many cqa datasets, such as coqa passage: my doorbell rings. on the step, i find the elderly chinese lady, small and slight, holding the hand of a little boy. in her other hand, she holds a paper carrier bag …… who is at the door? the elderly chinese lady is she carrying something? yes a paper carrier bag … … figure 1 : an example from coqa. @xcite , quac @xcite and qblink @xcite , are proposed. generally, a dialogue contains a long passage and some short questions about this passage. the current question may rely on previous questions or answers. here is an example from coqa dataset in figure 1 . only if we know the conversation history can we understand that the third question "what?" besides, these methods occupy a lot of storage space or precious and limited graphics memory because of duplicated questions and passages. although, flowqa @xcite proposes a flow mechanism without concatenating previous questions, the hidden states of the passage are still duplicated many times to obtain the question-aware passage for each question. meanwhile, it can not utilize dialogue history selectively. to address these issues, we propose our new architecture, i.e., gated recurrent memory reader (grmr). it integrates traditional mrc models into a generalized sequence-to-squence (seq2seq) @xcite framework to generate extractive answers. for one thing, the deeper representation of historical conversation can be utilized selectively by our recurrent structure and gated mechanism. for another thing, our model uses original questions and passages without redundancy, which saves lots of storage and memory. experiments on the conversational question answering (coqa) dataset show that our model achieves comparable results to most models with the least space. ## related work in the field of generative conversations, large language models have become the prevailing approach. these models (openai, 2022, 2023; zhang and yang, 2023b) are typically pretrained using vast amounts of unsupervised text data and subsequently fine-tuned using supervised instruction data. this supervised instruction data is often obtained through human annotations or distilled from existing large-scale models (zhang and yang, 2023a). in the field of extractive conversations, the answers are typically extracted directly from the original passages by the model. this task is often seen as an extension of single-turn machine reading comprehension to multi-turn conversations. prior to the era of pretraining, attention mechanisms were commonly employed in various subdomains of question answering, such as bidaf @xcite and rception (zhang and wang, 2020) in classic single-turn machine reading comprehension, bidaf++ (yatskar, 2018) in conversational machine reading comprehension, and other methods based on multi-modal or structured knowledge question answering @xcite . subsequently, pre-trained models based on transformer @xcite like elmo @xcite or bert @xcite are employed in a wide range of natural language processing tasks @xcite . in question answering tasks, these pre-trained models are either combined with designed attention structures through embeddingstyle @xcite or directly fine-tuned using concatenated historical dialogues. however, regardless of the method used, the previous dialogue history needs to be encoded and interacted with in each turn of the conversation. it is not possible to save the previous dialogue states and directly use them. ## conclusion we propose a novel structure, grmr, for conversational question answering. traditional extractive mrc models are integrated into a generalized sequence-to-sequence framework. gated mechanism and recurrent memory enable the model to consider the latent semantics of conversation history selectively and deeply with less space. the experiments show that this is a successful attempt to integrate extraction and generation in conversational question answering.