Combining Impression Feature Representation for Multi-turn Conversational Question Answering (2020)

Abstract: Multi-turn conversational Question Answering (ConvQA) is a practical task that requires the understanding of conversation history, such as previous QA pairs, the passage context, and current question. It can be applied to a variety of scenarios with human-machine dialogue. The major challenge of this task is to require the model to consider the relevant conversation history while understanding the passage. Existing methods usually simply prepend the history to the current question, or use the complicated mechanism to model the history. This article proposes an impression feature, which use the word-level inter attention mechanism to learn multi-oriented information from conversation history to the input sequence, including attention from history tokens to each token of the input sequence, and history turn inter attention from different history turns to each token of the input sequence, and self-attention within input sequence, where the input sequence contains a current question and a passage. Then a feature selection method is designed to enhance the useful history turns of conversation and weaken the unnecessary information. Finally, we demonstrate the effectiveness of the proposed method on the QuAC dataset, analyze the impact of different feature selection methods, and verify the validity and reliability of the proposed features through visualization and human evaluation.

## introduction conversational question answering (convqa) is a new question answering task that requires a comprehension of the context, which has recently received more and more attention @xcite @xcite @xcite . since conversation is one of the most natural ways for humans to seek information, it carries over context through the dialogue flow. specifically, we ask other people a question, dependending on their answer, we follow up with a new question, and second answer with additional information will be given based on what has been discussed @xcite . therefore, multi-turn conversational question answering is formed in this way. it can be used in many fields as a personal assistant systems, such as, customer service, medical, finance, education, etc. moreover, with the rapid development of artificial intelligence technology in theory and practical applications, many personal assistant products have been launched in the market, such as alibaba alime, apple siri, amazon alexa, etc. although these assistants are capable to cover some simple tasks, they cannot handle complicated information-seeking conversations that require multiple turns of interaction @xcite . in the tasks of two recent multi-turn convqa datasets, coqa @xcite and quac @xcite , given a passage, a question, and the conversation context preceding the question, the task is to predict a span of passage as the answer or give an abstrctive answer based on the passage. so the machine has to understand a text passage and converstion history to answer a series of questions. each conversation in the quac dataset is obtained by two annotators playing the roles of teacher (informationprovider) and student (information-seeker) respectively. during the conversation, the student only has access to the heading of passage and tries to learn about a hidden wikipedia passage by asking a sequence ## related work convqa is closely related to machine reading comprehension (mrc) and conversational system. the convqa task is similar to the machine reading comprehension task @xcite , but the major difference from mrc is that the questions in convqa are organized in conversations @xcite , such as coqa @xcite , quac @xcite . some questions rely on the historical questions or answers through pronouns. for instance, there are two questions q 1 , q 2 and an answer a 1 to q 1 . q 1 :"who is going to have a birthday?". a 1 :"grandma li.". q 2 :"where she was born?". here, the pronoun "she" of q 2 associates q 2 with grandma li of a 1 , which indicates that the a 2 depend on a 1 . if the qa model does not use q 1 and a 1 , then it does not know who she refers to in q 2 , making it difficult for the model to accurately answer q 2 . however, the questions of traditional mrc datasets (such as squad @xcite and squad2.0 @xcite ) are independent of each other and have no relevance. compared with the traditional mrc task, multi-turn convqa based on mrc adds multiple turns of conversation history to the original mrc task, making the convqa task more suitable for human daily conversation habits. the exsiting methods for convqa in @xcite and @xcite determine whether the token in the question and the passage appear in each round of the historical conversation, and take the distance from the history turn of answers to the current question as the relative position, finally use the embedding of the relative position as an input of bert encoder @xcite . these methods are simple and effective, but they are not applicable to some no span-based answers. because the token in the abstractive answer may be synonymous with a word in the historical answer, not the same word. in this case, the relative position is invalid. moreover, a large amount of redundant information may also be introduced, and there may be a possibility of over-learning. for example, for a long passage, the author divides the passage into several sub-passages, and learns the relationship between each sub-passage and the answers of the historical rounds. if a question is only related to one of the sub-passages, suppose p 0 , and has nothing to do with another sub-passage p 1 . the information learned by p 1 and the largely redundant information of history conversation turns might play a negative interference role for the model to find the answer, while answering the current question q k . therefore, this paper focuses on how to select historical context and integrate its information into current question and passage. convqa is very similar to the background based conversations (bbcs) which recently proposed in the field of conversational systems. the latter is proposed to generate a more informative response based on unstructured background knowledge. but most of the research is aimed at topic-specific field @xcite , such as the conversation for movies @xcite and diverse set of topics of wikipedia @xcite . therefore, question answering based on reading comprehension and bbcs, these two tasks have in common that when responsing to each current sentence, not only the passage or background, but also the historical conversational context must be considered. the difference is that the former pays more attention to the ability of the model to understand the passage. when answering questions, the passage is mainly learned, and the historical conversation is supplemented to make the answer more accurate. the latter pays more attention to the ability of the model to understand the conversational context. when making a response, the model mainly learns conversational context, and assists with reference to background knowledge, the purpose is to enable the conversation to continue while making the response more informative. in terms of model structure, rnn-based structure and bert-based model @xcite have certain effectiveness on convqa, mrc and bbcs tasks. the rnn-based model @xcite can learn the impact of historical questions and answers on the current question and passage, but it cannot learn the deep bidirectional context representation. the bert-based model is proved to greatly improve the performance of convqa @xcite , but it lacks reasonable integration into the history turns of conversation. therefore, this paper proposes a method to model the history turns of questions and answers, generate impression features, and integrate them into the current questio and passage to improve model performance. ## conclusion and future work based on the general framework for convqa, we propose a new feature named impression feature, and combine the proposed feature with token embedding, position embedding and segment embedding as the input of bert encoder. then we introduce an impression feature selection method to select the important history information. extensive experiments show the effectiveness of our method. finally, we perform an in-depth analysis to show the different attention methods under different setting. future work will consider to integrate multi-oriented information and a free-form answer type for convqa.