End-to-end Spoken Conversational Question Answering: Task, Dataset and Model (2022)

Abstract: In spoken question answering, the systems are designed to answer questions from contiguous text spans within the related speech transcripts. However, the most natural way that human seek or test their knowledge is via human conversations. Therefore, we propose a new Spoken Conversational Question Answering task (SCQA), aiming at enabling the systems to model complex dialogues flow given the speech documents. In this task, our main objective is to build the system to deal with conversational questions based on the audio recordings, and to explore the plausibility of providing more cues from different modalities with systems in information gathering. To this end, instead of directly adopting automatically generated speech transcripts with highly noisy data, we propose a novel unified data distillation approach, DDNet, which effectively ingests cross-modal information to achieve fine-grained representations of the speech and language modalities. Moreover, we propose a simple and novel mechanism, termed Dual Attention, by encouraging better alignments between audio and text to ease the process of knowledge transfer. To evaluate the capacity of SCQA systems in a dialogue-style interaction, we assemble a Spoken Conversational Question Answering (Spoken-CoQA) dataset with more than 40k question-answer pairs from 4k conversations. We first show that the performance of the existing state-of-the-art methods significantly degrade on our dataset, hence demonstrating the necessity of incorporating cross-modal information to achieve good performance gains. Our experimental results demonstrate that our proposed method achieves superior performance in spoken conversational question answering. Codes and datasets will be made publicly available.

## introduction conversational question answering (cqa) has been studied extensively over the past few years * equal contribution. current cqa research mainly focuses on leveraging written text sources in which the answer can be extracted from a large document collection. however, humans communicate with each other via spontaneous speech (e.g., meetings, lectures, online conversations), which convey rich information. consider our multimodal experience, fine-grained representations of both audio recordings and text documents are considered to be of paramount importance. thus, we learn to draw useful relations between modalities (speech and language), which enables us to form fine-grained multimodal representations for end-to-end speech-and-language learning problems in many real-world applications, such as voice assistant and chat robot. in this paper, we propose a novel and challenging spoken conversational question answering task -scqa. an overview pipeline of this task is shown in figure 1 . collecting such a scqa dataset is a non-trivial task, as in contrast to current cqa tasks, we build our scqa with two main goals as follows: (1) scqa is a multi-turn conversational spoken question answering task, which is more challenging than only text-based task; (2) existing cqa methods rely on a single modality (text) as the context source. however, plainly leveraging uni-modality information is naturally undesirable for end-to-end speech-and-language learning problems since the useful connections between speech and text are elusive. thus, employing data from ## task definition in this section, we propose the novel scqa task and collect a spoken-coqa (s-coqa) dataset, which uses the spoken form of multi-turn dialogues and spoken documents to answer questions in multiturn conversations. given a spoken document d s , we use d t to denote the clean original text and d a to denote the asr transcribed document. we also have q a 1:@xmath0, q a 2 , ..., q a l }, which is a collection of l-turn asr transcribed spoken questions q s 1:l , as well as a t 1:@xmath1, a t 2 , ..., a t l } which are the corresponding answers to the questions in clean texts. the objective of scqa task is then to generate the answer a t l for question q a l , given document d a , multi-turn history questions q a 1:l-1 ={q a 1 , q a 2 , ..., q a l-1 }, and reference answers a t 1:l-1 = {a t 1 , a t 2 , ..., a t l-1 }. in other words, our task in the testing phase can be formulated as however, in the inference stage, these additional information of d t and q t 1:l are not needed. ## ddnet in this section, we propose ddnet to deal with the scqa task, which is illustrated in figure 2 . we first describe the embedding generation process for both audio and text data. next, we propose dual attention to fuse the speech and textual modalities. after that, we present the major components of the ddnet module. finally we describe a simple yet effective distillation strategy in the proposed ddnet to learn enriched representations in the speech-text domain comprehensively. ## experiments and results in this section, we first describe the collection and filtering process of our proposed spoken-coqa dataset in detail. next, we introduce several stateof-the-art language models as our baselines, and then evaluate the robustness of these models on our proposed spoken-coqa dataset. finally, we provide a thorough analysis of different components of our method. note that we use the default settings in all evaluated methods. data collection. we detail the procedures to build spoken-coqa as follows. first, we select the conversational question-answering dataset coqa @xcite foot_1 as our basis data since it is one of the largest public cqa datasets. coqa contains around 8k stories (documents) and over 120k questions with answers. the average dialogue length of coqa is about 15 turns, and the answers areis in free-form texts. in coqa, the training set and the development set contain 7,199 and 500 conversations over the given stories, respectively. therefore, we use the coqa training set as our reference text of the training set and the coqa development set as the test set in spoken-coqa. next, we employ the google text-to-speech system to transform the questions and documents in coqa into the spoken form, and adopt cmu sphinx to transcribe the processed spoken contents into asr transcriptions. in doing so, we collect more than 40g audio data, and the data duration is around 300 hours. the asr transcription has a kappa score of 0.738 and word error rates (wer) of 15.9%, which can be considered sufficiently good since it is below the accuracy threshold of 30% wer @xcite . for the test set, we invite 5 human native english speakers to read the sentences of the documents and questions. the sentences of one single document are assigned to a single speaker to keep consistency, while the questions in one example may have different speakers. all speech files are sampled at 16khz, following the common approach in the speech community. we provide an example of our spoken-coqa dataset in ## ablation study we conduct ablation studies to show the effectiveness of several components in ddnet in this section and appendix. ## conclusions in this paper, we have presented scqa, a new spoken conversational question answering task, for enabling human-machine communication. we make our effort to collect a challenging dataset -spoken-coqa, including multi-turn conversations and pas-sages in both text and speech form. we show that the performance of existing state-of-the-art models significantly degrade on our collected dataset, hence demonstrating the necessity of exploiting cross-modal information in achieving strong results. we provide some initial solutions via knowledge distillation and the proposed dual attention mechanism, and have achieved some good results on spoken-coqa. experimental results show that ddnet achieves substantial performance improvements in accuracy. in future, we will further investigate the different mechanisms of integrating speech and text content, and our method also opens up the possibility for downstream spoken language tasks.