M eeting QA : Extractive Question-Answering on Meeting Transcripts (2023)

Abstract: With the ubiquitous use of online meeting platforms and robust automatic speech recognition systems, meeting transcripts have emerged as a promising domain for natural language tasks. Most recent works on meeting transcripts primarily focus on summarization and extraction of action items. However, meeting discussions also have a useful question-answering (QA) component, crucial to understanding the discourse or meeting content, and can be used to build interactive interfaces on top of long transcripts. Hence, in this work, we leverage this inherent QA component of meeting discussions and introduce MeetingQA, an extractive QA dataset comprising of questions asked by meeting participants and corresponding responses. As a result, questions can be open-ended and actively seek discussions, while the answers can be multi-span and distributed across multiple speakers. Our comprehensive empirical study of several robust baselines including long-context language models and recent instruction-tuned models reveals that models perform poorly on this task (F1 = 57.3) and severely lag behind human performance (F1 = 84.6), thus presenting a challenging new task for the community to improve upon.

## introduction millions of meetings occur every day worldwide, which results in vast amounts of meeting transcripts. meeting transcripts are typically long documents, often domain-specific depending on the subject matter, and contain a lot of information. basic tasks such as catching up with a missed meeting, looking up a specific discussion or response to a query can be time-consuming. these tasks can be facilitated by nlp systems, including summarization and question-answering. to this end, several publicly available small-scale corpora of meeting 1 meetingqa data and code is available at https:// archiki.github.io/meetingqa.html figure 1 : representative example from meeting transcript segment in meetingqa. the question and annotated answer are highlighted in red and blue respectively. prior nlp work on meeting transcripts mainly focuses on summarization @xcite @xcite . however, lack of annotated data impedes research on other important nlp tasks in this domain. to address this gap, we introduce a question-answering (qa) task based on conversations in meeting transcripts. specifically, we consider questions asked by participants during the meeting and aim to extract corresponding answer spans from relevant discussions among meeting participants (refer to figure 1 ). this task has several practical applications such as building an interactive meeting browser/interface for navigating through transcripts and informing tasks such as meeting summarization and handling action items involving qa pairs @xcite . while standard qa datasets consist of human generated questions either based on short supplied contexts @xcite @xcite or are answered using a large collection of documents @xcite @xcite , our task setting is challenging yet interesting in several ways. first, meeting transcripts are long documents and qa systems still struggle to understand long contexts @xcite . second, successfully answering questions asked within meetings requires robust understanding of the conversation and discourse that takes place both before and after a question. third, the multi-party spoken text falls under a different domain when compared to typical text documents. while standard long documents rarely include any meaningful (nonrhetorical) questions, multi-party meetings often involve discussions asked by one participant and answered by the rest, allowing us to use these questions to create a qa dataset. furthermore, the conversational nature of transcribed text differs from written documents and may contain disfluencies and other artifacts. finally, instead of using annotator-generated questions (like in @xcite ), questions asked by participants are more open-ended and discussion-seeking, with interesting answer types that can be multi-span and/or contributed by multiple speakers (e.g., figure 1 ). to this end, we first introduce our dataset meet-ingqa, created by annotating meetings transcripts from the popular ami (augmented multi-party interaction) corpus, containing over 100 hours of meetings @xcite , via a robust annotation pipeline . meetingqa comprises of 7,735 questions asked by participants across 166 meetings. unlike other datasets, questions in meet-ingqa are less concise (12 words on average) and reflect queries asked in a conversational setting. the answers include realistic situations such as rhetorical questions, multiple discontiguous spans and/or contributions from multiple speakers. next, on meetingqa dataset, we test diverse models designed for long input contexts such as longformer @xcite , and bigbird @xcite as well as roberta @xcite , and deberta-v3 @xcite with as much meeting context surrounding the question as possible. to incorporate the multi-span nature of answers in our dataset, we design and experiment with multi-span variants of the aforementioned models. furthermore, we also investigate how well recent instruction-tuned large language models fare at answering questions from meetingqa. lastly, we create a silver-annotation pipeline using me-diasum @xcite , a corpus containing 463.6k short interview transcripts, to provide additional training data. we find that the best performance is achieved by finetuned short-context models (@xmath0.3). overall, we show that models struggle to identify rhetorical questions and selecting which utterances constitute the answer. thus, model performance significantly trails behind human performance on meetingqa (@xmath1.6), leaving a large potential for future improvements on this challenging task. ## methods in this section, we investigate the difficulty level of our new meetingqa for state-of-the-art qa systems and establish strong baseline results. we describe strategies for retrieving contexts from transcripts in section 3.1, followed by different qa models in section 3.2, and silver data annotation for data augmentation methods in section 3.3. ## experiments and results evaluation metrics. following @xcite we report macro-averaged f1 on the entire test set as well as on specific answer types (section 2.2). 5 however, f1 treats sequences as bagof-words, and thus, there can be a non-significant 5 we also report exact match (em) scores in appendix c. overlap between a random span and the target span for large span lengths. to address this, @xcite propose reporting intersection-over-union (iou) defined as: training settings. we measure performance of various models in both finetuned and zero-shot settings. first, we directly finetune the base pretrained model on the model on meetingqa. next, to supplement training data we explore intermediatetraining @xcite with squad v2.0 @xcite 6 or a combination including silver data from section 3.3 prior to finetuning on meetingqa, increasing the training data by 5x and 10x respectively. additional details on checkpoints, hyperparameters, and training are present in appendix b. turn-based baseline. we devise a straightforward algorithm called turn-based baseline that is inspired by the automatic silver data annotation algorithm explained in section 3.3. in the turnbased baseline, when a speaker asks a question, the predicted answer includes all the subsequent utterances of other speakers until the same speaker gets another turn (stopping criterion). note that, turn-based baseline assumes all questions can be answered and always provides single-span answers, although the predictions may be multi-speaker. ## conclusion in this work, we present meetingqa, an extractive qa dataset based on meeting transcripts to identify answers to questions asked during discussion among meeting participants. detailed analysis of the data reveals it is a challenging real-world task. baseline experiments with a wide variety of models show the current performance lags behind human performance by at least 25 and 44 overall f1 points for finetuned and zeroshot models respectively. this demonstrates that current qa systems find our task challenging, leaving tremendous scope for improvement. we hope that future works will aim to bridge this gap and our work fosters research in nlp tasks (especially qa) on other text domains such as meeting transcripts.