QMS um: A New Benchmark for Query-based Multi-domain Meeting Summarization (2021)

Abstract: Meetings are a key component of human collaboration. As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may or may not have attended the meetings about the key decisions made and the tasks to be completed. However, it is hard to create a single short summary that covers all the content of a long meeting involving multiple people and topics. In order to satisfy the needs of different types of users, we define a new query-based multi-domain meeting summarization task, where models have to select and summarize relevant spans of meetings in response to a query, and we introduce QMSum, a new benchmark for this task. QMSum consists of 1,808 query-summary pairs over 232 meetings in multiple domains. Besides, we investigate a locate-then-summarize method and evaluate a set of strong summarization baselines on the task. Experimental results and manual analysis reveal that QMSum presents significant challenges in long meeting summarization for future research. Dataset is available at https://github.com/Yale-LILY/QMSum .

## introduction meetings remain the go-to tool for collaboration, with 11 million meetings taking place each day in the usa and employees spending six hours a week, on average, in meetings @xcite . the emerging landscape of remote work is making meetings even more important and simultaneously taking a toll on our productivity and wellbeing @xcite . the proliferation of meetings makes it hard to stay on top of this sheer volume of information and increases the need for automated methods for accessing key information exchanged during them. meeting summarization @xcite @xcite meeting transcript summarize the whole meeting. the meeting was mainly related to ...... turn 0: project manager: we have been provided with some technical tools to communicate. in this task, a model is required to summarize the contents that users are interested in and query. @xcite is a task where summarization models are leveraged to generate summaries of entire meetings based on meeting transcripts. the resulting summaries distill the core contents of a meeting that helps people efficiently catch up to meetings. most existing work and datasets on meeting summarization @xcite pose the problem as a single document summarization task where a single summary is generated for the whole meeting. unlike news articles where people may be satisfied with a high-level summary, they are more likely to seek more detailed information when it comes to meeting summaries such as topics @xcite , opinions, actions, and decisions @xcite . this poses the question of whether a single paragraph is enough to summarize the content of an entire meeting? figure 1 shows an example of a meeting about "remote control design". the discussions in the meeting are multi-faceted and hence different users might be interested in different facets. for example, someone may be interested in learning about the new trends that may lead to the new product standing out, while others may be more interested in what other attendees thought about different elements of the design. it is challenging to compress or compose a short summary that contains all the salient information. alternatively, summarization systems should adopt a more flexible and interactive approach that allows people to express their interests and caters to their diverse intents when generating summaries @xcite @xcite . with comprehensive consideration of the multigranularity meeting contents, we propose a new task, query-based meeting summarization. to enable research in this area, we also create a highquality multi-domain summarization dataset. in this task, as shown in figure 1 , given a query and a meeting transcript, a model is required to generate the corresponding summary. the query-based approach is a flexible setup that enables the system to satisfy different intents and different levels of granularity. besides the annotated queries and corresponding gold summaries at different levels of granularity, our new dataset contains a rich set of annotations that include the main topics of each meeting and the ranges of relevant text spans for the annotated topics and each query. we adopt a hierarchical annotation structure that could not only assist people to find information faster, but also strengthen the models' summarization capacity. in this paper, we employ a two-stage meeting summarization approach: locate-then-summarize. specifically, given a query, a model called locator is used to locate the relevant utterances in the meeting transcripts, and then these extracted spans are used as an input to another model called summarizer to generate a query-based summary. we present and evaluate several strong baselines based on state-of-the-art summarization models on qm-sum. our results and analysis from different perspectives reveal that the existing models struggle in solving this task, highlighting the challenges the models face when generating query-based meeting summaries. we are releasing our dataset and baselines to support additional research in queryfocused meeting summarization. overall, our contributions are listed as follows: ## data construction in this section, we show how we collected meeting data from three different domains: academic meetings, product meetings, and committee meetings. in addition, we show how we annotated the three types of meeting data while ensuring annotation quality for query-based meeting summarization. ## method in this section, we first define the task of querybased meeting summarization, then describe our two-stage locate-then-summarize solution in detail. ## experiments in this section, we introduce the implementation details, effectiveness of locator, experimental results and multi-domain experiments on qmsum. ## analysis in this section, we conduct comprehensive analysis of query types and errors in the model output. ## conclusion we propose a new benchmark, qmsum, for querybased meeting summarization task. we build a locate-then-summarize pipeline as a baseline and further investigate variants of our model with different locators and summarizers, adopt different training settings including cross-domain and multidomain experiments to evaluate generalizability, and analyze the task difficulty with respect to query types. the new task and benchmark leave several open research directions to explore: 1) how to process the long meeting discourses; 2) how to make a meeting summarization model generalize well; 3) how to generate summaries consistent with both meeting transcripts and queries. 4) how to reduce the annotation cost for meeting summarization. erty and privacy rights of the original authors: both of the collected meeting transcripts and recruited annotators. we ensure that the dataset construction process is consistent with the intellectual property and privacy rights of the original authors of the meetings. all the meeting transcripts we collected are public and open to use according to the regulation 6 7 8 9 . the annotation process is consistent with the intellectual property and privacy rights of the recruited annotators as well. compensation for annotators we estimated the time for annotating one meeting is around 1 -2 hours. therefore, we paid annotators around @xmath028 for each committee meeting. to further encourage annotators to work on annotations, we proposed bonus mechanism: the bonus of each of the 5th to 8th meetings would be @xmath15, and so on. some of the authors also did annotations and they were paid as well. steps taken to avoid potential problems the most possible problems which may exist in the dataset is bias problem and the inconsistency among queries, annotated summaries and original meeting contents. with regard to bias problem, we find that product meeting dataset rarely contains any explicit gender information, but annotators still tended to use 'he' as pronoun. to avoid the gender bias caused by the usage of pronouns, we required annotators to replace pronouns with speaker information like 'project manager', 'marketing' to avoid the problem. also, when designing queries based on query schema list, we found that annotators usually used the same query schema, which might lead to bias towards a certain type of query. therefore, we asked the annotators to use different schemas as much as possible. for the inconsistency problem, each annotation step was strictly under supervision by 'experts' which are good at annotation and could be responsible for reviewing.