M eeting B ank: A Benchmark Dataset for Meeting Summarization (2023)

Abstract: As the number of recorded meetings increases, it becomes increasingly important to utilize summarization technology to create useful summaries of these recordings. However, there is a crucial lack of annotated meeting corpora for developing this technology, as it can be hard to collect meetings, especially when the topics discussed are confidential. Furthermore, meeting summaries written by experienced writers are scarce, making it hard for abstractive summarizers to produce sensible output without a reliable reference. This lack of annotated corpora has hindered the development of meeting summarization technology. In this paper, we present MeetingBank, a new benchmark dataset of city council meetings over the past decade. MeetingBank is unique among other meeting corpora due to its divide-and-conquer approach, which involves dividing professionally written meeting minutes into shorter passages and aligning them with specific segments of the meeting. This breaks down the process of summarizing a lengthy meeting into smaller, more manageable tasks. The dataset provides a new testbed of various meeting summarization systems and also allows the public to gain insight into how council decisions are made. We make the collection, including meeting video links, transcripts, reference summaries, agenda, and other metadata, publicly available to facilitate the development of better meeting summarization techniques.

## introduction an astonishing 55 million meetings happen in the u.s. each week @xcite . with the extensive use of video conferencing software, e.g., microsoft teams, google meet and zoom, it has become easier than ever before to record meetings. while these recordings provide a wealth of human intelligence and actionable knowledge, the temporal nature of sound makes it difficult for users to navigate and 1 our dataset can be accessed at: meetingbank.github.io search for specific content @xcite . a summarization system that produces text summaries from transcripts can help, by providing users with great flexibility in navigating recordings, including but not limited to: meetings, interviews, podcasts, lectures, movies and tv series @xcite @xcite @xcite . effective meeting summarization requires annotated datasets. most summarizers, including fewshot and prompt-based @xcite , will benefit directly from benchmark datasets containing hundreds of thousands of document-summary pairs such as xsum @xcite , multi-news @xcite , govreport @xcite , plos @xcite . however, datasets for meeting summarization are relatively scarce, small, or unrepresentative. icsi and ami are two benchmark datasets @xcite that consist of only 75 and 140 meetings, respectively. other existing datasets for meetings are developed for speech recognition, or are in languages other than english and do not have reference summaries @xcite . creating an annotated meeting dataset poses several challenges. first, meetings often contain confidential or proprietary information, making it difficult to share them publicly. moreover, accurately annotating meeting summaries is a labor-intensive process, even for experienced writers familiar with the meeting topics @xcite . effective meeting summaries should capture key issues discussed, decisions reached, and actions to be taken, while excluding irrelevant discussions @xcite . there thus is a growing need for innovative approaches to construct a meeting dataset with minimal human effort to support advanced meeting solutions. an increasing number of city governments are releasing their meetings publicly to encourage transparency and engage residents in their decision making process. in this paper, we present a systematic approach to develop a city council meeting dataset. a city council is the legislative branch of local government. the council members are responsible for making decisions on a range of issues that affect the city and its citizens. these decisions may include creating and approving annual budgets, setting tax rates, confirming appointments of officers, enacting and enforcing ordinances, and setting policies on issues such as land use, public safety, and community development. figure 1 provides an example of a regular meeting of the city council of boston held on may 4, 2022. we present meetingbank, a benchmark dataset created from the city councils of 6 major u.s. cities to supplement existing datasets. it contains 1,366 meetings with over 3,579 hours of video, as well as transcripts, pdf documents of meeting minutes, agenda, and other metadata. it is an order of magnitude larger than existing meeting datasets @xcite . on average, a council meeting is 2.6 hours long and its transcript contains over 28k tokens, making it a valuable testbed for meeting summarizers and for extracting structure from meeting videos. to handle the max sequence length constraint imposed by abstractive summarizers, we introduce a divide-and-conquer strategy to divide lengthy meetings into segments, align these segments with their respective summaries from minutes documents, and keep the segments simple for easy assembly of a meeting summary. this yields 6,892 segment-level summarization instances for training and evaluating of performance. our repository can be further enhanced through community efforts to add annotations such as keyphrases and queries @xcite . to summarize, this paper presents the following contributions: ## existing datasets in this section, we review existing meeting datasets, discuss the techniques used to create reference summaries for them and identify research challenges that require attention in this area. icsi and ami are widely used datasets for meetings. icsi @xcite is a benchmark of 75 meetings that occurred naturally among speech researchers in a group seminar setting. each meeting lasts approximately an hour. ami @xcite ) contains 100 hours of recorded meetings, including 140 scenario-based meetings where groups of four participants assume roles within a fictitious company to design a product. meetings typically last around 30-40 minutes, with roles including a project manager, user interface designer, production designer, and marketing expert. a wide range of annotations are performed by annotators, including speech transcriptions, dialogue acts, topics, keywords, extractive and abstractive summaries. although small in size, these datasets offer a valuable testbed for evaluating meeting summarization systems @xcite @xcite @xcite @xcite . our study complements recent datasets for meetings such as elitr and qmsum. @xcite developed elitr, a dataset of 120 english and 59 czech technical project meetings spanning 180 hours of content. their minutes are created by participants and specially-trained annotators. @xcite developed the qmsum system which extracts relevant utterances from transcripts and then uses the utterances as input for an abstractor to generate query-focused summaries. human annotators are recruited to collect queries and compose summaries. they annotate a limited number of 25 committee meetings from the welsh parliament, 11 from the parliament of canada, as well as ami and icsi meetings for query-focused summarization. summarization datasets have been developed for genres similar to meetings, such as podcasts, interviews, livestreams and tv series. spotify @xcite released a dataset of 100,000 podcasts to support podcast search and summarization. the dataset includes automatic transcripts with word- thank you. next item, please. item 18 report from human resources. recommendation to purchase access workers compensation insurance for a total premium amount not to exceed 505,134 citywide. any public comment? no public comment on this item. oc motion, but counter appearances are a second. taken my customers and they asked customer pearce, do you want to? he said, you want to. short staff update. yes. i'm not sure which which item it is if it's 18, 19 or 20, but i just. was hoping to get a brief staff report on on the insurance. on which item? we'll do it on 18 is fine. okay alex vasquez will get the step forward. good evening, mayor and city council. i'm going to turn it over to jolene richardson. she's our risk manager and she'll give a brief overview of this particular report. even the mayor and council. this is for the city's annual renewal, for the excess workers compensation insurance, which is important for us to continue to provide coverage for our employees. it also helps us to reduce our negative financial consequences for our high exposures or losses that may result from injuries or deaths due to accidents, fire or terrorist attacks and earthquakes during work hours. this coverage will be obtained through the city's casualty. broker for a record. alliant insurance services. this year's policy for excess workers compensation will continue to provide 150 million and coverage access of 5 million self-insured retention at a premium of @xmath0505,134, for the period of july 1, 2020 through july 1, 2021. level time alignments and creator-provided podcast descriptions are used as reference summaries. me-diasum @xcite is a dataset of media interviews containing 463.6k transcripts from npr and cnn, with overview and topic descriptions used as reference summaries. streamhover @xcite used crowd workers to annotate 370 livestream videos for both extractive and abstractive summaries. summscreen @xcite consists of tv series transcripts and human-written recaps extracted from community websites. unlike meetings, these datasets have a smaller number of participants, usually one or two, and do not involve decision making. meetings can also occur through online chats, as opposed to face-to-face. samsum @xcite is a dataset of 16k chat dialogs with manually annotated abstractive summaries. the conversations are short, ranging from 3 to 30 utterances. fo-rumsum @xcite net forums, with associated human-written summaries. dialogsum @xcite ) contains 13k dialogs gathered from multiple websites. medical consultations conducted through online chats have also been used to create consultation summaries @xcite @xcite . our paper focuses on developing a dataset of naturallyoccurring meeting conversations to aid in the development of summarization systems from transcripts. ## creation of meetingbank there is a growing need to make public meetings more accessible and inclusive for citizens to engage with their local officials and have their voices heard. a 2020 report from the american academy of arts and sciences 2 reveals that only 11% of americans attend public meetings to discuss local issues. organizations such as the councildataproject.org and codeforamerica.org are working to improve the search infrastructure of public meetings. creation of a city council meeting dataset could provide a valuable testbed for meeting summarizers, and an effective summarizer could make public meetings more accessible by allowing citizens to navigate meetings more efficiently, thus promoting citizen engagement. we begin by compiling a list of the top 50 cities in the u.s. by population 3 and narrow it down to include only cities that regularly release meetings 2 amacad.org/ourcommonpurpose/recommendations 3 www.infoplease.com/us/cities/ top-50-cities-us-population-and-rank with accompanying minutes documents, and have downloadable videos on their city council websites. an example of a public meeting and its minutes can be seen in figure 1 . we consult with our legal team and reach out to city councils when necessary to ensure compliance with licensing and data policies. 4 our dataset for this release includes 1,366 meetings from six cities or municipalities spanning over a decade, including seattle, washington; king county, washington; denver, colorado; boston, massachusetts; alameda, california; and long beach, california. using minutes documents as-is for the development of summarization systems can be challenging. this is because minutes are often provided in pdf format and do not always align with the flow of meeting discussions. for instance, minutes may include a section on the mayor's update that provides detailed information on appointments of officers, but this is only briefly mentioned in the meeting. in general, minutes are more formal and comprehensive records of meetings, including information such as the date, location, attendees, summary of main points discussed, decisions made, and action items assigned. minutes are distributed to the stakeholders after the meeting. in contrast, meeting summaries tend to be shorter and less formal, focusing on the key points discussed in a meeting. we propose a divide-and-conquer strategy for creating reference meeting summaries. it involves dividing lengthy meetings into segments, aligning them with their corresponding summaries from minutes documents, and keeping the segments simple for easy assembly of a meeting summary. to start, we extract a list of council bill (cb) numbers discussed at the meeting by parsing the minutes and city council websites. 5 for each bill, we then identify a short description that summarizes its content, which serves as the reference summary. next, we use the bill number to obtain the corresponding meeting segment, including its start and end time, by referencing the index of the meeting on the city council website (see figure 1 ). the transcript of that segment serves as the source text for the summarizer. after filtering out noisy and too short segments, 6 we have a total of 6,892 segment-level instances in our dataset ( ## data analysis we measure the level of abstraction in meeting summaries by quantifying the amount of reused text. higher abstraction poses more challenges to the meeting summarization systems. we employ two common measures, coverage and density @xcite , to evaluate segment-level reference summaries. results are illustrated in figure 3 , with coverage on x-axis and density on y-axis. the coverage score measures the percentage of summary words that appear in the source transcript. e.g., a summary of 10 words that includes 8 words from the source transcript and 2 new words has a coverage score of 0.8 (= 8/10). as shown in the figure, the coverage score for city council meeting summaries is in the range of 0.7-0.9 for most cities. this suggests that, unlike news, these meeting summaries tend to include discussion points verbatim rather than performing abstraction. given a compression rate of over 90%, an effective summarizer should focus on accurately identifying content to be included in the summary. a density score evaluates how much a summary can be characterized as a set of extractive fragments. for example, a summary of 10 words made up of two extractive fragments of length 1 and 6 and three new words would have a density score of 3.7 = (1 2 + 6 2 )/10. a summary with long consecutive text fragments taken from the source transcript would yield a high density score. we observe that seattle and boston have the highest density scores among all cities studied, while denver has the lowest score, indicating a high degree of editing is performed to produce the minutes for denver. we note that certain resolutions and ordinances are read out plainly at the council meetings and included in the minutes, making the summaries often have higher density scores than those of news documents. the coverage and density measures can be influenced by a range of factors such as the length and complexity of meetings and the preferences of the minute-takers. the diversity of meeting summaries highlights the complexity of this task. ## performance of existing systems we evaluate state-of-the-art summarization systems on city council meetings, focusing on segments of the meetings rather than entire transcripts due to the length constraint imposed by abstractive summarizers. we split our dataset into train, validation and test sets, containing 5169, 861, 862 instances respectively. each summarizer is given the transcript of a meeting segment and tasked with generating a concise summary. the results are reported for the test set of our meeting dataset. extractive. our extractive methods include the oracle, lead, lexrank and textrank @xcite . the extractive oracle foot_1 selects the highest-scoring sentences from the input transcript, adding one sentence at a time until the combined r1 and r2 score can no longer be improved. the lead-n baseline selects the first n sentences of the input. lexrank and textrank, both graph-based methods, determine the importance of sentences by analyzing their centrality in the graph structure. both methods are set to extract two sentences from a transcript segment, which is the average number of sentences in the reference summaries. abstractive with fine-tuning. we investigate five best-performing neural abstractive summarizers. these include bart-large @xcite , a denoising autoencoder that is trained to reconstruct original text from corrupted input, pegasus @xcite , a model that is trained to regenerate missing key sentences, longformer @xcite , a model designed to handle long sequences gpt-3 with prompting. large language models like gpt-3, t5 and palm @xcite @xcite have developed advanced capabilities due to increased computation, parameters, training data size @xcite . when prompted, gpt-3 can generate a summary of a source text by identifying the most important information. the text-davinci-003 version of gpt-3 is used in this study, with a prompt asking the model to summarize the text in two sentences @xcite . evaluation metrics. we use a variety of automatic evaluation metrics to assess the quality of transcript summaries. these metrics are broadly grouped into three categories: (a) traditional metrics comparing system and reference summaries based on lexical overlap, including rouge @xcite , rougewe (w/ embeddings), bleu @xcite and me-teor @xcite ; (b) new metrics making use of contextualized embeddings to measure semantic similarity, e.g., bertscore @xcite and moverscore @xcite ; (c) question answering-based metrics, where the hypothesis is that high-quality summaries should contain informative content and act as a surrogate for the original document in satisfying users' infor-mation needs. we leverage summarization evaluation toolkits provided by @xcite , sacre-bleu @xcite , qaeval @xcite and summertime @xcite to report results using these metrics. ## city-by-city analysis we investigate the characteristics that make effective training instances for meeting summarization by conducting a series of ablations. we begin by informativeness: how well does the summary capture the main points of the meeting segment? a good summary should contain all and only the important information of the source. ## human evaluation we evaluate the performance of seven state-of-theart summarization systems, including fine-tuned abstractive models hmnet, bart, pegasus, dialoglm, 10 we fine-tune the bart summarizers for the same number of steps in both cases to mitigate the impact of varying number of training instances. gpt-3 with prompting, and traditional extractive models lexrank and lead to best assess the effectiveness of system-generated meeting summaries. all abstractive models have been fine-tuned on the train split of our city councils dataset to achieve the best possible results. to ensure high quality in the assessment of summaries, we have worked with imerit.net, a labor sourcing company, to recruit experienced evaluators from the u.s. and india to perform annotations. the workers are registered on appen.com, a crowdsourcing platform, to complete the tasks and deliver results. a total of three workers from the united states and six workers from india participate in our evaluations, including pilot annotations. 11 the workers are asked to watch a video segment, typically 30 minutes or less, read the transcript, and then evaluate the quality of each system summary based on five criteria: informativeness, factuality, fluency, coherence, and redundancy. these criteria are outlined in in a document and using the remaining sentences to regenerate them, making it particularly well-suited for this task and effective at identifying important content from the transcripts. we find that gpt-3 achieves the highest overall score of 4.21 according to human evaluations across all criteria. this aligns with recent studies that demonstrate gpt-3's near-human performance in news summarization @xcite . on our meeting dataset, gpt-3 shows exceptional performance in terms of fluency and coherence, receiving scores of 4.52 and 4.41 respectively. however, its results are less impressive in terms of informativeness and factuality with scores of 3.74 and 3.82, but still on par with the best abstractive model, pegasus. our findings suggest that meeting summarization solutions should continue to focus on capturing the main discussion points and staying true to the original content. ## conclusion we created a benchmark dataset from city council meetings and tested various summarization systems including extractive, abstractive with fine-tuning, and gpt-3 with prompting on this task. our findings indicate that gpt-3 is well received in human assessments, but it falls short in terms of informativeness and factual consistency. our meetingbank dataset could be a valuable testbed for researchers designing advanced meeting summarizers and for extracting structure from meeting videos.