The Second Automatic Minuting ( A uto M in) Challenge: Generating and Evaluating Minutes from Multi-Party Meetings (2022)

Abstract: We would host the AutoMin generation chal- lenge at INLG 2023 as a follow-up of the first AutoMin shared task at Interspeech 2021. Our shared task primarily concerns the automated generation of meeting minutes from multi-party meeting transcripts. In our first venture, we ob- served the difficulty of the task and highlighted a number of open problems for the community to discuss, attempt, and solve. Hence, we invite the Natural Language Generation (NLG) com- munity to take part in the second iteration of AutoMin. Like the first, the second AutoMin will feature both English and Czech meetings and the core task of summarizing the manually- revised transcripts into bulleted minutes. A new challenge we are introducing this year is to devise efficient metrics for evaluating the quality of minutes. We will also host an optional track to generate minutes for European parliamentary sessions. We carefully curated the datasets for the above tasks. Our ELITR Minuting Corpus has been recently accepted to LREC 2022 and publicly released. We are already preparing a new test set for evaluating the new shared tasks. We hope to carry forward the learning from the first AutoMin and instigate more community attention and interest in this timely yet chal- lenging problem. INLG, the premier forum for the NLG community, would be an appropriate venue to discuss the challenges and future of Automatic Minuting. The main objective of the AutoMin GenChal at INLG 2023 would be to come up with efficient methods to auto- matically generate meeting minutes and design evaluation metrics to measure the quality of the minutes.

## introduction ever since most of our interactions went virtual, the need for automatic support to run online meetings became essential. due to frequent meetings and the resulting context switching, people are experiencing an information overload @xcite of epic proportions. hence a tool to automatically summarize a meeting proceeding would be a valuable addition to the virtual workplace. automatic minuting @xcite is close to summarization; however, there are subtle differences. while summarization is motivated towards generating a concise and coherent summary of the text, minuting is more inclined towards adequately capturing the contents of the meeting (where coverage is probably more significant than coherence and conciseness). summarizing spoken multi-party dialogues @xcite comes with its own challenges: incorrect/noisy automated speech recognition (asr) outputs, long discourse, topical shifts, the dialogue turns, redundancies and small talk, etc. hence we deem automatic minuting to be more difficult than text summarization (figure 2 in appendix a shows an envisaged demonstration of the task). with the automin challenge, we want to explore the various problems associated with the task and their potential solutions from the perspective of a multi-year joint community initiative. apart from the main task of summarizing meeting transcripts into concise, bulleted minute items, another crucial task is to develop efficient evaluation measures to judge the quality of the automatically generated minutes. it is a known fact that the current popular methods of automatic summarization evaluation (e.g., rouge @xcite ) do not guarantee critical quality parameters like fluency, adequacy, grammatical correctness, etc. @xcite , which is why we have to primarily rely on human evaluation metrics in our shared task. the proposed instance of the automin challenge will venture into developing automatic/semi-automatic evaluation metrics to measure the quality of generated minutes. summarizing the participants' ideas for this challenge and the anticipated follow-up dis-cussions, we will try to define an ideal meeting summary. since the task suffers from resource scarcity, we would launch an initiative where interested parties could donate their meetings to prepare a public multimodal, multilingual dataset of real meetings. ## first automin @ interspeech 2021 the automin foot_1 shared task at interspeech 2021 @xcite was a first of its kind with this problem. it generated considerable interest in the speech and nlp community. twenty-seven teams from diverse geographical regions registered, and finally, ten teams (both from academia and industry) actively participated in the challenge. almost 70 people attended the shared task virtual event. the first automin consisted of one main task (task a) and two supporting tasks (task b and task c), relying on a dataset of transcripts and minutes from primarily technical meetings in english and czech @xcite . considering the current non-availability of largescale domain datasets on multiparty meeting summarization, the best recipe for automatic minuting that evolved out from the first automin is roughly the following: training a deep neural model on available dialogue summarization datasets (sam-sum @xcite ; dialsum @xcite ; etc.) and further fine-tuning it on the minuting or meeting summarization datasets (ami @xcite ; icsi @xcite ; au-tomin @xcite )), accompanied by some intelligent pre-and post-processing steps. ## task overview we would continue with the tasks in the previous automin challenge in the current iteration. however, the new additions would be: (1) automatically generating the meeting minutes of parliamentary sessions as part of task a, and (2) designing appropriate evaluation schema/metrics to evaluate the generated minutes as a new task d. ## dataset description we provide the automin 2023 participants with elitr minuting corpus @xcite ; however, we would allow them to use any external datasets if they explicitly describe them in their system reports. ## baseline evaluations we provide our participants with the baseline codes for automatic minuting (task a) here. 5 the details of the experiments are described in @xcite . it includes initial exploration using off-theshelf text summarization models for future investigations. for generating abstractive meeting minutes we use bart @xcite , bertsum @xcite , bert2bert @xcite , led @xcite , pegasus @xcite , roberta2roberta @xcite , and t5 @xcite models. for extractive meeting summaries we use tf-idf-based summarizer @xcite , an unsupervised extractive summarizer, textrank @xcite , lexrank @xcite , luhn algorithm @xcite , and lsa @xcite based summarizer. these off-the-shelf text summarization models are not the best candidates for generating minutes which calls for further research on this challenging task for meetingspecific summarization or minuting models. ## automin 2023 procedure and timeline ## diversity and inclusion as our commitment to diversity and inclusion, like the previous iteration, we would like to make our event open-to-all (and possibly hybrid) in consultation with inlg 2023 chairs. we would especially reach out to organizations like widening nlp foot_5 (where our first organizer is also a chair) to help us reach the underrepresented groups and communities and encourage them to participate. we would july 2022 announcement at inlg 2022 august 2022 call for participation september 2022 training data release december 2023 test data release february 2023 system submission march 2023 evaluation notification april 2023 system report submission may 2023 system report review notification may 2023 camera-ready submission june 2023 proceedings appear in acl anthology july 2023 second automin at inlg 2023 ## conclusion automin is a very timely yet complex task for the speech and natural language processing community. given the array of problems this task had to offer, we are very excited to continue this iteration of the generation challenge at inlg 2023. we look forward to uncovering the several linguistic phenomena and insights that should go into action while a machine writes a minute and see how much we have progressed towards an acceptable automated minuting output. in that essence, task a and task d are of more interest to the nlg and summarization community than task b and task c.