Overview of the Second Shared Task on Automatic Minuting ( A uto M in) at INLG 2023 (2023)

Abstract: In this article, we report the findings of the second shared task on Automatic Minuting (AutoMin) held as a Generation Challenge at the 16th International Natural Language Generation (INLG) Conference 2023. The second Automatic Minuting shared task is a successor to the first AutoMin which took place in 2021. The primary objective of the AutoMin shared task is to garner participation of the speech and natural language processing and generation community to create automatic methods for generating minutes from multi-party meetings. Five teams from diverse backgrounds participated in the shared task this year. A lot has changed in the Generative AI landscape since the last AutoMin especially with the emergence and wide adoption of Large Language Models (LLMs) to different downstream tasks. Most of the contributions are based on some form of an LLM and we are also adding current outputs of GPT4 as a benchmark. Furthermore, we examine the applicability of GPT-4 for automatic scoring of minutes. Compared to the previous instance of AutoMin, we also add another domain, the minutes for EU Parliament sessions, and we experiment with a more fine-grained manual evaluation. More details on the event can be found at https://ufal.github.io/automin-2023/.

## introduction automatic minuting @xcite refers to the task of automatically creating meeting minutes from multi-party meeting conversations. since the pandemic, a significant portion of the global workforce, especially those in information technology (it) and it-enabled services, has gone virtual or preferring a hybrid mode of work. 1 meetings have always been important to ensure smooth coordination and success of projects, but the proportion of sessions which combine remote and onsite workforce and collaboration between geographically distant members has grown manifold. undeniably, frequent meetings create significant cognitive workload on people. to document the discussions, updates, tasks planned, minuting is an essential activity in meetings (be it online, in-person, or hybrid). usually minutes are jotted down by some member(s) in the meeting but due to the lack of any fixed standards for minuting, different minute-takers may have different perspectives depending on their background. despite that minutes of the same meeting by different authors may differ in certain aspects and content @xcite , this activity can be automated to some extent. there has been a body of research in this topic since the ami @xcite , icsi @xcite and calo meeting assistant (voss and @xcite projects. some interesting recent works on meeting and dialogue summarization include those by @xcite ; @xcite ; @xcite ; @xcite . we summarize our efforts on automatic minuting in @xcite ; @xcite . from the previous automin shared task @xcite , top-performing systems from @xcite ; @xcite showed the usability of a bart-based @xcite system trained on samsum corpus @xcite for the task. an exhaustive survey of abstractive meeting summarization research could be found in @xcite ; @xcite ; @xcite . for long, resource creation for meeting summarization was difficult because of privacy reasons (ami and icsi were the only publicly available ones and later we introduced elitr minuting corpus @xcite ). however quite recently, we see few datasets are made available to support research in this topic, incl. e.g. @xcite ; @xcite ; @xcite ; @xcite ; @xcite . recently, with the "generative ai revolution", pre-trained large language models (llms; @xcite @xcite have been employed for this task @xcite @xcite , demonstrating amazing output quality. based on the general public sentiment about the capabilities of llms, one could assume that automatic minuting belongs to one of the tasks that have suddenly become essentially solved. to verify the status in a rigorous way and to search for any open challenges that need to be addressed and also to assess how far we evolved since the last au-tomin @xcite , we continued with the second iteration of the automin shared task. a related effort along this direction was the di-alogsum generation challenge @xcite at inlg 2022. we proposed the second iteration of the au-tomin shared task as a generation challenge for inlg 2023 @xcite . essentially, with the current iteration of automin, we wanted to find out: we describe our shared task and present our findings in the remainder of the paper. ## tasks description we offered four tasks (task a, task b, task c, and task d) to automin participants @xcite . tasks a-c were known from the previous automin instance, task d was new and focused on evaluation of minutes. in the end, all the teams decided to take part only in the first and most important task, namely minuting from diarized transcript (task a). to compensate for the lack of participation in task d, we experimented with automatic evaluation using llms, see section 6.3. ## dataset description the datasets for automin 2023 cover three types of data: project meetings in both english and czech, as well as parliamentary sessions in english. basic statistics of the data are in the project meeting data was prepared from our own sources, while the parliamentary sessions were taken from the european parliament mostly as-is, we merely selected data which was useful for our purposes. ## evaluated systems we evaluate submissions from the participants, baselines we obtained using llms and also the reference minutes. kindly refer to appendix b for samples of the automatically created minutes. ## evaluation in this section, we describe the evaluation process carried out for automin 2023. we used human evaluations, traditional automatic evaluations as well as evaluations using llms. ## meta-analysis of automatic evaluation this section presents a comprehensive metaanalysis of the automatic evaluation in regard to human evaluation. the goal of this meta-analysis is to assess the usability of various automatic approaches, be it bart-score, bert-score, variants of rouge, or gpt-based evaluation introduced in section 6.3. ## conclusions and future plans this paper presented the results of automin 2023, the second instance of our shared task on automatic summarization of meeting transcripts into meeting minutes. the data of the shared task (inputs, references, submissions, scores etc.) are available in this repository: https://github.com/ufal/automin-2023-data as in the previous instance, the task was run on two languages (english and czech), with english receiving more attention from the participants. the submissions were scored manually at the level of full minutes (document level) and also averaging the scores assigned to individual minutes points (hunk level). we concluded that the more fine-grained hunk-level annotation is less useful; the most important question we would like to learn from our annotators is whether the minutes properly reflect the content and overall impression of the meeting. with hunk-level annotation, the annotation process is fragmented and even if each of the fragments is of a high quality, the aggregation of these scores does not answer the key question. this year, we added the domain of europarl sessions and observed that our preferred style of bulleted minutes is in sharp contrast with the officially released summaries. our annotators liked our style better and the official references did not score well. automin 2023 also responded to the emergence of large language models, applying them both to the task of summarization itself, as well as to the task of assigning scores to the minutes we received from the participants. we confirmed that gpt-4 excels at summarization, surpassing the reference in manual document-level scoring. on the other hand, the automatic evaluation carried out by asking gpt-4 to evaluate the minutes is unreliable. gpt-4 does not distinguish between the different qualities (adequacy, fluency, grammaticality, relevance, topicality) and reflects too much the grammaticality; it also prefers its own outputs. a rather surprising result this year is that the very simple rouge is very much in line with manual ranking of system pairs. (rouge-1 for adequacy and relevance, rouge-l for fluency and grammaticality). the complex metrics did not provide any substantial benefit or performed simply worse. we are again trying to secure funding for a future round of automin, aiming primarily at a reliable assessment on whether the gist of the meeting has been well preserved in the minutes, and on a strategy for discovery and scrutiny of summarization errors.