Team Synapse @ A uto M in 2023: Leveraging BART -Based Models for Automatic Meeting Minuting (2023)

Abstract: This paper describes the approach we followed for our submission to the Second Run of the Automatic Minuting Shared Task. Our methodology centers around employing BART-based models fine-tuned on diverse summarization corpora. The segmented meeting transcripts are fed into the models, generating summaries that are subsequently combined and formatted into the final meeting minutes.

## introduction the covid-19 pandemic has expedited digital transformation across industries, significantly impacting the conduct of meetings. with the restriction of physical gatherings, online meetings have emerged as the primary mode of communication and collaboration. this shift towards virtual meetings has highlighted the crucial need for automatic summarization of meeting transcripts. by harnessing the power of natural language processing (nlp), organizations can optimize their virtual collaboration, ensuring accurate documentation, streamlined processes, and enhanced information management. this paper presents our endeavor to develop a robust system for generating minutes from meeting transcripts, undertaken as part of the second run of the automatic minuting shared task @xcite . the development of this system for automatic minuting has been influenced by previous research in the field, serving as the basis for our work. in particular, we draw inspiration from the research which pioneered the use of bart summarization models for meeting summarization tasks @xcite . we begin by looking at the related works from the previous iteration of the automin shared task @xcite . then we provide a concise overview of the datasets utilized in the task, followed by a comprehensive description of the system architecture we implemented. the system overview encompasses detailed explanations of the pre-processing steps, the conducted experiments, and the post-processing techniques applied to refine the generated minutes. subsequently, we present our results and discuss potential avenues for improving the performance of our system. ## related work automatic meeting summarization is a relatively new use case compared to the traditional task of summarizing text. the first edition of the automin shared task @xcite provides valuable insight into the work done in this area and explores numerous methods with which the participants approached the task. the use of pre-trained language models, especially transformer-based architectures like bart @xcite , t5-base @xcite , and gpt-2 (garg and singh, 2021) was a prominent approach. these models were then fine-tuned on the task-specific dataset to improve performance. one approach used for multilingual summarization involved translation from czech to english, generating the minutes in english and then translating the results back to czech @xcite . incorporating other techniques such as coreference resolution and dialogue partitioning during pre-processing @xcite , syntactic phrase extraction, redundant word deletion, and vectorization with tf-idf scores @xcite attempted to enhance the quality of generated summaries. argumentation mining techniques were utilized @xcite to improve coherence and internal structure, highlighting the importance of organizational and contextual coherence in meeting minutes. ## dataset description we participated in task a of the automin 2023 shared task, the goal of which was to generate minutes from meeting transcripts. the task runs in two languages, english and czech, and separate meeting corpora were available for both languages. the first edition of the automin shared task @xcite used the elitr minuting corpus @xcite . in addition to that, this year, a new meeting corpus eu-roparlmin created from the european parliamentary debates was also made available to the participants for training. since we participated only in the minuting of english meeting transcripts, we will only describe the datasets corresponding to english. the elitr minuting corpus consists of 84, 36, and 12 transcript-minute instances for train, dev, 1 and test sets, respectively. the transcripts, which are text files, contain asr outputs of the meetings and therefore are not very refined. each transcript has one or more corresponding minutes generated in a specific format with details like the date, attendees, the purpose of the meeting, the summary (in bullet points), and the name of the annotator. some transcripts have additional information on the gender of the attendees and the alignment of the transcript and minutes. the europarlmin consists of 2065, 187, and 242 transcript-minute instances for train, dev, and test sets, respectively. each dataset contains directories labeled by the date of the session. each directory contains the transcripts and minutes of one or more chapters or sections of the meeting. chapters are split further into parts. during the europarlmin transcript revision, grammar and stylistic corrections were already incorporated, resulting in reduced cleaning requirements on our part compared to the elitr minuting corpus. the minutes follow a paragraph-style format and contain only a summary of the transcript. they do not report other details like date, list of attendees, etc., which were present in the minutes of the elitr minuting corpus. ## system overview in this section, we provide a comprehensive overview of the system architecture implemented for the automatic minuting of meeting transcripts. 1 the dev set also includes the two test sets from the first run of automin shared task. we begin by presenting the pre-processing steps undertaken to prepare the input data for the summarization models. next, we delve into the details of the experiments conducted, focusing on the fine-tuning of the bart summarization model @xcite on meeting summarization corpora. we then discuss the post-processing steps employed for the generation of concise minutes as the final output of our system. figure 1 shows the endto-end functioning of our system. the source code can be found at https://github.com/klesnkri/ automin-2023-team-synapse . 4.1 pre-processing as a first step, we pre-process the transcript data by splitting them into speaker-utterance pairs and normalizing the utterances. for elitr minuting corpus, we apply a series of text normalization techniques, including the removal of tags (e.g., <cough/>, <laugh/>, <censored/>) and asr stopwords and errors, deletion of punctuation at the start of sentences, removal of consecutive duplicate tokens and punctuation, and sentence normalization. figure 2 and figure 3 illustrate the steps involved in the pre-processing of elitr minuting corpus and an example of the raw text before and after pre-processing, respectively. figure 3: example of elitr minuting corpus preprocessing similarly, for europarlmin, we remove lines that were not speaker utterances, introduce the per-son entity so the speaker-utterance pairs have the same format as in elitr minuting corpus, remove punctuation, language codes, and other irrelevant information from the start of utterances, and normalize whitespaces. see figure 4 , for a detailed diagram. ## results we evaluate the generated summaries using the rouge-1, rouge-2, and rouge-l metrics on the development data. the automatic evaluation results are summarized in according to our experiments, the meet-ing_summary model with a segment length of 768 tokens is the most suitable for generating elitr minuting corpus minutes, while the bartlarge-cnn-samsum model with a segment length of 1024 tokens is the most appropriate for generating the europarlmin minutes. ## conclusion in this paper, we presented our approach for automatic minuting, focusing on fine-tuning the bart summarization model using meeting summarization corpora. for elitr corpus, we chose the meeting_summary model with a segment length of 768 tokens, and for europarlmin corpus, we settled on the bart-large-cnn-samsum model with a segment length of 1024 tokens. while our current approach yields promising results, there are areas for future improvements, such as exploring dialogue summarization models like dialoglm @xcite , which show potential in addressing the challenge of processing lengthy meeting transcripts. our intention is to refine our system continuously and advance the field of automatic minuting, ultimately providing more accurate and coherent meeting minutes.