Team Iterate @ A uto M in 2023 - Experiments with Iterative Minuting (2023)

Abstract: This report describes the development of our system for automatic minuting created for the AutoMin 2023 Task A. As a baseline, we utilize a system based on the BART encoder-decoder model paired with a preprocessing pipeline similar to the one introduced by the winning solutions at AutoMin 2021. We then further explore the possibilities for iterative summarization by constructing an iterative minuting dataset from the provided data, finetuning on it and feeding the model previously generated minutes. We also experiment with adding more context by utilizing the Longformer encoder-decoder model and finetuning it on the SAMSum dataset. Our submitted solution is of the baseline approach, since we were unable to match its performance with our iterative variants. With the baseline, we achieve a ROUGE-1 score of 0.368 on the ELITR minuting corpus development set. We finally explore the performance of Vicuna 13B quantized language model for summarization.

## introduction meeting minuting is the process of writing down the important contents discussed while reducing the overall length. it is generally necessary to create minutes to keep people who were not able to attend up to date and to have a reference to come back to. however, writing meeting minutes is a tedious process requiring a lot of concentration. additionally, most meetings lack a dedicated notetaker, therefore the additional cognitive load is placed on the meeting participants who are already under stress. since the covid pandemic hit, many meetings have shifted to the online space, and with the rise of the large language models, it is becoming technically possible to automate the tedious and taxing minuting process. numerous approaches for automatic minuting were shown at the first automin @xcite , most of them utilizing a pre-trained transformer model like @xcite or pegasus @xcite . such models however have their limitations, especially with their input size being constrained to 512 or 1024 tokens by the quadratic complexity of the attention mechanism. in our approach, we explored possible solutions to the issue of short context length, namely iterative summarization and the longformer model. we utilized a solution inspired by the winning one at automin 2021 by @xcite as a baseline. finally, we experimented with the new vicuna models, but we were unable to obtain the results by the task deadline. ## related works in automin 2021, the approaches with best results were of team hitachi @xcite and team abc @xcite . both of these teams solved the problem of limited model input length in a different way. team abc splits the transcript into fixed-size segments, preprocesses them with a rule-based system, then summarizes each segment separately using a bart model. they then filter the output to remove redundancies and concatenate the result. on the other hand, team hitachi utilize a segmenter based on the longformer architecture with a lstm recurrent network on top which assigns utterances to different topics. these topics are then summarized using a bart model and results are concatenated to form the final minutes. the approach from team hitachi scores slightly higher on adequacy while the system of team abc is ranked higher in fluency and grammatical correctness. notably, neither of these systems used the elitr minuting corpus data for training. we also list some notable systems that were not a part of the automin 2021. summ n by @xcite works by generating a coarse summary in multiple stages and then generating a final summary from them. it has a variable-length input as it can scale its number of stages. qmsum by @xcite utilizes a locate-then-summarize approach, which works by first locating parts of the transcript with a common topic and then summarizing them separately. in this, the approach is similar to team hitachi's. ## baseline system we use a baseline approach inspired by system of team abc from 2021. we use a pipeline with a bart model finetuned on the xsum @xcite and samsum datasets with a simple rule-based preprocessing system. the transcript is first cleaned of filler words and less common characters are removed to make the summary more fluent with the preprocessing code of @xcite . to satisfy the input length limitation of the bart model, the pipeline then splits the transcript into chunks of roughly 512 tokens. each of those chunks is then summarized into a separate bullet point. the resulting minutes are a concatenation of the chunk summaries. ## iterative approaches one of the biggest challenges for summarization transformer language models is the limited input length. this naturally limits the amount of context the model can process and therefore can severely interfere with the quality of the generated minutes, especially for conversations with a common topic that span several thousand tokens. there are approaches that try to counter this, notably the longformer mechanism, which modifies the attention mechanism to reduce the complexity, and others mentioned in section 2. for humans, a natural approach to creating meeting minutes is an incremental one. a notetaker listens to the conversation taking place and writes down the agreed-upon points, all the while keeping in mind what he has already noted. our intention was to imitate such a process. the summarization model would be fed a chunk of a transcript together with several previously generated minute points to both satisfy the input length constraint of the transformer models while providing the needed context for the minutes. ## evaluation and output samples commonly used approaches for automatic evaluation include rouge and bertscore, but these often fail to represent the real quality of a meeting minute, as they are unable to fully represent the informational content. we therefore fall back to a combination of manual evaluation (coarsely assessing the relevance, coverage and fluency of the generated minutes) and the automatic metrics of rouge and bertscore. we place most emphasis on the manual qualitative evaluation on the development set of elitr. we also ran automatic evaluations on the test and test2 sets. we found the baseline model to perform better than led in all the automatic metrics we computed, as shown in after the competition deadline, we also evaluated the outputs of the pipeline with the vicuna model prompted for meeting summarization. we found out it was similar to the the other models in bertscore but performed worse on rouge-1 precision and a bit better on rouge-1 recall, being more consistent across all the evaluation datasets. qualitatively, the result sample we examined seemed more informative than the output of our other models. had the results been available by the submission deadline, we would be submitting them. however, sometimes the outcome of the vicuna model was not summarization at all; sometimes it ignored the transcript completely and generated a generic outcome. this could be automatically filtered or, better, fixed with better prompting or finetuning. it should also be noted that the results from the vicuna model were on average considerably longer than of the baseline model, as also shown in we would like to point out that all the generated minutes differ in essence to the target minutes provided as a part of the dev set, as they often focus more on what is happening at the current time in the transcript instead of summarizing the outcomes and agreed upon points. we find this to be a major disparity that we believe will only be fixed with much increased context length. a comparison of samples of successfully generated summary points for the same transcript chunks can be found below. baseline: this week person11 is working on the collection is business for english. he has downloaded 1970 to 1980 pdf files which is extracted from the other3 thing of the office. 15000 sentences is crawled automatic using automatic clover for english, other3. he needs to clean all the data led: person11 is trying to record the call to see if they are able to summarize it automatically. they are trying to make it sound like they are discussing the issue of censorship. they need to apply the same type of adaptation for the nlp, project2 on the 12th and the 17th of february. they also need to adapt the vocabulary of the person7 talkage. they want to use english as english models. they will use it for the first dry run of a workshop on the monday seminar. vicuna person11 is working on a business project for other3, which involves cleaning and organizing a large amount of data in text format. person14 is collaborating with person6 on a language model for the project, and they are discussing how to use the model for organizing the data. ## conclusion although we were unable to pass the baseline with our approaches, we have several interesting findings.