Team Zoom @ A uto M in 2023: Utilizing Topic Segmentation And LLM Data Augmentation For Long-Form Meeting Summarization (2023)

Abstract: This paper describes Zoom’s submission to the Second Shared Task on Automatic Minuting at INLG 2023. We participated in Task A: generating abstractive summaries of meetings. Our final submission was a transformer model utilizing data from a similar domain and data augmentation by large language models, as well as content-based segmentation. The model produces summaries covering meeting topics and next steps and performs comparably to a large language model at a fraction of the cost. We also find that re-summarizing the summaries with the same model allows for an alternative, shorter summary.

## introduction we participated in the 2023 workshop on automatic minuting at inlg 2023 @xcite . we chose to focus our efforts on the technical meetings in task a, as it most closely aligns to a real-world use case of typical meetings. meeting summarization is a challenging task, for both technical and semantic reasons: on the other hand, meetings often follow an agenda, with specific topic being discussed sequentially. our model exploits this structure by breaking the meeting down into smaller chunks by topic. we also address the lack of data by generating artificial labels using a large language model (llm). ## related work submissions to the previous automin workshop @xcite used different approaches, including extractive summaries @xcite and abstractive summaries using pre-trained language models. the teams using neural abstractive models used different approaches to dealing with the long input length in the dataset, including truncating @xcite , chunking by length @xcite and topic segmentation @xcite . we found the summary quality to be heavily dependent on the quality of the segmentation and found that purely length-based segmentation leads to worse summaries than topicbased segmentation. because of the difficulties pointed out above, all of the best-performing systems in the previous automin did not make use of the training data at all. ## model architecture our main summarization model is derived from bart-large @xcite . because this model was only pretrained with 1024 position encodings, sequences longer than 1024 tokens cannot be processed by this model. while we have experimented with long-input models such as longformer @xcite and big bird (zaheer et al., 2020), we found those models not to perform better overall. in order to summarize a meeting, which is generally much longer than 1024 tokens, we use content vector segmentation (cvs) @xcite to break the meeting down into smaller segments. cvs uses a dynamic programming algorithm to find a globally optimal segmentation as determined by a scoring function. it operates on sentence embeddings, which we generate with a pre-trained sentence-bert @xcite . we observed that introducing more splits would almost always increase the score, which would almost always result in the meeting being split into the maximum number of segments. in order to allow for a variable number of segments per meeting, we introduced a cvs score improvement threshold of 1.2. only solutions which improve upon the current best by at least the threshold are considered an improvement. the desired segment length was tuned with the 1024-token limit of bart in mind, although the algorithm may still produce segments longer than that, in which case they are truncated. ## processing the organizers provided a training set of 84 meetings annotated with summaries (some multiple times) by human annotators @xcite . however, we found this training data unsuitable for the purposes of training our model. the annotation style is inconsistent across annotators and while we could apply the cvs segmentation to the source, we would need to create an alignment between source segments and summary sentences. we therefore opted to create pseudo-references by using a large language model. the automin data has been deidentified, meaning all names of persons, organizations and projects are replaced by "person[n]", "organiza-tion[n]" and "project[n]", respectively. we used models that had not been trained on such deidentified data, so we converted to pseudonyms instead: we replaced the "person[n]" tags by arbitrarily chosen english names, respecting the gender information present in the training set (for the test set, no such information is available, so we used male names for all participants). for projects and organizations, we randomly generated three-and four-letter acronyms, respectively, making sure not to use ones that represent well-known organiza-tions or projects. note that we made no attempt to determine the actual identities of the persons in the meetings (as required in the usage agreement), we simply replaced the generic tags by randomly chosen names. before submitting the results, we restored the original deidentification. for the llm, we used text-davinci-003 from openai, which is from the gpt-3.5 series. 1 we used two priming examples from the training set (see appendix a), which we annotated by hand to demonstrate the style that we wanted. we then sampled with a temperature of 0.7. we found that the outputs are quite sensitive to the choice of priming example. when choosing two examples that are about a similar topic, something similar to overfitting occurs and the generated summaries will always be about the same topic as the examples, regardless of source content. the priming examples allow us to influence the summary length, style and grammar to a certain extent, but not fully. for example, we would have preferred summaries to be completely in the simple present tense (as the priming examples), but in many generated summaries, the first sentence was in the present progressive, even when giving more explicit instructions to the llm. we reviewed model outputs on the automin 2021 development set. we generated one pseudo-reference for each segment of the training set for a total of 704 examples. we then fine-tuned a bart-large model that was previously trained on xsum @xcite and samsum @xcite foot_1 for 5 epochs using adam with a learning rate of 0.0002. we applied the model using beam search with beam size 4. the resulting model performs similarly to gpt-3 while being much smaller. the final summary for a meeting is the concatenation of all segment summaries. we also provided an alternative, shorter summary by applying the same bart model as above to the concatenated summaries, producing a summary of summaries (sos). an overall similar architecture was proposed by @xcite , which was developed contemporaneously to ours. also, @xcite also used a segment-then-summarize approach in automin 2021, although they did not use contentbased segmentation, llm pseudo-labels, or a summary of summaries. ## summary style in choosing our priming examples for the llm, we made a deliberate decision not to follow the summary style from the automin data. while this may cause our results to be unfairly rated by automatic metrics, we did not feel that the style of summary from the data was ideal for two reasons: first, full sentences in simple present tense aligns the data more closely to the samsum pretraining task, which we hypothesized to help with task transfer, especially given the small training set. second, the summaries from automin often omit attribution, i.e. they do not state who gave a particular piece of information. we feel that it is important for a summarization model to always attribute statements in its output. unless it is a very sophisticated system, the model has no way of distinguishing when a statement in the source represents a fact and when it represents an opinion. without attribution, a statement in the summary becomes a factual statement by the summarization model, which may not reflect reality. furthermore, if the summary states an opinion as fact, it could be misunderstood as the model actually holding that opinion. we therefore choose to alter the summary style to always include attribution and use full sentences in the simple present tense.