Tell me what I need to know: Exploring LLM -based (Personalized) Abstractive Multi-Source Meeting Summarization (2024)

Abstract: Meeting summarization is crucial in digital communication, but existing solutions struggle with salience identification to generate personalized, workable summaries, and context understanding to fully comprehend the meetings’ content.Previous attempts to address these issues by considering related supplementary resources (e.g., presentation slides) alongside transcripts are hindered by models’ limited context sizes and handling the additional complexities of the multi-source tasks, such as identifying relevant information in additional files and seamlessly aligning it with the meeting content.This work explores multi-source meeting summarization considering supplementary materials through a three-stage large language model approach: identifying transcript passages needing additional context, inferring relevant details from supplementary materials and inserting them into the transcript, and generating a summary from this enriched transcript.Our multi-source approach enhances model understanding, increasing summary relevance by ~9% and producing more content-rich outputs.We introduce a personalization protocol that extracts participant characteristics and tailors summaries accordingly, improving informativeness by ~10%.This work further provides insights on performance-cost trade-offs across four leading model families, including edge-device capable options.Our approach can be extended to similar complex generative tasks benefitting from additional resources and personalization, such as dialogue systems and action planning.

## introduction meeting summaries play a key role in professional settings @xcite @xcite , serving as references, updates for absentees, and reinforcements of key topics discussed. major virtual platforms (e.g., zoom 1 , microsoft 1 https://www.zoom.com/en/ai-assistant teams foot_0 , google meet foot_1 ) offer summarization systems already, highlighting their importance. current methods rely solely on transcripts @xcite and generate generic summaries, often failing to contextualize long discussions' content @xcite and to tailor information to individual preferences and productivity requirements. as such, there is a need for improved model comprehension and personalization in meeting summarization. additional content-related sources can be considered during the summarization process to enhance model comprehension, turning the task into multi-source summarization. however, traditional approaches of appending documents to transcripts are often limited by model context sizes (e.g., led @xcite , dialogled @xcite , llama @xcite ). while hierarchical @xcite and graph-based methods @xcite have been explored, they struggle with handling redundant or contradicting information and maintaining coherence throughout the additional input @xcite . recent advancements in question-answering, which face a conceptually close challenge when answering a query considering an arbitrarily large amount of sources @xcite , suggest retrieval augmented generation (rag) @xcite as a promising solution that efficiently filters relevant information from extensive document collections and uses language models to perform a task such as information inferring. as rag is not designed to identify contextual gaps in transcripts, a targeted approach is needed to pinpoint specific information requirements within the transcript, using rag for focused retrieval. otherwise, language models, already challenged by meeting summarization complexities (e.g., omission, repetition, irrelevance) red indicates the additional personalization modules. @xcite , may become overwhelmed by considering relevant documents in the summarization process. our proposed multi-source summarization pipeline (figure 1 ) mimics a human summarization process and distributes the inherent tasks of multi-source summarization across multiple large language models (llms) and an rag framework. our three-stage process, informed by multi-hop question-answering techniques @xcite and recent research in meeting summarization @xcite , includes: (1) gap identification where an llm analyzes the transcript, identifies context-deficient passages, and generates questions about missing information; (2) information inferring, using rag to retrieve relevant documents, process these questions, and insert inferred answers into the transcript; and (3) enriched transcript summarization, where a final llm generates an abstractive summary. personalized summaries are valuable in professional settings, as participants often write notes focused on points relevant to their projects and knowledge @xcite . current research on personalization mainly explores post-processing techniques @xcite . we explore salience control and personalization by extending our multi-source meeting summarization pipeline to automatically understand the target audience from the transcript. inspired by @xcite , we incorporate an upstream llm to extract participant information such as personality traits, project interests and observed knowledge level in the transcript. these characteristics are used to tailor a summary based on the participant's needs according to the identified gaps. we evaluate our pipeline using ms-ami, a 125sample multi-source dataset based on ami @xcite . our approach improves informativeness (+0.18 points) and relevance (+0.40 points) compared to single-source summarization, outperforming multi-source summarization with simple document concatenation. the pipeline shows better contextual understanding and provides more in-depth, relevant information. our personalization protocol further enhances informativeness (+0.33 points over a simple personalized baseline) for target readers, tailoring the content to individual preferences. while using gpt-4 turbo foot_2 (openai, 2024) as our primary model, we also assess smaller models from phi @xcite , gemini @xcite , and llama families for practical deployment scenarios. overall, gpt4 offers superior performance at the highest cost, while phi-3 mini provides a cost-effective alternative with similar quality but requires additional robustness measures for personalization. our contributions are summarized as follows: the dataset and code ware available through huggingface and the project-accompanying github repository: https://github.com/fkirste/ emnlp2024-personalized-meeting-sum 2 methodology our multi-source rag-based summarization pipeline (figure 1 ) enriches meeting transcripts with inferred information from supplementary materials, turning the multi-source summarization into a single-source task. an optional personalization protocol tailors summaries to specific readers by extracting participant information from the transcript and providing this info as the target audience to the generating llms. leveraging llms' zero-shot capabilities @xcite , proven effective for meeting summarization @xcite , our approach is suitable for real-world applications lacking in-domain datasets. prompt templates are detailed in appendix f. ## dataset for our experiments, due to the lack of an established multi-source meeting summarization dataset, we introduce ms-ami, an adapted version of ami @xcite , comprising 125 staged business meetings with processed supplementary content (whiteboard drawings, slides, notes). using gpt-4o foot_3 for ocr and image description @xcite , and aspose foot_4 for document text extraction, we create a multi-source dataset compatible with language models. each meeting's data is compiled into a json file, preserving original structures. we remove 12 samples from the initial 137 meetings due to processing errors. dataset statistics are in ## final considerations this paper presents a three-step rag-based pipeline using multiple llm instances to abstractly summarize englisch business meeting transcripts, considering supplementary files. we also explored how to use personas extracted from transcripts to introduce personalization and preferences in summaries. key findings show incorporating supplementary sources improves summary quality by at least 0.31 over the baseline (single-source), with an additional 0.11 improvement when distributing multi-source challenges (identifying, inferring, and linking salient content) across multiple sources. persona-based personalization, using dynamically generated participant personas, enhances relevance by up to 0.44 compared to a baseline with only the target audience's role information. our zeroshot setup performs well with significantly smaller models than gpt-4 turbo, revealing that phi-3 mini 128k produces good-quality summaries under a low-resource environment. this study provides initial insights into multi-source and personalized meeting summarization using llms and rag systems, leaving the development of more sophisticated approaches, such as multi-agent discussions for retrieval and personalization, and the development of a dynamic function to identify the best amount of resources to consider to future work.