Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective (2023)

Abstract: This paper studies how to effectively build meeting summarization systems for real-world usage using large language models (LLMs). For this purpose, we conduct an extensive evaluation and comparison of various closed-source and open-source LLMs, namely, GPT-4, GPT-3.5, PaLM-2, and LLaMA-2. Our findings reveal that most closed-source LLMs are generally better in terms of performance. However, much smaller open-source models like LLaMA-2 (7B and 13B) could still achieve performance comparable to the large closed-source models even in zero-shot scenarios. Considering the privacy concerns of closed-source models for only being accessible via API, alongside the high cost associated with using fine-tuned versions of the closed-source models, the opensource models that can achieve competitive performance are more advantageous for industrial use. Balancing performance with associated costs and privacy concerns, the LLaMA-2-7B model looks more promising for industrial usage. In sum, this paper offers practical insights on using LLMs for real-world business meeting summarization, shedding light on the trade-offs between performance and cost.

## introduction meetings are a widely used method for collaboration, with 55 million meetings occurring each week in the usa @xcite . with the rise of remote work, meetings have become even more crucial. while the widespread use of video conferencing software has made it easier to record meetings, the huge number of meetings makes it challenging to keep up with the information exchanged during them, emphasizing the need for automated methods of accessing key information. to address this issue, a summarization system that generates text summaries from meeting transcripts can be highly beneficial. however, the state-of-the-art neural summarization models @xcite @xcite require large domain-specific summarization datasets for model training, which is difficult to obtain in real-world industrial scenarios due to the lack of domain-specific annotated data. recently, the impressive capability of llms to solve a wide range of tasks even in zero-shot scenarios @xcite @xcite has drawn a lot of interest among practitioners to apply llms to solve real-world problems. however, despite the effectiveness of llms across several tasks, there is a scarcity of comparative analysis between llms in long conversational data, especially in tasks such as meeting summarization. thus, an extensive evaluation of llms in long meeting transcripts is an important step for the development of a real-world meeting summarization system that leverages llm technologies. in this regard, this paper aims to provide a comprehensive analysis of various llms, which includes closed-source models like gpt-3.5 (i.e., chatgpt foot_0 ), @xcite , and palm-2 @xcite , and two models (7b and 13b) of an open-source llm: llama-2 @xcite . our investigation focuses not only on the summarization quality of these models but also considers the cost-effectiveness and computational demands, providing a practical perspective to use such models in the real world. our experimental results show that while most closed-source models generally achieve better performance in meeting summarization datasets, the open-source llama-2 models still achieve comparable performance while being significantly smaller. for this reason, an extensive model like gpt-4, despite its marginally superior performance, is deemed not as cost-effective due to its substantially higher computational requirements, alongside posing privacy risks since such closed-source models are only accessible via api. furthermore, using fine-tuned versions of these closed-source models also significantly increases the api usage cost foot_1 . thus, the trade-off between performance, cost, and computational demands makes open-source models like llama-2 more favorable for industrial deployment. the insights from our research shed light on the practicalities of using llms for summarizing meeting transcripts. thus, providing valuable guidance for similar industrial applications. considering various factors, we employ llama-2-7b in a real-world setting to generate summaries from automatic speech recognition (asr)-generated transcripts @xcite @xcite @xcite of organizational meetings. below, we summarize our major contributions in this paper: ## related work in recent years, neural models based on the transformer architecture @xcite have led to state-of-the-art performance across various summarization datasets @xcite @xcite . however, these transformer-based models require domain-specific fine-tuning @xcite to achieve the optimum performance. thus, in real-world scenarios where in-domain labeled datasets are difficult to obtain, directly applying pre-trained transformers for zero-shot summarization may not yield great performance. in this regard, the impressive zero-shot performance of llms across various summarization datasets @xcite has drawn interest among practitioners to build real-world summarization systems via leveraging llms. in the real world, one interesting application of summarization is generating concise notes of long organizational meetings. though several datasets @xcite @xcite @xcite @xcite @xcite @xcite have been studied for the meeting summarization task in recent years, most of these datasets are not related to the business/organizational context. this makes these datasets less relevant for the evaluation of summarization models that require the generation of summaries from long organizational meetings. in this regard, some notable exceptions are the ami @xcite dataset, the icsi @xcite dataset, and the qm-sum @xcite dataset, as they consist of organizational meeting transcripts, contrary to most other datasets. since the development of a summarization system in a real-world industrial setting requires an extensive evaluation of the summarization model to ensure customer satisfaction, in this paper, we evaluate various llms in benchmark meeting summarization datasets. more specifically, we use the following datasets: ami, icsi, and qmsum. all these datasets are constructed from organizational meetings. we hope that our extensive evaluation of llms in these meeting summarization datasets will help researchers and industry professionals to harness the power of llms to build real-world meeting summarization systems. ## our methodology the objective of this research is to identify the most effective model to summarize organizational meetings that could be used in real-world applications in scenarios when in-domain labeled datasets are not available. therefore, in this paper, we study llms due to their impressive zero-shot capabilities. however, one major issue with existing llms is their limitation to handle long contexts @xcite . since organizational meetings are usually quite longer @xcite , the input limit restrictions of llms make it difficult to consider the whole transcript sequence. in this regard, we study how to handle the long input sequence issue in llms based on the following two approaches (an overview of our proposed approaches is also shown in figure 1 ): prompt: "summarize the following conversation: [truncated transcript]" (ii) summarization via chapterization: in our previous approach, the context of the whole transcript could not be given as input to the llms since only the truncated input sequence, consisting of the first n words is provided. while this approach is quite efficient in terms of cost, the performance of the summarization may not be optimal due to truncating the input sequence length. to address this issue, we propose summarization via chapterization. in this approach, we sequentially summarize every n words. here, we denote every n words as chapters and the summary generated in each chapter as chapter summary. afterward, we generate the final summary from these chapter summaries, either by concatenating the chapter summaries or by re-summarizing/re-writing the chapter summaries. to generate the summary of each chapter, we use the same prompt as we used in our previous approach of summarization via truncation. to generate the final summary from the chapter summaries, in addition to directly concatenating the chapter summaries, we investigate the following prompts: prompt (re-write): "rewrite the following text by maintaining coherency: [chapter summaries]" prompt (re-summarize): "summarize the following text: [chapter summaries]" ## experiments in this section, we first present our models along with their implementation details. next, we demonstrate the datasets we used for evaluation. finally, we demonstrate our experimental findings. ## using llms in real-world systems to deploy llms in the real world, we study the following aspects: cost and inference speed. cost: based on our experiments, we find that except for the palm-2 model, the closed-source llms usually outperform the open-source llama-2 models. while gpt-4 generally performs the best, it is also more costly. as of the writing of this paper, the pricing 7 in openai for the gpt series models are as follows: for gpt-4, the 8k context version costs 0.03@xmath0 per 1k output tokens, while for the 4k context version of gpt-3.5 that we use costs 0.0015@xmath1 per 1k output tokens. on average, it makes gpt-4 25 times more costly than gpt-3.5. meanwhile, for palm-2, the pricing 8 in google cloud is 0.0010@xmath2 per 1k tokens, making it about 2.5 times more costly than gpt-3.5. regarding llama-2, we were able to run it in a machine with 1 nvidia l4 gpu, while the llama-2-13b model was possible to run using 2 l4 gpus. however, using multiple gpus significantly increases the production cost. thus, in terms of real-world usage, llama-2-7b is more useful than llama-2-13b as we observe almost similar performance using these models across various datasets. inference speed: we also measure the inference speed of different llms. for this purpose, we collected 100 meeting transcripts from dialpad 9 that consist of real-world conversations. all the collected transcripts have at least 2500 words. based on our analysis using the summarization via truncation approach, we find that gpt-3.5 is the fastest, as it takes 2.5 seconds on average per transcript for inference, followed by palm-2 which takes about 3.2 seconds on average. while gpt-4 achieves the best performance in terms of rouge and bertscore metrics, it is also the slowest among the commercial closed-source llms since it takes about 11 seconds on average per transcript. we also benchmark the inference speed of llama-2-7b in gcp on a machine having 1 nvidia l4 gpu and find that it takes 15 seconds on average. deployment: while using the apis of closedsource models usually leads to faster inference speed, there are some drawbacks of using closedsource llms. for instance, businesses need to send their customer data using the 3rd-party api, which may lead to potential privacy risks. meanwhile, we also observe that these llm apis sometimes become too slow when the demand is high, leading to api request failure. by considering such cases of api failures, the average inference speed of gpt-4 is increased to 40 seconds per transcript (up from 11 seconds). thus, based on cost and inference speed trade-off, alongside the privacy concerns and the possibility of fine-tuning on in-domain datasets without additional deployment costs, llama-2-7b looks more promising for pro-8 https://cloud.google.com/vertex-ai/pricing 9 https://www.dialpad.com/ca/ duction usage. meanwhile, we can also leverage various model optimization @xcite techniques like quantization, pruning, distillation, etc. to further reduce the memory requirement as well as improve the inference speed. therefore, we select the llama-2-7b model for real-world usage and after further fine-tuning on our in-domain data, we successfully deploy it in our production environment in a machine having 1 nvidia l4 gpu. ## conclusion in this paper, our extensive study involving various llms led to several key insights on building a real-world meeting summarization system. while most closed-source llms usually outperform their open-source counterparts, striking a balance between cost and performance (while also addressing potential privacy risks) is crucial for practical, realworld deployment. this became evident in the case of gpt-4, which, while showing superior performance in most datasets, was considerably less costeffective. by considering the performance and cost trade-off, as well as the privacy concern, we deploy a summarization model based on llama-2-7b to generate live summaries of asr-generated meeting transcripts. this research thereby provides crucial insights and a clear perspective on deploying llms in real-world industrial settings. in the future, we will investigate the performance of llms by applying various optimization techniques.