Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization? (2024)

Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities to solve a wide range of tasks without being explicitly fine-tuned on task-specific datasets. However, deploying LLMs in the real world is not trivial, as it requires substantial computing resources. In this paper, we investigate whether smaller, Compact LLMs are a good alternative to the comparatively Larger LLMs to address significant costs associated with utilizing LLMs in the real world. In this regard, we study the meeting summarization task in a real-world industrial environment and conduct extensive experiments by comparing the performance of fine-tuned compact LLMs (FLAN-T5, TinyLLaMA, LiteLLaMA, etc.) with zero-shot larger LLMs (LLaMA-2, GPT-3.5, PaLM-2). We observe that most smaller LLMs, even after fine-tuning, fail to outperform larger zero-shot LLMs in meeting summarization datasets. However, a notable exception is FLAN-T5 (780M parameters), which achieves performance on par with zero-shot Larger LLMs (from 7B to above 70B parameters), while being significantly smaller. This makes compact LLMs like FLAN-T5 a suitable cost-efficient LLM for real-world industrial deployment.

## introduction the instruction following capabilities have made it possible for llms to achieve impressive performance in zero-shot scenarios @xcite @xcite , which has also led to an increase in using llms to solve realworld problems. for instance, in tasks like meeting summarization, llms have been widely utilized in recent times due to their impressive zero-shot performance @xcite . however, despite the effectiveness of llms in summarization, deploying llms in the real world to generate meeting summaries would also lead to an increase in production costs. while fine-tuning smaller language models @xcite , such as bart @xcite , pegasus @xcite , etc. led to state-of-the-art results across various summarization datasets, these models require large annotated datasets for model training, which are often difficult to obtain in real-world business scenarios. moreover, these smaller language models also do not have instruction-following capabilities @xcite . thus, they cannot be trained to properly follow specific instructions if there is a change in user requirements. @xcite is an llm proposed by openai which is widely considered the bestperforming llm currently available @xcite . gpt-4 generated responses are also used to fine-tune various llms that are significantly smaller in size in comparison to it @xcite . since using the gpt-4 api significantly increases the api usage cost @xcite , it is often not practical to use in real-world scenarios. in this regard, this paper studies whether compact/smaller llms can be fine-tuned in a way that can mimic the performance of gpt-4, while also significantly reducing the deployment cost of using llms in production for meeting summarization. more specifically, this paper aims to provide a comprehensive analysis of various smaller and larger llms, which includes larger llms like gpt-3.5 (i.e., chatgpt foot_0 ), palm-2 (google, 2023), llama-2 @xcite , as well as smaller llms like @xcite , tinyllama @xcite , etc. our experimental results show that most smaller llms, even after fine-tuning, fail to outperform larger zero-shot llms in meeting summarization datasets. however, a notable exception is a fine-tuned flan-t5-large, which achieves performance on par with much larger llms (from 7b to more than 70b) used in zero-shot settings, while being significantly smaller. this makes smaller llms like flan-t5 a suitable cost-efficient llm for real-world deployment. our extensive experiments would give insights into the cost-effective utilization of llms for summarizing business meeting transcripts. below, we summarize our major contributions in this paper: llms and compare their performance with larger llms in several meeting summarization datasets to address several limitations of using llms in the real world. ## related work fine-tuning language models @xcite @xcite based on the transformer architecture @xcite has led to state-of-the-art performance in various summarization datasets. since these transformerbased language models require domain-specific fine-tuning for best results, obtaining in-domain labeled data in real-world settings is not trivial. however, the notable zero-shot abilities of llms in summarization @xcite have attracted attention for their potential use in practical summarization systems where in-domain labeled datasets are not available. while zero-shot llms have demonstrated impressive performance in tasks that lack large annotated datasets @xcite @xcite , utilizing llms in the real world also has several limitations. for instance, gpt-4 is currently regarded as the best-performing llm in terms of various evaluation benchmarks. however, the api cost of using gpt-4 is significantly higher than of any other llms @xcite . while fine-tuned versions of less expensive closed-source llms could reach performance comparable to gpt-4, using fine-tuned versions of these llms for inference significantly increases the api cost foot_1 . since these closed-source llms are only available through apis, they pose potential privacy risks. to mitigate the above issues, various opensource llms have been proposed @xcite @xcite . some of the major advantages of using open-source llms are: (i) they are available for in-house deployment, (ii) they can be fine-tuned to achieve performance comparable to larger closed-source llms, and finally, (iii) the inference cost of using both zero-shot and finetuned versions are the same. thus, open-source llms could be a good alternative that addresses the limitations of closed-source llms. however, deployment of the open-source llms in a way that ensures customer satisfaction, i.e., high accuracy with low latency, would require expensive computing resources such as powerful gpus with large memory capacity. in addition, fine-tuning larger llms also requires scarce and costly computing resources which may not be available in many industries. while various optimization techniques @xcite like low-bit quantization @xcite , parameter-efficient fine-tuning (hu et al., 2021), etc. have been proposed recently to address the computational limitations, they often come with other issues, such as a drop in accuracy and an increase in latency. in this paper, we aim to address these issues by studying whether we can fine-tune smaller llms with instruction-following capabilities to mimic the performance of larger llms such as gpt-4 while ensuring low latency with minimized inference cost. ## our methodology the objective of this research is to study whether instruction-following llms that are smaller in size can be effectively utilized in a real-world system for meeting summarization to ensure performance comparable to the state-of-the-art larger llms while minimizing the inference cost. for this purpose, we select llms that have fewer than 2b parameters as the targeted compact llms for performance analysis. moreover, in real-world meeting summarization scenarios, users may have different requirements for the llms. for instance, some users may prioritize meeting summaries that are detailed and comprehensive, whereas others may prefer the meeting summaries to be short and concise. in such cases, the instruction following capability is important for the llms that would be deployed in production such that they can fulfill variations in user demands. therefore, in this paper, we also evaluate the performance of llms based on a diverse set of instructions to generate (i) long summary, (ii) medium length summary, and (iii) short summary. we follow the work of @xcite for prompt construction and use their summarization via truncation approach for each type of instruction. below are the examples of the prompts for each case. long: generate a long and descriptive summary of the following conversation. medium: generate a summary of the following conversation. short: generate a very short and concise summary of the following conversation. ## experiments in this section, we first present our models along with their implementation details. next, we demonstrate the datasets we used for evaluation. finally, we demonstrate our experimental findings. ## using llms in real-world systems to deploy llms in the real world, we study the following aspects: cost/gpu and inference speed. cost/gpu: as of the time of writing this paper, the pricing 14 in openai for the gpt series models are as follows: the 4k context version of gpt-3.5 that we use costs 0.0015@xmath0 per 1k output tokens. meanwhile, for palm-2, the pricing 15 in google cloud is 0.00025@xmath1 per 1k output characters. approximately, 1 token is con- ## conclusion in this paper, our extensive study involving various llms led to several key insights on building an efficient meeting summarization system for realworld usage. while most larger llms usually outperform their smaller counterparts, we find that flan-t5-large is an exception in this regard. on our in-domain dataset, with only 780m parameters, flan-t5-large not only outperforms larger zeroshot llms, but also it achieves comparable performance with larger fine-tuned llms. this makes flan-t5-large more suitable for real-world usage, especially in scenarios where the meetings are not too long. since the performance of flan-t5-large is still quite below in comparison to other larger llms on qmsum-i dataset that has longer meetings, future work should investigate the performance of flan-t5 by applying various chapterization techniques @xcite .