Query- OPT : Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization (2024)

Abstract: This work focuses on the task of query-based meeting summarization in which the summary of a context (meeting transcript) is generated in response to a specific query. When using Large Language Models (LLMs) for this task, a new call to the LLM inference endpoint/API is required for each new query even if the context stays the same. However, repeated calls to the LLM inference endpoints would significantly increase the costs of using them in production, making LLMs impractical for many real-world use cases. To address this problem, in this paper, we investigate whether combining the queries for the same input context in a single prompt to minimize repeated calls can be successfully used in meeting summarization. In this regard, we conduct extensive experiments by comparing the performance of various popular LLMs: GPT-4, Gemini, Claude-3, LLaMA2, Mistral, Phi-3, and Qwen-2 in single-query and multi-query settings. We observe that the capability to reliably generate the response in the expected format is usually limited to closedsource LLMs, with most open-source LLMs lagging behind (except Mistral). We conclude that multi-query prompting could be useful to optimize the inference costs by significantly reducing calls to the inference endpoints/APIs for the task of meeting summarization.

## introduction one key advantage of recent llms is their impressive instruction-following capabilities even in zero-shot scenarios (without fine-tuning on a particular task) @xcite @xcite . this instruction-following ability of llms has also led to an increase in utilizing llms for many real-world use cases @xcite . however, despite their impressive performance, deploying llms in the real world is not trivial, with one major obstacle being significant inference costs. thus, optimizing the inference cost while ensuring high accuracy and efficiency is important for practical applications. although several optimization techniques @xcite , such as knowledge distillation, posttraining quantization, etc. are utilized to minimize the cost associated with llms in production, these techniques cannot be applied to the closedsource llms like @xcite or gemini @xcite . for open-source llms @xcite , these techniques may come with different trade-offs. for instance, while quantization @xcite can reduce the gpu requirement and make it possible to do llm inference on less expensive hardware, it may also result in slower inference speed and reduced accuracy. another limitation is that the cost foot_0 associated with llm inference depends on the number of tokens processed by llms, which is true for both closed-source and open-source llms. this issue can be addressed by reducing either the context size or the number of calls to the inference endpoints. for the task of query-focused meeting summarization, the latter is highly preferable since there can be many queries that may require the full context of long meeting transcripts and so truncating the context size could lead to a drop in performance. in this paper, we study how we can minimize calls to the llm inference apis/endpoints in the meeting summarization task by optimizing the prompts. more specifically, we investigate whether we can combine the queries for the same context in a single prompt to minimize calls to the same inference endpoints for different query-context pairs. for this purpose, we created a new version of the qmsum dataset @xcite for the queryfocused meeting summarization task by combining all queries for the same context in a single prompt. we conduct extensive experiments with several open-source and closed-source llms and compare their performance in both single-query and multi-query versions of the dataset. our experimental results show that only certain closed-source llms were able to reliably answer all the queries given in a single prompt in the required format in the multi-query setting. meanwhile, most opensource llms, even after fine-tuning, fail to properly follow multi-query instructions to generate the response in the requested format. we also find a similar trend in zero-shot scenarios in some larger closed-source llms. our extensive experiments demonstrate the capabilities and limitations of different llms in following multi-query instructions in meeting summarization. this gives strong insights into utilizing llms in real-world settings to minimize the inference cost for similar applications. our major contributions are as follows: ## related work the impressive instruction-following capabilities of llms have led to their wide adoption in the real world for various tasks, which includes generating summaries from meeting transcripts @xcite . however, in many scenarios, users may require extracting other information from the transcripts rather than a generic summary of the meetings. in such cases, one straightforward way is to call the llm inference api/endpoint for the given query-transcript pair. however, this approach is not cost-effective, since the same transcript for a different query would be given as input again to the llm in different calls. thus, it will lead to a non-optimal usage cost for processing the same tokens in a transcript multiple times. one possible solution in this regard could be combining the queries in a single prompt, similar to the work of @xcite where they evaluated chatgpt foot_1 (i.e., gpt-3.5) in the open-domain question-answering task in about 100 samples from natural questions @xcite and webquestions @xcite datasets. while their evaluation shows that instruction-following llms like gpt-3.5 can respond to multiple queries in a single prompt, they did not investigate the following research questions: to address the above questions, in this paper, we conduct a comprehensive evaluation of popular closed-source and open-source llms in the qm-sum @xcite dataset for query-focused meeting summarization task to investigate their performance in following multi-query instructions to extract information from long conversations. ## our methodology in this section, we present our overall methodology to evaluate the multi-query instruction capabilities of llms. below, we describe our dataset construction procedure, evaluation approach, and the models used in these experiments. ## experimental results in this section, we present our experimental findings in the multi-query setting to investigate the following: (i) llms capability to generate responses in the expected format, (ii) effectiveness when compared with single-query settings, (iii) effects of finetuning, (iv) qualitative evaluation of the generated summaries, (v) performance robustness, and (vi) usage cost analysis. ## conclusion in this paper, our experimental findings involving various llms led to several key insights on building an efficient real-world query-focused meeting summarization system. while most closed-source llms demonstrate superior performance in the multi-query setting, the open-source mistral model with only 7b parameters performs on par with the closed-source llms, while also outperforming many of them. the relatively higher performance in terms of automatic metrics in the multi-query setting, alongside significant optimization of cost and inference latency would open up the opportunities to utilize multi-query prompts in real-world industrial settings. in the future, we will study how to utilize tiny llms @xcite similarly in the multi-query setting in the task of meeting summarization.