The economic trade-offs of large language models: A case study (2023)

Abstract: Contacting customer service via chat is a common practice. Because employing customer service agents is expensive, many companies are turning to NLP that assists human agents by auto-generating responses that can be used directly or with modifications. With their ability to handle large context windows, Large Language Models (LLMs) are a natural fit for this use case. However, their efficacy must be balanced with the cost of training and serving them. This paper assesses the practical cost and impact of LLMs for the enterprise as a function of the usefulness of the responses that they generate. We present a cost framework for evaluating an NLP model’s utility for this use case and apply it to a single brand as a case study in the context of an existing agent assistance product. We compare three strategies for specializing an LLM — prompt engineering, fine-tuning, and knowledge distillation — using feedback from the brand’s customer service agents. We find that the usability of a model’s responses can make up for a large difference in inference cost for our case study brand, and we extrapolate our findings to the broader enterprise space.

## introduction amidst increased automation, human agents continue to play an important role in providing excellent customer service. while many conversations are automated in text-based customer support, others are routed to human agents who can handle certain customer concerns more effectively. agents often handle multiple conversations at once, consulting customer account information and brand policies while maintaining these conversations. as agents are expensive to staff, many companies are seeking ways to make their work more efficient. liveperson's conversation assist, foot_0 illustrated in figure 1 , accelerates agents by automatically generating suggestions that the agent can either send, edit and then send, or ignore. conversation assist can both reduce agent response time and improve response quality, as a well-trained model may provide more consistent, higher quality responses than inexperienced agents or agents adversely impacted by external factors. these benefits lead to greater cost savings and increased customer satisfaction (csat) scores, not to mention providing a supervisory mechanism that is critical for brand control and model improvement. large language models (llms) are a natural fit for this technology, as they have achieved high performance on response generation tasks @xcite @xcite , but they are expensive to train and serve. for example, the inference cost for each response using a distilled gpt-2 model and an nvidia a100 gpu is ¢.0011, foot_1 while the inference cost using the gpt-3-based davinci model through openai's api is ¢1.10 (openai, 2023b). 3 llm economics and enterprise applications are highly fluid. first, individual partnership agreements may differ from the published api cost, and the rapid pace of innovation in the space will necessarily impact the cost of training and serving these models. second, as brands vary widely, a useful agent assistance model must be customized to the brand's use case and performance requirements. we propose a simple and flexible cost framework that can be applied to various llm and brand scenarios. this framework, expected net cost savings (encs), combines the probability and cost savings of an agent accepting or editing a response with the cost of generating the response. encs can be applied at the message level or in the aggregate. with one brand as a case study, we explore encs with various methods of model customization. using feedback from the brand's customer service agents, we evaluated fine-tuning, prompt engineering, and distillation to adapt and optimize gpt-2 @xcite , gpt-3 @xcite , and cohere @xcite . these strategies can lead to an agent usage rate of 83% (including both direct use and editing) and an annual cost savings of @xmath024,000 per year with any of the models, reducing the upfront r&d cost would be critical. on the other hand, a large enterprise brand who will save $950,000 per year over 20 million messages, will break-even on any r&d cost fairly quickly. as a model with lower inference cost will offset high r&d cost more quickly and lead to more savings over a longer period of time, inference cost is a much more important factor for a brand with high traffic. in appendix k, we provide a detailed example of the impacts of these costs. it is also worth noting that when choosing between in-house and third-party models, the difference in r&d and maintenance cost may not be as significant as one might expect. while an in-house model requires up-front investment to train and serve, openai and cohere's llms at the time of writing require a fair amount of effort to prompt engineer for the best performance and these prompts should be customized to some degree for different brands and scenarios. from a maintenance perspective, we similarly find that while an in-house model must be refreshed, prompts must also be redesigned as third-party providers update and release new models. brands might also wish to consider factors that are not accounted for in this framework. some brands would prefer to use an in-house model so that they can retain control over their data and protect their customer privacy by limiting access of their data to third-party vendors. an in-house model also provides more control over the model's suggestions, as well as control over when the model is updated or deprecated. especially as technology develops, models become less expensive to train, and the performance of open-source models improves, these factors may carry even more weight. ## conclusion in this case study, we demonstrated the utility of llms for agent assistance products, exploring 3 model adaptation strategies across 11 model configurations. based on feedback from real customer service agents, we found that bigger is not always better, as the distilled gpt-2 model resulted in greater cost-savings than gpt-3, despite lower quality responses, because, at the time of writing, its inference cost is so much lower. these results empower near-term decision-making for integrating models like these into production. however, with the rapidly shifting nlp landscape, a framework to assess the cost benefits of new technologies is critical to facilitate decisions about integrating them into products. the flexible framework presented in this paper, encs, enables nlp practitioners to invest in innovations that lead to tangible business benefits. we found that for this product, the impact of model quality far outweighs inference cost, pointing to the importance of continuing to push the state of the art, while considering practical expense. this framework empowers the nlp community to invest in the most cost-effective technology for their specific needs, even as that technology, its capability, and its pricing evolve.