Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models (2023)

Abstract: Prompting Large Language Models (LLMs) performs impressively in zero- and few-shot settings. Hence, small and medium-sized enterprises (SMEs) that cannot afford the cost of creating large task-specific training datasets, but also the cost of pretraining their own LLMs, are increasingly turning to third-party services that allow them to prompt LLMs. However, such services currently require a payment per call, which becomes a significant operating expense (OpEx). Furthermore, customer inputs are often very similar over time, hence SMEs end-up prompting LLMs with very similar instances. We propose a framework that allows reducing the calls to LLMs by caching previous LLM responses and using them to train a local inexpensive model on the SME side. The framework includes criteria for deciding when to trust the local model or call the LLM, and a methodology to tune the criteria and measure the tradeoff between performance and cost. For experimental purposes, we instantiate our framework with two LLMs, GPT-3.5 or GPT-4, and two inexpensive students, a k -NN classifier or a Multi-Layer Perceptron, using two common business tasks, intent recognition and sentiment analysis. Experimental results indicate that significant OpEx savings can be obtained with only slightly lower performance.

## introduction prompting pre-trained large language models (llms) aligned to follow instructions @xcite performs impressively well in zero-and few-shot settings. hence, small and medium-sized enterprises (smes) that cannot afford the cost of creating large task-specific training datasets for model fine-tuning, but also the cost of pretraining their own llms, are increasingly turning to third-party services that allow them to prompt llms. for example, smes that provide customer support chatbots prompt llms like @xcite to detect user intents and drive the chatbot-customer interaction @xcite . the best llms, however, currently require a payment per prompting call, and these payments become a significant operating expense (opex) for smes. furthermore, customer inputs (e.g., dialog turns) are often very similar over time, hence smes end up calling llms to handle inputs that may be very similar to inputs already handled by the llms in previous (already paid) calls. we introduce the online cost-aware teacher-student (ocats) framework that allows reducing the calls to a commercial llm, treated as a teacher model, by caching its previous responses and using them to train a local inexpensive student model. ocats includes criteria for deciding when to trust the student or call the teacher, and a methodology to tune the criteria and measure the tradeoff between performance and cost. unlike common teacherstudent training for knowledge distillation @xcite , here the teacher does not train the student on all the available instances (in our case, all the incoming customer inputs). also, unlike teacher-student approaches to self-training @xcite , the teacher is already reasonably effective (but expensive). in that sense, our work is closer to ac-tive learning @xcite , but ocats trains the student on labels provided by a teacher llm, not humans, and there is initially no large pool of unlabeled instances (customer inputs) to select from, as instances arrive online. ocats can be used with any service that allows prompting llms, and any kind of local student model. for experimental purposes, we instantiate ocats with gpt-3.5 or gpt-4 as the teacher, and a k-nn or multi-layer perceptron (mlp) classifier as the student, using an intent recognition dataset from the banking domain or a sentiment analysis dataset. experimental results indicate that significant opex savings can be obtained with only slightly lower performance. for example, the k-nn student can handle approximately two-thirds of the incoming instances (customer inputs) of the intent recognition task without calling the gpt-4 teacher (fig. 2 , left, red line) for a decrease of less than 0.5 percentage points in accuracy (fig. 2 , middle, red and black lines). ocats introduces discounted versions of common evaluation measures (e.g., accuracy) that allow an sme to quantify how much it prefers to lean towards fewer calls or less user frustration (different λ values in fig. 2 ). our main contributions are: (i) we introduce a general teacher-student framework that helps smes reduce the prompting calls to commercial llms and the corresponding opex costs by caching the responses of the llms and training inexpensive local student models. (ii) we introduce discounted versions of common evaluation measures that allow the smes to quantify how much they prefer fewer llm calls vs. increased user frustration (e.g., caused by lower accuracy) and tune the framework's criteria that decide when to trust the local student model or call the llm teacher accordingly. (iii) we instantiate the framework with gpt-3.5 or gpt-4 as teachers, and a k-nn or mlp classifier as students. (iv) we perform experiments on two well-known tasks for smes, intent recognition and sentiment analysis, and show that significant cost savings can be obtained with only slightly lower performance. this is a first step towards exploring the benefits of the proposed framework with more datasets, models, and business scenarios. ## framework architecture: the proposed framework (ocats) consists of three main components (fig. 1 ): a teacher, typically a resource-intensive model of-fering premium results; a student, a cost-effective model that is typically much smaller and simpler than the teacher; a cache, a repository of incoming instances (e.g., customer requests) that have already been processed by the teacher. we assume that the framework is employed to handle a task for which there is no available large dataset for supervised training, apart from a few incoming instances (possibly a handful per class) annotated with the ground truth (e.g., correct labels). this is a very common case for smes that cannot afford the cost of creating large task-specific training datasets, but can easily construct small numbers of demonstration instances. the teacher-student setting is online, as every incoming instance is handled at inference time as follows. first, the student is called to handle the instance. then some student-and task-specific criteria, which assess the reliability of the student's output, indicate if the student's output (e.g., label) should be used or if the teacher should be consulted. if the student's output is selected, it is returned as the response to the incoming instance. otherwise, the teacher is called to handle the instance. in the latter case, the instance along with the teacher's result are stored in the cache. depending on the type of student, periodic re-training takes place, to update the student with the cached instances. instantiations: in the experiments of this paper, we instantiate ocats with a gpt-3.5 or gpt-4 teacher, a distance-weighted k-nn or mlp classifier as the student, for a single-label classification task (intent recognition or sentiment analysis). in all cases, we represent each incoming instance (customer request) by its mpnet-based @xcite vector representation (text embedding) and we use two criteria (fig. 1 ) to decide when to use the student's response or invoke the teacher: (i) the entropy of the probability distribution (over the label set) produced by the student (k-nn or mlp) for the incoming instance, and (ii) the distance of the vector representation of the incoming instance from the centroid of the vector representations of the k most similar cached instances. consult nguyen et al. (2022) for other possible criteria. we leave other instantiations of ocats (other teachers, students, tasks, representations) for future work. ## main experiments here we discuss the experiments we conducted with the gpt-4 teacher, the k-nn student, and the banking intent recognition dataset. in the appendix, we report two additional sets of experiments, one where we replaced the k-nn student by an mlp (appendix e) keeping the rest of the setup unchanged, and one where we replaced the task/dataset (by sentiment analysis) and the teacher (by the cheaper gpt-3.5) otherwise keeping the setup of the initial experiments (appendix f). the additional experiments verify the conclusions of the experiments of this section. intent recognition dataset: in this section, we use banking77 @xcite , an intent recognition dataset from the banking customer service domain. it includes 13,083 customer messages. the ground truth assigns to each message a single label (intent) from the 77 available. the dataset is divided into training (10,003 instances) and test (3,080) subsets. appendix a shows more statistics. few-shot training and development sets: assuming that an sme can only afford to construct a small number of training instances per class, we use only 3 × 77 = 231 instances from the original training set of banking77, three per class, as a fewshot version of the training set. the 231 instances were manually selected to avoid unclear cases, e.g., similar instances with different ground truth labels. similarly, we created a few-shot development set of 13 × 77 = 1, 001 instances from the original training set, for hyperparameter tuning. incoming instances and evaluation measure: we use the original test set of banking77 as the incoming instances. we repeat each experiment with five random shufflings of the test set (to obtain five different streams of input instances) and report average scores over the shufflings. we set ϕ to accuracy, since the test set is balanced (appendix a). teacher: in this section, we use gpt-4 (openai, 2023) as the teacher, the most capable llm for few-shot in-context learning tasks at the time. each prompt includes instructions, demonstrators (incontext few-shot examples), and the incoming instance to be classified; see appendix b for details. student: in this section, a distance-weighted k-nn classifier is used as the student. vector representations of the incoming instances are generated with a sentence-transformer (reimers and gurevych, 2019) variation of mpnet @xcite . 2 appendix c provides more information on the distance weighting used. it also shows (fig. 7 ) that in a more conventional setting, where a large manually labeled training set is available, the k-nn classifier clearly outperforms gpt-4 in accuracy (92% vs. 82%). note that for the k-nn student, no retraining (fig. 1 ) is necessary, since the cache coincides with the memory of the k-nn classifier. the cache is initialized with the 3-shot training examples of the classes (231 instances in total). ## conclusions we introduced an online cost-aware teacher-student framework (ocats) to help smes reduce opex costs by caching the responses of commercial llms and training inexpensive local students. we also introduced discounted versions of common evaluation measures, allowing smes to quantify the trade-off between llm calls and user frustration. by instantiating ocats with a gpt-4 teacher and a k-nn student and experimenting with an intent recognition dataset from the banking domain (banking77), we showed that the calls to the teacher can be significantly reduced (by 1/3) with only a slight performance drop (0.37 percentage points). additional experiments with an mlp student on the same dataset led to the same findings (appendix e). further experiments with a gpt-3.5 teacher, the initial k-nn student, and a sentiment analysis dataset (lmr) also confirmed the conclusions of the previous experiments (appendix f). in future work, we plan to experiment with more datasets and tasks (e.g., question answering), and suggest adaptive policies for λ to allow higher opex costs (more frequent calls to the teacher) when the cache is cold and be more selective (calling the teacher less frequently) later on. we also plan to enhance ocats with indicators of how much we can trust the teacher responses (e.g., confidence of the teacher). finally, we intend to incorporate more financial metrics (e.g., student costs) in the discounted versions of the evaluation measures and study more complex strategies (e.g., game-theoretic, reinforcement learning) to select the thresholds that determine when to trust the student or call the teacher.