Paper: FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (ICLR-2024)

Reviewer comment (Experiments):
No justification for the chosen datasets. Do these map to diverse and realistic domains, tasks, and applications? The evaluation selection is fairly important to understand what realistic cost saving ranges are.

Author response:
***[Justification for the chosen datasets]***: We chose the datasets because they represented an important subset of real-world applications. For example, the HEADLINES and OVERRULING datasets represent the task of text classification and tagging, which is listed as an important application of GPT-4 by OPENAI (see, e.g., https://platform.openai.com/examples/default-tweet-classifier and https://platform.openai.com/examples/default-review-classifier). The SCIQ dataset measures the scientific question answering ability, which is key for QA applications such as the Khan academy pilot program (see https://openai.com/gpt-4). We have also added a discussion on this in the revised draft (see Section D in the appendix).

Reviewer comment (Experiments):
Are any of the datasets speculated to be in the training set of some of the models? Is there a way to test this?

Author response:
***[Are any datasets in the training set of some models]***: Data contamination is indeed important. Without access to the training datasets and documents of black-box commercial LLM services,  however, it is challenging to definitively detect if a data point has been seen during training.

Reviewer comment (Experiments):
Experiments are mainly conducted on QA and text classification tasks. From my understanding, a baseline of fine-tuning RoBERTa will achieve 90%+ accuracy on AGNews and CoQA.

Author response:
***[Experiments are mainly conducted on QA and text classification tasks]***: These tasks are selected partially because they are relatively easy to evaluate, which is why these datasets are commonly used to evaluate LLM research. In addition, several of our datasets are quite challenging. For example, on the COQA and SCIQ datasets, even GPT-4 is only able to achieve 40%-70% accuracy.

Reviewer comment (Experiments):
In terms of data efficiency, FrugalGPT also requires a training set to train the judge and the permutation function, and needs extra information to compute MPI.

Author response:
***[FrugalGPT requires a training set to train the judge and the permutation function, and need extra information to compute MPI]***: FrugalGPT is designed for applications involving a large number of LLM queries. Consider, for example, Yabble (https://openai.com/customer-stories/yabble), which analyzes customer posts for sentiment,  or, Jasper (https://www.jasper.ai/the-prompt/gpt4-future-of-gen-ai-business), which offers chatbots for varying domains. Here, as in many other industry applications of LLM, many tens of thousands of new queries per day are expected. In these cases, the cost of tuning FrugalGPT on a few thousand examples is well worth the reduction in cost when applying it to millions over time. We also conducted an additional study on the effects of FrugalGPT training set size in the HEADLINES dataset.  Specifically, we varied the data used to train FrugalGPT from 500 to 5000, and measured the accuracy of the FrugalGPT LLM cascade on the test set with 5000 samples. As shown in the following table, FrugalGPT’s performance was consistently better than GPT-4 (85.5%). We have added this in the revised paper (see Section B.5 in the Appendix). 

| Training set size            | Accuracy |
| :----------------: | :------: | 
| 500        |   85.9   |
| 1000           |  86.8  |
| 2000   | 86.8 |
| 3000   | 86.8 |
| 4000   | 87.2 |
| 5000   | 87.0 |

Reviewer comment (Experiments):
From my understanding, calling API is most common when there are only few training examples, or the task is complex. It would be more convincing to show that FrugalGPT works well in these cases.

Author response:
***[From my understanding, calling API is most common when there are only few training examples, or the task is complex]***: Many enterprises are interested in using LLMs on a large number of relatively repetitive tasks. For example, financial companies such as Stripe are using LLMs to perform large scale fraud detection (https://www.techcircle.in/2023/03/16/11-companies-using-gpt-4-in-consumer-products), and Yabble analyzes customer posts for sentiment (https://openai.com/customer-stories/yabble), which is similar to our evaluation datasets AGNEWS. Insurance companies are also using LLMs to process millions of claims. Also, FrugalGPT requires fewer labeled data points than fine tuning a model for the task would (since it only needs to train a router that can still use the big model).

Reviewer comment (Experiments):
Is FrugalGPT a task-specific or task-agnostic method? In the current experiments, are the parameters (permutation, judge model, threshold) trained separately for each task?

Author response:
***[Is FrugalGPT a task-specific or task-agnostic method]***: The FrugalGPT paradigm is task-agnostic, but different tasks do require additional training for better performance. For example, our experiments show the cost savings due to FrugalGPT in diverse tasks including price prediction, reading comprehension and news classification.

Reviewer comment (Reproducibility):
What's the cost of estimating MPI? Does that mean you need to run all LLMs over all training examples?

Author response:
***[What's the cost of estimating MPI]***: MPI was estimated based on each API’s performance on the training partitions. FrugalGPT was designed for applications requiring numerous and continuous queries to LLMs. In such applications, the cost of estimating MPI is often small (less than 5%) compared to processing queries during deployment.

Reviewer comment (Experiments):
What data is used to train the DistilBERT judge?

Author response:
***[What data is used to train the DistilBERT judge]***: The judge was trained on the training partition of each evaluation dataset. The judge training is data efficient: a few hundred examples are enough to obtain a high quality judge.

Reviewer comment (Reproducibility):
For classification tasks, is it possible to use the DistilBERT judge directly for inference?

Author response:
***[Is it possible to use the DistilBERT judge directly for inference for classification tasks]: Yes. One can use the judge to process a query with each possible label and pick the label with the maximum score returned by the judge. However, this performance is much worse than FrugalGPT. For example, on the HEADLINES dataset, directly using the judge for inference gives an accuracy of 60% performance compared to the FrugalGPT’s accuracy of 87%. This is because judging is easier than prediction. For example, for a given query, the judge may assign low scores to all possible labels. In this case, FrugalGPT would defer the query to the next LLM, which may give a high-quality answer. Since all assigned scores are low, directly using the judge for labeling can give poor predictions.

Reviewer comment (Experiments):
Since the judge model is doing a regression problem, what's the reason of using a 2-d output, and taking the maximum value of the last layer (normalized by softmax)?

Author response:
***[What’s the reason of a 2-d output for training the judge as a regression problem]***: This is because there is no numerical label for quality, and thus we cannot use the standard regression training paradigm directly. Instead, we train a model to predict correctness, which is binary and can be curated from the ground-truth labels. Then we use the model’s confidence as the judge’s quality estimation. We have added a detailed discussion in the revision (Section D in the appendix).

Reviewer comment (Experiments):
As generation diversity is mentioned as a reason for improving performance, would it be possible to sample outputs from the same LLM multiple times, and ensemble their answers. As shown in https://arxiv.org/abs/2203.11171 (Figure 2) this would also create a cost-performance curve, and the performance grows with more calls. To demonstrate the contribution of FrugalGPT, such curves should also be included for comparison.

Author response:
***[Compare with sampling outputs from the same LLM multiple times, and ensembling their answers]***: We have conducted an additional experiment for the model ensembles. In particular, we query ChatGPT 3 times with different temperatures (0, 0.1, 0.2) and then take the mode of the generation to be the final output. As shown in Table 5 in the appendix, sampling multiple times leads to marginal performance gains on the HEADLINES dataset. Yet, the gain is relatively smaller than FrugalGPT, whose accuracy can be 87% at the same cost.  We have added this in the appendix (Section B) too.

Reviewer comment (Evaluation):
In Figure 1(d), I'm a little confused why the leftmost dot of FrugalGPT is still better than the cheap models. At this budget point, I would assume FrugalGPT makes one single call to one API, and thus the dot of FrugalGPT should overlap with one of the model. Please help me understand this better.

Author response:
***[Why the leftmost dot of FrugalGPT is still better than the cheap models in Figure 1(d)]***: This is because the leftmost dot of FrugalGPT actually used a bit more budget than the cheapest model. In fact, it routed 2% of queries to ChatGPT. This leads to an accuracy gain of 0.7% (=78.7%-78%).

Reviewer comment (Experiments):
Regarding the distribution shift experiments, currently it's mainly done by changing the label distribution in the test set. It would be interesting to study and evaluate how FrugalGPT performs when there is distribution shift on the input text.

Author response:
***[It would be interesting to study and evaluate how FrugalGPT performs when there is distribution shift on the input text]***: We conducted new experiments for distribution shift on the input text. Specifically, we asked ChatGPT to rephrase each question without changing its  meaning. On the HEADLINES dataset, we observe that FrugalGPT is quite robust to such shifts. For example, GPT-4 achieves 84% overall performance on the shifted dataset, and FrugalGPT can reach the same accuracy with only 33% of the cost. We have added this analysis in the revised draft (see Section B.3 in the Appendix).

Reviewer comment (Experiments):
How much does the search space pruning help in practice, versus just ordering the APIs in increasing order of cost? The only evidence presented is anecdotal in the experimental section.

Author response:
***[How much does the search space pruning help in practice, versus just ordering the APIs in increasing order of cost?]***: Pruning does help identify the most effective set of LLMs to query. As an ablation study, we compare the performance of FrugalGPT and that of using three APIs in increasing order of cost: GPT-J, GPT-Neo, and GPT-4. We chose these three APIs because they roughly lie on the surface of the convex hull of all APIs in Figure 1. As shown below, pruning (FrugalGPT) brings additional 1-2% performance improvements compared to using a fixed ordering of APIs in increasing cost.

| Cost (fraction of GPT-4) | FrugalGPT | FrugalGPT with fixed API order|
| :----------------: | :------: | :------: | 
| 10%  | 86.8 |85.1 |
| 20%  | 86.9 |85.6 |
| 100%  | 87.2 | 85.8|

Reviewer comment (Experiments):
How much did changing the threshold T affect results? Is the method very sensitive to T? This information would be useful to practitioners.

Author response:
***[How much changing the threshold T affected results? Is the method very sensitive to T?]***: The threshold T does affect the performance, as it controls when a query is routed to an API. Fortunately, practitioners do not need to tune it themselves: the threshold is learned automatically during FrugalGPT’s training process (the optimization problem on the bottom of page 4).

Reviewer comment (Writing):
Some of the design space is constrained by only having access to the output (see comments on scorer in Questions section). Writing this in a discussion in the draft would help answer questions a reader might have on a first reading.

Author response:
***[Writing a discussion on the constraints of the design space]***: Thank you for the helpful suggestion! We have added a short discussion in the revised draft (see Section D in the appendix).

Reviewer comment (Experiments):
Why is a scorer necessary? Given that closed APIs generally don’t provide probability/logit outputs, this is not an option. However, when available, is a simpler proxy like confidence or probability thresholding sufficient? For example, using a much smaller learned calibration module at the query level as in https://arxiv.org/abs/2207.07411 and https://arxiv.org/abs/2207.05221. A logit-based score is more useful for routing at the token level.

Author response:
***[If APIs give probability/logit outputs, is a scorer still necessary?]***: The scorer is a flexible module to measure the quality of a generation; it does not necessarily need to be the particular form we used. If the APIs give probability/logit outputs that truly capture the quality of their generations, then leveraging these signals can certainly simplify the scorer design. However, in practice, the API’s probability and logit are often not calibrated, which is why having a scorer helps.