Paper: EcoAssistant: Using LLM Assistant More Affordably and Accurately (ICLR-2024)

Reviewer comment (Novelty):
Would you mind commenting on the novelty of the proposed approach, or say, the most surprising part of this paper?

Author response:
**Q: What is the most surprising part of this paper?**

The most surprising part of this work is the effectiveness of “indirect advising” (also explained in the general response): while the assistant hierarchy is designed to reduce the cost and the solution demonstration is to improve the success rate, their combination further reduces the cost. Such a synergistic effect emerges because the solution from a more powerful model (GPT-4) would be later used to guide a weaker model (GPT-3.5) automatically without a specialized design since the query-code database is shared across assistants. By looking at the solution from the more powerful model, the weaker model could solve more tasks and therefore reduce the overall cost by reducing the reliance on the more powerful model.

Reviewer comment (Reproducibility):
Though cost-efficiency is an appealing property, it’s very unclear what the back-off policy looks like in general.

Author response:
**Q3: how to design the policy for back off in the general case?**  

Our design of the assistant hierarchy provides a general back-off policy: the order of assistant invocation is based on the cost. Specifically, it always invokes a cheaper assistant before a more expensive one. This design aims to reduce the reliance on more expensive models in order to reduce the cost of the system.

Reviewer comment (Evaluation):
How do you decide whether a response is a success or not? Is it based on execution error?

Author response:
**Q2: how to decide whether a generated response is good enough to be put in the demonstration library**  

In EcoAssistant, we store a query-code pair only when the query is successfully addressed. By this way, the stored code snippets are all solutions to previous queries and therefore have good quality.

Reviewer comment (Experiments):
How does this method scale with # of APIs? For instance, the ToolLLM[1] paper had >16,000 APIs in their dataset. This would require some shortlisting using a retriever to make it compatible but I think adding that part would significantly help improve the novelty aspect of the paper.

Author response:
**Q1: How does this method scale with # of APIs?**

As we only provide the assistant with the API name and a fake key, we can include as many APIs as we would like as long as it can be fit into the context window. But it is possible that more APIs might confuse the model and decrease the success rate. In that case, we can directly use the API retriever from ToolLLM as a plug-and-play module for EcoAssistant. We would like to leave it to future work since our focus is the cost-effective deployment of LLM services.

Reviewer comment (Evaluation):
I think more error analysis would also be needed to identify what kind of queries are problematic for which models. For instance, if we can identify if smaller LLMs can easily answer easy queries then we don't need to ever invoke the larger LLMs - are you already doing this?

Author response:
**Q2a: More error analysis would also be needed to identify what kind of queries are problematic for which models.**

Thanks for the suggestion. We conduct error analysis on the Mixed-100 dataset (100 queries, details can be found in section 4.4), and the results are shown below. We identified 5 categories of error: 1) exceeding the context window limit of the model; 2) Inability to output correctly formatted code for execution; 3) incorrect answer or 4) partially-incorrect answer due to erroneous API use or problematic code; and 5) incorrect date (eg, asking for tomorrow’s weather but actually output today’s)

| Method    | exceeds context window | code format | incorrect answer | partially-incorrect answer  | incorrect date | total number of failures
|---------|----------|----------|---------|----------|----------|---------|
| GPT-3.5-turbo | 3 | 56 | 14 | 1 | 1 | 75 |
| GPT-4   | 3 | 0 | 25 | 7 | 6 | 40 |
| EcoAssistant  | 0 | 0 | 16 | 3 | 1 | 20 | 

We can see that the EcoAssistant demonstrated the best performance among the three, with a total of only 20 failures. It did not exceed the context window limit or have issues with code formatting. Compared with GPT-4, it showed fewer instances of incorrect answers, partially incorrect answers, and incorrect dates. This suggests that the solution demonstration enables EcoAssistant to leverage past solutions and process queries more accurately.

**Q2b: if we can identify if smaller LLMs can easily answer easy queries, then we don't need to ever invoke the larger LLMs**

This idea is already realized in our assistant hierarchy: because EcoAssistant would always start with the cheapest assistant, if it succeeds, then it will not invoke more expensive assistants.

Reviewer comment (Reproducibility):
Can you help me understand how do you define an exit criterion? For instance, what if the agent gets stuck in an infinite loop where the larger LLM and the smaller LLM agent keep going back and forth?

Author response:
**Q3: how do you define an exit criterion?**

For each query, the EcoAssistant will terminate after trying out all the assistants. In particular, it always invokes the cheapest assistant first, and invokes the next one only when the current one fails to address the query. The order of invocation is based on the cost of the LLM and each assistant will be invoked at most once for each query. If the last and most expensive assistant still fails, the EcoAssistant will terminate, so it won’t get stuck in an infinite loop.