Paper: Chain-of-Experts: When LLMs Meet Complex Operations Research Problems (ICLR-2024)

Reviewer comment (Experiments):
The results on ComplexOR seem not sense. A too small dataset.

Author response:
We understand the reviewer's concern. ComplexOR can be viewed as the first dataset that has been well annotated for complex operations research problems and it has great potential to evaluate the reasoning capabilities of LLMs. The dataset is not large-scale because it is rather labour intensive for domain experts to help annotate the problems.  To construct an annotated problem instance, we need to

1. Find problems with suitable complexity from sources like academic papers, textbooks, or tutorials
2. Manually extract the textual description of the problem
3. Construct the mixed integer programming model and double check if it is correct
4. Programming for this model
5. Design several test cases with varied difficulty

Since we are firmly aware of its value to the community, we continued to collect more samples after the paper submission. Till now, we have expanded the size of ComplexOR from 20 instances to 37 instances. The new experimental results on ComplexOR have been updated in the revision.

Reviewer comment (Writing):
The description of CoE is not clear. It should be well-moviated and started with several backgrounds.

Author response:
In revision, we will follow the reviewer's advice to better motivate the description of CoE. Our thought started with Solo Performance Prompting, a multi-agent system. While SSP shows potential in solving complex tasks like logical grid puzzles, it falls short in more domain-specific taks such as the OR problem. We identified two primary reasons for this limitation. Firstly, in SSP, the agents are prompt templates initialized by leader personas, which restricts their ability to integrate extensive knowledge. To address this limitation, we introduce 'experts' to play specific roles in the problem-solving process. Secondly, we find that the collaborative approach in SSP is too straightforward to tackle challenging problems. It lacks the ability to selectively choose experts based on the specific problem and refine based on external feedback. To this end, we propose forward thought construction and backward reflection mechanisms inspired by real-world collaboration processes.

Reviewer comment (Experiments):
Is the CoE suitable for other reasoning tasks? What is the difference if applied to other tasks?

Author response:
This is a good point. In this paper, the agents/experts for collaborative reasoning are customized from the domain of operations research. As to agent design, the **terminology interpreter** who is responsible for supplying knowledge about specific terminology is integrated with a scenario modeling knowledge base. The **programming expert** is designed to generate modeling code, specifically in the LP file format.

To extend CoE to support other reasoning tasks, the first step is to reconstruct proper agents for the target domain. Luckily, the collaborative reasoning framework among the agents with forward thought construction and backward reflection is re-usable. 

To evaluate the reusability of our framework, we conducted a preliminary experiment on Text2SQL task. Specifically, we selected 12 hard test instances related to a manufacturer scenario from the spider dataset[1]. In this experiment, we configured a CoE with two experts: a manufacturer terminology interpreter and a dataset grammar expert integrated with an SQLite grammar document[2]. We compared this setup with a standard prompting baseline. The results are as follows:

|       | Execution Accuracy | Exact Matching Accuracy |
| ----- | ------------------ | ----------------------- |
| GPT-4 | 50%                | 37.5%                   |
| CoE   | 75%                | 58.3%                   |

We regret that time constraints prevented us from conducting more comprehensive experiments.   In our upcoming revisions, we are committed to adding a formal experiment to further validate our approach.

Reviewer comment (Novelty):
I suggest the paper give more attention to the CoE framework.

Author response:
In this work, we focus on automatically solving complex OR problems. It is motivated by the real demand of our industry partners. Such an automatic tool can greatly help them to reduce the overhead of modeling and programming for operations research problems from diversified industry sectors. 

We totally agree with your comment that it would generate more impact if we focus on CoE and apply it to more types of reasoning tasks.  In the previous comment, we have validated its potential to support complex Text2SQL task.  We believe this is an exciting avenue for future research to explore if we can further improve the generality of our reasoning framework among various reasoning tasks.

Reference:

[1] https://paperswithcode.com/sota/text-to-sql-on-spider

[2] https://www.sqlite.org/lang.html

Reviewer comment (Evaluation):
ComplexOR is very small; I wonder how significant the results are.

Author response:
We understand the reviewer's concern. ComplexOR can be viewed as the first dataset that has been well annotated for complex operations research problems and it has great potential to evaluate the reasoning capabilities of LLMs. The dataset is not large-scale because it is rather labour intensive for domain experts to help annotate the problems.  To construct an annotated problem instance, we need to

1. Find problems with suitable complexity from sources like academic papers, textbooks, or tutorials.
2. Manually extract the textual description of the problem.
3. Construct the mixed integer programming model and double check if it is correct.
4. Programming for this model.
5. Design several test cases with varied difficulty.

Since we are firmly aware of its value to the community, we continued to collect more samples after the paper submission. Till now, we have expanded the size of ComplexOR from 20 instances to 37 instances. The new experimental results on ComplexOR have been updated in the revision, as shown in Table 1 (overall performance), Table 2 (ablation study), and Table 3 (Robustness under different LLM).

Reviewer comment (Experiments):
It would be interesting to compare to a baseline that applies CoE, and uses the same model to play all the different roles.

Author response:
This is an inspiring comment. We followed the reviewer's advice to implement a baseline that uses the same model. In this experiment, we use a uniform system prompt across all roles, without any additional knowledge bases. We also find that the removal of experts' features leads to a decrease in accuracy, which suggests that the CoE benefits from using specialized experts over a singular, generalized model. We have added this baseline in our revision.

|                     |           | LPWP     |          |           | ComplexOR |          |
| ------------------- | --------- | -------- | -------- | --------- | --------- | -------- |
|                     | Accuracy  | CE rate  | RE rate  | Accuracy  | CE rate   | RE rate  |
| CoE without Experts | 55.1%     | 4.0%     | 11.9%    | 18.8%     | 7.9%      | 15.0%    |
| Chain-of-Experts    | **58.9%** | **3.8%** | **7.7%** | **25.9%** | **7.6%**  | **6.4%** |

Reviewer comment (Theory):
Eq. 3 reads like applying the LLM to the prompt template outputs a new set of parameters, which does not align with what happens with prompting. At a higher level, do we really need the $\theta$ notations in Eqs. 2 and 3?

Author response:
We sincerely appreciate your comment and it actually pushes us to think about the roles of $\theta$ deeply. We agree that the use of $\theta$ in Equations 2 and 3 could be misleading as CoE don't actually generate new parameters through prompting. To clarify, we've revised our approach to align with the notation used in the paper Reflexion[3]. Here, we treat the Conductor as a policy, with its parameters indicated in the subscript. The revised equation is as follows:

$$Conductor_{\mathcal{F}^{\theta'}(\mathbb{PT}_t)}(e | s) = P_r\{E_{\phi_t} = e | S_t = s\}$$

Reviewer comment (Evaluation):
Lack of evaluation on individual expert agents, as well as the conductor

Author response:
In our original submission, we have conducted an ablation study to evaluate individual expert agents. The reviewer can check Figure 5 in Appendix A.4.2. Due to space limit, we regret that it was not put in the main body of the paper.

Reviewer comment (Experiments):
The comparison with other models might not be fair, since they call the LLM differently. Maybe add some measurements of how different methods use the LLMs.

Author response:
In our experiments, we unify the way to use the LLMs. Each method within an experiment utilizes the same model API, facilitated by the 'chat complete' interface LangChain framework[1]. Please let us know if we did not interprete the reviewer's comment correctly.

Reviewer comment (Experiments):
If we use other less competent LLMs, like smaller models or open sourced models, how much the performance will be affected?

Author response:
This is a good angle. If we use smaller models such as Llama 2, the performance degrades dramatically. We have conducted an experiment in chapter 4.7 to show the sensitivity to different types of LLMs. For smaller models such as Llama 2, we also conducted additional experiments on dataset LPWP and presented the results in the following:

|                  | Accuracy | CE rate | RE rate |
| ---------------- | -------- | ------- | ------- |
| Standard         | 0%       | 100%    | 0%      |
| Chain-of-Thought | 0%       | 100%    | 0%      |
| Reflexion        | 0%       | 100%    | 0%      |
| ReAct            | 0.7%     | 99.3%   | 0%      |
| Solo Performance | 0%       | 100%    | 0%      |
| Chain-of-Experts | 0.7%     | 99.3%   | 0%      |

In this experiment, we use the Llama2 13b model[2] as a foundational model, applying it across various baselines and the CoE on the LPWP dataset. Our findings revealed that the Llama2 model was largely ineffective in resolving the OR problems. Approaches like CoT and Reflexion showed no significant impact. Only 2 out of 289 instances were successfully resolved using ReAct and CoE, which incorporated external knowledge. These results suggest that Large Language Models must achieve a certain scale to effectively tackle complex problems.

Reviewer comment (Novelty):
While the paper focuses on tackling complex operations research problems, it doesn't seem to introduce any techniques specifically tailored for operations research challenges.

Author response:
**Techniques Designed for OR Problems?**

> While the paper focuses on tackling complex operations research problems, it doesn't seem to introduce any techniques specifically tailored for operations research challenges.

This is a good point. In this paper, the agents/experts for collaborative reasoning are customized from the domain of operations research. As to agent design, the **terminology interpreter** who is responsible for supplying knowledge about specific terminology is integrated with a scenario modeling knowledge base. The **programming expert** is designed to generate modeling code, specifically in the LP file format.

To extend CoE to support other reasoning tasks, the first step is to reconstruct proper agents for the target domain. Luckily, the collaborative reasoning framework among the agents with forward thought construction and backward reflection is re-usable. 

To test the reusability of CoE, we conducted a preliminary experiment using the Text2SQL dataset. Specifically, we selected 12 hard test instances related to a manufacturer scenario from the spider dataset[1]. Here, we configured a CoE with two experts: a manufacturer terminology interpreter and a dataset grammar expert integrated with an SQLite grammar document[2]. We compared this setup with a standard baseline. The results are as follows:

|       | Execution Accuracy | Exact Matching Accuracy |
| ----- | ------------------ | ----------------------- |
| GPT-4 | 50%                | 37.5%                   |
| CoE   | 75%                | 58.3%                   |

We regret that time constraints prevented us from conducting more comprehensive experiments.  In our upcoming revisions, we are committed to adding a formal experiment to further validation.

Reviewer comment (Novelty):
I believe the novelty of this work is somewhat limited, as several studies have already explored the "planning with feedback" approach with LLMs. Please refer to "A Survey on Large Language Model based Autonomous Agents (https://arxiv.org/pdf/2308.11432.pdf)" for more details. I think the authors should offer a more in-depth comparison with these existing works.

Author response:
**Comparison with Other Planning with Feedback Methods**

>  I think the authors should offer a more in-depth comparison with these existing works. Moreover, though the methodology is described as a multi-expert framework, it essentially relies on deploying various prompts to the same LLM.

We thank the reviewer for providing the reference. The unique features of our CoE include forward agent selection and backward reflecion. We followed the reviewer's advice and provided an in-depth comparison with these existing works in the following table.

|                  | Multi-agnets | Forward agent selection | External knowledge access | Refine by Feedback |
| ---------------- | ------------ | ----------------------- | ------------------------- | ------------------ |
| ReAct            | X            | X                       | ✓                         | X                  |
| Voyager          | X            | X                       | ✓                         | ✓                  |
| Ghost            | X            | X                       | ✓                         | ✓                  |
| SayPlan          | X            | X                       | ✓                         | ✓                  |
| MetaGPT          | ✓            | X                       | ✓                         | X                  |
| NLSOM            | ✓            | X                       | ✓                         | X                  |
| SSP              | ✓            | X                       | X                         | X                  |
| ChatEval         | ✓            | X                       | X                         | X                  |
| Chain-of-Experts | ✓            | ✓                       | ✓                         | ✓                  |



Reference:

[1] https://paperswithcode.com/sota/text-to-sql-on-spider

[2] https://www.sqlite.org/lang.html

Reviewer comment (Novelty):
Why can't the method proposed in this paper be represented through Solo Performance Prompting, and where exactly does it differ from Solo Performance Prompting? From the description, it seems that the approach is entirely representable under the Solo Performance Prompting framework.

Author response:
**Can CoE be Represented by the SSP Framework?**

> Why can't the method proposed in this paper be represented through Solo Performance Prompting, and where exactly does it differ from Solo Performance Prompting? 

There are two key differences between Solo Performance Prompting and our CoE. First, in terms of agent generation, Solo Performance Prompting prompts a single LLM to identify and simulate multiple agents. In contrast, the agents/experts in CoE are tailored to collaboratively solve an OR problem. For example, the Modeling Knowledge Supplement Expert is integrated with the knowledge of games-cutting edge modeling to its knowledge base. Second, in terms of collaborative reasoning framework, the agents in Solo Performance Prompting cooperate in a round-robin manner. In contrast, our CoE works with forward thought construction and backward reflection.

We also offer a case study in Appendix 4.3 to illustrate the differences of reasoning processes between Solo Performance Prompting  and our CoE.