Paper: TPE: Towards Better Compositional Reasoning over Conceptual Tools with Multi-persona Collaboration (ICLR-2024)

Reviewer comment (Writing):
The paper is poorly written. Many key concepts are not clearly defined. The method description is difficult to follow. Details about the experiment tasks are missing like what these tasks are about and how the evaluation metrics are computed.

Author response:
1) **Experimental Details**: We have meticulously defined key concepts, incorporating references from relevant literature. In our experimental design, we deliberately selected multiple source and multiple strategies-grounded response generation tasks, as explicitly outlined in the Introduction and Experiment sections. To enhance clarity, we have included examples and conducted a comprehensive statistical analysis of the corresponding datasets, presented in Figure 1 and the Appendix. In order to provide fair and comparable results, we choose conventional metrics such as BLEU, F1 as shown in Section 4.1 following previous works [1, 2, 3]. We also cite related literature for references.

Reviewer comment (Novelty):
The approach is simply multi-persona prompting, which has been done in various papers (the authors also cited those papers). I do not see a significant novelty compared to previous approaches.

Author response:
2) **Novelty**: Our work firstly introduces conceptual tool, serving as a significant extension and a new perspective to address complex dialogue situations, particularly for multi-source and multi-strategy dialogues. In addition, we also firstly apply  the multi-persona collaboration capability of LLMs into tool learning, while other multi-persona prompting method can not be directly used in this problem. In addition, the collaboration framework is also novel, as a tailored and explainable top-down dialogue planning framework to our targeted problems.

Reviewer comment (Reproducibility):
The approach requires too much human knowledge in designing the prompt. The prompt is highly customized to the problem setting.

Author response:
3) **Expert Knowledge**: We acknowledge that it is necessary to utilize expert knowledge when design the prompts. However, we also need to point out that it is exactly the novelty and values of the proposed method, following lots of other prompting method such as CoT [NeurIPS 2022], ReAct [ICLR 2023], Chameleon. Compared with these baselines, our method achieve better performance and can be applied into other dialogue response generation tasks.

Reviewer comment (Evaluation):
Performance gains compared to baselines are marginal.

Author response:
4) **Performance**: Despite the improvement may not substantial, it is important to consider the complexity of the task, dataset characteristics, and the nature of the baseline models. These two kind of dialogue response generation tasks are challenging and complicated, and we select current SOTA methods including supervised and unsupervised approaches. Illustratively, our method showcases a notable advancement with a remarkable 12 points enhancement in both Avg.B and F1 compared to the supervised state-of-the-art method. Furthermore, in surpassing all other unsupervised approaches grounded on ChatGPT, our method exhibits a superior performance with an approximately 2 points. Thus, we consider our performance improvement is **NOT** marginal. **These results not only highlight the substantial progress achieved by our approach but also underscore the significance of our motivations and innovative solutions in addressing these challenges compared to existing benchmarks. Our contributions extend beyond mere performance metrics, emphasizing the novel perspectives and solutions we bring to the forefront of these problems**. In this way, we believe that the insights gained from our approach are more important for the following reasons: 1. We shows that the LLM can call different conceptual tools automatically and dynamically to solve a complex task; 2) We validate the effectiveness and efficiency (i.e., less cost) of our proposed multi-persona collaboration framework; and 3) The results and analysis suggest that our approach holds the potential to contribute significantly to the advancement and application of dialogue systems within the realms of psychological therapy, educational tutoring and customized dialogue system.

Reviewer comment (Novelty):
Can you discuss the differences of your method compared to each baseline approach?

Author response:
Q1: Can you discuss the differences of your method compared to each baseline approach?

A1: To begin, our proposed TPE (Thinker-Planner-Executor) framework distinguishes itself from all baselines by harnessing the collaborative capabilities of multiple Language Model (LLMs) instances, each embodying distinct personas and playing varied roles in a top-down approach to address complex real-world problems effectively. Furthermore, we focus on elucidating the distinctions between TPE and three representative baselines—Chameleon, ReWOO, and ReAct—to provide a detailed comparative analysis. This approach allows for a clearer understanding of how TPE stands apart from specific baselines, emphasizing its unique contributions and advantages.

1. Chameleon: As shown in Table 13 and 14, it does not consider the argument dependency between different tools while TPE models the internal relationship such as the PERSONA and DOCUMENT in FoCus dataset. In addition, it can not perform well in multiple strategy dialogue response generation tasks since it generates the response for each strategy independently without considering the relationship, especially for multiple same strategies in single turn.

2. ReAct: This is a observation-dependent prompting method, which means all previous output must be feed during the output of all intermediate results, resulting in low efficiency and unnecessary cost. It is easily trap into local optima and loop, as shown in ToolQA [4].

3. ReWOO: To alleviate the problem of ReAct, this method decouples the tool planner and task executor. To better improve the performance and met the specfic requirements of target problems, we make some modifications: 1) Add Thinker modules, which prompt the LLMs generate more additional signals to help the planning. There are two functions: (a) similar with other methods which utilize LLMs to enrich the semantic information of the query in information retrieval [5]; and (b) the intermediate reasoning processing in CoT; 2) Design two variants for different situcations as introduced in Planner and Executor. For example, we observe that as the number of strategies increases, the number of evidence (\#E(number)) can become uncontrollable, making it impractical to present the results of ReWOO on the CIMA and PsyQA datasets. We put the analysis and explanations in the Appendix B.6. 

Lastly, it is crucial to highlight that our proposed method consistently outperforms all baselines, striking a balance between efficiency and effectiveness.

Reviewer comment (Theory):
Why can't we just use the same approach for functional tools to handle conceptual tools?

Author response:
Q2: Why can't we just use the same approach for functional tools to handle conceptual tools?

A2. Yes, we can. Actually, we indeed choose these approaches for functional tools as our baselines, such as Chameleon, ReWOO and ReAct. However, as indicated in our main experiments and analysis, our method achieves better performance than these strong baselines under both automatic and human evaluation. There are two reasons as explained in the paper: 1) some of the baselines for functional tools can not perform well when there are multiple calls of the same conceptual tools in the response generation, such as the multiple calls of the same strategy (2nd part of Executor in the Method Section). 2) the introduced top-down multi-persona collaboration framework brings more explainable and controllable generation, resulting in better performance (Experiment and Appendix).

In conclusion, it's crucial to highlight that existing prompting methods do not squarely address our specific target problem—multi-source and multi-strategy grounded response generation tasks. Our contribution extends beyond presenting a better method; fundamentally, we offer a novel perspective that addresses the inherent challenges of our targeted problem. This perspective represents a significant extension of current solutions, emphasizing the unique and valuable contribution our approach brings to the field.

Reviewer comment (Writing):
Some section description of the paper is not very clear to understand, for example, the formulation part of section 3.2; it's better to be illustrated with some examples.

Author response:
Thank you for your suggestions, we will definitely add more examples and formulations to better illustrate our method.

Reviewer comment (Writing):
It's a little difficult to understand the formulation symbols in section 3.2; it's better to illustrate with some intuitive example.

Author response:
A1: Please refer to the above answer for weakness.

Reviewer comment (Theory):
In this paper's experiment, the cognitive tools seem corresponding to the dialogue strategies. Is there more formal definition based on traditional dialogue theory for it like the definition for functional tools? Or does it change with different domain datasets?

Author response:
A2. Honestly, the cognitive tools (**conceptual tools** in the paper) corresponds **the multiple knowledge sources, and multiple conversational strategies** in the experiment. We formally define the conceptual tool as a cognitive concept used to help systematic or investigative thought in the Introduction section, and the instances of conceptual tools changes with different domain datasets, as same as functional tools. We indeed borrow some definitions of multi-strategy-grounded dialogue such as psychological therapy and we will follow the suggestions of reviewer U93G to ground the definition with literature in cognitive science.

Reviewer comment (Evaluation):
For Table 1's result, why is there no TPE in the Zero-shot ChatGPT methods? And for CIMA's F1 score, the ReAct method is much better than other methods; is there any possible explanation?

Author response:
A3: The primary challenge associated with zero-shot learning lies in the lack of control over the output, particularly in terms of format. This issue is pervasive across various prompting methods employed in tool learning, including Chameleon, ReWOO, and ReAct. A common drawback shared by these methods is the inability to determine when and how to invoke tools without the incorporation of demonstrations. In this way, all these methods utilize few-shot methods [1, 2, 3]. Ragarding the performance of CIMA, it is worthy noting that there arises a need to employ multiple instances of the same strategy within a single turn response generation, as exemplified in the second part of the Executor, for these multiple strategy dialogues. Notably, in such scenarios, observation-dependent prompting, as exemplified by ReAct, proves to be more advantageous compared to decoupling methods like those seen in Chameleon and ReWOO. This preference stems from the fact that the input for Chameleon and ReWOO remains the same when multiple calls of the same strategies are made. Additional details are provided in the Appendix, and the code implementation can be found in the attached materials.

[1] Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models

[2] ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models

[3] REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS

Reviewer comment (Theory):
The paper introduces several key components/definitions that are not well grounded in the literature. For example, the definition of conceptual tools should be grounded from a more credible source in cognitive science.

Author response:
1) **Cognitive Science Literature**: Upon careful consideration of your comments, we acknowledge the importance of aligning our definitions with established concepts in cognitive science. In our initial draft, we mainly focus on tool learning and refer to definitions from Wikipedia and related literature. However, we understand the necessity of bolstering our literature review with additional citations and aligning our terms with the terms used in related cognitive science. To address this, we are currently in the process of reviewing the papers you suggested for inclusion in the revised manuscript and also others as listed below [1,2,3,4,5,6,7]. Our aim is to incorporate relevant findings from these sources, ensuring a more comprehensive and well-grounded discussion in the cognitive science domain.

Reviewer comment (Writing):
The paper mentions that the thinker tries to estimate the emotions of the persona involved, but emotions are not defined or referenced well and can mean many different things.

Author response:
2) **Emotion Definition**: We basically prompt the LLMs to consider the emotions of the users following the definition of previous works [8,9]. We will add the support material in a more clear way.

Reviewer comment (Writing):
It is suggested to look at the literature on Theory of Mind and try to define the thinker in a manner that aligns with the terms used in the literature.

Author response:
1) **Cognitive Science Literature**: Upon careful consideration of your comments, we acknowledge the importance of aligning our definitions with established concepts in cognitive science. In our initial draft, we mainly focus on tool learning and refer to definitions from Wikipedia and related literature. However, we understand the necessity of bolstering our literature review with additional citations and aligning our terms with the terms used in related cognitive science. To address this, we are currently in the process of reviewing the papers you suggested for inclusion in the revised manuscript and also others as listed below [1,2,3,4,5,6,7]. Our aim is to incorporate relevant findings from these sources, ensuring a more comprehensive and well-grounded discussion in the cognitive science domain.

Reviewer comment (Reproducibility):
The main paper lacks details about the implementation of thinker, planner, and executor. These details are mentioned in the Appendix, but they should be included in the main paper.

Author response:
3) **Implementation Details**: Thank you for your suggestions. Due to the page limit, we have to put some content in the Appendix, and we will re-organize the paper to meet the page limit and more clear.