Paper: Rational Decision-Making Agent with Internalized Utility Judgment (ICLR-2024)

Reviewer comment (Theory):
The state, action and decision step in the context is not well-defined. As Large Language Models (LLMs) take text as input and output text, it is unclear what state and action correspond to in the context. It is assumed the authors develop their method based on ReAct [1] and Tree-of-Thought [2] as illustrated in Figure 1, but they are not properly introduced in the methodology part. In addition, it is recommended that the authors emphasize that the reward is not given in the Markov Decision Process (MDP) of this paper, which is uncommon.

Author response:
Thanks for your suggestions about providing adequate background information to contextualize our methodology. To this end, we plan to reorganize the structure of our paper, placing the Related Work section after the Introduction in the final version. This will offer readers a more comprehensive understanding of existing approaches like ReAct and Tree-of-Thought, thereby facilitating a clearer grasp of the decision-making process in our RaDAgent framework.

For the fundamental concepts of MDP (i.e., state, action, reward), we would like to elucidate them. These concepts have different meanings in different scenarios.
- In the Game of 24 scenario:
    - _State_ refers to the current calculation result of the four numbers.
    - _Action_ denotes the selection of one of four arithmetic operations (+, −, ∗, /).
- In the Webshop scenario:
	- _State_ refers to the current web page information.
	- _Action_ refers to the web browsing operations (e.g., click, go back, purchase)
- In the ToolBench scenario:
    - _State_ encompasses the context of the current decision sequence, including historical API calls and their corresponding results.
    - _Action_ refers to the APIs that RaDAgent can call.

The concept of _Reward_ in our work diverges from the **traditional external performance measures** typical in common MDP approaches. Instead, we focus on showcasing the importance of **internal rationality** (i.e., internalized utility judgment). In this light, the Elo scores in RaDAgent serve as a form of 'reward', reflecting the utility judgment capability of the agent. These scores are internally derived, aligning with our goal of demonstrating self-sufficient decision-making prowess without reliance on external validations. We will incorporate these clarifications in the revised manuscript, ensuring that the definitions of state, action, and reward are explicitly tailored to their specific applications in the various scenarios we explore.

For the notations about "subsequent step" in Section 3.1, we are grateful for your observation and suggestions. We will revise the manuscript to refine the symbols and terms used to describe the decision steps.

Reviewer comment (Writing):
The "subsequent step" in Sec. 3.1 is unclear. In the preliminary section, decision steps have a subscript that stands for the number of steps, and "subsequent step" can be comprehended as a chain of steps following the current step. But in Sec. 3.1, the subsequent step with different subscript should be different versions of the next step. It is suggested to introduce \(n\), change \(\hat{d}\) to \(d_{n+1}\) (because \(\hat{d}\) has no subscript), and change \(\sum_j\) to \(\sum_{j=1}^{n+1}\).

Author response:
Thanks for your suggestions about providing adequate background information to contextualize our methodology. To this end, we plan to reorganize the structure of our paper, placing the Related Work section after the Introduction in the final version. This will offer readers a more comprehensive understanding of existing approaches like ReAct and Tree-of-Thought, thereby facilitating a clearer grasp of the decision-making process in our RaDAgent framework.

For the fundamental concepts of MDP (i.e., state, action, reward), we would like to elucidate them. These concepts have different meanings in different scenarios.
- In the Game of 24 scenario:
    - _State_ refers to the current calculation result of the four numbers.
    - _Action_ denotes the selection of one of four arithmetic operations (+, −, ∗, /).
- In the Webshop scenario:
	- _State_ refers to the current web page information.
	- _Action_ refers to the web browsing operations (e.g., click, go back, purchase)
- In the ToolBench scenario:
    - _State_ encompasses the context of the current decision sequence, including historical API calls and their corresponding results.
    - _Action_ refers to the APIs that RaDAgent can call.

The concept of _Reward_ in our work diverges from the **traditional external performance measures** typical in common MDP approaches. Instead, we focus on showcasing the importance of **internal rationality** (i.e., internalized utility judgment). In this light, the Elo scores in RaDAgent serve as a form of 'reward', reflecting the utility judgment capability of the agent. These scores are internally derived, aligning with our goal of demonstrating self-sufficient decision-making prowess without reliance on external validations. We will incorporate these clarifications in the revised manuscript, ensuring that the definitions of state, action, and reward are explicitly tailored to their specific applications in the various scenarios we explore.

For the notations about "subsequent step" in Section 3.1, we are grateful for your observation and suggestions. We will revise the manuscript to refine the symbols and terms used to describe the decision steps.

Reviewer comment (Reproducibility):
Prompts for each environment are not specified. Only the prompt for utility learning is listed; the prompt for each environment is missing in the appendix.

Author response:
Apologize for the absence of prompts. We list all prompts of our method and experiments as follows.

The decision-making prompt which is used to make decisions during the experience exploration process is listed as follows:
```
You are the Decision-Making GPT and can perform any task using the tree search method. 
The search method is as follows:
1. First, I will provide you with the task description and input details.
2. For each task, you need to call various functions through multiple steps. 
3. At each step, you need to give your thought to analyze the status now and what to do next, with a function call to actually execute your step.
After the call, you will get the call result, and you are now in a new state.
4. Each (thought-function) pair mentioned above is considered a tree node, and each trail is a tree path from the root to a terminal node. Therefore, the Monte Carlo search tree contains multiple trails.
5. Although you may not see previous trails, in each trail, I will first place you in an intermediate state determined by the "value" of the node, and then you make different choices from there.

Remember:
1. Always make a function call at each step.
2. If you believe you have gathered enough information, call the function "Finish: give_answer" to provide your answer for the task.
3. If you feel unable to handle the task from this step, call the function "Finish: give_up_and_restart".

Let\'s begin!
Task description: {task_description}
```

The task description of Game of 24 is as follows:
```
Use numbers and basic arithmetic operations (+ - * /) to obtain exactly one number 24. In each step, you are only allowed to choose two of the left numbers to obtain a new number. For example, 7 * 9 - 3 * 1 3 = 2 4.
Remember:
1. All of the numbers must be used and must be used ONCE. So Only when the left number is exactly 24, you will win. So you don't succeed when the left number = [2 4, 5]. You succeed when the left number = [2 4]. 
2. All the try takes exactly 3 steps, and the task ends when the count of left numbers is 1, and check whether the only left number equals 24.
3. When there are two numbers left, ALWAYS pre-compute and list all the operations' combined results( + - * / ), and find if there is a way to combine them to 24 before you make the function call. 
3.1. If There is a way, use function "play_24" to combine it.
3.2. If not, use function "give_up" to restart.
4. The status changes ONLY when you call function "play_24". If you only give thoughts, nothing happens.
5. "play_24" inputs only one step, if you want to combine many numbers, split it into multiple calls. 
```

The task description of Webshop is listed as follows:
```
You should use functions to help handle the web shopping task on a webshop site.
We have 2 Pages: Product Selection Page & Product Details Page.
You have access to the following functions:
1. search: at any time, you can search a product by keywords. Then you will go to the Product Selection Page which shows a list of related products.
2. select: after searching keywords, you can select a product on the Product Selection Page. Then you will go to the Product Details Page, which shows the details of the product you select.
3. buy: On the Product Details Page, you can buy a product. Then the shopping task is completed.
```

The task description of ToolBench is listed as follows:
```
You should use functions to help handle real-time user queries. Remember:
1. ALWAYS call \"Finish\" function at the end of the task. And the final answer should contain enough information to show to the user, If you can't handle the task, or you find that function calls always fail(the function is not valid now), use function Finish->give_up_and_restart.
2. Do not use origin tool names, use only subfunctions' names.
You have access to the following tools:\n
```

These prompts will be added in the Appendix section in the final version to provide more experimental details. Moreover, we plan to release all code including the method implementation and experiment settings on Github to ensure reproducibility after acceptance.

Reviewer comment (Experiments):
The authors could perform more ablations on the exploration process. The exploration process is controlled by multiple factors, such as temperature \(\tau\), initial ELO score \(\hat{v}\), and ELO coefficient \(r\). Because the probability of exploring a new decision is exponential to the initial ELO score, it is intuitive that RADAGENT might fall into local minima where the currently best solution is present in the branch but can still be improved. It would be better if an ablation study on the hyperparameters could be performed to better measure the robustness of RADAGENT.

Author response:
For your questions about the robustness of RaDAgent in terms of hypermeters (including temperature $\tau$, initial Elo score $\hat{v}$, and Elo coefficient $r$), I want to make clarification.

First, the selection of $r$ is based on the classic Elo rating algorithm. In the classic Elo rating system[1,2,3], the expected superiority is defined as:
$$
E = \frac{1}{1+10^{-\frac{(R1-R2)}{400}}}
$$ 
In this paper, we employ the base $e$ to compute the expected superiority for computational implementation convenience:
$$
E = \frac{1}{1+e^{-\frac{(R1-R2)}{K}}}
$$
To achieve equivalence between them, we can set $K$ as approximately $K = \frac{400}{\ln10} \approx 173.72$ to change the base. In this way, our calculation of the expected superiority equals the classic Elo rating algorithm.

[1] Elo et al., The rating of chessplayers: Past and present. 1978.

[2] Elo et. al., Logistic probability as a rating basis. 1986.

[3] The US Chess Rating system. http://www.glicko.net/ratings/rating.system.pdf, 2017.

Second, for the temperature $\tau$, it is initialized as $\tau = 173.72$ which is aligned with Elo coefficient $r$. This choice is motivated by our intention to base the exploration process on the expected superiority of decision steps. After multiple Elo comparisons have conducted, the Elo scores gradually converge to the accurate value. We draw inspiration from the Monte Carlo Tree Search algorithm[4,5] to anneal the exploration probability, thereby directing the exploration process towards more favorable decisions.

[4] Coulom et al., Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. Computers and Games. 2006.

[5] Silver et al., Mastering the game of Go with deep neural networks and tree search. Nature. 2016.

Third, for the initial Elo score of $\hat{d}$, we want to clarify the role of the initial Elo score in our RaDAgent. It is crucial to understand that in the Elo rating algorithm, the decision-making process is influenced by the relative difference between Elo scores of decision steps, rather than their absolute values. You can refer to Equation 1 in our paper. Furthermore, the essence of the Elo rating system lies in its dynamic nature, where scores are adjusted based on pairwise comparisons over time. This means that regardless of the initial score assigned to each decision step, the subsequent adjustments made through pairwise comparisons are what determine the final, accurate assessment of each decision's utility. Therefore, the initial score primarily serves as a starting point, and its specific value is not critical to the overall decision-making process. Consequently, the initial Elo score assigned to decision steps does not fundamentally impact the outcome of the comparisons. 

To further assess the impact of these hyperparameters, we plan to conduct additional experiments in the future. We appreciate your valuable suggestions and will incorporate them into our future work.

Reviewer comment (Experiments):
The reviewer asks how the method works with GPT-4, which is expected to give even better performance.

Author response:
We have conducted additional experiments integrating GPT-4 into our RaDAgent on the Game of 24 scenario . The experimental results are shown in the following Table:

| Method  | Succeed Rate |
|---------|-----------|
| ChatGPT | 43.0      |
| GPT-4   | 54.0      |

We found that RaDAgent, leveraging the enhanced capabilities of GPT-4, demonstrates its superiority over its ChatGPT version. This finding underscores the scalability and adaptability of RaDAgent. It suggests that as LLMs continue to become more sophisticated, our model can leverage these advancements to achieve even greater levels of decision-making efficiency and accuracy.

Reviewer comment (Reproducibility):
The revelation of the experimental details is inadequate. Notably, not all prompts used are revealed, and no examples are provided. Such omissions pose a challenge for reproducibility and make it difficult to pinpoint the source of the observed improvements.

Author response:
Q1: The experimental details if inadequate. Notably, not all prompts used are revealed, and no examples are provided.

A1: Apologize for the inadequate experimental details. We list all prompts of our method and experiments as follows.

The decision-making prompt which is used to make decisions during the experience exploration process is listed as follows:
```
You are the Decision-Making GPT and can perform any task using the tree search method. 
The search method is as follows:
1. First, I will provide you with the task description and input details.
2. For each task, you need to call various functions through multiple steps. 
3. At each step, you need to give your thought to analyze the status now and what to do next, with a function call to actually execute your step.
After the call, you will get the call result, and you are now in a new state.
4. Each (thought-function) pair mentioned above is considered a tree node, and each trail is a tree path from the root to a terminal node. Therefore, the Monte Carlo search tree contains multiple trails.
5. Although you may not see previous trails, in each trail, I will first place you in an intermediate state determined by the "value" of the node, and then you make different choices from there.

Remember:
1. Always make a function call at each step.
2. If you believe you have gathered enough information, call the function "Finish: give_answer" to provide your answer for the task.
3. If you feel unable to handle the task from this step, call the function "Finish: give_up_and_restart".

Let\'s begin!
Task description: {task_description}
```

The task description of Game of 24 is as follows:
```
Use numbers and basic arithmetic operations (+ - * /) to obtain exactly one number 24. In each step, you are only allowed to choose two of the left numbers to obtain a new number. For example, 7 * 9 - 3 * 1 3 = 2 4.
Remember:
1. All of the numbers must be used and must be used ONCE. So Only when the left number is exactly 24, you will win. So you don't succeed when the left number = [2 4, 5]. You succeed when the left number = [2 4]. 
2. All the try takes exactly 3 steps, and the task ends when the count of left numbers is 1, and check whether the only left number equals 24.
3. When there are two numbers left, ALWAYS pre-compute and list all the operations' combined results( + - * / ), and find if there is a way to combine them to 24 before you make the function call. 
3.1. If There is a way, use function "play_24" to combine it.
3.2. If not, use function "give_up" to restart.
4. The status changes ONLY when you call function "play_24". If you only give thoughts, nothing happens.
5. "play_24" inputs only one step, if you want to combine many numbers, split it into multiple calls. 
```

The task description of Webshop is listed as follows:
```
You should use functions to help handle the web shopping task on a webshop site.
We have 2 Pages: Product Selection Page & Product Details Page.
You have access to the following functions:
1. search: at any time, you can search a product by keywords. Then you will go to the Product Selection Page which shows a list of related products.
2. select: after searching keywords, you can select a product on the Product Selection Page. Then you will go to the Product Details Page, which shows the details of the product you select.
3. buy: On the Product Details Page, you can buy a product. Then the shopping task is completed.
```

The task description of ToolBench is listed as follows:
```
You should use functions to help handle real-time user queries. Remember:
1. ALWAYS call \"Finish\" function at the end of the task. And the final answer should contain enough information to show to the user, If you can't handle the task, or you find that function calls always fail(the function is not valid now), use function Finish->give_up_and_restart.
2. Do not use origin tool names, use only subfunctions' names.
You have access to the following tools:\n
```

These prompts will be added in the Appendix section in the final version to provide more experimental details. Moreover, we plan to release all code including the method implementation and experiment settings on Github to ensure reproducibility after acceptance.

Reviewer comment (Reproducibility):
I suggest revealing all experimental details so that the contribution can be properly evaluated and reproduced.

Author response:
Q1: The experimental details if inadequate. Notably, not all prompts used are revealed, and no examples are provided.

A1: Apologize for the inadequate experimental details. We list all prompts of our method and experiments as follows.

The decision-making prompt which is used to make decisions during the experience exploration process is listed as follows:
```
You are the Decision-Making GPT and can perform any task using the tree search method. 
The search method is as follows:
1. First, I will provide you with the task description and input details.
2. For each task, you need to call various functions through multiple steps. 
3. At each step, you need to give your thought to analyze the status now and what to do next, with a function call to actually execute your step.
After the call, you will get the call result, and you are now in a new state.
4. Each (thought-function) pair mentioned above is considered a tree node, and each trail is a tree path from the root to a terminal node. Therefore, the Monte Carlo search tree contains multiple trails.
5. Although you may not see previous trails, in each trail, I will first place you in an intermediate state determined by the "value" of the node, and then you make different choices from there.

Remember:
1. Always make a function call at each step.
2. If you believe you have gathered enough information, call the function "Finish: give_answer" to provide your answer for the task.
3. If you feel unable to handle the task from this step, call the function "Finish: give_up_and_restart".

Let\'s begin!
Task description: {task_description}
```

The task description of Game of 24 is as follows:
```
Use numbers and basic arithmetic operations (+ - * /) to obtain exactly one number 24. In each step, you are only allowed to choose two of the left numbers to obtain a new number. For example, 7 * 9 - 3 * 1 3 = 2 4.
Remember:
1. All of the numbers must be used and must be used ONCE. So Only when the left number is exactly 24, you will win. So you don't succeed when the left number = [2 4, 5]. You succeed when the left number = [2 4]. 
2. All the try takes exactly 3 steps, and the task ends when the count of left numbers is 1, and check whether the only left number equals 24.
3. When there are two numbers left, ALWAYS pre-compute and list all the operations' combined results( + - * / ), and find if there is a way to combine them to 24 before you make the function call. 
3.1. If There is a way, use function "play_24" to combine it.
3.2. If not, use function "give_up" to restart.
4. The status changes ONLY when you call function "play_24". If you only give thoughts, nothing happens.
5. "play_24" inputs only one step, if you want to combine many numbers, split it into multiple calls. 
```

The task description of Webshop is listed as follows:
```
You should use functions to help handle the web shopping task on a webshop site.
We have 2 Pages: Product Selection Page & Product Details Page.
You have access to the following functions:
1. search: at any time, you can search a product by keywords. Then you will go to the Product Selection Page which shows a list of related products.
2. select: after searching keywords, you can select a product on the Product Selection Page. Then you will go to the Product Details Page, which shows the details of the product you select.
3. buy: On the Product Details Page, you can buy a product. Then the shopping task is completed.
```

The task description of ToolBench is listed as follows:
```
You should use functions to help handle real-time user queries. Remember:
1. ALWAYS call \"Finish\" function at the end of the task. And the final answer should contain enough information to show to the user, If you can't handle the task, or you find that function calls always fail(the function is not valid now), use function Finish->give_up_and_restart.
2. Do not use origin tool names, use only subfunctions' names.
You have access to the following tools:\n
```

These prompts will be added in the Appendix section in the final version to provide more experimental details. Moreover, we plan to release all code including the method implementation and experiment settings on Github to ensure reproducibility after acceptance.

Reviewer comment (Experiments):
It would be helpful to conduct an extra experiment comparing the Elo-based evaluation and manually designed value evaluation.

Author response:
Q4: There's a conspicuous absence of an ablation study concerning the Elo-based value evaluation.

A4: We have conducted additional experiments to validate the robustness and efficacy of our Elo-based utility learning algorithm. In the Game of 24 scenario, we tested two variants of RaDAgent to assess the impact of different utility learning approaches:

**Simpler Utility Learning Prompt**: In this variant, we employed a more straightforward utility learning prompt to compare two decision sequences. The prompt is listed as follows:
```
Giving task description and candidate answers, I want you to choose one preferred answer which is more close to success.

*******************************
{{BEGIN_DESCRIPTION}}
your_task: {task_description}
your_query: {input_description}
{{END_DESCRIPTION}}
*******************************

*******************************
{{CANDIDATE_0_START}}
{candidate_A}
{{CANDIDATE_0_END}}
*******************************
{{CANDIDATE_1_START}}
{candidate_B}
{{CANDIDATE_1_END}}
*******************************
```
And the experimental results is as follows:

| Method  | Success Rate |
|---------|-----------|
| DFSDT   | 29.0      |
| SimplePrompt| 36.0      |
| RaDAgent| 43.0      |

From the results, we can observe that while there was a decrease in performance with the simpler prompt, RaDAgent still outperforms the best baseline DFSDT. This results highlights a couple of key points:
- **Impact of Prompt Design**: The experiment demonstrated that the design of the utility learning prompt does indeed impact the performance of the system. A more complex or carefully crafted prompt contributes to better utility assessment, leading to more effective decision-making.
- **Robustness of Utility Learning**: Despite the reduced performance with a simpler prompt, the fact that RaDAgent continued to outperform the baseline indicates the inherent robustness of our utility learning approach. It suggests that while the prompt design is significant, the core mechanics of our Elo-based utility learning algorithm are strong enough to maintain a competitive edge even under suboptimal conditions.

These findings unveil the need for further research into the optimal design of utility learning prompts. We plan to explore a broader range of prompt complexities and styles to fully understand their impact on the efficacy of the utility learning process in the future.

**Manual-Designed Strategy Comparison**: We also experimented with a manual-designed strategy for comparing decision sequences, instead of LLMs. The strategy is to compare which decision sequence is close to 24 (i.e., the difference between the final calculation result of four number and 24). The closer one wins. If their difference are the same, they tied. The experimental results are as follows:

| Method  | Success Rate |
|---------|-----------|
| DFSDT   | 29.0      |
| Manual  | 26.0      |
| RaDAgent| 43.0      |

The performance of this variant was notably inferior to that of the standard RaDAgent even DFSDT. This is the manual-designed strategy acts as an external performance measure and it may flawed or even erroneus, which would mislead the decision-making process. This outcome highlights the limitations of manual strategies and affirms the importance of the internal rationality embedded within our RaDAgent for decision-making.

Reviewer comment (Theory):
Why is the Boltzmann distribution used? How about using upper confidence tree for searching?

Author response:
Q2: How about using upper confidence tree for searching?

A2: We are thankful for your suggestion to consider the upper confidence tree (UCT) approach. UCT is indeed a powerful mechanism that uses multiple Monte Carlo simulations to accurately assess the utility of each decision node, with the precision of assessment improving as the number of simulations increases. However, in the context of LLM-based decision-making, conducting simulations for each decision step would entail numerous ChatGPT API calls. This would not only prove inefficient but also costly. Instead, we utilize LLM comparisons to estimate the utility of decision steps. By leveraging the Elo algorithm, which inherently provides a measure of expected performance based on historical comparisons, we achieve significant efficiency. The Elo algorithm's capacity for rapid utility evaluation without the need for extensive simulations is particularly advantageous in our application. Our experiments in Section 4.4 also demonstrate the superiority of RaDAgent in terms of efficiency.

Reviewer comment (Evaluation):
The paper mentions (Sec 3.2) the initial elo scores for a decision sequence (iteration 1) are fixed. How are comparisons performed? IE how is a "win" determined? I assume it is binary (task completed or not). In this case, are two decision sequences where both are "wins" but one is longer than the other compared? (Longer ones require more API calls I assume).

Author response:
We appreciate the opportunity to clarify these aspects of Elo rating details and comparison determination in the utility learning.

- **How are comparisons performed**: Regarding the comparisons, they are entirely agent-centric, devoid of any external or prior knowledge. This design choice is intentional: **Our goal is to validate the inherent capacity of the LLM-based agent to make rational utility judgments.** You can refer to Appendix A. Specifically, ChatGPT receives two decision sequences (i.e., ChatGPT calling messages) and then judges the better one to return its corresponding index.
- **How is a "win" determined**: The term 'win' in our context is not a binary indicator to judge whether a decision sequence successfully completes a task. Instead, it assesses which decision sequence demonstrates **superior performance** during the task completion process. Thus, even if two decision sequences both accomplish the task, our method will still judge the better one according to their decision sequence. This comparative approach aims to discern the relative efficacy of different decision sequences, rather than making an absolute determination of task completion. Additionally, the comparison result is triple: A wins B, B wins A, or a Tie, reflecting scenarios where decision sequences A and B exhibit similar performance levels.
-  **How to compare two that one is longer requiring more API calls**: Our empirical observations indicate that it depends on the specific context of decision sequences. For instance, if a shorter sequence fails while a longer one succeeds, RaDAgent will opt for the longer, successful sequence. Conversely, when both short and long sequences succeed, RaDAgent exhibits a tendency to favor the shorter, more efficient sequence. This preference aligns with intuitive human judgment, reinforcing the notion that LLM-based agents possess the capability to discern and prefer more efficient decision-making pathways.
- **Initial Elo score is fixed**: The initial Elo score is fixed does not mean the Elo score is fixed during the utility learning process. **The Elo score will be updated through comparison to converge to its accurate value.** Given a new decision sequence, RaDAgent will first assign it with an initial Elo score and then multiple comparisons will be conducted with existing decision sequences to adjust this decision sequence Elo score according to Equation 3 immediately. Hence, the initial Elo score just serves as a standardized starting point for evaluation, ensuring that subsequent Elo score adjustments in an unbiased and consistent manner.

Reviewer comment (Evaluation):
Building upon #2 above, the paper claims that there is a performance measure absent. That would mean that there is no real way of determining the final utility or "win". I think that there is some human-expert knowledge that is required for RADAGENT to be able to function. Could you please clarify this?

Author response:
We acknowledge the inquiry regarding the role of human-expert knowledge in RaDAgent's functionality. It is important to clarify that the integration of such knowledge is seen as complementary to, rather than a replacement for, the agent's inherent rational decision-making capabilities. While expert knowledge can indeed augment the agent’s decision-making process, making it more effective in certain contexts, reliance on such knowledge alone is not without drawbacks.

Primarily, external human expertise can sometimes be limited, flawed, or even erroneous. Additionally, it is impractical to infuse an all-encompassing array of prior knowledge into a general-purpose agent, especially considering the vast and varied nature of real-world tasks. Therefore, in our study, we emphasize that a general-purpose agent like RaDAgent should not solely depend on external human expertise. Instead, it should leverage its built-in rationality to make autonomous decisions, especially in scenarios where reliable external guidance is unavailable or insufficient.

In essence, our approach positions the agent's internal rationality as a fundamental aspect of its decision-making process, while still acknowledging the potential benefits of integrating human expertise where feasible and appropriate. This dual approach ensures that the agent remains adaptable and effective across a wide range of tasks, even in the absence of specific external inputs.

Reviewer comment (Experiments):
Why the specific use of Elo and not a different rating system like Glicko? There is not sufficient motivation for the choice of Elo.

Author response:
Our primary objective in this work was to underscore the significance of internal rationality for the agent in decision-making processes, particularly in evaluating the utility of each choice to determine the optimal one. In this context, the crucial factor was to employ a method that could provide a quantitative assessment of utility through a straightforward yet robust mechanism. We opted for the Elo rating system due to its simplicity and proven effectiveness, especially in environments where pairwise comparisons are pivotal. This system's straightforward nature aligns well with our objective to demonstrate the agent's capability to internalize utility judgment without overly complex calculations or dependencies. Our experimental results affirm that the Elo rating algorithm successfully showcases the effectiveness of internalizing utility judgment in the decision-making process.

While Elo was chosen for its current applicability, the reviewer's suggestion of exploring alternative rating systems like Glicko is indeed valuable. We plan to consider this insightful direction in our future research to further enhance our understanding of rational decision-making in the agent.