Paper: C-MCTS: Safe Planning with Monte Carlo Tree Search (ICLR-2024)

Reviewer comment (Experiments):
The time complexity of C-MCTS seems higher than normal MCTS. In the expansion of C-MCTS, the safety critic is used to evaluate all actions. This greatly increases the computing resources when applying in the large action space domains or using more MCTS simulations. In addition, considering C-MCTS uses additional resources to tune a Lagrange multiplier, the comparison between C-MCTS and CC-MCP is unfair. The authors should provide the detailed time complexity for C-MCTS, and the time for the pre-training phase for comparison.

Author response:
Thank you for your question! You are right that we must evaluate the safety critic for each action the agent might take in a state. However, as our safety critic is a neural network that output cost estimates for all the available actions through single forward pass (just like in a DQN) its constant time and only little overhead in addition to CC-MCP and MCTS.

Please also find below a more rigorous explanation of incurred costs that we also lined out to reviewer q5zX and that we have added to the appendix of the paper:

In terms of computational cost per simulation across the different algorithmic phases:
- Selection: CC-MCP is the most computationally expensive, requiring more operations to select the best child node. MCTS and C-MCTS have identical operation counts.
- Expansion: C-MCTS incurs additional costs due to the safety critic's prediction. MCTS and CC-MCP require no additional computation during expansion.
- Backpropagation: CC-MCP backs up Q-values for both reward and cost, while MCTS and C-MCTS back up only Q-values for reward.
- Rollout: Computational cost is identical for C-MCTS, MCTS, and CC-MCP.
C-MCTS and MCTS algorithms were implemented in Python, while the benchmark CC-MCP uses a C++ implementation. Comparing actual execution times was unfair since our implementation was not optimized for hardware efficiency. Also, such an optimization would highly depend on the hardware platform (e.g. CPUs vs GPUs). For instance, added cost in C-MCTS's expansion phase is highly parallelizable, with good potential for effective GPU utilization. So, for our analysis we instead compare the number of simulations (planning iterations) required by each algorithm as a performance metric.

Reviewer comment (Presentation):
P2, Figure 1. As introduced in Section 3.1, the training phase uses MCTS to solve MDP, which should also be added to the figure for better understanding.

Author response:
Thank you for this suggestion. We changed the Figure accordingly.

Reviewer comment (Writing):
P2, Section 2.1. Use bold font to emphasize sets (especially for $C$ and $\hat{c}$) in definitions and equations for clear reading. All policies $\Pi$ is undefined in equation (1).

Author response:
Thank you for pointing this out. Following the notation convention of [1], $\mathbf{C} = \\{C_m\\}\_{1\ldots M}$ is a set of $M$ non-negative cost functions, with $\mathbf{\hat{c}} = \{c_m\}_{1\ldots M} \in [0,1]$ their respective thresholds. In the text, we assume only one constraint function $C$ with its respective threshold $\hat{c}$, to simplify the notation.

In any case, we have re-written this point more clearly in the new version of the text.

[1] Lee, Jongmin, et al. "Monte-Carlo tree search for constrained POMDPs." Advances in Neural Information Processing Systems 31 (2018).

Reviewer comment (Writing):
P3, Section 2.2, "Lee et al. (2018) proposed Cost-Constrained Monte Carlo Planning (CC-MCP)". This is incorrect since Lee et al. proposed Cost-Constrained POMCP (CC-POMCP) instead of CC-MCP. Although it applies to CMDPs, the authors should revise the statement to clarify this.

Author response:
Thank you for pointing this out. We have corrected this in the new version of the paper.

Reviewer comment (Writing):
P3, Section 3.1. Clarify what "e.g., in simulation" means. Does it mean to vary parameters during MCTS simulation in the pre-training phase? In addition, $V_{C}^{k,*}$ is undefined.

Author response:
Good point! The correct sentence is: “Instead of tuning the Lagrange multiplier online, C-MCTS varies this parameter in a pre-training phase (in simulation) to obtain different sets of trajectories with different safety levels (“data gathering” in Fig. 1).

Our approach is as follows: we have a low- and a high-fidelity simulator that approximate a real-world problem/environment. The pre-training phase (“data gathering” phase in Figure 1) consists of varying the Lagrange multiplier \lambda and finding the optimal (MCTS) policy and V_C for this value of \lambda (“solve” the MDP). Here, the low-fidelity simulator is used for the planning phase of MCTS, while the high-fidelity simulator is used for training the cost approximation V_C.

Also, we have added the definition for $V_{C}^{k, *}$

Reviewer comment (Theory):
P4, Section 3.1.1, Definition 3. In equation (5), it seems meaningless to define $\mathop{\min}_{a'} {Q^{\pi}_{c}(s,a')} <= Q^{\pi}_{c}(s,a)$ when $\forall a' \in A$.

Author response:
Correct! As we are striving to find a feasible solution that does not violate the constraints, this first part of Eq. 5 has no effect on the design of a policy leading to minimum (or no) constraint violations. We simply added the term there to highlight clearer the connection to related work [1,2].

[1] Kumar, Aviral, et al. "When should we prefer offline reinforcement learning over behavioral cloning?." arXiv preprint arXiv:2204.05618 (2022).

[2] Liu, Zuxin, et al. "Constrained decision transformer for offline safe reinforcement learning." arXiv preprint arXiv:2302.07351 (2023).

Reviewer comment (Theory):
P4, Proposition 2, "This is not only numerically infeasible, but it is also due to the utilization of the low-fidelity simulator in the MCTS planner." It is still unclear why this is numerically infeasible. In common sense, even if a low-fidelity simulator is employed, it is still likely that the critic can provide perfect estimation for some state-action pairs, especially for those near the terminal states.

Author response:
Good point. We were thinking along the lines of numerical precision of the estimator, but we agree with the argument that an estimation can be precise for selected classes of problems, especially in state-action pairs near the terminal states. Therefore, we have relaxed the proposition as follows:

“Proposition 2. Let $B = \{(s,a) | s \in S\ \text{and}\ a \in A\}$ be the set of all state-action pairs for a given~\ac{MDP}. Then, there exist $B_p \subseteq B$, a set of state-action pairs for which the trained safety critic would over-estimate the expected discounted cumulative cost and $B_n \subseteq B$, a set of state-action pairs for which the trained safety critic would under-estimate it.

What Proposition 2 indicates is that the trained safety critic will under-estimate or over-estimate the expected cost of every state-action pair defined in the underlying MDP of the high-fidelity simulator. If this was not true, this would imply that the safety critic provides the perfect prediction at least for all state-action pairs. This is rarely the case, both due to numerical precision issues, as well as due to the utilization of the low-fidelity simulator in the MCTS planner, which potentially predicts sequences of safe or unsafe next states that are different compared to the actual ones, especially for state-action pairs that are far from the terminal states.”

Still, we would argue that this does not change the behavior of the algorithm in case of cost over- or under-estimation by the safety critic, hence does not affect Corollary 1.

Reviewer comment (Writing):
P6, Section 4, "the selected environments clearly "emulate" the complexity that stems from domains with large state/action spaces (Schrittwieser et al., 2020b; Afsar et al., 2022)." Please clarify the reason for citing these two papers here. They do not seem to have any discussion about the selected environments.

Author response:
We apologize for not being clear here. Our intention was to conclude the discussion of this paragraph, related to selecting representative problems/environments that are not computationally intensive [1], to be able to run several MCTS planning iterations in a “reasonable” amount of time in moderate hardware infrastructure. Considering the fact that we are evaluating with 100 different seeds for the results to have statistical significance [2], this requirement becomes even more pressing.

Having this in mind, previous work using MCTS (or tree-based planners in general), has been able to address problems with extremely large state (e.g., Atari, go, chess and sogi in [3]) and action spaces (e.g., see representative references in [4] for approaches that facilitate some form of clustering of potential actions). Therefore, we have tried – to the best of our ability – to define “simple” environments that provide insights on properties and the quality of the final, feasible solution of our algorithm, with respect to the constraint formulation, considering that scalability issues of MCTS have already been addressed in previous work.

We have changed the text in the newest version of the paper to better clarify our views.

[1] Togelius, Julian, and Georgios N. Yannakakis. "Choose your weapon: Survival strategies for depressed AI academics." arXiv preprint arXiv:2304.06035 (2023).

[2] Henderson, Peter, et al. "Deep reinforcement learning that matters." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

[3] Schrittwieser, Julian, et al. "Mastering atari, go, chess and shogi by planning with a learned model." Nature 588.7839 (2020): 604-609.

[4] Afsar, M. Mehdi, Trafford Crump, and Behrouz Far. "Reinforcement learning based recommender systems: A survey." ACM Computing Surveys 55.7 (2022): 1-38.

Reviewer comment (Experiments):
P8, Section 4.2, "with a grid search". Did the optimized $\alpha_0$ and $\epsilon$ highly affect the final performance? The authors should provide the grid search results in the appendix for reference.

Author response:
We agree that the results of the grid search should be mentioned in the paper. Unfortunately, we did not keep the full results which is why we just started the grid search again. While the results will not be ready by the rebuttal deadline, we will include them in the final version of the paper.

Reviewer comment (Theory):
About proof of Proposition 1 in P.4. "... guaranteed to find the optimal solution in the k-th MDP." UCT's proof is asymptotic, and we need an infinite number of simulations to guarantee the convergence to the optimal. Also, we have no way of knowing whether we actually reached the optimal (in my understanding). Does this proof assume that we do an infinite number of simulations on each iteration? Is that valid?

Author response:
Fair point. As MCTS with upper confidence bounds converges asymptotically to the optimal policy [1], usually a time- or computational budget-limit is used to terminate learning [2,3]. In our case, since we are interested in a feasible solution, we terminate the training process (search for $\lambda^*$ effectively) in the data gathering phase only when enough data has been gathered and the cost constraints are satisfied (see ``data gathering'' phase in Fig. 1).

To solve each of the MDPs we use a sufficiently high number of simulations that ensures a nearly-optimal policy (refer Figure 5, comparing performance for low and high number of simulations). When the costs observed are close to the constraint (but not violated), determined by $\hat{c}-\epsilon \leq V_{C}^{k, *} \leq \hat{c}$,  the iterative process is terminated.

We have adapted the text accordingly to clarify this.

[1] Kocsis, Levente, and Csaba Szepesvári. "Bandit based monte-carlo planning." European conference on machine learning. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006.

[2] Silver, David, et al. "Mastering the game of Go with deep neural networks and tree search." nature 529.7587 (2016): 484-489.

[3]  Schrittwieser, Julian, et al. "Mastering atari, go, chess and shogi by planning with a learned model." Nature 588.7839 (2020): 604-609.

Reviewer comment (Experiments):
I may have overlooked, but how long does this step take in the actual experiments in Section 4? (Is it the "Solve MDP" step in Fig. 1?)

Author response:
To solve the k-th MDP during the iterative training process, we use 1024 simulations for Rocksample(5,7) and Rocksample(7,8), and 512 simulations for Rocksample(11,11) and Safe Gridworld (Refer to column 'Planning Iterations' in Table 2 in A.2).

Reviewer comment (Evaluation):
How meaningful is it to count the number of cost violations? In what problem settings are we not allowed to violate the cost even during the simulations?

Author response:
Let us answer depending on the actual question you ask:
- If the question is related to the type of constraints selected for the experimental evaluation, then we would like to clarify that the safety critic can be trained to approximate any type of safety or cost constraint, like number of cost violations, magnitude of violation or combinations of them. We selected environments that count the number of constraint violations to enable a more clear evaluation of the results and comparison with the baseline.
- If the question is related to how we evaluate C-MCTS, e.g. in Figs. 2 (bottom row) and 4 (bottom row): please note that here we count the cost constraint violations during /actual/ deployment of the agent (of course we still run the agent in a simulator, but here we do not count the violations that the agent would do in the simulation step of MCTS itself). Hence, this would lead to an actual possibly harmful execution of an action in the real world. Here we also stick to the convention that constraints should be hard and not being violated.

Reviewer comment (Evaluation):
For Fig. 2, how was the actual execution time of the three algorithms? Also, for the Safe GridWorld experiment, did C-MCTS with $2^9$ planning iterations and CC-MCP with $2^{20}$ planning iterations take a similar amount of time?

Author response:
In terms of computational cost per simulation across the different algorithmic phases:
- Selection: CC-MCP is the most computationally expensive, requiring more operations to select the best child node. MCTS and C-MCTS have identical operation counts.
- Expansion: C-MCTS incurs additional costs due to the safety critic's prediction. MCTS and CC-MCP require no additional computation during expansion.
- Backpropagation: CC-MCP backs up Q-values for both reward and cost, while MCTS and C-MCTS back up only Q-values for reward.
- Rollout: Computational cost is identical for C-MCTS, MCTS, and CC-MCP.
C-MCTS and MCTS algorithms were implemented in Python, while the benchmark CC-MCP uses a C++ implementation. Comparing actual execution times was unfair since our implementation was not optimized for hardware efficiency. Also, such an optimization would highly depend on the hardware platform (e.g. CPUs vs GPUs). For instance, added cost in C-MCTS's expansion phase is highly parallelizable, with good potential for effective GPU utilization. So, for our analysis we instead compare the number of simulations (planning iterations) required by each algorithm as a performance metric.

We have also added this information to the appendix of the paper.

Reviewer comment (Experiments):
About Fig. 4. What is one episode here? Does one episode for C-MCTS and CC-MCP require a similar amount of time? If the latter is faster, could we run CC-MCP 100 times and select the path based on the majority?

Author response:
Here, one episode corresponds to an agent starting from one of the four squares in the bottom-left of the grid and reaching the green square in the top-right corner. Fig 4. was evaluated for 100 episodes i.e., both CC-MCP and C-MCTS were run 100 times each. C-MCTS uses $2^9$ planning iterations and CC-MCP uses $2^{20}$ planning iterations, hence the former is much cheaper. Here, running CC-MCP more times wouldn’t improve its performance since the cost estimates for CC-MCP are dependent on the planning model. A model mismatch in the planner has introduced inaccuracies, rendering additional runs less beneficial.

Reviewer comment (Writing):
I got confused with the notation of $C, c, \hat{c}$. Are these sets or functions?

Author response:
We apologize for not being clear at this point. Following the notation convention of [1], $\mathbf{C}=\\{C_m\\}\_\{1\\ldots M\}$ is a set of $M$ non-negative cost functions, with $\mathbf{\hat{c}} = \{c_m\}_{1\ldots M} \in [0,1]$ their respective thresholds. In the text, we assume only one constraint function $C$ with its respective threshold $\hat{c}$, to simplify the notation.

In any case, we have re-written this point more clearly in the new version of the text.

[1] Lee, Jongmin, et al. "Monte-Carlo tree search for constrained POMDPs." Advances in Neural Information Processing Systems 31 (2018).