Paper: The Update-Equivalence Framework for Decision-Time Planning (ICLR-2024)

Reviewer comment (Theory):
What are the objectives of the algorithms (global policy learners and DTP algorithms)? Do they compute an equilibrium of the game, or do they just compute policies for the next iteration? What is the solution concept applied for the games studied?

Author response:
> What are the objectives of the algorithms (global policy learners and DTP algorithms)?

The definitions of these classes of algorithms (global policy learners and DTP algorithms) specify the structure of the updates they induce, not their objectives. For global policy learners, the update is over the whole policy at once, whereas for DTP planning algorithms, it is only over a single decision-point.

> Do they compute an equilibrium of the game

Whether or not a global policy learner or DTP algorithm computes an equilibrium of the game depends on the algorithm and the game. For example, counterfactual regret minimization and policy gradient can both be considered as global policy learners. However, while counterfactual regret minimization computes equilibria in two-player zero-sum games, policy gradient does not (though it sometimes still performs well in practice).

> do they just compute policies for the next iteration?

Global policy learners compute policies for the next iteration. Decision-time planning algorithms compute policies for the current decision point.

> What is the solution concept applied for the games studied?

Solution concepts aren't necessarily the best way of thinking about the submission's framework in generality. The submission shows that decision-time planning algorithms can inherit the one-step guarantees of global algorithms. These guarantees can be related to solution concepts, but don't have to be. Of the algorithms studied in the experiments section, MD-UES possesses a guarantee on expected return improvement, while the results of Sokota et al. (2023) suggest that MMD-UES may possess guarantees related a decrease in exploitability. These guarantees are related to Nash equilibria in the sense that Nash equilibria are fixed points of the updates. However, at decision-time, we are only doing one-step improvements (and therefore not converging arbitrarily close to equilibria).

Reviewer comment (Writing):
What does \( g \) represent in \( U^{global}: (\pi_t, g) \rightsquigarrow \pi_{t+1} \) when you introduce the global policy learners?

Author response:
> What does $g$ represent when you introduce the global policy learners?

As is defined in Section 2, $g$ just represents the game itself (i.e., the states, actions, transition function, reward functions, etc.).

Reviewer comment (Theory):
It is said that squiggly arrows are used to indicate randomness in the output. When you write \( U_Q^{global} : \Delta(\mathbb{A}_i) \times \mathbb{R}^{|\mathbb{A}_i|} \rightsquigarrow \Delta(\mathbb{A}_i) \), the output is already a distribution over \(\mathbb{A}_i\). Do you mean that there is a further randomization over the output distribution (i.e., a distribution over \(\mathbb{A}_i\) is randomly chosen)?

Author response:
> Do you mean that there is a further randomization over the output distribution (i.e., a distribution over $\mathbb{A}_i$ is randomly chosen)?

Yes.

Reviewer comment (Writing):
What is \(\mathcal{P}_\pi\) in Algorithm 1? It seems undefined. What is \(G\) samples in the algorithm?

Author response:
> What is $\mathcal{P}_{\pi}$ in Algorithm 1? It seems undefined.

Thanks for pointing this out. $\mathcal{P}_{\pi}(h)$ is the probability of reaching history $h$ using joint policy $\pi$. We added this to the notation in Section 2 to clarify this issue.

> What is $G$ samples in the algorithm?

$G$ is the return, as is stated in the pseudocode text. We added this to the notation in Section 2 to clarify this issue.

Reviewer comment (Writing):
Several notations are never defined, such as $S^t$, $A^t$, $H^{t+1}$, $h_i^t$, $h_i$, etc.

Author response:
> Several notations are never defined, such as $S^t, A^t, H^{t+1}, h_i^t, h_i$, etc.

The listed notations listed by the reviewer are defined in Section 2 of the submission:
- "$s \in \mathbb{S}$ to notate Markov states"
- "$a \in \mathbb{A} = \times_i \mathbb{A}_i$ to notate joint actions"
- "$h \in \mathbb{H} = \times_i \mathbb{H}_i$ to notate histories"
- "$h_i \in \mathbb{H}_i = \cup_t (\mathbb{O}_i \times \mathbb{A}_i)^t \times \mathbb{O}_i$ to notate i’s decision points"
- "we use capital letters to denote random variables"

Reviewer comment (Writing):
The definitions of global policy learner and decision-time planning are not clear.

Author response:
> The definitions of global policy learner and decision-time planning are not clear.

Could the reviewer specify exactly what is unclear?

Reviewer comment (Theory):
Proposition 3.2 shows the connection between $\mathcal{U}^{global}$ and $\mathcal{U}^{global}_Q$. However, it is unclear how Algorithm 1 could convert a global policy learner to a decision-time planner, as claimed in the paper.

Author response:
> Proposition 3.2 shows the connection between $\mathcal{U}^{\text{global}}$ and $\mathcal{U}_Q^{\text{global}}$. However, it is unclear how could Algorithm 1 converts a global policy learner to a decision-time planner, as claimed in the paper.

Algorithm 1 is a decision-time planning algorithm (by construction) and takes the local update function of a global policy learner $\mathcal{U}_Q^{\text{global}}$ as input (by construction).

Reviewer comment (Theory):
It is unclear what is meant by policy iteration local updating function, how to plug it into Algorithm 1, and why it can turn the algorithm into Monte Carlo search.

Author response:
> What does it mean by policy iteration local updating function?

We don't use the term "policy iteration local updating function" in the submission. We do use the term "policy iteration's local update function". This is defined in the text: 
"The local update induced by policy iteration can be written as $U_Q^{\text{global}} \colon (\bar{\pi}, \bar{q}) \mapsto \sigma$ where $\Sigma_{a \in \text{arg max } \bar{q}} \sigma(a) = 1$, where $\bar{\pi}$ and $\bar{q}$ refer to generic elements of $\Delta(\mathbb{A}_i)$ and $\mathbb{R}^{|\mathbb{A}_i|}$, respectively."

> How to plug it into Algorithm 1?

$U^{\text{global}}_Q$ is applied on the final line of the pseudocode, as is written in the text.

> Why it can turn the algorithm to Monte Carlo search?

As is discussed in the text, Algorithm 1, given policy iteration's $U^{\text{global}}_Q$, coincides with Monte Carlo search. Both estimate action values using rollouts and select an action with a maximal action value.

Reviewer comment (Theory):
Does Proposition 3.2 rely on the definition of global policy learner operating with action-value feedback?

Author response:
> Does Proposition 3.2 rely on the definition of global policy learner operating with action-value feedback?

Yes, because Proposition 3.2 makes is a statement about global policy learners operating with action-value feedback it is inherently tied to the definition of global policy learners operating with action-value feedback.

Reviewer comment (Theory):
How to guarantee the desired connection between $\mathcal{U}^{global}$ and $\mathcal{U}^{global}_Q$ exists, and if so, how to obtain the corresponding function?

Author response:
> How to guarantee the desired connection between $\mathcal{U}^{\text{global}}$ and $\mathcal{U}^{\text{global}}_Q$ exist, and if so, how to obtain the corresponding function?

There is no such guarantee. The reviewer may misunderstand the purpose of this connection. It's not to show this connection exists for every $\mathcal{U}^{\text{global}}$ (it doesn't); it's to exploit this connection for specific $\mathcal{U}^{\text{global}}$ for which the connection does exist. For example, the desired connection exists for mirror descent and magnetic mirror descent, which allowed us to derive the search algorithms stuied in the paper.

Reviewer comment (Experiments):
The requirement that algorithm needs to exhaust all the search budget seems inefficient and the performance highly depends on this computational costs.

Author response:
There may be a misunderstanding about the usage of terminology here. In search literature, it is common to say "until search budget is exhausted" to mean "do search for as long as is convenient". There is no requirement to fully exhaust any particular budget. Indeed, the class of algorithms we empirically investigate fall under the class "anytime algorithms" which is a term that means that they can be stopped at any time.

Reviewer comment (Presentation):
I would prefer a separate "related works" section to make the presentation more clear.

Author response:
To clarify, there is already a separate related works section (Section 5) with about a page of content.

Reviewer comment (Novelty):
How does the proposed framework of update equivalence fundamentally differ from the traditional subgame-based approach in DTP? Can authors put the differences or advantages into several points so that the ideas are clear?

Author response:
Conceptually they can be seen as doing two different things. Subgame-based approaches define an auxiliary game and try to solve that game as best as possible. Update equivalence approaches try to approximate a pre-defined update step.

An advantage of subgame-based approaches is that one can sometimes achieve a larger improvement by solving the entire subgame than would be possible by performing a single update step. An advantage of update equivalence-based approaches is that can you sidestep the need to construct a subgame, which can be expensive or intractable in imperfect information settings. For example, even in 7-card Hanabi, it becomes very difficult to employ public belief state-based methods because the public belief state is too large to compute exactly.