Paper: Best Possible Q-Learning (ICLR-2024)

Reviewer comment (Theory):
Lemma 2 is a crucial lemma for justifying the best possible operator. However, the second equality appears to be incorrect. Could the authors explain how it holds?

Author response:
We made a correction in the revision (the colored line). The $\max _{a_i}$ should be factored out. After the correction, the proof still holds because $a^{'*}_i$ maximizes the second term but does not maximize the first term.

Reviewer comment (Theory):
Is it a common setting that the reward function is only dependent on the current state and the next state (without the dependency on the action)? To me this is not commonly seen.

Author response:
The setting is still general because if two joint actions lead to the same transition dynamics in the environment, the two actions should have the same effects, and we do not need to discriminate the actions. The effects of actions are reflected in the distribution of $s'$.

Imagine that we are playing a computer game. There is a certain state where the left key and the right key can lead to the same next state or the same distribution of the next state. Do we need to give the right key a higher reward to encourage the players to always select the right key?

Reviewer comment (Theory):
I think (3) (where $Q(s,\boldsymbol{a})$ is equipped with some arbitrary $\boldsymbol{a}$) is not the expected return of the optimal joint policy.

Author response:
$Q(s, \boldsymbol{a})$ means the expected return if you start in state $s$, take an arbitrary joint action $\boldsymbol{a}$, and then forever after act according to the optimal policy in the environment. (https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#value-functions)

Reviewer comment (Theory):
In the proof of lemma 1, I notice that there is an underlying assumption that the actions taken by other agents do not restrict the allowable actions of each agent. If this restriction is considered, does the best possible learning strategy still apply?

Author response:
Yes, fully decentralized learning requires the assumption that the agents can independently make decisions. If we consider the restriction of the allowable actions of each agent, we can train the agents in a modified MDP. If the selected joint action breaks the restriction, the MDP ends and the agents receive a very large negative reward. Then the optimal joint policy in the modified MDP will be equal to the optimal joint policy in the original MDP, and agents can independently make decisions in the modified MDP.

Reviewer comment (Writing):
The 3rd line under problem (6), could the authors provide exactly where is the reference in (Puterman, 1994)?

Author response:
Theorem 7.1.9. of (Puterman, 1994) shows that there is at least one deterministic optimal joint policy. Therefore, when there is only one optimal joint policy, the optimal joint policy must be deterministic.

Reviewer comment (Theory):
In Lemma 2, the line  
$=\gamma  P_i^*(s^′|s, a_i)  \max_{a_i^′} (\max_{a_{-i}^′} Q(s^′, a_i^′ , a^′_{-i} ) − Q^{k−1}(s^′, a_i^′ ))$  
appears to be incorrect. The $\max_{a_i}$ cannot be factored out here since the maximum actions can be different for the two Q value terms inside the parenthesis and there is a minor sign. Apart from this, the remainder of the lemma appears to be correct.

Author response:
>  1.a.Lemma 2

Sorry for this mistake. The $\max _{a_i}$ should be factored out. We have corrected it in the revision (the colored line). After the correction, the proof still holds because $a^{'*}_i$ maximizes the second term but does not maximize the first term.

Reviewer comment (Theory):
In Lemma 3, the proof’s second inequality appears to be valid but seems to omit intermediate steps. The absence of these steps makes it challenging to follow the reasoning. Elaborating on these steps would enhance the clarity of the proof.

Author response:
>  1.b.Lemma 3 and 1.c.Lemma 4:

For the second inequality of Lemma 3, we replace the expectation over transition probabilities with the infinite norm (which is larger). We will elaborate on the proofs of these two lemma in the final version with additional pages.

Reviewer comment (Theory):
In Lemma 4, the statement “Similar to the proof of Lemma 2, we can easily prove $\max_{a_{−i}} Q(s, a_i, a_{−i}) \leq Q^k_i (s, a_i)$” seems valid, but it may not be as straightforward as implied, especially considering the nuances in this context. The simplified operator incorporates an additional max operator (8), compared to operator (6) used in Lemma 2. A more detailed elaboration of the steps involved in this proof would be beneficial.

Author response:
>  1.b.Lemma 3 and 1.c.Lemma 4:

For the second inequality of Lemma 3, we replace the expectation over transition probabilities with the infinite norm (which is larger). We will elaborate on the proofs of these two lemma in the final version with additional pages.

Reviewer comment (Theory):
The conditions in the convergence proof for both operators seem too strong to hold in practice. There seems to be a lack of sufficient attention to how these conditions might be met, leading to doubts around the practical relevance of the operators.

Author response:
> 2. As I will detail in the following, the conditions in the convergence proof for both operators seem too strong to hold in practice. There seems to be lack of sufficient attention to how these conditions might be met, thus leading to doubts around the practical relevance of the operators.

There are four questions in this part. For the first two questions, in theoretical analysis, it is reasonable to assume that the expected values over transition probabilities are computed exactly at every step. Even Q-learning makes such assumptions. In the implementation, as DQN, we use TD-error by sampling from a non-stationary replay buffer to fit the expected values under $\tilde{P}_i$. This approximation technique is commonly adopted, so we believe your main concern is the next two questions about approximation error.

For the last two questions, the main concern is the overestimation caused by approximation error. We use Eq. 10, the weighted max update, to overcome this problem. We can choose a large $\lambda$ to mitigate overestimation. This technique is proposed in Hysteretic IQL to solve the overestimation in Distributed IQL (as discussed in Appendix B), and has been verified to be practical in decentralized learning. This solution similar to quantile regression has been used in many RL methods, e.g., implicit Q-learning. Therefore, we follow this practical solution, and experimental results show that this solution can avoid severe overestimation.

Reviewer comment (Theory):
An approximation error in computing the expectations, when combined with the max operation, could cause overestimation of Q values. This issue, briefly mentioned in the text following Eq. (10), lacks a clear explanation. The approximation error would also lead to violation of the Q value upper bound, undermining the convergence guarantees for both operators.

Author response:
> 2. As I will detail in the following, the conditions in the convergence proof for both operators seem too strong to hold in practice. There seems to be lack of sufficient attention to how these conditions might be met, thus leading to doubts around the practical relevance of the operators.

There are four questions in this part. For the first two questions, in theoretical analysis, it is reasonable to assume that the expected values over transition probabilities are computed exactly at every step. Even Q-learning makes such assumptions. In the implementation, as DQN, we use TD-error by sampling from a non-stationary replay buffer to fit the expected values under $\tilde{P}_i$. This approximation technique is commonly adopted, so we believe your main concern is the next two questions about approximation error.

For the last two questions, the main concern is the overestimation caused by approximation error. We use Eq. 10, the weighted max update, to overcome this problem. We can choose a large $\lambda$ to mitigate overestimation. This technique is proposed in Hysteretic IQL to solve the overestimation in Distributed IQL (as discussed in Appendix B), and has been verified to be practical in decentralized learning. This solution similar to quantile regression has been used in many RL methods, e.g., implicit Q-learning. Therefore, we follow this practical solution, and experimental results show that this solution can avoid severe overestimation.

Reviewer comment (Theory):
For the explanation on the algorithm and theoretical justification, the writing is unclear in general. There are several arguments I cannot verify, and I suspect that there are flaws in the proof of the convergence. Please see Questions below. This also largely hinders my understanding of the algorithm.

Author response:
In theory we use the real transition probabilities, and in algorithm we collect experiences to estimate the value under the transition probabilities by TD-error. It is the standard paradigm in reinforcement learning. **The proofs are based on the real transition probabilities**, so the proofs are correct. In algorithm, we address the overestimation using a practical technique (Eq. 10), which is a commonly used technique. Experimental results show that this technique can avoid overestimation.

Reviewer comment (Theory):
In the Equation in Lemma 1 (the definition of $Q_i(s,a_i)$), should the $r$ be $r_i(s,a_i)$? If it should, should we also include $\max_{r_i(s,a_i)}$ in the definition of $Q_i(s,a_i)$? It's a bit strange if the reward does not depend on the current state and the action chosen by the player.

Author response:
> In the Equation in Lemma 1 (the definition of $Q_i\left(s, a_i\right)$ ), should the $r$ be $r_i\left(s, a_i\right)$ ?

As defined in section 2.1, the reward is $r(s,s')$, which does not depend on joint action. The setting is still general because if two joint actions lead to the same transition dynamics in the environment, the two actions should have the same effects, and we do not need to discriminate the actions. The effects of actions are reflected in the distribution of $s'$.

Imagine that we are playing a computer game. There is a certain state where the left key and the right key can lead to the same next state or the same distribution of the next state. Do we need to give the right key a higher reward to encourage the players to always select the right key?

Reviewer comment (Theory):
Again, in the definition of $Q_i(s,a_i)$, what is the search space of $P_i(\cdot|s,a_i)$? Do you search for "all possible transitions probabilities" from $(s,a_i)$? If it is, then why not the solution simply be $Q_i(s,a_i)=r + \max_{s', a_i'} Q_i(s', a_i')$? That is, the optimal solution of $P_i(\cdot|s,a_i)$ should put all probability to the state $s'$ that has the highest value of $\max_{a_i'}Q_i(s',a_i')$.

Author response:
> Again, in the definition of $Q_i\left(s, a_i\right)$, what is the search space of $P_i\left(\cdot \mid s, a_i\right)$ ?

Yes, the search space is "all possible transitions probabilities", but the situation you proposed might be impossible to happen in the environment. Look at the definition of $P_i$ (Eq. 1), if $P_{env}$ is stochastic, no matter what $\pi_{-i}$ is, we cannot put all probability to the state that has the highest value.

Reviewer comment (Theory):
I have difficulty understanding the first inequality in the equation series in Lemma 2, i.e., the step that replaces $P_{\text{env}}$ with $P_i^\star$ with a $\geq$ inequality. Can you explain that inequality?

Author response:
> I have difficulty understanding the first inequality in the equation series in Lemma 2, i.e., the step that replaces $P_{env}$ with $P_i^{\star}$ with a $\geq$ inequality. Can you explain that inequality?

Look at the first "=" line in lemma 2, there are two terms. $\max_{a_{-i}}P_{env}$ achieves the highest first term, and $P_i^*$ achieves the highest second term. If we replace $\max_{a_{-i}}P_{env }$ with $P_i^*$ in the first term, the first term decreases and we have a $\ge$.

Reviewer comment (Theory):
In Page 5, it is said that "When M is sufficiently large, given any $(s,a')$ pair, any $P_i(s,a_i)$ can be found in a replay buffer." I thought the set of $P_i(s,a_i)$ is infinitely large since these are continuous values, how is it possible that a finite set can contain all possible values of $P_i(s,a_i)$? Again, it's unclear what you mean by $P_i(s,a_i)$ can be found in replay buffer, since replay buffer only contains $(s,a,r,s')$ tuples.

Author response:
> I thought the set of $P_i$ is infinitely large since these are continuous values, how is it possible that a finite set can contain all possible values of $P_i$.

Look at the definition of $P_i$ (Eq. 1), the number of $P_i$ depends on the number of $\pi_{-i}$. In the lines after Eq. 6, we have claimed that we only consider the deterministic policies, because when there is only one optimal joint policy, the optimal joint policy must be deterministic. Deterministic $\pi_{-i}$ is countable, and thus $P_i$ is countable.

Reviewer comment (Writing):
I don't understand the following sentence in page 5: "Simplified best possible operator ... does not care about the relation between transition probabilities of different state-action pairs in the same buffer."

Author response:
> I don't understand the following sentence in page 5: "Simplified best possible operator ... does not care about the relation between transition probabilities of different state-action pairs in the same buffer."

In the simplified best possible operator, each transition independently contributes to the update of the operator. Even though the transitions are in the same buffer, they are not related during the update of the operator.

Reviewer comment (Theory):
In Page 5, it is said that "BQL ideally needs only $|A|$ buffers.. which is very efficient." Suppose that every player has $K$ actions, is $|A|=K^N$? I'm not sure one can call this very efficient, given that this is exponentially large, and running a centralized Q-learning (so the number of joint action is $K^N$) should also just incur the same amount of computation.

Author response:
>  In Page 5 , it is said that "BQL ideally needs only $|A|$ buffers.. which is very efficient" Suppose that every player has $K$ actions, is $|A|=K^N$ ? I'm not sure one can call this very efficient, given that this is exponentially large, and running a centralized Q-learning (so the number of joint action is $\mathrm{K}^{\wedge} \mathrm{N}$ ) should also just incur the same amount of computation.

Yes, $|A|=K^N$. The reason why you think it is not efficient is that the problem itself is complex. Decentralized learning is more complex than centralized learning, and centralized Q-learning is an ideal solution for decentralized learning. If our method can achieve the same amount of computation with centralized Q-learning, we can say that our method is efficient. 

We know that you expect a decentralized learning method which is much more efficient than centralized Q-learning and can guarantee the convergence to optimal policy. However, it seems to be impossible. If such a decentralized method really exists, for any single-agent Q-learning problem, we can factorize its action space, convert it to a decentralized MARL problem, and solve it using this method efficiently. It will be a huge revolution for Q-learning.

Reviewer comment (Experiments):
**BQL is not "fully decentralized"?** In the tabular setting, Algorithm 1 assumes that you can ensure that agents independently explore, but it's not clear that's a realistic assumption in this "fully decentralized setting". BQL- is more realistic and still seems to outperform baselines, but it seems like the key ingredient to getting BQL- to work is ensuring enough exploration. It would be useful to compare how well BQL does based on how much exploration is allowed (e.g., by changing the distance of the initialization to an equilibrium). It would also be useful to talk about whether sufficient exploration is a realistic assumption in practice.

Author response:
We only need a pre-defined exploration schedule for each agent. In training, the agents independently follow the exploration schedule without centralized control, so the decentralized assumption still holds. The key ingredient to getting BQL- to work is not enough exploration but the operator, because BQL- adopts the same exploration method with IQL, H-IQL, MA2QL-, and I2Q.

Reviewer comment (Theory):
**No intuition for the 2-step update when the theoretical assumptions are broken:** The paper leans heavily on asymptotic intuitions, but a lot of the wins in 4.3 and 4.4 seem to come from sample efficiency. Is there any intuition for this? More generally, there seems to be a gap between theory and practice, esp., the improved performance of BQL when there is only a single replay buffer $\mathcal{D}$. Are there any intuitions for why BQL works in this case?

Author response:
On the one hand, as analyzed in the last paragraph of section 2.4, if the non-stationary single replay buffer sufficiently goes through all possible transition probabilities, the theory still holds. On the other hand, the intuition behind the operators is optimistic value estimation, which has been verified to be useful in decentralized learning, e.g., H-IQL. However, since the optimistic target in H-IQL is the highest state value, the overestimation problem is severe in a stochastic setting. The optimistic target in BQL is the expected value over possible transition probabilities, so BQL can avoid overestimation and achieve better performance.

Reviewer comment (Theory):
Are the theoretical proofs a reduction of the "fully decentralized setting" to the CTDE setting? If so, what are the assumptions required? If not, what is the gap?

Author response:
Decentralized learning is very different from CTDE. As analyzed in section 2.1, due to the lack of other agents' actions, the transition probabilities become non-stationary, which breaks the assumption of convergence and optimality of existing RL methods. Therefore,  the main challenge of decentralized learning is to deal with non-stationary transition probabilities. However, in CTDE and joint learning, the transition probabilities are stationary, so they do not need to consider this problem at all. Therefore, we can say that the theory of BQL has no connection with CTDE and joint Q-learning. BQL is not an extension of I2Q. I2Q tries to directly obtain the optimal transition probabilities, but BQL obtains the value under the optimal transition probabilities by new operators. BQL is an extension of Hysteretic IQL to the stochastic setting, as discussed in Appendix B. Our method and baselines adopt the same settings and hyper-parameters, so we believe the comparison is fair.