Paper: Generative Reinforcement Learning with Transformers (ICLR-2024)

Reviewer comment (Theory):
Page 4 --> Is it required to have only one communicating class for this discussion to be true? If yes, it needs to be set as an assumption (with a discussion on resultant limitations as most practical domains violate this).

Author response:
* Yes, one communicating class is required. The original CnC paper makes an irreducibility assumption on the Markov Chain formed by the combination of MDP+policy, which implies one communicating class. If this holds (along with aperiodicity and positive recurrence), then it also holds for the augmented chain. We can revise the text so as to not sweep this under the rug, but point out that it is a very mild assumption commonly assumed by nearly all real world policy evaluation algorithms which have theoretical guarantees, and far larger than the class of problems for which decision transformers can sensibly address.

Reviewer comment (Theory):
The counting method used to train the action-conditioned model seems to work only because the MDP is not discounted, otherwise it could be unlikely to have a $n_{za}$ more than 2.

Author response:
* The case of discounted return is more complicated, but the situation is not hopeless. We are aware of two ways of dealing with discounted return. a) Quantize the return, which introduces quantization error; b) Compute the horizon stochastically (e.g. discounting can be seen as the return being geometrically distributed to gamma, so one can keep flipping biased coins to determine an effective horizon for each update, which introduces additional variance to the estimation but provides an unbiased estimate of the discounted return) suitable for a count based estimator. We will provide these details in the next revision.

Reviewer comment (Theory):
Page 5, the first paragraph of Return-Action Conditioning --> it is said that all the information in $\xi(s)$ is completely irrelevant to estimating the $Q$ function. It then follows that all such information is also irrelevant to $\xi(s|a,z)$ as well (as it is part of $Q$). Why??

Author response:
* Re: Page 5, the first paragraph of Return-Action Conditioning: thankyou for spotting this, this was a very poor exposition on our part, and is completely incorrect. We will remove this clunky sentence in the next revision. What we intended to stress here is that there are multiple ways of implementing conditioning, and we found this to be one of the key challenges in getting the transformer architecture to work well on the challenging domain of chess.

Reviewer comment (Presentation):
There is lack of summary of distributional RL in the related work.

Author response:
* We will add some extra references to distributional RL, we omitted many since most works focus on various types of bootstrapping which goes beyond the CnC formulation (which predated distributional RL), but the point is well taken and should be discussed.

Reviewer comment (Experiments):
The paper focuses on comparing discriminative and generative decomposition of \xi (z | s, a) for estimating the value function. Yet it's not clear how well the value-based approach compared to the decision transformer approach which directly updates the policy. I suggest adding these experiments for a more comprehensive comparison.

Author response:
* We take the position that the decision transformer method is a non-starter for the class of ergodic MDPs (especially stochastic environments) and that further demonstration of this is not necessary.

Reviewer comment (Experiments):
The major limitation of this paper is, although it provides some evidence and insights, showing the potential of estimating value function using RL, it's not conclusive whether the value function hypothesis is true. The behavior cloning baseline is too weak to reach a conclusion. Also, the generative approach has not shown advantage over discriminative decomposition, even that it was finetuned on a strong foundation model. This work brings more questions than answers and I do wish there are more answers resolved in this paper.

Author response:
* There is no advantage in terms of final playing strength using pre-training to the generative approach, only shorter training time/sample efficiency. Our latest results suggest the generative decomposition is 50 elo weaker (it was 100 elo weaker in our submission).

Reviewer comment (Experiments):
Could you run experiments on decision transformers and show how the value-based approach compares to the policy-based approach? If we want to validate the value function hypothesis, shouldn't we collect evidence that the value-based discriminative / generative decomposition is better than policy-based approach?

Author response:
* We take the position that the decision transformer method is a non-starter for the class of ergodic MDPs (especially stochastic environments) and that further demonstration of this is not necessary.

Reviewer comment (Experiments):
I might miss it, but how does the finetuned generated decomposition approach compared to discriminative decomposition in the experiments?

Author response:
* There is no advantage in terms of final playing strength using pre-training to the generative approach, only shorter training time/sample efficiency. Our latest results suggest the generative decomposition is 50 elo weaker (it was 100 elo weaker in our submission).

Reviewer comment (Experiments):
The paper feels like a proof-of-concept because the proposed method does not conclusively answer the question asked in the paper (about the Value Function Hypothesis).

Author response:
* Our claim is that our results only support the value function hypothesis in the sense that both value-based methods beat a behavioral cloning baseline. We see our work as more provocative rather than conclusive on this matter.

Reviewer comment (Evaluation):
The paper feels like a proof-of-concept because the proposed method does not beat a baseline (a superiority of discriminative methods was observed).

Author response:
* Our claim is that our results only support the value function hypothesis in the sense that both value-based methods beat a behavioral cloning baseline. We see our work as more provocative rather than conclusive on this matter.

Reviewer comment (Theory):
In Section 3, the paragraph "connecting value function to the logarithmic loss" contains a statement claiming that a set of augmented states converges to a stationary distribution, which is confusing and seems like a "type mismatch" problem.

Author response:
* Re: A statement claiming that a set of augmented states converges to a stationary distribution: thanks for spotting, this should be “...whose distribution for n->inf will converge to …”

Reviewer comment (Theory):
In the decomposition approach in Section 3, the main components are $\xi(s|a,z)$ and $\xi(z|a)$, both of which are rather unorthodox in RL: the former requires the knowledge of future returns ($z$), and the latter gives the distribution over returns only given an action (without knowledge of the state $s$).

Author response:
* In general one does need to accurately model the xi(z|a) term, however in chess many moves (with the rare exception of promotions) are not good in themselves across arbitrary board states. A helpful example of where this is essential can be found in the Atari game Freeway, where the goal is to move the avatar to the top of the screen, here pressing up/down has a significant effort on the return distribution across nearly all states.

Reviewer comment (Reproducibility):
In the decomposition approach in Section 3, it is unclear how such objects are used for inference; a pseudo-code could improve clarity here.

Author response:
* For inference, we plug the learnt terms into Equation 1 for the discriminative decomposition, and in the generative decomposition we plug the learnt terms first into Equation 2 and then into Equation 1. We will make this more explicit in the next revision.

Reviewer comment (Theory):
There should be an assumption in Section 3 that $|\mathcal Z|<\infty$.

Author response:
* Yes, the return space either needs to be finite or quantised.

Reviewer comment (Evaluation):
In Section 5.1 (Experiments), the evaluation protocol is confusing regarding whether the computation of the ELO metric and the early stopping rule described in the "main result" is standard and whether there is a citation to refer to.

Author response:
* Using BayeElo for elo computation is standard in the literature. A stopping rule for learnt agent performance is also typically used since degenerate games can take in excess of 800 moves (which are almost always drawn in uninteresting ways), and using it allows for more games in less time, which leads to better elo estimates in practice. We will add some citations to similar setups in the next revision.

Reviewer comment (Evaluation):
In Section 5.1, Table 1 implies that the generative approach has ELO roughly smaller than 100 compared to the discriminative approach, and according to the table's caption, the latter is twice as likely to win. This seems to conflict with the statement that "the two main decompositions are close in performance".

Author response:
* An elo gap of 100 implies that the stronger player has a 64% chance of winning, and this is probably what we should have said in the caption. In chess terms, two players with this rating gap would be considered to be close enough to have an enjoyable game with an uncertain outcome, whereas say a 600 rating gap there would be almost no point playing. We can clarify this further in the next revision.