Paper: RL Algorithms are Information-State Policies in the Bayes-Adaptive MDP (ICLR-2024)

Reviewer comment (Writing):
The motivation and contribution of the paper is unclear, making it hard to understand the main purpose and value of the work.

Author response:
**Motivation and contribution** The reviewer pointed out that the motivation and contribution of the paper were unclear. We motivate BAMDPs as a way to formally specify RL problems, allowing us to analyze RL algorithms just like MDPs allow us to analyze policies. This radically different perspective makes it straightforward to derive many insights and analyze the extent to which RL algorithms successfully navigate the explore-exploit trade-off, and which naturally has many implications for the design of RL algorithms. We have concretized this more in the abstract and introduction to make our work’s motivation and contribution clearer.

Reviewer comment (Presentation):
The presentation of the paper is unfocused, which makes it difficult to follow its main concept and insights.

Author response:
**Unfocused presentation** We have moved the less directly applicable section on potential-based shaping to the appendix to focus the story around the more practical insights.

Reviewer comment (Experiments):
The practicalness of the analysis is unclear due to the lack of practical examples or insights regarding the usage of the analysis, such as how the insights could be used to improve or design more practical RL algorithms.

Author response:
**Lack of practical examples** We flesh out and further emphasize practical examples in our manuscript, specifically:
    - Our framework can be used to inform the design of better RL algorithms - it shows that guaranteeing convergence to the optimal policy for the underlying MDP is generally undesirable because it comes at the cost of over-exploring, and algorithms should instead be designed to explore exactly when that is expected to increase return. This is a radical departure from the mainstream view in RL of convergence to the optimal MDP policy as the gold standard. We have emphasized this more in the abstract, introduction and optimality sections.
    - We also flesh out the Negative Surprise example in the Effects of Reward Shaping section, clarifying exactly what assumptions of the algorithm and task distribution must hold for Negative Surprise rewards to be beneficial, to show how our framework provides practically useful insights for reward shaping design.
We understand the remaining concern about directly demonstrating practical use. It is now clear to us that it would greatly benefit the reader to see the BAMDP value decomposition we derived applied outside of our toy Caterpillar problem, in a way that demonstrates the practical relevance to AI designers. We have fleshed out our example of how the value signaled by “Empowerment”-based intrinsic motivation (measured as mutual information) can be decomposed into two terms signaling the Value of Opportunity (represented as a negative surprise term) and the Value of Information (represented as an entropy bonus term). To highlight this usage we have moved it from the appendix up into the main text (in the Effects of Reward Shaping section), and mentioned this corollary in the introduction. We find that our decomposition provides a new perspective on the empirical behavior of “Empowerment”-driven agents, more accurately describing the observed behaviors than the prior interpretation of it motivating agents to seek states of maximum influence [1].

Specifically, since Empowerment signals a positive Value of Information for exploring a wide range of actions, and a negative Value of Opportunity for unpredictable transitions. The latter component explains why we see empowerment-driven agents barricade themselves in corners to avoid danger: upon dying they respawn in random locations, which is highly unpredictable. Though this is counterintuitive behavior looking like “Empowerment,” it is directly evident when observing the value decomposition.

We believe this has greatly strengthened our paper by clearly demonstrating to readers how our framework can be leveraged to gain significant novel insights on a practical example.

[1] Mohamed, S., & Jimenez Rezende, D. (2015). Variational information maximisation for intrinsically motivated reinforcement learning. Advances in neural information processing systems, 28.

Reviewer comment (Writing):
Since multiple concepts such as BAMDP, RL algorithms, and reward shaping are used in the paper, it should be clear how each one is related and why it is important under a main message of the paper. The paper does not do this, leading to confusion.

Author response:
**Motivation and contribution** The reviewer pointed out that the motivation and contribution of the paper were unclear. We motivate BAMDPs as a way to formally specify RL problems, allowing us to analyze RL algorithms just like MDPs allow us to analyze policies. This radically different perspective makes it straightforward to derive many insights and analyze the extent to which RL algorithms successfully navigate the explore-exploit trade-off, and which naturally has many implications for the design of RL algorithms. We have concretized this more in the abstract and introduction to make our work’s motivation and contribution clearer.

Reviewer comment (Writing):
The motivation and contribution of the paper is not clearly provided. It is unclear for what purpose RL algorithms are being described in the context of BAMDP — whether it is to specify a proper way to use RL algorithms for BAMDP or to show that RL algorithms are not suitable for BAMDP.

Author response:
**Motivation and contribution** The reviewer pointed out that the motivation and contribution of the paper were unclear. We motivate BAMDPs as a way to formally specify RL problems, allowing us to analyze RL algorithms just like MDPs allow us to analyze policies. This radically different perspective makes it straightforward to derive many insights and analyze the extent to which RL algorithms successfully navigate the explore-exploit trade-off, and which naturally has many implications for the design of RL algorithms. We have concretized this more in the abstract and introduction to make our work’s motivation and contribution clearer.

Reviewer comment (Theory):
It is not clear why the paper considers BAMDP. For meta-learning, considering BAMDP is reasonable since meta-learning tries to address multiple tasks in real-world applications together, which can be modeled as different task MDPs. However, it is unclear whether the authors are arguing that real-world problems traditionally modeled with MDPs are actually more suited to be modeled with BAMDPs.

Author response:
**Modeling RL problems with BAMDPs instead of MDPs** The reviewer questioned the use of BAMDPs, rather than MDPs, to model RL problems outside of meta-learning. BAMDPs are necessary to fully specify RL problems because the underlying task MDP cannot capture the initial uncertainty and information-gathering that RL algorithms do to learn to maximize return, whereas these are baked into the BAMDP transition function and augmented state space. To reuse existing MDP theory on RL algorithms, we must frame RL problems as MDPs in which the algorithms act like policies, but RL algorithms are not valid policies in the task MDPs because they learn from experience, and thus select actions based on more than just the current task MDP state.

Reviewer comment (Writing):
The term "hand-written information-state policies" appears in the abstract but is not clearly described in the text, making its precise meaning unclear.

Author response:
**Hand-written information-state policies** We have amended this vague use of language. Information-state policies are policies that act on states augmented with the information gathered thus far. Hand-written refers to the fact that they are explicitly hard-coded by the RL practitioner rather than learned or otherwise optimized.

Reviewer comment (Theory):
The rationale for applying reward shaping in analyzing the regret is unclear, and the importance of this choice should be justified more clearly.

Author response:
**Lack of practical examples** We flesh out and further emphasize practical examples in our manuscript, specifically:
    - Our framework can be used to inform the design of better RL algorithms - it shows that guaranteeing convergence to the optimal policy for the underlying MDP is generally undesirable because it comes at the cost of over-exploring, and algorithms should instead be designed to explore exactly when that is expected to increase return. This is a radical departure from the mainstream view in RL of convergence to the optimal MDP policy as the gold standard. We have emphasized this more in the abstract, introduction and optimality sections.
    - We also flesh out the Negative Surprise example in the Effects of Reward Shaping section, clarifying exactly what assumptions of the algorithm and task distribution must hold for Negative Surprise rewards to be beneficial, to show how our framework provides practically useful insights for reward shaping design.
We understand the remaining concern about directly demonstrating practical use. It is now clear to us that it would greatly benefit the reader to see the BAMDP value decomposition we derived applied outside of our toy Caterpillar problem, in a way that demonstrates the practical relevance to AI designers. We have fleshed out our example of how the value signaled by “Empowerment”-based intrinsic motivation (measured as mutual information) can be decomposed into two terms signaling the Value of Opportunity (represented as a negative surprise term) and the Value of Information (represented as an entropy bonus term). To highlight this usage we have moved it from the appendix up into the main text (in the Effects of Reward Shaping section), and mentioned this corollary in the introduction. We find that our decomposition provides a new perspective on the empirical behavior of “Empowerment”-driven agents, more accurately describing the observed behaviors than the prior interpretation of it motivating agents to seek states of maximum influence [1].

Specifically, since Empowerment signals a positive Value of Information for exploring a wide range of actions, and a negative Value of Opportunity for unpredictable transitions. The latter component explains why we see empowerment-driven agents barricade themselves in corners to avoid danger: upon dying they respawn in random locations, which is highly unpredictable. Though this is counterintuitive behavior looking like “Empowerment,” it is directly evident when observing the value decomposition.

We believe this has greatly strengthened our paper by clearly demonstrating to readers how our framework can be leveraged to gain significant novel insights on a practical example.

[1] Mohamed, S., & Jimenez Rezende, D. (2015). Variational information maximisation for intrinsically motivated reinforcement learning. Advances in neural information processing systems, 28.

Reviewer comment (Novelty):
(Motivation) Whereas the formulation is interesting, BAMDP is also known to be an intractable problem in general, so it is unclear what benefit this new perspective can bring.

Author response:
**Motivation** The reviewer questioned the benefit of using the BAMDP, because they are known to be generally intractable to solve. 
We should have made it clearer that we never intend to solve the BAMDP directly, we simply use it as a conceptual framework to model and analyze the behavior of RL algorithms.  We don’t think solving the BAMDP is necessarily a good approach to designing optimal RL algorithms- heuristic approximate RL algorithms can be very effective. The BAMDP is valuable because it allows us to formalize what these algorithms should ideally aim to approximate, clarifying when the approximations are appropriate, and giving insights into how to design them- for instance how to shape their rewards to approximate the optimal Q function. We have made this more clear in the introduction.

Reviewer comment (Novelty):
(Implications) The paper does not fully clarify how the introduced perspective should help analyzing existing RL algorithms and building more advanced algorithms in the future.

Author response:
**Practical implications** We have made the practical implications of our framework much clearer and more concrete:
    - We highlight more in the abstract, introduction and optimality sections the powerful implications our framework has for the design and development of improved RL algorithms: guaranteeing convergence to the optimal policy for the underlying MDP generally comes at the cost of over-exploring, and algorithms should instead be designed to explore exactly when gathering that additional information is expected to increase return. This is a radical departure from the mainstream view in RL of convergence to the optimal MDP policy as the gold standard.
    - We dive deeper into the Negative Surprise example in the Effects of Reward Shaping section, clarifying exactly what assumptions of the algorithm and task distribution must hold for Negative Surprise rewards to be beneficial, to demonstrate the non-obvious insights our theory can provide Rl practitioners trying to select the appropriate shaping function

Reviewer comment (Novelty):
Despite providing an original interpretation of RL as the problem of solving a BAMDP and reporting insights such as the value decomposition that explicitly separates exploration and exploitation contributions to the value, the paper falls short of providing a coherent set of implications resulting from this new perspective, beyond a few informal considerations over reward shaping methods.

Author response:
**Practical implications** We have made the practical implications of our framework much clearer and more concrete:
    - We highlight more in the abstract, introduction and optimality sections the powerful implications our framework has for the design and development of improved RL algorithms: guaranteeing convergence to the optimal policy for the underlying MDP generally comes at the cost of over-exploring, and algorithms should instead be designed to explore exactly when gathering that additional information is expected to increase return. This is a radical departure from the mainstream view in RL of convergence to the optimal MDP policy as the gold standard.
    - We dive deeper into the Negative Surprise example in the Effects of Reward Shaping section, clarifying exactly what assumptions of the algorithm and task distribution must hold for Negative Surprise rewards to be beneficial, to demonstrate the non-obvious insights our theory can provide Rl practitioners trying to select the appropriate shaping function

Reviewer comment (Theory):
To meet the standard for acceptance, the paper should narrow its scope (e.g., present the contribution as a study of reward shaping through the BAMDP perspective) and provide more formal and deeper implications from the analysis, such as a study on how different shaping methods impact Bayesian regret and under which assumptions or priors those methods can be considered optimal.

Author response:
We greatly appreciate reviewer t2gv taking the time to consider our response and the updates to our manuscript. We forgot to mention in our previous response that we also found the suggestion to narrow the scope of the paper to be helpful, and we have moved our section on potential-based shaping to the appendix to focus the story around the more practical insights. 

Re: the reviewer's request for clarification, this implication is indeed equivalent to the notion of minimizing **Bayes Regret**, specifically. We should have been more precise-- adversarial or otherwise non-Bayesian regret minimization would not necessarily have the issue of sacrificing return to guarantee convergence to the optimal task policy. Its difference from Bayes Regret is that it is not aware of the distribution of tasks the designer intends to use the algorithm in, so it may still over-explore or under-exploit with respect to that. 

In the claim quoted by the reviewer, we were referring to the fact that many RL practitioners, particularly in deep RL, do not consider the regret of their algorithms at all- instead using the final performance of their algorithms after training converges as their metric, reported on a few benchmark environments. Thus they only consider convergence results for their algorithms, such as for policy gradient or Q-learning style algorithms. 

Although Bayes Regret is not a novel concept, this perspective is novel: solving RL problems can generally be understood as minimizing Bayes Regret, because RL algorithms are designed for and evaluated on the distribution of tasks they’re expected to encounter in practice. Our BAMDP framework provides a way to express and analyze the Bayes regret of any RL algorithm (the regret of an RL algorithm cast as a policy in the BAMDP is equivalent to its Bayes-Regret in the corresponding RL problem), explaining many findings made by RL researchers, *even when not explicitly doing Bayesian RL*, such as which reward shaping functions are effective in different settings. It is thus valuable for guiding RL practitioners in designing algorithms and shaping functions that intentionally and appropriately balance exploration and exploitation in the types of tasks they expect to use them on.

We thank the reviewer for their question which has highlighted an aspect of our paper that should be clearer, and will revise the manuscript to clarify this.

Reviewer comment (Theory):
What is the point of framing the RL problem as BAMDP, when it is well known that BAMDP cannot be solved efficiently in general? Especially, given that provably efficient RL algorithms exist (under somewhat restrictive assumptions), does this mean that the provided formulation is missing some of the structure of the underlying problem?

Author response:
**Question 1** The reviewer questioned the validity of the BAMDP formulation, citing the existence of provably efficient RL algorithms which appears incompatible with the intractability of the BAMDP.
We thank the reviewer for pointing out these efficient algorithms; we have amended our statement about theoretical RL aiming to converge with unlimited interactions, and clarified our own vague use of “efficient”. We used “efficient” to mean making the optimal exploration-exploitation trade-off, so neither over-exploring past the point of diminishing returns, nor exploiting before having enough information to get high rewards. Algorithms that find the optimal MDP policy in polynomial time complexity are not necessarily efficient in this way- always finding this optimal policy often comes at the cost of over-exploring. If the discount factor were high enough that finding the optimal MDP policy is not over-exploration and if we made those same assumptions of underlying structure, then we would arrive at the same results in our framework as a special case.

Reviewer comment (Theory):
While setting a prior over the tasks might be reasonable in meta-RL or analogous settings, in which some knowledge of the task distribution can be collected during training, is this also reasonable in standard RL? Any MDP can be seen as a sample from a very uninformative prior over all the possible MDPs, but this does not seem to provide any benefit.

Author response:
**Question 2** The reviewer questioned the assumption of a prior over MDPs in the formulation of RL problems. 
Our use of the word prior should have been clearer- by prior we mean the distribution of tasks the RL algorithm will encounter. We have amended the definition in the BAMDP section and changed language in other places that p(M)/the prior is mentioned to clarify this. An RL algorithm is useful insofar as it performs well on the types of tasks it’s expected to encounter in practice, while it does not matter if it performs poorly on very unrealistic tasks. RL algorithms and reward shaping functions are always designed with specific types of task in mind, and evaluated on such tasks (e.g. maze navigation, atari games, driving simulators) i.e. assumptions about the task distribution are central in the design and evaluation of virtually all RL algorithms, even though this is often not stated explicitly. The task distribution may not be expressible in closed form, but we do not need to construct and solve the BAMDP directly- we show in the Effects of Reward Shaping section how simple features of this distribution can be used to inform the design of helpful reward shaping functions.

Reviewer comment (Novelty):
Can the authors discuss how their framework will help produce a deeper analysis of existing RL algorithms and possibly guide the development of improved algorithms?

Author response:
We greatly appreciate reviewer t2gv taking the time to consider our response and the updates to our manuscript. We forgot to mention in our previous response that we also found the suggestion to narrow the scope of the paper to be helpful, and we have moved our section on potential-based shaping to the appendix to focus the story around the more practical insights. 

Re: the reviewer's request for clarification, this implication is indeed equivalent to the notion of minimizing **Bayes Regret**, specifically. We should have been more precise-- adversarial or otherwise non-Bayesian regret minimization would not necessarily have the issue of sacrificing return to guarantee convergence to the optimal task policy. Its difference from Bayes Regret is that it is not aware of the distribution of tasks the designer intends to use the algorithm in, so it may still over-explore or under-exploit with respect to that. 

In the claim quoted by the reviewer, we were referring to the fact that many RL practitioners, particularly in deep RL, do not consider the regret of their algorithms at all- instead using the final performance of their algorithms after training converges as their metric, reported on a few benchmark environments. Thus they only consider convergence results for their algorithms, such as for policy gradient or Q-learning style algorithms. 

Although Bayes Regret is not a novel concept, this perspective is novel: solving RL problems can generally be understood as minimizing Bayes Regret, because RL algorithms are designed for and evaluated on the distribution of tasks they’re expected to encounter in practice. Our BAMDP framework provides a way to express and analyze the Bayes regret of any RL algorithm (the regret of an RL algorithm cast as a policy in the BAMDP is equivalent to its Bayes-Regret in the corresponding RL problem), explaining many findings made by RL researchers, *even when not explicitly doing Bayesian RL*, such as which reward shaping functions are effective in different settings. It is thus valuable for guiding RL practitioners in designing algorithms and shaping functions that intentionally and appropriately balance exploration and exploitation in the types of tasks they expect to use them on.

We thank the reviewer for their question which has highlighted an aspect of our paper that should be clearer, and will revise the manuscript to clarify this.

Reviewer comment (Writing):
The use of \citep and \citet in the paper is sometimes confusing. \citet should be used only when the papers' authors' names are part of the sentence.

Author response:
We have also fixed the use of \citep and \citet, added a more representative reference for regret bounds, and added the full stop after the last bullet in the introdution.

Reviewer comment (Writing):
The itemize at the end of the Introduction is missing a full stop after the last bullet point.

Author response:
We have also fixed the use of \citep and \citet, added a more representative reference for regret bounds, and added the full stop after the last bullet in the introdution.

Reviewer comment (Experiments):
The primary weakness of this paper is the absence of a direct practical application of the ideas developed. This is natural, given the conceptual nature and theoretical focus of the work, but some experimental evidence supporting the main ideas would make the paper much more accessible to the experimental RL community.

Author response:
We understand their concern about the need for more explicit direct practical applications of the ideas, so we updated the manuscript to show more clear and concrete practical implications. Specifically, 

- in the abstract, introduction and optimality sections we highlight a powerful implication our work has for the design of RL algorithms, i.e. that exploring until finding the optimal policy for the underlying MDP is not the optimal approach for the problem itself, and algorithms should instead explore if and only if further information is expected to maximize overall return. This is a radical departure from the mainstream view in RL of convergence to the optimal MDP policy as the gold standard.
- In the Effects of Reward Shaping section we have expanded on the negative surprise reward shaping example, to show more concretely how an RL practitioner could determine when it is beneficial- i.e., when the vast majority of surprise correlates well with negative outcomes on the distribution of trajectories the agent actually experiences.

Reviewer comment (Experiments):
It would be very helpful to have numerical experiments illustrating key aspects of the discussion of reward shaping provided in Sec. 6 on a non-trivial problem.

Author response:
Our analysis in the Effects of Reward Shaping section should be predictive of the experimental performance of agents, for each type of shaping function and corresponding conditions on the task distributions described- e.g. adding prediction error rewards will improve performance in problems where the prediction error of an observation is an accurate signal for the value of that observation’s information content. We updated the manuscript fleshing out the negative prediction error example in this section to make this more concrete.

Reviewer comment (Theory):
Why is \( R(r_t | s_t, a_t) \) included in the definition of \(\bar{T}\) in bullet point four of the BAMDP definition on page 3? Can you clarify how \(\bar{T}( \cdot | \bar{s}_t, a_t)\) remains a pdf/pmf?

Author response:
$R(r_t|s_t,a_t)$ is included in the BAMDP transition function because the history component $h_t$ of the BAMDP state includes all the rewards received over the agent’s trajectory. This captures how the RL algorithm learns the reward function $R$ from its interactions with the task MDP. This remains a pdf because the product $T(s_{t+1}|s_t,a_t)R(r_t|s_t,a_t)$ is equal to the joint probability of transitioning to MDP state $s_{t+1}$ and receiving reward $r_t$.

Reviewer comment (Novelty):
The analysis in the appendix seems to consist primarily of proving Theorem 3.1 and providing computations for the caterpillar example; are there any key innovations in the analysis that you consider worth highlighting?

Author response:
The main innovation of this paper was this new conceptual perspective, the use of which makes analysis of RL algorithms very straightforward. We updated the manuscript to emphasize this point.

Reviewer comment (Experiments):
Do you have any ideas for how the discussion in Sec. 6 might be illustrated experimentally?

Author response:
Our analysis in the Effects of Reward Shaping section should be predictive of the experimental performance of agents, for each type of shaping function and corresponding conditions on the task distributions described- e.g. adding prediction error rewards will improve performance in problems where the prediction error of an observation is an accurate signal for the value of that observation’s information content. We updated the manuscript fleshing out the negative prediction error example in this section to make this more concrete.