Paper: Non-ergodicity in reinforcement learning: robustness via ergodic transformations (ICLR-2024)

Reviewer comment (Writing):
In the introduction the definition of ergodic is not clearly introduced. For example, ergodicity can be defined by a process which is stationary and every invariant random variable of the process is almost surely equal to a constant.

Author response:
**Problem description.** We have thoroughly revised the problem setting section to define the concept of ergodicity more clearly. We are here using an subscript $k$ on the time $t$ to clarify that we are considering discrete time steps and to distinguish it from the continuous-time analysis we do when comparing our contributions to risk-sensitive reinforcement learning.

Reviewer comment (Writing):
As the authors discuss, non-ergodic reward functions have been studied within RL — the authors write “none of these works, as a consequence of non-ergodicity, question the use of the expectation operator in the objective function” which seems a little vague.

Author response:
**Connections to related work.** The main contribution of this paper is raising the question of ergodicity of reward functions in RL and how it impacts the use of the expected value as an optimization objective, as well as proposing a first approach toward resolving the problem. Other approaches that we are aware of consider non-ergodicity of MDPs but do not make the connection to the divergence between expected rewards and long-term rewards in such settings. Majeed and Hutter prove that the Q-learning algorithm still converges under non-ergodic dynamics. In the coin-toss game, the PPO algorithm also converges. And it does converge toward the policy that yields the maximum expected return. However, this policy yields a return of 0 in the long run with probability 1. Convergence itself does not necessarily help if the optimization objective is the problem.

Reviewer comment (Novelty):
The work seems poorly motivated with it being unclear as to why the examples given aren’t resolved by using other existing approaches such as risk-sensitive objectives (though these approaches are discussed).

Author response:
**Risk-sensitive RL.** As we discuss in Section 5, for a specific class of reward functions (logarithmic), the risk-sensitive transformation is an ergodicity transformation. For such settings, the risk-sensitive transformation does exactly what we propose to do in this paper. This is not the case for other classes of reward functions - which is intuitive: a fixed transformation should only work well for a class of reward functions and not for all potential classes one could think of. We used the risk-sensitive transformation in the coin-toss game, whose dynamics are exponential, i.e., opposite from what the risk-sensitive transformation would need. In this case, the transformation fails.

Reviewer comment (Theory):
The paper at times lacks precision and consistency with formal details — for example the paper begins with a discrete time analysis then makes a diversion to studying continuous time.

Author response:
**Problem description.** We have thoroughly revised the problem setting section to define the concept of ergodicity more clearly. We are here using an subscript $k$ on the time $t$ to clarify that we are considering discrete time steps and to distinguish it from the continuous-time analysis we do when comparing our contributions to risk-sensitive reinforcement learning.

Reviewer comment (Theory):
Some formalities are missing or unclear — for example the transition dynamics aren’t specified and it’s unclear why a subindex is needed on the time index.

Author response:
**Problem description.** We have thoroughly revised the problem setting section to define the concept of ergodicity more clearly. We are here using an subscript $k$ on the time $t$ to clarify that we are considering discrete time steps and to distinguish it from the continuous-time analysis we do when comparing our contributions to risk-sensitive reinforcement learning.

Reviewer comment (Novelty):
Can the authors clarify the contribution beyond what has already been studied towards the goal of RL with non-ergodic reward functions e.g. what is new compared to Majeed and Hutter (2018).

Author response:
**Connections to related work.** The main contribution of this paper is raising the question of ergodicity of reward functions in RL and how it impacts the use of the expected value as an optimization objective, as well as proposing a first approach toward resolving the problem. Other approaches that we are aware of consider non-ergodicity of MDPs but do not make the connection to the divergence between expected rewards and long-term rewards in such settings. Majeed and Hutter prove that the Q-learning algorithm still converges under non-ergodic dynamics. In the coin-toss game, the PPO algorithm also converges. And it does converge toward the policy that yields the maximum expected return. However, this policy yields a return of 0 in the long run with probability 1. Convergence itself does not necessarily help if the optimization objective is the problem.

Reviewer comment (Novelty):
How does this approach compare with risk-sensitive RL?

Author response:
**Risk-sensitive RL.** As we discuss in Section 5, for a specific class of reward functions (logarithmic), the risk-sensitive transformation is an ergodicity transformation. For such settings, the risk-sensitive transformation does exactly what we propose to do in this paper. This is not the case for other classes of reward functions - which is intuitive: a fixed transformation should only work well for a class of reward functions and not for all potential classes one could think of. We used the risk-sensitive transformation in the coin-toss game, whose dynamics are exponential, i.e., opposite from what the risk-sensitive transformation would need. In this case, the transformation fails.

Reviewer comment (Theory):
The proposed notion of (non-)ergodicity and how it relates to the underlying sequential decision-making problem is not clearly defined, seriously undermining the proposed approach and its potential relevance to existing work. This is a major drawback and my most serious concern. RL is typically viewed as a family of methods for (approximately) solving MDPs, where the goal is to find a policy -- a mapping from states to distributions over actions -- maximizing some notion of expected (discounted, average, or total) reward. For a fixed policy, a Markov chain (and corresponding Markov reward process) is induced over the underlying MDP. In RL, (non-)ergodicity is a property of these induced Markov chains and reward processes, so specifying a policy is necessary before one can consider (non-)ergodicity. In the paper, these issues are not discussed, resulting in lack of clarity at several critical points detailed in the questions section below. A critical issue is left unaddressed in the paper: **what is the relationship between the (non-)ergodicity of the reward process and the (non-)ergodicity of the underlying sequential decision-making process?**

Author response:
**Ergodicity notions.** In the RL literature, ergodicity is typically defined as the existence of a finite $N$ such that any state in the Markov chain can be reached from any other state in at most $N$ steps. We could now also understand the coin-toss game as a Markov chain, where the different states in the chain would represent the “wealth” of the agent. As long as we consider finite episode lengths, a finite path exists from any starting state to any other state in the chain. Nevertheless, in the limit, the agent will end up in an absorbing state (wealth equal to zero) with probability one. Thus, in the limit, due to the existence of an absorbing state, the game would also be non-ergodic from the MDP perspective. On the other hand, a non-ergodic MDP, which, for instance, has an absorbing state in which we end up with non-zero probability, should, in general, also be non-ergodic from the reward perspective. This is because, in the infinite time limit, we will end up in the absorbing state with probability 1. A case where the two definitions would not coincide would be if we define the reward dynamics themselves to follow a geometric Brownian motion while the agent moves in a grid world in which all states can be reached or if we have a grid world with absorbing state where all states, including the absorbing one, give the same reward. However, typically, ergodicity of the MDP should imply ergodicity of the reward function and vice versa.

Reviewer comment (Theory):
The variance-stabilizing transform developed in Sec. 4, though interesting, is insufficiently motivated. Importantly, the mechanism whereby it produces an ergodic, transformed reward sequence is unclear. This weakens support for the validity of the approach.

Author response:
**Variance-stabilizing transform.** The problem that this work attempts to solve is how to estimate a function that transforms a given time-series into one with ergodic increments. There is no established measure of how ergodic a time-series is, so instead, we take the approach of using a proxy metric, that is a metric that is easier to measure and still results in a transformed time series with ergodic increments. This motivates our choice of the variance-stabilizing transform as (assuming the incremental mean has no underlying trend) if the incremental variance of a random process is constant, then it will have ergodic increments. Of course, there is a class of random processes that have ergodic increments but varying incremental variance. A consequence of our proxy measure approach is that we cannot make any inferences on these processes.

Reviewer comment (Theory):
Sec. 5 points out an interesting connection between risk-sensitive RL and the transformation proposed in Sec. 4, yet it remains unclear for what classes of problems (i.e., what classes of MDPs, rewards, and policies) the $\hat{\text{I}}$to process of Eq. (4) is a reasonable model for the reward process.

Author response:
**Ito process.** Equation (4) assumes that the reward function is an Ito process. This is a very general type of stochastic processes, as the two functions $f$ and $g$ can be nonlinear or even stochastic. It is also used more as an analysis tool. We state for which underlying Ito process the risk-sensitive transformation would be an ergodicity transformation and find that we can get valuable insights from this perspective.

Reviewer comment (Experiments):
The experiments are limited: performance improvements over baseline methods are observed only on the simple coin toss betting problem and Reacher; comparable performance to baseline is observed on Cartpole.

Author response:
**Experiments.** In the experiments, we see indeed that the improvements for the cart-pole are only modest. This implies that either the reward function is ergodic or that the difference between the time average and expected value is relatively small so it does not show too clearly in the experiments. For the reacher, on the other hand, we see a clear improvement.

Reviewer comment (Theory):
How does your notion of (non-)ergodicity relate to that considered in (Puterman, 2014), (Sutton & Barto, 2018), and many recent works on RL theory?

Author response:
**Ergodicity notions.** In the RL literature, ergodicity is typically defined as the existence of a finite $N$ such that any state in the Markov chain can be reached from any other state in at most $N$ steps. We could now also understand the coin-toss game as a Markov chain, where the different states in the chain would represent the “wealth” of the agent. As long as we consider finite episode lengths, a finite path exists from any starting state to any other state in the chain. Nevertheless, in the limit, the agent will end up in an absorbing state (wealth equal to zero) with probability one. Thus, in the limit, due to the existence of an absorbing state, the game would also be non-ergodic from the MDP perspective. On the other hand, a non-ergodic MDP, which, for instance, has an absorbing state in which we end up with non-zero probability, should, in general, also be non-ergodic from the reward perspective. This is because, in the infinite time limit, we will end up in the absorbing state with probability 1. A case where the two definitions would not coincide would be if we define the reward dynamics themselves to follow a geometric Brownian motion while the agent moves in a grid world in which all states can be reached or if we have a grid world with absorbing state where all states, including the absorbing one, give the same reward. However, typically, ergodicity of the MDP should imply ergodicity of the reward function and vice versa.

Reviewer comment (Writing):
On page 2, the reward function is defined both as $R : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ and $R(T) = \sum_{t_k=0}^T r(t_k)$, which appear incompatible; which is correct?

Author response:
**Notation.** We agree that the notation has not been clear enough in the submitted version, and we have thoroughly revised it, also defining more clearly what we mean by ergodicity and how exactly it relates to RL. The reward typically depends on the current state and action. I.e., we have $r(a(t_k), s(t_k))$. We abbreviate this for conciseness as $r(t_k)$, as we now define in the paper.

Reviewer comment (Writing):
What is the definition of $t_k$ in Eq. (1)? If $t_k$ denotes timestep, then what is $r(t_k)$ and how does it relate to $\mathcal{S}$ and $\mathcal{A}$?

Author response:
**Notation.** We agree that the notation has not been clear enough in the submitted version, and we have thoroughly revised it, also defining more clearly what we mean by ergodicity and how exactly it relates to RL. The reward typically depends on the current state and action. I.e., we have $r(a(t_k), s(t_k))$. We abbreviate this for conciseness as $r(t_k)$, as we now define in the paper.

Reviewer comment (Writing):
What is the expectation in Eq. (2) taken w.r.t.?

Author response:
**Expectation.** Given a stochastic environment, such as in the coin-toss game, a policy induces a stochastic process. The expected return then gives us a measure of how good this policy is. So conventionally, in RL, we would maximize over policies to find the one with the highest expected return. In that sense, we take the expected value over trajectories induced by a certain policy.

Reviewer comment (Reproducibility):
What is the action space in Sec. 2.1? What policies were used in the experiment in Fig. 1?

Author response:
**Action space.** In Section 2.1, the action space is the discrete decision of playing or not playing the game. Since not playing it results in no change in wealth, in Figure 1, we show the dynamics if one decides to play the game.

Reviewer comment (Theory):
Under what conditions on the underlying problem is Eq. (4) a good reward model?

Author response:
**Ito process.** Equation (4) assumes that the reward function is an Ito process. This is a very general type of stochastic processes, as the two functions $f$ and $g$ can be nonlinear or even stochastic. It is also used more as an analysis tool. We state for which underlying Ito process the risk-sensitive transformation would be an ergodicity transformation and find that we can get valuable insights from this perspective.

Reviewer comment (Experiments):
The risk-sensitive transformation is not applied successfully in any setting.

Author response:
**Application of risk-sensitive transformation.** The risk-sensitive transformation we discuss in the paper has been applied in various works, for instance, Prashanth et al. 2022, Noorani et al. 2022, and Fei et al. 2021. As this transformation is not a contribution of our paper, we did not provide a successful example ourselves. Instead, we wanted to show that the risk-sensitive transformation is an ergodicity transformation given a specific form of reward function. However, when the reward dynamics are very different from this form, as in the coin-toss game, such fixed transformations fail. Therefore, a more generic approach, such as the transformation we propose, is required.

Reviewer comment (Presentation):
Could the author clarify the meaning of 't_k'? If it represents the 'k'-th step within one episode, then the use of 'R(t)' in Figure 4(a) should be reconsidered, as this should represent the cumulative rewards of each episode. It is essential for the author to provide a clearer explanation regarding the interpretation of the vertical axis.

Author response:
**Notation.** The variable $t_k$ in our paper indeed represents the time step. We here added the indice $k$ to clearly distinguish it from the $t$ in Section 5, where we consider a continuous-time setting. Figure 4 shows the return for the cart-pole and reacher experiments at $t_k=T$ over the number of episodes. The parameter $T$ has also in the problem setting been defined as the duration of an episode.

Reviewer comment (Experiments):
The author's analysis focuses solely on the case of an exponential distribution. However, given the multitude of reward functions in simulations or real environment, it becomes challenging to determine whether they exhibit ergodicity. A broader exploration of different reward functions would enhance the paper's comprehensiveness.

Author response:
**Different distributions.** We focus our analysis on one particular example for which we know that the rewards are non-ergodic and show that state-of-the-art reinforcement learning algorithms cannot solve it - despite the simple dynamics. The fact that even such a simple example can pose severe challenges for established methods should be a good reason to re-evaluate the usage of the expectation operator for highly complex environments where RL is usually applied. In Section 6, we discuss in more detail for which types of reward functions our ideas are most important. Apart from non-ergodic stochastic processes, we particularly highlight environments where we have an absorbing barrier. We can often find such settings in safe learning settings where reaching forbidden states, e.g., a robot crashing into an obstacle, automatically ends the episode. A risky policy, for which there is a non-zero probability of reaching the forbidden state when rolled out many times for a short duration, might now, on average, yield a high reward, as it only sometimes crashes. However, if we let the same policy run for a long time, we will almost surely eventually reach the forbidden state. Thus, also here time average and expected value do not coincide, and, for the sake of robustness, we might want to go for optimizing the time average.

Reviewer comment (Evaluation):
Could the author provide a more detailed explanation for the observed difference in improvement between the Reacher and Cartpole environments? Does this suggest that the reward dynamics in Cartpole are ergodic, while they are non-ergodic in Reacher?

Author response:
**Cart-pole example.** In the cart-pole environment, the differences between the runs are only modest. This suggests that the cart-pole environment is either ergodic or that the non-ergodicity here plays a less significant role. For a non-ergodic process, time average and expected value are different, but they may still be close to each other. For the reacher, this apparently does not hold. The difference between testing and training is because we change the link length, as we also mention in the corresponding section.