M AXIMUM E NTROPY RL (P ROVABLY ) S OLVES S OME R OBUST RL P ROBLEMS
A BSTRACT
Many potential applications of reinforcement learning (RL) require guarantees that the agent will perform well in the face of disturbances to the dynamics or reward function.
In this paper, we prove theoretically that maximum entropy ( MaxEnt
RL maximizes a lower bound on a robust RL objective, and thus can be used to learn policies that are robust to some disturbances in the dynamics and the reward function.
While this capability of MaxEnt RL has been observed empirically in prior work to the best of our knowledge our work provides the ﬁrst rigorous proof and theoretical characterization of the MaxEnt RL robust set
While a number of prior robust RL algorithms have been designed to handle similar disturbances to the reward function or dynamics these methods typically require additional moving parts and hyperparameters on top of a base RL algorithm In contrast
, our results suggest that MaxEnt RL by itself is robust to certain disturbances , without requiring any additional modiﬁcations
While this does not imply that MaxEnt RL is the best available robust RL method, MaxEnt RL is a simple robust RL method with appealing formal guarantees.
I NTRODUCTION
Many real-world applications of reinforcement learning (RL) require control policies that not only maximize reward but continue to do so when faced with environmental disturbances, modeling errors, or errors in reward speciﬁcation.
These disturbances can arise from human biases and modeling errors, from non-stationary aspects of the environment, or actual adversaries acting in the environment.
A number of works have studied how to train RL algorithms to be robust to disturbances in the environment (e.g.,
Morimoto & Doya (2005); Pinto et al.
Tessler et al.
However, designing robust
RL algorithms requires care, typically requiring an adversarial optimization problem and introducing additional hyperparameters (Pinto et al., 2017; Tessler et al., 2019).
Instead of designing a new robust RL algorithm, we will instead analyze whether an existing RL method, MaxEnt RL, offers robustness to perturbations.
MaxEnt RL methods are based on the maximum entropy principle, and augmented the expected reward objective with an entropy maximization term (Ziebart et al., 2008).
Prior work has conjectured that such algorithms should learn robust policies (Haarnoja et al., 2017; Huang et al., 2019), on account of the close connection between maximum entropy methods and robustness in supervised learning domains (Grünwald et al., 2004; Ziebart, 2010).
However, despite decades of research on maximum entropy RL methods (Kappen, 2005; Todorov, 2007; Toussaint, 2009; Theodorou et al., 2010), a formal characterization of the robustness properties of such approaches has proven elusive.
To our knowledge, no prior work formally proves that MaxEnt RL methods are robust, and no prior work characterizes the disturbances against which these methods are robust.
Showing how to obtain robust policies from existing MaxEnt RL methods, which already constitute a signiﬁcant portion of the RL methods in use today (Abdolmaleki et al., 2018; Haarnoja et al., 2018a; Vieillard et al., 2020), would be useful because it would enable practitioners to leverage existing, tried-and-true methods to solve robust RL problems.
Stochastic policies, of the sort learned with MaxEnt RL, inject noise into the actions during training, thereby preparing themselves for deployment in environments with disturbances.
For example, in the robot pushing task shown in Fig. 1, the policy learned by MaxEnt RL pushes the white puck to the goal using many possible routes.
In contrast, (standard) RL learns a deterministic policy, always using the same route to get to the goal.
Now, imagine that this environment is perturbed by adding the red barrier in Fig.
While the policy learned by (standard) RL always collides with this obstacle, the policy learned by MaxEnt RL uses many routes to solve the task, and some fraction of these routes continue to solve the task even when the obstacle is present.
While a number of prior works have articulated the intuition that the stochastic policies learned via MaxEnt RL should be robust to disturbances (Levine, 2018; Abdolmaleki et al., 2018; Lee et al., 2019), no prior work has actually shown that MaxEnt RL policies are provably robust, nor characterized the set of disturbances to which they are robust.
Applications of MaxEnt RL methods to problems that demand robustness are likely hampered by a lack of understanding of when such methods are robust, what kinds of reward functions should be used to obtain the desired type of robustness, and how the task should be set up.
The goal in our work is to make this notion precise, proving that MaxEnt RL is already a robust RL algorithm, and deriving the robust set for these policies.
The main contribution of this paper is a theoretical characterization of the robustness of existing
MaxEnt RL methods.
Speciﬁcally, we show that the MaxEnt RL objective is a nontrivial lower bound on the robust RL objective, for a particular robust set.
Importantly, these two objectives use different reward functions: to learn a policy that maximizes a reward function on a wide range of dynamics functions, one should apply MaxEnt RL to a pessimistic version of that reward function.
MaxEnt
RL is not robust with respect to the reward function it is trained on.
As we will show, the robust set for these methods is non-trivial, and in some important special cases can be quite intuitive.
To our knowledge, our results are the ﬁrst to formally show robustness of standard MaxEnt RL methods to dynamics perturbations and characterize their robust set.
Importantly, our analysis and experiments highlight that robustness is only achieved for relatively large entropy coefﬁcients.
We validate our theoretical results on a set of illustrative empirical experiments.
R ELATED W ORK
The study of robustness in controls has a long history, with robust or H ∞ control methods proposing provably robust solutions under various assumptions on the true dynamics (Zhou et al., 1996; Doyle et al., 2013).
RL offers an appealing alternative, since robust RL methods can in principle learn robust policies without knowing the true dynamics.
However, policies learned by standard RL algorithms often ﬂounder in the face of environmental disturbances (Rajeswaran et al., 2016; Peng et al., 2018).
The problem of learning robust policies that achieve high reward in the face of adversarial environmental disturbances, often called robust RL , has been well studied in the literature (Bagnell et al., 2001; Nilim & Ghaoui, 2003; Morimoto & Doya, 2005; Pinto et al., 2017; Tessler et al., 2019;
Russel & Petrik, 2019; Kamalaruban et al., 2020; Russel et al., 2020; 2021; Derman et al., 2021).
Prior robust RL methods are widely-applicable but often require many additional hyperparameters or components.
For example, Bagnell et al.
(2001) modify the Bellman backup in a Q-learning by solving a convex optimization problem in an inner loop, and Tessler et al.
(2019) trains an additional adversary policy via RL.
Robust RL is different from robust control in the traditional sense, which focuses on stability independent of any reward function (Zhou et al., 1996; Doyle et al., 2013).
Robust RL problem involves estimating a policy’s returns under “similar” MDPs, a problem that has been studied in terms of distance metrics for MDPs (Lecarpentier et al., 2020).
Robust RL is different from maximizing the average reward across many environments, as done by methods such as domain randomization (Sadeghi & Levine, 2016; Rajeswaran et al., 2016; Peng et al., 2018).
Closely related to robust RL are prior methods that are robust to disturbances in the reward function (Hadﬁeld-Menell et al., 2017; Bobu et al., 2020; Michaud et al., 2020), or aim to minimize a cost function in addition to maximizing reward (Chow et al., 2017; Achiam et al., 2017;
Chow et al., 2019; Carrara et al., 2019; Tang et al., 2020; Thananjeyan et al., 2021).
Robust RL is also different from the problem of learning transferable or generalizable RL agents (Lazaric, 2012; Zhang et al., 2018; Justesen et al., 2018; Cobbe et al., 2019), which focuses on the average-case performance on new environments, rather than the worst-case performance.
P RELIMINARIES
We assume that an agent observes states s t , takes actions a t ∼ π ( a t
| s t ) , and obtains rewards r ( s t , a t ) .
The initial state is sampled s 1 ∼ p 1 ( s 1 ) , and subsequent states are sampled s t + 1 ∼ p ( s t
+ 1 | s t , a t ) .
We will use p π ( τ ) to denote the probability distribution over trajectories for policy π .
Episodes have T steps, which we summarize as a trajectory τ (cid:44) ( s 1 , a 1 , · · · , s T , a T ) .
Without loss of generality, we can assume that rewards are undiscounted, as any discount can be addressed by modifying the dynamics to transition to an absorbing state with probability 1 − γ .
The standard RL objective is:
[Equation]
t = r
( s
[Equation]
a t ) (cid:21) , where p π ( τ ) =
p 1 ( s 1 )
T (cid:89) t = p ( s [Equation] s [Equation]
( a
[Equation] s [Equation] dτ is the distribution over trajectories when using policy π .
In fully observed MDPs, there always exists a deterministic policy as a solution (Puterman, 2014).
The MaxEnt RL objective is to maximize the sum of expected reward and conditional action entropy.
Formally, we deﬁne this objective in terms of a policy π , the dynamics p and the reward function r :
[Equation]
1) where H π [ a t | s t ] = (cid:82)
A π ( a t | s t ) log 1 π ( a t
| s t ) d a t denotes the entropy of the action distribution.
The entropy coefﬁcient α balances the reward term and the entropy term; we use α = 1 in our analysis.
As noted above, the discount factor is omitted because it can be subsumed into the dynamics.
Our main result will be that maximizing the MaxEnt RL objective (Eq. 1) results in robust policies.
We quantify robustness by measuring the reward of a policy when evaluated on a new reward function ˜ r or dynamics function ˜ p , which is chosen adversarially from some set:
[Equation]
This robust RL objective is deﬁned in terms of the sets of dynamics
˜ P and reward functions ˜ R .
Our goal is to characterize these sets.
The robust RL objective can be interpreted as a two-player, zero-sum game.
The aim is to ﬁnd a Nash equilibrium.
Our goal is to prove that MaxEnt RL (with a different reward function ) optimizes a lower bound this robust objective, and to characterize the robust sets ˜ P and ˜ R for which this bound holds.
M AX E NT RL AND R OBUST C ONTROL
In this section, we prove the conjecture that MaxEnt RL is robust to disturbances in the environment.
Our main result is that MaxEnt RL can be used to maximize a lower bound on a certain robust RL objective.
Importantly, doing this requires that MaxEnt RL be applied to a different, pessimistic, version of the target reward function.
Before presenting our main result, we prove that MaxEnt RL is robust against disturbances to the reward function.
This result is a simple extension of prior work, and we will use this result for proving our main result about dynamics robustness
4. R OBUSTNESS TO A DVERSARIAL R EWARD F UNCTIONS
We ﬁrst show that MaxEnt RL is robust to some degree of misspeciﬁcation in the reward function.
This result may be useful in practical settings with learned reward functions (Fu et al., 2018;
Xu &
Denil, 2019; Michaud et al., 2020) or misspeciﬁed reward function (Amodei et al., 2016; Clark &
Amodei, 2016).
Precisely, the following result will show that applying MaxEnt RL to one reward function, r ( s t , a t ) , results in a policy that is guaranteed to also achieve high return on a range of other reward functions, ˜ r ( s t , a t ) ∈ ˜ R :
Theorem 4.1.
Let dynamics p
( s t + 1 | s t , a t ) , policy π ( a t
| s t ) , and reward function r ( s t , a t ) be given.
Assume that the reward function is ﬁnite and that the policy has support everywhere (i.e., π ( a t | s t ) > 0 for all states and actions).
Then there exists a positive constant (cid:15) > 0 such that the
MaxEnt RL objective J MaxEnt is equivalent to the robust RL objective deﬁned by the robust set ˜ R ( π ) :
[Equation]
where the adversary chooses a reward function from the set
[Equation]
Thus, when we use MaxEnt RL with some reward function r , the policy obtained is guaranteed to also obtain high reward on all similar reward functions ˜ r that satisfy Eq.
We discuss how MaxEnt
RL can be used to learn policies robust to arbitrary sets of reward functions in Appendix A.10.
The proof can be found in Appendix A.2, and is a simple extension of prior work (Grünwald et al., 2004;
Ziebart et al., 2011).
Our proof does not assume that the reward function is bounded nor that the policy is convex.
While the proof is straightforward, it will be useful as an intermediate step when proving robustness to dynamics in the next subsection.
The robust set ˜ R corresponds to reward functions ˜
r that are not too much smaller than the original reward function: the adversary cannot decrease the reward for any state or action too much.
The robust set ˜ R depends on the policy, so the adversary has the capability of looking at which states the policy visits before choosing the adversarial reward function ˜ r .
For example, the adversary may choose to apply larger perturbations at states and actions that the agent frequently visits.
4. R OBUSTNESS TO A DVERSARIAL D YNAMICS
We now show that MaxEnt RL learns policies that are robust to perturbations to the dynamics.
Importantly, to have MaxEnt RL learn policies that robustly maximize one a reward function, we will apply MaxEnt RL to a different, pessimistic reward function:
[Equation]
The log( · ) transformation is common in prior work on learning risk-averse policies (Mihatsch &
Neuneier, 2002).
The entropy term rewards the policy for vising states that have stochastic dynamics, which should make the policy harder for the adversary to exploit.
In environments that have the same stochasticity at every state (e.g., LG dynamics), this entropy term becomes a constant and can be ignored.
In more general settings, computing this pessimistic reward function would require some knowledge of the dynamics.
Despite this limitation, we believe that our results may be of theoretical interest, taking a step towards explaining the empirically-observed robustness properties of MaxEnt RL.
To formally state our main result, we must deﬁne the range of “similar” dynamics functions against which the policy will be robust.
We use the following divergence between two dynamics functions:
[Equation]
This divergence is large when the adversary’s dynamics ˜
p
( s t + 1 | s t , a t ) assign low probability to a transition with high probability in the training environment ˜ p .
Our main result shows that applying
MaxEnt RL to the pessimistic reward function results in a policy that is robust to these similar dynamics functions:
Theorem 4.2.
Let an MDP with dynamics p ( s t + 1 | s t , a t ) and reward function r ( s t , a t ) > 0 be given.
Assume that the dynamics have ﬁnite entropy (i.e., H [ s t + 1 | s t , a t ] is ﬁnite for all states and actions).
Then there exists a constant (cid:15) >
H π ∗ [ a t | s t ] such that the MaxEnt RL objective with dynamics p and reward function ¯ r is a lower bound on the robust RL objective
[Equation]
where the robust set is deﬁned as
[Equation]
In deﬁning the robust set, the adversary chooses a dynamics function from the robust set ˜ P independently at each time step; our next result (Lemma 4.3) will describe the value of (cid:15) in more detail.
The proof, presented in Appendix A.4, ﬁrst shows the robustness to reward perturbations implies robustness to dynamics perturbations and then invokes Theorem 4.1 to show that MaxEnt RL learns policies that are robust to reward perturbations.
This result can be interpreted in two ways.
First, if a user wants to acquire a policy that optimizes a reward under many possible dynamics, we should run MaxEnt RL with a speciﬁc, pessimistic reward function ¯ r ( s t , a t , s t + 1 ) .
This pessimistic reward function depends on the environment dynamics, so it may be hard to compute without prior knowledge or a good model of the dynamics.
However, in some settings (e.g., dynamics with constant additive noise), we might assume that
H [ s t + 1 | s t , a t ] is approximately constant, in which case we simply set the MaxEnt RL reward to ¯ r ( s t , a t , s t + 1 )
= log r ( s t , a t ) .
Second, this result says that every time a user applies MaxEnt RL, they are (implicitly) solving a robust RL problem, one deﬁned in terms of a different reward function.
This connection may help explain why prior work has found that the policies learned by MaxEnt RL tend to be robust against disturbances to the environment (Haarnoja et al., 2019).
Theorem 4.2 relates one MaxEnt RL problem to another (robust) MaxEnt RL problem.
We can also show that MaxEnt RL maximizes a lower bound on an unregularized RL problem.
Corollary 4.2.1.
Under the same assumptions as Theorem 4.1 and 4.2, the MaxEnt RL problem is a lower bound on the robust RL objective:
[Equation]
where the robust sets ˜ P
( π ) and ˜ R ( π ) as deﬁned as in Theorem 4.1 and 4.2.
Our next result analyzes the size of the robust set by providing a lower bound on (cid:15) .
Doing so proves that Theorem 4.2 is non-vacuous and will also tell us how to increase the size of the robust set.
Lemma 4.3.
Assume that the action space is discrete.
Let a reward function r ( s t , a t ) and dynamics p ( s t
+
| s t , a t ) be given.
Let π ( a t | s t ) be the corresponding policy learned by MaxEnt RL.
Then the size of (cid:15) (in Eq. 5) satisﬁes the following:
[Equation]
This result provides an exact expression for (cid:15) .
Perhaps more interesting is the inequality, which says that the size of the robust set is at least as large as the policy’s entropy.
For example, if MaxEnt
RL learns a policy with an entropy of 10 bits, then (cid:15) ≥ 10 .
This result immediately tells us how to increase the size of the robust set: run MaxEnt RL with a larger entropy coefﬁcient α .
Many popular implementations of MaxEnt RL automatically tune the entropy coefﬁcient α so that the policy satisﬁes an entropy constraint (Haarnoja et al., 2018b).
Our derivation here suggests that the entropy constraint on the policy corresponds to a lower bound on the size of the robust set.
The constraint in the deﬁnition of the robust set holds in expectation.
Thus, the adversary can make large perturbations to some transitions and smaller perturbations to other states, as long as the average perturbation is not too large.
Intuitively, the constraint value (cid:15) is the adversary’s “budget,” which it can use to make large changes to the dynamics in just a few states, or to make smaller changes to the dynamics in many states.
4. W ORKED E XAMPLES
This section provides worked examples of our robustness results.
The aim is to build intuition for what our results show and to show that, in simple problems, the robust set has an intuitive interpretation.
Reward robustness.
We present two worked examples of reward robustness.
To simplify analysis, we consider the following, smaller robust set, which effectively corresponds to a weaker adversary:
[Equation]
For the ﬁrst example
[Equation]
deﬁne a 2-armed bandit with the following reward function and corresponding robust set:
[Equation]
Fig.
2 (left) plots the original reward function, r as a red dot, and the collection of reward functions, ˜
R , as a blue line.
In the center plot we plot the expected reward for each reward in ˜ R as a function of the action a .
The robust RL problem in this setting is to choose the policy whose worst-case reward (dark blue line) is largest.
The right plot shows the MaxEnt RL objective.
We observe that the robust
RL objective and the MaxEnt RL objectives are equivalent.
For the second example , we use a task with a 1D, bounded action space A =
[ − 10 , 10] and a reward function composed of a task-speciﬁc reward r task and a penalty for deviating from some desired action a ∗ : r ( s , a ) (cid:44) r task ( s , a ) − ( a − a ∗ ) 2 .
The adversary will perturb this desired action by an amount ∆ a and decrease the weight on the control penalty by 50%, resulting in the following reward function: ˜ r ( s , a ) (cid:44) r task ( s , a ) − 12 ( a − ( a ∗ + ∆ a )) 2 .
In this example, the subset of the robust set in
Eq.
6 corresponds to perturbations ∆ a that satisfy
[Equation]
Thus, MaxEnt RL with reward function r yields a policy that is robust against adversaries that perturb a ∗ by at most ∆ a = O ( √ (cid:15) ) .
See Appendix A.3 for the full derivation.
Dynamics robustness.
The set of dynamics we are robust against, ˜ P , has an intuitive interpretation as those that are sufﬁciently close to the original dynamics p ( s t + 1 | s t , a t ) .
This section shows that, in the case of linear-Gaussian dynamics (described at the end of this subsection), this set corresponds to a bounded L2 perturbation of the next state.
Because the robust set in Theorem 4.2 is deﬁned in terms of the policy, the adversary can intelligently choose where to perturb the dynamics based on the policy’s behavior.
Robustness against this adversary also guarantees robustness against an adversary with a smaller robust set, that does not depend on the policy:
[Equation]
[Table]
[Equation]
We will use this subset of the robust set in the following worked example.
Consider an MDP with 1D, bounded, states and actions s t , a t ∈ [ − 10 , 10] .
The dynamics are p ( s t + 1 | s t , a t ) =
N ( s t + 1 ; µ =
A s t + B a t , σ = 1) the reward function is r ( s t , a t ) =
(cid:107) s t (cid:107) 22 , and episodes have ﬁnite length T .
Note that the dynamics entropy H [ s t + 1 | s t , a t ] is constant.
We assume the adversary modiﬁes the dynamics by increasing the standard deviation to √ 2 and shifts the bias by an amount β , resulting in the dynamics ˜
p
( s t + 1 | s t , a t )
=
N ( s t + 1 ; µ =
A s t + B a t + β, σ = √ 2) .
The robust set deﬁned in Theorem 4.2 speciﬁes that the adversary can choose any value of β that satisﬁes 12 β 2 + log(8 √ π ) + log(20) ≤ (cid:15) .
To apply MaxEnt RL to learn a policy that is robust to any of these adversarial dynamics, we would use the pessimistic reward function speciﬁed by
Theorem 4.2: ¯ r ( s t , a t , s t + 1 )
= 2 T log (cid:107) s t (cid:107) 2 .
See Appendix A.7 for the full derivation.
4. L IMITATIONS OF A NALYSIS
We identify a few limitations of our analysis that may provide directions for future work.
First, our deﬁnition of the robust set is different from the more standard H ∞ and KL divergence constraint sets used in prior work.
Determining a relationship between these two different sets would allow future work to claim that MaxEnt RL is also robust to these more standard constraint sets (see Lecarpentier et al.
On the other hand, MaxEnt RL may not be robust to these more conventional constraint sets.
Showing this may inform practitioners about what sorts of robustness they should not except to reap from MaxEnt RL.
A second limitation is the construction of the augmented reward: to learn a policy that will maximize reward function r under a range of possible dynamics functions, our theory says to apply MaxEnt RL to a different reward function, ¯ r , which includes a term depending on the dynamics entropy.
While this term can be ignored in special MDPs that have the same stochasticity at every state, in more general MDPs it will be challenging to estimate this augmented reward function.
Determining more tractable ways to estimate this augmented reward function is an important direction for future work.
N UMERICAL S IMULATIONS
This section will present numerical simulations verifying our theoretical result that MaxEnt RL is robust to disturbances in the reward function and dynamics function.
We describe our precise implementation of MaxEnt RL, standard RL, and all environments in Appendix B.
Comparison with prior robust RL methods.
We compare MaxEnt RL against two recent robust
RL methods, PR-MDP and NR-MDP (Tessler et al., 2019).
These methods construct a two-player game between a player that chooses actions and a player that perturbs the dynamics or actions.
This recipe for robust RL is common in prior work (Pinto et al., 2017), and we choose to compare to Tessler et al.
(2019) because it is a recent, high-performing instantiation of this recipe.
We also include the
DDPG baseline from Tessler et al.
We evaluate all methods on the benchmark tasks used in Tessler et al.
(2019), which involve evaluating in an environment where the masses are different from the training environment.
We ran MaxEnt RL with both small and large entropy coefﬁcients, evaluating the ﬁnal policy for 30 episodes and taking the average performance.
We repeated this forrandom seeds to obtain the standard deviation.
The results shown in Fig.
3 suggest that MaxEnt RL with a large entropy coefﬁcient α is at least competitive, if not better, than prior purpose-designed robust RL methods.
Note that MaxEnt RL is substantially simpler than PR-MDP and NR-MDP.
Intuition for why MaxEnt RL is robust.
To build intuition into why MaxEnt RL should yield robust policies, we compared MaxEnt RL (SAC (Haarnoja et al., 2018a)) with standard RL (TD3 (Fujimoto et al., 2018)) on two tasks.
On the pusher task, shown in Fig.
1, a new obstacle was added during evaluation.
On the button task, shown in Fig.
4b, the box holding the button was moved closer or further away from the robot during evaluation.
Note that the robust RL objective corresponds to an adversary choosing the worst-case disturbances to these environments.
Fig.
1 (right) and Fig.
4b (center) show that MaxEnt RL has learned many ways of solving these tasks, using different routes to push the puck to the goal and using different poses to press the button.
Thus,
[Figure]
when we evaluate the MaxEnt RL policy on perturbed environments, it is not surprising that some of these strategies continue to solve the task.
In contrast, the policy learned by standard RL always uses the same strategy to solve these tasks, so the agent fails when a perturbation to the environment makes this strategy fail.
Quantitatively, Fig. 4a and 4b (right) show that the MaxEnt RL policy is more robust than a policy trained with standard RL.
In many situations, simply adding noise to the deterministic policy found by standard RL can make that policy robust to some disturbances.
MaxEnt RL does something more complex, dynamically adjusting the amount of noise depending on the current state.
This capability allows
MaxEnt RL policies to have lower entropy in some states as needed to ensure high reward.
We study this capability in the 2D navigation task shown in Fig.
The agent starts near the top left corner and gets reward for navigating to the bottom right hand corner, but incurs a large cost for entering the red regions.
The policy learned by MaxEnt RL has low entropy initially to avoid colliding with the red obstacles, and then increases its entropy in the second half of the episode.
To study robustness, we introduce a new “L”-shaped obstacle for evaluation.
The policy learned by MaxEnt RL often navigates around this obstacle, whereas the policy from standard RL always collides with this obstacle.
Adding independent Gaussian noise to the actions from the standard RL policy can enable that policy to navigate around the obstacle, but only at the cost of entering the costly red states.
Testing for different types of robustness.
Most prior work on robust RL focuses on changing static attributes of the environment, such as the mass or position of objects (Tessler et al., 2019;
Kamalaruban et al., 2020).
However, our analysis suggests that MaxEnt RL is robust against a wider range of perturbations, which we probe in our next set of experiments.
First, we introduced perturbations in the middle of an episode.
We took the pushing task shown in Fig.
1 and, instead of adding an obstacle, perturbed the XY position of the puck after 20 time steps.
By evaluating the reward of a policy while varying the size of this perturbation, we can study the range of disturbances to which MaxEnt RL is robust.
We measured the performance of MaxEnt RL policies trained with different entropy coefﬁcients α .
The results shown in
Fig.
6 indicate all methods perform well on the environment without any disturbances, but only the MaxEnt RL trained with the largest entropy coefﬁcient is robust to larger disturbances.
This experiment supports our theory that the entropy coefﬁcient determines the size of the robust set.
Our analysis suggests that MaxEnt RL is not only robust to random perturbations, but is actually robust against adversarial perturbations.
We next compare MaxEnt RL and standard RL in this setting.
We trained both algorithms on a peg insertion task shown in Fig.
7 (left) .
Visualizing the learned policy, we observe that the standard RL policy always takes the same route to the goal, whereas the MaxEnt RL policy uses many different routes to get to the goal.
For each policy we found the worst-case perturbation to the hole location using CMA-ES (Hansen, 2016).
For small perturbations, both standard RL and MaxEnt RL achieve a success rate near 100%.
However, for perturbations that are 1.5cm or 2cm in size, only MaxEnt RL continues to solve the task.
For larger perturbations neither method can solve the task.
In summary, this experiment highlights that MaxEnt
RL is robust to adversarial disturbances, as predicted by the theory.
Finally, our analysis suggests that MaxEnt RL is also robust to perturbations to the reward function.
To test this theoretical result, we apply MaxEnt RL on four continuous control tasks from the standard OpenAI Gym (Brockman et al., 2016) benchmark.
We compare to SVG-0 (Heess et al., 2015) (which uses stochastic policy gradients) and to ﬁctitious play (Brown, 1951).
In the RL setting, ﬁctitious play corresponds to modifying standard RL to use an adversarially-chosen reward function for each Bellman update.
We evaluate the policy on an adversarially chosen reward function, chosen from the set deﬁned in Equation 2.
The analytic solution for this worst-case reward function is ˜ r
( s t , a t ) =
r ( s t , a t ) − log π ( a t | s t ) .
Both MaxEnt RL and standard RL can maximize the cumulative reward (see Fig.
11 in Appendix B), but only MaxEnt RL succeeds as maximizing the worst-case reward, as shown in Fig. 8.
In summary, this experiment supports our proof that MaxEnt
RL solves a robust RL problem for the set of rewards speciﬁed in Theorem 4.1.
D ISCUSSION
In this paper, we formally showed that MaxEnt RL algorithms optimize a bound on a robust RL objective.
This robust RL objective uses a different reward function than the MaxEnt RL objective.
Our analysis characterizes the robust sets for both reward and dynamics perturbations, and provides intuition for how such algorithms should be used for robust RL problems.
To our knowledge, our work is the ﬁrst to formally characterize the robustness of MaxEnt RL algorithms, despite the fact that numerous papers have conjectured that such robustness results may be possible.
Our experimental evaluation shows that, in line with our theoretical ﬁndings, simple MaxEnt RL algorithms perform competitively with (and sometimes better than) recently proposed adversarial robust RL methods on benchmarks proposed by those works.
Of course, MaxEnt RL methods are not necessarily the ideal approach to robustness: applying such methods still requires choosing a hyperparameter (the entropy coefﬁcient), and the robust set for
MaxEnt RL is not always simple.
Nonetheless, we believe that the analysis in this paper, represents an important step towards a deeper theoretical understanding of the connections between robustness and entropy regularization in RL.
We hope that this analysis will open the door for the development of new, simple algorithms for robust RL.
Acknowledgments
We thank Oﬁr Nachum, Brendan O’Donoghue, Brian Ziebart, and anonymous reviewers for their feedback on an early drafts.
BE is supported by the Fannie and John Hertz Foundation and the National Science Foundation GFRP (DGE 1745016)
A. U SEFUL L EMMAS
Before stating the proofs
[Equation]
we recall and prove two (known) identities.
Lemma A.1.
[Equation]
This identity says that the negative entropy function is the Fenchel dual of the log-sum-exp function.
This identity can be proven using calculus of variations:
Proof.
We start by ﬁnding the function f
( x, y ) that optimizes the RHS of Eq. 7.
Noting that the RHS is a convex function of f ( x, y ) , we can ﬁnd the optimal f ( x, y ) by taking the derivative and setting it equal to zero:
[Equation]
Setting this derivative equal to zero
[Equation]
we see that solution f ∗ ( x, y ) satisﬁes
[Equation]
Now
[Equation]
we observe that p ( y ) = 1 :
[Equation]
We then have
[Equation]
Taking the log of both sides [Equation] we have
[Equation]
Multiplying both sides by -1 and taking the expectation w.r
[Equation]
t. p ( x, y )
[Equation]
we obtain the desired result:
[Equation]
This lemma also holds for functions with more variables, a result that will be useful when proving dynamics robustness
[Equation]
Lemma A.2.
[Equation]
Proof.
Simply apply Lemma A.1 to the random variables ( x, y ) and z .
A. P ROOF OF T HEOREM 4.
This section will provide a proof of Theorem 4.1.
Proof.
We start by restating the objective for robust control with rewards:
[Equation]
We now employ the KKT conditions (Boyd et al., 2004, Chpt. 5.5.3).
If the robust set ˜ R is strictly feasibly, then there exists a constraint value (cid:15) ≥
(the dual solution) such that the constrained problem is equivalent to the relaxed problem with Lagrange multiplier λ = 1 :
[Equation]
To clarify exposition, we parameterize ˜ r by its deviation from the original reward, ∆ r ( s t , a t ) (cid:44) r ( s t , a t ) − ˜ r ( s t , a t ) .
We also deﬁne ρ πt ( s t , a t ) as the marginal distribution over states and actions visited by policy π at time t .
We now rewrite the relaxed objective in terms of ∆ r ( s t , a t ) :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The last line follows from applying Lemma A.1.
One important note about this proof is that the KKT conditions are applied to the problem of optimizing the reward function ˜ r , which is a convex optimization problem.
We do not require that the policy π ( a t | s t ) or the dynamics p ( s t + 1 | s t , a t ) be convex.
A. W ORKED E XAMPLE OF R EWARD R OBUSTNESS
We compute the penalty for Example 1 in Section 4.3 using a Gaussian integral:
[Equation]
A. P ROOF OF T HEOREM 4.
Proof.
We now provide the proof of Theorem 4.2.
The ﬁrst step will be to convert the variation in dynamics into a sort of variation in the reward function.
The second step will convert the constrained robust control objective into an unconstrained (penalized) objective.
The third step will show that the penalty is equivalent to action entropy.
Step 1: Dynamics variation → reward variation.
Our aim is to obtain a lower bound on the following objective:
[Equation]
We start by taking a log-transform of this objective, noting that this transformation does not change the optimal policy as the log function is strictly monotone increasing.
[Equation]
Note that we assumed that the reward was positive, so the logarithm is well deﬁned.
We can write this objective in terms of the adversarial dynamics using importance weights.
log
[Equation]
Both inequalities are applications of Jensen’s inequality.
As before, our assumption that the rewards are positive ensures that the logarithms remain well deﬁned.
While the adversary is choosing the dynamics under which we will evaluate the policy, we are optimizing a lower bound which depends on a different dynamics function.
This step allows us to analyze adversarially-chosen dynamics as perturbations to the reward .
Step 2: Relaxing the constrained objective To clarify exposition, we will parameterize the adversarial dynamics as a deviation from the true dynamics:
[Equation]
The constraint that the adversarial dynamics integrate to one can be expressed as
[Equation]
Using this notation
[Equation]
we can write the lower bound on the robust control problem (Eq. 9) as follows:
[Equation]
and
[Equation]
The constraint in Eq.
12 is the deﬁnition of the set of adversarial dynamics ˜ P , and the constraint in
Eq.
13 ensures that ˜ p ( s t + 1 | s t , a t ) represents a valid probability density function.
Note that ∆ r ( s t + 1 , s t , a t ) = 0 is a strictly feasible solution to this constrained optimization problem for (cid:15) > 0 .
Note also that the problem of optimizing the function ∆ r ( s t , a t ) is a convex optimization problem.
We can therefore employ the KKT conditions (Boyd et al., 2004, Chpt. 5.5.3).
If the robust set ˜ P is strictly feasibly, then there exists (cid:15) ≥
(the dual solution) such that the set of solutions ˜ p to the constrained optimization problem Eq.
11 are equivalent to the set of solutions to the following relaxed objective with Lagrange multiplier λ = 1 :
[Equation]
This step does not require that the policy π ( a t | s t ) or the dynamics p ( s t + 1 | s t , a [Equation] ) be convex.
Our next step is to show that the constraint does not affect the solution of this optimization problem.
For any function ∆ r ( s t + 1 , s t , a t ) , we can add a constant c ( s t , a t ) and obtain the same objective value but now satisfy the constraint.
We construct c ( s t , a t ) as
[Equation]
First, we observe that adding c ( s t , a t ) to ∆ r does not change the objective:
[Equation]
Second, we observe that the new reward function ∆ r ( s t + 1 , s t , a t )
+ c ( s t , a t ) satisﬁes the constraint in Eq.
[Equation]
Thus, constraining ∆ r to represent a probability distribution does not affect the solution (value) to the optimization problem, so we can ignore this constraint without loss of generality.
The new, unconstrained objective is
[Equation]
Step 3:
The penalty is the Fenchel dual of action entropy.
We deﬁne ρ πt ( s t , a t , s t + 1 ) as the marginal distribution of transitions visited by policy π at time t .
We now apply Lemma A.2 to Eq. 16:
[Equation]
Summary.
We have thus shown the follow:
[Equation]
Taking the exponential transform of both sides, we obtain the desired result.
A. R OBUSTNESS TO B OTH R EWARDS AND D YNAMICS
While Theorem 4.2 is phrased in terms of perturbations to the dynamics function, not the reward function, we now discuss how this result can be used to show that MaxEnt RL is simultaneously robust to perturbations in the dynamics and the reward function.
Deﬁne a modiﬁed MDP where the reward is appended to the observation.
Then the reward function is the last coordinate of the observation.
In this scenario, robustness to dynamics is equivalent to robustness to rewards.
A. H OW B IG IS THE R OBUST S ET ( (cid:15) )?
Our proof of Theorem 4.2 used duality to argue that there exists an (cid:15) for which MaxEnt RL maximizes a lower bound on the robust RL objective.
However, we did not specify the size of this (cid:15) .
This raises the concern that (cid:15) might be arbitrarily small, even zero, in which case the result would be vacuous.
In this section we provide a proof of Lemma 4.3, which provides a lower bound on the size of the robust set.
Proof.
Our proof proceeds in three steps.
We ﬁrst argue that, at optimality, the constraint on the adversarial dynamics is tight.
This will allow us to treat the constraint as an equality constraint, rather than an inequality constraint.
Second, since this constraint holds with equality, then we can rearrange the constraint and solve for (cid:15) in terms of the optimal adversarial dynamics.
The third step is to simplify the expression for (cid:15) .
Step 1: The constraint on ∆ r holds with equality.
Our aim here is to show that the constraint on ∆ r in Eq.
12 holds with equality for the optimal ∆ r (i.e., that which optimizes Eq. 11).
The objective in Eq.
11 is linear in ∆ r , so there must exist optimal ∆ r at the boundary of the constraint (Rockafellar, 1970, Chapter 32).
Step 2: Solving for (cid:15) .
Since the solution to the constrained optimization problem in Eq.
11 occurs at the boundary, the constraint in Eq.
12 holds with equality, immediately telling us the value of (cid:15) :
[Equation]
where ∆ r is the solution to Eq.
This identity holds for all states s t .
The second line above expresses the expectation over trajectories as an expectation over states, which will simplify the analysis in the rest of this proof.
The factor of T is introduced because we have removed the inner summation.
Step 3:
Simplifying the expression for (cid:15) .
To better understand this value of (cid:15) , we recall that the following identity (Lemma A.2) holds for this optimal ∆ r :
[Equation]
We next take the log( · ) of both sides and rearrange terms:
[Equation]
Next, we substitute the deﬁnition of ∆ r (Eq. 10) and factor log ρ ( a t , s t + 1 | s t ) : log (cid:90)(cid:90)
[Equation]
Substituting this expression into Eq.
[Equation]
we obtain a more intuitive expression for (cid:15) :
[Equation]
The last line follows from our assumption that the state space is discrete, so the entropy H ˜ p [ s t + 1 | s t , a t ] is non-negative.
This same result will hold in environments with continuous state spaces as long as the (differential) entropy H ˜
p
[ s t + 1 | s t , a t ] is non-negative.
A. W ORKED E XAMPLE OF D YNAMICS R OBUSTNESS
We calculate the penalty using a Gaussian integral:
[Equation]
We make two observations about this example.
First, the penalty would be inﬁnite if the adversary dynamics ˜
p had the same variance as the true dynamics, p : the adversary must choose dynamics with higher variance.
Second, the penalty depends on the size of the state space.
More precisely, the penalty depends on the region over which the adversary applies their perturbation.
The adversary can incur a smaller penalty by applying the perturbation over a smaller range of states.
A. T EMPERATURES
Many algorithms for MaxEnt RL (Haarnoja et al., 2018a;
Fox et al., 2016; Nachum et al., 2017) include a temperature α > 0 to balance the reward and entropy terms:
[Equation]
We can gain some intuition into the effect of this temperature on the set of reward functions to which we are robust.
In particular, including a temperature α results in the following robust set:
[Equation] [Equation]
In the second line, we simply moved the temperature from the objective to the constraint by redeﬁning u s t ( a t ) → 1 α u s t ( a t ) .
We visualize the effect of the temperature in Fig.
First, we ﬁx a reward function r , and plot the robust set R αr for varying values of α .
Fig.
9 (left) shows the somewhat surprising result that increasing the temperature (i.e., putting more weight on the entropy term) makes the policy less robust.
In fact, the robust set for higher temperatures is a strict subset of the robust set for lower temperatures:
[Equation]
This statement can be proven by simply noting that the function e − xα is an increasing function of α in Equation 19.
It is important to recognize that being robust against more reward functions is not always desirable.
In many cases, to be robust to everything, an optimal policy must do nothing.
We now analyze the temperature in terms of the converse question: if a reward function r (cid:48) is included in a robust set, what other reward functions are included in that robust set?
To do this, we take a reward function r (cid:48) , and ﬁnd robust sets R α r that include r (cid:48) , for varying values of α .
As shown in
Fig.
9 (right), if we must be robust to r (cid:48) and use a high temperature, the only other reward functions to which we are robust are those that are similar, or pointwise weakly better, than r (cid:48) .
In contrast, when using a small temperature, we are robust against a wide range of reward functions, including those that are highly dissimilar from our original reward function (i.e., have higher reward for some actions, lower reward for other actions).
Intuitively, increasing the temperature allows us to simultaneously be robust to a larger set of reward functions.
A. M AX E NT S OLVES R OBUST C ONTROL FOR R EWARDS
In Sec. 4.1, we showed that MaxEnt RL is equivalent to some robust-reward problem.
The aim of this section is to go backwards: given a set of reward functions, can we formulate a MaxEnt RL problem such that the robust-reward problem and the MaxEnt RL problem have the same solution?
Lemma A.3.
For any collection of reward functions R , there exists another reward function r such that the MaxEnt RL policy w.r.t.
r is an optimal robust-reward policy for R :
[Equation]
We use set containment, rather than equality, because there may be multiple solutions to the robustreward control problem.
Proof.
Let π ∗ be a solution to the robust-reward control problem:
[Equation]
Deﬁne the MaxEnt RL reward function as follows:
[Equation]
Substituting this reward function in Equation 1, we see that the unique solution is π = π ∗ .
Intuitively, this theorem states that we can use MaxEnt RL to solve any robust-reward control problem that requires robustness with respect to any arbitrary set of rewards, if we can ﬁnd the right corresponding reward function r for MaxEnt RL.
One way of viewing this theorem is as providing an avenue to sidestep the challenges of robust-reward optimization.
Unfortunately, we will still have to perform robust optimization to learn this magical reward function, but at least the cost of robust optimization might be amortized.
In some sense, this result is similar to Ilyas et al.
A. F INDING THE R OBUST R EWARD F UNCTION
In the previous section, we showed that a policy robust against any set of reward functions R can be obtained by solving a MaxEnt RL problem.
However, this requires calculating a reward function r ∗ for MaxEnt RL, which is not in general an element in R .
In this section, we aim to ﬁnd the MaxEnt reward function that results in the optimal policy for the robust-reward control problem.
Our main idea is to ﬁnd a reward function r ∗ such that its robust set, R r ∗ , contains the set of reward functions we want to be robust against, R .
That is, for each r i ∈ R , we want r i ( s t , a t ) =
r ∗ ( s t , a t ) + u s t ( a t ) for some u s t ( a t ) satisfying
[Equation]
Replacing u with r (cid:48) − r ∗ , we see that the MaxEnt reward function r must satisfy the following constraints:
[Equation]
We deﬁne R ∗ ( R ) as the set of reward functions satisfying this constraint w.r.t.
reward functions in R :
[Equation]
Note that we can satisfy the constraint by making r ∗ arbitrarily negative, so the set R ∗ ( R ) is nonempty.
We now argue that all any applying MaxEnt RL to any reward function in r ∗ ∈ R ∗ ( R ) lower bounds the robust-reward control objective.
Lemma A.4.
Let a set of reward functions R be given, and let r ∗ ∈ R ∗ ( R ) be an arbitrary reward function belonging to the feasible set of MaxEnt reward functions.
Then
[Equation]
While we know that MaxEnt RL can be used to solve arbitrary robust-reward control problems exactly, doing so requires that we already know the optimal policy (§ A.9).
Instead, we use the approach outlined in Sec.
A.10, which allows us to approximately solve an arbitrary robust-reward control problem without knowing the solution apriori.
This approach (“LowerBound + MaxEnt”) achieves near-optimal minimax reward.
Note that this bound holds for all feasible reward functions and all policies, so it also holds for the maximum r ∗ :
[Equation]
Deﬁning π ∗ = arg max π J MaxEnt ( π, r ∗ ) , we get the following inequality:
[Equation]
Thus, we can ﬁnd the tightest lower bound by ﬁnding the policy π and feasibly reward r ∗ that maximize Equation 20:
[Equation]
[Table]
It is useful to note that the constraints are simply L OG S UM E XP functions, which are convex.
For continuous action spaces, we might approximate the constraint via sampling.
Given a particular policy, the optimization problem w.r.t.
r has a linear objective and convex constraint, so it can be solved extremely quickly using a convex optimization toolbox.
Moreover, note that the problem can be solved independently for every state.
The optimization problem is not necessarily convex in π .
A. A NOTHER C OMPUTATIONAL E XPERIMENT
This section presents an experiment to study the approach outlined above.
Of particular interest is whether the lower bound (Eq 20) comes close to the optimal minimax policy.
We will solve robust-reward control problems on 5-armed bandits, where the robust set is a collection of 5 reward functions, each is drawn from a zero-mean, unit-variance Gaussian.
For each reward function, we add a constant to all of the rewards to make them all positive.
Doing so guarantees that the optimal minimax reward is positive.
Since different bandit problems have different optimal minimax rewards, we will normalize the minimax reward so the maximum possible value is 1.
Our approach, which we refer to as “LowerBound + MaxEnt”, solves the optimization problem in
Equation 21 by alternating between (1) solving a convex optimization problem to ﬁnd the optimal reward function, and (2) computing the optimal MaxEnt RL policy for this reward function.
Step 1 is done using CVXPY, while step 2 is done by exponentiating the reward function, and normalizing it to sum to one.
Note that this approach is actually solving a harder problem: it is solving the robust-reward control problem for a much larger set of reward functions that contains the original set of reward functions.
Because this approach is solving a more challenging problem, we do not expect that it will achieve the optimal minimax reward.
However, we emphasize that this approach may be easier to implement than ﬁctitious play, which we compare against.
Different from experiments in Section 4.1, the “LowerBound + MaxEnt” approach assumes access to the full reward function, not just the rewards for the actions taken.
For fair comparison, ﬁctitious play will also use a policy player that has access to the reward function.
Fictitious play is guaranteed to converge to the optimal minimax policy, so we assume that the minimax reward it converges to is optimal.
We compare against two baselines.
The “pointwise minimum policy” ﬁnds the optimal policy for a new reward function formed by taking the pointwise minimum of all reward functions:
˜ r ( a t ) =
min r ∈ R r ( a t ) .
This strategy is quite simple and intuitive.
The other baseline is a “uniform policy” that chooses actions uniformly at random.
We ran each method on the same set of 10 robust-reward control bandit problems.
In Fig.
10, we plot the (normalized) minimax reward obtained by each method on each problem, as well as the average performance across all 10 problems.
The “LowerBound + MaxEnt” approach converges to a normalized minimax reward of 0.91, close to the optimal value of 1.
In contrast, the “pointwise minimum policy” and the “uniform policy” perform poorly, obtaining normalized minimax rewards of 0.56 and 0.60, respectively.
In summary, while the method proposed for converting robustreward control problems to MaxEnt RL problems does not converge to the optimal minimax policy, empirically it performs well.
B E XPERIMENTAL D ETAILS B. D YNAMICS R OBUSTNESS E XPERIMENTS (F IG . 4)
We trained the MaxEnt RL policy using the SAC implementation from TF Agents (Guadarrama et al., 2018) with most of the default parameters (unless noted below).
Fig.
4a
We used the standard Pusher-v2 task from OpenAI Gym (Brockman et al., 2016).
We used a ﬁxed entropy coefﬁcient of 1 e − 2 for the MaxEnt RL results.
For the standard RL results, we used the exact same codebase to avoid introducing any confounding factors, simply setting the entropy coefﬁcient to a very small value 1 e − 5 .
The obstacle is a series of three axis-aligned blocks with width 3cm, centered at (0.32, -0.2), (0.35, -0.23), and (0.38, -0.26).
We chose these positions to be roughly along the perpendicular bisector of the line between the puck’s initial position and the goal.
We used 100 episodes of length 100 for evaluating each method.
To decrease the variance in the results, we ﬁxed the initial state of each episode:
[Equation]
environment Metaworld (Yu al., 2020) using a maximum episode length of 151.
For this experiment, we perturbed the environment by modifying the observations such that the button appeared to be offset along the Y axis.
We recorded the average performance over 10 episodes.
For this environment we used an entropy coefﬁcient of 1 e 1 for MaxEnt
Fig.
6 We used the standard Pusher-v2 task from OpenAI Gym (Brockman et al., 2016).
We modiﬁed the environment to perturb the XY position of the puck at time t = 20 .
We randomly sampled an angle θ ∼ Unif [0 , 2 π ] and displaced the puck in that direction by an amount given by the disturbance size.
For evaluating the average reward of each policy on each disturbance size, we used 100 rollouts of length 151.
Fig.
5 We used a modiﬁed version of the 2D navigation task from Eysenbach et al.
with the following reward function:
[Equation]
Episodes were 48 steps long
Fig.
7 We used the peg insertion environment from Eysenbach et al.
Episodes were at most 200 steps long, but terminated as soon as the peg was in the hole.
The agent received a reward of +100 once the peg was in the hole, in addition to the reward shaping terms described in Eysenbach et al.
[Figure]
[Equation]
Following Haarnoja et al.
(2018a)
, we use an entropy coefﬁcient of α = 1 / 5 .
In Fig.
8, the thick line is the average over ﬁve random seeds (thin lines).
Fig.
11 shows the evaluation of all methods on both the expected reward and the minimax reward objectives.
Note that the minimax reward can be computed analytically as
[Equation]
The mean for arm i , µ i , is drawn from a zero-mean, unit-variance Gaussian distribution, µ i ∼ N (0 , 1) .
When the agent pulls arm i , it observes a noisy reward r i ∼ N ( µ i , 1) .
The thick line is the average over 10 random seeds (thin lines).
C R OBUST RL A BLATION E XPERIMENTS
We ran additional ablation experiments to study whether simple modiﬁcations to the robust RL baseline from Section 5 (Tessler et al., 2019) would improve results.
These experiments will help discern whether the good performance of MaxEnt RL, relative to the PR-MDP and NR-MDP baselines, comes from using a more recent RL algorithm (SAC instead of DDPG), or from the entropy regularization.
We used the following ablations:
larger network : We increased the width of all neural networks to 256 units, so that the network size the exactly the same as for the MaxEnt RL method.
dual critic :
Following TD3 (Fujimoto et al., 2018), we added a second Q function and used the minimum over two (target) Q functions.
This change affects not only the actor and critic updates, but also the updates to the adversary learned by PR/NR-MDP.
more exploration : We increased the exploration noise from 0.2 to 1.0.
We implemented these ablations by modifying the open source code released by Tessler et al.
The results, shown in Fig.
12, show that these changes do not signiﬁcantly improve the performance of the NR-MDP or PR-MDP.
Perhaps the one exception is on the Hopper-v2 task, where PR-MDP with the larger networks is now the best method for large relative masses.
The dual critic ablation generally performs worse than the baseline.
We hypothesize that this poor performance is caused by using the (pessimistic) minimum over two Q functions for updating the adversary, which may result in a weaker adversary.
This experiment suggests that incorporating the dual critic trick into the action robustness framework may require some non-trivial design decisions.
[Figure]