Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrations
s)
AffiliationAddress email
Abstract
Providing densely shaped reward functions for RL algorithms is often exceedinglychallenging, motivating the development of RL algorithms that can learn fromeasier-to-specify sparse reward functions.
This sparsity poses new explorationchallenges; one common response is to use demonstrations to provide initial signalabout regions of the state space with high rewards.
However, prior RL from demonstrations algorithms introduce significant complexity and many hyperparameters,making them hard to implement and tune.
We introduce Monte Carlo augmented
Actor-Critic (MCAC), a parameter free modification to standard actor-critic algorithms which initializes the replay buffer with demonstrations and computes amodified Q -value by taking the maximum of the standard temporal distance (TD)target and a Monte Carlo estimate of the reward-to-go.
This encourages explorationin the neighborhood of high-performing trajectories by encouraging high Q -valuesin corresponding regions of the state space.
Experiments across 5 continuouscontrol domains suggest that MCAC can be used to significantly increase learningefficiency for a number of prior RL and RL-from-demonstrations algorithms.
Introduction
Reinforcement learning has been successful in learning complex skills in many environments [Mnihet al., 2015, Silver et al., 2016, Schulman et al., 2017], but providing dense, informative rewardfunctions for RL agents is often very challenging [Xie et al., 2019, Krakovna et al., 2020, Wu et al.
This is particularly challenging for high-dimensional control tasks, in which there may bea large number of factors that influence the agent’s objective.
In many settings, it may be mucheasier to provide sparse reward signals that simply convey high-level information about task progress,such as whether an agent has completed a task or has violated a constraint.
However, optimizing RLpolicies given such reward signals can be exceedingly challenging, as sparse reward functions maynot be able to meaningfully distinguish between a wide range of different policies.
This issue can be mitigated by leveraging demonstrations, which provide initial signal about desiredbehaviors.
Though demonstrations may often be suboptimal in practice, they should still serve toencourage exploration in promising regions of the state space, while allowing the agent to explorebehaviors which may outperform the demonstrations.
Prior work has considered a number of waysto leverage demonstrations to improve learning efficiency for reinforcement learning, including byinitializing the policy to match the behavior of the demonstrator [Rajeswaran et al., 2018, Peng et al.
,2019], using demonstrations to explicitly constrain agent exploration [Thananjeyan et al., 2020, 2021,
Wilcox et al., 2021], and introducing auxiliary losses to incorporate demonstration data into policyupdates [Nair et al., 2018, Hester et al., 2018, Gao et al., 2018].
While these algorithms have shown impressive performance in improving the sample efficiency of RL algorithms, they add significantcomplexity and hyperparameters, making them difficult to implement and tune for different tasks.
We present Monte Carlo augmented Actor-Critic (MCAC), which introduces an easy-to-implement,but highly effective, change that can be readily applied to existing actor-critic algorithms without theintroduction of any additional hyperparameters and only minimal additional complexity.
The idea isto encourage initial optimism in the neighborhood of successful trajectories, and progressively reducethis optimism during learning so that it can continue to explore new behaviors.
To operationalizethis idea, MCAC introduces two modifications to existing actor-critic algorithms.
First, MCACinitializes the replay buffer with task demonstrations.
Second, MCAC computes a modified target
Q -value for critic updates by taking the maximum of the standard temporal distance targets used inexisting actor critic algorithms and a Monte Carlo estimate of the reward-to-go.
The intuition is that
Monte Carlo value estimates can more effectively capture longer-term reward information, making itpossible to rapidly propagate reward information from demonstrations through the learned Q -function.
This makes it possible to prevent underestimation of values in high-performing trajectories early inlearning, as high rewards obtained later in a trajectory may be difficult to initially propagate backto earlier states with purely temporal distance targets [Wright et al., 2013].
Experiments on fivecontinuous control domains suggest that MCAC is able to substantially accelerate exploration forboth standard RL algorithms and recent RL from demonstrations algorithms in sparse reward tasks.
Related Work
2.1 Reinforcement Learning from Demonstrations
One standard approach for using demonstrations for RL first uses imitation learning [Argall et al.
,2009] to pre-train a policy, and then fine-tunes this policy with on-policy reinforcement learningalgorithms [Schaal et al., 1997, Kober and Peters, 2014, Peng et al., 2019, Rajeswaran et al., 2018].
However, initializing with suboptimal demonstrations can hinder learning and using demonstrationsto initialize only a policy is inefficient, since they can also be used for Q -value estimation.
Other approaches leverage demonstrations to explicitly constrain agent exploration.
Thananjeyan et al.[2020] and Wilcox et al.
[2021] propose model-based RL approaches that uses suboptimal demonstrations to iteratively improve performance by ensuring consistent task completion during learning.
Similarly, Jing et al.
[2020] also uses suboptimal demonstrations to formulate a soft-constraint onexploration.
However, a challenge with these approaches is that they introduce substantial algorithm complexity, making it difficult to tune and utilize these algorithms in practice.
For example,while Thananjeyan et al.
[2020] does enable iterative improvement upon suboptimal demonstrations,it requires learning a model of system dynamics and a density estimator to capture the support ofsuccessful trajectories making it challenging to scale to high-dimensional observations.
Finally, many methods introduce auxiliary losses to incorporate demonstrations into policy updates [Kim et al., 2013, Gao et al., 2018, Kang et al., 2018].
Deep Deterministic Policy Gradientsfrom Demonstrations (DDPGfD)
[Vecerik et al., 2018] maintains all the demonstrations in a separatereplay buffer and uses prioritized replay to allow reward information to propagate more efficiently.
Nair et al.
[2018], and Hester et al.
[2018] use similar approaches, where demonstrations are maintained separately from the standard replay buffer and additional policy losses encourage imitating thebehavior in the demonstrations.
Meanwhile, RL algorithms such as AWAC [Nair et al., 2021] pretrainusing demonstration data to constrain the distribution of actions selected during online exploration.
While these methods often work well in practice, they often increase algorithmic complexity andintroduce several additional hyperparameters that are difficult and time consuming to tune.
Bycontrast, MCAC does not increase algorithmic complexity, is parameter-free, and can easily bewrapped around any existing actor-critic algorithm.
2.2 Improving Q -Value Estimates
The core contribution of this work is an easy-to-implement, yet highly effective, method for stabilizingactor-critic methods for sparse reward tasks using demonstrations and an augmented Q -value target.
There has been substantial literature investigating learning stability challenges in off-policy deep
Q -learning and actor-critic algorithms.
See Van Hasselt et al.
[2018] for a more thorough treatment of the learning stability challenges introduced by combining function approximation, bootstrapping,and off-policy learning, as well as prior work focused on mitigating these issues.
One class of approaches focuses on developing new ways to compute target Q -values [Konidariset al., 2011, Wright et al., 2013, Van Hasselt et al., 2016, Schulman et al., 2016].
Van Hasselt et al.[2016] computes target Q -values with two Q -networks, using one to select actions and the other tomeasure the value of selected actions, which helps to prevent the Q -value over-estimation commonlyobserved in practice in many practical applications of Q -learning.
TD3 [Fujimoto et al.
2018]attempts to address overestimation errors by taking the minimum of two separate Q -value estimates,but this can result in underestimation of the true Q -value target.
Kuznetsov et al.
[2020] uses anensemble of critics to adaptively address estimation errors in Q -value targets, but introduces a numberof hyperparameters which must be tuned separately for different tasks.
Bellemare et al.
[2016]and Ostrovski et al.
[2017] consider a linear combination of a TD-1 target and Monte Carlo target.
Wright et al.
[2015], Schulman et al.
[2016] and Sharma et al.
[2018] consider a number of differentestimators of a policy’s value via n -step returns, which compute Q -targets using trajectories with ncontiguous transitions followed by a terminal evaluation of the Q -value after n steps.
Each of thesetargets make different bias and variance tradeoffs that can affect learning dynamics.
Similar to our work, Wright et al.
[2013] explore the idea of taking a maximum over a bootstrappedtarget Q -value (TD-1 target) and a Monte Carlo estimate of the return-to-go to improve fitted Qiteration.
However, Wright et al.
[2013] focuses only on fully offline Q -learning and only considerssimple low-dimensional control tasks using Q -learning with linear value approximation.
There arenumerous reasons to extend these ideas to online deep RL.
First, deep RL algorithms are oftenunstable, and using the ideas in Wright et al.
[2013] to improve Q estimates is a promising way toalleviate this, as we empirically verify.
Second, while offline learning has many important applications,online RL is far more widely studied, and we believe it is useful to study the effects these ideashave in this setting.
To the best of our knowledge, MCAC is the first application of these ideas toonline actor-critic algorithms with deep function approximation, and we find that it yields surprisingimprovements in RL performance on complex high-dimensional continuous control tasks.
Problem Statement
We consider a Markov Decision Process (MDP) described by a tuple ( S , A , p, r, γ, T ) with a stateset S , an action set A , a transition probability function p : S ×
A × S → [0 , 1] , a reward functionr : S ×
A → R , a discount factor γ , and finite time horizon T .
In each state s t ∈ S the agent choosesan action a t ∈ A and observes the next state s t +1 ∼ p ( ·| s t , a t ) and a reward r ( s t , a t ) ∈ R .
Theagent acts according to a policy π , which induces a probability distribution over A given the state,π ( a t | s t ) .
The agent’s goal is to find the policy π ∗ which at any given s t ∈ S maximizes the expecteddiscounted sum of rewards,
[Equation]
where τ = ( s 0 , a 0 , s 1 , a 1 , . .
s T ) and τ ∼ π indicates the distribution of trajectories induced byevaluating policy π in the MDP.
In this work, we make additional assumptions specific to the class of problems we study.
First,we assume that all transitions in the replay buffer are elements of complete trajectories; this is areasonable assumption as long as all transitions are collected from rolling out some policy in the MDP.
Second, we assume the agent has access to an offline dataset D offline of (possibly suboptimal) taskdemonstrations.
Finally, we focus on settings where the reward function is sparse in the sense thatmost state transitions do not induce a change in reward, causing significant exploration challenges forreinforcement learning algorithms.
Preliminaries: Actor-Critic Algorithms
For a given policy π , its state-action value function Q π is defined as
[Equation] [Equation]
Actor-critic algorithms learn a sample-based approximation to Q π , denoted Q πθ , and a policy π ϕwhich selects actions to maximize Q πθ , with a function approximator (typically a neural network)parameterized by θ and ϕ respectively.
During the learning process, they alternate between regressing
Q πθ to predict Q π and optimizing π ϕ to select actions with high values under Q πθ .
Exactly computing Q π targets to train Q πθ is typically intractable for arbitrary policies in continuous
MDPs, motivating other methods for estimating them.
One such method is to simply collect trajectories ( s t , a t , r t , s t +1 , . . .
s T − 1 , a T − 1 , r T − 1 , s T ) by executing the learned policy π ϕ from state s t ,computing a Monte Carlo estimate of the reward-to-go defined as follows:
[Equation]
and fitting Q
πθ to these targets.
However, Q targetMC can be a high variance estimator of the reward-to-go
[Schulman et al., 2016, Suttonand Barto, 2018], motivating the popular one-step temporal difference target (TD-1 target) to helpstabilize learning:
[Equation]
where a t +1 ∼ π ϕ ( s t +1 ) , which is recursively defined based on only a single ( s t , a t , s t +1 , r t ) transition.
Here θ ′ gives the parameters of a lagged target network as in [Van Hasselt et al., 2016].
Therehas also been recent interest in methods which computing TD-n targets, which instead sum up rewardsfor the next n steps and then incorporate Q -values from step n +
[Wright et al., 2015, Schulmanet al., 2016, Sharma et al., 2018].
Monte Carlo augmented Actor-Critic
We discuss the intuition behind MCAC in Section 5.1 and then describe how MCAC is implementedin practice in Section 5.
5.1 MCAC Algorithm
The objective of MCAC is to efficiently convey information about sparse rewards from suboptimaldemonstrations to Q πθ in order to accelerate policy learning while still maintaining learning stability.
To do this, MCAC combines two different methods of computing targets for fitting Q -functions toenable efficient value propagation throughout the state-action space while also learning a Q -valueestimator with low enough variance for stable learning.
To operationalize this idea, MCAC defines anew Q -function target for training Q πθ by taking the maximum of the Monte Carlo Q -target (eq 4.2)and the temporal difference Q -target (eq 4.3): max (cid:2) Q targetTD ( s t , a t ) , Q target MC ( s t , a t ) (cid:3) .
The idea here is that early in learning, a Q -function trained only with temporal difference targetswill have very low values throughout the state-action space as it may be very difficult to propagateinformation about delayed rewards through the temporal difference target for long-horizon tasks withsparse rewards [Van Hasselt et al., 2018].
On the other hand, the Monte Carlo Q -target can easilycapture long-term rewards, but often dramatically underestimate Q -values for poorly performingtrajectories [Wright et al., 2013].
Thus, taking a maximum over these two targets serves to initiallyboost the Q -values for transitions near high performing trajectories while limiting the influence ofunderestimates from the Monte Carlo estimate.
MCAC can also be viewed as a convenient way to balance the bias and variance properties of the
Monte Carlo and temporal difference Q -targets.
The Monte Carlo Q -target is well known to bean unbiased estimator of policy return, but have high variance [Wright et al., 2013].
Conversely,temporal difference targets are known to be biased, but have much lower variance
[Wright et al.
,2013, Van Hasselt et al., 2016].
Thus, as the temporal difference target is typically negatively biased(an underestimate of the true policy return) on successful trajectories early in learning due to thechallenge of effective value propagation, computing the maximum of the temporal difference and
Monte Carlo targets helps push the MCAC target value closer to the true policy return.
Conversely,the Monte Carlo targets are typically pessimistic on unsuccessful trajectories, and their high variancemakes it difficult for them to generalize sufficiently to distinguish between unsuccessful trajectoriesthat are close to being successful and those that are not.
Thus, computing the maximum of the
Require:
Offline dataset D offline .
Require: Total training episodes N , batch size M , Pretraining Steps N p
Require:
Episode time horizon T .
1: Initialize replay buffer R :
= D offline .
2: Initialize agent π ϕ and critic Q
πθ using data from D offline .
3: for i ∈ { 1 , . . .
, N p } do 4:
Sample B ⊊ R such that |B| =
M .
Optimize Q πθ on B to minimize loss in eq (5.4).
Optimize policy π ϕ to maximize Q θ .
6: end for 7: for i ∈ { 1 , . . .
, N } do 8:
Initialize episode buffer E = {} .
Observe state s i 1 .
10: for j ∈ { 1 , . . .
, T } do 11:
Sample and execute a it ∼ π θ ( s ij ) , observing s ij +1 , r ij .
12: τ ij ← ( s ij , a ij , s ij +1 , r ij ) 13:
E ← E ∪ { τ ij } .
Sample B ⊊ R such that |B| =
M .
Optimize Q πθ on B to minimize loss in eq (5.4).
Optimize policy π ϕ to maximize Q θ .
16: end for 17: for τ ij ∈ E do 18:
Compute Q targetMC- ∞ ( s ij , a ij ) as in eq (5.2).
19: τ ij ← ( s ij , a ij , s ij +1 , r ij , Q targetMC- ∞ ( s ij , a ij )) .
20: end for 21:
R ← R ∪ E .
22: end for temporal difference and Monte Carlo targets also helps to prevent excessive pessimism in evaluatingunsuccessful trajectories.
Notably, MCAC does not constrain its Q -targets explicitly based on thetransitions in the demonstrations, making it possible for the policy to discover higher performingbehaviors than those in the demonstrations as demonstrated in Section 6.
5.2 MCAC Practical Implementation
MCAC can be implemented as a wrapper around any actor-critic RL algorithm; we consider 5 optionsin experiments (Section 6).
MCAC starts with an offline dataset of suboptimal demonstrations D offline ,which are used to initialize a replay buffer R .
Then, during each episode i , we collect a full trajectoryτ i , where the j th transition ( s ij , a ij , s ij +1 , r ij ) in τ i is denoted by τ ij .
Next, consider any actor-critic method using a learned Q function approximator Q θ ( s t , a t ) that, for agiven transition τ ij = ( s ij , a ij , s ij +1 , r ij ) ∈ τ i ⊊ R , would be updated by minimizing a loss,
[Equation]
where ℓ is an arbitrary differentiable loss function and Q target is the target value for regressing Q θ .
We note that Q target is defined by the choice of actor-critic method.
To implement MCAC, we firstcalibrate with temporal difference targets (which provide infinite-horizon Q estimates) by computingthe infinite horizon analogue of the Monte Carlo target defined in Equation 4.2, which assumes thelast observed reward value will repeat forever and uses this to add an infinite sum of discountedrewards, and is given by
[Equation]
Then, we simply replace the target with a maximum over the original target and the Monte Carlotarget defined in Equation 5.2, given by
[Equation]
[Figure]
This results in the following loss function for training Q θ :
[Equation]
The full training procedure alternates between updating Q πθ using the method described above,followed by optimizing the policy
π ϕ using any standard policy update method.
The full MCACalgorithm is described in Algorithm 1.
Experiments
In the following experiments we study (1) whether MCAC enables more efficient learning when builton top of standard actor-critic RL algorithms and (2) whether MCAC can be applied to improve prioralgorithms for RL from demonstrations.
See the supplementary material for code and instructions onhow to run experiments for reproducing all results in the paper and additional experiments studyingthe impact of demonstration quantity, quality, and other algorithmic choices such as whether topretrain learned networks on demonstration data before online interaction.
6.1 Experimental Procedure
All experiments were run on a set of 24 Tesla V100 GPUs through a combination of Google Cloudresources and a dedicated lab server.
We aggregate statistics over 10 random seeds for all experiments,reporting the mean and standard error across the seeds with exponential smoothing.
Details onhyperparameters and implementation details are provided in the supplementary material.
6.2 Domains
We consider the five long-horizon continuous control tasks shown in Figure 1.
All tasks haverelatively sparse rewards, making demonstrations critical for performance.
We found that withoutdemonstrations, SAC and TD3 made little to no progress on these tasks.
Pointmass Navigation:
The first domain is a pointmass 2D navigation task (Figure 1(a))
withtime horizon T = 100 , where the objective is to navigate around the red barrier from start set S to agoal set G by executing 2D delta-position controls.
If the agent collides with the barrier it receives areward of − 100 and the episode terminates.
At each time step, the agent receives a reward of − 1 if itis not in the goal set and 0 if it is in the goal set.
To increase the difficulty of the task, we perturbthe state with zero-mean Gaussian noise at each timestep.
The combination of noisy transitions andsparse reward signal makes this a very difficult exploration task where the agent must learn to make itthrough the slit without converging to the local optima of avoiding both the barrier and the slit.
The demonstrator is implemented as a series of proportional controllers guiding it from the startingset to the slit, through the slit, and to the goal set.
The actions are clipped to fit in the action space,and trajectories are nearly optimal.
The agent is provided with 20 demonstrations.
Object Manipulation in MuJoCo
We next consider two object manipulation tasks designed inthe MuJoCo physics simulator [Todorov et al., 2012], where the objective is to extract a block from a tight configuration on a table (Block Extraction, Figure 1(b)) and push each of a set of 3 blocksforward on the plane (Sequential Pushing, Figure 1(c)).
In the Block Extraction task, the actionspace consists of 3D delta position controls and an extra action dimension to control the degree towhich the gripper is opened.
In the Sequential Pushing environment, this extra action dimension isomitted and the gripper is kept closed.
In the Block Extraction domain, the agent receives a rewardof − 1 for every timestep that it hasn’t retrieved the red block and 0 when it has.
In the Sequential
Pushing domain, the reward increases by 1 for each block the agent pushes forward.
Thus, the agentreceives a reward of − 3 when it has made no progress and 0 when it has completed the task.
The
Block Extraction task is adapted from Thananjeyan et al.
[2021] while the Sequential Pushing task isadapted from Wilcox et al.
We use a time horizon of T = 50 for the Block Extraction taskand a longer T = 150 for the Sequential Pushing task since it is of greater complexity.
The block extraction demonstrator is implemented as a series of proportional controllers guiding thearm to a position to grip the block, followed by an instruction to close the gripper and a controllerto lift.
We provide the agent with 50 demonstrations with zero-mean noise injected in the controlsto induce suboptimality.
For the sequential pushing environment, the demonstrator uses a series ofproportional controllers to slowly push one block forward, move backwards, line up with the nextblock, and repeat the motion until all blocks have been pushed.
Because it moves slowly and movesfar back from each block it pushes, demonstrations are very suboptimal.
For Sequential Pushing, theagent is provided with 500 demonstrations due to the increased difficulty of the task.
Robosuite Object Manipulation
Finally, we consider two object manipulation tasks built ontop of Robosuite [Zhu et al., 2020], a collection of robot simulation tasks using the MuJoCo physicsengine.
We consider the Door Opening task (Figure 1(d)) and the Block Lifting task (Figure 1(e)).
Inthe Door Opening task, a Panda robot with 7 DoF and a parallel-jaw gripper must turn the handle of adoor in order to open it.
The door’s location is randomized at the start of each episode.
The agentreceives a reward of -1 if it has not opened the door and a reward of 0 if it has.
In the Block Liftingtask, the same Panda robot is placed in front of a table with a single block on its surface.
The robotmust pick up the block and lift it above a certain threshold height.
The block’s location is randomizedfor each episode and the agent receives a reward of − 1 for every timestep it has not lifted the blockand a reward of 0 when it has.
Both Robosuite tasks use a time horizon of T = 50 .
For both Robosuite tasks, demonstrators are trained using SAC on a version of the task with ahand-designed dense reward function, as in the Robosuite benchmarking experiments [Zhu et al.
In order to ensure suboptimality, we stop training the demonstrator policy before convergence.
For each Robosuite environment we use the trained demonstrator policies to generate 100 suboptimaldemonstrations for training MCAC and the baselines.
6.3 Algorithm Comparisons
We empirically evaluate the following baselines both individually and in combination with MCAC.
Allmethods are provided with the same demonstrations which are collected as described in Section 6.2.
See Appendix ??
for more in depth details on implementation and training.
Behavior Cloning: Direct supervised learning on the offline suboptimal demonstrations.
Twin Delayed Deep Deterministic Policy Gradients (TD3) [Fujimoto et al., 2018]: State of the artactor-critic algorithm which trains a deterministic policy to maximize a learned critic.
Soft Actor-Critic (SAC) [Haarnoja et al., 2018]: State of the art actor-critic algorithm which trainsa stochastic policy which maximizes a combination of the Q value of the policy and the expectedentropy of the policy to encourage exploration.
Generalized Q Estimation (GQE): Version of the complex return estimation method from Schulmanet al.
[2016] for actor-critic methods, which computes a weighted average over TD- i estimates for arange of i .
GQE is implemented on top of SAC (see supplement for more details).
We tune the rangeof horizons considered and the eligibility trace value (corresponding to λ in GAE).
Overcoming Exploration from Demonstrations (OEFD)
[Nair et al., 2018]: OEFD builds on
DDPG [Lillicrap et al., 2015] by adding a loss which encourages imitating demonstrations and alearned filter which determines when to activate this loss.
Our implementation does include hindsightexperience replay since it is not applicable to most of our environments.
[Figure]
Advantage Weighted Actor-Critic (AWAC)
[Nair et al., 2021]:
A recent offline reinforcementlearning algorithm designed for fast online fine-tuning.
We also implement versions of each of the above RL algorithms with MCAC (TD3 + MCAC, SAC +
MCAC, GQE + MCAC, OEFD + MCAC, AWAC + MCAC).
The behavior cloning comparison serves to determine whether online learning is beneficial in general,while the other comparisons study whether MCAC can be used to accelerate reinforcement learningfor commonly used actor-critic algorithms (TD3, SAC, and GQE, which is essential SAC withcomplex returns) and also for recent algorithms for RL from demonstrations (OEFD, and AWAC).
6.4 Results
In Section 6.4.1 we study whether MCAC enables more efficient learning when built on top of
SAC and TD3, which are standard actor-critic RL algorithms widely used in the literature, and in
Section 6.4.2 we study whether MCAC can provide similar benefits when applied to recent RLfrom demonstration algorithms (OEFD and AWAC).
Additionally, in the supplement we provideexperiments involving other baselines, investigating the sensitivity of MCAC to the quality andquantity of demonstration data, and investigating its sensitivity to pretraining.
6.4.1 MCAC and Standard RL Algorithms
In Figure 2, we study the impact of augmenting SAC, TD3 and GQE with the MCAC target Q -function.
Note that all methods, both with and without MCAC, initialize their replay buffers with the sameset of demonstrations.
The results in Figure 2 suggest that MCAC is able to significantly acceleratelearning for both TD3 and SAC across all environments, and is able to converge to performance thatis either on-par or better than the demonstrations.
Furthermore, in the Pointmass Navigation and
Block Lifting task, SAC and TD3 are unable to make much task progress without MCAC.
MCACalso significantly accelerates learning for GQE for the Pointmass Navigation, Block Extraction, and
Sequential Pushing environments.
In the Door Opening and Block Lifting environments, MCACleaves performance largely unchanged since GQE already achieves very performance on par with thenext best algorithm even without MCAC.
[Figure]
6.4.2 MCAC and RL From Demonstrations Algorithms
In Figure 3, we study the impact of augmenting OEFD [Nair et al., 2018] and AWAC [Nair et al., 2021]with the MCAC target Q -function.
Results suggest that MCAC significantly improves the learningefficiency of OEFD on the Pointmass Navigation, Sequential Pushing, and Block Lifting tasks, butdoes not have a significant positive or negative affect on performance for the Block Extraction and
Door Opening tasks.
MCAC significantly improves the performance of AWAC on the Pointmass
Navigation and Sequential Pushing environments, stabilizing learning while the versions without
MCAC see performance fall off during online fine tuning.
On the other 3 environments where AWACis able to immediately converge to a stable policy after offline pre-training, MCAC has no significantnegative effect on its performance.
Discussion, Limitations, and Ethical Considerations
We present Monte Carlo augmented Actor-Critic (MCAC), a simple, yet highly effective, changethat can be applied to any actor-critic algorithm in order to improve reinforcement learning fromdemonstrations.
We present empirical results suggesting that MCAC often significantly improves,and does not significantly degrade, performance when applied to two state-of-the-art actor-critic RLalgorithms and two RL from demonstrations algorithms on five different continuous control domains.
Despite strong empirical performance, MCAC also has some limitations.
We found that whileencouraging higher Q -value estimates is beneficial for sparse reward tasks, when rewards are denseand richly informative, MCAC is not helpful and can even hinder learning by overestimating Q -values.
From an ethical standpoint, reinforcement learning algorithms such as MCAC can be used to automateaspects of a variety of interactive systems, such as robots, online retail systems, or social mediaplatforms.
However, poor performance when policies haven’t yet converged, or when hand-designedreward functions do not align with true human intentions, has the potential to cause significant harmto the humans using these systems.
Before deploying RL algorithms such as MCAC in real-worldsettings, it is critical to ensure that appropriate precautions are taken to minimize harm and damage.
Lastly, while MCAC shows convincing empirical performance, in future work it would be helpful toprovide theoretical analysis on its convergence properties and its effect on the bias-variance tradeoff.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] The Abstract and introduction claim that we introducea new reinforcement learning algorithm and experiments on a number of domainssuggesting that it accelerates learning for sparse reward tasks.
This is consistent withthe
algorithm
presented in Section 5 and the experiments in Figures 2 and 3 in Section 6(b)
Did you describe the limitations of your work?
[Yes] Yes, see Section [-
-]7(
c
) Did you discuss any potential negative societal impacts of your work?
[Yes] Yes, see
Section 7(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes] Yes, this work conforms to the ethics guidelines in the sense that it doesnot have direct negative societal impacts that we know of and is fully reproducible, withreleased code and results reported over 10 random seeds.
That being said, reinforcementlearning research in general, including our work, has possible ethical concerns relatedto misuse in systems which interact with humans, such as robots, online retail systems
and social media platforms.
Specifically, poorly specified reward functions can cause
unexpected consequences when algorithms discover behaviors different from what
the
human reward designer intended.
Suboptimal behaviors during policy optimization
also have potential to cause harm.
We discuss these concerns in detail in Section 7.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
There are no
theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A]
There are no theoreti- cal results.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] A zip file
containing all code is included in the supplementary material.
This code is associated
with a README that can be used to replicate all experiments.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] All training details are described at a high level in Section 6 andin detail in the supplementary material for all experiments.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All results in all plots are reported over 10 random seeds,with the mean displayed in the plot with standard error shading to indicate the marginof error over the 10 runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Yes, see the beginning of Section 6.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Yes, the sources of
all experimental domains used are cited in Section 6.2.
We also cite the sources of all
algorithms we build on in Section 6.3 and reference implementations used of these
algorithms in the supplement.
(b) Did you mention the license of the assets?
[N/A]
There were no explicit licencesassociated with the datasets/code we used, but we cited the papers they came from and
the Github repositories associated with them.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Yes, we have included code and datasets to reproduce all results in this paper in the
supplement.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] All data was collected in computational simulations, so
no
explicit human permission was required.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
No data collected was explicitly collected by
human users, so this was not necessary.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
There were no human subject studies.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
There were no human subject studies.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
There were no human subject studies.