You Only Live Once: Single-Life Reinforcement Learning via Learned Reward Shaping
Anonymous Author(s)
AffiliationAddress email
Abstract
Reinforcement learning algorithms are typically designed to learn a performantpolicy that can repeatedly and autonomously complete a task, typically startingfrom scratch.
However, many real-world situations operate under a different set ofassumptions: the goal might not be to learn a policy that can do the task repeatedly,but simply to perform a new task successfully once, ideally as quickly as possible,and while leveraging some prior knowledge or experience.
For example, imaginea robot that is exploring another planet, where it cannot get help or supervisionfrom humans.
If it needs to navigate to a crater that it has never seen before insearch of water, it does not really need to acquire a policy for reaching cratersreliably, it only needs to reach this particular crater once.
It must do so without thebenefit of episodic resets and tackle a new, unknown terrain, but it can leverageprior experience it acquired on Earth.
We formalize this problem setting, whichwe call single-life reinforcement learning (SLRL), where an agent must completea task once while contending with some form of novelty in a single trial withoutinterventions, given some prior data.
In this setting, we find that algorithmsdesigned for standard episodic reinforcement learning can struggle, as they havetrouble recovering from novel states especially when informative rewards arenot provided.
Motivated by this observation, we also propose an algorithm, Qweighted adversarial learning (Q WALE ), that addresses the dearth of supervisionby employing a distribution matching strategy that leverages the agent’s priorexperience as guidance in novel situations.
Our experiments on several singlelife continuous control problems indicate that methods based on our distributionmatching formulation are 20-60% more successful because they can more quicklyrecover from novel, out-of-distribution states.
Introduction
When building autonomous agents for the natural world, often the goal is not to learn a performantpolicy but rather to get something done, perhaps even suboptimally.
For example, an agent exploringon Mars looking for water will only need to complete its mission a single time.
As another example, a rescue robot will need to recover valuables from a particular burning building only once.
While the agent may have access to prior data about its task, a challenge arises from the fact the agent is inevitably going to have to contend with some form of novelty.
In the prior examples, the agent on Mars may have to contend with unknown terrain and environmental conditions, and the rescue agent may find certain paths in the building unpassable due to the fire.
We formalize this setting as the
SLRL) setting where the agent is evaluated on its ability to complete a in a single autonomously without episodic resets
Importantly the given online contains an aspect of novelty not present in the data although the task objective remains the same
The objective is to complete the given task as quickly as possible rather than learn a policy that can repeatedly complete the
We find that algorithms designed for episodic policy learning can struggle to complete single-lifetasks, even when initialized with prior data.
These algorithms empirically struggle to recover fromnovel states.
In episodic RL, the agent can rely on a reset to recover from an unfamiliar state.
Incontrast, in SLRL, the agent will inevitably fall off the distribution of prior data and must find its wayback to a good state distribution on its own.
We find that fine-tuning a pre-trained value function viaonline RL will not explicitly encourage the agent to get back on distribution.
We hypothesize thatbiasing exploration towards the known distribution represented in prior data and incentivizing theagent to stay there may be suboptimal from a policy learning perspective but may enable the agent toget the task done more quickly, which is what we care about in SLRL.
While shaped rewards mayhelp the agent find its way back and ultimately complete the desired task, the agent may often finditself in a sparse reward environment or with access to rewards that are not informative enough to beguided towards task completion.
Adversarial imitation learning (AIL) approaches such as GAIL ([22]) can potentially provide thedesired reward shaping via distribution matching.
However, using existing AIL methods naively maynot give the intended behavior in the SLRL setting due to two main shortcomings.
First, such methodsassume expert demonstrations are given as prior data, but in SLRL, we may be given suboptimaloffline prior data.
Second, AIL methods train the agent to match the entire distribution of priordata, which may be key to learning an optimal policy, but may be a drawback in our setting, as theagent might not be consistently guided towards task completion.
To address these shortcomings, wepropose a method in which different states in the prior data are weighted different amounts by theirestimated Q -value.
More concretely, we propose a Q -weighted AIL approach that incentivizes theagent to move towards states in the prior data with higher value than its current state, so that agentmay be guided consistently towards states closer to task completion.
Our contributions are as follows.
First, we formalize the SLRL problem setting, which we believe tobe a useful framework for modeling many situations in the real world.
We next provide an intuitiveargument and empirical analysis suggesting that learned reward shaping via distribution matching isbetter suited for this setting than finetuning without additional reward shaping.
We identify challengesthat uniquely arise in the SLRL setting with existing distribution matching approaches and proposea new approach, Q -weighted adversarial learning (Q WALE ), which is less sensitive to the qualityof prior data available and provides the agent with a shaped reward towards completing the desiredtask a single time.
Through our experiments, we explore the performance of different approaches in
SLRL.
We find that Q WALE can meaningfully guide the agent to explore the state distribution in itsprior data to complete the desired novel task 20-60% more successfully on four separate domainscompared to existing distribution matching approaches and RL fine-tuning.
Related Work
Autonomous RL .
In the context of deep RL, agents typically (but not always) are trained in episodicsetting and are evaluated on the quality of the learned policy.
Several recent works have developedalgorithms that can learn without episodic resets [19, 7, 67, 44, 46, 16, 17, 25, 45].
Like our work,such methods aim to make it possible to learn without any episodic resets, but are typically stillfocused on acquiring an effective policy that can perform the task repeatedly, typically by trainingsome auxiliary controller to enable the policy to “retry” the task multiple times without resets.
Incontrast, our aim is to develop an algorithm that can solve the task once, but as quickly as possible,which introduces a unique set of challenges as we discussed above.
Continual RL .
There is a rich literature on reinforcement learning in the continuing setting [30,43, 49, 64, 39, 29, 59] that considers maximizing the average reward accumulated over an infinitehorizon without episodic resets.
Such works often also consider regret minimization as the objective.
SLRL is closely related, and can be viewed as a special case where the agent has access to a prioroffline dataset and aims to solve a single task as quickly as possible in a new domain.
While the focusin continual learning is on general “lifelong” methods or on exploration, our focus is on effectivelyleveraging prior data in a setting that is meant to be reflective of real-world tasks (for example, inrobotics).
Leveraging offline data in online RL .
Learning expert policies given prior interaction data has beenextensively studied in imitation learning [2, 40, 14], inverse RL [34, 9, 69, 70], RL for sparse rewardsettings [4, 33, 38, 21, 53] and offline RL [58, 28, 27, 24, 60, 33].
Across all these diverse topics, thegoal is to learn a competent policy that can solve the task efficiently whereas the objective in thiswork is to complete the task in a single trial as quickly as possible.
To this end, we build on recentadversarial approaches to inverse RL [22, 12, 47, 52, 26, 68] to encourage agent’s state visitationtowards expert prior data where the agent is likely to be successful.
Prior methods have also studiedadversarial inverse RL and imitation learning with non-expert data [54, 48, 61, 55, 57, 56, 6, 3].
However, as we discuss in Section 5, these approaches need to be adapted for the SLRL setting tobe efficient at completing the task and to handle novelty, for example when the dynamics may havechanged.
Transfer and adaptation in RL .
Many prior works have studied the problem of adapting in presenceof shifts between train and test settings, often in a specific problem setting such as sim2real transfer [42, 51, 36, 31] or fast adaptation via meta-learning [10, 35, 32, 71, 11].
A common theme in theseworks is that the algorithm can often train in preparation for adaptation at test-time, thus affecting theprior experiences it may collect.
In contrast, the SLRL setting lays algorithmic emphasis on onlineexploration and adaptation, as the agent has access to fixed prior dataset of experiences.
Other transferlearning approaches adapt the weights of the policy to a new environment or task, either through rapidzero-shot adaptation [20, 65] or through extended episodic online training [23, 41, 8, 63, 62].
Unlikethe latter, we focus on adaptation within a single episode, but, unlike the former, with a focus onextended exploration and learning over tens of thousands of timesteps.
This problem setting leads tounique challenges, namely that the agent must autonomously recover from mistakes, hence requiringa distinct approach.
Preliminaries
In this section, we describe some preliminaries before formalizing our problem statement in thefollowing section.
We consider an agent that operates in a Markov decision process (MDP) consistingof the tuple M =
( S , A , P , R , ρ 0 , γ ) , where S is the state space, A is the agent’s action space, P ( s t +1 | s t , a t ) represents the environment’s transition dynamics, R : S
→ R indicates the rewardfunction, ρ 0 : S → R denotes the initial state distribution, and γ ∈ [0 , 1) denotes the discountfactor.
In typical reinforcement learning, the objective is find a policy π that maximizes J ( π ) =
119 E τ ∼ π
[ (cid:80) ∞ t =
γ t R ( s t )] .
Although our method is a reward-driven RL algorithm, we utilize concepts from imitation learningto overcome sparse rewards, utilizing potentially suboptimal prior data.
To this end, we build onadversarial imitation learning (AIL), which uses prior data D prior in the form of expert demonstrations(we will relax this requirement) to recover the expert’s policy.
One such method is GAIL [22], whichfinds a policy π θ that minimizes the Jensen-Shannon divergence between its stationary distributionand the expert data.
It does so by training a discriminator network D : S ×
A → (0 , 1) , alternatingupdates with updates to the policy π .
Concretely, D and π are learned by optimizing the following:
[Equation]
where D JS is the Jensen-Shannon divergence and ρ ∗ ( s, a ) is the optimal state-action occupancymeasure of the expert π E (represented by demonstrations).
In Section 6, we will see that AIL-stylediscriminator-based approaches can be adapted to the SLRL problem setting without demonstrationdata, and will therefore form the basis of our method.
Single-Life Reinforcement Learning
In this section, we formalize our problem setting, the single-life reinforcement learning (SLRL)problem.
The defining characteristic of SLRL is that the agent is given a single “life”, i.e. trial, tocomplete a desired task, with the trial ending when the task is completed.
The agent must completethe task autonomously, without access to any human interventions or resets.
In the real world, when faced with situations where a task must be completed once, an agent typicallyhas some prior knowledge.
E.g., an agent tasked with finding water on Mars may have experiencelooking for water in a desert on Earth.
We will therefore assume access to offline prior data of somesort that the agent may use for pretraining and during its single life.
In many cases such as the Marsexample, we may not have expert prior data of the desired task in the desired environment.
Hence, weemulate this setting by providing the agent with prior data from a related environment and deployingits single life on a domain with a distribution shift.
We can formalize this setting as follows.
We are given prior data D prior , which consists of transitionsfrom some source MDP M source .
The agent will then interact with a target MDP defined by M target =( S , A , ˜ P , R , ˜ ρ 0 , γ ) .
We assume that the the target MDP has an aspect of novelty not present in thesource MDP, such as different dynamics ˜ P ( s t +1 | s t , a t ) or a different initial state distribution ˜ ρ .
Naturally, the more similar the domains are, the easier the problem becomes, and the effectiveness ofany algorithm will be strongly dependent on the degree of similarity, though formalizing a preciseassumption on similarity between the source and target domain is difficult.
The reward between thetwo MDPs is the same, meaning the agent is still trying to accomplish the same task in the targetdomain as in the source.
The problem setting may be extended to include multiple source MDPs or aseries of target MDPs.
The goal in SLRL is to accumulate as much reward as possible in a single trial in M target .
Maintainingthe same notation as the previous section, the SLRL problem aims to maximize J = (cid:80) ht
=
γ t R ( s t ) ,where h is the trial horizon, which may be ∞ .
In general, we expect the task reward to be such thatlearning only from task rewards during the single life is difficult, for example because the reward isvery sparse or even awarded only upon successful completion of the task (which can only happenonce in the entire single life deployment).
We assume that there are no sink states beyond a terminalsuccess state, such that it is possible for the agent to autonomously recover from any mistake.
Note that this setup is essentially the same as the widely studied regret minimization problem inexploration [30].
However, while regret minimization is typically studied in the context of RLexploration theory, our aim with SLRL is to study a particular special case of the more general regretminimization framework that is meant to reflect a realistic setting in real-world RL (e.g., robotics)where an agent with prior experience must solve a task in a single (potentially long) trial.
As we analyze in Section 6, algorithms designed for episodic policy learning do not perform wellin the SLRL problem setting, even when the policy and replay buffer are pre-trained and seededwith the prior data, because they do not quickly recover from mistakes to get back onto a good statedistribution.
In Section 5, we will discuss an approach based on distribution matching that attemptsto address this issue.
Q -weighted Adversarial Learning (Q WALE )
In this section, we will present our method for addressing SLRL, which we call Q WALE .
The keyinsight in Q WALE is to utilize the prior data D prior to handle the sparse and uninformative rewardinformation in the target domain.
Our first observation is that the framework of AIL already providesa reasonable starting point, though it is not sufficient by itself: existing AIL methods assume thatthe prior data consists of expert demonstrations, and they train the agent to match the entire demodistribution, which is not necessarily the ideal distribution to match.
Since our goal is not to learn apolicy that repeatedly performs the task, but rather to solve it as quickly as possible once, we do notactually want to learn to imitate prior data, but rather to seek out states that resemble the best statesin the prior data, with better states being more preferred.
This is especially important when the priordata is not actually optimal, but might consist of arbitrarily suboptimal states.
We will discuss howthis can be accomplished with a modification of AIL which, instead of treating prior data as equallydesirable, preferentially drives the agent toward states that resemble the best states in the prior data.
5.1 Algorithm Description
Our algorithm’s desired behavior is to lead the agent to nearby states within distribution of theprior data if it is out of distribution and to nearby states closer to task completion if in distribution. GAIL [22] offers a framework to shape the state-action distribution towards that of an expert. Inparticular, GAIL minimizes D JS ( ρ π ( s, a ) || ρ ∗ ( s, a )) (ignoring the policy entropy H ( π ) ).
In SLRL,we relax this assumption, as we may only have access to suboptimal offline prior data. As such, wedesire an algorithm that will be agnostic to the quality of prior data. Hence, instead of learning apolicy that aims to uniformly match the entire state-action distribution of the prior data, we want tomatch the state-action pairs that will lead to task completion.
The problem of matching the state-action distribution has been studied in stochastic optimal control,particularly in REPS [37]. Roughly, this problem can be framed as the RL problem with an additionalconstraint encouraging the learned distribution to be close to the distribution of the prior data. Thesolution to this optimization problem gives us that our desired target state-action distribution has theform ρ ∗ target ( s, a ) ∝ ρ β ( s, a ) exp( Q π target ( s, a ) − V π target ( s )) , where the corresponding policy can bederived by using π target = ρ ∗ target ( s, a ) / (cid:82) A ρ ∗ target ( s, a ′ )
d a ′ using [50]. For a formal description of theproblem, see Appendix A.1. Thus, replacing ρ ∗ target as the target distribution instead of ρ ∗ , we propose Q -weighted adversariallearning (Q WALE ), which minimizes D JS ( ρ π || ρ ∗ target ) as follows: [Equation]
where the constant E ρ β [exp( Q π target ( s, a ) − V π target ( s ))] can be absorbed into the learning rate usedfor gradient descent.
Hence, Q WALE trains a weighted discriminator using a fixed Q -function Q ( s, a ) trained in the source MDP to distinguish between useful transitions and ones that may be less useful.
This Q -functionmay be obtained through RL pretraining, which is what we use for our experiments, or a variety ofother ways, such as offline RL or Monte Carlo estimation.
We train the discriminator in a similarmanner as GAIL, where the positives come from offline data and negatives from online experience.
When training the discriminator, we weight the positive states s by exp( Q ( s, a ) − b ) , where b isan implementation detail treated as a hyperparameter.
Following [1], we exclude V π target ( s ) in theweighting term in our practical implementation.
Including the term b changes the bias on thediscriminator, which in effect just adds a constant to the reward.
In practice, to avoid having totune b , we just use the value of the most recent state as b , i.e. b = Q ( s t , a t ) .
We include additionaldiscussion in the Appendix.
We normalize the Q -values to be between 0 and 1 by subtracting the minand dividing by the (max - min) and train the Q -weighted discriminator in alternating updates thatfinetune the policy and critic in an AIL fashion.
In this manner, Q WALE extends AIL to the generalsetting with any prior data.
Intuitively, the goal of our weighted discriminator training procedure is to obtain a discriminatorthat, when used as a reward, will drive the agent toward states that it believes would lead to betteroutcomes than its present state, based on the prior data.
The Q -function quantifies the agent’s belieffrom the prior data that a particular state will lead to high reward, making this a natural choice forestimating how desirable a state is at any given time.
Hence, using Q -values to weight the examplesfor the discriminator will cause the discriminator to prefer states that are closer to the goal over statesthat are further away.
This is significantly different from the behavior we would expect to see if wewere simply imitating the optimal policy, which would give equal weight to all of the transitionsalong an optimal path.
5.2 Practical Implementation
We optimize our objective using maximum entropy off-policy RL with the SAC algorithm ([18]),modified in a similar manner as we did with GAIL in the previous section.
In particular, we learnan additional discriminator q ϕ ( prior | s t ) , optimized using standard cross-entropy loss, which is
Single Trial Deployment 2: Require: D prior , test MDP M
, pretrained critic and
; 3: Initialize timestep
=4: while task not complete do 5:
Take a ∼ π ( · | s t ) , and observe r t and s t +1 6:
D online ← D online ∪ { ( s t , a t , r t , s t +1 ) } 7: ϕ ← ϕ − η ∇ ϕ L ( ϕ ) // Update discriminator according to Eq. 2 8: r ′ ( s t ) = r ( s t ) − log(1 − q ϕ ( prior | s t )) 9:
Q ( s, a ) , π ← SAC ( Q ( s, a ) , π, D prior ∪ D online , r ′ ) 10:
Increment t weighted accordingly.
[Equation]
The discriminator is used to modify the rewards when updating off-policy from all experience–priorand online.
At single trial test time, the actor and critic are optionally initialized with the pretrainedweights, and the replay buffer is initialized with the offline data.
For details such as networkarchitecture and hyperparameters, see Appendix A. We present the full algorithm in Algorithm 1.
Experiments
The goal of our experiments is to answer the following questions: (1) How does Q WALE compare to prior reinforcement learning and distribution matching approaches in single-life RL settings
(2) Dodistribution matching approaches help agents learn to recover from novel situations in single-life RL?
(3) How does Q WALE compare to different variants of adversarial imitation learning, with differentprior datasets?
6.1 Experimental Setup
To answer the above questions, we construct four single-life RL domains with varying prior datasetsand sources of novelty, and then measure performance both in terms of speed of task completion andoverall single-life success.
In this subsection, we describe this experimental set-up in detail.
Environments .
We consider the following four problem domains.
First, in the Tabletop-Organizationenvironment from the EARL benchmark [45], the agent is tasked with bringing a mug to one of fourdifferent locations designated by a goal coaster.
The prior data always has the same starting positionof the mug.
In the target environment, the starting position is in a new location unseen in the priordata.
Second, the Pointmass setting tasks an agent to move in 2D from its starting location at theorigin (0 , 0) to the point (100 , 0) .
The target environment introduces a dynamics shift in the form of astrong “wind”, where the agent is involuntarily pushed upward in the y-direction each step.
Third, weconstruct a modified HalfCheetah environment, in which it is difficult but feasible for the cheetah torecover when flipped over.
The target environment includes hurdles that the cheetah must jump over,as the prior data does not include these obstacles.
Finally, we evaluate on a modified Franka-Kitchenenvironment, adapted from [15], where the task is to close a microwave and a hinged cabinet.
Theprior data only contains trajectories of closing the microwave and the hinged cabinet separately, sothe agent must figure out online how to complete both tasks in a row.
In other words, both objects areopen at the start of single-life RL, and the agent has only previously seen instances where only one is open.
For the latter two environments, dense rewards are given, and the discriminator-based reward isadded to the extrinsic reward during single-life training.
These environments are shown in Figure 2.
Further details on the environments are given in Appendix A.
Comparisons.
To answer question (1), we compare Q WALE to three alternative methods: (a) SACfine-tuning, which pre-trains a policy and value function in the source setting and fine-tunes fora single, long episode in the target environment, (b) SAC-RND, which additionally includes an
RND exploration bonus [5] during single-life fine-tuning, and (c) GAIL-s, which runs generativeadversarial imitation learning [22, 26] where the discriminator only operates on the current states .
We choose for the discriminator to only look at s so that it is less susceptible to dynamics shiftbetween the source data and target environment.
We additionally compare to GAIL-sa, which passesboth the current state and action to the discriminator.
All methods use soft actor-critic (SAC) [18] asthe base RL algorithm.
We present additional comparisons in the Appendix.
Prior datasets .
For all four environments, we evaluate SLRL using data collected through RL as ourprior data.
More specifically, we run SAC in the source MDP in the standard episodic RL setting for
K steps and take the last 50,000 transitions as the prior data.
K is chosen such that the prior datacontains some good transitions but has not converged to an optimal policy yet.
While we are able torun episodic RL in the source MDP, this is not a requirement for SLRL, as long as prior data in thesource MDP is available.
For all methods, including Q WALE , GAIL variants, and SAC fine-tuningvariants, the policy and value function are pretrained in this manner for the initialization of single-life
RL.
We note that AIL methods like GAIL typically assume that the prior data consists of expertdemonstrations but we apply the algorithm only using mixed quality prior data, unless otherwisenoted.
In particular, Section 6.4 further evaluates AIL methods using demos as prior data, using 10demonstrations for the Tabletop environment and 3 demonstrations for the Pointmass domain.
Weinclude such experiments to answer question (3), i.e. to investigate how the quality of prior data mayaffect performance.
Evaluation Metrics.
To evaluate each method in each environment, we report the average and mediannumber of steps taken before task completion across 10 seeds along with the standard error andsuccess rate (out of 10).
During single-life RL, for all environments, the agent is given a maximum of200,000 steps to complete the task.
If it has not completed the task after 200k steps, then 200k islogged as the total number of steps, and the run is marked as unsuccessful.
6.2 Results using mixed data as prior data
In this subsection, we aim to answer our first experimental question and study how Q WALE performscompared to prior reinforcement learning and distribution matching approaches in single-life RLsettings.
As seen in Table 1, we find that Q WALE achieves the lowest average and median numberof steps as well as highest number of successes on three out of the four domains, and performscomparably to the other methods on the fourth environment, Franka-Kitchen.
On the Tabletop,
Pointmass, and Cheetah environments, Q WALE takes 20-40% fewer steps on average as the nextbest performing method.
GAIL also outperforms finetuning SAC across three of the four domains.
These results demonstrate the suitability of distribution-matching approaches over RL finetuning inthe SLRL setting.
We see that guidance particularly towards a good state distribution is important,as we compare to finetuning SAC with an exploration bonus through random network distillation([5]).
From Table 1, although RND may improve performance, particularly in the Tabletop domain,it generally does not perform as well as the distribution matching approaches, especially Q WALE ,showing that simply increasing exploration is not enough.
Furthermore, these results show that theadditional shaping provided by weighting the prior data by Q -value when training the discriminatorcan significantly improve guidance towards the goal.
While GAIL gives equal weight to all transitionsalong an optimal path, the agent in Q WALE is consistently guided towards states in the prior datawith higher Q -value, leading to more efficient and reliable single-life task completion.
Additionalcomparisons are presented in the Appendix.
6.3 Recovering from novel situations in SLRL
Next, to answer question (2), we analyze how Q WALE helps agents learn to recover from novelsituations in SLRL.
First, we analyze the state visitation of SAC finetuning in the online phase forthe Tabletop and Pointmass domains.
A key challenge of SLRL (and also fully autonomous RLin general) is that if the agent falls off of the distribution, it cannot rely on resets to get back ontrack.
Since there is a gap between the source and target domains, the agent will inevitably find [Table] itself in states that are out of distribution from the prior data.
A value function pre-trained on thesource data could in principle be used to evaluate states and guide the agent back towards good states,but it will be inaccurate on states outside of the prior data [13]; then, when the value of some ofthose out-of-distributions states is overestimated, the value function may misguide the policy awayfrom good states.
Hence, fine-tuning a pre-trained value function via online RL will not explicitlyencourage the agent to get back on distribution, especially in sparse reward envs.
As a result, theagent may spend a lot of time (perhaps infinite time) drifting once it falls out of distribution, whichwe see occurs in Figure 3.
On the other hand, distribution-matching methods like Q WALE will explicitly encourage the agent toget back on distribution, by giving higher rewards on distribution than off distribution.
In Figure 3,we visualize Q WALE ’s state visitation in the Tabletop and Pointmass domains throughout a singlelifetime.
We color the trajectories according to timestep for both methods as well as by reward(discriminator score).
The coloring in the timestep-colored plots is highly correlated with that inthe reward-colored plots, showing how the reward gradually guides the agent towards the goal.
Inparticular, when the agent is out of distribution, the agent is incentivized to explore states that willlead it closer back to the prior state distribution, and when the agent is within distribution, it isincentivized to move to states closer to the goal, leading to efficient task completion.
6.4 Using demos as prior data
Finally, we evaluate the performance of different discriminator-based approaches in the two SLRLproblem settings–Tabletop and Pointmass–where demonstration data is available as prior data.
Wecompare using AIL with a state-only discriminator (GAIL-s) as well as with a state-action discriminator (GAIL-sa) to our proposed Q -weighted discriminator method (Q WALE ).
With the latter method,we have access to the same Q -function pretrained when collecting prior data using standard RL forthe mixed data experiments above, but we do not initialize any of the algorithms at test time with thepretrained policy and critic weights.
From Figure 4, the GAIL variants are both able to consistently solve the task in the Tabletopdomain, but unsurprisingly, GAIL-sa does especially poorly in the Pointmass domain, where thedynamics have changed at test time.
Compared to the two GAIL variants, Q WALE gives a significantimprovement in both domains.
These results demonstrate how more detailed reward shaping towardsthe completion of the desired task can be helpful in the SLRL setting even with demonstrations asprior data.
Moreover, comparing these results with those in Table 1, while access to expert dataas prior data unsurprisingly improves the performance of GAIL methods, it can also improve theperformance of Q WALE .
Conclusion
In this paper, we formalized and studied a problem setting underlying single-life reinforcementlearning: settings where an agent needs to autonomously complete a task once while drawing uponprior experience from a related environment.
We found that standard fine-tuning via RL is ill-suitedfor this problem because the algorithm struggles to recover from mistakes and novel situations.
Wehypothesized that this happens since resets in episodic RL prevent algorithms from needing to recover,whereas single-life RL and continuing settings in general do demand the agent to find its way back togood states on its own.
We then postulated that distribution matching methods that aim to match thedistribution of related prior data may help agents recover via reward shaping, and presented a newdistribution matching method, Q WALE , that weights examples by their Q -value.
Our experimentsverified that distribution matching approaches indeed do make better use of prior data, and that Q WALE outperforms prior distribution matching methods on four single-life RL problems.
While Q WALE can efficiently complete novel target tasks in a single episode without any interventions,important limitations remain.
No algorithm, including Q WALE , was able to complete the target taskwith
100% success, indicating that future works should aim to improve an algorithm’s ability tosolve tasks consistently.
Moreover, the methods that we evaluated all used a pre-trained policy andvalue function from the source domain, which may be difficult to obtain in some source scenarios, asopposed to only obtaining some demonstrations or offline data.
Finally, it would be interesting toexplore problems with greater degrees of novelty between the source and target environments.
In ourexperimental domains, the reward and state space between the source and target MDPs were assumedto stay the same.
Eventually, we hope that this work may be extended to the case where the rewardmay be different in the online trial from the prior data.
We expect that such settings would placeeven greater importance on autonomy and exploration, requiring sophisticated strategies for bothrecovering to known states and exploring new strategies.
By publicly releasing and open-sourcing theenvironments and code upon publication, we hope that future work can more easily explore theseinteresting questions and continue to make progress on allowing RL agents to autonomously completetasks within a single lifetime.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We do
not expect any negative societal impacts that are unique to this work compared to other
works on reinforcement learning.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Included in the
supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Included in the supplemental material.
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes] See experiments in Section ?? .
(d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)?
[Yes] Included in the supplemental
material.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant
risks, with links to Institutional Review
Board
IRB) approvals, if applicable?
[N/ hourly wage to participants amountspent