You Only Live Once: Single-Life Reinforcement Learning via Learned Reward Shaping
Anonymous Author(s)
AffiliationAddress email
Abstract
Reinforcement learning algorithms are typically designed to learn a performantpolicy that can repeatedly and autonomously complete a task, typically startingfrom scratch.
However, many real-world situations operate under a different set ofassumptions: the goal might not be to learn a policy that can do the task repeatedly,but simply to perform a new task successfully once, ideally as quickly as possible,and while leveraging some prior knowledge or experience.
For example, imaginea robot that is exploring another planet, where it cannot get help or supervisionfrom humans.
If it needs to navigate to a crater that it has never seen before insearch of water, it does not really need to acquire a policy for reaching cratersreliably, it only needs to reach this particular crater once.
It must do so without thebenefit of episodic resets and tackle a new, unknown terrain, but it can leverageprior experience it acquired on Earth.
We formalize this problem setting, whichwe call single-life reinforcement learning (SLRL), where an agent must completea task once while contending with some form of novelty in a single trial withoutinterventions, given some prior data.
In this setting, we find that algorithmsdesigned for standard episodic reinforcement learning can struggle, as they havetrouble recovering from novel states especially when informative rewards arenot provided.
Motivated by this observation, we also propose an algorithm, Qweighted adversarial learning (Q WALE ), that addresses the dearth of supervisionby employing a distribution matching strategy that leverages the agent’s priorexperience as guidance in novel situations.
Our experiments on several singlelife continuous control problems indicate that methods based on our distributionmatching formulation are 20-60% more successful because they can more quicklyrecover from novel, out-of-distribution states.
Introduction
When building autonomous agents for the natural world, often the goal is not to learn a performantpolicy but rather to get something done, perhaps even suboptimally.
For example, an agent exploringon Mars looking for water will only need to complete its mission a single time.
As another example, a rescue robot will need to recover valuables from a particular burning building only once.
While the agent may have access to prior data about its task, a challenge arises from the fact the agent is inevitably going to have to contend with some form of novelty.
In the prior examples, the agent on Mars may have to contend with unknown terrain and environmental conditions, and the rescue agent may find certain paths in the building unpassable due to the fire.
We formalize this setting as the
SLRL) setting where the agent is evaluated on its ability to complete a in a single autonomously without episodic resets
Importantly the given online contains an aspect of novelty not present in the data although the task objective remains the same
The objective is to complete the given task as quickly as possible rather than learn a policy that can repeatedly complete the
We find that algorithms designed for episodic policy learning can struggle to complete single-lifetasks, even when initialized with prior data.
These algorithms empirically struggle to recover fromnovel states.
In episodic RL, the agent can rely on a reset to recover from an unfamiliar state.
Incontrast, in SLRL, the agent will inevitably fall off the distribution of prior data and must find its wayback to a good state distribution on its own.
We find that fine-tuning a pre-trained value function viaonline RL will not explicitly encourage the agent to get back on distribution.
We hypothesize thatbiasing exploration towards the known distribution represented in prior data and incentivizing theagent to stay there may be suboptimal from a policy learning perspective but may enable the agent toget the task done more quickly, which is what we care about in SLRL.
While shaped rewards mayhelp the agent find its way back and ultimately complete the desired task, the agent may often finditself in a sparse reward environment or with access to rewards that are not informative enough to beguided towards task completion.
Adversarial imitation learning (AIL) approaches such as GAIL ([21]) can potentially provide thedesired reward shaping via distribution matching.
However, using existing AIL methods naively maynot give the intended behavior in the SLRL setting due to two main shortcomings.
First, such methodsassume expert demonstrations are given as prior data, but in SLRL, we may be given suboptimaloffline prior data.
Second, AIL methods train the agent to match the entire distribution of priordata, which may be key to learning an optimal policy, but may be a drawback in our setting, as theagent might not be consistently guided towards task completion.
To address these shortcomings, wepropose a method in which different states in the prior data are weighted different amounts by theirestimated Q -value.
More concretely, we propose a Q -weighted AIL approach that incentivizes theagent to move towards states in the prior data with higher value than its current state, so that agentmay be guided consistently towards states closer to task completion.
Our contributions are as follows.
First, we formalize the SLRL problem setting, which we believe tobe a useful framework for modeling many situations in the real world.
We next provide an intuitiveargument and empirical analysis suggesting that learned reward shaping via distribution matching isbetter suited for this setting than finetuning without additional reward shaping.
We identify challengesthat uniquely arise in the SLRL setting with existing distribution matching approaches and proposea new approach, Q -weighted adversarial learning (Q WALE ), which is less sensitive to the qualityof prior data available and provides the agent with a shaped reward towards completing the desiredtask a single time.
Through our experiments, we explore the performance of different approaches in
SLRL.
We find that Q WALE can meaningfully guide the agent to explore the state distribution in itsprior data to complete the desired novel task 20-60% more successfully on four separate domainscompared to existing distribution matching approaches and RL fine-tuning.
Related Work
Autonomous RL .
In the context of deep RL, agents typically (but not always) are trained in episodicsetting and are evaluated on the quality of the learned policy.
Several recent works have developedalgorithms that can learn without episodic resets [18, 6, 62, 42, 44, 15, 16, 24, 43].
Like our work,such methods aim to make it possible to learn without any episodic resets, but are typically stillfocused on acquiring an effective policy that can perform the task repeatedly, typically by trainingsome auxiliary controller to enable the policy to “retry” the task multiple times without resets.
Incontrast, our aim is to develop an algorithm that can solve the task once, but as quickly as possible,which introduces a unique set of challenges as we discussed above.
Continual RL .
There is a rich literature on reinforcement learning in the continuing setting [29,41, 47, 60, 37, 28, 55] that considers maximizing the average reward accumulated over an infinitehorizon without episodic resets.
Such works often also consider regret minimization as the objective.
SLRL is closely related, and can be viewed as a special case where the agent has access to a prioroffline dataset and aims to solve a single task as quickly as possible in a new domain.
While the focusin continual learning is on general “lifelong” methods or on exploration, our focus is on effectivelyleveraging prior data in a setting that is meant to be reflective of real-world tasks (for example, inrobotics).
Leveraging offline data in online RL .
Learning expert policies given prior interaction data has beenextensively studied in imitation learning [1, 38, 13], inverse RL [33, 8, 64, 65], RL for sparse rewardsettings [3, 32, 36, 20, 50] and offline RL [27, 26, 23, 56, 32].
Across all these diverse topics, the goalis to learn a competent policy that can solve the task efficiently whereas the objective in this work isto complete the task in a single trial as quickly as possible.
To this end, we build on recent adversarialapproaches to inverse RL [21, 11, 45, 49, 25, 63] to encourage agent’s state visitation towards expertprior data where the agent is likely to be successful.
Prior methods have also studied adversarialinverse RL and imitation learning with non-expert data [51, 46, 57, 52, 54, 53, 5, 2].
However, aswe discuss in Section 6, these approaches need to be adapted for the SLRL setting to be efficient atcompleting the task and to handle novelty, for example when the dynamics may have changed.
Transfer and adaptation in RL .
Many prior works have studied the problem of adapting in presenceof shifts between train and test settings, often in a specific problem setting such as sim2real transfer [40, 48, 35, 30] or fast adaptation via meta-learning [9, 34, 31, 66, 10].
A common theme in theseworks is that the algorithm can often train in preparation for adaptation at test-time, thus affecting theprior experiences it may collect.
In contrast, the SLRL setting lays algorithmic emphasis on onlineexploration and adaptation, as the agent has access to fixed prior dataset of experiences.
Other transferlearning approaches adapt the weights of the policy to a new environment or task, either through rapidzero-shot adaptation [19, 61] or through extended episodic online training [22, 39, 7, 59, 58].
Unlikethe latter, we focus on adaptation within a single episode, but, unlike the former, with a focus onextended exploration and learning over tens of thousands of timesteps.
This problem setting leads tounique challenges, namely that the agent must autonomously recover from mistakes, hence requiringa distinct approach.
Preliminaries
In this section, we describe some preliminaries before formalizing our problem statement in thefollowing section.
We consider an agent that operates in a Markov decision process (MDP) consistingof the tuple M =
( S , A , p, R , ρ, γ ) , where S is the state space, A is the agent’s action space,p ( s t +1 | s t , a t ) represents the environment’s transition dynamics, R :
S → R indicates the rewardfunction, ρ : S → R denotes the initial state distribution, and γ ∈ [0 , 1) denotes the discountfactor.
In typical reinforcement learning, the objective is find a policy π that maximizes J ( π ) =
118 E τ ∼ π
[ (cid:80) ∞ t =
γ t R ( s t )] .
Although our method is a reward-driven RL algorithm, we utilize concepts from imitation learningto overcome sparse rewards, utilizing potentially suboptimal prior data.
To this end, we build onadversarial imitation learning (AIL), which uses prior data D prior in the form of expert demonstrations(we will relax this requirement) to recover the expert’s policy.
One such method is GAIL [21], whichfinds a policy π θ that minimizes the Jensen-Shannon divergence between its stationary distributionand the expert data.
It does so by training a discriminator network D : S ×
A → (0 , 1) , alternatingupdates with updates to the policy π .
Concretely, D and π are learned by optimizing the following:min π max D ∈ (0 , 1) S×A
E π [log( D ( s, a ))]
+ E π E [log(1 − D ( s, a ))] − λH ( π ) .
In Section 5, we will seethat AIL-style discriminator-based approaches can be adapted to the SLRL problem setting withoutdemonstration data, and will therefore form the basis of our method.
Single-Life Reinforcement Learning
In this section, we formalize our problem setting, the single-life reinforcement learning (SLRL)problem.
The defining characteristic of SLRL is that the agent is given a single “life”, i.e. trial, tocomplete a desired task, with the trial ending when the task is completed.
The agent must completethe task autonomously, without access to any human interventions or resets.
In the real world, when faced with situations where a task must be completed once, an agent typicallyhas some prior knowledge.
E.g., an agent tasked with finding water on Mars may have experience looking for water in a desert on Earth.
We will therefore assume access to offline prior data of somesort that the agent may use for pretraining and during its single life.
In many cases such as the Marsexample, we may not have expert prior data of the desired task in the desired environment.
Hence, weemulate this setting by providing the agent with prior data from a related environment and deployingits single life on a domain with a distribution shift.
We can formalize this setting as follows.
We are given prior data D prior , which consists of transitionsfrom some source MDP M source .
The agent will then interact with a target MDP defined by M target =( S , A , p, R , ρ, γ ) .
We assume that the the target MDP has an aspect of novelty not present in thesource MDP, such as different dynamics p ( s t +1 | s t , a t ) or a different initial state distribution ρ .
Naturally, the more similar the domains are, the easier the problem becomes, and the effectiveness ofany algorithm will be strongly dependent on the degree of similarity, though formalizing a preciseassumption on similarity between the source and target domain is difficult.
The reward between inthe two MDPs is the same, meaning the agent is still trying to accomplish the same task in the targetdomain as in the source.
The problem setting may be extended to include multiple source MDPs or aseries of target MDPs.
The goal in SLRL is to accumulate as much reward as possible in a single trial in M target .
Maintainingthe same notation as the previous section, the SLRL problem aims to maximize J = (cid:80) ht
=
γ t R ( s t ) ,where h is the trial horizon, which may be ∞ .
In general, we expect the task reward to be such thatlearning only from task rewards during the single life is difficult, for example because the reward isvery sparse or even awarded only upon successful completion of the task (which can only happenonce in the entire single life deployment).
We assume that there are no sink states beyond a terminalsuccess state, such that it is possible for the agent to autonomously recover from any mistake.
Note that this setup is essentially the same as the widely studied regret minimization problem inexploration [29].
However, while regret minimization is typically studied in the context of RLexploration theory, our aim with SLRL is to study a particular special case of the more general regretminimization framework that is meant to reflect a realistic setting in real-world RL (e.g., robotics)where an agent with prior experience must solve a task in a single (potentially long) trial.
As we analyze in Section 5, algorithms designed for episodic policy learning do not perform wellin the SLRL problem setting, even when the policy and replay buffer are pre-trained and seededwith the prior data, because they do not quickly recover from mistakes to get back onto a good statedistribution.
In Section 6, we will discuss an approach based on distribution matching that attemptsto address this issue.
Single-Life Performance of Online RL
Now that we have described the problem setting, we now empirically analyze online RL, which isdesigned for episodic learning, in the SLRL setting.
Namely, we consider finetuning SAC [17], whichpre-trains a policy and value function in the source setting and fine-tunes during the single trial in thetarget environment.
As we will see in Section 7, finetuning SAC performs poorly in the SLRL setting.
We save thedetails of the experimental setup until Section 7, but to first motivate the use of distribution-matchingapproaches in our problem setting, we analyze the state visitation of SAC finetuning in the onlinephase for the Tabletop and Pointmass domains (see Figure 2).
A key challenge of SLRL (and alsofully autonomous RL in general) is that if the agent falls off of the distribution, it cannot rely onresets to get back on track. Since there is a gap between the source and target domains, the agentwill inevitably find itself in states that are out of distribution from the prior data. A value functionpre-trained on the source data could in principle be used to evaluate states and guide the agent backtowards good states, but it will be inaccurate on states outside of the prior data [12]; then, when thevalue of some of those out-of-distributions states is overestimated, the value function may misguidethe policy away from good states. Hence, fine-tuning a pre-trained value function via online RL willnot explicitly encourage the agent to get back on distribution, especially in sparse reward envs.
As aresult, the agent may spend a lot of time (perhaps infinite time) drifting once it falls out of distribution,which we see occurs in Figure 2.
On the other hand, distribution-matching methods like GAIL [21] will explicitly encourage theagent to get back on distribution, by giving higher rewards on distribution than off distribution.
However, existing adversarial imitation learning methods assume that the prior data consists ofexpert demonstrations, and they train the agent to match the entire demo distribution, which is notnecessarily the ideal distribution to match.
In the following section, we will discuss a method thataims to address these shortcomings.
Q -weighted Adversarial Learning (Q WALE )
In this section, we will present our method for addressing SLRL, which we call Q WALE .
The keyinsight in Q WALE is to utilize the prior data D prior to handle the sparse and uninformative rewardinformation in the target domain.
Our first observation is that the framework of AIL already providesa reasonable starting point, though it is not sufficient by itself: rather than using only the task reward,which is too sparse to be useful in a single episode, we can bias the agent to seek out states thatare similar to those seen in the prior data.
However, since our goal is not to learn a policy thatrepeatedly performs the task, but rather to solve it as quickly as possible once, we do not actuallywant to learn to imitate prior data, but rather to seek out states that resemble the best states in theprior data, with better states being more preferred.
This is especially important when the prior data isnot actually optimal, but might consist of arbitrarily suboptimal states.
We will discuss how this canbe accomplished with a modification of AIL which, instead of treating prior data as equally desirable,preferentially drives the agent toward states that resemble the best states in the prior data.
6.1 Algorithm Description
In SLRL, we may have shifts in dynamics online, in which case matching state-action distributions asdone in GAIL ([21]) may not be appropriate. GAIL with a state discriminator will help lead the agentback to the prior data distribution but by matching the entire demo state distribution, the discriminatordoes not necessarily incentivize the agent to go towards task completion.
Our algorithm’s desiredbehavior is to lead the agent to nearby states within distribution of the prior data if it is out ofdistribution and to nearby states closer to task completion if in distribution
Our proposed methodfor shaping relies on the intuition that as states gradually get closer to task completion, even withinexpert data, they should have gradually higher values as well.
We propose Q -weighted adversarial learning (Q WALE ), which trains a Q -weighted discriminator.
Inorder to use the prior data effectively, we use a fixed Q -function Q ( s, a ) trained in the source MDPto distinguish between useful transitions and ones that may be less useful.
This Q -function may beobtained through RL pretraining, which is what we use for our experiments, or a variety of otherways, such as offline RL or Monte Carlo estimation.
We train the discriminator in a similar manneras GAIL, where the positives come from the offline data and negatives from online experience.
Totake into account the varied quality of the data, we use the intuition that the closer a state is to taskcompletion, the higher its value should be.
In particular, a state in the prior data should get smallerweight if it has worse value than the agent’s current state, so the agent is consistently incentivizedto move towards states in the prior data with higher value than its current state.
Therefore, whentraining the discriminator, we weight the positive states s by exp( Q ( s, a ) − b ) and the negatives byexp( − Q ( s, a ) + b ) , where b is an implementation detail, which we discuss in the Appendix.
Wenormalize the Q -values to be between 0 and 1 and train the Q -weighted discriminator in alternating
Single Trial Deployment 2: Require: D prior , test MDP M test , pretrained critic Q ( s, a ) , and ( ) policy π ; 3: Initialize: replay online transitions D online ; parameters ϕ i q prior | s t ) , timestep t =4: while task not complete do 5: Sample a ∼ π ( · | s t ) 6: ϕ ← ϕ − η ∇ ϕ L ( ϕ ) //
Update discriminator according to Eq. 1 7: r ′ ( s t ) =
r ( s t ) + log q θ ( prior | s t ) 8: Q ( s, a ) , π ← SAC ( Q ( s, a ) , π, D prior ∪ D online , r ′ ) 9: Increment t updates with SAC updates that finetune the policy and critic in an AIL fashion.
In this manner,
Q WALE extends AIL to the general setting with any prior data.
The goal of our weighted discriminator training procedure is to obtain a discriminator that, whenused as a reward, will drive the agent toward states that it believes would lead to better outcomesthan its present state, based on the prior data.
The Q -function quantifies the agent’s belief from theprior data that a particular state will lead to high reward, making this a natural choice for estimatinghow desirable a state is at any given time.
Hence, using Q -values to weight the examples for thediscriminator will cause the discriminator to prefer states that are closer to the goal over states thatare further away.
This is significantly different from the behavior we would expect to see if we weresimply imitating the optimal policy, as this would give equal weight to all of the transitions along anoptimal path.
When the reward is more complex, using Q -values as weights generalizes this intuition.
6.2 Practical Implementation
We optimize our objective using maximum entropy off-policy RL with the SAC algorithm ([17]),modified in a similar manner as we did with GAIL in the previous section.
In particular, we learnan additional discriminator q θ ( prior | s t ) , optimized using standard cross-entropy loss, which isweighted accordingly:
[Equation]
The discriminator is used to modify the rewards when updating off-policy from all experience–priorand online.
At single trial test time, the actor and critic are optionally initialized with the pretrainedweights, and the replay buffer is initialized with the offline data.
For details such as networkarchitecture and hyperparameters, see Appendix ?? . We present the full algorithm in Algorithm 1.
Experiments
The goal of our experiments is to answer the following questions: (1) How does Q WALE compare toprior reinforcement learning and distribution matching approaches in single-life RL settings?
(2) Dodistribution matching approaches help agents learn to recover from novel situations in single-life RL?(3)
How does Q WALE compare to different variants of adversarial imitation learning, with differentprior datasets?
7.1 Experimental Setup
To answer the above questions, we construct four single-life RL domains with varying prior datasetsand sources of novelty, and then measure performance both in terms of speed of task completion andoverall single-life success.
In this subsection, we describe this experimental set-up in detail.
Environments .
We consider the following four problem domains.
First, in the Tabletop-Organizationenvironment from the EARL benchmark [43], the agent is tasked with bringing a mug to one of fourdifferent locations designated by a goal coaster.
The prior data always has the same starting positionof the mug.
In the target environment, the starting position is in a new location unseen in the priordata.
Second, the Pointmass setting tasks an agent to move in 2D from its starting location at theorigin (0 , 0) to the point (100 , 0) .
The target environment introduces a dynamics shift in the form of astrong “wind”, where the agent is involuntarily pushed upward in the y-direction each step.
Third, weconstruct a modified HalfCheetah environment, in which it is difficult but feasible for the cheetah torecover when flipped over.
The target environment includes hurdles that the cheetah must jump over, as the prior data does not include these obstacles.
Finally, we evaluate on a modified Franka-Kitchenenvironment, adapted from [14], where the task is to close a microwave and a hinged cabinet.
Theprior data only contains trajectories of closing the microwave and the hinged cabinet separately, sothe agent must figure out online how to complete both tasks in a row.
In other words, both objects areopen at the start of single-life RL, and the agent has only previously seen instances where only one isopen.
For the latter two environments, dense rewards are given, and the discriminator-based reward isadded to the extrinsic reward during single-life training.
These environments are shown in Figure 3.
Further details on the environments are given in Appendix ?? .
Comparisons.
To answer question (1), we compare Q WALE to three alternative methods: (a) SACfine-tuning, which pre-trains a policy and value function in the source setting and fine-tunes fora single, long episode in the target environment, (b) SAC-RND, which additionally includes an
RND exploration bonus [4] during single-life fine-tuning, and (c) GAIL-s, which runs generativeadversarial imitation learning [21, 25] where the discriminator only operates on the current states .
We choose for the discriminator to only look at s so that it is less susceptible to dynamics shiftbetween the source data and target environment.
We additionally compare to GAIL-sa, which passesboth the current state and action to the discriminator.
All methods use soft actor-critic (SAC) [17] asthe base RL algorithm.
Prior datasets .
For all four environments, we evaluate SLRL using data collected through RL as ourprior data.
More specifically, we run SAC in the source MDP in the standard episodic RL setting for
K steps and take the last 50,000 transitions as the prior data.
K is chosen such that the prior datacontains some good transitions but has not converged to an optimal policy yet.
While we are able torun episodic RL in the source MDP, this is not a requirement for SLRL, as long as prior data in thesource MDP is available.
For all methods, including Q WALE , GAIL variants, and SAC fine-tuningvariants, the policy and value function are pretrained in this manner for the initialization of single-life
RL.
We note that AIL methods like GAIL typically assume that the prior data consists of expertdemonstrations but we apply the algorithm only using mixed quality prior data, unless otherwisenoted.
In particular, Section 7.4 further evaluates AIL methods using demos as prior data, using 10demonstrations for the Tabletop environment and 3 demonstrations for the Pointmass domain.
Weinclude such experiments to answer question (3), i.e. to investigate how the quality of prior data mayaffect performance.
Evaluation Metrics.
To evaluate each method in each environment, we report the average and mediannumber of steps taken before task completion across 10 seeds along with the standard error andsuccess rate (out of 10).
During single-life RL, for all environments, the agent is given a maximum of200,000 steps to complete the task.
If it has not completed the task after 200k steps, then 200k islogged as the total number of steps, and the run is marked as unsuccessful.
7.2 Results using mixed data as prior data
In this subsection, we aim to answer our first experimental question and study how Q WALE performscompared to prior reinforcement learning and distribution matching approaches in single-life RLsettings.
As seen in Figure 4 and Table 1, we find that Q WALE achieves the lowest average andmedian number of steps as well as highest number of successes on three out of the four domains,and performs comparably to the other methods on the fourth environment, Franka-Kitchen.
On the
Tabletop and Pointmass environments, Q WALE takes less than half of the number of steps on averageas the next best performing method.
It is possible that the method does not work as well on the
Franka-Kitchen environment because the pretrained Q -function may be quite inaccurate, as there is
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
a global distribution shift in the state space at test time–the agent has never seen both objects openbefore in the prior data.
GAIL also outperforms finetuning SAC across three of the four domains.
These results demonstrate the suitability of distribution-matching approaches over RL finetuning inthe SLRL setting.
We see that guidance particularly towards a good state distribution is important,as we compare to finetuning SAC with an exploration bonus through random network distillation([4]).
From Figure 4, although RND may improve performance, particularly in the Tabletop domain,it generally does not perform as well as the distribution matching approaches, especially Q WALE ,showing that simply increasing exploration is not enough.
Furthermore, these results show that theadditional shaping provided by weighting the prior data by Q -value when training the discriminatorcan significantly improve guidance towards the goal.
While GAIL gives equal weight to all transitionsalong an optimal path, the agent in Q WALE is consistently guided towards states in the prior datawith higher Q -value, leading to more efficient and reliable single-life task completion.
7.3 Analysis of distribution matching approaches
Next, to answer question (2), we analyze how Q WALE helps agents learn to recover from novelsituations in SLRL.
To do so, we visualize Q WALE ’s state visitation in the Tabletop and Pointmassdomains throughout a single lifetime.
We color the trajectories according to timestep for bothmethods as well as by reward (discriminator score).
The coloring in the timestep-colored plots ishighly correlated with that in the reward-colored plots, showing how the reward gradually guides theagent towards the goal.
In particular, when the agent is out of distribution, the agent is incentivized toexplore states that will lead it closer back to the prior state distribution, and when the agent is withindistribution, it is incentivized to move to states closer to the goal, leading to efficient task completion.
7.4 Using demos as prior data
Finally, we evaluate the performance of different discriminator-based approaches in the two SLRLproblem settings–Tabletop and Pointmass–where demonstration data is available as prior data.
Wecompare using AIL with a state-only discriminator (GAIL-s) as well as with a state-action discriminator (GAIL-sa) to our proposed Q -weighted discriminator method (Q WALE ).
With the latter methods,we have access to the same Q -function pretrained when collecting prior data using standard RL forthe mixed data experiments above, but we do not initialize any of the algorithms at test time with thepretrained policy and critic weights.
From Figure 6, the GAIL variants are both able to consistently solve the task in the Tabletop domain, but unsurprisingly, GAIL-sa does especially poorly in the Pointmass domain, where thedynamics have changed at test time.
Compared to the two GAIL variants, Q WALE gives a significantimprovement in both domains.
These results demonstrate how more detailed reward shaping towardsthe completion of the desired task can be helpful in the SLRL setting even with demonstrations asprior data.
Moreover, comparing these results with those in Table 1, while access to expert dataas prior data unsurprisingly improves the performance of GAIL methods, it can also improve theperformance of Q WALE .
Conclusion
In this paper, we formalized and studied a problem setting underlying single-life reinforcementlearning: settings where an agent needs to autonomously complete a task once while drawing uponprior experience from a related environment.
We found that standard fine-tuning via RL is ill-suitedfor this problem because the algorithm struggles to recover from mistakes and novel situations.
Wehypothesized that this observation stems from the fact that resets in episodic RL prevent algorithmsfrom needing to recover, whereas single-life RL and continuing settings in general do demand theagent to find its way back to good states on its own.
We then postulated that distribution matchingmethods that aim to match the distribution of related prior data may help agents recover via rewardshaping, and presented a new distribution matching method, Q WALE , that weights examples by their Q -value.
Our experiments verified that distribution matching approaches indeed do make better useof prior data, and that Q WALE is competitive with or outperforms prior distribution matching methodson four single-life RL problems.
While Q WALE can efficiently complete novel target tasks in a single episode without any interventions,important limitations remain.
No algorithm, including Q WALE , was able to complete the target taskwith
100% success, indicating that future works should aim to improve an algorithm’s ability tosolve tasks consistently.
Moreover, the methods that we evaluated all used a pre-trained policy andvalue function from the source domain, which may be difficult to obtain in some source scenarios,as opposed to only obtaining some demonstrations or offline data.
Finally, it would be interestingto explore problems with greater degrees of novelty between the source and target environments.
We expect that such settings would place even greater importance on autonomy and exploration,requiring sophisticated strategies for both recovering to known states and exploring new strategies.
By publicly releasing and open-sourcing the environments and code upon publication, we hope thatfuture work can more easily explore these interesting questions and continue to make progress onallowing RL agents to autonomously complete tasks within a single lifetime.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 8.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We do
not expect any negative societal impacts that are unique to this work compared to other
works on reinforcement learning.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Included in the
supplemental material.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Included in the supplemental material
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See experiments in Section 7.
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] Included in the supplemental
material.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]