EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL
Author s)
Abstract
Reinforcement learning (RL) in long horizon and sparse reward tasks is notoriouslydifficult and requires a lot of training steps.
A standard solution to speed up theprocess is to leverage additional reward signals, shaping it to better guide thelearning process.
In the context of language-conditioned RL, the abstraction andgeneralisation properties of the language input provide opportunities for more efficient ways of shaping the reward.
In this paper, we leverage this idea and proposean automated reward shaping method where the agent extracts auxiliary objectivesfrom the general language goal.
These auxiliary objectives use a question generation (QG) and question answering (QA) system: they consist of questions leadingthe agent to try to reconstruct partial information about the global goal using itsown trajectory.
When it succeeds, it receives an intrinsic reward proportional to itsconfidence in its answer.
This incentivizes the agent to generate trajectories whichunambiguously explain various aspects of the general language goal.
Our experimental study using various BabyAI [7] environments shows that this approach,which does not require engineer intervention to design the auxiliary objectives,improves sample efficiency by effectively directing exploration.
Introduction
One of the main challenges of Reinforcement Learning (RL) research is to train agents able ofabstraction, generalisation and communication.
Languages, be they natural or formal, afford thesedesirable properties [11].
Based on this insight, many papers have tried to leverage the abilitiesof language in RL to enable communication and improve generalisation and sample efficiency[3, 22, 12, 36].
The domain can be subdivided into language-conditioned RL (LC-RL) in whichlanguage conditions the formulation of the problem[2, 12] and language-assisted RL, where languagehelps the agent to learn [15, 10, 1, 9].
In the present paper, we focus on the LC-RL framework wherethe agent initially receives a language instruction and must act to optimise the corresponding rewardfunction.
Unfortunately, the corresponding RL algorithms are sample inefficient, especially due tothe fact that the reward function is sparse when it is restricted to completing the goal.
To tackle the reward sparsity issue, one idea is to densify the reward by decomposing the general goalinto sub-goals and rewarding them individually.
This idea is based on a decomposition principle whichhypothesises that a general goal can be decomposed into a set of easier ones.
Previous works assumea strict decomposition of the general goal into sub-goals [3, 17].
However, strict decompositionrequires that that each high-level goal can be reached through an exact series of low-level policies,an assumption which fails when all the primitive actions are not known in advance.
More recently,the ELLA method was proposed as a language reward shaping technique that is sample efficient anddoes not require a strict decomposition: it rewards interesting auxiliary objectives, without requiringrigid ordering and correspondence with sub-goals [23].
However, all these methods suffer from the need for expert input.
Indeed, they require task-specificengineering for determining at least one of the following elements: the set of sub-goals or auxiliaryobjectives, the relevant ones and when they are achieved, and the appropriate reward.
While all thesepoints can be addressed more or less easily in simple game-like environments, they get demandingwhen the environment is more complex.
Therefore, it seems desirable to find an alternative rewardshaping approach that minimises expert involvement.
We would like the agent to generate its ownauxiliary objectives and create intrinsic rewards, just like a human would do.
The Natural Language Processing ( NLP ) field often suffers from the same need of expert input,e.g. for the evaluation of automatic summary tasks.
In that context, some successful approacheshave developed reference-less metrics [31], [29], based on question generation ( QG ) and questionanswering ( QA ).
These techniques assess the quality of a generated text by measuring the quantityof information from the source conserved in the generated text (broadly, how well one can answerquestions about the original text using the generated text).
These metrics are said reference-lessbecause they do not require a comparison with a man-made example to evaluate the quality of a text.
Contributions In our work, we build on these reference-less metrics to circumvent the need of expertinput for generating auxiliary objectives.
We adapt it and propose a novel QG/QA framework for
RL called EAGER.
In EAGER, an agent reuses the initial language goal sentence to generate aset of questions (QG): each of these self-generated questions defines an auxiliary objective.
Here,generating a question consists in masking a word of the initial language goal.
Then the agent triesto answer these questions (guess the missing word) only by observing its trajectory so far.
Whenit manages to answer a question correctly (QA) it obtains an intrinsic reward proportional to itsconfidence in the answer.
The QA module is trained using a set of successful example trajectories.
Ifthe agent follows a path too different from correct ones at some point in its trajectory, the QA modulewill not answer the question correctly, resulting in zero intrinsic reward.
The sum of all the intrinsicrewards measures the quality of a trajectory in relation to the given goal.
In other words, maximizingthis intrinsic reward incentivizes the agent to produce behaviour that unambiguously explains variousaspects of the given goal.
To the best of our knowledge, EAGER is the only framework that can automatically 1) generaterelevant auxiliary objectives, 2) determine their completion and 3) return the appropriate intrinsic reward.
This approach only assumes the agent has access to a dataset of demonstrated behavioursassociated to global language commands, which enables it to pre-train its question answering module.
Thus our work brings the following contributions:
• We create a QG/QA metric providing to an agent an information-rich measure of the quality of its trajectory given a goal.
• We propose the EAGER framework that lets the agent guide its own learning process by generatingauxiliary objectives and producing intrinsic rewards without requiring any expert intervention.•
We show that EAGER retains the good properties of ELLA without requiring task-specific expertknowledge, by leveraging properties of language.•
We experiment EAGER with the BabyAI platform [7]: we compare our approach against ELLA(SOTA on BabyAI) and RIDE (a non-language based reward shaping approach using intrinsicmotivation), showing its robustness and sample efficiency. Furthermore, although we use example trajectories to train
the QA
, their use is much more
parsimonious
than
training
an agent using
behavioural cloning
[7], as we
show
in Appendix
C.
Related work
Language-conditioned RL.
We place our work in the LC-RL setting, where an agent learns apolicy to execute language commands [24, 7, 20, 21].
We reuse the BabyAI platform [7], widelyused in this domain as it enables to decouple exploration challenges from perception challenges.
Ituses a synthetic language exhibiting interesting combinatorial properties with possible conjunction ofproperties, and procedural generation to avoid overfitting [8].
Here, we consider instruction followingagents which receive external instructions and rewards [14, 5, 17].
Language as an abstraction in hierarchical RL.
Several approaches leverage language forabstraction in hierarchical RL.
One approach uses language for training a low-level instructionfollowing policy, then learns a high-level policy that generates the sequence of low-level goals [17].
Another one explicitly decomposes high-level tasks into low-level ones as in policy sketches [3].
ELLA [23] also uses language to decompose high-level tasks but relaxes the strict decompositionconstraint by replacing sub-goals with auxiliary objectives.
We adopt the same flexible framework in our approach but automatise the decomposition
Language for exploration and reward shaping.
Reward shaping is a form of guidance thatsupplies additional rewards to the agent to direct its learning process.
Among approaches studyinghow language can shape rewards and exploration, LEARN [12] proposes to map intermediate naturallanguage instruction to intermediate rewards.
Similarly, [35] enables reward shaping using naturallanguage through a narration-guided method.
The high-level tasks are decomposed into low-leveltasks and rewarded using narration.
ELLA [23] is positioned in the same paradigm but with fewerassumptions about the environment or the structure of the task.
Other approaches assume that an oracle provides language descriptions of environment states whichare used as state abstraction to generate novelty intrinsic rewards and guide exploration [25, 34].
This extends classical approaches to intrinsically motivated exploration [4], such as count-basedmethods [6] or RIDE [28].
Here, we also use language to generate intrinsic rewards, but we do notneed language descriptions of states.
Besides, other forms of intrinsic motivation systems, such as
IMAGINE [10], first learn language-parameterized reward functions through interaction with a socialpeer, then autonomously use language to generate diverse and novel goals, using the learned rewardfunction for self-supervision.
Asking questions in RL.
Beyond reward shaping, some methods consider agents that use languageto ask questions to external knowledge sources.
In QWA [36], questions are used to identify sub-tasksand prune irrelevant actions.
In AFK [19], questions are used to obtain world knowledge that helpscompleting tasks.
Natural Language Processing One of the sources of QG/QA methods is the thriving field ofquestion generation from natural language processing and information retrieval [16].
Our approach is inspired from text generation methods where QG and QA are used to measure the quality of agenerated text without using a human reference [31, 29].
Problem statement
We set ourselves in the standard framework of LC-RL with an augmented Partial Observation Markov
Decision Process [33] M defined by the tuple ( S , A , Z , T , O , G , V , I , R , γ ), with S the state space, Z the observation space, A the actions space, G the goal space, V the vocabulary of goal instructions.
O stands as the observation function O : S → Z , which maps states to the observations space.
I isthe instruction function I : G → V isize , which maps any goal in G to the set of language instructions,which correspond to sequences of isize symbols (the empty symbol ϵ belongs to V to allow variableinstruction sizes).
R is a goal-conditioned state action reward function, with R : S ×
A × G → [0 , 1]the extrinsic reward received for some goal g ∈ G and γ the discount factor. For simplicity, we notein the following r gt = R ( s t , a t , g ) as the reward obtained at step t of any episode with goal g . At each time step t , the agent receives an observation o t
∈ Z following the observation function O : S → Z and selects an action a t ∈
A to reach a goal g ∈ G , expressed by ω g = I ( g ) .
T : S ×
A → P ( S ) is the transition function.
Using RL, we search for an optimal goal-conditionedpolicy π ∗ , such that π ∗ : S × V isize → A maximises the discounted expected return R π g t =
E π [ (cid:80) Tk {+=
+} γ k r gt + k +1 ] .
We consider in this work sparse reward problems, where r ( s, a, g ) returns 1for any state s such that d ( s, g ) ≤ ϵ , for a given distance function d and a specified threshold ϵ , and 0otherwise.
Moreover, we assume a limited number of steps at most H steps. These two conditionsresult in a hard exploration problem.
To deal with those problems, many methods aim at densifying rewards, by focusing on auxiliaryobjectives during training, whose accomplishment can help the agent to reach the goal g at hand. Inprevious work, such as ELLA [23], the selection of relevant objectives required the intervention of anexpert (in the form of expert annotations and example trajectories), which can be problematic becausenew expert trade-offs have to be established for each new environment. Thus, an automated waymust be found to recover the relevant auxiliary objectives, measure their completion, and associatethe appropriate reward. Rather than relying on expert knowledge for defining auxiliary objectives, we assume that wehave access to a set of trajectories of successful examples coupled to their respective instructions{ ( τ 0 , ω g 0 ) , ..., ( τ n , ω gn ) } , where τ n =
( o i , a i ) i ∈
[ | 0 ,k | ] with k the number of steps. For any goalinstruction ω g , we consider a function f that aims at generating a set of auxiliary goals of g , suchthat all g ′ ∈ f
( ω g ) belong to G g , with G g ⊂ G the set of goals that help training the agent towards g . Then, for a trajectory τ and any g ′ ∈ f ( ω g ) , a function h determines the probability h ( τ, g ′ ) ∈
[0 , 1]that the auxiliary objective g ′ has been achieved.
We train h using the example demonstrations suchthat we can leverage its generalisation abilities to use h for unseen trajectories.
Please note that Behavioural Cloning methods also rely on the exploitation of expert trajectories, by learning the policy π BC that maximises the log-likelihood L BC : [Equation] where [Equation]
However, the latter uses demonstrations to train an agent to copy an expert based on a data setof example trajectories. This technique is only effective when the agent is close enough to thedemonstrated examples. A large number of demonstrations are therefore required to cover the spaceof states that the agent may encounter.
Methods such as GAIL [18] can partially circumvent thisproblem by forcing the agent to stay on known trajectories, but this impairs generalisation.
Ourmethod is much more parsimonious in the use of such examples: because it generalises well, theagent can receive intrinsic rewards even in areas not encountered in the expert demonstrations (see Appendix ?? ).
Method
In this section, we first introduce EAGER, our automated reward shaping method based on QG/QA,then we present a practical implementation in the context of the BabyAI benchmark.
Figure 1provides a graphical overview of EAGER.
4.1 EAGER
We need an automatic evaluation method that is fine-grained enough to rank various trajectoriesdepending on a language instruction.
But various successful policies can generate a set of validtrajectories for the same goal.
It could be deceiving to rank them based only on their final results, asan overly complex trajectory seems intuitively worse than a simpler one even if the result is the same.
To address this issue, the EAGER framework consists of an agent learning module, a questiongeneration module QG (automatic, but not learned) and a learned question answering module QA ..
These two modules fulfil the role of the functions f and h defined in Section 3.
EAGER is inspiredfrom works like QuestEval [31] and Data-QuestEval [29], developed for natural language generation.
For instance, for abstractive summarization, by generating questions from the original text (QG) andtrying to answer them using the summary QA, this method measures the quantity of informationshared between both texts.
In our work, we draw the analogy with the NLP task with the goal replacing the original text andthe trajectory replacing the summary.
We use the QG/QA system to verify that a trajectory containsthe same level of information as the language instruction, meaning that the goal is contained inthe trajectory.
As the goal can be contained in a lot of different trajectories, we also favour simpletrajectories.
If one can easily answer the question, that means the trajectory is simple.
The QG module We assume that the goal linguistic instruction ω g of g ∈ G is a highly expressivelanguage instruction (e.g Put the red ball next to the blue box , open the red door , ...), containingby themselves enough world knowledge to generate questions by masking words.
Thus, the QGmodule returns a list of k questions QG k
( ω g ) that can be seen as auxiliary objectives.
Each questionis formed by masking one word in the linguistic instruction.
Crucially, the choice of words to maskcan be done automatically without any expert knowledge of the task, or the environment, for instancemasking all nouns and adjectives.
These questions can be seen as auxiliary objectives guiding theagent during training.
Besides, being formulated in natural language, these auxiliary objectives areeasily interpreted.
The QA module Let ˜ A be the set of possible answers generated automatically from the list oftokens masked by the QG.
Thus, ˜ A contains ( q, a ∗ ) pairs of questions and expected answers.
The
QA module returns the probability for all ˜ a ∈ ˜ A , that ˜ a answers question q , given a trajectoryτ t , where τ t = ( o i , a i ) i ∈
[ | 0 ,t | ] are state-action pairs and t the time step.
We note ˜ a ∗ =
QA ( q, τ t )the answer greedily generated from the module and QA (˜ a ∗ | q, τ t ) the associated probability.
Theauxiliary objectives from the QG are considered achieved when the QA answers them successfully.
In our work, the QA module is pre-trained using full example trajectories generated by a bot, see Section 5.1, without any type of annotations to guide it.
Besides, at time step t , there is no guaranteethat the trajectories contain enough information to correctly answer a question.
Thus we include a«no_answer» token in ˜
A to prevent the QA from answering correctly by chance.
Moreover, since the
QA module takes the whole trajectory, once it has answered a question, it can also answer it at thenext step.
To avoid giving a reward that does not have direct link to the current step, every time aquestion q is answered, it is removed from the set of questions: We note Q t the active set of questions,the initial set is Q 0 = QG k ( ω g ) and once a question is answered, we apply Q t ← Q t − 1 \ { q } .
Architecture of the QA The QA is used to compositionally chain low-level tasks.
To do this,it relies on the episodic transformer [27].
This architecture uses multimodal transformers (overlanguage, visual observation, and a list of actions) that have access to the full episode.
For any timestep t and any question, a ( x 1: L , v 1: t , a 1: t ) tuple is given as input to the QA module.
The languageinput x 1: L is the question, it is a sequence of L tokens with x i ∈ N .
The visual input v 1: t is the list ofobservations v t ∈ R W × H × 3 .
Finally, the action input a 1: t is a list of discrete actions.
As output, thenetwork returns the probability distribution over the set of possible answers ˜ A .
Intrinsic reward The main difficulty in ranking trajectories is that many trajectories can sharethe same description and thus can answer questions correctly.
However, a human assessing severaltrajectories, with the same extrinsic reward, relatively to the same goal would prioritise the simplestone.
To account for this, we make the reward proportional to the confidence in the answer at time stept : QA (˜ a ∗ | q, τ t ) .
An overly complex trajectory being harder to understand for the QA, the answershould be given with less confidence, so it should obtain less reward than a direct and clear trajectory,even with the same number of correct answers.
In this paper, we keep the same desired properties for the shaped reward function as the ones givenin ELLA:
the reward shaping should not change the optimal policy that prevails before the rewardshaping (policy invariance), and the reward should encourage sample efficient exploration based onauxiliary-objectives.
Using intrinsic rewards, we modify the global reward from r to r ′ through apolicy invariant transformation [26], which ensure that the new policy π ∗ is optimal for both M and
M ′ = ( S , A , T , G ′ , γ ).To do so, we ensure that only successful trajectories get the same return withor without the reward shaping by substracting the shaped reward at the final time step N of a successfultrajectory: r ′ N = r N − (cid:80) t ∈ T S γ t − N r ′ t with r ′ t = λ (cid:80) q ∈Q t (cid:80) ˜
a ∗ is correct answer QA (˜ a ∗ | q, τ t ) , where
T S is the set of time steps where a bonus is applied.
r t is the reward given at time step t , in the caseof sparse reward studied here r t ̸ = 0 only if t corresponds to the last step of a successful trajectory.
In ELLA Mirchandani et al.
[23] the authors prove that this transformation is policy invariant.
Thus,as long as the policy produces unsuccessful trajectories, the agent is guided by the shaping reward.
Then once it has learn to successfully complete an instruction, the shaping reward is substracted atthe last step and the agent improves using only the extrinsic reward.
Further details are provided inthe appendix.
The QA system is pre-trained with successful trajectories, which prevents reward hacking.
Indeed, ifthe agent individually completes the auxiliary objectives without a meaningful trajectory, the QAdoes not consider the trajectory meaningful and answers: "no_answer", preventing the agent to getrewarded.
Using the above notations and concepts, we can define a metric to measure the adequacy of a trajectoryto a goal, that corresponds to our cumulative intrinsic reward over a trajectory τ of length N , up to aλ factor:
[Equation]
Algorithm Our algorithmic procedure is given in Algorithm 1.
At the beginning of an episode, the
QG takes the goal and returns a set of questions related to it.
Then the QA module is applied at eachstep over the active set of questions Q t .
When a question is answered correctly, the shaped rewardfunction returns a bonus λ QA (˜ a ∗ | q, τ t ) , where λ is a scaling factor, to the agent and the answeredquestion is removed from the set of active questions.
: θ 0 initial policy parameters, λ bonus scaling factor,
ENV the environment and OPTIMISE an RL optimiser for k=0,..., n step do ω g , o 0 , done 0 ← ENV .reset ()
Q 0 = { q 1 , ..., q k } ← QG k ( ω g ) t ←while done t not True do a t ← π θ
( o t ) o t +1 , r t , done t +1 ← ENV ( a t ) r ′ t , Q t +1 ← QA_SHAPE ( Q t , τ t , r t ) if done t +1 is True then
N ← t r ′ N ← NEUTRALISE ( r ′ 1: N ) end if end while
Update θ k +1 ← OPTIMISE( r ′ 1: N ) end for function QA_SHAPE( Q t , τ t , r t ) for q in Q t do ˜ a ∗ ← QA ( q, τ t ) if ˜ a ∗ is correct answer to q then r ′ t =
r t + λ QA
(˜ a ∗ | q, τ t )
Q = Q \ { q } end if end for return r ′ t , Q end function function NEUTRALISE( r ′ 1: N ) r ′
N ← r ′ N − (cid:80) t ∈ T S γ t − N r ′ t return r ′ N end function
Then we tune λ to ensure that no unsuccessful trajectories can get more reward than a successful onefrom the optimal policy π ∗ .
The higher bound for the shaped reward of unsuccessful trajectories isλ k and the lower bound for the reward of a successful trajectory is γ H r H , where k is the number ofquestions generated by the QG and H is the maximum length of an episode.
Thus we obtain
[Equation]
Note that by making a less conservative hypothesis, i.e. assuming in the worst case the successfultrajectory takes N<H steps, we could obtain a higher λ leading to faster learning [23].
4.2 A particular instance of the method in the BabyAI framework
We now explain how to adapt the EAGER method to train an RL agent in BabyAI, a languageconditioned environment where the agent has a limited number of steps to complete a language goal.
In this environment the agent receives a reward if and only if it finishes the task successfully.
The BabyAI benchmark contains tasks with highly expressive language instructions e.g Put the redbox next to the green key , ...).
Thus they are rich enough to generate questions by masking words.
In practice, we mask nouns and adjectives: this form of QG is very simple and can be automatedusing standard NLP techniques, thus it does not require expert knowledge.
For instance, for the goal Put the red ball next to the blue box , using the token «question» as a mask we generate 4 questionsamong which Put the «question» ball next to the blue box .
The environments in our experiments are partially observable.
Thus, our agent takes sequences ofobservations ( o 1 , o 2 , ..., o t ) as inputs of a recurrent network [13].
Experiments
5.1 Experimental settings
We use the BabyAI [7] platform to run our experiments.
This platform relies on a gridworldenvironment (MiniGrid) to generate a set of complex instructions-following environments.
It hasbeen specifically designed for research on grounded language learning and related sample efficiencyproblems.
The gridworld environment is populated with several entities: the agent, boxes, balls,doors, and keys of 6 different colors.
These entities are placed in rooms of 8 × 8 tiles that areconnected by doors that could be locked or closed.
The agent can do 6 primitive navigation actionssuch as forward , toggle , pick up to solve the language instruction (for instance Pick up thered box ).
It only has access to partial observations of its environment inside which irrelevant objectsare randomly added.
Moreover, the observations are in a symbolic space using a compact encoding,with 3 input values per grid cell, 2 8 × 8 × 3 values in total.
When the agent completes the task after
N steps, it receives the reward r N = 1 − 0 .
9 NH , where H is the maximum number of steps.
Duringtraining, all rewards are scaled up by a factor of 20 to ensure a good propagation of the rewards.
If theagent fails, the reward is 0 .
We focus our tests on tasks of varying complexity: PutNextTo, Unlockand Sequence.
The task can also take place in one room local or two rooms Medium .
To train the QA module through supervised learning, we build a dataset of example trajectoriesassociated to language goals using a bot provided in BabyAI, then we generate related questions andanswers.
To obtain a QA that can operate on various tasks, we use a mix of the PutNextTo, PickUp,
Open, and Sequence tasks for generating training trajectories.
This dataset is only used to train the
QA, not to bootstrap policy learning.
During training, we give as input of the QA the full trajectory— the list of observations and actions — and all the questions generated by the QG and we use thecross-entropy loss over its output distribution to update it.
To train the QA to answer: "no_answer"and prevent it from guessing the answer by chance, we randomly associate some trajectories andquestions from unrelated goals.
For instance, we associate the trajectory from the goal: take the redbox next to the blue ball and associate it with the question: take the red «question» next to the bluekey generated from the goal take the red box next to the blue key .
The QA must learn to pay attentionto all the details of the question.
It has to see that the trajectory describes an agent taking the red box but placing it next to an object other than the one in the question.
Thus the good answer to thequestion is no_answer and not box .
Moreover, we empirically show in Section 5.4 that the QA is more efficient when it learns from abroad distribution of trajectories for similar tasks.
The intuition behind such behaviour is that a QAtrained on a narrow distribution of successful bot-generated trajectories will not recognise the noisytrajectories of the agent when it starts training.
Thus the QA will too often answer "no_answer",resulting in no intrinsic reward and hurting the reward shaping efficiency.
To produce a widerdistribution using the given procedural bot, we force the bot to take with a certain probability arandom action at each time step and only keep successful noisy trajectories as training examples.
More details on the QA pre-training are given in Appendix B.
All the tasks we use for training
the
QA and the agent are summarised with examples in Appendix A.
To evaluate our reward shaping framework, we use the Proximal Policy Optimization ( PPO ) algorithm[30], but our reward shaping method is algorithm agnostic.
We compare our framework to PPOwithout reward shaping, ELLA and RIDE.
RIDE [28] is an exploration method that does not uselanguage and addresses sparse reward problem by rewarding impactful change in state.
We use Nvidia Tesla V100 with 10 cores to train our model and we use 4 seeds in each experiment.
Figure 2 presents learning curves for EAGER, ELLA and RIDE across 4 environments.
5.2 How does EAGER perform when sparsity increases?
In the PutNextTo and Unlock tasks, EAGER obtains results better than ELLA (SOTA in BabyAI)without using expert knowledge.
It also performs significantly better than RIDE for the tasks PutNextand Sequence and slightly better for Unlock.
The better performance of EAGER with respect to
RIDE is not surprising as the EAGER agent receives some indications based on example trajectoriesthrough the QA module.
For Unlock-Medium, EAGER overcomes a bottleneck.
The general goal being Open the «colour»door , the agent has to first pick up the key of the corresponding colour before reaching the doorto open it.
ELLA rewards picking up keys, via the PICK low-level instruction chosen via expertknowledge.
EAGER reaches better performance (see the statistical test in Appendix C) without theneed for expert knowledge.
Moreover, although this is not its main purpose, EAGER gets a similar orbetter sample efficiency for most tasks.
5.3 How does EAGER perform with a sequence of tasks under a temporal constraint?
The Sequence task adds a temporal constraint by chaining two tasks using ’before’ or ’after’ togetherwith a high number of instructions (over 1.5M instructions in comparison with PutNext-Medium with1440 instructions).
Moreover both EAGER and ELLA decompose the goal into auxiliary objectives.
This decomposition does not retain the temporal constraint, there is no notion of doing one auxiliaryobjective before another.
Our tests show that EAGER retains strong performance, doing better than RIDE and ELLA.
Theslow progress of EAGER at the beginning can be attributed to the time when the agent is not goodenough to efficiently trigger an intrinsic reward signal from the QA module.
Indeed, at the beginning,trajectories are noisy and it is more difficult for the QA to exploit a trajectory with more rooms.
As aresult, it over-responds "no_answer" leading to a lesser intrinsic reward.
5.4 Is EAGER robust to QA performance?
At first glance, the reliability of the QA looks crucial to our method.
However, the QA could bedifficult to train in some environments e.g. if you want the QA to learn to answer in a large setof answers from a small number of example trajectories.
This is why we tested the robustness ofour method using the PutNextTo-Local task with two metrics: the success rate SR of the QA afterpre-training and the distribution of example trajectories.
For the former, we take the same QA atdifferent training epochs and we determine its SR over a test dataset, then we train the agent usingthe reward shaping provided by this QA.
Figure 3 shows the robustness of our method with agentsthat display similar training curves as soon as SR > 0 .
As explained in Section 5.1, we added noise to trajectories generated by the bot to compensate fora too narrow trajectory distribution.
Figure 4 shows training curves for two environments for QAtrained on wide trajectory distribution (WD) and narrow distribution (ND).
The reward shapingmethod trained on (WD) learns faster because they efficiently reward the agent early in training.
Conclusion
In this work, we have proposed to leverage the abstraction and generalisation properties of language tobuild an automatic reward shaping method in the context of long horizon and sparse reward tasks.
Ourlearning agent generates its own questions from the goal and rewards itself for correctly answeringthem, resulting in an efficient curriculum over auxiliary objectives.
This is to be contrasted with
ELLA [23] where expert knowledge is required for choosing auxiliary objectives.
Besides, we do notcall upon an oracle for getting linguistic description of environment states as in [25].
Limitations and Future Work EAGER assumes the QA system was pre-trained using a preexisting set of example trajectories.
Next steps will consist in investigating how to remove thislimitation, e.g. by implementing autotelic strategies based on QG/QA learned online.
Besides, inthis work we tested our method on BabyAI, a 2D environment with synthetic language.
In the future,we would like to consider a more complex language, generating more complex questions than theone obtained by masking, and testing our method on more realistic environments with true humaninstructions, as in the ALFRED dataset [32].
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See the Method and Experiments Section 4 and 5.
(b) Did you describe the limitations of your work?
[Yes] See the Limitations paragraph ofin Section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We do
not foresee major negative societal impacts of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We do believe our work conforms to them.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We give the
URL in the introduction.
(b) Did you specify all training details (e.g., data splits, hyperparameters, how they werechosen)?
[Yes] See the Section 5 and the Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We ran our main experiments with multiple random seeds
and plotted curves with standard deviations.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the end of section 5.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use open-sourceenvironments, cited in the code and in the paper text.(b)
Did you mention the license of the assets?
[Yes] We inherit the license in our attachedcode repository.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]We include the code of our approach
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]