Original: Language-Goal Imagination to Foster Creative Exploration in Deep RL
Revised: Language-Goal Imagination to Foster Creative Exploration in Deep RL

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: Developmental machine learning studies how artiﬁcial agents can model the way children learn open-ended repertoires of skills.
Revised: Developmental machine learning studies how artiﬁcial agents can model the way children learn open-ended repertoires of skills.

Original: Children are known to use language and its compositionality as a tool to imagine descriptions of outcomes they never experienced before and target them as goals during play.
Revised: Children are known to use language and its compositionality as a tool to imagine descriptions of outcomes they never experienced before and target them as goals during play.

Original: We introduce IMAGINE , an intrinsically motivated deep RL architecture that models this ability.
Revised: We introduce IMAGINE , an intrinsically motivated deep RL architecture that models this ability.

Original: Such imaginative agents, like children, beneﬁt from the guidance of a social peer who provides language descriptions.
Revised: Such imaginative agents, like children, beneﬁt from the guidance of a social peer who provides language descriptions.

Original: To take advantage of goal imagination, agents must be able to leverage these descriptions to interpret their imagined goals.
Revised: To take advantage of goal imagination, agents must be able to leverage these descriptions to interpret their imagined goals.

Original: This generalization is made possible by modularity: a decomposition between learned goal-achievement reward function and policy relying on deep sets, gated attention and object-centered representations.
Revised: This generalization is made possible by modularity: a decomposition between learned goal-achievement reward function and policy relying on deep sets, gated attention and object-centered representations.

Original: We introduce the
Revised: We introduce the

Original: Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity.
Revised: Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity.

Original: 1. Introduction
Revised: 1. Introduction

Original: Building autonomous machines that can discover and learn a variety of skills is a long-standing goal in Artiﬁcial Intelligence.
Revised: Building autonomous machines that can discover and learn a variety of skills is a long-standing goal in Artiﬁcial Intelligence.

Original: In this quest, we can draw inspiration from the way children learn and explore open-ended environments (Asada et al., 2009).
Revised: In this quest, we can draw inspiration from the way children learn and explore open-ended environments (Asada et al., 2009).

Original: In such exploration, intrinsic motivations are key to trigger spontaneous exploration, for the mere purpose of experiencing novelty, surprise or learning progress (Kaplan & Oudeyer, 2007; Kidd & Hayden, 2015).
Revised: In such exploration, intrinsic motivations are key to trigger spontaneous exploration, for the mere purpose of experiencing novelty, surprise or learning progress (Kaplan & Oudeyer, 2007; Kidd & Hayden, 2015).

Original: Children also leverage the properties of language to assimilate years of experience embedded in their culture, in only a few years (Tomasello, 1999; Bruner, 1991).
Revised: Children also leverage the properties of language to assimilate years of experience embedded in their culture, in only a few years (Tomasello, 1999; Bruner, 1991).

Original: As they discover language, their goal-driven exploration changes. 
Revised: As they discover language, their goal-driven exploration changes. Accepted to the Workshop on Language in Reinforcement Learning

Original: Piaget (1926) identiﬁed a form of egocentric speech where children narrate their ongoing activities.
Revised: Piaget (1926) identiﬁed a form of egocentric speech where children narrate their ongoing activities.

Original: Later, Vygotsky (1978) showed that they can generate novel plans and goals by using the expressive generative properties of language.
Revised: Later, Vygotsky (1978) showed that they can generate novel plans and goals by using the expressive generative properties of language.

Original: This paper presents I ntrinsic M otivations A nd G oal
Revised: This paper presents I ntrinsic M otivations A nd G oal

Original: IN vention for E xploration ( IMAGINE ): a learning architecture which leverages natural language ( NL ) interactions with a descriptive social partner ( SP ) to explore procedurallygenerated scenes and interact with objects.
Revised: IN vention for E xploration ( IMAGINE ): a learning architecture which leverages natural language ( NL ) interactions with a descriptive social partner ( SP ) to explore procedurallygenerated scenes and interact with objects.

Original: IMAGINE discovers meaningful environment interactions through its own exploration (Figure 1a) and episode-level NL descriptions provided by SP (1b).
Revised: IMAGINE discovers meaningful environment interactions through its own exploration (Figure 1a) and episode-level NL descriptions provided by SP (1b).

Original: These descriptions are turned into targetable goals by the agent (1c).
Revised: These descriptions are turned into targetable goals by the agent (1c).

Original: It learns to represent them by jointly training a language encoder mapping NL to goal embeddings and a goal-achievement reward function (1d).
Revised: It learns to represent them by jointly training a language encoder mapping NL to goal embeddings and a goal-achievement reward function (1d).

Original: The latter provides training signals for policy learning (ticks in Figure 1d-e).
Revised: The latter provides training signals for policy learning (ticks in Figure 1d-e).

Original: More importantly, IMAGINE can invent new goals by composing known ones, and trains on them autonomously via internal signals (1f).
Revised: More importantly, IMAGINE can invent new goals by composing known ones, and trains on them autonomously via internal signals (1f).

Original: Related work.
Revised: Related work.

Original: The idea that language understanding is grounded in one’s experience of the world and should not be secluded from the perceptual and motor systems has a long history in Cognitive Science (Glenberg & Kaschak, 2002;
Revised: The idea that language understanding is grounded in one’s experience of the world and should not be secluded from the perceptual and motor systems has a long history in Cognitive Science (Glenberg & Kaschak, 2002;

Original: Zwaan & Madden, 2005).
Revised: Zwaan & Madden, 2005).

Original: This vision was transposed to intelligent systems (Steels, 2006; McClelland et al., 2019), applied to human-machine interaction (Dominey, 2005; Madden et al., 2010) and recently to deep RL via frameworks such as BabyAI (Chevalier-Boisvert et al., 2019).
Revised: This vision was transposed to intelligent systems (Steels, 2006; McClelland et al., 2019), applied to human-machine interaction (Dominey, 2005; Madden et al., 2010) and recently to deep RL via frameworks such as BabyAI (Chevalier-Boisvert et al., 2019).

Original: In their review of RL algorithms informed by NL , Luketina et al.
Revised: In their review of RL algorithms informed by NL , Luketina et al.

Original: (2019) distinguish between language-conditional problems where language is required to solve the task and language-assisted problems where language is a supplementary help.
Revised: (2019) distinguish between language-conditional problems where language is required to solve the task and language-assisted problems where language is a supplementary help.

Original: In the ﬁrst category, most works propose instruction-following agents (R. K. Branavan et al., 2010;
Revised: In the ﬁrst category, most works propose instruction-following agents (R. K. Branavan et al., 2010;

Original: Chen & Mooney, 2011; Bahdanau et al., 2019; Co-Reyes et al., 2018; Jiang et al., 2019; Goyal et al., 2019; Cideron et al., 2019).
Revised: Chen & Mooney, 2011; Bahdanau et al., 2019; Co-Reyes et al., 2018; Jiang et al., 2019; Goyal et al., 2019; Cideron et al., 2019).

Original: Although our system is language-conditioned , it is not language-instructed : it is never given any instruction or reward but sets its own goals and learns its own internal reward function.
Revised: Although our system is language-conditioned , it is not language-instructed : it is never given any instruction or reward but sets its own goals and learns its own internal reward function.

Original: Bahdanau et al.
Revised: Bahdanau et al.

Original: and Fu et al.
Revised: and Fu et al.

Original: (2019) also learn a reward function but require extensive expert knowledge (expert dataset and known environment dynamics respectively), whereas our agent uses experience generated by its own exploration.
Revised: (2019) also learn a reward function but require extensive expert knowledge (expert dataset and known environment dynamics respectively), whereas our agent uses experience generated by its own exploration.

Original: Imagining goals by composing known ones only works in association with systematic generalization (Bahdanau et al., 2018; Hill et al., 2019): generalizations of the type grow any animal + grasp any plant → grow any plant .
Revised: Imagining goals by composing known ones only works in association with systematic generalization (Bahdanau et al., 2018; Hill et al., 2019): generalizations of the type grow any animal + grasp any plant → grow any plant .

Original: These were found to emerge in instruction-following agents, including generalizations to new combinations of motor predicates, object colors and shapes (Hermann et al., 2017;
Revised: These were found to emerge in instruction-following agents, including generalizations to new combinations of motor predicates, object colors and shapes (Hermann et al., 2017;

Original: Hill et al., 2019; Bahdanau et al., 2019).
Revised: Hill et al., 2019; Bahdanau et al., 2019).

Original: Because systematic generalization can only occur when objects share common attributes (e.g. type, color), we represent objects as singleslot object ﬁles (Green & Quilty-Dunn, 2017): separate entities characterized by shared attributes.
Revised: Because systematic generalization can only occur when objects share common attributes (e.g. type, color), we represent objects as singleslot object ﬁles (Green & Quilty-Dunn, 2017): separate entities characterized by shared attributes.

Original: We introduce a new object-centered inductive bias: object-based modular architectures based on Deep Sets (Zaheer et al., 2017).
Revised: We introduce a new object-centered inductive bias: object-based modular architectures based on Deep Sets (Zaheer et al., 2017).

Original: Contributions.
Revised: Contributions.

Original: We introduce goal imagination using language compositionality to drive exploration of intrisically motivated agents and a modular architecture to enable it.
Revised: We introduce goal imagination using language compositionality to drive exploration of intrisically motivated agents and a modular architecture to enable it.

Original: 2. Problem Deﬁnition
Revised: 2. Problem Deﬁnition

Original: Open-ended learning environment.
Revised: Open-ended learning environment.

Original: We consider a setup where agents evolve in an environment ﬁlled with objects and without prior on the set of possible interactions.
Revised: We consider a setup where agents evolve in an environment ﬁlled with objects and without prior on the set of possible interactions.

Original: An agent decides what to learn by setting its own goals, and has no access to external rewards.
Revised: An agent decides what to learn by setting its own goals, and has no access to external rewards.

Original: However, to allow the agent to learn relevant skills, a social partner ( SP ) watches the scene to guide the agent.
Revised: However, to allow the agent to learn relevant skills, a social partner ( SP ) watches the scene to guide the agent.

Original: Following a developmental approach (Asada et al., 2009), we propose a hard-coded surrogate SP modelling important aspects of human development:
Revised: Following a developmental approach (Asada et al., 2009), we propose a hard-coded surrogate SP modelling important aspects of human development:

Original: • At the beginning of each episode, the agent chooses a goal by formulating a sentence.
Revised: • At the beginning of each episode, the agent chooses a goal by formulating a sentence.

Original: SP then provides agents with optimal learning opportunities by organizing the scene with: 1) the required objects to reach the goal (relevant scene) 2) procedurally-generated distracting objects (new scene and providing further discovery opportunities).
Revised: SP then provides agents with optimal learning opportunities by organizing the scene with: 1) the required objects to reach the goal (relevant scene) 2) procedurally-generated distracting objects (new scene and providing further discovery opportunities).

Original: This is close to a developmental scaffolding modelling the Zone of Proximal Development (ZPD) introduced by Vygotsky to describe infant-parent learning dynamics (Vygotsky, 1978).
Revised: This is close to a developmental scaffolding modelling the Zone of Proximal Development (ZPD) introduced by Vygotsky to describe infant-parent learning dynamics (Vygotsky, 1978).

Original: • At the end of each episode, SP utters a set of sentences describing achieved and meaningful outcomes (except sentences from a test set).
Revised: • At the end of each episode, SP utters a set of sentences describing achieved and meaningful outcomes (except sentences from a test set).

Original: Linguistic guidance given through descriptions are a key component of how parents "teach" language to infants.
Revised: Linguistic guidance given through descriptions are a key component of how parents "teach" language to infants.

Original: It contrasts with instruction following (providing a linguistic command and then a reward), rarely seen in real parent-child interactions (Tomasello, 2009; Bornstein et al., 1992).
Revised: It contrasts with instruction following (providing a linguistic command and then a reward), rarely seen in real parent-child interactions (Tomasello, 2009; Bornstein et al.

Original: Learning objective.
Revised: Learning objective.

Original: This paper investigates how goal imagination can lead agents to creatively explore their environment and discover interesting interactions with objects around.
Revised: This paper investigates how goal imagination can lead agents to creatively explore their environment and discover interesting interactions with objects around.

Original: In this quest, SP guides agents towards a set of interesting outcomes by uttering NL descriptions.
Revised: In this quest, SP guides agents towards a set of interesting outcomes by uttering NL descriptions.

Original: Through compositional recombinations of these sentences, goal imagination aims to drive creative exploration, to push agents to discover outcomes beyond the set of outcomes known by
Revised: Through compositional recombinations of these sentences, goal imagination aims to drive creative exploration, to push agents to discover outcomes beyond the set of outcomes known by

Original: SP .
Revised: SP .

Original: We evaluate this desired behavior by two metrics: 1) the generalization of the policy to new language goals unknown to SP , (test set deﬁned in Supp.
Revised: We evaluate this desired behavior by two metrics: 1) the generalization of the policy to new language goals unknown to SP , (test set deﬁned in Supp.

Original: Section 1) ; 2) goal-oriented exploration metrics.
Revised: Section 7) ; 2) goal-oriented exploration metrics.

Original: We measure generalization for each goal as the success rate SR over 30 episodes and report SR the average over goals.
Revised: We measure generalization for each goal as the success rate SR over 30 episodes and report SR the average over goals.

Original: We evaluate exploration with the interesting interaction count ( I  C ).
Revised: We evaluate exploration with the interesting interaction count ( I  C ).

Original: Given a set of interaction,
Revised: Given a set of interaction,

Original: I  C measures the penchant of agents to explore interactions of this set, (details in Supp.
Revised: I  C measures the penchant of agents to explore interactions of this set, (details in Supp.

Original: Section 2).
Revised: Section 8).

Original: We report means and std over 10 seeds and statistical signiﬁcance using a two-tail Welch’s t-test at level α =0 .
Revised: We report means and std over 10 seeds and statistical signiﬁcance using a two-tail Welch’s t-test at level α =0 .

Original: (Colas et al., 2019).
Revised: (Colas et al., 2019b).

Original: 3. Methods 3.1. The Playground environment
Revised: 3. Methods 3.1. The Playground environment

Original: We introduce Playground , a simple environment designed to study the impact of goal imagination on exploration and generalization by disentangling it from the problems of perception and fully-blown NL understanding.
Revised: We introduce Playground , a simple environment designed to study the impact of goal imagination on exploration and generalization by disentangling it from the problems of perception and fully-blown NL understanding.

Original: Playground is a continuous 2 D world, with procedurally-generated scenes containing N =3 objects, from 32 different object types ( e.g. dog, cactus, sofa, water, etc. ), organized into 5 categories ( animals, furniture, plants, etc ), see Figure 1.
Revised: Playground is a continuous 2 D world, with procedurally-generated scenes containing N =3 objects, from 32 different object types ( e.g. dog, cactus, sofa, water, etc. ), organized into 5 categories ( animals, furniture, plants, etc ), see Figure 1.

Original: Agent perception and embodiment.
Revised: Agent perception and embodiment.

Original: Agents have access to state vectors describing the scene: the agent’s body and the objects.
Revised: Agents have access to state vectors describing the scene: the agent’s body and the objects.

Original: Each object is represented by a set of features describing its type, position, color, size and whether it is grasped.
Revised: Each object is represented by a set of features describing its type, position, color, size and whether it is grasped.

Original: The agent can perform bounded translations in the 2 D plane, grasp and release objects with its gripper.
Revised: The agent can perform bounded translations in the 2 D plane, grasp and release objects with its gripper.

Original: It can make animals and plants grow by bringing them the right supply (food or water for animals, water for plants).
Revised: It can make animals and plants grow by bringing them the right supply (food or water for animals, water for plants).

Original: Goal and description A simple grammar generates the descriptions of the 256 achievable goals, such as go bottom left, grasp red cat or grow green animal .
Revised: Goal and description A simple grammar generates the descriptions of the 256 achievable goals, such as go bottom left, grasp red cat or grow green animal .

Original: Supp.
Revised: Supp.

Original: Sectioncompletely details the environment and the grammar
Revised: Sectioncompletely details the environment and the grammar

Original: 3.2. The IMAGINE Architecture
Revised: 3.2. The IMAGINE Architecture

Original: IMAGINE agents build a repertoire of goals and train two internal models: 1) a goal-achievement reward function R to predict whether a given description matches a behavioral trajectory; 2) a policy Π to achieve behavioral trajectories matching descriptions.
Revised: IMAGINE agents build a repertoire of goals and train two internal models: 1) a goal-achievement reward function R to predict whether a given description matches a behavioral trajectory; 2) a policy Π to achieve behavioral trajectories matching descriptions.

Original: The architecture is presented in
Revised: The architecture is presented in

Original: Figure 2 and follows this logic:
Revised: Figure 2 and follows this logic:

Original: The Goal Generator samples a target goal g target from known and imagined goals ( G known ∪G im ) .
Revised: The Goal Generator samples a target goal g target from known and imagined goals ( G known ∪G im ) .

Original: The agent ( RL Agent ) interacts with the environment using its policy Π conditioned on g target .
Revised: The agent ( RL Agent ) interacts with the environment using its policy Π conditioned on g target .

Original: State-action trajectories are stored in mem (Π) .
Revised: State-action trajectories are stored in mem (Π) .

Original: SP ’s descriptions of the last state are considered as potential goals G SP ( s T ) =
Revised: SP ’s descriptions of the last state are considered as potential goals G SP ( s T ) =

Original: D SP ( s T ) .
Revised: D SP ( s T ) .

Original: mem ( R ) stores positive pairs ( s T , G SP ( s T )) and infers negative pairs ( s T , G known \G SP ( s T )) .
Revised: mem ( R ) stores positive pairs ( s T , G SP ( s T )) and infers negative pairs ( s T , G known \G SP ( s T )) .

Original: The agent then updates: • Goal Gen. : G known ← G known ∪G SP ( s T ) and G im ← Imagination ( G known ) .
Revised: The agent then updates: • Goal Gen. : G known ← G known ∪G SP ( s T ) and G im ← Imagination ( G known ) .

Original: • Language Encoder
Revised: • Language Encoder

Original: ( L e ) and Reward Function ( R ) are updated using data from mem ( R ) .
Revised: ( L e ) and Reward Function ( R ) are updated using data from mem ( R ) .

Original: • RL agent : We sample a batch of state-action transitions ( s , a , s (cid:48) ) from mem (Π) .
Revised: • RL agent : We sample a batch of state-action transitions ( s , a , s (cid:48) ) from mem (Π) .

Original: Then, we use Hindsight Replay and R to bias the selection of substitute goals to train on ( g s ) and compute the associated rewards ( s , a , s (cid:48) , g s , r ) .
Revised: Then, we use Hindsight Replay and R to bias the selection of substitute goals to train on ( g s ) and compute the associated rewards ( s , a , s (cid:48) , g s , r ) .

Original: Finally, the policy and critic are trained via RL.
Revised: Finally, the policy and critic are trained via RL.

Original:  Goal generator.
Revised: [Figure] Goal generator.

Original: It is a generative model of NL goals.It
Revised: It is a generative model of NL goals.

Original: [Figure] Hindsight [Figure] generates target goals g target for data collection and substitute goals g s for hindsight replay.
Revised: It generates target goals g target for data collection and substitute goals g s for hindsight replay.

Original: When goal imagination is disabled, the goal generator samples uniformly from the set of known goals G known , sampling random vectors if empty.
Revised: When goal imagination is disabled, the goal generator samples uniformly from the set of known goals G known , sampling random vectors if empty.

Original: When enabled, it samples with equal probability from G known and G im (set of imagined goals).
Revised: When enabled, it samples with equal probability from G known and G im (set of imagined goals).

Original: G im is generated using a mechanism grounded in construction grammar that leverages the compositionality of language to imagine new goals from G known .
Revised: G im is generated using a mechanism grounded in construction grammar that leverages the compositionality of language to imagine new goals from G known .

Original: The heuristic consists in computing sets of equivalent words : words that appear in two sentences that only differ by one word.
Revised: The heuristic consists in computing sets of equivalent words : words that appear in two sentences that only differ by one word.

Original: For example, from grasp red lion and grow red lion , grasp and grow can be considered equivalent and from grasp green tree one can imagine a new goal grow green tree (see Figure 1f).
Revised: For example, from grasp red lion and grow red lion , grasp and grow can be considered equivalent and from grasp green tree one can imagine a new goal grow green tree (see Figure 1f).

Original: Among them, some are meaningless, some are syntactically correct but infeasible (e.g. grow red lamp ) and some belong to G test , or even to G train before they are encountered by the agent and described by SP .
Revised: Among them, some are meaningless, some are syntactically correct but infeasible (e.g. grow red lamp ) and some belong to G test , or even to G train before they are encountered by the agent and described by SP .

Original: The pseudo-code and all imaginable goals are provided in Supp.
Revised: The pseudo-code and all imaginable goals are provided in Supp.

Original: Section 3.
Revised: Section 9.

Original: Object-centered modular architectures.
Revised: Object-centered modular architectures.

Original: The goalachievement reward function, policy and critic leverage novel modular-attention ( MA ) architectures based on Deep
Revised: The goalachievement reward function, policy and critic leverage novel modular-attention ( MA ) architectures based on Deep

Original: Sets (Zaheer et al., 2017), gated attention mechanisms (Chaplot et al., 2017) and object-centered representations.
Revised: Sets (Zaheer et al., 2017), gated attention mechanisms (Chaplot et al., 2017) and object-centered representations.

Original: The idea is to ensure efﬁcient skill transfer between objects, no matter their position in the state vector.
Revised: The idea is to ensure efﬁcient skill transfer between objects, no matter their position in the state vector.

Original: This is done through the combined use of a shared neural network that encodes object-speciﬁc features and a permutation-invariant function to aggregate the resulting latent encodings.
Revised: This is done through the combined use of a shared neural network that encodes object-speciﬁc features and a permutation-invariant function to aggregate the resulting latent encodings.

Original: The shared network independently encodes, for each object, an affordance between this object (object observations), the agent (body observations) and its current goal. 
Revised: The shared network independently encodes, for each object, an affordance between this object (object observations), the agent (body observations) and its current goal. The goal embedding, generated by the language encoder L e , is ﬁrst cast [Figure]

Original: The goal embedding, generated by the language encoder L e , is ﬁrst cast into an attention vector in [0 , 1] , then fused with the concatenation of object and body features via an Hadamard product (gated-attention (Chaplot et al., 2017)). 
Revised: into an attention vector in [0 , 1] , then fused with the concatenation of object and body features via an Hadamard product (gated-attention (Chaplot et al., 2017)). The resulting objectspeciﬁc encodings are aggregated by a permutation-invariant function and mapped to the desired output via a ﬁnal network (e.g. into actions or action-values). Supp. Section 11. provides visual representations and details about the reward function, policy and critic based on this architecture.

Original: The resulting object-
Revised: As IMAGINE achieve near perfect generalizations on training goals: SR = 0 . 95 ± 0 . 05 , we focus on the testing set performance.

Revised: Supp.

Revised: Sections 8 to 10 provide more results.

Revised: Global generalization performance.

Revised: Figure 3a shows SR on the set of testing goals, when the agent starts imagining new goals early (after 6 · 10 3 episodes), half-way (after 48 · 10 3 episodes) or when not allowed to do so.

Revised: Imagining goals leads to signiﬁcant improvements in generalization.

Revised: A particular generalization: growing plants.

Original: [Figure]
Revised: Agents learn to grow animals from SP ’s descriptions, but are never told they could grow plants. When evaluated ofﬂine on the growing-plants goals before goal imagination, agents’ policies perform a sensible zero-shot generalization and bring them water or food with equal probability, as they would do for animals (Figure 3b, left). As they start imagining these goals, their behavior adapts (Figure 3b, right). A reward function with good zero-shot abilities only provides positive rewards when the agent brings water.

Revised: The policy slowly adapts to this internal reward signal and pushes agents to bring more water.

Original: speciﬁc encodings are aggregated by a permutation-invariant function and mapped to  the desired output via a ﬁnal network (e.g. into actions or action-values).
Revised: We name this: behavioral adaptation . Exploration. Figure 3c presents the I  C metric computed on the set of interactions related to G test and demonstrates the exploration boost triggered by goal imagination.

Original: Supp.
Revised: Supp.

Revised: Section 8 additional results on exploration.

Revised: Modularity is Essential to Goal Imagination.

Original: Section 5. provides visual representations and details about the  reward function, policy and critic based on this architecture.
Revised: Flat architectures ( FA ) consider the whole scene at once. We compared policies based on MA and in Table 4. Both used MA reward function as the FA reward function showed poor performance on G train . MA shows stronger generalization and is the only architecture allowing an additional boost

Original: 4. Experiments and Results
Revised: [Table] with goal imagination. Only MA policy can leverage the novel reward signals coming from imagined goals and show behavioral adaptation . 5. Discussion and Conclusion

Original: As IMAGINE achieve near perfect generalizations on training goals: SR = 0 .
Revised: IMAGINE is an autonomous learning architecture that leverages NL interactions with a social partner. IMAGINE sets its own goals and builds behavioral repertoires without external rewards. As such, it is distinct from traditional instructionfollowing RL agents.

Original: 95 ± 0 . 
Revised: This is done through the joint training of a language encoder for goal representation and a goalachievement reward function to generate internal rewards. Our proposed modular architectures with gated-attention enable efﬁcient out-of-distribution generalization of the reward function and policy. The ability to imagine new goals by composing known ones leads to further improvements over initial generalization abilities and fosters exploration beyond the set of interactions relevant to SP . Our agent even tries to grow pieces of furniture with supplies, a behavior that can echo the way a child may try to feed his doll.

Original: 05 , we focus on the testing set performance.
Revised: IMAGINE does not need externally-provided rewards but learns which behaviors are interesting from language-based interactions with SP . In contrast with hand-crafted reward functions, NL descriptions provide an easy way to guide machines towards relevant interactions. In addition, Section 10 shows that agents can learn to achieve goals from a relatively small number of descriptions, paving the way towards human-provided descriptions. 6. Ackowledgement

Original: Supp.
Revised: Cédric Colas and Tristan Karch are partly funded by the French Ministère des Armées - Direction Générale de l’Armement. Nicolas Lair is supported by ANRT/CIFRE contract No. 151575A20 from Cloud Temple Supplementary Material

Original: Sections 2 to 4 provide more results.
Revised: This supplementary material provides additional methods, results and discussion, as well as implementation details. • Section 7 gives a complete description of our setup and of the Playground environment. • Section 8 presents a focus on exploration and how it is inﬂuenced by goal imagination. • Section 9 presents a focus on the goal imagination mechanism we use for IMAGINE and identiﬁes key properties of imagination.

Original: Global generalization performance.
Revised: In Section 10, we investigate the case of more realistic feedback from SP . • Section 11 gives all necessary implementation details. 7. Complete Description of the Playground Environment and Its Language Environment description.

Original: Figure 3a shows SR on the set of testing goals, when the agent starts imagining new goals early (after 6 · 10 3 episodes), half-way (after 48 · 10 3 episodes) or when{+is a disc of diameter 
 .with an initial position (0 , 0) . Objects have sizes uniformly sampled from [0 . 2 , 0 . 3] and their initial positions are randomized so that they are+} not allowed to do so.
Revised: The environment is a 2 D square: [ − 1 . 2 , 1 . 2] 2 . The agent {+is a disc of diameter 
 .with an initial position (0 , 0) . Objects have sizes uniformly sampled from [0 . 2 , 0 . 3] and their initial positions are randomized so that they are+} not in contact with each other.

Original: Imagining goals leads to signiﬁcant improvements in generalization.
Revised: The agent has an action space of size 3 bounded in [ − 1 , 1] . The ﬁrst two actions control the agent’s continuous 2 D translation (bounded to 0 . 15 in any direction). The agent can grasp objects by getting in contact with them and closing its gripper (positive third action), unless it already has an object in hand.

Original: A particular generalization: growing plants.
Revised: Objects include 10 animals, 10 plants,pieces of furniture and 2 supplies. Admissible categories are animal, plant, furniture, supply and living_thing (animal or plant), see Figure 5. Objects are assigned a color attribute (red, blue or green). Their precise color is a continuous

Original: Agents learn to grow animals from SP ’s descriptions, but are never told they could grow plants.
Revised: RGB code uniformly sampled from RGB subspaces associated with their attribute color. Each scene contains 3 of these procedurally-generated objects (see paragraph about the Social Partner below). Agent perception.

Original: When evaluated ofﬂine on the growing-plants goals before goal imagination, agents’ policies perform a sensible zero-shot generalization and bring them water or food with equal probability, as they would do for animals (Figure 3b, left).
Revised: At time step t , we can deﬁne an observation o t as the concatenation of body observations ( 2 Dposition, gripper state) and objects’ features. These two types of features form affordances between the agent and the objects around. These affordances are necessary to understand the meaning of object interactions like grasp . The state s t used as input of the models is the concatenation of o t and ∆ o t =

Revised: o t − o 0 to provide a sense of time.

Original: As they start imagining these goals, their behavior adapts (Figure 3b, right).
Revised: This is required to acquire the understanding and behavior related to the grow predicate, as the agent needs to observe and produce a change in the object’s size. Social Partner. SP has two roles: • Scene organization :

Original: A reward function with good zero-shot abilities only provides positive rewards when the agent brings water. The policy slowly adapts to this internal reward signal and pushes agents to bring more water.
Revised: SP organize the scene according to the goal selected by the agent. When the agent selects a goal, it communicates it to SP . If the goal starts by the word grow , SP adds a procedurally-generated supply (water or food for animals, water for plants) of any size and color to the scene.

Original: We name this: behavioral adaptation .
Revised: If the goal contains an object (e.g. red cat ), SP adds a corresponding object to the scene (with a procedurally generated size and RGB color).

Original: Exploration.
Revised: Remaining objects are generated procedurally. As a result, the objects required to fulﬁll a goal are always present and the scene contains between 1 ( grow goals) and 3 ( go goals) random objects. Note that all objects are procedurally generated (random initial position, RGB color and size). • Scene description : SP provides NL descriptions of interesting outcomes experienced by the agent at the end of episodes.

Original: Figure 3c presents the I
Revised: By default, SP respects the 3 following properties: precision : descriptions are accurate, exhaustiveness : it provides all valid descriptions for each episode and full-presence : it is always available. Sectioninvestigates relaxations of the last two assumptions. It takes the ﬁnal state of an episode ( s T ) as input and returns matching NL descriptions: D SP ( s T ) ⊂D SP .

Original:  C metric computed on the set of interactions related to G test and demonstrates the exploration boost triggered by goal imagination.
Revised: When SP provides descriptions , the agent considers them as targetable goals . This mapping D SP →G train simply consists in removing the ﬁrst you token (e.g. turning you grasp red door into the goal grasp red door ). Given the set of previously discovered goals ( G known ) and new descriptions D SP ( s T ) , the agent infers the set of goals that were not achieved: G na ( s T ) = G known \ D SP ( s T ) , where \ indicates the complement.

Original: Supp.
Revised: Grammar. We now present the grammar that generates descriptions for the set of goals achievable in the Playground environment ( G A ) . Bold and { } refer to sets of words while italics refers to particular words:

Original: Section 2 additional results on exploration.
Revised: Go: (e.g. go bottom left) • go + zone 2. Grasp: (e g. grasp any animal) • grasp + color ∪ { any } + object type ∪ object category • grasp + any + color + thing 3. Grow: (e

Original: Modularity is Essential to Goal Imagination.
Revised: g. grow blue lion) • grow + color ∪ { any } + living thing ∪ { living_thing, animal, plant } • grow + any + color + thing Word sets are deﬁned by: • zone = { center, top, bottom, right, left, top left, top right, bottom left, bottom right } • color = { red, blue, green } • object type = living thing ∪ furniture ∪ supply • object category =

Original: Flat architectures ( FA ) consider the whole scene at once.
Revised: { living_thing , animal , plant , furniture , supply } • living thing = animal ∪ plant • animal = { dog, cat, chameleon, human, ﬂy, parrot, mouse, lion, pig, cow } • plant = { cactus, carnivorous, ﬂower, tree, bush, grass, algae, tea, rose, bonsai } • furniture = { door, chair, desk, lamp, table, cupboard, sink, window, sofa, carpet } • supply = { water, food } • predicate = { go, grasp, grow } Training and testing goals sets The set of achievable goals is partitioned into training ( G train ) and testing ( G test ) . Goals from G test are intended to evaluate the ability of our agent to explore the set of achievable outcomes beyond the set of outcomes described by SP .

Original: We compared policies based on MA and in Table 4.
Revised: G test maximizes the compound divergence with a null atom divergence with respect to G train : testing sentences (compounds) are out of the distribution of G train sentences, but their words (atoms) belong to the distribution of words in G train (Keysers et al., 2019). SP only provides descriptions from G train . Note that some goals might be syntactically valid but not achievable. This includes all goals of the form grow + color ∪ { any } + furniture ∪ { furniture } (e.g. grow red lamp ).

Original: Both used MA reward function as the FA reward function showed poor performance on G train .
Revised: Besides all goals related to the object ﬂower are unknown to the SP and cannot be uttered. The agent can encounter ﬂower but is never able to identify the object by its name. This will prevent the agent to imagine new goals about ﬂower object. IMAGINE Pseudo-Code.

Original: MA shows stronger generalization and is the only architecture allowing an additional boost
Revised: Algorithm 1 outlines the pseudo-code of our learning architecture. See Main Section 3.2 for high-level descriptions of each module and function. Algorithm

Original: MA *
Revised: 1: Input: env, SP 2: Initialize: L e , R , Π , mem ( R ) , mem (Π) , G known , G im # Random initializations for networks # empty sets for memories and goal sets 3: for e =1: N episodes do 4: if G known (cid:54) =

Original: FA
Revised: Ø then 5: sample g NL from G known ∪G im 6: g ← L e ( g NL ) 7: else 8: sample g from N (0 , I ) 9: s 0 ← env.reset() 10: for t =1: T do 11: a t ← π ( s t − 1 ,g ) 12: s t ← env.step( a t ) 13: mem Π .add ( s t − 1 ,a t ,s t ) 14: G SP ← SP .get_descriptions

Original: Im.
Revised: ( s T ) 15:

Original: 76 ± 0 .0 .
Revised: G known ←G known ∪ G SP 16: mem ( R ) .add ( s T , g NL ) for g NL in G SP 17: if goal imagination allowed then 18: G im ← Imagination ( G known ) # see Algorithm19:

Original: 15 ± 0 .
Revised: Batch Π ← ModularBatchGenerator ( mem (Π)) # Batch Π =

Original: No Im.
Revised: ( s,a,s (cid:48) ) } 20: Batch Π ← Hindsight ( Batch Π , R , G known , G im ) # Batch Π = { ( s,a,r,g,s (cid:48) ) } where r = R ( s,g ) 21: Π ← RL_Update (Batch Π ) 22: if e % reward_update_freq

Original: 51 ± 0 .0 .
Revised: = =0 then 23: Batch R ← ModularBatchGenerator ( mem ( R )) 24: L e , R← LE&RewardFunctionUpdate ( Batch R )

Original: 17 ± 0 .p-val 4 . 8 e0.
Revised: Category Object Type furniture animal plant living thing supply dog cat chameleon human fly cactus carnivorous flower tree bush grass aglae tea rose bonsai parrot mouse lion pig cow cupboard sink window sofa carpet door chair desk lamp table water food Non

Original: with goal imagination.
Revised: -imaginable goals

Original: Only MA policy can leverage the novel reward signals coming from imagined goals and show behavioral adaptation .
Revised: Grasp any ﬂower , Grasp blue ﬂower , Grasp green ﬂower , Grasp red ﬂower ,

Original: 5. Discussion and Conclusion
Revised: Grow any ﬂower , Grow blue ﬂower , Grow green ﬂower , Grow red ﬂower

Original: IMAGINE is an autonomous learning architecture that leverages NL interactions with a social partner.
Revised: Imaginable goals Grasp blue door , Grasp green dog , Grasp red tree , Grow green dog Grasp any animal , Grasp blue animal , Grasp green animal , Grasp red animal Grasp any ﬂy , Grasp blue ﬂy , Grasp green ﬂy , Grasp red ﬂy

Original: IMAGINE sets its own goals and builds behavioral repertoires without external rewards.
Revised: Grow any algae , Grow any bonsai , Grow any bush , Grow any cactus Grow any carnivorous , Grow any grass , Grow any living_thing , Grow any plant Grow any rose , Grow any tea , Grow any tree , Grow blue algae Grow blue bonsai , Grow blue bush , Grow blue cactus , Grow blue carnivorous

Original: As such, it is distinct from traditional instructionfollowing RL agents.
Revised: Grow blue grass , Grow blue living_thing , Grow blue plant , Grow blue rose Grow blue tea , Grow blue tree , Grow green algae , Grow green bonsai Grow green bush , Grow green cactus , Grow green carnivorous , Grow green grass Grow green living_thing , Grow green plant , Grow green rose , Grow green tea

Original: This is done through the joint training of a language encoder for goal representation and a goalachievement reward function to generate internal rewards.
Revised: Grow green tree , Grow red algae , Grow red bonsai , Grow red bush Grow red cactus , Grow red carnivorous , Grow red grass , Grow red living_thing Grow red plant , Grow red rose , Grow red tea , Grow red tree

Original: Our proposed modular architectures with gated-attention enable efﬁcient out-of-distribution generalization of the reward function and policy.
Revised: Interesting Interactions. Interesting interactions are trajectories of the agent that humans could infer as goaldirected. If an agent brings water to a plant and grows it, it makes sense for a human.

Original: The ability to imagine new goals by composing known ones leads to further improvements over initial generalization abilities and fosters exploration beyond the set of  interactions relevant to SP .
Revised: If it then tries to do this for a lamp, it also feels goal-directed, even though it does not work. This type of behavior characterizes the penchant of agents to interact with objects around them, to try new things and, as a result, is a good measure of exploration. Sets of interesting interactions. We consider three sets of interactions: 1) interactions related to training goals; 2) to testing goals; 3) the extra set.

Original: Our agent even tries to grow pieces of furniture with supplies, a behavior that can echo the way a child may try to feed his doll.
Revised: This extra set contains interactions where the agent brings water or food to a piece of furniture or to another supply. Although such behaviors do not achieve any of the goals, we consider them as interesting exploratory behaviors. Indeed, they testify that agents try to achieve imagined goals that are meaningful from the point of view of an agent that does not already know that doors cannot be grown, i.e. corresponding to a meaningful form of generalization after discovering that animals or plants can be grown (e.g. grow any door ).

Original: IMAGINE does not need externally-provided rewards but learns which behaviors are  interesting from language-based interactions with SP .
Revised: The Interesting Interaction Count metric. We count the number of interesting interactions computed over all ﬁnal transitions from the last 600 episodes (1 epoch). Agents do not need to target these interactions, we just report the number of times they are experienced. Indeed, the agent does not have to target a particular interaction for the trajectory to be interesting from an exploratory point of view.

Revised: The

Revised: HER mechanism ensures that these trajectories can be replayed to learn about any goal, imagined or not.

Revised: Computed on the extra set, the Interesting Interaction Count ( I  C ) is the number of times the agent was found to bring supplies to a furniture or to other supplies over the last epoch:

Revised: [Equation]

Revised: where δ i,t =1 if interaction i was achieved in episode t ,otherwise and I is the set of interesting interactions (here from the extra set) performed during an epoch.

Revised: Agents that are allowed to imagine goals achieve higher scores in the testing and extra sets of interactions, while maintaining similar exploration scores on the training set, see Figures 6a to 6c.

Revised: [Figure]

Revised: 9. Focus on Goal Imagination 9.1. Goal Imagination Algorithm

Revised: Algorithm 2 presents the algorithm underlying our goal imagination mechanism.

Revised: This mechanism is inspired from the Construction Grammar (CG) literature and generates new sentences by composing known ones (Goldberg, 2003).

Revised: It computes sets of equivalent words by searching for sentences with an edit distance of 1 : sentences where only one word differs.

Revised: These words are then labelled equivalent, and can be substituted in known sentences.

Revised: Note that the goal imagination process ﬁlters goals that are already known.

Revised: Although all sentences from G train can be imagined, there are ﬁltered out of the imagined goals as they are discovered.

Revised: Imagining goals from G train before they are discovered drives the exploration of IMAGINE agents.

Revised: In our setup, however, this effect remains marginal as all the goals from

Revised: G train are discovered in the ﬁrst epochs.

Revised: Imagined goals.

Revised: We run our goal imagination mechanism based on the Construction Grammar Heuristic ( CGH ) from

Revised: G train .

Revised: After ﬁltering goals from G train , this producesnew imagined sentences.

Revised: Table 2 presents the list of these goals while Figure 7 presents a Venn diagram of the various goal sets.

Revised: Among these 136 goals, 56 belong to the testing set G test .

Revised: This results in a coverage of 87 .

Revised: 5% of G test , and a precision of 45% .

Revised: In goals that do not belong to G test , goals of the form Grow + { any } ∪ color + furniture ∪ supplies (e.g. Grow any lamp ) are meaningful to humans, but are not achievable in the environment ( impossible ).

Revised: Goal Imagination Pseudo-Code.

Revised: The pseudo-code of the goal imagination mechanism is presented in Algorithm 2.

Revised: The edit distance between two sentences refers to the number of words to modify to transform one sentence into the other.

Revised: NL in G known do {Computing word equivalences} 6: new_goal_template

Revised: = True 7: for g m in goal_template do 8: if edit_distance ( g NL ,g m ) < 2 then 9: new_goal_template =

Revised: False 10: if edit_distance ( g NL ,g m )=

Revised: =1 then 11: w 1 ,w 2 ← get_non_matching_words ( g NL ,g m ) 12: if w 1 and w 2 not in any of word_eq sets then 13: word_eq .add

Original: {+({ w 1 ,w 2 }) 14: else 15: for eq_set in word_eq do 16: if w 1 ∈ eq_set or w 2 ∈ eq_set then 17: eq_set = eq_set ∪ { w 1 ,w 2 } 
+}
Revised: {+({ w 1 ,w 2 }) 14: else 15: for eq_set in word_eq do 16: if w 1 ∈ eq_set or w 2 ∈ eq_set then 17: eq_set = eq_set ∪ { w 1 ,w 2 } 
+}

Revised: : if new_goal_template then 19: goal_template .add

Revised: ( g NL ) 20: for g in goal_template do {Generating new sentences} 21: for w in g do 22: for eq_set in word_eq do 23: if w ∈ eq_set then 24: for w (cid:48) in eq_set do 25: g im ← replace( g , w , w (cid:48) ) 26: if g im / ∈G known then 27:

Revised: G im =

Revised: G im ∪ { g im } 28: G im =

Revised: G im \G known {ﬁltering known goals.}

Revised: [Table]

Revised: G train .

Revised: (Note that known goals are ﬁltered from the set of imagined goals.

Revised: However, any goal from G train can be imagined before it is encountered in the interaction with SP .)

Revised: Goals from G test

Revised: All goals from G test that are imaginable, see Table

Revised: [Table]

Revised: Go bottom top , Go left right , Grasp red blue thing ,

Revised: Grow blue red thing , Go right left , Go top bottom ,

Revised: Grasp green blue thing , Grow green red thing , Grasp green red thing

Revised: Grasp blue green thing , Grasp blue red thing , Grasp red green thing .

Revised: [Table]

Revised: Go center bottom , Go center top , Go right center , Go right bottom ,

Revised: Go right top , Go left center , Go left bottom , Go left top ,

Revised: Grow green cupboard , Grow green sink , Grow blue lamp , Go center right ,

Revised: Grow green window , Grow blue carpet , Grow red supply , Grow any sofa ,

Revised: Grow red sink , Grow any chair , Go top center , Grow blue table ,

Revised: Grow any door , Grow any lamp , Grow blue sink , Go bottom center ,

Revised: Grow blue door , Grow blue supply , Grow green carpet , Grow blue furniture ,

Revised: Grow green supply , Grow any window , Grow any carpet , Grow green furniture ,

Revised: Grow green chair , Grow green food , Grow any cupboard , Grow red food ,

Revised: Grow any table , Grow red lamp , Grow red door , Grow any food ,

Revised: Grow blue window , Grow green sofa , Grow blue sofa , Grow blue desk ,

Revised: Grow any sink , Grow red cupboard , Grow green door , Grow red furniture ,

Revised: Grow blue food , Grow red desk , Grow red table , Grow blue chair ,

Revised: Grow red sofa , Grow any furniture , Grow red window , Grow any desk ,

Revised: Grow blue cupboard , Grow red chair , Grow green desk , Grow green table ,

Revised: Grow red carpet , Go center left , Grow any supply , Grow green lamp ,

Revised: Grow blue water , Grow red water , Grow any water , Grow green water,

Revised: Grow any water , Grow green water.

Revised: 9.2. How Does Goal Imagination Properties Impact Performance?

Revised: Properties of imagined goals.

Revised: We propose to characterize goal imagination mechanisms by two properties:

Revised: Coverage : the fraction of G test found in G im 2.

Revised: Precision : the fraction of the imagined goals that are achievable.

Revised: We compare our goal imagination mechanism based on the construction grammar heuristic ( CGH ) to variants characterized by:

Revised: Lower coverage:

Revised: To reduce the coverage of CGH while maintaining the same precision, we simply ﬁlter half of the goals that would have been imagined by CGH .

Revised: This ﬁltering is probabilistic, resulting in different imagined sets for different runs.

Revised: It happens online, meaning that the coverage is always half of the coverage that CGH would have had at the same time of training.

Revised: Lower precision: To reduce precision while maintaining the same coverage, we sample a random sentence (random words from the words of G train ) for each goal imagined by CGH that does not belong to G test .

Revised: Goals from G test are still imagined via the CGH mechanism.

Revised: This variants only doubles the imagination of sentences that do not belong to G test .

Revised: Oracle: Perfect precision and coverage is achieved by ﬁltering the output of CGH , keeping only goals from G test .

Revised: Once the 56 goals that CGH can imagine are imagined, the oracle variants adds the 8 remaining goals: those including the word ﬂower that the agent cannot imagine.

Revised: Random goals: Each time CGH would have imagined a new goal, it is replaced by a randomly generated sentence, using words from the words of G train .

Revised: It leads to near null coverage and precision and works as a baseline.

Revised: Note that all variants imagine goals at the same speed as the

Revised: CGH algorithm.

Revised: They simply ﬁlter or add noise to its output.

Revised: For each variant, we measure the precision and coverage at the end of experiments, when all goals from G train have been discovered and report them in Figure 3.

Revised: CGH vs Oracle Figure 9a shows that CGH achieves a generalization performance on par with the oracle.

Revised: Reducing the coverage of the goal imagination mechanism still brings signiﬁcant improvements in generalization.

Revised: Effect of lower coverage on generalization In Figure 9a, the Low Coverage condition seems to achieve good results,

Revised: [Table]

Original: In contrast with hand-crafted reward functions, NL descriptions provide an easy way to guide machines towards relevant interactions.
Revised: close to CGH . The Low Coverage variant of our goal imagination mechanism only covers 43 . 7% the test set with a 45% precision. To further study the impact of a lower coverage, we report in Figure 9b, success rates on testing goals of type grow + plant and compare with the no imagination baseline (green).

Revised: We split in two: goals that were imagined (blue), and goals that were not (orange).

Revised: The generalization performance on goals from G test that the agent imagined (blue) are not signiﬁcantly higher than the generalization performance on goals from G test that were not imagined.

Revised: As they are both signiﬁcantly higher than the no imagination baseline, this implies that training on imagined goals boosts zero-shot generalization on similar goals that were not imagined.

Revised: Effect of lower precision on generalization Reducing the precision of imagined goals (gray curve) seems to impede generalization (no signiﬁcant difference with the no imagination baseline), as shown in Figure 9a.

Revised: 9.3. How Does Goal Imagination Properties Impact Exploration?

Revised: Figure 9 presents the I  C exploration scores on the training, testing and extra sets for the different goal imagination mechanisms above.

Revised: Let us discuss each of these scores:

Revised: Training interactions.

Revised: In Figure 9c, we see that decreasing the precision (Low Precision and Random Goal conditions) affects exploration on interactions from the training set, where it falls below the exploration of the no imagination baseline.

Revised: This is due to the addition of meaningless goals forcing agent to allow less time to meaningful interactions relatively.

Revised: Testing interactions.

Revised: Figure 9d shows that all goal imagination heuristics enable a signiﬁcant exploration boost.

Revised: The random goal baseline acts as a control condition.

Revised: It demonstrates that the generalization boost is not

Revised: due to a mere effect of network regularization introduced by adding random goals (no signiﬁcant effect w.r.t.

Revised: the no imagination baseline).

Revised: In the same spirit, we also ran a control using random goal embedding, which did not produce any signiﬁcant effects.

Revised: We also see that the highest exploration scores on interactions from the test set comes from the oracle.

Revised: Because it shows high coverage and precision, its spends more time on the diversity of interactions from the testing set.

Revised: What is more surprising is the exploration score of the low coverage condition, higher than the exploration score of CGH .

Revised: With an equal precision, CGH should show better exploration, as it covers more test goals.

Revised: However, the Low Coverage condition, by spending more time exploring each of its imagined goals (it imagined fewer), probably learned to master them better, increasing the robustness of its behavior towards those.

Revised: This insight advocates for the use of goal selection methods based on learning progress (Forestier &

Revised: Oudeyer, 2016; Colas et al., 2019a).

Original: In addition, Section 4 shows that agents  can learn to achieve goals from a relatively small number of descriptions, paving the way towards human-provided descriptions.
Revised: Agents could estimate their learning progress on imagined goals using their internal reward function and its zero-shot generalization. Focusing on goals associated to high learning progress might help agents ﬁlter goals they can learn about from others. Extra interactions. Figure 9e shows that only the goal imagination mechanisms that invent goals not covered by the testing set manage to boost exploration in this extra set.

Revised: The oracle perfectly covers the testing set, but does not generate goals related to other objects (e.g. grow any lamp ).

Revised: 10. Focus on SP feedback

Revised: In this section, we study the relaxation of 2 of the 3 properties imposed on SP and wonder if we could Use More

Revised: Realistic Feedbacks.

Revised: We study the relaxation of the fullpresence and exhaustiveness assumptions of SP .

Revised: We ﬁrst relax full-presence while keeping exhaustiveness (blue, yellow and purple curves).

Revised: When SP has a 10% chance of being present (yellow), imaginative agents show generalization performance on par with the unimaginative agents trained in a full-presence setting (green), see Figure 8).

Revised: However, when the same amount of feedback is concentrated in the ﬁrst 10% episodes (purple), goal imagination enables significant improvements in generalization (w.r.t. green).

Revised: This is reminiscent of children who require less and less attention as they grow into adulthood and is consistent with Chan et al.

Revised: Relaxing exhaustiveness , SP only provides one positive and one negative description every episode (red) or in 50% of the episodes (gray).

Revised: Then, generalization performance matches the one of unimaginative agents in the exhaustive setting (green).

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: 11. Architecture and implementation details 11.1. Architecture

Revised: Figure 2 shows the architecture of IMAGINE and its different modules.

Revised: In this section, we give more details on each modules :

Revised: Language encoder.

Revised: The language encoder ( L e ) embeds

Revised: NL goals ( L e : G NL → R 100 ) using an LSTM (Hochreiter & Schmidhuber, 1997).

Revised: It is trained jointly with the reward function.

Revised: L e acts as a goal translator, turning the goal-achievement reward function, policy and critic into language-conditioned functions.

Revised: Reward function.

Revised: Learning a goal-achievement reward function ( R ) is framed as binary classiﬁcation:

Revised: R ( s , g ): S × R 100 →{ 0 , 1 } .

Revised: We use the MA architecture described in Section 3.2 with attention vectors α g , a shared network NN R with output size 1 and a logical OR aggregation.

Revised: NN R computes object-dependent rewards r i in [0 , 1] from the object-speciﬁc inputs and the goal embedding.

Revised: The ﬁnal binary reward is computed by NN OR which outputswhenever ∃ j : r j > 0 .

Revised: We pre-trained a neural-networkbased OR function to enable end-to-end training.

Revised: Figure 10a shows the detailed architecture of the reward function.

Revised: The overall reward function can be written as:

Revised: [Equation]

Revised: Data.

Revised: Interacting with the environment and SP , the agent builds a set of entries [ s T , g, r ] with g ∈ G known where r ∈{ 0 , 1 } rewards the achievement of g in state s T :

Revised: r =if g ∈G SP ( s T ) and 0 otherwise.

Revised: L e and R are periodically updated jointly by backpropagation on this dataset.

Revised: Multi-goal RL agent.

Revised: Our agent is controlled by a goalconditioned policy Π (Schaul et al., 2015) based on the MA architecture (see Figure 10b).

Revised: It uses an attention vector β g , a shared network NN Π , a sum aggregation and a mapper

Revised: NN a that outputs the actions.

Revised: Similarly, the critic produces action-values via γ g , NN Q and NN a-v respectively, see equations below.

Revised: Both are trained using DDPG (Lillicrap et al., 2015), although any other off-policy algorithm can be used.

Revised: Figure 10b shows the detailed architecture of the reward function.

Revised: The overall function can be written as

Revised: [Equation]

Revised: Hindsight Experience Replay ( HER ).

Revised: Our agent uses a form of Hindsight Experience Replay (Andrychowicz et al., 2017).

Revised: In multi-goal RL with discrete sets of goals, HER is traditionally used to modify transitions sampled from the replay buffer.

Revised: It replaces originally targeted goals by others randomly selected from the set of goals (Andrychowicz et al., 2017; Mankowitz et al., 2018).

Revised: This enables to transfer knowledge between goals, reinterpreting trajectories in the light of new goals.

Revised: In that case, a reward function is required to compute the reward associated to that new transition (new goal).

Revised: To improve on random goal replay, we favor goal substitution towards goals that actually match the state and have higher chance of leading to rewards.

Revised: In IMAGINE , we scan a set of 40 goal candidates for each transition, and select substitute goals that match the scene when possible, with probability p = 0 .

Revised: 11.2. Implementation details

Revised: Reward function inputs and hyperparameters.

Revised: The following provides extra details about the inputs.

Revised: • The object-dependent sub-state s obj ( i ) contains information about both the agent’s body and the corresponding object i :

Revised: [Equation]

Revised: o obj ( i ) ] where o body and o obj ( i ) are body- and obj i -dependent observations, and

Revised: ∆ o tbody =

Revised: o tbody − o 0 body and ∆ o tobj ( i ) =

Original: {+o tobj ( i ) − o 
+}
Revised: {+o tobj ( i ) − o 
+}

Revised: obj ( i ) measure the difference between the initial and current observations.

Revised: • The attention vector α g that is integrated with s obj ( i ) through an Hadamard product to form the model input: x gi =

Revised: s obj ( i ) (cid:12) α g .

Revised: This attention vector is a simple mapping from g to a vector of the size of s obj ( i ) contained in [0 , 1] size ( s obj ( i ) ) .

Revised: This cast is implemented by a one-layer neural network with sigmoid activations NN cast such that α g = NN cast ( g ) .

Revised: For the three architectures the number of hidden units of the LSTM and the sizes of the hidden layers of fully connected networks are ﬁxed to 100 .

Revised: NN parameters are initialized using He initialization (He et al., 2015) and we use one-hot word encodings.

Revised: The LSTM is implemented using rnn.

Revised: BasicLSTMCell from tensorﬂow 1.15 based on Zaremba et al.

Revised: The states are initially set to

Revised: zero

Revised: The LSTM ’s weights are initialized uniformly from [ − 0 .

Revised: and the biases initially set to zero.

Revised: The LSTM use a

Revised: tanh

Revised: activation function whereas the NN are using ReLU activation functions in their hidden layers and sigmoids at

Revised: there output

Revised: Reward function training schedule.

Revised: The architecture are trained via backpropagation using the Adam Optimizer (Kingma & Ba, 2014).

Revised: The data is fed to the model in batches of 512 examples.

Revised: Each batch is constructed so that

Revised: [Figure]

Revised: it contains at least one instance of each goal description g NL (goals discovered so far).

Revised: We also use a modular buffer to impose a ratio of positive rewards of 0 .

Revised: 2 for each description in each batch.

Revised: When trained in parallel of the policy, the reward function is updated once every 1200 episodes.

Revised: Each update corresponds to up to 100 training epochs (batches).

Revised: We implement a stopping criterion based on the

Revised: F 1 -score computed from a held-out test set uniformly sampled from the last episodes ( 20% of the last 1200 episodes (2 epochs)).

Revised: The update is stopped when the F 1 -score on the held-out set does not improve for 10 consecutive training epochs.

Revised: RL implementation and hyperparameters.

Revised: In the policy and critic architectures, we use hidden layers of size 256 and ReLU activations.

Revised: Attention vectors are cast from goal embeddings using single-layer neural networks with sigmoid activations.

Revised: We use the He initialization scheme for (He et al., 2015) and train them via backpropagation using the Adam optimizer ( β 1 =0 .

Revised: 9 ,β 2 =0 .

Revised: (Kingma & Ba, 2014).

Revised: Our learning algorithm is built on top of the OpenAI Baselines implementation of HER - DDPG .

Revised: 1 We leverage a parallel implementation with 6 actors.

Revised: Actors share the same policy and critic parameters but maintain their own memory and conduct their own updates independently.

Revised: Updates are then summed to compute the next set of parameters broadcast to all actors.

Revised: Each actor is updated for 50 epochs with batches of size 256 every 2 episodes of environment interactions.

Revised: Using hindsight replay, we enforce a ratio p =0 .

Revised: 5 of transitions associated with positive rewards in each batch

Revised: Computing resources.

Revised: The RL experiments containconditions of 10 seeds each, and 4 conditions with 5 seeds ( SP study).

Revised: Each run leverages 6 cpus ( 6 actors) for about 36 h for a total of 2 . 5 cpu years.

Revised: Experiments presented in this paper requires machines with at least 6 cpu cores.