Original: GROUNDING LANGUAGE TO AUTONOMOUSLYACQUIRED SKILLS VIA GOAL GENERATION
Revised: GROUNDING LANGUAGE TO AUTONOMOUSLYACQUIRED SKILLS VIA GOAL GENERATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We are interested in the autonomous acquisition of repertoires of skills.
Revised: We are interested in the autonomous acquisition of repertoires of skills.

Original: Languageconditioned reinforcement learning ( LC - RL ) approaches are great tools in this quest, as they allow to express abstract goals as sets of constraints on the states.
Revised: Languageconditioned reinforcement learning ( LC - RL ) approaches are great tools in this quest, as they allow to express abstract goals as sets of constraints on the states.

Original: However, most LC - RL agents are not autonomous and cannot learn without external instructions and feedback.
Revised: However, most LC - RL agents are not autonomous and cannot learn without external instructions and feedback.

Original: Besides, their direct language condition cannot account for the goal-directed behavior of pre-verbal infants and strongly limits the expression of behavioral diversity for a given language input.
Revised: Besides, their direct language condition cannot account for the goal-directed behavior of pre-verbal infants and strongly limits the expression of behavioral diversity for a given language input.

Original: To resolve these issues, we propose a new conceptual approach to language-conditioned RL : the Language-Goal-Behavior architecture ( LGB ).
Revised: To resolve these issues, we propose a new conceptual approach to language-conditioned RL : the Language-Goal-Behavior architecture ( LGB ).

Original: LGB decouples skill learning and language grounding via an intermediate semantic representation of the world.
Revised: LGB decouples skill learning and language grounding via an intermediate semantic representation of the world.

Original: To showcase the properties of LGB , we present a speciﬁc implementation called
Revised: To showcase the properties of LGB , we present a speciﬁc implementation called

Original: DECSTR .
Revised: DECSTR .

Original: DECSTR is an intrinsically motivated learning agent endowed with an innate semantic representation describing spatial relations between physical objects.
Revised: DECSTR is an intrinsically motivated learning agent endowed with an innate semantic representation describing spatial relations between physical objects.

Original: In a ﬁrst stage ( G → B ), it freely explores its environment and targets selfgenerated semantic conﬁgurations.
Revised: In a ﬁrst stage ( G → B ), it freely explores its environment and targets selfgenerated semantic conﬁgurations.

Original: In a second stage ( L → G ), it trains a languageconditioned goal generator to generate semantic goals that match the constraints expressed in language-based inputs.
Revised: In a second stage ( L → G ), it trains a languageconditioned goal generator to generate semantic goals that match the constraints expressed in language-based inputs.

Original: We showcase the additional properties of
Revised: We showcase the additional properties of

Original: LGB w.r.t.
Revised: LGB w.r.t.

Original: both an end-to-end LC - RL approach and a similar approach leveraging non-semantic, continuous intermediate representations.
Revised: both an end-to-end LC - RL approach and a similar approach leveraging non-semantic, continuous intermediate representations.

Original: Intermediate semantic representations help satisfy language commands in a diversity of ways, enable strategy switching after a failure and facilitate language grounding.
Revised: Intermediate semantic representations help satisfy language commands in a diversity of ways, enable strategy switching after a failure and facilitate language grounding.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Developmental psychology investigates the interactions between learning and developmental processes that support the slow but extraordinary transition from the behavior of infants to the sophisticated intelligence of human adults (Piaget, 1977; Smith & Gasser, 2005).
Revised: Developmental psychology investigates the interactions between learning and developmental processes that support the slow but extraordinary transition from the behavior of infants to the sophisticated intelligence of human adults (Piaget, 1977; Smith & Gasser, 2005).

Original: Inspired by this line of thought, the central endeavour of developmental robotics consists in shaping a set of machine learning processes able to generate a similar growth of capabilities in robots (Weng et al., 2001;
Revised: Inspired by this line of thought, the central endeavour of developmental robotics consists in shaping a set of machine learning processes able to generate a similar growth of capabilities in robots (Weng et al., 2001;

Original: Lungarella et al., 2003).
Revised: Lungarella et al., 2003).

Original: In this broad context, we are more speciﬁcally interested in designing learning agents able to: 1) explore open-ended environments and grow repertoires of skills in a self-supervised way and 2) learn from a tutor via language commands.
Revised: In this broad context, we are more speciﬁcally interested in designing learning agents able to: 1) explore open-ended environments and grow repertoires of skills in a self-supervised way and 2) learn from a tutor via language commands.

Original: The design of intrinsically motivated agents marked a major step towards these goals.
Revised: The design of intrinsically motivated agents marked a major step towards these goals.

Original: The Intrinsically Motivated Goal Exploration Processes family (IMGEPs), for example, describes embodied agents that interact with their environment at the sensorimotor level and are endowed with the ability to represent and set their own goals, rewarding themselves over completion (Forestier et al., 2017).
Revised: The Intrinsically Motivated Goal Exploration Processes family (IMGEPs), for example, describes embodied agents that interact with their environment at the sensorimotor level and are endowed with the ability to represent and set their own goals, rewarding themselves over completion (Forestier et al., 2017).

Original: Recently, goal-conditioned reinforcement learning ( GC - RL ) appeared like a viable way to implement
Revised: Recently, goal-conditioned reinforcement learning ( GC - RL ) appeared like a viable way to implement

Original: IMGEPs and target the open-ended and self-supervised acquisition of diverse skills.
Revised: IMGEPs and target the open-ended and self-supervised acquisition of diverse skills.

Original: Goal-conditioned RL approaches train goal-conditioned policies to target multiple goals (Kaelbling, 1993; Schaul et al., 2015).
Revised: Goal-conditioned RL approaches train goal-conditioned policies to target multiple goals (Kaelbling, 1993; Schaul et al., 2015).

Original: While most GC - RL approaches express goals as target features (e.g. target block positions (Andrychowicz et al., 2017), agent positions in a maze (Schaul et al., 2015) or target images (Nair et al., 2018)), recent approaches started to use language to express goals, as language can express sets of constraints on the state space (e.g. open the red door ) in a more abstract and interpretable way (Luketina et al., 2019).[Figure]
Revised: While most GC - RL approaches express goals as target features (e.g. target block positions (Andrychowicz et al., 2017), agent positions in a maze (Schaul et al., 2015) or target images (Nair et al., 2018)), recent approaches started to use language to express goals, as language can express sets of constraints on the state space (e.g. open the red door ) in a more abstract and interpretable way (Luketina et al., 2019).

Original: However, most GC - RL approaches – and language-based ones ( LC - RL ) in particular – are not intrinsically motivated and receive external instructions and rewards.
Revised: However, most GC - RL approaches – and language-based ones ( LC - RL ) in particular – are not intrinsically motivated and receive external instructions and rewards.

Original: The IMAGINE approach is one of the rare examples of intrinsically motivated LC - RL approaches (Colas et al., 2020).
Revised: The IMAGINE approach is one of the rare examples of intrinsically motivated LC - RL approaches (Colas et al., 2020).

Original: In any case, the language condition suffers from three drawbacks.
Revised: In any case, the language condition suffers from three drawbacks.

Original: 1) It couples skill learning and language grounding.
Revised: 1) It couples skill learning and language grounding.

Original: Thus, it cannot account for goal-directed behaviors in pre-verbal infants (Mandler, 1999).
Revised: Thus, it cannot account for goal-directed behaviors in pre-verbal infants (Mandler, 1999).

Original: Direct conditioning limits the behavioral diversity associated to language input: a single instruction leads to a low diversity of behaviors only resulting from the stochasticity of the policy or the environment.
Revised: Direct conditioning limits the behavioral diversity associated to language input: a single instruction leads to a low diversity of behaviors only resulting from the stochasticity of the policy or the environment.

Original: This lack of behavioral diversity prevents agents from switching strategy after a failure.
Revised: This lack of behavioral diversity prevents agents from switching strategy after a failure.

Original: To circumvent these three limitations, one can decouple skill learning and language grounding via an intermediate innate semantic representation.
Revised: To circumvent these three limitations, one can decouple skill learning and language grounding via an intermediate innate semantic representation.

Original: On one hand, agents can learn skills by targeting conﬁgurations from the semantic representation space.
Revised: On one hand, agents can learn skills by targeting conﬁgurations from the semantic representation space.

Original: On the other hand, they can learn to generate valid semantic conﬁgurations matching the constraints expressed by language instructions.
Revised: On the other hand, they can learn to generate valid semantic conﬁgurations matching the constraints expressed by language instructions.

Original: This generation can be the backbone of behavioral diversity: a given sentence might correspond to a whole set of matching conﬁgurations.
Revised: This generation can be the backbone of behavioral diversity: a given sentence might correspond to a whole set of matching conﬁgurations.

Original: This is what we propose in this work.
Revised: This is what we propose in this work.

Original: Contributions.
Revised: Contributions.

Original: We propose a novel conceptual RL architecture, named LGB for Language-GoalBehavior and pictured in Figure 1 (right).
Revised: We propose a novel conceptual RL architecture, named LGB for Language-GoalBehavior and pictured in Figure 1 (right).

Original: This LGB architecture enables an agent to decouple the intrinsically motivated acquisition of a repertoire of skills (Goals → Behavior) from language grounding (Language → Goals), via the use of semantic goal representation.
Revised: This LGB architecture enables an agent to decouple the intrinsically motivated acquisition of a repertoire of skills (Goals → Behavior) from language grounding (Language → Goals), via the use of semantic goal representation.

Original: To our knowledge, the
Revised: To our knowledge, the

Original: LGB architecture is the only one to combine the following four features:
Revised: LGB architecture is the only one to combine the following four features:

Original: • It is intrinsically motivated: it selects its own (semantic) goals and generates its own rewards, • It decouples skill learning from language grounding, accounting for infants learning, • It can exhibit a diversity of behaviors for any given instruction, • It can switch strategy in case of failures.
Revised: • It is intrinsically motivated: it selects its own (semantic) goals and generates its own rewards, • It decouples skill learning from language grounding, accounting for infants learning, • It can exhibit a diversity of behaviors for any given instruction, • It can switch strategy in case of failures.

Original: Besides, we introduce an instance of LGB , named DECSTR for DE ep sets and C urriculum with
Revised: Besides, we introduce an instance of LGB , named DECSTR for DE ep sets and C urriculum with

Original: S eman T ic goal R epresentations.
Revised: S eman T ic goal R epresentations.

Original: Using DECSTR , we showcase the advantages of the conceptual decoupling idea.
Revised: Using DECSTR , we showcase the advantages of the conceptual decoupling idea.

Original: In the skill learning phase, the DECSTR agent evolves in a manipulation environment and leverages semantic representations based on predicates describing spatial relations between physical objects.
Revised: In the skill learning phase, the DECSTR agent evolves in a manipulation environment and leverages semantic representations based on predicates describing spatial relations between physical objects.

Original: These predicates are known to be used by infants from a very young age (Mandler, 2012).
Revised: These predicates are known to be used by infants from a very young age (Mandler, 2012).

Original: DECSTR autonomously learns to discover and master all reachable conﬁgurations in its semantic representation space.
Revised: DECSTR autonomously learns to discover and master all reachable conﬁgurations in its semantic representation space.

Original: In the language grounding phase, we train a Conditional Variational Auto-Encoder ( C - VAE ) to generate semantic goals from language instructions.
Revised: In the language grounding phase, we train a Conditional Variational Auto-Encoder ( C - VAE ) to generate semantic goals from language instructions.

Original: Finally, we can evaluate the agent in an instruction-following phase by composing the two ﬁrst phases.
Revised: Finally, we can evaluate the agent in an instruction-following phase by composing the two ﬁrst phases.

Original: The experimental section investigates three questions: how does DECSTR perform in the three phases?
Revised: The experimental section investigates three questions: how does DECSTR perform in the three phases?

Original: How does it compare to end-to-end LC - RL approaches?
Revised: How does it compare to end-to-end LC - RL approaches?

Original: Do we need intermediate representations to be semantic?
Revised: Do we need intermediate representations to be semantic?

Original: Code and videos can be found at https://sites.google.com/view/decstr/.
Revised: Code and videos can be found at https://sites.google.com/view/decstr/.

Original:  RELATED WORK
Revised:  RELATED WORK

Original: Standard language-conditioned RL.
Revised: Standard language-conditioned RL.

Original: Most approaches from the LC - RL literature deﬁne instruction following agents that receive external instructions and rewards (Hermann et al., 2017; Chan et al., 2019; Bahdanau et al., 2018; Cideron et al., 2019; Jiang et al., 2019;
Revised: Most approaches from the LC - RL literature deﬁne instruction following agents that receive external instructions and rewards (Hermann et al., 2017; Chan et al., 2019; Bahdanau et al., 2018; Cideron et al., 2019; Jiang et al., 2019;

Original: Fu et al., 2019), except the IMAGINE approach which introduced intrinsically motivated agents able to set their own goals and to imagine new ones (Colas et al., 2020).
Revised: Fu et al., 2019), except the IMAGINE approach which introduced intrinsically motivated agents able to set their own goals and to imagine new ones (Colas et al., 2020).

Original: In both cases, the language-condition prevents the decoupling of language acquisition and skill learning, true behavioral diversity and efﬁcient strategy switching behaviors.
Revised: In both cases, the language-condition prevents the decoupling of language acquisition and skill learning, true behavioral diversity and efﬁcient strategy switching behaviors.

Original: Our approach is different, as we can decouple language acquisition from skill learning.
Revised: Our approach is different, as we can decouple language acquisition from skill learning.

Original: The language-conditioned goal generation allows behavioral diversity and strategy switching behaviors.
Revised: The language-conditioned goal generation allows behavioral diversity and strategy switching behaviors.

Revised: [Figure]

Original: Goal-conditioned RL with target coordinates for block manipulation.
Revised: Goal-conditioned RL with target coordinates for block manipulation.

Original: Our proposed implementation of LGB , called DECSTR , evolves in a block manipulation domain.
Revised: Our proposed implementation of LGB , called DECSTR , evolves in a block manipulation domain.

Original: Stacking blocks is one of the earliest benchmarks in artiﬁcial intelligence (e.g. Sussman (1973); Tate et al.
Revised: Stacking blocks is one of the earliest benchmarks in artiﬁcial intelligence (e.g. Sussman (1973); Tate et al.

Original: and has led to many simulation and robotics studies (Deisenroth et al., 2011; Xu et al., 2018; Colas et al., 2019a).
Revised: and has led to many simulation and robotics studies (Deisenroth et al., 2011; Xu et al., 2018; Colas et al., 2019a).

Original: Recently, Lanier et al.
Revised: Recently, Lanier et al.

Original: and
Revised: and

Original: Li et al.
Revised: Li et al.

Original: (2019) demonstrated impressive results by stacking up to 4 and 6 blocks respectively.
Revised: (2019) demonstrated impressive results by stacking up to 4 and 6 blocks respectively.

Original: However, these approaches are not intrinsically motivated, involve hand-deﬁned curriculum strategies and express goals as speciﬁc target block positions.
Revised: However, these approaches are not intrinsically motivated, involve hand-deﬁned curriculum strategies and express goals as speciﬁc target block positions.

Original: In contrast, the DECSTR agent is intrinsically motivated, builds its own curriculum and uses semantic goal representations (symbolic or language-based) based on spatial relations between blocks.
Revised: In contrast, the DECSTR agent is intrinsically motivated, builds its own curriculum and uses semantic goal representations (symbolic or language-based) based on spatial relations between blocks.

Original: Decoupling language acquisition and skill learning.
Revised: Decoupling language acquisition and skill learning.

Original: Several works investigate the use of semantic representations to associate meanings and skills (Alomari et al., 2017; Tellex et al., 2011; Kulick et al., 2013).
Revised: Several works investigate the use of semantic representations to associate meanings and skills (Alomari et al., 2017; Tellex et al., 2011; Kulick et al., 2013).

Original: While the two ﬁrst use semantic representations as an intermediate layer between language and skills, the third one does not use language.
Revised: While the two ﬁrst use semantic representations as an intermediate layer between language and skills, the third one does not use language.

Original: While DECSTR acquires skills autonomously, previous approaches all use skills that are either manually generated (Alomari et al., 2017), handengineered (Tellex et al., 2011) or obtained via optimal control methods (Kulick et al., 2013).
Revised: While DECSTR acquires skills autonomously, previous approaches all use skills that are either manually generated (Alomari et al., 2017), handengineered (Tellex et al., 2011) or obtained via optimal control methods (Kulick et al., 2013).

Original: Closer to us, Lynch & Sermanet (2020) also decouple skill learning from language acquisition in a goalconditioned imitation learning paradigm by mapping both language goals and images goals to a shared representation space.
Revised: Closer to us, Lynch & Sermanet (2020) also decouple skill learning from language acquisition in a goalconditioned imitation learning paradigm by mapping both language goals and images goals to a shared representation space.

Original: However, this approach is not intrinsically motivated as it relies on a dataset of human tele-operated strategies.
Revised: However, this approach is not intrinsically motivated as it relies on a dataset of human tele-operated strategies.

Original: The deterministic merging of representations also limits the emergence of behavioral diversity and efﬁcient strategy-switching behaviors.
Revised: The deterministic merging of representations also limits the emergence of behavioral diversity and efﬁcient strategy-switching behaviors.

Original:  METHODS
Revised:  METHODS

Original: This section presents our proposed Language-Goal-Behavior architecture ( LGB ) represented in Figure 1 (Section 3.1) and a particular instance of the LGB architecture called DECSTR .
Revised: This section presents our proposed Language-Goal-Behavior architecture ( LGB ) represented in Figure 1 (Section 3.1) and a particular instance of the LGB architecture called DECSTR .

Original: We ﬁrst present the environment it is set in [3.2], then describe the implementations of the three modules composing any LGB architecture: 1) the semantic representation [3.3]; 2) the intrinsically motivated goal-conditioned algorithm [3.4] and 3) the language-conditioned goal generator [3.5].
Revised: We ﬁrst present the environment it is set in [3.2], then describe the implementations of the three modules composing any LGB architecture: 1) the semantic representation [3.3]; 2) the intrinsically motivated goal-conditioned algorithm [3.4] and 3) the language-conditioned goal generator [3.5].

Original: We ﬁnally present how the three phases described in Figure 1 are evaluated [3.6].
Revised: We ﬁnally present how the three phases described in Figure 1 are evaluated [3.6].

Original: 3. THE LANGUAGE - GOAL - BEHAVIOR ARCHITECTURE
Revised: 3. THE LANGUAGE - GOAL - BEHAVIOR ARCHITECTURE

Original: The LGB architecture is composed of three main modules.
Revised: The LGB architecture is composed of three main modules.

Original: First, the semantic representation deﬁnes the behavioral and goal spaces of the agent.
Revised: First, the semantic representation deﬁnes the behavioral and goal spaces of the agent.

Original: Second, the intrinsically motivated GC - RL algorithm is in charge of the skill learning phase.
Revised: Second, the intrinsically motivated GC - RL algorithm is in charge of the skill learning phase.

Original: Third, the language-conditioned goal generator is in charge of the language grounding phase.
Revised: Third, the language-conditioned goal generator is in charge of the language grounding phase.

Original: Both phases can be combined in the instruction following phase.
Revised: Both phases can be combined in the instruction following phase.

Original: The three phases are respectively called G → B for Goal → Behavior, L → G for Language → Goal and L → G → B for Language → Goal → Behavior, see Figure 1 and Appendix A. Instances of the
Revised: The three phases are respectively called G → B for Goal → Behavior, L → G for Language → Goal and L → G → B for Language → Goal → Behavior, see Figure 1 and Appendix A. Instances of the

Original: LGB architecture should demonstrate the four properties listed in the introduction: 1) be intrinsically motivated; 2) decouple skill learning and language grounding (by design); 3) favor behavioral diversity; 4) allow strategy switching.
Revised: LGB architecture should demonstrate the four properties listed in the introduction: 1) be intrinsically motivated; 2) decouple skill learning and language grounding (by design); 3) favor behavioral diversity; 4) allow strategy switching.

Original: We argue that any LGB algorithm should fulﬁll the following constraints.
Revised: We argue that any LGB algorithm should fulﬁll the following constraints.

Original: For LGB to be intrinsically motivated (1), the algorithm needs to integrate the generation and selection of semantic goals and to generate its own rewards.
Revised: For LGB to be intrinsically motivated (1), the algorithm needs to integrate the generation and selection of semantic goals and to generate its own rewards.

Original: For LGB to demonstrate behavioral diversity and strategy switching (3, 4), the language-conditioned goal generator must efﬁciently model the distribution of semantic goals satisfying the constraints expressed by any language input.
Revised: For LGB to demonstrate behavioral diversity and strategy switching (3, 4), the language-conditioned goal generator must efﬁciently model the distribution of semantic goals satisfying the constraints expressed by any language input.

Original: 3. ENVIRONMENT
Revised: 3. ENVIRONMENT

Original: The DECSTR agent evolves in the Fetch Manipulate environment: a robotic manipulation domain based on MUJOCO (Todorov et al., 2012) and derived from the Fetch tasks (Plappert et al., 2018), see
Revised: The DECSTR agent evolves in the Fetch Manipulate environment: a robotic manipulation domain based on MUJOCO (Todorov et al., 2012) and derived from the Fetch tasks (Plappert et al., 2018), see

Original: Figure 2.
Revised: Figure 2.

Original: Actions are 4-dimensional: 3D gripper velocities and grasping velocity.
Revised: Actions are 4-dimensional: 3D gripper velocities and grasping velocity.

Original: Observations include the Cartesian and angular positions and velocities of the gripper and the three blocks.
Revised: Observations include the Cartesian and angular positions and velocities of the gripper and the three blocks.

Original: Inspired by the framework of Zone of Proximal Development that describes how parents organize the learning environment of their children (Vygotsky, 1978), we let a social partner facilitate DECSTR ’s exploration by providing non-trivial initial conﬁgurations.
Revised: Inspired by the framework of Zone of Proximal Development that describes how parents organize the learning environment of their children (Vygotsky, 1978), we let a social partner facilitate DECSTR ’s exploration by providing non-trivial initial conﬁgurations.

Original: After a ﬁrst period of autonomous exploration, the social partner initializes the scene with stacks of 2 blocks 21% of times, stacks of 3 blocks 9% of times, and a block is initially put in the agent’s gripper 50% of times.
Revised: After a ﬁrst period of autonomous exploration, the social partner initializes the scene with stacks of 2 blocks 21% of times, stacks of 3 blocks 9% of times, and a block is initially put in the agent’s gripper 50% of times.

Original: This help is not provided during ofﬂine evaluations.
Revised: This help is not provided during ofﬂine evaluations.

Original: 3. SEMANTIC REPRESENTATION
Revised: 3. SEMANTIC REPRESENTATION

Original: Semantic predicates deﬁne the behavioral space.
Revised: Semantic predicates deﬁne the behavioral space.

Original: Deﬁning the list of semantic predicates is deﬁning the dimensions of the behavioral space explored by the agent.
Revised: Deﬁning the list of semantic predicates is deﬁning the dimensions of the behavioral space explored by the agent.

Original: It replaces the traditional deﬁnition of goal spaces and their associated reward functions.
Revised: It replaces the traditional deﬁnition of goal spaces and their associated reward functions.

Original: We believe it is for the best, as it does not require the engineer to fully predict all possible behaviors within that space, to know which behaviors can be achieved and which ones cannot, nor to deﬁne reward functions for each of them.
Revised: We believe it is for the best, as it does not require the engineer to fully predict all possible behaviors within that space, to know which behaviors can be achieved and which ones cannot, nor to deﬁne reward functions for each of them.

Original: Semantic predicates in DECSTR.
Revised: Semantic predicates in DECSTR.

Original: We assume the DECSTR agent to have access to innate semantic representations based on a list of predicates describing spatial relations between pairs of objects in the scene.
Revised: We assume the DECSTR agent to have access to innate semantic representations based on a list of predicates describing spatial relations between pairs of objects in the scene.

Original: We consider two of the spatial predicates infants demonstrate early in their development (Mandler, 2012): the close and the above binary predicates.
Revised: We consider two of the spatial predicates infants demonstrate early in their development (Mandler, 2012): the close and the above binary predicates.

Original: These predicates are applied to all permutations of object pairs for the 3 objects we consider: 6 permutations for the above predicate and 3 combinations for the close predicate due to its order-invariance.
Revised: These predicates are applied to all permutations of object pairs for the 3 objects we consider: 6 permutations for the above predicate and 3 combinations for the close predicate due to its order-invariance.

Original: A semantic conﬁguration is the concatenation of the evaluations of these 9 predicates and represents spatial relations between objects in the scene.
Revised: A semantic conﬁguration is the concatenation of the evaluations of these 9 predicates and represents spatial relations between objects in the scene.

Original: In the resulting semantic conﬁguration space { 0 , 1 } 9 , the agent can reachphysically valid conﬁgurations, including stacks of 2 or 3 blocks and pyramids, see examples in
Revised: In the resulting semantic conﬁguration space { 0 , 1 } 9 , the agent can reachphysically valid conﬁgurations, including stacks of 2 or 3 blocks and pyramids, see examples in

Original: Figure 2.
Revised: Figure 2.

Original: The binary reward function directly derives from the semantic mapping: the agent rewards itself when its current conﬁguration c p matches the goal conﬁguration c
Revised: The binary reward function directly derives from the semantic mapping: the agent rewards itself when its current conﬁguration c p matches the goal conﬁguration c

Original: p = g .
Revised: p = g .

Original: Appendix B provides formal deﬁnitions and properties of predicates and semantic conﬁgurations.
Revised: Appendix B provides formal deﬁnitions and properties of predicates and semantic conﬁgurations.

Original: 3. INTRINSICALLY MOTIVATED GOAL - CONDITIONED REINFORCEMENT LEARNING
Revised: 3. INTRINSICALLY MOTIVATED GOAL - CONDITIONED REINFORCEMENT LEARNING

Original: This section describes the implementation of the intrinsically motivated goal-conditioned RL module in DECSTR .
Revised: This section describes the implementation of the intrinsically motivated goal-conditioned RL module in DECSTR .

Original: It is powered by the Soft-Actor Critic algorithm ( SAC ) (Haarnoja et al., 2018) that takes as input the current state, the current semantic conﬁguration and the goal conﬁguration, for both the critic and the policy.
Revised: It is powered by the Soft-Actor Critic algorithm ( SAC ) (Haarnoja et al., 2018) that takes as input the current state, the current semantic conﬁguration and the goal conﬁguration, for both the critic and the policy.

Original: We use Hindsight Experience Replay ( HER ) to facilitate transfer between goals (Andrychowicz et al., 2017).
Revised: We use Hindsight Experience Replay ( HER ) to facilitate transfer between goals (Andrychowicz et al., 2017).

Original: DECSTR samples goals via its curriculum strategy, collects experience in the environment, then performs policy updates via SAC .
Revised: DECSTR samples goals via its curriculum strategy, collects experience in the environment, then performs policy updates via SAC .

Original: This section describes two particularities of our RL implementation: the self-generated goal selection curriculum and the object-centered network architectures.
Revised: This section describes two particularities of our RL implementation: the self-generated goal selection curriculum and the object-centered network architectures.

Original: Implementation details and hyperparameters can be found in Appendix C.
Revised: Implementation details and hyperparameters can be found in Appendix C.

Original: Goal selection and curriculum learning.
Revised: Goal selection and curriculum learning.

Original: The DECSTR agent can only select goals among the set of semantic conﬁgurations it already experienced.
Revised: The DECSTR agent can only select goals among the set of semantic conﬁgurations it already experienced.

Original: We use an automatic curriculum strategy (Portelas et al., 2020) inspired from the CURIOUS algorithm (Colas et al., 2019a).
Revised: We use an automatic curriculum strategy (Portelas et al., 2020) inspired from the CURIOUS algorithm (Colas et al., 2019a).

Original: The DECSTR agent tracks aggregated estimations of its competence ( C ) and learning progress ( LP ).
Revised: The DECSTR agent tracks aggregated estimations of its competence ( C ) and learning progress ( LP ).

Original: Its selection of goals to target during data collection and goals to learn about during policy updates (via HER ) is biased towards goals associated with high absolute LP and low C .
Revised: Its selection of goals to target during data collection and goals to learn about during policy updates (via HER ) is biased towards goals associated with high absolute LP and low C .

Original: Automatic bucket generation.
Revised: Automatic bucket generation.

Original: To facilitate robust estimation, LP is usually estimated on sets of goals with similar difﬁculty or similar dynamics (Forestier et al., 2017; Colas et al., 2019a).
Revised: To facilitate robust estimation, LP is usually estimated on sets of goals with similar difﬁculty or similar dynamics (Forestier et al., 2017; Colas et al., 2019a).

Original: While previous works leveraged expert-deﬁned goal buckets , we cluster goals based on their time of discovery, as the time of discovery is a good proxy for goal difﬁculty: easier goals are discovered earlier.
Revised: While previous works leveraged expert-deﬁned goal buckets , we cluster goals based on their time of discovery, as the time of discovery is a good proxy for goal difﬁculty: easier goals are discovered earlier.

Original: Buckets are initially empty (no known conﬁgurations).
Revised: Buckets are initially empty (no known conﬁgurations).

Original: When an episode ends in a new conﬁguration, the N b = 5 buckets are updated.
Revised: When an episode ends in a new conﬁguration, the N b = 5 buckets are updated.

Original: Buckets are ﬁlled equally and the ﬁrst buckets contain the conﬁgurations discovered earlier.
Revised: Buckets are ﬁlled equally and the ﬁrst buckets contain the conﬁgurations discovered earlier.

Original: Thus goals change buckets as new goals are discovered.
Revised: Thus goals change buckets as new goals are discovered.

Original: Tracking competence, learning progress and sampling probabilities.
Revised: Tracking competence, learning progress and sampling probabilities.

Original: Regularly, the DECSTR agent evaluates itself on goal conﬁgurations sampled uniformly from the set of known ones.
Revised: Regularly, the DECSTR agent evaluates itself on goal conﬁgurations sampled uniformly from the set of known ones.

Original: For each bucket, it tracks the recent history of past successes and failures when targeting the corresponding goals (last W = 1800 self-evaluations).
Revised: For each bucket, it tracks the recent history of past successes and failures when targeting the corresponding goals (last W = 1800 self-evaluations).

Original: C is estimated as the success rate over the most recent half of that history C = C recent .
Revised: C is estimated as the success rate over the most recent half of that history C = C recent .

Original: LP is estimated as the difference between C recent and the one evaluated over the ﬁrst half of the history ( C earlier ).
Revised: LP is estimated as the difference between C recent and the one evaluated over the ﬁrst half of the history ( C earlier ).

Original: This is a crude estimation of the derivative of the C curve w.r.t.
Revised: This is a crude estimation of the derivative of the C curve w.r.t.

Original: time:
Revised: time:

Original: LP = C recent - C earlier .
Revised: LP = C recent - C earlier .

Original: The sampling probability P i for bucket i is:
Revised: The sampling probability P i for bucket i is:

Original: [Equation]
Revised: [Equation]

Original: In addition to the usual LP bias (Colas et al., 2019a), this formula favors lower C when LP is similar.
Revised: In addition to the usual LP bias (Colas et al., 2019a), this formula favors lower C when LP is similar.

Original: The absolute value ensures resampling buckets whose performance decreased (e.g. forgetting).
Revised: The absolute value ensures resampling buckets whose performance decreased (e.g. forgetting).

Original: Object-centered architecture.
Revised: Object-centered architecture.

Original: Instead of fully-connected or recurrent networks, DECSTR uses for the policy and critic an object-centered architecture similar to the ones used in Colas et al.
Revised: Instead of fully-connected or recurrent networks, DECSTR uses for the policy and critic an object-centered architecture similar to the ones used in Colas et al.

Original: Karch et al.
Revised: Karch et al.

Original: (2020), adapted from Deep-Sets (Zaheer et al., 2017).
Revised: (2020), adapted from Deep-Sets (Zaheer et al., 2017).

Original: For each pair of objects, a shared network independently encodes the concatenation of body and objects features and current and target semantic conﬁgurations, see Appendix Figure 4.
Revised: For each pair of objects, a shared network independently encodes the concatenation of body and objects features and current and target semantic conﬁgurations, see Appendix Figure 4.

Original: This shared network ensures efﬁcient transfer of skills between pairs of objects.
Revised: This shared network ensures efﬁcient transfer of skills between pairs of objects.

Original: A second inductive bias leverages the symmetry of the behavior required to achieve above ( o i , o j ) and above ( o j , o i ) .
Revised: A second inductive bias leverages the symmetry of the behavior required to achieve above ( o i , o j ) and above ( o j , o i ) .

Original: To ensure automatic transfer between the two, we present half of the features (e.g. those based on pairs ( o i , o j ) where i < j ) with goals containing one side of the symmetry (all above ( o i , o j ) for i < j ) and the other half with the goals containing the other side (all above ( o j , o i ) for i < j ).
Revised: To ensure automatic transfer between the two, we present half of the features (e.g. those based on pairs ( o i , o j ) where i < j ) with goals containing one side of the symmetry (all above ( o i , o j ) for i < j ) and the other half with the goals containing the other side (all above ( o j , o i ) for i < j ).

Original: As a result, the above ( o i , o j ) predicates fall into the same slot of the shared network inputs as their symmetric counterparts above ( o j , o i ) , only with different permutations of object pairs.
Revised: As a result, the above ( o i , o j ) predicates fall into the same slot of the shared network inputs as their symmetric counterparts above ( o j , o i ) , only with different permutations of object pairs.

Original: Goals are now of size 6 : 3 close and 3 above predicates, corresponding to one side of the above symmetry.
Revised: Goals are now of size 6 : 3 close and 3 above predicates, corresponding to one side of the above symmetry.

Original: Skill transfer between symmetric predicates are automatically ensured.
Revised: Skill transfer between symmetric predicates are automatically ensured.

Original: Appendix C.1 further describes these inductive biases and our modular architecture.
Revised: Appendix C.1 further describes these inductive biases and our modular architecture.

Original: 3. LANGUAGE - CONDITIONED GOAL GENERATION
Revised: 3. LANGUAGE - CONDITIONED GOAL GENERATION

Original: The language-conditioned goal generation module ( LGG ) is a generative model of semantic representations conditioned by language inputs.
Revised: The language-conditioned goal generation module ( LGG ) is a generative model of semantic representations conditioned by language inputs.

Original: It is trained to generate semantic conﬁgurations matching the agent’s initial conﬁguration and the description of a change in one object-pair relation.
Revised: It is trained to generate semantic conﬁgurations matching the agent’s initial conﬁguration and the description of a change in one object-pair relation.

Original: A training dataset is collected via interactions between a DECSTR agent trained in phase G → B and a social partner.
Revised: A training dataset is collected via interactions between a DECSTR agent trained in phase G → B and a social partner.

Original: DECSTR generates semantic goals and pursues them.
Revised: DECSTR generates semantic goals and pursues them.

Original: For each trajectory, the social partner provides a description d of one change in objects relations from the initial conﬁguration
Revised: For each trajectory, the social partner provides a description d of one change in objects relations from the initial conﬁguration

Original: c
Revised: c

Original: i to the ﬁnal one c f .
Revised: i to the ﬁnal one c f .

Original: The set of possible descriptions contains 102 sentences, each describing, in a simpliﬁed language, a positive or negative shift for one of the 9 predicates (e.g. get red above green ).
Revised: The set of possible descriptions contains 102 sentences, each describing, in a simpliﬁed language, a positive or negative shift for one of the 9 predicates (e.g. get red above green ).

Original: This leads to a dataset D of 5000 triplets: ( c i , d , c f ) .
Revised: This leads to a dataset D of 5000 triplets: ( c i , d , c f ) .

Original: From this dataset, the LGG is learned using a conditional Variational Auto-Encoder ( C - VAE )
Revised: From this dataset, the LGG is learned using a conditional Variational Auto-Encoder ( C - VAE )

Original: (Sohn et al., 2015).
Revised: (Sohn et al., 2015).

Original: Inspired by the contextconditioned goal generator from Nair et al.
Revised: Inspired by the contextconditioned goal generator from Nair et al.

Original: (2019), we add an extra condition on language instruction to improve control on goal generation.
Revised: (2019), we add an extra condition on language instruction to improve control on goal generation.

Original: The conditioning instruction is encoded by a recurrent network that is jointly trained with the VAE via a mixture of Kullback-Leibler and cross-entropy losses.
Revised: The conditioning instruction is encoded by a recurrent network that is jointly trained with the VAE via a mixture of Kullback-Leibler and cross-entropy losses.

Original: Appendix C.2 provides the list of sentences and implementation details.
Revised: Appendix C.2 provides the list of sentences and implementation details.

Original: By repeatedly sampling the LGG , a set of goals is built for any language input.
Revised: By repeatedly sampling the LGG , a set of goals is built for any language input.

Original: This enables skill diversity and strategy switching : if the agent fails, it can sample another valid goal to fulﬁll the instruction, effectively switching strategy.
Revised: This enables skill diversity and strategy switching : if the agent fails, it can sample another valid goal to fulﬁll the instruction, effectively switching strategy.

Original: This also enables goal combination using logical functions of instructions: and is an intersection, or is an union and not is the complement within the known set of goals.
Revised: This also enables goal combination using logical functions of instructions: and is an intersection, or is an union and not is the complement within the known set of goals.

Original: 3. EVALUATION OF THE THREE LGB PHASES
Revised: 3. EVALUATION OF THE THREE LGB PHASES

Original: Skill learning phase G → B :
Revised: Skill learning phase G → B :

Original: DECSTR explores its semantic representation space, discovers achievable conﬁgurations and learns to reach them.
Revised: DECSTR explores its semantic representation space, discovers achievable conﬁgurations and learns to reach them.

Original: Goal-speciﬁc performance is evaluated ofﬂine across learning as the success rate ( SR ) over 20 repetitions for each goal.
Revised: Goal-speciﬁc performance is evaluated ofﬂine across learning as the success rate ( SR ) over 20 repetitions for each goal.

Original: The global performance SR is measured across either the set of 35 goals or discovery-organized buckets of goals, see Section 3.4.
Revised: The global performance SR is measured across either the set of 35 goals or discovery-organized buckets of goals, see Section 3.4.

Original: Language grounding phase L → G :
Revised: Language grounding phase L → G :

Original: DECSTR trains the LGG to generate goals matching constraints expressed via language inputs.
Revised: DECSTR trains the LGG to generate goals matching constraints expressed via language inputs.

Original: From a given initial conﬁguration and a given instruction, the LGG should generate all compatible ﬁnal conﬁgurations (goals) and just these.
Revised: From a given initial conﬁguration and a given instruction, the LGG should generate all compatible ﬁnal conﬁgurations (goals) and just these.

Original: This is the source of behavioral diversity and strategy switching behaviors.
Revised: This is the source of behavioral diversity and strategy switching behaviors.

Original: To evaluate LGG , we construct a synthetic, oracle dataset O of triplets ( c i , d , C f ( c i , d )) , where C f ( c i , d ) is the set of all ﬁnal conﬁgurations compatible with ( c i , d ) .
Revised: To evaluate LGG , we construct a synthetic, oracle dataset O of triplets ( c i , d , C f ( c i , d )) , where C f ( c i , d ) is the set of all ﬁnal conﬁgurations compatible with ( c i , d ) .

Original: On average, C f in O contains 16 .
Revised: On average, C f in O contains 16 .

Original: 7 conﬁgurations, while the training dataset
Revised: 7 conﬁgurations, while the training dataset

Original: D only contains 3 .
Revised: D only contains 3 .

Original: We are interested in two metrics: 1)
Revised: We are interested in two metrics: 1)

Original: The Precision is the probability that a goal sampled from the LGG belongs to C f (true positive / all positive); 2)
Revised: The Precision is the probability that a goal sampled from the LGG belongs to C f (true positive / all positive); 2)

Original: The Recall is percentage of elements from C f that were found by sampling the LGG 100 times (true positive / all true).
Revised: The Recall is percentage of elements from C f that were found by sampling the LGG 100 times (true positive / all true).

Original: These metrics are computed on 5 different subsets of the oracle dataset, each calling for a different type of generalization (see full lists of instructions in Appendix C.2):
Revised: These metrics are computed on 5 different subsets of the oracle dataset, each calling for a different type of generalization (see full lists of instructions in Appendix C.2):

Original: Pairs found in D , except pairs removed to form the following test sets.
Revised: Pairs found in D , except pairs removed to form the following test sets.

Original: This calls for the extrapolation of known initialization-effect pairs ( c i , d ) to new ﬁnal conﬁgurations c f
Revised: This calls for the extrapolation of known initialization-effect pairs ( c i , d ) to new ﬁnal conﬁgurations c f

Original: ( D contains only 20% of C f on average).
Revised: ( D contains only 20% of C f on average).

Original: Pairs that were removed from D , calling for a recombination of known effects d on known c i .
Revised: Pairs that were removed from D , calling for a recombination of known effects d on known c i .

Original: Pairs for which the c i was entirely removed from D .
Revised: Pairs for which the c i was entirely removed from D .

Original: This calls for the transfer of known effects d on unknown c i . 4.
Revised: This calls for the transfer of known effects d on unknown c i . 4.

Original: Pairs for which the d was entirely removed from D .
Revised: Pairs for which the d was entirely removed from D .

Original: This calls for generalization in the language space, to generalize unknown effects d from related descriptions and transpose this to known c i . 5.
Revised: This calls for generalization in the language space, to generalize unknown effects d from related descriptions and transpose this to known c i . 5.

Original: Pairs for which both the c i and the d were entirely removed from D .
Revised: Pairs for which both the c i and the d were entirely removed from D .

Original: This calls for the generalizations 3 and 4 combined.
Revised: This calls for the generalizations 3 and 4 combined.

Original: Instruction following phase L → G → B :
Revised: Instruction following phase L → G → B :

Original: DECSTR is instructed to modify an object relation by one of the 102 sentences.
Revised: DECSTR is instructed to modify an object relation by one of the 102 sentences.

Original: Conditioned on its current conﬁguration and instruction, it samples a compatible goal from the LGG , then pursues it with its goal-conditioned policy.
Revised: Conditioned on its current conﬁguration and instruction, it samples a compatible goal from the LGG , then pursues it with its goal-conditioned policy.

Original: We consider three evaluation settings: 1) performing a single instruction; 2) performing a sequence of instructions without failure; 3) performing a logical combination of instructions.
Revised: We consider three evaluation settings: 1) performing a single instruction; 2) performing a sequence of instructions without failure; 3) performing a logical combination of instructions.

Original: The transition setup measures the success rate of the agent when asked to perform the 102 instructions 5 times each, resetting the environment each time.
Revised: The transition setup measures the success rate of the agent when asked to perform the 102 instructions 5 times each, resetting the environment each time.

Original: In the expression setup, the agent is evaluated on 500 randomly generated logical functions of sentences, see the generation mechanism in Appendix C.2.
Revised: In the expression setup, the agent is evaluated on 500 randomly generated logical functions of sentences, see the generation mechanism in Appendix C.2.

Original: In both setups, we evaluate the performance in 1-shot ( SR 1 ) and 5-shot ( SR 5 ) settings.
Revised: In both setups, we evaluate the performance in 1-shot ( SR 1 ) and 5-shot ( SR 5 ) settings.

Original: In the 5-shot setting, the agent can perform strategy switching , to sample new goals when previous attempts failed (without reset).
Revised: In the 5-shot setting, the agent can perform strategy switching , to sample new goals when previous attempts failed (without reset).

Original: In the sequence setup, the agent must execute 20 sequences of random instructions without reset (5-shot).
Revised: In the sequence setup, the agent must execute 20 sequences of random instructions without reset (5-shot).

Original: We also test behavioral diversity.
Revised: We also test behavioral diversity.

Original: We ask DECSTR to follow each of the 102 instructions 50 times each and report the number of different achieved conﬁgurations.
Revised: We ask DECSTR to follow each of the 102 instructions 50 times each and report the number of different achieved conﬁgurations.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: Our experimental section investigates three questions: [4.1]: How does DECSTR perform in the three phases?
Revised: Our experimental section investigates three questions: [4.1]: How does DECSTR perform in the three phases?

Original: [4.2]: How does it compare to end-to-end language-conditioned approaches?
Revised: [4.2]: How does it compare to end-to-end language-conditioned approaches?

Original: [4.3]: Do we need intermediate representations to be semantic?
Revised: [4.3]: Do we need intermediate representations to be semantic

Original:  4. HOW DOES DECSTR PERFORM IN THE THREE PHASES ?
Revised: ? 4. HOW DOES DECSTR PERFORM IN THE THREE PHASES ?

Original: This section presents the performance of the DECSTR agent in the skill learning, language grounding, and instruction following phases.
Revised: This section presents the performance of the DECSTR agent in the skill learning, language grounding, and instruction following phases.

Original: Skill learning phase G → B : Figure 3 shows that DECSTR successfully masters all reachable conﬁgurations in its semantic representation space.
Revised: Skill learning phase G → B : Figure 3 shows that DECSTR successfully masters all reachable conﬁgurations in its semantic representation space.

Original: Figure 3a shows the evolution of SR computed per bucket.
Revised: Figure 3a shows the evolution of SR computed per bucket.

Original: Buckets are learned in increasing order, which conﬁrms that the time of discovery is a good proxy for difﬁculty.
Revised: Buckets are learned in increasing order, which conﬁrms that the time of discovery is a good proxy for difﬁculty.

Original: Figure 3b reports C , LP and sampling probabilities P computed online using self-evaluations for an example agent.
Revised: Figure 3b reports C , LP and sampling probabilities P computed online using self-evaluations for an example agent.

Original: The agent leverages these estimations to select its goals:
Revised: The agent leverages these estimations to select its goals:

Original: ﬁrst focusing on the easy goals from bucket 1, it moves on towards harder and harder buckets as easier ones are mastered (low LP , high C ).
Revised: ﬁrst focusing on the easy goals from bucket 1, it moves on towards harder and harder buckets as easier ones are mastered (low LP , high C ).

Original: Figure 3c presents the results of ablation studies.
Revised: Figure 3c presents the results of ablation studies.

Original: Each condition removes one component of DECSTR : 1) Flat replaces our object-centered modular architectures by ﬂat ones; 2) w/o Curr.
Revised: Each condition removes one component of DECSTR : 1) Flat replaces our object-centered modular architectures by ﬂat ones; 2) w/o Curr.

Original: replaces our automatic curriculum strategy by a uniform goal selection; 3) w/o Sym. does not use the symmetry inductive bias; 4)
Revised: replaces our automatic curriculum strategy by a uniform goal selection; 3) w/o Sym. does not use the symmetry inductive bias; 4)

Original: In w/o SP , the social partner does not provide non-trivial initial conﬁgurations.
Revised: In w/o SP , the social partner does not provide non-trivial initial conﬁgurations.

Original: In the Expert buckets condition, the curriculum strategy is applied on expert-deﬁned buckets, see Appendix D.1.
Revised: In the Expert buckets condition, the curriculum strategy is applied on expert-deﬁned buckets, see Appendix D.1.

Original: The full version of LGB performs on par with the
Revised: The full version of LGB performs on par with the

Original: Expert buckets oracle and outperforms signiﬁcantly all its ablations.
Revised: Expert buckets oracle and outperforms signiﬁcantly all its ablations.

Original: Appendix E.3 presents more examples of learning trajectories, and dissects the evolution of bucket compositions along training.
Revised: Appendix E.3 presents more examples of learning trajectories, and dissects the evolution of bucket compositions along training.

Original: [Figure]
Revised: [Figure]

Revised: S u cc e ss

Original: [Table]
Revised: R a t e DECSTRw/o Curr. Exp. Buckets w/o Asym.

Revised: Flatw/o ZPD (c)

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Language grounding phase L → G : The LGG demonstrates the 5 types of generalization from
Revised: Language grounding phase L → G : The LGG demonstrates the 5 types of generalization from

Original: Table 1.
Revised: Table 1.

Original: From known conﬁgurations, agents can generate more goals than they observed in training data (1, 2).
Revised: From known conﬁgurations, agents can generate more goals than they observed in training data (1, 2).

Original: They can do so from new initial conﬁgurations (3).
Revised: They can do so from new initial conﬁgurations (3).

Original: They can generalize to new sentences (4) and even to combinations of new sentences and initial conﬁgurations (5).
Revised: They can generalize to new sentences (4) and even to combinations of new sentences and initial conﬁgurations (5).

Original: These results assert that DECSTR generalizes well in a variety of contexts and shows good behavioral diversity.
Revised: These results assert that DECSTR generalizes well in a variety of contexts and shows good behavioral diversity.

Original: Instruction following phase L → G → B : Table 2 presents the 1-shot and 5-shot results in the transition and expression setups.
Revised: Instruction following phase L → G → B : Table 2 presents the 1-shot and 5-shot results in the transition and expression setups.

Original: In the sequence setups, DECSTR succeeds in L = 14 .
Revised: In the sequence setups, DECSTR succeeds in L = 14 .

Original: 9 ± 5 .
Revised: 9 ± 5 .

Original: 7 successive instructions (mean ± stdev over 10 seeds).
Revised: 7 successive instructions (mean ± stdev over 10 seeds).

Original: These results conﬁrm efﬁcient language grounding.
Revised: These results conﬁrm efﬁcient language grounding.

Original: DECSTR can follow instructions or sequences of instructions and generalize to their logical combinations.
Revised: DECSTR can follow instructions or sequences of instructions and generalize to their logical combinations.

Original: Strategy switching improves performance ( SR 5 - SR 1 ).
Revised: Strategy switching improves performance ( SR 5 - SR 1 ).

Original: DECSTR also demonstrates strong behavioral diversity: when asked over 10 seeds to repeat 50 times the same instruction, it achieves at least 7 .different conﬁgurations, 15 . 6 on average and up to 23 depending on the instruction.
Revised: DECSTR also demonstrates strong behavioral diversity: when asked over 10 seeds to repeat 50 times the same instruction, it achieves at least 7 .different conﬁgurations, 15 . 6 on average and up to 23 depending on the instruction.

Original: 4. DO WE NEED AN INTERMEDIATE REPRESENTATION ?
Revised: 4. DO WE NEED AN INTERMEDIATE REPRESENTATION ?

Original: This section investigates the need for an intermediate semantic representation.
Revised: This section investigates the need for an intermediate semantic representation.

Original: To this end, we introduce an end-to-end LC - RL baseline directly mapping Language to Behavior ( L → B ) and compare its performance with DECSTR in the instruction following phase ( L → G → B ).
Revised: To this end, we introduce an end-to-end LC - RL baseline directly mapping Language to Behavior ( L → B ) and compare its performance with DECSTR in the instruction following phase ( L → G → B ).

Original: The LB baseline.
Revised: The LB baseline.

Original: To limit the introduction of confounding factors and under-tuning concerns, we base this implementation on the DECSTR code and incorporate deﬁning features of IMAGINE , a stateof-the-art language conditioned RL agent (Colas et al., 2020).
Revised: To limit the introduction of confounding factors and under-tuning concerns, we base this implementation on the DECSTR code and incorporate deﬁning features of IMAGINE , a state- of-the-art language conditioned RL agent (Colas et al., 2020).

Original: We keep the same HER mechanism, object-centered architectures and RL algorithm as DECSTR .
Revised: We keep the same HER mechanism, object-centered architectures and RL algorithm as DECSTR .

Original: We just replace the semantic goal space by the 102 language instructions.
Revised: We just replace the semantic goal space by the 102 language instructions.

Original: This baseline can be seen as an oracle version of the IMAGINE algorithm where the reward function is assumed perfect, but without the imagination mechanism.
Revised: This baseline can be seen as an oracle version of the IMAGINE algorithm where the reward function is assumed perfect, but without the imagination mechanism.

Original: Comparison in the instruction following phase L → B vs L → G → B : After training the LB baseline for 14 K episodes, we compare its performance to DECSTR ’s in the instruction-following setup.
Revised: Comparison in the instruction following phase L → B vs L → G → B : After training the LB baseline for 14 K episodes, we compare its performance to DECSTR ’s in the instruction-following setup.

Original: In the transition evaluation setup, LB achieves SR 1 = 0 .
Revised: In the transition evaluation setup, LB achieves SR 1 = 0 .

Original: 76 ± 0 .
Revised: 76 ± 0 .

Original: 001 : it always manages to move blocks close to or far from each other, but consistently fails to stack them.
Revised: 001 : it always manages to move blocks close to or far from each other, but consistently fails to stack them.

Original: Adding more attempts does not help:
Revised: Adding more attempts does not help:

Original: SR 5 = 0 .
Revised: SR 5 = 0 .

Original: 76 ± 0 .
Revised: 76 ± 0 .

Original: The LB baseline cannot be evaluated in the expression setup because it does not manipulate goal sets.
Revised: The LB baseline cannot be evaluated in the expression setup because it does not manipulate goal sets.

Original: Because it cannot stack blocks, LB only succeeds in 3 .
Revised: Because it cannot stack blocks, LB only succeeds in 3 .

Original: 01 ± 0 .
Revised: 01 ± 0 .

Original: 43 random instructions in a row, against 14 . 9 for DECSTR ( sequence setup).
Revised: 43 random instructions in a row, against 14 . 9 for DECSTR ( sequence setup).

Original: We then evaluate LB ’s diversity on the set of instructions it succeeds in.
Revised: We then evaluate LB ’s diversity on the set of instructions it succeeds in.

Original: When asked to repeat 50 times the same instruction, it achieves at least 3 .
Revised: When asked to repeat 50 times the same instruction, it achieves at least 3 .

Original: different conﬁgurations, 4 . 2 on average and up to 5 .
Revised: different conﬁgurations, 4 . 2 on average and up to 5 .

Original: 2 depending on the instruction against 7 . 8 , 17 . 1 , 23 on the same set of instructions for DECSTR .
Revised: 2 depending on the instruction against 7 . 8 , 17 . 1 , 23 on the same set of instructions for DECSTR .

Original: We did not observe strategy-switching behaviors in LB , because it either always succeeds (close/far instructions) or fails (stacks).
Revised: We did not observe strategy-switching behaviors in LB , because it either always succeeds (close/far instructions) or fails (stacks).

Original: Conclusion.
Revised: Conclusion.

Original: The introduction of an intermediate semantic representation helps DECSTR decouple skill learning from language grounding which, in turns, facilitates instruction-following when compared to the end-to-end language-conditioned learning of LB .
Revised: The introduction of an intermediate semantic representation helps DECSTR decouple skill learning from language grounding which, in turns, facilitates instruction-following when compared to the end-to-end language-conditioned learning of LB .

Original: This leads to improved scores in the transition and sequence setups.
Revised: This leads to improved scores in the transition and sequence setups.

Original: The direct language-conditioning of LB prevents the generalization to logical combination and leads to a reduced diversity in the set of mastered instructions.
Revised: The direct language-conditioning of LB prevents the generalization to logical combination and leads to a reduced diversity in the set of mastered instructions.

Original: Decoupling thus brings signiﬁcant beneﬁts to LGB architectures.
Revised: Decoupling thus brings signiﬁcant beneﬁts to LGB architectures.

Original: 4. DO WE NEED A SEMANTIC INTERMEDIATE REPRESENTATION ?
Revised: 4. DO WE NEED A SEMANTIC INTERMEDIATE REPRESENTATION ?

Original: This section investigates the need for the intermediate representation to be semantic.
Revised: This section investigates the need for the intermediate representation to be semantic.

Original: To this end, we introduce the LGB - C baseline that leverages continuous goal representations in place of semantic ones.
Revised: To this end, we introduce the LGB - C baseline that leverages continuous goal representations in place of semantic ones.

Original: We compare them on the two ﬁrst phases.
Revised: We compare them on the two ﬁrst phases.

Original: The LGB - C baseline.
Revised: The LGB - C baseline.

Original: The LGB - C baseline uses continuous goals expressing target block coordinates in place of semantic goals.
Revised: The LGB - C baseline uses continuous goals expressing target block coordinates in place of semantic goals.

Original: The skill learning phase is thus equivalent to traditional goalconditioned RL setups in block manipulation tasks (Andrychowicz et al., 2017; Colas et al., 2019a;
Revised: The skill learning phase is thus equivalent to traditional goalconditioned RL setups in block manipulation tasks (Andrychowicz et al., 2017; Colas et al., 2019a;

Original: Li et al., 2019; Lanier et al., 2019).
Revised: Li et al., 2019; Lanier et al., 2019).

Original: Starting from the DECSTR algorithm, LGB - C adds a translation module that samples a set of target block coordinates matching the targeted semantic conﬁguration which is then used as the goal input to the policy.
Revised: Starting from the DECSTR algorithm, LGB - C adds a translation module that samples a set of target block coordinates matching the targeted semantic conﬁguration which is then used as the goal input to the policy.

Original: In addition, we integrate deﬁning features of the state-of-the-art approach from Lanier et al.
Revised: In addition, we integrate deﬁning features of the state-of-the-art approach from Lanier et al.

Original: (2019): non-binary rewards (+1 for each well placed block) and multi-criteria HER , see details in Appendix D.2.
Revised: (2019): non-binary rewards (+1 for each well placed block) and multi-criteria HER , see details in Appendix D.2.

Original: Comparison in skill learning phase G → B : The LGB - C baseline successfully learns to discover and master all 35 semantic conﬁgurations by placing the three blocks to randomly-sampled target coordinates corresponding to these conﬁgurations.
Revised: Comparison in skill learning phase G → B : The LGB - C baseline successfully learns to discover and master all 35 semantic conﬁgurations by placing the three blocks to randomly-sampled target coordinates corresponding to these conﬁgurations.

Original: It does so faster than DECSTR : 708 · 10 3 episodes to reach SR = 95% , against 1238 · 10 3 for DECSTR , see Appendix Figure 6.
Revised: It does so faster than DECSTR : 708 · 10 3 episodes to reach SR = 95% , against 1238 · 10 3 for DECSTR , see Appendix Figure 6.

Original: This can be explained by the denser learning signals it gets from using HER on continuous targets instead of discrete ones.
Revised: This can be explained by the denser learning signals it gets from using HER on continuous targets instead of discrete ones.

Original: In this phase, however, the agent only learns one parameterized skill: to place blocks at their target position.
Revised: In this phase, however, the agent only learns one parameterized skill: to place blocks at their target position.

Original: It cannot build a repertoire of semantic skills because it cannot discriminate between different block conﬁgurations.
Revised: It cannot build a repertoire of semantic skills because it cannot discriminate between different block conﬁgurations.

Original: Looking at the sum of the distances travelled by the blocks or the completion time, we ﬁnd that DECSTR performs opportunistic goal reaching: it ﬁnds simpler conﬁgurations of the blocks which satisfy its semantic goals compared to LGB - C .
Revised: Looking at the sum of the distances travelled by the blocks or the completion time, we ﬁnd that DECSTR performs opportunistic goal reaching: it ﬁnds simpler conﬁgurations of the blocks which satisfy its semantic goals compared to LGB - C .

Original: Blocks move less ( ∆ dist = 26 ±cm), and goals are reached faster ( ∆ steps = 13 ± 4 , mean ± std across goals with p-values > 1 . 3 · 10 −and 3 .
Revised: Blocks move less ( ∆ dist = 26 ±cm), and goals are reached faster ( ∆ steps = 13 ± 4 , mean ± std across goals with p-values > 1 . 3 · 10 −and 3 .

Original: 2 · 10 − 19 respectively).
Revised: 2 · 10 − 19 respectively).

Original: [Table]
Revised: [Table]

Original: Comparison in language grounding phase L → G : We train the LGG to generate continuous target coordinates conditioned on language inputs with a mean-squared loss and evaluate it in the same setup as DECSTR ’s LGG , see Table 3.
Revised: Comparison in language grounding phase L → G : We train the LGG to generate continuous target coordinates conditioned on language inputs with a mean-squared loss and evaluate it in the same setup as DECSTR ’s LGG , see Table 3.

Original: Although it maintains reasonable precision in the ﬁrst two testing sets, the LGG achieves low recall – i.e. diversity – on all sets.
Revised: Although it maintains reasonable precision in the ﬁrst two testing sets, the LGG achieves low recall – i.e. diversity – on all sets.

Original: The lack of semantic representations of skills might explain the difﬁculty of training a language-conditioned goal generator.
Revised: The lack of semantic representations of skills might explain the difﬁculty of training a language-conditioned goal generator.

Original: Conclusion.
Revised: Conclusion.

Original: The skill learning phase of the LGB - C baseline is competitive with the one of
Revised: The skill learning phase of the LGB - C baseline is competitive with the one of

Original: DECSTR .
Revised: DECSTR .

Original: However, the poor performance in the language grounding phase prevents this baseline to perform instruction following.
Revised: However, the poor performance in the language grounding phase prevents this baseline to perform instruction following.

Original: For this reason, and because semantic representations enable agents to perform opportunistic goal reaching and to acquire repertoires for semantic skills, we believe the semantic representation is an essential part of the LGB architecture.
Revised: For this reason, and because semantic representations enable agents to perform opportunistic goal reaching and to acquire repertoires for semantic skills, we believe the semantic representation is an essential part of the LGB architecture.

Original:  DISCUSSION AND CONCLUSION
Revised:  DISCUSSION AND CONCLUSION

Original: This paper contributes LGB , a new conceptual RL architecture which introduces an intermediate semantic representation to decouple sensorimotor learning from language grounding.
Revised: This paper contributes LGB , a new conceptual RL architecture which introduces an intermediate semantic representation to decouple sensorimotor learning from language grounding.

Original: To demonstrate its beneﬁts, we present DECSTR , a learning agent that discovers and masters all reachable conﬁgurations in a manipulation domain from a set of relational spatial primitives, before undertaking an efﬁcient language grounding phase.
Revised: To demonstrate its beneﬁts, we present DECSTR , a learning agent that discovers and masters all reachable conﬁgurations in a manipulation domain from a set of relational spatial primitives, before undertaking an efﬁcient language grounding phase.

Original: This was made possible by the use of object-centered inductive biases, a new form of automatic curriculum learning and a novel language-conditioned goal generation module.
Revised: This was made possible by the use of object-centered inductive biases, a new form of automatic curriculum learning and a novel language-conditioned goal generation module.

Original: Note that our main contribution is in the conceptual approach, DECSTR being only an instance to showcase its beneﬁts.
Revised: Note that our main contribution is in the conceptual approach, DECSTR being only an instance to showcase its beneﬁts.

Original: We believe that this approach could beneﬁt from any improvement in GC - RL (for skill learning) or generative models (for language grounding).
Revised: We believe that this approach could beneﬁt from any improvement in GC - RL (for skill learning) or generative models (for language grounding).

Original: Semantic representations.
Revised: Semantic representations.

Original: Results have shown that using predicate-based representations was sufﬁcient for DECSTR to efﬁciently learn abstract goals in an opportunistic manner.
Revised: Results have shown that using predicate-based representations was sufﬁcient for DECSTR to efﬁciently learn abstract goals in an opportunistic manner.

Original: The proposed semantic conﬁgurations showcase promising properties: 1) they reduce the complexity of block manipulation where most effective works rely on a heavy hand-crafted curriculum (Li et al., 2019;
Revised: The proposed semantic conﬁgurations showcase promising properties: 1) they reduce the complexity of block manipulation where most effective works rely on a heavy hand-crafted curriculum (Li et al., 2019;

Original: Lanier et al., 2019) and a speciﬁc curiosity mechanism (Li et al., 2019); 2) they facilitate the grounding of language into skills and 3) they enable decoupling skill learning from language grounding, as observed in infants (Piaget, 1977).
Revised: Lanier et al., 2019) and a speciﬁc curiosity mechanism (Li et al., 2019); 2) they facilitate the grounding of language into skills and 3) they enable decoupling skill learning from language grounding, as observed in infants (Piaget, 1977).

Original: The set of semantic predicates is, of course, domain-dependent as it characterizes the space of behaviors that the agent can explore.
Revised: The set of semantic predicates is, of course, domain-dependent as it characterizes the space of behaviors that the agent can explore.

Original: However, we believe it is easier and requires less domain knowledge to deﬁne the set of predicates, i.e. the dimensions of the space of potential goals, than it is to craft a list of goals and their associated reward functions.
Revised: However, we believe it is easier and requires less domain knowledge to deﬁne the set of predicates, i.e. the dimensions of the space of potential goals, than it is to craft a list of goals and their associated reward functions.

Original: A new approach to language grounding.
Revised: A new approach to language grounding.

Original: The approach proposed here is the ﬁrst simultaneously enabling to decouple skill learning from language grounding and fostering a diversity of possible behaviors for given instructions.
Revised: The approach proposed here is the ﬁrst simultaneously enabling to decouple skill learning from language grounding and fostering a diversity of possible behaviors for given instructions.

Original: Indeed, while an instruction following agent trained on goals like put red close to green would just push the red block towards the green one, our agent can generate many matching goal conﬁgurations.
Revised: Indeed, while an instruction following agent trained on goals like put red close to green would just push the red block towards the green one, our agent can generate many matching goal conﬁgurations.

Original: It could build a pyramid, make a blue-green-red pile or target a dozen other compatible conﬁgurations.
Revised: It could build a pyramid, make a blue-green-red pile or target a dozen other compatible conﬁgurations.

Original: This enables it to switch strategy , to ﬁnd alternative approaches to satisfy a same instruction when ﬁrst attempts failed.
Revised: This enables it to switch strategy , to ﬁnd alternative approaches to satisfy a same instruction when ﬁrst attempts failed.

Original: Our goal generation module can also generalize to new sentences or transpose instructed transformations to unknown initial conﬁgurations.
Revised: Our goal generation module can also generalize to new sentences or transpose instructed transformations to unknown initial conﬁgurations.

Original: Finally, with the goal generation module, the agent can deal with any logical expression made of instructions by combining generated goal sets.
Revised: Finally, with the goal generation module, the agent can deal with any logical expression made of instructions by combining generated goal sets.

Original: It would be of interest to simultaneously perform language grounding and skill learning, which would result in “overlapping waves” of sensorimotor and linguistic development (Siegler, 1998).
Revised: It would be of interest to simultaneously perform language grounding and skill learning, which would result in “overlapping waves” of sensorimotor and linguistic development (Siegler, 1998).

Original: Semantic conﬁgurations of variable size.
Revised: Semantic conﬁgurations of variable size.

Original: Considering a constant number of blocks and, thus, ﬁxed-size conﬁguration spaces is a current limit of DECSTR .
Revised: Considering a constant number of blocks and, thus, ﬁxed-size conﬁguration spaces is a current limit of DECSTR .

Original: Future implementations of LGB may handle inputs of variable sizes by leveraging Graph Neural Networks as in Li et al.
Revised: Future implementations of LGB may handle inputs of variable sizes by leveraging Graph Neural Networks as in Li et al.

Original: Corresponding semantic conﬁgurations could be represented as a set of vectors, each encoding information about a predicate and the objects it applies to.
Revised: Corresponding semantic conﬁgurations could be represented as a set of vectors, each encoding information about a predicate and the objects it applies to.

Original: These representations could be handled by Deep
Revised: These representations could be handled by Deep

Original: Sets (Zaheer et al., 2017).
Revised: Sets (Zaheer et al., 2017).

Original: This would allow to target partial sets of predicates that would not need to characterize all relations between all objects, facilitating scalability.
Revised: This would allow to target partial sets of predicates that would not need to characterize all relations between all objects, facilitating scalability.

Original: Conclusion In this work, we have shown that introducing abstract goals based on relational predicates that are well understood by humans can serve as a pivotal representation between skill learning and interaction with a user through language.
Revised: Conclusion In this work, we have shown that introducing abstract goals based on relational predicates that are well understood by humans can serve as a pivotal representation between skill learning and interaction with a user through language.

Original: Here, the role of the social partner was limited to: 1) helping the agent to experience non-trivial conﬁgurations and 2) describing the agent’s behavior in a simpliﬁed language.
Revised: Here, the role of the social partner was limited to: 1) helping the agent to experience non-trivial conﬁgurations and 2) describing the agent’s behavior in a simpliﬁed language.

Original: In the future, we intend to study more intertwined skill learning and language grounding phases, making it possible to the social partner to teach the agent during skill acquisition.
Revised: In the future, we intend to study more intertwined skill learning and language grounding phases, making it possible to the social partner to teach the agent during skill acquisition.

Original: ACKNOWLEDGMENTS
Revised: ACKNOWLEDGMENTS

Original: Anonymized
Revised: This work was performed using HPC resources from GENCI-IDRIS (Grant 20XX-AP010611667), the MeSU platform at Sorbonne-Universit´e and the PlaFRIM experimental testbed. C´edric Colas is partly funded by the French Minist`ere des Arm´ees - Direction G´en´erale de l’Armement.

Original: A LGB PSEUDO - CODE
Revised: A LGB PSEUDO - CODE

Original: Algorithm 1 and 2 present the high-level pseudo-code of any algorithm following the LGB architecture for each of the three phases.
Revised: Algorithm 1 and 2 present the high-level pseudo-code of any algorithm following the LGB architecture for each of the three phases.

Original: 1: Require Env E 2: Initialize policy Π , goal sampler G s , buffer B 3: loop 4: g ← G s .sample() 5: ( s, a, s (cid:48) , g, c p , c (cid:48) p ) traj ← E
Revised: 1: Require Env E 2: Initialize policy Π , goal sampler G s , buffer B 3: loop 4: g ← G s .sample() 5: ( s, a, s (cid:48) , g, c p , c (cid:48) p ) traj ← E

Original: .rollout( g ) 6:
Revised: .rollout( g ) 6:

Original: G s .update( c Tp ) 7:
Revised: G s .update( c Tp ) 7:

Original: B .update( ( s, a, s (cid:48) , g, c p , c (cid:48) p ) traj ) 8: Π .update( B ) 9: return Π , G s 10: 11: 12:
Revised: B .update( ( s, a, s (cid:48) , g, c p , c (cid:48) p ) traj ) 8: Π .update( B ) 9: return Π , G s 10: 11: 12:

Original: (cid:46)
Revised: (cid:46)

Original: → Goal phase 1: Require Π , E, G s , social partner SP 2: Initialize language goal generator LGG 3: dataset ← SP .interact( E, Π , G s ) 4: LGG .update(dataset) 5: return LGG (cid:46) Language → Behavior phase 6: Require E, Π , LGG, SP 7: loop 8: instr.
Revised: 1: Require Π , E, G s , social partner SP 2: Initialize language goal generator LGG 3: dataset ← SP .interact( E, Π , G s ) 4: LGG .update(dataset) 5: return LGG (cid:46) Language → Behavior phase 6: Require E, Π , LGG, SP 7: loop 8: instr.

Original: ← SP .listen() 9: loop (cid:46) Strategy switching loop 10: g ← LGG .sample(instr.
Revised: ← SP .listen() 9: loop (cid:46) Strategy switching loop 10: g ← LGG .sample(instr.

Original: , c 0 ) 11: c
Revised: , c 0 ) 11: c

Original: Tp ← E .rollout
Revised: Tp ← E .rollout

Original: ( g ) 12: if g ==
Revised: ( g ) 12: if g ==

Original: c
Revised: c

Original: Tp then break
Revised: Tp then break

Original: B S EMANTIC PREDICATES AND APPLICATION TO FETCH MANIPULATE
Revised: B S EMANTIC PREDICATES AND APPLICATION TO FETCH MANIPULATE

Original: In this paper, we restrict the semantic representations to the use of the close and above binary predicates applied to M = 3 objects.
Revised: In this paper, we restrict the semantic representations to the use of the close and above binary predicates applied to M = 3 objects.

Original: The resulting semantic conﬁgurations are formed by: c p =
Revised: The resulting semantic conﬁgurations are formed by: c p =

Original: [ c ( o 1 , o 2 ) , c ( o 1 , o 3 ) , c ( o 2 , o 3 ) , a ( o 1 , o 2 ) , a ( o 2 , o 1 ) , a ( o 1 , o 3 ) , a ( o 3 , o 1 ) , a ( o 2 , o 3 ) , a ( o 3 , o 2 )] , where c() and a() refer to the close and above predicates respectively and ( o 1 , o 2 , o 3 ) are the red, green and blue blocks respectively.
Revised: [ c ( o 1 , o 2 ) , c ( o 1 , o 3 ) , c ( o 2 , o 3 ) , a ( o 1 , o 2 ) , a ( o 2 , o 1 ) , a ( o 1 , o 3 ) , a ( o 3 , o 1 ) , a ( o 2 , o 3 ) , a ( o 3 , o 2 )] , where c() and a() refer to the close and above predicates respectively and ( o 1 , o 2 , o 3 ) are the red, green and blue blocks respectively.

Original: Symmetry and asymmetry of close and above predicates.
Revised: Symmetry and asymmetry of close and above predicates.

Original: We consider objects o 

Revised: We consider objects o

Original: and o 2 .
Revised: and o 2 .

Original: • close is symmetric: “ o 1 is close to o 2 ” ⇔ “ o 2 is close to o 1 ”.
Revised: • close is symmetric: “ o 1 is close to o 2 ” ⇔ “ o 2 is close to o 1 ”.

Original: The corresponding semantic mapping function is based on the Euclidean distance, which is symmetric.
Revised: The corresponding semantic mapping function is based on the Euclidean distance, which is symmetric.

Original: • above is asymmetric: “ o 1 is above o 2 ” ⇒ not “ o 2 is above o 1 ”.
Revised: • above is asymmetric: “ o 1 is above o 2 ” ⇒ not “ o 2 is above o 1 ”.

Original: The corresponding semantic mapping function evaluates the sign of the difference of the object Z -axis coordinates.
Revised: The corresponding semantic mapping function evaluates the sign of the difference of the object Z -axis coordinates.

Original: C THE DECSTR ALGORITHM C. INTRINSICALLY MOTIVATED GOAL - CONDITIONED RL
Revised: C THE DECSTR ALGORITHM C. INTRINSICALLY MOTIVATED GOAL - CONDITIONED RL

Original: Overview.
Revised: Overview.

Original: Algorithm 3 presents the pseudo-code of the sensorimotor learning phase ( G → B ) of
Revised: Algorithm 3 presents the pseudo-code of the sensorimotor learning phase ( G → B ) of

Original: DECSTR .
Revised: DECSTR .

Original: It alternates between two steps:
Revised: It alternates between two steps:

Original: • Data acquisition.
Revised: • Data acquisition.

Original: A DECSTR agent has no prior on the set of reachable semantic conﬁgurations.
Revised: A DECSTR agent has no prior on the set of reachable semantic conﬁgurations.

Original: Its ﬁrst goal is sampled uniformly from the semantic conﬁguration space.
Revised: Its ﬁrst goal is sampled uniformly from the semantic conﬁguration space.

Original: Using this goal, it starts interacting with its environment, generating trajectories of sensory states s , actions a and conﬁgurations c p .
Revised: Using this goal, it starts interacting with its environment, generating trajectories of sensory states s , actions a and conﬁgurations c p .

Original: The last conﬁguration c
Revised: The last conﬁguration c

Original: Tp achieved in the episode after T time steps is considered stable and is added to the set of reachable conﬁgurations.
Revised: Tp achieved in the episode after T time steps is considered stable and is added to the set of reachable conﬁgurations.

Original: As it interacts with the environment, the agent explores the conﬁguration space, discovers reachable conﬁgurations and selects new targets.
Revised: As it interacts with the environment, the agent explores the conﬁguration space, discovers reachable conﬁgurations and selects new targets.

Original: • Internal models updates.
Revised: • Internal models updates.

Original: A DECSTR agent updates two models: its curriculum strategy and its policy.
Revised: A DECSTR agent updates two models: its curriculum strategy and its policy.

Original: The curriculum strategy can be seen as an active goal sampler.
Revised: The curriculum strategy can be seen as an active goal sampler.

Original: It biases the selection of goals to target and goals to learn about.
Revised: It biases the selection of goals to target and goals to learn about.

Original: The policy is the module controlling the agent’s behavior and is updated via RL.
Revised: The policy is the module controlling the agent’s behavior and is updated via RL.

Original: 1: Require: env E , # buckets N b , # episodes before biased init.
Revised: 1: Require: env E , # buckets N b , # episodes before biased init.

Original: n unb , self-evaluation probability pself eval , noise function σ () 2: Initialize: policy Π , buffer B ,goal sampler G s , bucket sampling probabilities p b , language module LGG .
Revised: n unb , self-evaluation probability p , noise ) 2: Initialize: policy Π , buffer B , sampler G s , bucket sampling probabilities p b , language module LGG .

Original: 3: loop 4: self eval ← random() <
Revised: 3: loop 4: self eval ← random() <

Original: p self eval (cid:46)
Revised: p self eval (cid:46)

Original: If True then evaluate competence 5: g ← G s .sample(self eval, p b ) 6: biased init ← epoch < n unb (cid:46)
Revised: If True then evaluate competence 5: g ← G s .sample(self eval, p b ) 6: biased init ← epoch < n unb (cid:46)

Original: Bias initialization only after n unb epochs 7: s 0 , c 

Revised: Bias initialization only after n unb epochs 7: s 0 , c

Original: p ← E .reset( biased init ) (cid:46) c 0 : Initial semantic conﬁguration 8: for t = 1 : T do 9: a t ← policy ( s t , c t , g ) 10: if not self eval then 11: a t ← a t + σ () 12: s t +1 , c t +1 p ← E .step( a t ) 13: episode ← ( s, c, a, s (cid:48) , c (cid:48) ) 14:
Revised: p ← E .reset( biased init ) (cid:46) c 0 : Initial semantic conﬁguration 8: for t = 1 : T do 9: a t ← policy ( s t , c t , g ) 10: if not self eval then 11: a t ← a t + σ () 12: s t +1 , c t +1 p ← E .step( a t ) 13: episode ← ( s, c, a, s (cid:48) , c (cid:48) ) 14:

Original: G s .update( c T ) 15:
Revised: G s .update( c T ) 15:

Original: B .update(episode) 16: g ← G s .sample( p b ) 17: batch ← B .sample( g ) 18: Π .update(batch) 19: if self eval then 20: p b ← G s .update LP()
Revised: B .update(episode) 16: g ← G s .sample( p b ) 17: batch ← B .sample( g ) 18: Π .update(batch) 19: if self eval then 20: p b ← G s .update LP()

Original: Policy updates with a goal-conditioned Soft Actor-Critic.
Revised: Policy updates with a goal-conditioned Soft Actor-Critic.

Original: Readers familiar with Markov Decision Process and the use of SAC and HER algorithms can skip this paragraph.
Revised: Readers familiar with Markov Decision Process and the use of SAC and HER algorithms can skip this paragraph.

Original: We want the DECSTR agent to explore a semantic conﬁguration space and master reachable conﬁgurations in it.
Revised: We want the DECSTR agent to explore a semantic conﬁguration space and master reachable conﬁgurations in it.

Original: We frame this problem as a goal-conditioned MDP (Schaul et al., 2015):
Revised: We frame this problem as a goal-conditioned MDP (Schaul et al., 2015):

Original: M = ( S , G p , A , T , R , γ ) , where the state space S is the usual sensory space augmented with the conﬁguration space C p , the goal space G p is equal to the conﬁguration space G p =
Revised: M = ( S , G p , A , T , R , γ ) , where the state space S is the usual sensory space augmented with the conﬁguration space C p , the goal space G p is equal to the conﬁguration space G p =

Original: C p , A is the action space, T : S ×
Revised: C p , A is the action space, T : S ×

Original: A × S → [0 , 1] is the unknown transition probability, R : S ×
Revised: A × S → [0 , 1] is the unknown transition probability, R : S ×

Original: A → { 0 , 1 } is a sparse reward function and γ ∈ [0 , 1] is the discount factor.
Revised: A → { 0 , 1 } is a sparse reward function and γ ∈ [0 , 1] is the discount factor.

Original: Policy updates are performed with Soft Actor-Critic ( SAC ) (Haarnoja et al., 2018), a state-of-the-art off-policy actor-critic algorithm.
Revised: Policy updates are performed with Soft Actor-Critic ( SAC ) (Haarnoja et al., 2018), a state-of-the-art off-policy actor-critic algorithm.

Original: We also use Hindsight Experience Replay ( HER ) (Andrychowicz et al., 2017).
Revised: We also use Hindsight Experience Replay ( HER ) (Andrychowicz et al., 2017).

Original: This mechanism enables agents to learn from failures by reinterpreting past trajectories in the light of goals different from the ones originally targeted.
Revised: This mechanism enables agents to learn from failures by reinterpreting past trajectories in the light of goals different from the ones originally targeted.

Original: HER was designed for continuous goal spaces, but can be directly transposed to discrete goals (Colas et al., 2019a).
Revised: HER was designed for continuous goal spaces, but can be directly transposed to discrete goals (Colas et al., 2019a).

Original: In our setting, we simply replace the originally targeted goal conﬁguration by the currently achieved conﬁguration in the transitions fed to SAC .
Revised: In our setting, we simply replace the originally targeted goal conﬁguration by the currently achieved conﬁguration in the transitions fed to SAC .

Original: We also use our automatic curriculum strategy: the LP - C -based probabilities are used to sample goals to learn about.
Revised: We also use our automatic curriculum strategy: the LP - C -based probabilities are used to sample goals to learn about.

Original: When a goal g is sampled, we search the experience buffer for the collection of episodes that ended in the conﬁguration c
Revised: When a goal g is sampled, we search the experience buffer for the collection of episodes that ended in the conﬁguration c

Original: p = g .
Revised: p = g .

Original: From these episodes, we sample a transition uniformly.
Revised: From these episodes, we sample a transition uniformly.

Original: The HER mechanism substitutes the original goal with one of the conﬁgurations achieved later in the trajectory.
Revised: The HER mechanism substitutes the original goal with one of the conﬁgurations achieved later in the trajectory.

Original: This substitute g has high chances of being the sampled one.
Revised: This substitute g has high chances of being the sampled one.

Original: At least, it is a conﬁguration on the path towards this goal, as it is sampled from a trajectory leading to it.
Revised: At least, it is a conﬁguration on the path towards this goal, as it is sampled from a trajectory leading to it.

Original: The HER mechanism is thus biased towards goals sampled by the agent.
Revised: The HER mechanism is thus biased towards goals sampled by the agent.

Original: Object-Centered Inductive Biases.
Revised: Object-Centered Inductive Biases.

Original: In the proposed Fetch Manipulate environment, the three blocks share the same set of attributes (position, velocity, color identiﬁer).
Revised: In the proposed Fetch Manipulate environment, the three blocks share the same set of attributes (position, velocity, color identiﬁer).

Original: Thus, it is natural to encode a relational inductive bias in our architecture.
Revised: Thus, it is natural to encode a relational inductive bias in our architecture.

Original: The behavior with respect to a pair of objects should be independent from the position of the objects in the inputs.
Revised: The behavior with respect to a pair of objects should be independent from the position of the objects in the inputs.

Original: The architecture used for the policy is depicted in Figure 4.
Revised: The architecture used for the policy is depicted in Figure 4.

Original: A shared network ( NN shared ) encodes the concatenation of: 1) agent’s body features; 2) object pair features; 3) current conﬁguration ( c p ) and 4) current goal g .
Revised: A shared network ( NN shared ) encodes the concatenation of: 1) agent’s body features; 2) object pair features; 3) current conﬁguration ( c p ) and 4) current goal g .

Original: This is done independently for all object pairs.
Revised: This is done independently for all object pairs.

Original: No matter the location of the features of the object pair in the initial observations, this shared network ensures that the same behavior will be performed, thus skills are transferred between object pairs.
Revised: No matter the location of the features of the object pair in the initial observations, this shared network ensures that the same behavior will be performed, thus skills are transferred between object pairs.

Original: A sum is then used to aggregate these outputs, before a ﬁnal network ( NN policy ) maps the aggregation to actions a .
Revised: A sum is then used to aggregate these outputs, before a ﬁnal network ( NN policy ) maps the aggregation to actions a .

Original: The critic follows the same architecture, where a ﬁnal network NN critic maps the aggregation to an action-value Q .
Revised: The critic follows the same architecture, where a ﬁnal network NN critic maps the aggregation to an action-value Q .

Original: Parallel encoding of each pair-speciﬁc inputs can be seen as different modules trying to reach the goal by only seeing these pair-speciﬁc inputs.
Revised: Parallel encoding of each pair-speciﬁc inputs can be seen as different modules trying to reach the goal by only seeing these pair-speciﬁc inputs.

Original: The intuition is that modules dealing with the pair that should be acted upon to reach the goal will supersede others in the sum aggregation.
Revised: The intuition is that modules dealing with the pair that should be acted upon to reach the goal will supersede others in the sum aggregation.

Original: [Figure]
Revised: [Figure]

Original: Although in principle our architecture could work with combinations of objects ( 3 modules), we found permutations to work better in practice ( 6 modules).
Revised: Although in principle our architecture could work with combinations of objects ( 3 modules), we found permutations to work better in practice ( 6 modules).

Original: With combinations, the shared network would need to learn to put block A on block B to achieve a predicate above ( o i , o j ) , and would need to learn the reverse behavior (put B on A ) to achieve the symmetric predicate above ( o j , o i ) .
Revised: With combinations, the shared network would need to learn to put block A on block B to achieve a predicate above ( o i , o j ) , and would need to learn the reverse behavior (put B on A ) to achieve the symmetric predicate above ( o j , o i ) .

Original: With permutations, the shared network can simply learn one of these behaviors (e.g. A on B ).
Revised: With permutations, the shared network can simply learn one of these behaviors (e.g. A on B ).

Original: Considering the predicate above ( o A , o B ) , at least one of the modules has objects organized so that this behavior is the good one: if the permutation ( o B , o A ) is not the right one, permutation ( o A , o B ) is.
Revised: Considering the predicate above ( o A , o B ) , at least one of the modules has objects organized so that this behavior is the good one: if the permutation ( o B , o A ) is not the right one, permutation ( o A , o B ) is.

Original: The symmetry bias is explained in Section 3.4.
Revised: The symmetry bias is explained in Section 3.4.

Original: It leverages the symmetry of the behaviors required to achieve the predicates above ( o i , o j ) and above ( o j , o i ) .
Revised: It leverages the symmetry of the behaviors required to achieve the predicates above ( o i , o j ) and above ( o j , o i ) .

Original: As a result, the two goal conﬁgurations are:
Revised: As a result, the two goal conﬁgurations are:

Original: [Equation]
Revised: [Equation]

Original: = [ c ( o 1 , o 2 ) , c ( o 1 , o 3 ) , c ( o 2 , o 3 ) , a ( o 2 , o 1 ) , a ( o 3 , o 1 ) , a ( o 3 , o 2 )] , where g 1 is used in association with object permutations ( o i , o j ) with i <
Revised: [ c ( o 1 , o 2 ) , c ( o 1 , o 3 ) , c ( o 2 , o 3 ) , a ( o 2 , o 1 ) , a ( o 3 , o 1 ) , a ( o 3 , o 2 )] , where g 1 is used in association with object permutations ( o i , o j ) with i <

Original: j and g 2 is used in association with object permutations ( o j , o i ) with i < j .
Revised: j and g 2 is used in association with object permutations ( o j , o i ) with i < j .

Original: As a result, the shared network automatically ensures transfer between predicates based on symmetric behaviors.
Revised: As a result, the shared network automatically ensures transfer between predicates based on symmetric behaviors.

Original: Implementation Details.
Revised: Implementation Details.

Original: This part includes details necessary to reproduce results.
Revised: This part includes details necessary to reproduce results.

Original: The code is available at https://sites.google.com/view/decstr/.
Revised: The code is available at https://sites.google.com/view/decstr/.

Original: Parallel implementation of SAC - HER .
Revised: Parallel implementation of SAC - HER .

Original: We use a parallel implementation of SAC (Haarnoja et al., 2018).
Revised: We use a parallel implementation of SAC (Haarnoja et al., 2018).

Original: Each of the 24 parallel worker maintains its own replay buffer of size 10 6 and performs its own updates.
Revised: Each of the 24 parallel worker maintains its own replay buffer of size 10 6 and performs its own updates.

Original: Updates are summed over the 24 actors and the updated network are broadcast to all workers.
Revised: Updates are summed over the 24 actors and the updated network are broadcast to all workers.

Original: Each worker alternates between 2 episodes of data collection and 30 updates with batch size 256 .
Revised: Each worker alternates between 2 episodes of data collection and 30 updates with batch size 256 .

Original: To form an epoch, this cycle is repeated 50 times and followed by the ofﬂine evaluation of the agent on each reachable goal.
Revised: To form an epoch, this cycle is repeated 50 times and followed by the ofﬂine evaluation of the agent on each reachable goal.

Original: An epoch is thus made of 50 × 2 × 24 = 2400 episodes.
Revised: An epoch is thus made of 50 × 2 × 24 = 2400 episodes.

Original: Goal sampler updates.
Revised: Goal sampler updates.

Original: The agent performs self-evaluations with probability self eval = 0 .
Revised: The agent performs self-evaluations with probability self eval = 0 .

Original: During these runs, the agent targets uniformly sampled discovered conﬁgurations without exploration noise.
Revised: During these runs, the agent targets uniformly sampled discovered conﬁgurations without exploration noise.

Original: This enables the agent to self-evaluate on each goal.
Revised: This enables the agent to self-evaluate on each goal.

Original: Goals are organized into buckets.
Revised: Goals are organized into buckets.

Original: Main Section 3.4 presents our automatic bucket generation mechanism.
Revised: Main Section 3.4 presents our automatic bucket generation mechanism.

Original: Once buckets are formed, we compute C , LP and P , based on windows of the past W = 1800 self-evaluation interactions for each bucket.
Revised: Once buckets are formed, we compute C , LP and P , based on windows of the past W = 1800 self-evaluation interactions for each bucket.

Original: Modular architecture.
Revised: Modular architecture.

Original: The shared network of our modular architecture NN shared is a 1-hidden layer network of hidden size 256 .
Revised: The shared network of our modular architecture NN shared is a 1-hidden layer network of hidden size 256 .

Original: After all pair-speciﬁc inputs have been encoded through this module, their output (of size 84 ) are summed.
Revised: After all pair-speciﬁc inputs have been encoded through this module, their output (of size 84 ) are summed.

Original: The sum is then passed through a ﬁnal network with a hidden layer of size 256 to compute the ﬁnal actions (policy) or action-values (critic).
Revised: The sum is then passed through a ﬁnal network with a hidden layer of size 256 to compute the ﬁnal actions (policy) or action-values (critic).

Original: All networks use
Revised: All networks use

Original: ReLU activations and the Xavier initialization.
Revised: ReLU activations and the Xavier initialization.

Original: We use Adam optimizers, with learning rates 10 − 3 .
Revised: We use Adam optimizers, with learning rates 10 − 3 .

Original: The list of hyperparameters is provided in Table
Revised: The list of hyperparameters is provided in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Automatically tune 
Revised: Automatically tune the

Original: [Table]
Revised: [Table]

Original: Computing resources.
Revised: Computing resources.

Original: The sensorimotor learning experiments contain 8 conditions: 2 of 10 seeds and 6 of 5 seeds.
Revised: The sensorimotor learning experiments contain 8 conditions: 2 of 10 seeds and 6 of 5 seeds.

Original: Each run leverages 24 cpus ( 24 actors) for about 72 h for a total of 9 . 8 cpu years.
Revised: Each run leverages 24 cpus ( 24 actors) for about 72 h for a total of 9 . 8 cpu years.

Original: Experiments presented in this paper requires machines with at least 24 cpu cores.
Revised: Experiments presented in this paper requires machines with at least 24 cpu cores.

Original: The language grounding phase runs on a single cpu and trains in a few minutes.
Revised: The language grounding phase runs on a single cpu and trains in a few minutes.

Original: C. L ANGUAGE - CONDITIONED GOAL GENERATOR
Revised: C. L ANGUAGE - CONDITIONED GOAL GENERATOR

Original: Language-Conditioned Goal Generator Training.
Revised: Language-Conditioned Goal Generator Training.

Original: We use a conditional Variational AutoEncoder ( C - VAE )
Revised: We use a conditional Variational AutoEncoder ( C - VAE )

Original: (Sohn et al., 2015).
Revised: (Sohn et al., 2015).

Original: Conditioned on the initial conﬁguration and a sentence describing the expected transformation of one object relation, it generates compatible goal conﬁgurations.
Revised: Conditioned on the initial conﬁguration and a sentence describing the expected transformation of one object relation, it generates compatible goal conﬁgurations.

Original: After the ﬁrst phase of goal-directed sensorimotor training, the agent interacts with a hard-coded social partner as described in Main Section 3.
Revised: After the ﬁrst phase of goal-directed sensorimotor training, the agent interacts with a hard-coded social partner as described in Main Section 3.

Original: From these interactions, we obtain a dataset oftriplets: initial conﬁguration, ﬁnal conﬁguration and sentence describing one change of predicate from the initial to the ﬁnal conﬁguration.
Revised: From these interactions, we obtain a dataset oftriplets: initial conﬁguration, ﬁnal conﬁguration and sentence describing one change of predicate from the initial to the ﬁnal conﬁguration.

Original: The list of sentences used by the synthetic social partner is provided in Table 5.
Revised: The list of sentences used by the synthetic social partner is provided in Table 5.

Original: Note that red , green and blue refer to objects o 1 , o 2 , o 3 respectively.
Revised: Note that red , green and blue refer to objects o 1 , o 2 , o 3 respectively.

Original: Content of test sets.
Revised: Content of test sets.

Original: We describe the 5 test sets:
Revised: We describe the 5 test sets:

Original: Test set 1 is made of input pairs ( c i , s ) from the training set, but tests the coverage of all compatible ﬁnal conﬁgurations C f , 80% of which are not found in the training set.
Revised: Test set 1 is made of input pairs ( c i , s ) from the training set, but tests the coverage of all compatible ﬁnal conﬁgurations C f , 80% of which are not found in the training set.

Original: In that sense, it is partly a test set.
Revised: In that sense, it is partly a test set.

Original: Test set 2 contains two input pairs: { [0 1 0 0 0 0 0 0 0] , put blue close to green } and { [0 0 1 0 0 0 0 0 0] , put green below red } corresponding to 7 and 24 compatible ﬁnal conﬁgurations respectively.
Revised: Test set 2 contains two input pairs: { [0 1 0 0 0 0 0 0 0] , put blue close to green } and { [0 0 1 0 0 0 0 0 0] , put green below red } corresponding to 7 and 24 compatible ﬁnal conﬁgurations respectively.

Original: Test set 3 corresponds to all pairs including the initial conﬁguration c i =
Revised: Test set 3 corresponds to all pairs including the initial conﬁguration c i =

Original: [1 1 0 0 0 0 0 0 0] ( 29 pairs), with an average of 13 compatible ﬁnal conﬁgurations.
Revised: [1 1 0 0 0 0 0 0 0] ( 29 pairs), with an average of 13 compatible ﬁnal conﬁgurations.

Original: Test set 4 corresponds to all pairs including one of the sentences put green on top of red and put blue far from red , i
Revised: Test set 4 corresponds to all pairs including one of the sentences put green on top of red and put blue far from red , i

Original: e. 20 pairs with an average of 9 .
Revised: e. 20 pairs with an average of 9 .

Original: 5 compatible ﬁnal conﬁgurations.
Revised: 5 compatible ﬁnal conﬁgurations.

Original: [Table]
Revised: [Table]

Original: Close 0 →
Revised: Close 0 →

Original: Put block A close to block B , Bring block A and block B together , ( × 3 )
Revised: Put block A close to block B , Bring block A and block B together , ( × 3 )

Original: Get block A and block B close from each other , Get block
Revised: Get block A and block B close from each other , Get block

Original: A close to block B .
Revised: A close to block B .

Original: Close 1 →
Revised: Close 1 →

Original: Put block A far from block B , Get block
Revised: Put block A far from block B , Get block

Original: A far from block B , ( × 3 )
Revised: A far from block B , ( × 3 )

Original: Get block A and block B far from each other , Bring block A and block B apart ,
Revised: Get block A and block B far from each other , Bring block A and block B apart ,

Original: Above 0 →
Revised: Above 0 →

Original: Put block A above block B , Put block A on top of block B , ( × 6 )
Revised: Put block A above block B , Put block A on top of block B , ( × 6 )

Original: Put block B under block A , Put block B below block A .
Revised: Put block B under block A , Put block B below block A .

Original: Above 1 →
Revised: Above 1 →

Original: Remove block A from above block B , Remove block A from block B , ( × 6 )
Revised: Remove block A from above block B , Remove block A from block B , ( × 6 )

Original: Remove block B from below block A , Put block B and block A on the same plane ,
Revised: Remove block B from below block A , Put block B and block A on the same plane ,

Original: Put block A and block B on the same plane .
Revised: Put block A and block B on the same plane .

Original: Testing on logical expressions of instructions.
Revised: Testing on logical expressions of instructions.

Original: To evaluate DECSTR on logical functions of instructions, we generate three types of expressions:
Revised: To evaluate DECSTR on logical functions of instructions, we generate three types of expressions:

Original: 1. 100 instructions of the form “A and B” where A and B are basic instructions corresponding to shifts of the form above 0 → 1 (see Table 5).
Revised: 1. 100 instructions of the form “A and B” where A and B are basic instructions corresponding to shifts of the form above 0 → 1 (see Table 5).

Original: These intersections correspond to stacks ofor pyramids.
Revised: These intersections correspond to stacks ofor pyramids.

Original: 200 instructions of the form “A and B” where A and B are above and close instructions respectively.
Revised: 200 instructions of the form “A and B” where A and B are above and close instructions respectively.

Original: B can be replaced by “not B” with probability 0.5.
Revised: B can be replaced by “not B” with probability 0.5.

Original: 200 instructions of the form “(A and B) or (C and D))”, where A, B, C, D are basic instructions:
Revised: 200 instructions of the form “(A and B) or (C and D))”, where A, B, C, D are basic instructions:

Original: A and C are above instructions while B and D are close instructions.
Revised: A and C are above instructions while B and D are close instructions.

Original: Here also, any
Revised: Here also, any

Original: instruction can be replaced by its negation with probability 0.5.
Revised: instruction can be replaced by its negation with probability 0.5.

Original: Implementation details.
Revised: Implementation details.

Original: The encoder is a fully-connected neural network with two layers of size 128 and ReLU activations.
Revised: The encoder is a fully-connected neural network with two layers of size 128 and ReLU activations.

Original: It takes as input the concatenation of the ﬁnal binary conﬁguration and its two conditions: the initial binary conﬁguration and an embedding of the NL sentence.
Revised: It takes as input the concatenation of the ﬁnal binary conﬁguration and its two conditions: the initial binary conﬁguration and an embedding of the NL sentence.

Original: The NL sentence is embedded with an recurrent network with embedding size 100 , tanh non-linearities and biases.
Revised: The NL sentence is embedded with an recurrent network with embedding size 100 , tanh non-linearities and biases.

Original: The encoder outputs the mean and log-variance of the latent distribution of size 27 .
Revised: The encoder outputs the mean and log-variance of the latent distribution of size 27 .

Original: The decoder is also a fully-connected network with two hidden layers of size 128 and ReLU activations.
Revised: The decoder is also a fully-connected network with two hidden layers of size 128 and ReLU activations.

Original: It takes as input the latent code z and the same conditions as the encoder.
Revised: It takes as input the latent code z and the same conditions as the encoder.

Original: As it generates binary vectors, the last layer uses sigmoid activations.
Revised: As it generates binary vectors, the last layer uses sigmoid activations.

Original: We train the architecture with a mixture of KullbackLeibler divergence loss ( KD loss ) w.r.t a standard Gaussian prior and a binary Cross-Entropy loss ( BCE loss ) .
Revised: We train the architecture with a mixture of KullbackLeibler divergence loss ( KD loss ) w.r.t a standard Gaussian prior and a binary Cross-Entropy loss ( BCE loss ) .

Original: The combined loss is BCE loss + β × KD loss with β = 0 .
Revised: The combined loss is BCE loss + β × KD loss with β = 0 .

Original: We use an Adam optimizer, a learning rate of 5 × 10 − 4 , a batch size of 128 and optimize for 150 epochs.
Revised: We use an Adam optimizer, a learning rate of 5 × 10 − 4 , a batch size of 128 and optimize for 150 epochs.

Original: As training is fast ( ≈ 2 min on a single cpu), we conducted a quick hyperparameter search over β , layer sizes, learning rates and latent sizes (see Table 6).
Revised: As training is fast ( ≈ 2 min on a single cpu), we conducted a quick hyperparameter search over β , layer sizes, learning rates and latent sizes (see Table 6).

Original: We found robust results for various layer sizes, various β below 1 .
Revised: We found robust results for various layer sizes, various β below 1 .

Original: and latent sizes above 9 .
Revised: and latent sizes above 9 .

Original: [Table]
Revised: [Table]

Original: D B ASELINES AND ORACLE
Revised: D B ASELINES AND ORACLE

Original: The language-conditioned LB baseline is fully described in the main document.
Revised: The language-conditioned LB baseline is fully described in the main document.

Original: D. E XPERT BUCKETS ORACLE
Revised: D. E XPERT BUCKETS ORACLE

Original: In the E XPERT B UCKETS oracle, the automatic bucket generation of DECSTR is replaced with an expert-predeﬁned set of buckets using a priori measures of similarity and difﬁculty.
Revised: In the E XPERT B UCKETS oracle, the automatic bucket generation of DECSTR is replaced with an expert-predeﬁned set of buckets using a priori measures of similarity and difﬁculty.

Original: To deﬁne these buckets, one needs prior knowledge of the set of unreachable conﬁgurations, which are ruled out.
Revised: To deﬁne these buckets, one needs prior knowledge of the set of unreachable conﬁgurations, which are ruled out.

Original: The 5 predeﬁned buckets contain all conﬁgurations characterized by:
Revised: The 5 predeﬁned buckets contain all conﬁgurations characterized by:

Original: • Bucket 1 : a single close relation between a pair of objects and no above relations (4 conﬁgurations).
Revised: • Bucket 1 : a single close relation between a pair of objects and no above relations (4 conﬁgurations).

Original: • Bucket 2 : 2 or 3 close relations and no above relations (4 conﬁgurations).
Revised: • Bucket 2 : 2 or 3 close relations and no above relations (4 conﬁgurations).

Original: • Bucket 3 : 1 stack of 2 blocks and a third block that is either away or close to the base, but is not close to the top of the stack (12 conﬁgurations).
Revised: • Bucket 3 : 1 stack of 2 blocks and a third block that is either away or close to the base, but is not close to the top of the stack (12 conﬁgurations).

Original: • Bucket 4 : 1 stack of 2 blocks and the third block close to the stack, as well as pyramid conﬁgurations (9 conﬁgurations).
Revised: • Bucket 4 : 1 stack of 2 blocks and the third block close to the stack, as well as pyramid conﬁgurations (9 conﬁgurations).

Original: • Bucket 5 : stacks of 3 blocks (6 conﬁgurations).
Revised: • Bucket 5 : stacks of 3 blocks (6 conﬁgurations).

Original: These buckets are the only difference between the E XPERT B UCKETS baseline and DECSTR .
Revised: These buckets are the only difference between the E XPERT B UCKETS baseline and DECSTR .

Original: D. LGB - C BASELINE
Revised: D. LGB - C BASELINE

Original: The LGB - C baseline represent goals not as semantic conﬁgurations but as particular 3 D targets positions for each block, as deﬁned for example in Lanier et al.
Revised: The LGB - C baseline represent goals not as semantic conﬁgurations but as particular 3 D targets positions for each block, as deﬁned for example in Lanier et al.

Original: and
Revised: and

Original: Li et al.
Revised: Li et al.

Original: The goal vector size is also 9 and contains the 3D target coordinates of the three blocks.
Revised: The goal vector size is also 9 and contains the 3D target coordinates of the three blocks.

Original: This baselines also implements decoupling and, thus, can be compared to DECSTR in the three phases.
Revised: This baselines also implements decoupling and, thus, can be compared to DECSTR in the three phases.

Original: We keep as many modules as possible common with DECSTR to minimize the amount of confounding factors and reduce the under-ﬁtting bias.
Revised: We keep as many modules as possible common with DECSTR to minimize the amount of confounding factors and reduce the under-ﬁtting bias.

Original: The goal selection is taken from DECSTR , but converts semantic conﬁguration into speciﬁc randomly-sampled target coordinates for the blocks, see Figure 5.
Revised: The goal selection is taken from DECSTR , but converts semantic conﬁguration into speciﬁc randomly-sampled target coordinates for the blocks, see Figure 5.

Original: The agent is not conditioned on its current semantic conﬁguration nor its semantic goal conﬁguration.
Revised: The agent is not conditioned on its current semantic conﬁguration nor its semantic goal conﬁguration.

Original: For this reason, we do not apply the symmetry bias.
Revised: For this reason, we do not apply the symmetry bias.

Original: The binary reward is positive when the maximal distance between a block and its target position is below 5 cm, i.e. the size of a block (similar to (Andrychowicz et al., 2017)).
Revised: The binary reward is positive when the maximal distance between a block and its target position is below 5 cm, i.e. the size of a block (similar to (Andrychowicz et al., 2017)).

Original: To make this baseline competitive, we integrate methods from a state of the art block manipulation algorithm (Lanier et al., 2019).
Revised: To make this baseline competitive, we integrate methods from a state of the art block manipulation algorithm (Lanier et al., 2019).

Original: The agent receives positive rewards of 1, 2, 3 when the corresponding number of blocks are well placed.
Revised: The agent receives positive rewards of 1, 2, 3 when the corresponding number of blocks are well placed.

Original: We also introduce the multi-criteria HER from Lanier et al.
Revised: We also introduce the multi-criteria HER from Lanier et al.

Original: Finally, we add an additional object-centered inductive bias by only considering, for each Deep Sets module, the 3D target positions of the corresponding pair.
Revised: Finally, we add an additional object-centered inductive bias by only considering, for each Deep Sets module, the 3D target positions of the corresponding pair.

Original: That is, for each object pair, we ignore the 3D positions of the remaining object, yielding to a vector of size 6.
Revised: That is, for each object pair, we ignore the 3D positions of the remaining object, yielding to a vector of size 6.

Original: Language grounding is based on a C - VAE similar to the one used by DECSTR .
Revised: Language grounding is based on a C - VAE similar to the one used by DECSTR .

Original: We only replace the cross-entropy loss by a mean-squared loss due to the continuous nature of the target goal coordinates.
Revised: We only replace the cross-entropy loss by a mean-squared loss due to the continuous nature of the target goal coordinates.

Original: We use the exact same training and testing sets as with semantic goals.
Revised: We use the exact same training and testing sets as with semantic goals.

Original: E A DDITIONAL RESULTS E. C OMPARISON DECSTR - LGB - C IN SKILL LEARNING PHASE
Revised: E A DDITIONAL RESULTS E. C OMPARISON DECSTR - LGB - C IN SKILL LEARNING PHASE

Original: Figure 6 presents the average success rate over the 35 valid conﬁgurations during the skill learning phase for DECSTR and the LGB - C baseline.
Revised: Figure 6 presents the average success rate over the 35 valid conﬁgurations during the skill learning phase for DECSTR and the LGB - C baseline.

Original: Because LGB - C cannot pursue semantic goals as such, we randomly sample a speciﬁc instance of this semantic goal: target block coordinates that satisfy the constraints expressed by it.
Revised: Because LGB - C cannot pursue semantic goals as such, we randomly sample a speciﬁc instance of this semantic goal: target block coordinates that satisfy the constraints expressed by it.

Original: Because LGB - C is not aware of the original semantic goal, we cannot measure success as the ability to achieve it.
Revised: Because LGB - C is not aware of the original semantic goal, we cannot measure success as the ability to achieve it.

Original: Instead, success is deﬁned as the achievement of the corresponding speciﬁc goal: bringing blocks to their respective targets within an error margin of 5 cm each.
Revised: Instead, success is deﬁned as the achievement of the corresponding speciﬁc goal: bringing blocks to their respective targets within an error margin of 5 cm each.

Original: In short, DECSTR targets semantic goals and is evaluated on its ability to reach them.
Revised: In short, DECSTR targets semantic goals and is evaluated on its ability to reach them.

Original: LGB - C targets speciﬁc goals and is evaluated on its ability to reach them.
Revised: LGB - C targets speciﬁc goals and is evaluated on its ability to reach them.

Original: These two measures do not match exactly.
Revised: These two measures do not match exactly.

Original: Indeed, LGB - C sometimes achieves its speciﬁc goal but, because of the error margins, does not achieve the original semantic goal.
Revised: Indeed, LGB - C sometimes achieves its speciﬁc goal but, because of the error margins, does not achieve the original semantic goal.

Original: [Figure]
Revised: [Figure]

Original: E. A UTOMATIC BUCKET GENERATION .
Revised: E. A UTOMATIC BUCKET GENERATION .

Original: Figure 7 depicts the evolution of the content of buckets along training (epochs 1 , 50 and 100 ).
Revised: Figure 7 depicts the evolution of the content of buckets along training (epochs 1 , 50 and 100 ).

Original: Each pie chart corresponds to a reachable conﬁguration and represents the distribution of conﬁgurations into buckets across 10 different seeds.
Revised: Each pie chart corresponds to a reachable conﬁguration and represents the distribution of conﬁgurations into buckets across 10 different seeds.

Original: Blue, orange, green, yellow, purple represent buckets 1 torespectively and grey are undiscovered conﬁgurations.
Revised: Blue, orange, green, yellow, purple represent buckets 1 torespectively and grey are undiscovered conﬁgurations.

Original: At each moment, the discovered conﬁgurations are equally spread over the 5 buckets.
Revised: At each moment, the discovered conﬁgurations are equally spread over the 5 buckets.

Original: A given conﬁguration may thus change bucket as new conﬁgurations are discovered, so that the ones discovered earlier are assigned buckets with lower indexes.
Revised: A given conﬁguration may thus change bucket as new conﬁgurations are discovered, so that the ones discovered earlier are assigned buckets with lower indexes.

Original: Goals are organized by their bucket assignments in the Expert Buckets condition (from top to bottom).
Revised: Goals are organized by their bucket assignments in the Expert Buckets condition (from top to bottom).

Original: After the ﬁrst epoch (left), DECSTR has discovered all conﬁgurations from the expert buckets 1 and 2 , and some runs have discovered a few other conﬁgurations.
Revised: After the ﬁrst epoch (left), DECSTR has discovered all conﬁgurations from the expert buckets 1 and 2 , and some runs have discovered a few other conﬁgurations.

Original: After 50 epochs, more conﬁgurations have been discovered but they are not always the same across runs.
Revised: After 50 epochs, more conﬁgurations have been discovered but they are not always the same across runs.

Original: Finally, after 100 epochs, all conﬁgurations are found.
Revised: Finally, after 100 epochs, all conﬁgurations are found.

Original: Buckets are then steady and can be compared to expert-deﬁned buckets.
Revised: Buckets are then steady and can be compared to expert-deﬁned buckets.

Original: It seems that easier goals (top-most group) are discovered ﬁrst and assigned in the ﬁrst-easy buckets (blue and orange).
Revised: It seems that easier goals (top-most group) are discovered ﬁrst and assigned in the ﬁrst-easy buckets (blue and orange).

Original: Hardest conﬁgurations (stacks of 3 , bottom-most group) seem to be discovered last and assigned the last-hardest bucket (purple).
Revised: Hardest conﬁgurations (stacks of 3 , bottom-most group) seem to be discovered last and assigned the last-hardest bucket (purple).

Original: In between, different runs show different compositions, which are not always aligned with expert-deﬁned buckets.
Revised: In between, different runs show different compositions, which are not always aligned with expert-deﬁned buckets.

Original: Goals from expert-deﬁned buckets 3 and 4 (third and fourth group from the top) seem to be attributed different automatic buckets in different runs.
Revised: Goals from expert-deﬁned buckets 3 and 4 (third and fourth group from the top) seem to be attributed different automatic buckets in different runs.

Original: This means that they are discovered in different orders depending on the runs.
Revised: This means that they are discovered in different orders depending on the runs.

Original: In summary, easier and harder goals from expert buckets 1 - 2 and 5 respectively seem to be well detected by our automatic bucket generations.
Revised: In summary, easier and harder goals from expert buckets 1 - 2 and 5 respectively seem to be well detected by our automatic bucket generations.

Original: Goals in medium-level expected difﬁculty as deﬁned by expert buckets seem not to show any signiﬁcant difference in difﬁculty for our agents.
Revised: Goals in medium-level expected difﬁculty as deﬁned by expert buckets seem not to show any signiﬁcant difference in difﬁculty for our agents.

Original: E. DECSTR LEARNING TRAJECTORIES
Revised: E. DECSTR LEARNING TRAJECTORIES

Original: Figure 8 shows the evolution of internal estimations of the competence C , the learning progress
Revised: Figure 8 shows the evolution of internal estimations of the competence C , the learning progress

Original: LP and the associated sampling probabilities P .
Revised: LP and the associated sampling probabilities P .

Original: Note that these metrics are computed online by
Revised: Note that these metrics are computed online by

Original: [Figure]
Revised: [Figure]

Original: DECSTR , as it self-evaluates on random discovered conﬁgurations.
Revised: DECSTR , as it self-evaluates on random discovered conﬁgurations.

Original: Learning trajectories seem to be uniform across different runs
Revised: Learning trajectories seem to be uniform across different runs

Original: [Figure]
Revised: [Figure]

Original: and buckets are learned in increasing order.
Revised: and buckets are learned in increasing order.

Original: This conﬁrms that the time of discovery is a good proxy for goal difﬁculty.
Revised: This conﬁrms that the time of discovery is a good proxy for goal difﬁculty.

Original: In that case, conﬁgurations discovered ﬁrst end up in the lower index buckets and are indeed learned ﬁrst.
Revised: In that case, conﬁgurations discovered ﬁrst end up in the lower index buckets and are indeed learned ﬁrst.

Original: Note that a failing automatic bucket generation would assign goals to random buckets.
Revised: Note that a failing automatic bucket generation would assign goals to random buckets.

Original: This would result in uniform measures of learning progress across different buckets, which would be equivalent to uniform goal sampling.
Revised: This would result in uniform measures of learning progress across different buckets, which would be equivalent to uniform goal sampling.

Original: As Main
Revised: As Main

Original: Figure 3c shows, DECSTR performs much better than the random goals conditions.
Revised: Figure 3c shows, DECSTR performs much better than the random goals conditions.

Original: This proves that our automatic bucket algorithm generates useful goal clustering
Revised: This proves that our automatic bucket algorithm generates useful goal clustering

Original: [Figure]
Revised: [Figure]