Language-Conditioned Goal Generation: a New Approach to Language Grounding in RL
Abstract
In the real world, linguistic agents are also embodied agents: they perceive and act in the physical world.
The notion of Language Grounding questions the interactions between language and embodiment: how do learning agents connect or ground linguistic representations to the physical world ?
This question has recently been approached by the Reinforcement Learning community under the framework of instruction-following agents.
In these agents, behavioral policies or reward functions are conditioned on the embedding of an instruction expressed in natural language.
This paper proposes another approach: using language to condition goal generators.
Given any goal-conditioned policy, one could train a language-conditioned goal generator to generate language-agnostic goals for the agent.
This method allows to decouple sensorimotor learning from language acquisition and enable agents to demonstrate a diversity of behaviors for any given instruction.
We propose a particular instantiation of this approach and demonstrate its beneﬁts.
1. Introduction
Language Grounding describes the idea that language acquisition is strongly shaped by one’s experience of the physical world.
This idea emerged in Cognitive Science (Harnad, 1990; Glenberg & Kaschak, 2002; Zwaan & Madden, 2005) and quickly inspired Artiﬁcial Intelligence approaches in natural language processing (Roy & Reiter, 2005), humanmachine interactions (Dominey, 2005; Madden et al., 2010) and more recently deep Reinforcement Learning (deep RL)
(Luketina et al., 2019).
In the RL community, this has taken the form of languageconditioned agents (Hermann et al., 2017; Chan et al., 2019; Equal
Bordeaux, FR. Paris-Sorbonne, Paris, FR. <cedric.colas@inria.fr>. Accepted to the 1 st Workshop on Language in Reinforcement
Learning , ICML 2020. Copyright 2020 by the author(s). Bahdanau et al., 2018; Cideron et al., 2019; Jiang et al., 2019; Zhong et al., 2019; Waytowich et al., 2019; Colas et al., 2020b).
Language can be used to build representations (Waytowich et al., 2019) or to characterize the dynamics of the environment (Zhong et al., 2019).
However, it is mostly used to represent instructions or goals (Hermann et al., 2017;
Chan et al., 2019; Bahdanau et al., 2018; Cideron et al., 2019; Jiang et al., 2019; Colas et al., 2020b).
In these approaches, natural language ( NL ) sentences are embedded through recurrent networks and merged with the agent’s state to form the input of the policy or reward function.
The language encoder is then trained jointly with the former (Hermann et al., 2017; Chan et al., 2019; Jiang et al., 2019;
Hill et al., 2019) or the latter (Bahdanau et al., 2018; Colas et al., 2020b).
These approach demonstrate beneﬁts over traditional goal-conditioned methods:
Targeting abstract goals.
Language can express abstract goals characterized by sets of properties the scene should verify (e g. block A above block B ).
This contrasts with previous goal-as-state approaches, where goals are speciﬁc states of the agent (e
g. target pixel image, target block positions).
Systematic generalization.
Previous language- conditioned approaches
demonstrate strong generalization capabilities including generalizations to new
combinations
of action verbs and object attributes
colors, shapes
categories etc.
) (Hermann et al., 2017; Bahdanau et al., 2018; Hill et al., 2019; Colas et al., 2020b).
However, this comes with drawbacks:
Language becomes a pre-requisite for sensorimotor learning.
Pre-verbal infants are known to demonstrate goal-oriented behavior (Wood et al., 1976).
Languageconditioned agents require language inputs to act in the world and, thus, cannot account for this decoupling.
Lack of behavioral diversity.
Policies directly conditioned on language embeddings, generate a low behavioral diversity.
For a given instruction in a given context: a unique behavior for a deterministic policy and minor, noise-induced behavioral variations for stochastic ones.
This paper proposes to leverage the abstraction and generalization capacities of language while 1) decoupling language
[Figure]
acquisition from sensorimotor learning; 2) allowing behavioral diversity.
This is achieved via language-conditioned goal generation (Figure 1).
Instead of using language to condition policies directly, we use it to condition a generator of language-agnostic goals.
Language-agnostic goals can be represented by speciﬁc states (e.g. pixels inputs)
(Péré et al., 2018;
LaversanneFinot et al., 2018; Nair et al., 2018; Warde-Farley et al., 2018; Nair et al., 2019; Pong et al., 2019) or handcrafted representations (e.g. target block coordinates) (Schaul et al., 2015; Andrychowicz et al., 2017; Florensa et al., 2018;
Racaniere et al., 2019; Fournier et al., 2019; Colas et al., 2019).
These can come from any source: uniform sampling of the goal space (Schaul et al., 2015), sampling from low-density areas (Pong et al., 2019), from high learning progress areas (Fournier et al., 2019; Colas et al., 2019), using generative models of states (Nair et al., 2018; 2019) or generative models targeting intermediate difﬁculty (Florensa et al., 2018;
Racaniere et al., 2019).
In our approach, language-conditioned goal generators are a source of language-agnostic goals among others.
As a result, pure sensorimotor training does not require any language input: agents can simply use any other source of goals.
Language-conditioned goal generators are generative models of language-agnostic goals conditioned on instructions (e.g. Variational Auto-Encoders, Generative Adversarial
Networks).
For any given instruction, agent can thus sample a set of matching goals.
This results in behavioral diversity, a set of diverse behaviors matching any given instruction.
Contributions.
This paper introduces a novel approach to the problem of language grounding in RL agents: languageconditioned goal generation.
This leverages the capability of language to represent abstract goals and to generalize, while avoiding the lack of behavioral diversity and the dependence to language inputs usually associated with language-conditioned agents.
This paper presents a particular implementation of this approach and disentangles language acquisition from policy learning by assuming access to pre-trained goal-conditioned policies.
Our policies are pre-trained to reach high-level conﬁgurations characterizing spatial relations between objects in the scene.
2. Methods
Section 2.1 describes the learning environment and the pretrained goal-conditioned policies.
Section 2.2 describes a particular implementation of a language-conditioned goal generator and how it is used for language grounding.
2.1. Behavioral Policies Conditioned on Abstract Semantic Representations
The agents we consider in this paper were pre-trained without language input by an algorithm presented in a companion paper (Akakzia et al., 2020).
The Fetch Manipulate environment Agents evolve in the Fetch Manipulate environment: a robotic manipulation domain based on MUJOCO (Todorov et al., 2012) and adapted from the Fetch tasks (Plappert et al., 2018).
Agents are 4 DoF robotic arms that face 3 colored blocks on a table (see Figure 2).
They are given innate representations called semantic conﬁgurations that characterize spatial relations between blocks.
During sensorimotor training, agents discovered all reachable conﬁgurations in that space and learned to master them.
Semantic conﬁgurations.
In contrast to traditional approaches, goals are not deﬁned as particular targets for each block but as high-level semantic conﬁgurations .
These conﬁgurations are based on two spatial predicates infants demonstrate early in their development (Mandler, 2012): the close and the above binary predicates.
These two predicates are applied to all permutations of object pairs, i.e.permutations for the 3 objects we consider.
Because the close predicate is order invariant, we only need to evaluate it on 3 object combinations.
The above predicate being order dependent, we need all 6 permutations.
The resulting binary vector of size 9 forms the semantic conﬁguration .
It represents the spatial relations between objects in the scene.
In the resulting semantic conﬁguration space { 0 , 1 } 9 , the agent can reach 35 physically valid conﬁgurations, including stacks of 2 or 3 blocks and pyramids.
Supplementary
Section 5 provides formal deﬁnitions and properties of predicates and semantic conﬁgurations.
Figure 2 displays visual representations of some example conﬁgurations.
2.2. Language-Conditioned Goal Generation
The language-conditioned goal generator, or language module , generates semantic conﬁgurations matching the agent’s initial conﬁguration and a sentence describing an expected transformation of a relation between a pair of objects.
This section explains its training and use for language grounding.
Training the language module.
The languageconditioned goal generator is implemented by a conditional
Variational Auto-Encoder ( C - VAE ) (Sohn et al., 2015) trained in a supervised setting with a standard Gaussian prior.
The training data is collected via interactions between a trained agent and a social partner.
For each goal-directed trajectory the agent performs, the social partner provides one description from a set of 102 sentences.
Each description comments on a single transformation in the object relations, i.e. one positive of negative shift for one of the 9 predicates (e.g. get red above green ).
NL descriptions are encoded via a recurrent network that is jointly trained with the C - VAE .
Supplementary Sectionprovides the list of sentences and implementation details.
Language grounding.
At test time, agents are instructed by one sentence to transform a relation between objects.
The trained language module acts as a translator: agent can sample a goal conﬁguration matching their current state and the instruction.
The sampled conﬁguration is the output of the trained C - VAE decoder conditioned on the initial state and instruction, using as input a latent code sampled from the prior.
This module effectively enables agents to ground
NL in their internal semantic representations and set of sensorimotor skills.
We consider three evaluation settings: 1) performing a single instruction; 2) performing a sequence of instructions; 3) performing a logical combination of instructions.
As the agent can generate a set of goals matching any instruction, it can easily combine these sets to perform logical functions of instructions: and is an intersection, or is an union and not is the complement within the set of goals the agent discovered during sensorimotor training.
Given sets of compatible goal conﬁgurations, agents can also try again : ﬁnd other goal conﬁgurations that match the required instruction when previous attempts failed.
3. Experiments
We ﬁrst train a language-conditioned goal generator from a training dataset D collected via interactions between the agent and a social partner (Section 2.2).
For a given initial conﬁguration and a given sentence, we want the language module to generate all compatible ﬁnal conﬁgurations, and just these.
Language-conditioned goal generation performance.
To evaluate the language module, we construct a synthetic, oracle dataset O of triplets ( c i , s ,
C f
( c i , s )) , where c i is the initial conﬁguration, s is the sentence describing the expected transformation and
C f
( c i , s ) is the set of all ﬁnal conﬁgurations compatible with ( c i , s ) .
Note that, on average, C f in O contains 16 .
7 conﬁgurations, while the training dataset D only contains 3 . 4 (20%) .
We are interested in two metrics: 1)
The Precision (P) is the probability that a goal sampled from the generator belongs to C f ; 2)
The Recall (R) is the ratio of goals from C f found after sampling the language-conditioned generator 100 times.
We compute these metrics on 5 different sets of input pairs ( c i ,s ) , each calling for a different type of generalization:
Pairs found in D , except pairs removed to form the following test sets.
This calls for the extrapolation of known initialization-effect pairs ( c i ,s ) to new ﬁnal conﬁgurations c f
( D contains only 20% of C f on average).
Pairs that were removed from D , calling for a recombination of known effects s on known c i .
Pairs for which the c i was entirely removed from D .
This calls for the transfer of known effects s on unknown c i . 4.
Pairs for which the s was entirely removed from D .
This calls for generalization in the language space, to generalize unknown effects s from related sentences and transpose this to known c i . 5.
Pairs for which both the c i and the s were entirely removed from D .
This calls for the generalizationsand 4 combined.
As a result, D does not contain neither all 102 sentences, nor all initial conﬁgurations.
Our language module demonstrates these 5 types of generalization (see Table 1).
Agents can generate goals from situations they never encountered (Test 3).
They can generalize the meaning of sentences they never heard (Test 4) and even apply the latter to unknown situations (Test 5).
We detail the testing sets in Supplementary Section
[Table] [Table]
Grounding language in sensorimotor behavior.
We investigate how the language module interacts with the sensorimotor skills of the agent.
We consider three evaluation settings.
In the transition setup, we look at the average success rate of the agent when asked to perform theinstructions 5 times each, resetting the environment each time.
In the expression setup, we evaluate the agent on 500 randomly generated logical functions of sentences.
In both setups, we give the agent 5 attempts, enabling it to resample new compatible goals when the previous failed (without reset).
Success rates after 1 (SR1) and 5 (SR5) attempts are reported in Table 2.
In the sequence setup, we ask the agent to execute 20 random sequences of instructions without reset and report the average number of successes before the agent fails: N s = 14 .
9 ± 5 .
7 (std).
The 10 RL agents are evaluated with the 10 C - VAE models evaluated above.
These results show that the language module efﬁciently implements language grounding.
Agents achieve instructed transitions almost all the time, resampling alternative goals when previous ones failed.
They only fail when a previous trajectory kicked the blocks out of reach.
Akakzia et al.
(2020) presents a language-conditioned baseline for sensorimotor learning, and showed it could not master the task.
[Table]
4. Discussion
This paper introduces language-conditioned goal generation : a new approach to the problem of language grounding in RL agents.
It shows the following advantages over traditional language-conditioned RL agents:
• Decoupled language grounding.
Our approach enables agents to decouple language acquisition from sensorimotor learning, as it is observed in infants (Wood et al., 1976).
However, nothing in the architecture prevents language from being grounded during sensorimotor learning.
This would result in "overlapping waves" of sensorimotor and linguistic development (Siegler, 1998).
• Behavioral diversity.
The language module generates a diversity of goals matching any instruction (see recall metrics in Table 1).
This results in a behavioral diversity that language-conditioned agents cannot demonstrate.
Indeed, language-conditioned agents translate a given instruction into a unique goal embedding.
Conditioned on put red close_to green they would produce a speciﬁc behavior (e.g. pushing the red block towards the green one) that is dictated by the policy conditioned on the corresponding goal embedding.
Our agent can sample a set of diverse matching goal conﬁgurations.
It could build a pyramid, make a blue-green-red pile or target a dozen other compatible conﬁgurations.
• Trying again.
Generating a diversity of goals matching any instruction enables agents to try again : to ﬁnd alternative approaches to satisfy a same instruction when previous attempts failed.
Table 2 shows that beneﬁting from several attempts signiﬁcantly improves the chances of success.
This could also improve robustness to perturbed environments where former successful behaviors fail.
• Logical expressions.
Generating sets of compatible goals makes it easy to scale language understanding to any logical combination of instructions.
This cannot be achieved with language-conditioned policies.
One strength of the language-conditioned approach is their capability of systematic generalization (Hermann et al., 2017; Bahdanau et al., 2018;
Hill et al., 2019; Colas et al., 2020b).
Our language module also seems to demonstrate generalization abilities: agents can generate goals from situations they never encountered (Type 3).
They can generalize the meaning of sentences they never heard (Type 4) and even apply the latter to unknown situations (Type 5).
Type 4, especially, involves recombinations of action verbs and attributes similar to Hill et al.
; Colas et al.
(2020b).
Language is not always about instructing what to do , but can sometimes be used to give hints about how to do (Schwartz et al., 2019; Zhong et al., 2019).
In addition to languageconditioned goal generation, we could add an optional language-based context embedding.
C - VAE were already used to generate goals matching initial states in Nair et al.
However, the additional condition on instructions brings the beneﬁts of language use: 1) the representation of abstract goals; 2) the systematic generalization capacities.
In addition instructions enable agents to control their goal generation.
Because image-based goal generation was shown to work in Nair et al.
; Pong et al.
; Nair et al.
(2019), we believe our languageconditioned goal generator could be trained to generate imaged-based goals.
Further work could investigate the extension of languageconditioned goal generator to diverse goal-conditioned settings such as, for example, Quality-Diversity algorithms.
These algorithms train populations of diverse and highperforming solutions to a problem (Cully & Demiris, 2017).
Each solution is associated with its behavioral characterization : a low-dimensional description of its behavior.
Our language-conditioned goal generator could be used to map language instructions to that behavioral space, enabling a language-based control of these diversity-seeking evolutionary algorithms (Mouret & Clune, 2015; Lehman & Stanley, 2011; Conti et al., 2018; Colas et al., 2020a).
Acknowledgments
Cédric Colas is partly funded by the French Ministère des
Armées - Direction Générale de l’Armement.
5. Formal Deﬁnition of Semantic Conﬁgurations
Semantic conﬁgurations are based on a collection of formal sys- tems known as predicate logic or ﬁrst-order logic .
They use k -ary relations to describe possible connections between k quantiﬁed variables.
This paper focuses on spatial binary predicates characterizing spatial relations between pairs of physical objects.
We provide formal deﬁnitions, properties and examples below.
Binary predicates Consider a ﬁnite set of objects O = { o 1 , o 2 , ..., o M } .
A binary predicate p associated with a semantic relation r is an expression that takes as input any ordered pair of objects ( o i , o
j ) ∈
O 2 .
p ( o i , o j ) is said true if and only if " o i r o j " is veriﬁed.
For simplicity, we refer to p and r interchangeably.
Examples of binary predicates.
We consider the objects o
and o 2 .
• The expression " o 1 is close to o 2 " describes the predicate close evaluated on ( o 1 , o 2 ).
• The expression " o 2 is above o 1 " describes the predicate above evaluated on ( o 2 , o 1 ).
Semantic mapping functions.
To achieve symbol grounding into non-symbolic sensorimotor interactions using predicates, we deﬁne a semantic mapping function f associated with the binary predicate p as the probability that p is true given the states of the considered objects.
Formally, if we consider the objects o i , o j and their respective states s i , s j , then:
[Equation]
i , s j ) .
This paper assumes oracle deterministic semantic mapping func- tions, i.e. f is a Boolean function in { 0 , 1 } .
Practically, we hardcode a function, assumed internal to the agent, that uses predeﬁned ﬁxed thresholds to determine whether a predicate is true or false given the states of the considered objects.
For example, for the close predicate, it outputs 1 if
and only if the Euclidean distance between the two considered objects is below a deﬁned threshold.
For the sake of simplicity, we omit the word deterministic .
Symmetry and asymmetry.
Consider a ﬁnite set of objects O = { o 1 , o 2 , ..., o M } and a binary predicate p .
The predicate p is said to be symmetric if and only if, for any ordered pair of objects ( o i , o
j ) ∈
O 2 , " o i r o j " and " o j r o i " are equivalent.
As a result, the corresponding semantic mapping function f needs to be symmetric, i.e. f ( o i , o j ) =
f
( o j , o i ) .
The predicate p is said to be asymmetric iff, for any ordered pair ( o i , o j ) ∈
O 2 , " o i r o j " implies not "
o j r
o
i ".
Examples.
We consider the objects o
and o 2 .
• close is symmetric: " o 1 is close to o 2 " ⇔ " o 2 is close to o 1 ".
The corresponding semantic mapping function is based on the Euclidean distance, which is symmetric.
• above is asymmetric: " o 1 is above o 2 " ⇒ not " o 2 is above o 1 ".
The corresponding semantic mapping function evaluates the sign of the difference of the object Z -axis coordinates.
Effective number of predicate relations.
Consider a ﬁnite set of M objects O = { o 1 ,o 2 ,...,o M } and a binary predicate p .
• If p is not symmetric, then the effective number of relations K p that can be described without redundancy is equal to the number of permutations of 2 objects among M , i.e.
[Equation]
• If p is symmetric, then the effective number of relations K p is equal to the number of combinations of 2 objects among M , i.e.
[Equation]
Semantic conﬁgurations based on spatial relations.
Let ( p i )
i ∈
[1 ..P ] be a list of P binary predicates.
The concatenation of the evaluations of the semantic mapping functions f i on the
K pi pairs of objects forms a semantic conﬁguration .
It is an abstract representation of a scene which characterizes all relations deﬁned by the ( p i ) predicates among the M objects.
This deﬁnes a binary semantic conﬁguration space C p
= { 0 , 1 } K c , where
K c = (cid:80) Pi =1 K p i .
If any world conﬁguration can be mapped to C p , not all conﬁgurations are reachable (e.g. o 1 cannot be above and below o 2 at the same time).
Semantic representation space in Fetch Manipulate .
In the Fetch Manipulate environment, we restrict semantic representations to the use of the close and above binary predicates applied on M =3 objects.
The resulting semantic conﬁgurations are formed by:
[Equation] [Equation]
[Equation]
2 )] , where c() and a() refer to the close and above predicates respectively and ( o 1 , o 2 , o 3 ) are the red, green and blue blocks respec- tively.
6. Language-Conditioned Goal Generator Training
We use a conditional Variational Auto-Encoder ( C - VAE )
(Sohn et al., 2015).
Conditioned on the initial conﬁguration and a sentence describing the expected transformation of one object relation, it generates compatible goal conﬁgurations.
After the ﬁrst phase of goal-directed sensorimotor training, the agent interacts with a hard-coded social partner as described in Main Section 2.2.
From these interactions, we obtain a dataset of 5000 triplets: initial con- ﬁguration, ﬁnal conﬁguration and sentence describing one change of predicate from the initial to the ﬁnal conﬁguration.
The list of sentences used by the synthetic social partner are provided in
Content of test sets.
We describe the 5 test sets:
Test set 1 is made of input pairs ( c i , s ) from the training set, but tests the coverage of all compatible ﬁnal conﬁgurations
C f , 80% of which are not found in the training set.
In that sense, it is partly a test set.
Test set 2 contains two input pairs: { [0 1 0 0 0 0 0 0 0] , put blue close_to green } and { [0 0 1 0 0 0 0 0 0] , put green below red } corresponding to 7 and 24 compatible ﬁnal conﬁgurations respectively.
Test set 3 corresponds to all pairs including the initial conﬁg- uration c i =
[1 1 0 0 0 0 0 0 0] ( 29 pairs), with an average of 13 compatible ﬁnal conﬁgurations.
Test set 4 corresponds to all pairs including one of the sen- tences put green on_top_of red and put blue far_from red , i.e. 20 pairs with an average of 9 .
5 compatible ﬁnal conﬁgurations.
Test set 5 is all pairs that include both the initial conﬁguration of test set 3 and one of the sentences of test set 4 , i.e. 2 pairs with 6 and 13 compatible goals respectively.
Note that pairs of set 5 are removed from sets 3 and 4 .
[Table]
Close
Put block A close_to block B , 0 →
Bring block B and block A together ,
Put block B close_to block A ,
Bring block A and block B together ,
Get block B and block A close_from each_other ,
Get block
A close_to block B
Get block A and block B close_from each_other ,
Get block B close_to block A .
Close
Put block A far_from block B , 1 →
Get block A far_from block B ,
Put block B far_from block A ,
Get block B far_from block A ,
Get block A and block B far_from each_other ,
Bring block A and block B apart ,
Get block B and block A far_from each_other ,
Bring block B and block A apart .
Above
Put block A above block B , 1 →
Put block A on_top_of block B ,
Put block B under block A ,
Put block B below block A .
Above
Remove block A from_above block B , 1 →
Remove block A from block B ,
Remove block B from_below block A ,
Put block B and block A on_the_same_plane , Put block A and block B on_the_same_plane .
Testing on logical expressions of instructions.
To evaluate our agents on logical functions of instructions, we generate three types of expressions:
1. 100 instructions of the form "A and B" where A and B are basic instructions corresponding to shifts of the form above 0 → 1 (see Table 3).
These intersections correspond to stacks of 3 or pyramids.
200 instructions of the form "A and B" where A and B are above and close instructions respectively.
B can be replaced by "not B" with probability 0.5.
200 instructions of the form "(A and B) or (C and D))", where A, B, C, D are basic instructions:
A and C are above instructions while B and D are close instructions.
Here also, any instruction can be replaced by its negation with probability 0.
Hyperparameters.
The encoder is a fully-connected neural network with two layers of size 128 and ReLU activations.
It takes as input the concatenation of the ﬁnal binary conﬁguration and its two conditions: the initial binary conﬁguration and an embedding of the NL sentence.
The NL sentence is embedded with an recurrent network with embedding size 100 , tanh non-linearities and biases.
The encoder outputs the mean and log-variance of the latent distribution of size 27 .
The decoder is also a fully-connected network with two hidden layers of size 128 and ReLU activations.
It takes as input the latent code z and the same conditions as the encoder.
As it generates binary vectors, the last layer uses sigmoid activations.
We train the architecture with a mixture of Kullback- Leibler divergence loss ( KD loss ) w.r.t a standard Gaussian prior and a binary Cross-Entropy loss ( BCE loss ) .
The combined loss is loss =
BCE loss + β × KD loss with β = 0 .
We use an Adam optimizer with a learning rate of 5 × 10 − 4 , a batch size of 128 and optimize for 150 epochs.
As training is fast ( ≈ 2 min on a single cpu), we conducted a quick hyperparameter search over β , layer sizes, learning rates and latent sizes (see Table 4).
We found robust results for various layer sizes, various β below 1 and latent sizes above 9 .
[Table]