I N A N UTSHELL , THE H UMAN A SKED FOR T HIS : L ATENT G OALS FOR F OLLOWING T EMPORAL S PECIFICATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
We address building agents whose goal is to execute out-of distribution (OOD) multi-task instructions expressed in temporal logic (TL) by using deep reinforcement learning (DRL).
Recent works provided evidence that the agent’s neural architecture is a key feature when DRL agents are learning to solve OOD tasks in TL.
Yet, the studies on this topic are still in their infancy.
In this work, we propose a new deep learning conﬁguration with inductive biases that lead agents to generate latent representations of their current goal, yielding a stronger generalisation performance. We use these latent-goal networks within a neuro-symbolic framework that executes multi-task formally-deﬁned instructions and contrast the performance of the proposed networks against employing different state-of-the-art (SOTA) architectures when generalising to unseen instructions in OOD environments.
I NTRODUCTION
Building agents that generalise their learning to satisfy new speciﬁcations is a crucial goal of artiﬁcial intelligence (Oh et al., 2017; Bahdanau et al., 2019; Hill et al., 2021).
Deep reinforcement learning (DRL) holds promise in autonomous agents that tackle complex scenarios (Silver et al., 2017; Samvelyan et al., 2019), which motivates ongoing research with DRL algorithms following human instructions expressed in natural language (Yu et al., 2018; Mao et al., 2019).
Unfortunately, generalisation in DRL is linked to training autonomous agents with large numbers of instructions requiring to build manually those natural language expressions and their corresponding reward functions, which prevents such methods from scaling efﬁciently (Lake, 2019; Vaezipoor et al., 2021).
These considerations inspired research on agents learning from formally speciﬁed instructions (Wen & Topcu, 2016;
Alshiekh et al., 2018; Simão et al., 2021) as a substitute for natural language.
Formal languages (Huth & Ryan, 2004) offer desirable properties such as unambiguous semantics and compositional syntax, allowing to automatically generate large amounts of training instructions and their corresponding reward functions.
Earlier contributions in this area rely on the compositional nature of formal languages, often employing multiple policy networks to execute temporal logic (TL) instructions (Andreas et al., 2017; Icarte et al., 2018; Kuo et al., 2020).
However, these approaches do not scale well with the number of instructions since policy networks are a computationally costly resource and, consequently, these earlier studies are restricted to relatively small state-spaces environments that require less computation, e.g., non-visual settings.
More recent contributions (León et al., 2020; Vaezipoor et al., 2021) have presented DRL frameworks that are capable of generalizing to large numbers of out-of-distribution (OOD, i.e., never seen during training) instructions while relying on a single policy network.
Those latter studies evidence that deep learning architectures within DRL agents are key towards the ability of agents to generalise formal instructions.
However, network architectures in previous literature have mostly followed a standard conﬁguration where all the network’s layers that are not used to encode information (e.g., convolutional layers for images, recurrent layers for text) have equal access to both human instructions and environment’s observation.
We propose a novel conﬁguration to help agents generalise better by having a task-agnostic representation of the current state concatenated to a latent goal that is form by processing both observation of the agent and human instruction.
As a motivation example, let us consider two scenarios: 1) a robot that is at the center of an empty room with a red square at the bottom right corner, 2) a robot at the same position in a room that is identical to the one in (1) except that the red square is now green. Intuitively, we can say that, if we give the instruction “go to the red square” in (1), the goal of the robot is the same as if we say “go to the green square” in (2), because the two tasks can be abstracted as “turn to face the bottom right of the room, then move straight”.
In a nutshell, computing the human instruction together with the current state of the environment – being at the center of the room, with the object asked by the human in the bottom right corner – allows to deduce that the optimal policy is the same in both scenarios.
Our contributions are listed as follows:
• We propose a new deep learning conﬁguration (the latent-goal architecture) that helps agents to generalise better when solving multi-task instructions in OOD scenarios.
• We are ﬁrst to evaluate the performance of multiple state-of-the-art (SOTA) neural networks targeting generalisation when following temporal logic instructions.
• Through ablation studies, we ﬁnd that networks generating independent pieces of information within their output are specially beneﬁted from being used within latent-goal architectures.
The remaining of the paper is structured as follows: Section 2 brieﬂy introduces the key concepts needed to follow this work.
Section 3 presents the formal language used to generate instructions and the neuro-symbolic framework that agents use in our experiments. Then, Section 4 details the proposed network conﬁguration.
Section 5 includes the experimental settings, empirical results and ablation studies.
Last, Sections 6 and 7 discuss related works and conclusions, respectively.
B ACKGROUND
We develop agents aimed to execute OOD instructions expressed in TL while navigating in partially observable (p.o.) environments.
Speciﬁcally, agents operate with a ﬁxed perspective and a limited visual range.
Below we introduce the basic concepts used in the rest of the paper.
Reinforcement learning.
Our p.o. environment is modelled as a p.o.
Markov decision process (POMDP).
A POMDP is a tuple M = (cid:104) S, A, P, R, Z, O, γ (cid:105) where (i) S is the set of states s, s (cid:48) , . . .
; (ii) A is the set of actions a, a (cid:48) , . . .
; (iii) P : S ×
A × S → [0 , 1] is the (probabilistic) transition function ; (iv) R : S ×
A × S → R is the reward function ; (v) Z is the set of observations z, z (cid:48) , . . . .
(vi) O : S ×
A × S → Z is the observation function .
(vii) γ ∈ [0 , 1) is the discount factor .
At each time step t , the agent chooses an action a t ∈
A triggering an update in the current state from s t to s t +1 ∈ S according to P .
Then, R t =
R ( s t , a, s t +1 ) provides the reward associated with the transition, and O ( s t , a t , s t +1 ) generates a new observation
o t +1 for the agent.
Intuitively, the goal of the learning agent is to choose the policy π that maximizes the expected sum of discounted rewards:
V π ( x ) def = E π (cid:104)(cid:80) t ≥
γ t r t (cid:105) , where γ ∈ [0 , 1) is the discount factor (see Sutton & Barto (2018)).
Relational networks.
Relational networks (RelNets) are a particular kind of neural network whose structure is explicitly designed for reasoning about relations (Kemp & Tenenbaum, 2008).
Previous contributions on solving formal instructions with DRL have been mainly focused on multi-layer perceptrons (MLPs) and recurrent networks (Goodfellow et al., 2016), yet RelNets are of particular interest in our context since they hold promise to improve generalization in DRL (Santoro et al., 2018).
Their central principle is to constrain the functional form of the neural network in the relational module so that it captures the core common properties needed to reason about the relations between entities (e.g., objects) and their properties (e.g. color), see Santoro et al.
Recently, relational networks have been enhanced with the inclusion of the widely-known key-value attention mechanism (Bahdanau et al., 2015).
This method relies on the computation of attention scores that are used to decide in which portions of the input data the layer using this mechanism should focus on.
L EARNING TO SOLVE SATTL INSTRUCTIONS
In this section we detail the shared features of our agents.
Concretely, Sec.
3.1 introduces the formal language we use to procedurally generate instructions, while Sec.
3.2 details the neuro-symbolic framework that we use while testing the different neural network conﬁgurations.
3. S AFETY - AWARE T ASK T EMPORAL L OGIC
We investigate agents learning compositionally instructions in temporal logic.
Below, we formally deﬁne the language from which we generate instructions to be executed by our agents.
Speciﬁcally, we extend task temporal logic (TTL), deﬁned in León et al.
(2020) to study the ability of neural networks to learn systematically from instructions about reachability goals such as "eventually reach a sword".
In this work, we evaluate agents that handle safety constrains as well, e.g. "walk on soil or stone until you reached a sword". Thus, we extend TTL to safety-aware task temporal logic (SATTL).
Deﬁnition 1 (SATTL) .
Let AP be a set of propositional atoms .
The sets of literals l , atomic tasks α , and temporal formulas T in SATTL are inductively deﬁned as follows:
[Equation]
Literals l are positive ( + p ) or negative ( − p ) propositional atoms, or disjunctions thereof.
Atomic tasks α are obtained by applying the temporal operator until ( U ) to pairs of literals.
An atom α = lUl (cid:48) is read as it is the case that l until l (cid:48) holds .
Temporal formulas T are built from atomic tasks by using sequential composition ( ; ) and non-deterministic choice ( ∪ ).
Note that U and ∪ are different operators.
We use an explicit positive operator ( + ) so that both positive and negative tasks have the same length.
This is beneﬁcial for learning negative literals ( − p ) when instructions are given visually, as highlighted in (León et al., 2020).
The temporal operators eventually (cid:5) and always (cid:50) can be respectively deﬁned as (cid:5) l ≡ true Ul and (cid:50) l ≡ lUend , where end is a particular atom true only at the end of the episode.
As TTL, SATTL is interpreted over ﬁnite traces λ , which in this context refers to ﬁnite sequences of states and actions.
We denote time steps, i.e., instants, on the trace as λ [ j ] , for 0 ≤ j < | λ | , where | λ | is the length of the trace.
While, λ [ i, j ] is the (sub)trace between instants i and j .
A model is a tuple N = (cid:104)M , L (cid:105) , where M is a POMDP, and L : S → 2 AP is a labelling of states in S with atoms in AP .
Deﬁnition 2 (Satisfaction) .
Let N be a model and λ a ﬁnite path.
We deﬁne the satisfaction relation | = for literals l , atomic tasks α , and temporal formulas T on path λ as follows:
[Equation]
iff for some 0 ≤ j < | λ | , ( N , λ
[ j, | λ | ])
|
= l (cid:48) , and for every t ∈ [0 , j ) , ( N , λ [ t, j − 1] [Equation]
| = l [Equation]
iff for some 0 ≤ [Equation] | λ | [Equation]
( N , λ [0 , j [Equation] | = T and ( N , λ [ j + 1 , | λ | ]
[Equation]
= T (cid:48)
[Equation]
Intuitively, by Def. 2 an atomic task α = c
α Ug α is satisﬁed if the “safety” condition c α remains true until goal g α is fulﬁlled.
In the context of this work we are not interested in strict safety warranties but in agents trying to reach a goal while aiming to minimize the violation of additional conditions.
Thus, we may say our agent has satisﬁed α even though it has not fulﬁlled c
α
at every time step before g α .
Formally, in those cases we consider traces that are not starting at the beginning of the episode, but from the state after the last violation.
The agent is penalised for this behaviour through the reward function.
Examples of tasks we use are α 1 =
− grass U (+ axe ∨ + sword ) “avoid grass until you reach an axe or a sword” and α 2 = (+ soil ∨ + mud ) U axe “move through soil or mud until reaching an axe”.
It is not difﬁcult to prove that SATTL is a fragment of the well-known Linear-time
Temporal Logic over ﬁnite traces (LTL f ) (De Giacomo & Vardi, 2013).
We provide a translation of
SATTL into LTL f and the corresponding proof of truth preservation in Appendix F.
3. N EURO - SYMBOLIC AGENTS
We extend the neuro-symbolic framework from León et al.
(2020), intended for DRL agents solving unseen TTL instructions, and adapted it to work with SATTL.
In this section, we provide details about this framework, that entails a symbolic and a neural module.
Intuitively, given a formal instruction T in SATTL, the symbolic module (SM) decomposes T into a sequence of atomic tasks α to be solved sequentially by the neural module (NM), which is instantiated as a DRL algorithm. [Figure]
Symbolic module.
Formally, the ﬁrst component of the SM is the extractor E , which transforms the complex formula T into a list K consisting of all the sequences of atomic tasks α that satisfy T .
As it is common in the literature (Kuo et al., 2020; Vaezipoor et al., 2021), the SM have access to an internal labelling function L I : Z → 2 AP , where L I is the restriction of L to the observation space and maps the observation into the set of atoms (e.g, reached_sword is an atom).
Intuitively the labelling function acts as an event detector that ﬁres when the propositions in AP hold in the environment.
The second functionality is the progression function P , which given the output of L I and the current list K , selects the next task α for the NM and updates K .
Once α is selected, the NM follows the given instruction until it is satisﬁed or until the time horizon is reached, while the role of the SM is to evaluate the fulﬁllment of the task based on the signals from L I .
From the outputs of L I , the SM generates a reward function R , which provides the reward signal for the NM.
Note that from the NM perspective, this signal is the standard reward signal of a POMDP – but in our case associated to the instruction–. Given a task α , R is deﬁned so that the agent receives a positive reward ( +1 ) if the goal condition becomes true ( L I ( z t ) =
g α ), a strong penalisation ( − 1 ) if the agent is neither satisfying the goal nor the safety condition ( L I ( z t ) (cid:54) =
g α or c α ), and a small penalisation ( − 0 . 05 )
if none of the previous is true.
The latter penalisation is meant to encourage the agent to reach the goal.
The pseudocode of the SM and its functions E and P are included in Appendix A.
Since the inner functioning of our symbolic module is analogous to the one in León et al.
(2020), where authors show how the SM can process complex formulas T given a NM that has learnt to execute tasks
α , we devote the remaining sections to the role of the deep learning architecture within the NM when solving atomic tasks, which are the main scope of this work (see end of Sec. 3.1 and Figure 2 for examples of atomic tasks).
Neural module.
The NM consists of a DRL algorithm interacting with the environment to maximize the discounted cumulative reward from R .
All our agents use the same A2C algorithm, a synchronous version of A3C (Mnih et al., 2016).
To facilitate a fair comparison, all the deep learning architectures use the same visual encoder and output layers for the respective benchmarks.
Appendix C provides further detail about these features.
As anticipated, the core of our approach is the deep learning architecture, detailed below.
L ATENT - GOAL ARCHITECTURES
We presented the language employed to generate instructions (Sec 3.1) and the neuro-symbolic framework that use our agents (Sec 3.2).
Now we move towards the main focus of this work, presenting a new neural network conﬁguration that improves OOD generalisation when executing novel multi-task instructions.
Literature about autonomous agents with generalisation-specialized architectures such as relational deep layers (Shanahan et al., 2020) or modular hierarchical networks (Mittal et al., 2020), commonly follows a standard model architecture set-up where the output of a visual encoder is passed through a central module (CM), which in our context is either a relational layer or an MLP, followed by a recurrent layer.
Unfortunately, deep learning agents struggle to generalise instructions compositionally (Lake, 2019), even when generalising between task-similar instructions.
Consequently, we propose an architecture conﬁguration that induces agents to compute both human instructions and sensory inputs in a dedicated channel, generating a latent representation of the current goal.
We call this type of deep learning model a latent-goal architecture .
Figure 1 right shows our proposed conﬁguration when instructions are given as text; while Figure 1 left illustrates the standard architecture from previous literature for comparison (Hill et al., 2020; Küttler et al., 2020; Vaezipoor et al., 2021).
In our conﬁguration, the key feature is that the decision-making layers receive two input streams L g and L s .
L g refers to the latent goal, i.e., a representation of the agent’s current goal given the state and the human instructions, and L s is a vector that contains information about all sensory input processed by the agent except for the instruction.
For a latent-goal architecture to generalise effectively there are two central design properties: i)
L s should be task-agnostic, i.e., the layers computing L s should have no access to what the current task is.
ii)
L g should be passed through an information bottleneck, e.g., a low-dimensional fully-connected layer (FC).
Condition i) forces the layers computing L s to generate representations of the states as generic as possible.
Granting no access to the current task these layers cannot discern which elements provide penalisation or rewards, encapsulating information in a fashion that facilitates generalisation when executing novel instructions.
Condition ii) ensures that decision-making layers rely on the generic representation from L s to extract most of the data from the environment, while the layers processing L g should focus on providing enough information about the goal.
E XPERIMENTS
Here we start presenting the experimental settings and continue with the main experimental results that highlighting the beneﬁts of latent-goal architectures.
We ﬁnish with ablation studies to better understand the inner working of the novel conﬁguration.
Empirical settings.
We use two benchmarks.
One is a Minecraft-inspired environment – Minecraft for short – widely used in the RL-TL literature (Andreas et al., 2017; Vaezipoor et al., 2021).
Particularly, we use the variant from León et al.
(2020) that works with procedural maps and instructions.
In this setting, instructions are visually given as part of the observation.
The second benchmark is MiniGrid (Chevalier-Boisvert et al., 2018), which is also a procedural benchmark where instructions are given as text through a separated channel.
Agents are trained in maps of random size n × n , for n =
[7 − 10] , and evaluated in maps of size n = 7 , 14 , 22 .
[-Instructions are SATTL tasks (see α 
-]
Figure 2 illustrates maps from both are generalisation to unseen instructions, thus results under label “Train”, refer to performance with training instructions, while “Test” to OOD instructions.
In MiniGrid, where agents require grounding words to objects, OOD speciﬁcations are formed with combinations of symbols and objects that were explicitly excluded from training, e.g., if an OOD instructions has the safety condition “avoid orange lava” it means that both color “orange” and shape “lava” where only present in training instructions of the type “reach orange lava”.
In Minecraft, where tasks are depicted with objects themselves,
OOD instructions refer to tasks with zero-shot, i.e., never seen, objects.
Since speciﬁcations are human-given, we assume access to the resolution of the visual instructions and select an encoder generating a visual embedding where instructions are independent from the rest the observation.
This allows the separation of the task from the rest of the observation, which is beneﬁcial for the latent-goal architectures that generate task-agnostic streams of data.
The separation also facilitates a fair comparison between the text and visual instruction domains.
Both benchmarks have at least 200K tasks for training and 65K OOD speciﬁcations.
Further detail is given in Appendix B.
Networks and baselines.
All the networks follow either a standard or a latent goal (LG) conﬁguration (see Sec. 4).
The encoder is a convolutional neural network (CNN) (LeCun et al., 2015) while the recurrent layer in the output is either a long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997), that we use in multi-layered architectures, or a bidirectional recurrent independent mechanism (BRIM) in the case of hierarchical models (Mittal et al., 2020).
In MiniGrid, where instructions are given as text, we use a bidirectional LSTM (Schuster & Paliwal, 1997).
We consider four variants for the central modules within the architectures (with detailed information given in Appendix C.3):
• Relation net (RN) from (Santoro et al., 2017), designed to dispense FC and LSTM layers with relational reasoning.
• Multi-head attention net (MHA) (Zambaldi et al., 2018), which combines relational connections with key-value attention.
• PrediNet (Shanahan et al., 2020), also combining relational and attention operations but speciﬁcally designed to form propositional representations of the output.
• MLP consisting of a single FC layer, which is meant to act as baseline against the relationbased modules.
Results with multi-layer architectures.
Fig. 9 shows the performance of the different multilayer networks (see Fig. 4 for the reference of a random walker).
Results that are labelled as CM networks (e.g., “RN”) refer to standard conﬁgurations, whose CM uses the network of the label.
Correspondingly, results with the LG superscript (e.g., RN LG ) refer to latent-goal conﬁgurations.
Overall, we see that the latent-goal architecture improves the performance of all the networks in most scenarios.
And is always the best performer in the hardest setting ( n = 22 ), where the best latent-goal model achieves 33% and 22% better performance compared to the best standard architectures in
Minecraft and MiniGrid, respectively.
Training results and tables with the speciﬁc values are given in
Appendix D, together with a two-way analysis of variance (ANOVA, independent variables being network architecture in the central modules, and the architecture conﬁguration) that also conﬁrms the statistical impact of the latent-goals.
We also observe that the PrediNet is the network that beneﬁts the most from the new conﬁguration (up to 74%) and PrediNet LG becomes clearly the best architecture.
Noticeably, the two most different results with latent-goal architectures come from the networks that combine relational and attention mechanisms (MHA and PrediNet).
This point if further examined in the ablation below.
Results with hierarchical architectures.
We demonstrate that latent goals can improve the performance of hierarchical architectures.
Speciﬁcally, in line with the two design conditions highlighted in
Sec. 4, we evaluate empirically a hierarchical architecture that restricts the amount of information shared between layers of different hierarchy, while providing instruction and observation data to the bottom layer of the architecture and a task-agnostic input of the environment to the top-level of the hierarchy (see Fig. 8 in the Appendix for an illustration).
In these experiments we ﬁx the PrediNet as the default network for the different CMs – note that the PrediNet LG achieved the best results in the previous evaluation – and replaced the last recurrent layer of the architectures with a
BRIM. BRIMs are modularized hierarchical networks that hold promise for OOD generalisation in RL.
A particularly interesting feature of BRIMs for our method is that they employ a sparse attention mechanism to communicate between different hierarchies of layers.
Speciﬁcally, each layer is composed of various modules, and at each time step only the top k modules (according to a key-value attention mechanism) will be able to communicate with modules from different layers, where k is a hyperparameter.
Thus, when using latent goals with BRIMs we remove the FC layer that acted as information-bottleneck above.
An illustration of the architectures using BRIMs is given in Appendix C.3.
We do not perform any hyperparameter optimization with BRIMs and use directly the same conﬁguration from the RL experiments in Mittal et al.
Fig. 4 contrasts the performance when using a vanilla BRIM layer (BRIM) against using a residual connection (He et al., 2016) (ResBRIM) or latent goals (BRIM LG ).
We see that inducing the latent goals improves the performance of the hierarchical network in all settings.
Hence, we see that deep hierarchical architectures can also capitalize on specialized hierarchy levels where bottom layers produce latent representations of the goal while the higher levels receive task-agnostic information of the state.
5. A BLATIONS
Contrasting MHA and PrediNet with their latent-goal counterparts, we see that the particular differences of the PrediNet with respect to the MHA internal functioning beneﬁts the proposed latent-goal architecture.
After a closer inspection, we see that the most signiﬁcant difference comes from a channeling of the input within the PrediNet.
This network encourages a kind of the semantic separation of the representations that it learns, making an element-wise subtraction of the extracted features, an operation not found in the MHA.
Additionally, we see that the PrediNet modiﬁes its output to explicitly represent positions of the pair of features selected by each of its heads, also not present in the MHA.
Fig.
5 inspects the impact of individually removing these distinct features within the PrediNet and whether the PrediNet is beneﬁcial in both CMs or it is better to use an MHA in any of the modules.
We ﬁnd that the element-wise subtraction is the key feature within the PrediNet since its PNMHA LG uses an in LG removal reduces the performance of PrediNet LG to that of an MHA LG .
None of the networks using MHA in one of the outperform having a PrediNet in both.
Regarding the outlayer of MHA in MC-14x14, we believe that receiving instructions not preprocessed by a recurrent layer is preventing the MHA to overﬁt as much as it does in MiniGrid, thus achieving better performance in maps of close-to-distribution size.
Still, note that MHA LG outperforms MHA in three out of four OOD results.
As for the reason why the standard PrediNet does not outperform the MHA in the same line that PrediNet LG outperforms MHA LG , from Shanahan et al.
we note that at the core of the PrediNet information is organised into small pieces that are processed in parallel channels limiting the ways these pieces can interact.
This pressures the network to learn representations where each separate piece of information has independent meaning and utility.
The result is a representation whose component parts are amenable to recombination.
Such feature is highly beneﬁcial for generalisation, but supposes that the following recurrent layer needs to combine those pieces while producing a meaningful output for both actor and critic.
As highlighted in Santoro et al.
(2017), recurrent layers struggle when relating independent pieces of information.
The latent-goal conﬁguration alleviates this problem by decoupling the problem of goal abstraction and state representation into separated channels.
Consequently, a latent-goal architecture helps to exploit the full potential of the highly re-usable outputs produced by PrediNets.
Consequently, when substituting the element-wise subtraction with a FC layer (as we do with PrediNet LGnoSub ), the output is no longer generating independent pieces of information from the parallel channels, thus reducing their utility and aligning the representations obtained from the PrediNet with the ones generated by an MHA, which employs a single channel of information.
Last, Fig. 6 provides the results of ablation studies regarding conditions (i) and (ii) from Sec. 4.
We observe that, for larger bottleneck sizes, generalisation performance drops signiﬁcantly as the agent no longer relies on the generic representations from L s since biased information from L g is no longer constrained.
That is, larger bottlenecks do not limit the information passed through L g biasing the agent towards the objects used in the training tasks.
We see a similar degradation in performance when providing information about the task through L s , conﬁrming that the beneﬁts are drawn on the generic representations from an input that cannot tell on its own which objects are beneﬁcial or dangerous.
Additionally, we perform control experiments in Minecraft to verify that our agents are learning compositionally from the training instructions, specially with negation, which previous work (Hill et al., 2020;
León et al., 2020) have found particularly challenging.
We ﬁnd evidence that with standard architectures only the MLP and MHA are correctly following the OOD tasks.
In the case of architectures inducing latent goals we observe that MLP LG , MHA LG and PrediNet LG are able to execute correctly the instructions when applied to OOD objects.
Results and further discussion is given in Appendix E.
R ELATED WORK
Applying RL to autonomous agents executing formal speciﬁcations has become a thriving area of research.
The earliest works combining RL and TL focused on tackling temporally extended goals, naturally described as a non-Markovian reward decision process (NRMDP), by generating an equivalent extended MDP were RL algorithms can be applied (Bacchus et al., 1996).
Recent contributions in this line investigate increasingly diverse and expressive languages (Brafman et al., 2018;
Camacho et al., 2019) or multi-agent systems (León & Belardinelli, 2020;
Hammond et al., 2021).
Beyond TL, we ﬁnd methods such as reward machines (a type of ﬁnite state machine)
(Icarte et al., 2018;
Xu et al., 2020; Icarte et al., 2019), or RL-speciﬁc formal languages such as SPECTRL (Jothimurugan et al., 2019).
Closer to our line of work, Kuo et al.
(2020) presents a novel RL framework to follow OOD combinations of known tasks expressed in linear-time temporal logic (LTL) by training multiple networks (one per LTL operator and per object).
In a similar line, Araki et al.
(2021) introduces a hierarchical reinforcement learning framework aimed to learn policies that are optimal and composable while relying on different neural networks each specialized in one subtask.
León et al.
(2020) provides evidence that a DRL agent relying on a single neural network can learn compositionally from the training instructions when the convolutional layers are designed according to the environment where agents operate.
Vaezipoor et al.
(2021) tackles a similar problem in non-visual scenarios by pretraining a neural network to learn to interpret LTL instructions in navigation-based environments.
Our work advances the state-of-the-art in agents following formal instructions by presenting a novel network architecture that improves the performance of DRL algorithms in OOD environments.
Due to the multi-modular nature of the proposed conﬁguration, our work also has links to modular frameworks.
Modular conﬁgurations hold promise in improving generalisation through its natural compositionality (Jothimurugan et al., 2019).
Examples of modular frameworks include DRL agents that combine general control policies, where a global controller is able to combine with different lower-level modules in series of locomotion tasks (Huang et al., 2020).
Closer to this work Karkus et al.
(2020) propose a framework that makes use of a perception and a planner networks to selects the most suitable human-designed instruction for each time step to be forwarded to a controller network.
This differs from our proposed framework, where latent goals are sparsely communicated and learned from scratch without using any predeﬁned human tasks as embedding of the goal.
C ONCLUSIONS
By relying on the assumption that human instructions can be separated from the sensory input, we have presented a novel deep learning conﬁguration that induces situated agents to generate latent representations of their current goal.
Particularly, we have seen that deep learning can draw on architectures where output layers take decisions based on both task-agnostic representations of the environment’s state and the output of a channel that processes observations and human instructions together.
This conﬁguration induces the agent to produce abstract representations of their current complex goal, yielding a better OOD generalisation, as long as the channel processing both human instructions and observations is forced through a small-enough information bottleneck.
We have provided evidence of the strong potential this conﬁguration has with multiple networks and settings.
Additionally, a detailed ablation have shown that networks producing independent and composable outputs are the most beneﬁted from the proposed architecture.
We believe our ﬁndings may have a broad impact within the communities working with DRL, OOD generalisation and formal methods.
Further, this work poses interesting questions about task abstraction and generalisation.
It also remains open how well DRL agents can generalise when the symbolic part may not provide reliable feedback on task progression.
We think that addressing these questions, among many others, will push the boundaries of applicability in RL.
A T HE SYMBOLIC MODULE
In this section we include the pseudo-code of the SM and the internal extractor ( E ) and progression ( P ) functions, whose procedures are explained in Sec.
3.2 of the main text.
Algorithm 1 shows the general functioning of the SM and its iterations with the NM.
Within the SM,
Algorithm 2 details the Extractor E .
Given the complex instruction T , E generates a list K with all the possible sequences of tasks α that satisfy T .
Note that α are the tasks that the neural module (NM) is expected to learn/solve compositionally.
Algorithm 3 refers to the Progression function P , which updates K according to the true evaluation p α given by the internal labelling function L I , that indicates to the SM how the previous task has been solved, and selects the next task α (cid:48) .
B E NVIRONMENT DETAILS
Below expand the description of the benchmarks introduced in the main text:
Minecraft-inpired.
In this environment agents have four actions: Up, Down, Left, Right to move one square from their current position, according to the given direction. Speciﬁcally, we use a conﬁguration where maps are procedurally generated by randomly selecting populations of objects, their placement and agent’s starting position, and where each tile within has a resolution of 9x9 gray (1 channel) pixels. We generate 50 objects split into various sets.
Each object is represented by a matrix of 9x9 values. These values were procedurally generated with pseudo-aleatory numbers (we used a ﬁxed seed).
The global set of objects is referred to as X , where the total number of objects is |X| =
The set is partitioned into pretraining set |X 1 | = 35 , training set |X 2 | = 20 and a test set |X 3 | = 20 , where X = X 1 ∪ X 3 , X 2 ⊂ X 1 and X 1 ∩ X 3 = ∅ .
MiniGrid.
In this environment there are three possible actions: Move Straight, Turn Left,
Turn Right .
Here objects are formed from compositions of colors and shapes. We expand the number of shapes and from MiniGrid to have eleven colors and eight shapes.
We refer to the total set of colors and shapes as C and F respectively.
We also divide the global set of instructions in four categories as follows:
• Reachability atomic goals, which have the form “ True U + p ”.
• Negative reachability goals, e.g., “ True U − p 1 ∨ − p 2 ” or “ True U − p ”.
• Positive instructions such as “ + p 1 U + p 2 ” or “ + p 1 ∨ + p 2 U + p 3 ∨ + p 4 ”.
• Negative conditions, e.g., “ − p 1 U + p 2 ∨ + p 3 ”.
With atomic reachability goals we use sets C 1 and F 1 while training and C 2 and F 2 for testing, where | C 1 | = 8 , | F 1 | = 6 , C = C 1 ∪ C 2 , C 1 ∩ C 2 = ∅ , F = F 1 ∪ F 2 and F 1 ∩ F 2 = ∅ .
The remaining categories of instructions use C 3 , F 3 in training and C 4 , F 4 for testing, where | C 3 | = 8 , | F 3 | = 6 , do 6: Seq ( α 7: end deterministic choices 10:
Initialize choice list: CL 11:
LK ← length ( K ) 12: for all Seq ∈ K do 13: for all choice ∈ α do 14: CL .append(choice) Generate a clone per choice Seq (cid:48) ← Seq 16:
K .append
( Seq
) 17: end for 18: end for 19:
Initialize counter c ← −20: for i in range ( length ( K )) do 21: if i % LK == 0 then 22: c + =23: end if 24:
We append a different choice to each sequence cloned K
[ i ] .append ( CL [ c ]) 25: end for 26: end if 27: end for 28: return K
C = C 3 ∪ C 4 , C 3 ∩ C 4 = ∅ , F = F 3 ∪ F 4 and F 3 ∩ F 4 = ∅ .
Also, C 2 ⊂ C 3 , C 4 ⊂ C 1 , F 2 ⊂ Fand F 4 ⊂ F 1 .
Intuitively, we use reachability goals to ground some shapes and color for our agents, and the rest of the instructions for the remaining elements.
Then we test the agents with the unseen colors and shapes for each category.
Task generation.
We procedurally generate tasks as detailed in Algorithm 4.
Where task type ( α type ) refers to one of the four categories of instructions as detailed in MiniGrid paragraph in
Appendix B. Note that in Minecraft all task types have the same population of training objects since all the test objects are zero-shot.
Experiment Architecture.
All the agents in this work use a pretrained encoder.
While pretraining we use only instructions of the form (cid:5) l = true Ul , with objects from X 1 in Minecraft, or C 1 and Fin MiniGrid.
This pretraining stage helps the encoder to differentiate objects within the environment.
Once pretrained, the encoder remains frozen to prevent overﬁtting.
The training stage is 70M timesteps in Minecraft or 50M in MiniGrid, with maps of training size and populated with objects from the training sets.
For testing, all the training parameters are frozen and the agents are evaluated in sets of 500 maps populated with OOD objects from the test sets.
At the beginning of each episode, a task α is procedurally generated (e.g., α 1 or α 2 above).
Train and test tasks are generated with objects from the corresponding set.
In Minecraft, tasks include zero-shot (i.e., unseen) objects.
Once we selected a task, a new map is generated where the agent is placed in an aleatory position, some or all the goal objects are randomly placed and some or all the objects for the constraint are also randomly placed.
Additional objects from the corresponding set are also included in the map as distractions.
1: Function P , Input: K , p α 2: if p α (cid:54) = ∅ then 3: for all Seq ∈ K do 4: if p α fulﬁlls Seq [0] then 5: Seq .pop ( Seq [0] ) 6: else 7: K .pop
( Seq ) 8: end if 9: end for 10: end if 11: α (cid:48)
← ∅ 12: if K == ∅ then 13: return α (cid:48) 14: else 15:
Select the head of the ﬁrst sequence: α (cid:48) ← K [0][0] 16: for all Seq ∈ K do 17: // look for a non-deterministic choice 18: if α (cid:48) & Seq [0] can be combined then 19: α (cid:48)
← α (cid:48) ∪ Seq [0] 20: end if 21: end for 22: end if 23: return α (cid:48)
1: Input: is _ test 2: Randomly select task type α type 3: if is _ test then 4:
Pop ← Collection of test objects for α type 5: else 6:
Pop ← Collection of train objects for α type 7: end if 8: Randomly select number of goals n g range
1 , 2] 9: if α type !
= 1 then 10:
Defaults number of safety constrains: n c =11: else 12:
Defaults number of safety constrains: n c =13: end if 14: if α type =
= 3 then 15:
Randomly select n c in range [1 , 2] 16: end if 17: Form an instruction I with n g goals and n c safety constrains by selecting different objects from
Pop for every constrain and goal.
18: return I
C E MPIRICAL DETAIL
Here we include further details about the empirical evaluation.
The ﬁnal version of the paper will include a link to the the published code.
C. H ARDWARE
For training we use various computing clusters with GPUs such as Nvidia Tesla K80, and GTX 1080.
We employ Intel Xeon CPUs and 7700k and consume 5 GB of RAM per every three independent runs working in parallel.
Running concurrently, each experiment typically takes days in the
Note that these environments are specially suited for agents capable of relational reasoning (Kemp &
Tenenbaum, 2008), i.e., for an agent to perform well it needs to 1) relate the objects that appear in the ﬁrst rows of the visual input and their relative positions with respect to the SATTL symbols and 2) relate the instruction with what it perceives around itself.
For more details about hyperparameters, environments, hardware and computing time we refer to Appendix C in the supplemental material together with the code. C. RL HYPERPARAMETERS All of our agents are trained with A2C. Speciﬁcally, we use a discount factor γ = 0 . 99 , a value loss weight of 0 . 5 and a batch size of 512 .
We work with a scheduled learning rate using a starting value of 8 e − 5 , 6 e − 5 at the step 30 M, and 4 e − 5 at 55 M. In the case of BRIMs, which are signiﬁcantly larger networks, we use a constant learning rate of 3 e − 5 . These values offered the best results when Obs
Task
Encoder
Central module a t −
BRIM bottom layer
BRIM top layer a t −π ( a t )
V t residual connection h t h t −
Obs
Task
Encoder
Encoded
Obs
Encoded
Task
Central modulea t −
BRIM bottom layer
Central module
BRIM top layer a t −π ( a t )
V t
L s h t / L g h t −
Green cells within the BRIM layer refer to activated modules while gray cells are deactivated.
The bottom layer receives as input the output of the central module and the hidden state from the upper layer in the previous time step ( h (cid:48)
t − 1 ).
The upper layer receives the output from the bottom layer ( h t ).
Only strong connections (the ones between activated modules) are shown.
In the case of ResBRIM the upper layer also receives the output of the Central module.
Bottom: BRIM LG .
The architecture is similar to the ResBRIM but the top layer receives as additional input an embedding from the observation exclusively ( L s ).
Consequently, the top layer has only access to the current goal through the lower layers of the hierarchy.
testing after a grid-search in a range values between 1 e − 5 and 1 e − 3 .
The entropy-exploration term was ﬁxed to H = 1 e − 3 after testing with values in the range [1 e − 5 , 1 e − 2 ] .
C. N ETWORKS
Our models use 3-layer convolutional networks in both settings.
Particularly, in Minecraft we use a kernel of 3x3 in the ﬁrst and second layers and 1x1 in the third one.
The number of channels in each layer are 8, 16 and 1 while the strides are 3, 3 and 1 respectively.
Note that this encoder has been designed bearing in mind the resolution of the tiles within so that the latent representations (i.e., the outputs of the encoder) of the tasks are independent from the latent representations of the original observation z .
This inductive bias is exploited in the latent-goal architectures.
In MiniGrid all kernels and strides are of size 2.
The number of channels are 16, 32 and 32 respectively.
Note that encoders are pretrained and then frozen.
With MiniGrid’s text instructions we use a bidirectional recurrent layer of size 32.
In multi-layer conﬁgurations the other two recurrent layers are LSTMs of size 128.
The information bottleneck is a FC layer of size 16.
For the central modules in the multilayered architectures, we use the best hyperparameters from a previous ablation study in Shanahan et al.
Particularly, in the case of the MLP we use a single fully-connected layer with 640 units.
The RN has a central hidden size of 256 units and an output size of 640 units.
The MHA employs 32 head with a key size of 16 and value size of 20.
Last, the
PrediNet has 32 heads, with key and relation sizes of 16.
The latent-goal variants use a FC layer of size 16 for the information bottleneck.
Regarding BRIMs we use the hyperparameters recommended for RL in Mittal et al.
Speciﬁcally, we use 2 hierarchical BRIM layers, with 6 modules of 50 LSTM units each.
Four modules ( k ) are active at a given time step (top- k mechanism).
Figureillustrates the different BRIM conﬁgurations that we evaluate in Sec. 5 in the main text when working with visual instructions, i.e., the Minecraft setting.
D A DDITIONAL EMPIRICAL ANALYSIS
In this section we ﬁrst include the training results and tables with the speciﬁc values used to generate the bar-charts.
Then, we detail the results of the statistical analysis (a two-way ANOVA) of the results with multi-layered networks.
Last, we provide the training plots of the experiments in the main document. D. T RAINING PERFORMANCE AND DETAILED VALUES Figs. 9-11 illustrate the training results of the experiments in the main section.
Tables 1-4 provide the speciﬁcs values in a table format.
From the training results, we observe that latent-goal conﬁgurations can improve the performance also the with training instructions and that this improvement becomes more noticeable as environments become larger in size, i.e., as we get further from the training distribution of environments.
The fact that the effects of latent-goal networks are most noticeable with unseen instructions aligns with the conclusions drawn from the main document, where we stated that this architectures are specially conceived to strength agents’ ability to generalise to new tasks.
D. S TATISTICAL ANALYSIS
To conﬁrm that the standard error of the different conditions do not overlap each other, we perform a two-way analysis of variance (two-way ANOVA) with the independent variables being the neural
[Table]
networks used in the central modules and the architecture conﬁguration, i.e., latent goal (ours) or standard conﬁguration.
The 2-way ANOVA is done across the maps of different size of Minecraft and MiniGrid respectively.
Note that to evaluate the different hypotheses, we need to check whether the F value is greater than its corresponding F-critic or not, and to conﬁrm that the P-value is smaller than Alpha.
Table 5 shows the results of the ANOVA analysis with the results from Minecraft.
We see that it conﬁrms that both the architecture conﬁguration and the neural network have a signiﬁcant impact of the performance of the agent.
Noticeably, the ANOVA results show that there is not a signiﬁcant interaction between the two variables, meaning that the neural network is not relevant in the general improvement in performance that latent goals grant in this setting.
Still, note that this is analysis of the statistical impact averaged across all the networks.
As studied in Sec.
5.1 the latent goals do have a different impact with the PrediNet than with other networks.
Table 6 shows the results of the ANOVA test with MiniGrid.
Here we see that neural networks have a signiﬁcant impact in the training performance of the agent but that is not the case with the architecture, i.e., using latent goals does not have a signiﬁcant impact in the global training performance.
Nevertheless, we see that this changes with unseen instructions (test) where the latent goals have a stronger impact in performance than the neural network, being both statistically relevant.
Last, from the interactions’ results we see that both in training and test the interaction of the two
[Table]
[Table]
variables have a signiﬁcant impact in the ﬁnal performance,
i
[Table]
e., the effect in performance that latent goals have is strongly dependant on the neural network and vice versa.
Regarding the differences in how network and architecture impact in Minecraft and MiniGrid, we believe that these are motivated by the additional difﬁculty of MiniGrid observation and action settings.
Speciﬁcally, MiniGrid only allows agents to move forward or turn around
[Table]
and agents can only observe what is in front of them.
This requires agents to rely more on their memory than they do in Minecraft, where they can observe the objects around them in any direction and have actions to move in four different directions.
This point also motivated the need of using an schedule in the introduction of larger training maps in MiniGrid, as detailed in Appendix D.
D. T RAINING PLOTS
Figure 7 shows the training plots of the standard (top), latent-goal (middle) and BRIM (bottom) conﬁgurations in Minecraft (left) and MiniGrid (right).
In the later it can be appreciated a general decrease of performance the 20M mark.
This is because we noticed that in MiniGrid, where the navigational features force the agent to further rely on memory, we noticed that agents struggled when learning with the default conﬁguration of random sizes with n =
[7 − 10] .
To alleviate this, through the ﬁrst 20 M steps we give a higher chance (70%) to maps of size 7, to help agents to learn from the instructions before navigating through larger maps.
Note that good training performance, as shown in these training plots, does not necessarily mean good OOD performance.
This is highlighted in Table 1.
E C OMPOSITIONAL LEARNING
Here we assess whether our agents are learning compositionally from SATTL instructions and check if they are really following the given safety constraints when applied to OOD objects or if they are just learning to reach the goal g α .
We focus our study on their performance with negated constraints, since previous research highlighted as the most challenging type of instructions Hill et al.
León et al.
Compositional learning Lake (2019) (also known as systematic learning Hill et al.
(2020)) refers to the ability of understanding and producing novel utterances by combining already known primitives Chomsky & Lightfoot (2002).
In our context, an agent that is learning compositionally should be able to solve the instruction − c U + p if the primitives c and p are known and it already learnt to solve − c (cid:48)
U + p (cid:48) .
[Table] or a standard conﬁguration. [Table] Table 7 shows the results from a control experiment where we track the performance of the agents according to tasks of the form − c U
+ p when giving reliable, partially occluded or deceptive instructions.
Speciﬁcally, every row shows the test performance of the multi-layered networks trained in Sec. 5 when providing rewards according to the instruction − c U + p , which intuitively means "avoid c until reaching p ".
The ﬁrst row shows the performance of the agents when the SM provides reliable instructions to the NM, i.e., the instruction provided is the one used to generate rewards.
The second row shows the performance under the same setting, but where the extended observation provided to the NM is a reachability goal (true U + p ).
Intuitively, for the second row the agents are given the right goal p but not the safety constraint, i.e., partially occluding information of the real task.
The third row shows the performance when the SM gives deceptive information abut the safety constrain ( + cU + p ), i.e., the SM is "telling" the NM to go through the objects that actually should be avoided.
Agents learning systematically should show worse performance with partially occluded instructions than with the reliable ones, but still better than a random walker.
Additionally, the worst performance should come when provided deceptive instructions.
From Table 7 we see that all the variants using the MHA and MLP modules follow the rule 1 st row > 2 nd row > random > 3 rd (compositional rule, or c.r. for short).
Notably, the best performance comes from a variant that does not use relational networks nor attention (MLP LG ).
Still, this does not imply that MLPs are better suited for negation since agents were trained in a much wider variety of tasks and the MHA, MHA LG and PrediNet LG outperformed the MLP LG in the general evaluation test.
In the case of the RN, the c.r. is satisﬁed but the general low performance and the small difference between the results of the ﬁrst and the second rows (14 and 13.3 respectively) indicates weak generalisation.
This is worse with the M-RN and the PrediNet having both equal or better performance with partially occluded instructions than when receiving the real task as input.
Such results imply that these networks have not correctly learnt to generalise negated instructions.
This is not the case with the PrediNet LG , whose results correctly follow the c.r..
In addition, the worse performance of the PrediNet LG than the MHA with partially occluded and deceptive instructions suggests that the PrediNet LG advantage over the MHA in maps of OOD size ( see Table 1 comes from a better ability of the former to execute safety constraints when compared with the later.
Note that the larger the map the harder it is to ﬁnd p ; thus the bigger chances of accumulating penalizations due to "violations" of the safety condition. [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] Discussion To the best of our knowledge, the only two works that include some empirical evidence of emergent compositional learning with negation, i.e., achieving a performance 50% better than chance are Hill et al.
León et al.
However, in those works negated instructions are interpreted as "something different from", e.g., "not p " intuitively meant "get something different from p ".
Instead, here we have an interpretation of negated atoms more aligned with propositional and temporal logic, and also to natural language, where "not p " intuitively means that p must be false.
Hence, we believe this is the ﬁrst work to show that DRL agents are capable of learning the abstract operator of negation in its classical interpretation and successfully apply it to new utterances.
F SATTL AND LTL f
We stated that SATTL is a fragment of the widely-used LTL f .
The syntax of LTL f is deﬁned as follows:
[Equation]
Since both LTL f and SATTL are deﬁned ofer ﬁnite traces, we can directly introduce the following translations:
[Equation]
Deﬁnition 3.
Translations τ from Task Temporal Logic to LTL f are deﬁned as follows:
[Equation]
T We immediately prove that translation τ preserve the interpretation of formulae in SATTL.
Proposition 1.
Given a model N and trace λ , for every formula T in SATTL,
[Equation]
Proof.
The proof is by induction on the structure of formula T .
The base case follows immediately by the semantics of SATTL and LTL f .
As for the induction step, the case of interest is for formulae of type T ; T (cid:48) .
In particular, ( N , λ ) | =
T ; T (cid:48) iff for some 0 ≤ j < | λ | , ( N , λ [0 , j ]) | =
T and ( N , λ
[ j + 1 , | λ ])
| =
T (cid:48) .
By induction hypothesis, this is equivalent to ( N , λ [0 , j ]) | =
lUl (cid:48) and ( N , λ [ j + 1 , | λ ]) | =
τ ( T (cid:48) ) in the case of
T = lUl (cid:48) .
Finally, this is equivalent to ( N , λ ) | =
lU ( l (cid:48)
∧ τ ( T (cid:48) )) .
The cases for T =
T 1 ; T 2 and
T = T 1 ∪ T 2 are dealt with similarly.
Finally, the case for T ∪ T (cid:48) follows by induction hypothesis and the distributivity of ∨ .