A GENT , DO YOU SEE IT NOW ? SYSTEMATIC GENERALISATION IN DEEP REINFORCEMENT LEARNING
A BSTRACT
Systematic generalisation, i.e., the algebraic capacity to understand and execute unseen tasks by combining already known primitives, is one of the most desirable features for a computational model.
Good adaptation to novel tasks in open-ended settings relies heavily on the ability of agents to reuse their past experience and recombine meaningful learning pieces to tackle new goals.
In this work, we analyse how the architecture of convolutional layers impacts on the performance of autonomous agents when generalising to zero-shot, unseen tasks while executing human instructions.
Our findings suggest that the convolutional architecture that is correctly suited to the environment the agent will interact with, may be of greater importance than having a generic convolutional network trained in the given environment.
I NTRODUCTION
A major long-term goal of artificial intelligence research is the creation of generalist agents than can handle a never-ending stream of challenges.
Open-ended research (Wang et al., 2019; 2020; ?
; Team et al., 2021) conceives algorithms that continuously improve themselves by creating both challenges and solutions.
However, for this kind of algorithms to find meaningful solutions there is major concern on the generation of effective curriculum learning procedures that balance the right amount of optimization and exploration, facilitating generalisation.
To achieve this, a relevant topic in open-ended research is investigating the drivers of generalisation in autonomous agents.
Systematic generalisation, also known as compositional generalisation (Lake, 2019), concerns the human capacity for compositional learning, that is, the algebraic capacity to understand and execute novel tasks by combining already known primitives.
For instance, once a human agent understands the instruction “get wood " and the meaning of iron , they will understand the task “get iron ".
This is a desirable feature for a computational model, as it suggests that once the model is able to understand the components of a task, it should be able to satisfy tasks with the same components in possibly different combinations.
The ability of neural-based agents to learn systematically and generalise beyond the training environment is a recurrent point of debate in machine learning (ML).
In the context of autonomous agents solving human-given instructions, deep reinforcement learning (DRL) holds considerable promise (Co-Reyes et al., 2019; Luketina et al., 2019).
Various research have shown that DRL agents can exhibit some degree of systematic generalisation with natural language (Yu et al., 2018;
Bahdanau et al., 2019; Hill et al., 2021).
However, it has become evident that systematic generalisation only emerges when the right environmental drivers are present (Mao et al., 2019; Lake, 2019), e.g. a fixed visual perspective.
One of the latest research in this area (Hill et al., 2020) focused on the emergence of generalisation with abstract logical operators.
This work shows that DRL agents can generalise positive instructions from as few as six different specifications, while for negated instructions agents required a much larger variety of examples ( ∼ 100 ).
Consequently those findings suggest that when designing curriculum learning or open-ended algorithms for agents working with logical operators, e.g., “do not step into water", we should consider environments with larger varieties of instructions that we would when tackling positive instructions, e.g., “step into grass".
Nevertheless, in this work we find that systematically generalisation can emerge with instructions with logic operators such as negation as early as with specifications without them and provide evidence that the architecture of the convolutional layers, i.e., the way that agents “see" their environment, plays a key role in the emergence of systematicity.
Contributions In this work we aim to answer the question: can systematic learning emerge in DRL agents when prompted with abstract logic operators, e.g., disjunction, when learning from a limited variety ( ∼ 6 ) of examples?
Specifically, we make the following contributions:
• We provide empirical evidence of emergent systematic generalisation in DRL agents with abstract operators – including negation, which previous works have struggled with – when learning from as few as six training instructions.
Note that the previous latest research highlighted that significantly larger numbers ( ∼ 100 ) were required (Hill et al., 2020).
• We find that the architecture of the convolutional layers (c l.) – a background feature in previous literature – can be key when generalising to zero-shot tasks.
Specifically, when variety of instructions is limited, we find that only those convolutional networks whose architecture is adapted to their environment generalise in a systematic fashion.
The remaining of this document is structured as follows: Section 2 introduces the key concepts needed to understand our research.
Section 3 presents the formal language we use to provide instructions, while Section 4.2 specify the peculiarities of our DRL agents, focusing on the CNN architecture.
Section 5 details the experimental settings and results obtained.
Finally, Sections 6 and 7 report on related works and discussion, respectively.
S YSTEMATIC GENERALISATION OF ABSTRACT OPERATORS
Our situated agents operate with a fixed visual perspective and limited visual range.
Thus, we work with partially observable (p.o.) environments.
Our target is to assess whether systematic learning is emerging in such environments while stressing that the convolutional neural network (CNN) architecture is key to facilitate generalisation when the number of training examples is limited.
This section briefly introduces the concepts needed to present our work.
Reinforcement Learning.
In RL p.o. environments are typically modelled as a POMDP.
Definition 1 (POMDP) .
A p.o.
Markovian Reward Decision Process is a tuple M = ⟨ S, A, P, R, Z, O, γ ⟩ where (i) S is the set of states s, s ′ , . . . .
(ii) A is the set of actions a, a ′ , . . . .
(iii) P ( s ′ | s, a ) :
S ×
A × S → [0 , 1] is the (probabilistic) transition function .
(iv) R ( s, a, s ) :
S ×
A × S → R is the Markovian reward function .
(v) Z is the set of observations z, z ′ , . . . .
(vi)
O ( s, a, s ) :
S ×
A × S → Z is the observation function .
(vii) γ ∈ [0 , 1) is the discount factor .
At every timestep t , the agent chooses an action a t ∈
A updating the current state from s t to s t +1 ∈ S according to P .
Then, R t =
R ( s t , a, s t +1 ) provides the reward associated with the transition, and O ( s t , a t , s t +1 ) generates an observation o t +1 that will be provided to the agent.
Intuitively, the goal of the learning agent is to choose the policy π that maximizes the expected discounted reward from any state s .
Given the action-value function Q π ( s, a ) =
E [ R t | s t =
s, a ] ,
RL searches a policy that maximizes the action-value function Q ∗ ( s, a ) = max π Q π ( s, a ) .
For more detail we refer to Sutton & Barto (2018).
Convolutional neural network (CNN).
A CNN is a specialized type of neural network model (LeCun et al., 2015) designed for working with visual data.
Particularly, a CNN is a network with one or more convolutional layers (c.l.).
Relevant features of c.l. for this study include kernels , a matrix of weights which is slid across an input image and multiplied so that the output is enhanced in a desirable manner, and stride , which measures the step-size of the kernel slid.
After the last convolutional layer, the output is a tensor that we refer to as visual embedding .
We point to Goodfellow et al.
for a complete description of CNNs.
Evaluating the emergence of systematic generalisation.
As anticipated in Sec. 1, we aim to assess whether a DRL agent is able to generalize systematically from instructions with abstract operators.
Consequently, we do not focus on the raw reward collected by the agent , but rather on whether the agent is correctly following new instructions.
To evaluate this, we work in two popular settings involving procedural maps where agents need to fulfill reachability goals, in line with previous literature (Hill et al., 2020; Icarte et al., 2019; Kuo et al., 2020).
In our experiments agents are trained with instructions such as “get gold" (reach gold) or “get something different from mud" (i.e., reach an object that is not mud).
Then, we evaluate the agents with unseen instructions, e.g., “get something different from iron".
It is evident that a learning agent achieving significantly better rewards than a random walker does not imply systematic learning, e.g., the agent could accumulate larger rewards just by navigating better than a random walker and reaching all the objects promptly.
For this reason, after training, we evaluate generalisation in settings called binary choice maps (BCMs).
In these test maps, agents are evaluated with zero-shot reliable instructions, i.e., unseen reliable instructions that point to objects granting a positive reward, or zero-shot deceptive instructions, i.e., instructions pointing to objects granting a penalisation – a negative reward –.
Since during training we always provide reliable instructions, agents that show systematic learning – not complete systematicity but at least similar to closely related works (Bahdanau et al., 2019; Hill et al., 2020) – should accumulate significantly higher rewards with zero-shot reliable instructions than with deceptive ones (we estimate around two times higher or greater to be a good reference in our settings, detailed in Sec. 5).
T ASK T EMPORAL L OGIC
Since we are concerned about generalisation with logic operators, we start by defining a language in temporal logic (TL)
(Huth & Ryan, 2004) whose atomic components we use to train the DRL algorithm.
Task temporal logic (TTL) is an expressive, learning-oriented TL language interpreted over finite traces, i.e, over finite episodes.
Given a set AT of atomic tasks α, β, . . .
, the syntax of
TTL is defined as follows:
Definition 2 (TTL) .
Every formula ϕ in TTL is built from atomic tasks α ∈ AT by using negation “ ∼ " (on atoms only), sequential composition “ ; " and non-deterministic choice of atoms “ ∨ " and formulae “ ∪ ":
[Equation]
Intuitively, an atomic task α corresponds to a reachability goal in the TL literature, in the sense that the fulfilment condition associated with α will eventually hold.
Task α ∼ encapsulates a special form of negation; informally it means that something different from α will eventually hold.
Choosing this form of negation allows us to contrast our results with those of previous studies of this operator in natural language (Hill et al., 2020).
Note that the negation symbol is intentionally positioned after the atomic task.
We found that this feature helps during training with visual instructions, as it forces the neural module (Sec. 4.2) to process the same number of symbols to distinguish negative from positive tasks (we refer to Sec.
E.1 in the supplemental material for further detail).
Formulae ϕ ; ϕ ′ intuitively express that ϕ ′ follows ϕ in sequential order; whereas ϕ ∪ ϕ ′ means that either ϕ or ϕ ′ holds.
We then also introduce an abbreviation for the concurrent (non-sequential) composition ∩ as follows: ϕ ∩ ϕ ′ ≡ ( ϕ ; ϕ ′ ) ∪ ( ϕ ′ ; ϕ ) .
We say that our language TTL is learning-oriented as its logical operators and their positions in formulae are so chosen as to help the training process of the learning agent.
TTL is interpreted over finite traces λ , where | λ | denotes the length of the trace.
We denote time steps, i.e., instants, on the trace as λ [ j ] , for 0 ≤ j < | λ | ; whereas λ [ i, j ] is the (sub)trace between instants i and j .
In order to define the satisfaction relation for TTL, we associate with every atomic task α an atomic proposition p α , which represents α ’s fulfilment condition.
Note again that, in this context, α is a reachability goal , typically expressed in TL as an eventuality ⋄ p α .
Then, a model is a tuple N = ⟨M , L ⟩ , where M is a POMDP, and L : S → 2 AP is a labelling of states in S with truth evaluations of atoms p α in some set AP of atoms.
Definition 3 (Satisfaction) .
Let N be a model and λ a finite trace.
We define the satisfaction relation | = for tasks T and formulae ϕ on trace λ inductively as follows:
( N , λ ) | =
α iff for some 0 ≤ j < | λ | , p α ∈ L ( λ [ j ]) ( N , λ ) | = α ∼ iff for some 0 ≤ j < | λ | , for some q ̸ = p α , q ∈ L ( λ [ j ]) and p α / ∈ L ( λ [ j ]) ( N , λ ) |
=
α ∨ α ′ iff ( N , λ ) | = α or ( N , λ ) | = α ′ ( N , λ ) | = ϕ ; ϕ ′
iff for some 0 ≤ j < | λ | , ( N , λ [0 , j ]) | = ϕ
and ( N , λ
[ j + 1 , | λ − 1 | ]) |
= ϕ ′ ( N , λ ) | = ϕ ∪ ϕ ′
iff ( N , λ ) | = ϕ or ( N , λ ) | = ϕ ′
By Def. 3, an atomic task α indeed corresponds to a formula ⋄ p α of temporal logic, where p α is the fulfilment condition associated with α .
It can be immediately inferred that TTL is a fragment of the widely-used Linear-time Temporal Logic over finite traces (LTL f )
(De Giacomo & Vardi, 2013).
We provide a translation of TTL into LTL f and the corresponding proof of truth preservation in Sec.
F in the appendix.
To better understand the language we present a toy example.
Example 1.
The specification “get something different from grass, then collect grass and later use either the workbench or the toolshed" can be expressed in TTL as:
[Equation]
TTL is designed to be expressive enough to encapsulate both the tasks described in (Andreas et al., 2017), a popular benchmark in the RL-TL literature (León & Belardinelli, 2020; Toro Icarte et al., 2018; De Giacomo et al., 2019), and the negated instructions from studies about negation in natural language (Hill et al., 2020), i.e., encapsulating “not wood" as “get something different from wood".
Remark.
As a foundational work assessing the ability of learning agents to generalise abstract operators with unseen instructions, we intentionally restricted TTL syntax with atomic operators to negation and disjunction.
This allows to carefully study the ability of DRL agents to generalise with these two important operators.
Given a task T and an event detector L I : Z → 2 AP that maps observations into sets of atoms in
AP .
The reward function for the DRL agent is defined as follows:
[Equation]
( p t ) =   d if p t = ∅ , where d ∈ R and d < 0; i if p t = p α , for α occurring in T and T ̸ = α ∼ , or if p t ̸
= p
α , for α occurring in T and T ≡ α ∼ , where i ∈ R and i ≥ 0; c otherwise, where c ∈ R and c ≪ d .
where L I ( z t ) =
p t for any time step t .
Intuitively, the agent is rewarded when the event detector L I fires a signal that satisfies the current task, and penalised when L I signals an event not related to T .
A small penalisation is also given at each time step to induce the agent to promptly solve T .
In a similar fashion to previous literature applying temporal logic to RL (TL-RL), we rely on a neuro-symbolic framework composed by a symbolic module (SM) and a neural module (NM), the former specializes in generalising formulae operators while the later is a DRL algorithm that learns atomic operators in a systematic fashion. Note that our main contribution is the analysis of the impact of different CNN architectures in DRL agents. Nevertheless, as bridging work between RL-TL and systematic generalisation literature, we detail both modules in the section below to provide a complete picture of our method.
4. S YMBOLIC M ODULE
Given an instruction ϕ in TTL, the first task of the symbolic module (SM) (detailed in Algorithmin Appendix B), is to decompose ϕ into sequences of atomic tasks for the neural module (NM).
Intuitively, the SM reduces the problem into a progression of POMDPs, which is a well-known procedure in the literature on RL with TL specifications (Brafman et al., 2018; Toro Icarte et al., 2018).
In particular, a task extractor E transforms ϕ into the set K of all sequences of tasks T that satisfy ϕ
(see Algorithm 2).
As standard in the literature (Andreas et al., 2017; Kuo et al., 2020), we assume that the SM has access to an internal labelling function (a.k.a.
event detector 1 ) L
I : Z → 2 AP , which maps the agent’s observations into sets of atoms in AP .
Note that L I might differ from the labelling L originally defined in Sec. 3, since L is defined on states.
For our approach to be effective, observations have to contain enough information to decide the truth of atoms, so that both the agent’s internal and the model’s overall labelling functions remain aligned.
Given L I and the sequences of tasks in K , a progression function P selects the next task from K that the NM has to solve.
Once a task is solved, P updates K and selects the next task.
This process is repeated until ϕ is satisfied or the time limit is reached.
We include the pseudocode for K and P subroutines in Algorithms 2 andof the Appendix respectively.
When forwarding the current task T to the NM, the SM expands observation z with the subformula corresponding to T .
In the current context, z is a square visual input with a fixed perspective.
In our empirical setting, this is expanded either by adding extra rows to z containing T (see Figurecenter) or providing T in a separated "language" channel (see Figure 1 right).
We now illustrate the inner working of the SM.
Example 2.
{+Consider one of the complex specifications used in Sec. 5: ϕ 
+}
=
(( wood ; grass ) ∪ ( iron ; axe )); workbench ; ( toolshed ∼ ) .
Given ϕ 1 , the task extractor E outputs set K = {⟨ [ wood ] , [ grass ] , [ workbench ] , [ toolshed ∼ ] ⟩ , ⟨ [ iron ] , [ axe ] , [ workbench ] , [ toolshed ∼ ] ⟩} with two available sequences of tasks.
Set K is forwarded to the progression function P and since we have two lists with different initial elements, both positive, P selects [ wood ∨ iron ] as task T to extend the next observations of the NM.
The NM iterates with the environment using the extended observations as input.
At each iteration, functions L I and R respectively evaluate the progression of T and reward the NM until T is satisfied.
In our example, if the NM got wood , the second sequence and [ wood ] are discarded and the next task becomes [ grass ] .
This process is repeated until K is empty or the time limit is reached.
Note that the main focus of this work is to assess generalisation with abstract operators such as negation and disjunction.
Thus our experiments and results in the main document will focus on generalising atomic tasks T .
For generalisation of TTL formulae we refer to Appendix B.
4. D EEP RL A GENTS
We aim to demonstrate that DRL agents can generalise logic TTL operators from a limited variety of instructions and the major role that convolutional layers play in the ability of agents towards generalisation.
In this section we detail the agents used in our experiments.
Common features.
We use A2C, a synchronous version of the DRL algorithm A3C
(Mnih et al., 2016).
Agents are trained in procedurally generated maps, using a fixed perspective which is known to help agents with generalisation (Mao et al., 2019).
The deep learning architecture is composed by a CNN, followed by a fully-connected (FC) layer and an LSTM (LeCun et al., 2015).
When instructions are provided as text we use a bidirectional LSTM concatenate its output with the one from the CNN.
Agents only differ in the architecture of their CNN.
Further detail is given in Appendix A
Convolutional architectures.
In preliminary studies, we observed that agents with different FC and recurrent layer configurations can exhibit similar generalisation abilities.
However, as detailed in Sec. 5, different convolutional configurations yield significantly different results.
Particularly, while all the convolutional architectures we test achieve a similarly good performance with training instructions, only specific configurations correctly execute zero-shot tasks.
We find that the ability to generalise is correlated to the degree of alignment between the CNN architecture and the environment.
In our gridworld settings, we say that a CNN is weakly aligned (WA) with an environment when the kernel dimensions of the c.l. and stride size are factors of the setting’s tile resolution.
If a CNN is WA and the two first dimensions (length and width) of its visual embedding correspond to the number of input tiles in the original observation, we say that this CNN is strongly aligned (SA).
Note that the number of channels of the output, i.e., the third dimension of the embedding, does not influence the alignment.
Intuitively, the key feature is that a SA CNN is one that produces an independent output for each object in the environment.
Last, when a CNN is not WA or SA, we say it is not aligned (NA).
When aiming compositional learning from ∼ 6 training instructions, we find that only SA networks (when giving visual instructions) or SA and WA networks (when instructions are given in a separated channel) correctly execute zero-shot tasks.
E XPERIMENTS
We evaluate our framework in procedurally generated grid-worlds.
The first setting (Figure 1 left) is designed along the lines of a Minecraft-inspired (Minecraft for sort) benchmark widely used in
RL-TL (Andreas et al., 2017; Toro Icarte et al., 2018; De Giacomo et al., 2019), which we adapt to match the 2D maps from Hill et al.
(2020) (the last study about negation).
The action set consists of 4 individual actions: Up, Down, Left, Right , that move the agent one tile in the given direction.
In this setting each tile has a resolution of 9x9x1 values.
The DRL algorithm receives the observation extended with visual instructions depicting the target objects (Figure 1 center).
We also evaluate the agents in the popular MiniGrid benchmark (Chevalier-Boisvert et al., 2018), depicted in Figureright.
Here each tile has a resolution of 8x8x3 values, and the action set is Move forward, Turn left,
Turn right .
In this benchmark instructions are given through a separated language channel.
For each setting we test four different CNN configurations.
The first two are popular architectures from previous literature that we use as baselines: Atari-CNN (Mnih et al., 2015; 2016) which is NA with Minecraft and WA with MiniGrid, and ResNet (Espeholt et al., 2018; Hill et al., 2020) WA with
Minecraft and NA with MiniGrid.
We contrast these networks with different architectures designed to be SA with each environment and that we refer to as Env-CNN.
Last, for each environment we also test the performance of the same Env-CNNs, but when the c.l. are frozen during training, i.e., the c.l. always use a set of randomly selected parameters.
We refer to these networks as Env-CNN RF .
Schemes of the architectures are given in Appendix A.1.
In every experimental setting we refer to the global set of objects as X .
X is split into training ( X 1 ), validation ( X 2 ) and test sets ( X 3 )
where X = X 1 ∪ X 2 ∪ X 3 , X 1 ∩ X 2 = ∅ , X 1 ∩ X 3 = ∅ , and
X 2 ∩ X 3 = ∅ .
We contrast the results of learning with training sets of different size that we refer to as small ( |X 1 | = 6 ) and large ( |X 1 | = 20 ).
In Minecraft, where instructions are visual, agents are trained with objects from X 1 only.
In MiniGrid, where agents require to ground words to objects, we train the agents with positive instructions from X , while instructions concerning the negation or disjunction operators refer only to objects in X 1 .
Agents are trained with atomic tasks T .
Tasks are procedurally generated with objects from the corresponding set.
For instance, with the small setting we have six different objects, meaning that agents are learning the negation operator from six training instructions.
At each episode the agent navigates receiving rewards according to T .
The episode ends when the agent receives a positive reward (meaning that T is satisfied) or the time limit is reached.
See Appendix B for the reward function.
During training we periodically evaluate the performance of the agents with the corresponding validation set X 2 , that we use to select the best learning weights in each run, i.e., to perform early stopping when necessary.
Further details about object set generation, reward values, environment mechanics and training and validation plots are included in Appendix A. [Table]
better while in ’Deceptive’ lower is better.
Agents show a stronger systematic generalisation the bigger the difference between respective reliable and deceptive instructions.
Positive and negative labels refer to atomic positive or negative instruction respectively pointing to one of the two objects present.
In disjunction 2 nd choice the instructions refers to two objects, but only the second is present in the map.
Disjunction 2 choices points to two objects in a map with 4 items. Results are bolded where performance with reliable instructions is greater or equal than two times the respective performance with deceptive instructions. [Table] Results about systematic learning.
Table 1 presents the raw reward results in X 1 and X 3 with the different CNNs.
Note that “train" refers to results with objects from the training set but with the weights that achieve the best rewards in X 2 .
Therefore, all the networks can accumulate higher rewards in training if we let them overfit (See Figure 3 in appendix for training and validation plots).
Note that results in Table 1 alone do not yield a strong correlation between the alignment of the CNN and the generalisation ability.
Yet, as anticipated in Sec. 2 this correlation becomes evident when evaluating whether the agents are actually following the test instructions.
Table 2 shows the results in a particular set that we call binary choice maps (BCMs).
Here maps have only two objects (except in “disjunction 2 choices", with four), one granting a positive reward ( i = +1 )
the other a penalisation ( c = − 1 ).
In “disjunction 2 choices" two objects penalise, and two provide a reward.
Since agents are always trained with reliable instructions pointing to objects that grant a positive reward, an agent that is correctly generalising to zero-shot instructions should collect significantly higher rewards with reliable instructions than with deceptive ones (i.e., the later instructions are pointing to objects that grant a negative reward).
Roughly, agents should accumulate with reliable instructions two times greater rewards than when given deceptive instructions to show similar generalisation to the experiments in previous studies about negation with 100 training instructions (Hill et al., 2020).
The first strong pattern that we extract from Table 2 is that only architectures with some form of alignment (WA and SA) give evidence of systematic learning when no more than six training instructions are available.
NA architectures show minimal to none generalisation (e.g. better performance with deceptive instructions than with reliable) in all the small settings.
Withobjects, NA architectures also fail when instructions are visual (Minecraft) while struggling with the disjunction operator when instructions come in a dedicated channel (Minigrid).
In the latter setting, architectures with some alignment (WA) show good generalisation performance.
Yet, when instructions are visual they only learn to generalise positive and negative instructions, and exclusively in the larges setting.
Notably, architectures that are strongly aligned (SA) show good generalisation even when the weights of the c.l. are not trained (Env-CNN RF ), further evidencing the high influence of the architecture configuration in how the agent generalises.
Remarkably, the Env-CNN (the fully trained SA architecture) is the only configuration that systematically achieves a performance greater than 1 (the performance of a random agent) with reliable instructions while getting less than 1 with deceptive instructions across all the settings.
That is, SA architectures are the only configurations that have systematically learnt to generalise from the training instructions to guess not only the desired object/s, but also which undesired items should be avoided according to the test instruction .
Last, we highlight that in preliminary studies we also explored additional overfitting-prevention methods, including dropout and autoencoders, to help NA networks.
However, the results do not vary from the ones obtained by the train-validation-test set division that we carry here and that evidence the benefits of using architectures that are aligned with its environment.
We included the disjunction test where agents struggled the most, due to space constrains, additional results are included in Appendix C.
R ELATED WORK
Here we include the closest related literature to this research:
Systematic Generalisation.
Generalisation beyond training instructions has been widely studied in the RL literature.
For instance, Oh et al.
(2017) presents a framework that relies on hierarchical RL and task decomposition of instructions in English to enhance generalisation in DRL.
Later work from
Mao et al.
(2019) introduces a neuro-symbolic concept learner that jointly learns visual concepts, words, and semantic parsing of sentences from natural supervision.
Yu et al.
(2018) present a model that reports strong generalisation in a 2D environment that is aimed at language understanding for positive instructions.
Closer to our line of work, there is growing literature (see e.g., Smolensky (1988); Bahdanau et al.
; Lake (2019)) focused on finding the right drivers enabling systematic learning from natural language instructions, e.g., fixed perspectives and varied training data.
Recent work (Hill et al., 2020) suggests that systematic generalisation is not a binary question, but an emergent property of agents interacting with a situated environment (McClelland et al., 2010) and explores the drivers that enable agents generalise an abstract operator such as negation.
We expand the state of this line of works by first, providing evidence that systematic learning can emerge with logic operators from as few examples as with positive instructions, second, giving evidence that the
CNN architecture is a key driver –that previous research was oblivious about– towards generalisation.
Reinforcement Learning and Temporal Logic.
Training autonomous agents to solve multi-task goals expressed in temporal logic is drawing growing attention from the research community.
Examples include following instructions expressed as TL formulae by using logic progression and
RL (Littman et al., 2017; Andreas et al., 2017), tackling continuous-action environments (Yuan et al., 2019), multi-agent systems (León & Belardinelli, 2020) and studies on the applicability of different
TL languages (Camacho et al., 2019).
Complex goals expressed as temporal formulae can also be easily decomposed by using other techniques, including finite state automata (De Giacomo et al., 2019; Icarte et al., 2018) and progression (Toro Icarte et al., 2018).
More recent work has further exploited automata techniques, combining hierarchical approaches and reward machines – a particular form of automata– (Furelos-Blanco et al., 2021) or high-level planning with low-level RL solutions (Jothimurugan et al., 2021).
These works focus on solving non-Markovian models by extending the state space in a minimal form (to minimally impact the state space) so that the RL agent learns from an equivalent Markovian model, where an optimal behavior is also optimal in the original system (Bacchus et al., 1996).
A drawback of the mentioned literature is their exclusive focus on learning to solve compositions of training tasks.
For this reason Kuo et al.
(2020) targets generalisation of temporal logic instructions with DRL at the expense of requiring an additional neural network for each new symbol or object encountered.
Our work advances the state of the art by presenting a framework that can execute zero-shot TL formulae while relying on a single neural network.
D ISCUSSION AND CONCLUSIONS
We have empirically demonstrated that systematic learning can emerge with abstract operators including negation as early as with positive instructions.
An obvious limitation of our work is the use of regular gridworlds in our experiments.
However, this feature does does not lessen our contributions since previous closely-related work have struggled to provide evidence of generalisation with abstract operators in similar settings.
As we previously discussed in Sec. 1, the ability of deep learning agents for compositional reasoning is a recurrent topic in the literature.
Our results suggest that future research should pay close attention to the CNN architecture employed in their benchmarks before calling into question whether it is possible for agents to generalise.
Another limitation is that — in line with related work on generalisation (Lake & Baroni, 2018; Bahdanau et al., 2019; Hill et al., 2020) – we do not give nomological explanations for the generalisation ability (or lack thereof) that we evidence.
Yet, this is beyond the scope of our contribution, which is proving the emergence of generalisation and the key elements missing in previous works in similar settings.
As a foundational bridging work between the literature of formal methods applied to RL (FM-RL) and systematic generalisation, we use a language that is less expressive than the latest contributions in the field.
This constraint was needed to carry a careful experimental evaluation of the systematic learning with the different types of operators.
Given the positive results, more expressive languages can be used in future iterations.
Nevertheless, we presented the first framework that, leveraging on a learning-oriented syntax and the compositional structure of logical formulae, is capable of executing zero-shot complex formulae in temporal logic while relying on a single network. As anticipated in Sec. 6, this contrasts with dominant methods in the FM-RL literature that rely on a new policy network per task (e.g., De Giacomo et al. ; Icarte et al. ; Jothimurugan et al.
Furelos-Blanco et al.
(2021)) or a new network per word and symbol Kuo et al.
Our findings evidence that from as few as six different examples agents with aligned architectures learn not only how to follow training commands, but also abstract information about how symbols and operators in the given language compose and how the combination of those symbols influences what the agent should do.
This contrasts with Hill et al.
(2020) – the only previous study highlighting some generalisation with unseen negated instructions – which suggests that ∼ 100 instructions are required with negated instructions.
Using the same form of negation, we find that the CNN architecture plays a key role when learning from limited examples.
Still, we observe in our experiments that the ResNet used in Hill et al.
(2020) shows evidence of generalising negation with 20 instructions.
This also contrasts with Hill et al.
(2020), where the ResNet performed barely better than random when trained with 40 negated instructions.
We find that this difference comes from our use of a validation set, not common in RL literature and consequently not present in previous research.
We provide empirical evidence of this point in Appendix D.
This latest point also evidences the importance of adopting the traditional train-validation-test set split from supervised learning literature into RL research.
In the context of open-ended research, our results are also specially relevant as they highlight a key feature for the ability of agents to acquire new knowledge. Moreover, the good generalisation results with the randomnly-parameterized convolutional layers in both settings suggests that finding a suited architecture may be more beneficial than training a generic convolutional configuration when aiming open-ended agents. Consequently, when working in visually continuous settings – where the prior-knowledge about object resolution cannot be applied – our research indicates that methods for automatic generation of deep learning architectures are a potential way forward (Elsken et al., 2019; Song et al., 2020).
The impact of making an explicit train-validation-test splits when assessing the
RL agent ability is also of special importance for open-ended literature, where often environments are generated by learning agents Wang et al.
Campero et al.
Future works should assess performance in zero-shot environments in a similar fashion to the supervised learning literature
LeCun et al.
A E XPERIMENT DETAILS
Here we provide further details about the experimental setting and training procedure.
A. A RCHITECTURE CONFIGURATIONS
Figure 2 includes schemes of the neural network architectures from Section 5.
On the left, we see the general setting for Minecraft, where tasks instructions are embedded within the observation, and MiniGrid, where instructions are given in a separated channel.
Note that we intentionally use a significantly different number of channels in the last layers with Env-CNN minecraft and EnvCNN minigrid .
We do this to evidence that the number of channels in the visual embedding does not affect the ability of SA architectures to generalise training instructions.
In Sec 5 we observed that both SA architectures show good generalisation performance in their respective settings.
A. E NVIRONMENT
Map generation Training maps are procedurally-generated with a random number of objects that ranges from 2 to 8, there are no restrictions on the number of objects on a training map that can be the same as long as one of the objects solves the current task.
As explained in the main text, the action set consists of four individual actions in Minecraft: Up, Down, Left, Right , that move the agent one slot from its current position according to the given direction; and three actions in MiniGrid:
Forward,
Turn Left, Turn Right .
If the agent moves against a wall, its position remains the same.
Note that moving against walls is discouraged by the internal reward function that provides a negative reward at each timestep if the agent has not solved the current task.
Instruction generation.
As detailed in the main text, instructions are procedurally generated with objects from the corresponding set.
For instance, with the small setting we have 6 different objects, meaning that agents are learning the negation operator from 6 training instructions.
Consequently, the total number of training instructions in the small setting is 42 (6 positive, 6 negative anddisjunctions –we do not instruct disjunctions with the same object twice–).
Object-set generation.
In the Minecraft-inspired environment, we procedurally generate objects as a random matrix of 9x9 values.
Small and large training sets use 6 to 20 procedural objects as described in the main text.
For validation and test sets we use 6 objects respectively.
In MiniGrid, objects are not procedural values but compositions of colors, e.g., red, and shapes, e.g., key.
Here the small set uses 2 shapes and 3 colors while training with negative instructions and disjunction.
In validation we use an additional shape combined with training colors.
In test we use 2 new colors, the validation shape and a new test shape.
In the large setting, training has 4 shapes and 5 colors, an additional shape with training colors for validation, and 2 new colors with a a test shape and the validation shape in test.
Rewards.
In both settings the agents received a penalisation of -1 when reaching an object that does not satisfy the task, and +1 when the opposite occurs.
We explored different penalisation values per time-step to stop the agents from local-optima policies, e.g., not reaching any object, while still making beneficial avoiding undesired objects when possible.
We did a grid-search in the range [ − 0 .
01 , − 0 .
2] and achieved the best results with − 0 .
1 in Minecraft and − 0 .
05 in MiniGrid.
A. T RAINING HYPERPARAMETERS AND PLOTS
The A2C algorithm uses a learning rate of 8 e − 5 , this value for the learning rate is the one that give us the best results when doing grid search between 1 e − 5 and 1 e − 3 , except for the ResNet that shows better results with the value of 2 e − 5 that we use in the experiments.
The entropy and value loss regularization terms are 1 e − 3 (best results we found in the range [ 1 e − 4 ,0] and 0 .
5 (as in Mnih et al.
(2016)) respectively.
The expected-reward discount factor is 0 .
99 , which is typical value for this term.
Batch size is set to 512 values, which gives the best results in the range of ( [128 , 2048] ).
Figure 3 shows the training and validation plots of the runs evaluated in Sec. 5.
For each run we selected the weights that achieve the peak performance in the validation set.
A. H ARDWARE
The various architectures were trained on computing clusters with GPUs such as Nvidia Tesla
K80, Testla T4 or TITAN Xp.
We employed Intel Xeon CPUs and consumed 7 GB of RAM per 5 independent runs working in parallel.
Each experiment typically takes 2 days to train on this hardware.
All the results presented in the main text and supplementary material are obtained from the aforementioned five independent runs per experiment.
B G ENERALISING TTL FORMULAE
In this section we include the pseudo-code of the of the symbolic module, whose procedures are explained in Sec. 4.1 of the main text and provide experimental results when agents generalise complex, i.e., multi-task, instructions.
Algorithm 1 details the Symbolic Module (SM), which given a TTL instruction ϕ , decomposes the instruction into atomic tasks T to be solved by the Neural Module (NM).
Algorithm 2 details the extractor E , an internal function of the SM that given the instruction formula generates a set K of sequences of tasks that satisfy ϕ .
Algorithm 3 refers to the progression function P , which updates K according to the true evaluation p given by the internal labelling function L I that indicates to the SM how the previous task was solved.
B. G ENERALISING COMPLEX TTL FORMULAE
We use five complex instructions test the agents (see Table 3). The first two are the hardest specifications from Andreas et al.
; Toro Icarte et al.
The three additional instructions were defined by aiming for combinations of negated tasks and inclusive/exclusive non-deterministic choices with non-atomic components together with sequential instances.
We recall that operator ∩ can be defined in terms of sequential composition ; and non-deterministic choice ∪ .
Generalisation with TTL formulae. We evaluate the performance of the same agents trained with atomic tasks T when executing complex (multi-task) instructions given as TTL formulae ϕ that refer to zero-shot atomic tasks.
Table 4 shows the average reward and number of actions needed inmaps executing the complex instructions.
In line with the results with TTL tasks from Table 1, we see that a stronger alignment yields better the final performance (i.e., SA > WA > NA accumulated reward).
do 4:
5: for all Seq K do 6:
Seq .append
( T ) 7: end for 8: else 9:
Save the first element: FC ← T [
Save the last element: SC ← T [ − 1] 11:
Nseqs ← length ( K 12: for all Seq K do 13:
Generate a clone:
Seq ′ ← Seq 14:
K .append
( Seq ′ ) 15: end for 16: for i in range ( length ( K )) do 17: if i < Nseqs then 18:
K [ i ] .append ( FC ) 19: else 20: i
( SC ) 21: end if 22: end for 23: end if 24: end for 25: Output
C D ISJUNCTION RESULTS
This section includes the results with disjunction anticipated in Sec. [-5: ϕ 
 =-]5.
Table 5 shows the results of the different architectures when following reliable, i.e., pointing to the right object, and deceptive, i.e., pointing to the wrong object, zero-shot instructions with a non-deterministic choice. Instructions labelled “Disjunction 1 st choice" refer to those where only the first object within the instruction is present in the map.
“Disjunction 2 nd choice" is a similar setting with the second object. In “Disjunction 2 choices" both objects are present in the map.
As detailed in the main text, agents learning compositionally should be able to follow the zero-shot formulae and consequently do well with reliable instructions while poorly with deceptive ones.
1: Function P , Input: K , p 2: if p ̸ = ∅ then 3: for all Seq ∈ K do 4: if p fulfills Seq [0] then 5:
Seq .pop
( Seq [0] ) 6: else 7:
K .pop
( Seq ) 8: end if 9: end for 10: end if 11: T ← ∅ 12: if K is not ∅ then 13:
Select the head T of a sequence in K : 14: if T ≡ α then 15: //Aiming to form a non-deterministic choice 16: for all Seq ∈ K do 17: if T ==
Seq [0] then 18: continue 19: else if Seq [0] ≡ α then 20:
T ← T ∨ Seq [0] 21: return T 22: end if 23: end for 24: end if 25: end if 26: Output: T
[Table]
D T HE STRUGGLE WITH NEGATED INSTRUCTIONS
This section section aims to bring light to an apparent contradiction between our empirical results and those from the state-of-the-art about generalisation with negation Hill et al.
In Sec. 5 in the main text, we observe that the architecture of the CNN plays a key role in the ability of agents to generalise when the number of instructions is limited.
Still, in Table 2 we see that
the
ResNet shows good generalisation in both settings when learning from 20 negated instructions.
This seems to contradict the results from Hill et al.
(2020) where the same ResNet struggles to generalise with 40 negated instructions.
Nevertheless, we find that this is caused by the particular form that neural networks overfit to the negation operator.
[Table] [Table] In particular, consider the TTL task copper ∼ , which intuitively means “get something different from copper".
When agents overfit they no longer interpret the negation operator as a separated element [Table] but rather as if the operator was part of the object’s name.
Consequently, for an overfitting agent, copper ∼ is a single word that represents all the objects of the training set except copper.
This overfitting phenomena produces policies that, when facing non-training objects, are no longer able to interpret negation correctly, leading to a behavior different from what an agent learning systematically is expected to do.
We find that this phenomena, first described in Hill et al.
(2020), happens only after agents overfit to the training set.
This means that through a validation set, we can perform early stopping – as we do in Sec. 5 – and use policies that correctly generalise to new instructions without requiring a larger number of training instructions.
Table 6 presents the results of an ablation study about the use of a validation set with a ResNet agent trained until convergence in the Minecraft large training set.
We see that there is little difference with positive instructions, but results are completely opposite in the case of negated tasks, that go from an agent correctly generalising negation – when applying a validation set and early stopping – to one that does the opposite from what the specification requires – when no validation set is used –.
Consequently, the struggle with negated instructions in previous research seems to be motivated by the different ways deep learning overfits when learning different abstract operators .
E R EQUIREMENTS FOR L OGICAL O PERATORS
Below we include additional details about the requirements we faced when aiming a neuro-symbolic agent that generalises the negation and non-deterministic choice operators. [Table]
E. N EGATION
When learning positive and negative instances concurrently using visual instructions, we noticed that the position of the negation operator impacted on the agent’s behavior in training.
Specifically, placing the negation symbol after the negated object helps the agent to learn both types of instances concurrently, while, when placed otherwise, the agent focus only on the positive tasks first, which can lead to saddle points.
We believe that this result is not related to linguistic processing.
Note that all of our architectures in
Minecraft process the specification as a whole, as part of the observational input processed by the convolutional layer.
Thus, the order is not important from the point of view of temporal processing.
We believe the difference was caused because affirmative formulas are represented with one symbol, e.g., "get wood" is represented as "wood"; while negation is represented by two symbols, e.g., "get an object different from wood" as "wood ∼ ".
By placing the negation operator after the object we force the neural network to "observe" the first two inputs to differentiate affirmative tasks from negative ones.
An alternative approach that could have a similar effect is adding an explicitly positive operator so that both negative and positive formulas have the same length
. E. N ON - DETERMINISTIC C HOICE
Training the agent to effectively learn non-deterministic choice ∪ required episodes where only one of the two disjuncts of the non-deterministic choice was present, as well as some other episodes with the two objects present simultaneously.
Intuitively, if we have the instruction "get axe or get gold", there should be training episodes where gold is present in the map, but there are no axes, and similarly, other maps where there are axes, but there is not gold.
This prevents the agent from biasing with the object that appears either in the first or second position of the instruction.
Additionally, the agent should also interact with episodes where both objects are present.
Otherwise, the policy network of the agent will allocate a similarly high probability to go after the two objects, instead of to the one that is closer or "safer", i.e, far from non-desired objects.
F T RANSLATING TTL INTO LTL f
It can be easily inferred that our task temporal logic (TTL) is a fragment of the more popular lineartime temporal logic over finite traces (LTL f ) Baier & Katoen (2008).
The syntax of LTL f is defined as follows:
[Equation]
Note that both TTL and LTL f are defined over finite traces.
Below we provide truth-preserving translations of TTL into LTL f .
Definition 4.
Translation τ from TTL to LTL f is defined as follows:
τ ( α ) =
⋄ p
α , where p α is the fulfilment condition associated with task
α τ ( α ∼ ) =
⋄ ( (cid:87) β ̸
= α p β ∧ ¬ p α ) τ ( α ∨ α ′ )
=
τ ( α ) ∨ τ ( α ′ )
[Equation] =  
[Equation] if T = α [Equation]
[Equation]
̸
[Equation]
if T
[Equation]
⋄ (
[Equation]
( T
[Equation]
if T = α ∨ α ′ [Equation] [Equation] ( T
[Equation]
T ′ ) [Equation]
if T = T [Equation] T
[Equation]
(( T [Equation] T [Equation]
[Equation]
T ′ ))
if T = T 1 ∪ T
We immediately prove that translation τ preserve the interpretation of formulae in TTL.
Proposition 1.
Given a model N and trace λ , for every formula T in TTL,
[Equation]
In the main body of this work we stated that TTL is a fragment of LTL f , and supported such claim by explaining that translations τ 1 and τ 2 preserve the interpretation of formulae in TTL.
We include now a detailed proof for this claim:
Proposition 2.
Given a model N and trace λ , for every formula T in TTL,
[Equation]
Proof.
The proof is by induction on the structure of formula T .
The base case follows immediately by the semantics of TTL and LTL f .
As for the induction step, the case of interest is for formulae of type T ; T ′ .
In particular, consider the case for T =
α .
We then have that ( N , λ ) | =
α ; T ′ iff for some 0 ≤ j < | λ | , ( N , λ [0 , j ]) |
= α and ( N , λ
[ j + 1 , | λ ]) | =
T ′ .
By induction hypothesis, this is equivalent to ( N , λ [0 , j ]) | =
⋄ p α and ( N , λ
[ j + 1 , | λ ])
|
=
τ ( T ′ ) .
Finally, this is equivalent to ( N , λ ) | =
⋄ ( p α ∧ τ ( T ′ )) .
The other cases.
are dealt with similarly.
Finally, the case for T ∪ T ′ follows by induction hypothesis and the distributivity of ⋄ over ∨ .