M ODULAR C ONTINUAL L EARNING IN A U NIFIED V ISUAL E NVIRONMENT
A BSTRACT
A core aspect of human intelligence is the ability to learn new tasks quickly and switch between them ﬂexibly.
Here, we describe a modular continual reinforcement learning paradigm inspired by these abilities.
We ﬁrst introduce a visual interaction environment that allows many types of tasks to be uniﬁed in a single framework.
We then describe a reward map prediction scheme that learns new tasks robustly in the very large state and action spaces required by such an environment.
We investigate how properties of module architecture inﬂuence efﬁciency of task learning, showing that a module motif incorporating speciﬁc design principles (e g. early bottlenecks, low-order polynomial nonlinearities, and symmetry) signiﬁcantly outperforms more standard neural network motifs, needing fewer training examples and fewer neurons to achieve high levels of performance.
Finally, we present a meta-controller architecture for task switching based on a dynamic neural voting scheme which allows new modules to use information learned from previouslyseen tasks to substantially improve their own learning efﬁciency
I NTRODUCTION
In the course of everyday functioning, people are constantly faced with real-world environments in which they are required to shift unpredictably between multiple, sometimes unfamiliar, tasks (Botvinick & Cohen, 2014).
They are nonetheless able to ﬂexibly adapt existing decision schemas or build new ones in response to these challenges (Arbib, 1992).
How humans support such ﬂexible learning and task switching is largely unknown, both neuroscientiﬁcally and algorithmically (Wagner et al., 1998; Cole et al., 2013).
We investigate solving this problem with a neural module approach in which simple, task-specialized decision modules are dynamically allocated on top of a largely-ﬁxed underlying sensory system (Andreas et al., 2015; Hu et al., 2017).
The sensory system computes a general-purpose visual representation from which the decision modules read.
While this sensory backbone can be large, complex, and learned comparatively slowly with signiﬁcant amounts of training data, the task modules that deploy information from the base representation must, in contrast, be lightweight, quick to be learned, and easy to switch between.
In the case of visually-driven tasks, results from neuroscience and computer vision suggest the role of the ﬁxed general purpose visual representation may be played by the ventral visual stream, modeled as a deep convolutional neural network (Yamins & DiCarlo, 2016;
Razavian et al., 2014).
However, the algorithmic basis for how to efﬁciently learn and dynamically deploy visual decision modules remains far from obvious.
[Figure] In standard supervised learning it is often assumed that the output space of a problem is prespeciﬁed in a manner that just happens to ﬁt the task at hand – e g. for a classiﬁcation task, a discrete output with a ﬁxed number of classes might be determined ahead of time, while for a continuous estimation problem, a one-dimensional real-valued target might be chosen instead.
This is a very convenient simpliﬁcation in supervised learning or single-task reinforcement learning contexts, but if one is interested in the learning and deployment of decision structures in a rich environment deﬁning tasks with many different natural output types, this simpliﬁcation becomes cumbersome.
To go beyond this limitation, we build a uniﬁed environment in which many different tasks are naturally embodied.
Speciﬁcally, we model an agent interacting with a two-dimensional touchscreenlike GUI that we call the TouchStream , in which all tasks (discrete categorization tasks, continuous estimation problems, and many other combinations and variants thereof) can be encoded using a single common and intuitive – albeit large – output space.
This choice frees us from having to hand-design or programmatically choose between different output domain spaces, but forces us to confront the core challenge of how a naive agent can quickly and emergently learn the implicit “interfaces” required to solve different tasks.
We then introduce Reward Map Prediction (ReMaP) networks, an algorithm for continual reinforcement learning that is able to discover implicit task-speciﬁc interfaces in large action spaces like those of the TouchStream environment.
We address two major algorithmic challenges associated with learning ReMaP modules.
First, what module architectural motifs allow for efﬁcient task interface learning?
We compare several candidate architectures and show that those incorporating certain intuitive design principles (e.g. early visual bottlenecks, low-order polynomial nonlinearities and symmetry-inducing concatenations) signiﬁcantly outperform more standard neural network motifs, needing fewer training examples and fewer neurons to achieve high levels of performance.
Second, what system architectures are effective for switching between tasks?
We present a meta-controller architecture based on a dynamic neural voting scheme, allowing new modules to use information learned from previously-seen tasks to substantially improve their own learning efﬁciency.
In § 1 we formalize the TouchStream environment.
In § 2, we introduce the ReMaP algorithm.
In § 3, we describe and evaluate comparative performance of multiple ReMaP module architectures on a variety of TouchStream tasks.
In § 4, we describe the Dynamic Neural Voting meta-controller, and evaluate its ability to efﬁciently transfer knowledge between ReMaP modules on task switches. [Figure]
R ELATED W ORK
Modern deep convolutional neural networks have had signiﬁcant impact on computer vision and artiﬁcial intelligence (Krizhevsky et al., 2012), as well as in the computational neuroscience of vision (Yamins & DiCarlo (2016)).
There is a recent but growing literature on convnet-based neural modules , where they have been used for solving compositional visual reasoning tasks (Andreas et al., 2015;
Hu et al., 2017).
In this work we apply the idea of modules to solving visual learning challenges in a continual learning context.
Existing works rely on choosing between a menu of pre-speciﬁed module primitives, using different module types to solve subproblems involving speciﬁc input-output datatypes, without addressing how these modules’ forms are to be discovered in the ﬁrst place.
In this paper, we show a single generic module architecture is capable of automatically learning to solve a wide variety of different tasks in a uniﬁed action/state space, and a simple controller scheme is able to switch between such modules.
Our results are also closely connected with the literature on lifelong (or continual) learning (Kirkpatrick et al., 2016; Rusu et al., 2016).
A part of this literature is concerned with learning to solve new tasks without catastrophically forgetting how to solve old ones (Zenke et al., 2017; Kirkpatrick et al., 2016).
The use of modules obviates this problem, but instead shifts the hard question to one of how newly-allocated modules can be learned effectively.
The continual learning literature also directly addresses knowlege transfer to newly allocated structures (Chen et al., 2015; Rusu et al., 2016;
Fernando et al., 2017), but largely addresses how transfer learning can lead to higher performance, rather than addressing how it can improve learning speed.
Aside from reward performance, we focus on issues of speed in learning and task switching, motivated by the remarkably efﬁcient adaptability of humans in new task contexts.
Existing work in continual learning also largely does not address which speciﬁc architecture types learn tasks efﬁciently, independent of transfer.
By focusing ﬁrst on identifying architectures that achieve high performance quickly on individual tasks (§ 3), our transfer-learning investigation then naturally focuses more on how to efﬁciently identify when and how to re-use components of these architectures (§ 4).
Most of these works also make explicit a priori assumptions about the structure of the tasks to be encoded into the models (e.g. output type, number of classes), rather than address the more general question of emergence of solutions in an embodied case, as we do.
Meta-reinforcement learning approaches such as Wang et al.
; Duan et al.
(2016), as well as the schema learning ideas of e.g. Arbib (1992); McClelland (2013)
typically seek to address the issue of continual learning by having a complex meta-learner extract correlations between tasks over a long timescale.
In our context most of the burden of environment learning is placed on the individual modules, so our meta-controller can thus be comparatively light-weight compared to typical meta-reinforcement approaches.
Unlike our case, meta-learning has mostly been limited to small state or action spaces.
Some recent work in general reinforcement learning (e.g. Ostrovski et al.
Dulac-Arnold et al.
(2015)) has addressed the issue of large action spaces, but has not sought to address multitask transfer learning in these large action spaces.
T HE T OUCH S TREAM E NVIRONMENT
Agents in a real-world environment are exposed to many different implicit tasks, arising without predeﬁned decision structures, and must learn on the ﬂy what the appropriate decision interfaces are for each situation.
Because we are interested in modeling how agents can do this on-the-ﬂy learning, our task environment should mimic the unconstrained nature of the real world.
Here, we describe the
TouchStream environment, which attempts to do this in a simpliﬁed two-dimensional domain.
Our problem setup consists of two components, an “environment” and an “agent,” interacting over an extended temporal sequence (Fig. 1).
At each timestep t , the environment emits an RGB image x t of height H and width W , and a scalar reward r t .
Conversely, the agent accepts images and rewards as input and chooses an action a t in response.
The action space A available to the agent consists of a two-dimensional pixel grid { 0 , . . .
, H − 1 } × { 0 , . . .
, W − 1 } ⊂ Z 2 , of the same height and width as its input image.
The environment is equipped with a policy (unknown to the agent) that on each time step computes image x t and reward r t as a function of the history of agent actions { a 0 , . . . , a t − 1 } , images { x 0 , . . . , x t − 1 } and rewards { r 0 , . . .
, r t − 1 } .
In this work, the agent is a neural network, composed of a visual backbone with ﬁxed weights, together with a meta-controller module whose parameters are learned by interaction with the environment.
The agent’s goal is to learn to enact a policy that maximizes its reward obtained over time.
Unlike an episodic reinforcement learning context, the TouchStream environment is continuous: throughout the course of learning the agent is never signaled when it should reset to some “initial” internal state.
However, unlike the traditional continuous learning context of e.g. Sutton & Barto (1998), a
TouchStream may implicitly deﬁne many different tasks, each of which is associated with its own characteristic reward schedule.
The agent experiences a continual stream of tasks, and any implicit association between reward schedule and state reset must be discovered by the agent.
By framing the action space A of the agent as all possible pixel locations and the state space as any arbitrary image, a very wide range of possible tasks are uniﬁed in this single framework, at the cost of requiring the agents’ action space to be congruent to its input state space, and thus be quite large.
This presents two core efﬁciency challenges for the agent: on any given task, it must be able to both quickly recognize what the “interface” for the task is, and transfer such knowledge across tasks in a smart way.
Both of these goals are complicated by the fact that both the large size of agent’s state and action spaces.
Although we work with modern large-scale computer vision-style datasets and tasks in this work, e.g. ImageNet (Deng et al.
and MS-COCO (Lin et al.
, we are also inspired by visual psychology and neuroscience, which have pioneered techniques for how controlled visual tasks can be embodied in real reinforcement learning paradigms (Horner et al., 2013; Rajalingham et al., 2015).
Especially useful are three classes of task paradigms that span a range of the ways discrete and continuous estimation tasks can be formulated – including Stimulus-Response, Match-To-Sample, and Localization tasks (Fig. 2).
Stimulus-Response Tasks:
The Stimulus-Response (SR) paradigm is a common approach to physically embodying discrete categorization tasks (Gaffan & Harrison, 1988).
For example, in the simple two-way SR discrimination task shown in Fig.
2a, the agent is rewarded if it touches the left half of the screen after being shown an image of a dog, and the right half after being shown a butterﬂy.
SR tasks can be made more difﬁcult by increasing the number of image classes or the complexity of the reward boundary regions.
In our SR experiments, we use images and classes from the ImageNet dataset (Deng et al., 2009).
Match-To-Sample Tasks:
The Match-to-Sample (MTS) paradigm is another common approach to assessing visual categorization abilities (Murray & Mishkin, 1998).
In the MTS task shown in Fig.
2b, trials consist of a sequence of two image frames – the “sample” screen followed by the “match” screen – in which the agent is expected to remember the object category seen on the sample frame, and then select an onscreen “button” (really, a patch of pixels) on the match screen corresponding to the sample screen category.
Unlike SR tasks, MTS tasks require some working memory and more localized spatial control.
More complex MTS tasks involve more sophisticated relationships between the sample and match screen.
In Fig.
2c, using the MS-COCO object detection challenge dataset (Lin et al., 2014), the sample screen shows an isolated template image indicating one of the 80 MS-COCO classes, while the match screen shows a randomly-drawn scene from the dataset containing at least one instance of the sample-image class.
The agent is rewarded if its chosen action is located inside the boundary of an instance (e.g. the agent “pokes inside”) of the correct class.
This MS-COCO
MTS task is a “hybrid” of categorical and continuous elements, meaning that if phrased as a standard supervised learning problem, both categorical readout (i.e. class identity) and a continous readout (i.e. object location) would be required.
Localization: Fig.
2d shows a two-step continuous localization task in which the agent is supposed to mark out the bounding box of an object by touching opposite corners on two successive timesteps, with reward proportionate to the Intersection over Union (IoU) value of the predicted bounding box relative to the ground truth bounding box IoU =
Area ( B GT ∩ ˆ B ) Area ( B GT ∪ ˆ B ) .
In localization, unlike the SR and
MTS paradigms, the choice made at one timestep constrains the agent’s optimal choice on a future timestep (e.g. picking the upper left corner of the bounding box on the ﬁrst step contrains the lower right opposite corner to be chosen on the second).
Although these tasks can become arbitrarily complex along certain axes, the tasks presented here require only ﬁxed-length memory and future prediction.
That is, each task requires only knowledge of the past k b timesteps, and a perfect solution always exists within k f timesteps from any point.
The minimal required values of k b and k f are different across the various tasks in this work.
However, in the investigations below, we set these to the maximum required values across tasks, i.e. kb = 1 and kf = 2.
Thus, the agent is required to learn for itself when it is safe to ignore information from the past and when it is irrelevant to predict past a certain point in the future.
We will begin by considering a restricted case where the environment runs one semantic task indefinitely, showing how different architectures learn to solve such individual tasks with dramatically different levels of efﬁciency (§ 2-3).
We will then expand to considering the case where the environment’s policy consists of a sequence of tasks with unpredictable transitions between tasks, and exhibit a meta-controller that can cope effectively with this expanded domain (§ 4).
R EWARD M AP P REDICTION
The TouchStream environment necessarily involves working with large action and state spaces.
Methods for handling this situation often focus on reducing the effective size of action/state spaces, either via estimating pseudo-counts of state-action pairs, or by clustering actions (Ostrovski et al., 2017; Dulac-Arnold et al., 2015).
Here we take another approach, using a neural network to directly approximate the (image-state modulated) mapping between the action space and reward space, allowing learnable regularities in the state-action interaction to implicitly reduce the large spaces into something manageable by simple choice policies.
We introduce an off-policy algorithm for efﬁcient multitask reinforcement learning in large action and state spaces: Reward Map Prediction, or ReMaP.
2. R E M A P N ETWORK A LGORITHM
As with any standard reinforcement learning situation, the agent seeks to learn an optimal policy π = p ( a t | x t ) deﬁning the probability density p over actions given image state x t .
The ReMaP algorithm is off-policy, in that π is calculated as a simple ﬁxed function of the estimated reward.
A ReMaP network M Θ is a neural network with parameters Θ , whose inputs are a history over previous timesteps of (i) the agent’s own actions, and (ii) an activation encoding of the agent’s state space; and which explicitly approximates the expected reward map across its action space for some number of future timesteps.
Mathematically:
[Equation]
where k b is the number of previous timesteps considered; k f is the length of future horizon to be considered; Ψ t − k b : t is the history [ ψ ( x t − k b ) , . . .
, ψ ( x t )] of state space encodings produced by ﬁxed backbone network ψ ( · ) , h t − k b : t − 1 is the history [ a t − k b . . .
, a t − 1 ] of previously chosen actions, and each m i ∈ map ( A , R ) – that is, a map from action space to reward space.
The predicted reward maps are constructed by computing the expected reward obtained for a subsample of actions drawn randomly from A :
[Equation]
(1) where r t + j is the predicted reward j steps into the future horizon.
Having produced k f reward prediction maps, one for each timestep of its future horizon, the agent needs to determine what it believes will be the single best action over all the expected reward maps (cid:104) m 1 t , m 2 t , . . .
, m k f t [Equation]
The ReMaP algorithm formulates doing so by normalizing the predictions across each of these k f maps into separate probability distributions, and sampling an action from the distribution which has maximum variance.
That is, the agent computes its policy π as follows:
[Equation]
where
[Equation]
is a normalization that removes the minimum of the map,
[Equation]
ensures it is a probability distribution parameterized by functional family f ( · ) , and VarArgmax is an operator which chooses the input with largest variance.
The sampling procedure described in equation (2) uses two complementary ideas to exploit spatial and temporal structure to efﬁciently explore a large action space.
Since rewards in real physical tasks are spatially correlated, the distribution-based sampler in Equation (4) allows for more effective exploration of potentially informative actions than would the single-point estimate of an apparent optimum (e.g. an (cid:15) -greedy policy).
Further, in order to reduce uncertainty, the ReMaP algorithm explores timesteps with greatest reward map variance.
The VarArgmax function nonlinearly upweights the timeframe with highest variance to exploit the fact that some points in time carry disproportianate relevance for reward outcome, somewhat analagously to how max-pooling operates in convolutional networks.
Although any standard action selection strategy can be used in place of the one in (2) (e.g. pseudo (cid:15) -greedy over all k f maps), we have empirically found that this policy is effective at efﬁciently exploring our large action space.
The parameters Θ of a ReMaP network are learned by gradient descent on the loss of the reward prediction error Θ ∗ =
argmin Θ L [ m t ( a t ) , r t , ; Θ] with map m jt compared to the true reward r t + j .
Only the reward prediction in m t corresponding to the action chosen at timestep t participates in loss calculation and backpropagation of error signals.
A minibatch of maps, rewards, and actions is collected over several consecutive inference passes before performing a parameter update.
The ReMaP algorithm is summarized in 1.
Algorithm 1: ReMaP – Reward Map
Prediction
Initialize ReMaP network M
Initialize state and action memory buffers Ψ t − k b : t and h t − k b : t − 1 for timestep t = 1,T do
Observe x t , encode with state space network ψ ( · ) , and append to state buffer Subsample set of potential action choices a t uniformly from A Produce k f expected reward maps of a t from eq.
Select action according to policy π as in (2) Execute action a t in environment, store in action buffer, and receive reward r t Calculate loss for this and previous k f − 1 timesteps if t ≡ 0 mod batch size then Perform parameter update
Throughout this work, we take our ﬁxed backbone state space encoder to be the VGG-16 convnet, pretrained on ImageNet (Simonyan & Zisserman, 2014).
Because the resolution of the input to this network is 224x224 pixels, our action space
A = { 0 , . . .
, 223 } × { 0 , . . .
By default, the functional family f used in the action selection scheme in Eq.
(4) is the identity, although on tasks beneﬁting from high action precision (e.g. Localization or MS-COCO MTS), it is often optimal to sample a low-temperature Boltzmann distribution with f ( x ) =
e − x/T .
Reward prediction errors are calculated using the cross-entropy loss (where logits are smooth approximations to the Heaviside function in analogy to eq.
E FFICIENT N EURAL M ODULES FOR T ASK L EARNING
The main question we seek to address in this section is: what speciﬁc neural network structure(s) should be used in ReMaP modules?
The key considerations are that such modules (i) should be easy to learn, requiring comparatively few training examples to discover optimal parameters Θ ∗ , and (ii) easy to learn from, meaning that an agent can quickly build a new module by reusing components of old ones.
Intuitive Example: As an intuition-building example, consider the case of a simple binary StimulusResponse task, as in Fig.
2a (“if you see a dog touch on the right, if a butterﬂy touch on the left").
One decision module that is a “perfect” reward predictor on this task is expressed analytically as:
[Equation]
where H is the Heaviside function, a x and a y are the x and y components of the action a ∈ A relative to the center of the screen, and W is a 1 × | Ψ t | matrix expressing the class boundary (bias term omitted for clarity).
If W Ψ t is positive (i.e. the image is of a dog) then a x must also be positive (i.e. touch is on the right) to predict positive reward; conversly, if W Ψ t is negative (i.e. butterﬂy), a x must be negative (i.e. left touch) to predict reward.
If neither of these conditions hold, both terms are equal to zero, so the formula predicts no reward.
Since vertical location of the action does not affect reward, a y is not involved in reward calculation on this task.
Equation (5) has three basic ideas embedded in its structure:
• there is an early visual bottleneck , in which the high-dimensional general purpose feature representation Ψ t is greatly reduced in dimension (in this case, from the 4096 features of VGG’s FC layer, to 1) prior to combination with action space, • there is a multiplicative interaction between the action vector and (bottlenecked) visual features, and • there is symmetry , e.g. the ﬁrst term of the formula is the sign-antisymmetric partner of the second term, reﬂecting something about the spatial structure of the task.
In the next sections, we show these three principles can be generalized into a parameterized family of networks from which the visual bottleneck (the W parameters), and decision structure (the form of equation (5)) can emerge naturally and efﬁcienty via learning for any given task of interest.
3. T HE EMS M ODULE
In this section we deﬁne a generic ReMaP module which is lightweight, encodes all three generic design principles from the “perfect” formula, and uses only a small number of learnable parameters.
Deﬁne the concatenated square nonlinearity as
[Equation]
and the concatenated ReLU nonlinearity (Shang et al.
as
[Equation]
where ⊕ denotes vector concatenation.
The CReS nonlinearity is then deﬁned as the composition of
CReLU and Sq , e.g.
[Equation]
The CReS nonlinearity introduces multiplicative interactions between its arguments via its Sq component and symmetry via its use of CReLU .
Deﬁnition.
The ( n 0 , n 1 , . .
. , n k ) - Early Bottleneck-Multiplicative-Symmetric (EMS) module is the
ReMaP module given by
[Equation]
where W i and b i are learnable parameters, Ψ are features from the ﬁxed visual encoding network, and a is the action vector in A .
[Figure]
The EMS structure builds in each of the three principles described above.
The B stage represents the early bottleneck in which visual encoding inputs are bottlenecked to size n 0 before being combined with actions, and then performs k CReS stages, introducing multiplicative symmetric interactions between visual features and actions.
From this, the “perfect” module deﬁnition for the binary SR task in eq.
(5) then becomes a special case of a two-layer EMS module.
Note that the visual features to be bottlenecked can be from any encoder; in practice, we work with both fully connected and convolutional features of the VGG-16 backbone.
In the experiments that follow, we compare the EMS module to a wide variety of alternative control motifs, in which the early bottleneck, multiplicative, and symmetric features are ablated.
Multiplicative nonlinearity and bottleneck ablations use a spectrum of more standard activation functions, including ReLU , tanh , sigmoid, elu (Clevert et al., 2015), and CReLU forms.
In late bottleneck (fully-ablated) architectures – which are, effectively, “standard” multi-layer perceptrons (MLPs) – action vectors are concatenated directly to the output of the visual encoder before being passed through subsequent stages.
In all, we test 24 distinct architectures.
Detailed information on each can be found in the Supplement.
3. E XPERIMENTS
We compared each architecture across 12 variants of visual SR, MTS, and localization tasks, using ﬁxed visual encoding features from layer FC6 of VGG-16.
Task variants ranged in complexity from simple (e.g. a binary SR task with ImageNet categories) to more challenging (e.g. a many-way
ImageNet MTS task with result buttons appearing in varying positions on each trial).
The most complex tasks are two variants of localization, either with a single main salient object placed on a complex background (similar to images used in Yamins & DiCarlo (2016)), or complex scenes from MS-COCO (see Fig.
3b).
Details of the tasks used in these experiments can be found in the
Supplement.
Module weights were initialized using a normal distribution with µ = 0 . 0 , σ = 0 .
01 , and optimized using the ADAM algorithm (Kingma & Ba (2014)) with parameters β 1 = 0 .
9 , β 2 = 0 .and
(cid:15) =
1 e − 8 .
Learning rates were optimized on a per-task, per-architecture basis in a cross-validated fashion.
For each architecture and task, we ran optimizations from ﬁve different initialization seeds to obtain mean and standard error due to initial condition variability.
For fully-ablated “late-bottleneck” modules, we measured the performance of modules of three different sizes (small, medium, and large), where the smallest version is equivalent in size to the EMS module, and the medium and large versions are much larger (Table S1).
Emergence of Decision Structures:
A key feature of ReMaP modules is that they are able to discover de novo the underlying output domain spaces for a variety of qualitatively distinct tasks (Fig.
3; more examples in Fig. S2).
The emergent decision structures are highly interpretable and reﬂect the true interfaces that the environment implicitly deﬁnes.
The spatiotemporal patterns of learning are robust across tasks and replicable across initial seedings, and thus might serve as a candidate model of interface use and learning in humans.
In general, we observe that the modules typically discover
[Figure]
a c
[Figure]
EMS
No symm
No mult
None (large)
No mult/symm
None (medium)
None (small)
[Figure]
-N-AUC.
the underlying “physical structures” needed to operate the task interface before learning the speciﬁc decision rules needed to solve the task. For example, in the case of a discrete MTS categorization task (Fig. 3a), this involves the quick discovery of onscreen “buttons” corresponding to discrete action choices before these buttons are mapped to their semantic meaning. In the case of in the MS-COCO MTS task (Fig. 3b), we observe the initial discovery of high salience object boundaries, and followed by category-speciﬁc reﬁnement.
It is important to note that the visual backbone was trained on a categorization task, quite distinct from the localization task in MS-COCO MTS.
Thus, the module had to learn this very different decision structure, as well as the class boundaries of MS-COCO, from scratch during training.
Efﬁciency of the EMS module: The efﬁciency of learning was measured by computing the taskaveraged, normalized area under the learning curve (TA-N-AUC) for each of the 24 modules tested, across all 12 task variants.
Fig.
4a-d shows characteristic learning curves for several tasks, summarized in the table in Fig.
4e.
Results for all architectures for all tasks are shown in Supplement
Figure S1.
We ﬁnd that the EMS module is the most efﬁcient across tasks (0.997 TA-N-AUC).
Moreover, the EMS architecture always achieves the highest ﬁnal reward level on each task.
Increasing ablations of the EMS structure lead to increasingly poor performance, both in terms of learning efﬁciency and ﬁnal performance.
Ablating the low-order polynomial interaction (replacing
Sq with CReLU ) had the largest negative effect on performance (0.818 TA-N-AUC), followed in importance by the symmetric structure (0.944 TA-N-AUC).
Large fully-ablated models (no bottleneck, using only ReLU activations) performed signiﬁcantly worse than the smaller EMS module and the single ablations (0.717 TA-N-AUC), but better than the module with neither symmetry nor multiplicative interactions (0.566 TA-N-AUC).
Small fully-ablated modules with the same number of parameters as EMS were by far the least efﬁcient (0.403 TA-N-AUC) and oftentimes achieved much lower ﬁnal reward.
In summary, the main conceptual features by which the special-case architecture in eq.
(5) solves the binary SR task are both individually helpful, combine usefully, and can be parameterized and efﬁciently learned for a variety of visual tasks.
These properties are critical to achieving effective task learning compared to standard MLP structures.
In a second experiment focusing on localization tasks, we tested an EMS module using convolutional features from the ﬁxed VGG-16 feature encoder, reasoning that localization tasks could beneﬁt from ﬁner spatial feature resolution.
We ﬁnd that using visual features with explicit spatial information [Figure] substantially improves task performance and learning efﬁciency on these tasks (Fig. 5).
To our knowledge, our results on MS-COCO are the ﬁrst demonstrated use of reinforcement learning to achieve instance-level object segmentations.
Reward curves (measuring bounding box IoU) in Fig.
5a show little difference between any of the late bottleneck modules at any size.
The only models to consistently achieve an IoU above 0.4 are the EMS-like variants, especially with convolutional features.
For context, a baseline SVR trained using supervised methods to directly regress bounding boxes using the same VGG features results in an IoU of 0.
D YNAMIC N EURAL V OTING FOR T ASK S WITCHING
So far, we’ve considered the case where the TouchStream consists of only one task.
However, agents in real environments are often faced with having to switch between tasks, many of which they may be encountering for the ﬁrst time.
Ideally, such agents would repurpose knowledge from previously learned tasks when it is relevant to a new task.
Formally, we now consider environment policies consisting of sequences of tasks T
= { τ 1 , τ 2 , ..., τ Ω } , each of which may last for an indeterminate period of time.
Consider also a set of modules M , where each module corresponds to a task-speciﬁc policy π ω ( a | x ) =
p ( a t | x t , τ ω ) .
When a new task begins, we cue the agent to allocate a new module M Ω+1 which is added to the set of modules M .
In the learning that follows allocation, the weights in old modules are held ﬁxed while the parameters in the new module M Ω+1 are trained.
However, the output of the system is not merely the output of the new module, but instead is a dynamically allocated mixture of pathways through the computation graphs of the old and new modules.
This mixture is determined by a meta-controller (Fig. 6).
The meta-controller is itself a neural network which learns a dynamic distribution over (parts of) modules to be used in building the composite execution graph.
Intuitively, this composite graph is composed of a small number of relevant pathways that mix and match parts of existing modules to solve the new task, potentially in combination with new module components that need to be learned.
4. D YNAMIC N EURAL V OTING
We deﬁne a meta-controller that assigns weights to each layer in each module in M .
Let p iω be the weight associated with the ith layer in module ω .
These weights are probabilistic on a per layer basis, e.g. p iω ≥ 0 and (cid:80) ω
p iω = 1 and can be interpreted as the probability of the controller selecting the ith layer l iω for use in the execution graph, with distribution π i = { p iω } .
For such an assignment of weights, the composite execution graph deﬁned by the meta-controller is generated by computing the sum of the activations of all the components at layer i weighted by the probabilities p iω .
These values are then passed on to the next layer where this process repeats.
Mathematically, the composite layer at stage i can be expressed as
[Equation]
[Figure]
A
[Figure]
+ compositelayer visualinput +
[Figure]
[Figure]
p 1 ω M
[Figure]
ω p 2 ω M
[Figure]
p
[Figure]
p [Figure]
[Figure]
where M iω ( · ) is the operator that computes the ith layer of module ω , and ˜ l 0 :
= ψ ( x t ) is the original encoded input state.
The question now is, where do these probabilistic weights come from?
The core of our procedure is a dynamic neural voting process in which the controller network learns a Boltzmann distribution over module activations to maximize reward prediction accuracy.
This process is performed at each module layer, where the module weightings for a given layer are conditioned on the results of voting at the previous layer.
That is,
[Equation]
where ppp i
= ( p i 0 , p i 1 , ..., p i Ω ) are the module weights at layer i , ⊕ is concatenation, and
W i ∈
R (Ω · L ) × Ω is a learnable weight matrix of the controller.
This voting procedure operates in an online fashion, such that the controller is continously learning its meta-policy while the agent is taking actions.
As deﬁned, the meta-controller constitutes a fully-differentiable neural network and is learned by gradient descent online.
A useful reﬁnement of the above mechanism involves voting across the units of M .
Speciﬁcally, the meta-controller now assigns probabilistic weights p
i,jω to neuron n i,jω (the jth unit in layer i of module ω ).
In contrast to the layer-voting scheme, the dynamically generated execution graph computed by the meta controller now becomes composite neurons with activations:
[Equation]
which are concatenated to form the composite layer ˜ l i M .
The generalization of equation (7) to the single-unit voting scheme then becomes:
[Equation]
where ppp i,j = ( p i,j 0 , p i,j 1 , ..., p i,j Ω ) are the unit-level weights across modules, and W i,j ∈ R Ω × Ω .
Empirically, we ﬁnd that the initialization schemes of the learnable controller parameters are an important consideration in the design, and that two specialized transformations also contribute slightly to its overall efﬁciency.
For details on these, please refer to the Supplement.
The dynamic neural voting mechanism achieves meta-control through a neural network optimized online via gradient descent while the modules are solving tasks, rather than a genetic algorithm that operates over a longer timescale as in the work of Fernando et al.
Moreover, in contrast to the work of Rusu et al.
the voting mechanism eliminates the need for fully-connected adaptation layers between modules, thus substantially reducing the number of parameters required for transfer.
4. S WITCHING E XPERIMENTS
[Figure]
“No-switch” switches: Our ﬁrst experiments tested how the dynamic neural voting mechanism would respond to “no-switch” switches, i.e. ones in which although a switch cue was given and
a new module allocated, the environment policy’s task did not actually change (Fig 7).
We ﬁnd that in such cases, performance almost instantly approaches pre-switch levels (e.g. there is very little penalty in attempting an uneccessary switch).
Moreover, we ﬁnd that the weightings the controller applies to the new module is low: in other words, the system recognizes that no new module is needed and acts accordingly by concentrating its weights on the existing module.
These results show that, while we formally assume that the agent is cued as when task switches occurs, in theory it could implement a completely autonomous monitoring policy, in which the agent simply runs the allocation procedure if a performance “anomoly” occurs (e.g. a sustained drop in reward).
If the system determines that the new module was unneeded, it could simply reallocate the new module for a later task switch.
In future work, we plan to implement this policy explicitly.
“Real” switches: We next tested how the dynamic voting controller handled switches in which the environment policy substantially changed after the switching cue.
Using both the EMS module and (for control) the large fully-ablated module as described in § 3.2, the dynamic neural voting controller was evaluated on 15 switching experiments using multiple variants of SR and MTS tasks.
Speciﬁcally, these 15 switches cover a variety of distinct (but not mutually exclusive) switching types including:
• addition of new classes to the dataset (switch indexes 2, 7, 11 in the table of Fig.
8) • replacing the current class set entirely with a new non-overlapping class set (switch ids. 1, 3) • addition of visual variability to a previously less variable task (switch id. 6) • addition of visual interface elements e.g. new buttons (switch id. 8) • transformation of interface elements
e.g. screen rotation (switch ids. 12, 13, 14, 15) • transitions between different task paradigms e.g. SR to MTS tasks and vice-versa (switch ids. 4, 5, 9, 10).
Controller hyperparameters were optimized in a cross-validated fashion (see Appendix G.1), and optimizations for three different initialization seeds were run to obtain mean and standard error.
Figures 8a and b show characteristic post-switch learning curves for the EMS module for both the
Layer Voting and Single-Unit Voting methods.
Additional switching curves can be found in the
Supplement.
Cumulative reward gains relative to learning from scratch were quantiﬁed by Relative
Gain in
[Table] [Table]
, where M is the module trained from scratch on
[Table]
a r d
R e l a ti v e
AU C
G a i n
[Table]
a b c d e
T r a n s
[Table]
n
[Table]
[Figure]
the second task, and M switch is the module transferred from an initial task using the dynamic voting controller.
We ﬁnd that the dynamic voting controller allows for rapid positive transfer of both module types across all 15 task switches, and the general Single-Unit voting method is a somewhat better transfer mechanism than the Layer Voting method (Fig. 8c).
Both the EMS module and the large fully-ablated module, which was shown to be inefﬁcient on single-task performance in § 3.2, beneﬁt from dynamic neural voting (Fig. 8 d).
EMS modules are more “switchable”:
To quantify how fast switching gains are realized, we use
Transfer Gain: TGain = ∆ max T ∆ max , where T ∆ max = argmax (∆ t ) is the time where the maximum amount of reward difference between M switch and M occurs, and ∆ max is the reward difference at that time.
Qualitatively, a high score on the Transfer Gain metric indicates that a large amount of relative reward improvement has been achieved in a short amount of time (see Figure S7 for a graphical illustration of the relationship between the RGain and TGain metrics).
While both the
EMS and large fully-ablated modules have positive Transfer Gain, EMS scores signiﬁcantly higher on this metric, i.e. is signiﬁcantly more “switchable” than the large fully-ablated module (Fig. 8e).
We hypothesize that this is due to the EMS module being able to achieve high task performance with signiﬁcantly fewer units than the larger fully-ablated module, making the former easier for the dynamic neural voting controller to operate on.
C ONCLUSION AND F UTURE D IRECTIONS
In this work, we introduce the TouchStream environment, a continual reinforcement learning framework that uniﬁes a wide variety of spatial decision-making tasks within a single context.
We describe a general algorithm (ReMaP) for learning light-weight neural modules that discover implicit task interfaces within this large-action/state-space environment.
We show that a particular module architecture (EMS) is able to remain compact while retaining high task performance, and thus is especially suitable for ﬂexible task learning and switching.
We also describe a simple but general dynamic task-switching architecture that shows substantial ability to transfer knowledge when modules for new tasks are learned.
A crucial future direction will be to expand insights from the current work into a more complete continual-learning agent.
We will need to show that our approach scales to handle dozens or hundreds of task switches in sequence.
We will also need to address issues of how the agent determines when to build a new module and how to consolidate modules when appropriate (e.g. when a series of tasks previously understood as separate can be solved by a single smaller structure).
It will also be critical to extend our approach to handle visual tasks with longer horizons, such as navigation or game play with extended strategic planning, which will likely require the use of recurrent memory stores as part of the feature encoder.
From an application point of view, we are particularly interested in using techniques like those described here to produce agents that can autonomously discover and operate the interfaces present in many important real-world two-dimensional problem domains, such as on smartphones or the internet (Grossman, 2007).
We also expect many of the same spatially-informed techniques that enable our ReMaP/EMS modules to perform well in the 2-D TouchStream environment will also transfer naturally to a three-dimensional context, where autonomous robotics applications (Devin et al., 2016) are very compelling.
S UPPLEMENTARY M ATERIAL A T ASK V ARIANTS
The EMS module and all ablation controls were evaluated on a suite of 13 stimulus-response, match-to-sample, localization and MS-COCO MTS variants:
1. 2-way SR - standard binary SR task 2.
4-way double binary SR - four class variant of SR, where each class is assigned either to the right or left half of the action space 3.
4-way quadrant SR - four class variant of SR, where each class is assigned to only a quadrant of the action space 4.
2-way stationary MTS - standard binary MTS task with stereotyped and non-moving match screens
5. 2-way stationary horiz-ﬂip MTS - two class variant MTS task where the match templates’ horizontal placement is randomly chosen, but conﬁned within the same vertical plane 6. 2-way stationary vert-motion MTS - two class variant MTS task where the match templates’ vertical position is randomly chosen, but each class is conﬁned to a speciﬁc side 7. 2-way stationary vert-motion horiz-ﬂip MTS - two class variant MTS task where the match templates’ positions are completely random 8. 4-way 2-shown MTS - four class variant MTS task where only two class templates are shown on the match screen (appearing with random horizontal location as well) 9.
4-way 2-shown vert-motion MTS - same as above, but with random vertical motion for the templates 10.
4-way 4-shown stationary MTS - four class variant MTS task where all four class templates are shown on the match screen, but with ﬁxed positions.
4-way 4-shown permuted MTS - same as above, but with randomly permuted locations of all match templates 12.
Localization - Localization task 13.
MS-COCO MTS - 80-way MTS task using the MS-COCO detection challenge dataset, where match screens are randomly samples scenes from the dataset
B E XPERIMENT DETAILS AND DATASETS
Stimulus-Response Experiment Details: Image categories used are drawn from the Image-Net 2012 ILSVR classiﬁcation challenge dataset Deng et al.
Four unique object classes are taken from the dataset: Boston Terrier, Monarch Butterﬂy, Race Car, and Panda Bear.
Each class hasunique training instances, and 50 unique validation instances.
Match-To-Sample Experiment Details: Sample screen images drawn from the same Image-Net class set as the Stimulus-Response tasks.
One face-centered, unobstructed class instance is also drawn from the Image-Net classiﬁcation challenge set and used as a match screen template image for that class.
Class template images for the match screen were held ﬁxed at 100x100 pixels.
For all variants of the MTS task, we keep a six pixel buffer between the edges of the screen and the match images, and a twelve pixel buffer between the adjascent edges of the match images themselves.
Variants without vertical motion have the match images vertically centered on the screen.
Localization Experiment Details: The Localization task uses synthetic images containing a single main salient object placed on a complex background (similar to images used in Yamins & DiCarlo (2016);
Yamins et al.
There are a total of 59 unique classes in this dataset.
In contrast to other single-class localization datasets (e.g. Image-Net) which are designed to have one large, face-centered, and centrally-focused object instance and for which a trivial policy of “always poke in image corners” could be learned, this synthetic image set offers larger variance in instance scale, position, and rotation so the agent is forced into learning non-trivial policies requiring larger precision in action selection.
MS-COCO MTS Experiment Details
This task uses the entire MS-COCO detection challenge dataset Lin et al.
On every timestep, a sample screen chosen from one of the 80 MS-COCO classes.
These are constructed to be large, unobstructed, face centered representations of the class.
For the match screen, we sample a random scene from MS-COCO containing any number of objects, but containing at least a single instance of the sample class.
The agent is rewarded if its action is located inside any instance of the correct class.
Both modules use sample actions from a low-temperature
Boltzmann policy from eq.
(4), which was empirically found to result in more precise reward map prediction.
C M ODULES C. U
[Equation]
[Table]
N one t anh ( s m a ll )
Normalized Validation AUC 1.
C. T HE C ONVOLUTIONAL -EMS M ODULE
This is a "Convolutional Bottleneck" extension of the EMS module shown in the paper, where skip connections link the conv5 and the FC6 representation of the visual backbone.
Here, the "scenelevel" representation stored in the FC6 ReMaP memory buffer is tiled spatially to match the present convolution dimensions (here 14x14), and concatenated onto its channel dimension.
A series of 1x convolutions plays the role of a shallow visual bottleneck, before the activations are vectorized and concatenated with A as input to the CReS layers of the standard EMS module.
The results in the paper are shown for a bottleneck consisting of a single tanh and two CReS convolutions, with 128 units each.
The Downstream layers use 128 units each as well.
The motivation for the convolutional bottleneck is that lower-level features are useful for complex spatial tasks such as Localization and Object Detection, and hence may result in a more precise policy.
By tiling the entire scene-level representation along the convolution layer’s channel dimension, a form of multiplicative template-matching is possible between objects that must be memorized (e.g.
MS-COCO MTS templates) and what is inside the present scene.
D E XHAUSTIVE A BLATION S TUDY
In all, we investigated 23 distinct ablations on the EMS module, across all twelve task variants outlined in sec A (Fig. S1).
Symmetry ablations replace CReS with the activation x (cid:55)→ ReLU ( x ) ⊕ x
Multiplicative ablations are denoted by specifying the nonlinearity used in place of CReS (where this is one of ReLU , tanh , sigmoid, elu Clevert et al.
(2015), or CReLU Shang et al.
This additionally includes one partial symmetry ablation (denoted “partial symm”) where only the visual bottleneck is symmetric, and one which ablates the ReLU from the “no symm” module (denoted “no symm/partial-mult”).
[Table]
[Figure]
[Table]
D. H YPERPARAMETERS
Learning rates for the ADAM optimizer were chosen on a per-task basis through cross-validation on a grid between [ 10 − 4 , 10 − 3 ] for each architecture.
Values used in the present study may be seen in
Table S2.
E A DDITIONAL MS-COCO R EWARD M APS
Five additional reward map examples for the MS-COCO MTS task are provided in in Figure S2.
Examples are plotted over the course of learning.
F A DDITIONAL L EARNING C URVES F. S INGLE -T ASK A BLATION E XPERIMENTS
Learning trajectories for seven additional tasks are provided in Figure S3.
Modules capable of convergence on a task were run until this was acheived, but AUC values for a given task are calculated at the point in time when the majority of models converge.
F. D YNAMIC V OTING C ONTROLLER AND EMS MODULE T ASK -S WITCHING E XPERIMENTS
Additional trajectories for ten unshown switching curves are provided in Figure S4.
G D YNAMIC V OTING C ONTROLLER AUGMENTATIONS G. L EARNABLE P ARAMETER I NITIALIZATIONS
Here we describe the weight initialization scheme that was found to be optimal for use with the dynamic voting controller.
For simplicity, consider the layer-voting mechanism, with learnable
Sample
Match
[Figure] [Figure]
EMS
No symm
No mult None (large) [Equation] medium
[Equation]
None
[Equation]
small)
[Figure] [Figure]
weight matricies W i and biases b i .
The intended biasing scheme is achieved through initializing the elements of these parameters to:
[Equation]
This initialization technique was also generalized for use with the single-unit voting mechanism.
For the switching experiments presented in section § 4.2, we sweep the hyperparameters on a narrow band around the default scheme.
The ranges for these are: µ 0 ∈ [0 . 01 , 0 . 005] , b 0 ∈ [0 . 1 , 0 . 01] , µ 1 ∈ [0 . 01 , 0 . 02] , and b 1 ∈ [0 . 1 , 0 . 2 , 0 .
G. T ARGETED T RANSFORMATIONS
Two additional switching mechanisms were added to the controller to augment its ability to switch between taks which are remappings of the action space or reward policy of a preexisting module.
G.2. A CTION T RANSFORMATIONS
we note that efﬁcient modules are those which can effectively produce a minimal representation of the interaction between action space A and observation x t .
If the agent’s optimal action space shifts to A (cid:48) while the remainder of the task context remains ﬁxed, the controller should allow for rapid targeted remapping A (cid:55)→ A (cid:48) .
Since we formulate the modules as ReMaP Networks, and A is an input feature basis, we can achieve remappings of this form through a fully-connected transformation:
[Equation]
(12) where a τ =
[ h t − kb : t − 1 , a t ] is the vector of action histories, and W a and b embed a τ into new action space A (cid:48) using only a small number of learnable parameters.
Pseudo Identity-Preserving Transformation
In practice, we initialize the parameters in eq.
(12) such that the transformation is pseudo identity-preseving , meaning that the representation learned at this level in the original module is not destroyed prior to transfer.
This is done by initializing W a to be an identity matrix I | a τ | with a small amount of Gaussian noise (cid:15) ∼ N (0 . 0 , σ 2 ) added to break symmetry.
b is initialized to be a vector of ones of size | a τ | .
G.
R EWARD M AP T RANSFORMATIONS
Each of the k f maps m t ( x ) reﬂects the agent’s uncertainty in the environment’s reward policy.
If the task context remains stationary, but the environment transitions to new reward schedule R (cid:48) that no longer aligns with the module’s policy π , the controller could to this transition by e.g. containing a mechanism allowing for targeted transformation of m ( x ) and hence also π .
One complication that arises under ReMaP is that since each task-module learns its optimal action space internally, m ( x ) are in the basis of R rather than A .
Therefore, transformations on the map distribution must also re-encode A before mapping to R (cid:48) .
In this work, we investigate a shallow “adapter” neural network that lives on top of the existing module and maps R (cid:55)→ R (cid:48) .
Its ﬁrst and second layers are deﬁned by
[Equation] [Equation]
where g ( a τ ) is a similar transformation on A as above, (cid:12) denotes elementwise multiplication, W 1 is a learnable matrix embedding into a hidden state, and W 2 ∈ R | l 1 |×|R (cid:48) | is a learnable matrix embedding into R (cid:48)
Pseudo Identity-Preserving Transformation Similar to the transformation on the action space, we modify the reward-map transformation to be pseudo identity-preserving as well.
This is done by modifying eq.
(13) such that the original maps are concatenated on to the beginning of the transformation input vector:
[Equation]
The intended map-preserving transformation is accomplished via initializing W 1 and W 2 as:
[Equation]
G. T ARGETED T RANSFORMATION H YPERPARAMETERS
Both of the targeted transformations have several hyperparameters.
We conducted a grid search to optimize these in a cross-validated fashion, on a set of test task switches designed to be solved by one of the targeted transformations (Fig. S5).
Each was conducted independently of the dynamic voting controller, and independently of the other transformation.
Optimal hyperparameters found in these experiments were ﬁxed for use in the integrated dynamic voting controller, and were not further optimized afterwards.
Action Transformation Hyperparameters We conducted three tests using the stimulus-response paradigm: class reversal (in which the left class becomes the right class and vice-versa), a horizontal rotation of the reward boundaries (such that right becomes up and left becomes down), and a “switch” to the original task (intended to test the identity-preserving component).
In this work, we ﬁnd that a single, non-activated linear transformation ( f in (12)) is optimal for this new state-space embedding, using k b ∗ 2 units, and initialized such that the idendity-preserving transformation weights have σ = 0 .
The learning rate for this transformation was found to be optimal at 0 . 1 .
Reward Map Transformation Hyperparameters We conducted two tests using the stimulusresponse paradigm: a “squeezing” task (where there is no longer any reward dispensed on the lower half of the screen), and a “switch” to the original task (intended to test the identity-preserving component).
In this work, we ﬁnd the optimal activations in eq.
(15) to be f ( · ) =
CReS and g ( · )
= ReLU , with 4 units in the hidden layer.
(cid:15) in the weight initialization scheme was found optimal at 0.001, and an initial bias of 0.01.
The optimal learning rate for this transformation was found to be 0.01.
G. T RANSFORM A BLATION
A study was conducted to determine the relative beneﬁt of the targeted transformations (Fig. S6), where it was determined that the primary contribution of the dynamic neural controller was in fact the voting mechanism (although the transformations did supplement this as well).
G. D EPLOYMENT SCHEME OF TASK MODULES
When cued into task transition, the controller freezes the learnable parameters of the old task-module, and deploys a new unitialized task-module.
The controller then initializes the action and reward map transformation networks as described in G.2 on top of the old module.
These transformations are also voted on inside the dynamic neural controller at every timestep.
H S WITCHING M ETRICS
Figure S7 graphically illustrates the metrics used inside the paper to quantify switching performance:
RGain and TGain .
a
Stimulus-Response image: reward map:
Environment Reward
Policy Reversal
Task
Switch
Training
Trials b
Training
Trials
Stimulus-Response image: reward map:
Environment Reward
Policy Rotation
Task
Switch c
Training
Trials
Stimulus-Response image: reward map:
Environment Reward
Policy Squeeze
Task
Switch
R e l a ti v e
AU
C
G a i n              15 − 0.
[Figure]
[Equation]
[Figure]
[Equation]