J ELLY B EAN W ORLD : A T ESTBED FOR N EVER -E NDING L EARNING
A BSTRACT
Machine learning has shown growing success in recent years.
However, current machine learning systems are highly specialized, trained for particular problems or domains, and typically on a single narrow dataset.
Human learning, on the other hand, is highly general and adaptable.
Never-ending learning is a machine learning paradigm that aims to bridge this gap, with the goal of encouraging researchers to design machine learning systems that can learn to perform a wider variety of inter-related tasks in more complex environments.
To date, there is no environment or testbed to facilitate the development and evaluation of never-ending learning systems.
To this end, we propose the Jelly Bean World testbed.
The Jelly Bean
World allows experimentation over two-dimensional grid worlds which are ﬁlled with items and in which agents can navigate.
This testbed provides environments that are sufﬁciently complex and where more generally intelligent algorithms ought to perform better than current state-of-the-art reinforcement learning approaches.
It does so by producing non-stationary environments and facilitating experimentation with multi-task, multi-agent, multi-modal, and curriculum learning settings.
We hope that the Jelly Bean World will prompt new interest in the development of never-ending learning, and more broadly general intelligence.
I NTRODUCTION
Machine learning has witnessed growing success across a multitude of applications over the past years.
However, despite these successes, current machine learning systems are each highly specialized to solve one or a small handful of problems.
They have much narrower learning capabilities as compared to humans, often learning just a single function or model based on statistical analysis of a single dataset.
One reason for this is that current machine learning paradigms are restricted and specialized to a particular problem and/or dataset.
An alternative learning paradigm that more closely resembles the generality, diversity, competence, and cumulative nature of human learning is never-ending learning (Mitchell et al., 2018).
The thesis of never-ending learning is that we will never truly understand machine learning until we can build computer programs that, like people: (i) learn many different types of knowledge or functions, (ii) from years of diverse, mostly selfsupervised experience, (iii) in a staged curricular fashion, where previously learned knowledge enables learning further types of knowledge, and (iv) where self-reﬂection and the ability to formulate new representations and new learning tasks enable the learner to avoid stagnation and performance plateaus.
Building computer programs with these properties necessitates well-deﬁned and robust ways to evaluate whether a system is indeed capable of never-ending learning.
However, there are currently no ways to achieve that.
There only exists one large-scale case study on never-ending learning with the Never-Ending Language Learning (NELL) system by Mitchell et al.
(2018), which uses the internet as the environment with which the system interacts.
While the internet does have signiﬁcant complexity, it is unwieldy to use as a testbed.
It is very difﬁcult to focus on a particular aspect of the system or the environment, or to tweak the algorithm and restart experiments to observe the effects of changes.
Furthermore, oftentimes tasks require manual annotation which can be very expensive.
Thus, a good testbed for never-ending learning (and machine learning more generally) needs to provide the experimenter with a high degree of control.
To this end, we propose a novel evaluation framework—the Jelly Bean World (JBW) —that can enable and facilitate research towards the goal never-ending learning.
We have designed the JBW to be highly versatile, enabling evaluation of systems that have any number of the aforementioned abilities.
We consider never-ending learning in the context of reinforcement learning.
Let s t ∈ S denote the state of the environment at time t ,
a t ∈
A denote the action performed by the learning agent at time t , ω t ∈ Ω denote the observation of the world that the learning agent receives at time t , and r t ∈ R denote the reward provided to the learning agent at time t .
The distribution of the next state of the world s t ∼ T ( s t − 1 , a t − 1 ) has the Markov property (i.e., it depends only on the previous state and action) and the initial state of the world is given by a distribution s 0 ∼ W .
The observation ω t ∼ O
( s t ) depends only on the current state of the world (perhaps deterministically).
The reward r t is given by a function R ( s t − 1 , a t − 1 , s t ) of the current state, the previous state, and the previous action taken.
The environment is a tuple containing all these elements E (cid:44) ( W, T, O, R ) .
Then, the goal of reinforcement learning is to ﬁnd a learning algorithm π that, given the history of previous observations, actions, and rewards, outputs the next action so that the obtained reward is maximized.
We deliberately blur the distinction between the policy and the algorithm that learns the policy, which is why we call π a “learning algorithm.”
This formalism does not distinguish between learning algorithms that are highly specialized to a single task and learning algorithms that are capable of learning a wide variety of tasks and adapting to richer and more complex environments, which are hallmarks of general intelligence .
In order to more formally describe general intelligence, we posit that there is an underlying measure of complexity of the environment E such that: (i) highly specialized and non-general learning algorithms can perform well in environments with low complexity, but (ii) environments with high complexity require successful learning agents to possess more general learning capabilities.
It is in these more complex environments where we can characterize never-ending learning.
We can formalize this notion of complexity by letting π ∗ be the (computable) learning algorithm that maximizes expected reward in an environment E .
Then we deﬁne the complexity of E to be the length of the shortest program (Turing machine) that implements π ∗ :
[Equation]
[Table]
We can equivalently deﬁne complexity ( E ) =
K ( π ∗ ) , where K ( · ) is the Kolmogorov complexity and is related to the minimum description length and minimum message length (Nannen, 2010;
Kolmogorov, 1963).
The Kolmogorov complexity of the environment K ( E ) is bounded below by
K ( π ∗ ) minus a constant.
This bound is shown in Section A.1 of the appendix.
Never-ending learning is in many respects similar to lifelong learning , also called continual learning (Chen & Liu, 2018).
Like never-ending learning, lifelong learning is distinguished from multi-task learning by the never-ending nature of the learning problem.
However, in never-ending learning, and unlike multi-task learning and lifelong learning (to the best of our understanding), a well-deﬁned set of tasks is not assumed a priori.
Rather, never-ending learning is more similar to real-world settings in this respect, where the notion of a task or subtask naturally emerges from the complexity of the environment, and the distinction between tasks is not always so sharp.
Regardless, due to their similarities, a good testbed for never-ending learning will also be a good testbed for lifelong learning.
In contrast to most popular reinforcement learning settings, never-ending learning focuses on environments with high complexity .
In never-ending learning, agents can only exist in a single reset-free environment (i.e., we explicitly disallow the agent π from learning across multiple episodes or in multiple environments, which is closer to human learning).
We require π to only have access to a single episode.
During its lifetime, π can only use the information provided by its past observations { ω t } and actions { a t } in a single world.
Thus, never-ending learning explicitly removes the distinction between training and testing that is common to many other classical machine learning paradigms.
Additionally, note that in the general reinforcement learning formalism, s t can contain information about t , and the reward function R can be time-varying, thus rendering the environment non-stationary.
In never-ending learning, we are interested in the full generality of non-stationary environments, as the assumption of stationarity is not realistic in even simple adversarial and multi-agent settings.
We thus argue that a testbed for never-ending learning should have the following properties:
Non-Episodic: It should disallow agents from resetting the environment and “retrying”.
The testbed should also force them to only learn within a single environment (i
e., not transfer information across environments).
This is in contrast with most popular reinforcement learning environments and, as we show in Section 4, poses signiﬁcant challenges to existing algorithms.
Non-Stationary: The testbed should allow for easy experimentation with non-stationary environments, where the reward R can depend on time.
[Table]
Built on fast
MiniGridsimulator
Adversarial Games like Go (Silver et al
[Equation]
, 2017) , StarCraft (Vinyals et al
[Equation]
and Dota (OpenAI, 2019) (cid:55) (cid:51) (cid:55) (cid:51) ∼ (cid:55)
Games end when the player wins or loses
Non-stationary(without assumptionsabouttheadversaries)
Each game has a single ﬁxed rewardfunction
Agents observe the game video frames and the game state
There is limited control over things like the adversary’scompetence
Experimentsaretypically extremely computationallyexpensive
DeepMind Lab (Beattie et al., 2016) (cid:55) ∼ ∼ (cid:55) (cid:55) ∼ Levels have a time limit
Levels have diﬀerent rewards but same physics
Levels have predeﬁnedrewards
Agents only observe the game video frames
The complexity of each level is ﬁxed
Requires rendering of a 3D world
Malmö and MineRL (Johnson et al., 2016) , (Guss et al., 2019) ∼ (cid:55) (cid:51) (cid:51) (cid:55) (cid:55)
Tasks have a pre-speciﬁed timelimitbutthatistypically verylong
StationaryrewardsandmapgenerationisbasedonPerlinnoise
Supports 6 complex tasks but also allows for new ones
Agents observe the game video frames and the game state
Modifying the task complexity/richness is diﬃcult and expensive
Requires rendering of a 3D world and slow training of large models
[Table]
(cid:51) (cid:51) (cid:51) (cid:51) (cid:51) (cid:51)
Agents live “forever” in an inﬁnite open world
The rewards and the world can both be non-stationary
Composableanddynamictasksaresupported
Vision and scent are designed to be complementary
Modifying the task complexity/richness is very easy
Experimentscanruneﬃcientlyonsmallmachines
Multi-Task: It should support settings in which reward is maximized not by learning how to perform a single task repetitively, but by learning how to perform a general variety of tasks, and learning how to switch between them and/or combine them to better perform other tasks (e.g., by composing them).
We posit that, at a sufﬁciently high level of task complexity, optimal learning agents will be required—either explicitly or implicitly—to perform abstract reasoning over concepts and make informed decisions about actions in the environment.
Multi-Modal: It should support multiple data modalities that agents receive as input.
These modalities should not contain the same information, but rather be complementary to each other so that the agents are forced to learn from diverse types of experiences.
Multi-modality provides yet another way to increase the complexity of the world.
Controllable: It should be easy for experimenters to modify the complexity and richness of the learning problems in the testbed, make changes to it, and restart it (e.g., as opposed to NELL).
Efﬁcient: It should run on readily available hardware and allow for quick experimentation.
Ideally, we should not have to wait for days, weeks, or months (e.g., NELL) to obtain results.
Reproducible:
It should make it easy to reproduce results and experiments, which would facilitate scientiﬁc research.
This also requires that it allows for seamlessly saving and loading state and for reproducing results outside the environment in which they were ﬁrst obtained.
The testbed should also not require access to specialized hardware, which can be expensive.
Many of the above properties are means to increase the complexity of the world.
These properties are in fact very closely related to the characteristics of “AGI Environments, Tasks, and Agents” outlined by Laird & Wray III (2010) and later reﬁned by Adams et al.
The proposed JBW has all of these properties.
It aims to provide an easy way to create sufﬁciently complex environments allowing researchers to experiment with never-ending learning, while remaining simple enough to control the problem and enable rapid prototyping.
The JBW is a two-dimensional grid world with simple physics, but is extensible enough to admit a wide variety of complex and inter-related tasks.
We present a
MAP
Manages the infinite world map.
AGENTS
Manages agents and handles their interaction with the simulator.
SCENT
Simulates the diffusion of scent in the world.
Representedasavector.
VISION
Simulates the visual field of all managed agents.
FieldofView
Occlusion
Representedasa3Dtensor.
Advances time after all managed agents have acted, invoking modules as needed.
Simulator
Interface for reinforcement learning.
Environment
REWARD
Specifies the reward given to the agent for each possible state transition.
Collect[JellyBean] ∧ Avoid[Onion]
REWARD SCHEDULE
Specifies the reward for each time step.
Fixed / Periodic / Random
Asynchronoussimulationvisualizer.
Visualizer
Distributed simulations are also supported using MPI.
comparison with related work in Table 1, showcasing the ways in which the JBW is a novel and highly versatile evaluation framework.
The JBW is written in C++ and we provide C, Python, and
Swift APIs, and is available at https://github.com/eaplatanios/jelly-bean-world .
It is also worth noting that the JBW has already been used for the instruction of the “Deep Reinforcement
Learning” and “Never-Ending Learning” graduate courses at Carnegie Mellon University.
D ESIGN
The Jelly Bean World (JBW) consists of the following main modules (illustrated in Figure 1): (i) the simulator , which comprises the central component (the other modules only interact with the simulator), (ii) the environment , which provides a simple interface for performing reinforcement learning experiments in the never-ending learning setting as well as utilities for evaluating neverending learning systems, and (iii) the visualizer , which provides the ability to visualize and debug the behavior of learning agents.
Note that the visualizer is completely asynchronous and can be attached, reattached, and detached to and from existing simulator instances, without affecting the simulations.
2. S IMULATOR
The simulator manages a map and a set of agents .
At a high-level, the map is an inﬁnite twodimensional grid where each grid cell can contain items (e.g., jelly beans and onions) and/or agents.
Each item has a color and a scent that agents can perceive.
Each agent has a direction and a position, and can navigate the world map and collect or drop items.
The action space of each agent is: to turn, move, collect items, drop items, or do nothing.
The action space is conﬁgurable and can be constrained by the user.
These constraints are described later in this section.
Time in the simulator is discrete, and all agent-map interactions are turn-based , meaning that the simulator will ﬁrst wait for all managed agents to request an action and will then simultaneously execute all actions and advance the current time.
Thus, the simulator also controls the passage of time.
Map.
In order to truly support never-ending learning, we have designed the JBW map to be inﬁnite , meaning that it has no boundaries and agents can keep exploring it forever.
To achieve this, the map is a procedurally-generated two-dimensional grid.
We simulate it by dividing it into a collection of disjoint ( P × P )
-sized patches and only generating patches when an agent moves sufﬁciently close to them.
The map also contains items of various types which are distributed according to a pairwise-interaction point process over the two-dimensional grid (Baddeley & Turner, 2000).
More speciﬁcally, for a collection of items
I (cid:44) { I 0 , . .
, I m } , where I i =
( x i , t i ) , x i ∈
Z 2 is the position of the i th item, t i ∈ T is its type, and T is the set of all item types:
[Equation]
where f ( I i ) is the intensity of item
I i and g
( I i , I j ) is the interaction between I i and I j , which are provided as part of the item’s type.
The intensity function characterizes the (log) probability of the existence of an item
I i independent of other items in the world.
The interaction function can be understood as a description of the (log) probability of the existence of an item I i given the existence of all other items I j .
For example, the interaction function can be used to increase the log probability of an item when it appears near other items, producing a clustering effect.
Since the world is subdivided into ( P × P )-sized patches, the maximum distance of interaction between items is P .
[Figure] [Figure]
Item Types.
Each item type t ∈ T deﬁnes the following:
– Color: Fixed-size vector specifying the item color.
– Scent: Fixed-size vector specifying the item scent.
– Occlusion: Occlusion of an item (relevant to the vision modality, described later in this section).
– Intensity Function: Maps from item locations to real values.
– Interaction Functions: Collection of functions that map from pairs of item locations to real values.
The collection contains one function for each item type.
The number of item types and their properties are conﬁgurable.
The speciﬁc parametric forms for the intensity and interaction functions that are supported are described in Section A.2 of the appendix.
Note that each item type also speciﬁes additional properties that are described later in this section.
Procedural Generation.
When the simulator is instantiated the map is empty (i.e., no patches have been generated).
Whenever a new agent is added to the simulator, a patch centered at its location is generated.
In addition, whenever an existing agent moves sufﬁciently close to a region where no patch exists, a new patch is generated.
The patch generation process consists of two main steps: (i) add a new empty ( P × P )-sized patch to the collection of map patches (note that the new patch will be neighboring at least one existing patch and that all patches are disjoint), and (ii) ﬁll the new patch with items.
The second step is performed by using Metropolis-Hastings (MH) (Robert & Casella, 2010) to sample the items that the new patch contains, from the distribution deﬁned in Equation 1.
The proposal density we use is deﬁned as follows: (i) add a new item I m +
=
( x m +1 , t m +1 ) with probability 1 / (2 P 2 · |T | ) (i.e., uniform in position and type), and (ii) remove an existing item
I i with probability 1 / 2 m where m is the current number of items in the patch.
Before sampling, the patch is initialized by ﬁrst randomly selecting an existing patch and copying its items into the new patch.
This is intended to facilitate rapid mixing of the Markov chain, and reduce the number of MH iterations.
Note that if we use small patches and only sample new patches as the agents visit them, boundary effects may be observed due to the missing neighboring patches further away from the agent.
For this reason, we actually also sample all missing neighboring patches while sampling each new patch, but do not ﬁnalize them (i.e., they are still considered missing and may be resampled later on).
This prevents boundary effects during the procedural generation process.
An example is shown in Figure 2.
Each item has a color and a scent that is speciﬁed by its type and can be perceived by agents.
The
JBW thus supports two perception modalities, vision and scent .
These modalities are complementary and agents can beneﬁt by learning to combine them, as we explain at the end of this section.
Vision.
Each agent has a visual range property that speciﬁes how far they can see.
Vision is represented as a three-dimensional tensor, where the ﬁrst two dimensions correspond to the width and the height of the agent’s visual ﬁeld, and the third dimension corresponds to the color dimensionality.
The visual ﬁeld is always centered at the agent’s current position and the color observed at each cell within the visual ﬁeld is the sum of the color vectors of all items and agents located at that map location.
Agents also have a ﬁeld of view property that speciﬁes their ﬁeld of view angle (i.e., 180 ◦ denotes that the agent can only see the forward-facing half of the visual ﬁeld, whereas 360 ◦ denotes that the agent can see the whole visual ﬁeld).
The part of the visual ﬁeld that is outside an agent’s ﬁeld of view is masked out and appears black to the agent.
Another important aspect of vision is that items also have an occlusion property as part of their type.
This is used to simulate partial or complete visual occlusion (details in Section A.3).
If an item with occlusion 1 is in an agent’s visual ﬁeld, then the colors behind that item are not visible to the agent.
An example is shown in Figure 3.
Scent.
Scent is represented as a ﬁxed-dimensional vector, where each dimension can be used to model orthogonal/unrelated scents.
Each agent and each item has a pre-speciﬁed scent vector that is provided as part of the world conﬁguration (similar to their colors).
At each time step, agents can perceive the scent at their current grid position.
The physics of scent are described by a simple diffusion difference equation on the world grid.
We deﬁne the scent at location ( x, y ) at time t as:
S
[Equation] [Equation]
where λ is the rate of decay of the scent at each location, α is the rate of diffusion of the scent from neighboring grid cells, and C tx,y =
(cid:80) I ∈I tx,y scent ( I ) + (cid:80) A ∈A tx,y scent ( A ) , where I tx,y is the set of all items at time t and location ( x, y ) , and A tx,y is the set of all agents at time t and location ( x, y ) .
Our simulator ensures that the scent (or lack thereof) diffuses correctly, even as items are created, collected, dropped, and destroyed.
It does so by keeping track of the creation, collection, drop, and destruction times of each item in the world.
Note also that, while simulating this diffusion, we also take into account the non-ﬁxed patches that have been sampled in order to avoid boundary effects.
Vision-Scent Complementarity.
Vision and scent are complementary.
Vision has high precision, in the sense that the agent can see the actual color of each grid cell in its visual ﬁeld and can thus relatively accurately determine what items may exist in that cell.
However, it has low recall—the agent can only see as far as its visual range allows and it has no visual information about the rest of the map.
On the other hand, scent has low precision—the scent at the current cell is a linear combination of the scents of all items in the world and it may be very difﬁcult to learn to interpret and use it effectively.
However, scent has high recall—the scent at the current cell contains information about items in a much larger range.
Thus, learning to use both modalities will be beneﬁcial to agents.
In Section 4, we also provide some experimental results supporting this argument.
Constraints.
The simulator enforces multiple constraints on the actions that agents are allowed to take.
We have designed the following small set of constraints with the goal of providing a computationally efﬁcient way to support arbitrarily complex tasks and learning problems:
– Agent Collision: This occurs when multiple agents attempt to move to the same location at the same time.
This conﬂict can be resolved in one of three ways: (i) allow multiple agents to occupy the same location, (ii) ﬁrst-come-ﬁrst-serve (only allow the ﬁrst agent who made a move request for that location to actually move—this is the current default), or (iii) randomly choose one of the agents and satisfy their request (ignoring the requested action of the others).
– Item Blocking Movement:
Item types may specify that they block agent movement (e.g., a Wall item type).
This means that agents are not allowed to move to locations with items of that type.
– Item Collection Requirements: Item types may specify that in order to collect items of that type, an agent has to have ﬁrst collected a speciﬁed number of other items (e.g., collecting Wood may only be allowed if the agent has ﬁrst collected an Axe ).
– Item Collection Costs: Similar to the collection requirements, item types may specify that in order to collect items of that type, an agent has to drop or destroy a speciﬁed number of other items (e.g., collecting an Axe may require destroying a piece of Metal and a piece of Wood that the agent has previously collected).
Interface.
Users interact with the simulator programmatically.
JBW provides functions to add or remove agents from the world, query the current vision and scent perception of each agent, and to direct agents to perform actions.
Users can choose to add multiple agents to the world, thus enabling experimentation with multi-agent settings.
Multi-agent interactions provide another controllable source of complexity in the JBW.
Users can then request actions for each agent in the simulation (i.e., turn, move, do nothing, etc.).
Once all agents have requested actions, the simulator executes these actions and advances time, appropriately updating the state of the world.
Server/Client Support.
The JBW also provides a TCP server-client interface where the simulator can be setup to run as a server.
Users (i.e., clients) can then connect to the server, and interact with the simulator by sending messages to the server.
This allows use cases such as a class setting where students can each control an agent in a common simulated world, or perhaps a hackathon where participants can compete in a common world.
It also allows debugging and visualization tools to be attached and detached to and from running simulator instances, without affecting the simulations.
In fact, this is how our visualizer, which is described in Section 2.3, communicates with the simulator.
Persistence.
Simulations in the JBW can be saved to and loaded from ﬁles, which can then be distributed across platforms.
This facilitates reproducibility .
The simulator guarantees uniform random number generation behavior across all platforms and machines (e.g., in distributed settings).
The state of the pseudorandom number generator is also saved and loaded along with the simulation.
2. E NVIRONMENT
Environments manage simulator instances and provide an interface for performing reinforcement learning experiments using the JBW.
We provide implementations of the JBW environments for
OpenAI Gym (Brockman et al., 2016) in Python and for Swift RL (Platanios, 2019) in Swift.
JBW environments support batching by design, with support for parallel execution of the multiple simulator instances being managed (i.e., one simulator for each batch entry).
Perhaps the most important aspect of JBW environments is that they require the user to specify a reward schedule to use for each experiment.
This schedule effectively deﬁnes the tasks that the agents are learning to perform.
A reward schedule provides a function that, given a simulation time, returns a reward function to use at that time.
A reward function returns a scalar reward value, given the current and previous states of the agent and the world (e.g., the world map).
1 We provide a simple domain-speciﬁc language (DSL) for composing and combining multiple reward functions in arbitrary ways, to allow for the design of composable learning tasks.
This enables endless possibilities in the realms of multi-task learning, curriculum learning, and more generally never-ending learning.
Currently environments are limited to single agent reinforcement learning settings, but we plan to support multi-agent settings in the future (this is easy because the JBW simulator already supports multiple agents for each simulation).
2. V ISUALIZER
Visualization can be instrumental when developing, debugging, and evaluating never-ending learning systems.
To this end, we have implemented a real-time visualizer using Vulkan 2 in which the user can see any part of the simulated JBW, at any scale and simulation rate.
The visualizer utilizes the simulator server-client interface to visualize simulations running in different processes or on remote servers, in a fully asynchronous manner.
Rendering is multithreaded to provide a smooth and responsive user interface.
Finally, the visualizer can be attached, detached, and re-attached to existing simulation server instances, without affecting the running simulations.
L EARNING T ASKS
Learning tasks can be deﬁned in terms of reward functions and reward schedules , which were deﬁned in Section 2.2.
The JBW allows researchers to easily deﬁne their own reward functions and schedules, but it also provides a few primitives and ways to compose them in order to effortlessly allow for quick experimentation and prototyping.
In fact, all learning tasks used in Section 4 were deﬁned using these primitives.
The currently supported primitives are:
[Table]
We note that Collect is a sparse reward function, whereas Action and Explore are not.
For conciseness, we omit the v argument in reward functions when it is set to 1 and we also deﬁne Avoid[ i , v ] =
Collect[ i , − v ] .
E XPERIMENTS
[Table]
The goal of this section is to show how the non-episodic, non-stationary, multi-modal
[Table]
and multi-task aspects of the
JBW make it a challenging environment for existing machine learning algorithms, through a few example case studies.
For all experiments we use the simulator conﬁguration and item types shown in Tables 2 and 3.
3 Due to space, the case studies focus on the single-agent setting.
We use different agent models depending on which modalities are used in each experiment.
If vision is used, then the visual ﬁeld is passed through a convolution layer with stride 2 , 3 × 3 ﬁlters
[Table]
and 16 channels, and another one with stride 1 ,
2 × 2 ﬁlters, and 16 chan-
[Table]
nels.
The resulting tensor is ﬂattened and passed through a dense layer with size 512 .
If scent is used, then the scent vector is passed through two dense layers: one with size 32 , and one with size 512 .
If both modalities are being used, the two hidden representations are concatenated.
Finally, the result is processed by a Long
Short-Term Memory (LSTM) network (Hochreiter & Schmidhuber, 1997) which outputs a value for the agent’s current state, along with a distribution over actions.
Learning is performed using Proximal Policy Optimization (PPO); a popular on-policy reinforcement learning algorithm proposed by Schulman et al.
The experiments are implemented using
Swift for TensorFlow.
4. C ASE S TUDIES
For all experiments we evaluate performance using the reward rate metric, which is deﬁned as the amount of reward obtained per step, computed over a moving window.
The size of that window varies per experiment and is reported together with the results.
This is an appropriate metric for this task as we want to measure the improvement in the ability of an agent to learn (i.e., the gradient of the reward rate), while also making sure the agent does not get stuck (i.e., the reward rate goes to zero).
Whenever possible, we also report the results obtained by the greedy vision-based agent described in Section A.7 of our appendix.
The greedy agent makes additional assumptions about the world, and thus, doesn’t generalize to more complex environments, it provides a lower bound on the optimal reward rate.
Note that a perfect upper bound cannot be obtained as that would require solving an NP-hard discrete optimization problem.
[Figure]
Collect[JellyBean] ∧ Avoid[Onion]
[Figure]
( p t s / s )
Case Study #1: Non-Episodic.
The goal of this case study is to show that the JBW allows for experimenting with never-ending learning agents and to also show how current machine learning methods (e.g., PPO used to train an LSTM-based agent) are failing to effectively perform never-ending learning.
For this experiment we use the ﬁxed reward function Collect[JellyBean] ∧ Avoid[Onion] and let agents interact with the JBW for 10 million steps.
Our results are shown in Figure 4.
The agents seem to be learning effectively for the ﬁrst 1 million steps, but start to underperform later on, eventually getting stuck and being unable to collect any reward.
This is the case for multiple different learning agents that we experimented with; both using different models and using different learning algorithms, such as Deep Q-Networks (DQNs) proposed by Mnih et al. (2013).
After connecting the visualizer to observe what happens we see that all agents either: (i) get stuck in an area of the map that they have already explored and exhausted of jelly beans, or (ii) get stuck constantly rotating and not moving to new grid cells at all.
This indicates that the JBW is indeed challenging for current machine learning methods when it comes to never-ending learning.
Perhaps some sort of reward shaping or curriculum learning could help the agents.
However, our goal with this paper is not to solve these hard problems but rather point them out and show how the JBW provides a testbed with which to tackle them.
Case Study #2: Non-Stationary.
The goal of this case study is to demonstrate that the JBW allows for experimenting with non-stationary and multi-task learning problems.
To this end, we perform two experiments: (i) one using a cyclical/periodic reward function schedule where every 100, steps we alternate between the Collect[JellyBean] ∧ Avoid[Onion] and Avoid[JellyBean] ∧ Collect[Onion] reward functions, and (ii) one testing a couple of curriculum reward schedules for eventually learning to Collect[JellyBean] ∧ Avoid[Onion] .
The results are shown in Figureand we observe that current standard machine learning approaches are not able to efﬁciently alternate between different learning problems and are effectively learning each problem from scratch whenever they switch, eventually ending up unable to learn either one.
We also observe that agents who ﬁrst learn to collect jelly beans and then switch to the full reward function are able to learn to collect jelly beans and avoid onions faster than agents that ﬁrst learn to avoid onions or face the ﬁnal learning problem directly from the beginning.
Eventually all agents perform similarly, but this showcases how the JBW enables research in curriculum learning.
Case Study #3: Multi-Modal.
The goal of this case study is to: (i) show how computationally efﬁcient features, such as the ﬁeld of view mask and visual occlusion, allow for increasing the learning problem complexity in a controllable manner and, perhaps most importantly, (ii) show how the perception modalities of the JBW are complementary.
We thus perform three experiments.
For the ﬁrst two we use the ﬁxed reward function Collect[JellyBean] and for the last one we use
Collect[Onion] .
We change the reward function in order to show how easy it is to experiment with different tasks in the JBW.
In the ﬁrst experiment, we vary the ﬁeld of view of the agents.
The results are shown in the left plot of Figure 6.
We see that decreasing the ﬁeld of view allows us to make the learning task harder for agents, while maintaining the same computational cost for the environment.
Similarly, in the second experiment we measure the effect that visual occlusion has on performance.
The results are shown in the middle plot of Figure 6 and we observe that enabling visual occlusion makes the learning task harder.
Finally, with the third experiment our goal is to show that vision and scent are complementary.
The results are shown in the right plot of Figure 6.
We see that “vision” agents do better than “scent” agents, indicating that vision is perhaps an easier perception modality to use in the context of this learning task.
Surprisingly though, the “vision” agents also do better than the “vision+scent” agents.
This indicates a limitation of the model because, even though scent contains useful information that vision does not, the agents seem to get confused by it and do not seem able to use it properly.
It also shows the need for better multi-modal algorithms and the utility of the JBW in testing such algorithms.
The relative utility of scent and vision depends on the environment and the agent model.
We demonstrate this in Section A.4 by providing a different conﬁguration where “scent” agents are able to outperform both “vision” and “vision+scent” agents.
[Figure]
[Figure]
C ONCLUSION AND F UTURE E XTENSIONS
We presented a new testbed designed to facilitate experimentation with never-ending learning agents, where the complexity of the learning problems is higher than that of existing testbeds and evaluations, while maintaining controllability, performance, and reproducibility.
In order to produce more complex environments, the JBW supports non-stationary environments, with multiple distinct but inter-related tasks and complementary perception modalities.
The JBW also explicitly restricts learning to a single never-ending episode.
It is highly conﬁgurable and performant, and provides users with tools to easily save, load, distribute environments, and reproduce and visualize results.
We also showed how easily we can deﬁne learning tasks in the JBW, for which current machine learning methods struggle.
The space of potential extensions to the JBW is large.
Although the current intensity and interaction functions are stationary with respect to space (i.e., they are independent of position x, y ), it is not difﬁcult to deﬁne new non-stationary functions, in order to generate worlds with non-stationary item distributions.
The JBW supports multiple agents running asynchronously in the world, and so it would also be interesting to experiment with multi-agent settings.
However, agents currently don’t have an easy way to communicate with each other, and so adding a mechanism for communication, perhaps via new agent-item interactions (e.g., reading/writing note items), would be interesting.
Another way to add complexity is via items that can contain “strings” (e.g., notes) in an agent-speciﬁc language, or even natural language.
These notes could, for example, contain task speciﬁcations.
Scent currently does not interact with items in the world, meaning that it can pass through Wall items without any hinderance.
Thus, another possible extension would be to support more complex item-scent interactions.
It would be interesting to explore interactions between items and the properties of agents (e.g., a Telescope could extend the visual range of an agent while narrowing its ﬁeld of view).
Finally, another interesting way to add complexity is to generate richer relationships between item types, perhaps even an ontology.
We look forward to continue improving the JBW, and hope that a standardized testbed for never-ending learning will motivate research into more generally-intelligent learning algorithms.
A CKNOWLEDGMENTS
We thank the anonymous reviewers for their helpful comments and the Spring 2018 class in “NeverEnding Learning” at Carnegie Mellon University for the impulse and discussion that led to the development of this framework.
This research was supported in part by AFOSR grant FA95501710218.
A A PPENDIX A. C OMPLEXITY B OUND
We show that K ( E ) must be at least K ( π ∗ ) up to a constant.
We can write an algorithm ˆ π that enumerates all possible sequences of environment states, observations, actions, and rewards from time t up to time T : ( s t , ω t , a t , r t ) , . . .
, ( s T , ω T , a T , r T ) .
Then ˆ π computes the action that maximizes the expected reward E
[ R ( { r k } Tk = t )] .
Since the images im (ˆ π ) ⊆
A and im ( π ∗ ) are discrete, and lim T →∞ arg max a t E
[ R ( { r k } Tk = t )]
= arg max a t E
[ R ( { r k } ∞ k = t )] , there is a sufﬁciently large ﬁnite T such that the action computed by ˆ π is the same as that computed by π ∗ .
Note that ˆ π relies on a subroutine that simulates the environment E in order to ﬁrst enumerate the environment states, and the subroutine to perform the optimization is independent of E , and so K (ˆ π ) =
K ( E ) + c for a constant c .
Since K ( π ∗ ) ≤ K (ˆ π ) , we have that K ( E ) ≥ K ( π ∗ ) − c .
A. I NTENSITY AND I NTERACTION F UNCTIONS
The JBW currently only supports a small number of implemented intensity and interaction functions to control the distribution of items in the world.
However, it is very straightforward to implement new customized intensity and interaction functions.
Let ( x
, y ) ∈
Z 2 be a position and t ∈ T be an item type.
Intensity functions are indexed by item type, and so each item type is assigned its own intensity function:
f (( x, y ) , t ) (cid:44) f t ( x, y ) .
The JBW currently supports three intensity functions:
Zero : f t ( x, y ) = 0 .
Constant[ v ] : f t ( x, y )
= v .
RadialHash[ ∆ , s , c , k ] :
[Equation]
where ˆ M : R (cid:55)→ [0 , 1] is the linear interpolation of M ( (cid:98) t (cid:99) ) /
(2 32 − 1) and M ( (cid:98) t + 1 (cid:99) ) / (2 32 − 1) , and M : Z (cid:55)→ Z is the last mixing step of the 32-bit MurmurHash function (Appleby, 2008).
This provides pseudorandomness to the intensity function.
For interaction functions, let ( x 1 , y 1 ) be the input position of the ﬁrst item, t 1 be the type of the ﬁrst item, ( x 2 , y 2 ) be the position of the second item, and t 2 be the type of the second item.
Interaction functions are indexed by pairs of item types, so each pair of item types can be given its own interaction function:
g ((( x 1 , y 1 ) , t 1 ) , (( x 2 , y 2 ) , t 2 ))
(cid:44) g t 1 ,t 2 (( x 1 , y 1 ) , ( x 2 , y 2 )) .
The JBW currently supports four interaction functions:
Zero : g t 1 ,t 2 (( x 1 , y 1 ) , ( x 2 , y 2 )) =
PiecewiseBox[ U , V , u , v ] : g t 1 ,t 2 (( x 1 , y 1 ) , ( x 2 , y 2 )) =
[Equation] [Equation]
d < U
[Equation]
, if U ≤ d < V, 0 , otherwise , where d = ( x 1 − x 2 ) 2 + ( y 1 − y 2 ) 2 . 3.
Cross[ U , V , u , v , α , β ] : g t 1 ,t 2 (( x 1 , y 1 ) , ( x 2 , y 2 )) =
[Equation]
, if d = 0 ,
[Equation]
≤
[Equation]
, α, if
d (cid:54) = 0 , D ≤ U, v, if d = 0 , U < D ≤ V, β, if d (cid:54) = 0 , U < D ≤ V, 0 , otherwise , where d = min {| x 1 − x 2 | , | y 1 − y 2 |} and D = max {| x 1 − x 2 | , | y 1 − y 2 |} .
[ s , c , k , δ , u
v , α , β
is
U , V , u , v , α , β
are
[Equation]
M
Even though this is a small set of intensity and interaction functions it can allow for creating worlds with many interesting features (e.g., we use the Cross interaction function to create contiguous wall segments that are axis-aligned, and the PiecewiseBox interaction function to create irregularly shaped clusters of trees forming forests).
Note that the unspeciﬁed intensity and interaction functions in Table 3 are set to Zero by default.
A. F IELD OF VIEW AND V ISUAL O CCLUSION
To compute the color of a cell with respect to the agent’s ﬁeld of view, let the cell position be ( x, y ) and consider a circle of radius 12 centered at ( x, y ) .
Project this circle onto the circle of radius 1 centered at the agent position.
Let θ denote this projection (an arc).
Let θ fov be the arc on the agent’s circle centered on a point in the current agent direction.
The length of θ fov is speciﬁed by the ﬁeld-of-view parameter in the conﬁguration.
The color of cell c x,y is then computed as:
[Equation]
where ˆ c x,y is the original color of the cell.
In order to compute how much a cell at position ( x, y ) is occluded, we consider a circle of radius 12 centered at ( x, y ) , and project this circle onto the circle of radius 1 centered at the agent position.
Let θ denote this projection (an arc).
Each item in the agent’s visual ﬁeld is similarly projected onto the agent’s circle, each producing an arc θ i .
The color of the cell c x,y is then computed as:
[Equation]
where ˆ c x,y is the original color of the cell, and o i is the occlusion parameter of the i th item, as speciﬁed by the item’s type.
If a cell is affected by both the ﬁeld of view and visual occlusion, the above effects are composed (both multiplicative factors are applied to the original color).
A. R ELATIVE U TILITY OF S CENT AND V ISION
[Figure]
The relative difﬁculty of utilizing scent or vision to effectively navigate in the JBW environments depends on the environment conﬁguration as well as the method used by the learning agent.
For example, if the learning agent does not possess memory, it cannot remember the scent of any previously visited tile, and thus it will not be able to determine the direction from which the scent is diffusing.
Therefore, such methods will not beneﬁt from the information provided by the scent modality.
In addition, scent is not necessarily blocked by items that block movement (e.g., walls).
Thus,
[Table]
in environments with such items, scent is more difﬁcult to utilize, since the simple strategy of following paths of monotonically increasing scent could lead to a wall.
The agent could be fooled to believe an item is nearby when, in fact, it is behind a wall.
This is illustrated in Figureand is possibly one of the main reasons the “vision+scent” agent underperforms the “vision” agent in Figure 6.
In order to showcase how scent can provide useful information, we also designed a simpler environment conﬁguration that does not contain any walls.
The conﬁguration for this world is shown in Tables 4 and 5.
Given the task of collecting JellyBean s, we expect the scent modality to be very useful as long as the agent has some sort of memory.
The results of using an LSTM-
[Table]
based agent are shown in Figure 8.
We observe that the “scent” agent outperforms both the “vision” and the “vision+scent” agents.
This is the opposite pattern of what we observe in Figure 6.
5 This can partly be explained by the fact that, in this case, the visual ﬁeld of the agent is more restricted, thereby limiting the agent’s reliance on vision in its learning.
Ideally agents should be able to use each perception modality optimally and not be “confused” by the fact that one of them may be harder to utilize than the other.
Thus, these experiments showcase how the JBW can be used to evaluate multi-modal machine learning algorithms.
[Figure]
A. E XAMPLE OF S PATIAL N ON -S TATIONARITY
To demonstrate the ability to generate spatially non-stationary worlds in JBW
[Table]
we provide a conﬁgu-
[Table]
ration that makes use of nonstationary intensity and interaction functions.
The conﬁguration is shown in Tables 6 and 7, and a visualization is provided in Figure 9.
JellyBean s and Onion s appear together in clusters and, since this conﬁguration uses the non-stationary intensity function RadialHash , these clusters are arranged in concentric circles around the origin that are irregularly spaced.
RadialHash uses a hash function to induce a pseudorandom relationship between the distance to the origin and the likelihood of ﬁnding such clusters.
Wall s in this environment also have a non-stationary distribution.
In some regions, they are smaller and more frequent, whereas in other regions they are longer and appear more sporadically.
[Table]
A. P ERFORMANCE
The JBW is implemented in optimized C++, with performance being highly prioritized in both its design and its implementation.
This would allow less time and hardware resources to be spent simulating the world and more time and resources to be allocated for the machine learning algorithms.
Additionally, the JBW is perceptually quite simple, being a two-dimensional grid world with limited vision and scent inputs.
This allows the machine learning algorithm to focus less on perceptual information processing and more on abstract information processing, which we think is a hallmark of never-ending learning.
As a rough indication of performance, on a single core of an Intel Core i 5820K (released in 2014) at 3.5GHz, the JBW can generate 8 .
56 patches per second, each of size 64 × 64 (i.e., 35 , 062 grid cells), using the conﬁguration described in Section 4.
A. G REEDY A LGORITHM
As a benchmark and for the sake of comparison, we also implemented a simple greedy agent that searches its visual ﬁeld for cells of a particular color, and then computes the shortest path to those cells.
This algorithm makes the assumption that reward is maximized simply by collecting items of a single color, ad inﬁnitum.
It also assumes that this color is known a priori.
Additionally, it assumes the color of obstacles (items that block agent movement or that should be avoided as part of the reward function) is known a priori, and is distinct from the color of items that provide reward.
The shortest path it computes is such that it never goes through any such obstacles.
if best
= null or
shortest
best_path
assign
= shortest
path
if best_path = null
if the cell immediately in front of the
The algorithm is shown in pseudocode in Algorithm 1.
The function ShortestPath is simply
Dijkstra’s algorithm on a directed graph where each vertex corresponds to a unique agent position and direction within its visual ﬁeld ω t , and each edge corresponds to a possible action that transitions between agent states (Dijkstra, 1959).
Let c r be the color of items that provide reward, and c w be the color of items that block agent movement.
The algorithm returns a shortest path from the agent’s current position and direction to a cell that has a color γc r for any γ > 0 , while avoiding cells that have color γc w for any γ > 0 (we match any color in the direction of the vectors c r and c w in order to detect partially occluded items).
If no such path exists, ShortestPath returns null .
In the case where the agent’s ﬁeld of view is limited, ShortestPath only returns paths that pass through cells within the agent’s ﬁeld of view.
Also, in the experiment where the agent must additionally avoid
Onion items, ShortestPath avoids them in the same way that it avoids obstacles: it avoids cells that have color γc o for any γ >
where c
o is the color of the Onion item type.
In environments with visual occlusion, if items with high occlusion are arranged in a line (such as a wall), and the agent is adjacent to the wall and facing it, the portions of the wall further from the agent will be occluded by the portion of the wall closer to the agent.
Since we currently do not distinguish between empty cells and completely occluded cells, ShortestPath will return paths that may pass through the wall.
If no other paths are returned, the agent will continuously try to move through the wall and make no progress.
Similarly, in environments containing items that provide negative reward, the above greedy algorithm does not allow the agent to move over tiles with such items.
Thus, given sufﬁcient time, the agent will ﬁnd itself moving between two clusters of avoided items, where one cluster is shaped like a “ ∪ ” and the other like a “ ∩ ”.
And so when the agent moves downward into the “ ∪ ” of one cluster, it will make a 180-degree turn, leaving the cluster, and moving upward toward the other cluster.
The other cluster is shaped symmetrically, and the agent will repeat the behavior deterministically: enter the “ ∩ ”, make a 180-degree turn, leave the cluster, and move down toward the ﬁrst cluster again.
This will repeat indeﬁnitely.