E ND - TO -E ND E GOSPHERIC S PATIAL M EMORY
Anonymous authors
Paper under double-blind review
A BSTRACT
Spatial memory, or the ability to remember and recall speciﬁc locations and objects, is central to autonomous agents’ ability to carry out tasks in real environments.
However, most existing artiﬁcial memory modules have difﬁculty recalling information over long time periods and are not very adept at storing spatial information.
We propose a parameter-free module, Egospheric Spatial Memory (ESM), which encodes the memory in an ego-sphere around the agent, enabling expressive 3D representations.
ESM can be trained end-to-end via either imitation or reinforcement learning, and improves both training efﬁciency and ﬁnal performance against other memory baselines on both drone and manipulator visuomotor control tasks.
The explicit egocentric geometry also enables us to seamlessly combine the learned controller with other non-learned modalities, such as local obstacle avoidance.
We further show applications to semantic segmentation on the ScanNet dataset, where
ESM naturally combines image-level and map-level inference modalities.
Through our broad set of experiments, we show that ESM provides a general computation graph for embodied spatial reasoning, and the module forms a bridge between real-time mapping systems and differentiable memory architectures.
I NTRODUCTION
Egocentric spatial memory is central to our understanding of spatial reasoning in biology (Klatzky, 1998; Burgess, 2006), where an embodied agent constantly carries with it a local map of its surrounding geometry.
Such representations have particular signiﬁcance for action selection and motor control (Hinman et al., 2019).
For robotics and embodied AI, the beneﬁts of a persistent local spatial memory are also clear.
Such a system has the potential to run for long periods, and bypass both the memory and runtime complexities of large scale world-centric mapping.
Peters et al.
(2001) propose an EgoSphere as being a particularly suitable representation for robotics, and more recent works have utilized ego-centric formulations for planar robot mapping (Fankhauser et al., 2014), drone obstacle avoidance (Fragoso et al., 2018) and mono-to-depth (Liu et al., 2019).
In parallel with these ego-centric mapping systems, a new paradigm of differentiable memory architectures has arisen, where a memory bank is augmented to a neural network, which can then learn read and write operations (Weston et al., 2014; Graves et al., 2014; Sukhbaatar et al., 2015).
When compared to Recurrent Neural Networks (RNNs), the persistent memory circumvents issues of vanishing or exploding gradients, enabling solutions to long-horizon tasks.
These have also been applied to visuomotor control and navigation tasks (Wayne et al., 2018), surpassing baselines such as the ubiquitous Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997).
We focus on the intersection of these two branches of research, and propose Egospheric Spatial
Memory (ESM), a parameter-free module which encodes geometric and semantic information about the scene in an ego-sphere around the agent.
To the best of our knowledge, ESM is the ﬁrst end-to-end trainable egocentric memory with a full panoramic representation, enabling direct encoding of the surrounding scene in a 2.5D image.
We also show that by propagating gradients through the ESM computation graph we can learn features to be stored in the memory.
We demonstrate the superiority of learning features through the ESM module on both target shape reaching and object segmentation tasks.
For other visuomotor control tasks, we show that even without learning features through the module, and instead directly projecting image color values into memory, ESM consistently outperforms other memory baselines.
Through these experiments, we show that the applications of our parameter-free ESM module are widespread, where it can either be dropped into existing pipelines as a non-learned module, or end-to-end trained in a larger computation graph, depending on the task requirements.
R ELATED W ORK 2. M APPING
Geometric mapping is a mature ﬁeld, with many solutions available for constructing high quality maps.
Such systems typically maintain an allocentric map, either by projecting points into a global world co-ordinate system (Newcombe et al., 2011a; Whelan et al., 2015), or by maintaining a certain number of keyframes in the trajectory history (Newcombe et al., 2011b; Zhou et al., 2018;
Bloesch et al., 2018; Clark et al., 2018)
If these systems are to be applied to life-long embodied AI, then strategies must then be adopted in order to effectively select the parts of the map which are useful, and discard the rest from memory (Cadena et al., 2016).
Taking CodeSLAM (Bloesch et al., 2018) as an example, keyframe removal strategies are needed to prevent the map size from growing indeﬁnitely.
For robotics applications, prioritizing geometry in the immediate vicinity is a sensible prior.
Rather than taking a world-view to map construction, such systems often formulate the mapping problem in a purely ego-centric manner, performing continual re-projection to the newest frame and pose with ﬁxed-sized storage.
Peters et al.
(2001) outline an EgoSphere memory structure, with indexing via polar and azimuthal angles, with applications to humanoid robotics.
Fankhauser et al.
(2014) use ego-centric height maps, and demonstrate on a quadrupedal robot walking over obstacles.
Cigla et al.
(2017) use per-pixel depth Gaussian Mixture Models (GMMs) to maintain an ego-cylinder of belief around a drone, with applications to collision avoidance (Fragoso et al., 2018).
In a different application, Liu et al.
(2019) learn to predict depth images from a sequence of RGB images, again using ego reprojections.
These systems are designed to represent only at the level of depth and RGB features.
For mapping more expressive implicit features via end-to-end training, a fully differentiable long-horizon computation graph is required.
Any computation graph which satisﬁes this requirement is generally referred to as memory in the neural network literature.
2. M EMORY
The concept of memory in neural networks is deeply coupled with recurrence.
Naive recurrent networks have vanishing and exploding gradient problems (Hochreiter, 1998), which LSTMs (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Units (GRUs) (Cho et al., 2014)
mediate using additive gated structures.
More recently, dedicated differentiable memory blocks have become a popular alternative.
Weston et al.
(2014) applied Memory Networks (MemNN) to question answering, using hard read-writes and separate training of components.
Graves et al.
(2014) and Sukhbaatar et al.
(2015) instead made the read and writes ‘soft’ with the proposal of Neural Turing Machines (NTM) and
End-to-End Memory Networks (MemN2N) respectively, enabling joint training with the controller.
Other works have since conditioned dynamic memory on images, for tasks such as visual question answering (Xiong et al., 2016) and object segmentation (Oh et al., 2019).
Another distinct but closely related approach is self attention (Vaswani et al., 2017).
These approaches also use key-based content retrieval, but do so on a history of previous observations with adjacent connectivity.
Despite the lack of geometric inductive bias, recent results demonstrate the amenability of general memory (Wayne et al., 2018) and attention (Parisotto et al., 2019) to visuomotor control and navigation tasks.
Other authors have explored the intersection of network memory and spatial mapping for navigation, but have generally been limited to 2D aerial-view maps, focusing on planar navigation tasks.
Gupta et al.
(2017) used an implicit ego-centric memory which was updated with warping and conﬁdence maps for discrete action navigation problems.
Parisotto & Salakhutdinov (2017) proposed a similar setup, but used dedicated learned read and write operations for updates, and tested on simulated Doom environments.
Without consideration for action selection, Henriques & Vedaldi (2018) proposed a similar system, but instead used an allocentric formulation, and tested on free-form trajectories of real images.
Zhang et al.
(2018) propose a similar system, but with the inclusion of loop closure.
Our memory instead focuses on local perception, with the ability to represent detailed 3D geometry in all directions around the agent.
The beneﬁts of our module are complementary to existing 2D methods, which instead focus on occlusion aware planar understanding suitable for navigation.
M ETHOD
In this section, we describe our main contribution, the egospheric spatial memory (ESM) module, shown in Figure 1.
The module operates as an Extended Kalman Filter (EKF), with an egosphere image µ t ∈ R h s × w s × (2+1+ n ) and its diagonal covariance Σ t ∈ R h s × w s × (1+ n ) representing the state.
The egosphere image consists of 2 channels for the polar and azimuthal angles, 1 for radial depth, and n for encoded features.
The angles are not included in the covariance, as their values are implicit in the egosphere image pixel indices.
The covariance only represents the uncertainty in depth and features at these ﬁxed equidistant indices, and diagonal covariance is assumed due to the large state size of the images.
Image measurements are assumed to come from projective depth cameras, which similarly store 1 channel for depth and n for encoded features.
We also assume incremental agent pose measurements u t ∈ R 6 with covariance Σ u t ∈ R 6 × 6 are available, in the form of a translation and rotation vector.
The algorithm overview is presented in Algorithm 1.
1 Given: f m , F m , f o , F o 2 ¯ µ
[Equation]
f
[Equation] u t [Equation]
µ
[Equation] [Equation]
[Equation]
First, the motion step takes the state from the previous frame µ t − 1 , Σ t − 1 and transforms this into a predicted state for the current frame ¯ µ t , ¯Σ t via functions f m , F m and the incremental pose measurement u t with covariance Σ u t .
Then in the observation step, we use measured visual features ( v ti ∈ R h vi × w vi × (1+ n ) )
i ∈{ 1 ,...,m } with diagonal covariances (
V ti ∈
R
h
vi × w vi ×
(1+ n ) )
i ∈{ 1 ,...,m } originated from m arbitrary vision sensors, and associated pose measurements ( p ti ∈ R 6 ) i ∈{ 1 ,...,m } with covariances ( P ti ∈ R 6 × 6 ) i ∈{ 1 ,...,m } , to produce a new observation of the state ˆ µ t ∈ R h s × w s × (2+1+ n ) , again with diagonal covariance ˆΣ t ∈ R h s × w s × (1+ n ) , via functions f o and F o .
The measured poses also take the form of a translation and rotation vector.
Finally, the update step takes our state prediction ¯ µ t , ¯Σ t and state observation ˆ µ t , ˆΣ t , and fuses them to produce our new state belief µ t , Σ t .
We spend the remainder of this section explaining the form of the constituent functions.
All functions in Algorithm 1 involve re-projections across different image frames, using forward warping.
Functions f m , F m , f o and F o are therefore all built using the same core functions.
While the re-projections could be solved using a typical rendering pipeline of mesh construction followed by rasterization, we instead choose a simpler approach and directly quantize the pixel projections with variance-based image smoothing to ﬁll in quantization holes.
An overview of the projection and quantization operations for a single ESM update step is shown in Fig.
3. F ORWARD W ARPING
Forward warping projects ordered equidistant homogeneous pixel co-ordinates pc f 1 from frame f 1 to non-ordered non-equidistant homogeneous pixel co-ordinates ˜
pc f 2 in frame f 2 .
We use ˜ µ f 2 = { ˜ φ f 2 , ˜ θ f 2 , ˜ d f 2 , ˜
e f 2 } to denote the loss of ordering following projection from µ f 1 = { φ f 1 , θ f 2 , d f 1 , e f 2 } , where φ , θ , d and e represent polar angles, azimuthal angles, depth and encoded features respectively.
We only consider warping from projective to omni cameras, which corresponds to functions
f
o , F o , but the omni-to-omni case as in f m , F m is identical except with the inclusion of another polar co-ordinate transformation.
The encoded features are assumed constant during projection ˜
e f 2 = e f 1 .
For depth, we must transform the values to the new frame in polar co-ordinates, which is a composition of a linear transformation and non-linear polar conversion.
Using the camera intrinsic matrix K 1 , the full projection is composed of a scalar multiplication with homogeneous pixel co-ordinates pc f 1 , transformation by camera inverse matrix K − 1 1 and frame-to-frame T 12 matrices, and polar conversion f p :
[Equation]
Combined, this provides us with both the forward warped image ˜ µ f 2 = { ˜ φ f 2 , ˜ θ f 2 , ˜ d f 2 , ˜
e f 2 } , and the newly projected homogeneous pixel co-ordinates ˜ pc f
=
{ k ppr ˜ φ f 2 , k ppr ˜
θ f 2 , 1 }, where k ppr denotes the pixels-per-radian resolution constant.
The variances are also projected using the full analytic
Jacobians, which are efﬁciently implemented as tensor operations, avoiding costly autograd usage.
[Equation]
3. Q UANTIZATION , F USION AND S MOOTHING
Following projection, we ﬁrst quantize the ﬂoating point pixel coordinates ˜
pc f 2 into integer pixel co-ordinates pc f 2 .
This in general leads to quantization holes and duplicates.
The duplicates are handled with a variance conditioned depth buffer, such that the closest projected depth is used, provided that it’s variance is lower than a set threshold.
This in general prevents highly uncertain close depth values from overwriting highly certain far values.
We then perform per pixel fusion based on lines 6 and 7 in Algorithm 1 provided the depths fall within a set relative threshold, otherwise the minimum depth with sufﬁciently low variance is taken.
This again acts as a depth buffer.
Finally, we perform variance based image smoothing, whereby we treat each N × N image patch ( µ k,l ) k ∈{ 1 ,..,N } ,l ∈{ 1 ,..,N } as a collection of independent measurements of the central pixel, and combine their variance values based on central limit theory, resulting in smoothed values for each pixel in the image µ i,j .
Although we use this to update the mean belief, we do not smooth the variance values, meaning projection holes remain at prior variance.
This prevents the smoothing from distorting our belief during subsequent projections, and makes the smoothing inherently local to the current frame only.
The smoothing formula is as follows, with variance here denoted as σ 2 :
[Equation]
Given that the quantization is a discrete operation, we cannot compute it’s analytic jacobian for uncertainty propagation.
We therefore approximate the added quantization uncertainty using the numerical pixel gradients of the newly smoothed image G i,j , and assume additive noise proportional to the x and y quantization distances ∆ pc i,j :
[Equation]
3. N EURAL N ETWORK I NTEGRATION
The ESM module can be integrated anywhere into a wider CNN stack, forming an Egospheric Spatial
Memory Network (ESMN).
Throughout this paper we consider two variants, ESMN and ESMN-RGB, see Figure 2.
ESMN-RGB is a special case of ESMN, where RGB features are directly projected into memory, while ESMN projects CNN encoded features into memory.
The inclusion of polar angles, azimuthal angles and depth means the full relative polar coordinates are explicitly represented for each pixel in memory.
Although the formulation described in Algorithm 1 and Fig 1 allows for m vision sensors, the experiments in this paper all involve only a single acquiring sensor, meaning m = 1 .
We also only consider cases with constant variance in the acquired images V t = k var , and so we omit the variance images from the ESM input in Fig 2 for simplicity.
For baseline approaches, we compute an image of camera-relative coordinates via K − 1 , and then concatenate this to the RGB image along with the tiled incremental poses before input to the networks.
All values are normalized to 0 − 1 before passing to convolutions, based on the permitted range for each channel.
E XPERIMENTS
The goal of our experiments is to show the wide applicability of ESM to different embodied 3D learning tasks.
We test two different applications:
Image-to-action learning for multi-DOF control (Sec 4.1).
Here we consider drone and robot manipulator target reacher tasks using either ego-centric or scene-centric cameras.
We then assess the ability for ESMN policies to generalize between these different camera modalities, and assess the utility of the ESM geometry for obstacle avoidance.
We train policies both using imitation learning (IL) and reinforcement learning (RL).
Object segmentation (Sec 4.2).
Here we explore the task of constructing a semantic map, and the effect of changing the ESM module location in the computation graph on performance.
4. M ULTI -DOF V ISUOMOTOR C ONTROL
While ego-centric cameras are typically used when learning to navigate planar scenes from images (Jaderberg et al., 2016; Zhu et al., 2017; Gupta et al., 2017; Parisotto & Salakhutdinov, 2017), static scene-centric cameras are the de facto when learning multi-DOF controllers for robot manipulators (Levine et al., 2016; James et al., 2017; Matas et al., 2018; James et al., 2019b).
We consider the more challenging and less explored setup of learning multi-DOF visuomotor controllers from ego-centric cameras, and also from moving scene-centric cameras.
LSTMs are the de facto memory architecture in the RL literature (Jaderberg et al., 2016; Espeholt et al., 2018; Kapturowski et al., 2018; Mirowski et al., 2018; Bruce et al., 2018), making this a suitable baseline.
NTMs represent another suitable baseline, which have outperformed LSTMs on visual navigation tasks (Wayne et al., 2018).
Many other works exist which outperform LSTMs for planar navigation in 2D maze-like environments (Gupta et al., 2017; Parisotto & Salakhutdinov, 2017; Henriques & Vedaldi, 2018), but the top-down representation means these methods are not readily applicable to our multi-DOF control tasks.
LSTM and NTM are therefore selected as competitive baselines for comparison.
4.1. I MITATION L EARNING
For our imitation learning experiments, we test the utility the ESM module on two simulated visual reacher tasks, which we refer to as Drone Reacher ( DR ) and Manipulator Reacher ( MR ).
Both are implemented using the CoppeliaSim robot simulator (Rohmer et al., 2013), and its Python extension
PyRep (James et al., 2019a).
We implement DR ourselves, while MR is a modiﬁcation of the reacher task in RLBench (James et al., 2020).
Both tasks consist of 3 targets placed randomly in a simulated arena, and colors are newly randomized for each episode.
The targets consist of a cylinder, sphere, and "star", see Figure 3.
In both tasks, the target locations remain ﬁxed for the duration of an episode, and the agent must continually navigate to newly speciﬁed targets, reaching as many as possible in a ﬁxed time frame of 100 steps.
The targets are speciﬁed to the agent either as RGB color values or shape class id, depending on the experiment.
The agent does not know in advance which target will next be speciﬁed, meaning a memory of all targets and their location in the scene must be maintained for the full duration of an episode.
Both environments have a single bodyﬁxed camera, as shown in Figure 3, and also an external camera with freeform motion, which we use separately for different experiments.
For training, we generate an ofﬂine dataset of 100k 16-step sequences from random motions for both environments, and train the agents using imitation learning from known expert actions.
Action spaces of joint velocities ˙ q ∈ R 7 and cartesian velocities ˙ x ∈ R 6 are used for MR and DR respectively.
Expert translations move the end-effector or drone directly towards the target, and expert rotations rotate the egocentric camera towards the target via shortest rotation.
Expert joint velocities are calculated for linear end-effector motion via the manipulator Jacobian.
For all experiments, we compare to baselines of single-frame, dual-stacked LSTM with and without spatial auxiliary losses, and NTM. We condition the policy networks on both the image features and per-pixel relative cartesian co-ordinates.
We also compare against a network trained on partial oracle omni-directional images, masked at unobserved pixels, which we refer to as Partial-Oracle-Omni (PO2), as well as random and expert policies.
PO2 cannot see regions where the monocular camera has not looked, but it maintains a pixel-perfect memory of anywhere it has looked.
Full details of the training setups are provided in Appendix A.1.
The results for all experiments are presented in Table 1. [Figure] [Table] [Figure]
[Table]
ESMN-RGB stores RGB features in memory, while ESMN stores learnt features.
Ego-Centric Observations:
In this conﬁguration we take observations from body-mounted cameras.
We can see in Table 1 that for both DR and MR, our module signiﬁcantly outperforms other memory baselines, which do not explicitly incorporate geometric inductive bias.
Clearly, the baselines have difﬁculty in optimally interpreting the stream of incremental pose measurements and depth.
In contrast, ESM by design stores the encoded features in memory with meaningful indexing.
The
ESM structure ensures that the encoded features for each pixel are aligned with the associated relative polar translation, represented as an additional feature in memory.
When fed to the post-ESM convolutions, action selection can then in principle be simpliﬁed to target feature matching, reading the associated relative translations, and then transforming to the required action space.
A collection of short sequences of the features in memory for the various tasks are presented in Figure 4, with (a),
(b) and (d) coming from egocentric observations.
In all three cases we see the agent reach one target by the third frame, before re-orienting to reach the next.
We also observe that ESMN-RGB performs well when the network is conditioned on target color, but fails when conditioned on target shape id. This is to be expected, as the ability to discern shape from the memory is strongly inﬂuenced by the ESM resolution, quantization holes, and angular distortion. For example, the "star" shape in Figure 4 (a) is not apparent until t 5 .
However, ESMN is able to succeed, and starts motion towards this star at t 3 . The pre-ESM convolutional encoder enables ESMN to store useful encoded features in the ESM module from monocular images, within which the shape was discernible.
Figure 4 (a) shows the 3 most dominant ESM feature channels projected to RGB.
Freeform Scene-Centric Observations:
Here we explore the ability of ESM to generalize to unseen camera poses and motion, from cameras external to the agent.
The poses of these cameras are randomized for each episode during training, and follow random freeform rotations, with a bias to face towards the centre of the scene, and linear translations.
Again, we see that the baselines fail to learn successful policies, while ESM-augmented networks are able to solve the task.
The memories in these tasks take on a different proﬁle, as can be seen in Fig 4 (c) and (e).
While the memories from egocentric observations always contain information in the memory image centre, where the most recent monocular frame projects with high density, this is not the case for projections from arbitrarily positioned cameras which can move far from the agent, resulting in sparse projections into memory.
The targets in Fig 4 (e) are all represented by only 1 or 2 pixels.
The large apparent area in memory is a result of the variance-based smoothing, where the low-variance colored target pixels are surrounded by high-variance unobserved pixels in the ego-sphere.
Obstacle Avoidance: To further demonstrate the beneﬁts of a local spatial geometric memory, we augment the standard drone reacher task with obstacles, see Figure 5.
Rather than learning the avoidance in the policy, we exploit the interpretable geometric structure in ESM, and instead augment the policy output with a local avoidance component.
We then compare targets reached and collisions for different avoidance baselines, and test these avoidance strategies on random, ESMN-RGB and expert target reacher policies.
We see that the ESM geometry enables superior avoidance over using the most recent depth frame alone.
The obstacle avoidance results are presented in Table 2, and further details of the experiment are presented in Appendix A.2.
Camera Generalization: We now explore the extent to which policies trained from egocentric observations can generalize to cameras moving freely in the scene, and vice-versa.
The results of these transfer learning experiments are presented in Table 3.
Example image trajectories for both egocentric and free-form observations are presented in Figure 6.
Policy
Random
[Table]
Collisions
[Table]
Single
[Table]
Frame
[Table]
ESM Depth Map
[Table]
Truth
[Table]
[Figure]
[Table]
Shape
Color
Shape
Color
Shape
Color
Shape
LSTM
[Table]
Transferred
[Table]
4.1. R EINFORCEMENT L EARNING
Assuming expert actions in partially observable (PO) environments is inherently limited.
It is not necessarily true that the best action always rotates the camera directly to the next target for example.
In general, for ﬁnding optimal policies in PO environments, methods such as reinforcement learning (RL) must be used.
We therefore train both ESMN-RGB and an LSTM baseline on a simpler variant of the MR task via DQN (Mnih et al., 2015).
We refer to this variant as MR-Seq, where the manipulator must reach red, blue and then yellow spherical targets from egocentric observations, after which the episode terminates.
The only other difference to MR is that MR-Seq uses 128 ×images as opposed to 32 × 32 .
ESMN-RGB again outperforms the LSTM baseline, learning to reach all three targets, while the LSTM policy only succeeds in reaching one.
Full details of the RL setup and learning curves are given in Appendix A.3.
4. O BJECT S EGMENTATION
We now explore the suitability of the ESM module for object segmentation.
One approach is to perform image-level segmentation in individual monocular frames, and then perform probabilistic fusion when projecting into the map (McCormac et al., 2017).
We refer to this approach as Mono.
Another approach is to ﬁrst construct an RGB map, and then pass this map as input to a network.
This has the beneﬁt of wider context, but lower resolution is necessary to store a large map in memory, meaning details can be lost.
ESMN-RGB adopts this approach.
Zhi et al.
(2019) instead combine monocular predictions with multi-view optimization to gain the beneﬁts of wider surrounding context.
The ESMN architecture is also able to combine monocular inference with the wider map context, but does so by constructing a network with both image-level and map-level convolutions.
These ESM variants adopt the same broad architectures as shown in Fig 2, with the full networks speciﬁed in Appendix A.4.
We do not attempt to quote state-of-the-art results, but rather to further demonstrate the wide applications of the ESM module, and to explore the effect of placing the ESM module at different locations in the convolutional stack.
We evaluate segmentation accuracy based on the predictions projected to the ego-centric map.
With ﬁxed network capacity between methods, we see that ESMN outperforms both baselines, see Table 4 for the results, and Figure 7 for example predictions in a ScanNet reconstruction.
Further details are given in Appendix A.
[Table]
[Table]
C ONCLUSION
Through a diverse set of demonstrations, we have shown that ESM represents a widely applicable computation graph and trainable module for tasks requiring general spatial reasoning.
When compared to other memory baselines for image-to-action learning, our module outperforms these dramatically when learning both from ego-centric and scene-centric images.
One weakness of our method is that is assumes the availability of both incremental pose measurements of all scene cameras, and depth measurements, which is not a constraint of the other memory baselines.
However, with the ever increasing ubiquity of commercial depth sensors, and with there being plentiful streams for incremental pose measurements including visual odometry, robot kinematics, and inertial sensors, we argue that such measurements are likely to be available in any real-world application of embodied
AI.
We leave it to future work to investigate the extent to which ESM performance deteriorates with highly uncertain real-world measurements, but with full uncertainty propagation and probabilistic per pixel fusion, ESM is well suited for accumulating noisy measurements in a principled manner.
A A PPENDIX A. M ULTI -DOF I MITATION L EARNING A.1. O FFLINE D ATASETS
The image sequences for the ofﬂine datasets are captured following random motion of the agent in both the DR and MR tasks but known expert actions for each of the three possible targets in the scene are stored at every timestep.
The drone reacher is initialized in random locations at the start of each episode, whereas the manipulator reacher is always started in the same robot conﬁguration overlooking the workspace, as in the original RLBench reacher task. For the scene-centric acquisition, we instantiate three separate scene-centric cameras in the scene. In order to maximise variation in the dataset to encourage network generalization to arbitrary motions at test-time we reset the pose of each of these scene-centric cameras at every step of the episode, rather than having each camera follow smooth motions. Each new random pose has a rotational bias to face towards the scene-centre, to ensure objects are likely to be seen frequently.
By resetting the cameras poses on every time-step, we encourage the networks to learn to make sense of the pose information given to the network, rather than learning policies which fully rely on smoothly and slowly varying images. Both tasks use ego-centric cameras with a wider ﬁeld of view (FOV) than the scene-centric cameras. This is a common choice in robotics, where wide angle perception is especially necessary for bodymounted cameras. RLBench by default uses ego-centric FOV of 60 degrees and scene-centric FOV of 40 degrees, and we use the same values for our RLBench-derived MR task.
For the drone reacher task, we use ego-centric FOV of 90 degrees and scene-centric FOV of 55 degrees, to enable all methods to more quickly explore the perceptual ego-sphere.
For the manipulator reacher dataset, we also store a robot mask image, which shows the pixels corresponding to the robot for all ego-centric and scene-centric images acquired.
Known robot forward kinematics are used for generating the masking image.
All images in the ofﬂine dataset are 32 × 32 resolution.
A.1. T RAINING
To maximize the diversity from the ofﬂine datasets, a new target is randomly chosen from each of the three possible targets for each successive step in the unrolled time dimension in the training batch. This ensures maximum variation in sequences at train-time, despite a relatively small number of 100k 16-frame sequences stored in the ofﬂine dataset. This also strongly encourages each of the networks to learn to remember the location of every target seen so far, because any target can effectively be requested from the training loss at any time. Similarly, for the scene-centric cameras we randomly choose one of the three scene cameras at each successive time-step in the unrolled time dimension for maximum variation.
Again this forces the networks to make use of the camera pose information to make sense of the scene, and prevents overﬁtting on particular repeated sequences in the training set, instead encouraging generalization to fully arbitrary motions.
For these experiments, the baseline methods of Mono, LSTM, LSTM-Aux and NTM also receive the full absolute camera pose at each step rather than just incremental poses received by ESM, as we found this to improve the performance of the baselines.
For training the manipulator reacher policies, we additionally use the robot mask images to set high variance pixel values before feeding to the ESM module.
This prevents the motion of the robot from breaking the static-scene assumption adopted by ESM during re-projections.
We also provide the mask image as input to the baselines.
All networks use a batch size of 16, an unroll size of 16, and are trained for 250k steps using an
ADAM optimizer with 1 e − 4 learning rate.
None of the memory baselines cache the the internal state between training batches, and so the networks must learn to utilize the memory during the 16-frame unroll. 16 frames is on average enough steps to reach all 3 of the targets for both tasks.
A.1. N ETWORK A RCHITECTURES
The network architectures used in the imitation learning experiments are provided in Fig 8.
Both
LSTM baselines use dual stacked architectures with hidden and cell state sizes of 1024 .
For NTM we use a similar variant to that used by Wayne et al.
(2018), namely, we use sequential writing to the memory, and retroactive updates.
Regarding the 16-frame unroll, we again emphasize that 16 steps is on average enough time to reach all targets once.
In order to encourage generalization to longer sequences than only 16-steps, we limit the writable memory size to 10, and track the usage of these 10 memory cells with a usage indicator such that subsequent writes can preferentially overwrite the least used of the 10 cells.
This again is the same approach used by Wayne et al.
(2018), which is one of very few works to successfully apply NTM-style architectures to image-to-action domains.
It’s important to note that the use of retroactive updates makes the total memory size actually 20, as half of the cells are always reserved for the retroactive memory updates.
Regarding image padding at the borders for input to the convolutions, the Mono and LSTM/NTM baselines use standard zero padding, whereas ESMN-RGB and ESMN pad the outer borders with the wrapped omni-directional image.
A.1. A UXILIARY L OSSES
Motivated by the fact that many successful applications of LSTMs to image-to-actions learning involve the use of spatial auxiliary losses (Jaderberg et al., 2016; James et al., 2017; Sadeghi et al., 2017;
Mirowski et al., 2018), we also compare to an
LSTM which uses two such auxiliary proposals, namely the attention loss proposed in (Sadeghi et al., 2017) and a 3-dimensional Euler-based variant of the heading loss proposed in (Mirowski et al., 2018), which itself only considers 1D rotations normal to the plane of navigation.
Our heading loss does not compute the 1D rotational offset from North, as this is not detectable from the image input.
Instead, the networks are trained to predict the 3D Euler offset from the orientation of the ﬁrst frame in the sequence.
The modiﬁed LSTM network architecture is presented in Fig 9, and example images and classiﬁcation targets for the auxiliary attention loss are presented in Fig 10.
We emphasize that we did not attempt to tune these auxiliary losses, and applied them unmodiﬁed to the total loss function, taking the mean of the cross entropy loss for each, and linearly scaling so that the total auxiliary loss is roughly the same magnitude as the imitation loss.
Tuning auxiliary losses on different tasks is known to be challenging, and the losses can worsen performance without time-consuming manual tuning, as evidenced in the performance of the UNREAL agent (Jaderberg et al., 2016) compared to a vanilla RL-LSTM network in (Wayne et al., 2018).
We reproduce this general ﬁnding, and see that the untuned auxiliary losses do not improve performance on our reacher tasks.
In contrast, our
ESM module embeds strong inductive bias into the computation graph, requires no tuning at all, and consistently leads to successful policies on the different tasks.
A.1. F URTHER D ISCUSSION OF R ESULTS
The losses for each network evaluated on the training set and validation set during the course of training are presented in Fig 11.
We ﬁrst consider the results for the drone reacher task.
Firstly, we can clearly see from the DR-ego-rgb and DR-freeform-rgb tasks that the baselines struggle to interpret the stream of incremental pose measurements and depth, in order to select optimal actions in the training set, and this is replicated in the validation set, and in the ﬁnal task performance in Tab 1.
We can also see that ESMN is able to achieve lower training and validation losses than ESMN-RGB when conditioned on shape in the
DR-ego-shape and DR-freeform-shape tasks, and also expectedly achieves higher policy performance, shown in in Tab 1.
What we also observe is that the baselines have a higher propensity to over-ﬁt on training data.
Both the LSTM and NTM baselines achieve lower training set error than ESMN on the DR-ego-shape task, but not lower validation error.
In contrast, all curves for ESM-integrated networks are very similar between the training and validation set.
The ESM module by design performs principled spatial computation, and so these networks are inherently much more robust to overﬁtting.
Looking to the manipulator reacher task, we ﬁrst notice that the LSTM and NTM baselines are actually able to achieve lower losses than the ESM-integrated networks on both the training set and validation set for the MR-ego-rgb and MR-ego-shape tasks.
However, this does not translate to higher policy performance.
The reason for this is that the RLBench reacher task always initializes the robot in the same conﬁguration, and so the diversity in the ofﬂine dataset is less than that of the drone reacher ofﬂine dataset.
The scope of possible robot conﬁgurations in each 16-step window in the dataset is more limited.
In essence, the baselines are achieving well in both training and validation sets as a result of overﬁtting to the limited data distributions observed.
What these curves again highlight is the strong generalization power of ESM-integrated networks.
Despite seeing relatively limited robot conﬁgurations in the dataset, the ESM policies do not overﬁt on these, and are still able to use this data to learn general policies which succeed from unseen out-of-distribution images at test-time.
We also again observe the same superiority of ESMN over ESMN-RGB when conditioned on shape input in the training and validation losses.
A ﬁnal observation is that all methods fail to perform well on the MR-freeform-shape task.
We investigated this, and the weak performance is a combined result of low-resolution 32 × 32 images acquired in the training set and the large distance between the scene-centric cameras and the targets in the scene.
The shapes are often difﬁcult to discern from the monocular images acquired, and so little information is available for the methods to successfully learn a policy.
We expect that with higher resolution images, or with an average lower distance between the scene-cameras and workspace in the ofﬂine dataset, we would again observe the ESM superiority observed for all other tasks.
For this experiment, we increase the drone reacher environment by 2 × in all directions, resulting in an 8 × increase in volume.
We then also add 20 spherical obstacles into the scene with radius r = 0 .
1 m .
For the avoidance, we consider a bubble around the agent with radius R = 0 .
2 , and ﬂag a collision whenever any part of an obstacle enters this bubble.
Given the closest depth measurement available d closest , the avoidance algorithm simply computes an avoidant velocity vector v a whose magnitude is inversely proportional to the distance from collision, clipped to the maximum velocity | v | max . Equation 5 shows the calculation for the avoidance vector magnitude.
We run the avoidance controller at 10 × the rate of the ESM updates.
[Equation]
In order to prevent avoidant motion away from the targets to reach, we retrain the ESMN-RGB networks on the drone reacher task, but we train the network to also predict the full relative target location as an additional auxiliary loss.
When evaluating on the obstacle avoidance task, we prevent depth values within a ﬁxed distance of this predicted target location from inﬂuencing the obstacle avoidance.
This has the negative effect of causing extra collisions when the agent erroneously predicts that the target is close, but it enables the agent to approach and reach the target without being pushed away by the avoidance algorithm.
Regarding the performance against the baseline, we re-iterate that all monocular images have a very large ﬁeld-of-view of 90 degrees, and yet we still observe signiﬁcant reductions in collisions when using the full ESM geometry for avoidance, see Tab 2.
A. M ULTI -DOF R EINFORCEMENT L EARNING A.3. T RAINING
For the reinforcement learning experiment, we train both ESMN-RGB and an LSTM baseline on a similar sequential target reacher task as deﬁned in Section 4.1 via DQN (Mnih et al., 2015), where the manipulator must reach red, blue and then yellow targets from egocentric observations.
We use (128 × 128) images in this experiment rather than (32 × 32) as used in the imitation learning experiments.
We also use an unroll length of 8 rather than 16.
We use discrete delta end-effector translations, with ± 0 .
05 meters for each axis, with no rotation (resulting in an action size of 6).
We use a shaped reward of r = − len ( remaining _ targets ) −(cid:107) e − g (cid:107) 2 , where e and g are the gripper and current target translation respectively. A.3. N ETWORK A RCHITECTURES
In order to make the RL more tractable, and enable larger batch sizes, we use smaller network than used in the imitation learning experiments.
Both methods use a Siamese network to process the RGB and camera coordinate inputs separately, and consist of 2 convolution (conv) layers with 16 and 32 channels (for each branch).
We fuse these branches with a 32 channel conv and 1x1 kernel.
The remainder of the architecture then follows the same as in the imitation learning experiments, but we instead use channel sizes of 64 throughout.
The network output are 6 values corresponding to the Q-values for each action.
Both methods use a learning rate of 0 . 001 , target Q-learning τ = 0 .
001 , batch size of 128 , and leakyReLU activations.
We use an epsilon greedy strategy for exploration that is decayed over 100k training steps from 1 to 0.1.
We show the average return during RL training on the sequential reacher task over 5 seeds in Fig 12.
The ESMN-RGB policy succeeds in reaching alltargets on average, whereas the LSTM policy only succeeds in reaching 1.
A. OBJECT S EGMENTATION A.4. D ATASET
For the object segmentation experiment, we use down-sampled 60 × 80 and 120 × 160 images from the ScanNet dataset, which we ﬁrst RGB-Depth align.
We use a reduced dataset with frame skip of 30 to maximize diversity whilst minimizing dataset memory.
Many sequences contain slow camera motion, resulting in adjacent frames which vary very little.
We use the Eigen-13 classiﬁcation labels as training targets.
A.4. N ETWORK A RCHITECTURES
The Mono, ESMN-RGB and ESMN networks all exhibit a U-Net architecture, and output object segmentation predictions in an ego-sphere map.
ESMN-RGB and ESMN do so with a U-Net connecting the ESM output to the ﬁnal predictions, and Mono does so by projecting and probabilistically fusing the monocular segmentation predictions in a non-learnt manner.
The network architectures are all presented in Fig 13.
Regarding image padding at the borders for input to the convolutions, the Mono and LSTM/NTM baselines use standard zero padding, whereas ESMN pads the outer borders with the wrapped omni-directional image.
[Figure]
For training, all losses are computed in the ego-sphere map frame of reference, either following convolutions for ESMN and ESMNRGB, or following projection and probabilistic fusion for the Mono case.
We compute ground-truth segmentation training target labels by projecting the ground truth monocular frames to form an ego-sphere target segmentation image, see the right-hand-side of Fig 7 for an example.
We chose this approach over computing the ground truth segmentations from the complete ScanNet meshes for implementational simplicity.
All experiments use a batch size of 16, unroll size of 8 in the time dimension, and Adam optimizer with learning rate 1 e − 4 , trained for 250k steps.