E ND - TO -E ND E GOSPHERIC S PATIAL M EMORY
Anonymous authors
Paper under double-blind review
A BSTRACT
Spatial memory, or the ability to remember and recall speciﬁc locations and objects, is central to autonomous agents’ ability to carry out tasks in real environments.
However, most existing artiﬁcial memory modules have difﬁculty recalling information over long time periods and are not very adept at storing spatial information.
We propose a parameter-free module, Egospheric Spatial Memory (ESM), which encodes the memory in an ego-sphere around the agent, enabling expressive 3D representations.
ESM can be trained end-to-end via either imitation or reinforcement learning, and improves both training efﬁciency and ﬁnal performance against other memory baselines on both drone and manipulator visuomotor control tasks.
The explicit egocentric geometry also enables us to seamlessly combine the learned controller with other non-learned modalities, such as local obstacle avoidance.
We further show applications to semantic segmentation on the ScanNet dataset, where ESM naturally combines image-level and map-level inference modalities.
Through our broad set of experiments, we show that ESM provides a general computation graph for embodied spatial reasoning, and the module forms a bridge between real-time mapping systems and differentiable memory architectures.
I NTRODUCTION
Egocentric spatial memory is central to our understanding of spatial reasoning in biology (Klatzky, 1998; Burgess, 2006), where an embodied agent constantly carries with it a local map of its surrounding geometry.
Such representations have particular signiﬁcance for action selection and motor control (Wang et al., 2018; Hinman et al., 2019).
For robotics and embodied AI, the beneﬁts of a persistent local spatial memory are also clear.
Such a system has the potential to run indeﬁnitely, and bypass both the memory and runtime complexities of large scale world-centric mapping.
Peters et al. (2001) propose an EgoSphere as being a particularly suitable representation for robotics, and more recent works have utilized ego-centric formulations for planar robot mapping (Fankhauser et al., 2014), drone obstacle avoidance (Fragoso et al., 2018) and mono-to-depth from video (Liu et al., 2019).
In parallel with these explicit ego-centric mapping systems, a new paradigm of differentiable memory architectures has arisen, where an explicit memory bank is augmented to a neural network, which can then learn read and write operations (Weston et al., 2014; Graves et al., 2014; Sukhbaatar et al., 2015).
When compared to Recurrent Neural Networks (RNNs), the persistent memory circumvents issues of vanishing or exploding gradients, enabling solutions to complex long-horizon tasks.
These have also been applied to visuomotor control and navigation tasks (Wayne et al., 2018), surpassing baselines such as the ubiquitous Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997).
We focus on the intersection of these two branches of research, and propose Egospheric Spatial
Memory (ESM), a parameter-free module which encodes the memory in an ego-sphere around the agent.
To the best of our knowledge, ESM is the ﬁrst end-to-end trainable egocentric memory with full panoramic representation, enabling direct encoding of the surrounding scene in a 2.5D image. We emphasize the suitability of an ego-centric mapping computation graph for propagating gradients over long time horizons in an end-to-end manner, enabling an ego-centric memory of learned features.
We demonstrate the superiority of learning features through the ESM module on both target shape reaching and object segmentation tasks.
For other visuomotor control tasks, we show that even without learning features through the module, and instead directly projecting image color values into memory, ESM consistently outperforms other memory baselines.
Through these experiments, we show that the applications of our parameter-free ESM module are widespread, where it can either be dropped into existing pipelines as a non-learned module, or end-to-end trained in a larger computation graph, depending on the task requirements.
R ELATED W ORK 2. M APPING
Geometric mapping is a mature ﬁeld, with many solutions available for constructing high quality maps.
Such systems typically maintain an allocentric map, either by projecting points into a global world co-ordinate system (Newcombe et al., 2011a; Whelan et al., 2015), or by maintaining a certain number of keyframes in the trajectory history (Newcombe et al., 2011b; Zhou et al., 2018;
Bloesch et al., 2018; Clark et al., 2018)
If these systems are to be applied to life-long embodied AI, then strategies must then be adopted in order to effectively select the parts of the map which are useful, and discard the rest from memory (Cadena et al., 2016). Taking CodeSLAM (Bloesch et al., 2018) as an example, keyframe removal strategies are needed to prevent the map size from growing indeﬁnitely.
For robotics applications, prioritizing geometry in the immediate vicinity is a sensible prior.
Rather than taking a world-view to map construction, such systems often formulate the mapping problem in a purely ego-centric manner, performing continual re-projection to the newest frame and pose with ﬁxed-sized storage.
Peters et al.
(2001) outline an EgoSphere memory structure, with indexing via polar and azimuthal angles, with applications to humanoid robotics.
Fankhauser et al.
(2014) use ego-centric height maps, and demonstrate on a quadrupedal robot walking over obstacles.
Cigla et al.
(2017) use per-pixel depth Gaussian Mixture Models (GMMs) to maintain an ego-cylinder of belief around a drone, with applications to collision avoidance (Fragoso et al., 2018).
In a different application, Liu et al.
(2019) learn to predict depth images from a sequence of RGB images, again using ego reprojections.
These systems are designed to represent only at the level of depth and RGB features.
For mapping more expressive implicit features via end-to-end training, a fully differentiable long-horizon computation graph is required.
Any computation graph which satisﬁes this requirement is generally referred to as memory in the neural network literature.
2. M EMORY
The concept of memory in neural networks is deeply coupled with recurrence.
Naive recurrent networks have vanishing and exploding gradient problems (Hochreiter, 1998), which LSTMs (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Units (GRUs) (Cho et al., 2014)
mediate using additive gated structures.
More recently, dedicated differentiable memory blocks have become a popular alternative.
Weston et al.
(2014) applied Memory Networks (MemNN) to question answering, using hard read-writes and separate training of components.
Graves et al.
(2014) and Sukhbaatar et al.
(2015) instead made the read and writes ‘soft’ with the proposal of Neural Turing Machines (NTM) and
End-to-End Memory Networks (MemN2N) respectively, enabling joint training with the controller.
Other works have since conditioned dynamic memory on images, for tasks such as visual question answering (Xiong et al., 2016) and object segmentation (Oh et al., 2019).
Another distinct but closely related approach is self attention (Vaswani et al., 2017).
These approaches also use key-based content retrieval, but do so on a history of previous observations with adjacent connectivity.
Despite the lack of geometric inductive bias, recent results demonstrate the amenability of general memory (Wayne et al., 2018) and attention (Parisotto et al., 2019) to visuomotor control and navigation tasks.
Other authors have explored the intersection of network memory and spatial mapping for navigation, but have generally been limited to 2D aerial-view maps, focusing on planar navigation tasks.
Gupta et al.
(2017) used an implicit ego-centric memory which was updated with warping and conﬁdence maps for discrete action navigation problems.
Parisotto & Salakhutdinov (2017) proposed a similar setup, but used dedicated learned read and write operations for updates, and tested on simulated Doom environments. Without consideration for action selection, Henriques & Vedaldi (2018) proposed a similar system, but instead used an allocentric formulation, and tested on free-form trajectories of real images. Zhang et al. (2018) propose a similar system, but with the inclusion of loop closure.
Our memory instead focuses on local perception, with the ability to represent detailed 3D geometry in all directions around the agent.
The beneﬁts of our module are complementary to existing 2D methods, which instead focus on occlusion aware planar understanding suitable for navigation.
M ETHOD
In this section, we describe the formulation of the memory module.
The module operates as an
Extended Kalman Filter (EKF), with an egosphere image µ t ∈ R h s × w s × (2+1+ n ) and its diagonal covariance Σ t ∈ R h s × w s × (1+ n ) representing the state.
The egosphere image consists of 2 channels for the polar and azimuthal angles, 1 for radial depth, and n for encoded features.
The angles are not included in the covariance, as their values are implicit in the egosphere image pixel indices.
The covariance only represents the uncertainty in depth and features at these ﬁxed equidistant indices.
Image measurements are assumed to be captured from projective depth cameras, which similarly store 1 channel for depth and n for encoded features.
We also assume incremental agent pose measurements u t ∈ R 6 with covariance Σ u t ∈ R 6 × 6 are available, in the form of a translation and rotation vector.
The algorithm overview is presented in Algorithm 1.
1 Given: f m , F m , f o , F o 2 ¯ µ t = f m ( u t , µ t − 1 ) 3 ¯Σ t = F m ( u t , µ t − 1 , Σ t − 1 , Σ u t ) 4 ˆ µ t
=
f
o (( v ti , p ti ) i ∈ I ) 5 ˆΣ t =
F o (( v ti , p ti , V ti , P ti ) i ∈ I ) 6 K t =
¯Σ t (cid:104) ¯Σ t + ˆΣ t (cid:105) −7 µ t = ¯ µ t + K t [ˆ µ t − ¯ µ t ] 8 Σ t = [ I − K t ] ¯Σ t 9 return µ t , Σ t First, the motion step takes the state from the previous frame µ t − 1 , Σ t − 1 and transforms this into a predicted state for the current frame ¯ µ t , ¯Σ t via functions f m , F m and the incremental pose measurement u t with covariance Σ u t .
Then in the observation step, we use measured visual features ( v ti ∈ R h vi × w vi × (1+ n ) )
i ∈{ 1 ,...,m } with diagonal covariances (
V ti ∈
R
h
vi × w vi ×
(1+ n ) )
i ∈{ 1 ,...,m } originated from m arbitrary vision sensors, and associated pose measurements ( p ti ∈ R 6 ) i ∈{ 1 ,...,m } with covariances ( P ti ∈ R 6 × 6 ) i ∈{ 1 ,...,m } , to produce a new observation of the state ˆ µ t ∈ R h s × w s × (2+1+ n ) , again with diagonal covariance ˆΣ t ∈ R h s × w s × (1+ n ) , via functions f o and F o .
The measured poses also take the form of a translation and rotation vector.
Finally, the update step takes our state prediction ¯ µ t , ¯Σ t and state observation ˆ µ t , ˆΣ t , and fuses them to produce our new state belief µ t , Σ t .
We spend the remainder of this section explaining the form of the constituent functions.
All functions in Algorithm 1 involve re-projections across different image frames, using forward warping.
Functions f m , F m , f o and F o are therefore all built using the same core functions.
While the re-projections could be solved using a typical rendering pipeline of mesh construction followed by rasterization, we instead choose a simpler approach and directly quantize the pixel projections with variance-based image smoothing to ﬁll in quantization holes.
An overview of the projection and quantization operations for a single ESM update step is shown in Fig.
3. F ORWARD W ARPING
Forward warping projects ordered equidistant homogeneous pixel co-ordinates pc f 1 from frame f 1 to non-ordered non-equidistant homogeneous pixel co-ordinates ˜ pc f 2 in frame f 2 .
We use ˜ µ f 2 = { ˜ φ f 2 , ˜ θ f 2 , ˜ d f 2 , ˜ e f 2 } to denote the loss of ordering following projection from µ f 1 = { φ f 1 , θ f 2 , d f 1 , e f 2 } , where φ , θ , d and e represent polar angles, azimuthal angles, depth and encoded features respectively.
We only consider warping from projective to omni cameras, which corresponds to functions
f
o , F o , but the omni-to-omni case as in f m , F m is identical except with the inclusion of another polar co-ordinate transformation.
The encoded features are assumed constant during projection ˜
e f 2 = e f 1 .
For depth, we must transform the values to the new frame in polar co-ordinates, which is a composition of a linear transformation and non-linear polar conversion.
Using the camera intrinsic matrix K 1 , the full projection is composed of a scalar multiplication with homogeneous pixel co-ordinates pc f 1 , transformation by camera inverse matrix K − 1 1 and frame-to-frame T 12 matrices, and polar conversion f p :
[Equation]
Combined, this provides us with both the forward warped image ˜ µ f 2 = { ˜ φ f 2 , ˜ θ f 2 , ˜ d f 2 , ˜
e f 2 } , and the newly projected homogeneous pixel co-ordinates ˜ pc f {+
+} = { k ppr ˜ φ f 2 , k ppr ˜ θ f 2 , 1 }, where k ppr denotes the pixels-per-radian resolution constant.
The variances are also projected using the full analytic Jacobians, which are efﬁciently implemented as tensor operations, avoiding costly autograd usage.
[Equation]
3. Q UANTIZATION , F USION AND S MOOTHING
Following projection, we ﬁrst quantize the ﬂoating point pixel coordinates ˜ pc f 2 into integer pixel co-ordinates pc f 2 .
This in general leads to quantization holes and duplicates.
The duplicates are handled with a variance conditioned depth buffer, such that the closest projected depth is used, provided that it’s variance is lower than a set threshold.
This in general prevents highly uncertain close depth values from overwriting highly certain far values.
We then perform per pixel fusion based on lines 6 and 7 in Algorithm 1 provided the depths fall within a set relative threshold, otherwise the minimum depth with sufﬁciently low variance is taken.
This again acts as a depth buffer. Finally, we perform variance based image smoothing, whereby we treat each N × N image patch ( µ k,l ) k ∈{ 1 ,..,N } ,l ∈{ 1 ,..,N } as a collection of independent measurements of the central pixel, and combine their variance values based on central limit theory, resulting in smoothed values for each pixel in the image µ i,j .
Although we use this to update the mean belief, we do not smooth the variance values, meaning projection holes remain at prior variance.
This prevents the smoothing from distorting our belief during subsequent projections, and makes the smoothing inherently local to the current frame only.
The smoothing formula is as follows, with variance here denoted as σ 2 :
[Equation]
Given that the quantization is a discrete operation, we cannot compute it’s analytic jacobian for uncertainty propagation.
We therefore approximate the added quantization uncertainty using the numerical pixel gradients of the newly smoothed image G i,j , and assume additive noise proportional to the x and y quantization distances ∆ pc i,j :
[Equation]
3. N EURAL N ETWORK I NTEGRATION
The ESM module can be integrated anywhere into a wider CNN stack, forming an Egospheric Spatial
Memory Network (ESMN).
Throughout this paper we consider two variants, ESMN and ESMN-RGB, see Figure 2.
ESMN-RGB is a special case of ESMN, where RGB features are directly projected into memory, while ESMN projects CNN encoded features into memory.
The inclusion of polar angles, azimuthal angles and depth means the full relative polar coordinates are explicitly represented for each pixel in memory. Although the formulation described in Algorithm 1 and Fig 1 allows for m vision sensors, the experiments in this paper all involve only a single acquiring sensor, meaning m = 1 . We also only consider cases with constant variance in the acquired images V t = k var , and so we omit the variance images from the ESM input in Fig 2 for simplicity. For baseline approaches, we compute an image of camera-relative coordinates via K − 1 , and then concatenate this to the
RGB image along with the tiled agent incremental pose before input to the networks.
All values are normalized to 0 − 1 before passing to convolutions, based on the permitted range for each channel.
The goal of our experiments is to show the wide applicability of ESM to different embodied 3D learning tasks, where ESM outperforms existing memory baselines. We test two different applications: Image-to-action learning for multi-DOF control (Sec 4.1).
Here
we consider drone and robot manipulator target reacher tasks using either ego-centric or scene-centric cameras.
We then assess the ability for ESMN policies to generalize between these different camera modalities, and assess the utility of the ESM geometry for obstacle avoidance.
We train policies both using imitation learning (IL) and reinforcement learning (RL). Object segmentation (Sec 4.2). Here we explore the task of constructing a semantic map, and the effect of changing the ESM module location in the computation graph on performance. 4. M ULTI -DOF V ISUOMOTOR C ONTROL
While ego-centric cameras are typically used when learning to navigate planar scenes from images (Mnih et al., 2016; Jaderberg et al., 2016; Zhu et al., 2017; Gupta et al., 2017; Parisotto & Salakhutdinov, 2017), static scene-centric cameras are the de facto when learning multi-DOF controllers for robot manipulators (Levine et al., 2016; James et al., 2017; Matas et al., 2018; James et al., 2019b).
We consider the more challenging and less explored setup of learning multi-DOF visuomotor controllers from ego-centric cameras, and also from moving scene-centric cameras.
LSTMs are the de facto memory architecture in the RL literature (Mnih et al., 2016; Jaderberg et al., 2016;
Espeholt et al., 2018; Kapturowski et al., 2018), making this a suitable baseline.
NTMs represent another suitable baseline, which have outperformed LSTMs on visual navigation tasks (Wayne et al., 2018).
Many other works exist which outperform LSTMs for planar navigation in 2D maze-like environments (Gupta et al., 2017; Parisotto & Salakhutdinov, 2017; Henriques & Vedaldi, 2018), but the top-down representation means these methods are not readily applicable to our multi-DOF control tasks. LSTM and NTM are therefore selected competitive baselines for comparison. 4.1. I MITATION L EARNING For our imitation learning experiments, we test the utility the ESM module on two simulated visual reacher tasks, which we refer to as Drone Reacher ( DR ) and Manipulator Reacher ( MR ).
Both are implemented using the CoppeliaSim robot simulator (Rohmer et al., 2013), and its Python extension PyRep (James et al., 2019a).
We implement DR ourselves, while MR is a modiﬁcation of the reacher task in RLBench (James et al., 2020).
Both tasks consist of 3 targets placed randomly in a simulated arena, and colors are newly randomized for each episode.
The targets consist of a cylinder, sphere, and "star", see Figure 3.
In both tasks, the target locations remain ﬁxed for the duration of an episode, and the agent must continually navigate to newly speciﬁed targets, reaching as many as possible in a ﬁxed time frame of 100 steps.
The targets are speciﬁed to the agent either as RGB color values or shape class id, depending on the experiment.
The agent does not know in advance which target will next be speciﬁed, meaning a memory of all targets and their location in the scene must be maintained for the full duration of an episode.
Both environments have a single bodyﬁxed camera, as shown in Figure 3, and also an external camera with freeform motion, which we use separately for different experiments.
For training, we generate an ofﬂine dataset of 100k 16-step sequences from random motions for both environments, and train the agents using imitation learning from known expert actions.
Action spaces of joint velocities ˙ q ∈ R 7 and cartesian velocities ˙ x ∈ R 6 are used for MR and DR respectively.
Expert translations move the end-effector or drone directly towards the target, and expert rotations rotate the egocentric camera towards the target via shortest rotation.
Expert joint velocities are calculated for linear end-effector motion via the manipulator Jacobian.
For all experiments, we compare to baselines of single-frame, dual-stacked LSTM, and NTM, and condition the policy networks on both the image features and per-pixel relative cartesian co-ordinates.
We also compare against a network trained on partial oracle omni-directional images, masked at unobserved pixels, which we refer to as Partial-Oracle-Omni (PO2), as well as random and expert policies.
PO2 cannot see regions where the monocular camera has not looked, but it maintains a pixel-perfect memory of anywhere it has looked.
Full details of the training setups are provided in Appendix A.1.
The results for all experiments are presented in Table 1.
[Table]
ESMN-RGB stores RGB features in memory, while ESMN stores learnt features. Ego-Centric Observations:
In this conﬁguration we take observations from body-mounted cameras.
We can see in Table 1 that for both DR and MR, our module signiﬁcantly outperforms other memory baselines, which do not explicitly incorporate geometric inductive bias.
Clearly, the baselines have difﬁculty in optimally interpreting the stream of incremental pose measurements and depth.
In contrast, ESM by design stores the encoded features in memory with meaningful indexing.
The ESM structure ensures that the encoded features for each pixel are aligned with the associated relative polar translation, represented as an additional feature in memory.
When fed to the post-ESM convolutions, action selection can then in principle be simpliﬁed to target feature matching, reading the associated relative translations, and then transforming to the required action space.
A collection of short sequences of the features in memory for the various tasks are presented in Figure 4, with (a), (b) and (d) coming from egocentric observations.
In all three cases we see the agent reach one target by the third frame, before re-orienting to reach the next.
We also observe that ESMN-RGB performs well when the network is conditioned on target color, but fails when conditioned on target shape id. This is to be expected, as the ability to retrieve shape information from the memory is strongly inﬂuenced by the ESM resolution, quantization holes, and strong angular distortion.
However, with ESMN we train end-to-end with a pre-ESM convolutional encoder, and we can see that the issues are alleviated, with the network learning to store useful network encoded features in memory.
We show the activations of the 3 most dominate feature channels projected to RGB space during a short sequence in Figure 4 (a).
Scene-Centric Observations:
In this conﬁguration we use images from cameras external to the agent.
The poses of these cameras are randomized for each episode during training, and follow random freeform motion, with a rotational bias to face towards the centre of the scene.
Here we explore the ability of ESM to generalize to unseen camera poses and motion.
Again, we see that the baselines fail to learn successful policies, while ESM-augmented networks are able to solve the task.
The memories in these tasks take on a different proﬁle, as can be seen in Fig 4 (c) and (e).
While the memories from egocentric observations always contain information in the memory image centre, where the most recent monocular frame projects with high density, this is not the case for projections from arbitrarily positioned cameras.
External cameras can also move far from the agent, resulting in sparse projections into memory. The targets in Fig 4 (e) are all represented by only 1 or 2 pixels. The large apparent area in memory is a result of the variance-based smoothing, where the low-variance colored target pixels are surrounded by high-variance unobserved pixels in the ego-sphere. Obstacle Avoidance: To further demonstrate the beneﬁts of a local spatial geometric memory, we augment the standard drone reacher task with obstacles, see Figure 5.
Rather than learning the avoidance in the policy, we exploit the interpretable geometric structure in ESM, and instead augment the policy output with a local avoidance component. We then compare targets reached and collisions for different avoidance baselines, and test these avoidance strategies on random, ESMN-RGB and expert target reacher policies. We see that the ESM geometry enables superior avoidance over using the most recent depth frame alone. The obstacle avoidance results are presented in Table 2, and further details of the experiment are presented in Appendix A.2.
Generalization performance: We now explore the extent to which policies trained from egocentric observations can generalize to cameras moving freely in the scene, and vice-versa. The results of these transfer learning experiments are presented in Table 3. Example image trajectories for both egocentric and free-form observations are presented in Figure 6. Policy
Random
[Table]
Expert
Reached
Collisions
Reached
Collisions
Reached
Collisions
[Table]
i d
[Figure]
n ce
No Avoidance
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Single Depth Frame
[Table]
[Table]
[Table]
[Table]
ESM Depth Map
[Table]
[Table]
Ground Truth
[Table]
Drone Reacher
[Figure]
[Table]
[Figure]
Color
Shape
Color
Shape
Color
Shape Color Shape LSTM12.7(3.4) 4.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
4) Transfer 0.3(0.6) 0.5(0.7) 0.
[Table]
[Table]
[Table]
[Table] [Table] 4.1. R EINFORCEMENT L EARNING Assuming expert actions in partially observable (PO) environments is inherently limited.
It is not necessarily true that the best action always rotates the camera directly to the next target for example.
In general, for ﬁnding optimal policies in PO environments, methods such as reinforcement learning (RL) must be used.
We therefore train both ESMN-RGB and an LSTM baseline on a simpler variant of the MR task via DQN (Mnih et al., 2015). We refer to this variant as MR-Seq, where the manipulator must reach red, blue and then yellow spherical targets from egocentric observations, after which the episode terminates.
The only other difference to MR is that MR-Seq uses 128 ×images as opposed to 32 × 32 .
ESMN-RGB again outperforms the LSTM baseline, learning to reach all three targets, while the LSTM policy only succeeds in reaching one.
Full details of the RL setup and learning curves are given in Appendix A.3.
4. O BJECT S EGMENTATION
We now explore the suitability of the ESM module for object segmentation.
One approach is to perform image-level segmentation only in individual monocular frames, and then perform probabilistic fusion when projecting into the map (McCormac et al., 2017). We refer to this approach as Mono.
Another approach is to ﬁrst construct an RGB map, and then pass this map as input to a network. This has the beneﬁt of wider context, but lower resolution is necessary to store a large map in memory, meaning details can be lost.
ESMN-RGB adopts this approach. Zhi et al. (2019) take a different approach, and combine monocular predictions with multi-view optimization to gain the beneﬁts of wider surrounding context.
With ESMN, we construct a network with both image-level and map-level convolutions, which similarly offers the ability to combine monocular inference with the wider map context.
These variants adopt the same architectures as shown in Fig 2, with the full networks speciﬁed in Appendix A.4.
We do not attempt to quote state-of-the-art results, but rather to further demonstrate the wide applications of the ESM module, and to further explore the effect of placing the ESM module at different locations in the convolutional stack.
We evaluate segmentation accuracy based on the predictions projected to the ego-centric map.
With ﬁxed network capacity between methods, we see that ESMN outperforms both baselines, see Table 4 for the results, and
Figure 7 for example predictions in a ScanNet reconstruction.
Further details of this experiment are given in Appendix A.4. [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
as ESMN- and ESMN
C ONCLUSION
Through a diverse set of demonstrations, we have shown that ESM represents a widely applicable computation graph, and trainable module for tasks requiring general spatial reasoning.
When compared to other memory baselines for image-to-action learning, our module outperforms these dramatically when learning both from ego-centric and scene-centric images.
One weakness of our method is that is assumes the availability of both incremental pose measurements of all scene cameras, and depth measurements, which is not a constraint of the other memory baselines.
However, with the ever increasing ubiquity of commercial depth sensors, and with there being plentiful streams for incremental pose measurements including visual odometry, robot kinematics, and inertial sensors, we argue that such measurements are likely to be available in any real-world application of embodied AI.
We leave it to future work to investigate the extent to which ESM performance deteriorates with highly uncertain real-world measurements, but with full uncertainty propagation and probabilistic per pixel fusion, ESM is well suited for accumulating noisy measurements in a principled manner.
A A PPENDIX A. M ULTI -DOF I MITATION L EARNING A.1. O FFLINE D ATASETS
The image sequences for the ofﬂine datasets are captured following random motion of the agent in both the DR and MR tasks, but known expert actions for each of the three possible targets in the scene are stored at every timestep. At training time, a new target is then randomly chosen from each of the three possibilities for each successive step in the unrolled time dimension in the training batch. This ensures maximum variation in sequences at train-time, despite a relatively small number of only 100k 16-frame sequences stored in the ofﬂine dataset. All images in the dataset are 32 × 32 resolution.
All networks use a batch size of 16, an unroll size of and are trained for 250k steps using an ADAM optimizer with 1 e − 4 learning rate. None of the memory baselines cache the the internal state between training batches, and so the networks must learn to utilize the memory during the 16-frame unroll.
The network architectures used in the imitation learning experiments are provided in Fig 8. Both
LSTM baselines use dual stacked architectures. For NTM we use a similar variant to that used by Wayne et al. (2018), namely, we use sequential appending to the memory, and retroactive updates.
Regarding the 16-frame unroll, we ﬁnd that 16 steps is on average enough time to reach all targets once.
In order to encourage generalization to longer sequences than only 16-steps, we limit the memory size to 10, and track the usage of these 10 memory cells such that subsequent writes can preferentially overwrite the least used of the 10 cells. This again is the same approach used by
Wayne et al.
(2018), which is one of very few works to successfully apply NTM-style architectures to image-to-action domains. Regarding image padding at the borders for input to the convolutions, the Mono and LSTM/NTM baselines use standard zero padding, whereas ESMN-RGB and ESMN pad the outer borders with the wrapped omni-directional image.
A. O BSTACLE A VOIDANCE
For this experiment, we increase the drone reacher environment by 2 × in all directions, resulting in an 8 × increase in volume. We then also add 20 spherical obstacles into the scene with radius r = 0 . 1 m . For the avoidance, we consider a bubble around the agent with radius R = 0 .
2 , and ﬂag a collision whenever any part of an obstacle enters this bubble. Given the closest depth measurement available d closest , the avoidance algorithm simply computes an avoidant velocity vector v a whose magnitude is inversely proportional to the distance from collision, clipped to the maximum velocity | v | max , Eq 5 shows the calculation for the avoidance vector magnitude. We run the avoidance controller at 10 × the rate of the ESM updates. [Equation]
In order to prevent avoidant motion away from the targets to reach, we retrain the ESMN-RGB networks on the drone reacher task, but we train the network to also predict the full relative target location as an additional auxiliary loss.
When evaluating on the obstacle avoidance task, we prevent depth values within a ﬁxed distance of this predicted target location from inﬂuencing the obstacle avoidance.
This has the negative effect of causing extra collisions when the agent erroneously predicts that the target is close, but it enables the agent to approach and reach the target without being pushed away by the avoidance algorithm.
We train both ESMN-RGB and an LSTM baseline on a similar sequential target reacher task as deﬁned in Section ?? via DQN (Mnih et al., 2015), where the manipulator must reach red, blue and then yellow targets from egocentric observations.
We use (128 × 128) images in this experiment rather than (32 × 32) as used in the imitation learning experiments. We also use an unroll length of 8 rather than 16.
We use discrete delta end-effector translations, with ± 0 . 05 meters for each axis, with no rotation (resulting in an action size of 6).
We use a shaped reward of r = − len ( remaining _ targets ) −(cid:107) e − g (cid:107) 2 , where e and g are the gripper and current target translation respectively.
In order to make the RL more tractable, we use smaller network than used in the imitation learning experiments.
Both methods use a Siamese network to process the RGB and camera coordinate inputs separately, and consist of 2 convolution (conv) layers with 16 and 32 channels (for each branch).
We fuse these branches with a 32 channel conv and 1x1 kernel.
The remainder of the architecture then follows similarly to the imitation learning experiments, but we instead use channel sizes ofthroughout.
The network output are 6 values corresponding to the Q-values for each action.
Both methods use a learning rate of 0 . 001 , target Q-learning τ = 0 .
001 , batch size of 128 , and leakyReLU activations.
We use an epsilon greedy strategy for exploration that is decayed over 100k training steps from 1 to 0.1.
We show the average return during RL training on the sequential reacher task overseeds in Fig 9.
A. OBJECT S EGMENTATION
[Figure]
For the object segmentation experiment, we use both downsampled 60 × 80 and 120 × 160 images from the ScanNet dataset, which we ﬁrst RGB-Depth align.
We use a reduced dataset with frame skip of 30 to maximize diversity whilst minimizing dataset memory. Many sequences contain slow camera motion, resulting in adjacent frames which vary very little.
The Mono, ESMN-RGB and ESMN networks all exhibit a UNet architecture, and output object segmentation predictions in an ego-sphere map.
ESMN-RGB and ESMN do so with a U-Net connecting the ESM output to the ﬁnal predictions, and Mono does so by projecting and probabilistically fusing the monocular segmentation predictions in a non-learnt manner. The network architectures are all presented in Fig 10.
Regarding image padding at the borders for input to the convolutions, the Mono and LSTM/NTM baselines use standard zero padding, whereas ESMN pads the outer borders with the wrapped omnidirectional image. A.4. T RAINING
For training, all losses are computed in the ego-sphere map frame of reference, either following convolutions for ESMN and ESMN-RGB, or following projection and probabilistic fusion for the
Mono case. We compute ground-truth segmentation training target labels by projecting the ground truth monocular frames to form an ego-sphere target segmentation image. We chose this approach over computing the ground truth segmentations from the complete ScanNet meshes for implementational simplicity. All experiments use a batch size of 16, unroll size of 8 in the time dimension, and Adam optimizer with learning rate 1 e − 4 , trained for 250k steps.