Self-Improving Semantic Perception for Indoor Localisation ASL
ETH Z¨urich Ren´e Zurbr¨ugg ASL ETH Z¨urich zrene ethz ch
ASL Abel Gawel ASL ETH gawela
Abstract: We propose a novel robotic system that can improve its perception during deployment.
Contrary to the established approach of learning semantics from large datasets and deploying ﬁxed models, we propose a framework in which semantic models are continuously updated on the robot to adapt to the deployment environments.
By combining continual learning with self-supervision, our robotic system learns online during deployment without external supervision.
We conduct real-world experiments with robots localising in 3D ﬂoorplans.
Our experiments show how the robot’s semantic perception improves during deployment and how this translates into improved localisation, even across drastically different environments.
We further study the risk of catastrophic forgetting that such a continuous learning setting poses.
We ﬁnd memory replay an effective measure to reduce forgetting and show how the robotic system can improve even when switching between different environments.
On average, our system improves by 60% in segmentation and 10% in localisation accuracy compared to deployment of a ﬁxed model, and it maintains this improvement while adapting to further environments.
Introduction
Learning-based systems enable robots to partially understand the environment through, e.g., object detection or semantic classiﬁcation [1, 2].
Such understanding is a key requirement to enable many complex robotic applications such as autonomous driving or mobile manipulation [3, 4].
However, mobile robots are deployed in increasingly unstructured environments, where learning-based systems often fail [5, 6].
Anticipation of a wide variety of environmental conditions is required for safe operation, however difﬁcult if not impossible.
Thus, robotic actors with a high degree of autonomy are required to adapt to unexpected and changing conditions for robust operation.
Yet, deployment of learning-based systems typically means training a model on a variety of data and then using this ﬁxed model during deployment.
Since it is unattainable for ﬁxed datasets to model all environmental conditions and objects, robustness of current approaches to novel situations is severely limited.
In this work, we explore how learning can be used as a means to self-improve semantic perception during exploration of the environment.
Enabling robots to adapt on-the-job poses three main challenges: (i) Models need to be efﬁciently (re)trained to incorporate new data ( incremental / continual learning ).
(ii) Acquired knowledge should be kept while adapting to new tasks and environments (avert forgetting ).
(iii) Training signals are required during deployment, i.e. automatically harvested without manual human supervision in the loop ( self-supervision ).
For the ﬁrst two points, we build upon recent advancements in the ﬁeld of continual learning (CL) [7, 8], also referred to as incremental learning [9, 10], and lifelong learning [11, 12, 13].
This learning paradigm deals with settings where tasks or classes are presented incrementally, or in which the data distribution changes over time [7].
Key to our proposed system is the combination of CL and self-supervision.
In the literature, CL is usually evaluated by consecutively presenting a model with parts of existing datasets.
However, 5th Conference on Robot Learning (CoRL 2021), London, UK.
in order to continually learn during deployment, robots require streams of training signals without humans in the loop.
We make use of self-supervised pseudo labels , which can be generated from multi-sensor [14] or multi-task systems [15].
Our results show that even noisy, imperfect pseudo labels generate useful training signals and – what to our knowledge has not been discovered before – also work well with CL methods that were developed for perfect labels.
To validate our approach, we conduct experiments on a real physical system.
We deploy robots in diverse environments with existing 3D ﬂoorplans, in which the robot is required to localize.
Floorplans are ubiquitous and therefore enable easy deployment into any indoor environment, without the need to generate a map before or during the mission.
In applications like construction robotics, the robot’s tasks are also deﬁned in these plans.
However, robot localisation in ﬂoorplans is challenging.
While the plan only represents static building structure ( background ), the actual environments contain large amounts of un-modelled objects and clutter ( foreground ), potentially obstructing localization performance [16].
With our system, the robot adapts to each environment by learning to tell apart background from foreground and improving its localisation in all environments.
We further observe catastrophic forgetting when the robot switches between environments, and we identify CL methods that mitigate such forgetting.
Finally, our experiments show positive knowledge transfer where the robot improves in novel environments based upon learning in earlier environments.
In summary, our contributions are:
• A framework for self-improving perception; • A pseudo label generation method for background-foreground-segmentation; • Validation on a real-world robotic system; • Evaluation of a range of methods to prevent forgetting; • Evidence that CL with noisy pseudo labels yields signiﬁcant improvements over ﬁxed models.
All experimental data and code is available at https://github.com/ethz-asl/background_ foreground_segmentation and a summary video can be accessed at https://youtu.be/ awsynhkkFpk .
Related Work
Self-improving robotic agents is a both old and underexplored idea.
One framework in which agents are self-improving is reinforcement learning (RL).
With RL, robots have been learning to walk [17], grasp objects [18], or ﬂy [19].
All these systems indeed learn by self-improving over time, often failing in the beginning of the learning process.
Usually these learned models are ﬁxed once they acquire the necessary skills, unlike life-long learning.
This is because they require supervision signals, e.g. from simulators that are not available during deployment.
However, online adaptation of model-based RL has been shown for example in [20].
Self-improving robotic systems have also been described outside of RL.
For example, [21] and [22] describe online parameter optimisations for model predictive control.
The adaptive stereo vision of Tonioni et al.
[23] has been a particular inspiration for this work.
Very related is also Sofman et al.
[24], who learn a probabilistic model for terrain traversability in an online and self-supervised fashion.
[25] is a parallel work that uses self-supervised learning to improve localisation in a robotic map.
Interestingly, the mentioned previous works often do not explicitly address the problem of forgetting, which is a more prevalent problem in our semantic domain adaptation.
In continual learning , models are trained from non-stationary data distributions over a series of different tasks [8].
The main objective of CL is to optimize for the performance on each task or domain with which the network is presented at any given time, while achieving positive knowledge transfer between the tasks, and preventing performance on the previous tasks from decreasing.
This decrease on previous tasks is commonly referred to as catastrophic forgetting [26, 7].
Multiple techniques have been proposed to mitigate such forgetting, ranging from architectural modiﬁcations [27, 28] to regularization techniques [29, 30] and methods based on memories [31, 32, 33, 34] or generative models [35, 36].
A number of works have explored the use of CL techniques in the context of semantic segmentation [37, 38, 39, 40, 41].
They however experiment on a class-incremental setting, whereas we are interested in the scenario in which the deployment environment of the agent changes.
In this sense, the problem we aim at tackling is also related to a line of works that focus on domain adaptation [42, 43, 44].
However, these works generally assume that both the source and target domain are known at training time, while we tackle the problem of adapting to any (unknown) environment on the ﬂy.
Self-supervision is often used to learn useful image features in convolutional neural networks (CNNs).
These techniques include learning to (re)color images [45], to (un)rotate images [46], or to relatively position random crops [47].
However, these features require further (supervised) processing to relate them to e.g. classes.
In mobile agents, egomotion was found to be a promising self-supervision signal for a range of tasks [48].
Photometric consistency between video frames can be used to jointly learn camera calibration, visual odometry, depth estimation, and optical ﬂow [15].
A different line of works produces pseudolabels for segmentation by leveraging models trained on more available data.
Class activation maps of image classiﬁers [49] can be used to identify object regions as a segmentation proxy [50, 51, 52].
Furthermore, segmentation predictions can be reﬁned by optimizing over Mask R-CNN predictions, room layout estimation, and superpixels [53], by tracking and optical ﬂow [54], or by aggregating predictions in 3D space and projecting them back onto images [14].
While there is no direct prior work for background-foreground-segmentation, our proposed method builds up on similar ideas to use observable characteristics of the environment to produce a learning signal for the target task.
Proposed System
[Figure]
The segmentation still incorporates existing training data as a prior, but is not ﬁxed during deployment, as would be the established approach.
Instead, our segmentation is updated during deployment based on CL methods.
The continual learning uses self-supervised pseudo labels, which are available during deployment without manual labelling.
We mark signals from the deployment environment in orange and from the pretraining domain in green.
We propose a self-improving perception system that interlinks localisation within a map and semantic segmentation of the scene.
We deﬁne the semantics of the scene not as arbitrary class labels, but by the observable affordance that some parts of the scene are mapped ( background ) and some are not ( foreground ).
Therefore, we create pseudo labels based on the localisation in the map to train the semantic segmentation, and we use the segmentation into foreground and background to inform the localisation.
This creates a feedback loop that can yield improvements in both parts, as can be seen in Figure 1.
3.1 Semantically Informed Localisation
We localize the robot based on aligning 3D LiDAR scans with the given ﬂoorplan in the form of a 3D mesh as in [16].
Given the ﬂoorplan mesh
M , a pointcloud of the LiDAR scan P , and an initial alignment T ( t =0) mesh !
lidar , we ﬁnd subsequent robot poses as
[Equation]
We use point-to-plane ICP [55] 1 and ﬁlter out points of large distance and other criteria.
Complete parameters are reported in Appendix A.5.
To further divide the scan P into foreground and background points, we use additional information from a camera system mounted on top of the LiDAR.
Once camera images are semantically segmented, we ﬁlter P background ✓ P as those points p 2 P whose reprojected pixel in image frame is segmented as background and localise with ICP ( M, P ( t ) background , T ( t  1) mesh ! lidar ) .
3.2 Pseudolabel Generation
We generate pseudo labels for each camera based on the LiDAR pointcloud that is localised w.r.t. the ﬂoorplan.
First, for each point of the localised LiDAR scan, we calculate the distance to the closest plane of the mesh using fast intersection and distance computation [57].
We then check if the distance surpasses a given threshold  .
If so, the point is assigned the foreground class, otherwise the [Figure] [Figure] background class.
In the second stage, we project each point onto the respective camera frames and reﬁne the projection using SLIC superpixels [58].
In particular, we ﬁrst oversegment the image into a superpixel set S .
A superpixel s 2 S is then assigned a class according to a majority voting of the contained projected labels.
We further improve the segmentation by discarding superpixels whose depth variance surpasses a given threshold.
An overview of the approach is depicted in Figure 2.
These pseudo labels are not optimal 2 .
The goal however is not to generate perfect labels, but a useful training signal that can be generated on-the-ﬂy without requiring any external supervision.
As our experiments demonstrate, even this noisy learning signal substantially boosts performance.
3.3 Domain Adaptation with Continual Learning
To solve the task of background-foreground segmentation, we incrementally train a neural network architecture on different data sources.
We pre-train a CNN on the NYU-Depth v2 dataset [59], which contains 1449 images extracted from video sequences of indoor scenes, each with per-pixel semantic annotations.
We map the classes wall , ceiling , and ﬂoor to background and regard everything else as foreground.
This (pre-)training step allows the model to acquire prior knowledge as an inductive bias to perform the same segmentation task on subsequent environments that the agent is presented with.
During deployment, the network is then updated with self-supervision through the pseudolabels generated on the current scene.
With reference to the nomenclature often used in CL [8], we consider each new environment a task , and we assume task boundaries to be known, since for each task, the robot is given a different ﬂoorplan.
Every time the robot is moved to a new environment, the same scheme is applied, i.e., the network trained on the previous environments is updated by training on the pseudolabels from the current environment.
memory buffer =
RandomChoice( 10% out of previous scene) train data =
ShufﬂeTogether( current scene, memory buffer) while training not converged do foreach batch in train data do randomly augment batch train on batch end validate on 10% hold-out of current scene end
To prevent forgetting of information learned on the previous tasks, we adopt a method based on memory replay buffers.
When adapting to a new environment, each training batch is ﬁlled with the frames collected in the current environment, along with a small fraction of images collected in the previous environments.
Therefore, the model jointly optimizes over current and previous environments.
However, storing all observations from past environments in memory would come at huge costs.
Instead, a memory buffer for each previous environment only contains a random subset of all images and (pseudo-)labels.
Details of the buffer implementation and a comparison with other CL frameworks are described in Section 4.4.
Experimental Evaluation
We test and verify the applicability of the proposed framework in different steps of increasing complexity.
We ﬁrst validate that our robot can self-improve by deploying it into different unknown
[Table]
Robotic Platform.
We conduct all our experiments on a wheeled robotic ground vehicle [60] with the sensor setup shown in Figure 3a.
We calibrate all cameras to an IMU that we attach to the sensor system and then align trajectories from visual-inertial odometry and LiDAR to ﬁnd the extrinsic calibration between the LiDAR and the cameras.
The tracking prism is used to gather ground-truth with a total station for our evaluation and is mounted on the sensor system to be aligned with the optical center of the LiDAR.
For time synchronisation, we trigger all cameras simultaneously.
From images captured at 20 Hz we then take those closest to the timestamp of the LiDAR scans, which are captured at 5Hz.
Between the external total station and the robot, we correct time-offsets manually.
Evaluation Environments.
We deploy our proposed system into three different environments: a construction site, a parking garage, and an ofﬁce ﬂoor.
Architectural researchers provided us 3D meshes, constructed from dense 3D scans (Construction) and existing 2D ﬂoorplans supplemented with additional measurements (Garage and Ofﬁce).
Figure 3 shows these meshes together with the experimental setup.
In each environment, we steer the robot through multiple independent trajectories of 2-3 min while tracking the robot with a total station.
The coordinate system of the total station is always initialised to the origin of the building mesh, which we therefore set at corners visible to the total station.
To evaluate the learned segmentation models, we sample and annotate aroundground-truth segmentations per environment.
We then evaluate the segmentation predictions against ground-truth annotations within a static ﬁeld-of-view (FoV) mask of the LiDAR 4 .
Learning Setup.
We use a lightweight architecture based on Fast-SCNN [61] that we train for background-foreground segmentation.
We resize input images to a common size ( 480 ⇥ 640 pixels).
We ﬁrst train the model on the NYU dataset.
Then, when we deploy the robot into a new environment (cf.
Sec. 4.1), we ﬁll a replay buffer with samples from NYU in addition to training on the pseudolabels from the current environment.
When we evaluate the transfer from a ﬁrst environment to a second one (cf.
Sec. 4.2), we replay images from both NYU and the ﬁrst environment.
We set the replay fraction to 10% , but evaluate different replay regimes and strategies in Section 4.4.
Before feeding images to the network, we augment them with left-right ﬂipping and random perturbation of brightness and hue.
In each experiment, we hold out 10% of the training samples for validation and train our model using
Adam optimizer.
See Appendix A.2 for all training parameters and Appendix A.1 for runtimes.
4.1 Deployment in a new environment
To test the effectiveness of the pseudolabel training and the localisation based on ﬁltered pointclouds, we deploy the robot in a ﬁrst new environment.
There, the robot collects information in the form of pseudolabels, which are used to train the segmentation.
Afterwards, we deploy the robot over a different trajectory in the same environment and measure the performance of both segmentation and localisation.
In Table 1 we compare the performance obtained from our self-supervised foreground-background segmentation (‘self-improving’) with segmentation obtained from the network pre-trained on NYU
[Table]
(‘trained on NYU’), and with a baseline that does not semantically ﬁlter the pointcloud (‘no segmentation’).
We note that segmentation in general is important for good localisation and can prevent failure, as on the construction site.
The results also conﬁrm the effectiveness of the pseudolabels, as training on these yields improvements in both segmentation quality and localisation error in all metrics.
We conclude that the self-improving setup is working as expected and that through the feedback loop indeed the localisation improves the segmentation, and the segmentation improves the localisation.
4.2 Transfer into multiple environments
In a second stage of experiments, we evaluate the ability of our system to retain knowledge and still adapt to new domains when moved from a ﬁrst environment to a second (and third) one.
In particular, we iteratively train the same model on pseudolabels from a series of environments.
We then evaluate the extent of forgetting on both the pretraining data (NYU) and any environment before the last one, comparing our adopted method based on a replay buffer with simple ﬁne-tuning.
As shown in Table 2, using replay buffers improves the segmentation performance on the past tasks w.r.t.
the case in which no replay is adopted.
This indicates successful mitigation of forgetting and is even more prominent when measured on the pseudolabels and measuring forgetting on the NYU data, which we analyse in more detail in Table 5 in Appendix A.3.
Table 2 further shows that the forgetting of ﬁnetuning can cause localisation failure on the source environment.
Replay prevents such failure successfully.
In the CL literature, memory replay is usually studied with high-quality segmentation masks covering all pixels in the image [7].
Yet, from our observations we can conclude that memory replay is also effective when the replayed labels are noisy and imperfect.
We note that ﬁnetuning can result in better adaptation to the target environment, especially with regard to localisation.
This is expected, since the learning process of ﬁnetuning is fully tailored to the new environment.
This effect is known as stability-plasticity dilemma [62], i.e. old knowledge can inhibit learning of new knowledge, but increasing plasticity can in turn lead to forgetting.
In our experiments, the relative improvements of ﬁnetuning over memory replay that we observe on the target environment are marginal, suggesting that the chosen 10% replay ﬁnds a good balance.
Table 2 also highlights cases in which deployment in multiple environments is better than improving only towards one speciﬁc environment (Section 4.1), both with and without memory replay.
This indicates positive knowledge transfer between our evaluation environments.
For a self-improving robotic system this is a promising ﬁnding, showing that the robot not only adapts to the target environment, but generally improves at its task with every new deployment.
Finally, we observe that segmentation quality and localisation error are sometimes inconsistent, where a drop in segmentation quality in a past environment does not necessarily transfer into worse localisation.
This indicates potential for advanced methods which only keep in memory those observations that are important for the task at hand.
4.3 Online Learning
[Figure]
[ m [Figure] online localisation[-0 .0 .
 .m-]
[-I o U segmentation quality     0 .0 .
 .mean median time [s]-][Figure]
[ m ] localisation on reference trajectory
We measure how the robot improves online (row 1), but also evaluate snapshots of the model on 2 and 3)
Areas in green/red show changes (better/worse) with respect to a ﬁxed model.
We observe that segmentation quality increases over time and localisation error decreases.
The previous experiments have been conducted in a multi-mission fashion, in which a ﬁrst mission gathers data of the environment and the robot learns from it to improve in subsequent missions.
Figure 4 now shows a case study how the system can learn online during a mission by learning from the pseudolabels directly as they are generated.
We initialise the model pretrained on NYU and observe that over time segmentation quality increases and localisation error decreases.
Notably, already a few seconds of online learning increase the segmentation quality signiﬁcantly.
However, it takes longer until we measure a notable effect on the localisation.
Due to the limited time until the end of the trajectory, we are therefore unable to see if there is a feedback effect where the improved localisation would create better pseudolabels that can in turn increase the segmentation quality further.
For this, further investigation on longer deployments is necessary.
We present similar studies for the other environments in Appendix A.7.
4.4 Ablation Studies
Since CL methods have not been explored in a real-world robotic context, and even less in combination with noisy pseudolabels as training signal, we perform an extensive ablation of different methods and parameters to validate our use of replay buffers.
We ﬁrst evaluate two different strategies for memory replay.
In the ﬁrst strategy, which is the one that we adopt in the main experiments, on each source-to-target experiment (e.g., NYU !
Garage), we ﬁll the replay buffer with a fraction of samples from the source dataset(s)
(NYU in the example), which we select randomly.
We then ﬁll training batches from the replay buffer and target dataset according to their relative sizes.
In the second strategy, we ﬁll the replay buffer with the full source dataset but ﬁll training batches with a pre-deﬁned target-source ratio.
For instance, a ratio Garage : NYU = 4 :with a batch size of 10 indicates that batches on average contain 8 images from Garage and 2 images from NYU.
As shown in Table 3, milder replay regimes (larger target-source ratios, or smaller replay fractions) achieve higher performance on the target domain, but cause the amount of information retained from the source domain to drop.
This forgetting phenomenon is particularly evident in the adaptation tasks in which the semantic gap between source and domain is larger.
Indeed, for instance, in the NYU !
Garage experiment we observe a drop of 31.9% in mIoU on the NYU labels between a replay strategy with a fraction of 10% and simple ﬁne-tuning, while the same decrease in performance when the target domain is Ofﬁce – semantically closer to the indoor dataset NYU – is 14.8%.
At the same time, the segmentation quality on the pseudo-labels of the target dataset follows an inverse trend, generally increasing for smaller amounts of replay.
This highlights the trade-off between the accuracy on the target and on the source domain.
Memory replay comes at the cost of storing images and labels of past environments, which does not scale well to more than a few environments.
We therefore further compare regularization techniques from the continual-learning literature, which try to prevent forgetting without storing as much data.
In particular, we investigate the use of a distillation approach, in which a regularization term is added to the cross-entropy loss, to encourage the network to retain the knowledge from previous tasks.
Similarly to [37], we apply this distillation either on the output logits produced by the network ( Output distillation ) or on the intermediate features ( Feature distillation ).
Furthermore, we evaluate
Elastic Weight Consolidation (EWC) [29].
We present further details in Appendix A.4.
As shown in Table 3, in our experiments replay buffers prove to be the most effective among the examined methods in minimizing the amount of forgetting on the NYU dataset, and generally allow attaining a good trade-off with the segmentation quality on the pseudo-labels from the target domain.
[Table]
Output
[Table] [Table]
[Table]
r
[Table]
r e p l a
[Table]
r
[Table]
l a r i s a
[Table]
t
[Table]
s with memory replay and regularisation methods.
We observe that memory replay is more successful than regularisation at preventing catastrophic forgetting in our application.
We also note that, with limited exceptions, both regularization approaches fail to maintain a good performance on the source dataset NYU.
We believe that for distillation methods this can be ascribed to the fact that, as opposed to related works that explored similar techniques in a class-incremental setting [37, 38], we conduct domain adaptation.
More importantly, unlike [37, 38] our supervision signal does not consist of accurate ground-truth annotations available at all pixels, but of noisy pseudo-labels that often cover a limited region of the image.
Finally, while similar considerations also apply for EWC, we believe that the limited effectiveness of this technique in our setting is also related to the method being designed for classiﬁcation as opposed to image segmentation.
4.5 Limitations
Our system required some manual intervention in the sense that a few parameters were changed across environments (ICP and superpixels, see Appendix).
While coming close, the system also did not run in real-time.
We are conﬁdent that both points can be tackled with better software implementations.
This study is limited to binary segmentation.
However, our framework itself is agnostic of the self-supervision method and can be directly extended to more classes based on multi-sensor [14] or multi-task systems [15], or temporal consistency [54].
More affordances can be observed through manipulating objects [63], temporal change [25], or context and spatial co-occurrence [64, 65].
Conclusion & Outlook
In this work we propose a framework for self-improving perception by combining CL with selfsupervision.
We study this on a real robotic system that localises in 3D ﬂoorplans.
Our experiments validate the gains of the self-improving systems in diverse environments.
In particular, we analyse the effects of knowledge transfer and forgetting when switching between environments.
We ﬁnd that memory replay is an effective solution that can mitigate forgetting, and observe that the system can even further improve as it transfers knowledge from previously seen, distinct environments.
The main ﬁnding from our evaluations of knowledge transfer and forgetting is that self-supervision and CL is an effective combination, even if the self-supervision is noisy.
This opens up exciting questions for future research.
Through the self-supervision approaches that we describe in Sectionsand 4.5, the same framework can be transferred to other perception tasks such as more semantic classes or perception for manipulation.
For some tasks, task boundaries may be less clear, and further research is necessary to lift this assumption.
Finally, we identify long-term dynamics and stability of self-improving systems as an interesting direction for future research. Acknowledgments
This work was partially funded by the Hilti Group.
Eberhard Unternehmungen kindly allowed us to conduct experiments on their premises.
Furthermore, we thank Selen Ercan for creating the building models, Florian Tschopp and his
VersaVIS board for all the multi-sensor calibration, Shen Kaiyue for initially implementing and testing various regularisation based continual learning methods and Andrei Cramariuc for GPU cluster support.