Self-Improving Semantic Perception for Indoor Localisation
Anonymous Author(s)
AfﬁliationAddress email
Abstract: We propose a novel robotic system that can improve its perceptionduring deployment.
Contrary to the established approach of learning semanticsfrom large datasets and deploying ﬁxed models, we propose a framework inwhich semantic models are continuously updated on the robot to adapt to thedeployment environments.
By combining continual learning with self-supervision,our robotic system learns online during deployment without external supervision.
We conduct real-world experiments with robots localising in 3D ﬂoorplans.
Ourexperiments show how the robot’s semantic perception improves during deploymentand how this translates into improved localisation, even across drastically differentenvironments.
We further study the risk of catastrophic forgetting that such acontinuous learning setting poses.
We ﬁnd memory replay an effective measureto reduce forgetting and show how the robotic system can improve even whenswitching between different environments.
On average, our system improves by60% in segmentation and 10% in localisation accuracy compared to deploymentof a ﬁxed model, and it maintains this improvement while adapting to furtherenvironments.
Introduction
Learning-based systems enable robots to partially understand the environment through, e.g., objectdetection or semantic classiﬁcation [1, 2].
Such understanding is a key requirement to enable manycomplex robotic applications such as autonomous driving or mobile manipulation [3, 4].
However,mobile robots are deployed in increasingly unstructured environments, where learning-based systemsoften fail [5, 6].
Anticipation of a wide variety of environmental conditions is required for safeoperation, however difﬁcult if not impossible.
Thus, robotic actors with a high degree of autonomyare required to adapt to unexpected and changing conditions for robust operation.
Yet, deploymentof learning-based systems typically means training a model on a variety of data and then using thisﬁxed model during deployment.
Since it is unattainable for ﬁxed datasets to model all environmentalconditions and objects, robustness of current approaches to novel situations is severely limited.
In this work, we explore how learning can be used as a means to self-improve semantic perceptionduring exploration of the environment.
Enabling robots to adapt on-the-job poses three main challenges: (i) Models need to be efﬁciently (re)trained to incorporate new data ( incremental / continuallearning ).
(ii) Acquired knowledge should be kept while adapting to new tasks and environments(avert forgetting ).
(iii) Training signals are required during deployment, i.e. automatically harvestedwithout manual human supervision in the loop ( self-supervision ).
For the ﬁrst two points, we buildupon recent advancements in the ﬁeld of continual learning (CL) [7, 8], also referred to as incrementallearning [9, 10], and lifelong learning [11, 12, 13].
This learning paradigm deals with settings wheretasks or classes are presented incrementally, or in which the data distribution changes over time [7].
Key to our proposed system is the combination of CL and self-supervision.
In the literature, CLis usually evaluated by consecutively presenting a model with parts of existing datasets.
However,in order to continually learn during deployment, robots require streams of training signals withouthumans in the loop.
We make use of self-supervised pseudo labels , which can be generated from multi-sensor [14] or multi-task systems [15].
Our results show that even noisy, imperfect pseudolabels generate useful training signals and – what to our knowledge has not been discovered before –also work well with CL methods that were developed for perfect labels.
To validate our approach, we conduct experiments on a real physical system.
We deploy robotsin diverse environments with existing 3D ﬂoorplans, in which the robot is required to localize.
Floorplans are ubiquitous and therefore enable easy deployment into any indoor environment, withoutthe need to generate a map before or during the mission.
In applications like construction robotics, therobot’s tasks are also deﬁned in these plans.
However, robot localisation in ﬂoorplans is challenging.
While the plan only represents static building structure ( background ), the actual environments containlarge amounts of un-modelled objects and clutter ( foreground ), potentially obstructing localizationperformance [16].
With our system, the robot adapts to each environment by learning to tell apartbackground from foreground and improving its localisation in all environments.
We further observecatastrophic forgetting when the robot switches between environments, and we identify CL methodsthat mitigate such forgetting.
Finally, our experiments show positive knowledge transfer where therobot improves in novel environments based upon learning in earlier environments.
In summary, ourcontributions are:
• A framework for self-improving perception;
• A pseudo label generation method for background-foreground-segmentation;
• Validation on a real-world robotic system;
• Evaluation of a range of methods to prevent forgetting;
• Evidence that CL with noisy pseudo labels yields signiﬁcant improvements over ﬁxed models.
All experimental data and code is avalable at anonymized-url and a summary video can be accessat anonymized-
url
Related Work
Self-improving robotic agents is a both old and underexplored idea.
One framework in whichagents are self-improving is reinforcement learning (RL).
With RL, robots have been learning towalk [17], grasp objects [18], or ﬂy [19].
All these systems indeed learn by self-improving over time,often failing in the beginning of the learning process.
Usually these learned models are ﬁxed oncethey acquire the necessary skills, unlike life-long learning.
This is because they require supervisionsignals, e.g. from simulators that are not available during deployment.
However, online adaptation ofmodel-based RL has been shown for example in [20].
Self-improving robotic systems have also been described outside of RL.
For example, [21] and [22]describe online parameter optimisations for model predictive control.
The adaptive stereo visionof Tonioni et al.
[23] has been a particular inspiration for this work.
Very related is also Sofmanet al.
[24], who learn a probabilistic model for terrain traversability in an online and self-supervisedfashion.
[25] is a parallel work that uses self-supervised learning to improve localisation in a roboticmap.
Interestingly, the mentioned previous works often do not explicitly address the problem offorgetting, which is a more prevalent problem in our semantic domain adaptation.
In continual learning , models are trained from non-stationary data distributions over a series ofdifferent tasks [8].
The main objective of CL is to optimize for the performance on each task or domainwith which the network is presented at any given time, while achieving positive knowledge transferbetween the tasks, and preventing performance on the previous tasks from decreasing.
This decreaseon previous tasks is commonly referred to as catastrophic forgetting [26, 7].
Multiple techniqueshave been proposed to mitigate such forgetting, ranging from architectural modiﬁcations [27, 28]to regularization techniques [29, 30] and methods based on memories [31, 32, 33, 34] or generativemodels [35, 36].
A number of works have explored the use of CL techniques in the context ofsemantic segmentation [37, 38, 39, 40, 41].
They however experiment on a class-incremental setting,whereas we are interested in the scenario in which the deployment environment of the agent changes.
In this sense, the problem we aim at tackling is also related to a line of works that focus on domainadaptation [42, 43, 44].
However, these works generally assume that both the source and targetdomain are known at training time, while we tackle the problem of adapting to any (unknown)environment on the ﬂy.
Self-supervision is often used to learn useful image features in convolutional neural networks (CNNs).
These techniques include learning to (re)color images [45], to (un)rotate images [46], or to relatively position random crops [47].
However, these features require further (supervised) processing to relatethem to e.g. classes.
In mobile agents, egomotion was found to be a promising self-supervision signalfor a range of tasks [48].
Photometric consistency between video frames can be used to jointly learncamera calibration, visual odometry, depth estimation, and optical ﬂow [15].
A different line of works produces pseudolabels for segmentation by leveraging models trained onmore available data.
Class activation maps of image classiﬁers [49] can be used to identify objectregions as a segmentation proxy [50, 51, 52].
Furthermore, segmentation predictions can be reﬁnedby optimizing over Mask R-CNN predictions, room layout estimation, and superpixels [53], bytracking and optical ﬂow [54], or by aggregating predictions in 3D space and projecting them backonto images [14].
While there is no direct prior work for background-foreground-segmentation, ourproposed method builds up on similar ideas to use observable characteristics of the environment toproduce a learning signal for the target task.
Proposed System
[Figure]
The segmentation still incorporates existing training data as a prior, but is not ﬁxed during deployment, as would be the established approach.
Instead, our segmentation is updated during deployment based on CL methods.
The continual learning uses self-supervised pseudo labels, which are available during deployment without manual labelling.
We mark signals from the deployment environment in orange and from the pretraining domain in green.
We propose a self-improving perception systemthat interlinks localisation within a map and semantic segmentation of the scene.
We deﬁne thesemantics of the scene not as arbitrary class labels, but by the observable affordance that someparts of the scene are mapped ( background ) andsome are not ( foreground ).
Therefore, we create pseudo labels based on the localisation in themap to train the semantic segmentation, and weuse the segmentation into foreground and background to inform the localisation.
This createsa feedback loop that can yield improvements inboth parts, as can be seen in Figure 1.
3.1 Semantically Informed Localisation
We localize the robot based on aligning 3D LiDAR scans with the given ﬂoorplan in the formof a 3D mesh as in [16].
Given the ﬂoorplan mesh
M , a pointcloud of the LiDAR scan P , and aninitial alignment T ( t =0) mesh !
lidar , we ﬁnd subsequent 127 robot poses as
[Equation]
We use point-to-plane ICP [55] 1 and ﬁlter out points of large distance and other criteria.
Completeparameters are reported in Appendix A.5.
To further divide the scan P into foreground and background points, we use additional informationfrom a camera system mounted on top of the LiDAR.
Once camera images are semantically segmented,we ﬁlter P background ✓ P as those points p 2 P whose reprojected pixel in image frame is segmentedas background and localise with ICP ( M, P ( t ) background , T ( t  1) mesh ! lidar ) .
3.2 Pseudolabel Generation
We generate pseudo labels for each camera based on the LiDAR pointcloud that is localised w.r.t.the ﬂoorplan.
First, for each point of the localised LiDAR scan, we calculate the distance to theclosest plane of the mesh using fast intersection and distance computation [57].
We then check if thedistance surpasses a given threshold  .
If so, the point is assigned the foreground class, otherwise thebackground class.
In the second stage, we project each point onto the respective camera frames andreﬁne the projection using SLIC superpixels [58].
In particular, we ﬁrst oversegment the image into a
[Figure] [Figure]
superpixel set S .
A superpixel s 2 S is then assigned a class according to a majority voting of thecontained projected labels.
We further improve the segmentation by discarding superpixels whosedepth variance surpasses a given threshold.
An overview of the approach is depicted in Figure 2.
These pseudo labels are not optimal 2 .
The goal however is not to generate perfect labels, but a usefultraining signal that can be generated on-the-ﬂy without requiring any external supervision.
As ourexperiments demonstrate, even this noisy learning signal substantially boosts performance.
3.3 Domain Adaptation with Continual Learning
To solve the task of background-foreground segmentation, we incrementally train a neural networkarchitecture on different data sources.
We pre-train a CNN on the NYU-Depth v2 dataset [59], whichcontains 1449 images extracted from video sequences of indoor scenes, each with per-pixel semanticannotations.
We map the classes wall , ceiling , and ﬂoor to background and regard everything else asforeground.
This (pre-)training step allows the model to acquire prior knowledge as an inductive biasto perform the same segmentation task on subsequent environments that the agent is presented with.
During deployment, the network is then updated with self-supervision through the pseudolabelsgenerated on the current scene.
With reference to the nomenclature often used in CL [8], we considereach new environment a task , and we assume task boundaries to be known, since for each task, therobot is given a different ﬂoorplan.
Every time the robot is moved to a new environment, the samescheme is applied, i.e., the network trained on the previous environments is updated by training onthe pseudolabels from the current environment.
memory buffer =
RandomChoice( 10% out of previous scene) train data =
ShufﬂeTogether( current scene, memory buffer) while training not converged do foreach batch in train data do randomly augment batch train on batch end validate on 10% hold-out of current scene end
To prevent forgetting of information learned on the previoustasks, we adopt a method based on memory replay buffers.
When adapting to a new environment, each training batch isﬁlled with the frames collected in the current environment,along with a small fraction of images collected in the previous environments.
Therefore, the model jointly optimizes overcurrent and previous environments.
However, storing all observations from past environments in memory would come at hugecosts.
Instead, a memory buffer for each previous environmentonly contains a random subset of all images and (pseudo-)labels.
Details of the buffer implementation and a comparison withother CL frameworks are described in Section 4.4.
Experimental Evaluation
We test and verify the applicability of the proposed framework in different steps of increasingcomplexity.
We ﬁrst validate that our robot can self-improve by deploying it into different unknown
[Table]
and in localisation plane
Robotic Platform.
We conduct all our experiments on a wheeled robotic ground vehicle [16] withthe sensor setup shown in Figure 3a.
We calibrate all cameras to an IMU that we attach to the sensorsystem and then align trajectories from visual-inertial odometry and LiDAR to ﬁnd the extrinsiccalibration between the LiDAR and the cameras.
The tracking prism is used to gather ground-truthwith a total station for our evaluation and is mounted on the sensor system to be aligned with theoptical center of the LiDAR.
For time synchronisation, we trigger all cameras simultaneously.
Fromimages captured at 20 Hz we then take those closest to the timestamp of the LiDAR scans, which arecaptured at 5Hz.
Between the external total station and the robot, we correct time-offsets manually.
Evaluation Environments.
We deploy our proposed system into three different environments: aconstruction site, a parking garage, and an ofﬁce ﬂoor.
Architectural researchers provided us 3Dmeshes, constructed from dense 3D scans (Construction) and existing 2D ﬂoorplans supplementedwith additional measurements (Garage and Ofﬁce).
Figure 3 shows these meshes together with theexperimental setup.
In each environment, we steer the robot through multiple independent trajectoriesof 2-3 min while tracking the robot with a total station.
The coordinate system of the total stationis always initialised to the origin of the building mesh, which we therefore set at corners visible tothe total station.
To evaluate the learned segmentation models, we sample and annotate around 30ground-truth segmentations per environment.
We then evaluate the segmentation predictions againstground-truth annotations within a static ﬁeld-of-view (FoV) mask of the LiDAR 4 .
Learning Setup.
We use a lightweight architecture based on Fast-SCNN [60] that we train forbackground-foreground segmentation.
We resize input images to a common size ( 480 ⇥ 640 pixels).
We ﬁrst train the model on the NYU dataset.
Then, when we deploy the robot into a new environment(cf.
Sec. 4.1), we ﬁll a replay buffer with samples from NYU in addition to training on the pseudolabelsfrom the current environment.
When we evaluate the transfer from a ﬁrst environment to a second one(cf.
Sec. 4.2), we replay images from both NYU and the ﬁrst environment.
We set the replay fractionto 10% , but evaluate different replay regimes and strategies in Section 4.4.
Before feeding images tothe network, we augment them with left-right ﬂipping and random perturbation of brightness and hue.
In each experiment, we hold out 10% of the training samples for validation and train our model using
Adam optimizer.
See Appendix A.2 for all training parameters and Appendix A.1 for runtimes.
4.1 Deployment in a new environment
To test the effectiveness of the pseudolabel training and the localisation based on ﬁltered pointclouds,we deploy the robot in a ﬁrst new environment.
There, the robot collects information in the formof pseudolabels, which are used to train the segmentation.
Afterwards, we deploy the robot over adifferent trajectory in the same environment and measure the performance of both segmentation andlocalisation.
In Table 1 we compare the performance obtained from our self-supervised foreground-backgroundsegmentation (‘self-improving’) with segmentation obtained from the network pre-trained on NYU
[Table]
forgetting
(‘trained on NYU’), and with a baseline that does not semantically ﬁlter the pointcloud (‘no segmentation’).
We note that segmentation in general is important for good localisation and can prevent failure,as on the construction site.
The results also conﬁrm the effectiveness of the pseudolabels, as trainingon these yields improvements in both segmentation quality and localisation error in all metrics.
Weconclude that the self-improving setup is working as expected and that through the feedback loopindeed the localisation improves the segmentation, and the segmentation improves the localisation.
4.2 Transfer into multiple environments
In a second stage of experiments, we evaluate the ability of our system to retain knowledge and stilladapt to new domains when moved from a ﬁrst environment to a second (and third) one.
In particular,we iteratively train the same model on pseudolabels from a series of environments.
We then evaluatethe extent of forgetting on both the pretraining data (NYU) and any environment before the last one,comparing our adopted method based on a replay buffer with simple ﬁne-tuning.
As shown in Table 2, using replay buffers improves the segmentation performance on the past tasksw.r.t.
the case in which no replay is adopted.
This indicates successful mitigation of forgetting and iseven more prominent when measured on the pseudolabels and measuring forgetting on the NYU data,which we analyse in more detail in Table 5 in Appendix A.3.
Table 2 further shows that the forgettingof ﬁnetuning can cause localisation failure on the source environment.
Replay prevents such failuresuccessfully.
In the CL literature, memory replay is usually studied with high-quality segmentationmasks covering all pixels in the image [7].
Yet, from our observations we can conclude that memoryreplay is also effective when the replayed labels are noisy and imperfect.
We note that ﬁnetuning can result in better adaptation to the target environment, especially withregard to localisation.
This is expected, since the learning process of ﬁnetuning is fully tailored tothe new environment.
This effect is known as stability-plasticity dilemma [61], i.e. old knowledgecan inhibit learning of new knowledge, but increasing plasticity can in turn lead to forgetting.
In ourexperiments, the relative improvements of ﬁnetuning over memory replay that we observe on thetarget environment are marginal, suggesting that the chosen 10% replay ﬁnds a good balance.
Table 2 also highlights cases in which deployment in multiple environments is better than improvingonly towards one speciﬁc environment (Section 4.1), both with and without memory replay.
Thisindicates positive knowledge transfer between our evaluation environments.
For a self-improvingrobotic system this is a promising ﬁnding, showing that the robot not only adapts to the targetenvironment, but generally improves at its task with every new deployment.
Finally, we observe that segmentation quality and localisation error are sometimes inconsistent,where a drop in segmentation quality in a past environment does not necessarily transfer into worselocalisation.
This indicates potential for advanced methods which only keep in memory thoseobservations that are important for the task at hand.
4.3 Online Learning
R M S E [ m ] online localisation 0 .0 .
.m
I o
U segmentation quality     0 .0 .
.mean
median time [s]
R M S E [ m ] localisation on reference trajectory
We measure how the robot improves online (row 1), but also evaluate snapshots of the model on reference data (rows 2 and 3).
Areas in green/red show changes (better/worse) with respect to a ﬁxed model.
We observe that segmentation quality increases over time and localisation error decreases.
The previous experiments have been conductedin a multi-mission fashion, in which a ﬁrst mission gathers data of the environment and therobot learns from it to improve in subsequentmissions.
Figure 4 now shows a case study howthe system can learn online during a mission bylearning from the pseudolabels directly as theyare generated.
We initialise the model pretrainedon NYU and observe that over time segmentation quality increases and localisation errordecreases.
Notably, already a few seconds ofonline learning increase the segmentation quality signiﬁcantly.
However, it takes longer untilwe measure a notable effect on the localisation.
Due to the limited time until the end of the trajectory, we are therefore unable to see if thereis a feedback effect where the improved localisation would create better pseudolabels that canin turn increase the segmentation quality further.
For this, further investigation on longer deployments is necessary.
We present similar studiesfor the other environments in Appendix A.7.
4.4 Ablation Studies
Since CL methods have not been explored in a real-world robotic context, and even less in combinationwith noisy pseudolabels as training signal, we perform an extensive ablation of different methods andparameters to validate our use of replay buffers.
We ﬁrst evaluate two different strategies for memory replay.
In the ﬁrst strategy, which is the one thatwe adopt in the main experiments, on each source-to-target experiment (e.g., NYU !
Garage), we ﬁllthe replay buffer with a fraction of samples from the source dataset(s)
(NYU in the example), whichwe select randomly.
We then ﬁll training batches from the replay buffer and target dataset accordingto their relative sizes.
In the second strategy, we ﬁll the replay buffer with the full source dataset butﬁll training batches with a pre-deﬁned target-source ratio.
For instance, a ratio Garage : NYU = 4 : 1with a batch size of 10 indicates that batches on average contain 8 images from Garage and 2 imagesfrom NYU.
As shown in Table 3, milder replay regimes (larger target-source ratios, or smaller replayfractions) achieve higher performance on the target domain, but cause the amount of informationretained from the source domain to drop.
This forgetting phenomenon is particularly evident inthe adaptation tasks in which the semantic gap between source and domain is larger.
Indeed, forinstance, in the NYU !
Garage experiment we observe a drop of 31.9% in mIoU on the NYU labelsbetween a replay strategy with a fraction of 10% and simple ﬁne-tuning, while the same decrease inperformance when the target domain is Ofﬁce – semantically closer to the indoor dataset NYU – is14.8%.
At the same time, the segmentation quality on the pseudo-labels of the target dataset followsan inverse trend, generally increasing for smaller amounts of replay.
This highlights the trade-offbetween the accuracy on the target and on the source domain.
Memory replay comes at the cost of storing images and labels of past environments, which does notscale well to more than a few environments.
We therefore further compare regularization techniquesfrom the continual-learning literature, which try to prevent forgetting without storing as much data.
In particular, we investigate the use of a distillation approach, in which a regularization term isadded to the cross-entropy loss, to encourage the network to retain the knowledge from previoustasks.
Similarly to [37], we apply this distillation either on the output logits produced by the network( Output distillation ) or on the intermediate features ( Feature distillation ).
Furthermore, we evaluate
Elastic Weight Consolidation (EWC) [29].
We present further details in Appendix A.4.
As shown in Table 3, in our experiments replay buffers prove to be the most effective among theexamined methods in minimizing the amount of forgetting on the NYU dataset, and generally allowattaining a good trade-off with the segmentation quality on the pseudo-labels from the target domain.
[Table]
Output
[Table] [Table]
[Table]
r
[Table]
r e p l a
[Table]
r
[Table]
l a r i s a
[Table]
t
[Table]
s with memory replay and regularisation methods.
We observe that memory replay is more successful than regularisation at preventing catastrophic forgetting in our application.
We also note that, with limited exceptions, both regularization approaches fail to maintain a goodperformance on the source dataset NYU.
We believe that for distillation methods this can be ascribedto the fact that, as opposed to related works that explored similar techniques in a class-incrementalsetting [37, 38], we conduct domain adaptation.
More importantly, unlike [37, 38] our supervisionsignal does not consist of accurate ground-truth annotations available at all pixels, but of noisypseudo-labels that often cover a limited region of the image.
Finally, while similar considerationsalso apply for EWC, we believe that the limited effectiveness of this technique in our setting is alsorelated to the method being designed for classiﬁcation as opposed to image segmentation.
4.5 Limitations
Our system required some manual intervention in the sense that a few parameters were changed acrossenvironments (ICP and superpixels, see Appendix).
While coming close, the system also did not runin real-time.
We are conﬁdent that both points can be tackled with better software implementations.
This study is limited to binary segmentation.
However, our framework itself is agnostic of theself-supervision method and can be directly extended to more classes based on multi-sensor [14] ormulti-task systems [15], or temporal consistency [54].
More affordances can be observed throughmanipulating objects [62], temporal change [25], or context and spatial co-occurrence [63, 64].
Conclusion & Outlook
In this work we propose a framework for self-improving perception by combining CL with selfsupervision.
We study this on a real robotic system that localises in 3D ﬂoorplans.
Our experimentsvalidate the gains of the self-improving systems in diverse environments.
In particular, we analysethe effects of knowledge transfer and forgetting when switching between environments.
We ﬁnd thatmemory replay is an effective solution that can mitigate forgetting, and observe that the system caneven further improve as it transfers knowledge from previously seen, distinct environments.
The main ﬁnding from our evaluations of knowledge transfer and forgetting is that self-supervisionand CL is an effective combination, even if the self-supervision is noisy.
This opens up excitingquestions for future research.
Through the self-supervision approaches that we describe in Sections 2and 4.5, the same framework can be transferred to other perception tasks such as more semanticclasses or perception for manipulation.
For some tasks, task boundaries may be less clear, and furtherresearch is necessary to lift this assumption.
Finally, we identify long-term dynamics and stability ofself-improving systems as an interesting direction for future research.