Real-time Mapping of Physical Scene Properties with an Autonomous Robot Experimenter
Abstract:
Neural ﬁelds can be trained from scratch to represent the shape and appearance of 3D scenes efﬁciently.
It has also been shown that they can densely map correlated properties such as semantics, via sparse interactions from a human labeller.
In this work, we show that a robot can densely annotate a scene with arbitrary discrete or continuous physical properties via its own fully-autonomous experimental interactions, as it simultaneously scans and maps it with an RGB-D camera.
A variety of scene interactions are possible, including poking with force sensing to determine rigidity, measuring local material type with single-pixel spectroscopy or predicting force distributions by pushing.
Sparse experimental interactions are guided by entropy to enable high efﬁciency, with tabletop scene properties densely mapped from scratch in a few minutes from a few tens of interactions. [Figure]
Introduction
An autonomous agent performing complex tasks in unpredictable environments (e g. tidying a home), must build a rich internal representation of its environment.
This representation should extend 6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand. beyond geometry, colour and traditional semantics to include physical properties from which robotic affordances may be inferred.
Like its biological counterparts, a robotic agent cannot infer these properties from vision alone [1].
In this work, we present a real-world robot that learns about its environment by autonomous exploration and experimentation.
The robot incrementally discovers physical scene properties by performing targeted physical interactions with the scene, measuring their effects and feeding the results into a jointly-optimised neural-ﬁeld MLP.
With only a handful of these autonomous robotic experiments, a task-driven internal representation of the scene is built from scratch, without any pre-training or external guidance.
We build upon the observation that the joint internal representation of shape and appearance learned by implicit scene representation models, and the smoothness and compactness priors present in these models, allow for ultra-efﬁcient, scene-wide propagation of user-provided labels [2].
We remove the human from the loop entirely, presenting the ﬁrst fully-autonomous, 3D scene understanding robot that combines active exploration and physical experimentation with a uniﬁed neural-ﬁeld representation as its underlying computational model.
The robot explores a scene with an RGB-D camera and simultaneously builds a map of semantic entropy, representing conﬁdence in its current predictions.
Considering this entropy, as well as kinematic feasibility and collision avoidance, the robot selects the optimal point of interaction and proceeds to take a physical measurement used for optimising the semantic head of the underlying MLP.
The real-time rendering of entropy guides the robot in selecting efﬁcient and beneﬁcial interactions, allowing for dense scene labelling with only a handful of experiments.
In contrast to iLabel [2], which assumes a static scene, and can therefore rely on a growing collection of keyframes over which to optimise semantics, the interactive nature of our framework results in a dynamic scene.
In addressing this, we make the observation that neural representations possess temporal memory, and thus propose a new strategy that uses only the latest keyframe during optimisation.
In this way, the neural representation remains current, allowing for continuous exploration in scenes with limited dynamics.
Autonomous discovery of physical scene properties potentially facilitates a broad range of practical downstream robotic tasks.
For example, automated tool exchange in a surface-cleaning robot; object and material sorting for automated sorting of recycling or laundry; improved task planning enabled by understanding object affordances and physical properties of objects; and improving the safety of autonomous robots by mitigating interactions with potentially hazardous objects or materials.
In summary, the key contributions of this paper are: 1) the ﬁrst fully-autonomous, neural-scene labelling robot, trained from scratch, in real-time, and capable of operating in the real-world; 2) active robotic experimentation achieved using a single MLP to drive entropy-guided automatic query generation, kinematic feasibility checking, collision avoidance and physical interaction planning and execution; 3) demonstration of the temporal memory characteristics of neural ﬁeld representations via single-frame optimisation and robustness to small temporal scene changes and 4) prediction of dense, continuous-valued semantics using a neural-ﬁeld model optimised with respect to sparse ground truths.
We substantiate these contributions with a series of real-world experiments, demonstrating the capability of our system to produce a variety of perceptual maps or scene labellings, based on the chosen sensor and action primitives.
Related work
We address the problem of fully-autonomous, unsupervised scene understanding via active experimentation.
Our solution operates at the intersection of several research domains: instance segmentation, interactive perception, affordance learning, and online scene understanding.
Instance segmentation:
Unseen Object Instance Segmentation (UOIS) is a necessary capability for robots operating in unstructured environments.
Xie et al.
[3] propose a class-agnostic RGBD instance segmentation method, using only simulated training data with reasonable sim-to-real generalisation.
Xiang et al.
[4] improve sim-to-real transfer by learning RGB-D feature embeddings from synthetic data using a contrastive loss, which allows their Unseen Clustering Network (UCN) to capitalise on both synthetic depth and non-photorealistic RGB data.
Gouda et al.
[5] ﬁne-tune a Mask
R-CNN [6] model to perform class-agnostic instance segmentation.
The current state-of-the-art in
UOIS is achieved by reﬁning the outputs of the UCN method with the graph-based RICE method [7].
Interactive object segmentation: Bohg et al. [8] ﬁrst provided a detailed deﬁnition of Interactive Perception (IP) and an analysis of its beneﬁts in solving robotic perception tasks. A common approach in interactive object segmentation is to exploit motion cues induced by physical interactions [9, 10, 11, 12].
While our work shares the broad objective of segmentation through interaction, it is not limited to rigid objects or segmenting based on geometric extent.
Instead, we demonstrate segmentations that are optimised for given downstream tasks, dictated by the chosen action primitive and sensor.
Interactive affordance learning: Le Goff et al.
[13] learn affordance-speciﬁc relevance maps, describing which actions can be applied at which locations.
The method relies on an off-theshelf point-cloud segmentation [14] and an online classiﬁer [15].
Nagarajan and Graumann [16] simultaneously train an interaction exploration policy and an image-based affordance segmentation model mapping image regions to the likelihood that they permit an action.
Both these prior works limit experimentation to simulated environments.
Although we do not consider affordances explicitly in this work, it is the ultimate objective of performing task-driven segmentation in the way that we do.
Online scene understanding and labelling: The iLabel framework [2] (built upon iMAP [17]), demonstrated that online, scene-speciﬁc training of a compact MLP model, which jointly encodes scene geometry, appearance and semantics allows for ultra-sparse interactive labelling and produces accurate dense semantic segmentations — outperforming existing pre-trained methods.
We remove the human-in-the-loop entirely, demonstrating the ﬁrst fully-autonomous neural-scene labelling robot.
To achieve this, we introduce several signiﬁcant contributions to allow for operation on a real-world robot, where scene dynamics, kinematics, constraints and interaction feasibility must be considered.
Method
We represent 3D scenes similarly to iMAP [17], with an MLP that maps a 3D coordinate to colour and volume density.
We build a sparse set of keyframes, selected incrementally based on information gain, and whose viewpoints span the scene.
The MLP parameters and camera poses are jointly and continually optimised via differential volume rendering of actively-sampled sparse pixels in this set of keyframes.
iLabel [2] adds a semantic head to the neural scene MLP in iMAP, allowing users to provide pixel-level annotations in the keyframes.
The joint optimisation in iMAP is then extended to include scene semantics, which are optimised through semantic rendering of the user annotations.
Both models are trained from scratch, in real-time and without any prior data and have demonstrated the power of the joint internal representations learnt by the neural-ﬁeld MLP.
3.1 Autonomous robot experimentation
The robot builds an internal representation of its environment via a series of autonomous experiments.
First, it actively selects interaction locations that are both feasible and information-rich (based on semantic entropy).
Second, the selected 2D image locations are mapped to the real-world coordinate system of the robot, and a physical interaction with the scene is planned and executed.
Third, the resulting measurement is processed and/or classiﬁed (see speciﬁcs in Section 3.5) to obtain the ground-truth semantic label (akin to the user-provided object class in [2]).
Finally, using the labels obtained in this manner, scene semantics are optimised through semantic rendering of the robotselected keyframe pixels.
As demonstrated previously [2], the resulting joint internal representation of shape, appearance and semantics of the neural-ﬁeld allows for the sparsely-annotated semantics to be propagated efﬁciently and densely throughout the scene.
3.2 Modes of interaction
Our framework facilitates the autonomous discovery and mapping of any measurable characteristic of a scene, provided that a suitable measurement sensor and interaction protocol can be deﬁned.
We demonstrate three particular interaction types: 1) predicting rigidity by top-down poking; 2) predicting material type using a single-pixel multiband spectrometer 1 ; and 3) predicting frictional force distributions by lateral pushing.
These modes constitute the basis of our fully automatic system described in the following sections.
3.3 Entropy-guided interactions
In [2], qualitative experiments demonstrated that well-placed user clicks, especially in regions where the model is performing poorly, are the most beneﬁcial in terms of improving segmentation quality.
This observation was exploited in an automatic query generation framework, whereby an uncertaintybased sampling was used to actively propose pixel positions for the user to label.
Similarly, we utilise softmax entropy to guide the physical interactions that the robot makes with the scene, encouraging interactions that are optimal in terms of information gain and thereby minimising the number of interactions required to produce optimal segmentations.
Softmax entropy, u S , is deﬁned as [18]:
[Equation]
with ˆ S c [ u, v ] the rendered semantic distribution and C the number of categories.
3.4 Interaction feasibility
In order to integrate uncertainty-based sampling into fully-autonomous robotic behaviour, kinematic feasibility and the particular mode of interaction have to be considered.
Queries deemed to be kinematically infeasible and/or unsuitable given the mode of interaction, are added to a query mask, preventing repeated testing.
For example, top-down interactions require masking of candidate points not approximately perpendicular to the surface normal, while the converse is true for lateral interactions.
Similarly, sensor type impacts interaction feasibility.
For example, when performing spectroscopic measurements, we mask regions of high curvature to encourage measurements perpendicular to the target surface.
More generally, we have found that interactions at object boundaries are generally undesirable, owing to a higher incidence of failures associated with misclassiﬁcations and object tipping.
We have therefore investigated two approaches to reduce the probability of boundary interactions: 1) Gaussian ﬁltering of the uncertainty maps and 2) uncertainty clipping.
While Gaussian ﬁltering yields reliable behaviour in scenes containing coherent objects, it performs poorly with overlapping objects (e.g. fabrics/clothing).
In contrast, clipping the uncertainty map by setting values above a threshold to zero produces more reliable results.
The robot focuses on regions of high uncertainty but ignores problematic regions of maximal uncertainty.
There is a trade-off between the degree of boundary suppression and uncertainty map degradation.
We determined empirically an optimal clipping threshold of 0.7 and kernel size of 41.
3.5 Semantic representation
In contrast to [2], the inputs to the semantic head of our neural-ﬁeld MLP can be one of several physical properties, measured via apposite affordances and modes of interaction.
Raw sensor measurements acquired by the robot need to be post-processed or converted into the target variable being predicted by the semantic head of the MLP.
For binary prediction tasks (e.g. rigidity), this may be as simple as applying a threshold to the raw measurement.
Multi-class target variables may require additional processing.
For example, when predicting material type from a multidimensional spectrometer reading, we use a pretrained multiclass SVM classiﬁer which outputs predeﬁned material classes, which are then fed to the semantic head.
In both scenarios, as in [19, 2] the semantic head of the MLP predicts a categorical value and can be optimised using cross-entropy loss.
We additionally demonstrate for the ﬁrst time the prediction of continuous-valued target variables in the semantic head of the MLP, where the ground-truths are sparse, in contrast to the dense ground-truths used in the optimisation of the colour and density heads.
For example, when predicting frictional force distributions, we feed the minimum (stiction) force required to move an object, directly to the semantic head and optimise using an L 1 loss.
3.6 Single frame optimisation
During lateral pushing, interactions between the robot and the scene may introduce object displacements.
This violates the static-scene assumption common to prior work in neural radiance ﬁelds.
While this assumption allows [17, 2] to optimise over an expanding set of keyframes, a dynamic scene potentially invalidates historic keyframes, ultimately leading to errors in the reconstruction.
We therefore clear the keyframe history and corresponding labels after each interaction in this mode.
Our experimentation has suggested that neural radiance representations possess some form of temporal memory characteristic over the labelled properties, whereby network weights adapt over time and maintain consistency with the dynamic scene, provided scene changes are comparatively small.
Experiments
A fundamental contribution of our work is the realisation of a fully-autonomous robot that operates in a real-world environment.
We demonstrate the ability of our system to perform a series of autonomous experiments, using the aforementioned interactive modes, to discover and predict a variety of physical scene properties.
We demonstrate the quantitative beneﬁts of entropy-guided experimentation and, in the case of rigidity and material-type classiﬁcation, compare segmentation performance against two state-of-the-art, class-agnostic segmentation techniques (see Sec.
4.2 for details).
Finally, we refer the reader to our supplementary video for additional results.
4.1 Experimental setup
We use a Franka Emika Panda robot, anchored to a table on which a variety of objects are arranged (Figure 1).
The robot is equipped with a Realsense D435 RGB-D sensor [20], tracked using the forward kinematics of the arm, which is controlled using ROS [21].
Prior to physical experiments, the robot builds a geometric reconstruction of the scene, to allow for collision-free motion planning.
For this purpose, a set of RGB-D keyframes is captured over a series of random motions in order to optimise the 3D neural ﬁeld, and subsequent querying of the network produces a collision mesh and normal map.
All objects of interest are placed within reach of the robot arm and any points located beyond this range, or on the plane of the table, are automatically labelled ‘table’.
4.2 Quantitative results
Rigidity and material-type prediction may be viewed as segmentation problems.
While training any popular instance segmentation technique (e.g. Mask R-CNN [6]) on the object classes present in our scenes (e.g. material types), is likely to produce high-quality segmentations, one would need to repeat this training for each scenario.
Therefore, instead of comparing against closed-set segmentation techniques, we consider two state-of-the-art class-agnostic instance segmentation approaches: 1)
Mask R-CNN trained to perform class-agnostic segmentation [5] and 2)
Unseen Clustering Network (UCN) [3] with RICE reﬁnement [7] (see Sec. 2 for details).
For each method, we perform instance segmentation on the keyframe and use the resulting instance mask to guide the robot-scene interaction.
In particular, the robot takes a single sensor reading as near to the centre of each instance in the mask as is feasible and propagates the measurement to the rest of the region.
The measurements are converted to categorical labels (binary rigidity or material-type) in the same manner as described in
Sec.
We report the mean Intersection over Union (mIoU) averaged over the ground-truth labels.
Table 1 shows the quantitative performance comparison against the Mask R-CNN and UCN baselines for each scene.
As expected, the baselines perform well for scenes 1 and 3, which contain geometrically-coherent objects and strong colour and depth cues.
Scene 2, however, is considerably more challenging for the colour and/or depth-based baselines, characterised by a signiﬁcant drop in performance.
In contrast, our autonomous approach performs well for all three scenes, with comparable results to the baselines in Scenes 1 and 3 and signiﬁcantly superior results in Scene 2.
4.3 Entropy exploration ablation study
Figure 2 illustrates material discovery in a complex scene containing wool (blue), cotton (yellow) and synthetic (green/pink) materials.
Prior to physical measurements, there is high uncertainty (red) across the entire scene, while the ﬁnal uncertainty map has high conﬁdence (blue) throughout.
We show the evolution of the uncertainty map through three consecutive interactions in Figure 3.
The robot is guided to a high-entropy pixel (unﬁlled circle).
On completion of the experiment, there is a clear, localised uncertainty reduction surrounding the target region (ﬁlled circle).
We observe that while the uncertainty in the localised region of measurement decreases, it often increases in more distant regions.
As the model accumulates information, it continuously adapts its predictions and corresponding conﬁdence, ultimately converging on an accurate representation.
This observation motivates the use of uncertainty as an exploration metric in neural implicit representations.
To substantiate the beneﬁts of entropy-driven exploration quantitatively, we conducted an ablation study comparing performance to random exploration.
We compare the evolution of mIoU and falseconﬁdence (where the model produces high-conﬁdence but incorrect predictions) with an increasing number of interactions for each technique.
Figure 5 demonstrates superior convergence rates for entropy-guided interaction in all three benchmark scenes in Figure 4 across both metrics.
4.4 Force measurement analysis
Figure 6 illustrates the stiction force maps produced by our framework for objects with uniform (top row) and non-uniform (bottom row) mass and friction distributions.
Note that in each scene the objects are displaced following the pushes performed by the robot.
The scene in the top row contains three cylindrical containers of varying mass and material.
As desired, the resulting force renderings match the varying masses.
This is potentially valuable information when planning for downstream manipulation tasks (e.g. distinguishing between full and empty containers).
A key observation in
Figure 6 is that the renderings for objects remain consistent despite displacement, demonstrating for the ﬁrst time a memory quality in neural ﬁeld representations.
In the bottom row of Figure 6, we demonstrate the ability of the robot to predict stiction force values reliably, even for complex geometries, with non-uniform mass and friction distributions.
We substantiate this quantitatively in Figure 7, where the robot interacts with a non-uniform rectangular box containing a 5 kg weight at one end.
We show that the output of our model, after three pushes, is comparable to that of a simple analytical physics model [22] with access to privileged information, including the contact surface area, mass distribution and friction coefﬁcient.
[Figure] [Figure]
highest uncertainty pixel. After interaction (ﬁlled marker), there is a localised reduction in uncertainty. [Table] [Figure]
[Figure]
Limitations
Several assumptions are made during the experimentation procedure: 1) rigid objects are never placed on top of ﬂexible objects; 2) the SVM classiﬁer has been trained on all materials and 3) pushed objects are both rigid and free to move across the tabletop.
The main limitations of our system relate to its self-awareness of errors.
We only include examples in which all measurements and inferences are correct.
In reality, this is not always the case, with each mode of measurement having an associated accuracy, and incorrect measurements impacting model performance.
We observed that, although the model recovers after an erroneous measurement, it involves signiﬁcantly more interactions.
An alternate solution would be to associate a likelihood with each measurement and allow the model to adapt accordingly.
Furthermore, with all scene interactions, there is a possibility of catastrophic disturbance, preventing the model from propagating labels effectively.
This scenario was observed several times during experimentation, requiring a manual reset.
Ideally, failures of this nature should be detected autonomously based on the corresponding surge in reconstruction loss.
Although several keyframes are captured during robot motion, interaction queries were only ever generated from the ﬁrst keyframe, which in our setup was predeﬁned as the initial viewpoint of the robot.
This has the advantage of limiting the attention of the robot to what we deemed to be the most interesting part of the scene.
However, it also meant the robot was unable to conduct [Figure] [Figure] wider investigation; in particular, the back surfaces of objects or those regions occluded in the initial viewpoint.
This was most evident in lateral pushing, where pushing from multiple angles could only be achieved with a careful choice of viewpoint and object positioning.
A future improvement would address this by allowing queries across multiple keyframes, though the robot will still be constrained by its kinematic limits.
The sensitivity of force measurement is limited by the robot joint torque sensors and subsequent estimation of external torques and wrenches.
This could be improved by using an external forcetorque or tactile sensor.
Using a spectrometer with greater bandwidth would allow for enhanced differentiation between material types.
Finally, while we propose that our system facilitates the discovery of a large variety of physical scene properties, we only consider three speciﬁc scenarios in this paper.
Conclusions
By taking advantage of the unsupervised decomposition capabilities of a 3D neural ﬁeld trained on a single scene, we have demonstrated a fully-automatic, real-world robotic platform that is able to build dense, accurate and physically meaningful representations of its environment.
The fact that this can be achieved from scratch and in real-time, without any pre-training or external guidance, gives the method potentially wide applicability in robotic scene learning and understanding.
In the future, we are excited to extend the system with new interactions, such as probing a scene with a gripper to build a grasping map, or using alternative sensors to measure properties such as temperature or surface roughness. Acknowledgments
Research presented in this paper has been supported by Dyson Technology Ltd.
We would like to thank Charles Collis at Dyson Technology Ltd. for his support and in enabling the secondment of
Iain Haughton from Dyson Technology Ltd. to the Dyson Robotics Lab, Imperial College.
We also wish to thank Barry Beard at Dyson Technology Ltd. for his help with using the spectroscopy sensor for material classiﬁcation.
Finally, we would like to thank Tristan Laidlow, Daniel Lenton, Kentaro
Wada and Shuaifeng Zhi who engaged in research discussions as well as the reviewers who provided valued feedback on our work.