Multi-Robot Scene Completion: Towards Task-Agnostic Collaborative Perception
Abstract: Collaborative perception learns how to share information among multiple robots to perceive the environment better than individually done.
Past research on this has been task-speciﬁc, such as detection or segmentation.
Yet this leads to different information sharing for different tasks, hindering the large-scale deployment of collaborative perception.
We propose the ﬁrst task-agnostic collaborative perception paradigm that learns a single collaboration module in a selfsupervised manner for different downstream tasks.
This is done by a novel task termed multi-robot scene completion , where each robot learns to effectively share information for reconstructing a complete scene viewed by all robots.
Moreover, we propose a spatiotemporal autoencoder (STAR) that amortizes over time the communication cost by spatial sub-sampling and temporal mixing.
Extensive experiments validate our method’s effectiveness on scene completion and collaborative perception in autonomous driving scenarios.
Our code is available at https://coperception.github.io/star/ .
Introduction
Single robot perception has been widely studied on tasks such as object detection [1] and semantic segmentation [2].
However, it suffers from various challenges, such as occlusion and sparsity in raw observations.
Collaborative perception is promising to alleviate those issues.
It provides more environment observations from different perspectives by information sharing to improve perception performance and robustness.
Amongst different collaboration strategies, feature-level collaboration [3, 4, 5] transmits the intermediate representations generated by deep neural networks (DNNs) of each robot.
Since these intermediate features are easy to compress and can preserve contextual information of the scene, feature-level collaboration demonstrates better performance-bandwidth trade-off compared to raw-data-level and output-level collaboration [6, 7].
However, existing feature-level collaboration methods [8, 4, 3] are fully supervised by task-speciﬁc losses to learn the entire model, including a feature extractor, a collaboration module, and a decoder, as shown in Fig.
1 (a).
Such a task-speciﬁc framework requires re-training the whole model for different perception tasks.
Besides, existing collaborative perception requires training data recordings to be synchronized among all robots in time, which is more demanding than data collection in single-robot perception.
How can we design a collaborative perception framework (1) independent from downstream tasks and (2) trainable from asynchronous datasets?
To answer this question, we propose a novel self-supervised learning task termed multi-robot scene completion .
It enables multiple robots to collaboratively use an autoencoder to reconstruct a complete scene based on shared latent features. The completed scene could then be fed into various
[Figure]
downstream tasks without additional training , as shown in Fig.
1 (b).
This allows us to decouple the collaboration training from downstream task learning.
Moreover, it seamlessly supports synchronous and asynchronous training datasets with different learning objectives: complete scene reconstruction if synchronous and individual view reconstruction if asynchronous.
Yet naive autoencoders are not designed to balance scene reconstruction performance and communication volume, which is an established criterion to evaluate collaborative perception.
To address this challenge, we further design a spatiotemporal autoencoder (STAR) inspired by the recent masked autoencoders (MAE)
It reconstructs a scene using a spatiotemporal mixture of patch tokens: some tokens are encoded from randomly sub-sampled patches in the current frame and others are cached from the past.
The sampling ensures that all patches in the mixture can jointly cover the whole spatial region while being self-disjoint.
This allows each robot to only transmit the sub-sampled tokens in the current frame instead of the entire latent feature maps, leading to much lower communication bandwidth than prior works.
Our key insight behind such an amortized communication cost is that features of many patches (e.g., static or nearly static) do not need to be shared in every frame.
In summary, our main contributions are threefold:
• We propose a brand-new task-agnostic collaborative perception framework based on multi-robot scene completion, decoupling the collaboration learning from downstream tasks.
• We propose asynchronous training and synchronous inference with a shared autoencoder to solve the proposed task, eliminating the need for synchronous data for collaboration learning.
• We develop a novel spatiotemporal autoencoder (STAR) that reconstructs scenes based on temporally mixed information.
It amortizes the spatial communication volume over time to improve the performance-bandwidth trade-off.
• We conduct extensive experiments to verify our method’s effectiveness for scene completion and downstream perception in autonomous driving scenarios.
Related Works
Collaborative perception.
Collaborative perception has been proposed to improve individual perception’s ﬂexibility, resilience, and efﬁciency [10, 11, 12].
With recent advances in deep learning, researchers have developed feature-level collaborative perception in which intermediate representations produced by deep neural networks (DNNs) from multiple viewpoints are propagated in a team of robots, e.g. , a swarm of drones [8, 3] or a group of vehicles [4, 6].
Existing works commonly consider a speciﬁc downstream task and use the corresponding loss function to learn a collaboration module such as a graph neural network (GNN) [4, 3], a Transformer [13, 14], and a convolutional neural network [5, 15].
Several downstream tasks have been investigated in collaborative scenarios, such as object detection [4], semantic segmentation [8], and depth estimation [3].
Unlike the existing task-speciﬁc collaborative perception sharing task-dependent representations, we deﬁne task-agnostic 3 collaborative perception as the feature-level collaborative perception sharing task-independent representations amongst multiple robots.
Scene completion.
Autonomous navigation [16] requires robots to understand the geometry and semantics of 3D scenes.
However, vision sensors only capture partial observations because of a limited ﬁeld of view and sparse sensing, leading to an incomplete spatial representation.
Therefore, scene completion (SC) has been proposed to infer the complete 3D scene geometry given sparse 2D/3D observations [17, 18, 19].
Following scene completion, semantic scene completion (SSC) has been introduced to jointly estimate both geometry and semantic information based on partial observation [2, 20, 21, 22].
On the one hand, single robot scene completion can rely on prior semantic knowledge to complete the partially-observed objects.
On the other hand, it is unrealistic to see through full occlusions.
Unlike single robot scene completion depending on prior knowledge, the multi-robot scene completion task utilizes information shared by teammates for scene completion.
Self-supervised representation learning.
Self-supervised representation learning (SSRL) aims to provide powerful features without the need for massive annotated datasets [23].
SSRL is generally composed of: (1) task-agnostic pre-training via carefully-designed self-supervised pretext tasks such as contrastive learning [24, 25] or autoencoding [26, 27, 9], and (2) task-speciﬁc adaptation to ﬁnetune the pre-trained model on the downstream tasks such as object detection or image classiﬁcation.
Masked autoencoder (MAE) achieves great performance with a simple reconstruction objective [9].
It employs an asymmetric architecture with a large encoder that only processes unmasked patches and a lightweight decoder that reconstructs the masked patches from the latent representation.
Recent works extend MAE into multimodal representation learning [28, 29], video [30, 31], and 2D image completion [32].
In this work, we employ similar autoencoding to learn the shared representations and enable quick adaptation to the downstream perception: the reconstructions could be seamlessly utilized by off-the-shelf individual perception model trained on single-view data without any ﬁne-tuning, bridging the gap between collaborative perception and individual perception.
Multi-Robot Scene Completion: Motivation, Formulation, and Evaluation
Motivation.
Even though single-robot data already requires arduous annotations like 3D bounding boxes and pixel-wise semantic labeling, multi-robot data even demands multiple times as much work.
To relax the task-dependent supervision for collaboration learning, we propose multi-robot scene completion to enable task-agnostic collaborative perception.
It can utilize self-supervision to learn shared representations instead of expensive task-dependent supervision.
We will introduce its overall workﬂow, training objective, and evaluation metrics hereafter.
Problem setup.
We consider N robots in the same geographical location simultaneously perceiving the 3D environment, such as a ﬂeet of autonomous vehicles located at a crossroad.
These robots communicate with each other about their observations to better understand the surrounding environment.
Each robot indexed by i is equipped with a 3D sensor such as a LiDAR to generate a binary occupancy grid map M i ∈ { 0 , 1 } H × W × C deﬁned in its local coordinate, where H , W , and
C respectively denote the length, width, and height resolution.
Feature extraction.
We employ intermediate collaboration with better performance-bandwidth trade-off [5].
Each robot encodes its observation into a feature map denoted by F i =
Θ ( M i ) , where Θ is a feature extractor.
Now F i ∈ R ¯ H
× ¯ W × ¯ C has lower spatial resolution ¯ H × ¯ W , while keeping a higher feature dimension ¯ C compared to the original map M i .
Then, each robot will broadcast F i to its peers as well as its pose ξ i ∈ se (3) deﬁned in the global coordinate.
Feature decoding.
The robot i receives the messages from the neighboring robots { F j , ξ j } j (cid:54) =
i , and then uses a decoder Φ and a pose-aware aggregator A for fusion, and output a completed occupancy grid map
ˆ Y i =
Φ ( A ( F i , ξ i , { F j , ξ j } j (cid:54) = i )) , where ˆ Y i has the same dimension and describe the same spatial range as M i yet is a more comprehensive spatial representation for the scene.
The pose-aware aggregator A transforms the feature maps of robot
j ( j (cid:54) = i ) into the coordinate of the target robot i , then sums all the coordinate-synchronized feature maps:
[Equation]
where Γ j → i ∈ SE (3) is the transformation from robot j ’s coordinate to i ’s, obtained by the exponential map of poses ξ j and ξ i .
Bi-linear interpolation is used to transform a discrete map, and the positions out of the spatial range ¯ H × ¯ W after transformation are padded with zero.
Training loss.
We treat the scene completion task as a binary classiﬁcation problem and use crossentropy loss to train a neural network composed of Θ and Φ .
Speciﬁcally, the ground-truth Y i ∈ { 0 , 1 } H × W × C deﬁned in the coordinate of robot i represents a multi-view occupancy voxel grid with two classes, i.e., free and occupied.
Therefore, the loss can be computed by:
[Equation]
where i is the robot index, k is the voxel index, L is the total number of the voxel ( L = H × W × C ), c is the number of class (2 in our case), ˆ y i,k,c is the predicted logits for the k -th voxel belonging to class c , y i,k,c is the k -th element of Y i and is a one-hot vector ( y i,k,c = 1 if voxel k of robot i belongs to class c ).
Here we show the training objective using synchronous multi-robot data created by aggregating multi-robot observations based on robots’ poses similar to Eq.
(1): Y i =
A ( M i , ξ i , { M j , ξ j } j (cid:54) = i ) ( j < M ) , where M is the number of robots for the ground truth generation.
Note that Θ and Φ can also be trained by individual view reconstruction on asynchronous data as shown in Eq.
(3), while being deployed on synchronous data for inference as discussed in Section 4.
Hallucinating the invisible scene is possible if the ground-truth Y i is generated with more robots than those involved in the training phase, i.e. , M >N .
Yet only synchronous training can achieve this because the ground truth requires aggregation of different viewpoints obtained synchronously.
Evaluation metrics.
We follow the evaluation protocol in single-robot scene completion [19, 33], which uses the voxel-level intersection over union (IoU) between predicted voxel labels ˆ Y
i and ground truth labels Y i for each robot.
Note that only non-empty voxels are evaluated.
STAR: Spatiotemporal Autoencoder
In addition to the multi-robot scene completion task, we also propose a novel architecture called
S aptio t emporal a utoencode r (STAR) to tackle this problem.
We will present our key design motivation, detailed modules, training, and inference procedures.
4.1 Design rationale
Partially broadcasting.
Inspired by the idea of ”masking” in MAE [9], we employ a similar asymmetric design as MAE yet with different purposes: MAE is to design a nontrivial self-supervisory task for pre-training via randomly masking, while the goal of STAR is to reduce the communication volume in multi-robot systems via partial broadcasting.
More speciﬁcally, STAR deploys an encoder at the sender robot to map the entire observation to an intermediate feature representation that is selectively transmitted to lower the bandwidth.
Meanwhile, STAR deploys a decoder at the receiver robot that reconstructs the original observation from the received partial representation.
Spatiotemporal amortization.
Simply applying random masking to a complete observation is not a good idea.
Unlike MAE mainly for object-level recognition, we aim at large-scale dynamic scene modeling.
Once objects are completely masked during encoding, the decoder cannot hallucinate the corresponding objects without such knowledge.
Therefore, we propose to exploit historical tokens to replace mask tokens during decoding.
In this way, we can amortize the communication cost over the temporal domain by spatial sub-sampling and temporal mixing.
Speciﬁcally, at the sender’s side, a part of the spatial patches are sampled from the entire observation in each timestamp.
Only the features of these subsampled spatial patches are communicated with the teammates.
Upon receiving the transmitted patch features, the receiver’s decoder combines these features with the historical patch features and reconstructs an entire observation.
Since the patches are subsampled in a spatially complementary manner, the temporally mixed patches jointly cover the whole spatial region.
Synchronization-free training.
Traditional collaborative perception approaches consider a synchronization training strategy that requires synchronous (potentially with a slight temporal latency) multi-robot recordings to train a feature-space collaboration strategy with task-speciﬁc loss functions [4, 5].
In this work, we try to relax the requirement that multiple robots simultaneously capture perception data by using single-view observations as the supervision during the training phase.
We will describe this in detail in section 4.3.
Asynchronous Training (Ours) Synchronous Inference … 𝑴 𝒊,𝒕
Completed Multiview Scene
Dec.
Enc.
Synchronous Training
Task Specific Losses
Scene Reconstruction Loss
Pose-aware Aggregation 𝑭 𝒋,𝒕 , 𝝃 𝒋 𝑭 𝒊,𝒕 , 𝝃 𝒊 𝑴 𝒊 𝑴 𝒋 𝑴 𝒌 Single-robot Perception Model 𝑴 𝒋,𝒕 𝑭 𝒊,𝒕 𝑭 𝒋,𝒕 𝑭 𝒊,𝒕 𝑭 𝒊,𝒕$ 𝑭 𝒋,𝒕 𝑭 𝒋,𝒕$𝟏 𝑭 𝒊,𝒕 𝑭 𝒊,𝒕$ 𝑭 𝒋,𝒕 𝑭 𝒋,𝒕$ 𝝃 𝒊 𝝃 𝒋 𝑴 𝒌,𝒕 𝑭 𝒌,𝒕 𝕋 𝕋 E n c . E n c .
D ec . E n c . E n c . D ec .
E n c . D ec . E n c . D ec .
D ec . Task Specific Outputs
E n c . D ec . E n c .
D ec . S S
S
4.2 Architecture
We consider a set of robots deploying the same neural network following [4, 5].
Each robot serves as both message sender and receiver during collaboration and is equipped with an encoder for observation abstraction and a decoder for view reconstruction.
STAR encoder.
Different from MAE [9], the STAR encoder uses a vision transformer (ViT) [34] backbone, which operates on all patches yet only sends out a subset ( spatial sub-sampling ).
Specifically, the entire grid map for robot i at time t denoted by M i,t is divided into multiple patches, and each patch is encoded with a linear projection with additional positional embedding, and then processed using a series of Transformer blocks to generate the ﬁnal message F i,t .
Note that we adopt a complementary transmission strategy in the temporal domain regarding the patch index ( i.e. , the observed spatial locations) to avoid the loss of information for the dynamic scenes.
STAR decoder.
Different from MAE which uses mask tokens to replace the missed patch embeddings, robot i as a receiver aggregates the historic tokens F j,t − 1 and the current tokens F j,t from robot j ( temporal mixing ), which approximately form a complete observation towards the entire spatial range.
Temporal embeddings are added to the tokens from the respective timestamps to enhance the temporal awareness before feeding them into Transformer blocks.
Note that here we use twotimestamp t and t − 1 as an example for a simple explanation.
The STAR decoder is also able to process more historical timestamps.
After decoding all robots’ views { ˆ M j,t } j (cid:54) =
i , the ultimate prediction of the complete view is computed by coordinate synchronization: ˆ Y i
=
A ( M i,t , ξ i , { ˆ M j,t , ξ j } j (cid:54) = i ) , and its calculation process is similar to Eq.
4.3 Training and inference
Asynchronous training.
The model is trained with single view ground-truth M i , and adopt crossentropy loss during training:
[Equation]
[Equation]
where i is the robot index, k is the voxel index, L is the total number of the voxels, c
= 2 is number of class, m i,k,c denotes the k -th element of M i and is a one-hot vector same as y i,k,c in
Eq. 2, ˆ m i,k,c is the prediction for the k -th voxel belonging to class c .
Note that the training loss is calculated voxel-wise with respect to the self-supervision signal from each robot’s single-view observation.
This design decouples the training phase from communication with other robots: the model on each robot does not require synchronous observations from neighbor robots in the training phase, making the training asynchronous (asynchronous training in Fig. 2).
This is greatly different from the training framework in previous collaborative perception works such as [5] (synchronous training in Fig. 2).
Our training framework can relax the need for the carefully-collected and hardto-annotate multi-robot dataset and can exploit a large amount of single-robot data to learn powerful as well as compact feature representations.
Synchronous inference.
During inference, each robot is equipped with the same model.
The sender robots’ encoders will encode and broadcast a subset of their current timestamp’s observation.
Then, the decoders on the receiver side will leverage the transmitted intermediate representation along with the pose information to reconstruct the corresponding view, optionally with historical features as described above.
Then, the receivers use corresponding pose information to aggregate the single observations into a multi-view completed scene.
We illustrate the pipeline on the left side of Fig. 2.
Experimental Results 5.1 Experimental setup
Dataset.
We conduct experiments on the V2X-Sim Dataset [15], a large-scale dataset that simulates urban multi-vehicle driving scenes with CARLA [35].
We use 80 scenes for training and ten scenes for testing.
The dataset is sampled at 5 Hz.
We pre-process the voxels grids with range [ − 32 m, 32 m ] in the x and y-axis and [ − 3 m, 2 m ] in the z-axis.
Finally, we can get the voxel grids with a spatial resolution of 256 × 256 × 13 .
Baselines.
The lower-bound refers to a single-robot perception model trained and tested using only individual observations.
The task-speciﬁc models all optimize the collaboration based on the speciﬁc perception head.
When2com [8] uses the attention mechanism to fuse the collaborators’ information.
Who2com [36] employs a handshake mechanism.
The V2VNet [4] trains a graph neural network to propagate the agent’s information.
DiscoNet [5] selectively fuse messages from the informative regions.
For task-agnostic models , we use a modiﬁed FaFNet [37] backbone as the
CNN baseline and substitute the detection head with a classiﬁcation head that outputs the logits for binary classiﬁcation.
VQ-VAE [26] learns a variational autoencoder to reconstruct the scene and employs a vector quantization technique to reduce communication costs.
Implementation details.
A 6-block ViT encoder with hidden dimension 384 is used for the STAR encoder.
Then an MLP is used to compress the intermediate representations to 32 dimensions and feed them to the decoder, where they are projected back to 256 dimensions and sent to a 4-layer transformer decoder.
An FaFNet [37] is used for single-robot object detection.
A UNet [38] serves the same purpose for the semantic segmentation task.
Note that all the perception models take the three-dimensional voxel grids as input and output results in bird’s eye view (BEV), such as bounding boxes and semantic labels.
Our models are all trained on single-view data.
Evaluation metrics.
For the scene completion task, we measure the completion quality using the intersection-over-union (IoU) at three different scales by down-sampling the voxels accordingly.
For the perception task, we report the average precision (AP) at thresholds 0.5 and 0.7 for vehicle detection, IoU for the vehicle category, and the overall mIoU for semantic segmentation.
5.2 Quantitative results on scene completion
We present the quantitative results of the multi-robot scene completion task in Table 1, measured by
IoU at different scales and the corresponding communication bandwidth.
Spatial resolution.
Among the three tested resolutions, we can see that, in general, a higher spatial resolution leads to a better completion quality: the spatial resolution 32 × 32 which has a patch size of 8 achieves the best performance.
Timestamps.
Our method allows the multi-robot system to amortize the spatial communication bandwidth over the temporal domain.
From Table 1, we can see that from timestamps 1 to 4, the performance only varies slightly while largely reducing the bandwidth.
Bandwidth .
Bandwidth is calculated to reﬂect the required data volume for communication per second.
A trade-off between performance and communication is clear.
The CNN baseline requires much higher bandwidth because multiple feature maps are transmitted during communication due
[Table] [Table]
[Table] [Table]
to the skip connections in the model.
STAR requires much lower bandwidth.
A ﬁner-grained spatial resolution with better performance requires a higher bandwidth.
5.3 Ablation studies on scene completion
We conduct several ablation studies to investigate the effectiveness of the key components in our method.
Results are presented in Table 2 and are discussed in detail below.
Patches to encode.
As shown in Table 2a, only encoding the patches that will be transmitted can result in a minor drop in performance.
Yet it can reduce some computations for computation-restricted robotic systems.
Timestamps to decode.
We investigate the effect of whether the decoder incorporates previous timestamps or just the current single timestamp combined with learnable mask tokens.
Results in
Table 2b indicate that historical information is essential.
Temporal embedding.
In the STAR decoder, we add temporal embedding to the patches of different timestamps, similar to the approach in [30, 31].
The ablation study in Table 2c shows that adding temporal embedding is beneﬁcial.
Masking strategy.
We compared our complementary masking strategy with the random masking strategy proposed in MAE [9] in Table 2d.
Results show that switching from complementary to random masking leads to a degradation in the completion performance.
5.4 Quantitative results on downstream perception
We directly feed the single-view to the single-robot perception model termed lower-bound without any ﬁne-tuning, and the results are shown in Table 3.
Our best STAR method improves the lowerbound by 25 . 9% and 22 . 8% in object detection (AP@IoU=0.7) and semantic segmentation (IoU of vehicle) respectively.
Achieved by simply combining the completion model with off-the-shelf single-robot perception models, these improvements are promising because our framework: (1) has no knowledge about downstream tasks ( task-agnostic ); (2) does not require synchronous data in the training phase ( synchronization-free ); (3) is learned without manual annotations ( self-supervised ).
We also investigated the single-robot perception model directly taking ground truth multi-view measurements without additional training, termed upper-bound , and ﬁnd that it can achieve nearly com-
[Table]
parable performance with DiscoNet [5] and V2VNet [4], both trained with full supervision using synchronous data for speciﬁc tasks.
This demonstrates the potential of our proposed task: when the completions approach the ground truth scenes, it can perform similarly to the upper bound on many downstream tasks.
Moreover, using a stronger single-robot perception model can further enhance the ﬁnal performance.
5.5 Qualitative results on scene completion and downstream perception
We present a few qualitative results on the scene completion and the downstream tasks in Figure 3.
Due to the limited space, more visualizations can be found in the appendix.
[Table]
[Figure]
Limitation
There is still a performance gap between our method and the upper bound on the downstream perception due to the non-perfect scene completion.
We believe when trained with more single-robot recordings, our method is able to achieve comparable performance to task-speciﬁc approaches while maintaining excellent ﬂexibility.
Currently the spatial tokens are sub-sampled randomly at individual timestamp, and we believe this could be improved in the future.
We also inherit the common limitation in most existing collaborative perception works: all experiments are on simulated datasets due to the lack of public real-world datasets.
We further ignore the inﬂuence of pose noises, although previous works [5] already revealed reasonable robustness.
Conclusion
We propose the ﬁrst task-agnostic collaborative perception paradigm, where a single collaboration module is learned and can be transferred to different downstream tasks.
Our key observation is that we can move communication between robots to the temporal domain, which achieves an excellent performance-bandwidth trade-off.
Also, our self-supervised learning method sheds new light on collaborative perception that reduces the importance of human annotations.
Acknowledgments
We thank the anonymous reviewers for their valuable comments in revising this paper.
This work was supported by the NSF CPS Program under Grant CMMI-1932187 and CNS-