Paper: An Extensible Framework for Open Heterogeneous Collaborative Perception (ICLR-2024)

Reviewer comment (Presentation):
The content of the Section 3 is somewhat limited, please make some extensions or merge it with other sections.

Author response:
Thanks for your advice! The proposed open heterogeneous collaborative perception problem holds significant practical significance, and therefore, it warrants a individual section to introduce it. In the revision, we have further expanded upon its content to provide a more comprehensive introduction to this highly practical and novel challenge.

Reviewer comment (Reproducibility):
The article has not mentioned how to extract features from different modalities and convert them into the required BEV features.

Author response:
Thanks for pointing it out. The feature encoding methods are as follows:

1. LiDAR modality

- **PointPillars[1]** ($\mathbf{L}^{(x)}_\mathit{P}$ in the paper) divides the point cloud data into pillars, which are vertical columns of points. They are then converted to a stacked pillar tensor and pillar index tensor. The encoder uses the stacked pillars to learn a set of features that can be scattered back to a 2D pseudo-image for a convolutional neural network. This forms the BEV feature.

- **SECOND[2]** ($\mathbf{L}^{(x)}_\mathit{S}$ in the paper) begins by voxelizing the 3D point cloud data, creating a 3D grid of voxels. Each voxel contains the point cloud data within its boundaries. SECOND then uses sparse convolutions to process these voxels. Stacking the voxel features along the height dimension results in a BEV feature map.

2. Camera modality

- **Lift-Splat[3]** ($\mathbf{C}^{(x)}_E$ and $\mathbf{C}^{(x)}_R$ in the paper) represents a more recent and innovative approach for camera modality. It starts by "lifting" the image features from an image encoder (here $\mathbf{C}^{(x)}_E$ uses EfficientNet and $\mathbf{C}^{(x)}_R$ uses ResNet50) with a predicted depth distribution into 3D space. After this, it "splats" these features onto a 2D grid, effectively projecting features onto a BEV plane. In our paper, we use this feature projection mechanism with two different image encoders.

[1] Fast Encoders for Object Detection from Point Clouds. CVPR 2019.

[2] SECOND: Sparsely Embedded Convolutional Detection. Sensors 2018.

[3] Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D. ECCV 2020.

Reviewer comment (Reproducibility):
The article has not mentioned how to perform spatial transformation and alignment of the features from heterogeneous agents in different coordinate systems. Please clarify this.

Author response:
Since agent generates BEV features in its own coordinate system, the ego agent requires spatial transformation to unify all coming features into its own coordinate system. This step requires knowing the (i) relative pose $^i\xi_j \in \mathbb{R}^{4\times 4}$ between vehicles and (ii) the physical size of the BEV feature $D_x$, $D_y$ in meters. We assume these meta information (global pose and BEV feature map physical size) will be sent in V2V communication. Note that relative pose can be calculated between global poses. 

To perform the spatial transformation, we adopt the ```affine_grid``` and ```grid_sample``` functions from ```torch.nn.functional``` package. 

Since the feature is in BEV space, we deprecate the z-axis component in $^i\xi_j$ and normalize it with $D_x$ and $D_y$. It results in a normalized and unitless affine matrix, which will be the input of ```affine_grid``` and ```grid_sample``` together with the feature map. With these two functions executed continuously, we will transform the feature map from the other agent's ego perspective to our own coordinate, just like the affine transformation of a 2D image.

We also supplement this in the Appendix.

Reviewer comment (Experiments):
During the training of the new-type agent, the new agent seems not to get collaborative information from other agents, which may have an impact on perception performance. This should be explained.

Author response:
As the variety of agent types in the scene grows, training without the use of collaborative information does not exhibit drawbacks; instead, it has a beneficial impact.

This is because collaborative training may result in unbalanced and insufficient training in situations with numerous agent types. Consider this scenario: in the OPV2V-H dataset, each scene averages 3 agents, and our experiment involves 4 types of agents. Given the nature of the training data, such collective training is prone to unbalanced training. This remains the case despite our implementation of extensive data augmentation and setting more epochs to promote convergence. 

We have conducted relevant experiments on OPV2V-H as follows:

| Infer $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ + $\mathbf{C}^{(336)}_R$ on OPV2V-H | AP30 | AP50 | AP70 |
|----------------------------------------------------------------------------------------------------------------|------|------|------|
| Collective Training + Pyramid Fusion | 0.899 | 0.881 | 0.769 |
| Back. Align. Training + Pyramid Fusion (HEAL) | **0.908** | **0.894** | **0.813** |

We see that, when we use collective training for newly added heterogeneous agent types, the performance is suboptimal to single-agent training based on backward alignment. Backward alignment does not involve collaboration, thereby circumventing the issues of training imbalance and insufficiency in collaborative training. Individual training enables new agents to converge more rapidly, aligning them to a unified feature space for collaboration. In real situations, the number of agent types is always growing, and it becomes increasingly difficult to collect enough spatio-temporally synchronized multi-agent data including types. Therefore, training on a single agent is an advantage. Therefore, the advantages demonstrated by training on a single agent are even more significant.

Reviewer comment (Experiments):
On the other hand, the Pyramid Fusion module and detection head haven’t been affected by new type agents. It means that new-type agents did not participate in collaborative perception during the training process. This could be better clarified.

Author response:
Thank you for your suggestion. We would like to clarify that new-type agent training does not involve any collaborative agents. This important detail will be reiterated in the main text for emphasis.

Reviewer comment (Experiments):
The claim of the performance of the proposed Pyramid Fusion should be supported by experiment results and observations, please supplement relevant ablation experiments.

Author response:
We provide a detailed ablation study on Pyramid Fusion on OPV2V-H dataset as follows. The same as our experiment setting, we gradually add $\mathbf{C}^{(384)}_E$, $\mathbf{L}^{(32)}_S$, $\mathbf{C}^{(336)}_R$ agents to the scene starting with one  $\mathbf{L}_p^{(64)}$ agent.


| AP50 | - | - | - | - | - | - | - |
|------|--|--|--|--|--|--|--|
| Foreground Estimator | Foreground Estimator  Supervision | Multiscale Structure | Encoder Layer | $\mathbf{L}_p^{(64)}$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ +$\mathbf{C}^{(336)}_R$ |
| x | x | x | ResNeXt | 0.744 | 0.794 | 0.856 | 0.857 |
| x | x | √ | ResNeXt | 0.754 | 0.802 | 0.857 | 0.855 |
| √ | √ | x | ResNeXt | 0.753 | 0.82 | 0.878 | 0.879 |
| √ | x | √ | ResNeXt | 0.745 | 0.799 | 0.87 | 0.864 |
| √ | √ | √ | vanilla CNN | 0.757 | 0.823 | 0.879 | 0.88 |
| √ | √ | √ | ResNeXt | **0.764** | **0.826** | **0.892** | **0.894** |
| **AP70** | - | - | - | - | - | - | - |
| Foreground Estimator | Foreground Estimator  Supervision | Multiscale Structure | Encoder Layer | $\mathbf{L}_p^{(64)}$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ +$\mathbf{C}^{(336)}_R$ |
| x | x | x | ResNeXt | 0.618 | 0.674 | 0.763 | 0.761 |
| x | x | √ | ResNeXt | 0.642 | 0.692 | 0.774 | 0.772 |
| √ | √ | x | ResNeXt | 0.633 | 0.699 | 0.785 | 0.782 |
| √ | x | √ | ResNeXt | 0.628 | 0.689 | 0.786 | 0.781 |
| √ | √ | √ | vanilla CNN | 0.65 | 0.723 | 0.806 | 0.807 |
| √ | √ | √ | ResNeXt | **0.662** | **0.726** | **0.812** | **0.813** |



It shows that all the component are beneficial to the performance gain.

Reviewer comment (Reproducibility):
What’s the structure of the encoder in the proposed framework?

Author response:
1. LiDAR modality

- $\mathbf{L}^{(x)}_\mathit{P}$ refers to PointPillar[1].

- $\mathbf{L}^{(x)}_\mathit{S}$ refers to SECOND[2].

2. Camera modality

- $\mathbf{C}^{(x)}_\mathit{E}$ refers to Lift-Splat[3] with EfficientNet as image encoder.

- $\mathbf{C}^{(x)}_\mathit{R}$ refers to Lift-Splat[3] with ResNet50 as image encoder.

We introduced them in the response to **W2**.

Reviewer comment (Reproducibility):
Will the number of input channels for the Pyramid Fusion module change when a new agent is added?

Author response:
The input feature channel for the Pyramid Fusion module will not change. While a new agent's model may encode features with a different channel number, we can append some additional network after the encoder to adjust the channel of features. The added network layers will be trained with the encoder together to achieve optimal collaboration performance.

Reviewer comment (Evaluation):
Why can the performance of this method be improved so much on AP70?

Author response:
Thanks for your insightful comment and this is a key question. We clarify that the performance gain comes from both the training strategy and fusion method are influencing factors, while **the training strategy based on backward alignment is more critical**. 

This is because, especially when there are many agent types, such as 4 four heterogeneous agent types shown in the paper, it is difficult to balance their training in collective training. In the OPV2V-H dataset, the average number of agents in the scene is only close to 3. The collective training method can easily fall into insufficient training for multiple agent types (for example, the 4 types in our paper). Moreover, the pre-training techniques employed in the HM-ViT paper are also insufficient in addressing the problem of imbalance, significantly when the number of agent types surpasses the number of agents present in the scene.

Our novel backward alignment does not require collaboration, thereby eliminating the issues of training imbalance and insufficiency of each agent type in collective training. It enables new agents have easier convergence and easier training with alignment to a unified feature space for collaboration. This is very important for precise detection, which is reflected in the AP70 metric.


| Infer $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ + $\mathbf{C}^{(336)}_R$ on OPV2V-H | AP30 | AP50 | AP70 |
|----------------------------------------------------------------------------------------------------------------|------|------|------|
| Collective Training + HM-ViT | 0.891 | 0.876 | 0.755 |
| Collective Training + Pyramid Fusion | 0.899 | 0.881 | 0.769 |
| Back. Align. Training + Pyramid Fusion (HEAL) | **0.908** | **0.894** | **0.813** |


From the table above, we see that "Collective Training + Pyramid Fusion" exhibits a slightly higher AP performance with some baseline methods. Thus, it demonstrates significant performance improvements in AP70 come from backward alignment.

References:

[1] Fast Encoders for Object Detection from Point Clouds. CVPR 2019.

[2] SECOND: Sparsely Embedded Convolutional Detection. Sensors 2018.

[3] Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D. ECCV 2020.

Reviewer comment (Experiments):
When comparing with other SOTA algorithms, is the same training strategy used for a fair comparison? Is the main performance boost from the training strategy or the multi-modal fusion design?

Author response:
Thank you very much for dedicating your time and expertise to review our submission. The depth of your analysis and the constructive criticism you have provided are immensely beneficial. I sincerely hope my reply can answer your questions.

> **W1:** (a) When comparing with other SOTA algorithms, is the same training strategy used for a fair comparison? (b) Is the main performance boost from the training strategy or the multi-modal fusion design?


(a) For the first question, given that HEAL represents a framework-level innovation with the novel training strategy, it is justifiable to conduct the comparsion. Additionally, for a fair assessment of the fusion module, we performed an extra experiment. In this experiment, we replaced our backward alignment based training with the collective training approach used in the baseline. The results are as follows:

| Infer $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ + $\mathbf{C}^{(336)}_R$ on OPV2V-H | AP30 | AP50 | AP70 |
|----------------------------------------------------------------------------------------------------------------|------|------|------|
| Collective Training + HM-ViT | 0.891 | 0.876 | 0.755 |
| Collective Training + Pyramid Fusion | 0.899 | 0.881 | 0.769 |
| Back. Align. Trainig + Pyramid Fusion **(HEAL)** | **0.908** | **0.894** | **0.813** |

We see that under the same training strategy, our fusion module slightly surpasses existing SOTA method HM-ViT.

Furthermore, we show the benchmark of pure LiDAR-based collaboration and pure Camera-based collaboration in Appendix. In the table (from Appendix) below , methods are also trained with the same strategy. Some factors, such as whether all detections are sorted by confidence before AP calculating and detection range setting, may lead to AP differences with other papers, but we got the same SOTA method ranking as in the HM-ViT paper. We believe it is a very fair comparison:

| Dataset | OPV2V | - | - | - | DAIR-V2X | - | - | - |
|---------|-------|--|--|--|----------|--|--|--|
| **Method** | LiDAR-based | LiDAR-based | Camera-based | Camera-based | LiDAR-based | LiDAR-based | Camera-based | Camera-based |
| - | AP50 | AP70 | AP50 | AP70 | AP30 | AP50 | AP30 | AP50 |
| No Fusion | 0.782 | 0.634 | 0.405 | 0.216 | 0.421 | 0.405 | 0.014 | 0.004 |
| F-Cooper | 0.821 | 0.632 | 0.469 | 0.219 | 0.723 | 0.620 | 0.115 | 0.026 |
| DiscoNet | 0.882 | 0.737 | 0.517 | 0.234 | 0.746 | 0.685 | 0.083 | 0.017 |
| AttFusion | 0.878 | 0.751 | 0.529 | 0.252 | 0.738 | 0.673 | 0.094 | 0.021 |
| V2XViT | 0.917 | 0.790 | 0.603 | 0.289 | 0.785 | 0.521 | 0.198 | 0.057 |
| CoBEVT | 0.935 | 0.821 | 0.571 | 0.261 | 0.787 | 0.692 | 0.182 | 0.042 |
| HMViT | 0.950 | 0.873 | 0.643 | 0.370 | 0.818 | 0.761 | 0.163 | 0.044 |
| Pyramid Fusion | **0.963** | **0.926** | **0.689** | **0.468** | **0.832** | **0.790** | **0.282** | **0.128** |



(b) For the second question, both the training strategy and the fusion method are key components, while the training strategy based on backward alignment is more critical. This is because, especially when there are many agent types, such as 4 four heterogeneous agent types shown in the paper, it is actually difficult to balance their training in collective training, **even with pre-train.** In the OPV2V-H dataset, the average number of agents in the scene is only close to 3, while we have 4 agent types. The collective training method can easily fall into insufficient training for each heterogeneous agent's collaboration. Moreover, the pre-training techniques employed in the HM-ViT paper are also inadequate in addressing the problem of imbalance, especially when the number of agent types surpasses the number of agents present in the training data.

Conversely, backward alignment does not involve collaboration, thereby circumventing the issues of training imbalance and insufficiency in collaborative training. Individual training enables new agents to converge more rapidly, aligning them to a unified feature space for collaboration. When we use collective training + Pyramid Fusion, the same training strategy with baselines, to train newly added heterogeneous agent types, the performance is also suboptimal to HEAL with backward alignment. Here are the experimental results.



| Infer $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ + $\mathbf{C}^{(336)}_R$ on OPV2V-H | AP30 | AP50 | AP70 |
|----------------------------------------------------------------------------------------------------------------|------|------|------|
| Collective Trainig + Pyramid Fusion | 0.899 | 0.881 | 0.769 |
| Back. Align. Trainig + Pyramid Fusion **(HEAL)** | **0.908** | **0.894** | **0.813** |



This demonstrates significant performance improvements from backward alignment.

Reviewer comment (Novelty):
The fusion model design (except the residual part) shows similarity with existing methods like who2com/where2comm/disconet. Please justify and highlight the differences and novelty. Please also benchmark the performance under the same training strategy with only different fusion networks so as to demonstrate the effectiveness of the proposed fusion module.

Author response:
> **W2:** The fusion model design (except the residual part) shows similarity with existing methods like who2com/where2comm/disconet. Please justify and highlight the differences and novelty. Please also benchmark the performance under the same training strategy with only different fusion networks to demonstrate the effectiveness of the proposed fusion module.

The proprietary design of Pyramid Fusion renders it uniquely compatible with the backward alignment training strategy. Not every fusion algorithm can work with our backward alignment. To simplify the explanation, we conceptualize collaborative perception as a graph structure, where the nodes represent the features of different agents, and the edges signify the interactions between these agents. It should be clarified that not all fusion methods can work with backward alignment, especially the algorithms that need pairwise edge interaction.

The following are the differences and novelty designs by Pyramid Fusion.

1. **Node level weight prediction** 
During the training of new agents, Pyramid Fusion does not require interaction with the features of other agents to predict an edge weight collectively. Instead, its weight is node-level, only derived from its foreground estimation. This modular design and independence are well-suited and more appropriate when no collaboration is involved in new agent training. 
As for DiscoNet/who2com/Where2comm, the weighting mechanisms are derived from pairwise interactions. For instance, DiscoNet and who2com concatenate ego and neighbor features to predict edge weights, whereas where2comm employs an attention mechanism with learnable parameters to calculate pairwise weights. A critical issue arises when training new agents alone without collaboration, as these methods are limited to one self-loop at the single-agent node on the collaboration graph.  For a self-loop input, a softmax function invariably outputs a weight of 1. This results in their edge weight prediction being ineffective during backward alignment, and new agent types do not adapt to the parameters of edge weight prediction. Consequently, during actual inference, the estimated edge weights between heterogeneous agents are erroneous, preventing the agents from collaborating effectively.


2. **Explicit foreground estimators**
In Pyramid Fusion, we have designed explicit foreground estimators and supervision for the predicted foreground maps. The foreground estimators predict the foreground probability for each BEV pixel for effective feature fusion. This supervision serves two crucial purposes: (i) It promotes the learning of foreground estimators, leading to more accurate weight predictions. (ii) More significantly, they offer feature alignment supervision signals during backward alignment. Since the parameters of the foreground estimator are also fixed for new heterogenous agent training, they go through the same individual supervision on foreground estimation, which encourages the encoder to produce features that align with those of the base agent. 
For DiscoNet/who2com/Where2comm, the absence of an estimator and supervision on weight prediction leads to suboptimal fusion and poor feature alignment.


3. **Multiscale network structure**
Our approach with multiscale feature fusion and supervision further facilitates the fusion. Particularly the multiscale foreground estimator, which, through supervision at varying granularities, ensures the alignment of features between new agents and the base agent during backward alignment. This aspect of our methodology ensures that the new agents are effectively aligned with the base agents.
For DiscoNet/who2com/Where2comm, they only employ a single-scale network structure, which is suboptimal to our multiscale design in feature learning and fusion.

We benchmark their performance under the same training strategy. As we mentioned previously, backward alignment is unsuitable for feature fusion algorithms with edge interaction. Thus, with the same training strategy, HEAL significantly outperforms other methods. Experiments on OPV2V-H dataset:

| AP50 | - | - | - | - |
|------|--|--|--|--|
| Fusion Backbone | $\mathbf{L}_p^{(64)}$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ +$\mathbf{C}^{(336)}_R$ |
| who2com | 0.742 | 0.09 | 0.016 | 0.017 |
| Where2comm | 0.743 | 0.773 | 0.774 | 0.771 |
| Ours | **0.764** | **0.826** | **0.892** | **0.894** |
| **AP70** | - | - | - | - |
| Fusion Backbone | $\mathbf{L}_p^{(64)}$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ +$\mathbf{C}^{(336)}_R$ |
| who2com | 0.621 | 0.05 | 0.006 | 0.006 |
| Where2comm | 0.639 | 0.674 | 0.679 | 0.678 |
| Ours | **0.662** | **0.726** | **0.812** | **0.813** |

Reviewer comment (Evaluation):
How to ensure each modality can be aligned to the common feature space? For example, it may be hard to extract the 3D features from camera/radar data which are expected to be as good as (aligned) as the LiDAR features. From 6, we can see that camera bev features are more vague than LiDAR features, which could not show the aligned effect. Please justify this design choice.

Author response:
>**W3:** How to ensure each modality can be aligned to the common feature space? For example, it may be hard to extract the 3D features from camera/radar data which are expected to be as good as (aligned) as the LiDAR features. From 6, we can see that camera bev features are more vague than LiDAR features, which could not show the aligned effect. Please justify this design choice.


We ensure that the features from the LiDAR agents and camera agents are aligned. We substantiate it from qualitative and quantitative aspects.

**For qualitative analysis:**

It should be clarified that, feature alignment means similar numerical intensity when describing the scene object, and it is reflected as similar brightness response in heat map visualization (as our figure 6 does).  The blurring of camera BEV features does not mean the misalignment in the feature space. Actually, it comes from the fact that the camera data has no scale information and cannot accurately restore the 3D scene like LiDAR. This is reflected in many camera-based detection paper like [1][2]. 

In the Appendix of our revision, we have supplement more visualizations, including agent features at different scales. These examples consistently show similar feature responses in the foreground/background regions. In our visualization, both camera and LiDAR features exhibit similar and distinct delineations between foreground and background areas, demonstrating their alignment in the unified feature space.

**For quantitative analysis:**

We experiment to measure their feature similarity. We adopt the LiDAR encoder and camera encoder on the same ego agent (to ensure accurate spatial alignment) and calculate the cosine similarity between the LiDAR feature and camera feature. We evaluate on the whole testset, and the results on OPV2V-H dataset are as follows:


| LiDAR and Camera Feature before fusion | Average Cosine Simiarlity |
|----------------------------------------|---------------------------|
| $\mathbf{F}_{j\rightarrow i}^{(1)}$ | 0.617 |
| $\mathbf{F}_{j\rightarrow i}^{(2)}$ | 0.650 |
| $\mathbf{F}_{j\rightarrow i}^{(3)}$ | 0.774 |


It is worth noticing that many existing works[3,4] have also considered aligning the BEV features from LiDAR and cameras to the same space to facilitate multi-modality perception. X-Align[3] uses a Cross-Modal Feature Alignment loss to align LiDAR and camera BEV features for more accurate BEV segmentation. BEVDistill[4] uses a dense feature distillation to promote the similarity between the camera feature and the LiDAR feature. These papers all demonstrated that such a design is reasonable and results in considerable performance enhancements. 

[1] Categorical Depth Distribution Network for Monocular 3D Object Detection. CVPR 2021.

[2] Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D. ECCV 2020.

[3] X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentation, WACV 2023.

[4] BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detection, ICLR 2023.

Reviewer comment (Experiments):
Can the network scale to different sensor modality combinations? For example, changing/fixing the ego modality with dynamic collaborator sensor modalities. What is the sensitivity of the network with respect to this sensor modality combination ratio?

Author response:
>**W4:** Can the network scale to different sensor modality combinations? For example, changing/fixing the ego modality with dynamic collaborator sensor modalities. What is the sensitivity of the network with respect to this sensor modality combination ratio?

HEAL is able to scale to different sensor modality combinations. Our training strategy inherently accounts for extreme LiDAR/camera ratio, because the new agents are trained independently (for example camera agent is trained without LiDAR collaborator). We conduct the LiDAR-Camera ratio experiment on OPV2V-H dataset as follows. The notation $\mathbf{L}_p^{(64)}$ + 3 $\mathbf{C}^{(384)}_E$ means ego agent is $\mathbf{L}_p^{(64)}$ and the collaborators are 3 $\mathbf{C}^{(384)}_E$ agents.


| Modality Ratio Ablation (AP50) | $\mathbf{L}_p^{(64)}$ + 3 $\mathbf{L}_p^{(64)}$ | $\mathbf{L}_p^{(64)}$ + 3 $\mathbf{C}^{(384)}_E$ | $\mathbf{C}^{(384)}_E$ + 3 $\mathbf{L}_p^{(64)}$ | $\mathbf{C}^{(384)}_E$ + 3 $\mathbf{C}^{(384)}_E$ |
|--------------------------------|-------------------------------------------------|--------------------------------------------------|--------------------------------------------------|---------------------------------------------------|
| V2XViT | 0.932 | 0.846 | 0.855 | 0.452 |
| CoBEVT | 0.919 | 0.849 | 0.727 | 0.432 |
| HM-ViT | 0.927 | 0.838 | 0.864 | 0.435 |
| HEAL | **0.954** | **0.848** | **0.87** | **0.458** |
| **Modality Ratio Ablation (AP70)** | **$\mathbf{L}_p^{(64)}$ + 3 $\mathbf{L}_p^{(64)}$** | **$\mathbf{L}_p^{(64)}$ + 3 $\mathbf{C}^{(384)}_E$** | **$\mathbf{C}^{(384)}_E$ + 3 $\mathbf{L}_p^{(64)}$** | **$\mathbf{C}^{(384)}_E$ + 3 $\mathbf{C}^{(384)}_E$** |
| V2XViT | 0.802 | 0.677 | 0.549 | 0.198 |
| CoBEVT | 0.775 | 0.715 | 0.419 | 0.218 |
| HM-ViT | 0.839 | 0.689 | 0.692 | 0.220 |
| HEAL | **0.918** | **0.748** | **0.799** | **0.267** |


We see that HEAL maintained commendable performance levels on the OPV2V-H dataset with different LiDAR-to-camera ratios. This showcases the resilience and adaptability of our approach in handling varying proportions of LiDAR and camera agents, ensuring consistent effectiveness across a range of modality ratios.

For different **sensor modality & detection model** combinations, we encourage reviewers to refer to Table 3, where we have showcased five distinct sensor setups and detection model combinations. This comprehensive analysis provides valuable insights into how our methodology adapts to and performs with various sensor configurations and detection algorithms, highlighting its versatility and effectiveness in diverse operational contexts.

Reviewer comment (Theory):
The author argued that "late fusion is suboptimal due to the communication latency". However, as shown in DiscoNet/V2X-ViT/V2VNet etc., intermediate fusion methods usually require larger bandwidth requirements than late fusion, which can even lead to potentially larger communication latency compared with late fusion.

Author response:
>**Q1:** The author argued that "late fusion is suboptimal due to the communication latency". However, as shown in DiscoNet/V2X-ViT/V2VNet etc., intermediate fusion methods usually require larger bandwidth requirements than late fusion, which can even lead to potentially larger communication latency compared with late fusion.


The communication cost of intermediate fusion can be significantly reduced with technical designs.

1. As shown in V2X-ViT/V2VNet/OPV2V **and our paper (Figure 4)**, we can compress the intermediate feature to a small data size *along channel dimension* with an autoencoder structure. In Figure 4 of our paper, we present experiments that compress the channel number of features for communication, and it performs well. 

2. As shown in Where2comm, we can also compress the feature *along spatial dimension*. We can select the most important feature (selected by our foreground estimator) for transmission. In Where2Comm's Appendix Table 3 and Figure 12, intermediate fusion with this technique has a communication volume already comparable to late fusion. 

Therefore, the communication cost problem can be solved in many ways, and it is also shown in our paper in Figure 4. Actually, there are currently many papers focusing on communication bandwidth and delay. These issues are significant but are not the focus of this work.

Reviewer comment (Evaluation):
What is the inference time of the proposed method?

Author response:
>**Q3:** What is the inference time of the proposed method?

The table below shows the inference time of all methods on one RTX A40 with batchsize 1. We see that HEAL is running at an acceptable FPS.

| Inference FPS (fusion module)  on OPV2V-H dataset | - | - | - | - | - |
|---------------------------------------------------|--|--|--|--|--|
| range $x \in$  [-102.4m, 102.4m], $y \in$ [-51.2m, 51.2m] | $N_{agent}$= 1 | $N_{agent}$= 2 | $N_{agent}$= 3 | $N_{agent}$= 4 | $N_{agent}$= 5 |
| AttFusion | 500.3 | 359.9 | 248.0 | 207.6 | 159.1 |
| DiscoNet | 477.6 | 383.3 | 292.6 | 235.8 | 196.3 |
| F-Cooper | 1025.7 | 869.4 | 757.3 | 681.0 | 595.5 |
| V2XViT | 54.1 | 30.83 | 19.78 | 15.50 | 12.07 |
| CoBEVT | 112.4 | 67.16 | 45.57 | 33.78 | 24.89 |
| HM-ViT | 6.08 | 5.56 | 4.60 | 3.95 | 2.99 |
| HEAL | 55.8 | 48.07 | 42.35 | 35.21 | 30.04 |

Reviewer comment (Experiments):
What is the influence of the modality choice in the base collaboration training?

Author response:
>**Q4:** What is the influence of the modality choice in the base collaboration training?

We really appreciate insightful thoughts and the highly meaningful question. Choosing agents with LiDAR modality as the collaboration base yields better results than using agents with camera modality. It is intuitive that LiDAR, compared to cameras, often achieves higher detection accuracy, which is beneficial for feature learning and helps construct a robust unified feature space with clear foreground-background differentiation. In our agent configuration, even though we provide depth information as supervision for cameras, their detection performance still falls short compared to LiDAR. Consequently, a HEAL model trained with a camera collaboration base exhibits a noticeable decrease in performance. In this ablation study, the agent combination is still  $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ +$\mathbf{C}^{(336)}_R$, but we change use different base agent type as the base type. The results are as follows. We see that using LiDAR as the collaboration base is significantly better than using cameras.


| **Base Agent Type Selection** | **AP30** | **AP50** | **AP70** |
|-------------------------------|----------|----------|----------|
| $\mathbf{L}_p^{(64)}$ | 0.908 | 0.894 | 0.813 |
| $\mathbf{L}^{(32)}_S$ | 0.904 | 0.892 | 0.805 |
| $\mathbf{C}^{(384)}_E$ | 0.804 | 0.794 | 0.714 |


**In real world, employing LiDAR modality to train a collaboration base is a totally viable and worthwhile solution.** This is also supported by existing datasets such as DAIR-V2X and V2V4Real, which provide the necessary data for such training. We have detailed our approach for practical implementation in the Appendix of our paper, emphasizing its feasibility and relevance in real-world applications.

Reviewer comment (Experiments):
There is no real-world experiments. There are some dataset like nuScene/nuPlan, Waymo and etc including data of different sensors. It would be nice to show some real examples.

Author response:
The DAIR-V2X[4] dataset used in our paper is a **real-world** multi-agent collaborative perception dataset. It is collaboratively collected by a vehicle and a roadside unit in the real world, equipped with multiple sensors, including cameras and LiDAR. In our Section 5.1, we provide a brief introduction to the dataset, and Section 5.3 is dedicated to extensive experimentation on it. 

nuScene, nuPlan and Waymo are all awesome autonoumous driving datasets, but they has only one data collection vehicle. They do not have temporally and spatially aligned multi-vehicle data, making them difficult to validate on our task. We believe that DAIR-V2X, as a real dataset widely used in collaborative perception (e.g. [6][7][8][9][10]), can verify the reliability of our algorithm.

In our paper's Table 3, we conducted experiments using the real-world DAIR-V2X dataset under our open heterogeneous collaborative perception setting. We tested three different combinations of agent types, and the results are presented as follows:


| Agent Types | $\mathbf{L}_p^{(40)}$ +$\mathbf{C}^{(288)}_E$ | - | $\mathbf{L}_p^{(40)}$+$\mathbf{L}^{(40)}_S$ | - | $\mathbf{L}_p^{(40)}$ +$\mathbf{C}^{(288)}_R$ | - |
|-------------|-----------------------------------------------|--|---------------------------------------------|--|-----------------------------------------------|--|
| - | AP50 $\uparrow$ | Training Param (M) $\downarrow$ | AP50 $\uparrow$ | Training Param (M) $\downarrow$ | AP50 $\uparrow$ | Training Param (M) $\downarrow$ |
| Late Fusion | 0.344 | 25.7 | 0.376 | 11.8 | 0.341 | 12.6 |
| F-Cooper | 0.626 | 38.8 | 0.545 | 24.9 | 0.611 | 25.8 |
| DiscoNet | 0.576 | 38.9 | 0.634 | 25.1 | 0.621 | 25.9 |
| AttFusion | 0.649 | 38.8 | 0.661 | 24.7 | 0.623 | 25.8 |
| V2XViT | 0.654 | 49.6 | 0.709 | 35.7 | 0.669 | 36.6 |
| CoBEVT | 0.638 | 43.8 | 0.692 | 29.8 | 0.66 | 30.7 |
| HM-ViT | 0.638 | 67.8 | 0.538 | 53.9 | 0.677 | 54.8 |
| **HEAL** | **0.658** | **25.7** | **0.770** | **11.8** | **0.681** | **12.6** |



The experimental outcomes align with those demonstrated on the OPV2V-H: HEAL outperforms other methods while maintaining the lowest training cost, validating the high performance and low training costs of our methods.

Reviewer comment (Novelty):
It would be interesting to include a bit discussion on related works for cooperation for driving tasks, e.g. [1][2][3]

Author response:
Thank you very much for recommending these driving related articles. In the revision, We have cited [1,2,3] and add discussions in our related works. We believe this will enhance the depth of the paper and inspire the readers.


Here, we provide a detailed discussion about how collaborative perception relates to cooperation in driving tasks in two main ways:
1) [1] and [3] concentrate on enhancing model training by incorporating multiple sources beyond just the ego agent. Collaborative perception benefits these agents by providing additional, complementary information during inference. This collaborative approach in driving tasks offers significant insights for improving the training process, thereby enhancing the generalizability of collaborative perception. Furthermore, collaborative perception can improve cooperative driving tasks by integrating collaboration modules into single-agent models, enhancing performance during inference.
2) [2] explores how sharing information benefits the ego agent's end-to-end driving task, which closely aligns with our collaborative perception approach. Investigating collaborative perception can further promote cooperation in driving tasks, as a more holistic perception contributes to overall driving enhancements. For example, our research, aimed at expanding the scope of collaboration, can be applied to collaborative driving systems to improve their overall efficiency and effectiveness.

Reviewer comment (Reproducibility):
I don't find a code release. Would be nice to release the code for supplementary or public github repo.

Author response:
We plan to release the code if the paper can be accepted. In the meantime, we are happy to discuss any technical detail to address your concerns and support reproducibility. We appreciate your understanding and look forward to sharing our complete code soon.

Reviewer comment (Theory):
The paper mentioned new agent privacy issue. I assume the late participate will require the new agent to access the fused feature to do the update according to equations in Section 4.3. Will the fused feature release some privacy of the old agents to the new agent?

Author response:
Thanks for your thoughtful question! There is no clear privacy leakage from the old agents to the new agents in inference stage. The privacy concerns discussed in our paper primarily relate to the model-detail privacy, which arises from collective training. During collective training, each agent type must provide its network structure for gradient backpropagation and parameter updates. 

In response to this question, we provide insights from two perspectives: sensor-data privacy and model-detail privacy.

- **Sensor-data privacy:** During the inference process, the transmitted features are derived from observations of the current scene. What they describe is a public driving scenario, so there are no privacy issues at the sensor data level. 

- **Model-detail privacy:** In the inference stage, it is challenging to deduce the model details of different agent types based solely on features, especially when both the network structure and parameters are unknown. In HEAL, the collaboration network (excluding the encoder) trained by base agents will be distributed to other agents, which may cause some concerns about leakage of the base agent model details. But a highly feasible solution is to use public multi-agent collaborative perception datasets for the collaboration base training, such as DAIR-V2X[4] and V2V4real[5]. This will completely eliminate model detail privacy issues in HEAL. We have delineated such a real-world solution in the Appendix, which comprehensively prevent the exposure of model privacy.

Reviewer comment (Experiments):
For the experiments setting, are there any case agent exits the cooperation during training? How the framework will do to deal with this situation.

Author response:
The number of input agents will not affect the collaboration process. This is equivalent to the situation you mentioned when an agent quits cooperation. 

The number of agents in the OPV2V-H data set varies from 2-7 and is not a fixed value itself. Therefore, our network is already trained with varying agent numbers. We use the foreground estimator for each agent to estimate its score map, normalize them with softmax, and then aggregate them into a weighted sum. For any number of agents, the aggregated features can always be considered to be the most important foreground features from all participants.

Reviewer comment (Evaluation):
The intermediate fusion method employed in this paper doesn't seem to address the issue of new agents joining as effectively as late fusion does.

Author response:
> **W1:** The intermediate fusion method employed in this paper doesn't seem to address the issue of new agents joining as effectively as late fusion does.

HEAL achieves significantly better detection performance than late fusion. As shown in Table 3 from our paper: 

| AP70 | OPV2V-H | OPV2V-H | OPV2V-H |
|---|--|---|---|
| **Agent Types** | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(64)}_S$ + $\mathbf{C}^{(336)}_R$ | $\mathbf{L}_p^{(32)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(32)}_S$ + $\mathbf{C}^{(336)}_R$ | $\mathbf{L}_p^{(16)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(16)}_S$ + $\mathbf{C}^{(336)}_R$ |
| Late Fusion | 0.685 | 0.639 | 0.457 |
| HEAL | 0.813 | 0.738 | 0.578 |

we see that HEAL significantly outperforms the late fusion with our feature-level fusion.

Furthermore, HEAL is as efficient as late fusion regarding new agent integration in training.  The new agent type's training process in HEAL is also performed on single agents without collaboration. This enables us to maintain comparable training parameters, FLOPs, and training time with late fusion.

Reviewer comment (Experiments):
This paper has only conducted experiments on two datasets, one of which is generated for the first time in this paper. It is hoped that the author can introduce more experiments to substantiate.

Author response:
> **W2:** It is hoped that the author can introduce more experiments to substantiate.

We add experiments on V2XSet[1] and OPV2V(culver city)[2]. We prepare the similar agent type setting in OPV2V-H and evaluate the collaborative scene-level AP when gradually adding new heterogeneous agents into the scene. The experimental results are as follows:

*  V2XSet dataset. We see that HEAL consistently achieves the best performance each time new heterogeneous agents are added. It can significantly outperforms late fusion by 18.3% in AP70 with our feature-level fusion.

| **AP50** | - | - | - | - |
|--|--|--|--|--|
| **Methods** | $\mathbf{L}_p^{(64)}$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(64)}_S$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(64)}_S$ +$\mathbf{C}^{(336)}_R$ |
| F-Cooper | 0.632 | 0.637 | 0.561 | 0.573 |
| DiscoNet | 0.367 | 0.346 | 0.354 | 0.351 |
| AttFusion | 0.637 | 0.641 | 0.641 | 0.643 |
| CoBEVT | 0.613 | 0.641 | 0.653 | 0.646 |
| HMViT | 0.612 | 0.610 | 0.694 | 0.706 |
| V2XViT | 0.609 | 0.678 | 0.750 | 0.743 |
| Late Fusion | 0.632 | 0.666 | 0.733 | 0.725 |
| HEAL | **0.658** | **0.719** | **0.763** | **0.763** |
| **AP70** | - | - | - | - |
| **Methods** | $\mathbf{L}_p^{(64)}$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(64)}_S$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{L}^{(64)}_S$ +$\mathbf{C}^{(336)}_R$ |
| F-Cooper | 0.466 | 0.453 | 0.353 | 0.353 |
| DiscoNet | 0.307 | 0.300 | 0.293 | 0.284 |
| AttFusion | 0.474 | 0.478 | 0.476 | 0.477 |
| CoBEVT | 0.412 | 0.420 | 0.398 | 0.372 |
| HMViT | 0.415 | 0.430 | 0.494 | 0.493 |
| V2XViT | 0.450 | 0.499 | 0.554 | 0.549 |
| Late Fusion | 0.473 | 0.492 | 0.569 | 0.558 |
| HEAL | **0.531** | **0.587** | **0.660** | **0.660** |

- OPV2V (culver city)

Since the OPV2V (culver city) has at most 3 agents in the scene, so we prepare 3 agent types in total. 

From the table below, we see that when new heterogeneous agents join the collaboration, HEAL notably outperforms other methods thanks to our effective training strategy and fusion module.

| AP50 | - | - | - |
|---|--|--|--|
| **Methods** | $\mathbf{L}_p^{(64)}$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{C}^{(336)}_R$ |
| F-Cooper | 0.548 | 0.539 | 0.603 |
| DiscoNet | **0.624** | 0.625 | 0.622 |
| AttFusion | 0.609 | 0.609 | 0.618 |
| CoBEVT | 0.606 | 0.622 | 0.647 |
| HMViT | 0.59 | 0.602 | 0.614 |
| V2XViT | 0.602 | 0.631 | 0.649 |
| Late Fusion | 0.569 | 0.589 | 0.616 |
| HEAL | 0.588 | **0.636** | **0.656** |
| **AP70** | - | - | - |
| **Methods** | $\mathbf{L}_p^{(64)}$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ | $\mathbf{L}_p^{(64)}$ +$\mathbf{C}^{(384)}_E$ +$\mathbf{C}^{(336)}_R$ |
| F-Cooper | 0.46 | 0.432 | 0.438 |
| DiscoNet | **0.503** | 0.505 | 0.505 |
| AttFusion | 0.492 | 0.494 | 0.483 |
| CoBEVT | 0.461 | 0.467 | 0.488 |
| HMViT | 0.44 | 0.45 | 0.46 |
| V2XViT | 0.459 | 0.483 | 0.487 |
| Late Fusion | 0.428 | 0.441 | 0.454 |
| HEAL | 0.487 | **0.522** | **0.533** |

In conclusion, HEAL presents two main advanatages over existing methods. (i) Comparing with collective training methods, it eliminates intractable multi-agent data collection with new and existing agent types, and significantly reduces the training costs. (ii) Comparing with late fusion, it shows noticeable performance gain with similar training costs.

Reviewer comment (Theory):
In this paper, the authors claim that agents newly joining the system may struggle to align well in the feature space due to data distribution differences. However, is training a unified feature space an effective solution? Given that data discrepancies arising from different sensors inherently result in domain differences, this discrepancy poses a significant challenge in the domain adaptation field. Is the method proposed in this paper suitable for addressing this issue?

Author response:
> **Q1:** In this paper, the authors claim that agents newly joining the system may struggle to align well in the feature space due to data distribution differences. However, is training a unified feature space an effective solution? Given that data discrepancies arising from different sensors inherently result in domain differences, this discrepancy poses a significant challenge in the domain adaptation field. Is the method proposed in this paper suitable for addressing this issue?

Training a unified feature space is an very effective solution when we want the high performance leveraged by feature fusion. Notably, aligning feature from different modalities to a shared space is an emerging practice. Existing single-agent multi-modality detection works[3,4] have achieved substantial success in unifying the BEV features from LiDAR and cameras to facilitate multi-modality perception. X-Align[3] proposes a Cross-Modal Feature Alignment loss to align LiDAR and camera BEV features for more accurate BEV segmentation. BEVDistill[4] uses a dense feature distillation to promote the similarity between the camera feature and the LiDAR feature. These papers all demonstrated that such unified feature space can brought considerable performance enhancements. 

In our open heterogeneous setting, this approach particularly shows its superiority. As delineated in Section 3 of our paper, we **cannot predetermine the sensor modality, sensor configuration, and detection model of the new agents**. A unified feature space constitutes a consistent objective for the heterogeneous alliance, simplifying the integration of new agents. They only need to adapt their own features to this unified space. Compare with collective training, it does not require cumbersome data collection and heavy training. Compared with late fusion, it shows higher performance and can be more resistant to noise.

Reviewer comment (Experiments):
The author raises a novel question, thus it would be prudent to utilize more datasets to verify the efficacy of the proposed method. This is because some schemes [1] solely employing distillation can achieve significant improvements in accuracy. In the open-source datasets they used, there are also newly joining agents, similar to the DAIR-V2X dataset used in this paper. It is hoped that the author can supplement with more extensive experiments to substantiate the reliability of the raised question and the effectiveness of the proposed method.

Author response:
> **Q2:** The author raises a novel question, thus it would be prudent to utilize more datasets to verify the efficacy of the proposed method. This is because some schemes[7] solely employing distillation can achieve significant improvements in accuracy. In the open-source datasets they used, there are also newly joining agents, similar to the DAIR-V2X dataset used in this paper. It is hoped that the author can supplement with more extensive experiments to substantiate the reliability of the raised question and the effectiveness of the proposed method.

Thanks for your suggestion; we have conducted additional experiments on V2XSet and OPV2V(culver city). See the response to **W2**. Experiment results show that HEAL holds the best performance compared with other intermediate fusion methods and late fusion. Note that in real-world deployment, we have also reduced the cost of data collection because we do not need to collect spatio-temporal synchronous cooperative data for new agents and existing agents in the same scene to perform collective training. It also significantly surpasses late fusion in terms of AP50 & AP70. 

Knowledge distillation is similar to other collective training methods, which bring heavy costs associated with data collection and training in real-world scenarios. When a new heterogeneous agent is to be integrated into the scene (acting as a student), it necessitates the presence of an existing agent type (acting as a teacher) to collect temporally synchronized sensor data together.  The open heterogeneous setting has never been considered in terms of accuracy alone. A significant contribution of HEAL is its capacity to prevent the complexities involved in data acquisition and collective training processes. Meanwhile, it can remarkably outperforms late fusion in detection performance.