Original: Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps
Revised: Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps

Original: Abstract
Revised: Abstract

Original: Multi-agent collaborative perception could significantly upgrade the perception performance by enabling agents to share complementary information with each other through communication.
Revised: Multi-agent collaborative perception could significantly upgrade the perception performance by enabling agents to share complementary information with each other through communication.

Original: It inevitably results in a fundamental trade-off between perception performance and communication bandwidth.
Revised: It inevitably results in a fundamental trade-off between perception performance and communication bandwidth.

Original: To tackle this bottleneck issue, we propose a spatial confidence map, which reflects the spatial heterogeneity of perceptual information.
Revised: To tackle this bottleneck issue, we propose a spatial confidence map, which reflects the spatial heterogeneity of perceptual information.

Original: It empowers agents to only share spatially sparse, yet perceptually critical information, contributing to where to communicate.
Revised: It empowers agents to only share spatially sparse, yet perceptually critical information, contributing to where to communicate.

Original: Based on this novel spatial confidence map, we propose Where2comm , a communication-efficient collaborative perception framework.
Revised: Based on this novel spatial confidence map, we propose Where2comm , a communication-efficient collaborative perception framework.

Original: Where2comm has two distinct advantages: i) it considers pragmatic compression and uses less communication to achieve higher perception performance by focusing on perceptually critical areas; and ii) it can handle varying communication bandwidth by dynamically adjusting spatial areas involved in communication.
Revised: Where2comm has two distinct advantages: i) it considers pragmatic compression and uses less communication to achieve higher perception performance by focusing on perceptually critical areas; and ii) it can handle varying communication bandwidth by dynamically adjusting spatial areas involved in communication.

Original: To evaluate Where2comm , we consider 3D object detection in both real-world and simulation scenarios with two modalities (camera/LiDAR) and two agent types (cars/drones) on four datasets: OPV2V, V2X-Sim, DAIR-V2X, and our original CoPerception-UAVs.
Revised: To evaluate Where2comm , we consider 3D object detection in both real-world and simulation scenarios with two modalities (camera/LiDAR) and two agent types (cars/drones) on four datasets: OPV2V, V2X-Sim, DAIR-V2X, and our original CoPerception-UAVs.

Original: Where2comm consistently outperforms previous methods; for example, it achieves more than 100 , 000 × lower communication volume and still outperforms DiscoNet and V2X-ViT on
Revised: Where2comm consistently outperforms previous methods; for example, it achieves more than 100 , 000 × lower communication volume and still outperforms DiscoNet and V2X-ViT on

Original: OPV2V.
Revised: OPV2V.

Original: Our code is available at https://github.com/MediaBrain-SJTU/where2comm .
Revised: Our code is available at https://github.com/MediaBrain-SJTU/where2comm .

Original:  Introduction
Revised:  Introduction

Original: Collaborative perception enables multiple agents to share complementary perceptual information with each other, promoting more holistic perception.
Revised: Collaborative perception enables multiple agents to share complementary perceptual information with each other, promoting more holistic perception.

Original: It provides a new direction to fundamentally overcome a number of inevitable limitations of single-agent perception, such as occlusion and longrange issues.
Revised: It provides a new direction to fundamentally overcome a number of inevitable limitations of single-agent perception, such as occlusion and longrange issues.

Original: Related methods and systems are desperately needed in a broad range of real-world applications, such as vehicle-to-everything-communication-aided autonomous driving [1–3], multirobot warehouse automation system [4, 5] and multi-UAVs (unmanned aerial vehicles) for search and rescue [6–8].
Revised: Related methods and systems are desperately needed in a broad range of real-world applications, such as vehicle-to-everything-communication-aided autonomous driving [1–3], multirobot warehouse automation system [4, 5] and multi-UAVs (unmanned aerial vehicles) for search and rescue [6–8].

Original: To realize collaborative perception, recent works have contributed high-quality datasets [9–11] and effective collaboration methods [12, 13, 2, 14–19].
Revised: To realize collaborative perception, recent works have contributed high-quality datasets [9–11] and effective collaboration methods [12, 13, 2, 14–19].

Original: In this emerging field, the current biggest challenge is how to optimize the trade-off between perception performance and communication bandwidth.
Revised: In this emerging field, the current biggest challenge is how to optimize the trade-off between perception performance and communication bandwidth.

Original: Communication systems in real-world scenarios are always constrained that they can hardly afford huge communication consumption in real-time, such as passing complete raw observations or a large volume of features.
Revised: Communication systems in real-world scenarios are always constrained that they can hardly afford huge communication consumption in real-time, such as passing complete raw observations or a large volume of features.

Original: Therefore,
Revised: Therefore,

Original: However, all previous works make a plausible assumption: once two agents collaborate, they are obligated to share perceptual information of all spatial areas equally .
Revised: However, all previous works make a plausible assumption: once two agents collaborate, they are obligated to share perceptual information of all spatial areas equally .

Original: This unnecessary assumption can hugely waste the bandwidth as a large proportion of spatial areas may contain irrelevant information for perception task.
Revised: This unnecessary assumption can hugely waste the bandwidth as a large proportion of spatial areas may contain irrelevant information for perception task.

Original: Figure 1 illustrates such a spatial heterogeneity of perceptual information.
Revised: Figure 1 illustrates such a spatial heterogeneity of perceptual information.

Original: To fill this gap, we consider a novel spatial-confidence-aware communication strategy.
Revised: To fill this gap, we consider a novel spatial-confidence-aware communication strategy.

Original: The core idea is to enable a spatial confidence map for each agent, where each element reflects the perceptually critical level of a corresponding spatial area.
Revised: The core idea is to enable a spatial confidence map for each agent, where each element reflects the perceptually critical level of a corresponding spatial area.

Original: Based on this map, agents decide which spatial area (where) to communicate about.
Revised: Based on this map, agents decide which spatial area (where) to communicate about.

Original: That is, each agent offers spatially sparse, yet critical features to support other agents, and meanwhile requests complementary information from others through multi-round communication to perform efficient and mutually beneficial collaboration.
Revised: That is, each agent offers spatially sparse, yet critical features to support other agents, and meanwhile requests complementary information from others through multi-round communication to perform efficient and mutually beneficial collaboration.

Original: Following this strategy, we propose Where2comm , a novel communication-efficient multi-agent collaborative perception framework with the guidance of spatial confidence maps; see Fig.
Revised: Following this strategy, we propose Where2comm , a novel communication-efficient multi-agent collaborative perception framework with the guidance of spatial confidence maps; see Fig.

Original: Where2comm includes three key modules: i) a spatial confidence generator, which produces a spatial confidence map to indicate perceptually critical areas; ii) a spatial confidence-aware communication module, which leverages the spatial confidence map to decide where to communicate via novel message packing, and who to communicate via novel communication graph construction; and iii) a spatial confidence-aware message fusion module, which uses novel confidence-aware multi-head attention to fuse all messages received from other agents, upgrading the feature map for each agent.
Revised: Where2comm includes three key modules: i) a spatial confidence generator, which produces a spatial confidence map to indicate perceptually critical areas; ii) a spatial confidence-aware communication module, which leverages the spatial confidence map to decide where to communicate via novel message packing, and who to communicate via novel communication graph construction; and iii) a spatial confidence-aware message fusion module, which uses novel confidence-aware multi-head attention to fuse all messages received from other agents, upgrading the feature map for each agent.

Original: Where2comm has two distinct advantages.
Revised: Where2comm has two distinct advantages.

Original: First, it promotes pragmatic compression at the feature level and uses less communication to achieve higher perception performance by focusing on perceptually critical areas.
Revised: First, it promotes pragmatic compression at the feature level and uses less communication to achieve higher perception performance by focusing on perceptually critical areas.

Original: Second, it adapts to various communication bandwidths and communication rounds, while previous models only handle one predefined communication bandwidth and a fixed number of communication rounds.
Revised: Second, it adapts to various communication bandwidths and communication rounds, while previous models only handle one predefined communication bandwidth and a fixed number of communication rounds.

Original: To evaluate Where2comm , we consider the collaborative 3D object detection task on four datasets: DAIR-V2X [11], V2X-Sim [9], OPV2V [10] and our original dataset CoPerception-UAVs.
Revised: To evaluate Where2comm , we consider the collaborative 3D object detection task on four datasets: DAIR-V2X [11], V2X-Sim [9], OPV2V [10] and our original dataset CoPerception-UAVs.

Original: Our experiments cover both real-world and simulation scenarios, two types of agents (cars and drones) and sensors (LiDAR and cameras).
Revised: Our experiments cover both real-world and simulation scenarios, two types of agents (cars and drones) and sensors (LiDAR and cameras).

Original: Results show that i) the proposed Where2comm consistently and significantly outperforms previous works in the performancebandwidth trade-off across multiple datasets and modalities; and ii) Where2comm achieves better trade-off when the communication round increases.
Revised: Results show that i) the proposed Where2comm consistently and significantly outperforms previous works in the performancebandwidth trade-off across multiple datasets and modalities; and ii) Where2comm achieves better trade-off when the communication round increases.

Original:  Related Works
Revised:  Related Works

Original: Multi-agent communication.
Revised: Multi-agent communication.

Original: The communication strategy in multi-agent systems has been widely studied [20].
Revised: The communication strategy in multi-agent systems has been widely studied [20].

Original: Early works [21–23] often use predefined protocols or heuristics to decide how agents communicate with each other.
Revised: Early works [21–23] often use predefined protocols or heuristics to decide how agents communicate with each other.

Original: However, it is difficult to generalize those methods to complex tasks.
Revised: However, it is difficult to generalize those methods to complex tasks.

Original: Recent works, thus, explore learning-based methods for complex scenarios.
Revised: Recent works, thus, explore learning-based methods for complex scenarios.

Original: For example,
Revised: For example,

Original: [Table]
Revised: [Table]

Original: CommNet [24] learns continuous communication in the multi-agent system.
Revised: CommNet [24] learns continuous communication in the multi-agent system.

Original: Vain [25] adopts the attention mechanism to help agents selectively fuse the information from others.
Revised: Vain [25] adopts the attention mechanism to help agents selectively fuse the information from others.

Original: Most of these previous works consider decision-making tasks and adopt reinforcement learning due to the lack of explicit supervision.
Revised: Most of these previous works consider decision-making tasks and adopt reinforcement learning due to the lack of explicit supervision.

Original: In this work, we focus on the perception task.
Revised: In this work, we focus on the perception task.

Original: Based on direct perception supervision, we apply supervised learning to optimize the communication strategy in both trade-off perception ability and communication cost.
Revised: Based on direct perception supervision, we apply supervised learning to optimize the communication strategy in both trade-off perception ability and communication cost.

Original: Collaborative perception.
Revised: Collaborative perception.

Original: As a recent application of multi-agent communication systems to perception tasks, collaborative perception is still immature.
Revised: As a recent application of multi-agent communication systems to perception tasks, collaborative perception is still immature.

Original: To support this area of research, there is a surge of high-quality datasets (e.g., V2X-Sim [9], OpenV2V [10], Comap[27] and DAIR-V2X[11]), as well as collaboration methods aimed for better performance-bandwidth trade-off (see comparisons in Table 1).
Revised: To support this area of research, there is a surge of high-quality datasets (e.g., V2X-Sim [9], OpenV2V [10], Comap[27] and DAIR-V2X[11]), as well as collaboration methods aimed for better performance-bandwidth trade-off (see comparisons in Table 1).

Original: When2com [12] proposes a handshake communication mechanism to decide when to communicate and create sparse communication graph.
Revised: When2com [12] proposes a handshake communication mechanism to decide when to communicate and create sparse communication graph.

Original: V2VNet [1] proposes multi-round message passing based on graph neural networks to achieve better perception and prediction performance.
Revised: V2VNet [1] proposes multi-round message passing based on graph neural networks to achieve better perception and prediction performance.

Original: DiscoNet [2] adopts knowledge distillation to take the advantage of both early and intermediate collaboration.
Revised: DiscoNet [2] adopts knowledge distillation to take the advantage of both early and intermediate collaboration.

Original: OPV2V [10] proposes a graph-based attentive intermediate fusion to improve perception performances.
Revised: OPV2V [10] proposes a graph-based attentive intermediate fusion to improve perception performances.

Original: V2X-ViT [26] introduces a novel heterogeneous multi-agent attention module to fuse information across heterogeneous agents.
Revised: V2X-ViT [26] introduces a novel heterogeneous multi-agent attention module to fuse information across heterogeneous agents.

Original: In this work, we leverage the proposed spatial confidence map to promote more compact messages, more sparse communication graphs, and more comprehensive fusion, resulting in efficient and effective collaboration.
Revised: In this work, we leverage the proposed spatial confidence map to promote more compact messages, more sparse communication graphs, and more comprehensive fusion, resulting in efficient and effective collaboration.

Original:  Problem Formulation
Revised:  Problem Formulation

Original: Consider N agents in the scene.
Revised: Consider N agents in the scene.

Original: Let X i
Revised: Let X i

Original: and
Revised: and

Original: Y
Revised: Y

Original: i be the observation and the perception supervision of the i th agent, respectively.
Revised: i be the observation and the perception supervision of the i th agent, respectively.

Original: The objective of collaborative perception is to achieve the maximized perception performance of all agents as a function of the total communication budge B and communication round K ; that is,
Revised: The objective of collaborative perception is to achieve the maximized perception performance of all agents as a function of the total communication budge B and communication round K ; that is,

Original: [Equation]
Revised: [Equation]

Original: where g ( · , · ) is the perception evaluation metric, Φ is the perception network with trainable parameter θ , and P ( k ) i
Revised: where g ( · , · ) is the perception evaluation metric, Φ is the perception network with trainable parameter θ , and P ( k ) i

Original: → j is the message transmitted from the i th agent to the j th agent at the k th communication round.
Revised: → j is the message transmitted from the i th agent to the j th agent at the k th communication round.

Original: Note that i) when B = K = 0 , there is no collaboration and ξ Φ (0 , 0) reflects the singleagent perception performance; ii) through optimizing the communication strategy and the network parameter, collaborative perception should perform well consistently at any communication bandwidth or round; and iii) we consider multi-round communication, where each agent serves as both a supporter (offering message to help others) and a requester (requesting messages from others).
Revised: Note that i) when B = K = 0 , there is no collaboration and ξ Φ (0 , 0) reflects the singleagent perception performance; ii) through optimizing the communication strategy and the network parameter, collaborative perception should perform well consistently at any communication bandwidth or round; and iii) we consider multi-round communication, where each agent serves as both a supporter (offering message to help others) and a requester (requesting messages from others).

Original: In this work, we consider the perception task of 3D object detection and present three contributions: i) we make communication more efficient by designing compact messages and sparse communication graphs; ii) we boost the perception performance by implementing more comprehensive message fusion; iii) we enable the overall system to adapt to varying communication conditions by dynamically adjusting where and who to communicate.
Revised: In this work, we consider the perception task of 3D object detection and present three contributions: i) we make communication more efficient by designing compact messages and sparse communication graphs; ii) we boost the perception performance by implementing more comprehensive message fusion; iii) we enable the overall system to adapt to varying communication conditions by dynamically adjusting where and who to communicate.

Original:  Where2comm : Spatial Confidence-Aware Collaborative Perception System
Revised:  Where2comm : Spatial Confidence-Aware Collaborative Perception System

Original: This section presents Where2comm , a multi-round, multi-modality, multi-agent collaborative perception framework based on a spatial-confidence-aware communication strategy; see the overview in Fig.
Revised: This section presents Where2comm , a multi-round, multi-modality, multi-agent collaborative perception framework based on a spatial-confidence-aware communication strategy; see the overview in Fig.

Original: Where2comm includes an observation encoder, a spatial confidence generator, the spatial confidence-aware communication module, the spatial confidence-aware message fusion module and a detection decoder.
Revised: Where2comm includes an observation encoder, a spatial confidence generator, the spatial confidence-aware communication module, the spatial confidence-aware message fusion module and a detection decoder.

Original: Among five modules, the proposed spatial confidence generator generates the spatial confidence map.
Revised: Among five modules, the proposed spatial confidence generator generates the spatial confidence map.

Original: Based on this spatial confidence map, the proposed spatial confidence-aware communication generates compact messages and sparse communication graphs to save communication bandwidth; and the proposed spatial confidence-aware message fusion module leverages informative spatial confidence priors to achieve better aggregation; also see an algorithmic summary in Algorithm 1 and the optimization-oriented design rationale in Section 7.3 in Appendix.
Revised: Based on this spatial confidence map, the proposed spatial confidence-aware communication generates compact messages and sparse communication graphs to save communication bandwidth; and the proposed spatial confidence-aware message fusion module leverages informative spatial confidence priors to achieve better aggregation; also see an algorithmic summary in Algorithm 1 and the optimization-oriented design rationale in Section 7.3 in Appendix.

Original: 4.1 Observation encoder
Revised: 4.1 Observation encoder

Original: The observation encoder extracts feature maps from the sensor data.
Revised: The observation encoder extracts feature maps from the sensor data.

Original: Where2comm accepts single/multimodality inputs, such as RGB images and 3D point clouds.
Revised: Where2comm accepts single/multimodality inputs, such as RGB images and 3D point clouds.

Original: This work adopts the feature representations in bird’s eye view (BEV), where all agents project their individual perceptual information to the same global coordinate system, avoiding complex coordinate transformations and supporting better shared cross-agent collaboration.
Revised: This work adopts the feature representations in bird’s eye view (BEV), where all agents project their individual perceptual information to the same global coordinate system, avoiding complex coordinate transformations and supporting better shared cross-agent collaboration.

Original: For the i th agent, given its input X i , the feature map is
Revised: For the i th agent, given its input X i , the feature map is

Original: F (0) i =
Revised: F (0) i =

Original: Φ enc ( X i ) ∈
Revised: Φ enc ( X i ) ∈

Original: R H
Revised: R H

Original: × W × D , where Φ enc ( · ) is the encoder, the superscript 0 reflects that the feature is obtained before communication and H, W, D are its height, weight and channel.
Revised: × W × D , where Φ enc ( · ) is the encoder, the superscript 0 reflects that the feature is obtained before communication and H, W, D are its height, weight and channel.

Original: All agents share the same BEV coordinate system.
Revised: All agents share the same BEV coordinate system.

Original: For the image input, Φ enc ( · ) is followed by a warping function that transforms the extracted feature from front-view to BEV.
Revised: For the image input, Φ enc ( · ) is followed by a warping function that transforms the extracted feature from front-view to BEV.

Original: For 3D point cloud input, we discretize 3D points as a BEV map and Φ enc ( · ) extracts features in BEV.
Revised: For 3D point cloud input, we discretize 3D points as a BEV map and Φ enc ( · ) extracts features in BEV.

Original: The extracted feature map is output to the spatial confidence generator and the message fusion module.
Revised: The extracted feature map is output to the spatial confidence generator and the message fusion module.

Original: 4.2 Spatial confidence generator
Revised: 4.2 Spatial confidence generator

Original: The spatial confidence generator generates a spatial confidence map from the feature map of each agent.
Revised: The spatial confidence generator generates a spatial confidence map from the feature map of each agent.

Original: The spatial confidence map reflects the perceptually critical level of various spatial areas.
Revised: The spatial confidence map reflects the perceptually critical level of various spatial areas.

Original: Intuitively, for object detection task, the areas that contain objects are more critical than background areas.
Revised: Intuitively, for object detection task, the areas that contain objects are more critical than background areas.

Original: During collaboration, areas with objects could help recover the miss-detected objects due to the limited view; and background areas could be omitted to save the precious bandwidth.
Revised: During collaboration, areas with objects could help recover the miss-detected objects due to the limited view; and background areas could be omitted to save the precious bandwidth.

Original: So we represent the spatial confidence map with the detection confidence map, where the area with high perceptually critical level is the area that contains an object with a high confidence score.
Revised: So we represent the spatial confidence map with the detection confidence map, where the area with high perceptually critical level is the area that contains an object with a high confidence score.

Original: To implement, we use a detection decoder structure to produce the detection confidence map.
Revised: To implement, we use a detection decoder structure to produce the detection confidence map.

Original: Given the feature map at the k th communication round, F ( k ) i , the corresponding spatial confidence map is
Revised: Given the feature map at the k th communication round, F ( k ) i , the corresponding spatial confidence map is

Original: [Equation]
Revised: [Equation]

Original: where the generator Φ generator ( · ) follows a detection decoder.
Revised: where the generator Φ generator ( · ) follows a detection decoder.

Original: Since we consider multi-round collaboration, Where2comm iteratively updates the feature map by aggregating information from other agents.
Revised: Since we consider multi-round collaboration, Where2comm iteratively updates the feature map by aggregating information from other agents.

Original: Once F ( k ) i is obtained, (1) is triggered to reflect the perceptually critical level at each spatial location.
Revised: Once F ( k ) i is obtained, (1) is triggered to reflect the perceptually critical level at each spatial location.

Original: The proposed spatial confidence map answers a crucial question that was ignored by previous works: for each agent, information at which spatial area is worth sharing with others.
Revised: The proposed spatial confidence map answers a crucial question that was ignored by previous works: for each agent, information at which spatial area is worth sharing with others.

Original: By answering this, it provides a solid base for efficient communication and effective message fusion.
Revised: By answering this, it provides a solid base for efficient communication and effective message fusion.

Original: 4.3 Spatial confidence-aware communication
Revised: 4.3 Spatial confidence-aware communication

Original: With the guidance of spatial confidence maps, the proposed communication module packs compact messages with spatially sparse feature maps and transmits messages through a sparsely-connected communication graph.
Revised: With the guidance of spatial confidence maps, the proposed communication module packs compact messages with spatially sparse feature maps and transmits messages through a sparsely-connected communication graph.

Original: Most existing collaboration perception systems [1, 2, 26] considers full feature maps in the messages and fully-connected communication graphs.
Revised: Most existing collaboration perception systems [1, 2, 26] considers full feature maps in the messages and fully-connected communication graphs.

Original: To reduce the communication bandwidth without affecting perception, we leverage the spatial confidence map to select the most informative spatial areas in the feature map (where to communicate) and decide the most beneficial collaboration partners (who to communicate).
Revised: To reduce the communication bandwidth without affecting perception, we leverage the spatial confidence map to select the most informative spatial areas in the feature map (where to communicate) and decide the most beneficial collaboration partners (who to communicate).

Original: Message packing.
Revised: Message packing.

Original: Message packing determines what information should be included in the to-besent message.
Revised: Message packing determines what information should be included in the to-besent message.

Original: The proposed message includes: i) a request map that indicates at which spatial areas the agent needs to know more; and ii) a spatially sparse, yet perceptually critical feature map.
Revised: The proposed message includes: i) a request map that indicates at which spatial areas the agent needs to know more; and ii) a spatially sparse, yet perceptually critical feature map.

Original: The request map of the i th agent is R ( k )
Revised: The request map of the i th agent is R ( k )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ∈ R H × W , negatively correlated with the spatial confidence map.
Revised: ∈ R H × W , negatively correlated with the spatial confidence map.

Original: The intuition is, for the locations with low confidence score, an agent is hard to tell if there is really no objects or it is just caused by the limited information (e.g. occlusion).
Revised: The intuition is, for the locations with low confidence score, an agent is hard to tell if there is really no objects or it is just caused by the limited information (e.g. occlusion).

Original: Thus, the low confidence score indicates there could be missing information at that location.
Revised: Thus, the low confidence score indicates there could be missing information at that location.

Original: Requesting information at these locations from other agents could improve the current agent’s detection accuracy.
Revised: Requesting information at these locations from other agents could improve the current agent’s detection accuracy.

Original: The spatially sparse feature map are selected based on each agent’s spatial confidence map and the received request maps from others.
Revised: The spatially sparse feature map are selected based on each agent’s spatial confidence map and the received request maps from others.

Original: Specifically, a binary selection matrix is used to represent each location is selected or not, where 1 denotes selected, and 0 elsewhere.
Revised: Specifically, a binary selection matrix is used to represent each location is selected or not, where 1 denotes selected, and 0 elsewhere.

Original: For the message sent from the i th agent to the j th agent at the k th communication round, the binary selection matrix is
Revised: For the message sent from the i th agent to the j th agent at the k th communication round, the binary selection matrix is

Original: [Equation]
Revised: [Equation]

Original: where ⊙ is the element-wise multiplication, R ( k − 1) j is the request map from the j th agent received at the previous round, Φ select ( · ) is the selection function which targets to select the most critical areas conditioned on the input matrix, which represents the critical level at the certain spatial location.
Revised: where ⊙ is the element-wise multiplication, R ( k − 1) j is the request map from the j th agent received at the previous round, Φ select ( · ) is the selection function which targets to select the most critical areas conditioned on the input matrix, which represents the critical level at the certain spatial location.

Original: We implement Φ select ( · ) by selecting the locations where the largest elements at in the given input matrix conditioned on the bandwidth limit; optionally, a Gaussian filter could be applied to filter out the outliers and introduce some context.
Revised: We implement Φ select ( · ) by selecting the locations where the largest elements at in the given input matrix conditioned on the bandwidth limit; optionally, a Gaussian filter could be applied to filter out the outliers and introduce some context.

Original: In the initial communication round, each agent selects the most critical areas from its own perspective as the request maps from other agents are not available yet; in the subsequent rounds, each agent also takes the partner’s request into account, enabling more targeted communication.
Revised: In the initial communication round, each agent selects the most critical areas from its own perspective as the request maps from other agents are not available yet; in the subsequent rounds, each agent also takes the partner’s request into account, enabling more targeted communication.

Original: Then, the selected feature map is obtained as Z ( k ) i
Revised: Then, the selected feature map is obtained as Z ( k ) i

Original: → j = M ( k ) i
Revised: → j = M ( k ) i

Original: →
Revised: →

Original: j ⊙ F ( k ) i ∈
Revised: j ⊙ F ( k ) i ∈

Original: R H
Revised: R H

Original: × W × D , which provides spatially sparse, yet perceptually critical information.
Revised: × W × D , which provides spatially sparse, yet perceptually critical information.

Original: Overall, the message sent from the i th agent to the j th agent at the k th communication round is P ( k ) i
Revised: Overall, the message sent from the i th agent to the j th agent at the k th communication round is P ( k ) i

Original: → j = ( R ( k ) i , Z ( k ) i
Revised: → j = ( R ( k ) i , Z ( k ) i

Original: → j ) .
Revised: → j ) .

Original: Note that i) R ( k ) i provides spatial priors to request complementary information for the i th agent’s need in the next round; the feature map Z ( k ) i → j provides supportive information for the i th agent’s need in the this round.
Revised: Note that i) R ( k ) i provides spatial priors to request complementary information for the i th agent’s need in the next round; the feature map Z ( k ) i → j provides supportive information for the i th agent’s need in the this round.

Original: They together enable mutually beneficial collaboration; ii) since Z ( k ) i
Revised: They together enable mutually beneficial collaboration; ii) since Z ( k ) i

Original: → j is sparse, we only transmit non-zero features and corresponding indices, leading to low communication cost; and iii) the sparsity of Z ( k ) i
Revised: → j is sparse, we only transmit non-zero features and corresponding indices, leading to low communication cost; and iii) the sparsity of Z ( k ) i

Original: → j is determined by the binary selection matrix, which dynamically allocates the communication budget at various spatial areas based on their perceptual critical level, adapting to various communication conditions.
Revised: → j is determined by the binary selection matrix, which dynamically allocates the communication budget at various spatial areas based on their perceptual critical level, adapting to various communication conditions.

Original: Communication graph construction.
Revised: Communication graph construction.

Original: Communication graph construction targets to identify when and who to communicate to avoid unnecessary communication that wastes the bandwidth.
Revised: Communication graph construction targets to identify when and who to communicate to avoid unnecessary communication that wastes the bandwidth.

Original: Most previous works [1, 2, 10] consider fully-connected communication graphs.
Revised: Most previous works [1, 2, 10] consider fully-connected communication graphs.

Original: When2com [12] proposes a handshake mechanism, which uses similar global features to match partners.
Revised: When2com [12] proposes a handshake mechanism, which uses similar global features to match partners.

Original: This is hard to interpret because two agents, which have similar global features, do not necessarily need information from each other.
Revised: This is hard to interpret because two agents, which have similar global features, do not necessarily need information from each other.

Original: Different from all previous works, we provide an explicit design rationale: the necessity of communication between the i th and the j th agents is simply measured by the overlap between the information that the i th agent has and the information that the j th agent needs.
Revised: Different from all previous works, we provide an explicit design rationale: the necessity of communication between the i th and the j th agents is simply measured by the overlap between the information that the i th agent has and the information that the j th agent needs.

Original: With the help of the spatial confidence map and the request map, we construct a more interpretable communication graph.
Revised: With the help of the spatial confidence map and the request map, we construct a more interpretable communication graph.

Original: For the initial communication round, every agent in the system is not aware of other agents yet.
Revised: For the initial communication round, every agent in the system is not aware of other agents yet.

Original: To activate the collaboration, we construct a fully-connected communication graph.
Revised: To activate the collaboration, we construct a fully-connected communication graph.

Original: Every agent will broadcast its message to the rest of the system.
Revised: Every agent will broadcast its message to the rest of the system.

Original: For the subsequent communication rounds, we examine if the communication between agent i and agent j is necessary based on the maximum value of the binary selection matrix M ( k ) i
Revised: For the subsequent communication rounds, we examine if the communication between agent i and agent j is necessary based on the maximum value of the binary selection matrix M ( k ) i

Original: → j , i.e. if there is at least one patch is activated, then we regard the connection is necessary.
Revised: → j , i.e. if there is at least one patch is activated, then we regard the connection is necessary.

Original: Formally, let A ( k ) be the adjacency matrix of the communication graph at the k th communication round, whose ( i, j ) th element is
Revised: Formally, let A ( k ) be the adjacency matrix of the communication graph at the k th communication round, whose ( i, j ) th element is

Original: [Equation]
Revised: [Equation]

Original: where h, w index the spatial area, reflecting message passing from the i th agent to the j th agent.
Revised: where h, w index the spatial area, reflecting message passing from the i th agent to the j th agent.

Original: Given this sparse communication graph, agents can exchange messages with selected partners.
Revised: Given this sparse communication graph, agents can exchange messages with selected partners.

Original: 4.4 Spatial confidence-aware message fusion
Revised: 4.4 Spatial confidence-aware message fusion

Original: Spatial confidence-aware message fusion targets to augment the feature of each agent by aggregating the received messages from the other agents.
Revised: Spatial confidence-aware message fusion targets to augment the feature of each agent by aggregating the received messages from the other agents.

Original: To achieve this, we adopt a transformer architecture, which leverages multi-head attention to fuse the corresponding features from multiple agents at each individual spatial location.
Revised: To achieve this, we adopt a transformer architecture, which leverages multi-head attention to fuse the corresponding features from multiple agents at each individual spatial location.

Original: The key technical design is to include the spatial confidence maps of all the agents to promote cross-agent attention learning.
Revised: The key technical design is to include the spatial confidence maps of all the agents to promote cross-agent attention learning.

Original: The intuition is that, the spatial confidence map could explicitly reflect the perceptually critical level, providing a useful prior for attention learning.
Revised: The intuition is that, the spatial confidence map could explicitly reflect the perceptually critical level, providing a useful prior for attention learning.

Original: Specifically, for the i th agent, after receiving the j th agent’s message P ( k ) j
Revised: Specifically, for the i th agent, after receiving the j th agent’s message P ( k ) j

Original: → i , it could unpack to retrieve the feature map Z ( k ) j → i and the spatial confidence map C ( k ) j = 1 − R ( k ) j .
Revised: → i , it could unpack to retrieve the feature map Z ( k ) j → i and the spatial confidence map C ( k ) j = 1 − R ( k ) j .

Original: We also include the ego feature map in fusion and denote Z ( k ) i
Revised: We also include the ego feature map in fusion and denote Z ( k ) i

Original: → i = F ( k ) i to make the formulation simple and consistent, where
Revised: → i = F ( k ) i to make the formulation simple and consistent, where

Original: Z ( k ) i
Revised: Z ( k ) i

Original: → i might not be sparse.
Revised: → i might not be sparse.

Original: To fuse the features from the j th agent at the k th communication round, the cross-agent/ego attention weight for the i th agent is
Revised: To fuse the features from the j th agent at the k th communication round, the cross-agent/ego attention weight for the i th agent is

Original: [Equation]
Revised: [Equation]

Original: where MHA W ( · ) is a multi-head attention applied at each individual spatial location, which outputs the scaled dot-product attention weight.
Revised: where MHA W ( · ) is a multi-head attention applied at each individual spatial location, which outputs the scaled dot-product attention weight.

Original: Note that i) the proposed spatial confidence maps contributes to the attention weight, as the features with higher perceptually critical level are more preferred in the feature aggregation; ii) the cross-agent attention weight models the collaboration strength with a H × W spatial resolution, leading to more flexible information fusion at various spatial regions.
Revised: Note that i) the proposed spatial confidence maps contributes to the attention weight, as the features with higher perceptually critical level are more preferred in the feature aggregation; ii) the cross-agent attention weight models the collaboration strength with a H × W spatial resolution, leading to more flexible information fusion at various spatial regions.

Original: Then, the feature map of the i th agent after fusing the messages in the k th communication round is
Revised: Then, the feature map of the i th agent after fusing the messages in the k th communication round is

Original: [Equation]
Revised: [Equation]

Original: where FFN( · ) is the feed-forward network and N i is the neighbors of the i th agent defined in the communication graph A ( k ) .
Revised: where FFN( · ) is the feed-forward network and N i is the neighbors of the i th agent defined in the communication graph A ( k ) .

Original: The fused feature F ( k +1)
Revised: The fused feature F ( k +1)

Original: i would serve as the i th agent’s feature in the ( k + 1) th round.
Revised: i would serve as the i th agent’s feature in the ( k + 1) th round.

Original: In the final round, we output F ( k +1)
Revised: In the final round, we output F ( k +1)

Original: i to the detection decoder to generate detections.
Revised: i to the detection decoder to generate detections.

Original: Sensor positional encoding.
Revised: Sensor positional encoding.

Original: Sensor positional encoding represents the physical distance between each agent’s sensor and its observation.
Revised: Sensor positional encoding represents the physical distance between each agent’s sensor and its observation.

Original: It adopts a standard positional encoding function conditioned on the sensing distance and feature dimension.
Revised: It adopts a standard positional encoding function conditioned on the sensing distance and feature dimension.

Original: The features are summed up with the positional encoding of each location before inputting to the transformer.
Revised: The features are summed up with the positional encoding of each location before inputting to the transformer.

Original: Compared to existing fusion modules that do not use attention mechanism [1] or only use agent-level attentions [12], the per-location attention mechanism adopted by the proposed fusion emphasizes the location-specific feature interactions.
Revised: Compared to existing fusion modules that do not use attention mechanism [1] or only use agent-level attentions [12], the per-location attention mechanism adopted by the proposed fusion emphasizes the location-specific feature interactions.

Original: It makes the feature fusion more targeted.
Revised: It makes the feature fusion more targeted.

Original: Compared to the methods that also use the per-location attention-based fusion module[2, 10, 26], the proposed fusion module leverages multi-head attention with two extra priors, including spatial confidence map and sensing distances.
Revised: Compared to the methods that also use the per-location attention-based fusion module[2, 10, 26], the proposed fusion module leverages multi-head attention with two extra priors, including spatial confidence map and sensing distances.

Original: Both assist attention learning to prefer high quality and critical features.
Revised: Both assist attention learning to prefer high quality and critical features.

Original: 4.5 Detection decoder
Revised: 4.5 Detection decoder

Original: The detection decoder decodes features into objects, including class and regression output.
Revised: The detection decoder decodes features into objects, including class and regression output.

Original: Given the feature map at the k th communication round F ( k ) i , the detection decoder Φ dec ( · ) generate the detections of i th agent by (cid:98) O ( k ) i =
Revised: Given the feature map at the k th communication round F ( k ) i , the detection decoder Φ dec ( · ) generate the detections of i th agent by (cid:98) O ( k ) i =

Original: Φ dec ( F ( k ) i )
Revised: Φ dec ( F ( k ) i )

Original: ∈
Revised: ∈

Original: R H
Revised: R H

Original: × W × 7 , where each location of (cid:98) O ( k ) i represents a rotated box with class ( c, x, y, h, w, cos α, sin α ) , denoting class confidence, position, size and angle.
Revised: × W × 7 , where each location of (cid:98) O ( k ) i represents a rotated box with class ( c, x, y, h, w, cos α, sin α ) , denoting class confidence, position, size and angle.

Original: The objects are the final output of the proposed collaborative perception system.
Revised: The objects are the final output of the proposed collaborative perception system.

Original: Note that (cid:98) O (0) i denotes the detections without collaboration.
Revised: Note that (cid:98) O (0) i denotes the detections without collaboration.

Original: 4.6 Training details and loss functions
Revised: 4.6 Training details and loss functions

Original: To train the overall system, we supervise two tasks: spatial confidence generation and object detection at each round.
Revised: To train the overall system, we supervise two tasks: spatial confidence generation and object detection at each round.

Original: As mentioned before, the functionality of the spatial confidence generator is the same as the classification in the detection decoder.
Revised: As mentioned before, the functionality of the spatial confidence generator is the same as the classification in the detection decoder.

Original: To promote parameter efficiency, our spatial confidence generator reuses the parameters of the detection decoder.
Revised: To promote parameter efficiency, our spatial confidence generator reuses the parameters of the detection decoder.

Original: For the multi-round settings, each round is
Revised: For the multi-round settings, each round is

Original: [Figure]
Revised: [Figure]

Original: supervised with one detection loss, the overall loss is L = (cid:80)
Revised: supervised with one detection loss, the overall loss is L = (cid:80)

Original: Kk =0 (cid:80) Ni L det (cid:16) (cid:98) O
Revised: Kk =0 (cid:80) Ni L det (cid:16) (cid:98) O

Original: ( k ) i , O i (cid:17) , where O i is the i th agent’s ground-truth objects, L det is the detection loss [28].
Revised: ( k ) i , O i (cid:17) , where O i is the i th agent’s ground-truth objects, L det is the detection loss [28].

Original: Training strategy for multi-round setting.
Revised: Training strategy for multi-round setting.

Original: To adapt to multi-round communication and dynamic bandwidth, we train the model under various communication settings with curriculum learning strategy [29].
Revised: To adapt to multi-round communication and dynamic bandwidth, we train the model under various communication settings with curriculum learning strategy [29].

Original: We first gradually increase the communication bandwidth and round; and then, randomly sample bandwidth and round to promote robustness.
Revised: We first gradually increase the communication bandwidth and round; and then, randomly sample bandwidth and round to promote robustness.

Original: Through this training strategy, a single model can perform well at various communication conditions.
Revised: Through this training strategy, a single model can perform well at various communication conditions.

Original:  Experimental Results
Revised:  Experimental Results

Original: Our experiments covers four datasets, both real-world and simulation scenarios, two types of agents (cars and drones) and two types of sensors (LiDAR and cameras).
Revised: Our experiments covers four datasets, both real-world and simulation scenarios, two types of agents (cars and drones) and two types of sensors (LiDAR and cameras).

Original: Specifically, we conduct camera-only 3D object detection in the setting of V2X-communication aided autonomous driving on OPV2V dataset [10], camera-only 3D object detection in the setting of drone swarm on the proposed CoPerception-UAVs dataset, and LiDAR-based 3D object detection on DAIR-V2X dataset [11] and V2X-Sim dataset [9].
Revised: Specifically, we conduct camera-only 3D object detection in the setting of V2X-communication aided autonomous driving on OPV2V dataset [10], camera-only 3D object detection in the setting of drone swarm on the proposed CoPerception-UAVs dataset, and LiDAR-based 3D object detection on DAIR-V2X dataset [11] and V2X-Sim dataset [9].

Original: The detection results are evaluated by Average Precision (AP) at Intersection-over-Union (IoU) threshold of 0 .
Revised: The detection results are evaluated by Average Precision (AP) at Intersection-over-Union (IoU) threshold of 0 .

Original: 50 and 0 .
Revised: 50 and 0 .

Original: The communication results count the message size by byte in log scale with base 2 .
Revised: The communication results count the message size by byte in log scale with base 2 .

Original: To compare communication results straightforward and fair, we do not consider any extra data/feature/model compression.
Revised: To compare communication results straightforward and fair, we do not consider any extra data/feature/model compression.

Original: 5.1 Datasets and experimental settings
Revised: 5.1 Datasets and experimental settings

Original: [Equation]
Revised: [Equation]

Original: [10] is a vehicle-to-vehicle collaborative perception dataset, co-simulated by
Revised: [10] is a vehicle-to-vehicle collaborative perception dataset, co-simulated by

Original: OpenCDA [10] and Carla [30].
Revised: OpenCDA [10] and Carla [30].

Original: It includes 12 K frames of 3D point clouds and RGB images with 230 K annotated 3D boxes.
Revised: It includes 12 K frames of 3D point clouds and RGB images with 230 K annotated 3D boxes.

Original: The perception range is 40m × 40m. For camera-only 3D object detection task on OPV2V, we implement the detector following CADDN [31].
Revised: The perception range is 40m × 40m. For camera-only 3D object detection task on OPV2V, we implement the detector following CADDN [31].

Original: The input front-view image size is (416 , 160) .
Revised: The input front-view image size is (416 , 160) .

Original: The front-view input feature map is transformed to BEV with resolution 0 .
Revised: The front-view input feature map is transformed to BEV with resolution 0 .

Original: 5 m/pixel.
Revised: 5 m/pixel.

Original: V2X-Sim.
Revised: V2X-Sim.

Original: V2X-Sim [9] is a vehicle-to-everything collaborative perception dataset, co-simulated by SUMO [32] and Carla, including 10K frames of 3D LiDAR point clouds and 501K 3D boxes.
Revised: V2X-Sim [9] is a vehicle-to-everything collaborative perception dataset, co-simulated by SUMO [32] and Carla, including 10K frames of 3D LiDAR point clouds and 501K 3D boxes.

Original: [Figure]
Revised: [Figure]

Original: CoPerception-UAVs.
Revised: CoPerception-UAVs.

Original: To enrich the collaborative perception datasets, we consider the swarm of unmanned aerial vehicles (UAV) and propose a UAV-swarm-based collaborative perception dataset:
Revised: To enrich the collaborative perception datasets, we consider the swarm of unmanned aerial vehicles (UAV) and propose a UAV-swarm-based collaborative perception dataset:

Original: CoPerception-UAVs, co-simulated by AirSim [34] and Carla [30], including 131.9K aerial images and 1.94M 3D boxes.
Revised: CoPerception-UAVs, co-simulated by AirSim [34] and Carla [30], including 131.9K aerial images and 1.94M 3D boxes.

Original: The perception range is 200m × 350m. For the camera-only 3D object detection task on CoPerception-UAVs, our detector follows DVDET [8].
Revised: The perception range is 200m × 350m. For the camera-only 3D object detection task on CoPerception-UAVs, our detector follows DVDET [8].

Original: The input aerial image size is (800 , 450) .
Revised: The input aerial image size is (800 , 450) .

Original: The aerial-view input feature map is transformed to BEV with the resolution of 0 .
Revised: The aerial-view input feature map is transformed to BEV with the resolution of 0 .

Original: 25 m/pixel, and the size is (192 , 352) ; see more details in Appendix.
Revised: 25 m/pixel, and the size is (192 , 352) ; see more details in Appendix.

Original: DAIR-V2X. DAIR-V2X [11] is the only public real-world collaborative perception dataset.
Revised: DAIR-V2X. DAIR-V2X [11] is the only public real-world collaborative perception dataset.

Original: Each sample contains two agents: a vehicle and an infrastructure, with 3D annotations.
Revised: Each sample contains two agents: a vehicle and an infrastructure, with 3D annotations.

Original: The perception range is 201.6m × 80m. Originally DAIR-V2X does not label objects outside the camera’s view, we relabel all objects to cover 360-degree detection range.
Revised: The perception range is 201.6m × 80m. Originally DAIR-V2X does not label objects outside the camera’s view, we relabel all objects to cover 360-degree detection range.

Original: We complement several intermediate fusion-based baselines on DAIR-V2X to comprehensively validate our method on real data.
Revised: We complement several intermediate fusion-based baselines on DAIR-V2X to comprehensively validate our method on real data.

Original: For
Revised: For

Original: LiDAR-based 3D object detection task
Revised: LiDAR-based 3D object detection task

Original: , our detector follows PointPillar [35].
Revised: , our detector follows PointPillar [35].

Original: We represent the field of view into a BEV map with size (200 , 504 , 64) and the resolution is 0 .
Revised: We represent the field of view into a BEV map with size (200 , 504 , 64) and the resolution is 0 .

Original: 4 m/pixel in length and width.
Revised: 4 m/pixel in length and width.

Original: 5.2 Quantitative evaluation
Revised: 5.2 Quantitative evaluation

Original: Benchmark comparison.
Revised: Benchmark comparison.

Original: Fig.
Revised: Fig.

Original: 3 compares the proposed Where2comm with the previous methods in terms of the trade-off between detection performance (AP@IoU=0.50) and communication bandwidth; also see exact values in Table 3 of Appendix.
Revised: 3 compares the proposed Where2comm with the previous methods in terms of the trade-off between detection performance (AP@IoU=0.50) and communication bandwidth; also see exact values in Table 3 of Appendix.

Original: We consider single-agent detection without collaboration ( (cid:98) O
Revised: We consider single-agent detection without collaboration ( (cid:98) O

Original: (0) i ), When2com [12], V2VNet [1], DiscoNet [2], V2X-ViT [26] and late fusion, where agents directly exchange the detected 3D boxes.
Revised: (0) i ), When2com [12], V2VNet [1], DiscoNet [2], V2X-ViT [26] and late fusion, where agents directly exchange the detected 3D boxes.

Original: The red curve comes from a single Where2comm model evaluated at varying bandwidths.
Revised: The red curve comes from a single Where2comm model evaluated at varying bandwidths.

Original: We see that the proposed Where2comm : i) achieves a far-more superior perception-communication trade-off across all the communication bandwidth choices and various collaborative perception tasks, including camera-only 3D object detection from aerial view and car front view, and LiDAR-based 3D object detection; ii) achieves significant improvements over previous state-of-the-arts on both real-world (DAIR-V2X) and simulation scenarios, improves the
Revised: We see that the proposed Where2comm : i) achieves a far-more superior perception-communication trade-off across all the communication bandwidth choices and various collaborative perception tasks, including camera-only 3D object detection from aerial view and car front view, and LiDAR-based 3D object detection; ii) achieves significant improvements over previous state-of-the-arts on both real-world (DAIR-V2X) and simulation scenarios, improves the

Original: SOTA performance by 7.7% on DAIR-V2X, 6.62% on CoPerception-UAVs, 25.81% on OPV2V, 1.9% on V2X-Sim; iii) achieves the same detection performance of previous state-of-the-arts with extremely less communication volume: 5128 times less on CoPerception-UAVs, more than 100K times less on OPV2V, 55 times less on V2X-Sim, 105 times less on DAIR-V2X.
Revised: SOTA performance by 7.7% on DAIR-V2X, 6.62% on CoPerception-UAVs, 25.81% on OPV2V, 1.9% on V2X-Sim; iii) achieves the same detection performance of previous state-of-the-arts with extremely less communication volume: 5128 times less on CoPerception-UAVs, more than 100K times less on OPV2V, 55 times less on V2X-Sim, 105 times less on DAIR-V2X.

Original: Multi-round evaluation.
Revised: Multi-round evaluation.

Original: Fig.
Revised: Fig.

Original: 4 presents the performances of Where2comm at communication rounds ranging from 1 to 3.
Revised: 4 presents the performances of Where2comm at communication rounds ranging from 1 to 3.

Original: Each curve comes from a single Where2comm model with a certain communication round evaluated at varying bandwidths.
Revised: Each curve comes from a single Where2comm model with a certain communication round evaluated at varying bandwidths.

Original: Results show that 1 communication round is good, more rounds are even better.
Revised: Results show that 1 communication round is good, more rounds are even better.

Original: Multi-round communication steadily improves the performance-bandwidth trade-off across all three datasets, reflecting its effectiveness and robustness.
Revised: Multi-round communication steadily improves the performance-bandwidth trade-off across all three datasets, reflecting its effectiveness and robustness.

Original: This encourages the agents to actively collaborate without worrying the performance degradation.
Revised: This encourages the agents to actively collaborate without worrying the performance degradation.

Original: This also validates that Where2comm can well work at various communication bandwidths and rounds.
Revised: This also validates that Where2comm can well work at various communication bandwidths and rounds.

Original: Robustness to localization noise.
Revised: Robustness to localization noise.

Original: We follow the localization noise setting in V2VNet and V2X-ViT (Gaussian noise with a mean of 0m and a standard deviation of 0m-0.6m) and conduct experiments on all the three datasets to validate the robustness against realistic localization noise.
Revised: We follow the localization noise setting in V2VNet and V2X-ViT (Gaussian noise with a mean of 0m and a standard deviation of 0m-0.6m) and conduct experiments on all the three datasets to validate the robustness against realistic localization noise.

Original: Where2comm is more robust to the localization noise than previous SOTAs.
Revised: Where2comm is more robust to the localization noise than previous SOTAs.

Original: Fig.
Revised: Fig.

Original: 5 shows the detection performances as a function of localization noise level in CoPerception-UAVs, OPV2V and V2X-Sim datasets, respectively We see: i) overall the collaborative perception performance degrades with the increasing
Revised: 5 shows the detection performances as a function of localization noise level in CoPerception-UAVs, OPV2V and V2X-Sim datasets, respectively We see: i) overall the collaborative perception performance degrades with the increasing

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: localization noise, while where2comm outperforms previous SOTAs (When2com, V2VNet,DiscoNet) under all the localization noise.
Revised: localization noise, while where2comm outperforms previous SOTAs (When2com, V2VNet,DiscoNet) under all the localization noise.

Original: ii) where2comm keeps being superior to No Collaboration while
Revised: ii) where2comm keeps being superior to No Collaboration while

Original: V2VNet fails when noise is over 0.4m and DiscoNet fails when noise is over 0.5m on CoPerceptionUAVs.
Revised: V2VNet fails when noise is over 0.4m and DiscoNet fails when noise is over 0.5m on CoPerceptionUAVs.

Original: The reasons are: i) the powerful transformer architecture in fusion module attentively select the most suitable collaborative feature; ii) the spatial confidence map helps filter out noisy features, these two designs work together to mitigate noise localization distortion effects.
Revised: The reasons are: i) the powerful transformer architecture in fusion module attentively select the most suitable collaborative feature; ii) the spatial confidence map helps filter out noisy features, these two designs work together to mitigate noise localization distortion effects.

Original: 5.3 Qualitative evaluation
Revised: 5.3 Qualitative evaluation

Original: Visualization of spatial confidence map.
Revised: Visualization of spatial confidence map.

Original: Fig.
Revised: Fig.

Original: 6 illustrates how Where2comm is empowered by the proposed spatial confidence map.
Revised: 6 illustrates how Where2comm is empowered by the proposed spatial confidence map.

Original: In the scene, Drone 1’s view is occluded by a tall building.
Revised: In the scene, Drone 1’s view is occluded by a tall building.

Original: With
Revised: With

Original: Drone 2’s help, Drone 1 is able to detect through occlusion.
Revised: Drone 2’s help, Drone 1 is able to detect through occlusion.

Original: Fig.
Revised: Fig.

Original: 6 (a-d) shows Drone 1’s observation, spatial confidence map (1), binary selection matrix (2), and ego attention weight (3).
Revised: 6 (a-d) shows Drone 1’s observation, spatial confidence map (1), binary selection matrix (2), and ego attention weight (3).

Original: Fig.
Revised: Fig.

Original: 6 (f-h) shows Drone 2’s observation and message sent to Drone 1, including the request map (opposite of confidence map) and the sparse feature map, achieving efficient communication.
Revised: 6 (f-h) shows Drone 2’s observation and message sent to Drone 1, including the request map (opposite of confidence map) and the sparse feature map, achieving efficient communication.

Original: Fig.
Revised: Fig.

Original: 6 (i) shows the attention weight for Drone 1 to fuse Drone 2’s messages, which is sparse, yet highlights the objects’ positions.
Revised: 6 (i) shows the attention weight for Drone 1 to fuse Drone 2’s messages, which is sparse, yet highlights the objects’ positions.

Original: Fig.
Revised: Fig.

Original: 6 (e) and (j) compares the detection results before and after the collaboration with Drone 2.
Revised: 6 (e) and (j) compares the detection results before and after the collaboration with Drone 2.

Original: We see that the proposed spatial confidence map contributes to spatially sparse, yet perceptually critical message, which effectively helps Drone 1 detect occluded objects.
Revised: We see that the proposed spatial confidence map contributes to spatially sparse, yet perceptually critical message, which effectively helps Drone 1 detect occluded objects.

Original: Visualization of detection results.
Revised: Visualization of detection results.

Original: Fig.
Revised: Fig.

Original: 7 shows that compared to No Collaboration , When2com and DiscoNet , Where2comm is able to achieves more complete and accurate detection results.
Revised: 7 shows that compared to No Collaboration , When2com and DiscoNet , Where2comm is able to achieves more complete and accurate detection results.

Original: The reason is that When2com employs a scalar to denote the agent-to-agent attention, which cannot distinguish which spatial area is more informative; DiscoNet employs a MLP-based fusion weight learning, which cannot well capture the complex collaboration attention; while Where2comm can zoom in to critical spatial areas in a cell-level resolution and leverage the spatial confidence map and sensing distances as priors to achieve more comprehensive fusion.
Revised: The reason is that When2com employs a scalar to denote the agent-to-agent attention, which cannot distinguish which spatial area is more informative; DiscoNet employs a MLP-based fusion weight learning, which cannot well capture the complex collaboration attention; while Where2comm can zoom in to critical spatial areas in a cell-level resolution and leverage the spatial confidence map and sensing distances as priors to achieve more comprehensive fusion.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 5.4 Ablation studies
Revised: 5.4 Ablation studies

Original: Effect of Gaussian filter in perceptually critical area selection.
Revised: Effect of Gaussian filter in perceptually critical area selection.

Original: Fig.
Revised: Fig.

Original: 8 compares two versions of the selection matrix (2) with and without Gaussian filter.
Revised: 8 compares two versions of the selection matrix (2) with and without Gaussian filter.

Original: We see that applying Gaussian filter improves the overall performance.
Revised: We see that applying Gaussian filter improves the overall performance.

Original: The reason is that: i) Gaussian filter could help filter out the outliers in the input map, selecting more robust critical regions; ii) it considers the context, benefiting the independent feature selection at each certain location by providing more information.
Revised: The reason is that: i) Gaussian filter could help filter out the outliers in the input map, selecting more robust critical regions; ii) it considers the context, benefiting the independent feature selection at each certain location by providing more information.

Original: Effect of components in spatial confidence-aware message fusion.
Revised: Effect of components in spatial confidence-aware message fusion.

Original: Tab. 2 assesses the effectiveness of the proposed fusion with two priors.
Revised: Tab. 2 assesses the effectiveness of the proposed fusion with two priors.

Original: We see that: i) per-location multi-head attention (MHA) outperforms the vanilla attention by 10.84% on OPV2V on AP@0.50, because MHA leverages information from multiple heads, better capturing cross-agent attention; and ii)
Revised: We see that: i) per-location multi-head attention (MHA) outperforms the vanilla attention by 10.84% on OPV2V on AP@0.50, because MHA leverages information from multiple heads, better capturing cross-agent attention; and ii)

Original: As two informative priors, both sensing position encoding (SPE) and spatial confidence map (SCM) can consistently improve the performance.
Revised: As two informative priors, both sensing position encoding (SPE) and spatial confidence map (SCM) can consistently improve the performance.

Original: Especially, the version with all three designs improves the detection performance by 22.06% on OPV2V on AP@0.50.
Revised: Especially, the version with all three designs improves the detection performance by 22.06% on OPV2V on AP@0.50.

Original:  Conclusion and limitation
Revised:  Conclusion and limitation

Original: We propose Where2comm , a novel communication-efficient collaborative perception framework.
Revised: We propose Where2comm , a novel communication-efficient collaborative perception framework.

Original: The core idea is to exploit a spatial confidence map at each agent to promote pragmatic compression, assisting agents to decide what to communicate with whom, and whose information to aggregate.
Revised: The core idea is to exploit a spatial confidence map at each agent to promote pragmatic compression, assisting agents to decide what to communicate with whom, and whose information to aggregate.

Original: Each agent offers spatially sparse, yet perceptually critical features to support other agents; meanwhile, requests complementary information from others in multi-round communication.
Revised: Each agent offers spatially sparse, yet perceptually critical features to support other agents; meanwhile, requests complementary information from others in multi-round communication.

Original: Comprehensive experiments covering multi-type agents and multi-modality inputs show that Where2comm achieves far superior trade-off between perception performance and communication bandwidth.
Revised: Comprehensive experiments covering multi-type agents and multi-modality inputs show that Where2comm achieves far superior trade-off between perception performance and communication bandwidth.

Original: Limitation and future work.
Revised: Limitation and future work.

Original: The current work focuses on perceptually critical spatial areas.
Revised: The current work focuses on perceptually critical spatial areas.

Original: In future, we plan to expand a similar idea to the temporal dimension and determine critical time stamps.
Revised: In future, we plan to expand a similar idea to the temporal dimension and determine critical time stamps.

Original: More cost will be reduced by exploring when to communicate.
Revised: More cost will be reduced by exploring when to communicate.

Original: We also expect that more methods on pragmatic compression and emergent communication could be applied to collaborative perception.
Revised: We also expect that more methods on pragmatic compression and emergent communication could be applied to collaborative perception.

Original: Acknowledgment.
Revised: Acknowledgment.

Original: This research is partially supported by the National Key R&D Program of China under Grant 2021ZD0112801, National Natural Science Foundation of China under Grant 62171276, the Science and Technology Commission of Shanghai Municipal under Grant 21511100900, CCFDiDi GAIA Research Collaboration Plan 202112 and CALT Grant 2021-01.Appendix 7.1 Highlights of our contribution To sum up, our contributions are: • We propose a novel fine-grained spatial-aware communication strategy, where each agent can decide where to communicate and pack messages only related to the most perceptually critical spatial areas.
Revised: This research is partially supported by the National Key R&D Program of China under Grant 2021ZD0112801, National Natural Science Foundation of China under Grant 62171276, the Science and Technology Commission of Shanghai Municipal under Grant 21511100900, CCFDiDi GAIA Research Collaboration Plan 202112 and CALT Grant 2021-01.

Original: This strategy not only enables more precise support for other agents, but also more targeted request from other agents in multi-round communication.

Original: • We propose Where2comm , a novel collaborative perception framework based on the spatial-aware communication strategy.

Original: With the guidance of the proposed spatial confidence map, Where2comm leverages novel message packing and communication graph learning to achieve lower communication bandwidth, and adopts confidence-aware multi-head attention to reach better perception performance.

Original: • We conduct extensive experiments to validate Where2comm achieves state-of-the-art performancebandwidth trade-off on multiple challenging real/simulated datasets across views and modalities.

Original: 7.2 Detailed information about the system pipeline

Original: Alg.

Original: 1 presents the pipeline of our multi-round spatial confidence-aware collaborative perception system.

Original: 7.3 Detailed information about the optimization problem of collaborative perception

Original: The constrained optimization in Sec.3 is the mathematical formation of collaborative perception.

Original: It is hard to obtain the global optimum due to hard constrains and non-differentialability of binary variables.

Original: Therefore, the proposed Where2comm essentially introduces an auxiliary variable and decomposes the original problem into two sub-optimization problems, each one of which is easy to solve.

Original: To understand the details, let us consider a setting of fixed communication bandwidth and communication round, K = 1 , B =

Original: [ B 1 ] .

Original: Then, the optimization is

Original: [Equation]

Original: Since there is only one round, we do not consider the request map, then, the message sent from the i th agent to the j th agent is P i → j = Z i → j = M i

Original: → j ⊙ F i , whose spatial sparsity is determined by the binary selection mask M i → j .

Original: Note that M i

Original: → j determines where to communicate, and is the key of the proposed Where2comm.

Original: Then, the original optimization is equivalent to max θ,

Original: [Equation]

Original: where F i can attribute to the network Φ θ ( · ) and input data X i and b 1 =

Original: B 1 /D with D the channel number of F i .

Original: Due to the binary constrains, it is hard to optimize (4) directly.

Original: Instead, we decompose (4) into two sub-optimization problems and optimize the binary selection matrix M i →

Original: j and the network parameters θ once at a time: i) obtain a feasible binary selection matrix M i → j by optimizing a proxy constrained problem; ii) given the feasible binary selection matrix M i

Original: → j , optimize the perception network parameter θ .

Original: The constraint is satisfied in i) and the perception goal is achieved in ii).

Original: Specifically, two sub-optimization problems are

Original: • Obtain a feasible binary selection matrix M i → j .

Original: This essentially optimizes where to allocate the communication bandwidth.

Original: Intuitively, the spatial confidence reflects the perceptually critical level, so that those spatial regions with higher spatial confidence will provide more critical information to help the partners and should have a higher priority be selected.

Original: Following this spirit, we consider a proxy constrained problem as follows,

Original: [Equation]

Original: where C i is the spatial confidence map.

Original: Note that i) even this optimization problem has hard constraints and non-differentialability of binary variables, it has an analytical solution that naturally satisfies all the constraints in (4); and ii) even we cannot solve the original objective, this proxy objective still carries the similar idea to promote better, yet more compact perception.

Original: This solution is obtained by selecting those spatial regions whose corresponding elements in M rank top-

Original: b 1 .

Original: The detailed steps of selection function are: i

Original: ) arrange

Original: the elements

Original: in the

Original: input

Original: matrix in descending order; ii) given the communication budget constrain, decide the total number ( b 1 ) of communication regions; iii) set the spatial regions of M , where elements rank in top- b 1 as the 1 and 0 verses.

Original: • Given the feasible binary selection matrix, optimize the network parameter θ .

Original: This essentially optimizes the perception performance.

Original: The sub-problem is

Original: [Equation]

Original: This can be solved by standard supervised learning .

Original: For example, the perception evaluation metric g ( · ) can be evaluated by the detection loss calculated between detections and the ground-truth and the

Original: [Figure]

Original: detection loss is optimized with an Adam optimizer.

Original: We thus get the optimized perception network parameter θ .

Original: Note that this sub-problem does not involve any constraints and is thus easy to optimize.

Original: 7.4 Detailed information about the module design

Original: Observation encoder.

Original: Here we elaborate on the warping functions for the monocular camera, where the depth is unknown and estimated.

Original: Instead of directly projecting 2D features to flat ground space, we first lift them to 3D voxel space and then collapse them to the BEV.

Original: This design considers all the possible depths/altitudes, introducing flexibility in the projection, and mitigating the distortion effect caused by information loss in imaging.

Original: The detailed steps are: 1) Categorical Depth Distribution

Original: Network (CaDDN [31]), which is a recent and effective method to warp image feature to BEV feature, is applied to estimate the depth distribution for each image feature point.

Original: Each feature point is wrapped from the 2D image space to the 3D physic space according to the known camera parameters.

Original: The 3D voxel features are flattened to BEV features.

Original: Briefly, the warping function is unfolded as follows: for each image feature point locates at ( u, v ) , given the estimated categorical depth d i , and the known camera projection matrix P ∈ R 3 × 4 , 3D physical space coordinates [ x, y, z ] T is calculated conditioned on the image feature coordinates [ u, v, d i ] T based on the projection function: [ u, v, d i ] T = P · [

Original: x, y, z,

Original: T

Original: Spatial confidence-aware message packing.

Original: Fig.

Original: 9 presents the detail about the spatial confidenceaware message packing module.

Original: For the message from agent i to agent j at k th communication round, the module takes the spatial confidence map C ( k ) i of agent

Original: i and the request map R ( k − 1) j of agent j as input, and outputs the message P ( k ) i

Original: → j including the masked feature map Z ( k ) i

Original: → j and the request map of agent i .

Original: Spatial confidence-aware communication graph construction.

Original: Fig.

Original: 10 presents the comparisons on the communication graph with previous works.

Original: Fully connected versus agent-level partially connected versus ours spatial-decouple partially connected communication.

Original: Fully connected communication results in a large amount of bandwidth usage, growing on the order of O ( N 2 ) , where N is the number of agents in a network.

Original: Agent-level partially connected communication prune irrelevant connections between agents while may erroneously sever the information connection.

Original: Spatial-decouple partially connected communication could further flexibly prune irrelevant connections per-location and can substantially reduce the overall network complexity.

Original: Spatial confidence-aware message fusion.

Original: Fig.

Original: 11 presents the detail about the spatial confidenceaware message fusion module.

Original: Given the received messages {P ( k ) j → i , j ∈ N i } , each agent i attentively augments the features with the received messages at each location.

Original: And the request map R ( k ) j in the received message is firstly decoded to the confidence map C ( k ) j via a point-wise minus.

Original: Then the per-location multi-head attention are separately operated at each location in parallel, it takes the features and the corresponding confidence scores as input, and outputs the augmented features.

Original: Sensor positional encoding.

Original: Sensor positional encoding is conditioned on the physical distance between the known sensor coordinates and each BEV gird’s coordinate in the 3D physic space.

Original: It is introduced to provide spatial prior, as the smaller the sensing distance is, the clear the observation would be.

Original: Mathematically, similar to the position encoding in [36], our sensor positional encoding is given by SPE ( dis, 2 p )

Original: =

Original: sin ( dis/ 10000 2 p/D ) , SPE ( dis, 2 p +1)

Original: = cos ( dis/ 10000 2 p/D ) where dis is the physical distance, p is the dimension, D is the total channel dimension of the BEV feature map, sin and cos denote the sine and cosine functions.

Original: 7.5 Detailed information about experimental settings

Original: Implementation details.

Original: For camera-only 3D object detection task on OPV2V, we implement the detector following CADDN [31].

Original: The model is trained 100 epoch with initial learning rate of 1 e- 3 , and decay by 0.1 at epoch 80.

Original: For LiDAR-based 3D object detection task, our detector follows

Original: MotionNet [33].

Original: We train 120 epoch with learning rate 1 e- 3 .

Original: For the camera-only 3D object detection task on CoPerception-UAVs, our detector follows the CenterNet [28] with DLA-34 [37] backbone.

Original: The model is trained 140 epoch with learning rate 5 e- 4 .

Original: [Table]

Original: [Table]

Original: Inference strategy in multi-round setting.

Original: For the single-round communication, all the communication budget are used in this broadcast communication round.

Original: For the two-round communication, a small bandwidth (about 20%) is allocated to activate the collaboration; for the next round, the remained relatively large (about 80%) bandwidth is allocated to transmit the targeted information to meet agents’ request.

Original: For more than two rounds communication setting, we strategically allocate communication budget across multiple communication rounds.

Original: For the initial broadcast round, a small bandwidth (about 20%) is allocated to activate the collaboration; for the next round, a relatively large (about 60%) bandwidth is allocated to transmit the targeted information to meet agents’ request; then, the bandwidth is gradually reduced, accounting for the communication degradation with the increasing rounds.

Original: Communication volume.

Original: Our communication volume is the same as DiscoNet [2], the only difference is that our log base is 2 , while it is 10 , so our number is about 3 .

Original: 32 times theirs.

Original: The base 2 is chosen to align with the metric bit/byte, this is, communication volume counts the message size by byte in log scale with base 2 .

Original: Mathematically for the selected sparse feature map Z ( k ) i

Original: → j =

Original: M (

Original: [Equation]

Original: i →

Original: [Equation]

Original: ⊙ F ( k ) i

Original: [Equation]

Original: , the communication volume is

Original: [Equation]

Original: (5) where | · | denotes the L0 norm counting the non-zero elements in the binary selection matrix, this is, the total spatial girds need to be transmitted, and for each feature point D denotes the channel dimension, 32 is multiplied as float32 data type is used to represent each number, 8 is divided as the metric byte is used.

Original: 7.6 Benchmarks

Original: We conduct extensive experiments on all the available collaborative perception benchmarks.

Original: Tab.presents the overall performance on the four datasets, CoPerception-UAVs, OPV2V [10], V2X-

Original: [Figure]

Original: [Figure]

Original: detection, respectively.

Original: The objects occluded by a tall building can be detected through transmitting spatially sparse, yet perceptually critical message.

Original: Sim1.0 [2] and DAIR-V2X [11].

Original: And we further benchmark the updated V2X-Sim2.0 [9] in Fig.and Tab. 4.

Original: For this LiDAR-based 3D object detection task, our detector follows PointPillar [35].

Original: We see that where2comm consistently achieves significant improvements over previous methods on all the benchmarks.

Original: 7.7 Visualization Visualization of collaboration in OPV2V and V2X-Sim.

Original: Fig. 13 and Fig.

Original: 14 illustrates how Where2comm is empowered by the proposed spatial confidence map on OPV2V and V2XSim dataset.

Original: In the scene, with Vehicle 2’s help, Vehicle 1 is able to detect the missed objects in the

Original: [Figure]

Original: [Figure]

Original: single view.

Original: Fig.

Original: 13 (a-d) shows Vehicle 1’s spatial confidence map, binary selection matrix, ego attention weight and the detection results by its own observation.

Original: Fig.

Original: 13 (e-f) shows Vehicle 2’s message sent to Drone 1, including the request map (opposite of confidence map) and the sparse feature map, achieving efficient communication.

Original: Fig.

Original: 13 (g) shows the attention weight for Vehicleto fuse Vehicle 2’s messages, which is sparse, yet highlights the objects’ positions.

Original: Fig.

Original: 13 (d) and (h) compares the detection results before and after the collaboration with Vehicle 2.

Original: We see that the proposed spatial confidence map contributes to spatially sparse, yet perceptually critical message, which effectively helps Vehicle 1 detect occluded objects.

Original: Visualization of detection results.

Original: Fig.

Original: 15 shows that Where2comm qualitatively outperforms the state-of-the-art methods in CoPerception-UAVs, V2X-Sim and OPV2V datasets.

Original: 7.8 Ablation on bandwidth allocation

Original: Fig.

Original: 16 shows the bandwidth allocation ablation study in multi-round communication setting.

Original: We see that allocating more bandwidth in the second and subsequent communication rounds achieves a better performance-bandwidth trade-off than allocating all bandwidth in the initial communication round, and the gain is stable for different bandwidth allocation strategies.

Original: The reason is that multi-round communication employs a request map in the second and subsequent communication rounds to denote the spatial area where each agent needs more information, which enables more targeted and efficient communication.

Original: 7.9 Discussion on the realistic limitations

Original: There are many challenges in a collaborative perception system.

Original: In this work, we focus on the biggest challenge in current collaborative perception systems; that is, the trade-off between communication bandwidth and perception performance.

Original: This challenge has been actively addressed in previous works [13, 12, 1, 2].

Original: Because collaborative perception is enabled and also severely limited by the communication capacity, which is critically reflected in the highly dynamic and limited bandwidth in real-world communication systems.

Original: Where2comm flexibly adapts to various communication bandwidths, achieving superior performance-bandwidth trade-off.

Original: Here we further discuss other realistic limitations, assess the robustness of our system and future improvements to be done.

Original: • For other realistic communication issues such as latency , where2comm communicates strategically when necessary, rather than all the time or everywhere, to reduce the possibility of encountering communication problems.

Original: In addition, a prediction module could be integrated to estimate the missed or delayed frames according to the historically received frames.

Original: And by focusing on the informative spatial regions, where2comm can reduce the estimation difficulty.

Original: • For the time synchronisation issue, by using the powerful transformer architecture-based fusion module, where2comm can attentively augment the features with the received asynchronous features from other agents.

Original: In addition, where2comm can introduce positional encoding conditioned on delay time and easily extend to global multi-head attention to further reduce the effects of time synchronization.

Original: • For the noisy localization issue, where2comm exchanges the intermediate features among agents, which has a relatively low spatial resolution, thus is relatively robust to noisy pose.

Original: In addition, where2comm can easily extend to a deformable transformer architecture like [38] to further alleviate the feature distortion caused by the noisy localization.

Original: • For the attack issue, by focusing on specific spatial regions and attentively fusing the received features from other agents, where2comm is relatively less likely to be attacked.

Original: • For the data availability , where2comm works on both RGB and point cloud modalities, and is sensor friendly, so it can be deployed on cheap camera sensors and lidar sensors.

Original: 7.10 CoPerception-UAVs dataset details

Original: CoPerception-UAVs dataset collects data from drones, see Fig.

Original: As the rapid development of an unmanned aerial vehicle (UAV) significantly enhances human’s ability to perceive the world from an aerial perspective.

Original: UAV-based systems have been widely used in numerous applications, including search and rescue, security and surveillance, photography, geographical mapping, as well as traffic monitoring.

Original: Through collaboration, UAV swarm can further distribute multiple tasks and achieve higher flexibility, stronger robustness, and a larger perception range, leading to significant advantages in harsh and complex environments.

Original: Unfortunately, collaborative perception mainly focuses on the vehicles and ignores the UAV literature.

Original: To provide more diverse views and challenging benchmark for the collaborative perception community , here we present the first comprehensive large-scale collaborative perception dataset for UAV swarm so far.

Original: Since building a dataset in the real world is too expensive and laborious, in this initial version, we consider a virtual dataset based on the co-simulation of AirSim [34] and Carla [30], where

Original: AirSim simulates the UAV swarms and Carla simulates the complex background scenes and dynamic foreground objects.

Original: In the simulation, we consider that the UAV swarm is flying over diverse simulated scenes at various altitudes.

Original: Each UAV has a sensing device to collect RGB images, a computation device to perceive the environment with a perception model, and a communication device to transmit perception information among UAVs.

Original: In this setting, the UAV swarm is able to achieve 2D/3D object detection, pixel-wise or bird’s-eye-view (BEV) semantic segmentation in a collaborative manner.

Original: Our dataset consists of 131.9k synchronous images collected fromcoordinated UAVs flying at 3 altitudes over 3 simulated towns with 2 types of swarm formation.

Original: To enable the model training and testing, each image is fully annotated with the pixel-wise semantic segmentation labels, 2D bounding boxes of vehicles, as well as 3D bounding boxes on the ground and the semantic mask from BEV view.

Original: This benchmark can enable the evaluation of collaborative perception methods on the important perception tasks: 2D/3D/BEV object detection and semantic segmentation.

Original: The dataset details as unfolded as follows.

Original: Data collection.

Original: Our proposed dataset is collected by the co-simulation of CARLA [30] and

Original: AirSim[34] (both under MIT license).

Original: We use CARLA to generate complex simulation scenes and traffic flow; and use AirSim to simulate UAV swarm flying in the scene.

Original: The flight route of UAVs is controlled by AirSim and sample data are collected randomly at about 4-second intervals.

Original: Map creation.

Original: The simulation scenes, including the road layout, static objects, and traffic flow, are created based on CARLA [30] simulation.

Original: We take three open-source maps ( town4 to town6 ) provided by CARLA as the basic road layouts, which are the three largest maps in scale.

Original: To increase the complexity and diversity of the scenes and make the perception tasks more challenging, we customize the original maps, adding and replacing various buildings, vegetation, roadblocks, barriers, and other static objects with various assets provided by CARLA.

Original: Traffic flow creation.

Original: Moving vehicles in the scene are managed through CARLA.

Original: Hundreds of vehicles are spawned in each scene by script spawn_npc.py provided by CARLA.

Original: The initial location and motion trajectory of each vehicle is determined by the map’s road layout.

Original: Sensor setup.

Original: Each UAV is equipped with 5 RGB cameras in 5 directions and 5 semantic cameras collecting semantic ground truth for RGB cameras.

Original: The cameras include a bird’s eye view camera and four cameras facing forward, backward, right, and left with a pitch degree of − 45 ◦ .

Original: Each camera has an FoV of 90 ◦ and the resolution is 800 × 450 .

Original: On each UAV, all the cameras are fixed and their internal relative position and rotation degree are invariable.

Original: The translation (x, y, z) and rotation (w, x, y, z in quaternion) of each camera in both global and ego coordinates are recorded during data collection.

Original: With such a sensor setting, a UAV at the height of 40 m can mostly cover an area of 200 m × 200 m .

Original: Formation flying.

Original: The UAV swarm moves and executes tasks in the three-dimensional space, where the situation could be much more complex than those of vehicles or roadside units.

Original: In our proposed dataset, we take into consideration two main factors that may affect the perception and collaboration patterns of UAV swarms: flight formation and altitude.

Original: Each UAV swarm consists of 5 UAVs.

Original: We arrange two types of formation modes for a UAV swarm: discipline mode, where all 5 UAVs keeps a consistent and relatively static array, and dynamic mode, where each UAV navigates independently in the scene; see Fig.

Original: The former simulates the situation where the swarm of UAVs is executing a same specific task such as exploring an unknown area, search and rescue; while the latter simulates the monitoring and patrolling tasks in the city.

Original: Fully-annotated data are provided in our proposed dataset, including synchronous images with pixel-wise semantic labels, 2D & 3D bounding boxes of vehicles, and BEV semantic map; see 19.

Original: Camera data.

Original: We collect synchronous images from all cameras on 5 UAVs, which is 25 images in a sample.

Original: Camera intrinsics and extrinsics in global coordinate are provided to support coordinate transformation across various UAVs.

Original: In total, 123.8K images are collected for the discipline swarm mode and 8.1K for the dynamic swarm mode.

Original: Bounding boxes.

Original: During data collection, 3D bounding boxes of vehicles are recorded at the same moment with images, including location (x, y, z), rotation (w, x, y, z in quaternion) in the global coordinate and their length, width and height.

Original: The location (x, y, z) is the center of the bounding box.

Original: Then we provide 2D bounding boxes by projecting the 3D bounding boxes to the image perspective plane of each camera, resulting in 1.94M 3D bounding boxes and 3.6M 2D bounding boxes in total.

Original: Data usage.

Original: In total, CoPerception-UAVs has 131.9K aerial images and 1.94M 3D boxes.

Original: We randomly split the samples into train/validation/test, resulting 91,175/19,500/20,250 images, and 1,316,536/303,888/319,576 3D bounding boxes.

Original: The dataset is organized in a similar way with the widely-used autonomous driving dataset, nuScenes [39]; so it can be used directly with the well-established nuScenes-devkit.