Original: Learning Control Admissibility Models with Graph Neural Networks for Multi-Agent Navigation
Revised: Learning Control Admissibility Models with Graph Neural Networks for Multi-Agent Navigation

Original: Abstract: Deep reinforcement learning in continuous domains focuses on learning control policies that map states to distributions over actions that ideally concentrate on the optimal choices in each step.
Revised: Abstract: Deep reinforcement learning in continuous domains focuses on learning control policies that map states to distributions over actions that ideally concentrate on the optimal choices in each step.

Original: In multi-agent navigation problems, the optimal actions depend heavily on the agents’ density.
Revised: In multi-agent navigation problems, the optimal actions depend heavily on the agents’ density.

Original: Their interaction patterns grow exponentially with respect to such density, making it hard for learning-based methods to generalize.
Revised: Their interaction patterns grow exponentially with respect to such density, making it hard for learning-based methods to generalize.

Original: We propose to switch the learning objectives from predicting the optimal actions to predicting sets of admissible actions, which we call control admissibility models (CAMs), such that they can be easily composed and used for online inference for an arbitrary number of agents.
Revised: We propose to switch the learning objectives from predicting the optimal actions to predicting sets of admissible actions, which we call control admissibility models (CAMs), such that they can be easily composed and used for online inference for an arbitrary number of agents.

Original: We design CAMs using graph neural networks and develop training methods that optimize the CAMs in the standard model-free setting, with the additional beneﬁt of eliminating the need for reward engineering typically required to balance collision avoidance and goal-reaching requirements.
Revised: We design CAMs using graph neural networks and develop training methods that optimize the CAMs in the standard model-free setting, with the additional beneﬁt of eliminating the need for reward engineering typically required to balance collision avoidance and goal-reaching requirements.

Original: We evaluate the proposed approach in multiagent navigation environments.
Revised: We evaluate the proposed approach in multiagent navigation environments.

Original: We show that the CAM models can be trained in environments with only a few agents and be easily composed for deployment in dense environments with hundreds of agents, achieving better performance than state-of-the-art methods.
Revised: We show that the CAM models can be trained in environments with only a few agents and be easily composed for deployment in dense environments with hundreds of agents, achieving better performance than state-of-the-art methods.

Original:  Introduction
Revised:  Introduction

Original: Multi-agent navigation is a longstanding problem with a wide range of practical applications such as in manufacturing [1, 2], transportation [3] and surveillance [4].
Revised: Multi-agent navigation is a longstanding problem with a wide range of practical applications such as in manufacturing [1, 2], transportation [3] and surveillance [4].

Original: The goal is to control many agents to all achieve their goals while avoiding collisions and deadlocks.
Revised: The goal is to control many agents to all achieve their goals while avoiding collisions and deadlocks.

Original: Such problems are multi-objective in nature and the control methods depend heavily on the density of the agents, with the computational complexity growing exponentially in such density [5, 6, 7, 8].
Revised: Such problems are multi-objective in nature and the control methods depend heavily on the density of the agents, with the computational complexity growing exponentially in such density [5, 6, 7, 8].

Original: Recently, reinforcement learning approaches have shown promising results on multi-agent navigation problems [9, 10, 11, 12, 13], but there are still two well-known sources of the difﬁculty.
Revised: Recently, reinforcement learning approaches have shown promising results on multi-agent navigation problems [9, 10, 11, 12, 13], but there are still two well-known sources of the difﬁculty.

Original: First, balancing safety and goal-reaching requirements often lead to ad hoc reward engineering that is highly dependent on the environments and the density of the agents [14, 15].
Revised: First, balancing safety and goal-reaching requirements often lead to ad hoc reward engineering that is highly dependent on the environments and the density of the agents [14, 15].

Original: Second, it is shown that such neural control policies leaned on RL approaches are hard to generalize when the agent density in the environment changes, which leads to distribution drifts.
Revised: Second, it is shown that such neural control policies leaned on RL approaches are hard to generalize when the agent density in the environment changes, which leads to distribution drifts.

Original: The core difﬁculty for generalizability is that the RL approaches are designed to capture the optimal actions speciﬁc to the training environments and rewards.
Revised: The core difﬁculty for generalizability is that the RL approaches are designed to capture the optimal actions speciﬁc to the training environments and rewards.

Original: Nevertheless, such optimal actions may change rapidly when the density and interaction patterns change during deployment.
Revised: Nevertheless, such optimal actions may change rapidly when the density and interaction patterns change during deployment.

Original: The unnecessary optimization of control policies in the training environments makes it fundamentally hard to transfer the learned results to new environments and achieve compositionality in the deployment to varying numbers of agents.
Revised: The unnecessary optimization of control policies in the training environments makes it fundamentally hard to transfer the learned results to new environments and achieve compositionality in the deployment to varying numbers of agents.

Original: We propose a new learning-based approach to multi-agent navigation that shifts the focus from learning the optimal control policy to a set-theoretic representation of admissible control policies to achieve compositional inference.
Revised: We propose a new learning-based approach to multi-agent navigation that shifts the focus from learning the optimal control policy to a set-theoretic representation of admissible control policies to achieve compositional inference.

Original: By decoupling the multi-agent navigation tasks, we avoid the ad-hoc reward engineering using a simple goal-reaching preference function and a learnable set-theoretic component for collision avoidance.
Revised: By decoupling the multi-agent navigation tasks, we avoid the ad-hoc reward engineering using a simple goal-reaching preference function and a learnable set-theoretic component for collision avoidance.

Original: We use graph neural networks (GNNs) to represent the set of admissible control actions at each state as Control Admissibility Models (CAM), which are trained with a small number of agents during training with sparse rewards.
Revised: We use graph neural networks (GNNs) to represent the set of admissible control actions at each state as Control Admissibility Models (CAM), which are trained with a small number of agents during training with sparse rewards.

Original: In online inference, we compose the CAMs and apply goal-reaching preference functions to infer the speciﬁc action to take for each agent.
Revised: In online inference, we compose the CAMs and apply goal-reaching preference functions to infer the speciﬁc action to take for each agent.

Original: We show that CAMs can be learned purely from the data collected online in the standard RL
Revised: We show that CAMs can be learned purely from the data collected online in the standard RL

Original: [Table]
Revised: [Table]

Original: ++XBZa9wWdZTJCTkl58Qj16RB7kmTtAgjCXkmr+TNQefFeXc+FqMlp8gckz9wPn8AAXGRFg==</latexit> ˙ ↵ <latexit sha1_base64="JkKTFfwisCbSEMGu5M6dyZLQvIs=">AAAB8XicbVBNSwMxEJ2tX7V+VT16CRahgpRdKeqx6MVjBfuB7VKyabYNTbJLkhXL0n/hxYMiXv033vw3pu0etPXBwOO9GWbmBTFn2rjut5NbWV1b38hvFra2d3b3ivsHTR0litAGiXik2gHWlDNJG4YZTtuxolgEnLaC0c3Ubz1SpVkk7804pr7AA8lCRrCx0kM3HrLy0xnCp71iya24M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/OLp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjlp0zGiaGSzBeFCUcmQtP3UZ8pSgwfW4KJYvZWRIZYYWJsSAUbgrf48jJpnle8i0r1rlqqXWdx5OEIjqEMHlxCDW6hDg0gIOEZXuHN0c6L

Revised: ++XBZa9wWdZTJCTkl58Qj16RB7kmTtAgjCXkmr+TNQefFeXc+FqMlp8gckz9wPn8AAXGRFg==</latexit> ˙ ↵ <latexit sha1_base64="JkKTFfwisCbSEMGu5M6dyZLQvIs=">AAAB8XicbVBNSwMxEJ2tX7V+VT16CRahgpRdKeqx6MVjBfuB7VKyabYNTbJLkhXL0n/hxYMiXv033vw3pu0etPXBwOO9GWbmBTFn2rjut5NbWV1b38hvFra2d3b3ivsHTR0litAGiXik2gHWlDNJG4YZTtuxolgEnLaC0c3Ubz1SpVkk7804pr7AA8lCRrCx0kM3HrLy0xnCp71iya24M6Bl4mWkBBnqveJXtx+RRFBpCMdadzw3Nn6KlWGE00mhm2gaYzLCA9qxVGJBtZ/OLp6gE6v0URgpW9Kgmfp7IsVC67EIbKfAZqgXvan4n9dJTHjlp0zGiaGSzBeFCUcmQtP3UZ8pSgwfW4KJYvZWRIZYYWJsSAUbgrf48jJpnle8i0r1rlqqXWdx5OEIjqEMHlxCDW6hDg0gIOEZXuHN0c6L

Original: +58zFtzTjZzCH/gfP4AOJCP+g==</latexit> 
Revised: +58zFtzTjZzCH/gfP4AOJCP+g==</latexit>

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: setting, with no expert or human guidance.
Revised: setting, with no expert or human guidance.

Original: We propose relabelling through backpropagation (Section 2.3) for the CAMs to learn rich information from the transitions even given sparse rewards.
Revised: We propose relabelling through backpropagation (Section 2.3) for the CAMs to learn rich information from the transitions even given sparse rewards.

Original: As shown in Figure 1(d), the learned models of the admissible actions are typically complex and suitable for
Revised: As shown in Figure 1(d), the learned models of the admissible actions are typically complex and suitable for

Original: GNN representations.
Revised: GNN representations.

Original: Moreover, in online inference, we decompose the state graph of the agent into much smaller subgraphs and aggregate the CAMs from them.
Revised: Moreover, in online inference, we decompose the state graph of the agent into much smaller subgraphs and aggregate the CAMs from them.

Original: Such compositionality allows us to train CAMs with only a small number of agents, directly deploy them in much denser environments
Revised: Such compositionality allows us to train CAMs with only a small number of agents, directly deploy them in much denser environments

Original: and achieve generalizable learning and inference.
Revised: and achieve generalizable learning and inference.

Original: We evaluate the proposed methods in various challenging multi-agent environments, including robot arms, cars, and quadcopters.
Revised: We evaluate the proposed methods in various challenging multi-agent environments, including robot arms, cars, and quadcopters.

Original: Experiments show that the CAM generalizes very well.
Revised: Experiments show that the CAM generalizes very well.

Original: While the training environments only have at most 3 agents, the learned CAM models can be directly deployed to hundreds of agents in comparable environments and achieve a very high success rate and low collision rate.
Revised: While the training environments only have at most 3 agents, the learned CAM models can be directly deployed to hundreds of agents in comparable environments and achieve a very high success rate and low collision rate.

Original: We analyze the beneﬁts of the proposed methods compared to state-of-the-art approaches.
Revised: We analyze the beneﬁts of the proposed methods compared to state-of-the-art approaches.

Original: Furthermore, we show a zero-shot transfer to a multi-agent chasing game, demonstrating that the trained CAM generalizes to other tasks that are not limited to navigation.
Revised: Furthermore, we show a zero-shot transfer to a multi-agent chasing game, demonstrating that the trained CAM generalizes to other tasks that are not limited to navigation.

Original: Related Work.
Revised: Related Work.

Original: Multi-Agent Navigation.
Revised: Multi-Agent Navigation.

Original: Classical Multi-Agent Path Finding is one of the most popular deﬁnitions of the multi-agent navigation task, where the state space lies in a pre-constructed graph shared among agents [16].
Revised: Classical Multi-Agent Path Finding is one of the most popular deﬁnitions of the multi-agent navigation task, where the state space lies in a pre-constructed graph shared among agents [16].

Original: Under this setting, heuristic-based methods have been used to generate high-quality solutions, such as Priority-Based Search and Conﬂict-Based Search [17, 18].
Revised: Under this setting, heuristic-based methods have been used to generate high-quality solutions, such as Priority-Based Search and Conﬂict-Based Search [17, 18].

Original: Learning-based methods have also been proposed to accelerate the planning time, using CNN and GNN [19, 20, 21].
Revised: Learning-based methods have also been proposed to accelerate the planning time, using CNN and GNN [19, 20, 21].

Original: In this work, we focus more on the continuous state space with reactive planning.
Revised: In this work, we focus more on the continuous state space with reactive planning.

Original: Traditional methods, including Optimal Reciprocal Collision Avoidance [22], Buffered
Revised: Traditional methods, including Optimal Reciprocal Collision Avoidance [22], Buffered

Original: Voronoi Cells [23], and Artiﬁcial Potential Functions [24, 25, 26, 27], are designed manually to address speciﬁc structural properties.
Revised: Voronoi Cells [23], and Artiﬁcial Potential Functions [24, 25, 26, 27], are designed manually to address speciﬁc structural properties.

Original: On the other hand, learning-based multi-agent methods using reinforcement learning and imitation learning [9, 10, 11, 12, 13, 28, 29], may or may not generalize when there is covariant shift for test tasks.
Revised: On the other hand, learning-based multi-agent methods using reinforcement learning and imitation learning [9, 10, 11, 12, 13, 28, 29], may or may not generalize when there is covariant shift for test tasks.

Original: Recently, MACBF [30] incorporates multi-agent safe certiﬁcates [31, 32, 33] into a learning-based framework, which shows the generalization capability while preserving the safety properties.
Revised: Recently, MACBF [30] incorporates multi-agent safe certiﬁcates [31, 32, 33] into a learning-based framework, which shows the generalization capability while preserving the safety properties.

Original: Nevertheless, in [30] the learning of the neural controller requires a reference controller, which may constrain the potential of the neural controller, when the actions dissimilar to the reference actions could yield better rewards and safer results.
Revised: Nevertheless, in [30] the learning of the neural controller requires a reference controller, which may constrain the potential of the neural controller, when the actions dissimilar to the reference actions could yield better rewards and safer results.

Original: Though learned in an online fashion, our work only assumes sparse reward and avoids the reward hacking problem.
Revised: Though learned in an online fashion, our work only assumes sparse reward and avoids the reward hacking problem.

Original: Graph Neural Networks.
Revised: Graph Neural Networks.

Original: Graph Neural Networks (GNN), including Graph Convolutional Networks [34], Message Passing Neural Networks [35], Interaction Networks [36], are deep neural networks that operate on graph-structured data [37].
Revised: Graph Neural Networks (GNN), including Graph Convolutional Networks [34], Message Passing Neural Networks [35], Interaction Networks [36], are deep neural networks that operate on graph-structured data [37].

Original: Graph neural networks are permutation-equivariant to the orders of nodes on graph [38, 39], which is important for homogeneous multi-agent problems since the order among agents should not matter.
Revised: Graph neural networks are permutation-equivariant to the orders of nodes on graph [38, 39], which is important for homogeneous multi-agent problems since the order among agents should not matter.

Original: In robotics, the effectiveness of GNN has been shown on tasks such as path planning [19, 20], motion planning [40, 41], coverage and exploration [42, 43], and SLAM [44].
Revised: In robotics, the effectiveness of GNN has been shown on tasks such as path planning [19, 20], motion planning [40, 41], coverage and exploration [42, 43], and SLAM [44].

Original: In this work, we use GNN as an architecture to implement CAM due to the multiagent navigation problem can be modeled as a graph-based problem in nature.
Revised: In this work, we use GNN as an architecture to implement CAM due to the multiagent navigation problem can be modeled as a graph-based problem in nature.

Original: However, we believe that the proposed CAM could generalize to other inputs that are not graphs.
Revised: However, we believe that the proposed CAM could generalize to other inputs that are not graphs.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /uf5qYmug4yJJDVUkMVHUcqRkSi/Hw2ZosTwqSWYKGazIjLGChNjW6rYErzlk1dJ56LuXdYbD41a87aoowwncArn4MEVNOEeWtAGAhKe4RXeHOO8OO/Ox2K05BQ7x/AHzucPczyRYA==</latexit>
Revised: /uf5qYmug4yJJDVUkMVHUcqRkSi/Hw2ZosTwqSWYKGazIjLGChNjW6rYErzlk1dJ56LuXdYbD41a87aoowwncArn4MEVNOEeWtAGAhKe4RXeHOO8OO/Ox2K05BQ7x/AHzucPczyRYA==</latexit>

Original: A
Revised: A

Original:  Control Admissibility Models
Revised:  Control Admissibility Models

Original: We present Control Admissibility Models (CAMs), a general representation of the feasible control set.
Revised: We present Control Admissibility Models (CAMs), a general representation of the feasible control set.

Original: CAM is task-compositional, robust to reward design, and can be trained with an online pipeline similar to model-free RL.
Revised: CAM is task-compositional, robust to reward design, and can be trained with an online pipeline similar to model-free RL.

Original: With graph decomposition, CAM can plan a high-quality solution for a large number of robots, which could be signiﬁcantly out of the distribution of the training tasks.
Revised: With graph decomposition, CAM can plan a high-quality solution for a large number of robots, which could be signiﬁcantly out of the distribution of the training tasks.

Original: 2.1 Control Admissibility Models (CAMs)
Revised: 2.1 Control Admissibility Models (CAMs)

Original: We deﬁne CAMs as a general representation of the feasible actions.
Revised: We deﬁne CAMs as a general representation of the feasible actions.

Original: In general, CAMs are function approximators that behave as the characteristic functions of such sets.
Revised: In general, CAMs are function approximators that behave as the characteristic functions of such sets.

Original: By evaluating the CAM values we can efﬁciently determine whether an action is feasible.
Revised: By evaluating the CAM values we can efﬁciently determine whether an action is feasible.

Original: Formally,
Revised: Formally,

Original: Deﬁnition 1 (Control Admissibility Models and Admissible Set) .
Revised: Deﬁnition 1 (Control Admissibility Models and Admissible Set) .

Original: Given CAM φ , state x , action space A , the admissible set is deﬁned as ∆( φ, x ) : { a | a ∈ A , φ ( x, a ) ≥ 0 } .
Revised: Given CAM  , state x , action space A , the admissible set is deﬁned as  (  , x ) : { a | a 2 A ,  ( x, a )  0 } .

Original: Compositionality of CAMs.
Revised: Compositionality of CAMs.

Original: An important feature of CAMs is that they can be composed together.
Revised: An important feature of CAMs is that they can be composed together.

Original: Given two CAMs φ 1 and φ 2 , we want to ﬁnd the admissible actions that satisfy both these two CAMs:
Revised: Given two CAMs  1 and  2 , we want to ﬁnd the admissible actions that satisfy both these two CAMs:

Original: [Equation]
Revised: [Equation]

Original: Proposition 1 (Composition) .
Revised: Proposition 1 (Composition) .

Original: If φ 1 and φ 2 are CAMs, then ∆( φ 3 , x ) deﬁnes ∆( φ 1 , x ) ∩ ∆( φ 2 , x ) , given φ 3 = min { φ 1 , φ 2 } .
Revised: If  1 and  2 are CAMs, then  (  3 , x ) deﬁnes  (  1 , x ) \  (  2 , x ) , given  3 = min {  1 ,  2 } .

Original: 2.2 Graph Neural Networks for CAMs
Revised: 2.2 Graph Neural Networks for CAMs

Original: Though CAM can be applied to various single-agent problems (see Section 3.1 as an example), we focus on multi-agent problems in this paper.
Revised: Though CAM can be applied to various single-agent problems (see Section 3.1 as an example), we focus on multi-agent problems in this paper.

Original: At each timestep, the state for each agent is an egocentric graph G = (cid:104) V, E (cid:105) , where V are the vertices and E are the edges.
Revised: At each timestep, the state for each agent is an egocentric graph G = h V, E i , where V are the vertices and E are the edges.

Original: Vertices V is composed of agent vertices V a and static obstacle vertices V o .
Revised: Vertices V is composed of agent vertices V a and static obstacle vertices V o .

Original: Edges E are connecting all the neighbor vertices to the current agent, i.e. E = { ( v ja , v ia ) | a j ∈ E ( a i ) ∪ { ( v jo , v ia ) 
Revised: Edges E are connecting all the neighbor vertices to the current agent, i.e. E = { ( v ja , v ia ) | a j 2 E ( a i ) [ { ( v jo , v ia ) |

Original: | o j ∈ E ( a i ) } , where E denotes the neighbor vertices.
Revised: o j 2 E ( a i ) } , where E denotes the neighbor vertices.

Original: The one-hot vector is used as the features of V , denoting the types of vertices.
Revised: The one-hot vector is used as the features of V , denoting the types of vertices.

Original: For the features of edges E , we use relative positions and the states of the two connected vertices, along with a one-hot vector to indicate the edge types.
Revised: For the features of edges E , we use relative positions and the states of the two connected vertices, along with a one-hot vector to indicate the edge types.

Original: Zero paddings are adopted if necessary [45].
Revised: Zero paddings are adopted if necessary [45].

Original: We provide more details on the representation of graphs in the Appendix.
Revised: We provide more details on the representation of graphs in the Appendix.

Original: The CAM φ has two components: The ﬁrst component is a GNN layer, which transforms the graph G to a hidden state vector h for each agent.
Revised: The CAM  has two components: The ﬁrst component is a GNN layer, which transforms the graph G to a hidden state vector h for each agent.

Original: The second component is a fully-connected layer f , which takes the hidden state h and an action a , and predicts the admissibility score φ ( G, a ) =
Revised: The second component is a fully-connected layer f , which takes the hidden state h and an action a , and predicts the admissibility score  ( G, a ) =

Original: f ( h, a ) .
Revised: f ( h, a ) .

Original: Such a design enables fast computation of the admissibility scores for all the possible state-action pairs.
Revised: Such a design enables fast computation of the admissibility scores for all the possible state-action pairs.

Original: We can merge the computation of the hidden state for all state-action pairs which share the same input graph into one forward passing in the GNN layer.
Revised: We can merge the computation of the hidden state for all state-action pairs which share the same input graph into one forward passing in the GNN layer.

Original: We describe the GNN layer with the following details.
Revised: We describe the GNN layer with the following details.

Original: The vertices and the edges are ﬁrst embedded into latent space as m (0) =
Revised: The vertices and the edges are ﬁrst embedded into latent space as m (0) =

Original: g m ( v ) , n (0) =
Revised: g m ( v ) , n (0) =

Original: g n ( e ) , given fully-connected layers g m , g n .
Revised: g n ( e ) , given fully-connected layers g m , g n .

Original: Taking m, n , the GNN aggregates the local information for each vertex from the neighbors through K GNN layers.
Revised: Taking m, n , the GNN aggregates the local information for each vertex from the neighbors through K GNN layers.

Original: For the k -th layer, it performs the message passing with 2 fully-connected layers f ( k ) m and f ( k ) n :
Revised: For the k -th layer, it performs the message passing with 2 fully-connected layers f ( k ) m and f ( k ) n :

Original: [Equation]
Revised: [Equation]

Original: We assign the hidden state h with the ﬁnal node embedding m ( K ) i , where i corresponds to the current agent.
Revised: We assign the hidden state h with the ﬁnal node embedding m ( K ) i , where i corresponds to the current agent.

Original: We use max as the aggregation operator to gather the local geometric information due to its empirical robustness to achieve the order invariance [30, 39].
Revised: We use max as the aggregation operator to gather the local geometric information due to its empirical robustness to achieve the order invariance [30, 39].

Original: We use the residual connection to update the vertex embedding and edge embedding due to its simplicity and robust performance for deep layers [46, 47].
Revised: We use the residual connection to update the vertex embedding and edge embedding due to its simplicity and robust performance for deep layers [46, 47].

Original: Moreover, each agent’s hidden state h is only updated based on its egocentric graph, which is entirely invariant to the other agents’ hidden states.
Revised: Moreover, each agent’s hidden state h is only updated based on its egocentric graph, which is entirely invariant to the other agents’ hidden states.

Original: This architecture decentralizes the decision-making of the whole swarm.
Revised: This architecture decentralizes the decision-making of the whole swarm.

Original: 2.3 Training CAMs in Reinforcement Learning
Revised: 2.3 Training CAMs in Reinforcement Learning

Original: We train CAMs in the same online setting as model-free reinforcement learning procedures.
Revised: We train CAMs in the same online setting as model-free reinforcement learning procedures.

Original: At each time step, the agent perceives the observation from the environment and takes action.
Revised: At each time step, the agent perceives the observation from the environment and takes action.

Original: The state-action pairs for every transition are labeled as safe or unsafe and then appended to the replay buffer.
Revised: The state-action pairs for every transition are labeled as safe or unsafe and then appended to the replay buffer.

Original: The CAM is updated every time a certain number of transitions are collected.
Revised: The CAM is updated every time a certain number of transitions are collected.

Original: The main differences between our training approach and the standard RL methods are three-fold: (i) instead of using an actor-network, the agent chooses the action among the sampled actions based on the admissibility score generated by CAM.
Revised: The main differences between our training approach and the standard RL methods are three-fold: (i) instead of using an actor-network, the agent chooses the action among the sampled actions based on the admissibility score generated by CAM.

Original: (ii) the label is binary for each state-action pair instead of calculating the cumulative future rewards.
Revised: (ii) the label is binary for each state-action pair instead of calculating the cumulative future rewards.

Original: (iii) the training objective is non-bootstrapped, which avoids the overestimation issue and is more stable.
Revised: (iii) the training objective is non-bootstrapped, which avoids the overestimation issue and is more stable.

Original: We design these three improvements around the underlying structure of the admissibility problem.
Revised: We design these three improvements around the underlying structure of the admissibility problem.

Original: We provide further explanations as follows.
Revised: We provide further explanations as follows.

Original: 1: Input : Preference function ω exploration noise

Original: 2: Initialize CAM

Original: 3: Initialize replay

Original: 4: for episode =

Original: M do 5:for do 6:
Revised: do 5: do 6:

Original: Sample candidate

Original: A 7: {

Original: Select

Original: Execute actions

Original: Label the transition ( t a t s t y by checking whether s t danger set 11:

Original: Store transition ( a through backpropagation 13:

Original: Sparse Reward Setting.
Revised: Sparse Reward Setting.

Original: We model the problem as the standard Markov Decision Process (MDP)
Revised: We model the problem as the standard Markov Decision Process (MDP)

Original: M = ( S , A , T, r, γ ) [48].
Revised: M = ( S , A , T, r,  ) [48].

Original: S is the state space represented as graphs G .
Revised: S is the state space represented as graphs G .

Original: The action space A is continuous.
Revised: The action space A is continuous.

Original: The reward r is sparse: it only returns non-zero values when the agent enters the region to avoid and the region to reach.
Revised: The reward r is sparse: it only returns non-zero values when the agent enters the region to avoid and the region to reach.

Original: These certain regions can also be dynamic, i.e., danger regions centering around other agents.
Revised: These certain regions can also be dynamic, i.e., danger regions centering around other agents.

Original: Preference Function.
Revised: Preference Function.

Original: Given current state G and goal state T , we assume a preference function over actions, i.e., ω ( G, T , · ) , is given.
Revised: Given current state G and goal state T , we assume a preference function over actions, i.e., ! ( G, T , · ) , is given.

Original: The preference function solely focuses on the goal-reaching.
Revised: The preference function solely focuses on the goal-reaching.

Original: For instance, it can simply be the L2 distance from the next state to the goal, since CAM will achieve the obstacle avoidance.
Revised: For instance, it can simply be the L2 distance from the next state to the goal, since CAM will achieve the obstacle avoidance.

Original: In the experiments, we try to choose the preference functions as simple as possible.
Revised: In the experiments, we try to choose the preference functions as simple as possible.

Original: We refer readers to the Appendix for more details.
Revised: We refer readers to the Appendix for more details.

Original: Action Selection via Sampling.
Revised: Action Selection via Sampling.

Original: At every timestep, the agent chooses an action among a batch of actions sampled from the action space.
Revised: At every timestep, the agent chooses an action among a batch of actions sampled from the action space.

Original: Given a state x , the admissibility value every action a is computed with φ ( x, a ) .
Revised: Given a state x , the admissibility value every action a is computed with  ( x, a ) .

Original: If there exist one or multiple actions that satisfy { a | φ ( x, a ) ≥ 0 } , given the goal state
Revised: If there exist one or multiple actions that satisfy { a |  ( x, a )  0 } , given the goal state

Original: T and a preference ω ( x, T , · ) over actions, we choose the action that has the highest preference score in the admissible set.
Revised: T and a preference ! ( x, T , · ) over actions, we choose the action that has the highest preference score in the admissible set.

Original: Namely, arg max a { ω ( x, T , a ) | φ ( x, a ) ≥ 0 } .
Revised: Namely, arg max a { ! ( x, T , a ) |  ( x, a )  0 } .

Original: If there does not exist any admissible action, in order to fulﬁll the admissibility property maximally, the agent chooses the action that has the highest admissibility score, arg max a { φ ( x, a ) } + N , where N is an exploration noise.
Revised: If there does not exist any admissible action, in order to fulﬁll the admissibility property maximally, the agent chooses the action that has the highest admissibility score, arg max a {  ( x, a ) } + N , where N is an exploration noise.

Original: We model the exploration noise under the uniform distribution in our experiment.
Revised: We model the exploration noise under the uniform distribution in our experiment.

Original: Training Loss of CAM.
Revised: Training Loss of CAM.

Original: Given admissible transitions R 0 ⊆ S × A and inadmissible transitionsR d
Revised: Given admissible transitions R 0 ✓ S ⇥ A and inadmissible transitions

Original: ⊆ S × A , we train the CAM with the following objective:
Revised: R d ✓ S ⇥ A , we train the CAM with the following objective:

Original: [Equation]
Revised: [Equation]

Original: where γ 1 , γ 2 , γ 3 ≥ 0 are the coefﬁcients to enlarge the margin of the learned boundary.
Revised: where  1 ,  2 ,  3  0 are the coefﬁcients to enlarge the margin of the learned boundary.

Original: Instead of simply classifying the transitions with the ﬁrst and the second loss terms, we encourage the forward invariance property by adding the third term, similar to Control Lyapunov Functions and Control
Revised: Instead of simply classifying the transitions with the ﬁrst and the second loss terms, we encourage the forward invariance property by adding the third term, similar to Control Lyapunov Functions and Control

Original: Barrier Functions [31].
Revised: Barrier Functions [31].

Original: In practice, we approximate ˙ φ ( x, a ) by φ ( x (cid:48) , a (cid:48) ) − φ ( x, a ) , where ( x (cid:48) , a (cid:48) ) is the next state-action pair on the trajectory.
Revised: In practice, we approximate ˙  ( x, a ) by  ( x 0 , a 0 )   ( x, a ) , where ( x 0 , a 0 ) is the next state-action pair on the trajectory.

Original: Intuitively, once all pairs of consecutive transitions satisfy this condition, the transitions would form a forward invariant set, and any trajectory starting from inside the invariant set will never cross the admissible boundary.
Revised: Intuitively, once all pairs of consecutive transitions satisfy this condition, the transitions would form a forward invariant set, and any trajectory starting from inside the invariant set will never cross the admissible boundary.

Original: The λφ ( x, a ) term encourages the trajectory to be asymptotically admissible, even if the agent enters the inadmissible region sometimes.
Revised: The  ( x, a ) term encourages the trajectory to be asymptotically admissible, even if the agent enters the inadmissible region sometimes.

Original: Relabeling Through Backpropagation.
Revised: Relabeling Through Backpropagation.

Original: We ﬁrst label the collected transitions as admissible and inadmissible based on whether the next state is in the danger region.
Revised: We ﬁrst label the collected transitions as admissible and inadmissible based on whether the next state is in the danger region.

Original: Once the whole trajectory terminates, we relabel these transitions through backpropagation.
Revised: Once the whole trajectory terminates, we relabel these transitions through backpropagation.

Original: Intuitively, suppose the agent encounters the collision along the trajectory.
Revised: Intuitively, suppose the agent encounters the collision along the trajectory.

Original: In that case, it means that starting from some timestep, it enters the inadmissible region and fails to get back to the admissible set (similar to Intermediate
Revised: In that case, it means that starting from some timestep, it enters the inadmissible region and fails to get back to the admissible set (similar to Intermediate

Original: Value Theorem [49]).
Revised: Value Theorem [49]).

Original: Without relabeling, the labels of the transitions from this step to the collision event will be admissible, which is incorrect.
Revised: Without relabeling, the labels of the transitions from this step to the collision event will be admissible, which is incorrect.

Original: To solve this issue, we relabel these critical transitions.
Revised: To solve this issue, we relabel these critical transitions.

Original: A state-action pair ( x, a ) on the trajectory is relabelled as inadmissible, if (i) the next state-action pair ( x (cid:48) , a (cid:48) ) is inadmissible, and (ii) no admissible action is found at state x (cid:48) - namely, max a (cid:48) φ ( x (cid:48) , a (cid:48) ) < 0 .
Revised: A state-action pair ( x, a ) on the trajectory is relabelled as inadmissible, if (i) the next state-action pair ( x 0 , a 0 ) is inadmissible, and (ii) no admissible action is found at state x 0 - namely, max a 0  ( x 0 , a 0 ) < 0 .

Original: Intuitively, the relabelling propagates the inadmissibility from where the agent enters the danger region back to a state, where the agent can make a decision leading to admissible regions.
Revised: Intuitively, the relabelling propagates the inadmissibility from where the agent enters the danger region back to a state, where the agent can make a decision leading to admissible regions.

Original: The ﬁrst condition ensures the optimistic behavior of the CAM.
Revised: The ﬁrst condition ensures the optimistic behavior of the CAM.

Original: We only relabel those transitions which stay in the inadmissible region and encounter danger in the future.
Revised: We only relabel those transitions which stay in the inadmissible region and encounter danger in the future.

Original: Such optimism is important when no feasible trajectory exists at the early training stage.
Revised: Such optimism is important when no feasible trajectory exists at the early training stage.

Original: It encourages the agent to explore instead of predicting almost every transition to be inadmissible.
Revised: It encourages the agent to explore instead of predicting almost every transition to be inadmissible.

Original: The second condition ensures that there exist no admissible actions approximately by inspecting the admissibility score for all the actions.
Revised: The second condition ensures that there exist no admissible actions approximately by inspecting the admissibility score for all the actions.

Original: Furthermore, for the transitions that have potential admissible actions, they will not be relabelled as inadmissible easily at the early training stage since max a (cid:48) φ ( x (cid:48) , a (cid:48) ) < 0 is hard to meet for a moderately initialized CAM.
Revised: Furthermore, for the transitions that have potential admissible actions, they will not be relabelled as inadmissible easily at the early training stage since max a 0  ( x 0 , a 0 ) < 0 is hard to meet for a moderately initialized CAM.

Original: 2.4 Online Inference with Graph Decomposition
Revised: 2.4 Online Inference with Graph Decomposition

Original: At inference time, the agent behaves in the same manner as the training stage, except that there is no exploration noise during inference.
Revised: At inference time, the agent behaves in the same manner as the training stage, except that there is no exploration noise during inference.

Original: When the CAM is applied directly to a large swarm of agents, however, we ﬁnd that the performance degrades signiﬁcantly due to the distribution shift.
Revised: When the CAM is applied directly to a large swarm of agents, however, we ﬁnd that the performance degrades signiﬁcantly due to the distribution shift.

Original: Such degradation occurs considerably when the inference task has a much higher density of agents.
Revised: Such degradation occurs considerably when the inference task has a much higher density of agents.

Original: To solve this issue, we leverage the compositional property of CAMs and propose graph decomposition.
Revised: To solve this issue, we leverage the compositional property of CAMs and propose graph decomposition.

Original: Decomposition on Large Graphs.
Revised: Decomposition on Large Graphs.

Original: We can break down the whole graph G into subgraphs { G k } , where these subgraphs satisfy the training distribution, e.g., the number of agent vertices does not exceed a certain value.
Revised: We can break down the whole graph G into subgraphs { G k } , where these subgraphs satisfy the training distribution, e.g., the number of agent vertices does not exceed a certain value.

Original: We construct the subgraphs repeatedly until all the edges in G appear at least once in the subgraphs.
Revised: We construct the subgraphs repeatedly until all the edges in G appear at least once in the subgraphs.

Original: For each subgraph G k , and action a , we calculate the admissibility score as φ ( G k , a ) .
Revised: For each subgraph G k , and action a , we calculate the admissibility score as  ( G k , a ) .

Original: Though each subgraph deﬁnes a less strict subtask for the agent to perform, the admissible set for the original task should lie in the intersection of all these admissible sets of the subtasks.
Revised: Though each subgraph deﬁnes a less strict subtask for the agent to perform, the admissible set for the original task should lie in the intersection of all these admissible sets of the subtasks.

Original: As a result, these values on the subgraphs can be composed to represent the admissibility score on the entire graph φ ( G, a ) , which could be totally out of distribution.
Revised: As a result, these values on the subgraphs can be composed to represent the admissibility score on the entire graph  ( G, a ) , which could be totally out of distribution.

Original: To compose the admissibility values, we follow Proposition 1 and use φ ( G, a )
Revised: To compose the admissibility values, we follow Proposition 1 and use  ( G, a )

Original: =
Revised: =

Original: min G k ⊆ G { φ ( G k , a ) } .
Revised: min G k ✓ G {  ( G k , a ) } .

Original: Algorithm2 CAM Algorithm for Inference 1: Input : CAM φ , preference function ω , distribution P of training data 2: for t = 1, T do 3: for each agent’s current state G , goal state T do 4:
Revised: Algorithm 1: Input : CAM  , preference function ! , distribution P of training data 2: for t = 1, T do 3: for each agent’s current state G , goal state T do 4:

Original: Sample candidate actions { a } ⊆ A 5:
Revised: Sample candidate actions { a } ✓ A 5:

Original: Initialize φ ( G, a ) as ∞ for all sampled candidates 6: repeat 7:
Revised: Initialize  ( G, a ) as 1 for all sampled candidates 6: repeat 7:

Original: Sample G i ⊆ G , where G i ∼ P 8: φ ( G, a ) =
Revised: Sample G i ✓ G , where G i ⇠ P 8:  ( G, a ) =

Original: min( φ ( G, a ) , φ ( G i , a )) for all candidates a 9: until all edges on G are sampled at least once 10:
Revised: min(  ( G, a ) ,  ( G i , a )) for all candidates a 9: until all edges on G are sampled at least once 10:

Original: Select action for the agent according to φ ( G, a ) and ω ( G, T , a ) 11:
Revised: Select action for the agent according to  ( G, a ) and ! ( G, T , a ) 11:

Original: Execute the selected actions and observe the next states s t +
Revised: Execute the selected actions and observe the next states s t +

Original:  Experiments 3.1 Proof of Concept: CAM for Single Agent Environment
Revised:  Experiments 3.1 Proof of Concept: CAM for Single Agent Environment

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: -agent environment.
Revised: in a single-agent environment.

Original: The agent aims to reach the goal while avoiding obstacles.
Revised: The agent aims to reach the goal while avoiding obstacles.

Original: The black point denotes the starting state, and the red star denotes the goal state.
Revised: The black point denotes the starting state, and the red star denotes the goal state.

Original: The shaded blue regions denote the obstacles.
Revised: The shaded blue regions denote the obstacles.

Original: Given the learned
Revised: Given the learned

Original: CAM φ , action space A : {− 0 . 1 , 0 . 1 } 2 , and an arbitrary state x ∈ X : {− 3 , 3 } 2 , we illustrate max a ∈A φ ( x, a ) , min a ∈A φ ( x, a ) , and p ( φ ( x, a ) ≥ 0) respectively.
Revised: CAM  , action space A : {  0 . 1 , 0 . 1 } 2 , and an arbitrary state x 2 X : {  3 , 3 } 2 , we illustrate max a 2 A  ( x, a ) , min a 2 A  ( x, a ) , and p (  ( x, a )  0) respectively.

Original: We show that the learned CAM never leaves the admissible set, i.e.
Revised: We show that the learned CAM never leaves the admissible set, i.e.

Original: [Equation] , along the trajectory.
Revised: { x 2 X : max a 2 A  ( x, a )  0 } , along the trajectory.

Original: We begin by analyzing the CAM agent in a single-agent environment.
Revised: We begin by analyzing the CAM agent in a single-agent environment.

Original: We perform such analysis using a 2D minimal example environment, where we place three danger regions to form narrow passages, as shown in Figure 3.
Revised: We perform such analysis using a 2D minimal example environment, where we place three danger regions to form narrow passages, as shown in Figure 3.

Original: We illustrate max a ∈A φ ( x, a ) , min a ∈A φ ( x, a ) , and p ( φ ( x, a ) ≥ 0) respectively.
Revised: We illustrate max a 2 A  ( x, a ) , min a 2 A  ( x, a ) , and p (  ( x, a )  0) respectively.

Original: Intuitively, max a ∈A φ ( x, a ) indicates whether there exists any admissible actions that drive the agent to the admissible set.
Revised: Intuitively, max a 2 A  ( x, a ) indicates whether there exists any admissible actions that drive the agent to the admissible set.

Original: min a ∈A φ ( x, a ) indicates the possibility to enter the inadmissible sets if actions are picked randomly.
Revised: min a 2 A  ( x, a ) indicates the possibility to enter the inadmissible sets if actions are picked randomly.

Original: As what we show in the ﬁgure, the trajectory (blue dashed line) does not intersect with the admissibility boundary, i.e. { x ∈ X : max a ∈A φ ( x, a ) = 0 } .
Revised: As what we show in the ﬁgure, the trajectory (blue dashed line) does not intersect with the admissibility boundary, i.e. { x 2 X : max a 2 A  ( x, a ) = 0 } .

Original: This example stays consistent with the forward-invariance objective for learning the CAM: if the agent executes an admissible action a ∈ A : φ ( x, a ) ≥ 0 at every time step, then it will never leave the admissible set.
Revised: This example stays consistent with the forward-invariance objective for learning the CAM: if the agent executes an admissible action a 2 A :  ( x, a )  0 at every time step, then it will never leave the admissible set.

Original: 3.2 Multi-agent Navigation Experiments
Revised: 3.2 Multi-agent Navigation Experiments

Original: Experiment Setup.
Revised: Experiment Setup.

Original: We evaluate our methods under 4 types of environments: UR5, Car, Dynamic
Revised: We evaluate our methods under 4 types of environments: UR5, Car, Dynamic

Original: Dubins, and Drone.
Revised: Dubins, and Drone.

Original: We brieﬂy describe these 4 environments as follows: (i)
Revised: We brieﬂy describe these 4 environments as follows: (i)

Original: The UR5 environment is implemented with PyBullet [50] and contains 4 decentralized 5D robot arms with cubic obstacles.
Revised: The UR5 environment is implemented with PyBullet [50] and contains 4 decentralized 5D robot arms with cubic obstacles.

Original: We randomly select solely 2 robot arms from these four arms during training.
Revised: We randomly select solely 2 robot arms from these four arms during training.

Original: We ﬁx the conﬁgurations of the static obstacles and the goal states of the arms.
Revised: We ﬁx the conﬁgurations of the static obstacles and the goal states of the arms.

Original: In order to make the test task solvable, the initial
Revised: In order to make the test task solvable, the initial

Original: To evaluate our CAM approach thoroughly, we design diverse scenarios for all four environments.
Revised: To evaluate our CAM approach thoroughly, we design diverse scenarios for all four environments.

Original: For the UR5 environment, we make the number of agents vary from 2 to 4.
Revised: For the UR5 environment, we make the number of agents vary from 2 to 4.

Original: For scenarios with 2 orarms, the arms are selected randomly from the four decentralized arms.
Revised: For scenarios with 2 orarms, the arms are selected randomly from the four decentralized arms.

Original: For the other environments, we ﬁx the map size as 32x32 and vary the ratio of agents over obstacles from 1:1023 to 512:512.
Revised: For the other environments, we ﬁx the map size as 32x32 and vary the ratio of agents over obstacles from 1:1023 to 512:512.

Original: For each setting, we average the performance over 100 randomly generated test cases.
Revised: For each setting, we average the performance over 100 randomly generated test cases.

Original: Baselines.
Revised: Baselines.

Original: The baseline approaches we compare with include DDPG [53] and MACBF [30].
Revised: The baseline approaches we compare with include DDPG [53] and MACBF [30].

Original: DDPG is a standard RL approach to learning safe behavior through maximizing rewards.
Revised: DDPG is a standard RL approach to learning safe behavior through maximizing rewards.

Original: MACBF is a state-of-the-art approach for multi-agent safe control problems, which learns a policy jointly with neural safe certiﬁcates.
Revised: MACBF is a state-of-the-art approach for multi-agent safe control problems, which learns a policy jointly with neural safe certiﬁcates.

Original: We reimplement all the baselines with GNNs for a fair comparison.
Revised: We reimplement all the baselines with GNNs for a fair comparison.

Original: Note that our GNN implementation for DDPG is a multi-agent algorithm, since each agent is aware of the other agents by taking the egocentric graph as the input.
Revised: Note that our GNN implementation for DDPG is a multi-agent algorithm, since each agent is aware of the other agents by taking the egocentric graph as the input.

Original: We refer reader to Appendix for more details.
Revised: We refer reader to Appendix for more details.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: We evaluate our methods based on two metrics: the safe rate and the average reward.
Revised: We evaluate our methods based on two metrics: the safe rate and the average reward.

Original: The safe rate counts the number of collisions for each agent at every time step and averages over all the agents through the whole trial [30].
Revised: The safe rate counts the number of collisions for each agent at every time step and averages over all the agents through the whole trial [30].

Original: At each time step, the reward is -1 if the agent collides with another agent or the static obstacle.
Revised: At each time step, the reward is -1 if the agent collides with another agent or the static obstacle.

Original: A +10 reward will be given at the end of the trajectory, if the agent reaches its goal at any time step along the trajectory.
Revised: A +10 reward will be given at the end of the trajectory, if the agent reaches its goal at any time step along the trajectory.

Original: A small negative constant of -0.1 is added to the reward to encourage shorter paths for goal-reaching.
Revised: A small negative constant of -0.1 is added to the reward to encourage shorter paths for goal-reaching.

Original: The reward is accumulated throughout the trial and averaged over all the agents.
Revised: The reward is accumulated throughout the trial and averaged over all the agents.

Original: Overall Performance.
Revised: Overall Performance.

Original: In Figure 4, we demonstrate the overall performances for the UR5, Car,
Revised: In Figure 4, we demonstrate the overall performances for the UR5, Car,

Original: Dynamic Dubins, and Drone environments.
Revised: Dynamic Dubins, and Drone environments.

Original: Our method shows the generalization across different environments and densities of agents and obstacles.
Revised: Our method shows the generalization across different environments and densities of agents and obstacles.

Original: Though trained with only 3 agents, the safety rate of CAM remains nearly 100% for all the environments, up to 512 agents.
Revised: Though trained with only 3 agents, the safety rate of CAM remains nearly 100% for all the environments, up to 512 agents.

Original: Even if the density of agents and obstacles deviates from the training environment, our CAM approach can avoid the distribution shift, using decomposition on large graphs and the compositionality of CAMs.
Revised: Even if the density of agents and obstacles deviates from the training environment, our CAM approach can avoid the distribution shift, using decomposition on large graphs and the compositionality of CAMs.

Original: On the other side, though our implementation for MACBF and DDPG can take an arbitrary number of agents using GNNs, the performance degrades signiﬁcantly when the distribution of input graphs shifts in test environments.
Revised: On the other side, though our implementation for MACBF and DDPG can take an arbitrary number of agents using GNNs, the performance degrades signiﬁcantly when the distribution of input graphs shifts in test environments.

Original: We also inspect that the averaged reward for all the methods decreases in most cases when the number of agents increases.
Revised: We also inspect that the averaged reward for all the methods decreases in most cases when the number of agents increases.

Original: It is due to more agents failing to reach the goal when the timeout happens since it takes more time for agents to avoid each other.
Revised: It is due to more agents failing to reach the goal when the timeout happens since it takes more time for agents to avoid each other.

Original: We encourage the reader to view the video demonstrations in the supplementary material for all 4 environments.
Revised: We encourage the reader to view the video demonstrations in the supplementary material for all 4 environments.

Original: [Table]
Revised: [Table]

Original: In this section, we study the generalization of our CAM model for zero-shot transfer to other multiagent tasks, e.g., the chasing game.
Revised: In this section, we study the generalization of our CAM model for zero-shot transfer to other multiagent tasks, e.g., the chasing game.

Original: In the chasing game, each agent chases another agent to pursue.
Revised: In the chasing game, each agent chases another agent to pursue.

Original: The target agent is assigned randomly at the beginning of every episode.
Revised: The target agent is assigned randomly at the beginning of every episode.

Original: The agent’s goal is to maintain the distance to the assigned agent, with no collision with agents and obstacles.
Revised: The agent’s goal is to maintain the distance to the assigned agent, with no collision with agents and obstacles.

Original: We ﬁx the number of agents to 64 in the experiment.
Revised: We ﬁx the number of agents to 64 in the experiment.

Original: This new task is substantially different from the task for training, where the assigned goal is ﬁxed.
Revised: This new task is substantially different from the task for training, where the assigned goal is ﬁxed.

Original: We provide more details of the setting in the Appendix.
Revised: We provide more details of the setting in the Appendix.

Original: In Table 1, we show that our CAM method can adapt to the new task effectively due to its safety and goal-reaching decoupling property, preserving a relatively high safety rate and reward.
Revised: In Table 1, we show that our CAM method can adapt to the new task effectively due to its safety and goal-reaching decoupling property, preserving a relatively high safety rate and reward.

Original: In addition, we compare our method to the CAM without the graph decomposition.
Revised: In addition, we compare our method to the CAM without the graph decomposition.

Original: We observe that the graph decomposition works effectively in both environments.
Revised: We observe that the graph decomposition works effectively in both environments.

Original: The performance of the Car environment beneﬁts more from the graph decomposition.
Revised: The performance of the Car environment beneﬁts more from the graph decomposition.

Original: This advancement is because the density of Car agents in 2D is higher than that of the Drone agent in 3D. The higher density makes the test task more unlike the training task, which explains why the graph decomposition is more beneﬁcial for Car.
Revised: This advancement is because the density of Car agents in 2D is higher than that of the Drone agent in 3D. The higher density makes the test task more unlike the training task, which explains why the graph decomposition is more beneﬁcial for Car.

Original:  Limitations and Failure Modes
Revised:  Limitations and Failure Modes

Original: While the framework is generalizable in terms of scalability, it has limitations.
Revised: While the framework is generalizable in terms of scalability, it has limitations.

Original: Since the proposed
Revised: Since the proposed

Original: GNN architecture is entirely decentralized, it disables the agents from communicating and coordinating with each other in the swarm.
Revised: GNN architecture is entirely decentralized, it disables the agents from communicating and coordinating with each other in the swarm.

Original: As a result, it could be challenging for agents to identify and avoid critical situations requiring interactions among agents, e.g., dead-locks and states with very few feasible actions.
Revised: As a result, it could be challenging for agents to identify and avoid critical situations requiring interactions among agents, e.g., dead-locks and states with very few feasible actions.

Original: As shown in Figure 4, there is a slight drop in the safety rate of the Drone environment when the number of agents increases to 512.
Revised: As shown in Figure 4, there is a slight drop in the safety rate of the Drone environment when the number of agents increases to 512.

Original: To understand such a failure mode, we inspect these trajectories and ﬁnd that the corresponding CAM values are all negative at several consecutive preceding states right before the collision.
Revised: To understand such a failure mode, we inspect these trajectories and ﬁnd that the corresponding CAM values are all negative at several consecutive preceding states right before the collision.

Original: Such behavior indicates a very low probability of ﬁnding feasible actions in these states, which validates the necessity for inter-agent communication or global coordination when the density of agents is very high in local regions.
Revised: Such behavior indicates a very low probability of ﬁnding feasible actions in these states, which validates the necessity for inter-agent communication or global coordination when the density of agents is very high in local regions.

Original: We believe that a preference function considering the global coordination can be helpful for this issue.
Revised: We believe that a preference function considering the global coordination can be helpful for this issue.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed new learning-based control methods for multi-agent navigation problems by shifting the focus from training optimal control policies to training CAMs that implicitly represent a set of feasible actions.
Revised: We proposed new learning-based control methods for multi-agent navigation problems by shifting the focus from training optimal control policies to training CAMs that implicitly represent a set of feasible actions.

Original: We avoid the reward engineering by decoupling the multi-agent navigation tasks using a simple goal-reaching preference function and a learnable CAM for collision avoidance.
Revised: We avoid the reward engineering by decoupling the multi-agent navigation tasks using a simple goal-reaching preference function and a learnable CAM for collision avoidance.

Original: Our methods use GNNs to represent the CAM which takes egocentric graphs as inputs and proposes the relabelling with backtracing to learn from rich information even given sparse reward.
Revised: Our methods use GNNs to represent the CAM which takes egocentric graphs as inputs and proposes the relabelling with backtracing to learn from rich information even given sparse reward.

Original: In online inference, we propose the graph decomposition to deal with the covariate shift, which achieves a notably high success rate and low collision rate.
Revised: In online inference, we propose the graph decomposition to deal with the covariate shift, which achieves a notably high success rate and low collision rate.

Original: While unconventional compared to the types of models routinely used in multi-agent reinforcement learning, we demonstrate that CAM has several useful properties and is particularly effective in multi-agent settings that require compositionality and generalization for scaling at inference time.
Revised: While unconventional compared to the types of models routinely used in multi-agent reinforcement learning, we demonstrate that CAM has several useful properties and is particularly effective in multi-agent settings that require compositionality and generalization for scaling at inference time.

Original: In future work, we will study how to incorporate multi-agent communication and test its effectiveness in real-world scenarios.
Revised: In future work, we will study how to incorporate multi-agent communication and test its effectiveness in real-world scenarios.

Original: Acknowledgments
Revised: Acknowledgments

Original: We thank the anonymous reviewers for their helpful comments in revising the paper.
Revised: We thank the anonymous reviewers for their helpful comments in revising the paper.

Original: This material is based on work supported by the United States Air Force and DARPA under Contract No. FA875018-C-0092, AFOSR YIP FA9550-19-1-0041, NSF Career CCF 2047034, and Amazon ResearchAward. A [Table]
Revised: This material is based on work supported by the United States Air Force and DARPA under Contract No. FA875018-C-0092, AFOSR YIP FA9550-19-1-0041, NSF Career CCF 2047034, and Amazon Research

Original: HiddenState

Original: [Table]

Original: Concatenate

Original: [Table]

Original: /jyMmleVPzLSvW+Wq7d5HEU4BhO4Ax8uIIa3EEdGkDhEZ7hFd6QQi/oHX3MW1dQPnMEf4A+fwBlTo8G</latexit> + <latexit sha1_base64="fn6l+KBPqVjUDDAA/

Original: wSqrrhPcSU=">AAAB7XicbVBNSwMxEJ34WetX1aOXYBEEoexKUY9FLx4r2A9ol5JNs21sNlmSrFCW/gcvHhTx6v/x5r8xbfegrQ8GHu/NMDMvTAQ31vO+0crq2vrGZmGruL2zu7dfOjhsGpVqyhpUCaXbITFMcMkallvB2olmJA4Fa4Wj26nfemLacCUf7DhhQUwGkkecEuukZjeMsvNJr1T2Kt4MeJn4OSlDjnqv9NXtK5rGTFoqiDEd30tskBFtORVsUuymhiWEjsiAdRyVJGYmyGbXTvCpU/o4UtqVtHim/p7ISGzMOA5dZ0zs0Cx6U/E/r5Pa6DrIuExSyySdL4pSga3C09dxn2tGrRg7Qqjm7lZMh0QTal1ARReCv/jyMmleVPzLSvW+Wq7d5HEU4BhO4Ax8uIIa3EEdGkDhEZ7hFd6QQi/oHX3MW1dQPnMEf4A+fwBlTo8G</latexit>

Original: [Figure]

Original: MessagePassing

Original: MessagePassing

Original: B Details for the UR5 Environment

Original: Dynamics.

Original: Each UR5 agent controls a 5D robot arm.

Original: The state x = [ θ 1 , θ 2 , θ 3 , θ 4 , θ 5 ] ∈ [ − 2 π, 2 π ] 5 , representing the angles on the joints of base to shoulder, shoulder to lift, elbow, wrist 1, and wrist 2.

Original: The action is the angular velocity a = [ ˙ θ 1 , ˙ θ 2 , ˙ θ 3 , ˙ θ 4 , ˙ θ 5 ] ∈ [ − 1 , 1] 5 , and the dynamics to compute the next state x (cid:48) are

Original: [Equation]

Original: We choose dt = 0 .

Original: 1 in our experiment.

Original: Graph Representation.

Original: Each node v ∈ V is simply a zero scalar.

Original: For each agent.

Original: Each edge e : ( v i , v

Original: j ) ∈

Original: E is a 18D vector, which concatenates 4 vectors.

Original: The ﬁrst 4D vector is a one-hot vector representing the arm ID of v i .

Original: The second 5D vector represents the state of the arm v i .

Original: The third and fourth vectors follow the same representation as to the ﬁrst and the second vectors, but those of arm v j replace the values.

Original: The graph is complete for this environment, i.e., each arm can affect any other arm in the environment.

Original: Training Distribution.

Original: The training distribution is that only two arms should be in the graph.

Original: Collision and Goal-reaching Criteria.

Original: We deﬁne collision as the contact or penetration between one arm and another arm or static obstacle.

Original: If the state of the arm is within 0.3 distance of the goal, then the goal is reached.

Original: Preference Function ω .

Original: The preference function calculates the negative of distance to the goal based on the next state: ω ( G, T , a )

Original: =

Original: −|| f ( x G , a ) − T

Original: || 22 .

Original: Hyperparameters.

Original: For each time step, we sample N = 2000 candidate actions for every agent.

Original: The initial learning rate of the ADAM optimizer is 3e-4, and we decrease the learning rate with a factor of 0.5 when the performance in the validation environment meets a plateau until we reach a minimum learning rate of 1e-4.

Original: We set γ 1 , γ 2 , γ 3 as 0, 1e-1, 1e-2.

Original: We use a batch size of 256 and update the network every 20 trajectories.

Original: C Details for the Car Environment

Original: Dynamics.

Original: Each Car agent follows the Dubins’ Car dynamics [51].

Original: The state x = [ p x , p y , θ ] , p x , p y ∈ R , θ ∈ [0 , 2 π ) , representing the positions on 2D plane and the heading an- gle.

Original: We apply a constant v = 0 .

Original: as the velocity for each agent.

Original: The action is the angular velocity a = [ ˙ θ ] ∈ [ − 23 π, 2 3 π ] , and the dynamics to compute the next state x (cid:48) are

Original: [Equation]

Original: We choose dt = 1 in our experiment.

Original: Graph Representation.

Original: Each node v ∈ V is a one-hot vector indicating whether the current node is an agent or a static obstacle.

Original: Given each edge e : ( v i , v j ) ∈ E , the feature vector of the edge is [ I, sin( θ i ) , cos( θ i ) , sin( θ j ) , cos( θ j ) , p x,i − p x,j , p y,i − p y,j ] , where I is a one-hot vector, representing whether the current edge is obstacle-to-agent or agent-to-agent.

Original: For the obstacle-to-agent edges, sin( θ i ) , cos( θ i ) are padded as 0.

Original: We deﬁne each node’s neighbors, including agents and obstacles, as those nodes having a distance within 1.5 to the current node.

Original: We only connect edges from these neighbor nodes to each node.

Original: Training Distribution.

Original: The training distribution is that only 0-2 neighbor agents and 0-9 neighbor obstacles should be in the graph.

Original: Collision and Goal-reaching Criteria.

Original: We deﬁne the agent and the obstacle as circles with a radius of 0.15.

Original: The goals are circles with a radius of 0.3.

Original: The agent is in a collision if it is within a distance of 0.3 to other agents or obstacles.

Original: If the agent’s position is within a distance of 0.45 to the 2D goal, then the goal is reached.

Original: Preference Function ω .

Original: The preference function calculates the negative of distance to the goal based on the next state: ω ( G, T , a )

Original: = −|| ( p (cid:48)

Original: x , p (cid:48)

Original: y )

Original: − T

Original: || 22 , p (cid:48)

Original: x , p (cid:48)

Original: y ∈ f

Original: ( x G , a ) .

Original: Hyperparameters.

Original: For each time step, we sample N = 2000 candidate actions for every agent.

Original: The initial learning rate of the ADAM optimizer is 1e-3, and we decrease the learning rate with a factor of 0.5 when the performance in the validation environment meets a plateau until we reach a minimum learning rate of 1e-5.

Original: We set γ 1 , γ 2 , γ 3 as 0, 2e-2, 1e-2.

Original: We use a batch size of 256 and update the network every 10 trajectories.

Original: D Details for the Dynamic Dubins Environment

Original: Dynamics.

Original: The agent follows dynamic Dubins’ car and controls a 2D action.

Original: The state x = [ p x , p y , v, θ ] , p x , p y ∈ R , v ∈ [0 , 1] , θ ∈ [0 , 2 π ) , representing the position, velocity, and heading angle in the 2D space.

Original: The 2D action a ∈ A =

Original: [ − 1 , 1] 2 is composed of the acceleration rate q and the angular velocity ˙ θ , and the dynamics to compute the next state x (cid:48) is

Original: [Equation]

Original: We choose dt = 0 .

Original: 05 in our experiment.

Original: Graph Representation.

Original: Each node is a one-hot vector indicating whether the current node is an agent or a static obstacle.

Original: Given each edge e : ( i, j ) ∈ E , the feature vector of the edge is [ I, v i , v j , sin( θ i ) , cos( θ i ) , sin( θ j ) , cos( θ j ) , p x,i − p x,j , p y,i − p y,j ] , where I is a one-hot vector, representing whether the current edge is obstacle-to-agent or agent-to-agent.

Original: For the obstacle-to-agent edges, v i , sin( θ i ) , cos( θ i ) are padded as 0.

Original: We deﬁne each node’s neighbors, including agents and obstacles, as those nodes having a distance within 1.5 to the current node.

Original: We only connect edges from these neighbor nodes to each node.

Original: Training Distribution.

Original: The training distribution is that only 0-2 neighbor agents and 0-9 neighbor obstacles should be in the graph.

Original: Collision and Goal-reaching Criteria.

Original: We deﬁne the agent and the obstacle as circles with a radius of 0.15.

Original: The goals are circles with a radius of 0.3.

Original: The agent is in a collision if it is within a distance of 0.3 to other agents or obstacles.

Original: If the agent’s position is within a distance of 0.45 to the 2D goal, then the goal is reached.

Original: Preference Function ω .

Original: The preference function calculates the negative of distance to the goal based on the next state: ω ( G, T , a )

Original: = −|| ( p (cid:48)

Original: x , p (cid:48)

Original: y )

Original: − T

Original: || 22 , p (cid:48)

Original: x , p (cid:48)

Original: y ∈ f

Original: ( x G , a ) .

Original: Hyperparameters.

Original: For each time step, we sample N = 2000 candidate actions for every agent.

Original: The initial learning rate of the ADAM optimizer is 1e-3, and we decrease the learning rate with a factor of

Original: 0.5 when the performance in the validation environment meets a plateau until we reach a minimum learning rate of 1e-5.

Original: We set γ 1 , γ 2 , γ 3 as 0, 2e-2, 1e-2.

Original: We use a batch size of 256 and update the network every 10 trajectories.

Original: E Details for the Drone Environment

Original: Dynamics.

Original: Each Drone agent follows a 3D quadrotor model adopted from [52] and controls a 4D action.

Original: The state x = [ p x , p y , p z , v x , v y , v z , α, β, γ ] , p x , p y , p z ∈ R , v x , v y , v z ∈ [ − 1 , 1] , α, β, γ ∈ [ − π/ 2 , π/ 2] , representing the position and velocity in the 3D space, and the angles in pitch, roll, and yaw.

Original: The action is the instant acceleration q and the angular velocities, i.e., a = [ q, ˙ α, ˙ β, ˙ γ ] ∈ [ − 1 , 1] 4 , and the dynamics to compute the next state x (cid:48) is

Original: [Equation]

Original: 6) g is the gravity acceleration of the Earth as 9.8.

Original: We choose dt = 0 .

Original: 01 and k = 10 in our experiment.

Original: Graph Representation.

Original: Each node v ∈ V is a one-hot vector indicating whether the current node is an agent or a static obstacle.

Original: Given each edge e : ( v i , v j ) ∈ E , the feature vector of the edge is [ I, v x,i , v y,i , v z,i , sin( α i ) , cos( α i ) , sin( β i ) , cos( β i ) , v x,j , v y,j , v z,j , sin( α j ) , cos( α j ) , sin( β j ) , cos( β j ) , p x,i − p x,j , p y,i − p y,j , p z,i − p z,j ] , where I is a one-hot vector, representing whether the current edge is obstacle-to-agent or agent-to-agent.

Original: For the obstacleto-agent edges, v x,i , v y,i , v z,i , sin( α i ) , cos( α i ) , sin( β i ) , cos( β i ) , p z,i − p z,j are padded as 0.

Original: We deﬁne each node’s neighbors, including agents and obstacles, as those nodes having a distance within 1.5 to the current node.

Original: We only connect edges from these neighbor nodes to each node.

Original: Training Distribution.

Original: The training distribution is that only 0-2 neighbor agents and 0-9 neighbor obstacles should be in the graph.

Original: Collision and Goal-reaching Criteria.

Original: We deﬁne the agent as spheres with a radius of 0.15.

Original: The obstacles are vertical cylinders with a radius of 0.15 and inﬁnite height.

Original: The goals are spheres with a radius of 0.3.

Original: The agent is in a collision if it has a distance within 0.3 to other agents or obstacles in the 3D space.

Original: An agent reaches its goal if its 3D position has a distance of 0.45 to its goal.

Original: Preference Function ω .

Original: The preference function calculates the negative of an error between the action a and a linear–quadratic regulator (LQR) controller π .

Original: The LQR controller π takes the goal and the current state, and gives an action which only considers the goal-reaching: ω ( G, T , a )

Original: =

Original: −|| π ( x G , T ) − a || 22 .

Original: Hyperparameters.

Original: For each time step, we sample N = 2000 candidate actions for every agent.

Original: The initial learning rate of the ADAM optimizer is 1e-3, and we decrease the learning rate with a factor of 0.5 when the performance in the validation environment meets a plateau until we reach a minimum learning rate of 1e-5.

Original: We set γ 1 , γ 2 , γ 3 as 0, 1e-1, 1e-2.

Original: We use a batch size of 256 and update the network every 20 trajectories.

Original: F Details for the Chasing Game

Original: We provide more details for the chasing game in this section.

Original: We deﬁne the safety rates of the chasing game in the same way as those of the navigation task.

Original: The reward is deﬁned in another way, to reﬂect on the situation that one Drone agent could drift away from its target agent though it executes the most preferred action.

Original: We want to avoid penalizing such behavior because the most preferred action should not be discouraged if it is also safe at the same time.

Original: We deﬁne reward as follows:

Original: R = (cid:80) Tt =1 clip ( d t − 1 − d t , 0 , 2) , where d t − 1 and d t indicate the distances between the agent and the target agent in 2D or 3D space at t − 1 -th and t -th step.

Original: Such a reward keeps track of the agent’s improvement at every step while not penalizing on no progress.

Original: The preference for the agent follows the same way we deﬁned in the navigation tasks of the Car and Drone.

Original: We update the goal for each agent to be the position of the target agent at every step.

Original: G Details for the Experiments

Original: Reproducing the Baselines.

Original: The inputs to the networks of DDPG and MACBF are the same as those to the CAM.

Original: For DDPG, we use the GNN architectures as the actor and Q-critic network.

Original: The

Original: GNN actor takes an egocentric graph for each agent and outputs the action.

Original: The GNN critic takes the graph and action and outputs the Q value.

Original: For MACBF, since it requires differentiable dynamics, we implement an additional dynamic function using PyTorch to guarantee the differentiability.

Original: When computing the derivative of the CBF value over time, we assume the graph structure remains the same, i.e., no adding or removing edges.

Original: We also ﬁnd that replacing the temporary dataset of MACBF with the replay buffer, typically used for the RL methods, improves the stability of MACBF.

Original: Notes on why MA-DDPG is not deployed here [12] : Our GNN implementation deviates from the

Original: MA-DDPG algorithm because of scalability concerns.

Original: In MA-DDPG, the actor network for each agent is independent and does not share the weight.

Original: As a result, MA-DDPG requires the same number of agents for the training and inference tasks.

Original: However, in our experiment, since we focus on the generalization for scalability, the number of agents during inference varies, while our training tasks are ﬁxed to have three agents.

Original: Thus, it is impossible to deploy the MA-DDPG algorithm.

Original: Running the Experiments.

Original: All the methods are trained on CPU as Intel Core i7-11700F.

Original: The GPU is NVIDIA RTX 3080.

Original: We train each method until we reach convergence, which typically requires fewer than two days.

Original: Accelerating the Computation.

Original: Though the GNN can be deployed on decentralized devices at inference time, we conduct our experiment on one single desktop, which requires computation optimization.

Original: Assume at a time step of the inference time that there are 512 agents in the environment.

Original: We need to test 2000 actions for each agent, and each agent samples at least 100 subgraphs.

Original: [-We can infer that there are at least 100 × 2000 × 
-]
Revised: [-We can infer that there are at least 100 × 2000 × 
-]

Original: ≈ 1 .

Original: 02 × 10 8 forward operations for the GNN to compute.

Original: We need to accelerate such computation; otherwise, the computational cost will be intractable.

Original: We conduct three improvements for the acceleration.

Original: The ﬁrst improvement is to merge the computation of the hidden state for all state-action pairs which share the same input graph into one forward passing in the GNN layer, as mentioned in Section 2.2.

Original: The second improvement is to parallelize the computation of the CAM value by stacking the pairs of hidden states and actions as a matrix and then feeding it to the GPU.

Original: Such stacking enables the computation with batches of state-action pairs.

Original: The third improvement is to choose which agents to compute for each batch adaptively.

Original: By inspecting the

Original: CAM values of the computed batches, if we have already found admissible actions for one speciﬁc agent, then the rest of the state-action pairs for this agent do not need to be computed, and the agent chooses the action among those admissible actions that have been computed.

Original: By ﬁltering out the determined agents, we save the unnecessary computation of the state-action pairs of these agents.

Original: H Ablation Study: Varying Obstacle Density

Original: In the main paper, we test the scenarios where the densities of both the agents and the obstacles vary.

Original: In this section, we focus on how the obstacle density affects the performance of each method.

Original: We ﬁxed the map size to be 4x4 and increased the obstacle number from 0 to 20.

Original: We average the results over 100 randomly generated environments.

Original: We do not include the UR5 environment since the obstacle number for UR5 is ﬁxed to be 4.

Original: As shown, the CAM signiﬁcantly outperforms other methods and maintains the safety rate near 100%.

Original: On the other hand, MACBF and DDPG show a lack of robustness when the obstacle density increases.

Original: We also observe a remarkable degradation of the performance of MACBF for the Drone environment.

Original: When there are 0 obstacles, the safety rate of MACBF is close to 99%, but then the safety rate jumps to around 90% when there are

Original: [Figure]

Original: obstacles.

Original: Such a result shows that MACBF works better to avoid inter-agent collisions and cannot deal well with obstacle-agent collision avoidance.

Original: This result explains why the performance of

Original: MACBF improves while the density of obstacles decreases in the Drone environment in Figure 4.

Original: I Inference Time

Original: Figure 7 illustrates the running time during inference.

Original: We average the running time over all the agents and all the time steps in the test cases.

Original: With the three acceleration improvements mentioned in

Original: Section G, we can now reduce the running time to around 4.3 milliseconds at most.

Original: The computation time can further be improved when deployed in real-world experiments.

Original: In that case, each agent can compute the admissible set independently, since our GNN architecture is fully decentralized.

Original: It only requires observations of nearby agents and obstacles and does not need inter-agent communication.

Original: We ﬁnd that in Figure 7, the trends of the running time for Car and Drone are similar - the running time decreases and then increases as the density of agents grows.

Original: The running time ﬁrst decreases, since our method utilizes the GPU and computes the admissible actions of all the agents in parallel.

Original: However, when the number of agents grows to a certain threshold between 32 and 128, the memory of the GPU will reach its limitation and cannot take more agents beyond this threshold.

Original: If the number of agents is higher than this threshold, we need multiple GPU forward passes to cover all the agents.

Original: As a result, the running time increases afterward.

Original: More GPU memory should make the computation even faster in such a centralized computation scenario.

Original: J Details for the Single Agent Example

Original: In this section, we provide an additional single-agent experiment as a proof of concept.

Original: This example has more complex dynamics than the one in the main paper.

Original: We provide the details as follows.

Original: Dynamics.

Original: The agent here follows the single integrator and controls a 2D action.

Original: The state x = [ p x , p y ] , representing the position in the 2D space.

Original: The 2D action a = [ v x , v y ] ∈ A =

Original: [ − 1 , 1] 2 is composed of the velocities along x and y axis, and the dynamics to compute the next state x (cid:48) is x (cid:48)

Original: [Equation]

Original: We choose dt = 0 .

Original: in our experiment.

Original: In this example, we use an MLP to represent CAM, since the state here is solely a vector.

Original: Ratio of Agents over Obstacles

Original: M illi s e c o n d s

Original: Running Time (Per Agent Per Step)

Original: Car

Original: Drone

Original: Task Distribution.

Original: The goal state [ p x = 0 , p y = 1 . 7] , and the obstacles are all ﬁxed during training and test.

Original: The starting state of the agent is ﬁxed as [ p x = 0 , p y = − 1 . 7] .

Original: Collision and Goal-reaching Criteria.

Original: We deﬁne the agent as a circle with a radius of 0.15.

Original: The goals are circles with a radius of 0.15.

Original: The agent is in a collision if it intersects with the obstacles.

Original: If the agent intersects with the 2D goal, then the goal is reached.

Original: Preference Function ω .

Original: The preference function calculates the negative of distance to the goal based on the next state: ω ( x, T , a )

Original: = −|| ( p (cid:48)

Original: x , p (cid:48)

Original: y )

Original: − T

Original: || 22 , [ p (cid:48) x , p (cid:48)

Original: y ]

Original: = x (cid:48) =

Original: f ( x, a ) .

Original: J.1 Forward-Invariance Property

Original: To further investigate whether the learned CAM holds the forward-invariance property, we illustrate the boundary of the CAM { x | ∃ a 1 , a 2 ∈ A , φ ( x, a 1 ) ≥ 0 , φ ( x, a 2 ) < 0 } in Figure 8.

Original: We show that the trajectory never leaves the admissible region of CAM, i.e., the trajectory holds the forward-invariance property.

Original: Once the trajectory meets the boundary, it starts to follow the most preferred action among admissible actions, instead of the most preferred action among all sampled actions.

Original: Meanwhile, we ﬁnd that around 0.9% states in the boundary states violate the forwardinvariance property, i.e., they enter the inadmissible region after executing the most admissible action.

Original: We emphasize that since we only train on transitions from collected trajectories, CAM can be asymmetric and violate the forward-invariance property at some unseen states.

Original: We design the CAM to maintain the forward invariance and focus on states that could exist along possible trajectories.

Original: If the agent does not meet some states at any time step during inference, then we do not require these states to obey the forward invariance.

Original: Empirically, we have shown that as long as the training tasks and inference tasks are under the same distribution, the forward invariance often holds for inference trajectories once the CAM learns from enough training data.

Original: K Additional Single Agent Environment: Dynamic Dubins

Original: In this section, we provide an additional single-agent experiment as a proof of concept.

Original: This example has more complex dynamics than the one in the main paper.

Original: [Equation] [Equation] [Equation] [Equation]
Revised: Award.

Original: K.1 Forward-Invariance Property

Original: We sample around 40 trajectories starting from the initial states, and illustrate the collected transitions in Figure 9.

Original: The direction of each arrow indicates the action that CAM agent takes on each state x .

Original: The blue arrows represent the admissible region {

Original: Both red green arrows represent on

Original: [Equation]

Original: Only the red arrows represent those boundary transitions that violate the forward-invariance property.

Original: The black arrows represent the states in the inadmissible region { x | ∀ a ∈ A , φ ( x, a ) < 0 } .

Original: We show that around 98.6% of transitions along the trajectories are admissible and forward invariant.

Original: There are around 27.7% of states on the boundary.

Original: Around 0.4% of states violate the forwardinvariance property, i.e., the CAM agent crosses the boundary and enters the inadmissible region after executing the action.

Original: There are 1% of transitions that are inadmissible, but all of them return back to the admissible regions after executing sequences of most admissible actions.

Original: K.2 Details for the Additional Single Agent Environment

Original: Dynamics.

Original: The agent here is one dynamic Dubins’ car and controls a 2D action.

Original: The state x = [ p x , p y , v, θ ] , p x , p y ∈ R , v ∈ [0 , 1] , θ ∈ [0 , 2 π ) , representing the position, velocity, and heading angle in the 2D space.

Original: The 2D action a ∈ A =

Original: [ − 1 , 1] 2 is composed of the acceleration rate q and the angular velocity ˙ θ , and the dynamics to compute the next state x (cid:48) is

Original: [Equation]

Original: We choose dt = 0 .

Original: in our experiment.

Original: In this example, we use an MLP to represent CAM, since the state here is solely a vector.

Original: Note that the maximum acceleration rate is saturated here.

Original: At each time step, the agent can change the velocity by increasing or decreasing 0.05 at most.

Original: As a result, the agent needs to catch failure early on.

Original: Task Distribution.

Original: The goal state ( p x = 0 , p y = 1 . 7) , and the obstacles are all ﬁxed during training and test.

Original: The starting state of the agent is randomized within the ditribution of p x ∈ [ − 0 .

Original: 25] , p y = − 1 . 7 , v = 0 , θ = π 2 .

Original: Collision and Goal-reaching Criteria.

Original: We deﬁne the agent as a circle with a radius of 0.15.

Original: The goals are circles with a radius of 0.15.

Original: The agent is in a collision if it intersects with the obstacles.

Original: If the agent intersects with the 2D goal, then the goal is reached.

Original: Preference Function ω .

Original: The preference function calculates the negative of distance to the goal based on the next state: ω ( x, T ,

Original: [Equation]

Original: =

Original: [Equation]

Original: K.3 Performance Progress in the Additional Single Agent Environment

Original: After the training converges, we sample the initial state from p x = − 0 .

Original: 25 to p x = 0 .

Original: 25 with an interval equal to 0 .

Original: As shown in Figure 9, all the sampled trajectories reach the goal successfully with no collisions.

Original: We provide the video in the supplementary material with more details.

Original: Additionally, we show the progress during training in Figure 10.

Original: We observe that a great improvement in the success rate happens right after our algorithm relabels a relatively high number of transitions.

Original: This observation is consistent with our relabelling mechanism design, which aims to catch the failure early using backtracing.

Original: We also observe a slight improvement starting around the 700-th trajectory, where the success rate increases gradually from 95% to 100%.

Original: This happens at the same time when the ratio of admissible actions drops.

Original: We infer that at that speciﬁc time, the agent has explored some new states, which are critical for leading the agent to success.

Original: Though the agent is not familiar with these states initially, by collecting these new experiences and learning from them, the ratio becomes stable again and the success rate grows afterward.