E MERGENT R OAD R ULES I N M ULTI -A GENT D RIVING E NVIRONMENTS
A BSTRACT
For autonomous vehicles to safely share the road with human drivers, autonomous vehicles must abide by speciﬁc "road rules" that human drivers have agreed to follow.
"Road rules" include rules that drivers are required to follow by law – such as the requirement that vehicles stop at red lights – as well as more subtle social rules – such as the implicit designation of fast lanes on the highway.
In this paper, we provide empirical evidence that suggests that – instead of hard-coding road rules into self-driving algorithms – a scalable alternative may be to design multi-agent environments in which road rules emerge as optimal solutions to the problem of maximizing trafﬁc ﬂow.
We analyze what ingredients in driving environments cause the emergence of these road rules and ﬁnd that two crucial factors are noisy perception and agents’ spatial density.
We provide qualitative and quantitative evidence of the emergence of seven social driving behaviors, ranging from obeying trafﬁc signals to following lanes, all of which emerge from training agents to drive quickly to destinations without colliding.
Our results add empirical support for the social road rules that countries worldwide have agreed on for safe, efﬁcient driving.
I NTRODUCTION
Public roads are signiﬁcantly more safe and efﬁcient when equipped with conventions restricting how one may use the roads.
These conventions are, to some extent, arbitrary.
For instance, a “drive on the left side of the road” convention is, practically speaking, no better or worse than a “drive on the right side of the road” convention.
However, the decision to reserve some orientation as the canonical orientation for driving is far from arbitrary in that establishing such a convention improves both safety (doing so decreases the probability of head-on collisions) and efﬁciency (cars can drive faster without worrying about dodging oncoming trafﬁc).
In this paper, we investigate the extent to which these road rules – like the choice of a canonical heading orientation – can be learned in multi-agent driving environments in which agents are trained to drive to different destinations as quickly as possible without colliding with other agents.
As visualized in Figure 1, our agents are initialized in random positions in different maps (either synthetically generated or scraped from real-world intersections from the nuScenes dataset (Caesar et al., 2019)) and tasked with reaching a randomly sampled feasible target destination.
Intuitively, when agents have full access to the map and exact states of other agents, optimizing for trafﬁc ﬂow leads the agents to drive in qualitatively aggressive and un-humanlike ways.
However, when perception is imperfect and noisy, we show in Section 5 that the agents begin to rely on constructs such as lanes, trafﬁc lights, and safety distance to drive safely at high speeds.
Notably, while prior work has primarily focused on building driving simulators with realistic sensors that mimic LiDARs and cameras (Dosovitskiy et al., 2017; Manivasagam et al., 2020; Yang et al., 2020; Bewley et al., 2018), we focus on the high-level design choices for the simulator – such as the deﬁnition of reward and perception noise – that determine if agents trained in the simulator exhibit realistic behaviors .
We hope that the lessons in state space, action space, and reward design gleaned from this paper will transfer to simulators in which the prototypes for perception and interaction used in this paper are replaced with more sophisticated sensor simulation.
Code and Documentation for all experiments presented in this paper can be found in our Project Page 1 .
Our main contributions are as follows:
• We deﬁne a multi-agent driving environment in which agents equipped with noisy LiDAR sensors are rewarded for reaching a given destination as quickly as possible without colliding with other agents and show that agents trained in this environment learn road rules that mimic road rules common in human driving systems.
• We analyze what choices in the deﬁnition of the MDP lead to the emergence of these road rules and ﬁnd that the most important factors are perception noise and the spatial density of agents in the driving environment.
• We release a suite of 2D driving environments 2 with the intention of stimulating interest within the MARL community to solve fundamental self-driving problems.
R ELATED W ORKS
Reinforcement Learning Deep Reinforcement Learning (DeepRL) has become an popular framework that has been successfully used to solve Atari (Mnih et al., 2013), Strategy Games (Peng et al., 2017; OpenAI, 2018), and Trafﬁc Control (Wu et al., 2017a;
Belletti et al., 2018).
Vanilla Policy
Gradient (Sutton et al., 2000) is an algorithm that optimizes an agent’s policy by using monte-carlo estimates of the expected return.
Proximal Policy Optimization (Schulman et al., 2017) – which we use in this work – is an on-policy policy gradient algorithm that alternately samples from the environment and optimizes the policy using stochastic gradient descent.
PPO stabilizes the Actor’s training by limiting the step size of the policy update using a clipped surrogate objective function.
Multi-Agent Reinforcement Learning The central difﬁculties of Multi-Agent RL (MARL) include environment non-stationarity (Hernandez-Leal et al., 2019; 2017; Busoniu et al., 2008; Shoham et al., 2007), credit assignment (Agogino and Tumer, 2004; Wolpert and Tumer, 2002), and the curse of dimensionality (Busoniu et al., 2008; Shoham et al., 2007).
Recent works (Son et al., 2019; Rashid et al., 2018) have attempted to solve these issues in a centralized training decentralized execution framework by factorizing the joint action-value Q function into individual Q functions for each agent.
Alternatively, MADDPG (Lowe et al., 2017) and PPO with Centralized Critic (Baker et al., 2019) have also shown promising results in dealing with MARL Problems using policy gradients.
Emergent Behavior Emergence of behavior that appears human-like in MARL (Leibo et al., 2019) has been studied extensively for problems like effective tool usage (Baker et al., 2019), ball passing and interception in 3D soccer environments (Liu et al., 2019), capture the ﬂag (Jaderberg et al., 2019), hide and seek (Chen et al., 2020; Baker et al., 2019), communication (Foerster et al., 2016;
Sukhbaatar et al., 2016), and role assignment (Wang et al., 2020).
For autonomous driving and trafﬁc control (Sykora et al., 2020), emergent behavior has primarily been studied in the context of imitation learning (Bojarski et al., 2016; Zeng et al., 2019; Bansal et al., 2018; Philion and Fidler, 2020; Bhattacharyya et al., 2019). Zhou et al.
(2020) solve a similar problem as ours from the perspective of environment design but fail to account for real-world aspects like noisy perception, which are inherent for emergent rules.
In contrast to works that study emergent behavior in mixed-trafﬁc autonomy (Wu et al., 2017b), embedded rules in reward functions (Medvet et al., 2017; Talamini et al., 2020) and trafﬁc signal control (Stevens and Yeh, 2016), we consider a fully autonomous driving problem in a decentralized execution framework and show the emergence of standard trafﬁc rules that are present in transportation infrastructure.
P ROBLEM S ETTING
We frame the task of driving as a discrete time Multi-Agent Dec-POMDP (Oliehoek et al., 2016).
Formally, a Dec-POMDP is a tuple ρ n
[-=
 o i ∈ O , γ ∈ (0 , 1] is the discount factor, and T is the time horizon.-] We parameterize the policy π θ : o × a (cid:55)→ R + of the agents using a neural network with parameters θ .
In all our experiments, the agents share a common policy network.
Let the expected return for the i th agent be η
i ( π θ ) =
E τ (cid:104)(cid:80) T − 1 t =
γ t r i,t ( s t , a i,t ) (cid:105) , where τ = ( s 0 , a i, 0 , . . .
, s T − 1 , a i,T − 1 ) is the trajectory of the agent, s 0 ∼ ρ 0 , a i,t ∼ π θ ( a i,t | o i,t ) , and s t +1 ∼ P ( s t +1 | s t , (cid:83) ni
=
a i,t ) .
Our objective is to ﬁnd the optimal policy which maximizes the utilitarian objective (cid:80) η i .
Reward We use high-level generic rewards and avoid any extensive reward engineering.
The agents receive a reward of +1 if they successfully reach their given destination and -1 if they collide with any other agent or go off the road.
In the event of a collision, the simulation for that agent halts.
In an inter-agent collision, we penalize both agents equally without attempting to determine which agent was responsible for the collision.
We additionally regularize the actions of the agents to encourage smooth actions and add a normalized penalty proportional to the longitudinal distance of the agent from the destination to encourage speed.
We ensure that the un-discounted sum of each component of the reward for an agent over the entire trajectory is bounded 0 ≤ r ≤ 1 .
We combine all the components to model the reward r i,t received by agent i at timestep t as follows:
[Equation]
i
[Equation]
Reached the Destination at timestep t ] −
[Equation]
[ Collided for the ﬁrst time with an obstacle/agent at timesetep t ]
[Equation]
[Table]
where ( a i ) max is the maximum magnitude of action that agent i can take
Map and Goal Representation
We use multiple environments: four-way intersection, highway tracks, and real-world road patches from nuScenes (Caesar et al., 2019), to train the agents.
The initial state distribution ρ 0 is deﬁned by the drivable area in the base environment.
The agents “sense” a trafﬁc signal if they are near a trafﬁc signal and facing the trafﬁc signal.
These signals are represented as discrete values – 0, 0.5, 1.0 for the 3 signals and 0.75 for no signal available – in the observation space.
In all but our communication experiments, agents have the ability to communicate exclusively through the motion that other agents observe.
In our communication experiments, we open a discrete communication channel designed to mimic turn signals and discuss the direct impact on agent behavior.
Additionally, to mimic a satnav dashboard, the agents observe the distance from their goal, the angular deviation of the goal from the current heading, and the current speed.
LiDAR observations We simulate a LiDAR sensor for each agent by calculating the distance to the closest object – dynamic or static – along each of n equi-angular rays.
We restrict the range of the
LiDAR to be 50m. Human eyes themselves are imperfect sensors and are easily thwarted by weather, glare, or visual distractions; in our experiments, we study the importance of this “visual" sensor by introducing noise in the sensor.
We introduce x % lidar noise by random uniformly dropping (assigning a value of 0) x
% of the rays at every timestep (Manivasagam et al., 2020).
To give agents the capacity to infer the velocity and acceleration of nearby vehicles, we concatenate the LiDAR observations from T = 5 past timesteps.
P OLICY O PTIMIZATION 4. P OLICY P ARAMETERIZATION
In our experiments we consider the following two parameterizations for our policy network(s):
Fixed Track Model : We optimize policies that output a Multinomial Distribution over a ﬁxed set of discretized acceleration values.
This distribution is deﬁned by π φ ( a | o ) , where π φ is our policy network, a is the acceleration, and o is the observation.
This acceleration is used to drive the vehicle along
a
ﬁxed trajectory from the
source to target destination This model trains efﬁciently
but
precludes the emergence of lanes
Spline Model :
To
train agents that are capable of discovering lanes, we use a two -stage formulation inspired by Zeng et al.
(2019) in which
trajetories shapes are represented by clothoids and time-dependence is represented by a velocity proﬁle.
Our overall policy is factored into two “subpolicies" – a spline subpolicy and an acceleration subpolicy.
The spline subpolicy is tasked with predicting the spline along which the vehicle is supposed to be driven.
This subpolicy conditions on an initial local observation of the environment and predicts the spline (see Section 5).
We use a Centripetal Catmull Rom Spline (Catmull and
Rom, 1974) to parameterize this spatial path.
The acceleration subpolicy follows the same parameterization from Fixed Track Model, and controls the agent’s motion along this spline.
We formalize the training algorithm for this bilevel problem in Section 4.4.
Note that the ﬁxed track model is a special case of the spline model parameterization, where the spline is hard-coded into the environment.
These splines can be extracted from lane graphs such as those found in the HD maps provided by the nuScenes dataset (Caesar et al., 2019).
4. P ROXIMAL P OLICY O PTIMIZATION USING C ENTRALIZED C RITIC
We consider a Centralized Training with Decentralized Execution approach in our experiments.
During training, the critic has access to all the agents’ observations while the actors only see the local observations.
We use Proximal Policy Optimization (PPO)
(Schulman et al., 2017) and Generalized
Advantage Estimation (GAE) (Schulman et al., 2015) to train our agents.
Let V φ denote the value function.
To train our agent, we optimize the following objective:
[Equation]
where ˜ r ( φ ) =
π φ ( a | o ) π φold ( a | o ) , ˆ A is the Estimated Advantage, H ( · ) measures entropy, { (cid:15), c 1 , c 2 } are hyperparameters, and V target is the value estimate recorded during simulation.
Training is performed using a custom adaptation of SpinningUp (Achiam, 2018) for MARL and Horovod (Sergeev and
Balso, 2018).
The agents share a common policy network in all the reported experiments.
In our environments, the number of agents present can vary over time, as vehicles reach their destinations and new agents spawn.
To enforce permutation invariance across the dynamic pool of agents, the centralized critic takes as input the mean of the latent vector obtained from all the observations. Result:
Trained Subpolicies π θ and π φ π θ ← Spline Subpolicy, π φ ← Acceleration Subpolicy, V φ ← Value Function for Acceleration Control; for i = 1 . . .
N do /* Given π φ optimize π θ */ for k = 1 . . . K 1 do Collect set of Partial Trajectories D 1 ,k using a s ∼ π θ ( o s ) and a a ← arg max π φ ( a | o a ) ;
Compute and store the normalized rewards R ; end
Optimize the parameters θ using the objective L 2 ( θ ) and stored trajectories D 1 ; / * Given π θ φ vf φ */ for k = 1 . . .
K 2 do Collect set of Partial Trajectories D 2 ,k using a s ← arg max π θ ( a | o s ) and a a ∼ π φ ( o a ) ; Compute and store the advantage estimates ˆ A using GAE; end Optimize the parameters φ using the objective L 2 ( φ ) and stored trajectories D 2 ; end
4. S INGLE -S TEP P ROXIMAL P OLICY O PTIMIZATION
In a single-step MDP, the expected return modelled by the critic is equal to the reward from the environment as there are no future timesteps.
Hence, optimizing the critic is unnecessary in this context.
Let R t denote the normalized reward.
The objective function deﬁned in Sec 4.2 reduces to:
[Equation]
4. B ILEVEL O PTIMIZATION FOR J OINT T RAINING OF S PLINE /A CCELERATION S UBPOLICIES
In this section, we present the algorithm we use to jointly train two RL subpolicies where one subpolicy operates in a single step and the other operates over a time horizon T ≥ 1 .
The subpolicies operate oblivious of each other, and cannot interact directly.
The reward for the spline subpolicy is the undiscounted sum of the rewards received by the acceleration subpolicy over the time horizon.
Pseudocode is provided in Algorithm 1.
The algorithm runs for a total of N iterations.
For each iteration, we collect K 1 and K 2 samples from the environment to train the spline and acceleration subpolicy respectively.
We denote actions and observations using ( a s , o s ) and ( a a , o a ) (not to be confused with the notation used in Section 3) for the spline and acceleration subpolicy respectively.
E MERGENT S OCIAL D RIVING B EHAVIOR
We describe the various social driving behaviors that emerge from our experiments and analyze them quantitatively.
Experiments that do not require an explicit emergence of lanes – 5.1, 5.3, 5.4, 5.7 – use the ﬁxed track model.
We use the Spline Model for modeling lane emergence in Experiments 5.
and 5.5.
The results in Section 5.6 are a consequence of the increased number of training agents [Figure] and are observed using either agent type.
Qualitative rollouts are provided in our project page.
[Figure]
correlation exists between the acceleration, trafﬁc signal and distance from the intersection.
As the agent approaches the intersection, the effect of red signal on the actions is more prominent (characterized by the reduced variance).
However, the variance for green signal increasing, since agents need to marginally slow down upon detecting an agent in front of them.
With more
LiDAR noise, detecting the location of the intersection becomes diffcult so the agents prematurely slow down far from the intersection.
Agents with better visibility can potentially safely cross the intersection on a red light, hence the mean acceleration near the intersection goes down with increasing lidar noise.
Finally, with increasing number of agents, the variability of the actions increase due to presence of leading vehicles on the same path.
[Figure]
[Figure]
5. S TOPPING AT A T RAFFIC S IGNAL
Trafﬁc Signals are used to impose an ordering on trafﬁc ﬂow direction in busy 4-way intersections.
We simulate a 4-way intersection, where agents need to reach the opposite road pocket in minimum time.
We constrain the agents to move along a straight line path.
We employ the ﬁxed track model and agents learn to control their accelerations to reach their destination.
We study the agent behaviors by varying the number of training agents and their perception noise (Figure 2 & 3).
Note that the agents merely observe a ternary value representing the trafﬁc light’s state, not color.
To make the plots in this section
[Figure]
we visually inspect rollouts for each converged policy to ﬁnd a permutation of the ternary states that align with human red/yellow/green trafﬁc light conventions
[Figure]
low density
[Table]
density
[Table]
[Table]
[Table]
5. E MERGENCE OF L ANES
To analyze lane emergence, we relax the constraint on the ﬁxed agent paths in the setup of Section 5.1.
We use the Bilevel PPO Algorithm (Section 4.4) to train the two subpolicies.
The spline subpolicy predicts a deviation from a path along the roads’ central axis connecting the start position to the destination, similar to the GPS navigation maps used by human drivers.
The acceleration subpolicy uses the same formulation as Section 5.1.
To empirically analyze the emergence of lanes, we plot the “Normalized Lane Position" of the agents over time.
“Normalized Lane Position" is the directional deviation of agent from the road axis.
We consider the right side of the road (in the ego frame) to have a positive “Normalized Lane Position".
Figures 4 & 5 show the variation with lidar noise and number of training agents respectively.
Figureshows the spatial positions of the agents for the 8 agent perfect perception environment.
5. R IGHT OF W AY
For tasks that can be performed simultaneously and take approximately equal time for completion,
First In First Out (FIFO) scheduling strategy minimizes the average waiting time.
In the context of driving through an intersection where each new agent symbolizes a new task, the agent that arrives ﬁrst at the intersection should also be able to leave the intersection ﬁrst.
In other words, given any two vehicles, the vehicle arriving at the intersection ﬁrst has the “right of way" over the other vehicle.
Let the time at which agent i ∈ [ n ] arrives at the intersection be ( t a ) i and leaves the intersection be ( t d ) i .
If ∃ j ∈ [ n ] \ { i } , such that ( t a ) i < ( t a ) j and ( t d ) i > ( t d ) j , we say that agent i doesn’t respect j ’s right of way.
We evaluate this metric on a model trained on a nuScenes intersection (Figure 8).
We observe that, at convergence, the agents follow this right of way 85 .
25 ± 8 .
9 % of time (Figure 7).
[Figure] [Figure]
LiDAR [Figure] [Figure] pea k e r C on s i s t en cy 0.
[Figure]
[Figure] [Figure] | P
[Figure]
r s
[Figure]
C o e ff i c
i en t |
[Figure]
5. C OMMUNICATION
One way to safely traverse an intersection is to signal one’s intention to nearby vehicles.
We analyze the impact of perception noise on emergent communication at an intersection (Figure 8).
In particular, we measure the Speaker Consistency (SC), proposed in Jaques et al.
SC can be considered as the mutual information between an agent’s message and its future action.
We report the mutual information and the Pearson coefﬁcient (Freedman et al., 2007) between the agent’s heading and its sent message.
We limit the communication channel to one bit for simplicity.
Each car only receives a signal from the car in the front within − 30 ◦ and 30 ◦ .
Figure 9 shows that agents rely more heavily on communication at intersections when perception becomes less reliable
5. F AST L ANES ON A H IGHWAY
Highways have dedicated fast lanes to allow a smooth ﬂow of trafﬁc.
In this experiment, we empirically show that autonomous vehicles exhibit a similar behavior of forming fast lanes while moving on a highway when trained to maximize trafﬁc ﬂow.
We consider a straight road with a uni-directional ﬂow of trafﬁc.
Agents are spawned at random positions along the road’s axis.
A cc e l e r a t i o n R a t i n g
Every agent is assigned a scalar value called “Acceleration Rating," which scales the agent’s acceleration and velocity limits.
Thus, a higher acceleration rating implies a faster car.
The spline subpolicy predicts an optimal spline considering this acceleration rating.
Even though the agents can decide to move straight by design, it is clearly not an optimal choice as slower cars in front will hinder smooth trafﬁc ﬂow.
Figure 10 shows that agents are segregated into different lanes based on their
Acceleration Rating.
Figure 11 visualizes this behavior over time.
[Figure]
[Figure]
In this task, we evaluate the extent to which the agents learn to respect a minimum safety distance between agents while driving.
When agents are too close, they are at a greater risk of colliding; when agents are too far, they are not as efﬁcient traveling a → b .
To derive a human-like “safety distance”, we assume agents can change their velocity according to v
= u 2 + 2 ad , where v and u are the ﬁnal and initial velocities respectively, a is acceleration, and d is the distance for which the acceleration remains constant.
Hence, for an agent to stop entirely from a state with velocity v 0 , it needs at least a distance of v [Figure] a max in front of it, where a max is the maximum possible deceleration of the agent.
Our agents perceive the environment through LiDAR; thus, agents can estimate nearby agents’ velocity and acceleration.
We deﬁne the safe distance as the distance needed for a trailing agent to have a zero velocity in the leading agent’s frame.
We assume that the leading agent travels with a constant velocity, and as such, the safe distance is deﬁned by ∆ s 2 2 a max , where ∆ s is the relative velocity.
Any car having a distance greater than this can safely slow down.
Agents trained on nuScenes
Intersections obey this safety distance around 98.45% time (Figure 12).
[Figure]
S LOWING DOWN NEAR A C ROSSWALK
In this task, we evaluate if agents can detect pedestrians and slow down in their presence.
We augment the environment setup of Sec. 5.5, to include a crosswalk where at most 10 pedestrians are spawned at the start of every rollout.
The pedestrians cross the road with a constant velocity.
If any agent collides with a pedestrian, they get a collision reward of -1, and the simulation for that agent stops.
The KDE plots in Figure 13 show that the agents indeed detect the pedestrians, and most of them maintain a distance greater than 6m. To determine if the agents can safely stop and prevent collision with the pedestrians, we calculate a safe stopping distance of s
[Figure]
a max , where s is the velocity of the agent.
In the scatter plot, we observe that most agents adhere to this minimum distance and drive at a distance, which lies in the safe driving region.
D ISCUSSION
[Figure]
6. S TATISTICS OF H UMAN D RIVING
In some cases, the same statistics accumulated over agent trajectories that we use in Section 5 to quantitatively demonstrate emergence can also be accumulated over the human driving trajectories labeled in the nuScenes dataset.
In Figure 14, we visualize safety distance statistics across nuScenes trajectories and safety distance statistics across RL agents trained on nuScenes intersections sideby-side.
The nuScenes trainval split contains 64386 car instance labels, each with an associated trajectory.
For each location along the trajectory, we calculate the safety distance as described in
Section 5.6.
The same computation is performed over RL agent trajectories.
The agents trained in our
MDP tend to violate the safety distance more than human drivers.
However, in both cases, a safety distance is observed the vast majority of the time (green triangular region).
6. F UTURE W ORK
By parameterizing policies such that agents must follow the curve generated by the spline subpolicy at initialization (see Section 4.4), we prevent lane change behavior from emerging.
The use of a more expressive action space should address this limitation at the cost of training time.
Additionally, the fact that our reward is primarily based on agents reaching destinations means that convergence is slow on maps that are orders of magnitude larger than the vehicles’ dimensions.
One possible solution to training agents to navigate large maps would be to generate a curriculum of target destinations, as in
Mirowski et al.
C ONCLUSION
In this paper, we identify a lightweight multi-agent MDP that empirically captures the driving problem’s essential features.
We equip our agents with a sparse LiDAR sensor and reward agents when they reach their assigned target destination as quickly as possible without colliding with other agents in the scene.
We observe that agents in this setting rely on a shared notion of lanes and trafﬁc lights to compensate for their noisy perception.
We believe that dense multi-agent interaction and perception noise are critical ingredients in designing simulators that seek to instill human-like road rules in self-driving agents.  A CKNOWLEDGEMENT
This work was supported by NSERC and DARPA’s XAI program.
SF acknowledges the Canada
CIFAR AI Chair award at the Vector Institute.
AP acknowledges support by the Vector Institute.