MTL IGHT : E FFICIENT M ULTI -T ASK R EINFORCEMENT L EARNING FOR T RAFFIC S IGNAL C ONTROL
A BSTRACT
Traffic signal control has a great impact on alleviating traffic congestion in modern cities.
Deep reinforcement learning (RL) has been widely used for this task in recent years, demonstrating promising performance but also facing many challenges such as limited performances and sample inefficiency.
To handle these challenges, MTL IGHT is proposed to enhance the agent observation with a latent state, which is learned from numerous traffic indicators.
Meanwhile, multiple auxiliary and supervisory tasks are constructed to learn the latent state, and two types of embedding latent features, the task-specific feature and task-shared feature, are used to make the latent state more abundant.
Extensive experiments conducted on
CityFlow demonstrate that MTL IGHT has leading convergence speed and asymptotic performance.
We further simulate under peak-hour pattern in all scenarios with increasing control difficulty and the results indicate that MTL IGHT is highly adaptable.
INTRODUCTION
Traffic signal control aims to coordinate traffic signals across intersections to improve the traffic efficiency of a district or a city, which plays an important role in efficient transportation.
Most conventional methods aim to control traffic signals by fixed-time Koonce & Rodegerdts (2008) or hand-crafted heuristics Kouvelas et al.
(2014), which heavily rely on expert knowledge and in-depth excavation of regional historical traffic, making it difficult to migrate.
Recently, deep reinforcement learning (DRL) based methods Guo et al.
; Jintao et al.
(2020); Pan et al.
; He & Shin (2020);
Tong et al.
; Wang et al.
Gu et al.
; Liu et al.
; Xu et al.
Zhang et al.
(2021) employ a deep neural network to control an intersection where the network is learned by directly interacting with the environment.
However, due to the plenty of traffic indicators (number of vehicles, queue length, waiting time, speed, etc.), complex observation and the dynamic environment, the problem is challenging and remains unsolved.
Since the observation, reward and dynamics of each traffic signal are closely related to others, hence optimizing traffic signal control in a large-scale road network is naturally modeled as a multi-agent reinforcement learning (MARL) problem.
Most exiting works Wei et al.
(2019a)
; Zhang et al.
(2020b)
; Chen et al.
; Zheng et al.
(2019a) are proposed to learn the policy of each agent only conditioned on the raw observations of the intersection, while ignoring the help of the global state, which is accessible in smart city.
As stated in Zheng et al. (2019b), different metrics have a considerable impact on the traffic signal control task.
Hence, the observation design of agent should not only involve the raw observations of the intersection, but also the global state.
A good agent observation design could make full use of samples, and improves not only the policy performance but also the sample efficiency.
However, there are a huge amount of traffic indicators or metrics in the global state, and it is hard to subjectively design suited and non-redundant agent observation among these indicators.
On one hand, an overly concise observation design could not adequately and comprehensively represent the state characteristics and therefore affects the accuracy of the estimation of state transition and as well as influencing action selection.
In contrast, if an overly complex combination of metrics is used as an observation, the weights of different metrics are difficult to precisely define, and it may cause data redundancy and dimension explosion, which will not only increase the computational consumption, but also make the agent hard to learn.
[Table]
/fp1nm2zWn7hSAi8QtSQ2UaPn212AocBIRrjFDSvVdJ9ZeiqSmmJGsOkgUiRGeoBHpG8pRRJSXFhdk8MQoQxgKaR7XsFB/d6QoUmoaBaYy31PNe7n4n9dPdHjppZTHiSYczwaFCYNawDwOOKSSYM2mhiAsqdkV4jGSCGsTWtWE4M6fvEg6Z3X3vN64bdSaV2UcFXAEjsEpcMEFaIIb0AJtgMEjeAav4M16sl6sd+tjVrpklT0H4A+szx/yzpfb</latexit>
o t
[Figure]
In order to provide an adequate representation of the traffic signal control task, the latent state is introduced.
Specifically, the raw observation is identical to the intersection, which consists of several variables with concrete semantic meanings (i.e., the number of vehicles on each incoming lane and current signal phase).
Then, the raw observation is enhanced by the latent space.
To learn the latent space from the global state, multiple auxiliary and supervisory tasks are constructed, which are related to traffic signal control.
That is, several statistics of global state history are taken as inputs, a RNN-based network is employed firstly, and several branches are introduced subsequently to predict multiple types of statistics of the global state, such as the flow distribution and the travel time distribution, respectively.
To make the latent space more abundant, two types of embedding features are extracted: the task-specific feature and task-shared feature.
The former is extracted by the task-specific branch and represents the task-driven information, while the later is from the task-shared layer and could express more general underlying characteristics.
Hence, they are complementary to each other and are both used to enhance the raw observation.
Finally, conditioned on the enhanced observation, the policy is learned by DRL Mnih et al.
Note that the multiple tasks are learned simultaneously with the DRL, which makes the latent space more adaptive to the policy learning.
P ROBLEM S TATEMENT 2. P ROBLEM D EFINITION
We consider a multi-agent traffic signal control problem, the task is modeled as a Markov Game
Littman (1994), which can be denoted by a tuple G = < N , S , A , O , P , R , H , γ > .
N ≡ { 1 , . . . , n } is a finite set of agents, and each intersection in the scenario is controlled by an agent.
S is a finite set of global state space.
A denotes the action space for an individual agent.
The joint action a ∈
A ≡ A n is a collection of individual actions [ a i ] ni =1 .
At each timestep, each agent i receives an observation o i ∈ O , selects an action a i , results in the next state s ′ according to the transition function P ( s ′ | s, a ) and a reward r =
R
( s, a )
for each agent.
H is the time horizon and γ ∈ [0 , 1) is the discount factor.
2. A GENT D ESIGN
Each intersection in the system is controlled by an agent.
In the following, we introduce the state design, action design and reward design of the RL agent.
• Observation.
Our primitive observation consists of two parts: (1) the number of vehicles on each incoming lane f vt ; (2) current signal phase f st .
Both of them can be obtained directly from the simulator, the concepts are described in detail in Section B.4.
The raw observation of agent i is defined by
[Equation]
where f vt =
{ V l in 1 , V l in 2 , . . .
, V l inm } and l in = { l in 1 , . . .
, l inm } is a finite set of incoming lanes in the intersection.
Current signal phase f st = p k , k ∈ 1 , . .
, K , and K is the total number of phases.
Each phase p is represented as a one-hot vector.
Our goal is to learn latent space to enhance the raw observation to make better use of the sample.
[Table]
/7d/kk7d8Y6cxkWj1qzP4yiDKtgDB8ADx6AJzkELtAEGt+AePIIn6856sJ6tl1lpyZr37IJvsF4/ALzPmWk=</latexit> o spet
• Action.
The action of each agent is to choose the phase for the next time interval.
Note that the phases may organize in a sequential way in reality, while directly selecting a phase makes the traffic control plan more flexible.
Action of agent i is defined by
[Equation]
where f st = p k , k ∈ 1 , . .
, K .
• Reward.
We define the reward as the negative of the queue length on incoming lanes, which is generally accepted and reasonable in previous work Zheng et al.
(2019b)
; Huang et al.
; Zang et al.
; Zheng et al. (2019a); Wei et al. (2019b).
Reward of agent i is defined by
[Equation]
where q l inm is the queue length on incoming lane l inm .
METHOD
In this section, we will introduce the main modules of our proposed method MTL IGHT , which focuses on learning task-related task-shared latent state and task-specific latent state by introducing an auxiliary Multi-Task network to help policy learning.
The whole process of MTL IGHT is described in Algorithm 1, and the framework of MTL IGHT is shown in Fig.
MTL IGHT consists of a Multi-Task network and an agent network.
For the latter, Deep Q-Network (DQN) Mnih et al.
(2015) is employed as function approximator to estimate the Q-value function, which is consistent with the previous methods Chen et al.
Wei et al.
(2019b;a); Zheng et al.
(2019a)
; Wei et al.
The Multi-Task module adopts a hard parameter sharing paradigm Caruana (1997), which generally applied by sharing the hidden layers between all tasks, while keeping several task-specific output layers.
3. M ULTI -T ASK L EARNING FOR L ATENT S TATE
For each agent, its raw observation includes the number of vehicles f vt and the current signal phase f st .
Besides, several information from the global state is given, such as: the number of incoming cars in the last τ steps, denoted as f ct − τ : t =
[ f ct − τ , f ct − τ +1 , . . .
, f ct ] , the average travel time during the past τ steps, denoted as f trt − τ : t =
[ f trt − τ , f trt − τ +1 , . . .
, f trt ] , the queue length during the past τ steps, denoted as f qt − τ : t =
[ f qt − τ , f qt − τ +1 , . . . , f qt ] , the current vehicles during the past τ steps, which is denoted as f vrt − τ : t =
[ f vrt − τ , f vrt − τ +1 , . . .
, f vrt ] .
The Multi-Task module includes the following four tasks:
Flow distribution approximation.
We use T flow to denote the traffic distribution estimation task, i.e., to predict the mean µ f and variance σ 2 f of flow arrival rate from start up to the time step t .
The task could be denoted as:
[Equation]
Travel time distribution approximation.
We use T travel to denote the travel distribution estimation task, i e., to predict the mean µ tr and variance σ 2 tr of average travel time of vehicles that have completed the trip from start up to the time step t :
[Equation]
Next queue length approximation.
We use T queue to denote the next queue length estimation task, i
e., to predict the average number q of vehicles in queue at the next step:
[Equation]
Vehicles on the road approximation.
We use T vehicles to denote the vehicles on the road approximation task, i.e., to predict the number of vehicles V r existing in the system:
[Equation]
Note that vehicles that have completed the trips or have not yet entered the road network do not belong to these.
The above tasks act auxiliary tasks to learn the latent space.
Since the numbers of f ct − τ : t , f trt − τ : t , f q t − τ : t , f vr t − τ : t have different scales and their dimensions are different with f v t and f s t , four independent linear layers and ReLU functions are employed firstly to scale them respectively:
[Equation]
Then a linear layer and ReLU function is used to calculate the hidden state after concatenating all embedded inputs:
[Equation]
Based on H t , a task-shared network module is used to generate its task-shared latent feature ( o shrt , also called apparent state ).
Then, 4 independent branches are introduced for each task and calculate task-specific latent feature ( o spet , also called mental state ) from o shrt .
The specific implementation of network architecture is listed in the supplementary.
We use a single latent variable model to extract hierarchical latent features, which follows insights by Zhao et al.
That is, the mental state is output of the shared-layer after GRU in MultiTask network and could express more general underlying characteristics.
In contrast, the apparent state is the the concatenation of the output of the task-specific layer and represents the task-driven information.
In other words, the mental state is more coarse-grained, while apparent state is more fine-grained.
Hence, they are complementary to each other and both used in our method.
3. P OLICY WITH L ATENT S TATE
With the help of latent state, the agent observation is enhanced from o t to ( o t , o shr t , o spet ) .
For the policy π θ , the objective is to maximize the cumulative reward:
[Equation]
[Equation]
An agent that maximises Eq.
11 acts optimally under uncertainty and is called Bayes-optimal
Ghavamzadeh et al.
(2015), assuming we treat the knowledge over related tasks as our epistemic prior about the environment.
Multi-Task module minimizes the complexity of the model and give informative priors to the model.
Besides, it can minimize the representation bias in a way that push the learning algorithm to find a solution on a smaller area of representations on the intersection rather than on a large area of a single task.
This incentivises a faster and better convergence.
EXPERIMENT
We conduct the experiments on CityFlow Zhang et al.
(2019), an city-level open-source simulation platform for traffic signal control.
The simulator is used as the environment to provide state for traffic
[Figure]
signal control
, the agents execute actions by
Please refer to Appendix D.1 and Appendix D.2 for the detailed settings of road network and traffic flow configuration.
Baselines are described in detail in Appendix F.
4. P ERFORMANCE C OMPARISON
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Figure]
under real configurations.
Tab. 1 lists the comparative results, and it is evident that: 1) In general, RL methods perform better than conventional methods, and it indicates the advantage of the
RL.
Moreover, MTL IGHT is outperforms other methods in almost all cities and flow configurations, which demonstrates the effectiveness of the method.
2) MTL IGHT shows good generalization for different scenarios and configurations.
For example, M AX P RESSURE performs well in D Hangzhou with the Real , while under the
Synthetic traffic conditions, M AX P RESSURE shows significantly worse than other methods.
In contrast, MTL IGHT can not only achieve good performance under diverse configurations of D Hangzhou , but also shows great stability.
3) MTL IGHT outperforms I NDIVIDUAL RL,
M ETA L IGHT and P RESS L IGHT with 693.46, 461.80 and 432.38, respectively.
The reason is that they learn the traffic light’s policy only using its observation and ignore the influence of the neighbors, while
MTL IGHT considers the neighbors as the latent part of the environment to help learning.
The neighbor’s information is modeled in C O L IGHT and G ENERA L IGHT can adapt to a variety of flows, they both perform well.
While results of MTL IGHT is superiors to them in multiple scenarios, result- ing mean 42.5 and 398 improvement.
Compared to them, MTL IGHT benefits from prior knowledge learned from Multi-Task network to make more accurate decisions.
Fig.
4 shows the performances of all RL methods of D Hangzhou under Real traffic pattern, and it is obvious that MTL IGHT converges faster and has better asymptotic performance.
Fig.
5 shows the performances of all RL methods of D Hangzhou under Synthetic traffic pattern, we can conclude that
MTL IGHT converges quickly and learns effectively during the peak hour, while the other method have only a weak boost during training.
[Figure]
Fig. 8 and Tab. 5 illustrates the turning statistics of vehicle routes.
Take D Hangzhou Real as an example, the frequency of turning left and going straight is 14% and 86% respectively (turning right are not considered because they are free from the control by lights).
Fig.shows the percentage of each phase of RL methods, we can find: 1)
The total left-turn phase of MTL IGHT accounts for 15.3%, which is highly consistent with the leftturn frequency of 14%, which indicates that the strategy is interpretable.
The G ENERA L IGHT left-turn ratio of 10.9% is also close, but because it has an excessive proportion of straight phases, it may cause left-turn vehicles to be stranded, resulting in increased travel time.
3) I NDIVIDUAL RL tends to consider phase 1 and 2, which account for as much as 65.9%, M ETA L IGHT prefers to go straight, P RESS L IGHT is eccentric to phase 1, and C O LIGHT assigns a relatively even distribution to each phase, rather than aligning with the traffic flow direction.
These all demonstrate the limitations of other RL methods in multi-agent environments, while MTL IGHT can learn more stable strategies by introducing task-shared and task-specific latent states.
4. A BLATIONS
To better validate the contribution of each component, three variants of MTL IGHT are evaluated under a variety of scenarios, as shown in Tab. 1.
• B ASE only keeps the policy network and removes the Multi-Task network.
• B ASE + RAW only keeps the policy network and discards Multi-Task network, but directly uses the original input of Multi-Task module as part of the observation.
• B ASE + SHR retains the Multi-Task network and the policy, but only has task-shared latent state and removes task-specific latent state.
• B ASE + SPE retains the Multi-Task network and the policy.
In contrast to B ASE + SHR , B ASE + SPE has only the task-specific latent state and removes the task-shared latent state.
Note that MTL IGHT contains the whole modules: policy network, Multi-Task network with both task-specific latent state and task-shared latent state.
The quantitative evaluation results are presented in Tab. 1.
We can obtain the following findings: 1)
Among these 4 models, the performance of B ASE is the worst.
The reason is that it is hard to learn the effective policy independently in the multi-agent traffic signal control task, where the surrounding environment is changing dynamically, but B ASE has no sense of it. 2)
Compared with the B ASE and B ASE + RAW , the improvement of B ASE + SHR and B ASE + SPE demonstrate the effectiveness of the task-shared latent state o shrt and task-specific latent state o spet respectively.
o shrt reflects prior information that is constant over time with multiple related tasks , o spet reflects prior information that is align with the latest changing trends, both of them help policy to make Bayesian optimal decisions.
The o shrt and o spet are both effective because each of them is an efficient representations of environmental features.
Compared to them, the superiority of MTL IGHT indicates o shrt and o spet are complementary to each other.
Overall, all of the proposed components contribute positively to the final results.
CONCLUSION
We introduced MTL IGHT , an efficient Multi-Task reinforcement learning method for traffic signal control that can be scaled to complex multi-agent urban road networks of different scale.
We showed that MTL IGHT ’s latent structure learns a hierarchical latent representations of related tasks, separating the task-shared and task-specific latent states.
On several cities’ datasets we demonstrated that this latent representation inspired from related multiple tasks and conditioning the policy on it, allows an agent to adapt to the complex environment.
We conclude that maintaining prior approximations over related tasks helps compared to model-free approaches, especially when there is too much information in the environment and it cannot be fully expressed by artificial state design.
For the future, the latent prior could be learned from expert data prepared in advance using imitation learning techniques Song et al.
(2018) or by using existing multi-agent algorithms to pre-train MultiTask network
A A PPENDIX
You may include other additional sections here.
[Table]
B RELATED WORK B. C ONVENTIONAL AND A DAPTIVE T RAFFIC S IGNAL C ONTROL
Most conventional traffic signal control methods are designed based on fixed-time signal control
Webster (1958), actuated control Chiu (1992) or self-organizing traffic signal control Chiu & Chand (1993); Cools et al.
; Lowrie (1990); Svanes & Delaney (1981); Hunt et al.
These approaches rely on expert knowledge and often perform unsatisfactorily in complicated real-world situations.
To solve this problem, several optimization-based methods Roess et al.
; Varaiya (2013);
Kouvelas et al.
(2014) have been proposed to optimize average travel time, throughput, etc. , which decide the traffic signal plans according to the observed data instead of the human prior.
However, these approaches typically rely on strict assumptions which might not hold in the realworld cases Webster (1966).
Furthermore, the optimization problems are usually hard to tract and require significant computing power in complex scenarios.
B. RL- BASED T RAFFIC S IGNAL C ONTROL
RL-based traffic signal control methods aim to learn the policy from interactions with the environment.
Earlier studies use tabular Q-learning El-Tantawy et al.
; Abdoos et al.
Dusparic & Cahill (2009); Abdoos et al.
(2011) where the states in an environment are required to be discretized and low-dimensional.
To address the unmanageable large or continuous state space, recent advances employ deep RL with more complex continuous state representations (like images or feature vectors) to map the high-dimensional states into actions.
Efforts have been made to design strategies that formulate the task as a single agent Wei et al.
Mannion et al.
; Huang et al.
; Zang et al.
; Oroojlooy et al.
; Jiang et al.
(2021); Rizzo et al.
(2019) or some isolated intersections Zheng et al.
(2019b;a); Xiong et al.
Wei et al. (2019a); Chen et al.
; Oroojlooy et al.
; Zhang et al.
(2020b;a), i.e., each agent makes decision for its own.
The above methods are usually easy to scale, but they may have difficulty achieving globally optimal performance due to a lack of collaboration.
To solve the problem, another way is to consider jointly modeling the action between learning agents with centralized optimization
Van der Pol & Oliehoek (2016); Kuyer et al.
However, as the number of agents increases, joint optimization usually leads to dimensional explosion, which has inhibited the widespread adoption of such methods to a large-scale traffic signal control.
To overcome the difficulty, another type of methods are implemented in a decentralized manner, taking into account the collaboration between neighbors with appropriate reward and state design Arel et al.
; Nishi et al.
; Wei et al. (2019b); Xu et al.
Methods such as El-Tantawy et al.
; Chu et al.
(2019) add neighboring information into states, Nishi et al.
; Wei et al. (2019b); Yu et al.
Guo et al.
(2021) add neighbors’ hidden features into states, and Xu et al.
(2021) optimizes neighborhood travel time as an additional reward.
However, simple concatenation of neighboring information is not reasonable enough because the influence of neighboring intersections is not balanced.
Unlike the above methods that add neighbor information to the state, our method learns task-shared and task-specific latent states by constructing Multi-Task network.
B. M ULTI -T ASK L EARNING
Multi-Task Learning(MTL)
Caruana (1997) is a learning paradigm aims to jointly learn multiple related tasks so that the knowledge contained in a task can be leveraged by other tasks.
Past works
Oh et al.
; Zhang & Yang (2021); Ruder (2017); Ndirango & Lee (2019) have found that, by sharing a representation among related tasks and jointly learning all the tasks, better generalization can be achieved over independently learning each task.
Constructing auxiliary tasks to help the main task is a branch of Multi-Task Learning.
Reinforcement learning is known to be sample inefficient, transferring knowledge from other auxiliary tasks is a powerful tool for improving the learning efficiency Jaderberg et al.
; Lin et al.
; Lyle et al. (2021); Tongloy et al.
; Bellemare et al.
Lin et al.
(2019) combines different auxiliary tasks which provide gradient directions to speed up the training of the main reinforcement learning task.
In comparison, our work aims to transfer knowledge from the task-related auxiliary tasks as a prior to the main reinforcement learning task, to ultimately boost the performance.
Specifically, we model the Multi-Task network as a latent structure where the task-shared latent state is generated from early layers and the task-specific latent state is generated from deeper layers.
This incentivies the policy to learn the Bayers-optimal behaviours: the policy can take into account its uncertainty over the comprehensive information when choosing actions.
B. P RELIMINARIES
In this section, we first introduce some basic concepts related to traffic signal control (TSC) that have been widely recognized in previous work Wei et al.
(2019b)
; Zheng et al. (2019a); Zhang et al.
(2020b);
Wei et al. (2019a); Chen et al.
; Zang et al.
Note that the concepts can be easily generalized to other intersections with different structures.
• Incoming/Outgoing Lanes.
The incoming lanes refer to the lanes where the vehicles are about to enter the intersection.
It usually contains three basic types: ”left-turn”, ”straight” and ”rightturn” from inner to outer.
The outgoing lanes refer to the lanes where the vehicles are about to leave the intersection.
[Figure]
• Roadnet.
A roadnet is a part of a dataset that represents an area of a city.
A roadnet consists of signalized intersections, unsignalized intersections, and lanes connecting the intersections.
Generally, the lane lengths, number of lanes and relative locations of intersections vary from one roadnet to another.
• Phase.
Phase is a controller timing unit associated with the control of one or more movements, representing the permulation and combination of different traffic flows.
The 4-phase setting is the most common configuration in reality, illustrated in Fig. 6, but the number of phases can vary due to different intersection topologies (3-way, 5-way intersections, etc.).
• Queue Length.
Queue length is the number of vehicles waiting at an intersection due to a red light.
Vehicles on the incoming lane with a speed of less than 0.1m/s are considered to be waiting.
• Average Travel Time.
The travel time of a vehicle is the time discrepancy between entering and leaving a particular area.
Average travel time of all vehicles in a road network is the most frequently used measure to evaluate the performance of traffic signal control Wei et al.
(2019b;a); Zhang et al.
(2020b)
; Chen et al.
; Zheng et al. (2019a).
• Flow Distribution.
Flow distribution is the distribution of traffic entering the road network, which is generally expressed by the arrival rate of vehicles, i.e., the volume of traffic entering the road network per unit time.
• Vehicles on Road.
Vehicles on road indicate the running vehicle, i.e
vehicles that have entered the road network and have not reached the end point.
Vehicles on road can represent the real-time load on the road network.
C A LGORITHM
The algorithm is shown in Alg.
D D ATASETS D. R OAD N ETWORKS
The evaluation scenarios come from four real road network maps of different scales, including
Hangzhou (China), Jinan (China), New York (USA) and Shenzhen (China), illustrated in Fig.
The road networks and data of Hangzhou, Jinan and New York are from the public datasets 1 .
The road network map of Shenzhen is made by ourselves which is derived from OpenStreetMap 2 .
The road networks of Jinan and Hangzhou contain 12 and 16 intersections in 4 × 3 and 4 × 4 grids, respectively.
The road network of New York includes 48 intersections in 16 × 3 grid.
The road network of Shenzhen contains 33 intersections, which is not grid compared to other three maps. https://traffic-signal- github The facilitate future research.
2 Initialize B π 3 Initialize π θ
M ϕ
Clean up B π then
Calculate
Train M ϕ then
Collect the average total travel time of all vehicles as criteria
H a n g z h o u _ r e a l
H a n g z h o u _ p e a k J i n a n _ r e a l J i n a n _ p e a k N e w y o r k _ r e
[Figure]
D. F LOW CONFIGURATIONS
We run the experiments under two traffic flow configurations: real traffic flow and synthetic traffic flow.
The real traffic flow is real-world hourly statistical data with slight variance in vehicle arrival rates, as shown in Tab. 3.
Since the real-world strategies tend to break down during bottleneck period (peak hour), to better evaluate the performances of traffic light control methods in the flat-
[Table] [Table]
peak-flat scenario
, we use synthetic datasets, which have a more dramatic variance in vehicle arrival rates, as shown in Tab. 4.
A detailed description of traffic flow configurations is:
• Real .
The traffic flows of Hangzhou (China), Jinan (China) and New York (USA) are from the public datasets, which are processed from multiple sources.
The traffic flow of Shenzhen (China) is made by ourselves generated based on the traffic trajectories collected from 80 redlight cameras and 16 monitoring cameras in a hour.
The data statistics are listed in Tab. 3.
• Synthetic .
The Synthetic is a mixed traffic flow with a total flow of 4770 in one hour, to simulate a heavy peak.
The arrival rate changes every 10 minutes, which is used to simulate the uneven traffic flow distribution in the real world, the details of the vehicle arrival rate and cumulative traffic flow are shown in Tab. 4.
E E VALUATION C RITERIA
Following existing studies Wei et al.
(2019b;a); Xiong et al.
; Chen et al.
; Zang et al.
(2020), we use the average travel time to evaluate the performance of different methods for traffic signal control.
The average travel time indicates the overall traffic situation in an area over a period of time.
For a detailed definition of average travel time, see Section B.4.
Since the number of vehicles and the origin-destination (OD) positions are fixed, better traffic signal control strategies result in less average travel time.
F B ASELINES
Our method is compared with the following two categories of methods: conventional transportation methods and RL methods 3 .
Note that for a fair comparison all the RL methods are learned without any pre-trained parameters and the methods are evaluated under the same settings.
The results are obtained by running the source codes 4 .
All the baselines are run with three random seeds, and the mean is taken as the final result.
The action interval is five seconds for each method, and the horizon is 3600 seconds for each episode.
Specifically, the compared methods contain:
[Table]
F. C ONVENTIONAL METHODS
• M AX P RESSURE Varaiya (2013) is a leading conventional method, which greedily chooses the phase with the maximum pressure.
The pressure is defined as the difference of vehicle density between the incoming lane and the outgoing lane, and the vehicle density means the actual number of vehicles divided by the maximum permissible vehicle number.
• F IXEDTIME Koonce & Rodegerdts (2008) with random offset Roess et al.
(2004) executes each phase in a phase loop with a pre-defined span of phase duration, which is widely used for steady traffic.
• SOTL Cools et al.
(2013) specifies a pre-defined threshold for the number of waiting vehicles on approaching lanes.
Once the waiting vehicles exceeds the threshold, it will switch to the next phase.
F. RL- BASED METHODS
• I NDIVIDUAL RL.
Wei et al.
Independent control is performed for each agent in multiagent environment, each intersection is controlled by one agent.
The replay buffer and network parameters are not shared, and the model update is independent.
There is no information transfer between agents, and no neighbor information is considered.
• M ETA L IGHT Zang et al.
(2020) is a value-based meta reinforcement learning method via parameter initialization, which is based on MAML Finn et al.
M ETA L IGHT is originally a single-agent approach for meta-learning on multiple separate tasks.
Here we extend it to a multi-agent scenario without considering neighbor information.
• P RESS L IGHT Wei et al. (2019a) combines the traditional traffic method M AX P RESSURE Varaiya (2013) with RL technology together.
P RESS L IGHT is a RL method that optimizes the pressure of each intersection.
• C O L IGHT Wei et al.
(2019b) uses graph convolution and attention mechanism to model the neighbor information, and then further uses this neighbor information to optimize the queue length.
• G ENERA L IGHT Zhang et al.
(2020b) is a meta reinforcement learning method which uses generative adversarial network to generate diverse traffic flows and uses them to build training environments.