C ONDITIONAL E XPECTATION BASED V ALUE D ECOMPOSITION FOR S CALABLE O N
D
EMAND R IDE P OOLING
Anonymous authors
Paper under double-blind review
A BSTRACT
Owing to the beneﬁts for customers (lower prices), drivers (higher revenues), aggregation companies (higher revenues) and the environment (fewer vehicles), ondemand ride pooling (e.g., Uber pool, Grab Share) has become quite popular.
The signiﬁcant computational complexity of matching vehicles to combinations of requests has meant that traditional ride pooling approaches are myopic in that they do not consider the impact of current matches on future value for vehicles/drivers.
Recently, Neural Approximate Dynamic Programming (NeurADP) has employed value decomposition with Approximate Dynamic Programming (ADP) to outperform leading approaches by considering the impact of an individual agent’s (vehicle) chosen actions on the future value of that agent.
However, in order to ensure scalability and facilitate city-scale ride pooling, NeurADP completely ignores the impact of other agents actions on individual agent/vehicle value.
As demonstrated in our experimental results, ignoring the impact of other agents actions on individual value can have a signiﬁcant impact on the overall performance when there is increased competition among vehicles for demand.
Our key contribution is a novel mechanism based on computing conditional expectations through joint conditional probabilities for capturing dependencies on other agents actions without increasing the complexity of training or decision making.
We show that our new approach, Conditional Expectation based Value Decomposition (CEVD) outperforms NeurADP by up to 9.76% in terms of overall requests served, which is a signiﬁcant improvement on a city wide benchmark taxi dataset.
I NTRODUCTION
Taxi/car on Demand (ToD) services (e.g., UberX, Lyft, Grab) not only provide a comfortable means of transport for customers, but also are good for the environment by enabling sharing of vehicles over time (while being used to serve one request at any one point in time).
A further improvement of ToD is on-demand ride pooling (e.g., UberPool, LyftLine, GrabShare etc.), where vehicles are shared not only over time but also in space (on the taxi/car).
On-demand ride pooling reduces the number of vehicles required, thereby reducing emissions and trafﬁc congestion compared to Taxi/car onDemand (ToD) services.
This is achieved while providing beneﬁts to all the stakeholders involved: (a) Individual passengers have reduced costs due to sharing of space; (b) Drivers make more money per trip as multiple passengers (or passenger groups) are present; (c) For the aggregation company more customer requests can be satisﬁed with the same number of vehicles.
In this paper, we focus on this on-demand ride pooling problem at city scale, referred to as RidePool Matching Problem (RMP)
[Alonso-Mora et al.
Bei & Zhang (2018); Lowalekar et al.
The goal in an RMP is to assign combinations of user requests to vehicles (of arbitrary capacity) online such that quality constraints (e.g., delay in reaching destination due to sharing is not more than 10 minutes) and
matching constraints (one request can be assigned at most one vehicle, one vehicle must be assigned at most one request combination) are satisﬁed while maximizing an overall objective (e.g., number of requests, revenue).
Unlike the ToD problem that requires solving a bipartite matching problem between vehicles and customers, RMP requires effective matching on a tripartite graph of requests, trips (combinations of requests) and vehicles.
This matching on tripartite graph signiﬁcantly increases the complexity of solving RMP online, especially at city scale where there are hundreds or thousands of vehicles, hundreds of requests arriving every minute and request combinations have to be computed for each vehicle.
Due to this complexity and the need to make decisions online, most existing work related to solving
RMP has focused on computing best greedy assignments [Ma et al.
; Tong et al.
; Huang et al.
Lowalekar et al.
; Alonso-Mora et al.
While these scale well, they are myopic and, as a result, do not consider the impact of a given assignment on future assignments.
The closest works of relevance to this paper are by Shah et al.
[Shah et al.
(2020)] and Lowalekar et al.
[Lowalekar et al.
We speciﬁcally focus on the work by Shah et al.
as it has the best performance, while being scalable.
That work considers future impact of current assignment from an individual agents’ perspective without sacriﬁcing on scalability (to city scale).
However, a key limitation of that work is that they do not consider the impact of other agents (vehicles) actions on an agents’(vehicle) future impact, which as we demonstrate in our experiments can have a major effect (primarily because vehicles are competing for the common demand).
To that end, we develop a conditional expectation based value decomposition approach that not only considers future impact of current assignments but also of other agents state and actions through the use of conditional probabilities and tighter estimates of individual impact.
Due to these conditional probability based tighter estimates of individual value functions, we can scale the work by Guestrin et al.
[Guestrin & Parr (2002)] and Li et al.
[Li & Kochenderfer (2021)] to solve problems with no explicit coordination graphs and hundreds/thousands of homogeneous agents.
Unlike value decomposition approaches [Rashid & Whiteson (2018); Sunehag & Graepel (2018)] developed for solving cooperative Multi-Agent Reinforcement Learning (MARL) with tens of agents and under centralized training and decentralized execution set up, we focus on problems with hundreds or thousands of agents with centralized training and centralized execution (e.g., Uber, Lyft, Grab).
In this application domain of taxi on demand services, where improving 0.5%-1% is a major achievement [Lin et al.
, we demonstrate that our approach easily outperforms the existing best approach,
NeurADP [Shah et al.
by at least 3.8% and up to 9.76% on a wide variety of settings for the benchmark real world taxi dataset [NYYellowTaxi (2016)].
B ACKGROUND
In this section, we formally describe the RMP problem and also provide details of an existing approach for on-demand ride pooling called NeurADP, which we improve over.
Ride-pool Matching Problem (RMP) :
We consider a ﬂeet of vehicles/resources R with random initial locations, travelling on a predeﬁned road network G with intersections : L as nodes, road segments : E as edges and weights on edges indicate the travel time on the road segment.
Passengers that want to travel from one location to another send requests to a central entity that collects these requests over a time-window called the decision epoch ∆ .
The goal of the RMP is to match these collected requests U t to empty or partially ﬁlled vehicles that can serve them such that an objective
J is maximised subject to constraints on the delay D .
We upperbound D and consider the objective J to be the number of requests served.
Thus, RMP is deﬁned using the tuple [ G , U , R , D , ∆ , J ] 1 . Please refer Appendix A.1 for a detailed description.
Delay constraints : D consider two delays, { τ, 2 τ } .
τ denotes the maximum allowed pick-up delay which is the difference between the arrival time of a request and the time at which a vehicle picks the user up.
2 τ denotes the maximum allowed detour delay which is the difference between the time at which the user arrived at their destination in a shared cab and the time at which they would have arrived if they had taken a single-passenger cab.
Neural Approximate Dynamic Programming (NeurADP) for Solving RMP: Figure 1 provides the overall approach.
In this paper, there are two NeurADP [Shah et al.
contributions of relevance:
FV : To estimate F uture V alue of current actions, a method for solving the underlying Approximate Dynamic Program (ADP) [
Powell ( )] by considering neural network representations of value functions
DJV: To ensure scalability, D ecomposing the J oint V alue function into individual vehicle value functions by extending on the work of Russell et al.
[Russell & Zimdars (2003)].
F uture V alue ( FV ): ADP is similar to a Markov Decision Problem (MDP) with the key difference that the transition uncertainty is extrinsic to the system and not dependent on the action.
The ADP problem for RMP is formulated using the tuple (cid:104) S, A, ξ, T, J (cid:105) , where :
S :
The state of the system is represented as s t =
( r t , u t ) where r t is the state of all vehicles and u t contains all the requests waiting to be served.
The state is obtained in Step A of Figure 1.
A : At each time step there are a large number of requests arriving to the taxi service provider, however for an individual vehicle only a small number of such requests are reachable.
The feasible set of request combinations for each vehicle i at time t , F it is computed in Step B of Figure 1: [Equation] DetourDelay
[Equation]
a i
[Equation]
t is the decision variable that indicates whether vehicle i takes action f (a combination of requests) at a decision epoch t .
Joint actions across vehicles have to satisfy matching constraints: (i) each vehicle, i can only be assigned at most one request combination, f ; (ii) at most one vehicle, i can be assigned to a request j ; and (iii) a vehicle, i can be either assigned or not assigned to a request combination.
[Equation]
f
[Equation]
it
[Equation]
(cid:88)
[Equation]
a
[Equation]
ξ : denotes the exogenous information – the source of randomness in the system.
This would correspond to the user requests or demand.
ξ t denotes the exogenous information at time t .
T : denotes the transitions of system state.
In an ADP, the system evolution happens as ( s 0 , a 0 , s a 0 , ξ 1 , s 1 , a 1 , s a 1 , · · · , s t , a t , s at , · · · ) , where s t denotes the pre-decision state at decision epoch t and s at denotes the post-decision state [Powell (2007)].
The transition from state s t to s t +1 depends on the action vector a t and the exogenous information ξ t +1 .
Therefore,
[Equation]
It should be noted that T a ( ., . ) is deterministic as uncertainty is extrinsic to the system.
J : denotes the reward function and in RMP, this will be the revenue from a trip.
Let V ( s t ) denotes the value of being in state s t at decision epoch t , then using Bellman equation:
[Equation]
[Figure] [Figure]
on ]
[Figure] [Figure]
for ]
I n t e g e r
L i n ea r
P r og r a m
( I L P )
[Figure]
where γ is the discount factor.
Using post-decision state, this expression breaks down nicely:
[Equation]
The advantage of this two step value estimation is that the maximization problem in Equationcan be solved using a Integer Linear Program (ILP) with matching constraints indicated in expression 2.
Step D of Figure 1) provides this aspect of the overall algorithm.
The value function approximation around post-decision state, V a ( s at ) is a neural network and is updated (Step E of
Figure 1) by stepping forward through time using sample realizations of exogenous information (i.e. demand observed in data).
However, as we describe next, maintaining a joint value function is not scalable and hence we decompose and maintain individual value functions.
D ecomposing J oint V alue ( DJV ) : Non-linear value functions, unlike their linear counterparts, cannot be directly integrated into the ILP mentioned above.
One way to incorporate them is to evaluate the value function for all possible post-decision states and then add these values as constants.
However, the number of post-decision states is exponential in the number of resources/vehicles.
[Shah et al.
(2020)] introduced a two-step decomposition of the joint value function that converts it into a linear combination over individual value functions associated with each vehicle.
In the ﬁrst step, following [Russell & Zimdars (2003)], the joint value function is written as the sum over individual value functions : V ( s at ) =
(cid:80) i V i ( s at ) .
In the second step, the individual vehicles’ value functions are approximated.
They assumed that the long-term expected reward of a given vehicle is not signiﬁcantly affected by the speciﬁc actions another vehicle makes in the current decision epoch and thereby completely neglect the impact of the actions taken by other vehicles at the current time step.
Thus they model the value function using the pre-decision, rather than post-decision, state of other vehicles which gives :
[Equation]
where - i refers to all vehicles except vehicle i .
This allows NeurADP to get around the combinatorial explosion of the post-decision state of all vehicles.
NeurADP thus has the joint value function : V ( s at ) =
(cid:80) i V i ([ s i,at , s - it ]) .
They then evaluate these individual V i values (Step C of Figure 1) for all possible s i,at (from the individual value neural network) and then integrate the overall value function into the ILP as a linear function over these individual values.
This reduces the number of evaluations of the non-linear value function from exponential to linear in the number of vehicles.
C ONDITIONAL E XPECTATION BASED V ALUE D ECOMPOSITION , CEVD
One of the fundamental drawbacks in NeurADP is that each agent/vehicle 2 to a large extent is kept in oblivion about the values of the feasible actions for other agents/vehicles.
Since our problem execution (assignment of requests to agents) is centralized, this independence of individual agents (as shown in experimental results) leads to sub-optimal actions for the entire system.
While there are dependencies between agents, not all agents are dependent on each other and one mechanism typically employed to represent sparsely connected multi-agent systems is through the use of a coordination graph [ Guestrin & Parr (2002); Li & Kochenderfer (2021)], CG =
( X, E ) .
The joint value of the system with joint state s and joint action a in the context of a coordination graph is given by:
[Equation]
where f i ( . | . ) represents the value of agent i and f ij ( ., .
| . )
represents the impact of agent j ’s actions on agent i ’s value.
Such an approach is scalable if there are a few agents.
However, when considering thousands of agents and a central ILP which requires values for all different joint action pairs, there is a combinatorial explosion making the model non deployable in real time.
To put things into perspective, assuming each agent has | A | feasible actions (request combinations) and there are N (typically 1000 ) agents, the number of value evaluations jumps from N · | A | in DJV to
N · | A | 2 while using a coordination graph.
It should be further noted the A corresponds to request combinations and hence can increase combinatorially.
Thus, we need a mechanism that is scalable while considering the impact of i on other agents.
In the well known Expectation Maximization algorithm
[Dempster (1977)] for identifying missing data, the likelihood is calculated by introducing a conditional probability of unknown data given known data.
In a similar vein, our method to deal with the unknown impact of other agents is by considering conditional probability of agent j taking action a j given agent i takes action a i in state s .
This will ensure the overall value is dependent on individual agent values and not on joint values.
More speciﬁcally, the expected value of agent i is:
[Equation]
To make this broad idea of conditional expectation operational in case of RMP, we have to address multiple key challenges.
We describe these key challenges and our ways of addressing them below.
Figure 2 provides the overall method, with step (II) outlining the conditional expectation idea and the key difference from NeurADP described in Figure 1.
3. N O EXPLICIT / STATIC COORDINATION GRAPH
While it is clear that agents that are very far apart will not have any dependency, there is no explicit coordination graph that is present in RMP.
However, RMP has two characteristics that make it easier to identify neighbouring agents for any given agent:
• Agents that are nearby spatially are more probable to compete over the same set of requests and hence would have a dependency.
• Agents/vehicles do not have identity, i.e., they are all homogenous.
Due to these characteristics, we can cluster the intersections in the road network (to capture spatial dependencies) and consider agents at a time step in an intersection cluster as neighboring agents.
Due to homogeneity of agents, the only aspect of importance is whether there are agents (and not which speciﬁc agents) competing for the same requests.
Unlike previous works, the coordination graph keeps changing at each time step, as agents move between clusters.
At time t , an agent placed at an intersection belonging to cluster C k will coordinate with all other agents in cluster C k .
We deﬁne function M : L −→ [ K ] to map each intersection of the road network into one of the
K clusters : { C 1 , C 2 , . . .
, C K } based on the average travel times between intersections.
Because of clustering locations and assigning agents to location clusters, the total number of agents becomes less of an issue with respect to scalability.
3. H OW TO CONSIDER IMPACT OF OTHER AGENTS IN THE CLUSTER ?
Let us consider agent i present in cluster C k at time t .
The other agents in cluster C k are agents j 1 , j 2 , . . .
, j n and are termed its neigbours.
In NeurADP, agent i credited action f ∈ F it with value
V i ( s i,ft ) which is oblivious to the presence of neighbour agents.
Since the execution is centralized, agent i can however weigh the losses/gains its action f has on the cluster by getting useful feedback from the individual values of other agents.
Let us take a neighbour agent j ∈ C k ( j (cid:54)
=
i ) having feasible actions g 1 , g 2 , . .
, g k j ∈ F jt .
From agent i ’s perspective a conditional probability distribution
P ( Agent j takes action g | Agent i takes action f ) =
P j ( g | s it , f ) is formed and the feedback term from agent j is written as (cid:80) g ∈F tj P j ( g | s it , f )
V j ( s j,gt ) .
We do this for all the neighbours and take an average :
[Equation]
Now to calculate the value of agent i on taking action f , after getting this feedback, we take an afﬁne combination and write the new individual value as
[Equation]
where λ is a learnable parameter.
It should be noted that this individual value not only considers the future impact of current action, f , but also considers the impact on other agents.
Our overall ILP objective thus becomes :
[Equation]
subject to feasibility constraints in expression 2.
What should be the functional form for conditional probabilities?
Each request u it ∈
U t has a pickup location o it ∈ L .
Recall function M : L −→ [ K ] which maps each intersection to its cluster.
Every action f is associated with some user request u it , we deﬁne
[Equation]
which maps each action f to the corresponding request ( u it )
’s pickup intersection o it .
Deﬁne the composition function
[Equation]
that maps each action to a cluster by the pickup location of the action’s corresponding user request.
d : [ K ] × [ K ] −→ R + is deﬁned as the average travel time between 2 clusters.
We model the conditional probability of agent j taking action g given agent i takes action f as :
[Equation]
where α is a learnable parameter.
The normalizing constant is computed by summing over actions in F jt .
3. O VER / UNDER ESTIMATION OF INDIVIDUAL VALUES IN FV AND DJV :
NeurADP makes an optimistic assumption that individual agent will get to take the best action in the next time step.
Since central ILP decides the joint action, this can result in an overestimation or underestimation of individual agent value.
Due to this and other issues, in our experiments, we found that NeurADP values can have signiﬁcant errors compared to the discounted future rewards as shown in Figure 3.
We ﬁx this problem through two key enhancements:
• Controlling large Variance of exogeneous information : Recall from Equation 4, to calculate values of post action states, NeurADP employs
[Equation]
Here the exogenous information ξ t +1 is the global demand ( g t ) at time t .
Even within a small number of consecutive epochs, the global demand displays a signiﬁcant variance.
This results in individual values showing a large variance, instead of varying smoothly over time.
We thus consider expected discounted future demand,
[Equation]
(where T is large but ﬁnite horizon) which varies smoothly over time unlike the current demand.
In our approach, we consider exogenous information as ξ t +1 =
[ g t , F t ] .
• Enforcing values to be positive : NeurADP models the value function as a shared parameter Neural Network.
The ﬁnal layer of this Neural Network is a fully connected multi layer perceptron (MLP) having range as the entire real line.
However, as our objective function (number of requests served) is non-negative, negative values (admissible by NeurADP) are not reasonable.
We thus use a SoftPlus activation after the ﬁnal MLP to ensure that the computed values are strictly positive.
To evaluate the impact of the above modiﬁcations, in Figure 3 we plot the following : (cid:80) i V i ( s f it ,i t )
( f it is the action chosen for vehicle i at time t ) vs (cid:80) Tt (cid:48)
=
γ t (cid:48)
R t + t (cid:48) where R t is the total number of requests served at time t for NeurADP, ablation (the above two modiﬁcations) and CEVD.
Notice how the gap between the Estimated Value curve and the Discounted Reward curve is very small in the latter two (as it should be by the Bellman Equation 3), whereas the gap is quite signiﬁcant for NeurADP. Note that the height of the graphs is different and while these modiﬁcations improve the quality of the estimation, CEVD is responsible for the bulk of the performance gain.
A LGORITHM
Given a post decision state s i,ft , we need a paremterized function to compute ˆ V ( s i,ft ) .
Our joint function ˆ V θ,λ,α has 3 parameters : (i) θ : parameters of a Neural Network Based Individual Agent Value Function Estimator (as in NeurADP) 3 , (ii) λ : parameter to control the importance given to an agent’s neighbours while taking an afﬁne combination, (iii) α : parameter to control conditional probabilities P which controls the relative importance given to different feasible actions of neighbours.
We infer the parameters step by step.
Setting λ = 0 , reduces this function to NeurADP ( ˆ V θ, 0 , 0 ).
We ﬁrst estimate optimal θ ∗ (following the alogrithm in NeurADP).
This gives us the NeurADP parameters, which are a good starting point to estimate values at an individual level.
Now to estimate λ , we set α =
(this corresponds to uniform distribution over actions), and do a linear search on a set of sampled points on the real line to ﬁnd the optimal λ ∗ = max λ (cid:80) t =
T t =
(cid:80) | R | i =
(cid:18)
J i ( s it , f ) + γ ˆ V ( s i,ft ) (cid:19) × a i,ft subject to constraints in expression 2.
At this stage, we haven’t changed the preference over actions from an individual perspective, however for the central executor the values are now much more reﬁned as the individual over/under estimates have been smoothened by considering the neigbours.
Finally we estimate α by linear search on a set of sampled points on the real line to get optimal α ∗ = max α (cid:80) t =
T t =
(cid:80) | R | i =
(cid:18)
J i ( s it , f )+ γ ˆ V ( s i,ft ) (cid:19) × a i,ft subject to constraints in expression 2.
We can now compute values on unseen data using ˆ V θ ∗ ,λ ∗ ,α ∗ .
E XPERIMENTS
The goal of the experiments is to compare the performance of our approach CEVD to NeurADP[Shah et al.
(2020)](henceforth referred to as baseline), which is the current best approach for solving the RMP.
We make this comparison on a real-world dataset [NYYellowTaxi (2016)] across different RMP parameter settings.
We quantitatively justify our performance by comparing the service rate, i.e., the percentage improvement on the total requests served.
We vary the following parameters: the maximum allowed waiting time τ from 90 seconds to 150 seconds, the number of vehicles |R| from 500 to 1000 and the capacity C from 4 to 5.
The value of maximum allowable detour delay λ is taken as 2 ∗ τ .
The decision epoch duration ∆ is taken as 60 seconds.
Setup: We perform our experiments on the demand distribution from the publicly available
New
York Yellow Taxi Dataset [NYYellowTaxi (2016)].
The experimental setup is similar to the setup used by [Shah et al.
Street intersections are used as the set of locations L .
They are identiﬁed by taking the street network of the city from openstreetmap using osmnx with ’drive’ network type [Boeing (2017)].
Nodes that do not have outgoing edges are removed, i.e., we take the largest strongly connected component of the network.
The resulting network has 4373 locations (street intersections) and 9540 edges.
The travel time on each road segment of the street network is taken as the daily mean travel time estimate computed using the method proposed in [Santi et al.
We further cluster these intersections L into K clusters using K-Means Clustering based on the average travel times between different intersections.
We choose the value of K based on the number of vehicles.
K is chosen to be 100,150 and 200 for 500,750 and 1000 vehicles respectively.
Similar to previous work, we only consider the street network of Manhattan as a majority ( ∼ 75%) of requests have both pickup and drop-off locations within it.
The dataset contains data about past customer requests for taxis at different times of the day and different days of the week.
From this dataset, we take the following ﬁelds: (1) Pickup and drop-off locations (latitude and longitude coordinates) - These locations are mapped to the nearest street intersection.
Pickup time - This time is converted to appropriate decision epoch based on the value of ∆ .
The dataset contains on an average 322714 requests in a day (on weekdays) and 19820 requests during peak hour.
We evaluate the approaches over 24 hours on different days starting at midnight and take the average value over 5 weekdays (4 - 8 April 2016) by running them with a single instance of initial random location of taxis 4 .
CEVD is trained using the data for 8 weekdays (23 March - 1 April 2016) and it is validated on 22 March 2016.
For the experimental analysis, we consider that all vehicles have identical capacities.
Results
We compare CEVD to NeurADP (referred to as baseline).
Table 1 gives a detailed performance analysis for the service rates of CEVD and baseline.
Here are some key observations:
Effect of changing tolerance to delay, τ : CEVD obtains a 9.37% improvement over the baseline approach for τ = 90 seconds.
The difference between the baseline and CEVD decreases as τ increases.
The lower value of τ makes it difﬁcult for vehicles to accept new requests while satisfying the constraints for already accepted requests.
The neighbouring vehicles’ interactions in CEVD prevents a vehicle from picking up requests it values highly however which would have been more suitable given the delay constraints for some other vehicle and instead picks up requests which it might value less but still is feasible.
Thus the overall requests served increases.
Effect of changing the capacity, C : CEVD obtains a 9.76% gain over baseline for capacity 5. The [Table]
difference between the as vehicles, there is a larger scope for improvement if vehicles cooperate well.
Effect of changing the number of vehicles, |R| : CEVD obtains a 9.37% improvement over the baseline for capacity |R| = 500 .
The difference between the baseline and CEVD decreases as the number of vehicles increase as in the presence of a large number of vehicles, there will always be a vehicle that can serve the request.
As a result, the quality of assignments plays a smaller role.
We further analyse the improvements obtained by CEVD over baseline by comparing the number of requests served by both approaches at each decision epoch throughout the day.
Figure 4 shows the number of requests served by the baseline and CEVD at different decision epochs 5 .
As shown in the ﬁgure, initially at night time when the demand is low both approaches serve all available demand.
During the transition period from low demand to high demand period, the baseline algorithm starts to choose suboptimal actions without considering the impact of each vehicle’s action on the whole system while CEVD is able to capitalize on the joint action values and serve much more requests than the baseline.
The approach can be executed in real-time settings.
The average time taken to compute each batch assignment using CEVD is less than 60 seconds (for all cases) 6 .
These results indicate that using our approach can help ride-pooling platforms to better meet customer demand.
C ONCLUSION
Due to the matching required on a tri-partite graph between user requests, trips (combination of user requests) and vehicles, on-demand ride pooling is challenging.
Improving on existing methods, we provide a scalable novel value decomposition method based on conditional probabilities, where individual value is not only able to consider future impact of current matches but also impact on other agent values.
This new approach is able to outperform the best existing method in all settings of the benchmark taxi data set employed for on-demand ride pooling by margins of up to 9.79%.
To put this result in perspective, typically, an improvement of 1% is considered a signiﬁcant improvement on ToD for an entire city [Xu et al.
; Lowalekar et al.
A A PPENDIX A. R IDE - POOL M ATCHING P ROBLEM , RMP
Here, we provide speciﬁc details of the RMP problem.
G :
Following Alonso-Mora et al.
(2017), the road network is represented by a weighted graph ( L , E ) where L denotes the set of street intersections and E deﬁnes the adjacency of these intersections which captures the travel time for a road segment.
We assume that vehicles only pick up and drop people off at intersections.
U : =
× t U t , is the combination of requests that we observe at each decision epoch t .
Each request u jt ∈ U t is represented by the tuple: (cid:68) o jt , e jt , t (cid:69) , where o jt , e jt ∈ L denote the origin and destination and t denotes the arrival epoch of the request.
R : The set of resources/vehicles where each element i ∈ R is represented by the tuple (cid:68) c
i , p i ,(cid:126)l i (cid:69) .
c i denotes the capacity of the vehicle, i.e., the maximum number of passengers it can carry simultaneously, p i its current position and (cid:126)l i the ordered list of locations that the vehicle should visit next to satisfy the requests currently assigned to it.
D : { τ, λ } denotes the set of constraints on delay.
τ denotes the maximum allowed pick-up delay which is the difference between the arrival time of a request and the time at which a vehicle picks the user up.
λ denotes the maximum allowed detour delay which is the difference between the time at which the user arrived at their destination in a shared cab and the time at which they would have arrived if they had taken a single-passenger cab.
∆ : denotes the decision epoch duration.
J : represents the objective, with J it denoting the value obtained by serving request i at decision epoch t .
The goal of the online assignment problem is to maximize the overall objective over a given time horizon, T .
A. R ELATED W ORK
There are three main threads of existing work in solving RMP problems:
(i)
The ﬁrst set of approaches are traditional planning approaches that model RMP as an optimization problem Ropke & Cordeau (2009);
Ritzinger et al.
; Parragh et al.
The problem with this class of approaches is that they don’t scale to on-demand city-scale scenarios.
(ii)
The second set
of approaches are focused on making the best greedy assignments Ma et al.
; Tong et al.
; Huang et al.
Lowalekar et al.
; Alonso-Mora et al.
While these scale well, they are myopic and, as a result, do not consider the impact of a given assignment on future assignments.
(iii)
The third thread of methods Shah et al.
Lowalekar et al.
(2021) are focussed on use of
Reinforcement Learning (RL) or online Multi-Stage Stochastic Optimization to address the myopia associated with approaches from the second category.
These set of approaches consider future impact of current matches through the use of individual value function, they achieve this by ignoring the impact of other agents on the value (e.g., future revenue) of a vehicle.
With respect to our technical contributions on value decomposition, we improve the work of Guestrin et al.
[Guestrin & Parr (2002)] and Li et al.
[Li & Kochenderfer (2021)] that was previously applicable to tens of agents with coordination graphs, to scale to hundreds/thousands of homogeneous agents with no explicit coordination graphs.
The key difference is with regards to the use of conditional probability based dependencies amongst neighboring agents.
Another technical contribution that is of relevance is the value decomposition approaches [Rashid &
Whiteson (2018); Sunehag & Graepel (2018)] developed for solving cooperative Multi-Agent Reinforcement Learning (MARL).
These approaches have been developed to solve problems with tens of agents and under centralized training and decentralized execution set up, we focus on problems with hundreds or thousands of agents with centralized training and centralized execution (e.g., Uber,
Lyft, Grab).
A.
[Equation]
PickUpDelay ( f
[Equation]
, i )
[Equation]
, DetourDelay (
[Equation]
i )
[Equation]
τ
[Equation]
∀ f ∈ F it , ∀
i ∈ R
∀ f ∈ F it , ∀ i ∈ R , ∀ k ∈
[ K ] : ˆ V i ( s i,ft ) = 1 + λ (cid:20) V
( s i,ft ) + λ | C k | −(cid:88)
j ∈ C k
[Equation]
(cid:54) =
i (cid:88) g ∈F tj
[Equation]
( g | s it , f )
[Equation]
( s
[Equation]
g
[Equation]
) (cid:21) 11:
Solve the MIP : max (cid:88) i (cid:88)
f
[Equation]
ti (cid:18)
[Equation]
[Equation]
(cid:19)
[Equation]
a
[Equation]
,ft 12:
: (cid:88) f ∈F it a
i
[Equation]
t
[Equation]
∈ R (cid:88) i ∈R (cid:88) f
∈F it ;
j ∈ f a i,ft ≤ 1 ::: ∀ j ∈ U a i,ft ∈ { 0 , 1 } ::: ∀ i, f 13:
Assign actions
Simulate agents to pick requests
A. N EUR ADP A LGORITHM
=
T a
{+( s nt , a nt ) , s nt +
+}
=
T ξ ( s a,nt , ξ nt +
A. A DDITIONAL P LOTS
We provide additional graphs of number of requests served in Figure 5 with different settings.
A. P ARAMETER R ANGES FOR CEVD
To estimate λ we uniformly sample points in the range ( − 1 . 0 , 1 . 0) and choose the optimal λ ∗ as the value giving the largest rewards over a horizon.
Similarly, to estimate α we uniformly sample points in the range [ − 10 . 0 , 10 . 0] and choose the optimal α ∗ as the value giving the largest rewards over a horizon after choosing λ ∗ .