M ULTI - AGENT D EEP FBSDE R EPRESENTATION F OR L ARGE S CALE S TOCHASTIC D IFFERENTIAL G AMES
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper we present a deep learning framework for solving large-scale multiagent non-cooperative stochastic games using ﬁctitious play.
The HamiltonJacobi-Bellman (HJB) PDE associated with each agent is reformulated into a set of Forward-Backward Stochastic Differential Equations (FBSDEs) and solved via forward sampling on a suitably deﬁned neural network architecture.
Decision making in multi-agent systems suffers from curse of dimensionality and strategy degeneration as the number of agents and time horizon increase.
We propose a novel Deep FBSDE controller framework which is shown to outperform the current state-of-the-art deep ﬁctitious play algorithm on a high dimensional interbank lending/borrowing problem.
More importantly, our approach mitigates the curse of many agents and reduces computational and memory complexity, allowing us to scale up to 1,000 agents in simulation, a scale which, to the best of our knowledge, represents a new state of the art.
Finally, we showcase the framework’s applicability in robotics on a belief-space autonomous racing problem.
I NTRODUCTION
Stochastic differential games represent a framework for investigating scenarios where multiple players make decisions while operating in a dynamic and stochastic environment.
The theory of differential games dates back to the seminal work of Isaacs (1965) studying two-player zero-sum dynamic games, with a ﬁrst stochastic extension appearing in Kushner & Chamberlain (1969).
A key step in the study of games is obtaining the Nash equilibrium among players (Osborne & Rubinstein, 1994).
A Nash equilibrium represents the solution of non-cooperative game where two or more players are involved.
Each player cannot gain beneﬁt by modifying his/her own strategy given opponents equilibrium strategy.
In the context of adversarial multi-objective games, the Nash equilibrium can be represented as a system of coupled Hamilton-Jacobi-Bellman (HJB) equations when the system satisﬁes the Markovian property.
Analytic solutions exist only for few special cases.
Therefore, obtaining the Nash equilibrium solution is usually done numerically, and this can become challenging as the number of states/agents increases.
Despite extensive theoretical work, the algorithmic part has received less attention and mainly addresses special cases of differential games (e.g., Duncan & Pasik-Duncan (2015)), or suffers from the curse of dimensionality (Kushner, 2002).
Nevertheless, stochastic differential games have a variety of applications including in robotics and autonomy, economics and management.
Relevant examples include Mataramvura & Øksendal (2008), which formulate portfolio management as a stochastic differential game in order to obtain a market portfolio that minimizes the convex risk measure of a terminal wealth index value, as well as Prasad & Sethi (2004), who investigate optimal advertising spending in duopolistic settings via stochastic differential games.
Reinforcement Learning (RL) aims in obtaining a policy which can generate optimal sequential decisions while interacting with the environment.
Commonly, the policy is trained by collecting histories of states, actions, and rewards, and updating the policy accordingly.
Multi-agent Reinforcement
Learning (MARL) is an extension of RL where several agents compete in a common environment, which is a more complex task due to the interaction between several agents and the environment, as well as between the agents.
One approach is to assume agents to be part of environment (Tan, 1993), but this may lead to unstable learning during policy updates (Matignon et al., 2012).
On the other hand, a centralized approach considers MARL through an augmented state and action system, reducing its training to that of single agent RL problem.
Because of the combinatorial complexity, the centralized learning method cannot scale to more than 10 agents (Yang et al., 2019).
Another method is centralized training and decentralized execute (CTDE), however the challenge therein lies on how to decompose value function in the execute phase for value-based MARL.
Sunehag et al.
(2018) and Zhou et al.
(2019) decompose the joint value function into a summation of individual value functions.
Rashid et al.
(2018) keep the monotonic trends between centralized and decentralized value functions by augmenting the summation non-linearly and designing a mixing network (QMIX).
Further modiﬁcations on QMIX include Son et al.
; Mahajan et al.
The mathematical formulation of a differential game leads to a nonlinear PDE.
This motivates algorithmic development for differential games that combine elements of PDE theory with deep learning.
Recent encouraging results (Han et al., 2018; Raissi, 2018) in solving nonlinear PDEs within the deep learning community illustrate the scalability and numerical efﬁciency of neural networks.
The transition from a PDE formulation to a trainable neural network is done via the concept of a system of Forward-Backward Stochastic Differential Equations (FBSDEs).
Speciﬁcally, certain PDE solutions are linked to solutions of FBSDEs, and the latter can be solved using a suitably deﬁned neural network architecture.
This is known in the literature as the deep FBSDE approach.
Han et al.
; Pereira et al.
; Wang et al.
(2019b) utilize various deep neural network architectures to solve such stochastic systems.
However, these algorithms address single agent dynamical systems.
Two-player zero-sum games using FBSDEs were initially developed in Exarchos et al.
and transferred to a deep learning setting in Wang et al. (2019a).
Recently,Hu (2019) brought deep learning into ﬁctitious play to solve multi-agent non-zero-sum game, Han & Hu (2019) introduced the deep FBSDEs to a multi-agent scenario and the concept of ﬁctitious play, furthermore, Han et al. (2020) gives the convergence proof.
In this work we propose an alternative deep FBSDE approach to multi-agent non-cooperative differential games, aiming on reducing complexity and increasing the number of agents the framework can handle.
The main contribution of our work is threefold:
We introduce an efﬁcient Deep FBSDE framework for solving stochastic multi-agent games via ﬁctitious play that outperforms the current state of the art in Relative Square Error (RSE) and runtime/memory efﬁciency on an inter-bank lending/borrowing example.
We demonstrate that our approach scales to a much larger number of agents (up to 1, agents, compared to 50 in existing work).
To the best of our knowledge, this represents a new state of the art.
We showcase the applicability of our framework to robotics on a belief-space autonomous racing problem which has larger individual control and state space.
The experiments demonstrates that the decoupled BSDE provides the possibility of applications for competitive scenario.
The rest of the paper is organized as follows: in Section 2 we present the mathematical preliminaries.
In Section 3 we introduce the Deep Fictitious Play Belief FBSDE, with simulation results following in Section 4.
We conclude the paper and discuss some future directions in Section 5.
M ULTI - AGENT F ICTITIOUS P LAY FBSDE
Fictitious play is a learning rule ﬁrst introduced in Brown (1951) where each player presumes other players’ strategies to be ﬁxed.
An N -player game can then be decoupled into N individual decisionmaking problems which can be solved iteratively over M stages.
When each agent 1 converges to a stationary strategy at stage m , this strategy will become the stationary strategy for other players at stage m
+ 1 .
We consider a N -player non-cooperative stochastic differential game with dynamics d X ( t ) =
(cid:0) f
( X ( t ) , t )
+ G ( X ( t ) , t )
U ( t ) (cid:1)
d t + Σ
( X ( t ) , t ) d W ( t ) ,
X (0) =
X 0 , (1) where
X
=
( x 1 , x 2 , . . .
, x N ) is a vector containing the state process of all agents generated by their controls U
= ( u 1 , u 2 , . .
. , u N ) with x i ∈ R n x and u i ∈ R n u .
Here, f : R n x × [0 , T ] → R n x represents the drift dynamics, G : R n x × [0 , T ] → R n x × n u represents the actuator dynamics, and Σ : [0 , T ] × R n → R n x × n w represents the diffusion term.
We assume that each agent is only driven by its own controls so G is a block diagonal matrix with G i corresponding to the actuation of agent i .
Each agent is also driven by its own n w -dimensional independent Brownian motion W i , and denote
W =
( W 1 , W 2 , . . .
, W N ) .
Let U i be the set of admissible strategies for agent i ∈
I := { 1 , 2 , . . .
, N }
and U = ⊗ Ni
=1 U i as the
Kronecker product space of U i .
Given the other agents’ strategies, the stochastic optimal control problem for agent i under the ﬁctitious play assumption is deﬁned as minimizing the expectation of the cumulative cost functional J it
[Equation]
where g : R n x → R + is the terminal cost, and C i : [0 , T ] × R n x
× U → R + is the running cost for the i -th player.
In this paper we assume that the running cost is of the form
C ( X , u i,m , t ) =
q ( X ) + 12 u T i,m R u
i,m + X T Q
u i,m .
We use the double subscript u i,m to denote the control of agent i at stage m and the negative subscript − i as the strategies excluding player i , u − i = ( u 1 , . . .
, u i − 1 , u i +1 , . .
. , u N ) .
We can deﬁne value function of each player as
[Equation]
Assume that the value function in eq.
(3) is once differentiable w.r
[Equation]
and twice differentiable w.r.t.
x .
Then, standard stochastic optimal control theory leads to the HJB PDE
[Equation]
where h = C i ∗ + GU ∗ , 0 .
The double subscript of U ∗ , 0 denotes the augmentation of the optimal control u ∗ i,m = − R − 1 ( G T i V ix + Q T
i x ) and zero control u − i,m − 1 = 0 , and U 0 , − i denotes the augmentation of u i,m = 0 and u − i,m − 1 .
Here we drop the functional dependencies in the HJB equation for simplicity.
The detailed proof is in Appendix A.
The value function in the HJB PDE can be related to a set of FBSDEs
[Equation]
where the backward process corresponds to the value function.
The detailed derivation can be found in Appendix B. Note that the FBSDEs here differ from that of Han & Hu (2019) in the optimal control of agent i , G U ∗ , − i , in the forward process and compensation, V i T x
G U ∗ , 0 , in the backward process.
This is known as the importance sampling for FBSDEs and allows for the FBSDEs to be guided to explore the state space more efﬁciently.
D EEP F ICTITIOUS P LAY FBSDE C ONTROLLER
In this section, we introduce a novel and scalable Deep Fictitious Play FBSDE (SDFP) Controller to solve the multi-agent stochastic optimal control problem.
The framework can be extended to the partially observable scenario by combining with an Extended Kalman Filter, whose belief propagation can be described by an SDE for the mean and variance (see derivation in Appendix C).
By the natural of decoupled BSDE, the framework can also been extended to cooperative and competitive scenario.
In this paper, we demonstrate the example of competitive scenario.
3. N ETWORK A RCHITECTURE AND A LGORITHM
Inspired by the success of LSTM-based deep FBSDE controllers (Wang et al., 2019b; Pereira et al., 2019), we propose an approach based on an LSTM architecture similar to Pereira et al.
The beneﬁts of introducing LSTM are two-fold: 1) LSTM can capture the features of sequential data.
A performance comparison between LSTM and fully connected (FC) layers in the deep FBSDE framework has been elaborated in Wang et al.
(2019b); 2) LSTM signiﬁcantly reduces the memory complexity of our model since the memory complexity of LSTM with respect to time is O (1) in the inference phase compared with O ( T ) in previous work (Han et al., 2018), where T is the number of time steps.
The overall architecture of SDFP is shown in Fig.
1 and features the same time discretization scheme as Pereira et al.
Each player’s policy is characterized by its own copy 𝑋 ! 𝑈 !
𝑁𝑁 " 𝑁𝑁 # 𝑁𝑁 $ 𝑋 " 𝑈 " 𝑁𝑁 " 𝑁𝑁 # 𝑁𝑁 $ ⋱ ⋱
𝑋 % 𝑁𝑁 " 𝑁𝑁 # 𝑁𝑁 $&" 𝑁𝑁 $ ⋱ ⋯
𝑉 %" 𝑉 %# 𝑉 %$&" 𝑉 %$ ⋱ 𝑉 %∗ ℒ(𝑉 % ,𝑉 %∗ ) ⋯ 𝑁𝑁 $&" 𝑁𝑁 $&" of the network deﬁned in Fig.
At stage m , each player can access the stationary strategy of all other players from stage m − 1 .
During training within a stage, the initial value of each player is predicted by a FC layer parameterized by φ .
At each timestep, the optimal policy for each player is computed using the value function gradient prediction V ix from the recurrent network (consisting of FC and LSTM layers), parameterized by θ .
The FSDE and BSDE are then forward-propagated using the Euler integration scheme.
At terminal time T , the loss function for each player is constructed as the mean squared error between the propagated terminal value V iT and the true terminal value V i ∗ T computed from the terminal state.
The parameters φ and θ of each player can be trained using any stochastic gradient descent type optimizer such as Adam.
The detailed training procedure is shown in Algorithm 2.
3. M ITIGATING C URSE OF D IMENSIONALITY AND S AMPLE C OMPLEXITY
Scalability and sample efﬁciency are two crucial criteria of reinforcement learning.
In SDFP, as the number of agents increases, the number of neural network copies would increase correspondingly.
Meanwhile, the size of each neural network should be enlarged to gain enough capacity to capture the representation of many agents, leading to the infamous curse of dimensionality; this limits the scalability of prior works.
However, one can mitigate the curse of dimensionality in this case by taking advantage of the symmetric game setup.
We summarize merits of symmetric game as following:
Since all agents have the same dynamics and cost function, only one copy of the network is needed.
The strategy of other agents can be inferred by applying the same network.
Thanks to the symmetric property, we can applied invariant layer to extract invariant features to accelerate training and improve the performance with respect to the accumulate cost and RSE loss.
Sharing one network:
It’s important to note that querying other agents should not introduce additional gradient paths.
This signiﬁcantly reduces the memory complexity.
When querying other agents’ strategy, one can either iterate through each agent or feed all agents’ states to the network in a batch.
The latter approach reduces the time complexity by adopting the parallel nature of modern
GPU but requires O ( N 2 ) memory rather than O ( N ) for the ﬁrst approach.
[Figure]
Invariant Layers: The memory complexity can be further reduced with an invariant layer embedding (Zaheer et al., 2017).
The invariant layer utilizes a sum function along with the features in the same set to render the network invariant to permutation of agents.
We apply the invariant layer on X − i and concatenate the resulting features to the features extracted from X i .
However, vanilla invariant layer embedding will not reduce the memory complexity.
Thanks to the symmetric problem setup, one can apply a trick to reduce the invariant layer memory complexity form O ( N 2 ) to O ( N ) .
A detailed introduction to the invariant layer and our implementation can be found in Appendix D and
E. The full algorithm is outlined in Algorithm 1.
4: while above certain threshold (cid:15) do 5: for m ← 1 to M do 6: for all i ∈ I in parallel do 7:
Collect opponent agent’s policy which is same as i th policy: f m − 1 LSTM i ( · ) , f m − 1 FC i ( · ) 8: for l ← 1 to N gd do 9: for t ← 1 to T − 1 do 10: if Using Invariant Layer then 11: X t = f invariant ( X t ) 12: end if 13: for j ← 1 to B in parallel do 14: Compute network prediction for i th player: V jx i ,t = f m FC i ( f m LSTM i
( X jt ; θ l − 1 i )) 15: Compute i th optimal Control : u j,(cid:63)i,t = − R − 1 i ( G T i V jx i ,t + Q T
i x ji ) 16:
Infer − i th players’ network prediction and stop the gradient for them: V x − i ,t = f m − 1 FC i ( f m − 1 LSTM i ( X t ; θ i )) 17:
Compute − i th optimal Control and stop the gradient for them: u j,(cid:63) − i,t = − R − 1 − i ( G T − i V jx −
i ,t + Q T − i x j − i ) 18:
Sample noise ∆ W j ∼ N (0 , ∆ t ) 19:
Propagate FSDE:
X jt +1 =
f FSDE ( X jt , u j,(cid:63)i,t , u j, ∗− i,t , ∆ W j , t ) 20: Propagate BSDE: V ji,t +1 = f BSDE ( X jt , u j, ∗ i,t , ∆ W j , t ) 21: end for 22: end for 23:
Compute loss: L = 1 B (cid:80)
Bj =1 ( V j,(cid:63)T − V ji,T )24:
Gradient Update: θ l , φ l 25: end for 26: end for 27: end for 28: end while
[Figure]
S IMULATION R ESULTS
In this section, we demonstrate the capability of SFDP on two different systems in simulation.
We ﬁrst apply the framework to an inter-bank lending/borrowing problem, which is a classical multiplayer non-cooperative game with an analytic solution.
We compare against both the analytic solution and prior work (Han & Hu, 2019).
Different approaches introduced in Section 3.2 are compared empirically on this system.
We also apply the framework to a variation of the problem for which no analytic solution exists.
Finally, we showcase the general applicability of our framework in an autonomous racing problem in belief space.
All experiment conﬁgurations can be found in Ap-
[Figure] [Table] pendix J. we plot the results of 3 repeated runs with different seeds with the line and shaded region showing the mean and mean ± standard deviation respectively.
The hyperparameters and dynamics coefﬁcients used in the inter-bank experiments are the same as Han & Hu (2019) unless otherwise noted.
4. I NTER - BANK LENDING / BORROWING PROBLEM
We ﬁrst consider an inter-bank lending and borrowing model (Carmona et al., 2013) where the dynamics of the log-monetary reserves of N banks is described by the diffusion process
[Equation]
The state X it ∈ R denotes the log-monetary reserve of bank i at time t > 0 .
The control u it denotes the cash ﬂow to/from a central bank, where as a ( ¯ X − X it ) denotes the lending/borrowing rate of bank i from all other banks.
The system is driven by N independent standard Brownian motion W it , which denotes the idiosyncratic noise, and a common noise W 0 t .
The cost function has the form,
[Equation]
The derivation of the FBSDEs and analytic solution can be found in Appendix F.
We compare the result of implementation corresponding to Algorithm 2 on a 10-agent problem with analytic solution and previous work from Han & Hu (2019) with the same hyperparameters.
Fig.
3 shows the performance of our method compared with analytic solution.
The state and control trajectories outputted by the deep FBSDE solution are aligned closely with the analytic solution.
Table 1 shows the numerical performance compared with prior work by Han & Hu (2019).
Our method outperforms by Relative Square Error (RSE) metrics and computation wall time.
The RSE is deﬁned as following:
[Equation]
Where ˆ V
i is the analytic solution of value function for i th agents at intial state X j (0) .
The initial state X j (0) is new batch of data sampled from same distribution as X (0) in the training phase.
The batch size B is 256 for all inter-bank simulations.
V i is the approximated value function for i th agent by FBSDE controller, and ¯ V i is the average of analytic solution for i th agent over the entire batch.
Time/Memory Complexity Analysis: We empirically verify the time and memory complexity of different implementation approaches introduced in 3.2, which is shown in Fig. 4.
Note that all TITAN RTX
[Figure]
experiments hereon correspond to the symmetric SDFP implementation in Algorithm 1 We also test sample efﬁciency and generalization capability of the invariant layer on a 50-agent problem trained over 100 stages.
The number of initial states is limited during the training and the evaluation criterion is the terminal cost of the test set in which the initial states are different from the initial states during training.
Fig.
2 showcases the improvement in sample efﬁciency and generalization performance of invariant layer.
We suspect this is due to the network needing to learn with respect to a speciﬁc permutation of the input, whereas permutation invariance is built into the network architecture with invariant layer.
Importance sampling: An important distinction of SDFP from the baseline in Han & Hu (2019) is the importance sampling scheme, which helps the LSTM architecture achieve a fast convergence rate during training.
However, the baseline, which uses fully connected layer as backbone, is not suitable for importance sampling, as it would lead to an extremely deep network with fully connected layers from gradient topology perspective.
Sandler et al.
(2018) mentioned that the information loss is inevitable for this kind of fully connected deep network with nonlinear activation.
On the other hand, LSTM does not suffer from this problem because of the existence of long and short memory.
We illustrate the beneﬁts of importance sampling for LSTM backbone and gradient ﬂow of fully connected layer backbone in Appendix I.
High dimension experiment: We also analyze the performance of our framework and that of Han & Hu (2019) both with and without invariant layer on high dimensional problems.
We ﬁrst demonstrate the mitigation of the invariant layer on the curse of many agents.
Fig.
5 demonstrates the ablation experiment of the two deep FBSDE frameworks (SDFP and Han & Hu (2019)).
In order to illustrate that invariant layer can mitigate the curse of dimensionality, we also integrate invariant layers on Han & Hu (2019) and shows the performance in the same ﬁgure 5.
In this experiment, the weights of FBSDE frameworks with invariant layer are adjusted in order to dismiss the performance improvement resulting from increased weights from the invariant layers.
The total cost and RSE are computed by averaging the corresponding values over the last twenty stages of each run.
It can be observed from the plot that without invariant layer, the framework suffers from curse of many agents in the prediction of initial value as the RSE increases with respect to the number of agents.
On the other hand, RSE increases at a slower rate with invariant layer.
In terms of total cost, which is computed from the cost function deﬁned in eq.2, our framework enjoys the beneﬁts of importance sampling and invariant layer, and achieves better numerical results over the number of agents. We further analyse the inﬂuence of invariant layers in the training phase by demonstrating ﬁg.7. Invariant layer helps mintage over-ﬁtting phenomenon in the training and evaluation phase, meanwhile accelerating the training process, even though both of frameworks adopting same feature extracting backbone (LSTM) and importance sampling technique.
We also show that the invariant layer accelerates training empirically on a 500-agent problem.
Fig.shows that the FBSDE frameworks converge much faster with invariant layer than without it.
We suspect that the acceleration effect results from increased sample efﬁciency.
Note that the comparison is done on a 500-agent problem because the framework does not scale to 1000 agents without the invariant layer.
A comparison of the two frameworks with invariant layer only on a 1000-agent problem can be found in Fig 16, which shows similar results to the 500-agent problem. [Figure] [Figure]
Superlinear Simulation: We also consider a variant of dynamics in section 4.1,
[Equation]
Due to the nonlinearity in the drift term [Figure] analytic solution or simple numerical representation of the
Nash equilibrium does not exist (Han & Hu, 2019).
The drift rate a is set to 1 . 0 to compensate for the vanishing drift term caused by super-linearity.
Heuristically, the the distribution of control and state should be more concentrated than that of the linear dynamics.
We compare the state and control of a ﬁxed agent i at terminal time against analytic solution and deep FBSDE solution of the linear dynamics with the same coefﬁcients.
Fig. 8 is generated by evaluating the trained deep FBSDE model with a batch size of 50000.
It can be observed that the solution from super-linear dynamics is more concentrated as expected.
The terminal control distribution plot veriﬁes that the super-linear drift term pushes the state back to the average faster than linear dynamics and thus requires less control effort.
Since the numerical solution is not available in the superlinear case, we compare the total loss and training loss between baseline Han & Hu (2019) and our algorithm in the appendix 12.
[Figure]
4. B ELIEF S PACE A UTONOMOUS R ACING
In this section, we demonstrate the general applicability of our framework on an autonomous racing example in belief space.
We consider a 2-car autonomous racing competition problem with racecar dynamics
[Equation]
where x = [ x, y, v, θ ] T represent the x, y position, forward velocity and heading respectively.
Here we assume x, y, v, u acc ∈ R , u steer ∈ [ − 1 , 1] .
The goal of each player is to drive faster than the opponent, stay on the track and avoid collision.
An additional competition loss can be added to facilitate competition between players.
During the competition, players have access to the global augmented states and opponent’s history controller.
Additionally, we assume that stochasticity enters the system through the control channels and have a continuous-time noisy observation model with full-state observation.
The FBSDE derivation of belief space stochastic dynamics is included in Appendix G.
The framework for the racing problem is trained with batch size of 64, and 100 time steps over a time horizon of 10 seconds.
Since all the trials will run over 1 lapse of the circle, here we only show the ﬁrst 8 second result for neatness.
Fig.
13 demonstrate the capability of our framework.
When there is no competition loss, both of cars can stay in the track.
Since there is no competition between two cars, they demonstrate similar behaviors.
When we add competition loss on both cars, both of them try to cut the corner in order to occupy the leading position as shown in the second plot in Fig.
If competition loss is present in only one of the two cars, then the one with competition loss will dominate the game as shown in the botton subplots of Figure 13.
Notably the simulation is running in belief space where all states are estimated with observation noise and additive noise in the system.
The results emphasizes the generalization ability of our framework on more complex systems with higher state and control dimensions.
Fig.
9 shows a single trajectory of each car’s posterior distribution.
C ONCLUSION
In this paper, we propose a scalable deep learning framework for solving multi-agent stochastic differential game using ﬁctitious play.
The framework relies on the FBSDE formulation with importance sampling for sufﬁcient exploration.
In the symmetric game setup, an invariant layer is incorprated to render the framework agnostic to permutatoon of agents and further reduce the memory complexity.
The scalability of this algorithm, along with a detailed sensitivity analysis, is demonstrated in an inter-bank borrowing/lending example.
The framework achieves lower loss and scales to much higher dimensions than the state of the art.
The general applicability of the framework is showcased on a belief space autonomous racing problem in simulation.
A PPENDIX A M ULTI - AGENT HJB D ERIVATION
Applying Bellman’s principle to the value function equation 3 as following
[Equation]
Given the cost function assumption
[Equation]
the inﬁmum can be obtained explicitly using optimal control u ∗ i,m = − R − 1 ( G T i V ix
+ Q T
i x ) .
With that we can obtain the ﬁnal form of the HJB PDE as
[Equation]
B FBSDE D ERIVATION
Given the HJB PDE in equation 4, one can apply the nonlinear Feynman-Kac lemma Han & Hu (2019) to obtain a set of FBSDE as
[Equation]
Note that the forward process X is driven by the control of all agents other than i .
This means that agent
i searches the state space with Brownian motion only to respond to other agents’ strategies.
To increase the efﬁciency of the search, one can add any control from agent i to guide its exploration, as long as the backward process is compensated for accordingly.
In this work, since we consider problems with a closed form solution of the optimal control u i,m , we add it to the forward process for importance sampling from a new set of FBSDEs.
[Equation]
C C ONTINOUS T IME E XTENDED K ALMAN F ILTER
The Partial Observable Markov Decision Process is generally difﬁcult to solve within inﬁnite dimensional space belief.
Commonly, the Value function does not have explicit parameterized form.
Kalman ﬁlter overcome this challenge by presuming the noise distribution is Gaussian distribution.
In order to deploy proposed Forward Backward Stochastic Differential Equation (FBSDE) model in the Belief space, we need to utilize extended Kalman ﬁlter in continuous time Jazwinski (1970) correspondingly.
Given the partial observable stochastic system:
[Equation]
Where f is the stochastic state process featured by a Gaussian noise w ∼ N (0 , Q ) , h is the observation function while v ∼ N (0 , R ) is the observation noise.
Next, we consider the linearization of the stochastic dynamics in equation 20 represented as follows:
[Equation]
one can write the posterior mean state ˆ x and prior covariance matrix P − estimation update rule by
Simon (2006):
[Equation]
We follow the notation in (Simon, 2006), where x is the real state, ˆ x is the mean of state estimated by Kalman ﬁlter based on the noisy sensor observation, P − represents for the covariance matrix of the estimated state, nominal noise values are given as w 0 = 0 and v 0 = 0 , where superscript + is the posterior estimation and − is the prior estimation.
Then we can deﬁne a Gaussian belief dynamics as b (ˆ x k , P − k ) by the mean state ˆ x and variance P − of normal distribution N (ˆ x k , P − k )
The belief dynamics results in a decoupled FBSDE system as follows:
[Equation]
where:
[Equation]
D D EEP S ETS
A function f maps its domain from X to Y .
Domain X is a vector space R d and Y is a continuous space R .
Assume the function take a set as input:
X = { x 1 ...x N } , then the function f is indifferent if it satisﬁes property (Zaheer et al., 2017).
Property 1.
A function f : X → Y deﬁned on sets is permutation invariant to the order of objects in the set.
i.e. For any permutation function π : f ( { x 1 ...x N } )
=
f
( (cid:8) x π (1) ...
x π ( N ) (cid:9) )
In this paper, we discuss when f is a nerual network strictly.
Theorem 1 X has elements from countable universe.
A function f ( X ) is a valid permutation invariant function, i.e invariant to the permutation of X , iff it can be decomposed in the from ρ ( (cid:80) x
∈ X φ ( x )) , for appropriate function ρ and φ .
In the symmetric multi-agent system, each agents is not distinguishable.
This property gives some hints about how to extract the features of − i th agents by using neural network.
The states of − i th agents can be represented as a set:
X = { X 1 , X 2 , ..., X i − 1 , X i +1 , ..., X N } .
We want to design a neural network f which has the property of permutation invariant.
Speciﬁcally, φ is represented as a one layer neural network and ρ is a common nonlinear activation function.
E I NVARIANT L AYER ARCHITECTURE
The architecture of invariant layer is described in Fig.
The input of the layer is the states at time step t .
The Invariant Model module in Fig.
10 is described in Appendix D, where φ is a neural network and ρ is nonlinear activation function.
The speciﬁc conﬁguration of neural network in
Invariant model can be found in J.
Noticing that all the agents has the access to the global states, we deﬁne the state input features of neural network for i th agent as:
[Equation]
with shape of [ BS, N ] .
In the other word, we always put own feature at ﬁrst place.
For each agent i , there exists such feature tensor, then the shape of input tensor will become [ BS, N, N ] for invariant layer.
In invariant layer, we ﬁrst separate the input feature X t into two parts: X t,i and X t, − i .
Then the features of − i th agents X t, − i will be sent to the invariant model.
The shape of X t, − i will be [ B, N, N − 1] where N is the number of agents.
First,we could use neural network to map the feature into N f dimension space, where N f is the feature dimensions.
Then the shape of the tensor will become [ BS, N, N − 1 , N f ] , After summing up the features of all the element in the set, the dimension of the tensor would reduce to [ BS, N, 1 , N f ] , and we denote this feature tensor as F 1 .
However, the memory complexity is O ( N 2 × N f ) which is not tolerable when the number of agent
N increases.
Alternatively, we can simply mapping the feature tensor [ BS, N ] into desired feature dimension N f , then the tensor would become [ BS, N, N f ] , and we denote it to be F 2 .
Now we create another tensor which is the average of features of element in set with size [ BS, 1 , N f ] and we denote it to be ¯ F 2 .
Then we denote F (cid:48)
=
( ¯ F 2 × N − F 2 ) /
( N − 1) which has size of [ BS, N, N f ] .
We can ﬁnd that F (cid:48) 2 = F 1 , and the memory complexity of computing F (cid:48) 2 is just O ( N ) .
The derivation is true if the system is symmetric and the agents are not distinguishable.
The trick can be extended to high state dimension for individual agent.
𝑢 !
𝐿𝑆𝑇𝑀 𝐹𝐶 𝑉 ",!
$ 𝑉 !"
𝐹𝐶 !
𝑢 # 𝐿𝑆𝑇𝑀 𝐹𝐶 𝑉 ",%$ 𝑉 %$ ⋯ ⋯
𝐿𝑆𝑇𝑀 𝐹𝐶 𝑉 ",&’%$ 𝑉 &’%$ 𝑢 $%& 𝐿𝑆𝑇𝑀 𝐹𝐶 𝑉 ",&$ 𝑉 &$ 𝑢 $%# 𝑥 !
𝑥 % 𝑥 &’% 𝑥 &
[Figure]
F I NTERBANK
By pluging the running cost to the HJB function, one can have,
[Equation]
By computing the inﬁmum explicitly, the optimal control of player i is: u i ( X , t ) =
q ( ¯ X − X i ) −
V x,i ( X , t ) .
The ﬁnal form of HJB can be obtained as
[Equation]
[Equation]
Applying Feynman-Kac lemma to equation 22, the corresponding FBSDE system is
[Equation]
G B ELIEF C AR R ACING
The full stochastic model can be written as
[Equation]
Where dw is standard brownian motion.
We consider the problem of two cars racing on circle track.
The cost function of each car is designed as
[Equation]
Where d is Euclidean distance between two cars.
In this showcase [Equation] we use continuous time extended
Kalman Filter to propagate belief space dynamics described in equation 19.
The detailed algorithm for Belief space deep ﬁctitious play FBSDE can be found in Appendix
[Equation]
We introduce the concept of game by using an additional competition cost:
[Equation]
Where x i , y i is the x, y position of i th car.
When i th car is leading, the competition loss will be minor, and it will increase exponentially when the car is trailing.
Thanks to decoupled BSDE structure, each car can measure this competition loss separately and optimize the value function individually.
H A NALYTIC S OLUTION FOR I NTER -B ANK B ORROWING /L ENDING P ROBLEM
The analytic solution for linear inter-bank problem was derived in Carmona et al.
We provide them here for completeness.
Assume the ansatz for HJB function is described as:
[Equation]
[Figure]
Where η ( t ) , µ ( t ) are two scalar functions.
The optimal control under this ansatz is:
[Equation]
[Equation]
By pluginging the ansatz into HJB function derived in 22, one can have,
[Equation]
There exists the analytic solution for the Riccati equation described above as,
[Equation]
Where δ ± = − ( a + q ) ± √ R and R =
( a + q ) 2 + (1 − 1 /N 2 )( (cid:15) − q 2 )
I I MPORTANCE SAMPLING
Fig.
14 demonstrates how fully connected layers with importance sampling would lead to a extreme deep fully connected neural network.
Fig.
15 demonstrates how importance sampling helps increase convergence rate in FBSDE with LSTM backbone.
The experiment is conducted with 50 agents and 50 stages.
All the conﬁguration is identical except the existence of importance sampling.
J E XPERIMENT CONFIGURATIONS
This Appendix elaborates the experiment conﬁgurations for section 4.
For all the simulation in section 4, the number of SGD iteration is ﬁxed as N SGD = 100 .
We are using Adam as optimizer
[Figure]
In section 4.1, For the prediction of initial value function, all of frameworks are using 2 layers feed forward network with 128 hidden dimension.
For the baseline framework, we followed the suggested conﬁguration motioned in Han et al.
At each time steps, V x,i is approximated by three layers of feed forward network with 64 hidden dimensions.
We add batch norm Ioffe &
Szegedy (2015) after each afﬁne transformation and before each nonlinear activation function.
For
Deep FBSDE with LSTM backbone, we are using two layer LSTM parametrized by 128 hidden state.
If the framework includes the invariant layer, the number of mapping features is chosen to be 256.
The hyperparameters of the dynamics is listed as following:
[Equation]
In the simulation, the time horizon is separated into 40 time-steps by Euler method.
Learning rate is chosen to be 1E-3 which is the default learning rate for Adam optimizer.
The initial state for each agents are sampled from the uniform distribution [ δ 0 , δ 1 ] .
Where δ 0 is the constant standard deviation of state X ( t ) during the process.
In the evaluation, we are using 256 new sampled trajectory which are different from training trajectory to evaluate the performance in RSE error and total cost error.
The number of stage is set to be 100 which is enough for all framework to converge.
In section 4.2, the hyperparameter is listed as following:
[Equation]
The observation noise is sampled from Gaussian noise m ∼ N (0 , 0 .
01 I ) .
The time horizon is enrolled into 100 time-steps by Euler method.
In this experiments, the initial value V i is approximated a single trainable scale and V x,i ( t ) is approximated by two layers of LSTM parametrized withhidden dimensions.
The number of stage is set to be 10.
3: Initialize
: θ
, φ4: while LOSS is above certain threshold (cid:15) do 5: for m ← 1 to M do 6: for all i ∈ I in parallel do 7:
Collect opponent agent’s policy f m − 1 LSTM − i ( · ) , f m − 1 FC − i ( · ) 8: for l ← 1 to N gd do 9: for t ← 1 to T − 1 do 10: for j ← 1 to B in parallel do 11:
Compute network prediction for i th player: V ix i ,j,t = f mFC i ( f mLSTM i ( X jt ; θ l − 1 i )) 12:
Compute i th optimal Control: u j,(cid:63)i,t = − R − 1 i ( G T
i V ix
i ,j,t + Q T
i x ji ) 13:
Infer − i th players’ network prediction: V ix − i ,j,t = f m − 1 FC − i ( f m − 1 LSTM − i ( X t ; θ − i ))
Compute − i th optimal Control: u j,(cid:63) − i,t = − R − 1 − i ( G T − i V jx −
i ,t + Q T − i x j − i ) 15:
Sample noise ∆ W j ∼ N (0 , ∆ t ) 16:
Propagate FSDE:
X jt +1 =
f FSDE ( X jt , u j,(cid:63)i,t , u j, ∗− [Equation]
Propagate BSDE: V ij
[Equation]
18: end for 19: end for 20:
Compute loss: L = 1 B (cid:80)
Bj =1 ( V i,(cid:63)j,T − V ij,T )21:
Gradient Update: θ l , φ l 22: end for 23: end for 24: end for 25: end while