Deep Generalized Schrödinger Bridge
Abstract
Mean-Field Game (MFG) serves as a crucial mathematical framework in modeling the collective behavior of individual agents interacting stochastically with a large population.
In this work, we aim at solving a challenging class of MFGs in which the differentiability of these interacting preferences may not be available to the solver, and the population is urged to converge exactly to some desired distribution.
These setups are, despite being well-motivated for practical purposes, complicated enough to paralyze most (deep) numerical solvers.
Nevertheless, we show that
Schrödinger Bridge — as an entropy-regularized optimal transport model — can be generalized to accepting mean-ﬁeld structures, hence solving these MFGs.
This is achieved via the application of Forward-Backward Stochastic Differential Equations theory, which, intriguingly, leads to a computational framework with a similar structure to Temporal Difference learning.
As such, it opens up novel algorithmic connections to Deep Reinforcement Learning that we leverage to facilitate practical training.
We show that our proposed objective function provides necessary and sufﬁcient conditions to the mean-ﬁeld problem.
Our method, named Deep Generalized Schrödinger Bridge ( DeepGSB ), not only outperforms prior methods in solving classical population navigation MFGs, but is also capable of solving 1000-dimensional opinion depolarization , setting a new state-of-the-art numerical solver for high-dimensional MFGs. Our code will be made available at https://github.com/ghliu/DeepGSB .
Introduction
[Figure]
a new algorithmic connection between Schrödinger
Bridge (SB) and modelbased DeepRL for solving high-dimensional MFGs.
On a scorching morning, you navigated through the crowds toward the ofﬁce.
As you walked through a crosswalk, you were pondering the growing public opinion on a new policy over the past week, and were suddenly interrupted by the honking as the trafﬁc started moving...
From navigation in crowds to propagation of opinions and trafﬁc movement, examples of individual agents interacting with a large population are widespread in daily life and, due to their prevalence, appear as an important subject in multidisciplinary scientiﬁc areas, including economics [1, 2], opinion modeling [3–5], robotics [6, 7], and more recently machine learning [8–10].
Mathematically, the decision-making processes under these scenarios can be characterized by the Mean-Field Game [11–13] (MFG) , which models a noncooperative differential game on a ﬁnite horizon between a continuum population of rational agents.
Let u ( x, t ) be the value
Our DeepGSB is capable of solving a much wider class of MFGs in higher dimensional state spaces.
[Table]
function, also known as optimal cost-to-goal, that governs agents’ policies at each state x ∈ R d and time t ∈ [0 , T ] , and denote the resulting population density by ρ ( · , t ) ∈ P ( R d ) , where P ( R d ) is the set of probability measures on R d .
At the Nash equilibrium where no agent has the incentive to change his/her decision, MFG, at its most general form, solves the following partial differential equations (PDEs):
[Equation]
where ∇ , ∇· , and ∆ are respectively the gradient, divergence, and Laplacian operators.
These two PDEs are respectively known as the Hamilton-Jacobi-Bellman (HJB) and Fokker-Plank (FP) equations, which characterize the evolution of u ( x, t ) and ρ ( x, t ) .
They are coupled with each other through the Hamiltonian H ( x, p, ρ ) : R d × R d × P ( R d ) → R , which describes the dynamics of the game, and the mean-ﬁeld interaction F ( x, ρ ) : R d × P ( R d ) → R , which quantiﬁes the agent’s preference when interacting with the population.
The terminal condition G typically penalizes deviations from some desired target distribution ρ target , e.g., G ≈ D KL ( ρ ( · , T ) || ρ target ( · )) .
Given a solution ( u, ρ ) to (1), each agent acts accordingly and follows a stochastic differential equation(SDE)
[Equation]
where W t ∈ R
d is the Wiener process and σ ∈ R is some diffusion scalar.
At the mean-ﬁeld limit, i.e., when the number of agents goes to inﬁnity, the collective behavior of (2) yields the density ρ ( · , t ) .
Numerical methods for solving (1) have advanced rapidly with the aid of machine learning.
Seminar works such as Ruthotto et al.
[14] and Lin et al.
[15] approximated ( u, ρ ) with deep neural networks (DNNs) and directly penalized the violation of PDEs.
Despite showing preliminary successes, the underlying dynamics (2) were either degenerate ( e.g., σ := 0 )
[14], or completely discarded by instead regressing network outputs on the entire state space [15], which can scale unfavorably as the dimension d grows.
An alternative that avoids both limitations, i.e., it keeps the full stochastic dynamics in (2) while being computationally scalable, is to recast these PDEs to a set of forwardbackward SDEs (FBSDEs) by applying the nonlinear Feynman-Kac Lemma [17–19].
The FBSDEs analysis appears extensively in the theoretical study of MFG [20–23], yet development of scalable
FBSDEs-based solver has remained, surprisingly, limited.
Our work contributes to this direction.
Since ρ target is known in prior, in many cases there are direct interests to seek an optimal policy that guides the agents from an initial distribution ρ 0 to the exact ρ target , while respecting the structure of
MFG, particularly the MF interaction F ( x, ρ ) .
Lifting (1) to this setup, however, is highly nontrivial.
Indeed, replacing the soft penalty at u ( x, T ) =
D KL ( ρ || ρ target ) with a hard distributional constraint at ρ ( x, T ) =
ρ target yields an HJB whose boundary condition can only be deﬁned implicitly through
FP, which now contains two distributional constraints and resembles an optimal transport problem.
As such, despite being well-motivated, most prior methods have struggled to extend to this setup.
In this work, we show that Schrödinger Bridge ( SB ), as an entropy-regularized optimal transport problem [24–28], provides an elegant recipe for solving this challenging class of MFGs with distributional boundary constraints ( ρ 0 , ρ target ) .
Although SB is traditionally set up with F :=
[29–31], we show that SB-FBSDE [27], an FBSDE-based method for solving SB, can be generalized to accept nontrivial F ; hence solving MFG.
Interestingly, the new FBSDEs system admits a similar computational structure to temporal difference (TD) learning, leading to a framework that narrows the gap between SB and Deep Reinforcement Learning (DeepRL); see Fig.
This connection enables our method to take advantage of DeepRL techniques, such as target networks, replay buffer, actor-critic, etc , and, more importantly, to handle a wide class of MF interactions that need not be continuous nor differentiable.
This is in contrast to most existing works, which require differentiable [14, 15] or quadratic [32] structure on F , or discretize the state space [16].
We validate our method, called Deep Generalized Schrödinger Bridge (DeepGSB) , on various challenging MFGs from crowd navigation to high-dimensional opinion depolarization (where d = 1000), setting a state-of-the-art record in the area of numerical MFG solvers.
In summary, we present the following contributions.
• We present a novel numerical method, rooted in Schrödinger Bridge (SB), for solving a challenging class of Mean-Field Game where the population needs to converge exactly to the target distribution.
• The resulting method, DeepGSB , generalizes prior SB results to accepting ﬂexible mean-ﬁeld interaction ( e g., non-differentiable) and enjoys modern training techniques from DeepRL. • DeepGSB achieves promising empirical results in navigating crowd motion and depolarizing 1000-dimensional opinion dynamics, setting a new state-of-the-art numerical MFG solver.
Preliminary on Schrödinger Bridge (SB)
[Figure]
SDEs in SB, which are minimum-energy solution when (Ψ , (cid:98) Ψ) obey the PDEs in (4)
The SB problem was originally introduced in the 1930s for quantum mechanics [29, 33] and later draws broader interests with its connection to optimal transport and control [34–37].
Given a pair of boundary distributions ( ρ 0 , ρ T ) ,
SB seeks an optimal pair of stochastic processes of the forms:
[Equation]
While X t is a standard stochastic process starting from ρ 0 , ¯ X s evolves along the “ reversed ” time coordinate s :
= T − t from ρ T .
The base drift f and diffusion σ are typically known in prior and related to the Hamiltonian H .
Suppose Ψ , (cid:98) Ψ ∈ C 2 , 1 ( R d , [0 , T ]) solve the following coupled PDEs,
[Equation]
then the theory of SB suggests that the SDEs in (3) are optimal solution to an entropy-regularized ( i.e., minimum control) optimization problem.
Furthermore, the path-wise measure induced by (3a) along t ∈ [0 , T ] is equal almost surely to the path-wise measure induced by (3b) along s :
= T − t .
In other words, the two SDEs in (3) can be thought of as the “ reversed ” process to each other; and hence we also have X T ∼ ρ T and ¯ X T ∼ ρ 0 (see Fig. 2).
Due to the coupling constraints at the boundaries, solving (4) is no easier than solving (1).
Fortunately, recent advances [27, 28] have demonstrated a computationally scalable numerical method via the application of the nonlinear Feynman-Kac (FK) Lemma — a mathematical tool that recasts certain classes of PDEs into sets of forward-backward SDEs (FBSDEs) via some transformation.
These nonlinear FK transformations are parametrized in SB-FBSDE [27] by some DNNs with θ and φ , i.e.,
[Equation]
and the FBSDEs resulting from (4) and (5) yield the following objectives (see Appendix A.2):
[Equation]
The following lemma, as a direct consequence of Vargas [38], suggests that these objectives can be interpreted as the KL divergences between the parametrized path measures.
Lemma 1.
Let q θ and q φ be the path-wise densities of the parametrized forward and backward SDEs
[Equation]
Then, we have D
[Equation] and [Equation] Proof.
See Appendix A.3.2. Lemma 1 suggests that alternative minimization between L IPF ( φ ) and L IPF ( θ ) is equivalent to performing iterative KL projection [39], and is hence equivalent to applying the Iterative Proportional Fitting [40] (IPF) algorithm to solve parametrized SBs [24, 25].
Deep Generalized Schrödinger Bridge (DeepGSB) 3.1 Connection between the coupled PDEs in MFG and SB
[Figure] We begin by ﬁrst stating our problem of interest — MFG with hard distributional constraints ( ρ 0 , ρ target ) — in its mathematical form.
Similar to prior works [14, 15], we will adopt the controlafﬁne Hamiltonian, H ( x, ∇ u, ρ ) :
= 12 (cid:107) σ ∇ u (cid:107) 2 − ∇ u (cid:62) f
( x, ρ ) , given some base drift f and diffusion scalar σ .
Substituting this control-afﬁne Hamiltonian into the PDEs in (1) yields
[Equation]
which, as we brieﬂy discussed in Sec.1, differ from (1) in that the boundary condition of the HJB, u ( x, T ) , is now absorbed into FP and deﬁned implicitly through ρ ( x, T )
=
ρ target ( x ) .
Since analytic conversion between the boundary conditions of (1) and (7) exists only for highly degenerate 3 cases [41], this seemingly innocuous change sufﬁces to paralyze most prior methods.
Nevertheless, as (7) now describes a transformation between two distributions (from FP) while obeying some optimality (from HJB), it suggests a deeper connection to optimal transport, and hence the SB.
To bridge these new MFG PDEs (7) to the PDEs appearing in SB (4), we follow standard treatment [30] and apply the Hopf-Cole transform [42, 43]:
[Equation]
which, after some algebra (see Appendix A.4.1 for details), yields the following PDEs:
[Equation]
It can be seen that (9) generalizes (4) by introducing the MF interaction F .
Let (Ψ , (cid:98) Ψ) be the solution to these new MF-extended PDEs in (9), and recall the Hamiltonian adopted in (7), one can ﬁnd that
[Equation]
That is, the agent’s dynamic (2) coincides with the forward SDE (3a) in SB.
Hence, we have connected the MFG (1) and SB (4) frameworks through the PDEs in (7) and (9); see Fig. 3.
3.2 Generalized SB-FBSDEs with mean-ﬁeld interaction
With (9), we are ready to present our result that generalizes prior FBSDE for SB to MF interaction.
Theorem 2 (Generalized SB-FBSDEs) .
Suppose Ψ , (cid:98) Ψ ∈ C 2 , 1 and let f, F satisfy usual growth and
Lipchitz conditions [44, 45] .
Consider the following nonlinear FK transformations applied to (9):
[Equation]
where X t follows (3a) with X 0 ∼ ρ 0 .
Then, the resulting FBSDEs system takes the form:
[Equation]
r.
[Equation]
.(3a) [Equation]
Now, consider a similar transformation in (9) but instead w.r.t.
the “reversed” SDE ¯ X s ∼ (3b) and ¯
X 0 ∼ ρ target , i.e., Y s ≡ Y ( ¯ X s , s ) = log Ψ
( ¯ X s , s ) , and etc.
The resulting FBSDEs system reads
[Equation]
.r.t.(3b) :
[Equation]
Since Y t + (cid:98) Y t = log ρ ( X, t ) by construction, the functions f t and F t in (11) take the arguments f t := f t ( X t , exp( Y t + (cid:98) Y t )) and
F t :=
F t ( X t , exp( Y t + (cid:98) Y t )) .
Similarly, we have f s :=
f s ( ¯ X s , exp( Y s + (cid:98) Y s )) and F s :=
F s ( ¯ X s , exp( Y s + (cid:98) Y s )) in (12). Proof.
See Appendix A.3.3.
Just like how (9) generalizes (4), our results in Theorem 2 also generalize the ones appearing in vanilla
SB-FBSDE [27] (see (22) in Appendix A.2) by introducing nontrivial MF interaction F .
Despite seemingly complex compared to the original PDEs (9), these FBSDEs systems — namely (11) and (12) — stand as the foundation for developing scalable numerical methods, as they describe precisely how the values of Y ≡ log Ψ and
(cid:98) Y ≡ log (cid:98) Ψ shall change along the optimal SDEs (notice, e.g., that both Y t and Z t are functions of X t from (10)).
Essentially, the nonlinear FK Lemma provides a stochastic representation (in terms of Y and (cid:98) Y ) of the PDEs in (9) by expanding them w.r.t.
the optimal SDEs in (3) using the Itô formula [46].
Consequently, rather than solving the PDEs (9) in the entire function space as in the prior work [15], it sufﬁces to solve them locally around high probability regions characterized by (3), which leads to computationally scalable methods.
3.3 Design of the computational framework
Looking from Theorem 2, it sufﬁces to approximate Y θ ≈ Y and (cid:98) Y φ ≈ (cid:98) Y with some parametrized functions (we use DNNs), since one may infer Z θ ≈ σ ∇ Y θ and (cid:98)
Z φ ≈ σ ∇ (cid:98) Y φ , as suggested by (10), and then solve for ( X t , ¯ X s ) via (11a, 12a).
Below, we explore options of designing training objectives for ( θ, φ ) , with the aim to encourage ( Y θ , (cid:98) Y φ ) to satisfy the FBSDEs systems in (11, 12).
Option 1: L IPF .
Given how Theorem 2 generalizes the one in [27] (see (22) in Appendix A.2), it is natural to wonder if adopting the computation used to derive (6), e.g., L IPF ( φ ) : =
(cid:82) E [ d Y θt + d (cid:98) Y φt ] , sufﬁces to reach the FBSDE (11).
This is, unfortunately, not the case as one can verify that
[Equation]
Despite that (11) differs from (22) by the extra terms “ + F t ” in (11b) and “ − F t ” in (11c), the two terms cancel out in the sum of d Y θt + d (cid:98) Y φt , thereby yielding the same objectives that do not depend on F .
This implies that naively optimizing L IPF from [27] is insufﬁcient for solving FBSDE systems with nontrivial F .
We must seek additional objectives, if any, in order to respect the MF structure.
Option 2: L IPF + Temporal Difference objective L TD .
Let us revisit the relation between the FBSDEs (11, 12) and their PDEs counterparts — but this time the HJB in (7).
Take ( X t , Y t ) for example: The fact that Y t = log Ψ
( X t , t ) =
− u ( X t , t ) suggests an alternative interpretation of Y t as the stochastic representation of the HJB, which, crucially, can be seen as the continuous-time analogue of the Bellman equation [47].
Indeed, discretizing (11b) with some ﬁxed step size δt yields
[Equation]
which resembles a (non-discounted) Temporal Difference (TD) [48, 49] except that, in addition to the standard “ rewards ” (in terms of control and state costs), we also have a stochastic term.
This stochastic term, which vanishes in the vanilla Bellman equation upon taking expectations, plays a crucial role in characterizing the inherited stochasticity of the value function Y t .
With this interpretation in mind, we can construct suitable TD targets for our FBSDEs systems as shown below.
Proposition 3 (TD objectives L TD for (11, 12)) .
The single-step TD targets take the forms:
[Equation]
with (cid:100) TD 0 := log ρ 0 − Y θ 0 and TD 0 := log ρ target − (cid:98) Y φ 0 , and the multi-step TD targets take the forms:
[Equation]
15) where δ (cid:98) Y t :=
[Equation]
TD
[Equation]
Given these TD targets, we can construct
L TD
[Equation]
Proof. See Appendix A.3.4. It can be readily seen that the single-step TD targets in (14) obey a similar structure to (13), except deriving from different SDEs (11c, 12b).
Doing so reduces the computational overhead, as the related objectives for each parameter, e.g., L IPF ( θ ) and L TD ( θ ) , can be evaluated from the same expectation.
In practice, we ﬁnd that the multi-step objectives often yield better performance, as consistently observed in the DeepRL literature [50–52].
Additionally, common practices such as computing (cid:100) TD t and TD s using the exponential moving averaging ( i.e., target values) and replay buffers also help stabilize training.
Finally, the fact that the TD targets in (16) appear as the regressands implies that from a computational standpoint, the MF interaction F needs not to be continuous or differentiable.
Necessity and sufﬁciency of L IPF + L TD .
It remains unclear whether appending L TD to the objective sufﬁces for ( Y θ , (cid:98) Y φ ) to satisfy the FBSDEs (11, 12).
Below, we provide a positive result.
Proposition 4.
The functions ( Y θ , Z θ , (cid:98) Y φ , (cid:98) Z φ ) satisfy the FBSDEs (11,12) in Theorem 2 if and only if they are the minimizers of the combined losses L ( θ, φ ) :
=
L IPF ( φ ) + L TD ( φ ) + L IPF ( θ ) + L TD ( θ ) .
Proof.
See Appendix A.3.5.
Input: ( Y θ , (cid:98) Y φ , σ ∇ Y θ , σ ∇ (cid:98) Y φ ) for critic or ( Y θ , (cid:98) Y φ , Z θ , (cid:98) Z φ ) for actor-critic parametrization.
repeat
Sample X θ ≡ { X θt , Z θt , δW t } t ∈ [0 ,T ] from the forward SDE (11a); add X θ to replay buffer B . for k = 1 to K do
Sample on-policy X θ on and off-policy X θ off samples respectively from X θ and B . Compute L ( φ ) = L IPF ( φ ; X θ on ) + L TD ( φ ; X θ on ) + L TD ( φ ; X θ off ) + L FK ( φ ; X θ on ) . Update φ with the gradient ∇ φ L ( φ ) .
end for Sample ¯ X φ ≡ { ¯ X φs , (cid:98) Z φs , δW s } s ∈ [0 ,T ] from the backward SDE (12a); add ¯ X φ to replay buffer ¯ B . for k = 1 to K do Sample on-policy ¯ X φ on and off-policy ¯ X φ off samples respectively from ¯ X φ and ¯ B .
Compute L ( θ ) = L IPF ( θ ; ¯ X φ on ) + L TD ( θ ; ¯ X φ on ) + L TD ( θ ; ¯ X φ off ) + L FK ( θ ; ¯ X φ on ) . Update θ with the gradient ∇ θ L ( θ ) .
end for until converges Proposition 4 asserts the validity of the combined objectives L IPF + L TD in solving the generalized SB-FBSDEs in Theorem 2, and hence the MFG problem in (7).
It shall be interpreted as follows:
The minimizer of L IPF , as implied in Lemma 1, would always establish a valid “bridge” transporting between the boundary distributions ρ 0 and ρ target ; yet, without further conditions, this bridge needs not obey a “Schrödinger” bridge. While general IPF and Sinkhorn [16, 32], upon proper initialization or discretization, provides one way to ensure the convergence toward the “S”B, our Propositionsuggests an alternative by introducing the TD objectives L TD . This gives us ﬂexibility to handle generalized SB in MFGs where F becomes nontrivial or non-convex. Further, it naturally handles non-differentiable F , which can offer extra beneﬁts in many cases.
Option 3: L IPF + L TD + FK objective L FK .
Though it seems sufﬁcient to parametrize ( Y θ , (cid:98) Y φ ) then infer Z θ := σ ∇ Y θ and (cid:98) Z φ := σ ∇ (cid:98) Y φ , as suggested in previous options, in practice
we ﬁnd that parametrizing ( Z θ , (cid:98) Z φ ) with two additional DNNs then imposing the following FK objective, i.e., [Equation]
FK [Equation]
often offers extra robustness. These objectives aim to ensure that the nonlinear FK (10) holds.
Our DeepGSB is summarized in Alg. 1. Hereafter, we refer Option 2 and 3 respectively to DeepGSB critic and DeepGSB actor-critic , as Y θ and Z θ play similar roles of critic and actor networks [53, 54]. Remarks on convergence. Despite Alg.
1 sharing a similar alternating structure to IPF [16, 24, 27], the combined objective, e.g., L ( φ ) ∝ D KL ( ρ θ || ρ φ ) + E ρ θ [ L TD ( φ )] (cid:54) = D KL ( ρ φ || ρ θ ) is not equivalent to the (reversed) KL appearing in IPF. Instead, DeepGSB may be closer to trust region optimization [55], as both iteratively update the policy using samples from the previous stage while subjected to some KL penalty: π ( i +1) =
arg min π D KL ( π ( i ) ||
π ) + E π ( i ) [ L ( π )] . Hence, one can expect
DeepGSB to admit similar monotonic improvement and local convergence properties.
We leave more discussions to Appendix A.4.2.
Experiment
Instantiation of MFGs.
We validate our DeepGSB on two classes of MFGs, including classical crowd navigation ( d =2) and high-dimensional ( d =1000) opinion depolarization.
For crowd navigation, we consider three MFGs appearing in prior methods [14, 15], including (i) asymmetric obstacle avoidance, (ii) entropy interaction with a V-shape bottleneck, and (iii) congestion interaction on an S-shape tunnel.
We will refer to them respectively as GMM, V-neck, and S-tunnel.
The obstacles and the initial/target Gaussian distributions ( ρ 0 , ρ target ) are shown in Fig.
For opinion depolarization, we set ρ 0 and ρ target to two zero-mean Gaussians with varying variances for representing the initially polarized and desired moderated opinion distributions.
Finally, we consider zero and constant base drift f respectively for GMM and V-neck/S-tunnel, and adopt the polarized MF dynamics [4] for opinion MFG; see Sec.
4.2 for a detailed discussion.
GMM -   -
V-neck -   -
S-tunnel [Figure] MFGs and the high-dimensional opinion MFG [Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] MF interactions F . We follow standard treatments from the MFG theory [11] by noting that given a functional F ( ρ ) that quantiﬁes the MF cost w.r.t.
the population ρ , e.g., F entropy :=
E ρ [log ρ ] or F congestion := E x,y ∼ ρ [ [Equation]
1 ] , one can derive its associated MF interaction function F ( x, ρ ) by taking the functional derivative, i.e., δ F ( ρ ) δρ ( x )
=
F ( x, ρ ) .
Hence, the entropy and congestion MF interactions, together with the obstacle cost, follow (see Appendix A.4.3 for the derivation):
[Equation]
where 1 obs ( · ) is the (discontinuous) indicator of the problem-dependent obstacle set.
We summarize the MF interaction in Table 2.
Architecture & Hyperparameters.
We parameterize the functions with fully-connected DNNs for crowd navigation, and deep residual networks for high-dimensional opinion MFGs.
All networks adopt sinusoidal time embeddings and are trained with AdamW [56].
All SDEs in (11, 12) are solved with the Euler-Maruyama method.
Due to space constraints, we will focus mostly on the results of a ctor- c ritic parametrization DeepGSB-ac , and leave the discussion of c ritic parametrization DeepGSB-c , along with additional experimental details, to Appendix A.5.
4.1 Two-dimensional crowd navigation
Figure 5 shows the simulation results of our DeepGSB-ac on three crowd navigation MFGs.
We also report existing numerical methods [14–16] that are best-tuned on each MFG (see Appendix A.5.1 for details) but note that in practice, they either require softening F to be differentiable [14, 15] to yield reasonable results, or discretizing the state space [16], which can lead to prohibitive complexity.
We ﬁrst compare to Chen [16] on GMM (see Fig.
5a) as their method only applies to non-MF interaction, i.e., F :=
F ( x ) .
While DeepGSB-ac guides the population to smoothly avoid all obstacles (notice the sharp contours of Y around them), [16] struggles to escape due to the discretization of the state space (hence the policy).
To better examine the effect of ρ in F ( x, ρ ) , we next simulate the dynamics on V-neck (see Fig. 5b) with and without the MF interaction.
It is clear that our
DeepGSB-ac encourages the population to spread out once the entropy interaction F entropy is enabled, yet a similar effect is barely observed in [14].
We observed difﬁculties in balancing the MF interaction
F and the terminal penalty D KL ( ρ T || ρ target ) for [14], yet this problem is alleviated in DeepGSB by construction.
Lastly, we validate the robustness of our method on S-tunnel (see Fig.
5c) w.r.t.
varying diffusions σ = { 0 . 5 , 1 , 2 } .
Again, our DeepGSB-ac reaches the same ρ target despite being subject to different levels of stochasticity.
This is in contrast to [15], which, due to discarding the SDE dynamics, necessitates solving PDEs on the entire state space that may be sensitive to hyperparameters.
In short, our DeepGSB outperforms prior methods [14–16] by better respecting obstacles and MF interactions yet without losing convergence to ρ target , and its performance remains robust across different MFGs.
4.2 High-dimensional opinion depolarization
Next, we showcase our DeepGSB in solving high-dimensional MFGs in the application of opinion dynamics [3–5], where each agent now possesses a d -dimensional opinion x ∈ R d that evolves through the interactions with the population.
In light of increasing recent attention, we consider a particular class of opinion dynamics known to yield strong polarization [4], i.e., the agents’ opinions tend to partition into groups holding diametric views.
Take the party model [4] for instance:
Given a random information ξ ∈ R d sampled from some distribution independent of ρ , each agent updates [Figure]
[Figure] the opinion following a normalized polarize dynamic ¯ f polarize = f polarize / (cid:107) f polarize (cid:107) 12 , where f polarize ( x, ρ ; ξ ) : =
E y ∼ ρ
[ a ( x, y ; ξ )¯ y ] , a ( x, y ; ξ ) :
= (cid:26)if sign( (cid:104) x, ξ (cid:105) )
= sign( (cid:104) y, ξ (cid:105) )
− 1 otherwise , (18) and ¯ y
= y / (cid:107) y (cid:107) 12 .
The agreement function a ( x, y ; ξ ) indicates whether the two opinions x and y agree on the information ξ .
Intuitively, the dynamic in (18) suggests that the agents tend to be receptive to opinions they agree with, and antagonistic to opinions they disagree with.
As shown in Fig.
6a, this behavioral assumption, also known as biased assimilation [57, 58], can easily lead to polarization.
We can apply our MFG framework (7) to this polarized base drift (18), where, starting from some weakly polarized ρ 0 , we seek a policy that compensates the polarization tendency and helps guide the opinion towards a moderated distribution ρ target (assuming as Gaussian for simplicity).
We consider the entropy MF interaction F entropy as it encourages opinions diversity before reaching consensus.
As shown in Fig.
6b [Table]
in both lower- ( d =2) and higher- ( d =1000) dimensions [Table] our DeepGSB-ac successfully guides the opinion towards the desired distribution centered symmetrically at 0 ∈ R d , thereby mitigates the polarization.
Results of DeepGSB-c remain similar despite being more sensitive to hyperparameters; see Appendix A.5.2.
We highlight these state-of-the-art results on a challenging class of MFGs that, comparing to existing methods [14, 15], consider a more difﬁcult mean-ﬁeld dynamic ( f ( x, ρ ) vs. f ( x ) ) in an order of magnitude higher dimension ( d =1000 vs. d =100). 4.3 Discussion [Table] DeepGSB-ca vs. DeepGSB-c.
Table 3 compares actor-critic with critic parametrizations on crowd navigation MFGs.
While DeepGSB-ac typically achieves lower Wasserstein and TD errors, it seldom closes the consistency gap of L FK ( θ ) , as opposed to DeepGSB-c. In practice, the results of DeepGSB-c are visually indistinguishable from DeepGSB-ac, despite the different contours of Y ; see Appendix A.5.2 for more discussions.
DeepGSB works with intractable ρ target .
While the availability of the target density ρ target is a common assumption adopted in prior works [14, 15], in which ρ target is involved in computing the boundary loss, in most real-world applications, ρ target is seldom available.
Here, we show that
DeepGSB works well without knowing ρ target (and ρ 0 ) so long as we can sample from X 0 ∼ ρ 0 and ¯
X 0 ∼ ρ target .
This is similar to the setup of generative modeling [27].
In Fig.
9 (see Appendix A.5.2), we show that DeepGSB trained without the initial and terminal densities can converge equally well.
Crucially, this is because DeepGSB replies on a variety of other mechanisms ( e.g., self-consistency in single-step TD objectives and KL-matching in IPF objective) to generate equally informative gradients.
This is in contrast to [14, 15] where the training signals are mostly obtained by differentiating through
D KL ( ρ || ρ target ) ; consequently, their methods fail to converge in the absence of ρ target .
Conclusion, Limitation, and Boarder Impact
We present DeepGSB , a new numerical method for solving a challenging class of MFGs with distributional boundary constraints.
By generalizing prior FBSDE theory for Schrödinger Bridge to accepting mean-ﬁeld interactions, we show that practical training can be achieved via an intriguing algorithmic connection to DeepRL.
Our DeepGSB outperforms prior methods in crowd navigation
MFGs and sets a new state-of-the-art record in depolarizing 1000-dimensional opinion MFGs.
DeepGSB is mainly developed for MFGs in unconstrained state spaces such as R d .
Yet, it may be necessary to adopt domain-speciﬁc structures, e.g., constrained state spaces.
Additionally, the divergence in the IPF objectives may scale unfavorably as the dimension grows.
This may be mitigated by adopting a simpler regression from De Bortoli et al.
Study of Mean-Field Games (MFGs) possesses its own societal inﬂuence.
Thus, as a MFG solver, DeepGSB may pose a potential impact in offering solutions to previously unsolvable MFGs under more practical settings.
Acknowledgments and Disclosure of Funding
The authors would like to thank Yu-ting Chiang, Augustinos and Molei for their helpful supports and kind discussion.
The authors would also like to thank the anonymous Reviewer A4H3 for his/her initially harsh yet constructive comments on OpenReview, which led to substantial improvements of the theoretical results during rebuttal.