Deep Generalized Schrödinger Bridge
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Mean-Field Game (MFG) serves as a crucial mathematical framework in modelingthe collective behavior of individual agents interacting stochastically with a largepopulation.
In this work, we aim at solving a challenging class of MFGs in whichthe differentiability of these interacting preferences may not available to the solver,and the population is urged to converge exactly to some desired distribution.
Thesesetups are, despite being well-motivated for practical purposes, complicated enoughto paralyze most (deep) numerical solvers.
Nevertheless, we show that Schrödinger
Bridge — as an entropy-regularized optimal transport model — can be generalizedto accepting mean-ﬁeld structures, hence solving these MFGs.
This is achievedvia the application of Forward-Backward Stochastic Differential Equations theory,which, intriguingly, leads to a computational framework with a similar structure to
Temporal Difference learning.
As such, it opens up novel algorithmic connectionsto
Deep Reinforcement Learning that we leverage for facilitating practical training.
Our method, named Deep Generalized Schrödinger Bridge ( DeepGSB ), not onlyoutperforms prior methods in solving classical population navigation MFGs, but isalso capable of solving 1000-dimensional opinion depolarization , setting a newstate-of-the-art numerical solver for high-dimensional MFGs.
Introduction
In a scorching morning, you navigated through the crowds toward theofﬁce.
As you walked through a crosswalk, you were pondering thegrowing public opinion on a new policy over the past week and weresuddenly interrupted by the honking as the trafﬁc started moving...
From navigation in crowds to propagation of opinion and trafﬁc movement, examples of individual agents interacting with a large populationare widespread in daily life and, due to their prevalence, appear asan important subject in multidisciplinary scientiﬁc areas, includingeconomics [1, 2], opinion modeling [3–5], robotics [6, 7], and morerecently machine learning [8–10].
Mathematically, the decision-making processes under these scenarioscan be characterized by the Mean-Field Game [11–13] (MFG) , whichmodels a noncooperative differential game on a ﬁnite horizon betweena continuum population of rational agents.
Let u ( x, t ) be the value function, also known as optimalcost-to-goal, that governs agents’ policies at each state x ∈ R d and time t ∈ [0 , T ] , and denote theresulting population density by ρ ( · , t )
∈ P ( R d ) , where P ( R d ) is the set of probability measures on
R d .
At the Nash equilibrium where no agent has the incentive to change his/her decision, MFG, at its
Our DeepGSB is capable of solving a much wider class of MFGs in higher dimensional state spaces.
[Table]
most general form, solves the following partial differential equations (PDEs):
[Equation]
where ∇ , ∇· , and ∆ are respectively the gradient, divergence, and Laplacian operators.
2 Thesetwo PDEs are respectively known as the Hamilton-Jacobi-Bellman (HJB) and Fokker-Plank
(FP)equations, which characterize the evolution of u ( x, t ) and ρ ( x, t ) .
They are coupled with each otherthrough the Hamiltonian H ( x, p, ρ ) : R d × R d × P ( R d ) → R , which describes the dynamics ofthe game, and the mean-ﬁeld interaction F ( x, ρ ) :
R d × P ( R d ) → R , which quantiﬁes the agent’spreference when interacting with the population.
The terminal condition G typically penalizesdeviations from some desired target distribution ρ target , e.g., G ≈ D KL ( ρ ( · , T ) ||
ρ target ( · )) .
Given asolution ( u, ρ ) to (1), each agent acts accordingly and follows a stochastic differential equation(SDE)
[Equation]
where W t ∈ R
d is the Wiener process and σ ∈ R is some diffusion scaler.
At the mean-ﬁeld limit, i.e.,when the number of agents goes to inﬁnity, the collective behavior of (2) yields the density ρ ( · , t ) .
Numerical methods for solving (1) have advanced rapidly with the aid of machine learning.
Seminarworks such as Ruthotto et al.
[14] and Lin et al.
[15] approximated ( u, ρ ) with deep neural networks(DNNs) and directly penalized the violation of PDEs.
Despite showing preliminary successes, theunderlying dynamics (2) were either degenerate ( e.g., σ := 0 )
[14], or completely discarded byinstead regressing network outputs on the entire state space [15], which can scale unfavorably asthe dimension d grows.
An alternative that avoids both limitations, i.e., it keeps the full stochasticdynamics in (2) while being computationally scalable, is to recast these PDEs to a set of forwardbackward SDEs (FBSDEs) by applying the nonlinear Feynman-Kac Lemma [17–19].
The FBSDEsanalysis appears extensively in the theoretical study of MFG [20–23], yet development of scalable
FBSDEs-based solver has remained, surprisingly, limited.
Our work contributes to this direction.
Since ρ target is known in prior, in many cases there are direct interests to seek an optimal policy thatguides the agents from an initial distribution ρ 0 to the exact ρ target , while respecting the structure of
MFG, particularly the MF interaction F ( x, ρ ) .
Lifting (1) to this setup, however, is highly nontrivial.
Indeed, replacing the soft penalty at u ( x, T ) =
D KL ( ρ || ρ target ) with a hard distributional constraintat ρ ( x, T ) =
ρ target yields an HJB whose boundary condition can only be deﬁned implicitly through
FP, which now contains two distributional constraints and resembles an optimal transport problem.
As such, despite being well-motivated, most prior methods have struggled to extend to this setup.
In this work, we show that Schrödinger Bridge ( SB ), as an entropy-regularized optimal transportproblem [24–28], provides an elegant recipe for solving this challenging class of MFGs with distributional boundary constraints ( ρ 0 , ρ target ) .
Although SB is traditionally set up with F := {+
 [29–31],we+} show that SB-FBSDE [27], an FBSDE-based method for solving SB, can be generalized toaccept nontrivial F ; hence solving MFG.
Interestingly, the new FBSDEs system admits a similarcomputational structure to temporal difference (TD) learning, leading to a framework that narrows thegap between SB and Deep Reinforcement Learning (DeepRL); see Fig.
This connection enablesour method to take advantage of DeepRL techniques, such as target networks, replay buffer, actor-critic, etc , and, more importantly, to handle a wide class of MF interactions that need not becontinuous nor differentiable.
This is in contrast to most existing works, which require differentiable[14, 15] or quadratic [32] structure on F , or discretize the state space [16].
We validate our method,called Deep Generalized Schrödinger Bridge (DeepGSB) , on various challenging MFGs fromcrowd navigation to high-dimensional opinion depolarization (where d = 1000), setting a state-ofthe-art record in the area of numerical MFG solvers.
In summary, we present the following contributions.
• We present a novel numerical method, rooted in Schrödinger Bridge (SB), for solving a challengingclass of Mean-Field Game where the population needs to converge exactly to the target distribution.
• The resulting method, DeepGSB , generalizes prior SB results to accepting ﬂexible mean-ﬁeldinteraction ( e.g., non-differentiable) and enjoys modern training techniques from DeepRL.• DeepGSB achieves promising empirical results in navigating crowd motion and depolarizing1000-dimensional opinion dynamics, setting a new state-of-the-art numerical MFG solver.
Preliminary on Schrödinger Bridge (SB)
[Figure]
and backward (3b)
SDEs in SB, which are minimum-energy solution when (Ψ , (cid:98) Ψ) obey the PDEs in (4).
The SB problem was originally introduced in the 1930s for quantum mechanics [29, 33] and later draws broader interests with its connection to optimaltransport and control [34–37].
Given a pair of boundary distributions ( ρ 0 , ρ T ) ,
SB seeks an optimal pair of stochastic processes of the forms:
[Equation]
While X t is a standard stochastic process starting from ρ 0 , ¯ X s evolves alongthe “ reversed ” time coordinate s :
= T − t from ρ T .
The base drift f anddiffusion σ are typically known in prior and related to the Hamiltonian H .
Suppose Ψ , (cid:98) Ψ ∈ C 2 , 1 ( R d , [0 , T ]) solve the following coupled PDEs,
[Equation]
then the theory of SB suggests that the SDEs in (3) are “ optimal ” in the sense that they solve avariational optimization w.r.t.
minimum control ( i.e., entropy-regularized).
Furthermore, the pathwise measure induced by (3a) along t ∈ [0 , T ] is equal almost surely to the path-wise measureinduced by (3b) along s :
= T − t .
In other words, the two SDEs in (3) can be thought of as the“ reversed ” process to each other; and hence we also have X T ∼ ρ T and ¯ X T ∼ ρ 0 (see Fig. 2).
Due to the coupling constraints at the boundaries, solving (4) is no easier than solving (1).
Fortunately,recent advances [27, 28] have demonstrated a computationally scalable numerical method via theapplication of the nonlinear Feynman-Kac (FK) Lemma — a mathematical tool that recasts certainclasses of PDEs into sets of forward-backward SDEs (FBSDEs) via some transformation.
Thesenonlinear FK transformations are parametrized in SB-FBSDE [27] by some DNNs with θ and φ , i.e.,
Z θ ( · , ·
[Equation]
≈ σ ∇ log Ψ( · , ·
[Equation]
and
[Equation]
and the FBSDEs resulting from (4) and (5) yield the following objectives:
[Equation]
These objectives are alternatively minimized in the same manner as how Iterative Proportional Fitting [38] (IPF) algorithms solve parametrized SBs [24, 25].
In Appendix A.2 (see Lemma 5),we show that they can be related to KL divergences between the parametrized path measures, i.e., D KL ( ρ θ || ρ φ ) ∝ L IPF ( φ ) and D KL ( ρ φ || ρ θ ) ∝ L IPF ( θ ) , where ρ θ and ρ φ are the path measures of (5).
Deep Generalized Schrödinger Bridge (DeepGSB)
3.1 Connection between the coupled PDEs in MFG and SB
We begin by stating our problem of interest — MFG with hard distributional constraints ( ρ 0 , ρ target ) —in its mathematical form.
Similar to prior works [14, 15], we will adopt the control-afﬁne Hamiltonian,
H ( x, ∇ u, ρ ) :
= 12 (cid:107) σ ∇ u (cid:107) 2 − ∇ u (cid:62) f
( x, ρ ) , given some base drift f .
With this, the MFG (1) becomes
[Equation]
which, as we brieﬂy discussed in Sec.1, differs from (1) in that the boundary condition of the HJB,u ( x, T ) , is now absorbed into FP and deﬁned implicitly through ρ ( x, T )
=
ρ target ( x ) .
Since analyticconversion between the boundary conditions of (1) and (7) exists only for highly degenerate 3 cases[39], this seemingly innocuous change sufﬁces to paralyze most prior methods.
Nevertheless, as (7)now describes a transformation between two distributions (from FP) while obeying some optimality(from HJB), it suggests a deeper connection to optimal transport, and hence the SB.
To bridge these new MFG PDEs (7) to the PDEsappearing in SB (4), we follow standard treatment[30] and apply the Hopf-Cole transform [40, 41]:
[Equation]
which, after some algebra (see Appendix A.3.1 fordetails), yields the following PDEs:
[Equation]
It can be seen that (9) generalizes (4) by introducing the MF interaction F .
Let (Ψ , (cid:98) Ψ) be the solutionto these new MF-extended PDEs in (9), and recall the Hamiltonian adopted in (7), one can ﬁnd that
[Equation]
That is, the agent’s dynamic (2) coincides with the forward SDE (3a) in SB.
Hence, we have connectedthe MFG (1) and SB (4) frameworks through the PDEs in (7) and (9); see Fig. 3.
3.2 Generalized SB-FBSDEs with mean-ﬁeld interaction
With (9), we are ready to present our ﬁrst result that generalizes prior FBSDE for SB to MF interaction.
The derivation is standard but somewhat convoluted; we refer readers to Appendix A.3.2 for details.
Theorem 1 (Generalized SB-FBSDEs) .
Suppose Ψ , (cid:98) Ψ ∈ C 2 , 1 and let f, F satisfy usual growth and
Lipchitz conditions [42, 43] .
Consider the following nonlinear FK transformations applied to [Equation]
[Equation]
where X t follows (3a) with X 0 ∼ ρ 0 .
Then, the resulting FBSDEs system takes the form:
[Equation]
As (7) can be regarded as a generalized SB with nontrivial state cost (see Prop. 1 in [32] for a rigorous derivation), geneal IPF and Sinkhorn methods [16, 32], upon discretization in both the state space and time, still possess global convergence, providing the discretized problem remains convex
See App.
A.5.3 for more details.
Now, consider a similar transformation in (9) but instead w.r.t.
the “reversed” SDE ¯ X s ∼ (3b)
and¯ X 0 ∼ ρ target , i.e., Y s ≡ Y ( ¯ X s , s ) = log Ψ
( ¯ X s , s ) , and etc.
The resulting FBSDEs system reads
[Equation]
.t.
[Equation] [Equation]
Since Y t + (cid:98) Y t = log ρ ( X, t ) by construction, the functions f t and F t in (11) take the arguments
[Equation]
and
[Equation]
Similarly, we have f s := f s ( ¯ X s , exp( Y s + (cid:98) Y s )) and F s := F s ( ¯ X s , exp( Y s + (cid:98) Y s )) in (12).
Just like how (9) generalizes (4), our results in Theorem 1 also generalize the ones appearing in vanilla
SB-FBSDE [27] (see (22) in Appendix A.2) by introducing nontrivial MF interaction F .
Despiteseemingly complex compared to the original PDEs (9), these FBSDEs systems — namely (11) and(12) — stand as the foundation for developing scalable numerical methods, as they describe preciselyhow the values of Y ≡ log Ψ and
(cid:98) Y ≡ log (cid:98) Ψ shall change along the optimal SDEs (notice, e.g.,that both Y t and Z t are functions of X t from (10)).
Essentially, the nonlinear FK Lemma providesa stochastic representation (in terms of Y and (cid:98) Y ) of the PDEs in (9) by expanding them w.r.t.
theoptimal SDEs in (3) using the Itô formula [44].
Consequently, rather than solving the PDEs (9)in the entire function space as in the prior work [15], it sufﬁces to solve them locally around highprobability regions characterized by (3), which leads to computationally scalable methods.
3.3 Design of the computational framework
Looking from Theorem 1, it sufﬁces to approximate Y θ ≈ Y and (cid:98) Y φ ≈ (cid:98) Y with some parametrizefunctions (we use DNNs), since one may then infer Z θ ≈ σ ∇ Y θ and (cid:98)
Z φ ≈ σ ∇ (cid:98) Y φ , as suggestedby (10), and solve for ( X t , ¯ X s ) via (11a, 12a).
Below, we explore options of designing trainingobjectives for ( θ, φ ) , with the aim to encourage ( Y θ , (cid:98) Y φ ) to satisfy the FBSDEs systems in (11, 12).
Option 1: L IPF .
Given the resemblance between Theorem 1 and [27] (see (22) in App.
A.2), it isnatural to wonder if adopting the computation 5 that used to derive (6), e.g., L IPF ( φ ) :
= (cid:82) E [ d Y θt +d (cid:98) Y φt ] , sufﬁces to reach the FBSDE (11).
This is, unfortunately, not the case as one can verify that L
[Equation] Despite that (11) differs from (22) by the extra terms “ + F t ” in (11b) and “ − F t ” in (11c), the twoterms cancel out in the sum of d Y θt + d (cid:98) Y φt , thereby yielding the same objectives that do not dependon F .
This implies that naively minimizing L IPF from [27] is insufﬁcient for solving FBSDE systemswith nontrivial F .
We must seek additional objectives, if any, in order to respect the MF structure.
Temporal Difference (TD) objective.
Let us revisit the relation between the FBSDEs (11, 12)and their PDEs counterparts — but this time the HJB in (7).
Take ( X t , Y t ) for example: The factthat Y t = log Ψ
( X t , t ) =
− u ( X t , t ) suggests an alternative interpretation of Y t as the stochasticrepresentation of the HJB, which, crucially, can be seen as the continuous-time analogue of the Bellman equation [45].
Indeed, discretizing (11b) with some ﬁxed step size δt yields
[Equation]
which resembles a (non-discounted) TD [46, 47] except that, in addition to the standard “ rewards ” (interms of control and state costs), we also have a stochastic term.
This stochastic term, which vanishes
Input
Y θ φ parametrization actor parametrization repeat
Sample trajectories
Z δW t ∈ [ add replay B do
Sample
Compute L ( φ ) =
L IPF ( φ ; X on ) + L TD ( φ ; X on ) + L TD ( φ ; X off ) + (cid:107) (cid:98) Z φ − σ ∇ (cid:98) Y (cid:107) . Update φ with the gradient ∇ φ L ( φ end for Sample trajectories ¯ X ≡ { ¯ X s ,
(cid:98) Z s δW s } s ∈ [0 ,T ] from ; add
X to replay buffer for to do
Sample on policy
X respectively
B . Compute L ( θ ) =
L IPF ( θ ; ¯ X on ) + L TD ( θ ; ¯ X on ) + L TD ( θ ; ¯ X off ) + (cid:107) Z θ − σ ∇ Y (cid:107) . Update θ with gradient ∇ θ L ( θ ) .
end for until in the vanilla Bellman equation upon taking expectations, plays a crucial role in characterizing theinherited stochasticity of the value function Y t .
With this interpretation in mind, we aim to constructsuitable TD targets for our FBSDEs systems, as shown below.
We leave the proof to Appendix A.5.1.
Proposition 2 (TD objectives L TD for (11, 12)) .
The single-step TD targets take the forms:
[Equation]
TD
[Equation]
) with (cid:100) TD 0 := log ρ 0 − Y θ 0 and TD 0 := log ρ target − (cid:98) Y φ 0 , and the multi-step TD targets take the forms: [Equation]
[Equation]
where δ (cid:98) Y t := (cid:100) TD single t + δt − (cid:98) Y t and δY s :=
TD single s + δs − Y s .
Given these TD targets, we can construct L TD ( θ ) =
[Equation]
It can be readily seen that the single-step TD targets in (14) obey a similar structure to (13), exceptderiving from different SDEs (11c, 12b). Doing so reduces the computational overhead, as the relatedobjectives for each parameter, e.g., L IPF ( θ ) and L TD ( θ ) , can be evaluated from the same expectation.
In practice, we ﬁnd that these multi-step variants often yield better performance, as consistentlyobserved in the DeepRL literature [48–50].
Additionally, common practices such as computing (cid:100) TD tand TD s using the exponential moving averaging ( i.e., target values) and replay buffers also helpstabilize training.
Finally, the fact that the TD targets in (16) appear as the regressands implies thatfrom a computational standpoint, the MF interaction F needs not to be continuous or differentiable.
Option 2: L IPF + L TD .
It remains to be answered that whether appending L TD to the training issufﬁcient for ( Y θ , (cid:98) Y φ ) to satisfy the FBSDEs (11, 12). The proposition below gives a positive result.
Proposition 3.
The functions ( Y θ , Z θ , (cid:98) Y φ , (cid:98) Z φ ) satisfy the FBSDEs (11,12) in Theorem 1 if and onlyif they are the minimizers of the combined losses L ( θ, φ ) :
= L IPF ( φ ) + L TD ( φ ) + L IPF ( θ ) + L TD ( θ ) . Proposition 3 asserts the validity of the combined objectives L IPF + L TD in our DeepGSB (see Alg. 1).
Despite sharing a similar alternating structure to IPF [16, 24, 27], we stress that the combinedobjective, e.g., L ( φ ) ∝ D KL ( ρ θ || ρ φ )+ E ρ θ [ L TD ( ρ φ )] , dost not resemble the (reversed) KL projection,min ρ φ D KL ( ρ φ || ˜ ρ θ ) , in IPF. In this vein, DeepGSB may be closer to trust region optimization [51], which enjoys monotonic improvement with local convergence. We leave more discussions to
App. A.5.3 due to space constraint. Finally, we note that in additional to the parametrization( Y θ , (cid:98) Y φ ) , one may also keep ( Y θ 1 , Z θ 2 , (cid:98) Y φ 1 , (cid:98) Z φ 2 ) as four DNNs then impose a consistency loss ,
L FK ( θ ) : = (cid:107) Z θ − σ ∇ Y (cid:107) , to ensure that the nonlinear FK (10) holds. Dosing so often offers extrarobustness.
We refer these two choices respectively to DeepGSB critic and actor-critic .
Experiment
[Figure]
Instantiation of MFGs.
We validate our DeepGSB on two classes of MFGs,including classical crowd navigation ( d =2) and high-dimensional ( d =1000) opinion depolarization.
For crowd navigation, we consider three MFGs appearing inprior methods [14, 15], including (i) asymmetric obstacle avoidance, (ii) entropyinteraction with a V-shape bottleneck, and (iii) congestion interaction on an Sshape tunnel.
We will refer to them respectively as GMM, V-neck, and S-tunnel.
The obstacles and the initial/target Gaussian distributions ( ρ 0 , ρ target ) are shownin Fig.
For opinion depolarization, we set ρ 0 and ρ target to two zero-mean
Gaussians with varying variances for representing the initially polarized anddesired moderated opinion distributions.
Finally, we consider zero and constantbase drift f respectively for GMM and V-neck/S-tunnel, and adopt the polarized
MF dynamics [4] for opinion MFG; see Sec.
4.2 for a detailed discussion.
MF interactions F .
We follow standard treatments from the MFG theory[11] by noting that given a functional F ( ρ ) that quantiﬁes the MF cost w.r.t.
thepopulation ρ , e.g., F congestion :=
E x,y ∼ ρ [ 1 (cid:107) x − y (cid:107) 2 +1 ] or F entropy :=
E ρ [log ρ ] , onecan derive its associated MF interaction function F ( x, ρ ) by taking the functionalderivative, i.e., δ F ( ρ ) δρ ( x ) =
F ( x, ρ ) .
Hence, the congestion and entropy MFinteractions, together with the obstacle cost, follow:
[Equation]
where 1 obs ( · ) is the (discontinuous) indicator of the problemdependent obstacle set.
We leave the derivation of (17) to Appendix A.3.3 and summarize the MF interaction in Table 2.
[Table]
Architecture & Hyperparameters.
Recall the two parametrizations considered in Sec.3.2, i.e.,the critic ( Y θ , σ ∇ Y θ
, (cid:98) Y φ , σ ∇ (cid:98) Y φ ) , DeepGSB-c , and actor-critic ( Y θ , Z θ , (cid:98) Y φ , (cid:98) Z φ ) , DeepGSB-ac .
Inboth cases, we parametrize them with standard fully-connected DNNs for crowd navigation, anddeep residual networks for high-dimensional opinion MFGs.
All networks adopt sinusoidal timeembeddings and are trained with AdamW [52].
All SDEs in (11, 12) are simulated with the EulerMaruyama method.
Due to space constraints, we will focus mostly on the results of DeepGSB-acand leave the discussion of DeepGSB-c , alone with additional experimental details, to Appendix A.4.
4.1 Two-dimensional crowd navigation
Figure 5 shows the simulation results of our DeepGSB-ac on three crowd navigation MFGs.
We alsoreport existing numerical methods [14–16] that are best-tuned on each MFG (see Appendix A.4.1 fordetails) but note that in practice, they either require softening F to be differentiable [14, 15] to yieldreasonable results, or discretizing the state space [16], which can lead to prohibitive complexity.
We ﬁrst compare to Chen [16] on GMM (see Fig.
5a) as their method only applies to non-MF interaction, i.e., F :=
F ( x ) .
While DeepGSB-ac guides the population to smoothly avoid all obstacles(notice the sharp contours of Y around them), [16] struggles to escape due to the discretization of
[Figure] [Figure]
the state space (hence the policy). To better examine the effect of ρ in F ( x, ρ ) , we next simulatethe dynamics on V-neck (see Fig. 5b) with and without the MF interaction.
It is clear that our
DeepGSB-ac encourages the population to spread out once the entropy interaction F entropy is enabled,yet a similar effect is barely observed in [14].
We observed difﬁculties in balancing the MF interaction
F and the terminal penalty D KL ( ρ T || ρ target ) for [14], yet this problem is alleviated in DeepGSB byconstruction.
Lastly, we validate the robustness of our method on S-tunnel (see Fig.
5c) w.r.t.
varyingdiffusions σ = { 0 . 5 , 1 , 2 } .
Again, our DeepGSB-ac reaches the same ρ target despite being subject todifferent levels of stochasticity.
This is in contrast to [15], which, due to discarding the SDE dynamics,necessitates solving PDEs on the entire state space that may be sensitive to hyperparameters.
In short,our DeepGSB outperforms prior methods [14–16] by better respecting obstacles and MF interactionsyet without losing convergence to ρ target , and its performance remains robust across different MFGs.
Wasserstein distance to ρ target and FBSDEs violation, in terms of TD errors and nonlinear FK, averaged over 3 runs.
[Table]
Table 3 compares DeepGSB-ac with
DeepGSB-c. While DeepGSB-ac typically achieves lower Wasserstein and TD errors, it seldom closes the consistency gap of the nonlinear FK L FK ( θ )in (10), as opposed to DeepGSB-c.
In practice, the results from DeepGSB-c are visually indistinguishablefrom DeepGSB-ac, despite the different contours of Y ; we leave them to Appendix A.4.2 due to space constraint.
t =4.
t = 0.33 T 4.
t = 0.67 T 4.
t = T .
d = d =
10  d =   16 8 d =1000 (PCA embedding)
d = d = highly disagree neutral highly agree
Simulation of polarize dynamics f polarize in (18) when d =
Directional similarity of f polarize
Terminal distribution T after applying DeepGSB-ac
Directional similarity after applying DeepGSB-ac
[Figure]
where the directional similarity [3] counts the histogram of cosine angle between pairwise opinions at the terminal distribution ρ T .
(b) DeepGSB-ac guides ρ T toward moderated distributions, hence depolarizes the opinion dynamics.
We use the ﬁrst two principal components to visualize d =
4.2 High-dimensional opinion depolarization
Finally, we showcase our DeepGSB in solving high-dimensional MFGs in the application of opiniondynamics [3–5], where each agent now possesses a d -dimensional opinion x ∈ R d that evolvesthrough the interactions with the population.
In light of increasing recent attention, we consider aparticular class of opinion dynamics known to yield strong polarization [4], i.e., the agents’ opinionstend to partition into groups holding diametric views.
Take the party model [4] for instance:
Given arandom information ξ ∈ R d sampled from some distribution independent of ρ , each agent updateshis/her opinion following a normalized polarize dynamic ¯ f polarize = f polarize / (cid:107)
f polarize (cid:107) 12 , wheref polarize ( x, ρ ; ξ ) :=
E y ∼ ρ
[ a ( x, y ; ξ )¯ y ] , a ( x, y ; ξ ) :
= (cid:26)if sign( (cid:104) x, ξ (cid:105) )
= sign( (cid:104) y, ξ (cid:105) )
− 1 otherwise , (18) and ¯ y
= y / (cid:107) y (cid:107) 12 .
The agreement function a ( x, y ; ξ ) indicates whether the two opinions x and y agreeon the information ξ .
Intuitively, the dynamic in (18) suggests that the agents tend to be receptive toopinions they agree with, and antagonistic to opinions they disagree with.
As shown in Fig.
6a, thisbehavioral assumption, also known as biased assimilation [53, 54], can easily lead to polarization.
We can apply our MFG framework (7) to this polarized base drift (18), where, starting from someweakly polarized ρ 0 , we seek a policy that compensates the polarization tendency and helps guide thepopulation towards a moderated opinion distribution ρ target (assuming as Gaussian for simplicity).
Weconsider the entropy MF interaction F entropy as it encourages opinions diversity before reaching consensus.
As shown in Fig.
6b, in both lower-
( d =2) and higher- ( d =1000) dimensions, our DeepGSB-acsuccessfully guides the population towards the desired distribution centered symmetrically at 0 ∈ R d ,thereby mitigates the polarization.
Results of DeepGSB-c remain similar despite more sensitive tohyperparameters; see Appendix A.4.2.
We highlight these state-of-the-art results for solving thesechallenging MFGs that, comparing to existing methods [14, 15], consider a more difﬁcult mean-ﬁelddynamic ( f ( x, ρ ) vs. f ( x ) ) in an order of magnitude higher dimension ( d =1000 vs. d =100).
Conclusion, Limitation, and Broader Impact
We present DeepGSB , a new numerical method for solving a challenging class of MFGs withdistributional boundary constraints.
By generalizing prior FBSDE theory for Schrödinger Bridge toaccepting mean-ﬁeld interactions, we show that practical training can be achieved via an intriguingalgorithmic connection to DeepRL.
Our DeepGSB outperforms prior methods in crowd navigation MFGs and sets a new state-of-the-art record in depolarizing 1000-dimensional opinion MFGs.
DeepGSB is mainly developed for MFGs in unconstrained state spaces such as R d .
Yet, it maybe necessary to adopt domain-speciﬁc structures, e.g., constrained state spaces. Additionally, thedivergence terms appeared in the objectives may scale unfavorably as the dimension grows. Thismay, however, be mitigated by adopting a simpler regression re-formulation [24]. We leave this as apromising future direction.
Study of MFGs possesses its own societal inﬂuence.
Thus, as a MFGsolver, DeepGSB may pose a potential impact in offering solutions to previously unsolvable MFGsunder more practical settings, thereby facilitating new understanding of population behavior.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Sec.5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Sec.5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Theorem 1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provide the pseudocode in Alg. 1 and detail the hyperparameter in Table 6. The code will be madeavailable upon publication.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix A.4.1.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Table 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See Appendix A.4.1.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] Not applicable.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A] Not applicable.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] Not applicable.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] Not applicable.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A] Not applicable.