Deep Generalized Schrödinger Bridge
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Mean-Field Game (MFG) serves as a crucial mathematical framework in modelingthe collective behavior of individual agents interacting stochastically with a largepopulation.
In this work, we aim at solving a challenging class of MFGs in whichthe differentiability of these interacting preferences needs not available to the solver,and the population is urged to converge exactly to some desired distribution.
Thesesetups are, despite being well-motivated for practical purposes, complicated enoughto paralyze most (deep) numerical solvers.
Nevertheless, we show that Schrödinger
Bridge — as an entropy-regularized optimal transport model — can be generalizedto accepting mean-ﬁeld structures, hence solving these MFGs.
This is achievedvia the application of Forward-Backward Stochastic Differential Equations theory,which, intriguingly, leads to a computational framework with a similar structure to
Temporal Difference learning.
As such, it opens up novel algorithmic connectionsto
Deep Reinforcement Learning that we leverage for facilitating practical training.
Our method, named Deep Generalized Schrödinger Bridge ( DeepGSB ), not onlyoutperforms prior methods in solving classical population navigation MFGs, but isalso capable of solving 1000-dimensional opinion depolarization , setting a newstate-of-the-art numerical solver for high-dimensional MFGs.
Introduction
In a scorching morning, you navigated through the crowds toward theofﬁce.
As you walked through a crosswalk, you were pondering thegrowing public opinion on a new policy over the past week and weresuddenly interrupted by the honking as the trafﬁc started moving...
From navigation in crowds to propagation of opinion and trafﬁc movement, examples of individual agents interacting with a large populationare widespread in daily life and, due to their prevalence, appear asan important subject in multidisciplinary scientiﬁc areas, includingeconomics [1, 2], opinion modeling [3–5], robotics [6, 7], and morerecently machine learning [8–10].
Mathematically, the decision-making processes under these scenarioscan be characterized by the Mean-Field Game [11–13] (MFG) , whichmodels a noncooperative differential game on a ﬁnite horizon betweena continuum population of rational agents.
Let u ( x, t ) be the value function, also known as optimalcost-to-goal, that governs agents’ policies at each state x ∈ R d and time t ∈ [0 , T ] , and denote theresulting population density by ρ ( · , t )
∈ P ( R d ) , where P ( R d ) is the set of probability measures on
R d .
At the Nash equilibrium where no agent has the incentive to change his/her decision, MFG, at its
Our DeepGSB is capable of solving a much wider class of MFGs in higher dimensional state spaces.
[Table]
most general form, solves the following partial differential equations (PDEs):
[Equation]
where ∇ , ∇· , and ∆ are respectively the gradient, divergence, and laplacian operators.
2 Thesetwo PDEs are respectively known as the Hamilton-Jacobi-Bellman (HJB) and Fokker-Plank
(FP)equations, which characterize the evolution of u ( x, t ) and ρ ( x, t ) .
They are coupled with each otherthrough the Hamiltonian H ( x, p, ρ ) : R d × R d × P ( R d ) → R , which describes the dynamics ofthe game, and the mean-ﬁeld interaction F ( x, ρ ) :
R d × P ( R d ) → R , which quantiﬁes the agent’spreference when interacting with the population.
The terminal condition G typically penalizesdeviations from some desired target distribution ρ target , e.g., G ≈ D KL ( ρ ( · , T ) ||
ρ target ( · )) .
Given asolution ( u, ρ ) to (1), each agent acts accordingly and follows a stochastic differential equation(SDE)
[Equation]
where W t ∈ R
d is the Wiener process and σ ∈ R is some diffusion scaler.
At the mean-ﬁeld limit, i.e.,when the number of agents goes to inﬁnity, the collective behavior of (2) yields the density ρ ( · , t ) .
Numerical methods for solving (1) have advanced rapidly with the aid of machine learning.
Seminarworks such as Ruthotto et al.
[14] and Lin et al.
[15] approximated ( u, ρ ) with deep neural networks(DNNs) and directly penalized the violation of PDEs.
Despite showing preliminary successes, theunderlying dynamics (2) were either degenerate, e.g., σ := 0 or H :
= H ( x, ∇ u ) , or completelydiscarded by instead regressing network outputs on the entire state space, which can scale unfavorablyas the dimension d grows.
An alternative that avoids both limitations, i.e., it keeps the full stochasticdynamics in (2) while being computationally scalable, is to recast these PDEs to a set of forwardbackward SDEs (FBSDEs) by applying the nonlinear Feynman-Kac Lemma [17–19].
The FBSDEsanalysis appears extensively in the theoretical study of MFG [20–23], yet development of scalable
FBSDEs-based solver has remained, surprisingly, limited.
Our work contributes to this direction.
Since ρ target is known in prior, in many cases there are direct interests to seek an optimal policy thatguides the agents from an initial distribution ρ 0 to the exact ρ target , while respecting the structure of
MFG, particularly the MF interaction F ( x, ρ ) .
Lifting (1) to this setup, however, is highly nontrivial.
Indeed, replacing the soft penalty at u ( x, T ) =
D KL ( ρ || ρ target ) with a hard distributional constraintat ρ ( x, T ) =
ρ target yields an HJB whose boundary condition can only be deﬁned implicitly through
FP, which now contains two distributional constraints and resembles an optimal transport problem.
As such, despite being well-motivated, most prior methods have struggled to extend to this setup.
In this work, we show that an emerging machine learning model, called Schrödinger Bridge ( SB )[24–28], provides an elegant recipe for solving this challenging class of MFGs with distributionalboundary constraints ( ρ 0 , ρ target ) .
Although SB is traditionally set up as an entropy-regularizedoptimal transport model [29–31] with F := [-
 [29–31],we-]0 , we show that SB-FBSDE [27], an FBSDE-basedmethod for solving SB, can be generalized to accept nontrivial F ; hence solving MFG.
Interestingly,the new FBSDEs system admits a similar computational structure to temporal difference (TD)learning, leading to a framework that narrows the gap between SB and Deep Reinforcement Learning(DeepRL); see Fig.
This connection enables our method to take advantage of DeepRL techniques, such as target networks, replay buffer, actor-critic, etc , and, more importantly, to handle a wide classof MF interactions that need not require to be continuous nor differentiable.
This is in contrast to mostexisting works, which require differentiable [14, 15] or quadratic [32] structure on F , or discretize thestate space [16].
We validate our method, called Deep Generalized Schrödinger Bridge (DeepGSB) ,on various challenging MFGs from crowd navigation to high-dimensional opinion depolarization(where d = 1000), setting a state-of-the-art record in the area of numerical MFG solvers.
In summary, we present the following contributions.
• We present a novel numerical method, rooted in Schrödinger Bridge (SB), for solving a challenging class of Mean-Field Game where the population needs to converge exactly to the target distribution.
• The resulting method, DeepGSB , generalizes prior SB results to accepting ﬂexible mean-ﬁeldinteraction ( e.g., non-differentiable) and enjoys modern training techniques from DeepRL.• DeepGSB achieves promising empirical results in navigating crowd motion and depolarizing1000-dimensional opinion dynamics, setting a new state-of-the-art numerical MFG solver. [Figure]
Preliminary on Schrödinger Bridge (SB
[Figure]
SDEs in SB, which are minimum-energy solution when (Ψ , (cid:98) Ψ) obey the PDEs in (4)
The SB problem was originally introduced in the 1930s for quantum mechanics [29, 33] and later draws broader interests with its connection to optimaltransport and control [34–37].
Given a pair of boundary distributions ( ρ 0 , ρ T ) ,
SB seeks an optimal pair of stochastic processes of the forms:
[Equation]
While X t is a standard stochastic process starting from ρ 0 , ¯ X s evolves alongthe “ reversed ” time coordinate s :
= T − t from ρ T .
The base drift f anddiffusion σ are typically known in prior and related to the Hamiltonian H .
Suppose Ψ , (cid:98) Ψ ∈ C 2 , 1 ( R d , [0 , T ]) solve the following coupled PDEs,
[Equation]
then the theory of SB suggests that the SDEs in (3) are “ optimal ” in the sense that they solve avariational optimization w.r.t.
minimum control ( i.e., entropy-regularized).
Furthermore, the pathwise measure induced by (3a) along t ∈ [0 , T ] is equal almost surely to the path-wise measureinduced by (3b) along s :
= T − t .
In other words, the two SDEs in (3) can be thought of as the“ reversed ” process to each other; and hence we also have X T ∼ ρ T and ¯ X T ∼ ρ 0 (see Fig. 2).
Due to the coupling constraints at the boundaries, solving (4) is no easier than solving (1).
Fortunately,recent advances [27, 28] have demonstrated a computationally scalable numerical method via theapplication of the nonlinear Feynman-Kac (FK) Lemma — a mathematical tool that recasts certainclasses of PDEs into sets of forward-backward SDEs (FBSDEs) via some transformation.
Thesenonlinear FK transformations are parametrized in SB-FBSDE [27] by some DNNs with θ and φ , i.e.,
Z θ ( · , ·
[Equation]
≈ [Equation] ∇ log Ψ( · , · ) and (cid:98) Z φ ( · , ·
[Equation]
≈ σ ∇ log (cid:98) Ψ [Equation] [Equation]
[Equation]
, (5) and the FBSDEs resulting from (4) and (5) yield the following objectives:
[Equation]
These objectives are alternatively minimized in the same manner as how Iterative Proportional Fitting[38] (IPF) algorithms solve parametrized SBs [24, 25].
Once Z θ and (cid:98) Z φ are learned, the SDEs of SB in (3) can be simulated by substituting (5); see Fig. We refer readers to Appendix A.2 for adetailed review of nonlinear FK Lemma, SB-FBSDE [27], and the derivation of L IPF in (6).
Deep Generalized Schrödinger Bridge (DeepGSB)
3.1 Connection between the coupled PDEs in MFG and SB
We begin by stating our problem of interest — MFG with hard distributional constraints ( ρ 0 , ρ target ) —in its mathematical form.
Similar to prior works [14, 15], we will adopt the control-afﬁne Hamiltonian,
H ( x, ∇ u, ρ ) :
= 12 (cid:107) σ ∇ u (cid:107) 2 − ∇ u (cid:62) f
( x, ρ ) , given some base drift f .
With this, the MFG (1) becomes
[Equation]
which, as we brieﬂy discussed in Sec.1, differs from (1) in that the boundary condition of the HJB,u ( x, T ) , is now absorbed into FP and deﬁned implicitly through ρ ( x, T )
=
ρ target ( x ) .
Since analyticconversion between the boundary conditions of (1) and (7) exists only for highly degenerate 3 cases[39], this seemingly innocuous change sufﬁces to paralyze most prior methods.
Nevertheless, as (7)now describes a transformation between two distributions (from FP) while obeying some optimality(from HJB), it suggests a deeper connection to optimal transport, and hence the SB.
To bridge these new MFG PDEs (7) to the PDEsappearing in SB (4), we follow standard treatment[30] and apply the Hopf-Cole transform [40, 41]:
[Equation]
which, after some algebra (see Appendix A.3.1 fordetails), yields the following PDEs:
[Equation]
It can be readily seen that (9) generalizes (4) by introducing the MF interaction F .
Notice that thefunction F :=
F ( x, Ψ (cid:98) Ψ) shall take the product of Ψ and (cid:98) Ψ as its second argument, as the Hopf-Coletransform (8) implies Ψ( · , t ) (cid:98) Ψ( · , t )
=
ρ ( · , t ) . Let (Ψ , (cid:98) Ψ) be the solution to these new MF-extended PDEs in (9), and recall the control-afﬁne Hamiltonian we adopted in (7), then one can validate that
[Equation]
That is, the agent’s dynamic (2) coincides with the forward SDE (3a) in SB.
Hence, we have connectedthe MFG (1) and SB (4) frameworks through the PDEs in (7) and (9); see Fig. 3.
3.2 Generalized SB-FBSDEs with mean-ﬁeld interaction
With (9), we are ready to present our ﬁrst result that generalizes prior FBSDE for SB to MF interaction.
The derivation is standard but somewhat convoluted; we refer readers to Appendix A.3.2 for details.
Theorem 1 (Generalized SB-FBSDEs) .
Suppose Ψ , (cid:98) Ψ ∈ C 2 , 1 and let f, F satisfy usual growth and
Lipchitz conditions [42, 43] .
Consider the following nonlinear FK transformations applied to (9):
[Equation]
where X t follows (3a) with X 0 ∼ ρ 0 .
Then, the resulting FBSDEs system takes the form: FBSDEs
[Equation]
Now, consider a similar transformation in (9) but instead w.r.t.
the “reversed” SDE ¯ X s ∼ (3b)
and¯ X 0 ∼ ρ target , i.e., Y s ≡ Y ( ¯ X s , s ) = log Ψ
( ¯ X s , s ) , and etc.
The resulting FBSDEs system reads
[Equation]
.r.t.
[Equation] [Equation]
Since Y + (cid:98) Y = log ρ by construction, the functions f and F in (11,12) take the argumentsf := f ( · , exp( Y + (cid:98) Y )) and F
[Equation]
= F (
[Equation]
(cid:98) [Equation]
Just like how (9) generalizes (4), our results in Theorem 1 also generalize the ones appearing in vanilla
SB-FBSDE [27] (see (22) in Appendix A.2) by introducing nontrivial MF interaction F .
Despiteseemingly complex compared to the original PDEs (9), these FBSDEs systems — namely (11) and(12) — stand as the foundation for developing scalable numerical methods, as they describe preciselyhow the values of Y ≡ log Ψ and
(cid:98) Y ≡ log (cid:98) Ψ shall change along the optimal SDEs (notice, e.g.,that both Y t and Z t are functions of X t from (10)).
Essentially, the nonlinear FK Lemma providesa stochastic representation (in terms of Y and (cid:98) Y ) of the PDEs in (9) by expanding them w.r.t.
theoptimal SDEs in (3) using the Itô formula [44].
Consequently, rather than solving the PDEs (9)in the entire function space as in the prior work [15], it sufﬁces to solve them locally around highprobability regions characterized by (3), which leads to computationally scalable methods.
Parametrization.
Looking from Theorem 1, it sufﬁces to approximate Y ≈ Y θ and (cid:98) Y ≈ (cid:98) Y φ withtwo DNNs, since one can then infer Z ≈ σ ∇ Y θ and (cid:98) Z ≈
σ ∇ (cid:98) Y φ , as suggested by (10), and solvefor ( X t , ¯ X s ) via (11a, 12a).
We refer to this choice as the critic parametrization as they resemblethe critic networks [45].
An alternative parametrization, which offers extra robustness in practice, isto approximate ( Y, Z, (cid:98) Y , (cid:98) Z ) ≈ ( Y θ 1 , Z θ 2 , (cid:98) Y φ 1 ,
(cid:98) Z φ 2 ) 4 with four DNNs.
We call this the actor-criticparametrization since ( Z, (cid:98) Z ) share the same role as the actor networks [46] in propagating dynamics. Challenges in solving (11, 12) with IPF SB-FBSDE [27] proposed to solve their FBSDE with
IPF algorithm, which alternates optimizing the log-likelihood objectives log ρ 0 ∝ (cid:82) E
[ d Y t + d (cid:98) Y t ]and log ρ T ∝ (cid:82) E [ d Y s + d (cid:98) Y s ] .
Taking F := 0 in Theorem 1, these integrations will indeed recoverthe IPF objectives in (6), up to some independent constant. However, upon closer inspection, theseintegrations do not depend on the choice of F , since the two F s appearing in (11b,11c) or (12b,12c)will cancel each other out in the sum of d Y + d (cid:98) Y when computing the log-likelihoods.
Mathematically,this makes sense when ( Y, (cid:98) Y ) are the exact solutions to the FBSDE systems (11, 12), yet it can beproblematic when the parametrized ( Y θ , (cid:98) Y φ ) have not yet satisﬁed these equations. This implies that
The IPF algorithm, which underlies modern numerical methods for solving parametrized SBs [24, 25, 27, 28] , is insufﬁcient in solving the FBSDEs with MF interaction (11, 12) .
Hence, we must seek additional training objectives, if any, in order to develop methods that fullyrespect the MF interacting structure in these new FBSDEs systems.
As we will demonstrate in thefollowing subsection, this can be achieved by drawing an intriguing connection to DeepRL. 3.3 Design of computational framework via Deep Reinforcement Learning (DeepRL)
Let us revisit the relation between the FBSDEs (11, 12) and their PDEs counterparts — but this timethe HJB in (7).
Take ( X t , Y t ) for example: The fact that Y t = log Ψ
( X t , t ) =
− u ( X t , t ) suggestsan alternative interpretation of Y t as the stochastic representation of the HJB, which, crucially, canbe seen as the continuous-time analogue of the celebrated Bellman equation [47] for ﬁnite-horizonstochastic systems [48].
Indeed, discretizing (11b) with some ﬁxed step size δt yields
[Equation]
[Equation]
Input repeat
Sample
+ L TD ( φ
X on )
+ L TD ( φ
X off Z φ − σ ∇
Y φ for
Sample
Z add do
Sample
Compute θ X on )
+ L TD ( θ ; ¯ X on ) + L TD ( θ ; ¯ X off ) + Z θ − σ ∇ Y
∇ θ L θ end until converges which resembles a (non-discounted)
Temporal Difference [49, 50] (TD) except that, in addition to thestandard “ rewards ” (in terms of control and state costs), we also have a stochastic term.
This stochasticterm, which vanishes in the vanilla Bellman equation upon taking expectations, plays a crucial role incharacterizing the inherited stochasticity of the value function Y t .
With this interpretation in mind,we aim to construct suitable TD targets for our FBSDEs systems.
The results are summarized below.
Proposition 2 (TD targets for Generalized SB-FBSDEs) .
The single-step TD targets take the
[Equation]
[Equation]
Further, denote the incremental updates in (14) by δ (cid:98)
Y t :=
(cid:100) TD single t + δt − (cid:98) Y t and δY s :=
TD single s + δs − Y s .
Then, the multi-step TD targets take the forms:
[Equation]
where (cid:100) TD 0 :
= log ρ 0 − Y 0 and TD 0 :
= log ρ target − (cid:98) Y 0 ensure the boundary conditions. It can be readily seen that the single-step TD targets in (14) obey a similar structure to (13), exceptderiving from different SDEs (11c, 12b); we will discuss these choices of SDEs shortly. Given(14), one can construct their multi-step TD targets (15) straightforwardly.
In practice, we ﬁnd thatthese multi-step variants often yield better performance, as consistently observed in the DeepRLliterature [51–53].
Additionally, common practices such as computing (cid:100) TD t and TD s using theexponential moving averaging ( i.e., target values) and replay buffers also help stabilize training.
With
Proposition 2, we can construct our TD objectives as [Equation]
From (16), it should now be clear that these SDEs (11c, 12b) are deliberately chosen in Proposition 2so that the related objectives for each parameter, e.g., L IPF ( θ ) and L TD ( θ ) , can be evaluated fromthe same sampling expectation, e.g., E
¯
X s ∼ (3b) [ · · · ] . Doing so reduces the computational overhead. The fact that the TD targets in (16) appear as the regressing targets, i.e., labels , implies that froma computational standpoint, the MF interaction F needs not to be continuous or differentiable.
Wehighlight this distinction, together with the results in Theorem 1 and Proposition 2, as they not onlyfacilitate scalable methods for solving the challenging class of MFGs in (7), but also open up newalgorithmic opportunities in the intersection of SB and DeepRL.
Our DeepGSB is summarized in Alg. 1, which alternately optimizes φ and θ as in IPF, yet with thenew training objectives that account for the MF structure.
Note that for the actor-critic parametrization,we add a consistency loss , e.g., L FK ( θ ) :
= (cid:107) Z θ − σ ∇ Y (cid:107) , to ensure that the nonlinear FK relations in(10) approximately hold.
For the critic parametrization, these conditions are satisﬁed by construction.
Experiment
[Figure]
Instantiation of MFGs.
We testify our DeepGSB on two classes of MFGs,including classical crowd navigation ( d =2) and high-dimensional ( d =1000) opinion depolarization.
For crowd navigation, we consider three MFGs appearing inprior methods [14, 15], including (i) asymmetric obstacle avoidance, (ii) entropyinteraction with a V-shape bottleneck, and (iii) congestion interaction on an Sshape tunnel.
We will refer to them respectively as GMM, V-neck, and S-tunnel.
The obstacles and the initial/target Gaussian distributions ( ρ 0 , ρ target ) are shownin Fig.
For opinion depolarization, we set ρ 0 and ρ target to two zero-mean
Gaussians with varying variances for representing the initially polarized anddesired moderated opinion distributions.
Finally, we consider zero and constantbase drift f respectively for GMM and V-neck/S-tunnel, and adopt the polarized
MF dynamics [4] for opinion MFG; see Sec.
4.2 for a detailed discussion.
MF interactions F .
We follow standard treatments from the MFG theory[11] by noting that given a functional F ( ρ ) that quantiﬁes the MF cost w.r.t.
thepopulation ρ , e.g., F congestion :=
E x,y ∼ ρ [ 1 (cid:107) x − y (cid:107) 2 +1 ] or F entropy :=
E ρ [log ρ ] , onecan derive its associated MF interaction function F ( x, ρ ) by taking the functionalderivative, i.e., δ F ( ρ ) δρ ( x ) =
F ( x, ρ ) .
Hence, the congestion and entropy MFinteractions, together with the obstacle cost, follow:
[Equation]
where 1 obs ( · ) is the (discontinuous) indicator of the problemdependent obstacle set.
We leave the derivation of (17) to Appendix A.3.3 and summarize the MF interaction in Table 2.
[Table]
Architecture & Hyperparameters.
Recall the two parametrizations considered in Sec.3.2, i.e.,the critic ( Y θ , σ ∇ Y θ
, (cid:98) Y φ , σ ∇ (cid:98) Y φ ) , DeepGSB-c , and actor-critic ( Y θ , Z θ , (cid:98) Y φ , (cid:98) Z φ ) , DeepGSB-ac .
Inboth cases, we parametrize them with standard fully-connected DNNs for crowd navigation, anddeep residual networks for high-dimensional opinion MFGs.
All networks adopt sinusoidal timeembeddings and are trained with AdamW [54].
All SDEs in (11, 12) are simulated with the EulerMaruyama method.
Due to space constraints, we will focus mostly on the results of DeepGSB-acand leave the discussion of DeepGSB-c , alone with additional experimental details, to Appendix A.4.
4.1 Two-dimensional crowd navigation
Figure 5 shows the simulation results of our DeepGSB-ac on three crowd navigation MFGs.
We alsoreport existing numerical methods [14–16] that are best-tuned on each MFG (see Appendix A.4.1 fordetails) but note that in practice, they either require softening F to be differentiable [14, 15] to yieldreasonable results, or discretizing the state space [16], which can lead to prohibitive complexity.
We ﬁrst compare to Chen [16] on GMM (see Fig.
5a) as their method only applies to non-MF interaction, i.e., F :=
F ( x ) .
While DeepGSB-ac guides the population to smoothly avoid all obstacles(notice the sharp contours of Y around them), [16] struggles to escape due to the discretization ofthe state space (hence the policy).
To better examine the effect of ρ in F ( x, ρ ) , we next simulate
[Figure] [Figure]
the dynamics on V-neck (see Fig. 5b) with and without the MF interaction.
It is clear that our
DeepGSB-ac encourages the population to spread out once the entropy interaction F entropy is enabled,yet a similar effect is barely observed in [14].
We observed difﬁculties in balancing the MF interaction
F and the terminal penalty D KL ( ρ T || ρ target ) for [14], yet this problem is alleviated in DeepGSB byconstruction.
Lastly, we testify the robustness of our method on S-tunnel (see Fig.
5c) w.r.t.
varyingdiffusions σ = { 0 . 5 , 1 , 2 } .
Again, our DeepGSB-ac reaches the same ρ target despite being subject todifferent levels of stochasticity.
This is in contrast to [15], which, due to discarding the SDE dynamics,necessitates solving PDEs on the entire state space that may be sensitive to hyperparameters.
In short,our DeepGSB outperforms prior methods [14–16] by better respecting obstacles and MF interactionsyet without losing convergence to ρ target , and further enjoying robustness to hyperparameters.
Wasserstein distance to ρ target and FBSDEs violation, in terms of TD errors and nonlinear FK, averaged over 3 runs
[Table]
Table 3 compares DeepGSB-ac with
DeepGSB-c. While DeepGSB-ac typically achieves smaller Wasserstein distance to ρ target and TD errors, it seldom closes the consistency gap of thenonlinear FK L FK ( θ ) : = (cid:107) Z θ − σ ∇ Y (cid:107)in (10), as opposed to DeepGSB-c.
In practice, the results from DeepGSB-c are visually indistinguishablefrom DeepGSB-ac, despite the different contours of Y ; we leave them to Appendix A.4.2 due to space constraint.
t =4.
t = 0.33 T 4.
t = 0.67 T 4.
t = T .
d = d =
10  d =   16 8 d =1000 (PCA embedding)
d = d = highly disagree neutral highly agree
Simulation of polarize dynamics f polarize in (18) when d =
Directional similarity of f polarize
Terminal distribution T after applying DeepGSB-ac
Directional similarity after applying DeepGSB-ac
[Figure]
where the directional similarity [3] counts the histogram of cosine angle between pairwise opinions at the terminal distribution ρ T .
(b) DeepGSB-ac guides ρ T toward moderated distributions, hence depolarizes the opinion dynamics.
We use the ﬁrst two principal components to visualize d =
4.2 High-dimensional opinion depolarization
Finally, we showcase our DeepGSB in solving high-dimensional MFGs in the application of opiniondynamics [3–5], where each agent now possesses a d -dimensional opinion x ∈ R d that evolvesthrough the interactions with the population.
In light of increasing recent attention, we consider aparticular class of opinion dynamics known to yield strong polarization [4], i.e., the agents’ opinionstend to partition into groups holding diametric views.
Take the party model [4] for instance:
Given arandom information ξ ∈ R d sampled from some distribution independent of ρ , each agent updateshis/her opinion following a normalized polarize dynamic ¯ f polarize = f polarize / (cid:107)
f polarize (cid:107) 12 , wheref polarize ( x, ρ ; ξ ) :=
E y ∼ ρ
[ a ( x, y ; ξ )¯ y ] , a ( x, y ; ξ ) :
= (cid:26)if sign( (cid:104) x, ξ (cid:105) )
= sign( (cid:104) y, ξ (cid:105) )
− 1 otherwise , (18) and ¯ y
= y / (cid:107) y (cid:107) 12 .
The agreement function a ( x, y ; ξ ) indicates whether the two opinions x and y agreeon the information ξ .
Intuitively, the dynamic in (18) suggests that the agents tend to be receptive toopinions they agree with, and antagonistic to opinions they disagree with.
As shown in Fig.
6a, thisbehavioral assumption, also known as biased assimilation [55, 56], can easily lead to polarization.
We can apply our MFG framework (7) to this polarized base drift (18), where, starting from someweakly polarized ρ 0 , we seek a policy that compensates the polarization tendency and helps guide thepopulation towards a moderated opinion distribution ρ target (assuming as Gaussian for simplicity).
Weconsider the entropy MF interaction F entropy as it encourages opinions diversity before reaching consensus.
As shown in Fig.
6b, in both lower-
( d =2) and higher- ( d =1000) dimensions, our DeepGSB-acsuccessfully guides the population towards the desired distribution centered symmetrically at 0 ∈ R d ,thereby mitigates the polarization.
Results of DeepGSB-c remain similar despite more sensitive tohyperparameters; see Appendix A.4.2.
We highlight these state-of-the-art results for solving thesechallenging MFGs that, comparing to existing methods [14, 15], consider a more difﬁcult mean-ﬁelddynamic ( f ( x, ρ ) vs. f ( x ) ) in an order of magnitude higher dimension ( d =1000 vs. d =100).
Conclusion, Limitation, and Broader Impact
We present DeepGSB , a new numerical method for solving a challenging class of MFGs with distributional boundary constraints.
By generalizing prior FBSDE theory for Schrödinger Bridge to acceptingmean-ﬁeld interactions, we show that practical training can be achieved via an intriguing algorithmicconnection to DeepRL.
Our DeepGSB outperforms prior methods in population navigation MFGsand sets a new state-of-the-art record in depolarizing 1000-dimensional opinion MFGs.
Our DeepGSB is primarily developed for solving MFGs in unconstrained state spaces such as R d .
Yet, for applications in, e.g., biomedicines and robotics, it may be necessary to consider constrainedstate spaces or manifolds , so that the resulting methods further respect domain-speciﬁc structures.
Study of MFGs possesses its own societal inﬂuence.
Thus, as a numerical MFG solver, DeepGSBmay pose a potential impact in offering solutions to previously unsolvable MFGs under more practicalsettings, thereby facilitating new understanding of population behavior with interacting agents.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Sec.5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Sec.5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Theorem 1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Alg. 1.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix A.4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Table 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See Appendix A.4.1.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] Not applicable.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A] Not applicable.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] Not applicable.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] Not applicable.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A] Not applicable.