T EAM B ELIEF DAG F ORM : A C ONCISE R EPRESENTATION FOR T EAM -C ORRELATED G AME -T HEORETIC D ECISION M AKING
Machine, Inc
Strategy Robot, Inc. Optimized Markets, Inc. sandholm@cs.cmu.edu
A BSTRACT
In this paper, we introduce a new representation for team-coordinated gametheoretic decision making, which we coin team belief DAG form .
In our representation, at every timestep, a team coordinator observes the information that is public to all its members, and then decides on a prescription for all the possible states consistent with its observations.
Our representation unifies and extends recent approaches to team coordination.
Similar to the approach of Carminati et al.
(2021), our team belief DAG form can be used to capture adversarial team games, and enables standard, out-of-the-box game-theoretic techniques including no-regret learning (e.g., CFR and its state-of-the-art modern variants such as DCFR and
PCFR + ) and first-order methods.
However, our representation can be exponentially smaller, and can be viewed as a lossless abstraction of theirs into a directed acyclic graph.
In particular, like the LP-based algorithm of Zhang & Sandholm (2022), the size of our representation scales with the amount of information uncommon to the team; in fact, using linear programming on top of our team belief DAG form to solve for a team correlated equilibrium in an adversarial team games recovers almost exactly their algorithm.
Unlike that paper, however, our representation explicitly exposes the structure of the decision space, which is what enables the aforementioned game-theoretic techniques.
I NTRODUCTION
Decision making in teams has been a recent focus in computational game theory and reinforcement learning.
Teams that cannot communicate perfectly have particular problems that single players and perfectly-communicating teams do not have.
Examples include recreational games like Bridge (in which two teams compete adversarially), collusion in poker, military situations with restricted communications, various swindling settings, and many other real-world situations.
In general, computing optimal strategies in adversarial team games is NP -hard
(Chu & Halpern, 2001), even given the extensive form and assuming there are only two players and no adversary.
Despite this, there are cases that are tractable.
For example, if both teams exhibit so-called A-loss recall (Kaneko & Kline, 1995) or triangle-free interaction between two players (Farina & Sandholm, 2020), then polynomial-time algorithms are known to exist.
Until recently, techniques for solving adversarial team games were focused largely on column generation (Farina et al., 2018; 2021a; Zhang et al., 2021), which works well in some games in practice but has no theoretical guarantees and scales poorly in most games.
More recently, Zhang & Sandholm (2022) developed an algorithm for solving adversarial team games based on a novel tree decomposition of each player’s strategy space, and use it to devise a linear program.
They show parameterized complexity bounds on the runtime of the algorithm, and demonstrate strong practical performance.
Simultaneously, Carminati et al.
(2021) developed a conversion algorithm based on prior research in the multi-agent reinforcement learning community (e.g., Nayyar et al.
2013; Sokota et al. 2021).
Their algorithm converts an extensive-form adversarial team game into an exponentially-larger public team game .
In this paper, we unify these last two lines of work.
We give a representation of the decision problem faced by a team of players as a sequential decision problem (Farina et al., 2019a) on a directed acyclic graph (DAG), which we call the team belief DAG form .
Using a linear program to solve the resulting adversarial team game recovers almost exactly the algorithm of Zhang & Sandholm (2022).
However, our representation has several advantages.
First and most importantly, explicitly expressing the polytope as the set of flows over a DAG gives us a description of the set of strategies in terms of scaled extensions .
This description allows us to apply the vast theoretical literature on sequence-form games, especially with respect to regret minimization, to team games as well.
Without understanding the special structure of the DAG, this development is not possible.
Second, it is slightly more concise, owing to the slight differences in definitions of public states and ease of optimization of our method.
This will become clear in the experimental evaluation later in the paper.
Finally, it is conceptually cleaner and easier to understand, not requiring the extra machinery of tree decompositions.
The conceptual ease also allows easier theoretical statements and proofs with slightly tighter bounds.
Applying safe imperfect-recall abstraction (Lanctot et al., 2012) on top of the sequence form of the folding representation of Carminati et al.
(2021) for a given team yields a decision space that is also basically equivalent to our team belief DAG form.
However, this is a rather roundabout construction: their folding representation can have size exponentially larger than the team belief
DAG, so our framework yields exponentially-faster algorithms by avoiding the construction of that larger representation.
In experiments, we demonstrate that the state-of-the-art variants of counterfactual regret minimization—namely DCFR (Brown & Sandholm, 2019) or PCFR + (Farina et al., 2021b)—applied on top of our team belief DAG form almost always outperform linear programming in team games.
This represents the new practical state of the art for adversarial team games.
P RELIMINARIES
We now introduce the paradigm of tree-form sequential decision making (Farina et al., 2019a), which we will use throughout the paper.
2. T REE -F ORM S EQUENTIAL D ECISION M AKING
Definition 2.1.
A tree-form sequential decision-making problem (TFSDP) T is a rooted tree with node set S and labelled edges, in which each node is a decision node , at which the player takes an action, or an observation nodes , at which the player receives an observation.
The edges out of decision nodes are called actions , and the edges out of observation nodes are called observations .
The root node is denoted ∅ .
The set of leaf, or terminal , nodes is denoted Z T .
If s, s ′ ∈ S are two nodes, s ⪯ s ′ means that there is a directed path from s to s ′ .
If S, S ′ are set of nodes, S ⪯ S ′ means s ⪯ s ′ for some s ∈ S, s ′ ∈ S ′ .
A s is the set of labels on edges descending from node s (actions or observations).
The node reached by following edge a ∈ A s out of node s is denoted sa .
For two nodes s, s ′ ∈ S , s ∧ s ′ denotes the lowest common ancestor of s and s ′ .
A pure strategy is an assignment of one action to each decision node.
The sequence form 1 x ∈ { 0 , 1 } Z T of a pure strategy is the vector for which x
[ z ] = 1 if and only if the player plays all the actions on the root → z path.
A mixed strategy is a distribution over pure strategies.
The sequence form of a mixed strategy is the corresponding convex combination x ∈ [0 , 1] Z T .
2. O NLINE C ONVEX O PTIMIZATION
Online convex optimization (Zinkevich, 2003) is a framework for describing repeated interactions of a player with an arbitrary environment.
At each timestep, the player selects a strategy x t from a convex, compact set X ⊆ R m , and observes a (possibly adversarially chosen) utility vector u t ∈ R m .
The goal of the player is to minimize the regret after T timesteps:
[Equation]
A regret minimizer is any algorithm for the player that guarantees that R T is sublinear in T .
In this paper, we will be concerned with regret minimizers over the decision space X ⊆ [0 , 1] Z T of sequence-form mixed strategies in various sorts of decision problems.
D ECISION M AKING ON DAG S
In this section, we discuss how techniques that apply to sequential decision making, for example, no-regret learning, can be used on a decision problem that is a DAG, which may be of independent interest beyond our interest in teams.
Definition 3.1.
A DAG-form sequential decision problem (DFSDP) is an TFSDP on a DAG instead of a tree.
We will insist on the following technical conditions:
Observation nodes always have exactly one parent.
Nodes along every path alternate between decision nodes and observation nodes.
If p 1 and p 2 are two paths from the root ending at the same node, then the last node common to both p 1 and p 2 is a decision node.
The first two conditions are for expository simplicity and are without loss of generality; the final one is critical for our constructions to work.
The sequence form x ∈ { 0 , 1 } Z T of a pure strategy is the vector for which x
[ z ] = 1 if and only if the player plays all the actions on some root
→ z path 2 .
Mixed strategies and their sequence forms are defined as usual.
3. DFSDP S VIA S CALED E XTENSIONS
In this section, we show that the set of sequence-form strategies in a DFSDP can be expressed in terms of scaled extensions (Farina et al., 2019b).
Definition 3.2.
Given two nonempty, compact, convex sets X , Y and a linear map h : X → R ≥ 0 , the scaled extension of X with Y via h , is defined as
[Equation]
We now construct the set of sequence-form strategies in a given DFSDP.
We begin with the set
X ← { 1 } .
Then, for each decision node s , we perform the operation
X ← X
x [ s ] ◁ ∆
A s where
[Equation]
The technical condition ensures that x [ s ] ∈ [0 , 1] for every s .
The restriction of the resulting set X on the set of terminal states Z D is exactly the set of sequence-form mixed strategies.
Thus, we have shown:
Theorem 3.3.
The set of sequence-form strategies on a DFSDP can be expressed by scaled extension operations with simplices via functions h : X → [0 , 1] .
Corollary 3.4.
This set can be expressed as a polytope with O ( E ) constraints, where E is the number of edges.
3. R EGRET M INIMIZATION IN DFSDP S
Any set that can be built from scaled extensions and simplexes admits a regret minimizer that can be constructed starting from any simplex regret minimizer (Farina et al., 2019b).
This construction extends CFR (Zinkevich et al., 2007), and all its modern variants, to such sets.
In particular, applying
Proposition 1 of Farina et al.
(2019b) on top of Theorem 3.3 gives us:
Corollary 3.5.
Deterministic variants of CFR (e.g., CFR + , DCFR (Brown & Sandholm, 2019),
PCFR + (Farina et al., 2021b)) can be run on a DFSDP, with regret bounded by O ( |S|√ T ) after T timesteps and iteration time O ( E ) where E is the number of edges in the DAG.
Pseudocode for running CFR on an arbitrary DFSDP can be found in Algorithm 1.
T EAM S EQUENTIAL D ECISION M AKING
In this section, we define the team TFSDP , a generalization of the TFSDP to the setting where there are multiple collaborating players with uncommon information.
Definition 4.1.
A team TFSDP is a TFSDP in which decision nodes are partitioned into information sets , or infosets .
We will insist that decision nodes in the same information set have the same action set.
The action set in a given information set I will be denoted A I .
To disambiguate from non-team TFSDPs and to be consistent with literature on extensive-form games, we will use H for the set of nodes in a team TFSDP instead of S .
For an infoset I and an action
a ∈
A I , we will use Ia to denote { ha : h ∈ I } .
The team sequence σ ( h ) of a node h is the sequence of infosets reached and actions played by the team along the root → h path, including, if any, the infoset containing h itself.
The effective size of a set of nodes H ⊆
H is the number of distinct team sequences among the nodes in H .
A team TFSDP is timed if no infoset spans multiple layers of the decision problem.
That is, paths from the root to nodes in the same infoset must have the same length.
In this paper, we will only work with timed team TFSDPs 3 .
A pure strategy is an assignment of one action to each infoset.
Insisting on choosing actions on the infoset level ensures that team members are not making decisions based on information that they do not know.
A correlated strategy is a distribution over pure strategies 4 .
The sequence form of a correlated strategy is called its correlation form .
It is known that the polytope of correlation plans cannot be represented by polynomially many constraints unless P = NP (Chu & Halpern, 2001).
It will occasionally be useful to us to distinguish the individual players on a team.
Formally, if the team consists of n players, we partition the collection of infosets into n collections I 1 , . . .
, I n , where
I i is the set of infosets at which player i plays.
The player sequence σ i ( h ) of a node h for a player i is the sequence of infosets reached belonging to i , and actions played by i , on the root → h path, again including the infoset at h itself if i plays at h .
We will assume that each individual player on a team has perfect recall : if h, h ′ ∈
I i , then σ i ( h )
=
σ i ( h ′ ) .
A team TFSDP can also be thought of as a (single-agent) TFSDP with imperfect recall .
In this presentation, the team is represented as a single player whose memory is repeatedly updated to only contain the information of the team member who is currently playing.
The two models are equivalent.
P UBLIC S TATES AND THE T EAM B ELIEF DAG F ORM
Let T be a team TFSDP.
We will assume that T is completely inflated (Kaneko & Kline, 1995): there does not exist a partition I 1 ⊔
I 2 of any infoset I in T such that, for every h 1 ∈ I 1 and h 2 ∈ I 2 , there is some infoset I ′ such that I ′ a 1 ⪯ h
and I ′ a 2 ⪯ h 2 for different actions a 1 ̸ = a 2 ∈ A I ′ .
This assumption assumptions can be satisfied WLOG by preprocessing the tree.
Definition 5.1.
Two nodes h, h ′ in the same layer of T , and either both terminal or both nonterminal 5 , are connected if they have the same team sequence, or there is an infoset I such
that h ⪯ I and h ′ ⪯ I .
Connectedness induces a graph whose nodes are the decision nodes, which we call the connectivity graph .
Definition 5.2.
A public state is a connected component of the connectivity graph.
Intuitively, a public state is a set of nodes P such that whether P has been reached is common knowledge among the team.
The public states themselves induce a tree, which is known as the public tree .
We will be interested in two separate branching factors.
First, the team branching factor b is the largest number of actions at any decision node in T .
Second, the public branching factor b ′ is the branching factor of the public tree 6 .
The team belief DAG form of a team TFSDP T is a DFSDP D defined as follows.
All nodes in the team belief DAG form are identified with sets of nodes in the team TFSDP.
The initial node of D is a decision node corresponding to the set {∅} containing only the root node of the original problem.
At a decision node B in D , let I 1 , . .
, I m be all the infosets with nonempty intersection with B , and let J be the set of observation nodes (of T ) in set B .
The player selects a prescription a ∈ × i ∈ [ m ]
A
I i , consisting of one action a i in each infoset I i .
The decision problem then transitions to the observation node B a consisting of all children of nodes in B which are consistent with the actions of the player.
Formally,
[Equation]
At an observation node O in D , let P 1 , . . .
, P m be the public states with nonempty intersection with O .
Then the player transitions to the decision node O ∩ P i for some i ∈ [ m ] .
If a decision node B contains terminal nodes in D , then it is terminal.
The sets B identified with decision nodes in D are called beliefs .
In this construction, multiple decision nodes may be identified with the same belief.
The subtrees induced by these decision nodes are copies of each other.
Therefore, we merge them, forming a DAG.
This DAG is the team belief
DAG , and the DFSDP is the team belief DAG form .
For an example, see Figure 1; for pseudocode, see Algorithm 2.
To be consistent with the previous two sections and to distinguish a team TFSDP from its team belief
DAG form, we will use H to refer to the set of nodes in the former, and S to refer to the set of nodes in the latter.
Given a sequence-form mixed strategy x ′ in the team belief DAG form, we construct a correlation plan x in T as follows.
For each terminal node z in x , let Σ( z ) be the set of all terminal nodes in H sharing a team sequence and a level with z , that is, [Equation] level
[Equation]
Then Σ( z ) is a terminal decision node in D by construction.
We set x
[ z ] = x ′ [Σ( z )] .
Theorem 5.3.
The set of vectors x constructible in the above fashion is exactly the set of correlation plans of the team TFSDP.
The proof of this theorem and the proofs of all following theorems are in the appendix.
T HE S IZE OF A T EAM B ELIEF DAG
Since all our theoretical results depend on the size of the team belief DAG, it is critical to analyze that size.
The hardness result of Chu & Halpern (2001) means that our size bounds will not be polynomial.
However, we can still bound the size relative to natural parameters related to the complexity of the game.
The correspondence between our construction and that of Zhang & Sandholm (2022) allows us to achieve similar theoretical guarantees to that paper.
Here, we explicitly give such results in our language.
Let T be a team TFSDP with node set S and public state set P .
Theorem 6.1.
The team belief DAG of T has at most (cid:0) p
[Equation]
where p is the largest effective size of any public state and w is the largest effective size of any belief.
The parameter w is identical to the same-named parameter in Zhang & Sandholm (2022).
As discussed in that paper, w depends only on the amount of uncommon external information , that is, observations (as opposed to decisions) that are not common knowledge to the team.
In a certain family of team TFSDPs including those with team-public actions, we can do better.
Definition 6.2.
An n -player team TFSDP is k -private if, in every public state, there are at most k distinct player sequences.
That is, |{ σ i ( h ) : i ∈ [ n ] , h ∈ P }| ≤ k for every P ∈ P .
This is distinct from the effective size p , which is the total number of team sequences in P .
In particular, in games with team-public actions (such as poker), where each player has at most t private types, we have k ≤ nt .
In a k -private team TFSDP, it is possible that w = ( k/n ) n , so Theorem 6.1 gives a bound of (2 bp )
( k/n ) n b ′ |P| , which is bad.
However, we can improve upon this through a more careful analysis.
Theorem 6.3.
The team belief DAG of a k -private team TFSDP has at most ( b + 1) k b ′ |P| edges.
If necessary, it is possible to “mix and match” the analyses of Theorems 6.1 and 6.3 when some public states have low w and some have low t .
To save the cumbersome notation, we will not do that here.
A DVERSARIAL T EAM G AMES
All our results so far have been given from the perspective of a single team.
However, they generalize very naturally to the case of multiple teams competing in a game.
An extensive-form adversarial team game Γ consists of:
a tree of nodes H , where the edges are labelled with actions.
The set of terminal nodes in H will, as usual, be denoted Z Γ ; 2.
a partition I of nonterminal nodes in H into infosets, where every node in a given infoset must have the same action set; 3.
a partition of I into three collections
I C ⊔
I ⊕ ⊔ I ⊖ , where I C is the collection of infosets at which chance acts, and I τ is the collection of infosets assigned to team τ ∈ {⊕ , ⊖} ; 4.
for each infoset I ∈ I C assigned to chance, a fixed distribution over the actions available at I ; and 5.
a utility function u : Z Γ → R , where u ( z ) denotes the utility for ⊕ in terminal node z .
The utility for ⊖ is − u
( z ) .
As usual, we will not demand that the teams have perfect recall.
An extensive-form game immediately defines a team TFSDP for each team τ , with information partition I τ .
For consistency with our definition of the team belief DAG form, we insist that non-chance nodes all have exactly 2 children.
A team correlated equilibrium (TMECor) in an adversarial team game is a Nash equilibrium of the game viewed as a two-player zero-sum game between the two teams.
A TMECor can be expressed as
[Equation]
the solution to a convex bilinear saddle-point problem
[Equation]
where A is the payoff matrix , whose bilinear form is the expected utility for ⊕ if ⊕ plays correlation plan x and ⊖ plays correlation plan y .
Let S and E be the total number of nodes and edges respectively across the team belief DAGs of both teams the following results follow from standard results on linear programming for saddle-point problems and regret minimization, and Corollary 3.5:
Corollary 7.1.
Nash equilibria in adversarial team games can be found via a linear program of size
O ( E + |Z Γ | ) .
Corollary 7.2.
In adversarial team games, after T iterations of a CFR variant, the average strategy for both players is an O ( S/ √ T ) -Nash equilibrium.
Each iteration takes time O ( E ) .
The former result is analogous to the results shown in Zhang & Sandholm (2022).
The latter, to our knowledge, is new.
In light of Corollary 3.5 and Theorems 6.1 and 6.3, Corollaries 7.1 and 7.2 immediately imply fixed-parameter runtime and convergence bounds for both the CFR family of algorithms and linear programming.
E XPERIMENTS
We experimentally investigate solving adversarial team games using the team belief DAG form.
Since all our experiments are in games with public actions, we preprocess with branching factor reduction in all cases.
8. E QUILIBRIUM -F INDING A LGORITHMS
We investigate the application of the following two categories of game-solving algorithm on the belief
DAG of the team:
CFR-like algorithms obtained from the scaled extension decomposition of the team belief DAG (Section 3.2).
We implemented the following state-of-the-art variants of CFR: Predictive CFR + (PCFR + ) (Farina et al., 2021b), Discounted CFR (DCFR) (Brown & Sandholm, 2019), and Linear CFR (LCFR) (Brown & Sandholm, 2019).
PCFR + and DCFR use quadratic averaging of iterates, while LCFR uses linear averaging.
PCFR + is a predictive regret minimization algorithm.
At each time t , we use the previous utility vector for each time as prediction for the next.
Each implementation is single-threaded.
Linear programming (LP) solvers applied directly on the bilinear saddle point formulation of the equilibrium problem (1).
To solve the linear program, we used the commercial solver Gurobi.
We investigated using both the barrier algorithm, and the (concurrent primal-dual) simplex algorithm.
For each algorithm, we experimented leaving the presolver on or off.
We allowed Gurobi to use up to four threads.
We compare solving an adversarial team games in team belief DAG form against two prior state-ofthe-art algorithms:
The tree-decomposition-based LP solver proposed by Zhang & Sandholm (2022) (henceforth “ ZS22 ”), which has already discussed at length in this paper.
We used the original implementation of the authors, which internally uses the barrier algorithm implemented by the commercial solver Gurobi.
As recommended by the authors, we turned Gurobi’s presolver off to avoid numerical instability and increase speed.
We allowed Gurobi to use up to four threads.
The single-oracle algorithm of Farina et al.
(2021a)
(henceforth “ FCGS21 ”).
FCGS21 iteratively refines the strategy of each team by solving best-response problems using a tight integer program derived from the theory of extensive-form correlation (von Stengel & Forges, 2008; Farina et al., 2021b).
We used the original code by the authors, which was implemented for three-player games in which a team of two players faces an opponent.
Like ZS22 and our LP-based solver, FCGS21 uses the commercial solver Gurobi to solve linear and integer linear programs.
We allowed Gurobi to use up to four threads.
All algorithms were run on a 64-core AMD EPYC 7282 processor.
Each algorithm was allocated a maximum of 4 threads, 60GBs of RAM, and a time limit of 6 hours.
8. G AME I NSTANCES
We ran experiments on the following standard, parametric benchmark games:
• n K r : n -player Kuhn poker with r ranks (Kuhn, 1950).
• n L brs : n -player Leduc poker with a b -bet maximum in each betting round, r ranks, and s suits (Southey et al., 2005).
• n D d : n -player Liar’s Dice with one d -sided die for each player (Lis`y et al., 2015).
• n G and n GL: n -player Goofspiel with 3 ranks.
GL is the imperfect-information variant.
These are the same games used by Zhang & Sandholm (2022) and Farina et al.
(2021a)
in their experimental evaluations.
We refer the reader to the latter paper for detailed descriptions of the games.
The size of each game, measured in terms of number of terminal states (leaves), is reported in the second column of Table 1.
Experimental results are summarized in Table 1.
Overall, we observe that our algorithms based on the team belief DAG form are generally 2-3 orders of magnitude faster than ZS22.
In games with low parameters p and k , our algorithms are also several orders of magnitude faster than FCGS21, validating the conclusion of Zhang & Sandholm (2022).
The difference in runtime between our algorithms and FCGS21 is especially dramatic in Leduc games.
In games with high parameters ( e.g. ,
K8 and 3 K12), on the other hand, FCGS21 is significantly more scalable, as it avoids the exponential dependence in the parameters at the cost of requiring the solution to integer programs, for which runtime guarantees are hard to give.
C ONCLUSION AND F UTURE R ESEARCH
We have given a new representation, the team belief DAG form , for the decision problem faced by a team of correlating players, which we have used to develop new algorithms for solving adversarial team games.
Our method enjoys the parameterized complexity bounds of Zhang & Sandholm (2022), and the extensibility and interpretability of Carminati et al.
Experiments show that modern variants of CFR applied with our team belief DAG form give state-of-the-art performance across multiple domains.
Possible directions for future research include:
1. devising a technique to allow the use of Monte Carlo CFR (MCCFR)
(Lanctot
et
al., 2009) in DFSDPs, and in particular in the team belief DAG form; 2. finding theoretically sound techniques for mitigating the exponential blowup in parameters w and k ; 3. finding a “best-of-both-worlds” algorithm that combines the strengths of our approach and the single-oracle-based methods; 4.
investigating extensions of our technique to correlated equilibrium in general-sum games; 5.
relaxing the assumption of timeability; 6. devising a construction that additionally generalizes the triangle-free interaction (Farina & Sandholm, 2020), a known polynomially-solvable subclass of the problem; and 7.
investigating the use of other standard game-theoretic techniques in two-player zero-sum games, such as abstraction, dynamic pruning, subgame solving, etc., in team games.
A CKNOWLEDGMENTS
This material is based on work supported by the National Science Foundation under grants IIS1718457, IIS-1901403, and CCF-1733556, and the ARO under award W911NF2010081.
A E XPERIMENTS T ABLE
Experimental results are summarized in Table 1.
Column ‘ Game ‘ indicates the game, and the set of players on Team ⊖ .
Columns ‘ Team ⊕ ’s DAG ’ and ‘ Team ⊖ ’s DAG ’ report the total number of vertices and edges in the team belief DAG for teams ⊕ and ⊖ respectively.
Column ‘ Team ⊕ value ’ reports the utility that team ⊕ can expect to gain at equilibrium.
Column ‘ P. S. Size ’ reports the largest effective size p of any public state.
Column ‘ k ’ reports the value of k for which both teams are k -private.
Column ‘ This paper, CFR ’ of Table 1 reports the time to convergence of the best CFR variant to an average team exploitability of less than 10 − 3 times the range of payoffs of the game.
Convergence plots for all CFR variants on all games can be found in the appendix.
Column ‘ This paper, LP ’ reports the runtime of the best of the LP algorithms (see Section 8.1) operating on the belief DAG form of the game, solving to to Gurobi’s default precision.
Column ‘ ZS22 ’ reports the time it took ZS22 to compute an equilibrium strategy for team ⊕ , again to
Gurobi’s default precision.
Finally, column ‘ FCGS21 ’ reports the time it took FCGS21 to compute an equilibrium strategy for team ⊕ with exploitability of less than 10 − 3 times the range of payoffs of the game.
The missing values in that column are due to the fact that the implementation of FCGS by the original authors only supported 3-player games.
B B RANCHING F ACTOR R EDUCTION
Since the branching factor b appears as the base of an exponential in Theorems 6.1 and 6.3, it is natural to ask whether it can reduced without affecting the other parameters.
This turns out to be true assuming team-public actions , which we now formalize.
[Table] [Table]
[Table]
[Table]
[Table] [Table] [Table] the team belief DAG form, compared to the prior state-of-the-art algorithms by Zhang & Sandholm (2022) (‘ ZS22 ’) and Farina et al.
(2021a)
(‘ FCGS21 ’), on several standard parametric benchmark games.
See the beginning of Appendix A for a detailed description of the meaning of each column, and Section 8.2 for a description of the games.
Missing or unknown values are denoted with ‘—’.
For each row (benchmark game), the background color of each runtime column is set proportionally to the ratio R with the best runtime for the row, according to the logarithmic color scale
[Figure]
Definition B.1.
A team TFSDP has team-public actions if, for all public states P containing decision nodes, for all edge labels ( i.e. , actions or observations) a ∈ (cid:83)
h ∈ P
A h , the set { ha : h ∈ P, a ∈ A h } is a union of public states.
Intuitively, this means that any action taken by a team member becomes common knowledge for the team.
The definition also allows for information other than the action to become common knowledge, and for some public states to give the team members private information.
Theorem B.2.
Given a team TFSDP T with public actions, there exists another realization-equivalent team decision problem T ′ such that the branching factor of T ′ is at most 2 at each decision node, the parameters p, n, k, w in T ′ are the same as in T , and |P| has increased by at most a constant factor.
Corollary B.3.
In a team TFSDP T with public actions, it is possible to create a team DAG for T with O ((2 p + 2) w b ′ |P| ) or O (3 k b ′ |P| ) edges.
C P ROOFS C.
T HEOREM 5.
We will show the claim for pure strategies, which is enough since mixed and correlated strategies come from taking convex combinations of pure strategies.
( ⇒ ) Consider a pure strategy π in T , inducing a correlation plan x .
Consider the pure strategy in the team belief DAG form in which the team chooses the prescription in each belief consistent with π , inducing a sequence form x ′
Let z be a terminal node in T , and suppose x [ z ] = 1 .
We need to demonstrate a path through
D leading to Σ( s ) such that π plays every action prescribed along that path.
Consider the path through D defined by following the prescriptions of π , and always selecting the public state that leads to z .
By construction of the team belief DAG form, this path must end exactly at z , so x ′ [Σ( z )]
= 1 .
Conversely, suppose that such a path exists.
Then, every infoset I ⪯ z must have appeared in some belief node B along the path, and, at that belief node, in order for Σ( z ) to still have been reachable, the team must have chosen the action at I leading to z .
Thus, the team plays all actions on the path from the root to z , so x
[ z ] = 1 .
( ⇐ )
Consider
a
pure
strategy π ′ in D , and let x ′ be its
sequence form
Define the pure strategy π in T as follows In
each public state P , π ′ induces a unique belief B ⊆ P
Let a be the prescription in π ′ at B .
At every infoset I ⊆ P , define π ( I ) =
a I if I intersects B , and arbitrarily otherwise.
We claim that π defined in this way is realization equivalent to
π ′ As before, let x be the correlation plan of π .
Suppose x ′ [Σ( z )] = 1 ; that is, there is a path through D ending at Σ( z ) at which
π ′ plays every prescription.
Then, at each public state P along this path, the belief B ⊆ P induced by π ′ must be exactly the unique belief used to construct π at P .
Thus, in particular, if P contains an infoset I ⪯ z , then I ∩ B ̸ = ∅ .
Thus, the team plays all actions on the root → z path, so x [ z ] = 1 .
Conversely, suppose x [ z ] = 1 .
Then, a straightforward induction shows that, at every public state P on the root → z path, the belief at B induced by π ′ must contain an ancestor of z and thus must be played to.
This completes the proof.
C. T HEOREM 6.
A belief B ⊆ P of a public state P ∈ P is uniquely identified by its sequence set σ ( B ) .
We have | σ ( B ) | ≤ w by definition.
Hence, there are at most (cid:0) p ≤ w (cid:1) such beliefs, and at most b w prescriptions a at B .
For each prescription a , the children of B a must be public state children of P ; in particular, there are at most b ′ of them.
Multiplying these gives the desired result.
C. T HEOREM 6.
Consider a public state P .
Each of the k player sequences corresponds to at most one information set in P .
Thus, to specify that player sequence’s contribution to a belief-prescription pair B a , it suffices to specify one of: either the player does not play to the sequence, or the player chooses one of her b possible actions at the sequence.
Thus, there are at most ( b + 1) k possible belief-prescription pairs
B a .
Each one has at most b ′ children as argued before.
The bound follows.
[Equation]
Consider a public state P of T .
If P contains no decision nodes, we leave it alone.
Otherwise, let
B be an arbitrary binary tree with leaf set
A :
= (cid:83)
h ∈ P
A h .
The internal nodes of B will be labelled with partial actions ˜ a , which we can think of as partial bitstrings of indices of actions in A .
For each node h ∈ P , we replace h with a modified copy of B wherein subtrees containing no nodes in A h have been pruned.
If h and h ′ are in the same infoset in P , then for every partial action (i.e., nonterminal node)
˜ a ∈ B we connect h ˜ a and h ′ ˜ a in an infoset.
This creates a new public tree T ′ , whose parameters we must now analyze.
For each public state P of T , the construction creates the internal nodes of a public subtree in T ′ with leaves corresponding to the child public states of P .
This adds at most as many public states as there are children of P in T , so the number of public states in T ′ is at most twice that of T .
The number of players n remains the same.
For each new public state P ∗ constructed in this process, we have P ∗ ⊆
P ˜ a := { h ˜ a : h ∈ P, ˜ a ⪯ a ∈ A h } for some ˜ a , where ⪯ denotes precedence in B (the subset may not be the whole set, because it is possible for the partial action ˜ a to have already revealed further common knowledge that was not available at P ).
Thus, every team or player sequence σ ( h ) in P identifies at most one unique team or player sequence in P ∗ —namely σ ( h ˜ a ) , if present—and thus p and k have not increased.
Finally, for each belief B ⊆ P , the largest belief in P ˜ a induced by B is B ˜ a , which has no larger effective size.
Hence, w has not increased.
This completes the proof.
D CFR C ONVERGENCE P LOTS
In this section, we show the performance of each of the three CFR variants that we implemented to perform no-regret learning on the team belief DAG.
As a rule of thumb, the predictive algorithm
PCFR+ (Farina et al., 2021b) is fastest when high precision (low team exploitability) is necessary.
For low precision, DCFR (Brown & Sandholm, 2019) is often the fastest algorithm in practice, especially in certain variants of Kuhn poker.
[Figure] [Figure]