Paper: DAG-Based Column Generation for Adversarial Team Games (ICLR-2024)

Reviewer comment (Presentation):
As mentioned by the authors, the basic idea of using the sequence form to speed up column generation/double oracle for extensive-form games was proposed by Bosansky et al. (2014). While the authors discuss this comparison frankly in Appendix A, this should also be reflected in the framing of the paper itself. For example, Bosansky et al.'s method could be introduced in the main body, and then the proposed DCG method framed as the natural analogue of that method for adversarial team games. The current framing of DCG can make it seem more complex than it actually is. Conceptually, Appendix A is arguably the most important part of the paper and should be moved into the main text and discussed earlier, potentially even in the introduction. If space is needed, less critical elements such as some ablation experiments or the discussion of the BRO could be moved to the appendix.

Author response:
**Comment**: About moving the related work in Appendix A to the main body of the paper.\
**A**: Thanks for giving us this interesting idea. We once put all related work in the main body of the paper, but the space is not enough. So, we summarized the related work in Section 3 and presented details in Appendix A. Specifically, at the beginning of Section 3, we present:

Solving Problem (1) is challenging because the team’s coordinated strategy space is exponential in the size of the game tree. With the procedure shown in Figure 1(c), CG can mitigate this challenge but still converges slowly in large games due to such a large strategy space. It has been shown that the TB-DAG-based algorithms are more efficient than CG in games with a small TB-DAG (Zhang et al., 2022b;c). However, solving Problem (1) through the TB-DAG is impractical in large games because the size of the TB-DAG is still exponential in the size of the game tree (Zhang et al., 2022c). To speed up, one straightforward idea is applying CG with the procedure shown in Figure 1(c) to the TB-DAG, which also is the direct application of the sequence-form double oracle (Bosansky et al., 2014) to the TB-DAG. That is, we compute a TMECor with a limited size of the TB-DAG, and then expand the DAG by computing a best response over the whole TB-DAG form strategy space, i.e., solving the problem: max_{\boldsymbol{x}} u_T (\boldsymbol{x}, \boldsymbol{y}_o). However, it is inefficient to compute a best response over the whole exponential-sized TB-DAG. As shown in experiments, this exponential size will make CG with the DAG-based BRO very inefficient. More details on related work are shown in Appendix A.

**Comment**: About presenting the viewpoint of applying the sequence-form double oracle Bosansky et al. (2014) algorithm to ATGs.\
**A**: Thanks again for your interesting idea. We once considered presenting the viewpoint of applying the sequence-form double oracle Bosansky et al. (2014) algorithm to ATGs. However, compared to the current viewpoint of applying CG to the TB-DAG, both viewpoints give us the same procedure and do not need the key features of the sequence-form double oracle algorithm in Bosansky et al. (2014), but using CG is the tradition in the mainstream literature on ATGs (Celli & Gatti, 2018; Farina et al., 2018; Zhang et al., 2021; Farina et al., 2021; Zhang et al., 2022b). Therefore, we chose the current viewpoint of applying CG to the TB-DAG:
Recall that the sequence-form double oracle algorithm in Bosansky et al. (2014) is only for two-player extensive-form perfect-recall zero-sum games. However, as we mentioned, an ATG is an imperfect-recall game. Thus, the algorithm in Bosansky et al. (2014) cannot be directly applied to original ATGs. To apply the algorithm in Bosansky et al. (2014) for solving ATGs, we need to transform the original strategy representation of ATGs into the TB-DAG form. Then, we can compute a TMECor with a limited size of the TB-DAG for the restricted game and then expand the TB-DAG by computing a best response over the whole TB-DAG of the original game. Unfortunately, it is inefficient to compute a best response over the whole exponential-sized TB-DAG. Therefore, the sequence-form double oracle algorithm in Bosansky et al. (2014) that is directly applied to the TB-DAG is inefficient in large games. To improve the scalability, similar to Section 3, we can propose a novel sequence-form oracle algorithm framework that includes 1) computing a coordinated best response, 2) transforming this coordinated best response to the TB-DAG form, and then 3) adding this equivalent TB-DAG form strategy to the restricted game. This algorithm framework has an extra transformation step and needs a suitable BRO to reduce the cost of the transformation. We can see that the above algorithm procedure is the same as our current DCG. We chose to present our algorithm from the viewpoint of applying CG to the TB-DAG because:

1.	As we mentioned, both viewpoints will result in the same algorithm.

2.	In the mainstream literature on ATGs (Celli & Gatti, 2018; Farina et al., 2018; Zhang et al., 2021; Farina et al., 2021; Zhang et al., 2022b), CG was used to solve ATGs because the team’s strategy space is in normal form and is exponentially larger than the strategy space of the opponent (adversary). 

3.	Because the strategy space of the opponent (adversary) is exponentially smaller than the team’s strategy space, we only expand the strategy space of the team. Thus, we do not need the key features of the sequence-form double oracle algorithm (Bosansky et al., 2014) in the restricted game (i.e., incomplete strategy and unallowed actions), as we mentioned in Appendix A.1.

We added Appendix A.2 in the new version to discuss the viewpoint of applying the sequence-form double oracle Bosansky et al. (2014) algorithm to ATGs.

Reviewer comment (Theory):
For the BRO, there should be discussion of the total size of the representation (number of nonzeros in the constraint matrix) as a function of variables such as the number of sequences, number of players, and game size. In the worst case (e.g., when the number of players is large), this size can be exponential in |H|, which is concerning, especially compared to past BROs that, as noted in Appendix A, do not suffer from this issue. For instance, in a polymatrix game with \(N\) players and \(A\) actions per player transformed naturally into extensive form, \( |H| = O((NA)^2)\) but every tuple of sequences would be relevant, resulting in a BRO representation size on the order of \(A^N\).

Author response:
**Comment**: About the representation size of BRO\
**A**: We have added Appendix A.3 in the new version to discuss the representation size of BRO. Specifically, for our pure BRO:
Our pure BRO ensures that only one joint action for each reachable belief will be used to expand the TB-DAG and then reduce the cost of the transformation. We exploit the team’s correlation property defined in Eq.(2a) (a probability flow) to improve the scalability of our BRO. In the original two-player von Stengel-Forges polytope (Von Stengel & Forges, 2008) for this probability flow, the constraints involve all relevant sequences. If we consider all relevant joint sequences for multiple team members, the number of constraints in our BRO will be exponential in the size of the game tree. To reduce the number of constraints, we consider only a subset of relevant joint sequences for nodes and sequences of the original game tree 
(i.e., $\\boldsymbol{\Sigma}^\bowtie_T$) and limit the relevant relation of an information set and a joint sequence to $\\boldsymbol{\Sigma}^\bowtie_T$. That is, $\\boldsymbol{\Sigma}$$^\bowtie_T$= {$\sigma_T(h_i) \mid h_i\in H\cup Z\\}\cup\\{(\sigma_i,\times_{j\in T\setminus\{i\}}\varnothing)\mid\sigma_i\in\Sigma_i,i\in T\\}$ and   $I_i\bowtie\sigma_{T\setminus\{i\}}$   only if   $(\sigma_i(I_i),\sigma_{T\setminus\{i\}})\in \\boldsymbol{\Sigma}^\bowtie_T$. 
We further add constraints in Eq.(2b) to ensure that $\xi(\sigma_T) =\prod_{i\in T}\xi(\sigma_T[i],\boldsymbol{\varnothing}_{-i})$ for   each $\sigma_T\in \\boldsymbol{\Sigma}^{\bowtie}_T$.
Therefore, our pure BRO shown in Program (3) has polynomial-sized constraints, i.e., $O(|H \cup Z||T |$) constraints.

In the previous version, the constraints for our BRO were not clearly presented. Now, we have clearly presented these constraints in the new version and highlighted them in the paper.

Reviewer comment (Experiments):
I would like to see experiments on team-vs-team games, where both teams would have to apply the column-generation method. In such scenarios, the advantage over the baseline DAG algorithm might be smaller due to the added inefficiency of performing column generation for both sides, though the method is still expected to perform very well.

Author response:
**Q**: (from above) How does the proposed technique (or, the natural generalization thereof) perform in the setting of team-vs-team games?\
**A**: First, in this paper, we focus on the ATGs with a team of players and an opponent, following the main mainstream literature on ATGs (Celli & Gatti, 2018; Farina et al., 2018; Zhang et al., 2021; Farina et al., 2021, Zhang et al. 2022a). Solving team-vs-team games could be the future work. 

Second, the baseline DAG directly solves the linear program for a TMECor after generating the whole TB-DAG. From results in Tables 1 and 2 for our current setting, we have seen that DAG is the fastest algorithm in games with very narrow game trees (the size of the TB-DAG is small) but runs out of memory in games with wide game trees (the size of the TB-DAG is large) where DAG performs worse than our algorithm. Similarly, we believe that in the setting of team-vs-team games, DAG will be faster than our algorithm in games with very narrow game trees but runs out of memory in games with wide game trees where DAG performs worse than our algorithm.

Reviewer comment (Experiments):
How does the proposed technique (or its natural generalization) perform in the setting of team-vs-team games?

Author response:
**Q**: (from above) How does the proposed technique (or, the natural generalization thereof) perform in the setting of team-vs-team games?\
**A**: First, in this paper, we focus on the ATGs with a team of players and an opponent, following the main mainstream literature on ATGs (Celli & Gatti, 2018; Farina et al., 2018; Zhang et al., 2021; Farina et al., 2021, Zhang et al. 2022a). Solving team-vs-team games could be the future work. 

Second, the baseline DAG directly solves the linear program for a TMECor after generating the whole TB-DAG. From results in Tables 1 and 2 for our current setting, we have seen that DAG is the fastest algorithm in games with very narrow game trees (the size of the TB-DAG is small) but runs out of memory in games with wide game trees (the size of the TB-DAG is large) where DAG performs worse than our algorithm. Similarly, we believe that in the setting of team-vs-team games, DAG will be faster than our algorithm in games with very narrow game trees but runs out of memory in games with wide game trees where DAG performs worse than our algorithm.

Reviewer comment (Theory):
Why does \( DCG_{\text{pure}} \) perform differently from \( DCG_{\text{linrelax}}^{\text{pure}} \)? Wouldn't an integer program solver first attempt to solve the linear relaxation anyway, leading to the same outcome?

Author response:
**Q**: Why does DCG_pure perform differently from DCG_linrelax^pure? Wouldn't an integer program solver first attempt to solve the linear relaxation anyway? (so they should do the same thing?)\
**A**: As we mentioned, the setting of DCG^linrelax_pure is: at each iteration, it solves the linear relaxation of the BRO first to see if it can output the optimal solution added to the restricted game; if it cannot do that, it solves the original mixed-integer BRO and adds all feasible solutions for the BRO from the CPLEX solution pool to the original game.

We use the default setting of the integer program solver, so we are not sure whether it first attempts to solve the linear relaxation. In addition to the linear relaxation, DCG^linrelax_pure usually adds more strategies to the restricted game than DCG_pure. As we mentioned in Section 4, the relaxed version DCG^linrelax_pure of DCG_pure, in most games with relatively narrow game trees, outperforms DCG_pure because it could reduce the cost of calling the mixed-integer BRO. However, in games with relatively wide game trees, DCG_pure outperforms DCG^linrelax_pure because DCG^linrelax_pure transforms too many coordinated strategies into the TB-DAG form, which incurs a very high cost in games with relatively wide game trees.

Reviewer comment (Evaluation):
No clear winner between \emph{pure} or \emph{linrelax} versions of the algorithm

Author response:
As we mentioned, the setting of \emph{linrelax} is: at each iteration, it solves the linear relaxation of the BRO first to see if it can output the optimal solution added to the restricted game; if it cannot do that, it solves the original mixed-integer BRO and adds all feasible solutions for the BRO from the CPLEX solution pool to the original game.

That is, \emph{linrelax} usually adds more strategies to the restricted game than \emph{pure}. Consider the relaxed version DCG^linrelax_pure of DCG_pure. As we mentioned in Section 4, the relaxed version DCG^linrelax_pure of DCG_pure, in most games with relatively narrow game trees, outperforms DCG_pure because it could reduce the cost of calling the mixed-integer BRO. However, in games with relatively wide game trees, DCG_pure outperforms DCG^linrelax_pure because DCG^linrelax_pure transforms too many coordinated strategies into the TB-DAG form, which incurs a very high cost in games with relatively wide game trees.

Reviewer comment (Novelty):
The proposed algorithm somewhat lacks originality, in the sense that it is an incremental improvement that recombines ideas already available in the literature. This lowers the contribution provided by the paper, even if my opinion is that this paper is interesting nonetheless.

Author response:
Our DCG does not just directly apply CG to the TB-DAG. Our work provides the first efficient TB-DAG CG algorithm, which includes a novel CG algorithm framework and solves many games that were previously unsolvable. Our DCG algorithm computes a pure coordinated best response (i.e., a coordinated strategy represented by a joint normal-form strategy of the team) in the original game tree first and then transforms it into the TB-DAG form. Our DCG has an extra transformation step, which does not exist in previous CG algorithms. Our DCG significantly overcomes the limitation of previous CG/double oracle algorithms for a TMECor with a novel CG algorithm framework and a suitable best response oracle. More details on related work are shown in Appendix A.

We provide a solid technical contribution in providing a novel CG framework for TMECor computation. For example, we show that the transformation cost is only polynomial in the size of the original game tree, which makes it clear that the transformation cost is not overly expensive. We also show that our algorithm converges exponentially faster than the state-of-the-art CG algorithms.

Reviewer comment (Writing):
In many occasions it is said that "by exploiting the team’s correlation property.. the BRO".. This property is not clear to me, and I think it should be better specified in the text.

Author response:
Thanks. We explicitly mentioned that the team’s correlation property is the probability flow defined in Eq.(2) in the new version.

Reviewer comment (Novelty):
The BRO algorithm presented in Section 3.3 is not completely novel. A similar algorithm was provided in "Subgame Solving in Adversarial Team Games" by Zhang et al., with the similar purpose of providing a more sparse BR (in that case sparsity is useful for reaching fewer private states in the same public state). My suggestion is to add a short reference to such a similar method.

Author response:
Their technique is different from ours. In Appendix A.4 of the new version, we discussed the technique in "Subgame Solving in Adversarial Team Games" by Zhang et al.:

Zhang et al. (2022a) provided a subgame technique to solve ATGs and proposed using CG for sparse solutions in the subgame solving algorithm. That is, they compute a best response starting from each reachable belief in each public state. To keep the number of these reachable beliefs small, they create sparse blueprints in the subgame solving algorithm by using a CG algorithm because “the support size of the blueprint generated by a CG algorithm, scales linearly with the number of iterations, which, under reasonable time constraints, rarely exceeds the hundreds” (Zhang et al., 2022a). Their sparse blueprint and our pure best response are both sparse solutions, but theirs is different from our suitable BRO:

 1. The goals are different: Our goal is to reduce the cost of the transformation in our DCG, but their goal is to reduce the number of times that the BRO is called.

 2. The sparsity concepts are different: Their sparse blueprint is about a small support size of the blueprint (a mixed strategy of the gadget game), but our pure best response is just about an action for each reachable state/belief.

 3. The approaches are different: We propose a new suitable BRO, i.e., pure BRO, to improve our DCG, but they just directly use an existing CG algorithm for a sparse blueprint, i.e., they do not provide a new BRO algorithm.

Reviewer comment (Novelty):
The column generation algorithm does not really add anything new compared to what is done in classical column generation algorithms.

Author response:
Thank you very much for your valuable comments.

**Comment**: About novelty and contribution:

**A**: Our algorithm DCG does not just directly apply previous CG algorithms to the TB-DAG. Our work provides the first efficient TB-DAG CG algorithm, which includes a novel CG algorithm framework and solves many games that were previously unsolvable. Our DCG algorithm computes a pure coordinated best response (i.e., a coordinated strategy represented by a joint normal-form strategy of the team) in the original game tree first and then transforms it into the TB-DAG form. Our DCG has an extra transformation step, which does not exist in previous CG algorithms. Our DCG significantly overcomes the limitation of previous CG algorithms for a TMECor with a novel CG algorithm framework and a suitable best response oracle. More details on related work are shown in Appendix A.

We provide a solid technical contribution in providing a novel CG framework for TMECor computation. For example, we show that the transformation cost is only polynomial in the size of the original game tree, which makes it clear that the transformation cost is not overly expensive. We also show that our algorithm converges exponentially faster than the state-of-the-art CG algorithms, and experimental results show that our algorithm is at least two orders of magnitude faster than the state-of-the-art baselines and solves games that were previously unsolvable. 

Thus, this paper creates a fundamental theory for applying the multiagent learning framework – Policy-Spaced Response Oracles (PSRO) (Lanctot et al., 2017) (a variant of CG) – to the TB-DAG for a TMECor.

Reviewer comment (Novelty):
The only addition made by the proposed algorithm is related to the implementation of the best response oracle, and this is not enough to constitute a contribution warranting publication in a top-tier AI conference such as ICLR.

Author response:
Thank you very much for your valuable comments.

**Comment**: About novelty and contribution:

**A**: Our algorithm DCG does not just directly apply previous CG algorithms to the TB-DAG. Our work provides the first efficient TB-DAG CG algorithm, which includes a novel CG algorithm framework and solves many games that were previously unsolvable. Our DCG algorithm computes a pure coordinated best response (i.e., a coordinated strategy represented by a joint normal-form strategy of the team) in the original game tree first and then transforms it into the TB-DAG form. Our DCG has an extra transformation step, which does not exist in previous CG algorithms. Our DCG significantly overcomes the limitation of previous CG algorithms for a TMECor with a novel CG algorithm framework and a suitable best response oracle. More details on related work are shown in Appendix A.

We provide a solid technical contribution in providing a novel CG framework for TMECor computation. For example, we show that the transformation cost is only polynomial in the size of the original game tree, which makes it clear that the transformation cost is not overly expensive. We also show that our algorithm converges exponentially faster than the state-of-the-art CG algorithms, and experimental results show that our algorithm is at least two orders of magnitude faster than the state-of-the-art baselines and solves games that were previously unsolvable. 

Thus, this paper creates a fundamental theory for applying the multiagent learning framework – Policy-Spaced Response Oracles (PSRO) (Lanctot et al., 2017) (a variant of CG) – to the TB-DAG for a TMECor.