From: ICLR.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Reject
comment: The paper presents tackles the problem of finding strategies that are -- unlike Nash which is safe -- both safe (non-exploitable, to some extent) and able to exploit the opponent. The proposed solution is a convex combination of exploitation and safety that is efficient to compute. Overall, the paper is borderline. Given that the objective and its analysis are not especially surprising, a lot rides on the thoroughness of the empirical results, which could be improved.

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: Response to Reviewer hmTT
comment: 

Thank you very much for your feedback! We are pleased that our response has clarified your concerns. During the discussions with other reviewers, we have provided additional experimental results to more thoroughly evaluate our method.

- Reviewer SWZo and f5FT proposed a new method EXP-STRATEGY. EXP-STRATEGY builds on the same gadget game of SES (Figure 1 in this paper). It is different from SES in that, it keeps opponent strategy fixed to the estimation in the right part of the gadget game while SES does not. SES exploits an estimated opponent infoset distribution, while EXP-STRATEGY exploits a full estimated opponent strategy. 
- As required by reviewer SWZo, we have done some comparisons between SES and EXP-STRATEGY in Leduc and FHP, confronted with opponents with different strengths ($\Pr_{shuffle}$), and we have committed to do more (our commitments are in a direct comment to this paper). In these experiments, we find that in some cases, SES can achieve lower exploitability than EXP-STRATEGY under the same evaluation results (experiment on FHP with opponent  $Pr_{shuffle}=0.3$). And EXP-STRATEGY usually has a wider range of evaluation performances than SES. Please refer to our 2nd&3rd responses to reviewer SWZo for details.
- As SWZo has pointed out, since EXP-STRATEGY is also a new algorithm, which can be regarded as a variant of SES, so the intention of this paper is not to "argue that SES is strictly superior". We believe the main contribution of this paper is that it is the first safe exploitation algorithm that builds on real-time search, which enables computationally efficient online adaptations


What's more, we are grateful for your suggestion on notations. We will refine them in the final revision.

Thank you again for your time and efforts in reviewing our work!

Best Regards,

The authors

From: ICLR.cc/2022/Conference/Paper1309/Reviewer_hmTT
title: Thank you!
comment: Thank you for your responses on many of the points that my review raised. Your feedback has helped me clarify a few aspects. Though, since my understanding of the underlying convex combination structure was fundamentally correct, I still somewhat stand behind my comment that, given the exceptional simplicity of the idea I would have preferred if the authors had done more to convince that the approach is robust experimentally, and had presented a significantly more thorough experimental evaluation. However, your clarifications and to the revised version you have uploaded have improved my opinion of the paper, and we are now discussing internally.

PS. As a minor comment that in no way affects the discussion about the paper, I have noticed that you are using the notation $\langle a, b \rangle$ to denote a pair $(a, b)$ in the revised version of the paper. As the $\langle\cdot,\cdot\rangle$ notation is (in my experience universally, as in, across disciplines) standard for the inner product of $a$ and $b$, I think you might want to pick a different symbol for tuples/pairs. As a final minor comment, I think that denoting the utility of a pair of strategies $(a,b)$ as $u^{\langle a,b\rangle}$ is very uncommon and serves as an unnecessary barrier of entry for your paper.

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: Thank you for your time and efforts in reviewing our work!
comment: Dear Reviewer,

Thank you for your time and efforts in reviewing our work. We have provided detailed clarification to address the issues raised in your comments. If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.

If you have any additional questions or comments, we would be happy to have further discussions.

Thanks,

The authors

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: Thanks for raising the score to 8!
comment: We would like to thank the reviewer for raising the score! We really appreciate the valuable comments and suggestions from the reviewer, which greatly help improve our work.

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: Thanks for raising the score and some updates (2/2) 
comment: | Leduc Evaluation ($\Pr_{\rm shuffle}=0.0$) | EXP-STRATEGY                 | SES                |
| ------------------------------------------ | ------------------- | ------------------ |
| $\alpha=p=0.0$                             | 1.81 ($\pm$ 0.09)   | 1.78 ($\pm$ 0.10)  |
| $\alpha=p=0.3$                             | 1.60 ($\pm$ 0.09)   | 1.86 ($\pm$ 0.15)  |
| $\alpha=p=0.7$                             | -1.13 ($\pm$ 0.85)  | 1.96 ($\pm$ 0.37)  |
| $\alpha=p=1.0$                             | -69.42 ($\pm$ 6.42) | -1.60 ($\pm$ 2.05) |

| Leduc Evaluation ($\Pr_{\rm shuffle}=0.3$) | EXP-STRATEGY                 | SES               |
| ------------------------------------------ | ------------------- | ----------------- |
| $\alpha=p=0.0$                             | 3.97 ($\pm$ 0.76)   | 4.00 ($\pm$ 0.72) |
| $\alpha=p=0.3$                             | 5.26 ($\pm$ 0.81)   | 4.36 ($\pm$ 1.15) |
| $\alpha=p=0.7$                             | 13.64 ($\pm$ 2.69)  | 5.63 ($\pm$ 2.58) |
| $\alpha=p=1.0$                             | -27.60 ($\pm$ 5.79) | 3.87 ($\pm$ 5.23) |

| Leduc Evaluation ($\Pr_{\rm shuffle}=0.7$) | EXP-STRATEGY                | SES                |
| ------------------------------------------ | ------------------ | ------------------ |
| $\alpha=p=0.0$                             | 8.47 ($\pm$ 2.65)  | 8.39 ($\pm$ 2.81)  |
| $\alpha=p=0.3$                             | 11.58 ($\pm$ 4.10) | 9.23 ($\pm$ 3.05)  |
| $\alpha=p=0.7$                             | 27.29 ($\pm$ 9.37) | 11.06 ($\pm$ 3.37) |
| $\alpha=p=1.0$                             | 5.92 ($\pm$ 9.55)  | 12.59 ($\pm$ 4.39) |

**Q3**: Adding discussion to the text discussing the differences between the two and how these differences are important.

**A3**: Both algorithms of SES and EXP-STRATEGY (proposed by the reviewer) have to construct a gadget game (defined in this paper, Figure 1). In the exploitation part of the game tree, both algorithms use the estimated probability $\hat{p}$ to sample which opponent infoset to enter. EXP-STRATEGY further fixes the opponent strategy to its estimation afterward, while SES does not. So, EXP-STRATEGY exploits the full estimated opponent strategy, while SES exploits an estimated distribution of opponent infosets.

Current experiments show that EXP-STRATEGY usually has a wider range of evaluation performances than SES. However, in some cases, SES can achieve lower exploitability than EXP-STRATEGY under the same evaluation results (experiment on FHP with opponent $\Pr_{\rm shuffle}=0.3$).

We admit that for a full comparison between these two algorithms, current experiments are not adequate. For instance, we should further investigate how modeling error impacts the results of both algorithms. Additionally, we will try our best to analyze the theoretical results of EXP-STRATEGY. Due to the time limit, we are not able to finish these in the discussion period. We have made commitments to finish these in the direct comment of the paper as the reviewer required.

In the final, we would like to thank the reviewer again for the insightful and beneficial comments on our work, especially for the constructive advice which enriches this work. Also, we appreciate your valuable time spent on our work!

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: Commitments
comment: 

We are grateful to all reviewers for their beneficial comments and advice on our paper. Especially, reviewer SWZo and f5FT propose an interesting method that is closely related to our algorithm SES. The method, which we call EXP-STRATEGY, builds on the same gadget game of SES (Figure 1 in this paper). It is different from SES in that, it keeps opponent strategy fixed to the estimation in the right part of the gadget game while SES does not. SES exploits an estimated opponent infoset distribution, while EXP-STRATEGY exploits a full estimated opponent strategy. Due to the time limit in the discussion period, we are not able to finish all comparisons between SES and EXP-STRATEGY.

As requested by reviewer SWZo, **we hereby promise to finish the following:**
 
2. **Complete experiments in FHP with more opponents (e.g., opponent $\Pr_{\rm shuffle}=0$).** (Half-done. Please refer to A2 in our second response to SWZo, and A1 in our third response to SWZo)
3. **Besides what we have explained in the responses to reviewer SWZo, we will discuss more in text about the differences and their significance. Our discussion will be based on new experiment results.** (Half-done. Please refer to A3 in our third response to SWZo)
4. **Add experiments on an additional game (such as Leduc)** (Done. Please refer to A2 in our third response to SWZo)

To be honest, it is hard to absolutely guarantee that a new theoretical result on EXP-STRATEGY can be achieved. For instance, the estimated opponent strategy is harder to analyze than just infoset distribution. But **we promise to**

1. **Try our best to theoretically characterize EXP-STRATEGY's guarantees in comparison to SES.**

Additionally, we also plan to complete experiments with various modeling errors $\epsilon$ to see how it impacts both algorithms, and which algorithm shows more robustness.

Thank all the reviewers again for their time and efforts on reviewing our work! If there are any additional questions or comments, we will be happy to have further discussions.

From: ICLR.cc/2022/Conference/Paper1309/Reviewer_Wp7Z
title: Follow-up
comment: Thank you for the response.

Regarding Q2:
I was not referring to generating opponents with different strengths, but rather the error in estimated opponent strategy (whose magnitude is governed by $\epsilon$, and each curve in a single plot in Fig 3). It probably isn't a huge deal, but should be included for completeness.

Regarding connections to [Ganzfried & Sandholm '15] and [Johanson et al '07] raised by other reviewers:
I think an unnecessary amount of confusion was caused (at least to me) by referencing over-loaded terms with different meanings from these 2 papers. For example, the term "Gift" was used in page 7, when more easily understood terms like exploitable or suboptimal could be used. Given that the notion of safety in SES is is quite different from [Ganzfried & Sandholm '15], I am not sure if these references help with clarity.

Regarding [Johanson et al '07]. One of the advantages SES holds is its requirement that only subgames are solved, as opposed to the full game. How does the objective between the 2 differ if say, there is only one subgame (i.e., the full game)?

Having re-read the other comments and the paper, I felt that one of the advantages touted by SES ---  being able to adapt to a possibly changing opponent model, was rather poorly elaborated on. In its current state, it appears to be a justification to dismiss a comparison with RNR. While I agree with sentiment in theory, I cannot think of a reasonable scenario where this advantage is significant.

From: ICLR.cc/2022/Conference/Paper1309/Reviewer_f5FT
title: Response
comment: 1. This means that SES, as used in the paper, is performing really only part of the possible exploitative behavior: it is exploiting the opponent's preflop mistakes in its postflop play, and that's it. In other words, SES does not exploit postflop mistakes at all, nor does it fully exploit preflop mistakes that predominantly affect the preflop (such as going all-in too often preflop). Am I understanding correctly?

1. My conclusion from seeing the table of RNR vs SES experiments is that RNR has a different, more broad, parameterization of the frontier of safety vs evaluation than SES. The parameter α in SES does not have the same "power" as the parameter p in RNR, and as such, this is expected. But it means that SES(α) and RNR(p=α) cannot be directly compared. For example, SES(α=1) only achieves evaluation 9.21, which RNR achieves with p<0.3. We should be comparing SES(α=1) to RNR(p) where p is chosen so that evaluation = 9.21, not to RNR(p=1). Without comparing the frontiers achieved by the methods, I cannot justify in my mind the stronger conclusions claimed in the above post.

1. On the case of $\tilde p = 0$, I make two comments. First, the same logic applies with $\tilde p \approx 0$ since then the division is by a small number, and I expect the values in the probability tables are small enough to run into this. Second, in my opinion it is likely that the lack of zeros in that table is due, once again, to lack of perfect convergence of equilibrium finding. In exact equilibrium (which is what is assumed for computing $\tau$ in Thm 1), I would expect many infosets played to with zero probability. For example, I expect that one should fold some starting hands such as 2-7 offsuit 100% of the time, and thus never reach the corresponding postflop infosets. 

    If the authors believe me to be wrong, I invite them to disprove me by reporting the values of $\tau$ as defined in the statement of Thm 1, ideally using an exact equilibrium generated by (say) LP rather than an approximate one.

From: ICLR.cc/2022/Conference/Paper1309/Reviewer_SWZo
title: Response to Additional Experiments
comment: Thanks for completing these experiments so quickly!

> We think the reviewer's intention in (2) is to compare with "p1 is forced to play the estimated opponent strategy" instead of "blueprint strategy"

Indeed, this was my intention. I apologize for not wording this clearly.

> Since the new P1 node does not exist in the original subgame , there is neither blueprint strategy nor estimated opponent strategy defined at this node. One potential way to address this problem is to use the estimated probability, which can be calculated from the estimated opponent strategy, as a substitute. Then this will be exactly the formulation of (2).

That's true. I apologize that my initial suggestion was not fully fleshed out. One "technically feasible" way to approach it that would not be the same as (2) would be to have P1 compute a new strategy for this node. While I still think this would be interesting to see, I don't think it's as important as (2). Given the short amount of remaining time, I suggest the authors forgo these experiments.

> To conclude, we find that although method (2) is close to SES in architecture, it does not ensure safety as SES does, when the estimation of the opponent's strategy is not accurate.

Is this an empirical claim based on the experiments above, or a theoretical claim?

> We would like to add some more comparisons if the reviewer requires them.

I think that the most important additional experiments (in my opinion) are the comparisons against (2) with other agents and other games (eg Leduc).

Other than these experiments, I think it would be good to show theoretical results for (2) analogous to Theorems 1 & 2 toward the end of giving a theoretical comparison to SES.

----

After some additional reflection, I am wondering whether it may make sense for the submission to be centered around the comparison between SES and (2) -- IE, whether the opponent's future strategy should be allowed to deviate from its estimated strategy, even when it has been assumed to have played according to its estimated strategy in the past. I think the experiments discussed above help to make that comparison. I think this comparison would be further strengthened by:
1. Theoretically characterizing (2)'s guarantees in comparison to SES
2. Completing the experiments above with other agents
3. Adding discussion to the text discussing the the differences between the two and how these differences are important
4. Adding experiments on an additional game (such as Leduc)

Lastly, I think the paper may be strengthened by approaching the writing with the tone of comparing SES and (2), rather than arguing that SES is strictly superior.

Given the changes that have been made so far, I think the submission merits a higher score than I originally gave it, so I am raising my score from a 5 to 6. If the submission were to complete 1-4 enumerated above, I think this would merit an additional improvement from a 6 to an 8. Unfortunately, it is unlikely there is sufficient time in the review period for the authors to complete these additional points. Thus, here is what I propose. If the authors i) make a good faith effort to make as much progress toward completing 1-4 as is possible in the remaining time and ii) publicly commit to completing the remaining points in *bolded text* as a direct comment on the paper (rather than as a reply to this comment), I will raise my score to an 8. While I am aware that there is no enforcement mechanism here should the authors not follow through on promises, I am inclined to trust them given their engagement in the review process so far if they publicly commit to do so as described above.

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: New Response to Reviewer f5FT (2/2)  
comment: **A3**: We admit that $\hat{p}$ should apply zero probability wherever $\tilde{p}$ applies zero probability. In this way, the algorithm will possibly not be able to find the most exploitative strategy against the opponent. However, SES is aimed at balancing between safety and exploitation but not exploitation alone. Intuitively, those infosets assigned with 0 probability by NE are most risky, which may be not suitable for a **"safe"** exploitation.

Additionally, in our experiment game FHP, we find that for most infosets $I_1^j$, $\tilde{p}(I_1^j)$ is not zero. In FHP, we have

|  Probability of Infoset on top of subgame ( $\tilde p(I_1^j)$ )     |   Fraction of Infoset with probability in the range   |
| -------------------- | ------ |
| $[10^{-1},1]$        | 0.0000 |
| $[10^{-2},10^{-1}]$  | 0.0000 |
| $[10^{-3},10^{-2}]$  | 0.2834 |
| $[10^{-4},10^{-3}]$  | 0.5677 |
| $[10^{-5},10^{-4}]$  | 0.1414 |
| $[10^{-6},10^{-5}]$  | 0.0074 |
| $[10^{-7},10^{-6}]$  | 0.0000 |
| $[10^{-8},10^{-7}]$  | 0.0000 |
| $[10^{-9},10^{-8}]$  | 0.0000 |
| $[10^{-10},10^{-9}]$ | 0.0000 |

The result is generated from 112 different blueprint strategies generated by different random seeds with 10,000,000 iterations each.

Therefore, the estimated opponent strategy $\hat{p}$ is still not strictly constrained by this condition. Together with empirical results in A2, we respectfully disagree with the reviewer that the safety guarantee of SES is trivial.

**A4**: We understand the reviewer's concern that, even when RNR(big) consumes 10M CFR iterations at each round, which is the same that SES uses to compute its blueprint, it is still possible that it has not converged. We will try RNR with more iterations to ensure its convergence and then report the results.

From: ICLR.cc/2022/Conference/Paper1309/Reviewer_f5FT
title: Response to response
comment: I feel that the response has only caused me to have more questions rather than fewer.

1. You claim that RNR is orders of magnitude less efficient computationally than SES. This is only possible if SES is used only in subgames "late" in the tree. I am assuming that this is what you are doing--in any case, you should specify explicitly in the paper how the subgames are defined in the experiments. Therefore, in particular, the early-game strategy is not exploitative at all, but rather identical to the blueprint, correct? I was previously (incorrectly, apparently) thinking of SES as being applied at every decision point in a nested fashion (as is common in the subgame solving literature).

1. Speaking of comparison to RNR, why is the opponent model used only for root probabilities, and not for modeling strategies during the subgame itself? Indeed, this seems like the only difference between SES and "RNR applied to the maxmargin gadget subgame". The latter would be an alternative to SES that carries over the theoretical guarantees of RNR in a more direct way. It seems like a strong baseline that should be compared against, if nothing else.

1. Building off the discussion with SWZo: My understanding of the revised Thm 1 is as follows. To give a nontrivial exploitability guarantee, it must be the case that $\hat p$ applies zero probability wherever $\tilde p$ applies zero probability, correct? (Otherwise, $\tau$ is infinite) (You should probably specify explicitly that the max in the definition of $\tau$ is only taken over $I^i_1$ such that $\tilde p(I^i_1) > 0$, which I'm assuming is actually the case; if this is not the case then we have bigger problems.) This seems like a very strong condition, especially considering that, in some sense intuitively, the mistakes generated by playing actions that are zero probability in equilibrium are exactly the "worst" mistakes, that should be exploited the most. In light of (2) above, this means in my mind that the safety guarantee of SES will be often simply trivial.

4. Why does RNR have worse performance than SES on all metrics? In particular, RNR(p=1) should be just computing a best response, and RNR(p=0) should be computing an NE, so they should be at least as good as SES(α=1) and SES(α=0) on the evaluation and exploitability metrics respectively. I assume this is only an artifact of RNR (even RNR(big)) not being given enough time to converge. I feel that the authors should make this more clear, as in that case the right takeaway is "SES is more computationally efficient than RNR" instead of "SES converges to better exploitation/exploitability than RNR". Indeed, the latter would be very surprising in my opinion if it were true.

From: ICLR.cc/2022/Conference/Paper1309/Reviewer_SWZo
title: Follow Up
comment: > We follow the common definition of exploitability in this paper.

I agree that the revised version uses the common definition of exploitability. I also agree that the statement of Theorem 1 was the source of my confusion. That being said, unless I am mistaken, this appears to have been a mistake on the part of the writing, rather than on the part of my interpretation. I have listed the point of confusion below for convenience. Please let me know if you feel that my interpretation was unfair.

The original submission uses the definitions:

 - Let $\tilde{p}(I_1^i)$ be the real reach probability,  and $\hat{p}(I_1^i)$ be its estimation.

The revised submission uses the definitions:

 - Let $\tilde{p}(I^i_1)$ be the reach probability given by $σ_1^*$. Let $\hat{p}(I_1^i)$ be the estimation of reach probability $p(I^i_1)$ given by the real opponent strategy.

While the statement has been clarified in the revision, that one of the two main theorems was misstated signals some amount of sloppiness, which makes me less inclined to trust the submission's proofs, which I have not checked in full detail.

> We provide both exploitability and expected utility results for mixing strategy.

Great, it may be nice to emphasize this in the submission.

> As suggested by the reviewer, we have added detailed comparisons with previous safe exploitation algorithms in Section 2, and empirical results in Section 5.3 in the revision.

Great, thanks for doing this.

Looking at these experiments, it seems like there are some questions regarding how to fairly compare an exploitation algorithm that uses subgame resolving to an exploitation algorithm that does not. This leads me to suggest two additional baselines. 1) Run RNR on top of the maxmargin gadget games; 2) Run the existing SES gadget game, but with the modification that $p_1$ is forced to play their blueprint strategy in $S_2$ (as notated in Figure 1. I am aware there is not much additional time to perform these experiments, but hope that these would only require small changes to the author's existing codebase to implement.

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: Response to Reviewer f5FT (2/2)
comment: **Q3:** "The main contribution of the paper is only a minimal step on top of known ideas (a convex combination of unsafe and maxmargin subgame solving)." "Not very novel/just incremental work over past work on subgame solving."

**A3**: Summing up A1 and A2, in spite of its simplicity in form, to our best knowledge, SES is the first algorithm that can efficiently perform safe exploitation in an "online" setting, where the opponent model is updated with streamed data. It relieves a serious restriction on practical use cases of previous algorithms. We also prove explicit bounds for both safety and exploitation, taking into account various practical factors such as opponent modeling error and blueprint strength. Our added experiments show that, with the aid of real-time search, SES is significantly more efficient than previous algorithm [Johanson et al '07] without search. Therefore, we respectfully insist that the advantages of our algorithm over previous works and its significance should not be underestimated.

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: Response to Reviewer f5FT (1/2)
comment: We appreciate the reviewer for the insightful comments.

**Q1:** "I'm not completely satisfied with the comparison to the past safe exploitation works [Johanson et al '07, Ganzfried & Sandholm '15]. As far as I can tell, the present paper simply uses a different parameterization of the Pareto frontier of exploitation vs exploitability -- it uses the mixing amount $\alpha$; the others use the exploitability, formulating the objective as 'maximize exploitation such that exploitability is at most $\varepsilon$'. These are not very different."

**A1:** We thank the reviewer to point out the comparions to [Johanson et al '07] and [Ganzfried & Sandholm '15]. **We have added detailed comparisons with previous safe exploitation algorithms in Section 2, and empirical results in Section 5.3 in the revision.**

We agree with the reviewer that our optimizing objective is not very different from "maximize exploitation such that exploitability is at most $\varepsilon$". However, we respectfully insist that the real-time search framework of SES plays a vital role. In added experiments, SES is much more efficient than p-RNR [Johanson 2007]. The reason is that p-RNR is a ***"global"*** algorithm, which means that whenever the opponent model or the exploitation level ($p$ in p-RNR) changes, it has to recompute the strategy for the whole game. In contrast, SES only efficiently computes safe exploitation strategies in subgames encountered, which are much smaller than the whole game. To our best knowledge, SES is the first algorithm that implements both "$\varepsilon$-safe exploitation" and "online adaptation" to a possibly changing opponent model.

The settings of [Ganzfried & Sandholm '15] and our paper are quite different. [Ganzfried & Sandholm '15] study strategies in ***"repeated games"***, and provide a full characterization of safe deviation from NE. In contrast, our work focuses on the safety of ***"stage game"*** strategies. The two definitions of "safety" are different.

Furthermore, we find that our algorithm is complementary to RWYWE proposed in [Ganzfried & Sandholm '15]. RWYWE generally proceeds as follows:

1. For each timestep $t$ in a repeated game, plays a $k^t$-safe best response $\pi_t^t$ with respect to the estimated opponent's strategy.
2. $k^{t+1} = k^t +$ expected utility of $\pi_i^t$ against the observed opponent's action $-$ NE value $v^*$

RWYWE achieves safety because it uses $k$ to track how much it has earned over NE in previous rounds, and then risk at most $k$ in exploitability. As stated in their paper, computing an $\varepsilon$-safe best response at every round is non-trivial. It is one of the main restrictions which limits RWYWE's scalability to even medium-sized games. If viewing SES as an efficient algorithm that approximates such $\varepsilon$-safe exploitation strategy, it is possible to integrate  SES into RWYWE, and provide more scalable algorithms for safe exploitation in repeated games as well.

**Q2:** "Further, LP solving in games is not that slow, especially with recent advances [Zhang & Sandholm '20]; in particular, the subgames in experiments seem tractable with LP."

**A2:** We admit that using LP to solve games in our experiments is feasible. However, as stated in A1,  opponent exploitation is an "online process" where an algorithm should adaptively respond at each round. Previous works [Johanson et al '07, Ganzfried & Sandholm '15] need to recompute a new $\varepsilon$-safe best response for the **whole** game at every round. It is still computationally hard for LP and CFR without real-time search. The added experiment in revision supports our argument that [Johanson et al '07] is much less efficient than our algorithm. From the other perspective, it is possible to use LP in the computation of SES, by using LP instead of CFR to solve the gadget game constructed by SES. Real-time search algorithms usually do not restrict which solver to use, and this is also true for SES.

What's more, in practice (especially in the real-time search domain), we do not require an extremely accurate Nash equilibrium. Usually, $10mbb/h$($0.01bb/h$) is a very accurate Nash equilibrium in practice. And [Zhang & Sandholm '20] also illustrate that Poker Specific DCFR is much faster than the best LP solver when we only require a strategy with exploitability of no more than $0.01$. DCFR only takes no more than 10 CPU seconds in River Endgame Solving while LP solver takes more than 100 CPU seconds.

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: Response to Reviewer SWZo (2/2)
comment: **Q4**: Also, while I do not dispute it is difficult to scale LP solvers to large games, the games considered in the submission are not that large.

**A4**: We admit that using LP to solve games in our experiments is feasible. However, the opponent exploitation setting is more than calculating a strategy once. Instead, it requires calculating a responding strategy with respect to a possibly changing opponent model at each round. [Ganzfried & Sandholm '15] uses LP to find $k^t$-safe best response at each round. Solving the whole game over and over again is still computationally hard. As stated in A3, [Johanson 2007] is also a global algorithm that is much less efficient than SES. In contrast, SES does not need to recompute a whole game strategy. Our added experiments in section 5.3 support this argument. Therefore, SES suffices the need for online playing against opponents. From the other perspective, it is possible to use LP in the computation of SES, by using LP instead of CFR to solve the gadget game constructed by SES. Real-time search algorithms usually do not restrict which solver to use, and this is also true for SES.

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: Response to Reviewer SWZo (1/2)
comment: We appreciate the reviewer for the insightful comments. Especially, we are grateful to the reviewer for pointing out typos in writing and citations, and we have made corrections accordingly in the revision. We provide clarification to address other concerns of the reviewer in the following.

**Q1**: "The term 'exploitability' is not generally used in a way that depends on the opponent's 'true distribution'." "SES is not, in fact, safe against arbitrary opponents."

**A1**: We follow the common definition of exploitability in this paper. We find that the original statement of Theorem 1 may cause the reviewer's confusion. We have refined this statement. In Theorem 1, the exploitability does not depend on the distribution of the best response. $\tau$ indicates how much $\hat p$ deviates from the distribution given by a Nash equilibrium $\langle \sigma_1^*, \sigma_2^* \rangle $ where $\sigma_2^*$ is constrained to be the same with $\sigma_2$ outside the subgame. Therefore, $\tau$ is a fixed value no matter what the best response is.

**Q2**: "That being said, it seems to me to be perfectly possible to provide theorems analogous to Theorem 1 and Theorem 2 for the 'mixing between a safe strategy and an exploitation strategy' approach."

**A2**: We provide both exploitability and expected utility results for mixing strategy. Let ${\rm mix}_p(\sigma_1, \sigma_1')$ be the mixing strategy which plays $\sigma_1$ with probability $p$, and $\sigma_1'$ with $1-p$. [Johanson 2007] has already provided a full characterization of its expected utility: $u({\rm mix}_p(\sigma_1, \sigma_1'), \sigma_2)=pu(\sigma_1, \sigma_2) + (1-p)u(\sigma_1', \sigma_2)$. As for exploitability, it can be shown that $\exp({\rm mix}_p(\sigma_1, \sigma_1'))\leq p\exp(\sigma_1)+(1-p)\exp(\sigma_1')$. The example in Figure 2 shows that the bound is tight.

In some circumstances, $\exp(\sigma_{unsafe})$ can be quite large. Therefore, the exploitability of the mixing strategy can also be large even for a rather small $p$. In contrast, as shown in Theorem 1, SES's exploitability does not rely on $\exp(\sigma_{unsafe})$, which makes our bound for SES tighter than that of mixed strategy in most cases.

**Q3**: "The submission dismisses Johanson 2007 and Ganzfried 2015 as baselines. However, I am not sure that this dismissal is well justified."

**A3**: As suggested by the reviewer, **we have added detailed comparisons with previous safe exploitation algorithms in Section 2, and empirical results in Section 5.3 in the revision.**

Experiments show that our algorithm is much more efficient than p-RNR proposed in [Johanson 2007]. p-RNR is a ***"global"*** algorithm, which means that whenever the opponent model or the exploitation level ($p$ in p-RNR) changes, it has to recompute the strategy for the whole game. In contrast, SES only needs to compute the whole strategy once, which is the blueprint. Thereafter, SES computes safe exploitation strategies in the encountered subgames, which are much smaller than the whole game. Consider a practical setting where opponent models are updated with new interaction data after each round. The efficiency of making online adaptations becomes important. Furthermore, in theory, [Johanson 2007] does not provide an explicit bound on exploitability, while Theorem 1 in our paper quantitatively takes into account some vital factors.

We have to clarify that there are major differences in the settings of [Ganzfried & Sandholm '15] and our paper. [Ganzfried & Sandholm '15] study strategies in ***"repeated games"***. In contrast, our work is aimed at finding approximately safe ***"stage game"*** exploitation strategies. The two definitions of "safety" are different.

Moreover, we find that our algorithm is complementary to RWYWE proposed in [Ganzfried & Sandholm '15]. RWYWE achieves safety by tracking how much it has earned over NE in previous rounds and risking at most this amount in exploitability. As stated in their paper, computing an $\varepsilon$-safe best response at every round is non-trivial. It is one of the main restrictions which limits RWYWE's scalability to even medium-sized games. If viewing SES as an approximate algorithm for calculating such $\varepsilon$-safe exploitation strategy, it is possible to integrate SES into [Ganzfried & Sandholm '15] and to provide more scalable algorithms for repeated games as well.

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: Response to Reviewer hmTT (2/2)
comment: **Q2**: "The technical results (Theorems 1 & 2) are not surprising at all, and just amount to bounding the two terms in the definition of $f(x)$ independently."

**A2**: We respectfully disagree with this claim on the significance of the theoretical results in this work. To our best knowledge, SES is the first to provide explicit bounds on both safety and exploitation, taking into account various practical factors such as opponent modeling error and blueprint strength. Besides, we find that Theorem 1 can possibly answer some fundamental problems insufficiently addressed by previous works. [Brown et al '17] found that unsafe search usually counterintuitively achieves rather low exploitability but they did not explain the reason. We believe that our Theorem 1 partially explains why unsafe search is usually safe. When testing the unsafe search, [Brown et al '17] use their blueprint strategy to calculate $\hat p$, which is close to $\tilde p$ created by $\sigma^*$. Here $\sigma^*=\langle \sigma_1^*, \sigma_2^* \rangle$ is a Nash equilibrium when constrained $\sigma_2^*$ to have same strategy as $\sigma_2$ outside the subgame. As a result, $\tau$ is small and the exploitability is low.

[McCracken et al '04] Peter McCracken and Michael Bowling. Safe strategies for agent modeling in games. Papers from the 2004 AAAI Fall Symposium.

[Johanson et al '07] Michael Johanson, Michael Bowling, and Martin Zinkevich. Computing robust counter-strategies. 2007.

[Ganzfried et al '15] Sam Ganzfried and Tuomas Sandholm. Safe opponent exploitation. *ACM Transactions on Economics and Computation (TEAC)*, 3(2):1–28, 2015a.

[Moravcik et al '16] Moravcik M, Schmid M, Ha K, et al. Refining subgames in large imperfect information games. Proceedings of the AAAI Conference on Artificial Intelligence. 2016, 30.

[Brown et al '17] Brown N, Sandholm T. Safe and nested endgame solving for imperfect-information games. Workshops at the thirty-first AAAI conference on artificial intelligence, 2017.

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: Response to Reviewer hmTT (1/2)
comment: We appreciate the reviewer for the insightful comments.

**Q1**: "The technique presented by the authors to interpolate between a Nash equilibrium strategy and a best response (say, for Player 1) simply boils down to the following: introduce a parameter $\alpha$ and optimize the objective $f(x)=(1-\alpha)\cdot \min_{y\in Y}x^TUy + \alpha x^TU\bar{y}$." "The presence of search seems just an implementation detail that obscures the above, straightforward idea."

**A1**: Technically, because SES uses search in subgames, the exact formulation is more complex than $f(x)=(1-\alpha)\cdot \min_{y\in Y}x^TUy + \alpha x^TU\bar{y}$. The main reason is that a subgame is **not** an imperfect information game. An imperfect information game has a **unique** root. But in a subgame, there are multiple roots (with the same public information). We give the exact formulation in the following.

$
\max_{v,x}~(1-\alpha) m + \alpha \hat p^Tv~~~~~(1)
$

$
s.t.~~~~~~v_I-m\geq CBV_1^{\sigma^S_2}(I)~~~~~(2)
$

$
Ex=e~~~~~(3)
$

$
F^T v-A_2^T x\leq 0~~~~~(4)
$

$
x\geq 0~~~~~(5)
$

The notations are borrowed from [Moravcik et al '16]. In the equation above, $m$ denotes the margin $\min_{I_1^j\in S_{\rm top}} \Big(v_1^{\sigma}(I_1^j)-CBV_1^{\sigma_2^S}(I_1^j)\Big)$ and $v_j=v_1^{\sigma}(I_1^j)$. And in the constraint, $x$ denotes the probability of reaching each state in the subgame. And $Ex=e$ constraints that the probability of reaching a state is equal to the probability of going out of it. And (4) is the constraint to guarantee that $x$ is achieved when the opponent best responds to our strategy.

We admit that our optimization objective for subgame search is a convex combination of exploitation and safety, but it is simple yet efficient and effective. This combination form is a commonly applied objective in safe opponent exploitation literature. As pointed out by another reviewer, this objective is closely related to "maximize exploitation such that exploitability is at most $\varepsilon$", which is used in [McCracken et al  04, Johanson et al '07].

We respectfully disagree with the point that "search seems just an implementation detail that obscures the above, straightforward idea". **We have added detailed comparisons with previous safe exploitation algorithms in Section 2, and empirical results in Section 5.3 in the revision.** Our algorithm performs better and is much more efficient. Whenever the opponent model is updated with new interaction data, the previous algorithm [Johanson et al '07] has to recompute an $\varepsilon$-safe best response in the whole game, which is computationally inefficient. In contrast, SES only searches for strategies in subgames encountered. The real-time search framework enables SES to make efficient online adaptations with respect to a possibly changing opponent model, which is generally infeasible for previous works. As stated in [Ganzfried et al '15], online calculation of safe exploitation is one of the main limitations on the algorithm's scalability. We respectfully insist that the novelty and significance of our paper should not be underestimated.

From: ICLR.cc/2022/Conference/Paper1309/Authors
title: Response to Reviewer Wp7Z
comment: We appreciate the reviewer for the insightful comments. We are also grateful to the reviewer for pointing out clarity in some definitions and descriptions, and we have addressed these issues in the revision. We also provide clarification to the reviewer's main concerns in the following.

**Q1**: "How useful these bounds introduced in the theorems are in practice, in particular for selecting an appropriate value of alpha?" "In practice, how do we know we have a bad estimation if there are no samples to compare with?"

**A1**: We agree with the reviewer that accurately estimating the opponent modeling error ($\epsilon$ in our paper) sometimes can be challenging for any algorithms which rely on opponent modeling. Instead of calculating $\epsilon$, we can use online learning algorithms (e.g., UCB1, Exp3) to choose among a fixed set of alphas based on online evaluation performances. This technique is also introduced in previous exploitation papers [Johanson et al' 07, Ganzfried et al' 15].

**Q2**: "How did the author produce opponents with different strengths in the experiment?"

**A2**: Following [Brown et al '18], we enumerate every infoset in the blueprint strategy and shift the action distribution randomly with probability $Pr_{shuffle}$ = 0.3 or 0.7. In shifting, we multiply the probability of each action by a random value sampled from $Uniform(0, 1)$, and then re-normalize the probability distribution.



[Johanson et al '07] Michael Johanson, Michael Bowling, and Martin Zinkevich. Computing robust counter-strategies. 2007.

[Ganzfried et al '15] Sam Ganzfried and Tuomas Sandholm. Safe opponent exploitation. *ACM Transactions on Economics and Computation (TEAC)*, 3(2):1–28, 2015a.

[Brown et al '18] Brown N, Sandholm T, Amos B. Depth-limited solving for imperfect-information games[J]. arXiv preprint arXiv:1805.08195

From: ICLR.cc/2022/Conference/Paper1309/Reviewer_Wp7Z
summary_of_the_paper: This paper proposes a method of subgame resolving in zero-sum extensive form games, blending in elements of theoretical safety, as well as more classical approaches based on opponent modeling. The method is fairly straightforward, utilizing a modified objective balancing between the safety objective (used in Moravcik et al) and a best-response like objective use in opponent modeling. To facilitate compatibility with modern solvers such as CFR, the authors propose gadgets transforming the refinement step into another zero-sum extensive form game.
main_review: This paper’s motivation is easy to follow and does a good job of introducing subgame search and opponent exploitation. I have not found any glaring errors in derivations or proofs. In terms of significance, this paper answers the natural question of how opponent weaknesses can be exploited without overly sacrificing safety. Experiments are sufficient.

One area this paper could do a lot better in terms of clarity. For example, the definition of CBR at the end of section 3.1 could be much more precise---the argument of CBR need only depend on player -p, and \sigma_p in the definition should really be CBR_p(\sigma_{-p}). In other areas, the authors could be more precise, e.g., in section 4.1, “under the assumption that the reach probabilities *remain the same*”, could be changed to “is \hat{p}(...)”. This is to disambiguate between \tilde{p} and \hat{p}. The gadget game presented seems correct to me, and Figure 1 was very useful in illustrating it. However, for readers unfamiliar with gadgets, the textual representation could be insufficient. For example, P2’s infosets should stretch over both the left and the right branches (which is briefly alluded to in point 1).

Theorem 1 provides a natural extension of the bounds of Moravcik et. al. One concern I have is how useful these bounds are in practice, in particular for selecting an appropriate value of alpha. In larger games (e.g., FHP), it is very likely that some information sets are never reached in practice even over many rounds of play. Hence, tau is not known, or may not even be estimated using data (this admittedly is a problem for any type of opponent modeling). The authors’ claim at the end of section 5: “In case of a bad estimation, we can always choose smaller \alpha to ensure safety”. In practice, how do we know we have a bad estimation if there are no samples to compare with?

The authors present experimental results with varying epsilon (estimation error of opponent strategy). What was the procedure in which this error was introduced?


summary_of_the_review: I vote for this paper to be accepted. Apart from a lack of polish, there are no glaring errors in the paper, and the key ideas in the paper are easy to follow and appreciate. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

From: ICLR.cc/2022/Conference/Paper1309/Reviewer_hmTT
summary_of_the_paper: The paper proposes a technique to smoothly interpolate between computing a Nash equilibrium strategy and computing a best response. The technique is presented in the form of search for an extensive-form imperfect information game. Experiments on two poker variants are presented showing that the technique can indeed interpolate between the Nash equilibrium strategy ("safety") and the best response ("exploitation").
main_review: To my understanding, the technique presented by the authors to interpolate between a Nash equilibrium strategy and a best response (say, for Player 1) simply boils down to the following:
- introduce a parameter $\alpha$ and optimize the objective $f(x) = (1-\alpha) \cdot \min_{y\in Y} \\{x^\top U y\\} + \alpha\cdot x^\top U \bar{y}$, where $x$ is a strategy of Player 1, $Y$ is the set of strategies of Player 2, and $\bar{y}$ is a "predicted" strategy for Player 2;
- solve the above optimization point (that is, solve $\max_{x \in X} f(x)$ where $X$ is the set of strategies of Player 1.
The presence of search seems just an implementation detail that obscures the above, straightforward idea.

If my understanding were to be confirmed correct, I find the paper very light theory-wise, and I definitely think that the idea should be stated for what it is instead of interspersing it with the search formalism, which ends up obscuring the idea and making it looks significantly more complex than it is.

Again assuming I didn't miss anything about the fundamental underlying idea of the paper, the technical results (Theorems 1 & 2) are not surprising at all, and just amount to bounding the two terms in the definition of $f(x)$ independently.

Given the limited technical contribution I would be significantly more satisfied with the paper if it had shown more compelling/insightful experimental evidence of the benefit of the approach. However, the paper only experiments on one domain (poker), and the experiments do not seem to show any particular interesting insight. For example, the experiments find that if the prediction $\bar{y}$ is accurate, then for large $\alpha$ the maximum of $f$ will be a best response to the strategy of the opponent, and if the prediction is not accurate, then one would be better off by picking a lower $\alpha$. I find that completely expected, but perhaps I missed some more subtle point?
summary_of_the_review: My biggest concern with the paper is about the significance. I believe that the idea behind the paper is simply optimizing a convex combination of functions, and that the search formalism that was imposed on top is mostly an implementation detail from a theory point of view. I also find the experiments unsurprising and not particularly insightful. Other than that, I do not believe the paper to be technically flawed or problematic in its claims. However, I find the significance concern is serious enough to prevent me from recommending acceptance. I look forward to a robust discussion with the authors on that point.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

From: ICLR.cc/2022/Conference/Paper1309/Reviewer_SWZo
summary_of_the_paper: The submission proposes performing safe opponent exploitation via decision-time planning using a novel gadget game. At the top of the gadget game, there is a chance node that goes to the maxmargin gadget game with probability (1 - alpha) and goes to the estimate posterior over private information with probability alpha. The submission proves that, for small enough alpha and posterior error, this gadget game produces a safe (non-exploitable) strategy. The submission presents experiments on Leduc Hold'em and Flop Hold'em Poker.
main_review: The main idea of the submission is somewhat obvious. However, I think it nevertheless offers value to the community, as opponent exploration is both an important and an under-explored research area (at least in my opinion). There are some small problems with the submission (see below) that I would like to see addressed. More generally, I like to see the authors put into the quality of the writing. As it currently stands, much of it is sloppy as does not read like a publication-ready conference paper.

### Exploitability Theory
I think the exploitability results are a bit confusing as currently written. The term "exploitability" is not generally used in a way that depends on the opponent's "true distribution". Exploitability defines a worst case loss. Unless I am misunderstanding something, the submission is talking about a worst case loss, subject to a constraint on the private information. I am not sure it is appropriate to refer to this quantity as exploitability --- SES is not, in fact, safe against arbitrary opponents.

The submission provides a nice example illustrating why its approach is superior to simply mixing between a safe strategy and an exploitation strategy. That being said, it seems to me to be perfectly possible to provide theorems analogous to Theorem 1 and Theorem 2 for the "mixing between a safe strategy and an exploitation strategy" approach. I think it would add value to the submission to work through these results, as it may be interesting to compare them to Theorem 1 and Theorem 2.


### Baselines
The submission dismisses Johanson 2007 and Ganzfried 2015 as baselines. However, I am not sure that this dismissal is well justified. The method in the submission already uses an estimate of the opponent strategy. It would be relatively to substitute this estimate for the "known" component of the strategy required by restricted Nash response (RNR) and test the exploitability. It may be that RNR performs perfectly well in this case. Also, while I do not dispute it is difficult to scale LP solvers to large games, the games considered in the submission are not that large. Therefore, it seems like both Johanson and Ganzfried offer reasonable baselines for the algorithm proposed in the submission.

### Small Problems

Paper cites benchmark Hanabi paper for superhuman Hanabi performance?

Paper cites Tian for superhuman bridge performance?

“Monte Carlo tree search (MCTS) can be viewed as a powerful policy improvement operator.” 
This sentence appears to be (hopefully accidentally) plagiarized from the AlphaGo Zero paper: “MCTS may therefore be viewed as a powerful policy improvement operator”.

Poorly constructed “They both have sounded monotonic improvement guarantees in theory, and show tremendous performance promotion empirically.”

“In fully-cooperative imperfect information games, search algorithms proposed in Tian et al. (2020b) and Lerer et al. (2020) are also proved to never be detrimental to the current policy”
This is only half true. Lerer et al.’s “never detrimental” is only asymptotic. After any finite amount of time (ie in all practical cases) there is no such guarantee.

“However, it may be overly conservative confronted with opponents with limited rationality and fail to take advantages of their weaknesses to obtain higher rewards.”
It has nothing to reference (should be they) 
Advantages is wrong grammatical number

“We construct a new gadget game to optimize this objective, which enable”

“Search techniques are proven to be essential in developing strong AI strategies”
This claim appears overly strong.

“Libratus (Brown et al., 2018) is the first superhuman poker AI which is considered the milestone in the research of imperfect information games”
This sentence is both grammatically incorrect and unnecessarily contentious. Does it add value to the paper to make a claim about which poker AI algorithm (between Libratus and DeepStack) was the first and which is considered the “milestone”? Also, is it appropriate to not even cite DeepStack here?

“search algorithms for subgame refinement have also shown promise in improving joint strategies in cooperative imperfect information games such as Hanabi (Bard et al., 2020)”
The paper being cited here does not involve sublime refinement.

“In that paper, Johanson et al. (2007) assumes that the”
assume

“But Ganzfried & Sandholm (2015a) relies on LP solver”
rely

“It is worthwhile mentioning that there also have been extensive research”
has

“However, this paper only only focuses”
fix

Notation Section:
It’s bad form to start sentences with symbols.

It’s pretty unusual for EFGs to use \sqsubseteq the way the submission defines it.
Having read further it appears that the submission uses \sqsubseteq the way it is normally used for EFGs. So the definition is incorrect.

In the matrix game shown in Table 1, there are two NEs.
Any interpolation of the two NE the submission gives is also a NE.

Throughout the paper Safe Subgame Exploitation is improperly capitalized.
summary_of_the_review: I would like the authors to address the comments above. Should they do so, I am willing to reconsider my score.

---

After discussions with the authors, I raised my score from a 5 to an 8.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

From: ICLR.cc/2022/Conference/Paper1309/Reviewer_f5FT
summary_of_the_paper: The paper introduces Safe Exploitation Search (SES), which achieves a balance between maxmargin solving (guaranteed to be safe, but non-exploitative) and unsafe subgame solving (unsafe, as the name suggests, but exploitative of the opponent's initial distribution at S_top) by mixing their objective functions. The paper provides both theoretical and experimental evidence that the method does a good job of balancing the two objectives. 
main_review: Strengths:

1. The paper is clean and easy to read.

2. Both the theoretical results and the experiments clearly support the main claims of the paper.

Weaknesses: 

1. The main contribution of the paper is only a minimal step on top of known ideas (a convex combination of unsafe and maxmargin subgame solving). The theoretical and experimental results are not at all surprising either.

2. Related to above: I'm not completely satisfied with the comparison to the past safe exploitation works [Johanson et al '07, Ganzfried & Sandholm '15]. As far as I can tell, the present paper simply uses a different parameterization of the Pareto frontier of exploitation vs exploitability--it uses the mixing amount $\alpha$; the others use the exploitability, formulating the objective as "maximize exploitation such that exploitability is at most $\varepsilon$". These are not very different. Further, LP solving in games is not that slow, especially with recent advances [Zhang & Sandholm '20]; in particular, the subgames in experiments seem tractable with LP.

[Zhang & Sandholm '20] Sparsified Linear Programming for Zero-Sum Equilibrium Finding
summary_of_the_review: Well-written, but in my view not very novel/just incremental work over past work on subgame solving. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.