Paper: Steering No-Regret Learners to Optimal Equilibria (ICLR-2024)

Reviewer comment (Theory):
p. 1: Maybe clarify what is meant by bounded regret. Is it constant regret per-iteration? Vanishing regret?

Author response:
We mean that the regret should be bounded by a sublinear function of $T$, as is the usual assumption. We've changed it to "sublinear regret".

Reviewer comment (Theory):
For the case of pure Nash equilibria, it is not clear to me that the problem is hard to begin with. The paper provides a lower bound where a simple game along with a specific type of very unnatural no-regret behavior can be hard to steer. Following results from e.g. [2-4], pure/strict Nash (e.g. like in Stag Hunt games explored in the paper) are locally asymptotically stable for effectively all no-regret algorithms and in fact arguably for all reasonable game dynamics. So here is how to informally stabilize all of them with a finite amount of money. Pad the target strategies with enough money at an initial phase of the algorithm. With a finite excess payoff any reasonable aggregate payoff based dynamics (not even no-regret) after some finite amount of investment will be in the attractor. This is a stronger convergence result since it is actually even guarantees last-iterate convergence (and it does so with a finite amount of money).

Author response:
*So here is how to informally stabilize all of them with a finite amount of money. Pad the target strategies with enough money at an initial phase of the algorithm. With a finite excess payoff any reasonable aggregate payoff based dynamics (not even no-regret) after some finite amount of investment will be in the attractor. This is a stronger convergence result since it is actually even guarantees last-iterate convergence (and it does so with a finite amount of money).*

First of all, we do not believe that there is a definite answer as to what constitutes "reasonable dynamics." For example, the papers cited by the reviewer do not (to the best of our understanding) capture regret matching, one of the most fundamental and natural learning dynamics. Making stronger assumptions regarding the class of update rules followed by the players would limit the scope of our results, and so instead we chose to treat the problem under the much broader condition of incuring sublinear regret. As we point out in the last paragraph of our related work section, there is a recent literature in mechanism design that operates under this exact assumption.

We continue our response below.

Reviewer comment (Novelty):
Why should we care enough for totally non-meaningful no-regret behaviors that do not capture any realistic algorithm either from an optimization or a behavioral point of view to be willing to accept paying an infinite amount of money when a finite amount should suffice for all reasonable dynamics?

Author response:
*Why should we care enough for totally non-meaningful no-regret behaviors that do not capture any realistic algorithm either from an optimization or a behavioral point of view to be willing to accept paying an infinite amount of money when finite amount should suffice for all reasonable dynamics?*

First of all, we do not believe that there is a definite answer as to what constitutes "reasonable dynamics." For example, the papers cited by the reviewer do not (to the best of our understanding) capture regret matching, one of the most fundamental and natural learning dynamics. Making stronger assumptions regarding the class of update rules followed by the players would limit the scope of our results, and so instead we chose to treat the problem under the much broader condition of incuring sublinear regret. As we point out in the last paragraph of our related work section, there is a recent literature in mechanism design that operates under this exact assumption.

We continue our response below.

Reviewer comment (Theory):
You also have an experiment with Exp3 where clearly a finite amount of money suffices to select the optimal equilibrium. Can you create a negative example where you would need unbounded money for Exp3? If not, then doesn't this show that the \(T^{\alpha}\) unbounded payoffs are too loose in comparison to real world lower bounds?

Author response:
*You also have an experiment with Exp3 where clearly a finite amount of money suffices to select the optimal equilibrium. Can you create a negative example where you would need unbounded money for Exp3? If not, then doesn't this show that the $T^\alpha$  unbounded payoffs are too loose in comparison to real world lower bounds?*

As we explained above, the answer depends on how we set the learning rate. If we use the usual vanishing learning rate, in general it shouldn't be possible to steer with a finite budget.

Reviewer comment (Writing):
I find the paper to be inaccessible and lack precision at times (for instance, a central notion to the paper such as optimal equilibrium is not well-defined—is it one that is Pareto-optimal or that maximizes some notion of welfare?).

Author response:
*A central notion to the paper such as optimal equilibrium is not well-defined—is it one that is Pareto-optimal or that maximizes some notion of welfare?*

By optimal we mean an equilibrium that, among the set of all equilibria, maximizes the objective of the mediator, as we already introduce in Definition 6.1 of the paper. This objective could be, for example, the social welfare or the revenue of a mechanism, but our results apply for any mediator objective function. Note that Sections 4 and 5 concern steering to a pure Nash equilibrium (without any underlying notion of optimality), which is why our notion of optimality is formally introduced later in Section 6 where we apply our earlier results to steer to optimal equilibria.

Reviewer comment (Experiments):
The experiments seem also unrelated to the theoretical results (Pure Nash vs. Correlated).

Author response:
*The experiments seem also unrelated to the theoretical results (Pure Nash vs. Correlated).*

There seems to be a misunderstanding here. As we explained above, our results also directly apply to correlated equilibria (see Section 6 of the paper). As a result, our experiments are directly under the umbrella of our theoretical results.

Reviewer comment (Theory):
The definition of the steering problem in terms of pure Nash equilibria is also odd, since pure Nash equilibria are not guaranteed to exist.

Author response:
*The definition of the steering problem in terms of pure Nash equilibria is also odd, since pure Nash equilibria are not guaranteed to exist.*

As we formalize in Section 6, our results directly apply to mixed-strategy Nash equilibria as well as correlated and communication equilibria. While pure-stratgy Nash equilibria are not guaranteed to exist in general, as the reivewer correctly points out, our results apply to any extensive-form game through the notion of a mediator-augmented game (Definition 6.1).

Reviewer comment (Writing):
Page 2: full feedback is not defined.

Author response:
*Page 2: full feedback is not defined*

The full feedback setting is introduced in the beginning of Section 5, and it means that the mediator can observe the players' strategies. We have added a reference to Section 5 in the revised version to help the reader.

Reviewer comment (Writing):
What is the meaning of the directness gap? I cannot read the math as the description given, an explanation would be appreciated.

Author response:
*What is the meaning of the directness gap?*

The directness gap quantifies how close players' strategies are from the direct (i.e., obedient) strategies---as prescribed by the target equilibrium.

Reviewer comment (Theory):
The steering problem is defined pure strategy Nash equilibria, when are pure strategy Nash equilibria guaranteed to exist in extensive form and normal-form games? How much of the theory provided by the authors apply to mixed Nash equilibria?

Author response:
*The steering problem is defined pure strategy Nash equilibria, when are pure strategy Nash equilibria guaranteed to exist in extensive form and normal-form games? How much of the theory provided by the authors apply to mixed Nash equilibria?*

As we formalize in Section 6, our results directly apply to mixed-strategy Nash equilibria as well as correlated and communication equilibria. While pure-stratgy Nash equilibria are not guaranteed to exist in general, as the reivewer correctly points out, our results apply to any extensive-form game through the notion of a mediator-augmented game (Definition 6.1).