Paper: On Local Equilibrium in Non-Concave Games (ICLR-2024)

Reviewer comment (Theory):
One has to be fair and recognize the novelty of the paper and the absence of pre-existing criteria for its assessment; nevertheless, it would be rational to ask for some justification of the proposed equilibrium notion other than computational complexity arguments. In a sense, what are real-world examples where the proposed notions of equilibria are already established as desirable states of a game?

Author response:
We appreciate your recognition of the novelty of our work and your interest in the practical relevance of our proposed equilibrium notion. We will use the dynamics of pricing in a competitive market as an example to illustrate the applicability of our $\Phi_{proj}(\delta)$-local correlated equilibrium concept in a real-world scenario. 

The market has multiple agents selling identical goods. Each agent sets a price $p^i_t$ for their goods at day $t >=1$. The reward everyday depends on the agent’s price: a high price decreases the demand of the goods but increases revenue per unit; while a low price could increase demand but decreases revenue per unit. In a complex market environment, the utility function in terms of $p_t^i$ could be non-concave so that even computing the optimal price given the knowledge of the utility functions is intractable. If each player adjusts their price using the online gradient descent algorithm, then our results show that their pricing strategy converges to a $\Phi_{proj}(\delta)$-local correlated equilibrium. In such equilibrium, each player would not regret to be more conservative (i.e., posting a slightly lower price $p_t- \delta$ everyday) or to be more aggressive (i.e., posting a slightly higher price $p_t + \delta$ everyday). This scenario demonstrates the equilibrium's relevance: in a market where agents have bounded rationality and make decisions based on localized price adjustments, our proposed equilibrium notion provides a stability guarantee. More generally, the proposed solution concepts may have practical implications, especially in multi-agent systems where agents encounter complex, non-concave utility functions and operate under the constraints of bounded rationality.

Reviewer comment (Theory):
What kinds of $\Phi(\delta)$ families would the authors consider as important for future study and of game-theoretic importance?

Author response:
A: This is a great question!  One family of $\Phi(\delta)$ that is important is a generalization of $\Phi_{int}(\delta)$. Recall that $\Phi_{int}(\delta)$ contains strategy modifications that interpolates between current strategy $x$ and a fixed strategy $x^*$. One can consider a strategy modification that interpolates between current strategy $x$ and a modified strategy $\phi(x)$ for $\phi \in \Phi’$ where $\Phi'$ is a set of (possibly not local) strategy modifications. $\Phi_{int}(\delta)$ then becomes a special case where $\phi(x) = x^*$ is a constant strategy modification. By a similar proof to that in section 4, it is easy to show that regret minimization under the new set reduces to $\Phi’$-regret minimization. We think obtaining a more fine-grained understanding of the complexity of $\Phi’$-regret minimization for more general classes of strategy modifications $\Phi’$ is an interesting future direction. For instance, is it possible to define a meaningful complexity measure of a set $\Phi’$ that characterizes the computational / oracle complexity of $\Phi’$-regret minimization.

Reviewer comment (Theory):
What is the connection of $\Phi(\delta)$-regret minimization and bounded rationality? Putting the computational theoretic aspects aside, we in a sense assume agents to be as rational as their first-order derivative dictates. Would assuming bounded rationality for the agents lead to tractable notions of equilibria as well?

Author response:
**A**: This is a great question! We believe $\Phi(\delta)$-regret minimization is a model of  bounded rationality. In this context, we refer to bounded rationality as the idea that agents have limitations in their decision-making processes, particularly in terms of the information they consider and the computational resources they have.

In $\Phi(\delta)$-regret minimization, the set $\Phi(\delta)$ only contains local deviations that are bounded by $\delta$. Thus agents are not fully rational as they ignore global deviations that might be more beneficial. Moreover, as you pointed out, assuming agents to be as rational as their first-order derivative dictates is a choice of bounded rationality. Furthermore, the cardinality and complexity of the set of $\Phi(\delta)$ reflect a degree of an agent’s rationality: A more rational agent, within the bounds of their capabilities, might consider a larger and more complex set of potential deviations compared to a less rational agent. Our results demonstrate that two sets of $\Phi(\delta)$ lead to tractable notions of local correlated equilibrium. This suggests that by assuming bounded rationality, agents can achieve stable outcomes even in complex non-concave environments. 

Exploring other models of bounded rationality, different equilibrium concepts, and learning dynamics that converge to these equilibria are very interesting future directions.

Reviewer comment (Theory):
What would qualitatively change if we assumed access to second-order information?

Author response:
**A**: We remark that by assuming access to second-order information, [DGSZ23] developed a second-order algorithm that converges to local Nash equilibrium although without concrete convergence rates. This result improves over first-order methods which usually get stuck in limit cycles and shows the power of second-order methods. If we assume access to second-order information, it is possible that we can develop algorithms with faster convergence to local correlated equilibrium than first-order algorithms. This is a very interesting future direction. 

[DGSZ23] Daskalakis, C., Golowich, N., Skoulakis, S., & Zampetakis, E. STay-ON-the-Ridge: Guaranteed Convergence to Local Minimax Equilibrium in Nonconvex-Nonconcave Games. COLT, 2023

Reviewer comment (Theory):
Part 1) of assumption 1 might be redundant given part 3, and part 2) might be redundant given part 1 and the compactness of strategy sets.

Author response:
Thank you for your thoughtful observations regarding the structure of Assumption 1.

Regarding your first point, we agree that Part 1 of Assumption 1 is implied by Part 3. Part 1 should be the definition of differentiable games while smooth games are differentiable games with additional smoothness assumptions. For clarity, we have removed Part 1 and added the definition of differentiable games in the paragraph above. 

Concerning Part 2, your insight is correct that it could be inferred from the combination of the former Part 1 and the compactness of strategy sets. Nevertheless, we have chosen to retain this part for explicitness, particularly to specify the Lipschitz constant $G$. This explicit specification is useful for some of our subsequent analyses and results. We have rephrased it in the revised version of the paper to clarify its role.

These changes, along with the revised introduction of differentiable and smooth games, are highlighted in the updated version of our paper.

Reviewer comment (Theory):
The local Nash definition studied in the paper considers only *pure* strategies, while local correlated equilibrium is studied in correlated **mixed** strategies. This raises the question of whether mixed local Nash equilibria can be efficiently computed, or if that is also out of reach. It appears to be out of reach since randomization would reduce the problem to a multilinear game (potentially infinite dimensional) for which computation of Nash is PPAD. A description of this point is important to understand the jump from pure to mixed strategies.

Author response:
Thank you for your insightful question on the complexity of mixed local Nash equilibria, which indeed warrants further discussion in our paper.

To address your question: Computing mixed local Nash equilibria is PPAD-hard for general-sum normal-form games, which are concave games. Since the agents have multi-linear utility functions, every local mixed Nash equilibrium is also a Nash equilibrium. Computing a Nash equilibrium in general-sum normal-form games is known to be PPAD-hard [DGP09, CDT09], so computing a mixed local Nash equilibrium is also PPAD-hard. [DSZ21] indeed considers pure local Nash equilibria, but they show that this is PPAD-hard even in two-player zero-sum games with non-concave utility functions. 

We will expand upon these points in the revised version of our paper, providing a clearer bridge between the concepts of pure and mixed strategies and their respective computational implications.

Reviewer comment (Theory):
It is unclear whether Lemma 1 assumes Lipschitz smoothness/continuity on the convex regrets.

Author response:
Thanks for this question.  Lemma 1 holds when there exists an online learning algorithm that has $\Phi$-regret against convex loss functions. In the context of lemma 1, we do not need other assumptions. $G$-Lipschitzness is required to achieve sublinear regret for specific algorithms like GD / OG, . But Lemma 1 still holds if there exists a no-$\Phi$-regret algorithm that does not require such assumptions.

Reviewer comment (Theory):
Clarification is needed on how part 2 of Lemma 1 relates to Hazan et al.’s [2] results and, in general, how the authors’ results relate to those on projected $\phi$ regret.

Author response:
Firstly, it's important to clarify that Part 2 of Lemma 1 and the results by Hazan et al. [HSZ17] study very different notions of regret (See examples below for an illustration), leading to distinct local equilibrium concepts. As such, they are not directly comparable. Moreover, we've included a detailed discussion in the related works section of Appendix A. The discussion is as follows.

#### The work most closely related to ours is [HSZ17]. The authors propose a notion of *$w$-smoothed local regret* against non-convex losses, and they also define a local equilibrium concept for non-concave games. They use the idea of *smoothing* to average the loss functions in the previous $w$ iterations and design algorithms with optimal $w$-smoothed local regret. The concept of regret they introduce suggests a  local equilibrium concept. However, their local equilibrium concept is very non-standard in that its local stability is not with respect to a distribution over strategy profiles sampled by this equilibrium concept. Moreover, the path to attaining this local equilibrium through decentralized learning dynamics remains unclear. The algorithms provided in [HSZ17] require that every agent $i$ experiences (over several rounds) the average utility function of the previous $w$ iterates, denoted as $F^t_{i,w}:=\frac{1}{w} \sum_{\ell=0}^{w-1} u_i^{t-\ell}(\cdot,x_{-i}^{t-\ell})$. Implementing this imposes a significant coordination burden on the agents. In contrast, we focus on a natural concept of local correlated equilibrium, which is  incomparable  to that of [HSZ17], and we also show that efficient convergence to this concept is achieved via decentralized gradient-based learning dynamics.''

The following two examples shows that the $w$-smoothed local regret and $\Phi_{proj}(\delta)$ (we will refer to it as $\delta$-projected regret) are incomparable. A sequence of actions may suffer linear $w$-smoothed local regret but constant $\delta$-projected regret (Example 1), and vise versa (Example 2).  
**Example 1**:
1. Loss sequence: $f^t(x) = |x|$ for every $t \in [T]$ over $X = [-1,1]$.
2. Action sequence: $x^t = 1/2$ for odd $t$; $x^t = -1/2$ for even $t$
3. Regret: For any $\eta < 1/2$ and $w \ge 1$, the $w$-local regret is at least $\Omega(T / w)$; For $\delta > 0$,  the $\delta$-projected regret of $\{x^t\}$ is at most $\delta$.

**Example 2**: 
1. Loss sequence: $f^t(x) = -x^2$ for every $t \in [T]$ over $X = [-1,1]$.
2. Action sequence: $x^t = 0$ for all $t \in [T]$
3. Regret: For any $\eta > 0$ and $w \ge 1$,  the $w$-local regret is $0$; For any $\delta < 1$,  the $\delta$-projected regret of $\{x^t\}$ is at least $\delta^2 T.$

Reviewer comment (Theory):
The naive regret bound in section 3.1 seems meaninglessly loose; having an additive Lipschitz continuity constant G suggests that the algorithm might make no progress at all.

Author response:
The regret bound of naive $\Phi$ regret minimization is $O(\delta G\sqrt{T \log|\Phi^\gamma|} + \gamma G T)$. By setting $\gamma = 1/T" we can get a regret bound that is sublinear in $T$. However, as we mentioned in the paper, this approach leads to a regret bound that depends on $ \log|\Phi^\gamma|$ while the per-iteration complexity depends on $|\Phi^\gamma|$, which is prohibitively high.

Reviewer comment (Theory):
Reg_proj does not have a learning rate in the step it takes; this seems to affect the notion that projected and external regret can, in general, be unrelated.

Author response:
Reg_proj (we will refer to it as projected regret) is parameterized by $\delta$, which is the largest $\ell_2$ norm of a possible deviation. As we demonstrated in the paper (example 1 and 2), projected regret is indeed incomparable with external regret in the sense that (1) a sequence of actions could have linear external regret but 0 projected regret; (2) a sequence of actions could have linear projected regret but 0 external regret. Despite the difference, we show that the online gradient descent algorithm achieves sublinear regret bounds for both notions of regret.