Paper: A Black-box Approach for Non-stationary Multi-agent Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
Lack of discussions on the definitions, assumptions, and results. For example, from Definition 4, if the non-stationarity degree of Markov game is zero, should this setting degenerates to the traditional stationary setting? It seems very straightforward, but I prefer to get confirmed from the paper.

Author response:
**Degenerates to the traditional stationary setting?** Yes, it degenerates to the traditional stationary setting when the non-stationarity degree of Markov game is zero. We have incorporated this point to the revised version.

Reviewer comment (Writing):
The introduction section contains many terms that have not been clearly defined, such as non-stationarity budget and black-box approach. I would prefer to see a revision on the introduction section including providing a high-level introduction to these concepts and giving some references on real-world scenarios where we really need this algorithm.

Author response:
**Introduction:** We add real-world multi-agent scenarios where non-stationarity is important in the revised version with references. We added an explicit definition of black-box algorithm. We also highlight the sentence defining nonstationarity budget in the Introduction. If there is any concept you think should be clarified, please tell us and we are happy to add accordingly.

Reviewer comment (Theory):
If the non-stationarity degree of Markov game is zero, should this setting degenerates to the traditional stationary setting?

Author response:
**Degenerates to the traditional stationary setting?** Yes, it degenerates to the traditional stationary setting when the non-stationarity degree of Markov game is zero. We have incorporated this point to the revised version.

Reviewer comment (Theory):
When $\Delta=0$, will the regret bounds (Corollary 1) degenerates to the regret bounds in the stationary case? Will it match the existing rate?

Author response:
Yes, the bound degenerates to the bound in the stationary case. However, it does not match the existing rate. When the environment is stationary, the algorithm is the vanilla Explore-then-Commit algorithm, which is suboptimal. We have incorporated this point to the revised version.

Reviewer comment (Theory):
I am not familiar with Assumption 1 and Assumption 2. Are they also common to be made in stationary MAML works?

Author response:
**Assumption 1, 2:** In stationary settings, i.e. $\Delta=0$, Assumption 1, 3 degenerate to the ordinary sample complexity assumptions, which is common in machine learning. Assumption 2 is not often made in literature, but we construct Protocol 1 (Algorithm 2 in the revised version) to convert an algorithm satisfying Assumption 3 to an algorithm satisfying Assumption 2.

Reviewer comment (Theory):
Algorithm 2, as outlined in the paper, necessitates a test EQ algorithm to detect nonstationarity in the case of an unknown budget. The assumption of the existence of a testing EQ trained offline appears to be a strong assumption in the context of online learning problems.

Author response:
- **TEST_EQ:** The regret from TEST_EQ is counted into the total regret. **The whole algorithm is run online and no offline training is needed.** In particular, this part of regret is calculated at the very beginning of the proof for Lemma 8 in the Appendix.

Reviewer comment (Theory):
The tightness of regret is not clear. For example, it is not evident whether the results achieved in this paper surpass the Bandit over Bandit approach with a sliding window multi-agent RL algorithm.

Author response:
- **Tightness:** We did consider constructing a lower bound for non-stationary games, but that seems too difficult. While the lower bound for non-stationary bandits are known for a long time, there is no non-trivial regret lower bounds for stationary normal form games with bandit feedback to the best of our knowledge. Hence it is too optimistic to hope for a regret lower bound for our setting directly.

Reviewer comment (Writing):
It would enhance clarity to provide an explanation for the definition of '(\Delta)-EQ' in Assumptions.

Author response:
- **$\epsilon$-EQ:** The definitions are in Definition 1,2,3 and the subsequent paragraph.

Reviewer comment (Theory):
It seems that the regret from training the test EQ is not factored into the results presented in Theorems 1 and 2. What implications might there be if the training cost is considered in the regret?

Author response:
- **TEST_EQ:** The regret from TEST_EQ is counted into the total regret. **The whole algorithm is run online and no offline training is needed.** In particular, this part of regret is calculated at the very beginning of the proof for Lemma 8 in the Appendix.

Reviewer comment (Theory):
Despite the authors discussing the inapplicability of the Bandit over RL approach to this problem, what if we were to consider a bandit over (sliding window) MARL approach for addressing nonstationary MARL? Due to the order of the regret in Theorems 1 and 2, it remains unclear whether the suggested method is superior to the bandit over bandit approach.

Author response:
- **Bandit over MARL:** **We discussed the challenges of bandit over MARL in the last paragraph of Section 3 and a more detailed explanation is provided in Appendix B.2.** A quick explanation is as follows. In games different actions are competing with different best response actions whereas in bandits all arms are competing with the single best arm. In games, comparing the value of two actions does not tell us which is better because the gap can be calculated toward different best responses. The high-level idea of Bandit over Bandit is to divide the whole time horizon into several segments, run an algorithm with known non-stationary budget in each of them and run an adversarial bandit algorithm on the segments. A crucial part is to use the sum of rewards in a segement as the reward for the adversarial bandit algorithm. The problem in the multi-agent setting is that the sum of rewards is meaningless for evaluating the performance as explained before. One may refer to the Appendix for a more formal explanation with formulae.  
Thank you very much for your reply and carefully examining the details. To explain the Bandit over MARL approach in detail, we further expand the notation. Let $\pi_t$ be the policy that the agent take by using the Bandit over MARL algorithm where $t\in[T]$. Let $\pi_t^*$ be the policy that the agent would take by committing to the best parameter $w^*$ in a segment, $t\in[H]$. The first term in the expansion of regret can be written as $$\sum_{t=1}^TV_i^M(\dagger, \pi_{t,-i})-\sum_{h=1}^H\sum_{t=1}^HV_i^M(\pi^*_t).$$ Since $\pi^*_t$ is competing with its best response, which may not equal the best response of $\pi_t$, this term cannot be bounded by regret guarantee of the base algorithm. In contrast, in the bandit setting, no matter what the policy is, it is always comparing with $a_t^*$, the optimal arm at timestep $t$. Hence in that scenario the corresponding term can be bounded with the regret of the base algorithm.

Reviewer comment (Theory):
The claimed regret bounds in Table 1 seem somewhat coarse given that the proposed notion of "total variation non-stationary budget" $\Delta$ scales linearly in $H \times S \times A$. Taking that into account, all of the regret bounds for multi-round games displayed in Table 1 look like they should scale linearly or superlinearly in horizon length and linearly in state size for general-sum Markov games and the square-root of state-size for zero-sum games. It seems one could get a similar regret bound (maybe even of order $O(\sqrt{T})$) by just flattening each game into a single round game with $H \times S \times A$ actions and having each player run Exp3 on those actions? The results in Table 1 could still be really interesting, but it’s difficult to put them into perspective because their main feature is $\Delta$, which the authors seem to be proposing the use of. It would be helpful if the author could clarify if $\Delta$ is already a commonly used quantity that is known to be small in practice.

Author response:
- **Total Variation:** We apologize for our typos in the paper. Following prior works, the nonstationarity takes the max over states and actions instead of taking sum. Similar notations are common in literature. For instance, the measure Wei & Luo (2021) use for single-agent RL is $$\Delta_{[t,t+1]}=H\sum_{h=1}^H\max_{s,a}\left|r_h^t(s,a)-r_h^{t+1}(s,a)\right|+H^2\sum_{h=1}^H\max_{s,a}\left\|r_h^t(s,a)-r_h^{t+1}(s,a)\right\|$$ which has an $O(H^2)$ difference in order from our measure. 
- **Flattening the Game**: Could you be more specific on how to flatten the game? If you mean treating each single player pure strategy as a bandit arm, the number would be exponential. At each timestep, the number of different pure-strategy policies is $A^S$ since we may choose $A$ different actions for each state. **The number of different combinations of single-step policies is hence $A^{SH}$, which is exponential**. Besides, each player running EXP.3 can achieve $O\left(\sqrt{T}\right)$ individual regret, but **there is no NE/CE/CCE-regret guarantee in the stationary game setting to the best of our knowledge**. Hence it is open to achieve $O\left(\sqrt{T}\right)$ regret in the harder non-stationary case using EXP.3.

Reviewer comment (Novelty):
I'm not familiar with restarting-based algorithms so I find it harder to evaluate the novelty/significance of the results in 5.2 and 5.3. However, the contributions listed in 5.1, namely the reduction of equilibria testing to single agent learning, are—to the best of my knowledge—already known (in the case of CCE testing) or very direct in the sense that it's a one-two sentence reduction and the first thing one would try (in the case of CE).

Author response:
- **Novelty**: In the non-stationary bandit literature, testing algorithms are usually used. While constructing TEST_EQ is relatively straightforward, we think it is important to state it rigorously and prove the correctness. This lays the foundation for future work that need testing algorithms in non-stationary games. This construction is a side contribution of this paper while the main technical contributions lie in comprehensively analyzing the difficulties incurred by the game setting (Section 3) and designing the (parameter-free) learning algorithm with no-regret guarantee (Section 4,5). In particular, designing such algorithms require many detailed while non-trivial adaptations. We raise two examples here. First, the test scheduling in Section 5.2 is tailored for Assumption 2. The scheduling we use assures that with high probability TEST_EQ terminates after running for desired number of timesteps, otherwise TEST_EQ may not generate a meaningful output. In prior works, when overlapping of different tests happen, their lengths get cut. The second example is that we use a new criterion to partition the time horizon into near-stationary intervals as mentioned in Remark 2. This technique improves the bound we get.

Reviewer comment (Writing):
The notation in the paper is a bit difficult to read in certain cases. In particular, the use of $\Delta$ in exponents could be clarified. (See below)

Author response:
- **$\Delta$**: We apologize for the confusion. $c_1^\Delta,c_2^\Delta,c_3^\Delta$ are constants depending on the base algorithms (not $c$ to the power $\Delta$). The purpose of writing in this way is to remind readers that it is the constant coefficient of $\Delta$ in the assumptions and to keep the notation concise.

Reviewer comment (Writing):
Can you clarify the difference between $c_1^\Delta$ versus $c_1$ and $c_2^\Delta$ versus $c_2$? Is $c_1^\Delta$ referring to $c_1$ to the power $\Delta$, or is it some other constant entirely?

Author response:
- **$\Delta$**: We apologize for the confusion. $c_1^\Delta,c_2^\Delta,c_3^\Delta$ are constants depending on the base algorithms (not $c$ to the power $\Delta$). The purpose of writing in this way is to remind readers that it is the constant coefficient of $\Delta$ in the assumptions and to keep the notation concise.

Reviewer comment (Theory):
Is your definition of a "total variation non-stationarity budget" $\Delta$ used in any other works? As mentioned previously, it seems like a very pessimistic quantity (as one would expect from a TV-like quantity) scaling linearly in # of states * # of players * # of actions * horizon.

Author response:
- **Total Variation:** We apologize for our typos in the paper. Following prior works, the nonstationarity takes the max over states and actions instead of taking sum. Similar notations are common in literature. For instance, the measure Wei & Luo (2021) use for single-agent RL is $$\Delta_{[t,t+1]}=H\sum_{h=1}^H\max_{s,a}\left|r_h^t(s,a)-r_h^{t+1}(s,a)\right|+H^2\sum_{h=1}^H\max_{s,a}\left\|r_h^t(s,a)-r_h^{t+1}(s,a)\right\|$$ which has an $O(H^2)$ difference in order from our measure.

Reviewer comment (Presentation):
Are the first 5 columns of Table 1 referring to single-timestep games?

Author response:
- **Table 1:** You are correct, the first five rows refer to single-timestep games.

Reviewer comment (Writing):
Is there a reason the terminology alternates between "Protocol" and "Algorithm"?

Author response:
- There was no particular reason for alternating between 'Protocol' and 'Algorithm'. We used 'Algorithm' for the main algorithms and 'Protocol' for the auxiliary algorithms. We have changed them all into 'Algorithm' in the revised version.

Reviewer comment (Writing):
Typo on page 6: chanllanges -> challenges, page 5: For simplicity, We -> For simplicity, we.

Author response:
- We have fixed our typos.

Reviewer comment (Novelty):
The paper seems to be the first work that considers the non-stationary Markov games, but this particular game setup is not well-motivated, except it is a natural generalization from the non-stationary bandit problem. I expect the author to provide some discussion on the motivation of this game setup, possibly with some real-world application scenarios.

Author response:
- **Motivating Examples:** Non-stationarity is ubiquitous in multi-agent systems, here are some concrete examples. First, consider traffic routing in a road network, each driver tries to get to their destination as soon as possible. Due to congestion, the action of one player can affect others' payoffs. The underlying environment can be changing due to weather conditions, pedestrain conditions and road policy changes. Second, consider the financial market. Stock markets, currency exchange, and other financial markets can be highly non-stationary due to constantly changing economic conditions, news events, and regulations. Traders need to adapt their investment strategies to stay profitable. Third, consider professional sports teams, where each of them adopts a strategy to play against other teams throughout a season. In different seasons, game rules, player rosters and the performance of each player can all be different, introducing non-stationarity. In particular, **at the end of Section 1.1**, we provide a potential application of adaptively tracking Nash Equilibria in the first example.

Reviewer comment (Theory):
Is Assumption 3 a reasonable assumption? I expect the author to add some discussion to justify this assumption (you did for Assumption 1, 2). I think only when \delta is small, there are existing algorithms that can satisfy this assumption.

Author response:
- **Assumption 3:** Assumption 3 is a direct corollary of the assumption Wei & Luo (2021) made on the stationary bandit algorithms. **Details can be found in the proof of Proposition 3 in the appendix**. We restate Assumption 1 of Wei & Luo (2021) as follows. Let $f_t:\Pi\to[0,1]$ be the reward function at timestep $t$ where $\Pi$ is the policy set of the player and $t\in[T]$. At timestep $t$, $f^\star_t=\max_{\pi\in\Pi}f_t(\pi)$ is the optimal mean reward and $R_t$ is the mean reward received by the player. The algorithm outputs an auxiliary quantity $\widetilde{f}\_t$ at timestep $t$ satisfying that with probability $1-\delta/T$, for all timesteps $t$, for some function $\rho:[T]\to\mathbb{R}$, $$\widetilde{f}\_t\geq\min_{\tau\in[1,t]}f_\tau^\star-\Lambda_{[1,t]},\quad \frac{1}{t}\sum_{\tau=1}^t\left(\widetilde{f}\_\tau-R_\tau\right)\leq\rho(t)+\Lambda_{[1,t]}.$$ Here $\Lambda$ is the non-stationarity measure. Using the fact that $$\max_{\tau\in[1,t]}f_\tau^\star-\min_{\tau\in[1,t]}f_\tau^\star\leq\Lambda_{[1,t]}$$ we have that for any timestep $s\in[1,t]$, $$\frac{1}{t}\sum_{\tau=1}^t\left(\max_{\tau'\in[1,t]}f_{\tau'}^\star-R_\tau\right)\leq\rho(t)+3\Lambda_{[1,t]}.$$ For single-agent RL, Q-UCB algorithm achieves $\rho(t)=\widetilde{O}\left(\sqrt{H^5SA/t}+H^3SA/t\right)$ and $\Lambda_{1,t}=\widetilde{O}(H^2\Delta)$. Thus, the average policy of behavior policies in $[1,t]$ satisfies Assumption 3 with $C_3(\epsilon,\delta)=\widetilde{O}(H^5SA/\epsilon^2)$ and $c_3^\Delta=\widetilde{O}(H^2)$.

Reviewer comment (Novelty):
It seems to me that the key idea of the proposed algorithm is largely based on that of Wei & Luo (2021). I expect the author to provide some discussion/summary on the challenges of extending their approach to the game setup and highlight some key differences between the two algorithms.

Author response:
- **Challenges extending MASTER to Games:** The challenges of extending non-stationary bandit algorithms to the game setup is **discussed extensively in Section 3**. The main challenge of extending MASTER to the game setup is as follows. The most crucial part of MASTER is to make use of the auxiliary quantity $\widetilde{f}\_t$ to design clever criteria that detect changes in the underlying environment that significantly affect the performance. Usually, $\widetilde{f}\_t$ is the optimistic value of the action. This quantity is easy to find for single-agent RL because many algorithms are UCB-based. However, MARL algorithms are more diverse. For instance, every agent running adversarial bandit algorithm converges to a CCE in general-sum Markov games, where such value is hard to extract. Even if we try to build an algorithm on top of a UCB-based algorithm, like Nash-UCB, it is hard to extend Assumption 1 in Wei & Luo (2021) to games. A quantity similar to $\min_{\tau\in[1,t]}f_\tau^\star$ is difficult to define because the best response is different for different policies and comparing mean rewards of best responses to different policies is pointless. This difficulty is also discussed in Section 3.  

- **Differences between MASTER and our algorithm:** First, we make sample complexity assumptions on the base algorithms. In contrast, MASTER assumes the existence of auxiliary quantity $\widetilde{f}_t$ and makes a more regret-like assumption. Second, we adopt an explore-commit-test paradigm. In contrast, MASTER automatically test for significant changes while running the learning algorithms, which is not easy to achieve for games as discussed in Section 3. Third, while sharing the same high-level idea of base algorithm, adaptation to sample-complexity-based assumption is made in our algorithm, as explained in the paragraph before Lemma 1. Actually, the idea of running multiple instances of base algorithm is widely used in non-stationary bandit algorithm since it was proposed in AdSwitch (Auer et al. (2019)). Lastly, the regret calculation is significantly different. As mentioned at the end of Section 5.2, we introduce a new partitioning of the horizon into near-stationary intervals to improve the bound.