Paper: Achieving Fairness in Multi-Agent MDP Using Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
The toy MDPs in the experiments are very small and have a short horizon.

Author response:
We added a new simulation figure with a longer horizon (H=10) in the new Figure 4 of the current revision, and we will add more simulations for the final version of this paper.

Reviewer comment (Experiments):
Although the authors discuss resource allocation throughout the paper and that problem setting seems to be the main goal, the experiments are conducted on randomly generated toy MDPs, which is surprising.

Author response:
Thanks for your suggestions. Our current simulation is on randomly generated toy MDPs because we want to speed up the simulation. Nonetheless, we agree with the reviewer that showing experiments on real resource allocation is more helpful. Although the rebuttal time period is too short to run simulations in this new setup, we will add these simulations on resource allocation in the final version.

Reviewer comment (Presentation):
The experimental results should have been presented in the main body of the paper, perhaps instead of the proof outlines.

Author response:
Thanks for your suggestion. In the final version, we plan to allocate some space (by shrinking the length of some other parts) in the main body to include experimental results.

Reviewer comment (Writing):
The paper is hard to follow, and both the writing and the outline need improvements for clarity and coherence.

Author response:
Thank you for your comment. Due to space limitations, we had to put some less important math details in Appendix C. We will revise the paper further to ensure that the main body is self-contained.

Reviewer comment (Writing):
Some of the notation is cluttered and overloaded (e.g., equations 10-13), making the math hard to understand.

Author response:
Thank you for your comment. Due to space limitations, we had to put some less important math details in Appendix C. We will revise the paper further to ensure that the main body is self-contained.

Reviewer comment (Presentation):
While proof outlines are appreciated, in their current form they are vague. Additionally, their positioning in the text is confusing; for example, the proof outline of Theorem 1 is not immediately after the theorem.

Author response:
Due to the limit of the paper length, we use the proof outline only to show readers a rough idea of our proving methods. If the reviewer wants to know some specific parts of the proof outline, we are happy to provide more details. The full rigorous proof is in the appendix.

Reviewer comment (Presentation):
Some section titles are too generic and lack useful information. For example, Section 5 is titled “Algorithm” whereas Section 6 is titled “Offline Fair MARL”; Section 5 could be renamed to something more informative, such as “Online Fair MARL.”

Author response:
Thanks for your suggestions. In the revision, we have changed the location of the proof outline and the section title as you suggested.

Reviewer comment (Experiments):
Why is the horizon extremely short (H=3) in the experiments in Appendix G?

Author response:
We use a short horizon to speed up the simulation. To address the concern of the horizon length, in the revision, we have added experiments of a slightly longer horizon (H=10) in the new Fig. 4.

Reviewer comment (Evaluation):
In Figure 3, why is the offline RL algorithm significantly outperforming the online RL algorithm? This is surprising because usually online performs better, and more importantly, the proposed online RL algorithm is optimistic whereas the offline RL algorithm is pessimistic.

Author response:
The reason is that in Fig 3, for each $K$, the offline RL algorithm uses the same data collected by the online algorithm. In other words, the high-quality data explored by the online algorithm is also used as the input of the offline RL algorithm. Thus, the offline algorithm gets the full benefits of the high-quality data collected by the online algorithm. Meanwhile, the online algorithm needs to balance between the exploitation of the collected data and future exploration, while the offline algorithm totally works on exploitation. This is why the offline RL algorithm works better than the online algorithm in Fig. 3. We also agree that if the offline algorithm uses different data (e.g., randomly generated data), then the online algorithm may perform better than the offline one when the offline RL algorithm does not get access to the high-quality data collected by online RL.

Reviewer comment (Novelty):
Fairness in (deep) RL, and in particular alpha-fairness, has been studied in multiple prior works. The Related Work section is too concise and does not give the existing literature enough merit. Example: Zimmer et al. 2021 is cited, but is claimed to focus on gini fairness. This is not true: they focus on a general class of fair SW functions, including alpha-fairness, and they have long-term fairness, and they propose a policy gradient algorithm. Another example: Ivanov et al. 2021 is missing, and they have long-term max-min fairness.

Author response:
Thank you for your suggestion. We have now modified our claim. We agree that Zimmer et al. 2021 indeed considered alpha-fairness. However, unlike those previous works, we provide regret bounds. Further, unlike all the other papers, we also consider an offline RL setup.  We have now added the above in the paper.
We did not find Ivanov et al. 2021. If the reviewer can provide the title or the link of the paper, we are happy to cite and compare with it in the final version.

Reviewer comment (Novelty):
In abstract, “the exploration of fairness in such systems for unknown environments remains open”. Yet many papers from section 2 are in model-free setting.

Author response:
While we agree that there are model-free algorithms with good empirical performance, these algorithms do not provide theoretical guarantees, e.g., none of the algorithms provide a regret bound. This is what we meant in the above sentence. In the revision, we modified the sentence as the following: “...provably efficient exploration of fairness in such systems for unknown environments remains open”.

Reviewer comment (Novelty):
Given the comment about the relation to literature, would it perhaps be more fair (pun not intended) to frame the paper as theoretically justifying existing approaches and proposing their new implementations rather than proposing new approaches?

Author response:
Thank you for the interesting question. Note that our proposed algorithm is indeed a new algorithm that can achieve regret bound on fairness. Further, to the best of our knowledge, we propose the first offline RL algorithm with provable sub-optimal guarantee. As the reviewer pointed out there are algorithms that have attained good empirical performance, however, they do not provide any theoretical guarantees. We hope that our work can influence others to provide theoretical performance bounds for those approaches as well.

Reviewer comment (Novelty):
Isn’t the online algorithm in 5.2 a variation/extension of UCB for bandits?

Author response:
We agree that Algorithm 1 seems very similar to UCB for bandits. However, there are subtle differences.  First, we want to emphasize that the construction of the “arms” and the confidence interval require specific consideration of the fairness objective. For example, we have to use the state-action-next-state occupancy measure as “arms” instead of the common state-action occupancy measure. Second, we consider an RL setting so the transition probability affects the performance, which is different from a classical UCB for bandits. In particular, we have to find the confidence interval of the transition probability. Finally, we have to find the upper confidence bound on the individual value function by combining the above. Details can be found in Appendix C.1.

Reviewer comment (Evaluation):
The occupancy measure is based on the frequency of appearance for each state-action pair, which may be biased and difficult to count since we cannot always experience the entire environment.

Author response:
We agree that the cardinality of S can be large in a multi-agent system and counting the state-action pair may be difficult in this case. However, in this paper, our focus is on dealing with fairness. We leave the possible state dimension reduction methods to future work. For example, one possible extension might be to consider linear MDP structure in [1-3]. For the concern that we cannot always experience the entire environment, our online algorithm indeed has an exploration term in the range of confidence intervals that puts priority on those unvisited state-action pairs.

Also note that we use a policy-gradient approach which might be more suitable for large state-space as they can leverage on function approximation using neural networks. However, the sample complexity bound characterization of such approaches has been left for future investigations. 

[1]. Neu, Gergely, and Nneka Okolo. "Efficient global planning in large mdps via stochastic primal-dual optimization." In International Conference on Algorithmic Learning Theory, pp. 1101-1123. PMLR, 2023.

[2]. Neu, Gergely, and Julia Olkhovskaya. "Online learning in MDPs with linear function approximation and bandit feedback." Advances in Neural Information Processing Systems 34 (2021): 10407-10417.

[3]. Nachum, Ofir, and Bo Dai. "Reinforcement learning via fenchel-rockafellar duality." arXiv preprint arXiv:2001.01866 (2020).

Reviewer comment (Experiments):
Could you please provide some examples to demonstrate the effectiveness of fairness in a multi-agent system?

Author response:
Certainly. The effectiveness of fairness is motivated by the literature on resource allocation in communication networking systems. For example, the max-min fairness is particularly effective when the overall system is limited by the agent/device that has the smallest utility. For example, the throughput over a route in a network (a set of links) is limited by the link with the smallest capacity. Thus, by maximizing the minimal agent’s (link’s) capacity, we effectively maximize the throughput of the network.

Reviewer comment (Evaluation):
I cannot understand what the experiment in the appendix demonstrates. Could you please provide a detailed description? (or how each agent can benefit from the fairness algorithm?)

Author response:
Thank you for this question. The fairness algorithm’s goal is to not necessarily benefit all agents, but to ensure that the disparity among agents is not very large. Usually, to achieve the fairness objective, some agents will need to sacrifice their rewards while some other agents will get more rewards (thus the result becomes “fair”). How the disparities are mitigated depends on the fairness objective. For example, in the max-min fairness, we tend to equalize the cumulative rewards achieved by each agent. In our simulations in the appendix, we consider all three types of fairness introduced in Section 3. For example, in Fig. 1(a), the blue curve approaches the red dashed line, which means that our Algorithm 1 eventually achieves the max-min fairness. In other words, the algorithm learned an optimal policy that can maximize the long-term return of the agent who has the smallest long-term return among all agents.

Reviewer comment (Theory):
The setting seems a bit restrictive. In particular, this paper considers the centralized setting and assumes all the $N$ agents have the same action $a$. This makes the problem no different from a single-agent MDP with $N$ different reward functions. Moreover, the assumption that the reward function is bounded from below by $\epsilon/ H$ seems an unnatural assumption.

Author response:
In our setting, we consider a centralized decision maker that learns the fair policy for all agents. Different agents can have different actions and different rewards, while the centralized decision-maker can view the actions of all agents as an aggregated action. However, the reward cannot be aggregated because of the fairness requirement. Therefore, the MDP is not equivalent to a single-agent MDP. While we consider that a central agent is taking decisions, our focus is on how those decisions can be fair to the agents.

Further note that we can consider the setup where each individual agent is making a decision (here, the action-space can be different for different agents). Our algorithm is still applicable and it is equivalent to a centralized-training-and-decentralized-execution framework [1-2]. In particular, the optimal joint action can be learned by a centralized agent, then each agent can execute its own action based on the global state. The consideration of a decentralized training framework where each agent can learn only from its local information has been left for future investigations. 

[1]. Chen, Gang. "A New Framework for Multi-Agent Reinforcement Learning--Centralized Training and Exploration with Decentralized Execution via Policy Distillation." arXiv:1910.09152 (2019).

[2]. Zimmer, Matthieu, Claire Glanois, Umer Siddique, and Paul Weng. "Learning fair policies in decentralized cooperative multi-agent reinforcement learning." ICML 2021.

Reviewer comment (Evaluation):
It seems unclear whether the fairness notions studied in this work include most of the common fairness metrics. For example, existing works also study Generalized Gini Social Welfare Function in the context of MARL.

Author response:
We agree that there are some other fairness types in the literature. However, in our paper, we have considered max-min, proportional, and alpha fairness, which are the most common/standard types in the literature of resource allocation (e.g., see Mo & Walrand (2000)). 

Jeonghoon Mo and Jean Walrand. Fair end-to-end window-based congestion control. IEEE/ACM Transactions on networking, 8(5):556–567, 2000.

Reviewer comment (Novelty):
Given existing works on UCB-based and pessimism-based RL, what are the technical novelties?

Author response:
For the novelty of the results, to the best of our knowledge, our work is the first to give the regret/suboptimality bound for both online and offline fair RL algorithms. We achieve sub-linear regret bound and order (nearly) optimal in $K$.

Regarding the technical novelty, although we acknowledge that certain components/steps of our methods (e.g., using confidence bound to quantify the uncertainty) may seem similar to some online convex work, our work, in general, is different. This is because the max-min, proportional, and alpha fairness objectives have not been studied in the RL literature, and the methods for a general concave objective do not fit here because the fairness objective (proportional and alpha fairness) is not always L-Lipchitz. This critical difference between the fairness objective and the general Lipschitiz continuous concave objective is also one of our motivations. Specifically, since the proportional and alpha fairness objective function can go to infinity and the regret can be unbounded, we have to make an additional requirement in Assumption 1 that the instant reward should not be too small. Related to this point, the technical challenge/novelty is how to connect this additional requirement to the performance of our algorithm. Indeed, as we show in the main results, this effect depends on the specific type of fairness, which is characterized by the coefficient $C_F$. In this regard, we state and prove Lemma 5 to show the dependency over different fairness functions, which we use to bound the regret. 

Furthermore, there are subtle differences in the algorithmic front as well. For example, we find that instead of a state-action occupancy measure, we need to consider a state-action-next state occupancy measure for efficient computation for an online approach. On the other hand, we have to modify the reward function in order to find the policy in the offline setup.

Reviewer comment (Experiments):
Can you run some toy-ish experiments to showcase the performance of the algorithm? In simulation experiments, how do we measure fairness-related regret?

Author response:
Thank you for your suggestion. Yes, we show simulation results in Appendix G. In all simulation figures, we plot the optimal fair policy (red horizontal dashed line). The regret is measured by the gap to this curve of the optimal fair policy. The empirical result also shows that the difference with the result achieved by our policy and the policy which is not fair (rather, it only maximizes the sum of the individual value functions).