Paper:
Fairness in Multi-Agent Sequential Decision-Making
We define a fairness solution criterion for multi-agent decision-making problems, where agents have local interests. This new criterion aims to maximize the worst performance of agents with a consideration on the overall performance. We develop a simple linear programming approach and a more scalable game-theoretic approach for computing an optimal fairness policy. This game-theoretic approach formulates this fairness optimization as a two-player zero-sum game and employs an iterative algorithm for finding a Nash equilibrium, corresponding to an optimal fairness policy. We scale up this approach by exploiting problem structure and value function approximation. Our experiments on resource allocation problems show that this fairness criterion provides a more favorable solution than the utilitarian criterion, and that our gametheoretic approach is significantly faster than linear programming.

Review:
This paper proposes a policy selection criterion (regularized maximin fairness) for multi-agent factored MDPs that is equivalent to the most utilitarian of egalitarian policies. The paper then presents an algorithm to find such a policy based on framing the problem as a zero sum problem between a utilitarian player and an egalitarian player. The algorithm operates over multiple iterations of solving standard utilitarian multi-agent factored MDPs.The paper is well written with nice theoretical proofs, and the idea of viewing the problem as a zero sum game is compelling. However, I believe this view has unnecessarily complicated the solution. Also, I am worried that the problem itself is not strongly motivated.- Since the problem being solved is ultimately centralized, the need for a different selection criteria must be due to a need for greater model expressiveness or conciseness. Its not clear to me that the problems couldn’t just be modeled in a different way (perhaps with additional states) using a standard global reward function that expresses the goal they wish to maximize (i.e. throughput). No argument is made that these problems don’t arise only due to lack of modeling skill. Even more importantly, utility theory (as used by MDPs) relies on the expected utility hypothesis (risk neutral), but it seems to me that preferences over policies given by regularized maximin fairness breaks this assumption by preferring (motivationally) to be risk adverse.For example, suppose we have a problem where two agents can choose either to A) receive (1, 1) or B) flip a fair coin and randomly receive (0, 4) or (4, 0). Each individual’s utility preference is to flip the coin, but the reward received by centralized maximin would have a value of 0 if the coin was chosen. Or course, this is not how the maximin criteria is defined, but is how it should be defined given the motivation. All of the examples given (assembly plan, telecommunications, traffic lights) would prefer the (1,1) solution, hence my concern with the motivation, and if the  regularized maximin fairness is the proper means of modeling these problems. Even if the problem is repeated (so expectation makes more sense) agents would still want to be risk adverse. - I believe the algorithm should be viewed as column generation to solve the zero-sum game. The variables are weights on deterministic joint-policies (i.e. p* along with some q* for which the policy maximizes) such that the resulting solution is a convex combination of these policies. The objective becomes the value of the minimum player. Column generation proceeds as per the best response problem in Line 4. Where q* = the convex combination of q*’s that make up the current basis. One immediate benefit of this view, is that line 3 needs not be solved each time, instead the new column can be added to the current basis and one (or more) of the old columns can be removed. - I don’t understand why line 4 of the algorithm 1 is considered a single-agent MDP. Doesn’t the agent need to optimize over all agent’s actions, which is only efficient because this is a factored multi-agent MDP? Calling this a single-agent MDP is misleading. - Pet peeve: Use of the word “obviously” to avoid explanation. This is basically argument by ridicule (“if you don’t understand this step, then you’re stupid”) and should be avoided. If you don’t want to explain a step, don’t explain it. No need to insult your reader.- The discussion on decentralized policies and Proposition 2 seems out of place in this paper. DecPOMDPs are quite a different beast, and its not clear how this work would help decentralized policies. Proposition 2 is a general statement for any optimal centralized policy and is already known.