Paper: Strategic Recommendations for Improved Outcomes in Congestion Games (ICLR-2024)

Reviewer comment (Writing):
In general, the writing is quite confusing. This includes a lack of robust mathematical presentations of results (Theorem 1 and 2) and several typos/grammar issues.

Author response:
You and the other reviewers have pointed out that the theorems are to be significantly improved, and that it is not clear how the RS works from reading section 2. This clearly is a big shortcoming which we must address. We will get to that as soon as possible with the revised manuscript.

1. We completely agree with you and the other reviewers on this point, and is a major part of our revision delivered to you ASAP.

Reviewer comment (Theory):
The main results (Theorem 1 and 2) do not give any guarantee on the convergence toward a social optimum (or at least, it lacks an explanation/justification). These theorems are not presented in a mathematically manner which hinders the readers to understand their contributions. As an heuristic scheme, the proposed algorithm is not supported by any performance analyses. It is quite disappointing as it can be seen from the experiments that as the number of states (recommendation options) increases, the proposed algorithm does drive the system closer to the social optimum.

Author response:
Theorems 1 and 2 guarantee that given assumption 1 the system can be manipulated to be in any target state. That said, we completely agree with you and the other reviewrs that our formulation of the theorems is vague and must be reworked. As it stands we have found that there are two concepts of accessibility and controllability from control theory that should fit to phrase these theorems with mathematical rigor.

Reviewer comment (Writing):
Some possible typos/ unclear sentences:  
   + p.6, first paragraph, Regarding Theorem 1?  
   + p.6, greater detail --> greater details  
   + p.6 "....reflects the action that most need agents to be assigned to them" ?  
   + notation r is used both for welfare function W (page 2) and reward function (page 5)  
   + In Equation (1), should z_t be a subscript instead of being a superscript?  
   + In Section 2.1., there is a sentence saying that " the goal of RS is to find P*(a)" and another sentence saying that the goal of RS is to find the policy Pi^*_RS.

Author response:
Thank you for pointing out all these typos. We will be sure to include them in our revision ASAP.

Reviewer comment (Writing):
What d*_a means exactly in the paragraph after Equation (3)?

Author response:
$d^*_a$  is the number of agents that take action $a$ at the social optimum.

Reviewer comment (Theory):
Given a congestion game model, how an RS can design a recommendation set that guarantees Assumption 1?

Author response:
Assumption 1 can not be guaranteed by designing the recommendation set, but is instead an assumption about the distribution of q-values, which the recommender system may not be in control of. We realized that assumption 1 must be reformulated and will do so ASAP in our review.

Reviewer comment (Theory):
How precisely does the proposed algorithm confuses the Q-leaner (as remarked in Conclusion)?

Author response:
The proposed algorithm selects states for q-learners such that the actions selected by q-learners lead to desired outcomes. After each action the q-learners update their q-values for that recommendation. If there were no recommendations, the dynamics would settle on the behaviour shown in figure 2. We call this "confusion" without recommendations the Q-learners would act differently, and because the recommender is effectively disrupting the learning process of the Q-learners by switching around the recommendations.

Reviewer comment (Theory):
The theoretical results (Theorem 1 and 2) appear somewhat unclear; they could benefit from more details or precision.

Author response:
You and the other reviewers have pointed out that the theorems are to be significantly improved, and that it is not clear how the RS works from reading section 2. This clearly is a big shortcoming which we must address. We will get to that as soon as possible with the revised manuscript.

Reviewer comment (Experiments):
On the experimental side, while the improvements for the improved recommendation system look quite interesting, they appear to be localized to a single network structure and it is not clear how general these results are.

Author response:
Thank you for this point. We will do our best to provide some intuition in the revision of how the heuristic algorithm works. In the paper we provide a generalized presentation of the algorithm. Our results use a specific instantiation of if. Hopefully when we get back with our theorems it will be clear that it should be possible to design algorithms like our heuristic one for different kinds of networks. We have another example of a network which we will include which is the initial network of the Braess Paradox. In this network the Nash Equilibrium is the socially Optimal Solution, and we show that we can rapidly increase the speed of convergence to the equilibrium by cleverly picking some recommendations. We will include this in the revision ASAP.

Reviewer comment (Experiments):
Since this is primarily an empirical paper, testing the heuristic algorithms on more general networks would provide much better evidence for the utility of this approach.

Author response:
Thank you for this point. We will do our best to provide some intuition in the revision of how the heuristic algorithm works. In the paper we provide a generalized presentation of the algorithm. Our results use a specific instantiation of if. Hopefully when we get back with our theorems it will be clear that it should be possible to design algorithms like our heuristic one for different kinds of networks. We have another example of a network which we will include which is the initial network of the Braess Paradox. In this network the Nash Equilibrium is the socially Optimal Solution, and we show that we can rapidly increase the speed of convergence to the equilibrium by cleverly picking some recommendations. We will include this in the revision ASAP.

Reviewer comment (Writing):
Was $\varepsilon$-greedy Q-learning ever defined? My understanding was that this means that each agent uniformly explores with probability $\varepsilon$; if so, how does this connect with Assumption 1?

Author response:
1. epsilon greedy means, as suggested, that agents uniformly explore with probability epsilon. Your comment makes clear that we should define that explicitly. This is not connected with assumption 1 which is an assumption on the initial distribution of q-values. We realize however that our assumption needs to be rephrased to: assume that for all agents and for all actions, the q-values are drawn from a distribution such that each argmax has a non-zero probability of occurring. We will formalize this and get back to you ASAP.

Reviewer comment (Theory):
What is the intuition behind the objective function of the RS in equation (2) in terms of KL divergence? Wouldn't a more natural objective be to minimize the weighted congestion of the actions of the Q-learners?

Author response:
2. If we understand correctly, you suggest that rather than looking at the KL divergence between action distributions and a target we minimize the congestion function directly. This point is well taken, and we believe that it will improve our presentation of the reward. In fact, what we care about is not the action distribution with a domain over $A^n$ but rather the action distribution with a domain over $A$: what is the proportion of players that selects each action? Thank you very much for pointing this out, we will modify it in the paper.

Reviewer comment (Reproducibility):
It might be useful to provide pseudocode for the subroutines in Algorithm 1, at least in the Appendix.

Author response:
4. We fully agree that it would be useful to add pseudocode for the subroutines in the appendix, and see that it is not easy or intuitive to understand how the heuristic algorithm works from the way it is presented. We are adding these to the appendix.

Reviewer comment (Experiments):
What is the (empirical or theoretical) computational cost of the heuristic algorithm, especially on generalizations to larger network topologies?

Author response:
6. The computational cost seems to be polynomial in the various sizes: the number of agents, the number of actions and the number of possible recommendations. We will try to give this full characterization to make it clearer.

Reviewer comment (Experiments):
The experiments seem to have only been done in the Braess paradox network and not on more general topologies, which raises concerns about generality.

Author response:
Thank you for this point. We will do our best to provide some intuition in the revision of how the heuristic algorithm works. In the paper we provide a generalized presentation of the algorithm. Our results use a specific instantiation of if. Hopefully when we get back with our theorems it will be clear that it should be possible to design algorithms like our heuristic one for different kinds of networks. We have another example of a network which we will include which is the initial network of the Braess Paradox. In this network the Nash Equilibrium is the socially Optimal Solution, and we show that we can rapidly increase the speed of convergence to the equilibrium by cleverly picking some recommendations. We will include this in the revision ASAP.

Reviewer comment (Theory):
Theorem 1 and Theorem 2 are not stated in rigorous ways. There is no quantification for what is meant by 'slower' and 'smaller'.

Author response:
You and the other reviewers have pointed out that the theorems are to be significantly improved, and that it is not clear how the RS works from reading section 2. This clearly is a big shortcoming which we must address. We will get to that as soon as possible with the revised manuscript.

Reviewer comment (Novelty):
For the motivation, could you justify why it is reasonable to study the Q-learning agents in routing games, given that it is known that any better-response dynamics converges to the NE of congestion games.

Author response:
Better-response dynamics converge to Nash, yes, but these dynamics are non-constructive. What do we mean. We mean that they do not provide the best response, they just assume that the best-response is played and thus Nash equilibrium is an almost trivial equilibrium point of these dynamics. Q-learning on the other hand is a constructive algorithm which can provably converge in MDPs. Its widespread application justifies its study in this model. Furthermore, as is shown in section1, Q-learning does not converge to the Nash equilibrium, which further justifies why we can not rely on best response analysis to know what to expect from practical RL algorithms when deployed IRL.

Reviewer comment (Theory):
In terms of the better social outcome result, it is known that for smooth games (including the class of congestion games), the PoA under NE is essentially the same as the PoA under CE/CCE. Therefore, in general, the social welfare improvement from steering the NE to CE in congestion games can be very little. Do you agree with this argument?

Author response:
This is a very interesting point. Indeed the PoA under CE and CCE are the same. This is because there are no CE/CCE which are different from Nash Equilibrium, also one of the results from Koessler et. al 2023 we cite in our paper. Thank you for the additional reference for such result. What this means for rational players is that there are no recommendations that can reduce the Price of Anarchy. This is quite the strong result. Our paper, shows that there are recommendations that can reduce the Price of Anarchy when the players are Q-learners. Given that Q-learning is a more realistic and constructive algorithm, we try to conclude that understanding whether or not recommendations can influence the outcome IRL can not be determined from rational player assumptions. Indeed the rational player assumptions would say that recommendations can not help avoid inefficient Nash equilibria.

Reviewer comment (Novelty):
There are some related works (e.g., [2,3]) regarding mechanism designs for content recommendation platforms, which also involve dynamical optimization over a congestion game structure. It would be nice to add some discussion of these works in the introduction or related work section.

Author response:
Thank you for the added references that we were not aware of. We will look at them closer and see how they may be added to the discussion. [2] very interesting, as they think about designing rewards such that content creators are effectively playing a potential game. [3] takes again the perspective of content providers (or creators) which may strategically create content that will be recommended by a mediator (recommender system)