Paper:
A Gang of Bandits
Multi-armed bandit problems formalize the exploration-exploitation trade-offs arising in several industrially relevant applications, such as online advertisement and, more generally, recommendation systems. In many cases, however, these applications have a strong social component, whose integration in the bandit algorithm could lead to a dramatic performance increase. For instance, content may be served to a group of users by taking advantage of an underlying network of social relationships among them. In this paper, we introduce novel algorithmic approaches to the solution of such networked bandit problems. More specifically, we design and analyze a global recommendation strategy which allocates a bandit algorithm to each network node (user) and allows it to "share" signals (contexts and payoffs) with the neghboring nodes. We then derive two more scalable variants of this strategy based on different ways of clustering the graph nodes. We experimentally compare the algorithm and its variants to state-of-the-art methods for contextual bandits that do not use the relational information. Our experiments, carried out on synthetic and real-world datasets, show a consistent increase in prediction performance obtained by exploiting the network structure.

Review:
This paper considers a setting where several 
      contextual linear bandits are connected by the underlying graph. The 
      assumptions is the weight vectors of the respective nodes are smooth over 
      the graph (are close for neighbors). This appears to a be a good and rich 
      enough model for recommendations in social networks. The algorithm 
      runs classical contextual linear bandits on each node (which node is being 
      tried is determined by nature), but then shares this information with 
      other nodes as well so they also update their model estimates and 
      confidence widths appropriately via Laplacian. Unfortunately, the 
      regret analysis provided is disappointing. The authors in fact provide the 
      analysis for a different algorithm than they propose and evaluate in the 
      experiments. This could be perhaps acceptable but I found no reason 
      whatsoever over for this. There is no discussion, how CB vs. TCB would 
      change the algorithm, its evaluation and the deployment. At least the 
      difference between TCB and CB should be discussed better than one could 
      use the results from [1]. It is therefore possible, that the TCB-kind 
      algorithm does not do well in the experiments or that for CB-kind it is 
      not possible to provide learning guarantees. Maybe it would be better 
      just to not include such analysis. Regarding the current analysis, 
      it is not clear how big the terms ln|M_t| and 2L can be. How do they 
      behave with T and n? This is needed in order to get some insight from the 
      provided regret bound. And how does the graph noise influence these 
      quantities? Furthermore, typically [4,9] the analyses for linear 
      bandits give at least a guideline to set the exploration parameter \alpha. 
      Unfortunately, we do get this from the analysis provided. What is the 
      dependence of the upper bound on alpha anyway? Is it hidden in some of the 
      quantities mentioned above? The experiment on the other hand are 
      treated very well. The descriptions are very detailed and well explained. 
      It is also great that the authors performed the experiments on real data 
      and described the practical details to make the experiments reproducible. 
      Finally, L186 - L190 The paper mentions that through graph, the 
      reward influences other nodes in the "lesser extent", "somehow 
      informatively" and "similar updates" are desired. However I do not see a 
      parameter for "somehow" or "similar" in the algorithm. Could that be done 
      by regularizing L by a multiple of I instead of I. In the experimental 
      section, the authors mention that the clustering acts as a regularizer. 
      Would not regularizing the Laplacian instead be a more principled way 
      to achieve this regularization? Other comments - Why it is 
      L(u1,...un) in the regret bound and not the regularized version, since 
      the Laplacian is regularized in the algorithm? After the rebuttal: 
      While this is a good work, I believe that including the theoretical 
      analysis of a changed algorithm with not entirely clear relationship 
      between the two, makes it confusing. I would increase my score if the 
      theoretical analysis was taken away.