An α -No-Regret Algorithm For Graphical Bilinear Bandits
Abstract
We propose the first regret-based approach to the Graphical Bilinear Bandits problem, where n agents in a graph play a stochastic bilinear bandit game with each of their neighbors.
This setting reveals a combinatorial NP-hard problem that prevents the use of any existing regret-based algorithm in the (bi-)linear bandit literature.
In this paper, we fill this gap and present the first regret-based algorithm for graphical bilinear bandits using the principle of optimism in the face of uncertainty.
Theoretical analysis of this new method yields an upper bound of ˜
O ( √ T ) on the α -regret and evidences the impact of the graph structure on the rate of convergence.
Finally, we show through various experiments the validity of our approach.
Introduction
In this paper we are interested in solving centralized multi-agent problems that involve interactions between agents.
For instance, consider the problem of the configuration of antennas in a wireless cellular network [Siomina et al., 2006] or the adjustment of turbine blades in a wind farm [van Dijk et al., 2016, Bargiacchi et al., 2018].
Choosing a parameter for an antenna (respectively adjusting a turbine blade) has an impact on its signal quality (respectively its energy collection efficiency) but also on the one of its neighboring antennas due to signal interference (respectively its neighboring turbine blade due to wind turbulence).
By considering each antenna or turbine blade as an agent, these problems can be modeled as a multi-agent multi-armed bandit problem [Bargiacchi et al., 2018] with the knowledge of a coordination graph [Guestrin et al., 2002] where each node represents an agent and each edge represents an interaction between two agents.
As a natural extension of the well-studied linear bandit to this multi-agent setting we focus on the graphical bilinear bandit setting introduced in Rizk et al.
[2021]: at each round, a central entity chooses an action for each agent and observes a bilinear reward for each couple of neighbors in the graph that is function of the actions chosen by the two neighboring agents.
The objective of the learner can either be to maximize the cumulative reward, which requires a trade-off between exploration and exploitation , or to identify the best action without being concerned about suboptimal rewards obtained during the learning process, known as the pure exploration setting [Bubeck et al., 2009, Audibert and Bubeck, 2010].
For graphical bilinear bandits, while the latter has already been addressed in Rizk et al.
[2021], maximizing the cumulative reward (and thus minimizing the associated regret) remains an open question that we address in this paper.
Contribution.
Given the NP-Hardness property of the problem and the impossibility of designing an algorithm that has sublinear regret with respect to the horizon T , we propose the first regret-based algorithm for the graphical bilinear bandit problem that gives a guarantee on the α -regret of ˜ O ( √ T ) where α (greater than 1 / 2 ) is a problem-dependent parameter and ˜
O hides the logarithmic factors.
The problem as well as the notion of α -regret are introduced in Section 2.
We provide a first algorithm and establish theoretical guarantees in Section 3.
In Section 4, we identify the problem-dependent parameters that one could use to improve both theoretical guarantees and empirical performances of the algorithm.
Accordingly, we state a refined bound that significantly improves α .
Finally, Sectiongives an experimental overview of the various theoretical bounds established for the regret.
Related work.
Centralized multi-agent bandit problems where the learner has to choose the actions of all the agents at each round implies to deal with parallelizing the learning process on the agents.
In the context of linear rewards where all the agents share the same reward function, Chan et al.
[2021] give a detailed analysis of the problem and show that a sublinear regret in T can be reached but with an additional cost specific to the parallelization.
The main difference with their work is that they consider independent agents, whereas we assume interactions between the agents.
Choosing an action for each node of a graph to then observe a global reward over the graph is a combinatorial bandit problem [Cesa-Bianchi and Lugosi, 2012], the number of joint arms scaling exponentially with the number of nodes.
This has been studied in the regret-based context [Chen et al., 2013, Perrault et al., 2020] where the notion of α -regret is introduced to cope with the NP-Hardness of the problem.
Most papers assume the availability of an oracle to solve the inner combinatorial optimization problem at each round.
In this paper, we do not make this assumption and we present an algorithm that takes into account the graph structure, which has not been considered in the combinatorial framework.
Bandit problems in graphs have been studied from several angles.
For instance, on the one hand, in
Valko et al.
[2014], Mannor and Shamir [2011], each node of the graph is an arm of the bandit problem, which, if pulled, gives information on its neighboring arms.
On the other hand, in Cesa-Bianchi et al.
[2013] each node is an instance of a linear bandit and the neighboring nodes are assumed to have similar unknown regression coefficients.
In this setting, the reward observed for each agent only depends on its chosen action which does not take into consideration agent interactions.
The graphical bilinear bandit setting has been first established in Rizk et al.
[2021] where they tackle the problem of identifying the 1 / 2 -best joint arm (which has an associated global reward equal to at least 1 / 2 of the optimal one).
They study the pure exploration setting while we tackle the regret-based one.
To the best of our knowledge, our paper is the first one to provide regret-based algorithms for the graphical bilinear bandits setting.
Problem setting
Let G =
( V, E ) be the directed graph defined by V the finite set of nodes representing the agents and
E the set of edges representing the agent interactions.
We assume that if ( i, j ) ∈ E then ( j, i ) ∈ E .
The Graphical Bilinear Bandit with a graph G consists in the following sequential problem [Rizk et al., 2021]: at each round t , each agent i ∈ V chooses an arm x ( i ) t in a finite arm set X and receives for each of its neighbors j a bilinear reward
[Equation]
where M ⋆ ∈ R d × d is an unknown matrix, and η ( i,j ) t a zero-mean σ -sub-Gaussian random variable.
For all agent i ∈ V , we denote N i the set of its neighboring agents.
Let n = Card ( V ) denote the number of nodes, m = Card ( E ) the number of edges and K = Card ( X ) the number of arms.
We assume that for all x ∈ X , ∥ x ∥ 2 ≤ L .
Let ∥ M ⋆ ∥ F be the Frobenius norm of M ⋆ with ∥ M ⋆ ∥ F ≤ S .
We assume that the expected reward of each edge is positive which gives for all ( x, x ′ ) ∈
X 2 , 0 ≤ x ⊤ M ⋆ x ′ ≤ LS .
For each round t , we define the global reward y t =
[Equation] [Equation]
i,j ) t as the sum of the local rewards obtained over the graph at round t , and (cid:16) x (1) ⋆ , . . .
, x ( n ) ⋆ (cid:17) =
arg max( x (1) ,...,x ( n ) )
(cid:80) ( i,j
) ∈
E x ( i )
⊤ M ⋆ x ( j ) as the optimal joint arm that maximizes the expected global reward.
The corresponding global pseudo-regret over T rounds is then defined as follows:
[Equation]
We know from Rizk et al.
[2021] that finding (cid:16) x (1) ⋆ , . . .
, x ( n ) ⋆ (cid:17) is NP-Hard with respect to the number of agents n .
Proposition 2.1 (Rizk et al.
[2021] Theorem 4.1) .
Consider a given matrix M ⋆ ∈ R d × d and a finite arm set X ⊂ R d .
Unless P=NP, there is no polynomial time algorithm guaranteed to find the optimal solution of
[Equation]
Hence, the objective of designing an algorithm with a sublinear regret in T is not feasible in polynomial time with respect to n .
However, some NP-hard problems are α -approximable (for some α ∈ (0 , 1] ), which means that there exists a polynomial-time algorithm guaranteed to produce solutions whose values are at least α times the value of the optimal solution.
For these problems, it makes sense to consider the α -pseudo-regret as in [Chen et al., 2013] which is defined for all α ∈ (0 , 1] as follows
[Equation]
and set the objective of designing an algorithm with a sublinear α -regret.
The linear form.
The reward can either be viewed as a bilinear reward as shown in (1) or as a linear reward in higher dimension [Jun et al., 2019] with y ( i,j ) t
[Equation]
where for any matrix A ∈ R d × d , vec ( A ) denotes the vector in R d 2 which is the concatenation of all the columns of A, and ⟨· , ·⟩ refers to the dot product between two vectors.
To simplify the notation, let us refer to any x ∈ X as a node-arm and define the arm set Z = { vec ( xx ′⊤ )
| ( x, x ′ ) ∈ X} where any z ∈ Z will be referred as an edge-arm .
If the arm x ( i ) t ∈ X represents the node-arm allocated to the node i ∈ V at time t , for each edge ( i,
j ) ∈
E we will denote the associated edge-arm by z ( i,j ) t
[Equation]
and define θ
[Equation]
= vec
[Equation]
M
[Equation]
) the vectorized version of the unknown matrix M ⋆ with ∥ θ ⋆ ∥ 2 ≤ S .
With those notations, the (now) linear reward can be rewritten as follows:
[Equation]
When needed, for any node-arms ( x, x ′ ) ∈
X 2 we use the abbreviation z xx ′ ≜ vec( xx ′⊤ ) .
Finally for any vector x ∈ R d and a symmetric positive-definite matrix A ∈ R d × d , we define ∥ x
∥ A ≜ √ x ⊤ A x .
Optimism in the face of uncertainty for Graphical Bilinear Bandit
Let us first notice that given the linear reward (2) that can be observed for each of the m edges ( i, j ) ∈ E , the central entity is trying to solve m parallel and dependent linear bandit problems.
The parallel aspect comes from the fact that at each round t the learner has to allocate all the m edge-arms of the graph (one for each edge ( i, j ) in E ) before receiving the corresponding m linear rewards.
The dependent aspect comes from the fact that the choice of most of the edge-arms are depending on other edge-arms.
Indeed, two adjacency edges ( i, j ) and ( j, k ) share the node j , hence their allocated edge-arms must share the node-arm x ( j ) t .
In this paper, we choose to design an algorithm based on the principle of optimism in the face of uncertainty Auer et al.
[2002], and in the case of a linear reward Li et al.
[2010], Abbasi-Yadkori et al.
[2011], we need to maintain an estimator of the true parameter θ ⋆ .
To do so, let us define for all rounds t ∈ { 1 , . .
, T } the OLS estimator of θ ⋆ as follows:
[Equation]
where,
[Equation]
i
[Equation]
, with λ > 0 a regularization parameter and
[Equation]
We define also the confidence set
[Equation]
where with probability 1 − δ , we have that θ ⋆ ∈ C t ( δ ) for all t ∈ { 1 , . .
, T } , and δ ∈ (0 , 1] .
Since the graphical bilinear bandit can be seen as a linear bandit in dimension d 2 and with K n possible arms, 3 the idea of our method is to overcome the NP-Hard optimization problem that one would have to solve at each round when directly applying the OFUL algorithm as defined in Abbasi-Yadkori et al.
Indeed, at each round, instead of looking for the best optimistic joint arm
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
Input : graph G =
( V, E ) ,
node-arm set X ( V 1 , V 2 ) =
Approx-MAX-CUT ( G ) for t = 1 to T do // Find the optimistic best couple of node-arms (cid:16) x
[Equation] [Equation]
(cid:17) =
arg
[Equation]
; // Allocate x t and x ′ t in the graph x
i
V 1 ; t i
Obtain for all
Compute end return θ t
Algorithm 2:
Input : G =
( V, E )
Set V 1 = ∅ , V 2 = ∅ for i in V do n 1 = Card ( { ( i,
j ) ∈ E | j ∈ V 1 } ) ; n 2 = Card ( { ( i,
j ) ∈ E | j ∈ V 2 } )
; if n 1 > n 2 then V 2 ← V 2 ∪ { i } else V 1 ← V 1 ∪ { i } ; end return ( V 1 , V 2 ) we consider the couple of node-arms ( x ⋆ , x ′ ⋆ ) =
arg max ( x,x ′ ) ∈X 2 ⟨ z xx ′
+ z x ′ x , θ ⋆ ⟩ maximizing the reward along one edge.
Given the optimal joint arm ( x (1) ⋆ , . . .
, x ( n ) ⋆ ) and the associated optimal edge-arms z ( i,j ) ⋆ = vec ( x ( i ) ⋆ x ( j ) ⋆ ) for all ( i, j ) ∈
E , we have
[Equation]
Hence, an alternative objective is to construct as many edge-arms z x ⋆ x ′ ⋆ and z x ′ ⋆ x
⋆ as possible in the graph.
Naturally, we do not have access to θ
⋆
, so we use the principle of optimism in the face of uncertainty, which is to find the couple ( x t , x ′ t ) such that
[Equation]
and then allocate the node-arms to maximize the number of locally-optimal edge-arms z x t x ′ t and z x ′ t x t .
The motivation of this alternative goal can be understood thanks to (4) where we know that the associated expected reward is better than the one obtained with the edge-arms created with the optimal joint arms.
Of course, one can also easily understand that for a general type of graph it is not possible to allocate those edge-arms everywhere ( i.e., to all the edges).
For example take a complete graph of three nodes, allocating the edge-arms z x t x ′ t and z x ′ t x t is equivalent to allocate x t and x ′ t to the nodes.
By doing so, two nodes will pull x t and the third one will pull x ′ t , which makes it inevitable to allocate sub-optimal and unwanted edge-arms of the form z x t x t .
The main concern with this method is to control how many unwanted edge-arms are drawn in each round (relative to the total number m of edges) in order to minimize their impact on the resulting regret.
Assigning x t to a subset of nodes and x ′ t to the complementary is equivalent to cutting the graph into two pieces and creating two distinct sets of nodes V 1 and V 2 such that V
=
V 1 ∪ V 2 and
V 1 ∩ V 2 = ∅ .
Pulling the maximum amount of optimal edge-arms thus boils down to finding a cut passing through the maximum number of edges.
This problem is known as the Max-Cut problem, which is also NP-Hard.
However, the extensive attention this problem has received allows us to use one of the many approximation algorithms (see, e.g., Algorithm 2) which are guaranteed to yield a cut passing through at least a given fraction of the edges in the graph.
Proposition 3.1.
Given a graph G =
( V, E ) , Algorithm 2 returns a couple ( V 1 , V 2 ) such that
[Equation]
Using Algorithm 2 we can thus divide the set of nodes V into two subsets V 1 and V 2 with the guarantee that at least m/ 2 edges connect the two subsets.
Combining this algorithm with the principle of optimism in the face of uncertainty for linear bandits Abbasi-Yadkori et al.
[2011] we obtain Algorithm 1 for which we can state the following guarantees on the α -regret.
Theorem 3.2.
Given the Graphical Bilinear Bandit problem defined in Section 2, let 0 ≤ γ ≤ 1 be a problem-dependent parameter defined by
[Equation]
and set α = 1+ γ 2 , then the α -regret of Algorithm 1 satisfies
[Equation]
where ˜ O hides the logarithmic factors.
One can notice that the first term of the regret-bound matches the one of a standard linear bandit that pulls sequentially Tm edge-arms.
The second term captures the cost of parallelizing m draws of edge-arms per round.
Indeed, the intuition behind this term is that the couple ( x t , x ′ t ) chosen at round t (and thus after having already pulled tm edge-arms and received tm rewards) is relevant to pull the ( tm + 1) -th edge-arm but not necessarily the ( m − 1) edge-arms that follows (from the ( tm + 2) -th to the ( tm + m ) -th one) since the reward associated to the ( tm + 1) -th
edge-arms could have led to change the central entity choice if it had been done sequentially.
In Chan et al.
[2021], they characterize this phenomenon and show that this potential change in choice occurs less and less often as we pull arms and get rewards, hence the dependence in O (log( Tm ))) .
What is γ and what value can we expect?
This parameter measures what minimum gain with respect to the optimal reward one could get by constructing the edge-arms of the form z xx .
For example, if there exists x 0 ∈ X such that ⟨ z x 0 x 0 , θ ⋆ ⟩ = 0 , then γ = 0 as well and we are in the worst case scenario where we can only have a guarantee on an α -regret with α = 1 / 2 .
In practice, having γ =is reached when given the couple ( x ⋆ , x ′ ⋆ ) =
arg max ( x,x ′ ) ∈X 2 ⟨ z xx ′
+ z x ′ x , θ ⋆ ⟩ , this x 0 is either x ⋆ or x ′ ⋆ .
In other words, if the unwanted edge-arms associated to the couple ( x ⋆ , x ′ ⋆ ) gives low rewards, then the guarantee on the regret will be badly impacted.
Hence we can wonder how we can prevent this phenomenon.
We answer this question in the next section by taking into account both the proportion of undesirable edge arms allocated and their potential rewards at the selection of the pair ( x t , x ′ t ) in order to improve in practice but also theoretically the performance of the algorithm.
Improved Algorithm for Graphical Bilinear Bandits
In this section, we want to capitalize on Algorithm 1 and its 1+ γ 2 -no-regret property to optimize the allocation of the arms x t and x ′ t such that the unwanted and suboptimal arms z x t x t and z x ′
t x ′ t penalize as less as possible the reward.
Indeed, in Algorithm 1, the choice of the couple ( x t , x ′ t ) is driven by the optimistic reward of the edge-arms z x t x ′ t and z x t x ′ t but not on the one of the unwanted arms z x t x t and z x ′ t x ′ t .
Here, the first improvement that we can provide is to include them in the selection of the couple ( x t , x ′ t ) as follows
[Equation]
However, this selection suggests that the number of unwanted edge-arms z x t x t and z x ′ t x ′ t will be equal to the number of optimal edge-arm z x t x ′ t and
z x ′ t x t (in the sense of optimism), which is in general not true.
Hence, one has to take into consideration the proportion of edges that will be allocated by suboptimal and unwanted edge-arms or by optimistically optimal edge-arms.
To do so, let us define m 1 (respectively m 2 ) the number of edges that goes from nodes in V(respectively V 2 ) to nodes in V 1 as well (respectively V 2 ) and m 1 → 2 (respectively m 2 → 1 ) the number of edges that goes from nodes in V 1 (respectively V 2 ) to nodes in V 2 (respectively V 1 ).
One can notice that by definition of the edge set E , we have m 1 → 2 = m 2 → 1 and that the total number of edges m = m 1 → 2 + m 2 → 1 + m 1 + m 2 .
By choosing the couple ( x t , x ′ t ) as follows
[Equation]
we are optimizing the total optimistic reward that one would obtain when allocating only two arms ( x, x ′ )
∈
X 2 in the graph.
This strategy is described in Algorithm 3.
[Equation]
Before stating the guarantees on the α -regret of this improved algorithm, let us recall that we defined ( x ⋆ , x ′ ⋆ )
= arg max
[Equation]
x,x ′ ) ∈X ⟨ z xx ′
+ z x ′ x , θ ⋆ ⟩ and let us define the couple (˜ x ⋆ , ˜ x ′ ⋆ ) such that
[Equation]
We define ∆ ≥ 0 to be the expected reward difference of allocating (˜ x ⋆ , ˜ x ′ ⋆ ) instead of ( x ⋆ , x ′ ⋆ ) ,
[Equation]
The new guarantees that we get on the α -regret of Algorithm 3 are stated in the following theorem
[Table]
Theorem 4.1.
Given the Graphical Bilinear Bandit problem defined as in Section 2, let γ be defined as in Theorem 3.2, let 0 ≤ ϵ ≤ 12 be a problem dependent parameter that measures the gain of optimizing on the suboptimal and unwanted arms defined as:
[Equation]
and set α = 1 − (cid:2) m 1+ m 2 m (1 − γ ) − ϵ (cid:3) where α ≥ 1 / 2 by construction, then the α -regret of
Algorithm 3 satisfies
[Equation]
where ˜ O hides the logarithmic factors.
Here one can see that the improvement happens in the α of the α -regret.
Let us analyze more precisely this term.
First of all, notice that in a complete graph (which is the worst case scenario in terms of graph type and guarantees of the cut), we get α ≥
+ γ
+ ϵ which shows that optimizing over the suboptimal arms already improves our bound by ϵ .
On the contrary, in the most favorable graphs, which are bipartite graphs ( i.e., graphs where all the m edges goes from nodes in V 1 to nodes in Vor vice versa), we have m 2 + m 1 = 0 and ϵ = 0 which gives α = 1 and makes the Algorithm 3 a no-regret algorithm.
What may also be of interest is to understand how α varies with respect to γ , ϵ and the quantity m 1 and m 2 for graphs that are between a complete graph and a bipartite graph.
We investigate experimentally this dependency in Section 5.
Experiments - Influence of the problem parameters on the regret
In this section, we give some insights on the problem-dependent parameters γ and ϵ and the corresponding α .
Let α
and α 2 be the α stated respectively in Theorem 3.2 and Theorem 4.1.
In the first experiment, we show the dependence of α 1 and α 2 on the graph type and the chosen approximation algorithm for the max-cut problem with respect to γ and ϵ .
We also highlight the differences between the two parameters α 1 and α 2 and the significant improvement in guarantees that one can obtain using Algorithm 3 depending on the type of the graph.
The results are presented in Table 1.
One can notice that the complete graph seems to be the one that gives the worst guarantee on the α -regret with respect to ϵ and γ .
Thus, we conducted a second experiment where we consider the worst case scenario in terms of the graph type – e.g., the complete graph– and where there is n =agents.
The second experiment studies the variation of ϵ and γ with respect to the unknown parameter matrix M ⋆ .
To design such experiments, we consider the arm-set X as the vectors ( e 1 , . .
. , e d ) of canonical base in R d , which implies by construction that the arm-set Z contains the vectors ( e 1 , . . .
, e d 2 ) of the canonical base in R d 2 .
We generate the matrix M ⋆ randomly in the following way: first, all elements of the matrix are drawn i.i.d.
from a standard normal distribution, and then we take the absolute value of each of these elements to ensure that the matrix only contains positive numbers.
The choice of the vectors of the canonical base as the arm-set allows us to modify the matrix M ⋆ and to illustrate in a simple way the dependence on γ and ϵ .
Consider the best couple ( i ⋆ , j ⋆ ) =
arg max ( i,j ) ∈
[Equation] [Equation]
z e
[Equation]
e j + z e j e i , θ ⋆ ⟩ , we want to see how the reward of the suboptimal
[Figure]
e e i⋆ , the closer ζ is to 1 the higher the reward of the unwanted arms.
The dimension d of the arm-set is 10 (which gives linear reward with unknown parameter θ ⋆ of dimension 100 ).
The plotted curve represents the average value of the parameters over 100 different matrices M ⋆ initiated randomly with positive values.
[Figure]
strategy in et al
We a of 5 nodes, we run the experiment ondifferent matrices as in Figure 1 with ζ = 0 and run it 10 different times to plot the average fraction of the edge-arms z e i⋆ e i⋆ and z e j⋆ e j⋆ impact the value of γ , ϵ and thus α .
Notice that the reward associated to the edge-arm z e i⋆ e i⋆ (respectively z e j⋆ e j⋆ ) is M ⋆i
⋆ i ⋆ (respectively M ⋆j ⋆ j ⋆ ).
Hence we define 0 ≤ ζ < 1 and set M ⋆i ⋆ i ⋆
=
M ⋆j ⋆
j ⋆ = ζ × 12 ( M ⋆i ⋆ j ⋆ + M ⋆j ⋆ i ⋆ ) .
We study the variation of γ , ϵ , α 1 and α 2 with respect to ζ .
The results are presented in Figure 1.
One can see that when the associated rewards of z e i⋆ e i⋆ and z e j⋆ e j⋆ are low (thus γ is low and ϵ high), Algorithm 3 gives a much better guarantees than Algorithm 1 since it will focus on other node-arms than e i ⋆ and e j ⋆ that will give a higher global reward.
Moreover, even when the unwanted edge-arm gives a high reward, the guarantees on the regret of Algorithm 3 are still stronger because it takes into consideration the quantities m 1 and m 2 of the constructed suboptimal edge-arms.
Finally, we design a last experiment that compares in practice the performance of Algorithm 1 and
Algorithm 3 with the Explore-Then-Commit algorithm by using the exploration strategy designed in
Rizk et al.
[2021] during the exploration phase, and by allocating the nodes in V 1 and V 2 with the best estimated couple ( x, x ′ ) =
arg max ( x,x ′ )
⟨ z xx ′
+ z x ′ x , ˆ θ t ⟩ during the commit phase.
Conclusion & discussions
In this paper, we presented the first regret-based algorithm for the stochastic graphical bilinear bandit problem with guarantees on the α -regret.
We showed that by exploiting the graph structure and the typology of the problem, one can both improve the performance in practice and have a better theoretical guarantee on its α -regret.
Also, we showed experimentally that our algorithm achieves a better performance than Explore-Then-Exploit on our synthetic datasets.
The method presented in this article can be extended in many ways.
First, one can consider cutting the graph into 3 or more pieces, which is equivalent to approximating the problem of a Max-k-Cut [Frieze and Jerrum, 1997] with k ≥ 3 .
With the knowledge of such a partition of nodes V 1 , . . .
, V k , one may want to look for a k -tuple of node-arms maximizing the optimistic allocated reward rather than a pair, therefore introducing an elegant tradeoff between the optimality of the solution and the computational complexity of the arms allocation.
One could also study this problem in the adversarial setting, in particular adapting adversarial linear bandit algorithms to our case.
Finally, our setting could be extended to the case where each agent has its own reward matrix.
A Proof of Proposition 3.
Proof.
Let consider the subgraph G ( i ) containing all the nodes that have been assigned to V 1 or V 2 at the end of iteration i of Algorithm 2.
Let us denote m
( i ) the number of edges in the graph G ( i ) .
At the first iteration, the algorithm chooses the node 1 , computes n 1 = 0 and n 2 = 0 , and then assigns node 1 to V 1 .
With only one node in G (1) , we have m (1) = 0 .
By denoting c ( i ) the number of additional cut edges induces by the assignment of node i at iteration i , we have
[Equation]
Indeed, at the end of iteration 1 , there is only one node assigned, hence the number of cut edges induced by this assignment is c (1) = 0 .
Suppose that (cid:80) pi
=1 c ( i ) ≥ m ( p )for a certain p ∈ { 1 , . . . , n − 1 } , let us prove that (cid:80) p +1 i
=1 c ( i ) ≥ m ( p +1) / 2 .
Indeed, at the iteration p + 1 , the algorithm chooses the node ( p + 1) and computes n 1 and n 2 .
Since n 1 represents the number of neighbors of the node ( p + 1) in V 1 , if the node p + 1 is added to V 2 , then 2 × n 1 edges would be cut (the factor 2 comes from the fact that between two nodes i and j , there are the edges ( i, j ) and ( j, i ) ).
Similarly, since n 2 represents the number of neighbors of the node ( p + 1) in V 2 , if the node ( p + 1) is added to V 1 , then 2 × n 2 edges would be cut.
Notice also that there is a total of 2 × n 1 + 2 × n 2 edges between the node ( p + 1) and the nodes in G ( p ) .
In the algorithm, the node ( p + 1) is added to V 1 or V 2 such that we cut the most edges, indeed one has c ( p
[Equation]
Hence,
[Equation] [Equation]
The number of edges that is added to the subgraph G ( p ) when adding the node ( p + 1) is equal to 2 n 1 + 2 n 2 = m ( p +1)
− m ( p ) , hence,
[Equation]
We have shown that (cid:80) 1 i =1 c ( i ) ≥ m (1) 2 and that if (cid:80) pi =1 c ( i ) ≥ m ( p ) 2 for a certain p ∈ { 1 , . . . , n − 1 } , then (cid:80) p +1 i
=1 c ( i ) ≥ m ( p +1).
Thus, (cid:80) pi =1 c ( i ) ≥ m ( p )for any p ∈ { 1 , . . . , n } , especially for p = n where G ( n ) = G .
By definition (cid:80) ni
=1 c ( i ) is the total number of edges that are cut which also means that n (cid:88) i =
[Equation]
B Proof of Theorem 3.2 and Theorem 4.
To properly derive the regret bounds, we will have to make connections between our setting and a standard linear bandit that chooses sequentially Tm arms.
For that matter, let us consider an arbitrary order on the set of edges E and denote E
[ i ] the i -th edge according to this order with i ∈ { 1 , . . .
, m } .
We define for all t ∈ { 1 , . . .
, T } and p ∈ { 1 , . . .
, m } the OLS estimator
[Equation]
where
[Equation]
with λ a regularization parameter and
[Equation]
We define also the confidence set
[Equation]
where with probability 1 − δ , we have that θ ⋆ ∈ C t,p ( δ ) for all t ∈ { 1 , . .
, T } , p ∈ { 1 , . . .
, m } and δ ∈ (0 , 1] .
Notice that the confidence set C t ( δ ) defined in Section 3 is exactly the confidence set C t,m ( δ ) defined here.
The definitions of the matrix
A t,m and the vector b t,m follow the same reasoning.
B.1 Proof of Theorem 3.
Proof.
Recall that ( x (1) ⋆ , . . .
, x ( n ) ⋆ ) =
arg max( x (1) ,...,x ( n ) )
(cid:80) ( i,j
) ∈
E x ( i )
⊤ M ⋆ x ( j ) is the optimal joint arm, and we define for each edge ( i,
j ) ∈
E the optimal edge arm z ( i,j ) ⋆
[Equation]
We recall that the α -pseudo-regret is
[Equation]
where the pseudo-regret R ( T ) is defined by
[Equation]
Let us borrow the notion of Critical Covariance Inequality introduced in [Chan et al., 2021], that is for a given round t ∈ { 1 , . .
, T } and p ∈ { 1 , . . .
, m } , the expected covariance matrix A t,p satisfies the critical covariance inequality if
[Equation]
Let us now define the event D t as the event where at a given round t , for all p ∈ { 1 , . . .
, m } , A t,p satisfies the critical covariance inequality (CCI).
We can write the pseudo-regret as follows:
[Equation]
We know that the approximation Max-CUT algorithm returns two subsets of nodes V 1 and V 2 such that there are at least m/ 2 edges between V 1 and V 2 , and to be more precise: at least m/ 4 edges from
V 1 to V 2 and at least m/ 4 edges from V 2 to V 1 .
Hence at each time t , if all the nodes of V 1 pull the node-arm x t and all the nodes of V 2 pull the node-arm x ′ t , we can derive the term ( a ) as follows:
[Equation]
] ⟨ z ( i,j ) t , θ ⋆ ⟩ − 1 [ i ∈ V 2 ∧ j ∈ V 1 ] ⟨ z ( i,j ) t , θ ⋆ ⟩ − 1 [ i ∈ V 1 ∧ j ∈ V 1 ] ⟨ z ( i,j ) t , θ ⋆ ⟩ − 1 [ i ∈ V
∧ j ∈ V 2 ] ⟨ z ( i,j ) t , θ ⋆ ⟩ .
Notice that (cid:80) ( i,
j ) ∈
E z ( i,j ) ⋆ = (cid:80) ( i,
j ) ∈
E 1 m (cid:80)
( k,l ) ∈
E z ( k,l ) ⋆ , so one has ( a ) =
T (cid:88) t =
[ D t ] (cid:88) ( i,
j ) ∈
E 1 [ i ∈ V 1 ∧ j ∈ V 2 ]   (cid:42)m (cid:88)
( k,l ) ∈
E z ( k,l ) ⋆ , θ ⋆ (cid:43) − ⟨ z ( i,j ) t , θ ⋆ ⟩ 
 (cid:124) (cid:123)(cid:122) (cid:125)
( a 1 )
+
T (cid:88) t =
[ D t ] (cid:88) ( i,
j ) ∈
E
[ i ∈ V 2 ∧ j ∈ V 1 ]   (cid:42)m (cid:88)
( k,l ) ∈
E z ( k,l ) ⋆ , θ ⋆ (cid:43) − ⟨ z ( i,j ) t , θ ⋆ ⟩ 
 (cid:124) (cid:123)(cid:122) (cid:125)
( a 2 )
+
T (cid:88) t =
[ D t ] (cid:88) ( i,
j ) ∈
E 1 [ i ∈ V 1 ∧ j ∈ V 1 ]   (cid:42)m (cid:88)
( k,l ) ∈
E z ( k,
[Equation]
Let us analyse the first term:
[Equation]
By defining ( x ⋆ , x ′ ⋆ ) =
arg max ( x,x ′ ) ∈X 2 ⟨ z xx ′
+ z x ′ x , θ ⋆ ⟩ , and noticing that in the case where a node i is in V 1 and a neighbouring node j in is V 2 , then z
( i,j ) t
[Equation]
x t x
[Equation]
t , we have,
[Equation]
Plugging this last inequality in (13) yields, with probability 1 − δ , ( a
[Equation]
≤
T (cid:88)
[Equation]
We define, as in Algorithm 1,
1 (cid:104) z ( i,j ) t =
z x t x ′ t
[Equation]
1 [ i ∈ V 1 ∧ j ∈ V 2 ] .
Then, one has, with probability 1 − δ ,
[Equation]
with (cid:112) β
[Equation]
δ
[Equation]
(cid:114)
[Equation]
(cid:17) + √ λS and where (14) uses the critical covariance inequality (12), (15) comes from the definition of the confidence set C t − 1 ,m ( δ ) (9) and (16) upper bounds the indicator functions by 1.
Using a similar reasoning, we obtain the same bound for ( a 2 ) :
[Equation]
Let us bound the terms ( a 3 ) and ( a 4 ) .
[Equation]
For all x ∈ X , let γ x be the following ratio
[Equation]
and let γ be the worst ratio
[Equation]
We have
[Equation]
where (21) comes from the fact that there is at most m/ 4 edges that goes from node in V 1 to other nodes in V 1 and that 1 [ D t ] ≤ 1 for all t .
The derivation of this bound for ( a 3 ) gives the same one for ( a 4 )
[Equation]
[Equation]
By rewriting ( a ) , we have :
[Equation]
In [Chan et al., 2021]
[Equation]
they bounded the term ( b ) as follows
[Equation]
We thus have the regret bounded by
R ( T
[Equation]
≤
[Equation]
which gives us
[Equation]
Let us bound the first term with the double sum as it is done in [Abbasi-Yadkori et al., 2011, Chan et al., 2021]:
[Equation]
where (24) uses the fact that for all a, x ≥ 0 , min( a, x ) ≤ max(2 , a ) log(1 + x ) , (25) uses the fact that (cid:80) Tt =
[Equation]
=
[Equation]
log
[Equation]
from Lemma 19.4 in Lattimore and Szepesvári [2018].
The final bound for the 1+ γ 2 -regret is
[Equation]
δ
[Equation]
B.2 Proof of Theorem 4.
Proof.
For the sake of completeness in the proof we recall that we defined the couples ( x ⋆ , x ′ ⋆ ) and (˜ x ⋆ , ˜ x ′ ⋆ ) and the quantity ∆ as follows:
[Equation]
and
[Equation]
And we recall that in Algorithm
[Equation]
the tuple ( x t , x ′ t , ˜ θ t − 1 ,m ) is obtained as follows:
[Equation]
We can write the regret R ( T ) as in the proof of Theorem
[Equation]
[Equation]
Here, ( b ) doesn’t change, we thus only focus on deriving ( a ) .
[Equation]
j
[Equation]
(where 1 [ D t ] ≤ 1 ) =
[Equation]
We have
[Equation]
By plugging the last upper bound in ( a ) and with probability 1 − δ
[Equation]
we have,
[Equation] [Equation]
) ⋆ , θ ⋆ ⟩ ≤
T (cid:88) t = (cid:88) ( i,j
) ∈ E ⟨ z ( i,j ) t , ˜ θ t − 1 ,m − θ ⋆ ⟩ −
T (cid:88) t = ∆ −
T (cid:88) t = (cid:88) ( i,
j ) ∈
E m 1 + mm γ ⟨ z ( i,j ) ⋆ , θ ⋆ ⟩ +
T (cid:88) t = (cid:88) ( i,
j ) ∈
E m
+ mm ⟨ z ( i,j ) ⋆ , θ ⋆ ⟩ =
T (cid:88) t = (cid:88) ( i,j
) ∈ E ⟨ z ( i,j ) t , ˜ θ t − 1 ,m − θ ⋆ ⟩ −
T (cid:88) t = ∆ +
T (cid:88) t = (cid:88) ( i,
j ) ∈
E m 1 + mm (1 − γ ) ⟨ z ( i,j ) ⋆ , θ ⋆ ⟩ =
T (cid:88) t = (cid:88) ( i,j
) ∈ E ⟨ z ( i,j ) t , ˜ θ t − 1 ,m − θ ⋆ ⟩ −
T (cid:88) t = (cid:88) ( i,
j ) ∈
E ϵ ⟨ z
( i,j ) ⋆ , θ ⋆ ⟩ +
T (cid:88) t = (cid:88) ( i,
j ) ∈
E m 1 + mm (1 − γ ) ⟨ z ( i,j ) ⋆ , θ ⋆ ⟩ =
T (cid:88) t = (cid:88) ( i,j
) ∈ E ⟨ z ( i,j ) t , ˜ θ t − 1 ,m − θ ⋆ ⟩ +
T (cid:88) t = (cid:88) ( i,j
) ∈ E (cid:20) m 1 + m
[Equation]
θ
[Equation]
By plugging ( a ) in the regret and with probability 1 − δ , we have,
R
[Equation]
which gives,
[Equation]
The upper bound of the right hand term follows exactly what we have already done for Theorem 3.
by applying the upper bounds (16) and (23)
C Additional information on the experiments C.
The number of nodes in each graph is equal to 100 .
The random graph corresponds to a graph where for two nodes i and j in V , the probability that ( i, j ) and ( j, i ) is in E is equal to 0 .
The results for the random graph are averaged over 100 draws.
The matching graph represents the graph where each node i ∈ V has only one neighbour: Card( N i )
= 1 .
C.2 Figure
The graph used in this experiment is a complete graph of 10 nodes.
The arm set X
= { e 1 , . .
. , e d } which gives Z = { e 1 , . .
, e d 2 } .
The matrix M ⋆ is randomly initialized such that all elements of the matrix are drawn i.i.d.
from a standard normal distribution, and then we take the absolute value of each of these elements to ensure that the matrix only contains positive numbers.
We plotted the results by varying ζ from 0 to 1 with a step of 0 .
We conducted the experiment on 100 different matrices M ⋆ randomly initialized as explained above and plotted the average value of the obtained γ , ϵ , α 1 and α 2 .
C.3 Figure
For the last experiment, we used a complete graph of 5 nodes.
The arm set X
= { e 1 , . .
. , e d } which gives Z = { e 1 , . .
, e d 2 } .
The matrix M ⋆ is randomly initialized as explained in the previous experiment.
We fixed ζ = 0 and the horizon T = 20000 .
We ran the experiment 10 times and plotted the average values (shaded curve) and the moving average curve with a window of 100 steps for more clarity.
The Explore-Then-Commit algorithm has an exploration phase of T/ 3 rounds and then exploits by pulling the couple ( x t , x ′ t ) =
arg max ( x,x ′ )
⟨ z xx ′
+ z x ′ x , ˆ θ t ⟩ .
Note that we set the exploration phase to T/ 3 because most of the time, it was sufficient for the learner to have the estimated optimal pair ( x t , x ′ t ) equal to the real optimal pair ( x ⋆ , x ′ ⋆ ) .
Machine used for all the experiments : Macbook
[Table]
The code is available here
[Table]