An ↵ -No-Regret Algorithm For Graphical Bilinear Bandits
Abstract
We propose the ﬁrst regret-based approach to the Graphical Bilinear Bandits problem, where n agents in a graph play a stochastic bilinear bandit game with each of their neighbors.
This setting reveals a combinatorial NP-hard problem that prevents the use of any existing regret-based algorithm in the (bi-)linear bandit literature.
In this paper, we ﬁll this gap and present the ﬁrst regret-based algorithm for graphical bilinear bandits using the principle of optimism in the face of uncertainty.
Theoretical analysis of this new method yields an upper bound of ˜
O ( p T ) on the ↵ -regret and evidences the impact of the graph structure on the rate of convergence.
Finally, we show through various experiments the validity of our approach.
Introduction
In this paper, we are interested in solving centralized multi-agent problems that involve interactions between agents.
For instance, consider the problem of the conﬁguration of antennas in a wireless cellular network [Siomina et al., 2006] or the adjustment of turbine blades in a wind farm [van Dijk et al., 2016, Bargiacchi et al., 2018].
Choosing a parameter for an antenna (respectively adjusting a turbine blade) has an impact on its signal quality (respectively its energy collection efﬁciency) but also on the one of its neighboring antennas due to signal interference (respectively its neighboring turbine blade due to wind turbulence).
By considering each antenna or turbine blade as an agent, these problems can be modeled as a multi-agent multi-armed bandit problem [Bargiacchi et al., 2018] with the knowledge of a coordination graph [Guestrin et al., 2002] where each node represents an agent and each edge represents an interaction between two agents.
As a natural extension of the well-studied linear bandit to this multi-agent setting we focus on the graphical bilinear bandit setting introduced in Rizk et al.
[2021]: at each round, a central entity chooses an action for each agent and observes a bilinear reward for each couple of neighbors in the graph that is function of the actions chosen by the two neighboring agents.
The objective of the learner can either be to maximize the cumulative reward, which requires a trade-off between exploration and exploitation , or to identify the best action without being concerned about suboptimal rewards obtained during the learning process, known as the pure exploration setting [Bubeck et al., 2009, Audibert and Bubeck, 2010].
For graphical bilinear bandits, while the latter has already been addressed in Rizk et al.
[2021], maximizing the cumulative reward (and thus minimizing the associated regret) remains an open question that we address in this paper.
Contribution.
Given the NP-Hardness property of the problem and the impossibility of designing an algorithm that has sublinear regret with respect to the horizon T , we propose the ﬁrst regret-based algorithm for the graphical bilinear bandit problem that gives a guarantee on the ↵ -regret of ˜ O ( p T ) where ↵ (greater than 1 / 2 ) is a problem-dependent parameter and ˜
O hides the logarithmic factors.
The problem as well as the notion of ↵ -regret are introduced in Section 2.
We provide a ﬁrst algorithm and establish theoretical guarantees in Section 3.
In Section 4, we identify the problem-dependent parameters that one could use to improve both theoretical guarantees and empirical performances of the algorithm.
Accordingly, we state a reﬁned bound that signiﬁcantly improves ↵ .
Finally, Sectiongives an experimental overview of the various theoretical bounds established for the regret.
Related work.
Centralized multi-agent bandit problems where the learner has to choose the actions of all the agents at each round implies to deal with parallelizing the learning process on the agents.
In the context of linear rewards where all the agents share the same reward function, Chan et al.
[2021] give a detailed analysis of the problem and show that a sublinear regret in T can be reached but with an additional cost speciﬁc to the parallelization.
The main difference with their work is that they consider independent agents, whereas we assume interactions between the agents.
Choosing an action for each node of a graph to then observe a global reward over the graph is a combinatorial bandit problem [Cesa-Bianchi and Lugosi, 2012], the number of joint arms scaling exponentially with the number of nodes.
This has been studied in the regret-based context [Chen et al., 2013, Perrault et al., 2020] where the notion of ↵ -regret is introduced to cope with the NP-Hardness of the problem.
Most papers assume the availability of an oracle to solve the inner combinatorial optimization problem at each round.
In this paper, we do not make this assumption and we present an algorithm that takes into account the graph structure, which has not been considered in the combinatorial framework.
Bandit problems in graphs have been studied from several angles.
For instance, on the one hand, in
Valko et al.
[2014], Mannor and Shamir [2011], each node of the graph is an arm of the bandit problem, which, if pulled, gives information on its neighboring arms.
On the other hand, in Cesa-Bianchi et al.
[2013] each node is an instance of a linear bandit and the neighboring nodes are assumed to have similar unknown regression coefﬁcients.
In this setting, the reward observed for each agent only depends on its chosen action which does not take into consideration agent interactions.
The graphical bilinear bandit setting has been ﬁrst established in Rizk et al.
[2021] where they tackle the problem of identifying the 1 / 2 -best joint arm (which has an associated global reward equal to at least 1 / 2 of the optimal one).
They study the pure exploration setting while we tackle the regret-based one.
To the best of our knowledge, our paper is the ﬁrst one to provide regret-based algorithms for the graphical bilinear bandits setting.
Problem setting
Let G =
( V, E ) be the directed graph deﬁned by V the ﬁnite set of nodes representing the agents and
E the set of edges representing the agent interactions.
We assume that if ( i, j ) 2 E then ( j, i ) 2 E .
The Graphical Bilinear Bandit with a graph G consists in the following sequential problem [Rizk et al., 2021]: at each round t , each agent i 2 V chooses an arm x ( i ) t in a ﬁnite arm set X and receives for each of its neighbors j a bilinear reward
[Equation]
where M ? 2 R d ⇥ d is an unknown matrix, and ⌘ ( i,j ) t a zero-mean  -sub-Gaussian random variable.
For all agent i 2 V , we denote N i the set of its neighboring agents.
Let n = Card ( V ) denote the number of nodes, m = Card ( E ) the number of edges and K = Card ( X ) the number of arms.
We assume that for all x 2 X , k x k 2  L .
Let k M ? k F be the Frobenius norm of M ? with k M ? k F  S .
We assume that the expected reward of each edge is positive which gives for all ( x, x 0 ) 2 X 2 , 0  x > M ? x 0  LS .
For each round t , we deﬁne the global reward y t
[Equation]
as the sum of the local rewards obtained over the graph at round t , and
[Equation] arg max( x (1) ,...,x ( n ) )
P ( i [Equation] j ) [Equation]
E x ( i ) >
M ? x ( j ) as the optimal joint arm that maximizes the expected global reward.
The corresponding global pseudo-regret over T rounds is then deﬁned as follows:
[Equation]
We know from Rizk et al.
[2021] that ﬁnding [Equation] x [Equation]
n [Equation] ⌘ is NP-Hard with respect to the number of agents n .
Proposition 2.1 (Rizk et al.
[2021] Theorem 4.1) .
Consider a given matrix M ? 2 R d ⇥ d and a ﬁnite arm set X ⇢ R d .
Unless P=NP, there is no polynomial time algorithm guaranteed to ﬁnd the optimal solution of
[Equation]
Hence, the objective of designing an algorithm with a sublinear regret in T is not feasible in polynomial time with respect to n .
However, some NP-hard problems are ↵ -approximable (for some ↵ 2 (0 , 1] ), which means that there exists a polynomial-time algorithm guaranteed to produce solutions whose values are at least ↵ times the value of the optimal solution.
For these problems, it makes sense to consider the ↵ -pseudo-regret as in [Chen et al., 2013] which is deﬁned for all ↵ 2 (0 , 1] as follows
[Equation]
and set the objective of designing an algorithm with a sublinear ↵ -regret.
The linear form.
The reward can either be viewed as a bilinear reward as shown in (1) or as a linear reward in higher dimension [Jun et al., 2019] with y ( i,j ) t
[Equation]
where for any matrix A 2 R d ⇥ d , vec ( A ) denotes the vector in R d 2 which is the concatenation of all the columns of A, and h · , · i refers to the dot product between two vectors.
To simplify the notation, let us refer to any x 2 X as a node-arm and deﬁne the arm set Z = { vec ( xx 0> ) | ( x, x 0 ) 2 X} where any z 2 Z will be referred as an edge-arm .
If the arm x ( i ) t 2 X represents the node-arm allocated to the node i 2 V at time t , for each edge ( i, j ) 2 E we will denote the associated edge-arm by z ( i,j ) t := vec ⇣ x ( i ) t x ( j ) > t ⌘ 2 Z and deﬁne ✓ ?
= vec ( M ? )
the vectorized version of the unknown matrix M ? with k ✓ ? k 2  S .
With those notations, the (now) linear reward can be rewritten as follows:
[Equation]
When needed, for any node-arms ( x, x 0 ) 2 X 2 we use the abbreviation z xx 0 , vec( xx 0> ) .
Finally for any vector x 2 R d and a symmetric positive-deﬁnite matrix A 2 R d ⇥ d , we deﬁne k x k A , p x > A x .
Optimism in the face of uncertainty for Graphical Bilinear Bandit
Let us ﬁrst notice that given the linear reward (2) that can be observed for each of the m edges ( i, j ) 2 E , the central entity is trying to solve m parallel and dependent linear bandit problems.
The parallel aspect comes from the fact that at each round t the learner has to allocate all the m edge-arms of the graph (one for each edge ( i, j ) in E ) before receiving the corresponding m linear rewards.
The dependent aspect comes from the fact that the choice of most of the edge-arms are depending on other edge-arms.
Indeed, two adjacency edges ( i, j ) and ( j, k ) share the node j , hence their allocated edge-arms must share the node-arm x ( j ) t .
In this paper, we choose to design an algorithm based on the principle of optimism in the face of uncertainty Auer et al.
[2002], and in the case of a linear reward Li et al.
[2010], Abbasi-Yadkori et al.
[2011], we need to maintain an estimator of the true parameter ✓ ? .
To do so, let us deﬁne for all rounds t 2 { 1 , . .
, T } the OLS estimator of ✓ ? as follows:
[Equation]
where,
[Equation]
, with  > 0 a regularization parameter and
[Equation]
We deﬁne also the conﬁdence set
[Equation]
where with probability 1   , we have that ✓ ? 2 C t (  ) for all t 2 { 1 , . .
. , T } , and  2 (0 , 1] .
Since the graphical bilinear bandit can be seen as a linear bandit in dimension d 2 and with K n possible arms, 3 the idea of our method is to overcome the NP-Hard optimization problem that one would have to solve at each round when directly applying the OFUL algorithm as deﬁned in Abbasi-Yadkori et al.
Indeed, at each round, instead of looking for the best optimistic joint arm
[Equation] [Equation]
E
Input : graph G =
( V, E ) ,
node-arm set X ( V 1 , V 2 ) =
Approx-MAX-CUT ( G ) for t = 1 to T do // Find the optimistic best couple of node-arms ⇣ x t , x 0 t , ˜
[Equation]
⌘ = arg max ( x,x 0 , ✓ ) 2 X 2 ⇥ C t  1 h
{+z xx 
 + z x 0 x , ✓ i ; // Allocate x t and x 0 t in the graph x ( i ) t = x t for all i in V 1 ; x ( i ) t = x 0 t for all+} i in V 2 ;
Obtain for all ( i, j ) in E the rewards y ( i,j ) t ;
Compute ˆ ✓ t as in (3) end return ˆ ✓ t
Algorithm 2:
Input : G =
( V, E )
Set V 1 = ; , V 2 = ; for i in V do n 1 = Card ( { ( i, j ) 2 E |
j 2 V 1 } )
; n 2 = Card ( { ( i, j ) 2 E | j 2 V 2 } ) ; if n 1 > n 2 then V V 2 [ { i } else V V 1 [ { i } ; end return ( V 1 , V 2 ) we consider the couple of node-arms ( x ? , x 0 ? ) =
arg max ( x,x 0 ) 2 X h z xx {+
+}
+ z x 0 x , ✓ ? i maximizing the reward along one edge.
Given the optimal joint arm ( x (1) ?
, x ( n ) ? ) and the associated optimal edge-arms z ( i,j ) ? = vec ( x ( i ) ? x ( j ) ? )
for all ( i, j ) 2 E , we have [Equation]
[Equation]
Hence, an alternative objective is to construct as many edge-arms z x ? x 0 ? and z x 0 ?
x ? as possible in the graph.
Naturally, we do not have access to ✓ ?
, so we use the principle of optimism in the face of uncertainty, which is to ﬁnd the couple ( x t , x 0 t ) such that
[Equation]
and then allocate the node-arms to maximize the number of locally-optimal edge-arms z x t x 0 t and z x 0 t x t .
The motivation of this alternative goal can be understood thanks to (4) where we know that the associated expected reward is better than the one obtained with the edge-arms created with the optimal joint arms.
Of course, one can also easily understand that for a general type of graph it is not possible to allocate those edge-arms everywhere ( i.e., to all the edges).
For example take a complete graph of three nodes, allocating the edge-arms z x t x 0 t and z x 0 t x t is equivalent to allocate x t and x 0 t to the nodes.
By doing so, two nodes will pull x t and the third one will pull x 0 t , which makes it inevitable to allocate sub-optimal and unwanted edge-arms of the form z x t x t .
The main concern with this method is to control how many unwanted edge-arms are drawn in each round (relative to the total number m of edges) in order to minimize their impact on the resulting regret.
Assigning x t to a subset of nodes and x 0 t to the complementary is equivalent to cutting the graph into two pieces and creating two distinct sets of nodes V 1 and V 2 such that V = V 1 [ V 2 and
V 1 \ V 2 = ; .
Pulling the maximum amount of optimal edge-arms thus boils down to ﬁnding a cut passing through the maximum number of edges.
This problem is known as the Max-Cut problem, which is also NP-Hard.
However, the extensive attention this problem has received allows us to use one of the many approximation algorithms (see, e.g., Algorithm 2) which are guaranteed to yield a cut passing through at least a given fraction of the edges in the graph.
Proposition 3.1.
Given a graph G =
( V, E ) , Algorithm 2 returns a couple ( V 1 , V 2 ) such that
[Equation]
Using Algorithm 2 we can thus divide the set of nodes V into two subsets V 1 and V 2 with the guarantee that at least m/ 2 edges connect the two subsets.
Combining this algorithm with the principle of optimism in the face of uncertainty for linear bandits Abbasi-Yadkori et al.
[2011] we obtain Algorithm 1 for which we can state the following guarantees on the ↵ -regret.
Theorem 3.2.
Given the Graphical Bilinear Bandit problem deﬁned in Section 2, let 0    1 be a problem-dependent parameter deﬁned by
[Equation]
and set ↵ = 1+  2 , then the ↵ -regret of Algorithm 1 satisﬁes
[Equation]
where ˜ O hides the logarithmic factors.
One can notice that the ﬁrst term of the regret-bound matches the one of a standard linear bandit that pulls sequentially Tm edge-arms.
The second term captures the cost of parallelizing m draws of edge-arms per round.
Indeed, the intuition behind this term is that the couple ( x t , x 0 t ) chosen at round t (and thus after having already pulled tm edge-arms and received tm rewards) is relevant to pull the ( tm + 1) -th edge-arm but not necessarily the ( m  1) edge-arms that follows (from the ( tm + 2) -th to the ( tm + m ) -th one) since the reward associated to the ( tm + 1) -th
edge-arms could have led to change the central entity choice if it had been done sequentially.
In Chan et al.
[2021], they characterize this phenomenon and show that this potential change in choice occurs less and less often as we pull arms and get rewards, hence the dependence in O (log( Tm ))) .
What is  and what value can we expect?
This parameter measures what minimum gain with respect to the optimal reward one could get by constructing the edge-arms of the form z xx .
For example, if there exists x 0 2 X such that h z x 0 x 0 , ✓ ? i = 0 , then  = 0 as well and we are in the worst case scenario where we can only have a guarantee on an ↵ -regret with ↵ = 1 / 2 .
In practice, having  =is reached when given the couple ( x ? , x 0 ? ) =
arg max ( x,x 0 ) 2 X h z xx {+
+}
+ z x 0 x , ✓ ? i , this x 0 is either x ? or x 0 ? .
In other words, if the unwanted edge-arms associated to the couple ( x ? , x 0 ? ) gives low rewards, then the guarantee on the regret will be badly impacted.
Hence we can wonder how we can prevent this phenomenon.
We answer this question in the next section by taking into account both the proportion of undesirable edge arms allocated and their potential rewards at the selection of the pair ( x t , x 0 t ) in order to improve in practice but also theoretically the performance of the algorithm.
Improved Algorithm for Graphical Bilinear Bandits
In this section, we want to capitalize on Algorithm 1 and its 1+  2 -no-regret property to optimize the allocation of the arms x t and x 0 t such that the unwanted and suboptimal arms z x t x t and z x 0 t x 0 t penalize as less as possible the reward.
respect to the optimal Max-Cut solution, which is not exactly the guarantee we are looking for: we need a guarantee as a proportion of the total number of edges. We thus have to be careful on the algorithm we choose.
Indeed, in Algorithm 1, the choice of the couple ( x t , x 0 t ) is driven by the optimistic reward of the edge-arms z x t x 0 t and z x t x 0 t but not on the one of the unwanted arms z x t x t and z x 0 t x 0 t .
Here, the ﬁrst improvement that we can provide is to include them in the selection of the couple ( x t , x 0 t ) as follows
[Equation]
However, this selection suggests that the number of unwanted edge-arms z x t x t and z x 0 t x 0 t will be equal to the number of optimal edge-arm z x t x 0 t and z x 0 t x t (in the sense of optimism), which is in general not true.
Hence, one has to take into consideration the proportion of edges that will be allocated by suboptimal and unwanted edge-arms or by optimistically optimal edge-arms.
To do so, let us deﬁne m 1 (respectively m 2 ) the number of edges that goes from nodes in V(respectively V 2 ) to nodes in V 1 as well (respectively V 2 ) and m 1 ! 2 (respectively m 2 ! 1 ) the number of edges that goes from nodes in V 1 (respectively V 2 ) to nodes in V 2 (respectively V 1 ).
One can notice that by deﬁnition of the edge set E , we have m 1 {+! 
+} = m 2 ! 1 and that the total number of edges m = m 1 ! 2 + m 2 {+! 
+} + m 1 + m 2 .
By choosing the couple ( x t , x 0 t ) as follows
[Equation]
we are optimizing the total optimistic reward that one would obtain when allocating only two arms ( x, x 0 ) 2 X 2 in the graph.
This strategy is described in Algorithm 3.
[Equation]
[Equation] [Equation] [Equation] Before stating the guarantees on the ↵ -regret of this improved algorithm, let us recall that we deﬁned ( x ? , x 0 ? )
= arg max
[Equation]
x,x 0 ) 2 X h z xx {+
+}
+ z x 0 x , ✓ ? i and let us deﬁne the couple (˜ x ? , ˜ x 0 ? ) such that
[Equation]
We deﬁne   0 to be the expected reward difference of allocating (˜ x ? , ˜ x 0 ? ) instead of ( x ? , x 0 ? ) , [Equation] =
[Equation]
The new guarantees that we get on the ↵ -regret of Algorithm 3 are stated in the following theorem
[Table]
Theorem 4.1.
Given the Graphical Bilinear Bandit problem deﬁned as in Section 2, let  be deﬁned as in Theorem 3.2, let 0  ✏  12 be a problem dependent parameter that measures the gain of optimizing on the suboptimal and unwanted arms deﬁned as:
[Equation]
and set ↵ = 1  [Equation] ⇤ where ↵  1 / 2 by construction, then the ↵ -regret of
Algorithm 3 satisﬁes
[Equation]
where ˜ O hides the logarithmic factors.
Here one can see that the improvement happens in the ↵ of the ↵ -regret.
Let us analyze more precisely this term.
First of all, notice that in a complete graph (which is the worst case scenario in terms of graph type and guarantees of the cut), we get [-α ≥ 
 + γ 
-]↵  1+  2 + ✏ which shows that optimizing over the suboptimal arms already improves our bound by ✏ .
On the contrary, in the most favorable graphs, which are bipartite graphs ( i.e., graphs where all the m edges goes from nodes in V 1 to nodes in Vor vice versa), we have m 2 + m 1 = 0 and ✏ = 0 which gives ↵ = 1 and makes the Algorithm 3 a no-regret algorithm.
What may also be of interest is to understand how ↵ varies with respect to  , ✏ and the quantity m 1 and m 2 for graphs that are between a complete graph and a bipartite graph.
We investigate experimentally this dependency in Section 5.
Experiments - Inﬂuence of the problem parameters on the regret
In this section, we give some insights on the problem-dependent parameters  and ✏ and the corresponding ↵ .
Let [-α 
-]↵ 1 and ↵ 2 be the ↵ stated respectively in Theorem 3.2 and Theorem 4.1.
In the ﬁrst experiment, we show the dependence of ↵ 1 and ↵ 2 on the graph type and the chosen approximation algorithm for the max-cut problem with respect to  and ✏ .
We also highlight the differences between the two parameters ↵ 1 and ↵ 2 and the signiﬁcant improvement in guarantees that one can obtain using Algorithm 3 depending on the type of the graph.
The results are presented in Table 1.
One can notice that the complete graph seems to be the one that gives the worst guarantee on the ↵ -regret with respect to ✏ and  .
Thus, we conducted a second experiment where we consider the worst case scenario in terms of the graph type – e.g., the complete graph– and where there is n =agents.
The second experiment studies the variation of ✏ and  with respect to the unknown parameter matrix M ? .
To design such experiments, we consider the arm-set X as the vectors ( e 1 , . .
. , e d ) of canonical base in R d , which implies by construction that the arm-set Z contains the vectors ( e 1 , . . .
, e d 2 ) of the canonical base in R d 2 .
We generate the matrix M ? randomly in the following way: ﬁrst, all elements of the matrix are drawn i.i.d.
from a standard normal distribution, and then we take the absolute value of each of these elements to ensure that the matrix only contains positive numbers.
The choice of the vectors of the canonical base as the arm-set allows us to modify the matrix M ? and to illustrate in a simple way the dependence on  and ✏ .
Consider the best couple
[Equation]
e j + z e j e i , ✓ ?
i , we want to see how the reward of the suboptimal at each round by applying the Algorithm 1, Algorithm 3 and the Explore-Then-commit algorithm (here named GBB-BAI) using the exploration strategy in Rizk et al.
We use a complete graph of 5 nodes, we run the experiment ondifferent matrices as in Figure 1 with ⇣ = 0 and run it 10 different times to plot the average fraction of the global reward edge-arms z e i ?
e i ?
and z e j ?
e j ? impact the value of  , ✏ and thus ↵ .
Notice that the reward associated to the edge-arm z e i ? e i ? (respectively z e j ? e j ? )
is M ?
i ?
i ? (respectively M ? j ? j ? ).
Hence we deﬁne 0  ⇣ < 1 and set M ? i ? i ? = M ?
j ?
j ?
=
⇣ ⇥ 12 ( M ? i ? j ? + M ?
j ? i ? ) . We study the variation of  , ✏ , ↵ 1 and ↵ 2 with respect to ⇣ .
The results are presented in Figure 1.
One can see that when the associated rewards of z e i ? e i ? and z e j ? e j ? are low (thus  is low and ✏ high), Algorithm 3 gives a much better guarantees than Algorithm 1 since it will focus on other node-arms than e i ?
and e j ? that will give a higher global reward.
Moreover, even when the unwanted edge-arm gives a high reward, the guarantees on the regret of Algorithm 3 are still stronger because it takes into consideration the quantities m 1 and m 2 of the constructed suboptimal edge-arms.
Finally, we design a last experiment that compares in practice the performance of Algorithm 1 and
Algorithm 3 with the Explore-Then-Commit algorithm by using the exploration strategy designed in
Rizk et al.
[2021] during the exploration phase, and by allocating the nodes in V 1 and V 2 with the best estimated couple ( x, x 0 ) =
arg max ( x,x 0 )
h z xx {+
+}
+ z x 0 x , ˆ ✓ t i during the commit phase.
Conclusion & discussions
In this paper, we presented the ﬁrst regret-based algorithm for the stochastic graphical bilinear bandit problem with guarantees on the ↵ -regret.
We showed that by exploiting the graph structure and the typology of the problem, one can both improve the performance in practice and have a better theoretical guarantee on its ↵ -regret.
Also, we showed experimentally that our algorithm achieves a better performance than Explore-Then-Exploit on our synthetic datasets.
The method presented in this article can be extended in many ways.
First, one can consider cutting the graph into 3 or more pieces, which is equivalent to approximating the problem of a Max-k-Cut [Frieze and Jerrum, 1997] with k  3 .
With the knowledge of such a partition of nodes V 1 , . . .
, V k , one may want to look for a k -tuple of node-arms maximizing the optimistic allocated reward rather than a pair, therefore introducing an elegant tradeoff between the optimality of the solution and the computational complexity of the arms allocation.
One could also study this problem in the adversarial setting, in particular adapting adversarial linear bandit algorithms to our case.
Finally, our setting could be extended to the case where each agent has its own reward matrix.
[-] ⟨ z ( i,j ) t , θ ⋆ ⟩ − 1 [ i ∈ V 2 ∧ j ∈ V 1 ] ⟨ z ( i,j ) t , θ ⋆ ⟩ − 1 [ i ∈ V 1 ∧ j ∈ V 1 ] ⟨ z ( i,j ) t , θ ⋆ ⟩ − 1 [ i ∈ V 
-]
[-T (cid:88) t = 
-]
[-T (cid:88) t = 
-]
[-E 
-]
[-T (cid:88) t = 
-]
[-E m 
-]