Learning Distributed and Fair Policies for Network Load Balancing as Markov Potential Game
Anonymous Author(s)
AfﬁliationAddress email
Abstract
This paper investigates the network load balancing problem in data centers (DCs)where multiple load balancers (LBs) are deployed, using the multi-agent reinforcement learning (MARL) framework.
The challenges of this problem consistof the heterogeneous processing architecture and dynamic environments, as wellas limited and partial observability of each LB agent in distributed networkingsystems, which can largely degrade the performance of in-production load balancing algorithms in real-world setups.
Centralised-training-decentralised-execution(CTDE)
RL scheme has been proposed to improve MARL performance, yet itincurs – especially in distributed networking systems, which prefer distributedand plug-and-play design scheme – additional communication and managementoverhead among agents.
We formulate the multi-agent load balancing problem as a Markov potential game, with a carefully and properly designed workloaddistribution fairness as the potential function.
A fully distributed MARL algorithmis proposed to approximate the Nash equilibrium of the game.
Experimental evaluations involve both an event-driven simulator and real-world system, where theproposed MARL load balancing algorithm shows close-to-optimal performance insimulations, and superior results over in-production LBs in the real-world system.
Introduction
In cloud data centers (DCs) and distributed networking systems servers are deployed on infrastructures with multiple processors to provide scalable services [1].
To optimise workload distributionand reduce additional queuing delay, load balancers (LBs) play a signiﬁcant role in such systems.
State-of-the-art network LBs rely on heuristic mechanisms [2–5] under the low-latency and highthroughput constraints of the data plane.
However, these heuristics are not adaptive to dynamicenvironments and require human interventions, which can lead to most painful mistakes in thecloud – mis-conﬁgurations.
RL approaches have shown performance gains in distributed system andnetworking problems [6–9], yet applying RL on the network load balancing problem is challenging.
First, unlike traditional workload distribution or task scheduling problem [6,7], network LBs havelimited observations over the system, including task sizes and actual server load states.
Being awareof only the number of tasks they have distributed, servers can be overloaded by collided elephant tasks and have degraded quality of service (QoS).
Second, to guarantee high service availability in the cloud, multiple LBs are deployed in DCs.
Network trafﬁc is split among all LBs.
This multi-agent setup makes LBs have only partial observation over the system.
Third, modern DCs are based onheterogeneous hardware and elastic infrastructures [10], where server capacities vary.
It is challenging to assign correct weights to servers according to their actual processing capacities, and this process conventionally requires human intervention – which can lead to error-prone conﬁgurations [3,5].
Lastbut not least, given the low-latency and high-throughput constraints in the distributed networkingsetup, the interactive training procedure of RL models and the centralised-training-decentralisedexecution (CTDE) scheme [11] can incur additional communication and management overhead.
In this paper, we study the network load balancing problem in multi-agent game theoretical approach,by formulating it as a Markov potential game through specifying the proper reward function, namelyvariance-based fairness.
We propose a distributed Multi-Agent RL (MARL) network load balancingmechanism that is able to exploit asynchronous actions based only on local observations and inferences.
Load balancing performance gains are evaluated based on both event-based simulations andreal-world experiments 1 .
[Table]
= [Table]
nsio5Ex4yiwnRHFoVn0puJ
[Table]
kLzRnKMeWUKaFvZWwIdWUoU2nZEPwFl9eJs2zqndR9e7OK7XrPI4iHMExnIIHl1CDW6hDAxgM4Ble4c2Rzovz7nzMWwtOPnMIf
[Table]
[Table] </ [Table]
[Figure] [Figure]
Related Work
[Figure]
Network Load Balancing Algorithms.
The main goal of network LBs is to fairly distributeworkloads across servers.
The system transition protocol of network load balancing system isdescribed in Alg. 1 and depicted in Fig.
Existing load balancing algorithms are sensitive topartial observations and inaccurate server weights.
Equal-Cost Multi-Path (ECMP) LBs randomlyassign servers to new requests [12–14], which makes them agnostic to server load state differences.
Weighted-Cost Multi-Path (WCMP) LBs assign weights to servers proportional to their provisionedresources ( e.g. CPU power)
However, the statically assigned weights may not correspondto the actual server processing capacity.
As depicted in Figure 2a, servers with the same IO speedyet different CPU capacities have different actual processing speed when applications have differentresource requirements.
Active WCMP (AWCMP) is a variant of WCMP and it periodically probeserver utilisation information (CPU/memory/IO usage) [5,18].
However, active probing can causedelayed observations and incur additional control messages, which degrades the performance ofdistributed networking systems.
Local Shortest Queue (LSQ) assigns new requests to the server withthe minimal number of ongoing networking connections that are locally observed [19,20].
It does notconcern server processing capacity differences.
Shortest Expected Delay (SED) derives the “expecteddelay” as locally observed server queue length divided by statically conﬁgured server processingspeed [2].
However, LSQ and SED are sensitive to partial observations and misconﬁgurations.
Asdepicted in Fig. 2b, the QoS performance of each load balancing algorithm degrades from the idealsetup (global observations and accurate server weight conﬁgurations) when network trafﬁc is splitacross multiple LBs or server weights are mis-conﬁgured 2 , which prevails in real-world cloud DCs.
In this paper, we propose a distributed MARL-based load balancing algorithm that considers dynamically changing queue lengths ( e.g. sub-ms in modern DC networks [21]), and autonomously adapts toactual server processing capacities, with no additional communications among LB agents or servers.
Markov Potential Games.
A potential game (PG) [22–25] has a special function called potentialfunction , which speciﬁes a property that any individual deviation of the action for one player will [Figure] change the value of its own and the potential function equivalently.
A desirable property of PG is thatpure NE always exists and coincides with the maximum of potential function in norm-form setting.
Self-play [26] is provably converged for PG.
Markov games (MG) is an extension of normal-formgame to a multi-step sequential setting.
A combination of PG and MG yields the Markov potentialgames (MPG) [27,28], where pure NE is also proved to exist.
Some algorithms [27,29,30] lyingin the intersection of game theory and reinforcement learning are proposed for MPG.
For example,independent nature policy gradient is proved to converge to NE for MPG [27].
Multi-Agent RL .
MARL [31] has been viewed as an important avenue for solving different typesof games in recent years.
For cooperative settings, a line of work based on joint-value factorisationhave been proposed, involving VDN [32], COMA [11], MADDPG [33], and QMIX [34].
For theseworks, a global reward is assigned to players within the team, but individual policies are optimised toexecute individual actions, known as the CTDE setting.
MPG satisﬁes the assumptions of the valuedecomposition approach, with the well-speciﬁed potential function as the joint rewards.
However,deploying CTDE RL models in real-world distributed system incurs additional communication latencyand management overhead for synchronising agents and aggregating trajectories.
By leveraging thespecial structure of MPG, independent learning approach can be more efﬁcient due to the decomposition of the joint state and action spaces, which is leveraged in the proposed methods.
Methods like MATRPO [35], IPPO [36] follow a fully decentralised setting, but for general cooperative games.
In terms of the distribution fairness, FEN [37] is proposed as a decentralised approach for fairreward distribution in multi-agent systems.
They deﬁned the fairness as coefﬁcient of variation anddecompose it for each individual agent.
Another work [38] proposes a decentralised learning methodfor fair policies in cooperative games.
However, the decentralised learning manner in these methodsare not well justiﬁed, while in this paper the load balancing problem is formally characterised as a
MPG and the effectiveness of distributed training is veriﬁed.
Methods
3.1 Problem Description
We formulate the load balancing problem into a discrete-time dynamic game with strong distributedand concurrent settings, where no centralised control mechanism exists among agents.
We let Mdenote the number of LB agents ( [ M ] denotes the set of servers { 1 , . . .
, M } )
and N denote thenumber of servers ( [ N ] denotes the set of servers { 1 , . . .
, N } ).
At each time step (or round) t 2 Hin a horizon H of the game, each LB agent i receives a workload w i ( t ) 2 W , where W is theworkload distribution, and the LB agent assigns a server to the task using its load balancing policy⇡ i 2 ⇧ , where ⇧ is the load balancing policy proﬁle.
At each time-step t , a LB agent
i takesan action a i ( t ) = { a ij ( t ) } Nj =1 , according to which the tasks w i ( t ) are assigned with distribution↵ i ( t ) .
↵ ij ( t ) is the probability mass of assigning tasks to server j ,
P
Nj =1 ↵ ij ( t ) = 1 .
Therefore,at each time step, the workload assigned to server j by the i -th LB is w i ( t ) ↵ ij ( t ) .
During eachtime interval, each server j is capable of processing a certain amount of workload v j based on theproperty of each server ( e.g. provisioned resources including CPU, memory, etc. ).
We have serverload state (remaining workload to process)
X j ( T )
[Equation]
max {
[Equation]
P
[Equation]
X ij ( T ) 3 .
Let l j denote the time for a server j toprocess all remaining workloads, which is also the potential queuing time for new-coming tasks,l
[Equation]
( t [Equation] =
[Equation]
l ij ( t ) .
Then transition fromtime step t to time step t + 1 is given in Alg.
Reward: r i ( t ) =
R ( l ( t ) , a i ( t ) ,  i ( t )) , where R is thereward function, l ( t ) =
[Equation]
i ( [Equation]
) denotes the estimated remaining time to processon each server, and  i ( t ) is a random variable that makes the process stochastic.
Deﬁnition 1.
(Makespan)
In the selﬁsh load balancing problem, the makespan is deﬁned as:
[Equation]
The network load balancing problem is featured as multi-commodity ﬂow problems and is NP-hard, 118 which makes it hard to solve with trivial algorithmic solution within micro-second level [39].
Thisproblem can be formulated as a constrained optimisation problem for minimizing the makespan overan horizon t 2 [ H ] :minimize
[Equation]
In modern realistic network load balancing system, the arrival of network requests is usually unpredictable in both its arriving rate and the expected workload, which introduces large stochasticityinto the problem.
Moreover, due to the existence of noisy measurements and partial observations,the estimation of makespan can be inaccurate, which indicates the actual server load states or processing capacities are not correctly captured.
Instant collisions of elephant workloads or bursts ofmouse workloads often happen, which do not indicate server processing capacity thus misleadingthe observation.
To solve this issue, we introduce fairness as an alternative of the original objectivemakespan.
Speciﬁcally, makespan is estimated on a per-server level, while the estimation of fairnesscan be decomposed to the LB level, which allows evaluating the individual LB performance withoutgeneral loss.
This is more natural in load balancing system due to the partial observability of LBs.
3.2 Distribution Fairness
We mainly introduce two types of load balancing distribution fairness: (1) variance-based fairness(VBF) and (2) product-based fairness (PBF).
It will be proved that optimization over either fairnesswill be sufﬁcient but not necessary for minimising the makespan.
Deﬁnition 2.
(Variance-based Fairness)
For a vector of time to ﬁnish all remaining jobs l =[ l 1 , . . .
, l N ] on each server j 2 [ N ] , let l ( t )
=
1 N
P
Nj =
P
Mi =1 l ij ( t ) , the variance-based fairness forworkload distribution is just the negative sample variance of the job time, which is deﬁned as: [Equation] [Equation] [Equation]
[Equation]
VBF deﬁned per LB is: F i ( l i ) =
1 N
P
Nj =1 l 2 ij ( t ) + l 2 i ( t ) , where l i ( t ) = 1 N
P
Nj =1 l ij ( t ) .
Lemma 3.
The VBF for load balancing system satisﬁes the following property:
[Equation]
This property makes VBF a good choice for the reward function in load balancing tasks.
We will seemore discussions in later sections.
Proof of the lemma is provided in Appendix B.1.
[Equation]
Proposition 4.
Maximising the VBF is sufﬁcient for minimising the makespan, subjective to the loadbalancing problem constraints (Eq.
(3) and (4) ): max F ( l ) )
min max j ( l j ) .
This also holds forper-LB VBF as max F i ( l i ) )
min max j ( l i ) .
Deﬁnition 5.
(Product-based Fairness [40])
For a vector of time to ﬁnish all remaining jobs l =[ l 1 , . . .
, l N ] on each server j 2 [ N ] , the product-based fairness for workload distribution is deﬁnedas:
F ( l ) = F ([ l 1 , . . .
, l N ]) =
Q j 2 [ N ] l j max( l ) .
PBF deﬁned per LB is: F i ( l i ) =
F ([ l i 1 , . . .
, l iN ]) =
Q j 2 [ N ] l ij max( l i ) .
Proposition 6.
Maximising the product-based fairness is sufﬁcient for minimising the makespan,subjective to the load balancing problem constraints (Eq.
(3) and (4) ): max F ( l ) )
min max( l ) .
Proofs of proposition 4 and 6 are in Appendix B.1 andB.2, respectively.
From proposition 4 and 6,we know that the two types of fairness can serve as an effective alternative objective for optimisingthe makespan, which will be leveraged in our proposed MARL method as valid reward functions.
3.3 Game Theory Framework
Markov decision game is deﬁned as MG ( H, M, S , A ⇥ M , P , ⇧ ⇥ M , r ⇥ M ) , ⇧ = { ⇧ i } , i 2 [ M ] ,where H is the horizon of the game, M is the number of player in the game, S is the state space,
A ⇥ M is the joint action space of all players,
A i is the action space of player i , P = { P h } ,
h 2 [ H ] isa collection of transition probability matrices P h : S ⇥
A ⇥ M ⇥ S !
[0 , 1] , r ⇥ M = { r i | i
[ M ] } , r i :
S ⇥
A ⇥ M !
R is the reward function for i -th player given the joint actions.
The stochastic policyspace for the i -th player in MG is deﬁned as ⇧ i : S ⇥
A i !
For the Markov decision game MG , the state value function V ⇡ i,h : S !
R and state-action valuefunction Q ⇡ i,h : S ⇥ A !
R for the i -th player at step h under policy ⇡ 2 ⇧ ⇥ M is deﬁned as:
V ⇡ i,h ( s[-) : = E ⇡ , P  H X h 
-]
[Equation]
[-X h 
-]
Deﬁnition 7.
( ✏ -approximate
Nash equilibrium)
Given a
Markov decision game
MG ( H, M, S , A ⇥ M , P , ⇧ ⇥ M , r ⇥ M ) , let ⇡  i be the policies of the players except for the 165 i -th player, the policies ( ⇡ ⇤ i , ⇡ ⇤ i ) is an ✏ -Nash equilibrium if 8 i 2 [ M ] , 9 ✏ > 0 , V ⇡ ⇤ i [Equation] ⇤
[Equation]
If ✏ = 0 , it is an exact Nash equilibrium.
Deﬁnition 8.
(Markov Potential Game)
A Markov decision game M ( H, M, S , A ⇥ M , P , ⇧ ⇥ M , r ⇥ M ) 168 is a Markov potential game (MPG) if
[Equation]
V
[Equation]
where  ( · ) is the potential function independent of the player index.
Lemma 9.
Pure NE (PNE) always exists for PG, local maximisers of potential function are PNE.
PNE also exists for MPG.
Theorem 10.
Multi-agent load balancing is MPG with the VBF F i ( l i ) as the reward r i for each LBagent i 2 [ M ] , then suppose for 8 s 2 S at step h 2 [ H ] , the potential function is time-cumulativetotal fairness:  ⇡ i ,
[Equation]
The proof of the theorem is based on Lemma 3, and it’s provided in Appendix B.3.
This theorem isessential for establishing our method, since it proves that multi-agent load balancing problem canbe formulated as a MPG with the time-cumulative VBF as its potential function.
Also, the choiceof per-LB VBF as reward function for individual agent is critical for making it MPG, it is easy toverify that PBF cannot guarantee such property.
From Lemma 9 we know the maximiser of potentialfunction is the NE of MPG, and from proposition 4 it is known that maximising the VBF gives thesufﬁcient condition for minimising the makespan.
Therefore, an effective independent optimisationwith respect to the individual reward function speciﬁed in the above theorem will lead the minimiserof makespan for load balancing tasks.
The effective independent optimisation here means the NE of
MPG is achieved.
[Table] [Table]
n
[Table]
a t c h N o r m
[Table]
m
L i n
[Table]
r
[Table]
a
[Table]
r
[Table]
o
[Table]
[Figure]
3.4 Distributed LB Method
With the above analysis, the load balancing problem can be formulated as an episodicversion of multi-player partially observable Markov game, which we denote as
POMG ( H, M, S , O ⇥ M , O ⇥ M , r ⇥ M ) , M is the number of LB agent in the game, O ⇥ Mcontains the observation space
O i for each player, O =
{ O h } , h 2 [ H ] is a collection of observationemission matrices, O i,h : S ⇥
O i !
[0 , 1] , r ⇥ M = { r i | i
[ M ] } , r i : O i ⇥ A ⇥ M !
R is the reward 191 function for i -th LB agent given the joint actions.
The stochastic policy space for the i -th agentin POMG is deﬁned as ⇧ i : O i ⇥ A i !
As discussed in Sec. 2, the partial observabilitycomes from the fundamental conﬁguration of network LBs in DC networks, which allows LBs toobserve only a partial of network trafﬁc and does not give LBs information about the tasks ( e.g.expected workload) distributed from each LB.
The reward functions in our experiments are variantsof distribution fairness introduced in Sec. 3.2.
The potential functions can be deﬁned accordinglybased on the two fairness indices.
The overview of the proposed distributed MARL framework isshown in Fig.
In MPG, independent policy gradient allows ﬁnding the maximum of the potential function, which isthe PNE for the game.
This inspires us to leverage the policy optimisation in a decomposed manner,i.e. , distributed RL for policy learning of each LB agent.
However, due to the partial observability ofthe system and the challenge of directly estimating the makespan (Eq.
(1)), each agent cannot have adirect access to the global potential function.
To address this problem, the aforementioned fairness(Sec. 3.2) can be deployed as the reward function for each agent, which makes the value function as avalid alternative for the potential function as an objective.
This also transforms the joint objective(makespan or potential) to individual objectives (per LB fairness) for each agent.
Proposition 4 and 6verify that optimising towards these fairness indices is sufﬁcient for minimising the makespan.
Alg. 2 shows the proposed distributed LB for load balancing problem, which is a partially observable MPG.
The distributed policy optimisation is based on Soft Actor-Critic (SAC) [41] algorithm,which is a type of maximum-entropy RL method.
It optimises the objective E [
P t  t r t + ↵ H ( ⇡ ✓ )] ,whereas H ( · ) is the entropy of the policy ⇡ ✓ .
Speciﬁcally, the critic Q network is updated withgradient r  E o,a ✓
[-Q  ( o, a )  r ( o, a )   E o 
-][Equation] [Equation] [Equation]
[ V ˜  [Equation] o 0 )] ◆  , where V ˜  ( o 0 ) =
E a 0 [ Q ˜
( o 0 , a 0 ) ↵ log ⇡ ✓ ( a 0 | o 0 )] and Q ˜  is the target Q network; the actor policy ⇡ ✓ is updated with the gradientr ✓ E o [ E a ⇠ ⇡ ✓ [ ↵ log ⇡ ✓ ( a | o )
Q  ( o, a )]] .
Other key elements of RL methods involve the observation,action and reward function, which are detailed as following.
Observation.
Each LB agent partially observes over the trafﬁc that traverses through itself, includingper-server-level and LB-level measurements.
For each LB, per-server-level observations consist of –
1: Initialise: 2:
LB policy ⇡ ✓ i and critic Q  i networks, replay buffer B i , 8 i 2 [ M ] ; 3: server processing speed function v j , 8 j 2 [ N ] ; 4: initial observed instant queue length on server j by the i -th LB: q ij = 0 , 8 i 2 [ M ] , j 2 [ N ] .
5: while not converge do 6:
Reset server load state X j (1) 0 , 8 j 2 [ N ] 7:
Each LB agent i ( i 2 [ M ] ) receives individual observation
o i (1) 8: for t = 1 , . .
. , N do 9:
Initialise distributed workload m ij , w i ( t ) 0 , i
[ M ] , j 2 [ N ] 10:
Get actions a i ( t ) { a ij ( t ) } Nj =1 = ⇡ ✓ i ( o i ( t )) , i 2 [ M ] 11: for job ˜ w arrived at LB i between timestep [ t , t + 1 ) do 12:
LB i assigns ˜ w to server j = arg min k 2 [ N ] q ik ( t )+1 a ik ( t ) 13: m ij m ij + ˜ w , w i ( t ) w i ( t ) + ˜ w 14: ↵ ij ( t ) m ij w i ( t ) 15: end for 16: for each server j do 17:
Update workload: X ij ( t + 1) max { X ij ( t ) + w i ( t ) ↵ ij ( t )  v j M , 0 } 18:
X j ( t + 1)
P
Mi =1 X ij ( t ) 19: end for 20:
Each agent receives individual reward r i ( t ) 21:
Each agent i collects observation
o i ( t + 1) , i
[ M ] 22:
Update replay buffer: B i =
B i
S ( a i ( t  1) , o i ( t ) , a i ( t ) , r i ( t ) , o i ( t + 1)) , i 2 [ M ] 23: end for 24:
Update critics with gradients: r  i E
( o i ,a i ,r i ,o 0 i )
⇠ B i ✓ Q  i ( o i , a
i )  r i   V ˜
i ( o 0 i ) ◆  25: where V ˜  i ( o 0 i )
=
E ( o
i ,a 0 i )
⇠ B i [ Q ˜  i ( o 0 i , a 0 i )  ↵ log ⇡ ✓ i ( a 0 i | o 0 i )] , i 2 [ M ] 26:
Update policies with gradients: - r ✓ i E o i
⇠ B i [ E a ⇠ ⇡ ✓ i [ ↵ log ⇡ ✓ i ( a i
| o i )  Q
i ( o i , a i )]] , i
[ M ] 27: end while 28: return ﬁnal models of learning agents for each server – the number of on going tasks, and sampled task duration and task completion time(TCT). Speciﬁcally, in Alg. 2 line 12-14, w i is the coming workload on servers assigned by i -th LB,and it is not observable for LB. q ik + 1 is the locally observed number of tasks on k -th server by i -th
LB, due to the real-world constraints of limited observability at the Transport layer.
The “+1” is fortaking into account the new-coming task.
Observations of task duration and TCT samples, along with
LB-level measurements which sample the task inter-arrival time as an indication of overall systemload state, are reduced to 5 scalars – i.e. average, 90th-percentile, standard deviation, discountedaverage and weighted discounted average 4 – as inputs for LB agents.
Action.
To bridge the different timing constraints between the control plane and data plane, each LBagent assigns the j -th server to newly arrived tasks using the ratio of two factors, arg min k 2 [ N ] q ik +1 a ik ,where the number of on-going tasks q ik helps track dynamic system server occupation at perconnection level – which allows making load balancing decision at µ s-level speed – and a ik isthe periodically updated RL-inferred server processing speed.
As in line 15 of Alg. 2, ↵ ij ( t ) is astatistical estimation of workload assignment distribution at time interval [ t, t + 1) .
Reward.
The individual reward for distributed MPG LB is chosen as the VBF (as Def. 2) of thediscounted average of sampled task duration measured on each LB agent, such that the LB groupjointly optimise towards the potential function deﬁned in Eq.
Task duration information isgathered as the time interval between the end of connection initialisation ( e.g. 3 -way handshakefor TCP trafﬁc) and the acknowledgement to the ﬁrst data packet ( e.g. the ﬁrst ACK packet for
TCP trafﬁc).
Given the limited and partial observability of LB agents, task duration informationapproximates the remaining workload l by measuring the queuing and processing delay for newcoming tasks on each server.
This PBF- and MS-based rewards are also implemented for CTDE
MARL algorithm as a comparison.
[Figure]
[Table]
Model.
The architecture of the proposed RL framework is depicted in Fig.
Each LB agent consistsof a replay buffer, and a pair of actor-critic networks, whose architecture is depicted on the topright.
There is also a pair of guiding actor-critic networks, with the same network architectures butupdated in a delayed and soft manner.
Each LB agent takes observations
o i ( t ) extracted from the dataplane ( e.g. numbers of ongoing tasks { q ij } , task duration, TCT) and actions from previous [-timestep 
 a-]timestepa i ( t  1) as inputs, and periodically generates new actions a i ( t ) , which is used to update the serverassignment function arg min
j 2 [ N ] q ij +1 a ij in the data plane.
The gated recurrent units (GRU) [42] areapplied for all agents to leverage the sequential history information for handling partial observability.
Evaluation
We developed (i) an event-based simulator (App. C.1) to study the distance between the NE achievedby the proposed algorithm and the NE achieved by the theoretical optimal load balancing policy(with perfect observation), and (ii) a realistic testbed (App. C.2) on physical servers in a DC networkproviding Apache web services, with real-world network trafﬁc [43], to evaluate the real-worldperformance of the proposed algorithm, in comparison with in-production state-of-the-art LB [3].
As depicted in Fig. 4a, in a moderate-scale real-world DC network setup with 2 LB agents and 7servers, after 120 episodes of training, the proposed distributed LB (Distr-LB) algorithm is able tolearn from the environment based on VBF as rewards, and it converges to offer better QoS than QMix.
Centralised RL agent (Centr-LB) has difﬁculties to learn within 120 episodes because of theincreased state and action space.
An empirical ﬁnding is that, by adding a log term to the VBF-basedreward for Distr-LB, we help LB agents to become more sensitive to close-to- 0 VBF during training( r x log f ( x ) >
r x f ( x ) when f ( x ) < 1 ), therefore achieving better load balancing performance.
As depicted in Fig. 4b, when comparing with in-production LB algorithms (WCMP, LSQ, SED),
Distr-LB shows clear performance gains and reduced TCT for both types of web pages – Wikipediapages require to query SQL databases thus they are more CPU-intensive, while static pages are
IO-intensive.
The comparison of TCT using different LB algorithms is shown in Table 1.
To evaluate the gap between the performance of Distr-LB and the theoretical optimal policy, weimplement in the simulator an Oracle LB, which has perfect observation (inaccessible in real world)over the system and minimises makespan for each load balancing decision.
Table 2 shows that, fordifferent types of applications, Distr-LB is able to achieve closer-to-optimal performance than QMix.
As the simulator is implemented based on the load balancing model formulated in this paper, our
[Table] [Table]
theoretical analysis can be directly applied, and VBF – as a potential function – helps independentcooperative LB agents to achieve good performance.
The additional log term shows empiricalperformance gains in real-world system, yet it is not necessarily the case in these simulation results.
On one hand, the generated trafﬁc of tasks in the simulation has higher expected workload ( > 1 smean and stddev), while the log terms is more sensitive to close-to- 0 variances, which is the casein real-world experimental setups.
On the other hand, though the simulator models the formulated LB problem, it fails to captures the complexity in the real-world system – e.g. Apache backlog,multi-processing optimisation, context switching, multi-level cache, network queues etc.
For instance,batch processing [44] helps reduce cache and instruction misses, yet yields similar processing timefor different tasks, thus the variance of task processing delay decreases and becomes closer to 0 inreal-world system.
The additional log term exaggerates the low variance differences to better evaluateload balancing decisions.
More detailed description about the simulator implementation can be foundin App C.1 and ablation study on reward engineering is presented in App E.2.1.
To evaluate the performance of Distr-LB in large-scale DC networks in real world, we scale up thereal-world testbed to have 6 LB agents and 20 servers and apply heavier network trafﬁc ( > 2000queries/s) to evaluate the performance of the LB algorithms that achieved the best performancein moderate scale setups, in comparison with in-production LB algorithms.
The test results after200 episodes of training are shown in Table 3, where Distr-LB achieves the best performance in allcases.
QMix also outperforms in-production LB algorithms.
But as a CTDE algorithm, similar to the Centr-LB, it requires agents to communicate their trajectories, which – after 200 episodes of training– become 221 MiB communication overhead at the end of each episode (episodic training), whereas95% -percentile per-destination-rack ﬂow rate is less than 1 MiB/s [45].
More details regarding thereal-world DC testbed implementation is in App.
C.2, training details are in App.
D, and ablationstudies – e.g. communication overhead of CTDE and centralised RL in real-world system, robustnessof MARL algorithms in dynamic DC network environments – can be found in App.
E.2.2.
Conclusion and Future Work
This paper proposes a distributed MARL approach for multi-agent load balancing problem, based on
Markov potential game formulation.
The proposed variance-based fairness for individual LB agent iscritical for this formulation.
Through this setting, the redundant communication overhead among
LB agents is removed, thus improving the overall training and deployment efﬁciency in real-worldsystems, with the local observations only.
Under such formulation, the effectiveness of our proposeddistributed LB algorithm together with the proposed fairness are both theoretically justiﬁed andexperimentally veriﬁed.
It demonstrates a performance gain over another commonly applied fairnessas well as centralised training methods like QMIX or centralised RL agent, in both simulation andreal-world tests with different scales. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No] To our
knowledge, the work has no potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] The proofs are
provided in Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] The source
code to reproduce both simulation and real-world experimental results has been open-
sourced.
The URL will be provided on acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running exper-
iments multiple times)?
[No] The error bars are replaced by cumulative distribution
function (CDF).
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No] This work is mainly based on
simulation and the information of computational resources are not relevant.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
This work does not
use existing assets.
(b) Did you mention the license of the assets?
[N/A]
This work does not use existing
assets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
This work does not use existing assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
This work does not use existing assets.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
This work does not use existing assets.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
It does not apply to this work.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
It does not apply to this work.
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
It does not apply to this work.