Learning Distributed and Fair Policies for Network Load Balancing as Markov Potential Game
Abstract
This paper investigates the network load balancing problem in data centers (DCs) where multiple load balancers (LBs) are deployed, using the multi-agent reinforcement learning (MARL) framework.
The challenges of this problem consist of the heterogeneous processing architecture and dynamic environments, as well as limited and partial observability of each LB agent in distributed networking systems, which can largely degrade the performance of in-production load balancing algorithms in real-world setups.
Centralised-training-decentralised-execution (CTDE) RL scheme has been proposed to improve MARL performance, yet it incurs – especially in distributed networking systems, which prefer distributed and plug-and-play design schemes – additional communication and management overhead among agents.
We formulate the multi-agent load balancing problem as a Markov potential game, with a carefully and properly designed workload distribution fairness as the potential function.
A fully distributed MARL algorithm is proposed to approximate the Nash equilibrium of the game.
Experimental evaluations involve both an event-driven simulator and real-world system, where the proposed MARL load balancing algorithm shows close-to-optimal performance in simulations, and superior results over in-production LBs in the real-world system.
Introduction
In cloud data centers (DCs) and distributed networking systems, servers are deployed on infrastructures with multiple processors to provide scalable services [1].
To optimise workload distribution and reduce additional queuing delay, load balancers (LBs) play a signiﬁcant role in such systems.
State-of-the-art network LBs rely on heuristic mechanisms [2–5] under the low-latency and highthroughput constraints of the data plane.
However, these heuristics are not adaptive to dynamic environments and require human interventions, which can lead to most painful mistakes in the cloud – mis-conﬁgurations.
RL approaches have shown performance gains in distributed system and networking problems [6–9], yet applying RL on the network load balancing problem is challenging.
First, unlike traditional workload distribution or task scheduling problem [6,7], network LBs have limited observations over the system, including task sizes and actual server load states.
Being aware of only the number of tasks they have distributed, servers can be overloaded by collided elephant tasks and have degraded quality of service (QoS).
Second, to guarantee high service availability in the cloud, multiple LBs are deployed in DCs.
Network trafﬁc is split among all LBs.
This multi-agent setup makes LBs have only partial observation over the system.
Third, modern DCs are based on heterogeneous hardware and elastic infrastructures [10], where server capacities vary.
It is challenging to assign correct weights to servers according to their actual processing capacities, and this process conventionally requires human intervention – which can lead to error-prone conﬁgurations [3,5].
Last ⇤ Equal contribution.
36th Conference on Neural Information Processing Systems (NeurIPS 2022). but not least, given the low-latency and high-throughput constraints in the distributed networking setup, the interactive training procedure of RL models and the centralised-training-decentralisedexecution (CTDE) scheme [11] can incur additional communication and management overhead.
In this paper, we study the network load balancing problem in multi-agent game theoretical approach, by formulating it as a Markov potential game through specifying the proper reward function, namely variance-based fairness.
We propose a distributed Multi-Agent RL (MARL) network load balancing mechanism that is able to exploit asynchronous actions based only on local observations and inferences.
Load balancing performance gains are evaluated based on both event-based simulations and real-world experiments 2 .
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table]
ApplicationServers
[Table]
/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWjm6nfeuTaiFg94DjhfkQHSoSCUbTS/VNP9MoVt+rOQJaJl5MK5Kj3yl/dfszSiCtkkhrT8dwE/YxqFEzySambGp5QNqID3rFU0YgbP5udOiEnVumTMNa2FJKZ+nsio5Ex4yiwnRHFoVn0puJ/XifF8MrPhEpS5IrNF4WpJBiT6d+kLzRnKMeWUKaFvZWwIdWUoU2nZEPwFl9eJs2zqndR9e7OK7XrPI4iHMExnIIHl1CDW6hDAxgM4Ble4c2Rzovz7nzMWwtOPnMIf+B8/gBhAo3d</latexit> w i
Related Work
Network Load Balancing Algorithms.
The main goal of network LBs is to fairly distribute workloads across servers.
The system transition protocol of network load balancing system is described in Alg. 1 and depicted in Fig.
Existing load balancing algorithms are sensitive to partial observations and inaccurate server weights.
Equal-Cost Multi-Path (ECMP) LBs randomly assign servers to new requests [12–14], which makes them agnostic to server load state differences.
Weighted-Cost Multi-Path (WCMP) LBs assign weights to servers proportional to their provisioned resources ( e.g. CPU power)
However, the statically assigned weights may not correspond to the actual server processing capacity.
As depicted in Fig. 2a, servers with the same IO speed yet different CPU capacities have different actual processing speed when applications have different resource requirements.
Active WCMP (AWCMP) is a variant of WCMP and it periodically probe server utilisation information (CPU/memory/IO usage) [5,18].
However, active probing can cause delayed observations and incur additional control messages, which degrades the performance of distributed networking systems.
Local Shortest Queue (LSQ) assigns new requests to the server with the minimal number of ongoing networking connections that are locally observed [19,20].
It does not concern server processing capacity differences.
Shortest Expected Delay (SED) derives the “expected delay” as locally observed server queue length divided by statically conﬁgured server processing speed [2].
However, LSQ and SED are sensitive to partial observations and misconﬁgurations.
As depicted in Fig.
2b, the QoS performance of each load balancing algorithm degrades from the ideal setup (global observations and accurate server weight conﬁgurations) when network trafﬁc is split across multiple LBs or server weights are mis-conﬁgured 3 , which prevails in real-world cloud DCs.
In this paper, we propose a distributed MARL-based load balancing algorithm that considers dynamically changing queue lengths ( e.g. sub-ms in modern DC networks [21]), and autonomously adapts to actual server processing capacities, with no additional communications among LB agents or servers.
Markov Potential Games.
A potential game (PG) [22–25] has a special function called potential function , which speciﬁes a property that any individual deviation of the action for one player will change the value of its own and the potential function equivalently.
A desirable property of PG is that pure NE always exists and coincides with the maximum of potential function in norm-form setting.
Self-play [26] is provably converged for PG.
Markov games (MG) is an extension of normal-form The stochastic Markov model of the simulation is detailed in the App. A [Figure]
[Figure]
[Table]
game to a multi-step sequential setting. A combination of PG and MG yields the Markov potential games (MPG) [27,28] [Table] where pure NE is also proved to exist.
Some algorithms [27,29,30] lying in the intersection of game theory and reinforcement learning are proposed for MPG.
For example, independent nature policy gradient is proved to converge to Nash equilibrium (NE) for MPG [27].
Multi-Agent RL .
MARL [31] has been viewed as an important avenue for solving different types of games in recent years.
For cooperative settings, a line of work based on joint-value factorisation have been proposed, involving VDN [32], COMA [11], MADDPG [33] [Table] and QMIX [34].
For these works, a global reward is assigned to players within the team [Table] but individual policies are optimised to execute individual actions, known as the CTDE setting.
MPG satisﬁes the assumptions of the value decomposition approach, with the well-speciﬁed potential function as the joint rewards.
However, deploying CTDE RL models in real-world distributed system incurs additional communication latency and management overhead for synchronising agents and aggregating trajectories.
These additional management and communication overheads can incur substantial performance degradation – constrained throughput and increased latency – especially in data center networks.
As listed in
Table 1, when we use active probing to measure server utilisation information, higher probing frequencies give improved measurement quality–in terms of CPU usage and number of on-going jobs on the servers.
However, higher probing frequencies also incur increased communication overhead, especially in large-scale data center networks.
The detailed experimental setups, as well as both qualitative and quantitative analysis of the impact of communication overhead, are described in
Sec. E.2.2. By leveraging the special structure of MPG, independent learning approach can be more efﬁcient due to the decomposition of the joint state and action spaces, which is leveraged in the proposed methods.
Methods like MATRPO [35], IPPO [36] follow a fully decentralised setting, but for general cooperative games.
In terms of the distribution fairness, FEN [37] is proposed as a decentralised approach for fair reward distribution in multi-agent systems.
They deﬁned the fairness as coefﬁcient of variation and decompose it for each individual agent.
Another work [38] proposes a decentralised learning method for fair policies in cooperative games.
However, the decentralised learning manner in these methods are not well justiﬁed, while in this paper the load balancing problem is formally characterised as a
MPG and the effectiveness of distributed training is veriﬁed.
Methods 3.1 Problem Description
We formulate the load balancing problem into a discrete-time dynamic game with strong distributed and concurrent settings, where no centralised control mechanism exists among agents.
We let M denote the number of LB agents ( [ M ] denotes the set of LB agents { 1 , . . .
, M } )
and N denote the number of servers ( [ N ] denotes the set of servers { 1 , . . .
, N } ).
At each time step (or round) t 2 H in a horizon H of the game, each LB agent i receives a workload w i ( t ) 2 W , where W is the workload distribution, and the LB agent assigns a server to the task using its load balancing policy ⇡ i 2 ⇧ , where ⇧ is the load balancing policy proﬁle.
At each time-step t , a LB agent i takes an action
a i ( t ) = { a ij ( t ) } Nj =1 , according to which the tasks w i ( t ) are assigned with distribution ↵ i ( t ) .
↵ ij ( t ) is the probability mass of assigning tasks to server j ,
P
Nj =1 ↵ ij ( t ) = 1 .
Therefore, at each time step, the workload assigned to server j by the i -th LB is w i ( t ) ↵ ij ( t ) .
During each time interval, each server j is capable of processing a certain amount of workload v j based on the property of each server ( e.g. provisioned resources including CPU, memory, etc. ).
We have server load state (remaining workload to process)
X j ( T ) =
[Equation]
max
[Equation]
P
[Equation]
P Mi =1 X ij ( T ) 4 .
Let l j denote the time for a server j to process all remaining workloads, which is also the potential queuing time for new-coming tasks, l j ( t ) =
[Equation]
[Equation] + P
[Equation]
=
P
Mi =
[Equation]
l [Equation] ( t ) .
Then transition from time step t to time step t + 1 is given in Alg.
Reward: r i ( t ) =
R ( l ( t ) , a i ( t ) ,  i ( t )) , where R is the reward function, l ( t ) =
P
Nj =1 l j ( t ) = P
Mi =1 l i ( t ) denotes the estimated remaining time to process on each server, and  i ( t ) is a random variable that makes the process stochastic.
Deﬁnition 1.
(Makespan)
In the selﬁsh load balancing problem, the makespan is deﬁned as:
[Equation]
The network load balancing problem is featured as multi-commodity ﬂow problems and is NP-hard, which makes it hard to solve with trivial algorithmic solution within micro-second level [39].
This problem can be formulated as a constrained optimisation problem for minimizing the makespan over an horizon t [Equation] ] : minimize [Equation]
[Equation]
In modern realistic network load balancing system, the arrival of network requests is usually unpredictable in both its arriving rate and the expected workload, which introduces large stochasticity into the problem.
Moreover, due to the existence of noisy measurements and partial observations, the estimation of makespan can be inaccurate, which indicates the actual server load states or processing capacities are not correctly captured.
Instant collisions of elephant workloads or bursts of mouse workloads often happen, which do not indicate server processing capacity thus misleading the observation.
To solve this issue, we introduce fairness as an alternative of the original objective makespan.
Speciﬁcally, makespan is estimated on a per-server level, while the estimation of fairness can be decomposed to the LB level, which allows evaluating the individual LB performance without general loss.
This is more natural in load balancing system due to the partial observability of LBs.
3.2 Distribution Fairness
We mainly introduce two types of load balancing distribution fairness: (1) variance-based fairness (VBF) and (2) product-based fairness (PBF).
It will be proved that optimization over either fairness will be sufﬁcient but not necessary for minimising the makespan.
Deﬁnition 2.
(Variance-based Fairness)
For a vector of time to ﬁnish all remaining jobs l = [ l 1 , . . .
, l N ] on each server j 2 [ N ] , let l ( t )
=
1 N
[Equation]
, the variance-based fairness for
[Equation]
workload distribution is just the negative sample variance of the job time, which is deﬁned as:
[Equation]
VBF deﬁned per LB is: F i ( l i ) =
1 N
P
Nj =1 l 2 ij ( t ) + l 2 i ( t ) , where l i ( t ) = 1 N
P
Nj =1 l ij ( t ) .
Lemma 3.
The VBF for load balancing system satisﬁes the following property:
[Equation]
This property makes VBF a good choice for the reward function in load balancing tasks.
We will see more discussions in later sections.
Proof of the lemma is provided in Appendix B.1.
Proposition 4.
Maximising the VBF is sufﬁcient for minimising the makespan, subjective to the load balancing problem constraints (Eq.
(3) and (4) ): max F ( l ) )
min max j ( l j ) .
This also holds for per-LB VBF as max F i ( l i ) )
min max j ( l i ) .
Deﬁnition 5.
(Product-based Fairness [40])
For a vector of time to ﬁnish all remaining jobs l = [ l 1 , . . .
, l N ] on each server j 2 [ N ] , the product-based fairness for workload distribution is deﬁned as: F ( l ) =
F ([ l 1 , . . .
, l N ]) =
Q j 2 [ N ] l j max( l ) .
PBF deﬁned per LB is: F i ( l i ) =
F ([ l i 1 , . . .
, l iN ]) =
Q j 2 [ N ] l ij max( l i ) .
Proposition 6.
Maximising the product-based fairness is sufﬁcient for minimising the makespan, subjective to the load balancing problem constraints (Eq.
(3) and (4) ): max F ( l ) )
min max( l ) .
Proofs of proposition 4 and 6 are in Appendix B.1 andB.2, respectively.
From proposition 4 and 6, we know that the two types of fairness can serve as an effective alternative objective for optimising the makespan, which will be leveraged in our proposed MARL method as valid reward functions.
3.3 Game Theory Framework
Markov game is deﬁned as MG ( H, M, S , A ⇥ M , P , r ⇥ M ) , where H is the horizon of the game, M is the number of player in the game, S is the state space,
A ⇥ M is the joint action space of all players,
A i is the action space of player i , P = { P h } , h 2 [ H ] is a collection of transition probability matrices
[-A ⇥ M ⇥ S ! [0 , 1] , r ⇥ M = { r i | i 
 [ M ] } , r i : S ⇥-]P
[Equation]
R is the reward function for i -th player given the joint actions.
The stochastic policy space for the i -th player in MG is deﬁned as ⇧ i :
[Equation]
For the Markov game MG , the state value function V ⇡ i,h : S !
R and state-action value function Q ⇡ i,h : S ⇥ A !
R for the i -th player at step h under policy ⇡ 2 ⇧ ⇥ M is deﬁned as:
V ⇡ i, [Equation] ( s ) : [Equation]
E ⇡
[Equation]
P 
H
[Equation]
[Equation]
h
[Equation]
i
[Equation]
[Equation]
[Equation]
a h
[Equation]
[Equation]
[Equation]

[Equation]
[Equation]
a
[Equation]
Deﬁnition 7.
( ✏ -approximate
Nash equilibrium)
Given a
Markov game
MG ( H, M, S , A ⇥ M , P , ⇧ ⇥ M , r ⇥ M ) , let ⇡  i be the policies of the players except for the i -th player, the policies ( ⇡ ⇤ i , ⇡ ⇤ i ) is an ✏ -Nash equilibrium if 8 i 2 [ M ]
[Equation]
✏
[Equation]
If ✏ = 0 , it is an exact Nash equilibrium.
Deﬁnition 8.
(Markov Potential Game)
A Markov game M ( H, M, S , A ⇥ M , P , ⇧ ⇥ M , r ⇥ M ) is a Markov potential game (MPG) if
[Equation]
where  ( · ) is the potential function independent of the player index.
Lemma 9.
Pure NE (PNE) always exists for PG, local maximisers of potential function are PNE.
PNE also exists for MPG.
Theorem 10.
Multi-agent load balancing is MPG with the VBF F i ( l i ) as the reward r i for each LB agent i {+
+} [ M ] , then suppose for 8 s 2 S at step h 2 [ H ] , the potential function is time-cumulative total fairness:  ⇡ i ,  ⇡ i ( s ) = P
Ht = h F ⇡ i ,  ⇡ i ( l ( t )) .
[Table]
L
[Table]
A
[Table]
e n t
L
[Table]
A g e n t
[Table]
r
[Table]
Actions
[Table]
</latexit>
[Figure]
( t )
[Figure]
<latexit sha1_base64="9vZpJtvgl20UdRxtNESNVfjJSwY=">AAAB/nicbVDLSsNAFJ3UV62vqLhyM9gKdWFJulCXBTcuK9gHNCFMptN22skkzEyEMgT8FTcuFHHrd7jzb5y2WWjrgQuHc+7l3nvChFGpHOfbKqytb2xuFbdLO7t7+wf24VFbxqnApIVjFotuiCRhlJOWooqRbiIIikJGOuHkduZ3HomQNOYPapoQP0JDTgcUI2WkwD6peBoFmo6zqrp0L7ws0OOsEthlp+bMAVeJm5MyyNEM7C+vH+M0IlxhhqTsuU6ifI2EopiRrOSlkiQIT9CQ9AzlKCLS1/PzM3hulD4cxMIUV3Cu/p7QKJJyGoWmM0JqJJe9mfif10vV4MbXlCepIhwvFg1SBlUMZ1nAPhUEKzY1BGFBza0Qj5BAWJnESiYEd/nlVdKu19yrmntfLzecPI4iOAVnoApccA0a4A40QQtgoMEzeAVv1pP1Yr1bH4vWgpXPHIM/sD5/AAsulNY=</latexit
[Figure]  [Figure] The proof of the theorem is based on Lemma 3, and it’s provided in Appendix B.3.
This theorem is essential for establishing our method, since it proves that multi-agent load balancing problem can be formulated as a MPG with the time-cumulative VBF as its potential function.
Also, the choice of per-LB VBF as reward function for individual agent is critical for making it MPG, it is easy to verify that PBF cannot guarantee such property.
From Lemma 9 we know the maximiser of potential function is the NE of MPG, and from proposition 4 it is known that maximising the VBF gives the sufﬁcient condition for minimising the makespan.
Therefore, an effective independent optimisation with respect to the individual reward function speciﬁed in the above theorem will lead the minimiser of makespan for load balancing tasks.
The effective independent optimisation here means the NE of
MPG is achieved.
3.4 Distributed LB Method
With the above analysis, the load balancing problem can be formulated as an episodic version of multi-player partially observable Markov game, which we denote as
POMG ( H, M, S , O ⇥ M , O ⇥ M , A ⇥ M , P , r ⇥ M ) , where M, H, S , A ⇥ M and P follow the same deﬁnitions as in Markov game
MG , O ⇥ M contains the observation space
O i for each player, O =
{ O h } , h 2 [ H ] is a collection of observation emission matrices, O i,h : S !
Pr( O i ) , r ⇥ M = { r i | i
[ M ] } , r i : O i ⇥ A ⇥ M !
R is the reward function for i -th LB agent given the joint actions.
The stochastic policy space for the i -th agent in POMG is deﬁned as ⇧ i : O i ! Pr( A i ) .
As discussed in Sec. 2, the partial observability comes from the fundamental conﬁguration of network LBs in DC networks, which allows LBs to observe only a partial of network trafﬁc and does not give LBs information about the tasks ( e.g. expected workload) distributed from each LB.
The reward functions in our experiments are variants of distribution fairness introduced in Sec. 3.2.
The potential functions can be deﬁned accordingly based on the two fairness indices.
The overview of the proposed distributed MARL framework is shown in Fig.
In MPG, independent policy gradient allows ﬁnding the maximum of the potential function, which is the PNE for the game.
This inspires us to leverage the policy optimisation in a decomposed manner, i.e. , distributed RL for policy learning of each LB agent.
However, due to the partial observability of the system and the challenge of directly estimating the makespan (Eq.
(1)), each agent cannot have a direct access to the global potential function.
To address this problem, the aforementioned fairness (Sec. 3.2) can be deployed as the reward function for each agent, which makes the value function as a valid alternative for the potential function as an objective.
This also transforms the joint objective (makespan or potential) to individual objectives (per LB fairness) for each agent.
Proposition 4 andverify that optimising towards these fairness indices is sufﬁcient for minimising the makespan.
1: Initialise: 2:
LB policy ⇡ ✓ i and critic Q  i networks, replay buffer B i , 8 i 2 [ M ] ; 3: server processing speed function v j , 8 j 2 [ N ] ; 4: initial observed instant queue length on server j by the i -th LB: q ij = 0 , 8 i 2 [ M ] , j 2 [ N ] .
5: while not converge do 6:
Reset server load state X j (1) 0 , 8 j 2 [ N ] 7:
Each LB agent i ( i 2 [ M ] ) receives individual observation
o i (1) 8: for t = 1 , . .
, H do 9:
Initialise distributed workload m ij , w i ( t ) 0 , i
[ M ] , j 2 [ N ] 10:
Get actions a i ( t ) { a ij ( t ) } Nj =1 = ⇡ ✓ i ( o i ( t )) , i 2 [ M ] 11: for job ˜ w arrived at LB i between timestep [ t , t + 1 ) do 12:
LB i assigns ˜ w to server j = arg min k 2 [ N ] q ik ( t )+1 a ik ( t ) 13: m ij m ij + ˜ w , w i ( t ) w i ( t ) + ˜ w 14: ↵ ij ( t ) m ij w i ( t ) 15: end for 16: for each server j do 17:
Update workload: X ij ( t + 1) max { X ij ( t ) + w i ( t ) ↵ ij ( t )  v j M , 0 } 18:
X j ( t + 1)
P
Mi =1 X ij ( t ) 19: end for 20:
Each agent receives individual reward r i ( t ) 21:
Each agent i collects observation
o i ( t + 1) , i
[ M ] 22:
Update replay buffer: B i =
B i
S ( a i ( t  1) , o i ( t ) , a i ( t ) , r i ( t ) , o i ( t + 1)) , i 2 [ M ] 23: end for 24:
Update critics with gradients: r  i E
( o i ,a i ,r i ,o 0 i )
⇠ B i ✓ Q  i ( o i , a
i )  r i   V ˜
i ( o 0 i ) ◆  25: where V ˜  i ( o 0 i )
=
E ( o
i ,a 0 i )
⇠ B i [ Q ˜  i ( o 0 i , a 0 i )  ↵ log ⇡ ✓ i ( a 0 i | o 0 i )] , i 2 [ M ] 26:
Update policies with gradients: - r ✓ i E o i
⇠ B i [ E a ⇠ ⇡ ✓ i [ ↵ log ⇡ ✓ i ( a i
| o i )  Q
i ( o i , a i )]] , i
[ M ] 27: end while 28: return ﬁnal models of learning agents Alg. 2 shows the proposed distributed LB for load balancing problem, which is a partially observable MPG. The distributed policy optimisation is based on Soft Actor-Critic (SAC) [41] algorithm, which is a type of maximum-entropy RL method. It optimises the objective E [
[Equation]
] , whereas H ( · ) is the entropy of the policy ⇡ ✓ .
Speciﬁcally, the critic Q network is updated with gradient r  E o,a ✓
[Equation]
◆  , where V ˜
[Equation]
o
[Equation]
) =
E
[Equation]
[ Q ˜
[Equation]
o
[Equation]
a
[Equation]
)  ↵ log ⇡ ✓ ( a 0 | o 0 )] and Q ˜  is the target Q network; the actor policy ⇡ ✓ is updated with the gradient r ✓ E o
[ E a ⇠ ⇡ ✓ [ ↵ log ⇡ ✓ ( a | o )
Q  ( o, a )]] .
Other key elements of RL methods involve the observation, action and reward function, which are detailed as following.
Observation.
Each LB agent partially observes over the trafﬁc that traverses through itself, including per-server-level and LB-level measurements.
For each LB, per-server-level observations consist of – for each server – the number of on going tasks, and sampled task duration and task completion time (TCT).
Speciﬁcally, in Alg.
2 line 12-14, w i is the coming workload on servers assigned by i -th LB, and it is not observable for LB.
q ik + 1 is the locally observed number of tasks on k -th server by i -th
LB, due to the real-world constraints of limited observability at the Transport layer.
The “+1” is for taking into account the new-coming task.
Observations of task duration and TCT samples, along with
LB-level measurements which sample the task inter-arrival time as an indication of overall system load state, are reduced to 5 scalars – i.e. average, 90th-percentile, standard deviation, discounted average and weighted discounted average 5 – as inputs for LB agents.
Action.
To bridge the different timing constraints between the control plane and data plane, each LB agent assigns the j -th server to newly arrived tasks using the ratio of two factors, arg min k 2 [ N ] q ik +1 a ik , where the number of on-going tasks q ik helps track dynamic system server occupation at perconnection level – which allows making load balancing decision at µ s-level speed – and a ik is [Figure] [Table] the periodically updated RL-inferred server processing speed.
As in line 14 of Alg. 2, ↵ ij ( t ) is a statistical estimation of workload assignment distribution at time interval [ t, t + 1) .
Reward.
The individual reward for distributed MPG LB is chosen as the VBF (as Def. 2) of the discounted average of sampled task duration measured on each LB agent, such that the LB group jointly optimise towards the potential function deﬁned in Eq.
Task duration information is gathered as the time interval between the end of connection initialisation ( e.g. 3 -way handshake for TCP trafﬁc) and the acknowledgement to the ﬁrst data packet ( e.g. the ﬁrst ACK packet for
TCP trafﬁc).
Given the limited and partial observability of LB agents, task duration information approximates the remaining workload l by measuring the queuing and processing delay for newcoming tasks on each server.
This PBF- and MS-based rewards are also implemented for CTDE
MARL algorithm as a comparison.
Model.
The architecture of the proposed RL framework is depicted in Fig.
Each LB agent consists of a replay buffer, and a pair of actor-critic networks, whose architecture is depicted on the top right.
There is also a pair of guiding actor-critic networks, with the same network architectures but updated in a delayed and soft manner.
Each LB agent takes observations
o i ( t ) extracted from the data plane ( e.g. numbers of ongoing tasks { q ij } , task duration, TCT) and actions from previous timestep a i ( t  1) as inputs, and periodically generates new actions a i ( t ) , which is used to update the server assignment function arg min
j 2 [ N ] q ij +1 a ij in the data plane.
The gated recurrent units (GRU) [42] are applied for all agents to leverage the sequential history information for handling partial observability.
Evaluation
We developed (i) an event-based simulator (App. C.1) to study the distance between the NE achieved by the proposed algorithm and the NE achieved by the theoretical optimal load balancing policy (with perfect observation), and (ii) a realistic testbed (App. C.2) on physical servers in a DC network providing Apache web services, with real-world network trafﬁc [43], to evaluate the real-world performance of the proposed algorithm, in comparison with in-production state-of-the-art LB [3].
[Table] [Table] Moderate-Scale Real-World Testbed: As depicted in Fig.
4a, in a moderate-scale real-world DC network setup with 2 LB agents and 7 servers, after 120 episodes of training, the proposed distributed LB (Distr-LB) algorithm is able to learn from the environment based on VBF as rewards, and it converges to offer better QoS than QMix.
Centralised RL agent (Centr-LB) has difﬁculties to learn within 120 episodes because of the increased state and action space.
An empirical ﬁnding is that, by adding a log term to the VBF-based reward for Distr-LB, we help LB agents to become more sensitive to close-to- 0 VBF during training ( r x log f ( x ) >
r x f ( x ) when f ( x ) < 1 ), therefore achieving better load balancing performance.
As depicted in Fig.
4b, when comparing with in-production LB algorithms (WCMP, LSQ, SED), Distr-LB shows clear performance gains and reduced TCT for both types of web pages – Wikipedia pages require to query SQL databases thus they are more
CPU-intensive, while static pages are IO-intensive.
The comparison of average TCT using different LB algorithms is shown in Table 2 ( 99 th percentile TCT in Table 12).
The proposed Distr-LB also shows superior performance than the RL-based solution (RLB-SAC) [40] because of (i) a well designed MARL framework, and (ii) the use of recurrent neural network to handle load balancing problem as a sequential problem.
NE Gap Evaluation with Simulation: To evaluate the gap between the performance of Distr-LB and the theoretical optimal policy, we implement in the simulator an Oracle LB, which has perfect observation (inaccessible in real world) over the system and minimises makespan for each load balancing decision.
Table 3 shows that, for different types of applications, Distr-LB is able to achieve closer-to-optimal performance than QMix.
As the simulator is implemented based on the load balancing model formulated in this paper, our theoretical analysis can be directly applied, and VBF – as a potential function – helps independent cooperative LB agents to achieve good performance.
The additional log term shows empirical performance gains in real-world system, yet it is not necessarily the case in these simulation results.
On one hand, the generated trafﬁc of tasks in the simulation has higher expected workload ( > 1 s mean and stddev), while the log terms is more sensitive to close-to- 0 variances, which is the case in real-world experimental setups.
On the other hand, though the simulator models the formulated LB problem, it fails to captures the complexity in the real-world system – e.g. Apache backlog, multi-processing optimisation, context switching, multi-level cache, network queues etc.
For instance, batch processing [44] helps reduce cache and instruction misses, yet yields similar processing time for different tasks, thus the variance of task processing delay decreases and becomes closer to 0 in real-world system.
The additional log term exaggerates the low variance differences to better evaluate load balancing decisions.
More detailed description about the simulator implementation can be found in App C.1 and ablation study on reward engineering is presented in App E.2.1.
Large-Scale Real-World Testbed: To evaluate the performance of Distr-LB in large-scale DC networks in real world , we scale up the real-world testbed to have 6 LB agents and 20 servers and apply heavier network trafﬁc ( > 2000 queries/s) to evaluate the performance of the LB algorithms that achieved the best performance in moderate scale setups, in comparison with in-production LB algorithms.
The test results after 200 episodes of training are shown in Table 4, where Distr-LB [Table] [Table] achieves the best performance in all cases.
QMix also outperforms in-production LB algorithms.
But as a CTDE algorithm, similar to the Centr-LB, it requires agents to communicate their trajectories, which – after 200 episodes of training – become 221 MiB communication overhead at the end of each episode (episodic training), whereas 95% -percentile per-destination-rack ﬂow rate is less than 1 MiB/s [45].
Scaling Experiments: Using the same large-scale real-world testbed with 6 LB agents and 20 servers, we conduct scaling experiments by applying network traces with different trafﬁc rates, comparing
LB methods with the best performances.
The 99 -th percentile QoS for both Wiki and static pages are shown in Table 5, 6.
As listed in Table 5 and 6, under low trafﬁc rates, when servers are all under utilised, the advantage of our proposed Distr-LB is not obvious because all resources are over-provisioned.
With the increase of trafﬁc rates (till servers are 100% saturated), our methods outperforms the best classical LB methods. More in-depth discussion and analaysis over the average job completion time for both types of pages in these scaling experiments are shown in Tableand 15 in App. E.2.2). More details regarding the real-world DC testbed implementation is in App.
C.2, training details are in App.
D, complete evaluation results (both moderate-scale and large-scale) are in App.
E and ablation studies – e.g. communication overhead of CTDE and centralised RL in real-world system, robustness of MARL algorithms in dynamic DC network environments – can be found in App.
E.2.
Conclusion and Future Work
This paper proposes a distributed MARL approach for multi-agent load balancing problem, based on
Markov potential game formulation.
The proposed variance-based fairness for individual LB agent is critical for this formulation.
Through this setting, the redundant communication overhead among
LB agents is removed, thus improving the overall training and deployment efﬁciency in real-world systems, with the local observations only.
Under such formulation, the effectiveness of our proposed distributed LB algorithm together with the proposed fairness are both theoretically justiﬁed and experimentally veriﬁed.
It demonstrates a performance gain over another commonly applied fairness as well as centralised training methods like QMIX or centralised RL agent, in both simulation and real-world tests with different scales.