Symbolic Distillation for Learned TCP Congestion Control
Anonymous Author(s)
AfﬁliationAddress
Abstract
Recent research in TCP congestion control (CC) has witnessed tremendous successwith deep reinforcement learning (RL) approaches, which use feedforward neuralnetworks (NN) to tackle complex environment conditions and make better decisions.
However, these “black box" policies lack interpretability, and reliability and, moreimportantly, cannot operate under the TCP datapath’s ultra-contingent latencyand computational constraints.
This paper proposes a novel two-stage solutionto achieve the best of both worlds: ﬁrst to train a deep RL agent, then distill its(over-)parameterized NN policy into white-box, light-weight rules in the formof symbolic expressions that are much easier to understand and to implementin constrained environments.
At the core of our proposal is a novel symbolicbranching algorithm that allows the rule to be “context-aware” of various networkconditions, eventually converting the NN policy into a symbolic tree.
The distilledsymbolic rules preserve and often improve performance over state-of-the-art NNpolicies while being faster and simpler than a standard neural network.
We validatethe performance of our distilled symbolic rules on both simulation and emulationnetwork systems.
Our code will be released upon acceptance.
Introduction
Congestion control (CC) is fundamental to Transmission Control Protocol (TCP) communication.
Congestion occurs when the data volume sent to a network reaches or exceeds its maximal capacity,in which case the network drops excess trafﬁc, and the performance unavoidably declines.
CCmitigates this problem by carefully adjusting the data transmission rate based on the inferred networkcapacities, aiming to send data as fast as possible without creating congestion.
For instance, a classicand best-known strategy, Additive-Increase/Multiplicative-Decrease (AIMD) [1], gradually increasesthe sending rate when there is no congestion but exponentially reduces the rate when the network iscongested.
It ensures that TCP senders fairly share the network capacity in the converged state.
Figure 1 shows an example where two TCP connections share a link between routers 1 and 2.
Whenthe shared link becomes a bottleneck, the CC algorithms running on sources A and B will alter thetrafﬁc rate based on the feedback to avoid congestion.
Efﬁcient CC algorithms have been the bedrockfor network services such as DASH video streaming, VoIP (voice-over-IP), VR/AR games, and IoT(Internet of Things), which ride atop the TCP protocol.
However, it is nontrivial to design a high-performance CC algorithm.
Over a long history, tens of CCproposals have been made, all with different metrics and strategies to infer and address congestion,and new designs are still emerging even today [2, 3].
There are two main challenges when designinga CC algorithm.
1 it needs to precisely infer whether the network is congested, and if so, howto adjust the sending rate, based on only partial or indirect observations .
Note that CC runs onend hosts while congestion happens in the network, so CC algorithms cannot observe congestiondirectly.
Instead, it can only rely on speciﬁc signals to infer the network status.
For instance, TCP
Cubic [4] uses packet loss as a congestion signal, and TCP Vegas [5] opts for delay increase.
2 CC algorithms operate within the OS kernel, where the computing and memory resources are limited,and they need to make real-time decisions to adjust the trafﬁc rates frequently (e.g., per round-triptime).
Therefore, the algorithm must be extraordinarily efﬁcient .
Spending a long time to compute anaction will signiﬁcantly offset the beneﬁt of congestion control.
[Figure]
In the long history of congestion control, most algorithms are implemented with manuallydesigned heuristics, including New Reno [6]
Vegas [5], Cubic [4], and BBR [2].
In TCP
New Reno, for example, the sender doubles thenumber of transmitted packets every RTT beforereaching a predeﬁned threshold, after which itsends one more packet every RTT.
If there isa timeout caused by packet loss, it halves thesending rate immediately.
Manually crafted CCshave been shown to be suboptimal and cannot support cases that escape the heuristics [7].
For example, packet loss-based CCs like Cubic [4] cannot distinguish packet drops caused by congestion or non-congestion-related events [7].
Researchers have tried to construct CC algorithms with machine learning approaches to address these limitations [7–11].
The insight isthat the CC decisions are dependent on trafﬁc patterns and network circumstances that can be exploited by deep reinforcement learning (RL) to learn a policy for each scenario.
The learned policycan perform more ﬂexible and accurate rate adjustment by discovering a mapping from experience,which can adapt to different network conditions and save manual tuning efforts for high performance.
Most notably, Aurora [7], a deep RL framework for Performance-oriented Congestion Control (PCC),trains a well-established PPO [12] agent to suggest sending rates as actions by observing the networkstatistics such as latency ratio, send ratio, and sent latency inﬂation.
It achieves competitive results onemulation environments Mininet [13] and Pantheon [14], demonstrating the potential of deep learningapproaches to outperform algorithmic, hand-crafted ones.
Despite its immense success, Aurorabeing a neural network based approach, is essentially a black-box to users or, in other words, lacksexplicit declarative knowledge [15].
They also require exponentially more computation resourcesthan traditional, algorithmic methods such as the widely deployed TCP-CUBIC [4].
1.1 Our Contributions
In this work, we develop a new algorithmic framework for performance-oriented congestion control(PCC) agents, which can 1 run as fast as classical algorithmic methods; 2 adjust the rate asaccurately as data-driven RL methods; and 3 be simpler than the original neural network, potentiallyimproving practitioner’s understanding of the model in a meaningful/actionable sense.
We solve this problem by grasping the opportunity enabled by advances in symbolic regression[16–19].
Symbolic regression bridges the gap between the infeasible search directly in the enormoussymbolic algorithms space and the differentiable training of overparameterized and un-interpretableneural networks.
At a high level, one can ﬁrst train an RL agent through gradient descent, then distill the learnedpolicy to obtain the data-driven optimized yet simpler and easier-to-understand symbolic rules.
Thisresults in a set of symbolic rules through data-driven optimization that meets TCP CC’s extremeefﬁciency and reliability demands.
However, considering the enormous volume of discrete symbolicspace, it is challenging to learn effective symbolic rules from scratch directly.
Therefore, in thispaper, we adopt a two-stage approach: we ﬁrst train a deep neural network policy with reinforcementlearning mimicking Aurora [7], and then distill the resultant policy into numerical symbolic rules,using symbolic regression (SR).
As the challenge, directly applying symbolic regression out of the box does not yield a sufﬁcientversatile expression that captures diverse networking conditions.
We hence propose a novel branchingtechnique for training and then aggregating a number of SymbolicPCC agents, each of which cater to asubset of the possible network conditions.
Speciﬁcally, we have multiple agents, each called a branch, and employ a light-weight “branch decider” to choose between the branches during deployment.
In order to create the branching conditions we partition the network condition space into adjacentnon-overlapping contexts, then regress symbolic rules in each context.
With this modiﬁcation, weenhance the expressiveness of the resulting SR equation and overcome the bias of traditional SRalgorithm to output rules mostly using numerical operators.
Our concrete technical contributions aresummarized as follows:
• We propose a symbolic distillation framework for TCP congestion control, which improvesupon the state-of-the-art RL solutions.
Our approach, SymbolicPCC , consists of two stages:ﬁrst training an RL agent and then distilling its policy network into ultra-compact and simple
[Figure]
rules in the symbolic expression form.
[Figure]
• We propose a novel branching technique that advances existing symbolic regression techniques
[Figure]
for training and aggregating multiple context-dependent symbolic policies, each of which
[Figure]
specializes for its own subset of network conditions.
A branch decider driven by light-weight
[Figure]
classiﬁcation algorithms determines which symbolic policy to use.
[Figure]
• Through our simulation and emulation experiments
SymbolicPCC achieves highly competitive
[Figure]
or even stronger performance results compared to their teacher policy networks while running
[Figure]
orders of magnitude faster.
The presented model uses a tree structure that is
light-weight
and
[Figure]
could be simpler for practitioners to reason about and improve manually while narrowing their
[Figure]
performance gap.
[Figure]
Related Works
[Figure]
Conventional TCP CC adopts aheuristic-based approach where theheuristic functions are manuallycrafted to adjust the trafﬁc rate in a deterministic manner.
Some proposalsuse packet loss as a signal for networkcongestion, e.g., Cubic [4], Reno [20],and NewReno [6], and others rely onthe variation of delay, e.g., Vegas [5].
Other CC designs combine packet lossand delay [21, 22].
Recently, different CC techniques specialized for data-center networks are alsoproposed [2, 3, 23].
Researchers have also investigated the use of machine learning to construct better heuristics.
Indigo [10] and Remy [11] use ofﬂine learning to obtain high-performance CC algorithms.
PCC [24]and PCC Vivace [9] opt for online learning to avoid any hardwired mappings between states andactions.
Aurora [7] utilizes deep reinforcement learning to obtain a new CC algorithm running inthe user space.
Orca [8] improves upon Aurora and designs a user-space CC agent that infrequentlyconﬁgures kernel-level CC policies.
Our proposal further improves these work.
On the other hand, Symbolic regression methods have [16–19] recently emerged for discoveringunderlying math equations that govern the observed data.
Algorithms with such a property are morefavorable for real-world deployment as they output white-box rules.
[17] use genetic programmingbased method while [19] uses a recurrent neural network to perform a search in the symbolic space.
We thus propose to synergize such numerical and data-driven approaches using symbolic regression(SR) in the congestion control domain.
We use SR by following a post-hoc method of ﬁrst training an
RL algorithm then distilling it into its symbolic rules.
Earlier methods that follow a similar proceduredo exist, e.g., [25] distills the learned policy as a soft decision tree.
They work on visual RL wherethe image observations are coarsely quantized into 10 × 10 cells, and the soft decision tree policyis learned over the 100 dimensional space.
[26] also aims to learn abstract rules using a commonsense-based approach by modifying Q learning algorithms.
Nevertheless, they fail to generalizebeyond the speciﬁc simple grid worlds they were trained in.
[27] learns from boolean feature sets, [28]directly approximates value functions based on a state-transition model, [29] optimizes risk-seekingpolicy gradients.
Other works on abstracting pure symbolic rules from data include attention-basedmethods [30], visual summary based methods [31], reward decomposition based methods [32], causal
RL Baseline
Ofine Dataset of Trajectories
Symbolic Policy (unbranched)
Branching Boundary Conditions
RL 1 RL 2 RL 3 RLtrain specialized RL agents
Sym1 Sym2 Sym
Sym
D D
D D
Ofine Datasets of Trajectories
Symbolic Policy (branched) symbolic distillation collect rollouts symbolic distillation collect rollouts reward clustering
Environment
Branch decider observations
Symbolic Policy (branched) branch
o b s e r v a t i o n s actions
Training SymbolicPCC
Evaluation/DeploymentSymbolicPCC clusteringalgorithm model based methods [33], markov chain based methods [34], and case-based expert-behaviourretrieval methods [35].
Methodology
Inspired from the idea of “teacher-student knowledge distillation” [36], our symbolic distillationtechnique is two-staged—ﬁrst train regular RL agents (as the teacher ), then distill the learned policynetworks into their white-box symbolic counterparts (as the student ).
In Section 3.1 we follow [7]’sapproach in Aurora and the training of teacher agents on the PCC-RL gym environment.
We alsobrieﬂy discuss the approach of applying symbolic regression to create a light-weight numerical-drivenexpression that approximates a given teacher’s behavior.
In Section 3.2 we look at the speciﬁcs ofsymbol spaces and attach internal attributes to aid long-term planning.
Finally, in Section 3.3 wediscuss our novel branching algorithm as a method for training, then ensembling multiple contextdependent symbolic agents during deployment.
3.1 Preliminaries: The PCC-RL Environment and the Symbolic Distillation Workﬂow
PCC-RL [7] is an open-source RL testbed for simulation of congestion control agents based on thepopular OpenAI Gym [37] framework.
We adopt it as our main playground.
It formulates congestioncontrol as a sequential decision making problem.
Time is ﬁrst divided into multiple periods called
MIs (monitor intervals), following [24].
At the onset of each MI, the environment provides the agentwith the history of statistic vector observations over the network, and the agent responds with adjustedsending rates for the following MI.
The sending rate remains ﬁxed during a single MI.
The network statistics provided as observations to the congestion control agent are 1 the latencyinﬂation, 2 the latency ratio, and 3 the sending ratio.
The agent is guided by reward signalsbased on its ability to react appropriately when detecting changes and trends in the vector statisticsof the PCC-RL environment.
It is provided with positive return for higher values of throughputs(packets/second) while being penalized for higher values of latency (seconds) and loss (ratio of sentvs.
acknowledged packets).
Training of Teacher Agents: We ﬁrst proceed to train RL agents using the PPO algorithm [12]similar to Aurora [7] in the PCC-RL gym environment till convergence.
Although they statisticallyperform very well [7], the PPO agents are entirely black-boxes; this makes it difﬁcult to explain itsunderlying causal rules directly.
Also, their over-parameterized neural network forms incur highlatency.
Hence, we choose to indirectly learn the symbolic representations using a student-teachertype knowledge distillation approach based on the teacher’s
(in this case, the RL agent) behaviors.
Distillation of Student Agents: Using the teacher agents, we collect complete trajectories, formallyknown as roll-outs in RL, in an inference mode—deterministic actions are enforced.
The observationsand their corresponding teacher actions are MI-aligned and stored as an ofﬂine dataset.
Note thatthis step is only performed once at the start of the distillation procedure and is reused in each of itsiterative steps.
A search space of operators and operands is also initialized (details are discussedshortly in Section 3.2).
Guesses for possible symbolic relations are taken, composed of randomoperators and operands from their respective spaces.
The stored observation trajectories are thenre-evaluated based on this rule to output corresponding actions.
The cross-entropy loss with respectto the teacher model’s actions from the same dataset is used as feedback.
This feedback drivesthe iterative mutation and pruning following a genetic programming technique [38, 39].
The bestcandidate policies are collected and forwarded to the next stage.
If the tree fails to converge or doesnot reach a speciﬁc threshold of acceptance, the procedure is restarted from scratch.
Our symbolicdistillation method is discussed with further details in the Appendix A.
3.2 A Symbolic Framework for Congestion Control
Deﬁning the Symbol Space for CC: Unlike visual RL [40], the PCC observation space is vectorbased, hence we directly plug them into the search space of our numerical-driven symbolic algorithm.
We henceforth call these observations as vector statistic symbols .
The distillation procedureas described earlier learns to chain these vector statistic symbols using a pre-deﬁned operatorspace.
Speciﬁcally, we employ three types of numerical operators.
The ﬁrst type of operators arearithmetic based which include + , − , ∗ , /, sin , cos , tan , cot , ( · ) 2 , ( · ) 3 , √· , exp , log , |·| .
The secondtype of operators are Boolean logic, such as is ( x < y ), is ( x ≤ y ), is ( x == y ), a | b , a & b , and ¬ a .
We also utilize a third type of high level operators – namely the slope_of ( observation history )which provides the average slope of an array of observations, and get_value_at ( observationhistory , index ).
The slope operator is especially useful when trying to detect trends of a speciﬁcstatistic vector over the provided monitor interval.
For instance, identifying latency increase ordecrease trends serves as one of the crucial indicators for adjusting sending rates.
Meanwhile theindex operator is observed from our experiments to be implicitly used for immediate responding —i.e.,based on the latest observations.
We note that the underlying decision procedure of the policy network could be efﬁciently representedin the form of a high-ﬁdelity tree-shaped form similar to Figure 4.
This decision tree contains saidcondition nodes and action nodes .
Each condition node forks into two leaf nodes based on the
Boolean result of its symbolic composition.
Attributes for Long Term CC Planning:
In addition to having these operators and operands aspart of the symbolic search space, we also attach a few attributes/ﬂags to the agent which are sharedacross consecutive MI processing steps and help with long-term planning.
One emerging behaviourin our SymbolicPCC agents is to use this attribute for remembering if the agent is in the process ofrecovering from a network overload or if the network is stable.
Indeed, a more straightforward optionfor such “multi-MI tracking” would be to just provide a longer history of the vector statistics intothe searching algorithms.
But this quickly becomes infeasible due to an exponential increase of thepossible symbolic expressions with respect to the length of vector statistic symbols .
3.3 Novel Branching Algorithm: Switching between Context-Dependent Policies
Unlike traditional visual RL environments, congestion control is a technically more demanding taskdue to the variety of possible network conditions.
The behaviour of the congestion control agentwill improve if its response is conditioned on the speciﬁc network context.
However, as this contextcannot be known by the congestion control agent, the traditional algorithms such as TCP Cubic [4]are forced to react in a slow and passive manner to generalize to both slow-paced and fast-pacedconditions.
Such a notion of context splitting and branching for training specialized agents canbe compared to earlier works in multi-task RL.
Speciﬁcally, Divide-and-Conquor [41] learns toseparate complex tasks into a set of local tasks, each of which can be used to learn a separate policy.
Meanwhile Distral [42] proposes optimizing a joint objective function by training a “distilled” policythat captures common behavior across tasks.
Hence, we propose to create n non-overlapping contexts for different network conditions, namely—bandwidth, latency, queue size, and loss rate.
We then train n individual RL agents in the PCC Gymby exposing them only to their corresponding network conditions.
We thus have a diverse set ofteachers which are each highly performant in their individual contexts.
Following the same approach
Yes &
Yes
No & Yes
Yes
No No
Yes
No & &
Start
Yes & &
No
Yes &
No & &
No
Yes Yes
No No
No
[Figure]
☐
[Figure]
& Yes
No
Condition nodes are represented as as described in Section 3.1, each of the agents are distilled—each one called a branch .
Finally,during deployment, based on the inference network conditions, the branch with the closest matchingboundary conditions/context is selected and the corresponding symbolic policy is used.
Partitioning the Networking Contexts:
A crucial point to note in the proposed branching procedure is to identify the most suitable branching context boundary values.
In other words, the bestboundary conditions for grouping need to be statistically valid, and plain hand-crafted boundaries arenot optimal.
This is because we do not have ground truths of any of the network conditions [43], letalone four of them together.
Therefore, we ﬁrst use a trained a RL agent on the default (maximal)bounds of network conditions (hereinafter known as the “baseline” agent).
We then evaluate thebaseline agent on multiple regularly spaced intervals of bandwidth, latency, queue size, and lossrate and store their corresponding return as well as observation trajectories.
To create the optimalgroupings, we simply use KMeans [44] to cluster the data based on their return.
Due to the inherentproportional relation of difﬁculty (or in this case the ballpark of return) with respect to a networkcontext, clear boundaries for the branches can be obtained by inspecting the extremes of each networkcondition within a speciﬁc cluster.
Our experimentally obtained branching conditions are furtherdiscussed in Section 4.2 and Table 1.
Branch Decider.
Since the network context is not known during deployment, this creates the needfor a branch decider module.
The branch decider reuses clusters labels from the training stagefor a K Nearest Neighbours [45] classiﬁcation.
The light-weight distance based metric is used toclassify the inference-time observation into one of the training groupings, and thereby executing thecorresponding branch’s symbolic policy.
Figure 3 illustrates our complete training and deploymenttechniques.
Lastly, in order to accommodate for branching, we have yet another long-term tracking attribute thatstores a history of branches taken in order to smooth over any erratic bouncing between brancheswhich are in non-adjacent contexts.
Experimental Settings and Results
Next, we discuss the abstract rules uncovered by SR, as well as validate the branching contexts.
In
Sections 4.3, 4.4, and 4.5 we provide emulation results on Mininet [13], a widely-used networkemulator that can emulate a variety of networking conditions.
Lastly in Section 4.6, we compare thecompute requirements and efﬁciencies of SymbolicPCC with conventional algorithms, RL-drivenmethods as well as their pruned and quantized variants.
More hyperparameter details are in Appendix
[Table]
4.1 Interpreting the Symbolic Policies
[Table]
The baseline symbolic policy distilled from the baseline RL agent is represented in its decision treeform in Figure 4.
One typical CC process presented by the tree is increasing the sending rate until thenetwork starts to “choke” and then balancing around that rate.
This process is guided with a seriesof conditions regarding to inﬂation and ratio signals, marked with circled numbers in Figure 4.
Thedetailed explanation is in the following.
Condition node 1 checks whether the vector statistic symbols are all stable—namely,whether the latency inﬂation is close to zero, while latency ratio and send ratio are close to one.
The sending rate starts to grow if the condition holds.
Condition node 2 identiﬁes if the networkis in a over-utilized status slope_of (latency inflation) increasing as the key indicator.
Itthe condition is true, the acceleration of sending rate will be reduced appropriately.
On the otherhand, condition node 3 is activated when the initial sending rate is too low or has been reducedextensively due to 2 .
4 is evaluated when major network congestion starts to occur due to increasedsending rates from the earlier condition nodes.
It checks both latency inﬂation and latency ratiosin an increasing state.
Its child nodes start reducing the sending rates and also ﬂip the internalstate attribute to 1 .
The latter is used to track if the agent is recovering from network congestion.
On the “False” side of 6 (i.e. internal state = 1 ), 7 and 8 tackle two stages of recovery,where the latency inﬂation ratio starts plateauing and then starts reducing.
11 indicates that stableconditions have been recovered again and the agent is at an optimal sending rate.
The internalstate is ﬂipped back again to 0 after this recovery.
4.2 Inspecting the Branching Conditions
285 As discussed in Section 3.3, a light-weight clustering algorithm divides the network conditions intomultiple non-overlapping subsets.
Table 1 summarizes the obtained boundary values.
The baselineagent is trained on all possible bandwidth, latency, queue size, and loss rate values, as depicted inthe ﬁrst row.
During the evaluation, bandwidth, latency, and loss rate are tested on linearly spacedvalues of step sizes 50 , 0 . 1 , and 0 .
01 , while queue sizes are exponentially spaced by powers ofe 2 respectively.
The return of the saved roll-outs are clustered using K-Means Clustering, and theoptimal cluster number is found to be 4 using the popular elbow curve [46] and silhouette analysis[47] methods.
By observing the maximum and minimum of each network condition individually inthe 4 clusters, respective boundary values are obtained.
A clear relation discovered is that higherbandwidths and lower latencies are directly related to higher baseline return.
Remark 1: Exceptions for non-overlapping contexts.
It is also to be noted that no such trend wasfound between the queue size and return, and hence all the 4 resultant branches were given the samequeue size.
A similar exception was made for the loss rates of Branches 2 and 3.
Remark 2: Interpreting the symbolic policies branches.
All the 4 distilled symbolic trees from thespecialized RL agents possess high structural similarity and share similar governing rules as to thatof the baseline agent in Section 4.1.
They majorly differ in the numerical thresholds and magnitudesof action nodes, i.e., by varying their “reaction speeds” and “reaction strengths”, respectively.
4.3 Emulation Performance on Lossy Network Conditions
The ability to differentiate between congestion-induced and random losses is essential to any PCCagent.
Figure 5a 1 shows a 25-second trace of throughput on a link where 1% of packets are randomly dropped [48].
As the link’s bandwidth is set to 30 Mbps, the ideal congestion controlwould aim to utilize it fully as depicted by the gray dotted line.
Baseline SymbolicPCC shows nearideal performance with its branched version pushing boundaries further.
In contrast, conventionalalgorithms, especially TCP CUBIC [4], repeatedly reduces its sending rates as a response to therandom losses.
Quantitative measures of mean square error with respect to the ideal line are providedin Table 2 as “Lossy ∆ 2 opt. ”.
This result proves that SymbolicPCC can effectively differentiatebetween packet loss caused by randomness and real network congestion.
4.4 Emulation Performance under Network Dynamics
[Figure]
[Figure]
Unstable network conditions are com- [Figure] mon in the real world and this testbenchmarks the agent’s ability ofquickly responding to network dynamics.
Figure 5b shows our symbolicagent’s ability to handle such conditions.
The beneﬁts of our novelbranching algorithm and switching between agents which each specializesin their own network context is clearlyvisible from faster response speeds.
Inthis case, the link was conﬁgured withits bandwidth alternating between
Mbps and 40 Mbps every 5 secondswith no loss.
Quantitative results from
Table 2 show the mean square errorwith respect to the ideal CC as “Unstable ∆ 2 opt. ”.
4.5 Link Utilization
and Network Sensitivities
Link utilization as measured from theserver side is deﬁned as the ratio of average throughput over the emulationperiod to the available bandwidth.
Asingle link is ﬁrst conﬁgured with defaults of 30 Mbps capacity, 30 ms oflatency, a 1000-packet queue, and 0%random loss.
To measure the sensitivity with respect to a speciﬁc condition,it is independently varied keeping the rest of the conditions constant.
An ideal CC preserves high linkutilization over the complete range of measurements.
From Figure 6, it is observed that our branched
SymbolicPCC provides near-capacity link-utilization at most tests and shows improvement over anyof the other algorithms.
4.6 Efﬁciency and Speed Comparisons
Since TCP congestion control lies on the fast path, ultra-fast responses are needed from the agents.
Due to its GPU compute requirements and slower runtimes, RL-based approaches such as Aurora are
[Figure] [Table]
constrained in their deployment settings (e
[Table]
g., user-space decisions). On the other hand, our symbolicpolicies are entirely composed of numerical operators, making them structurally and computationallyminimal. From our results in Table
[Table]
adding the branch decider incurs a slight overhead as comparedto the non-branched counterpart.
Nevertheless, it is preferable due to its increased versatility indifferent network conditions, as veriﬁed by Mininet emulation results.
SymbolicPCC enjoys over a23 × faster run times over Aurora, being reasonably comparable to PCC Vivace and TCP CUBIC.
We also compare between global magnitude pruned and dynamically quantized versions of Aurora.
Although these run faster than their baseline versions, they come at the cost of worse CC performance.
Discussions and Potential Impacts of SymbolicPCC
Interpretability – a universal boon for ML?
In the PCC domain, the model interpretability islinked to the wealth of domain knowledges.
By distilling a blackbox neural network into a white boxsymbolic rule, the congestion rule is made easier for the network congestion practitioners to locatethe bug and modify/improve manually.
[Table]
It may be a bit surprising that the distilled symbolic [Table] policy outperforms Aurora.
A natural question arises [Table] if it is due to a generalization ampliﬁcation thatsometimes happens for distillation in general or if it is [Table]
due to the symbolic representation.
We hypothesizethat the performance of a symbolic algorithm boils
[Table]
down to the nature of the environment it is employed
[Table] in. The congestion control problem is predominantlyrule-based, with deep RL models brought to deviserules more complex and robust than hand-crafted ones through iterative interaction.
It is onlynatural to observe that symbolic models outperform such PCC RL models when the distillation iscomposed of a rich operator space and dedicated policy denoising and pruning stages to boost theirrobustness and compactness further.
To justify this, in Table 3 we analyze the performance obtainedby decoupling distillation and symbolic representation : we ﬁrst distill a black-box NN half thesize of Aurora (“typical KD") and then further perform symbolic distillation on it.
On the possible limitations Some portion of the proposed approach is speciﬁc to the congestioncontrol setting, such as the return clustering and reward design.
Speciﬁc modiﬁcations are neededbefore the approach could be applied to other RL domains.
Conclusion and Future Work
This work studied the distillation of NN-based deep reinforcement learning agents into symbolicpolicies for performance-oriented congestion control in TCP.
Our branched symbolic frameworkenjoys better simplicity and efﬁciency while exhibiting comparable and often improved performancesover their black-box teacher counterparts on both simulation and rigorous emulation testbeds.
Ourresults point towards a fresh direction to make congestion control extremely light-weight and simpler,via a symbolic design.
Our future work aims for more integrated neural-symbolic solutions and fastermodel-free online training/ﬁne-tuning for performance-oriented congestion control.
Exploring thefairness between our learned CC and legacy CC is also an interesting next step.
Besides, we also aimto apply symbolic distillation to a wider range of systems and networking problems.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
We did not
include theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[No] Our codes and
data will be fully released upon acceptance.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
The assets we used are open-source.
The license information is available online.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[Yes] The data we are using is open source.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[No] Our data does not include personally identiﬁable
[Table]
information or offensive content.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A] [Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]
A
[Figure]
Equivalent
Require: Temporary dataset D train containing
X (numerical states), Y (actions)
Return: r : the root of symbolic policy tree
Maintain: S : the set of unsolved action nodes 1: Initializations 2: r ← newActionNode ( depth = 0) 3:
S ← { r } ; cnt ←4: While S (cid:54) = {} & cnt < cnt MAX : 5: cnt ← cnt +6: n ← pop ( S ) (cid:66)
Sample action node 7:
Y sub ← Y [ n. total _ condition] (cid:66) Slices 8:
IF Entropy ( Y sub ) <
Θ entropy : 9: n. policy ← Mean ( Y sub ) 10:
ELSE: (cid:66) Single action cannot ﬁt 11:
IF n. depth < depth MAX : 12:
With probability p 1 : (cid:66) Split condition 13: n ← newConditionNode () 14:
S ← S + { n.a LEFT , n.a RIGHT } 15:
With probability 1 − p 1 : (cid:66) De-noise 16: n. policy ← default action 17:
ELSE: (cid:66) Too deep, stop branching further 18:
With probability p 2 : 19:
X sub ← X
[ n. total _ condition] 20: n. policy ← runSR ( X sub , Y sub ) 21:
With probability p 3 : (cid:66) De-noise 22: n. policy ← default action 23:
With probability 1 − p 2 − p 3 : 24: n (cid:48) ← Sample ( pathToRoot ( n )) 25: removeSubtree ( n (cid:48) ) 26: n (cid:48) ← newConditionNode () 27:
We note that the decision procedure of a widerange of policy networks could be efﬁcientlyrepresented as high-ﬁdelity tree shaped symbolic policy.
In this tree structure, one basiccomponent – the condition node , has threekey properties: the condition , a LEFT , anda RIGHT , and could be written equivalentto one basic boolean operation, condition ∗a LEFT + ¬ condition ∗ a RIGHT , as explainedin Figure 7.
A careful and delicate “DRL behavior dataset”is to be generated and processed, which wespecify below.
Once having generated the
DRL behavior dataset, one could then applyone of the current symbolic regression benchmarks to parse out a symbolic rule that best ﬁtthe DRL behavior data.
We now specify how we build the DRL behavior dataset and process into a symbolic regression friendly format.
In general, the symbolicregression algorithms are able to evolve intoan expression that maps a vector x ∈ R d intoa scalar y ∈ R 1 , where d is the dimensionalityof the input vector.
To do so, they require adataset that stacks N Data samples of x and y ,into X ∈ R N Data ×
d and y ∈ R N Data × 1 , respectively.
Given these input/output samplepairs, i.e., ( X , y ) , a symbolic expression thatfaithfully ﬁt the data can be reliably recovered.
The overview of our symbolic distillation algorithm is provided in Table 4 and equivalentlyin Figure 8.
The genetic mutation is guided by a measuretermed program ﬁtness.
It is an indicator ofthe population of genetic programs’ performances.
The ﬁtness metric driving our evolu- tion is simply the MSE between the predicted action and the “expert” action (teacher model’s action).
We use the ﬁtness metric to determine the ﬁttest individuals of the population, essentially playing asurvival of the ﬁttest game.
These individuals are mutated before proceeding to following evolutionrounds.
We speciﬁcally follow 5 different evolution schemes, either one picked stochastically.
Theyare:
• Crossover: Requires a parent and a donor from two different evolution tournamets.
Thisscheme replaces (or) inserts a random subtree part of the donor into a random subtree partof the parent.
This mutant variant carries forth genetic material from both its sources.
• Subtree Mutation: Unlike crossover which brings “intelligent” subtrees into the parent,subtree mutation instead randomly generates it before replacing its parent.
This is more
aggressive as compared to the crossover counterpart and reintroduce extinct functions andoperators into the population to maintain diversity.
• Hoist Mutation: Being a bloat-ﬁghting mutation scheme, hoist mutation ﬁrst selects asubtree.
Then a subtree of that subtree is randomly chosen and hoists itself in the place ofthe original subtree chosen.• Point Mutation: Similar to subtree mutation, point mutation also reintroduces extinctfunctions and operators into the population to maintain diversity.
Random nodes of a tree areselected and replaced with other terminals and operators with the same arity as the chosenone.• Reproduction:
An unmodiﬁed clone of the winner is directly taken forth for the proceedingrounds.
B More experimental settings
In our training regime, the conﬁgured link bandwidth is between 100 − 500 pps, latency 50 − 500ms, queue size 2 − 2981 packets, and a loss rate between 0 − 5% .
In the MiniNet emulation, the linkbandwidth is between 0 − 100 mbps, latency 0 − 1000 ms, queue size 1 − 10000 packets, and a lossrate upto 8% .
The MiniNet conﬁguration is from its default setting, and we adopt this mismatch topurposely explore the model’s robustness.
C Extended discussions
The Interpretability.
The simple form of distilled symbolic rules provides more insights fornetworking researchers of what are the key heuristic for TCP CC.
Moreover, our success of usingsymbolic distillation for CC also paves the possibility of applying it to other systems and networkingapplications such as trafﬁc classiﬁcation and CPU scheduling tasks.
Need for Branching.
The branched training of multiple symbolic models, each in different trainingregimes, is designed to ease the optimization process.
It does not directly enforce similarity betweensolutions for the grouped states – therefore not causing brittleness .
This is assured as the symbolicmodel within any branch does not directly perform the same action for all scenarios within its regime,but contains multiple operations within itself to map states to actions based on the network stateobserved.
Also, during the inference/deployment stage, we use the branch-decider network whichchooses branches based on the observed state, not the bandwidths or latencies (in fact, these measuresare unavailable to the controller agent and cannot be observed).
[Table]
remove
[Figure]