Symbolic Distillation for Learned TCP Congestion Control
Anonymous Author(s)
AffiliationAddress email
Abstract
Recent research in TCP congestion control (CC) has witnessed tremendous successwith deep reinforcement learning (RL) approaches, which use feedforward neuralnetworks (NN) to tackle complex environment conditions and make better decisions.
However, these “black box" policies lack interpretability, and reliability and, moreimportantly, cannot operate under the TCP datapath’s ultra-contingent latencyand computational constraints.
This paper proposes a novel two-stage solutionto achieve the best of both worlds: first to train a deep RL agent, then distill its(over-)parameterized NN policy into white-box, light-weight rules in the formof symbolic expressions that are much easier to understand and to implementin constrained environments.
At the core of our proposal is a novel symbolicbranching algorithm that allows the rule to be “context-aware” of various networkconditions, eventually converting the NN policy into a symbolic tree.
The distilledsymbolic rules preserve and often improve performance over state-of-the-art NNpolicies while being orders of magnitude faster and interpretable.
We validatethe performance of our distilled symbolic rules on both simulation and emulationnetwork systems.
Our code will be released upon acceptance.
Introduction
Congestion control (CC) is fundamental to Transmission Control Protocol (TCP) communication.
Congestion occurs when the data volume sent to a network reaches or exceeds its maximal capacity,in which case the network drops excess traffic, and the performance unavoidably declines.
CCmitigates this problem by carefully adjusting the data transmission rate based on the inferred networkcapacities, aiming to send data as fast as possible without creating congestion.
For instance, a classicand best-known strategy, Additive-Increase/Multiplicative-Decrease (AIMD) [1], gradually increasesthe sending rate when there is no congestion but exponentially reduces the rate when the network iscongested.
It ensures that TCP senders fairly share the network capacity in the converged state.
Figure 1 shows an example where two TCP connections share a link between routers 1 and 2.
Whenthe shared link becomes a bottleneck, the CC algorithms running on sources A and B will alter thetraffic rate based on the feedback to avoid congestion.
Efficient and interpretable CC algorithms havebeen the bedrock for network services such as DASH video streaming, VoIP (voice-over-IP), VR/ARgames, and IoT (Internet of Things), which ride atop the TCP protocol.
However, it is nontrivial to design a high-performance CC algorithm.
Over a long history, tens of CCproposals have been made, all with different metrics and strategies to infer and address congestion,and new designs are still emerging even today [2, 3].
There are two main challenges when designinga CC algorithm.
1 it needs to precisely infer whether the network is congested, and if so, howto adjust the sending rate, based on only partial or indirect observations .
Note that CC runs onend hosts while congestion happens in the network, so CC algorithms cannot observe congestiondirectly.
Instead, it can only rely on specific signals to infer the network status.
For instance, TCP
Cubic [4] uses packet loss as a congestion signal, and TCP Vegas [5] opts for delay increase.
2 CC algorithms operate within the OS kernel, where the computing and memory resources are limited,and they need to make real-time decisions to adjust the traffic rates frequently (e.g., per round-triptime).
Therefore, the algorithm must be extraordinarily efficient .
Spending a long time to compute anaction will significantly offset the benefit of congestion control.
[Figure]
In the long history of congestion control, most [Figure] algorithms are implemented with manually-
[Figure]
designed heuristics, including New Reno [6]
[Figure]
Vegas [5], Cubic [4], and BBR [2].
In TCP
New Reno, for example, the sender doubles thenumber of transmitted packets every RTT beforereaching a predefined threshold, after which itsends one more packet every RTT.
If there isa timeout caused by packet loss, it halves the [Figure] sending rate immediately.
Manually crafted CCshave been shown to be suboptimal and cannotsupport cases that escape the heuristics [7].
Forexample, packet loss-based CCs like Cubic [4]cannot distinguish packet drops caused by congestion or non-congestion-related events [7].
Researchers have tried to construct CC algorithms with machine learning approaches to address these limitations [7–11].
The insight isthat the CC decisions are dependent on traffic patterns and network circumstances that can be exploited by deep reinforcement learning (RL) to learn a policy for each scenario.
The learned policycan perform more flexible and accurate rate adjustment by discovering a mapping from experience,which can adapt to different network conditions and save manual tuning efforts for high performance.
Most notably, Aurora [7], a deep RL framework for Performance-oriented Congestion Control (PCC),trains a well-established PPO [12] agent to suggest sending rates as actions by observing the networkstatistics such as latency ratio, send ratio, and sent latency inflation.
It achieves competitive results onemulation environments Mininet [13] and Pantheon [14], demonstrating the potential of deep learningapproaches to outperform algorithmic, hand-crafted ones.
Despite its immense success, Aurorabeing a neural network based approach, is essentially a black-box to users or, in other words, lacksexplicit declarative knowledge [15].
They also require exponentially more computation resourcesthan traditional, algorithmic methods such as the widely deployed TCP-CUBIC [4].
1.1 Our Contributions
In this work, we develop a new algorithmic framework for performance-oriented congestion control(PCC) agents, which can 1 run as fast as classical algorithmic methods; 2 adjust the rate asaccurately as data-driven RL methods; and 3 enjoy good interpretability.
We solve this problem by grasping the opportunity enabled by advances in symbolic regression[16–19].
Symbolic regression bridges the gap between the infeasible search directly in the enormoussymbolic algorithms space and the differentiable training of overparameterized and un-interpretableneural networks.
At a high level, one can first train an RL agent through gradient descent, then distill the learnedpolicy to obtain the data-driven optimized yet fully interpretable symbolic rules.
This results ina set of symbolic rules through data-driven optimization that meets TCP CC’s extreme efficiencyand reliability demands.
However, considering the enormous volume of discrete symbolic space, itis challenging to learn effective symbolic rules from scratch directly.
Therefore, in this paper, weadopt a two-stage approach: we first train a deep neural network policy with reinforcement learningmimicking Aurora [7], and then distill the resultant policy into numerical symbolic rules, usingsymbolic regression (SR).
As the challenge, directly applying symbolic regression out of the box does not yield a sufficientversatile expression that captures diverse networking conditions.
We hence propose a novel branchingtechnique for training and then aggregating a number of SymbolicPCC agents, each of which cater to asubset of the possible network conditions.
Specifically, we have multiple agents, each called a branch,and employ a light-weight “branch decider” to choose between the branches during deployment.
In order to create the branching conditions we partition the network condition space into adjacentnon-overlapping contexts, then regress symbolic rules in each context.
With this modification, weenhance the expressiveness of the resulting SR equation and overcome the bias of traditional SRalgorithm to output rules mostly using numerical operators.
Our concrete technical contributions aresummarized as follows:
• We propose a symbolic distillation framework for TCP congestion control, which improves upon
[Figure]
the state-of-the-art RL solutions.
Our approach, SymbolicPCC , consists of two stages: first
[Figure]
training an RL agent and then distilling its policy network into ultra-compact and interpretable
[Figure]
rules in the symbolic expression form.
[Figure]
• We propose a novel branching technique that advances existing symbolic regression techniques
[Figure]
for training and aggregating multiple context-dependent symbolic policies, each of which
[Figure]
specializes for its own subset of network conditions.
A branch decider driven by light-weight
[Figure]
classification algorithms determines which symbolic policy to use.
[Figure]
• Through our simulation and emulation experiments
SymbolicPCC achieves highly competitive
[Figure]
or even stronger performance results compared to their teacher policy networks while running
[Figure]
orders of magnitude faster.
The presented model uses a tree structure that is light-weight and
[Figure]
could be simpler for practitioners to reason about and improve manually while narrowing their [Figure] performance gap.
[Figure]
Related Works
[Figure]
Conventional TCP CC adopts aheuristic-based approach where theheuristic functions are manuallycrafted to adjust the traffic rate in a deterministic manner.
Some proposalsuse packet loss as a signal for networkcongestion, e.g., Cubic [4], Reno [20],and NewReno [6], and others rely onthe variation of delay, e.g., Vegas [5].
Other CC designs combine packet lossand delay [21, 22].
Recently, different CC techniques specialized for data-center networks are alsoproposed [2, 3, 23].
Researchers have also investigated the use of machine learning to construct better heuristics.
Indigo [10] and Remy [11] use offline learning to obtain high-performance CC algorithms.
PCC [24]and PCC Vivace [9] opt for online learning to avoid any hardwired mappings between states andactions.
Aurora [7] utilizes deep reinforcement learning to obtain a new CC algorithm running inthe user space.
Orca [8] improves upon Aurora and designs a user-space CC agent that infrequentlyconfigures kernel-level CC policies.
Our proposal further improves these work.
On the other hand, Symbolic regression methods have [16–19] recently emerged for discoveringunderlying math equations that govern the observed data.
Algorithms with such a property are morefavorable for real-world deployment as they output white-box rules.
[17] use genetic programmingbased method while [19] uses a recurrent neural network to perform a search in the symbolic space.
We thus propose to synergize such numerical and data-driven approaches using symbolic regression(SR) in the congestion control domain.
We use SR by following a post-hoc method of first training an
RL algorithm then distilling it into its symbolic rules.
Earlier methods that follow a similar proceduredo exist, e.g., [25] distills the learned policy as a soft decision tree.
They work on visual RL wherethe image observations are coarsely quantized into 10 × 10 cells, and the soft decision tree policyis learned over the 100 dimensional space.
[26] also aims to learn abstract rules using a commonsense-based approach by modifying Q learning algorithms.
Nevertheless, they fail to generalizebeyond the specific simple grid worlds they were trained in.
[27] learns from boolean feature sets, [28]directly approximates value functions based on a state-transition model, [29] optimizes risk-seekingpolicy gradients.
Other works on abstracting pure symbolic rules from data include attention-basedmethods [30], visual summary based methods [31], reward decomposition based methods [32], causalmodel based methods [33], markov chain
based methods [34], and case-based expert-behaviourretrieval methods [35].
RL Baseline
Ofine Dataset of Trajectories
Symbolic Policy (unbranched)
Branching Boundary Conditions
RL 1 RL 2 RL 3 RLtrain specialized RL agents
Sym1 Sym2 Sym
Sym
D D
D D
Ofine Datasets of Trajectories
Symbolic Policy (branched) symbolic distillation collect rollouts symbolic distillation collect rollouts reward clustering
Environment
Branch decider observations
Symbolic Policy (branched) branch
o b s e r v a t i o n s actions
Training SymbolicPCC
Evaluation/DeploymentSymbolicPCC clusteringalgorithm
Methodology
[Figure]
Inspired from the idea of “teacher-student knowledge distillation” [36], our symbolic distillationtechnique is two-staged—first train regular RL agents (as the teacher ), then distill the learned policynetworks into their white-box symbolic counterparts (as the student ).
In Section 3.1 we follow [7]’sapproach in Aurora and the training of teacher agents on the PCC-RL gym environment.
We alsobriefly discuss the approach of applying symbolic regression to create a light-weight numerical-drivenexpression that approximates a given teacher’s behavior.
In Section 3.2 we look at the specifics ofsymbol spaces and attach internal attributes to aid long-term planning.
Finally, in Section 3.3 wediscuss our novel branching algorithm as a method for training, then ensembling multiple contextdependent symbolic agents during deployment.
3.1 Preliminaries: The PCC-RL Environment and the Symbolic Distillation Workflow
PCC-RL [7] is an open-source RL testbed for simulation of congestion control agents based on thepopular OpenAI Gym [37] framework.
We adopt it as our main playground.
It formulates congestioncontrol as a sequential decision making problem.
Time is first divided into multiple periods called
MIs (monitor intervals), following [24].
At the onset of each MI, the environment provides the agentwith the history of statistic vector observations over the network, and the agent responds with adjustedsending rates for the following MI.
The sending rate remains fixed during a single MI.
The network statistics provided as observations to the congestion control agent are 1 the latencyinflation, 2 the latency ratio, and 3 the sending ratio.
The agent is guided by reward signalsbased on its ability to react appropriately when detecting changes and trends in the vector statisticsof the PCC-RL environment.
It is provided with positive return for higher values of throughputs(packets/second) while being penalized for higher values of latency (seconds) and loss (ratio of sentvs.
acknowledged packets).
Training of Teacher Agents: We first proceed to train RL agents using the PPO algorithm [12]similar to Aurora [7] in the PCC-RL gym environment till convergence.
Although they statisticallyperform very well [7], the PPO agents are entirely black-boxes; this makes it difficult to explain itsunderlying causal rules directly.
Also, their over-parameterized neural network forms incur highlatency.
Hence, we choose to indirectly learn the symbolic representations using a student-teachertype knowledge distillation approach based on the teacher’s
(in this case, the RL agent) behaviors.
Distillation of Student Agents: Using the teacher agents, we collect complete trajectories, formallyknown as roll-outs in RL, in an inference mode—deterministic actions are enforced.
The observations and their corresponding teacher actions are MI-aligned and stored as an offline dataset.
Note thatthis step is only performed once at the start of the distillation procedure and is reused in each of itsiterative steps.
A search space of operators and operands is also initialized (details are discussedshortly in Section 3.2).
Guesses for possible symbolic relations are taken, composed of randomoperators and operands from their respective spaces.
The stored observation trajectories are thenre-evaluated based on this rule to output corresponding actions.
The cross-entropy loss with respectto the teacher model’s actions from the same dataset is used as feedback.
This feedback drivesthe iterative mutation and pruning following a genetic programming technique [38, 39].
The bestcandidate policies are collected and forwarded to the next stage.
If the tree fails to converge or doesnot reach a specific threshold of acceptance, the procedure is restarted from scratch.
Our symbolicdistillation method is discussed with further details in the Appendix A.
3.2 A Symbolic Framework for Congestion Control
Defining the Symbol Space for CC: Unlike visual RL [40], the PCC observation space is vectorbased, hence we directly plug them into the search space of our numerical-driven symbolic algorithm.
We henceforth call these observations as vector statistic symbols .
The distillation procedureas described earlier learns to chain these vector statistic symbols using a pre-defined operatorspace.
Specifically, we employ three types of numerical operators.
The first type of operators arearithmetic based which include + , − , ∗ , /, sin , cos , tan , cot , ( · ) 2 , ( · ) 3 , √ · , exp , log , |·| .
The secondtype of operators are Boolean logic, such as is ( x < y ), is ( x ≤ y ), is ( x == y ), a | b , a & b , and ¬ a .
We also utilize a third type of high level operators – namely the slope_of ( observation history )which provides the average slope of an array of observations, and get_value_at ( observationhistory , index ).
The slope operator is especially useful when trying to detect trends of a specificstatistic vector over the provided monitor interval.
For instance, identifying latency increase ordecrease trends serves as one of the crucial indicators for adjusting sending rates.
Meanwhile theindex operator is observed from our experiments to be implicitly used for immediate responding —i.e.,based on the latest observations.
We note that the underlying decision procedure of the policy network could be efficiently representedin the form of a high-fidelity tree-shaped form similar to Figure 4.
This decision tree contains saidcondition nodes and action nodes .
Each condition node forks into two leaf nodes based on the
Boolean result of its symbolic composition.
Attributes for Long Term CC Planning:
In addition to having these operators and operands aspart of the symbolic search space, we also attach a few attributes/flags to the agent which are sharedacross consecutive MI processing steps and help with long-term planning.
One emerging behaviourin our SymbolicPCC agents is to use this attribute for remembering if the agent is in the process ofrecovering from a network overload or if the network is stable.
Indeed, a more straightforward optionfor such “multi-MI tracking” would be to just provide a longer history of the vector statistics intothe searching algorithms.
But this quickly becomes infeasible due to an exponential increase of thepossible symbolic expressions with respect to the length of vector statistic symbols .
3.3 Novel Branching Algorithm: Switching between Context-Dependent Policies
Unlike traditional visual RL environments, congestion control is a technically more demanding taskdue to the variety of possible network conditions.
The behaviour of the congestion control agentwill improve if its response is conditioned on the specific network context.
However, as this contextcannot be known by the congestion control agent, the traditional algorithms such as TCP Cubic [4]are forced to react in a slow and passive manner to generalize to both slow-paced and fast-pacedconditions.
Such a notion of context splitting and branching for training specialized agents canbe compared to earlier works in multi-task RL.
Specifically, Divide-and-Conquor [41] learns toseparate complex tasks into a set of local tasks, each of which can be used to learn a separate policy.
Meanwhile Distral [42] proposes optimizing a joint objective function by training a “distilled” policythat captures common behavior across tasks.
Hence, we propose to create n non-overlapping contexts for different network conditions, namely—bandwidth, latency, queue size, and loss rate.
We then train n individual RL agents in the PCC Gymby exposing them only to their corresponding network conditions.
We thus have a diverse set ofteachers which are each highly performant in their individual contexts.
Following the same approachas described in Section 3.1, each of the agents are distilled—each one called a branch .
Finally,
Yes &
Yes
No & Yes
Yes
No No
Yes
No & &
Start
Yes & &
No
Yes &
No & &
No
Yes Yes
No
No No
Yes
Key - Latency Ratio - [Figure] Ratio - Sent Latency Ination ☐ - Latest Value &
Yes
No
Condition nodes are during deployment, based on the inference network conditions, the branch with the closest matchingboundary conditions/context is selected and the corresponding symbolic policy is used.
Partitioning the Networking Contexts:
A crucial point to note in the proposed branching procedure is to identify the most suitable branching context boundary values.
In other words, the bestboundary conditions for grouping need to be statistically valid, and plain hand-crafted boundaries arenot optimal.
This is because we do not have ground truths of any of the network conditions [43], letalone four of them together.
Therefore, we first use a trained a RL agent on the default (maximal)bounds of network conditions (hereinafter known as the “baseline” agent).
We then evaluate thebaseline agent on multiple regularly spaced intervals of bandwidth, latency, queue size, and lossrate and store their corresponding return as well as observation trajectories.
To create the optimalgroupings, we simply use KMeans [44] to cluster the data based on their return.
Due to the inherentproportional relation of difficulty (or in this case the ballpark of return) with respect to a networkcontext, clear boundaries for the branches can be obtained by inspecting the extremes of each networkcondition within a specific cluster.
Our experimentally obtained branching conditions are furtherdiscussed in Section 4.2 and Table 1.
Branch Decider.
Since the network context is not known during deployment, this creates the needfor a branch decider module.
The branch decider reuses clusters labels from the training stagefor a K Nearest Neighbours [45] classification.
The light-weight distance based metric is used toclassify the inference-time observation into one of the training groupings, and thereby executing thecorresponding branch’s symbolic policy.
Figure 3 illustrates our complete training and deploymenttechniques.
Lastly, in order to accommodate for branching, we have yet another long-term tracking attribute thatstores a history of branches taken in order to smooth over any erratic bouncing between brancheswhich are in non-adjacent contexts.
Experimental Settings and Results
Next, we discuss the abstract rules uncovered by SR, as well as validate the branching contexts.
In
Sections 4.3, 4.4, and 4.5 we provide emulation results on Mininet [13], a widely-used networkemulator that can emulate a variety of networking conditions.
Lastly in Section 4.6, we compare thecompute requirements and efficiencies of SymbolicPCC with conventional algorithms, RL-drivenmethods as well as their pruned and quantized variants.
More hyperparameter details are in Appendix
C
4.1 Interpreting the Symbolic Policies
The baseline symbolic policy distilled from the baseline RL agent is represented in its decision treeform in Figure 4.
One typical CC process presented by the tree is increasing the sending rate until the
[Table]
network starts to “choke” and then balancing around that rate.
This process is guided with a seriesof conditions regarding to inflation and ratio signals, marked with circled numbers in Figure 4.
Thedetailed explanation is in the following.
Condition node 1 checks whether the vector statistic symbols are all stable—namely,whether the latency inflation is close to zero, while latency ratio and send ratio are close to one.
The sending rate starts to grow if the condition holds.
Condition node 2 identifies if the networkis in a over-utilized status slope_of (latency inflation) increasing as the key indicator.
Itthe condition is true, the acceleration of sending rate will be reduced appropriately.
On the otherhand, condition node 3 is activated when the initial sending rate is too low or has been reducedextensively due to 2 .
4 is evaluated when major network congestion starts to occur due to increasedsending rates from the earlier condition nodes.
It checks both latency inflation and latency ratiosin an increasing state.
Its child nodes start reducing the sending rates and also flip the internalstate attribute to 1 .
The latter is used to track if the agent is recovering from network congestion.
On the “False” side of 6 (i.e. internal state = 1 ), 7 and 8 tackle two stages of recovery,where the latency inflation ratio starts plateauing and then starts reducing.
11 indicates that stableconditions have been recovered again and the agent is at an optimal sending rate.
The internalstate is flipped back again to 0 after this recovery.
4.2 Inspecting the Branching Conditions
As discussed in Section 3.3, a light-weight clustering algorithm divides the network conditions intomultiple non-overlapping subsets.
Table 1 summarizes the obtained boundary values.
The baselineagent is trained on all possible bandwidth, latency, queue size, and loss rate values, as depicted inthe first row.
During the evaluation, bandwidth, latency, and loss rate are tested on linearly spacedvalues of step sizes 50 , 0 . 1 , and 0 .
01 , while queue sizes are exponentially spaced by powers ofe 2 respectively.
The return of the saved roll-outs are clustered using K-Means Clustering, and theoptimal cluster number is found to be 4 using the popular elbow curve [46] and silhouette analysis[47] methods.
By observing the maximum and minimum of each network condition individually inthe 4 clusters, respective boundary values are obtained.
A clear relation discovered is that higherbandwidths and lower latencies are directly related to higher baseline return.
Remark 1: Exceptions for non-overlapping contexts.
It is also to be noted that no such trend wasfound between the queue size and return, and hence all the 4 resultant branches were given the samequeue size.
A similar exception was made for the loss rates of Branches 2 and 3.
Remark 2: Interpreting the symbolic policies branches.
All the 4 distilled symbolic trees from thespecialized RL agents possess high structural similarity and share similar governing rules as to thatof the baseline agent in Section 4.1.
They majorly differ in the numerical thresholds and magnitudesof action nodes, i.e., by varying their “reaction speeds” and “reaction strengths”, respectively.
4.3 Emulation Performance on Lossy Network Conditions
The ability to differentiate between congestion-induced and random losses is essential to any PCCagent.
Figure 5a 1 shows a 25-second trace of throughput on a link where 1% of packets arerandomly dropped [48].
As the link’s bandwidth is set to 30 Mbps, the ideal congestion controlwould aim to utilize it fully as depicted by the gray dotted line.
Baseline SymbolicPCC shows nearideal performance with its branched version pushing boundaries further.
In contrast, conventional algorithms, especially TCP CUBIC [4], repeatedly reduces its sending rates as a response to therandom losses.
Quantitative measures of mean square error with respect to the ideal line are providedin Table 2 as “Lossy ∆ 2 opt. ”.
This result proves that SymbolicPCC can effectively differentiatebetween packet loss caused by randomness and real network congestion.
4.4 Emulation Performance under Network Dynamics
[Figure]
Unstable network conditions are common in the real world and this testbenchmarks the agent’s ability ofquickly responding to network dynamics.
Figure 5b shows our symbolicagent’s ability to handle such conditions.
The benefits of our novelbranching algorithm and switching between agents which each specializesin their own network context is clearlyvisible from faster response speeds.
Inthis case, the link was configured withits bandwidth alternating between
Mbps and 40 Mbps every 5 secondswith no loss.
Quantitative results from
Table 2 show the mean square errorwith respect to the ideal CC as “Unstable ∆ 2 opt. ”.
4.5 Link Utilizationand Network Sensitivities
Link utilization as measured from theserver side is defined as the ratio of average throughput over the emulationperiod to the available bandwidth.
Asingle link is first configured with defaults of 30 Mbps capacity, 30 ms oflatency, a 1000-packet queue, and 0%random loss.
To measure the sensitivity with respect to a specific condition,it is independently varied keeping the rest of the conditions constant.
An ideal CC preserves high linkutilization over the complete range of measurements.
From Figure 6, it is observed that our branched
SymbolicPCC provides near-capacity link-utilization at most tests and shows improvement over anyof the other algorithms.
4.6 Efficiency and Speed Comparisons
Since TCP congestion control lies on the fast path, ultra-fast responses are needed from the agents.
Due to its GPU compute requirements and slower runtimes, RL-based approaches such as Aurora areconstrained in their deployment settings (e.g., user-space decisions).
On the other hand, our symbolicpolicies are entirely composed of numerical operators, making them structurally and computationallyminimal.
From our results in Table 2, adding the branch decider incurs a slight overhead as compared
[Figure]
[Figure]
[Table]
to the non-branched counterpart.
Nevertheless, it is preferable due to its increased versatility indifferent network conditions [Table] as verified by Mininet emulation results.
SymbolicPCC enjoys over a23 × faster run times over Aurora, being reasonably comparable to PCC Vivace and TCP CUBIC.
We also compare between global magnitude pruned and dynamically quantized versions of Aurora.
Although these run faster than their baseline versions, they come at the cost of worse CC performance.
Discussions and Potential Impacts of SymbolicPCC
Interpretability – a universal boon for ML?
In the PCC domain, the model interpretability islinked to the wealth of domain knowledges.
By distilling a blackbox neural network into a white boxsymbolic rule, the congestion rule is made easier for the network congestion practitioners to locatethe bug and modify/improve manually. [Table]
It may be a bit surprising that the distilled symbolicpolicy outperforms Aurora.
A natural question arisesif it is due to a generalization amplification thatsometimes happens for distillation in general or if it isdue to the symbolic representation.
We hypothesizethat the performance of a symbolic algorithm boils [Table] down to the nature of the environment it is employedin.
The congestion control problem is predominantlyrule-based, with deep RL models brought to deviserules more complex and robust than hand-crafted ones through iterative interaction.
It is onlynatural to observe that symbolic models outperform such PCC RL models when the distillation iscomposed of a rich operator space and dedicated policy denoising and pruning stages to boost theirrobustness and compactness further.
To justify this, in Table 3 we analyze the performance obtainedby decoupling distillation and symbolic representation : we first distill a black-box NN half thesize of Aurora (“typical KD") and then further perform symbolic distillation on it.
On the possible limitations Some portion of the proposed approach is specific to the congestioncontrol setting, such as the return clustering and reward design.
Specific modifications are neededbefore the approach could be applied to other RL domains.
Conclusion and Future Work
This work studied the distillation of NN-based deep reinforcement learning agents into symbolic policies for performance-oriented congestion control in TCP.
Our branched symbolic framework enablesbetter interpretability and efficiency while exhibiting comparable and often improved performancesover their black-box teacher counterparts on both simulation and rigorous emulation testbeds.
Ourresults point towards a fresh direction to make congestion control extremely light-weight and moreinterpretable, via a symbolic design.
Our future work aims for more integrated neural-symbolic solutions and faster model performance-oriented congestion control.
Exploring the fairness between our learned CC and legacy CC is also an interesting next step.
Besides,we also aim to apply symbolic distillation to a wider range of systems and networking problems.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
We did not
include theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Our codes and data will be fully released upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
The assets we used are open-source.
The license information is available online.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] The data we are using is open source.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] Our data does not include personally identifiable
information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
A
Algorithm
[Figure]
Equivalent
Require: Temporary dataset D train containing
X (numerical states), Y (actions)
Return: r : the root of symbolic policy tree
Maintain: S : the set of unsolved action nodes 1: Initializations 2: r ← newActionNode ( depth = 0) 3:
S ← { r } ; cnt ←4: While S ̸ = {} & cnt < cnt MAX : 5: cnt ← cnt +6: n ← pop ( S )
▷ Sample action node 7:
Y sub ← Y [ n. total _ condition] ▷ Slices 8:
IF Entropy ( Y sub ) <
Θ entropy : 9: n. policy ← Mean ( Y sub ) 10:
ELSE: ▷ Single action cannot fit 11:
IF n. depth < depth MAX : 12:
With probability p 1 : ▷ Split condition 13: n ← newConditionNode () 14:
S ← S + { n.a LEFT , n.a RIGHT } 15:
With probability 1 − p 1 : ▷ De-noise 16: n. policy ← default action 17:
ELSE: ▷ Too deep, stop branching further 18:
With probability p 2 : 19:
X sub ← X
[ n. total _ condition] 20: n. policy ← runSR ( X sub , Y sub ) 21:
With probability p 3 : ▷ De-noise 22: n. policy ← default action 23:
With probability 1 − p 2 − p 3 : 24: n ′ ← Sample ( pathToRoot ( n )) 25: removeSubtree ( n ′ ) 26: n ′ ← newConditionNode () 27: S ← S + { n ′ .a LEFT , n ′ .a
RIGHT } 28: Return r
We note that the decision procedure of a widerange of policy networks could be efficientlyrepresented as high-fidelity tree shaped symbolic policy.
In this tree structure, one basiccomponent – the condition node , has threekey properties: the condition , a LEFT , anda RIGHT , and could be written equivalentto one basic boolean operation, condition ∗a LEFT + ¬ condition ∗ a RIGHT , as explainedin Figure 7.
A careful and delicate “DRL behavior dataset”is to be generated and processed, which wespecify below.
Once having generated the
DRL behavior dataset, one could then applyone of the current symbolic regression benchmarks to parse out a symbolic rule that best fitthe DRL behavior data.
We now specify how we build the DRL behavior dataset and process into a symbolic regression friendly format.
In general, the symbolicregression algorithms are able to evolve intoan expression that maps a vector x ∈ R d intoa scalar y ∈ R 1 , where d is the dimensionalityof the input vector.
To do so, they require adataset that stacks N Data samples of x and y ,into X ∈ R N Data ×
d and y ∈ R N Data × 1 , respectively.
Given these input/output samplepairs, i.e., ( X , y ) , a symbolic expression thatfaithfully fit the data can be reliably recovered.
The overview of our symbolic distillation algorithm is provided in Table 4 and equivalentlyin Figure 8.
The genetic mutation is guided by a measuretermed program fitness.
It is an indicator ofthe population of genetic programs’ performances.
The fitness metric driving our evolu- tion is simply the MSE between the predicted action and the “expert” action (teacher model’s action).
We use the fitness metric to determine the fittest individuals of the population, essentially playing asurvival of the fittest game.
These individuals are mutated before proceeding to following evolutionrounds.
We specifically follow 5 different evolution schemes, either one picked stochastically.
Theyare:
• Crossover: Requires a parent and a donor from two different evolution tournamets.
This
scheme replaces (or) inserts a random subtree part of the donor into a random subtree partof the parent.
This mutant variant carries forth genetic material from both its sources.
• Subtree Mutation: Unlike crossover which brings “intelligent” subtrees into the parent, subtree mutation instead randomly generates it before replacing its parent.
This is more
aggressive as compared to the crossover counterpart and reintroduce extinct functions and operators into the population to maintain diversity.
• Hoist Mutation: Being a bloat-fighting mutation scheme, hoist mutation first selects a subtree.
Then a subtree of that subtree is randomly chosen and hoists itself in the place ofthe original subtree chosen.• Point Mutation: Similar to subtree mutation, point mutation also reintroduces extinct functions and operators into the population to maintain diversity.
Random nodes of a tree areselected and replaced with other terminals and operators with the same arity as the chosenone.• Reproduction:
An unmodified clone of the winner is directly taken forth for the proceedingrounds.
B More experimental settings
In our training regime, the configured link bandwidth is between 100 − 500 pps, latency 50 − 500ms, queue size 2 − 2981 packets, and a loss rate between 0 − 5% .
In the MiniNet emulation, the linkbandwidth is between 0 − 100 mbps, latency 0 − 1000 ms, queue size 1 − 10000 packets, and a lossrate upto 8% .
The MiniNet configuration is from its default setting, and we adopt this mismatch topurposely explore the model’s robustness.
C Extended discussions
The Interpretability.
The interpretability of our approach enables researchers to verify the learnedmodel, which is critical for both performance and security objectives in practical deployments.
Italso provides more insights for networking researchers of what are the key heuristic for TCP CC.
Moreover, our success of using symbolic distillation for CC also paves the possibility of applying itto other systems and networking applications where performance and interpretability are both keyconsideration, such as traffic classification and CPU scheduling tasks.
Need for Branching.
The branched training of multiple symbolic models, each in different trainingregimes, is designed to ease the optimization process.
It does not directly enforce similarity betweensolutions for the grouped states – therefore not causing brittleness .
This is assured as the symbolicmodel within any branch does not directly perform the same action for all scenarios within its regime,but contains multiple operations within itself to map states to actions based on the network stateobserved.
Also, during the inference/deployment stage, we use the branch-decider network whichchooses branches based on the observed state, not the bandwidths or latencies (in fact, these measuresare unavailable to the controller agent and cannot be observed).
[Table]
remove
[Figure]