How ISPs manage Routing Traffic? by Abhilasha Dave, Rithika Shyam Chari, Swati Chaturvedi

## **Short Answer**

The majority of our daily hours are spent on the World Wide Web in one form or the other. This amount to billions and billions of bits of data zipping around per second, the traffic density is therefore mind boggling. We hardly realize the complexity of the background processes that take place to enable us hassle free, seamless internet interactions. We try to fathom the intricacies behind this in our topic. Though the breadth and depth of networks is exhaustive, we focus on the methodologies and algorithms adopted by Internet Service Providers (ISPs) to manage traffic efficiently in an ever busy packed network.

We start with the ways through which the ISPs route traffic by finding the shortest path. The algorithms discussed herein are the Djikastra algorithm and the Bellman-Ford algorithm.

Now the question to ask is, are efficient routing algorithms alone good enough to manage traffic on the internet? The answer is No. Even the best of routing algorithm do not suffice if the traffic management is badly handled. An everyday analogy would be to imagine a crowded traffic junction unregulated without traffic lights or a traffic cop!

We shall look at briefly the Congestion Management Framework deployed by Comcast across its whole network. This new framework was developed in response to complaints to the U.S. Federal Communications Commission (FCC), over its old system, where they throttled specific Peer to Peer networks as a workaround to temporarily ease out the congestion in their networks. The new congestion management framework was called protocol-agnostic. As the name suggests, it would neither examine nor affect specific applications or network protocols. So how does it exactly manage traffic? It targets the individual nodes which consume large bandwidth during congested time frames.

A very concise for managing traffic is explained as follows:
Basically, there are four stages for figuring out if the activity connected with a specific link modem is assigned as PBE or BE:

1. Figure out whether a port is in a Near Congestion State or not.

2. In the event that the answer to above is yes, then, figure out if any users are in an Extended High Utilization State.

3. In the event that yes, change those clients' priority to lower one from default high. If it is a No at either of the two above steps, no action is required.

4. In the event that a user’s movement has been assigned a lower priority, check user utilization periodically. If utilization has reduced beneath the prefixed thresholds, reassign the user’s priority as original top priority.

This above is implemented by Comcast using a new set of hardware and software. The Quality of Service concept is used and the appropriate QoS models are used to model the traffic given different objective functions and constraints. The detailed explanation of the QoS models is provided in the long answer.

## **Long Answer**

### **Dijkstra Algorith**

Dijkstra's calculation is a calculation for finding the most limited ways between hubs in a diagram, which may speak to, for instance, street systems. It was brought about by PC researcher Edsger W. Dijkstra in 1956 and distributed three years after the fact. The calculation exists in numerous variations; Dijkstra's unique variation found the most limited way between two nodes, yet a more regular variation alters a solitary hub as the "source" hub and finds briefest ways from the source to every single other hub in the chart, creating a briefest way tree.
For a given source hub in the chart, the calculation finds the most brief way between that hub and each other. It can likewise be utilized for finding the most limited ways from a solitary hub to a solitary destination hub by ceasing the calculation once the briefest way to the destination hub has been resolved. For instance, if the hubs of the diagram speak to urban communities and edge way costs speak to driving separations between sets of urban communities associated by an immediate street, Dijkstra's calculation can be utilized to locate the most brief course between one city and all different urban communities. Thus, the briefest way calculation is generally utilized as a part of system steering conventions, most outstandingly IS-IS and Open Shortest Path First (OSPF). It is likewise utilized as a subroutine in different calculations, for example, Johnson's.
Dijkstra's calculation basically utilizes broadness first pursuit with avaricious way to deal with think of the most brief separation between given two vertices.
Let the hub at which we are beginning be known as the underlying hub. We characterize the separation of hub "Y" as the separation from the underlying hub/beginning hub to hub 'Y'. Dijkstra's calculation will dole out some underlying separation values and will attempt to enhance them orderly.

1. Dole out to each hub a conditional separation esteem: set it to zero for our underlying hub and to infinity for every other hub. This progression connotes that toward the begin of the calculation, the beginning hub is at separation 0 from itself and different hubs are inaccessible.

2. Set the underlying hub as present. Stamp every other hub unvisited. Make an arrangement of all the unvisited hubs called the unvisited set.

3. For the present hub, consider the majority of its unvisited neighbors and ascertain their speculative separations. Contrast the recently figured provisional separation with the current appointed esteem and dole out the littler one. For instance, if the present hub An is set apart with a separation of 6, and theedge associating it with a neighbor B has length 2, then the separation to B (through A) will be 6 + 2 = 8. In the event that B was beforehand set apart with a separation more prominent than 8 then change it to 8. Something else, keep the present quality.

4. When we are done considering the majority of the neighbors of the present hub, check the present hub as went to and expel it from the unvisited set. A went by hub will never be checked again.

5. In the event that the destination hub has been stamped gone to (when arranging a course between two particular hubs) or if the littlest speculative separation among the hubs in the unvisited set is endlessness (when arranging a complete traversal; happens when there is no association between the underlying hub and remaining unvisited hubs), then stop. The calculation has wrapped up.

6. Something else, select the unvisited hub that is set apart with the littlest provisional separation, set it as the new "current hub", and backtrack to step 3.
Dijkstra Algorithm Pseudo Code.1

#### **Dijkstra Applications Extensions and Alternatives**

There are distinctive applications and unique situations where the Dijkstra calculation can be connected. Notwithstanding directing, the computation of separations can likewise be utilized for different zones where the Euclidian separation is not the premise, but rather time or cost is.

Notwithstanding the weighting of the edges, weights to the hub can likewise be indicated. This could be useful when the procedure of changing inside the hub "principle station" is connected with high expenses. For this situation, the hub gets its own weight, which must be considered while processing the most brief ways.

The Dijkstra calculation does not work with negative edge weights. In the event that you need to run a briefest way figuring with negative egde weights, different calculations, for example, the Bellman-Ford-calculation must be utilized.

The Dijkstra calculation is not reasonable for all applications or sorts of diagrams. Different calculations incorporate the Kruskal or Borùvka which are utilized to process least crossing trees in undirected charts.

### **Bellman Ford Algorithm**

Bellman Ford algorithm is use to find the shortest path in the graph. Bellman ford and Dijkstra algorithm have same end result that both algorithms provide shortest path. Bellman ford can support for negative weighted graph as well. This algorithm named after Richard Bellman and Lester Ford, Jr., they published it in 1958 and 1956 respectively. In 1957 Edward F. Moore published similar algorithm. Thus, it also called as Bellman-Ford-Moore algorithm [2].

**Flow chart of Bellman Ford Algorithm is given below**

- |v| = Number of vertices
- |v|-1 = Number of iteration
- v.d = Adjacent vertices
- u.d = Source
- w(u.v) = link cost

#### **Algorithm with example**

**Steps for Bellman Ford Algorithm:**

Here, packet needs to be transmitting from node S (source) to node D (destination).

**Step1:**
- Initially A = B = C = E = D = ∞
- S = 0.
- v.d = A = ∞
- u.d = S = 0
- Check the condition (v.d = ∞) > (u.d = 0) + (w(u.v = 6))
- Yes the condition is matching. Thus, we update v.d = A = 6.

**Step2:**

 This step we can take any of the nodes from A and B as a source node. In this step next source node is A.
- v.d = C and B = ∞
- u.d = A = 6
- w(u.v) = 5 for node C.
- w(u.v) = -3 for node B.
- For node C:
- Check condition (v.d = ∞)>(u.d = A = 6) + (w(u.v = 5))
- Yes the condition is matching. Thus, we update v.d = C = 11.
- For node B:
- Check condition (v.d = ∞)>(u.d = A = 6) + (w(u.v = -3))
- Yes the condition is matching. Thus, we update v.d = B = 3.

This step we will follow for all the nodes and set the value of all the nodes; that is our first iteration. In next iteration again follow the same steps; it will start from source S.

If after |v|-1 iteration still the values of nodes are being changed which means the negative cycle is present in the graph.

#### **Disadvantages of Bellman Ford**

- Count to infinity.
- Do not work for negative cycle.

To explain this both point we can take one example

Here, the packet should be transfer from node A to node E. One path is A-B-C-E but Bellman Ford will also check the path B-C-D-B which is actually a negative cycle because the path cost is 1-4+2 = -1 so each time packet rout from this cycle its path cost will reduce by one; packet will be route through this path for infinite time because every time it is reducing by one.
- When the router shuts down.

Imagine a big complex network in which N number of routers are connected and with that routers N number of nodes are connected so if one router shuts down so we cannot route to the node which are connected with that router.
- Slow algorithm as compare to Dijkstra algorithm.
- Cannot be used for large network.

#### **Difference between Bellman Ford and Dijkstra Algorithm:**

The main difference between Bellman Ford and Dijkstra is “Dijkstra is a greedy algorithm whereas Bellman Ford is not”. To understand this as shown in above figure we wanted to transfer packet between nodes S to A.

Firstly let’s focus on part a of the figure in this all the links are positive; shortest path for both the algorithm Dijkstra and Bellman Ford is same node S to node A and minimum link cost is 3.

Secondly let’s focus on part B of the figure in this the link cost from node B to A has been change from +2 to -2.Thus, the final output for both the algorithm will change because Dijkstra focus only on immediate nodes whereas Bellman Ford will focus on all-inclusive nodes. Thus, path cost from node S-B-A is less as compare to path cost between S-A so the shortest path for Bellman Ford algorithm is S-B-A. However, for Dijkstra the shortest path is S-A.

### **Congestion management fream work**

Even the best of routing algorithms shall not suffice if congestion in the networks is not properly managed. In this section we will look at how a major ISP manages it’s traffic congestion.

#### **Comcast’s Protocol Agnostic Congestion Management System**

Comcast is a US-based, large broadband internet service provider. During late 2008 , Comcast introduced an improvised Congestion Management System across all its networks. This new framework was developed in response to complaints by the public to the U.S. Federal Communications Commission (FCC) , over its old system, wherein they targeted specific Peer to Peer networks to ease out congestion in their networks. This new congestion management framework was termed as protocol-agnostic, implying that, it would neither examine nor affect specific applications or network protocols. Hence, this was termed to be a much fairer system for managing congestions. So who or what exactly does it target to manage traffic? It targets the individual nodes which are consuming large bandwidth during congestion time frames.

The framework was implemented using both new hardware and software. It involved a lot real world tests to analyze traffic, third party software, collecting data from the actual networks, several simulations and lab tests and took a lot of trial runs to perfect it[5].

Generally, one might wonder why there must be congestion at all in the first place when the network is said to be adequately equipped to handle the capacity. There are two types of network congestions namely regular and unpredictable.

Regular type of congestion is where traffic occurs when the usage spikes. To tackle this, Comcast has set up dynamic usage sensing, so that when usage spikes, adequate capacity is added to the network (as and when required). This procedure is equipped towards bringing extra limit to the system before it shows recognizable impacts.

Unpredictable type of congestion occurs when a sudden event takes place and all users are trying to use the same resources. This can occur during a disaster etc. Some part of the physical network might get damaged due to a calamity.

For an understanding of how exactly the system works, it is good to have a little knowledge on how a Comcast’s network is designed. It is also called the hybrid fiber co-ax network. It uses coaxial links to connect between every user’s cable modem to an optical node. The optical nodes are in-turn connected via fiber optic cables through distribution hubs to the Cable Modem termination System (CMTS). The CMTS is linked with higher level routers in the network, which, in turn, are connected to Comcast’s internet backbone infrastructure of high end servers and such. Each CMTS has many ports that handle data packets coming into and leaving it. Every link modem on the Comcast’s wide network is linked with the CMTS through its ports on the CMTS. These ports can be downstream or upstream ports. If the data transfer is from the server to the user nodes, then it is a downstream port. If data packets are transferred from the user node to the servers direction, they are called downstream ports. Both of these ports are susceptible to traffic congestion. The new congestion management system is used for ease congestion at both types of ports unlike the erstwhile system. Based upon the above-explained configuration of the system and the data pattern movements that were studied, it was found that the most likely place for congestion in a network are the CMTS ports. Accordingly, the framework measures the activity at the CMTS ports, and applies the predetermined policy at those ports. The new hardware deployed to implement the new framework are the Internet Protocol Detail Record ("IPDR") servers, Congestion Management servers, and PacketCable Multimedia ("PCMM") servers.

The following figure is representative of the Comcast’s network framework with respect to the Congestion Management System as explained above.

Each Comcast user’s link modem has a bootfile, which can be thought of as a configuration file. All network configuration updates will be sent through an updated config file to every connected cable modem via the network. Therefore all configuration settings required for enabling the congestion management is also sent through this file. Comcast created two new QoS variables as measuring pointers of internet traffic congestion at the cable modem terminal. They are Priority Best Effort (PBE) traffic and Best Effort (BE) traffic. Henceforth, all activity to and from the modems are assigned PBE or BE, with PBE being the default status. Traffic activity is assigned as BE only when both of the following two conditions are met:

- The bandwidth usage of an upstream or downstream port of a CMTS over a specific time frame must be nearing the point where the congestion might start affecting the other users’ internet experience. This is called the Near Congestion state. The quantitative parameter is explained in the following paras.
- A particular user must be contributing to high bandwidth consumption for a long period of time on that particular work. This is called the extended high consumption state.

Only when both the above conditions are met, is the priority of a user set to BE from the default PBE. It is therefore obvious that any congestion will affect the BE traffic before affecting the PBE traffic.

Quantitative parameter to determine when a CMTS port is in Near Congestion state:

For a CMTS port to enter the Near Congestion State, activity streaming to or from that CMTS port must surpass a predefined level – called the Port Utilization Threshold (PUT) for a particular timeframe defined as the Port Utilization Duration (PUD). The PUT on a CMTS port is measured as a percentage of the total average upstream or downstream data transmission for the specific port in the relevant time slot. The PUD on the CMTS port is measured in minutes.

After the above-mentioned evaluations the values of PUT have been set as 70% and the PUT as 15 minutes. This means that if any user utilizes greater than 70% of the total bandwidth of the total allotted bandwidth for that particular port for 15 minutes continuously, that user will be given a BE priority.

When the above data was used in actual field trials in the new congestion management framework, it was found that about 33.33% of the users had their priority status changed to the BE state on any given day. In Colorado Springs, CO, their biggest test market, on any given day, an average of 22 clients out of 6,016 users in the test had their status changed to BE sooner or later amid the day.

A user is changed back from BE to PBE movement when the user’s bandwidth utilization drops less than 50 percent of his allocated upstream or downstream transmission capacity in the time period of 15 minutes. These criteria are proposed to minimize and prevent user’s QoS wavering, i.e., a circumstance in which a specific client could cycle more than once between BE and PBE

Basically, there are four stages for figuring out if the activity connected with a specific link modem is assigned as PBE or BE:
- Figure out whether the CMTS port is in a Near Congestion State or not.
- In the event that the answer to above is yes, then, figure out if any users are in an Extended High Utilization State.
- In the event that yes, change those clients' movement to BE from PBE. If it is a No at either of the two above steps, no action is required.
- In the event that a user’s movement has been assigned BE, check user utilization periodically. If utilization has reduced beneath the prefixed thresholds, reassign the user’s priority as PBE.
- Impact of BE Quality of Service on Users' Broadband Experience:

At the point when a CMTS port is in a Near Congestion State and a link modem associated with that port is in an Extended High Consumption State, that link modem's status is assigned as BE. Contingent on the level of usage on the CMTS port, this assignment could possibly result in the user’s traffic movement being delayed or dropped in the worst case scenario. This is a result of the way that the CMTS handles traffic. CMTS ports have ‘scheduler’ that regulates all the data packets through a queue and handles them one by one. A specific number of data packets can be handled by the scheduler in any given slot of time. The PBE statusis assigned a higher priority to fill up the available bandwidth capacity while the BE traffic is processed on a space-available policy.
Normally, a user whose status is BE, the effects of traffic congestion may be indicated when a page loads sluggishly, a shared transfer takes fairly longer to finish, or a VoIP call sounds distorted.

Figure describes the hardware and the software overview of the Congestion Management System

The aforesaid methodology is implemented using three different application servers. The first level application server is an IPDR server, which monitors and collects the bandwidth usage at a particular cable modem node (via the CMTS). The IPDR software is denoted as noted as the Statistics Collection Server.

The second level application server is the Congestion Management server, which uses the Simple Network Management Protocol (SNMP) to calculate CMTS port current utilization and also to monitor whether a node has reached a Near Congestion State. At this point, the Congestion Management server investigates the corresponding IPDR data for the nodes that are being classified to be in an Extended High Consumption State. After zeroing in on such nodes, the Congestion Management server signals to the third level application server, called the PCMM application server (or the QoS-Quality of Service server), the nodes that have been noted to be in an Extended High Consumption State. Now, the priority status of these nodes shall be set to BE. The QoS servers instruct a given port to set the priority for high consumption nodes as a BE. They also manage the monitor change of status actions. We shall look into the QoS models used by the above mentioned QoS server in the following paragraphs.

#### **QoS Models**

The quality of service, from the perspective of a networking model, is the measure of users’ satisfaction of services that the communication model provides. It includes non erroneous data transmission with an optimal use of resources. In the wireless multihop networks, QoS deals with metrics such as bandwidth, jitter, delay and packet loss rate, energy consumption, stability and security. A QoS model encompasses all protocols that handle the communication between all levels of the architectural model. In the ensuing paragraphs, different mathematical models based on design objectives and its constraints are explained.

The following flowchart Figure, explains the process that is undertaken while developing a QoS model[4].

#### **Ad hoc networks**

Ad hoc networks are represented as a weighted directed complete graph G(V,A) where V represents the set of vertices associated with the nodes of the network and A the set of edges indicating a link between node I and j.

 pij = pji           ∀(i,j)∈A

Where,

**The QoS modeling representation of an ad-hoc network is as follows:**

pij=The energy per bit required to travel from node I to node j.

dij=The distance between I and j.

For simplicity purposes, the transmission energy from i to j is considered symmetric with j to i.

QoS models help to determine an optimal path according to certain objectives which are subject to a set of constraints. The constraints characterize the network.

Let P represent the routing path of a data packet from a source to a destination. x is the decision variable.

#### **Multicriterion based networks**

This type of QoS model takes into consideration multi criteria which are opposite to one another. Solving for the objective function for one constraint could produce counter intuitive results for another constraint. Therefore the objective function must be resolved taking into consideration all the constraints at the same time.

Multi criteria representation and mathematical model

min(f1(x),f2,...,fm(x))
Subject to
    hi(x) = 0       ,       i= 1,...,k
    ki(x) ≤ 0       ,       i= 1,...,l
    x ∈ D ⊂ IRn

Hi = Equality constraints
ki = inequality Constraints
D = Domain in which which the functions fi are defined.

The solution of multicriteria optimization is one which is the best compromise to fit all the constraints across all functions. It is to be noted that there are multiple solutions to such an optimization problem. Such a set of all optimal solutions is called Pareto Front.

#### **Minimum Energy Broadcasting network**

This QoS model deals with a minimization function that aims to minimize power consumption of a broadcast. Broadcasting is the process where you send a data packet from a source to all the active nodes in the network without expecting the acknowledgement message. The source node is therefore called a deaf node. The Minimum Energy Broadcast model aims to find the optimal tree starting from the source and spanning all the active nodes such that the tree’s energy cost in minimal.

The QoS model is as follows:

Subject to
            yi ≥ pijz;   & nbsp; (i,j) A
        Zij ≥ 1;           Sv, s∈S
(i,j)∈A
i ∈,j∈v\s
      Zij |p|-1 ;  ∈ u
(i,j)∈ ℘
  Zij {0,1}
           yi IR+     ∀i ∈ V

Zij = Variable characterizing the optimal tree
U = Set of all infeasible paths originated in s

#### **Minimum Energy Multicasting network**

This QoS model deals with a minimization function that aims to minimize power consumption of a multicast. Multicasting is the same as broadcasting, the only difference being that the acknowledgement message is expected to be sent.

The QoS model for this type is formulated as a mixed integer linear programming
model based on extracting a subgraph T∗ s , originating from the source s and
represents the bandwidth constrained multicast tree with minimal energy consumption.
From the undirected graph G(V,A), assume a binary variable z characterizing T∗
as:

i∈V
 Subject
 Σ
i∈s
j∈Ki(S)

M

S
TSi
FSi
Pimax
λ

The set of multicast nodes including the source node and all destination nodes.
Propagation loss exponent.
Set of time slots into which the bandwidth is partitioned.
Set of transmission schedule of node i, defined as the power assignment in each time slot.
A set of free time slot at node i defined as FSi = {k/Pik > 0, x ∈ S}
A maximum power value that a node i can use.
The minimum signal to interference plus noise ratio

#### **Multicriteria model based on four criteria**

This is a very common qoS model wherein four criteria namely energy consumed, delay, bandwidth and packet loss rate.

## **References**

[1] http://math.mit.edu/~rothvoss/18.304.3PM/Presentations/1-Melissa.pdf.

[2] Bellman Ford Algorithm with Example: https://www.youtube.com/watch?v=VxJaEquKr24 Published on Dec 13 2015.

[3] Bellman Ford algorithm

[4] Amine, Khalil. Mathematic Models for Quality of Service Purposes in Ad Hoc Networks. INTECH Open Access Publisher, 2011.

[5] Comcast's Protocol-Agnostic Congestion Management System.

[6] Dijkstra's algorithm