# DPDK

## Summary


Switches, routers, firewalls and other networking devices typically need to process large volumes of packets in real time. Traditionally, efficient packet processing was done using specialized and expensive hardware. Data Plane Development Kit (DPDK) enables us to do this on low-cost commodity hardware. By using commodity hardware, we can also move networking functions to the cloud and run them within virtualized environments. DPDK enables innovation around multi-core CPUs, edge computing, real-time security, NFV/SDN, low-latency applications and more.

DPDK was originally started at Intel. Later it was open sourced and subsequently brought under the guidance of the Linux Foundation. It uses BSD-3-Clause licensing. 

DPDK is not without its challenges. Not all NICs are supported. Support for Windows is limited. Being a low-level framework, it requires developers with relevant expertise. Debugging is harder.

## Discussion

### How did vendors achieve efficient packet processing before DPDK?

Before DPDK, specialized hardware did efficient packet processing. Such hardware might use custom ASICs, programmable FPGAs or Network Processing Units (NPUs). Sometimes low-level hardware-specific microcode or custom firmware could be present. Packet classification, flow control, TCP/IP processing, encryption/decryption, VLAN tagging, and checksum calculation example tasks that these specialized hardware did in an optimized manner.

However, such hardware were expensive to buy and maintain. Upgrades and security patches were time-consuming to apply and needed a full-time network administrator. One solution was move from specialized hardware to Commercial Off-the-Shelf (COTS) hardware. While this was more cost effective and easier to maintain, performance suffered. Packets moved from Network Interface Cards (NICs) to the operating system (OS), where they were processed via the OS kernel stack. 

Even with fast NICs, the kernel stack proved a bottleneck. System calls, interrupts, context switches, packet copying and per-packet processing brought down performance.  

DPDK solves the performance problem on COTS hardware. We get efficient packet processing without requiring expensive customized hardware. 


### How does DPDK improve packet processing?

DPDK **bypasses the kernel** and enables fast packet processing in userspace. It's essentially a set of network drivers and libraries. Environment Abstraction Layer (EAL) abstracts hardware-specific operations from applications. The figure shows how traditional processing with POSIX calls goes through the kernel space before packets reach the application. DPDK short-circuits this path and moves packets directly between NIC and userspace applications. 

Traditional processing is interrupt driven where the NIC interrupts the kernel when a packet arrives. DPDK uses **polling** instead and avoids the overhead associated with interrupts. This is performed by a Poll Mode Driver (PMD). 

Another important optimization is **zero copy**. In traditional networking, packets are copied from socket buffers in kernel space to user space. DPDK avoids this.  

For developers, DPDK's userspace approach is attractive since kernel need not be modified. Any network stack based on DPDK can be optimized for specific applications. 


### What's the packet processing model adopted by DPDK?

There are broadly two processing models:   

    + **Run-to-Completion**: A CPU core handles receive, processing and transmit of a packet. Multiple cores can be used with each core associated with a dedicated port. However, with Receive Side Scaling (RSS), traffic arriving at a single port can be distributed to multiple cores.
    + **Pipeline**: Each core is dedicated to a specific workload. For example, one core might handle receive/transmit of packets while other cores handle application processing. Packets are passed between cores via memory rings.For single-core multi-CPU deployments, one CPU is assigned to the OS and the other to the DPDK-based application. A less performant variant is when packets are accessed via the QPI interface. For multi-core deployments, we can assign more than one core to each port with or without hyperthreading. 

Deciding which model to use is not trivial. Some things to consider are cycles needed to process each packet, extent of data exchange across software modules, specific optimization at some cores, code maintainability, etc. Intel VTune Profiler can be used to analyze the efficiency of the pipeline model. 


### What other techniques does DPDK use to improve performance?

Apart from kernel bypass, polling and zero copy, there are a few more techniques used by DPDK:  

    + **Processor Affinity**: Ties specific processing to specific cores.
    + **Huge Pages**: Reduces TLB cache misses.
    + **Lockless Sync**: Queues are managed with the ring library. Enqueue and dequeue operations are lockless.
    + **I/O Batch**: Process a batch of packets rather than one at a time. This amortizes overheads in accessing the NIC.
    + **NUMA Aware**: Utilizes NUMA memory for better performance.
    + **Cache Alignment**: Align structures to 64-byte cache lines.These features are used by DPDK components such as EAL, memory manager, buffer manager, queue manager, flow classification, etc. 


### What are some metrics used to evaluate DPDK performance?

**Throughput** is the most common metric. Often this is quoted in Mbps, Gbps or Mega Packets Per Second (MPPS). When MPPS is used, packet size must be mentioned. At a low packet size of 64 bytes, throughput will be limited by MPPS rather than Mbps. The figure above shows throughput at L2. 

In 2015, per core L3 performance on Linux was 1.1 MPPS. On Intel DPDK, this was 28.5 MPPS. In 2010, DPDK could achieve L3 performance of 55 MPPS with 64-byte packets on an Intel Xeon-based system. This was improved to 255 MPPS (2014) and 347 MPPS (2016). 

**Latency** is important for low-latency applications. The figure above shows that DPDK reduces latency by a factor of ten on 64-byte packets. Likewise, for real-time applications, **jitter** is important. One study showed that DPDK reduces jitter from 10μs to 2μs. 


### Does DPDK need a TCP/IP stack to work?

DPDK doesn't include a TCP/IP stack. If an application requires a userspace networking stack, it can use F-Stack, mTCP, TLDK, Seastar and Accelerated Network Stack (ANS). These typically provide both blocking and non-blocking socket APIs. Some of these are based on FreeBSD implementation. 

By omitting a networking stack, DPDK doesn't have the inefficiency of a generic implementation. Applications can include networking modules optimized for their use cases. There might also be some use cases where no higher layer (above L2) processing is needed.


### Who in industry is using DPDK?

Load balancing, flow classification, routing, access control (firewall), and traffic policing are typical uses of DPDK. There's a misconception that DPDK is only for the telecom industry. However, DPDK has been used in cloud environments and enterprises alike. Traffic generators (TRex) and storage applications (SPDK) use DPDK. The figure above lists open-source projects powered by DPDK. 

Network Function Virtualization (NFV) and Software Defined Networking (SDN) are two technologies that leverage DPDK. Open vSwitch (OVS) when ported to DPDK has shown 7x performance improvement. 

In IoT applications, packets are tiny. DPDK reduces latency and allows more such packets to be processed per second. 

In 5G, the User Plane Function (UPF) processes user data packets. Delay, jitter and bandwidth are key performance metrics that need to be met. Some researchers have proposed DPDK for 5G UPF implementation.  For deploying UPF at edge networks, DPDK APIs can be used to interface the UPF application (UPF-C) and SmartNICs (UPF-U). 


### What are the challenges with DPDK?

DPDK requires low-level expertise. Developers should learn DPDK's programming model. They should know how to manage memories, pass packets without copying, and work with multi-core architectures. 

For example, PID namespace can cause problems with managing *fbarray*. Processes using *mmap* without specifying addresses can cause problems. Called *thread or core affinity*, threads must be assigned correctly to CPU cores for consistent performance. DPDK libraries offer developers many implementation choices. Getting these choices wrong can impact performance. 

Since kernel is bypassed, we lose all the protection, utilities (`ifconfig`, `tcpdump`) and protocols (ARP, IPSec) that the Linux kernel provides. Debugging and identifying root causes for networking problems are challenges. However, DPDK's tracing library and *LLTng* may help. A poor implementation can cause other processes/programs to fail. 

Finally, since polling is used instead of interrupts, DPDK causes 100% CPU usage even when only a few packets are being processed. 


### What are some alternatives to DPDK?

Faster packet processing via kernel bypass is possible using Snabbswitch, Netmap or StackMap. Like DPDK, these process packets in the userspace. Packets completely bypass the kernel stack. Snabbswitch is written in Lua while DPDK is in C.  PacketShader does kernel bypass for GPU-based hardware.  

An alternative approach is to modify the Linux kernel. Examples include eXpress Data Path (XDP) and network stacks based on Remote Direct Memory Access (RDMA). Other efficient tools include packet\_mmap (but doesn't bypass the kernel) and PF\_RING (with ZC drivers). 

Cloud providers may not wish to dedicate an entire NIC for a single offloaded application. Solarflare uses OpenOnload that uses the *EF\_VI* proprietary library. It creates a "hidden queue" at the NIC that userspace processes can access. This approach is sometimes called **bifurcated driver** or **queue splitting**. A similar approach exists in the virtualization world to avoid the overhead of passing packets from host to VM. Fake virtual interfaces allow packets to reach the VM directly. In general, modern NICs can bifurcate traffic to use or skip kernel stack. SR-IOV and VFIO technologies enable this.  

## Milestones

Sep  
2010

Intel creates **DPDK** and releases it under a permissive license. It uses the *3-Clause BSD License*. This allows developers to modify the source code without requiring them to open source those modifications. Intel's Venky Venkatesan is often called the "Father of DPDK". 

Sep  
2012

**DPDK v1.2.3r0** is released. On DPDK Git archives, this is the earliest version available. Release notes are available since v1.8 (December 2014).  

Oct  
2012

ETSI's white paper on NFV is published. This paper mentions DPDK as an enabling technology for NFV and cloud computing. This is attributed to DPDK's poll-mode Ethernet drivers that avoid interrupt-driven processing. 

2013

Website *DPDK.org* is launched and an **open source community** is formed. 

Sep  
2014

The first **DPDK Summit** is organized in San Francisco. Similar summits are organized regularly over the following years. Meanwhile in 2014, DPDK comes with **multi-vendor** CPU and NIC support. This support grows rapidly through 2015. 

Apr  
2015

Since DPDK doesn't offer a built-in TCP/IP stack, other open source projects attempt to provide this on top of DPDK. Examples of this include mTCP (v2 in Apr 2015), ANS (v16.08 in Aug 2016), and F-Stack (v1.11 in Nov 2017). 

Oct  
2015

DPDK releases new project **logos**, licensed under CC BY-IND 4.0. 

Apr  
2017

DPDK becomes a project under the guidance of **the Linux Foundation**, which provides a network platform for the growth of DPDK. By now, many open source projects have already adopted DPDK including MoonGen, mTCP, Ostinato, Lagopus, Fast Data (FD.io), Open vSwitch, OPNFV, and OpenStack. 

May  
2019

DPDK v19.05 is released with **support for Windows OS**. Only a "Hello World" sample application is supported. However, even four years later, the v23.07.0 documentation notes that DPDK for Windows is "a work in progress" and some source files may not compile. 

Nov  
2020

**DPDK v20.11** is released. It's claimed to be the biggest and most robust release. It's an LTS release that will be supported for two years. The two biggest contributors for this release are Intel and Nvidia, followed by Huawei and Broadcom. 

Dec  
2022

Intel publishes a technology guide explaining its *Data Streaming Accelerator (DSA)*. While DPDK aims to avoid data copying, sometimes this is unavoidable. Such copying operations can be offloaded to Direct Memory Access (DMA) accelerators such as DSA. DPDK library that enables this is `dmadev`, available since DPDK v21.11 (November 2021).  

Mar  
2023

Belkhiri et al. propose instrumenting DPDK libraries and collecting trace information. These traces can be used to debug performance issues and identify root causes. They claim that their approach is better than what existing tools VTune Amplifier (closed source), FlowWatcher-DPDK and DPDKStat are capable of. 

Sep  
2023

DPDK website shows that the current gold members include AMD, ARM, Ericsson, Intel, Marvell, Microsoft, Nvidia, Red Hat and ZTE. Silver members include 6Wind, Broadcom, Huawei, NXP, and more.