  It is becoming increasingly popular for distributed systems to exploit
network offload to alleviate load on the CPU. Remote Direct Memory Access
(RDMA) NICs (RNICs) are one such device, allowing applications to offload
remote memory accesses. However, RDMA still requires CPU intervention for
complex offloads, beyond simple remote memory access. As such, the offload
potential for RNICs is limited and RDMA-based systems usually have to work
around such limitations.
  We present RedN, a principled, practical approach to implementing complex
RNIC offloads, without requiring any hardware modifications. Using
self-modifying RDMA chains, we lift the existing RDMA verbs interface to a
Turing complete set of programming abstractions. We explore what is possible in
terms of offload complexity and performance with just a commodity RNIC. Through
a key-value store use case study, we show how to integrate complex RNIC
offloads into existing applications. RedN can outperform one and two-sided RDMA
implementations by up to 3x and 7.8x for key-value get operations and
performance isolation, respectively, and provide applications with failure
resiliency to OS and process crashes.
