Aya Rust Tutorial part 5: Using Maps
Tags: ebpf, networking, linux, rust
Published: 2024-06-19 00:38:30

<p>&copy; steve latif </p>

<p>Welcome to part 5. So far we have created a basic hello world program in Part <a href="https://dev.to/stevelatif/aya-rust-tutorial-part-four-xdp-hello-world-4c85">Four</a>.
In this chapter we will start looking at how to pass data
between the kernel and user space using Maps.</p>
<h1>Overview: What is a Map and why do we need them ?</h1>
<p>The eBPF verifier enforces a 512 byte limit per stack frame,
if you need to handle more data you can store data using
<a href="https://docs.kernel.org/bpf/maps.html">maps</a></p>
<ul>
<li>Maps are created on the kernel code</li>
<li>Maps are accessible from  user space using a system call and a key</li>
<li>Maps allow persistence across program invocations</li>
<li>Maps offer different storage types</li>
</ul>
<p>Maps in eBPF are a basic building block and we will be using them extensively
in the next sections. Our first example will build on the previous
example and will be a packet counter using an array.</p>
<p>Let&#39;s take a minute to look at the kernel code and
see the definitions there.
Maps are defined in <a href="https://elixir.bootlin.com/linux/latest/source/tools/lib/bpf/libbpf.c#L511">libbpf.h</a> </p>
<pre><code class="language-c">struct bpf<em>map</em>def {
    unsigned int type;
    unsigned int key_size;
    unsigned int value_size;
    unsigned int max_entries;
    unsigned int map_flags;
};
</code></pre>
<p>The different map types:</p>
<ul>
<li>BPF<em>MAP_TYPE</em>ARRAY                </li>
<li>BPF<em>MAP_TYPE_PERCPU</em>ARRAY         </li>
<li>BPF<em>MAP_TYPE_PROG</em>ARRAY           </li>
<li>BPF<em>MAP_TYPE_PERF_EVENT</em>ARRAY     </li>
<li>BPF<em>MAP_TYPE_CGROUP</em>ARRAY         </li>
<li>BPF<em>MAP_TYPE_CGROUP</em>STORAGE         </li>
<li>BPF<em>MAP_TYPE_CGROUP</em>STORAGE       </li>
<li>BPF<em>MAP_TYPE_PERCPU_CGROUP</em>STORAGE</li>
<li>BPF<em>MAP_TYPE</em>HASH                 </li>
<li>BPF<em>MAP_TYPE_PERCPU</em>HASH          </li>
<li>BPF<em>MAP_TYPE_LRU</em>HASH             </li>
<li>BPF<em>MAP_TYPE_LRU_PERCPU</em>HASH      </li>
<li>BPF<em>MAP_TYPE_LPM</em>TRIE             </li>
<li>BPF<em>MAP_TYPE_STACK</em>TRACE          </li>
<li>BPF<em>MAP_TYPE_ARRAY_OF</em>MAPS        </li>
<li>BPF<em>MAP_TYPE_HASH_OF</em>MAPS         </li>
<li>BPF<em>MAP_TYPE_INODE</em>STORAGE        </li>
<li>BPF<em>MAP_TYPE_TASK</em>STORAGE         </li>
<li>BPF<em>MAP_TYPE</em>DEVMAP               </li>
<li>BPF<em>MAP_TYPE_DEVMAP</em>HASH          </li>
<li>BPF<em>MAP_TYPE_SK</em>STORAGE           </li>
<li>BPF<em>MAP_TYPE</em>CPUMAP               </li>
<li>BPF<em>MAP_TYPE</em>XSKMAP               </li>
<li>BPF<em>MAP_TYPE</em>SOCKMAP              </li>
<li>BPF<em>MAP_TYPE</em>SOCKHASH             </li>
<li>BPF<em>MAP_TYPE_REUSEPORT</em>SOCKARRAY  </li>
<li>BPF<em>MAP_TYPE</em>QUEUE                </li>
<li>BPF<em>MAP_TYPE</em>STACK                </li>
<li>BPF<em>MAP_TYPE_STRUCT</em>OPS           </li>
<li>BPF<em>MAP_TYPE</em>RINGBUF              </li>
<li>BPF<em>MAP_TYPE_BLOOM</em>FILTER         </li>
<li>BPF<em>MAP_TYPE_USER</em>RINGBUF         </li>
<li>BPF<em>MAP_TYPE</em>ARENA</li>
</ul>
<p>are defined in <a href="https://elixir.bootlin.com/linux/latest/source/include/linux/bpf_types.h#L87">map</a></p>
<p>The corresponding aya definitions are documented <a href="https://docs.aya-rs.dev/aya_ebpf/maps/">here</a> 
for the kernel side. 
The corresponding user space entries are <a href="https://docs.aya-rs.dev/aya/maps/">here</a></p>
<p>Our initial example will be a simple per CPU packet counter that will print out 
from user space the number of packets arriving at an interface </p>
<p>In the C API helper functions are used on both the kernel and user space side. 
The helpers have the same names on both sides.
The kernel side helper functions have access to a pointer to the map and key.</p>
<p>On the user space side the helper functions use a syscall and file descriptor. </p>
<p>In Aya on the kernel <a href="https://docs.aya-rs.dev/aya_ebpf/maps/">side</a></p>
<ul>
<li>array::Array</li>
<li>bloom_filter::BloomFilter</li>
<li>hash_map::HashMap</li>
<li>hash_map::LruHashMap</li>
<li>hash_map::LruPerCpuHashMap</li>
<li>hash_map::PerCpuHashMap</li>
<li>lpm_trie::LpmTrie</li>
<li>per<em>cpu</em>array::PerCpuArray</li>
<li>perf::PerfEventArray</li>
<li>perf::PerfEventByteArray</li>
<li>program_array::ProgramArray</li>
<li>queue::Queue</li>
<li>ring_buf::RingBuf</li>
<li>sock_hash::SockHash</li>
<li>sock_map::SockMap</li>
<li>stack::Stack</li>
<li>stack_trace::StackTrace</li>
<li>xdp::CpuMap</li>
<li>xdp::DevMap</li>
<li>xdp::DevMapHash</li>
<li>xdp::XskMap</li>
</ul>
<p>While on the user space <a href="https://docs.aya-rs.dev/aya_ebpf/maps/">side</a>
We have the same function list.</p>
<p>As before generate the code from the template using the command</p>
<pre><code class="language-shell"> cargo generate https://github.com/aya-rs/aya-template
</code></pre>
<p>I called the project <code>xdp-map-counter</code></p>
<p>Lets set up the packet counter, on the eBPF side:</p>
<pre><code class="language-rust">#![no_std]
#![no_main]
 
use aya<em>ebpf::{bindings::xdp</em>action,
           macros::{xdp, map},
           programs::XdpContext,
           maps::PerCpuArray,
};
 
const CPU_CORES: u32 = 16;
 
#[map(name=&quot;PKT<em>CNT</em>ARRAY&quot;)]
static mut PACKET<em>COUNTER: PERCPU&lt;u32&gt; = PerCpuArray::with_max_entries(CPU</em>CORES , 0);
 
#[xdp]
pub fn xdp<em>map_counter(</em>ctx: XdpContext) -&gt; u32 {
    match try<em>xdp_map</em>counter() {
        Ok(ret) =&gt; ret,
        Err(<em>) =&gt; xdp_action::XDP</em>ABORTED,
    }
}
 
#[inline(always)] 
fn try<em>xdp_map</em>counter() -&gt; Result&lt;u32, ()&gt; {
    unsafe {
    let counter = PACKET_COUNTER
            .get<em>ptr</em>mut(0)
             .ok_or(())? ;
    *counter += 1;
    }
    Ok(xdp<em>action::XDP</em>PASS)
}
 
#[panic_handler]
fn panic(_info: &amp;core::panic::PanicInfo) -&gt; ! {
    unsafe { core::hint::unreachable_unchecked() }
}
</code></pre>
<p>The map will be created on the eBPF side. We will use a PerCpuArray in
this first example. Arrays are simple to work with. With the PerCpuArray
each CPU sees its own instance of the map, this means that it avoids
lock contention and is therefore the most performant way to get 
readings from eBPF to user land. The downside is that updating the 
values from user space can&#39;t be done safely.</p>
<p>The size of array must be known at build time so we set a constant with an 
upper bound on the number of cores on the system
<code>const CPU_CORES: u32 = 16</code></p>
<p>Then we can define a PerCpuArray with <code>CPU_CORES</code> entries initialized to 0.</p>
<pre><code class="language-rust">#[map(name=&quot;PKT<em>CNT</em>ARRAY&quot;)]
static mut PACKET<em>COUNTER: PerCpuArray&lt;u32&gt; = PerCpuArray::with_max_entries(CPU</em>CORES, 0);
</code></pre>
<p>The main work is in the <code>try<em>xdp</em>counter</code> function.
We get a pointer to the map and then increment the value</p>
<pre><code class="language-rust">    unsafe {
    let counter = PACKET_COUNTER
            .get<em>ptr</em>mut(0)
             .ok_or(())? ;
    *counter += 1;
    }
</code></pre>
<p>Note that the call to  <code>ok_or()</code> is required, failing to have the check 
here will fail the eBPF verifier. </p>
<p>The packet is then passed on to the networking stack.</p>
<pre><code class="language-rust">    Ok(xdp<em>action::XDP</em>PASS)
</code></pre>
<p>The code on the user space side:</p>
<pre><code class="language-rust">use anyhow::Context;
use aya::programs::{Xdp, XdpFlags};
use aya::{include<em>bytes</em>aligned, Bpf};
use aya::maps::PerCpuValues;
use aya::maps::PerCpuArray;
use aya_log::BpfLogger;
use clap::Parser;
use log::{warn, debug};
use aya::util::nr_cpus;
//use tokio::signal;
 
#[derive(Debug, Parser)]
struct Opt {
    #[clap(short, long, default_value = &quot;eth0&quot;)]
    iface: String,
}
 
#[tokio::main]
async fn main() -&gt; Result&lt;(), anyhow::Error&gt; {
    let opt = Opt::parse();
 
    env_logger::init();
    // Bump the memlock rlimit. This is needed for older kernels that don&#39;t use the
    // new memcg based accounting, see https://lwn.net/Articles/837122/
    let rlim = libc::rlimit {
        rlim<em>cur: libc::RLIM</em>INFINITY,
        rlim<em>max: libc::RLIM</em>INFINITY,
    };
    let ret = unsafe { libc::setrlimit(libc::RLIMIT_MEMLOCK, &amp;rlim) };
    if ret != 0 {
        debug!(&quot;remove limit on locked memory failed, ret is: {}&quot;, ret);
    }
    // This will include your eBPF object file as raw bytes at compile-time and load it at
    // runtime. This approach is recommended for most real-world use cases. If you would
    // like to specify the eBPF program at runtime rather than at compile-time, you can
    // reach for `Bpf::load_file` instead.
    #[cfg(debug_assertions)]
    let mut bpf = Bpf::load(include<em>bytes</em>aligned!(
        &quot;../../target/bpfel-unknown-none/debug/xdp-map-counter&quot;
    ))?;
    #[cfg(not(debug_assertions))]
    let mut bpf = Bpf::load(include<em>bytes</em>aligned!(
        &quot;../../target/bpfel-unknown-none/release/xdp-map-counter&quot;
    ))?;
    if let Err(e) = BpfLogger::init(&amp;mut bpf) {
        // This can happen if you remove all log statements from your eBPF program.
        warn!(&quot;failed to initialize eBPF logger: {}&quot;, e);
    }
    let program: &amp;mut Xdp = bpf.program<em>mut(&quot;xdp_map_counter&quot;).unwrap().try</em>into()?;
    program.load()?;
    program.attach(&amp;opt.iface, XdpFlags::default())
        .context(&quot;failed to attach the XDP program with default flags - try changing XdpFlags::default() to XdpFlags::SKB_MODE&quot;)?;
 
    let array = PerCpuArray::try<em>from(bpf.map_mut(&quot;PKT_CNT</em>ARRAY&quot;).unwrap())?;
 
    loop {
    let cc: PerCpuValues&lt;u32&gt; = array.get(&amp;0, 0)?;
    let mut total : u32 =  0;
    //println!(&quot;{:?} packets&quot;,  cc);
    for ii in 1..nr_cpus().expect(&quot;failed to get number of cpus&quot;) {
        print!(&quot;{} &quot;, cc[ii]);
        total += cc[ii];
    }
    println!(&quot;total: {} &quot;, total);
    std::thread::sleep(std::time::Duration::from_secs(1));
    }
    //signal::ctrl_c().await?;    
}
</code></pre>
<p>The array reference is created on the user space side here with name 
&#39;PKT<em>CNT</em>ARRAY&#39;	</p>
<pre><code class="language-rust">    let array = PerCpuArray::try<em>from(bpf.map_mut(&quot;PKT_CNT</em>ARRAY&quot;).unwrap())?;
</code></pre>
<p>and must match the name declared in the eBPF code </p>
<pre><code class="language-rust">#[map(name=&quot;PKT<em>CNT</em>ARRAY&quot;)]
static mut COUNTER: PerCpuArray&lt;u32&gt; = PerCpuArray::with<em>max_entries(CPU</em>CORES , 0);
</code></pre>
<p>Most of the rest of the code is boilerplate except for the loop at the end which checks every 
second for the results from the kernel eBPF code and then prints out the stats.</p>
<pre><code class="language-rust"> loop {
    let cc: PerCpuValues&lt;u32&gt; = array.get(&amp;0, 0)?;
    let mut total : u32 =  0;
    for ii in 1..nr_cpus().expect(&quot;failed to get number of cpus&quot;) {
        print!(&quot;{} &quot;, cc[ii]);
        total += cc[ii];
    }
    println!(&quot;total: {} &quot;, total);
    std::thread::sleep(std::time::Duration::from_secs(1));
    }
   
</code></pre>
<h2>Testing</h2>
<p>As we did before we can run it over the loopback interface.
Build as before</p>
<pre><code class="language-shell">cargo xtask build-ebpf
cargo build
</code></pre>
<p>Then run</p>
<pre><code class="language-shell">cargo xtask run -- -i lo
</code></pre>
<p>In another terminal ping the loopback interface</p>
<pre><code class="language-shell">
ping 127.0.0.1
</code></pre>
<p>In the terminal where you are running the <code>cargo run</code> command you should see </p>
<pre><code class="language-shell">0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 total: 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 total: 0 
0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 total: 2 
0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 total: 4 
0 0 6 0 0 0 0 0 0 0 0 0 0 0 0 total: 6 
0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 total: 8 
0 0 10 0 0 0 0 0 0 0 0 0 0 0 0 total: 10 
</code></pre>
<p>We can see packets arriving and being processed on one core. 
To run  a more stressful test, replace the ping with the 
following ssh command:</p>
<pre><code class="language-shell"> ssh 127.0.0.1 cat /dev/zero
</code></pre>
<p>You should see something like:</p>
<pre><code class="language-shell"> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 total: 0 
0 7 6 2 0 0 0 0 0 0 0 5 0 0 0 total: 20 
0 7 6 2 0 0 0 0 0 0 0 6 0 0 0 total: 21 
0 7 6 2 0 0 0 0 0 0 0 6 0 0 0 total: 21 
0 7 6 2 0 0 0 0 0 0 0 6 0 0 0 total: 21 
0 7 6 2 0 0 0 0 0 0 0 6 0 0 0 total: 21 
0 48 12 2 0 16 0 0 0 0 0 8 0 702 0 total: 788 
0 48 133 12 0 527 0 0 0 5 0 8 94 1978 558 total: 3363 
0 48 133 243 0 527 0 17 0 5 0 8 94 1978 3179 total: 6232 
0 48 133 243 0 645 0 144 64 23 0 8 94 1978 5800 total: 9180 
0 48 133 243 0 733 0 201 64 203 2 8 94 1978 8406 total: 12113 
0 48 133 243 0 903 0 333 64 228 2 8 94 1978 11027 total: 15061 
0 48 133 243 0 1447 0 548 64 237 2 8 94 1978 13136 total: 17938 
0 368 133 243 0 3908 0 548 64 237 2 8 94 1978 13136 total: 20719 
0 595 133 416 0 6529 0 548 64 237 2 8 94 1978 13136 total: 23740 
0 683 133 674 0 9294 0 548 64 237 2 8 94 1978 13136 total: 26851 
0 854 133 927 0 11544 0 548 64 296 2 440 94 1978 13136 total: 30016 
...
</code></pre>
<p>Run it for a few iterations before Ctrl-C ing it.</p>
<p>As before lets take a minute to look at the eBPF byte code which corresponds to 
the code in the XDP section of <code>xdp-map-counter/xdp-map-counter-ebpf/src/main.rs</code></p>
<pre><code class="language-rust">fn try<em>xdp_map</em>counter() -&gt; Result&lt;u32, ()&gt; {
    unsafe {
    let counter = COUNTER
            .get<em>ptr</em>mut(0)
             .ok_or(())? ;
    *counter += 1;
    }
    Ok(xdp<em>action::XDP</em>PASS)
}
</code></pre>
<p>Using <code>llvm-objdump</code> as before</p>
<pre><code class="language-shell">$ llvm-obj dump --section=xdp  -S target/bpfel-unknown-none/debug/xdp-map-counter
</code></pre>
<pre><code class="language-asm">target/bpfel-unknown-none/debug/xdp-map-counter:	file format elf64-bpf
 
Disassembly of section xdp:
 
0000000000000000 &lt;xdp<em>map</em>counter&gt;:
       0:	b7 06 00 00 00 00 00 00	r6 = 0
       1:	63 6a fc ff 00 00 00 00	*(u32 *)(r10 - 4) = r6
       2:	bf a2 00 00 00 00 00 00	r2 = r10
       3:	07 02 00 00 fc ff ff ff	r2 += -4
       4:	18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00	r1 = 0 ll
       6:	85 00 00 00 01 00 00 00	call 1
       7:	15 00 04 00 00 00 00 00	if r0 == 0 goto +4 &lt;LBB0_2&gt;
       8:	61 01 00 00 00 00 00 00	r1 = *(u32 *)(r0 + 0)
       9:	07 01 00 00 01 00 00 00	r1 += 1
      10:	63 10 00 00 00 00 00 00	*(u32 *)(r0 + 0) = r1
      11:	b7 06 00 00 02 00 00 00	r6 = 2
 
0000000000000060 &lt;LBB0_2&gt;:
      12:	bf 60 00 00 00 00 00 00	r0 = r6
      13:	95 00 00 00 00 00 00 00	exit
</code></pre>
<p>We see that byte code maps closely to the rust code.
After setting up parameters in the first 4 lines, in line 6 we
have a <code>call 1</code> that is a system call to the bpf helper 
<code>map<em>lookup</em>elem</code> 
That value gets assigned to r1 where it is incremented on
line 9
and is then assigned to a location in  memory.</p>