Introduction

Welcome to Building eBPF Programs with Aya: An introductory book about using the Rust Programming Language and Aya library to build extended Berkley Packet Filter (eBPF) programs.

Who Aya Is For

Rust is proving to be a popular systems programming language because of its safety features and excellent C interoperability. The safety features are less important in the context of eBPF as programs often need to read kernel memory, which is considered unsafe. However, what Rust combined with Aya does offer is a fast and efficient development experience:

Cargo for project scaffolding, build, test and debugging
Generation of Rust bindings to Kernel Headers with Compile-Once, Run-Everywhere (CO-RE) support
Easy code sharing between user-space and eBPF programs
Fast compile times
No runtime dependency on LLVM or BCC

Scope

The goals of this book are:

Get developers up to speed with eBPF Rust development. i.e. How to set up a development environment.
Share current best practices about using Rust for eBPF

Who This Book is For

This book caters towards people with either some eBPF or some Rust background. For those without any prior knowledge we suggest you read the "Assumptions and Prerequisites" section first. You can check out the "Other Resources" section to find resources on topics you might want to read up on.

Assumptions and Prerequisites

You are comfortable using the Rust Programming Language, and have written, run, and debugged Rust applications on a desktop environment. You should also be familiar with the idioms of the 2018 edition as this book targets Rust 2018.

You are familiar with the core concepts of eBPF

Other Resources

If you are unfamiliar with anything mentioned above or if you want more information about a specific topic mentioned in this book you might find some of these resources helpful.

Topic	Resource	Description
Rust	Rust Book	If you are not yet comfortable with Rust, we highly suggest reading this book.
eBPF	Cilium BPF and XDP Reference Guide	If you are not yet comfortable with eBPF, this guide is excellent.

How to Use This Book

This book generally assumes that you’re reading it front-to-back. Later chapters build on concepts in earlier chapters, and earlier chapters may not dig into details on a topic, revisiting the topic in a later chapter.

Source Code

The source files from which this book is generated can be found on GitHub.

eBPF Program Constraints

The eBPF Virtual Machine, where our eBPF programs will be run, is a constrained runtime environment:

There is only 512 bytes of stack (or 256 bytes if we are using tail calls).
There is no access to heap space and data must instead be written to maps.

Even applications written in C are restricted to a subset of language features:

no loops
no global variables
no variadic functions
no floating-point numbers
no passing structures as function arguments

While these limitations do not map 1:1 with Rust, we are still constrained:

We may not use the standard library. We use core instead.
core::fmt may not be used and neither can traits that rely on it, for example Display and Debug
As there is no heap, we cannot use alloc or collections.
We must not panic as the eBPF VM does not support stack unwinding, or the abort instruction.
There is no main function

Alongside this, a lot of the code that we write is unsafe, as we are reading directly from kernel memory.

Getting Started

In this section we'll walk you through the process of writing, building and running a simple eBPF program and userspace application.

Development Environment

Prerequisites

Before getting started you will need the Rust stable and nightly tool-chains installed on your system. This is easily achieved with [rustup]:

rustup install stable
rustup toolchain install nightly --component rust-src

Once you have the Rust tool-chains installed, you must also install the bpf-linker - for linking our eBPF program - and cargo-generate - for generating the project skeleton.

cargo +nightly install bpf-linker
cargo install --git https://github.com/cargo-generate/cargo-generate

Starting A New Project

To start a new project, you can use cargo-generate:

cargo generate https://github.com/dave-tucker/aya-template

This will prompt you for a project name. We'll be using myapp in this example

Hello XDP!

Example Project

While there are myriad trace points to attach to and program types to write we should start somewhere simple.

XDP (eXpress Data Path) programs permit our eBPF program to make decisions about packets that have been received on the interface to which our program is attached. To keep things simple, we'll build a very simplistic firewall to permit or deny traffic.

eBPF Component

Permit All

We must first write the eBPF component of our program. The logic for this program is located in myapp-ebpf/src/main.rs and currently looks like this:

#![no_std]
#![no_main]

#[panic_handler]
fn panic(_info: &core::panic::PanicInfo) -> ! {
    unreachable!()
}

#![no_std] is required since we cannot use the standard library.
#![no_main] is required as we have no main function.
The #[panic_handler] is required to keep the compiler happy, although it is never used since we cannot panic.

Let's expand this by adding an XDP program that permits all traffic.

First we'll add some imports:

use aya_bpf::bindings::xdp_action;
use aya_bpf::cty::c_long;
use aya_bpf::macros::xdp;
use aya_bpf::programs::XdpContext;

Then our application logic:

#[xdp]
pub fn xdp_firewall(ctx: XdpContext) -> u32 {
    match unsafe { try_xdp_firewall(ctx) } {
        Ok(ret) => ret,
        Err(_) => xdp_action::XDP_ABORTED,
    }
}

unsafe fn try_xdp_firewall(_ctx: XdpContext) -> Result<u32, c_long> {
    Ok(xdp_action::XDP_PASS)
}

#[xdp] indicates that this function is an XDP program
The try_xdp_firewall function returns a Result that permits all traffic
The xdp_firewall program calls try_xdp_firewall and handles any errors by returning XDP_ABORTED, which will drop the packet and raise a tracepoint exception.

Now we can compile this using cargo xtask build-ebpf

Verifying The Program

Let's take a look at the compiled eBPF program:

$ llvm-objdump -S target/bpfel-unknown-none/debug/myapp

target/bpfel-unknown-none/debug/myapp:  file format elf64-bpf


Disassembly of section xdp:

0000000000000000 <xdp_firewall>:
       0:       b7 00 00 00 02 00 00 00 r0 = 2
       1:       95 00 00 00 00 00 00 00 exit

We can see an xdp_firewall section here. r0 = 2 sets register 0 to 2, which is the value of the XDP_PASS action. exit ends the program.

Simple!

Completed Program

#![no_std]
#![no_main]

use aya_bpf::bindings::xdp_action;
use aya_bpf::cty::c_long;
use aya_bpf::macros::xdp;
use aya_bpf::programs::XdpContext;

#[panic_handler]
fn panic(_info: &core::panic::PanicInfo) -> ! {
    unreachable!()
}

#[xdp]
pub fn xdp_firewall(ctx: XdpContext) -> u32 {
    match unsafe { try_xdp_firewall(ctx) } {
        Ok(ret) => ret,
        Err(_) => xdp_action::XDP_ABORTED,
    }
}

unsafe fn try_xdp_firewall(_ctx: XdpContext) -> Result<u32, c_long> {
    Ok(xdp_action::XDP_PASS)
}

User-space Component

Now our eBPF program is complete and compiled, we need a user-space program to load it and attach it to a trace point. Fortunately, we have a program ready in myapp/src/main.rs which is going to do that for us.

Starting Out

The generated application has the following content:

fn main() {
    if let Err(e) = try_main() {
        eprintln!("error: {:#}", e);
    }
}

fn try_main() -> Result<(), anyhow::Error> {
    Ok(())
}

Let's adapt it to load our program.

We will add a dependency on ctrlc = "3.2" to myapp/Cargo.toml, then add the following imports at the top of the myapp/src/main.rs:

use aya::Bpf;
use aya::programs::{Xdp, XdpFlags};
use std::{
    convert::TryInto,
    env,
    thread,
    time::Duration,
    sync::Arc,
    sync::atomic::{AtomicBool, Ordering},
};

Then we'll adapt the try_main function to load our program:

fn try_main() -> Result<(), anyhow::Error> {
    let path = match env::args().nth(1) {
        Some(iface) => iface,
        None => panic!("not path provided"),
    };
    let iface = match env::args().nth(2) {
        Some(iface) => iface,
        None => "eth0".to_string(),
    };
    let mut bpf = Bpf::load_file(&path)?;
    let probe: &mut Xdp = bpf.program_mut("xdp")?.try_into()?;
    probe.load()?;
    probe.attach(&iface, XdpFlags::default())?;

    let running = Arc::new(AtomicBool::new(true));
    let r = running.clone();

    ctrlc::set_handler(move || {
        r.store(false, Ordering::SeqCst);
    }).expect("Error setting Ctrl-C handler");

    println!("Waiting for Ctrl-C...");
    while running.load(Ordering::SeqCst) {}
    println!("Exiting...");

    Ok(())
}

The program takes two positional arguments

The path to our eBPF application
The interface we wish to attach it to (defaults to eth0)

The line let mut bpf = Bpf::load_file(&path)?;:

Opens the file
Reads the ELF contents
Creates any maps
If your system supports BPF Type Format (BTF), it will read the current BTF description and performs any necessary relocations

Once our file is loaded, we can extract the XDP probe with let probe: &mut Xdp = bpf.program_mut("xdp")?.try_into()?; and then load it in to the kernel with probe.load().

Finally, we can attach it to an interface with probe.attach(&iface, XdpFlags::default())?;

Let's try it out!

$ cargo build
$ sudo ./target/debug/myapp ./target/bpfel-unknown-none/debug/myapp wlp2s0
Waiting for Ctrl-C...
Exiting...

That was uneventful. Did it work?

💡 HINT: Error Loading Program?

If you get an error loading the program, try changing XdpFlags::default() to XdpFlags::SKB_MODE

The Lifecycle of an eBPF Program

The program runs until CTRL+C is pressed and then exits. On exit, Aya takes care of detaching the program for us.

If you issue the sudo bpftool prog list command when myapp is running you can verify that it is loaded:

84: xdp  tag 3b185187f1855c4c  gpl
        loaded_at 2021-08-05T13:35:06+0100  uid 0
        xlated 16B  jited 18B  memlock 4096B
        pids myapp(69184)

Running the command again once myapp has exited will show that the program is no longer running.

Logging Packets

In the previous chapter, our XDP application ran for 10 seconds and permitted some traffic. There was however no output on the console, so you just have to trust that it was working correctly. Let's expand this program to log the traffic that is being permitted

Getting Data to User-Space

To get data from kernel-space to user-space we use an eBPF map. There are numerous types of maps to chose from, but in this example we'll be using a PerfEventArray.

While we could go all out and extract data all the way up to L7, we'll constrain our firewall to L3, and to make things easier, IPv4 only. The data structure that we'll need to send information to user-space will need to hold an IPv4 address and an action for Permit/Deny, we'll encode both as a u32.

Let's go ahead and add that to myapp-common/src/lib.rs

#[repr(C)]
pub struct PacketLog {
    pub ipv4_address: u32,
    pub action: u32,
}

#[cfg(feature = "user")]
unsafe impl aya::Pod for PacketLog {}

💡 HINT: Struct Alignment

Structs must be aligned to 8 byte boundaries. You can do this manually, or alternatively you may use #[repr(packed)]. If you do not do this, the eBPF verifier will get upset and emit an invalid indirect read from stack error.

We implement the aya::Pod trait for our struct since it is Plain Old Data as can be safely converted to a byte-slice and back.

eBPF: Map Creation

Let's create a map called EVENTS in myapp-ebpf/src/main.rs

use aya_bpf::macros::map;
use aya_bpf::maps::PerfMap;
use myapp_common::PacketLog;

#[map(name = "EVENTS")]
static mut EVENTS: PerfMap<PacketLog> = PerfMap::<PacketLog>::with_max_entries(1024, 0);

When the eBPF program is loaded by Aya, the map will be created for us.

Userspace: Map Creation

After our call to probe.attach() we'll add the following code.

use aya::maps::AsyncPerfEventArray;

let mut perf_array = AsyncPerfEventArray::try_from(bpf.map_mut("EVENTS")?)?;

Our perf_array is a mutable reference to the map that was created after the XDP program was loaded by Aya.

Writing Data

Now we've got our maps set up, let's add some data!

Generating Bindings To vmlinux.h

To get useful data to add to our maps, we first need some useful data structures to populate with data from the XdpContext. We want to log the Source IP Address of incoming traffic, so we'll need to:

Read the Ethernet Header to determine if this is an IPv4 Packet
Read the Source IP Address from the IPv4 Header

The two structs in the kernel for this are ethhdr from uapi/linux/if_ether.h and iphdr from uapi/linux/ip.h. If I were to use bindgen to generate Rust bindings for those headers, I'd be tied to the kernel version of the system that I'm developing on. This is where aya-gen comes in to play. It can easily generate bindings for using the BTF information in /sys/kernel/btf/vmlinux.

Once the bindings are generated and checked in to our repository they shouldn't need to be regenerated again unless we need to add a new struct.

Lets use xtask to automate this so we can easily reproduce this file in future.

We'll add the following content to xtask/src/codegen.rs

use aya_gen::btf_types;
use std::{
    fs::File,
    io::Write,
    path::{Path, PathBuf},
};

pub fn generate() -> Result<(), anyhow::Error> {
    let dir = PathBuf::from("myapp-ebpf/src");
    let names: Vec<&str> = vec!["ethhdr", "iphdr"];
    let bindings = btf_types::generate(Path::new("/sys/kernel/btf/vmlinux"), &names, false)?;
    // Write the bindings to the $OUT_DIR/bindings.rs file.
    let mut out = File::create(dir.join("bindings.rs"))?;
    write!(out, "{}", bindings).expect("unable to write bindings to file");
    Ok(())
}

This will generate a file called myapp-ebpf/src/bindings.rs. If you've chosen an application name other than myapp you'll need to adjust the path appropriately.

Add a new dependencies to xtask/Cargo.toml:

[dependencies]
aya-gen = { git = "http://github.com/alessandrod/aya", branch = "main" }

And finally, we must register the command in xtask/src/main.rs:

mod build_ebpf;
mod codegen;

use std::process::exit;

use structopt::StructOpt;
#[derive(StructOpt)]
pub struct Options {
    #[structopt(subcommand)]
    command: Command,
}

#[derive(StructOpt)]
enum Command {
    BuildEbpf(build_ebpf::Options),
    Codegen,
}

fn main() {
    let opts = Options::from_args();

    use Command::*;
    let ret = match opts.command {
        BuildEbpf(opts) => build_ebpf::build(opts),
        Codegen => codegen::generate(),
    };

    if let Err(e) = ret {
        eprintln!("{:#}", e);
        exit(1);
    }
}

Once we've generated our file using cargo xtask codegen from the root of the project.

These can then be accessed from within myapp-ebpf/src/main.rs:

mod bindings;
use bindings::{ethhdr, iphdr};

Getting Packet Data From The Context

The XdpContext contains two fields, data and data_end. data is a pointer to the start of the data in kernel memory and data_end, a pointer to the end of the data in kernel memory. In order to access this data and ensure that the eBPF verifier is happy, we'll introduce a helper function:

#[inline(always)]
unsafe fn ptr_at<T>(ctx: &XdpContext, offset: usize) -> Result<*const T, ()> {
    let start = ctx.data();
    let end = ctx.data_end();
    let len = mem::size_of::<T>();

    if start + offset + len > end {
        return Err(());
    }

    Ok((start + offset) as *const T)
}

This function will ensure that before we access any data, we check that it's contained between data and data_end. It is marked as unsafe because when calling the function, you must ensure that there is a valid T at that location or there will be undefined behaviour.

Writing Data To The Map

With our helper function in place, we can:

Read the Ethertype field to check if we have an IPv4 packet.
Read the IPv4 Source Address from the IP header

First let's add another dependency on memoffset = "0.6" to myapp-ebpf/Cargo.toml, and then we'll change our try_xdp_firewall function to look like this:

use memoffset::offset_of;

fn try_xdp_firewall(ctx: XdpContext) -> Result<u32, ()> {
    let h_proto = u16::from_be(unsafe { *ptr_at(&ctx, offset_of!(ethhdr, h_proto))? });
    if h_proto != ETH_P_IP {
        return Ok(xdp_action::XDP_PASS)
    }
    let source = u32::from_be(unsafe { *ptr_at(&ctx, ETH_HDR_LEN + offset_of!(iphdr, saddr))? });

    let log_entry = PacketLog{
        ipv4_address: source,
        action: xdp_action::XDP_PASS,
    };
    unsafe { EVENTS.output(&ctx, &log_entry, 0); }
    Ok(xdp_action::XDP_PASS)
}

💡 HINT: Reading Fields Using offset_of!

As there is limited stack space, it's more memory efficient to use the offset_of! macro to read a single field from a struct, rather than reading the whole struct and accessing the field by name.

Once we have our IPv4 source address, we can create a PacketLog struct and output this to our PerfEventArray

Reading Data

Going Async

In order to read from the AsyncPerfEventArray, we have to call AsyncPerfEventArray::open() for each online CPU, then we have to poll the file descriptor for events. While this is do-able using PerfEventArray and mio or epoll, the code is much less easy to follow. Instead, we'll use tokio to make our user-space application async.

Let's add some dependencies to myapp/src/Cargo.toml:

[dependencies]
aya = { git = "https://github.com/alessandrod/aya", branch="main", features=["async_tokio"] }
myapp-common = { path = "../myapp-common", features=["userspace"] }
anyhow = "1.0.42"
bytes = "1"
tokio = { version = "1.9.0", features = ["full"] }

And adjust our myapp/src/main.rs to look like this:

use aya::{
    maps::perf::AsyncPerfEventArray,
    programs::{Xdp, XdpFlags},
    util::online_cpus,
    Bpf,
};
use bytes::BytesMut;
use std::{
    convert::{TryFrom, TryInto},
    env, fs, net,
};
use tokio::{signal, task};

use myapp_common::PacketLog;

#[tokio::main]
async fn main() -> Result<(), anyhow::Error> {
    let path = match env::args().nth(1) {
        Some(iface) => iface,
        None => panic!("not path provided"),
    };
    let iface = match env::args().nth(2) {
        Some(iface) => iface,
        None => "eth0".to_string(),
    };

    let data = fs::read(path)?;
    let mut bpf = Bpf::load(&data, None)?;

    let probe: &mut Xdp = bpf.program_mut("xdp")?.try_into()?;
    probe.load()?;
    probe.attach(&iface, XdpFlags::default())?;

    let mut perf_array = AsyncPerfEventArray::try_from(bpf.map_mut("EVENTS")?)?;

    for cpu_id in online_cpus()? {
        let mut buf = perf_array.open(cpu_id, None)?;
        task::spawn(async move {
            let mut buffers = (0..10)
                .map(|_| BytesMut::with_capacity(1024))
                .collect::<Vec<_>>();

            loop {
                let events = buf.read_events(&mut buffers).await.unwrap();
                for i in 0..events.read {
                    let buf = &mut buffers[i];
                    let ptr = buf.as_ptr() as *const PacketLog;
                    let data = unsafe { ptr.read_unaligned() };
                    let src_addr = net::Ipv4Addr::from(data.ipv4_address);
                    println!("LOG: SRC {}, ACTION {}", src_addr, data.action);
                }
            }
        });
    }
    signal::ctrl_c().await.expect("failed to listen for event");
    Ok::<_, anyhow::Error>(())
}

This will now spawn a tokio::task to read each of the AsyncPerfEventArrayBuffers contained in out AsyncPerfEventArray. When we receive an event, we use read_unaligned to read our data into a PacketLog. We then use println! to log the event to the console. We no longer need to sleep, as we run until we receive the CTRL+C signal.

Running the program

$ cargo build
$ cargo xtask build-ebpf
$ sudo ./target/debug/myapp ./target/bpfel-unknown-none/debug/myapp wlp2s0
LOG: SRC 192.168.1.205, ACTION 2
LOG: SRC 192.168.1.21, ACTION 2
LOG: SRC 192.168.1.21, ACTION 2
LOG: SRC 18.168.253.132, ACTION 2
LOG: SRC 18.168.253.132, ACTION 2
LOG: SRC 18.168.253.132, ACTION 2
LOG: SRC 140.82.121.6, ACTION 2

Building eBPF Programs With Aya