Move the use of clang and llvm-objcopy from run-time to build-time. This
allows the integration tests to run on VMs with simpler userlands.
Create a new CI job to build the integration tests separately from
running them. Ship them from that job to the runner job using github
actions artifacts.
The aya-bpf-macros needed refactoring for:
1. Ease of testing
2. To be consistent with when we use K/V args vs. idents
3. To deprecate the use of `name` to change the exported name of a
function - we now use the symbol table.
Signed-off-by: Dave Tucker <dave@dtucker.co.uk>
This makes a few changes to the way that Aya reads the ELF object
files.
1. To find programs in a section, we use the symbols table. This allows
for cases where multiple programs could appear in the same section.
2. When parsing our ELF file we build symbols_by_section_index as an
optimization as we use it for legacy maps, BTF maps and now programs.
As a result of theses changes the "NAME" used in `bpf.prog_mut("NAME")`
is now ALWAYS the same as the function name in the eBPF code, making the
user experience more consistent.
Signed-off-by: Dave Tucker <dave@dtucker.co.uk>
For tests that do networking operations, this allows to have a
clean-state network namespace and interfaces for each test. Mainly, this
avoids "device or resource busy" errors when reusing the loopback
interface across tests.
In release, the trigger functions were being optimized out and the
tests did not work. Use core::hint::black_box to ensure that the
functions are not optimized out. Also, run these integration tests
in CI to ensure that we don't regress.
Trampoline cargo-in-cargo stdio through cargo:warning to ensure the user
sees all the output.
Use bpf-linker from git in CI so we can see what's going on there.
Remove the manual dependency tracking machinery in
integration-test/build.rs in favor of a build-dependency on
integration-ebpf. This required adding an empty lib.rs to create the
library target.
This allows integration-test/build.rs to be ignorant of bpf-linker.
Remove that in favor of the logic now in integration-ebpf.
Extract the symlink-to-bpf-linker logic from integration-test to xtask
and use it in a new build script in integration-ebpf, causing ebpf
probes to be rebuilt when bpf-linker changes. Previously bpf-linker
changes would rebuild integration-test, but not integration-ebpf,
resulting in stale tests.
Note that this still doesn't address the possibility that a new
bpf-linker is added to the PATH ahead of the cached one. Solving this in
the general case would require rebuild-if-changed-env=PATH *and*
rebuild-if-changed={every-directory-in-PATH} which would likely mean far
too much cache invalidation.
The ebpf probes require a nightly compiler. Before this change, if you
ran `cargo xtask integration-test` with a stable compiler toolchain as
default, or you ran `cargo +stable xtask integration-test`, you would
have seen an error like the one below. This is now fixed by running the
cargo build command in the integration-ebpf directory and making sure
to clear the RUSTUP_TOOLCHAIN env var.
```
--- stderr
/home/ajwerner/src/github.com/aya-rs/aya/test/integration-test/bpf/ring_buf_sched_tracepoint.bpf.c:18:21: warning: declaration of 'struct switch_args' will not be visible outside of this function [-Wvisibility]
int bpf_prog(struct switch_args* ctx)
^
1 warning generated.
error: the `-Z` flag is only accepted on the nightly channel of Cargo, but this is the `stable` channel
See https://doc.rust-lang.org/book/appendix-07-nightly-rust.html for more information about Rust release channels.
thread 'main' panicked at '"cargo" "build" "-p" "integration-ebpf" "-Z" "build-std=core" "--release" "--message-format=json" "--target" "bpfel-unknown-none" "--target-dir" "/home/ajwerner/src/github.com/aya-rs/aya/target/debug/build/integration-test-9bbcb3db5e9f8f57/out/integration-ebpf" exited
with status code 101:
', test/integration-test/build.rs:219:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: error while building userspace application
Caused by:
Child { stdin: None, stdout: None, stderr: None, .. } exited with status code 101:
```
Libbpf is used by xtasks, in the command, ensure that the submodules
are initialized. This eases the user-experience so that users don't
need to think about the submodule, while retaining all the benefits
of using a submodule vs forcing the user to manually check out libbpf
and stick it in some pre-defined place.
We use the symbol pointing to libbpf in xtask in the build script
to avoid repeating this constant.
Also, we install git in the vm so that we can init the submodule
when we build in the vm.
Emit "cargo:rerun-if-changed={}" for each transitive dependency on
integration-ebpf. In a normal world we'd just add integration-ebpf to
our build-dependencies, but cargo ignores this because integration-ebpf
has no library targets.
The matches crate has been archived now that `matches!` is in std.
However `assert_matches!` is still unstable in std, and the
assert_matches crate provides a more expressive form:
```
assert_matches!(foo, Ok(bar) => {
assert_eq!(bar, baz);
});
```
This feature is equivalent to async_tokio || async_std; removing it
avoids warnings emitted during `cargo hack check --feature-powerset`
where async is selected without either of the other features.
Use cargo hack to ensure clippy runs on the powerset of features.
in the integration tests we recenctly switched to using
our internal api to list programs. I was seeing times when
this would race and panic internally (program fd was deleted
by aya WHILE we were trying to get it). This ensures that
the list succeeded without panicking.
Signed-off-by: Andrew Stoycos <astoycos@redhat.com>
Add integration testing for link pinning and
loading/unloading of tracepoint, kprobe, and
uprobe programs.
Redo how we utilize bpftool to verify that programs
are loaded to be explicit with names. Also add a helper
to verify that a program is loaded AND linked.
Signed-off-by: Andrew Stoycos <astoycos@redhat.com>
We need bpftool to add tests for the link APIs since we don't yet have
and aya API for listing links.
Signed-off-by: Andrew Stoycos <astoycos@redhat.com>
"integration tests" as defined by Cargo produce a binary per file in the
tests directory. This is really not what we want and has a number of
downsides, but the main one is binary size.
Before:
tamird@pc:~/src/aya$ cargo xtask build-integration-test | xargs ls -lah
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `target/debug/xtask build-integration-test`
Compiling integration-test v0.1.0 (/home/tamird/src/aya/test/integration-test)
Finished dev [unoptimized + debuginfo] target(s) in 0.68s
-rwxrwxr-x 1 tamird tamird 34M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/bpf_probe_read-e03eb905a5e6209c
-rwxrwxr-x 1 tamird tamird 35M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/btf_relocations-57a4fbb38bf06064
-rwxrwxr-x 1 tamird tamird 31M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/elf-98b7a32d6d04effb
-rwxrwxr-x 1 tamird tamird 6.9M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/integration_test-0dd55ce96bfdad77
-rwxrwxr-x 1 tamird tamird 34M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/load-0718562e85b86d03
-rwxrwxr-x 1 tamird tamird 40M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/log-dbf355f9ea34068a
-rwxrwxr-x 1 tamird tamird 36M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/rbpf-89a1bb848fa5cc3c
-rwxrwxr-x 1 tamird tamird 34M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/relocations-cfe655c3bb413d8b
-rwxrwxr-x 1 tamird tamird 34M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/smoke-ccd3974180a3fd29
After:
tamird@pc:~/src/aya$ cargo xtask build-integration-test | xargs ls -lah
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `target/debug/xtask build-integration-test`
Compiling integration-test v0.1.0 (/home/tamird/src/aya/test/integration-test)
Finished dev [unoptimized + debuginfo] target(s) in 0.90s
-rwxrwxr-x 1 tamird tamird 47M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/integration_test-0dd55ce96bfdad77
Since we plan to run these tests in a VM, copying 10x fewer bytes seems
like a win.
Use the environment variable AYA_BUILD_INTEGRATION_BPF to indicate to
the build script that it should *actually* build bpf, otherwise emitting
empty files.
This allows metadata builds to skip costly build steps without
sacrificing ergonomics; all compile-time tools such as cargo clippy work
out of the box.
Cargo even gives each of these builds (depending on the value of the
environment variable) its own cache key, so they do not invalidate each
other when the user alternates between metadata and real builds.
This allows the lint action to move out of the VM.
- Add libbpf as a submodule. This prevents having to plumb its location
around (which can't be passed to Cargo build scripts) and also
controls the version against which codegen has run.
- Move bpf written in C to the integration-test crate and define
constants for each probe.
- Remove magic; each C source file must be directly enumerated in the
build script and in lib.rs.
Replace all `assert!(matches!(..))` with `assert_matches!(..)`.
Remove the now-unused build-integration-test xtask command whose logic
doesn't match that of the build-and-run command.
This doesn't add any value; use `cargo build --tests` with
`--message-format=json` instead; parse the output using `cargo_metadata`
to discover the location of the test binary.
Move test/integration-test/src/tests -> test/integration-test/tests to
conform to
https://doc.rust-lang.org/book/ch11-03-test-organization.html#integration-tests.
Tamir Duberstein
Dave Tucker
Tuetuopay
Andrew Werner
vadorovsky
Andrew Stoycos