Move the use of clang and llvm-objcopy from run-time to build-time. This
allows the integration tests to run on VMs with simpler userlands.
Create a new CI job to build the integration tests separately from
running them. Ship them from that job to the runner job using github
actions artifacts.
The aya-bpf-macros needed refactoring for:
1. Ease of testing
2. To be consistent with when we use K/V args vs. idents
3. To deprecate the use of `name` to change the exported name of a
function - we now use the symbol table.
Signed-off-by: Dave Tucker <dave@dtucker.co.uk>
This makes a few changes to the way that Aya reads the ELF object
files.
1. To find programs in a section, we use the symbols table. This allows
for cases where multiple programs could appear in the same section.
2. When parsing our ELF file we build symbols_by_section_index as an
optimization as we use it for legacy maps, BTF maps and now programs.
As a result of theses changes the "NAME" used in `bpf.prog_mut("NAME")`
is now ALWAYS the same as the function name in the eBPF code, making the
user experience more consistent.
Signed-off-by: Dave Tucker <dave@dtucker.co.uk>
For tests that do networking operations, this allows to have a
clean-state network namespace and interfaces for each test. Mainly, this
avoids "device or resource busy" errors when reusing the loopback
interface across tests.
In release, the trigger functions were being optimized out and the
tests did not work. Use core::hint::black_box to ensure that the
functions are not optimized out. Also, run these integration tests
in CI to ensure that we don't regress.
Trampoline cargo-in-cargo stdio through cargo:warning to ensure the user
sees all the output.
Use bpf-linker from git in CI so we can see what's going on there.
Remove the manual dependency tracking machinery in
integration-test/build.rs in favor of a build-dependency on
integration-ebpf. This required adding an empty lib.rs to create the
library target.
This allows integration-test/build.rs to be ignorant of bpf-linker.
Remove that in favor of the logic now in integration-ebpf.
Extract the symlink-to-bpf-linker logic from integration-test to xtask
and use it in a new build script in integration-ebpf, causing ebpf
probes to be rebuilt when bpf-linker changes. Previously bpf-linker
changes would rebuild integration-test, but not integration-ebpf,
resulting in stale tests.
Note that this still doesn't address the possibility that a new
bpf-linker is added to the PATH ahead of the cached one. Solving this in
the general case would require rebuild-if-changed-env=PATH *and*
rebuild-if-changed={every-directory-in-PATH} which would likely mean far
too much cache invalidation.
The ebpf probes require a nightly compiler. Before this change, if you
ran `cargo xtask integration-test` with a stable compiler toolchain as
default, or you ran `cargo +stable xtask integration-test`, you would
have seen an error like the one below. This is now fixed by running the
cargo build command in the integration-ebpf directory and making sure
to clear the RUSTUP_TOOLCHAIN env var.
```
--- stderr
/home/ajwerner/src/github.com/aya-rs/aya/test/integration-test/bpf/ring_buf_sched_tracepoint.bpf.c:18:21: warning: declaration of 'struct switch_args' will not be visible outside of this function [-Wvisibility]
int bpf_prog(struct switch_args* ctx)
^
1 warning generated.
error: the `-Z` flag is only accepted on the nightly channel of Cargo, but this is the `stable` channel
See https://doc.rust-lang.org/book/appendix-07-nightly-rust.html for more information about Rust release channels.
thread 'main' panicked at '"cargo" "build" "-p" "integration-ebpf" "-Z" "build-std=core" "--release" "--message-format=json" "--target" "bpfel-unknown-none" "--target-dir" "/home/ajwerner/src/github.com/aya-rs/aya/target/debug/build/integration-test-9bbcb3db5e9f8f57/out/integration-ebpf" exited
with status code 101:
', test/integration-test/build.rs:219:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: error while building userspace application
Caused by:
Child { stdin: None, stdout: None, stderr: None, .. } exited with status code 101:
```
Libbpf is used by xtasks, in the command, ensure that the submodules
are initialized. This eases the user-experience so that users don't
need to think about the submodule, while retaining all the benefits
of using a submodule vs forcing the user to manually check out libbpf
and stick it in some pre-defined place.
We use the symbol pointing to libbpf in xtask in the build script
to avoid repeating this constant.
Also, we install git in the vm so that we can init the submodule
when we build in the vm.
Emit "cargo:rerun-if-changed={}" for each transitive dependency on
integration-ebpf. In a normal world we'd just add integration-ebpf to
our build-dependencies, but cargo ignores this because integration-ebpf
has no library targets.
The matches crate has been archived now that `matches!` is in std.
However `assert_matches!` is still unstable in std, and the
assert_matches crate provides a more expressive form:
```
assert_matches!(foo, Ok(bar) => {
assert_eq!(bar, baz);
});
```
This feature is equivalent to async_tokio || async_std; removing it
avoids warnings emitted during `cargo hack check --feature-powerset`
where async is selected without either of the other features.
Use cargo hack to ensure clippy runs on the powerset of features.
in the integration tests we recenctly switched to using
our internal api to list programs. I was seeing times when
this would race and panic internally (program fd was deleted
by aya WHILE we were trying to get it). This ensures that
the list succeeded without panicking.
Signed-off-by: Andrew Stoycos <astoycos@redhat.com>
Add integration testing for link pinning and
loading/unloading of tracepoint, kprobe, and
uprobe programs.
Redo how we utilize bpftool to verify that programs
are loaded to be explicit with names. Also add a helper
to verify that a program is loaded AND linked.
Signed-off-by: Andrew Stoycos <astoycos@redhat.com>
"integration tests" as defined by Cargo produce a binary per file in the
tests directory. This is really not what we want and has a number of
downsides, but the main one is binary size.
Before:
tamird@pc:~/src/aya$ cargo xtask build-integration-test | xargs ls -lah
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `target/debug/xtask build-integration-test`
Compiling integration-test v0.1.0 (/home/tamird/src/aya/test/integration-test)
Finished dev [unoptimized + debuginfo] target(s) in 0.68s
-rwxrwxr-x 1 tamird tamird 34M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/bpf_probe_read-e03eb905a5e6209c
-rwxrwxr-x 1 tamird tamird 35M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/btf_relocations-57a4fbb38bf06064
-rwxrwxr-x 1 tamird tamird 31M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/elf-98b7a32d6d04effb
-rwxrwxr-x 1 tamird tamird 6.9M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/integration_test-0dd55ce96bfdad77
-rwxrwxr-x 1 tamird tamird 34M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/load-0718562e85b86d03
-rwxrwxr-x 1 tamird tamird 40M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/log-dbf355f9ea34068a
-rwxrwxr-x 1 tamird tamird 36M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/rbpf-89a1bb848fa5cc3c
-rwxrwxr-x 1 tamird tamird 34M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/relocations-cfe655c3bb413d8b
-rwxrwxr-x 1 tamird tamird 34M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/smoke-ccd3974180a3fd29
After:
tamird@pc:~/src/aya$ cargo xtask build-integration-test | xargs ls -lah
Finished dev [unoptimized + debuginfo] target(s) in 0.05s
Running `target/debug/xtask build-integration-test`
Compiling integration-test v0.1.0 (/home/tamird/src/aya/test/integration-test)
Finished dev [unoptimized + debuginfo] target(s) in 0.90s
-rwxrwxr-x 1 tamird tamird 47M Jul 12 15:21 /home/tamird/src/aya/target/debug/deps/integration_test-0dd55ce96bfdad77
Since we plan to run these tests in a VM, copying 10x fewer bytes seems
like a win.
Use the environment variable AYA_BUILD_INTEGRATION_BPF to indicate to
the build script that it should *actually* build bpf, otherwise emitting
empty files.
This allows metadata builds to skip costly build steps without
sacrificing ergonomics; all compile-time tools such as cargo clippy work
out of the box.
Cargo even gives each of these builds (depending on the value of the
environment variable) its own cache key, so they do not invalidate each
other when the user alternates between metadata and real builds.
This allows the lint action to move out of the VM.
- Add libbpf as a submodule. This prevents having to plumb its location
around (which can't be passed to Cargo build scripts) and also
controls the version against which codegen has run.
- Move bpf written in C to the integration-test crate and define
constants for each probe.
- Remove magic; each C source file must be directly enumerated in the
build script and in lib.rs.
Replace all `assert!(matches!(..))` with `assert_matches!(..)`.
Remove the now-unused build-integration-test xtask command whose logic
doesn't match that of the build-and-run command.
This doesn't add any value; use `cargo build --tests` with
`--message-format=json` instead; parse the output using `cargo_metadata`
to discover the location of the test binary.
Move test/integration-test/src/tests -> test/integration-test/tests to
conform to
https://doc.rust-lang.org/book/ch11-03-test-organization.html#integration-tests.
This change does a few things:
- it fixes a bug in the wrappers, where we were expecting the kernel to
return len=1 for b"\0" where it instead returns 0 and doesn't write
out the NULL terminator
- it makes the helpers more robust by hardcoding bound checks in
assembly so that LLVM optimizations can't transform the checks in a
way that the verifier can't understand.
- it adds integration tests
This fix aya wrong logic causing non entrypoint functions to not have
any BTF relocations working.
Also fix missing section_offset computation for instruction offset in
multiple spots.
This commit adds from_pin() which allows the creation of a Program
from a path on bpffs. This is useful to be able to call `attach` or
other APIs for programs that are already loaded to the kernel.
This differs from #444 since it implements this on the concrete program
type, not the Program enum, allowing the user to pass in any additional
context that isn't available from bpf_prog_info.
Signed-off-by: Dave Tucker <dave@dtucker.co.uk>
Before this chane, the check was always negative if the minor version
was less then 9. So, for example, the smoke test was skipped for kernel
6.1:
```
skipping as 6.1 does not meet version requirement of 5.9
```
Signed-off-by: Michal Rostecki <vadorovsky@gmail.com>
Simplifiy the relocation tests build process by removing the need for libbpf
at runtime. Its usage is replaced with local `__builtin_*` attributes.
This removes the need for the `LIBBPF_INCLUDE` env variable.
This commit adds documentation on how program names are parsed from
section names, as is used by `aya_obj::Object.programs` as HashMap keys,
and updates the examples into using program names.
Instead of returning anyhow>>Result<()> handle errors
'in-place' with unwrap or panic, for more informative
and user-friendly error messages on test failures.
Fixes#421.
Signed-off-by: Dmitry Savintsev <dsavints@gmail.com>
switch map() and map_mut() from returning a
`Result` to an `Option` since it's just getting
a value from a Hashmap, and to stay in line with
the Programs API.
Remove `MapError::MapNotFound`
Signed-off-by: Andrew Stoycos <astoycos@redhat.com>
Build completing tests passing
Refactor the Map API to better align
with the aya programs API. Specifically
remove all internal locking mechanisms
and custom Deref/DerefMut implementations.
They are replaced with a Map enum
and AsRef/AsMut implementations.
All Try_From implementations have been moved
to standardized enums, with a slightly
special one for PerfEventArray's.
Also cleanup/fix all associated tests and
documentation.
Signed-off-by: Andrew Stoycos <astoycos@redhat.com>
We often need to wait a bit until the program or link gets unloaded
after dropping.
Also, using a macro makes it clear in which particular test the panic
happened.
Signed-off-by: Michal Rostecki <vadorovsky@gmail.com>
libbpf 1.0 doesn't support multimaps defined in `maps` section, it
supports only BTF maps.
At the same time, we already test multimaps loading with the Rust
example (`integration-ebpf/src/bpf/map_test.rs`), so we can just remove
the failing C test.
Signed-off-by: Michal Rostecki <vadorovsky@gmail.com>
1. Removes OwnedLink
2. Allows Links to be converted into FdLink
3. Introduces a PinnedLink type to handle wrap FdLink when pinned and
support un-pinning
Signed-off-by: Dave Tucker <dave@dtucker.co.uk>
Tamir Duberstein
Dave Tucker
Tuetuopay
Andrew Werner
vadorovsky
Andrew Stoycos
Alessandro Decina
dependabot[bot]
Mary
Michal Rostecki
Matteo Nardi
Shenghui Ye
Dmitry Savintsev
abhijeetbhagat