This change implements additional symbol resolving for stack traces, which
improve the support for userspace stack traces.
A challenge with the stack traces obtained via ebpf is that they just contain
a raw address inside the virtual memory space of the process, which
requires some extra steps to be translated into a function name. This change
provides most of the infrastructure to provide the translation via a `DefaultResolver`
type which can resolve both kernelspace and userspace functions based on
a PID and address. In addition in adds a `SymbolResolver` trait that allows to customize
resolving behavior and extend it further.
I originally looked into just using the [StackTrace::resolve](https://docs.rs/aya/0.11.0/aya/maps/stack_trace/struct.StackTrace.html#method.resolve)
function. However I noticed that the information that is at the moment stored
in the StackTrace struct is not sufficient for resolving, e.g. due to missing the PID
and being unable to account for the virtual -> physical memory address translation.
Therefore this change uses a new resolver infrastructure.
Usage via a suitable EBPF program that sends stack traces and process IDs:
```rust
/// The BPF program populates this Queue with the process ID, kernel-space and user-space
/// stack trace IDs.
/// The former can be obtained via `bpf_get_current_pid_tgid()`, the stack
/// traces via `aya_bpf::maps::StackTrace`.
let mut stacks = Queue::<_, [u64; 3]>::try_from(bpf.map_mut("STACKS")?)?;
let stack_traces = StackTraceMap::try_from(bpf.map_mut("STACK_TRACES")?)?;
let resolver = DefaultResolver::new().unwrap();
loop {
match stacks.pop(0) {
Ok([pid_tgid, ktrace_id, utrace_id]) => {
let tgid = pid_tgid & 0xFFFFFFFF;
if let Ok(trace) = stack_traces.get(&(ktrace_id as u32), 0) {
for f in trace.frames() {
let mut symbol = SymbolInfo::unresolved_kernel(f.ip);
resolver.resolve(&mut symbol);
println!("Resolved kernel address: 0x{:x} to {:?}", f.ip, symbol);
}
}
if let Ok(trace) = stack_traces.get(&(utrace_id as u32), 0) {
for f in trace.frames() {
let mut symbol = SymbolInfo::unresolved_user(tgid as _, f.ip);
resolver.resolve(&mut symbol);
info!("Resolved pid {}, address: 0x{:x} to {:?}", tgid, f.ip, symbol);
}
}
}
}
}
```
BPF code:
```rust
static STACK_TRACES: StackTrace = StackTrace::with_max_entries(10, 0);
static STACKS: Queue<[u64; 3]> = Queue::with_max_entries(1024, 0);
pub fn testprobe(ctx: ProbeContext) -> u32 {
unsafe {
let pid_tgid = bpf_get_current_pid_tgid();
let ustack = STACK_TRACES.get_stackid(&ctx, BPF_F_USER_STACK as _);
let kstack = STACK_TRACES.get_stackid(&ctx, 0);
match (kstack, ustack) {
(Ok(kstack), Ok(ustack)) => {
if let Err(e) = STACKS.push(&[pid_tgid, kstack as _, ustack as _], 0) {
info!(&ctx, "Error pushing stack: {}", e);
}
},
_ => {}
}
0
}
}
```
Output when fetching stack traces of a kprobe on `sendmmsg` on a test program:
```
Resolved kernel address: 0xffffffff81a62691 to SymbolInfo { virtual_address: 18446744071589734033, object_address: Some(18446744071589734033), process_id: None, function_name: Some("__sys_sendmmsg"), object_path: None }
Resolved kernel address: 0xffffffff81a62870 to SymbolInfo { virtual_address: 18446744071589734512, object_address: Some(18446744071589734512), process_id: None, function_name: Some("__x64_sys_sendmmsg"), object_path: None }
Resolved kernel address: 0xffffffff81da30a3 to SymbolInfo { virtual_address: 18446744071593144483, object_address: Some(18446744071593144483), process_id: None, function_name: Some("do_syscall_64"), object_path: None }
Resolved kernel address: 0xffffffff81e0007c to SymbolInfo { virtual_address: 18446744071593525372, object_address: Some(18446744071593525372), process_id: None, function_name: Some("entry_SYSCALL_64_after_hwframe"), object_path: None }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x7fccf3b2adee to SymbolInfo { virtual_address: 140518238629358, object_address: Some(1195502), process_id: Some(22158), function_name: None, object_path: Some("/usr/lib/x86_64-linux-gnu/libc-2.31.so") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570cd66cc to SymbolInfo { virtual_address: 94856245241548, object_address: Some(947916), process_id: Some(22158), function_name: Some("tokio::io::async_fd::AsyncFdReadyGuard<Inner>::try_io"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570cd43e6 to SymbolInfo { virtual_address: 94856245232614, object_address: Some(938982), process_id: Some(22158), function_name: Some("quinn_udp:👿:UdpSocket::poll_send"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570ccd31f to SymbolInfo { virtual_address: 94856245203743, object_address: Some(910111), process_id: Some(22158), function_name: Some("<quinn::endpoint::EndpointDriver as core::future::future::Future>::poll"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570cc2400 to SymbolInfo { virtual_address: 94856245158912, object_address: Some(865280), process_id: Some(22158), function_name: Some("<core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570cd0e8d to SymbolInfo { virtual_address: 94856245218957, object_address: Some(925325), process_id: Some(22158), function_name: Some("tokio::runtime::task::harness::poll_future"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570cd160a to SymbolInfo { virtual_address: 94856245220874, object_address: Some(927242), process_id: Some(22158), function_name: Some("tokio::runtime::task::harness::Harness<T,S>::poll"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570c6ef25 to SymbolInfo { virtual_address: 94856244817701, object_address: Some(524069), process_id: Some(22158), function_name: Some("std:🧵:local::LocalKey<T>::with"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570c7e505 to SymbolInfo { virtual_address: 94856244880645, object_address: Some(587013), process_id: Some(22158), function_name: Some("tokio::runtime::basic_scheduler::Context::run_task"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570c7b2fa to SymbolInfo { virtual_address: 94856244867834, object_address: Some(574202), process_id: Some(22158), function_name: Some("tokio::macros::scoped_tls::ScopedKey<T>::set"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570c7de7c to SymbolInfo { virtual_address: 94856244878972, object_address: Some(585340), process_id: Some(22158), function_name: Some("tokio::runtime::basic_scheduler::BasicScheduler::block_on"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570c75a3d to SymbolInfo { virtual_address: 94856244845117, object_address: Some(551485), process_id: Some(22158), function_name: Some("tokio::runtime::Runtime::block_on"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570c5c597 to SymbolInfo { virtual_address: 94856244741527, object_address: Some(447895), process_id: Some(22158), function_name: Some("std::sys_common::backtrace::__rust_begin_short_backtrace"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570c59d44 to SymbolInfo { virtual_address: 94856244731204, object_address: Some(437572), process_id: Some(22158), function_name: Some("core::ops::function::FnOnce::call_once{{vtable.shim}}"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
02:46:16 [INFO] ayatest: [ayatest/src/main.rs:98] Resolved pid 22158, address: 0x564570ecc763 to SymbolInfo { virtual_address: 94856247297891, object_address: Some(3004259), process_id: Some(22158), function_name: Some("std::sys::unix:🧵:Thread:🆕:thread_start"), object_path: Some("/mnt/c/Users/matth/Code/rust/quinn/target/release/bulk") }
```
This commit moves the aya-log projects from the subtree and adds them to
the main cargo workspace. It also brings the BPF crates into the
workspace and moves the macro crates up a level since they aren't BPF
code.
Miri was disabled for aya-bpf as the previous config wasn't actually
checking anything.
CI, clippy, fmt and release configurations have all been adjusted
appropriately.
CI was not properly running for other supported arches which was also
ixed here.
Signed-off-by: Dave Tucker <dave@dtucker.co.uk>
Change 821ba0b243fd removed the `size > buf.len()` check, which was a
mistake, because we might write to a subslice of the whole buffer, so
then `buf` can be lower than `LOG_BUF_CAPACITY`.
This change compares `size` with `min::(buf.len(), LOG_BUF_CAPACITY)`
instead.
Fixes: 821ba0b243fd ("Ensure log buffer bounds")
Signed-off-by: Michal Rostecki <vadorovsky@gmail.com>
This change adds checks in `TagLenValue.write()` to ensure that the size
of written data doesn't exceed the buffer size.
Verifier in recent kernel versions requires the bound to be a constant
value, so using `buf.len()` does not work.
Signed-off-by: Michal Rostecki <vadorovsky@gmail.com>
This moves a large chunk of code from ebpf to shared so we can re-use
write_record_header and write_record_message and friends so that we
can write test cases to ensure that logs are properly formatted
given certain input.
Signed-off-by: Dave Tucker <dave@dtucker.co.uk>
eBPF verifier rejects programs which are not checking the bounds of the
log buffer before writing any arguments. This change ensures that
written log arguments.
In practice, it means that doing this kind of checks is not going to be
needed in eBPF program code anymore:
33a1aee2ea/echo-ebpf/src/main.rs (L47)
Tested on:
876f8b4551
Signed-off-by: Michal Rostecki <vadorovsky@gmail.com>
This change moves away argument formatting from eBPF to the userspace.
eBPF part of aya-log writes unformatted log message and all arguments to
the perf buffer and the userspace part of aya-log is formatting the
message after receiving all arguments.
Aya-based project to test this change:
https://github.com/vadorovsky/aya-log-exampleFixes: #4
Signed-off-by: Michal Rostecki <vadorovsky@gmail.com>
Signed-off-by: Tuetuopay <tuetuopay@me.com>
Co-authored-by: Tuetuopay <tuetuopay@me.com>
The `bpf_printk!` macro is a helper providing a convenient way to invoke the
`bpf_trace_printk` and `bpf_trace_vprintk` BPF helpers. It is implemented as
a macro because it requires variadic arguments.