Skip to content
boolescale
All posts

A performance stack for a hot-path Rust service

We're building Garm, a gateway that sits between an AI agent and its tools and decides whether each tool call is allowed. That puts it on the path of every tool call the agent makes. If it adds noticeable latency, people turn it off, and a security control that's switched off protects nothing.

So performance had to be designed in from the start, not fixed later. The second pull request in the repository, before any real logic existed, was the performance harness. This post walks through what's in it and why.

Start with the questions, not the tools

"Add benchmarks" is too vague to be useful. Each tool in the stack answers one specific question:

QuestionToolWhen it runs
What counts as fast enough?A budgets fileRead by humans on every review
How fast is this one function?divanLocally, cargo bench
Did this PR make it slower?CodSpeedEvery PR, in CI
Does this allocate?dhatEvery test run
What latency does a user actually see?HDR histogram load testNightly
Why is it slow?samply / flamegraphBy hand, when investigating

If a tool doesn't answer one of these questions, it doesn't go in.

1. Budgets before code

The first file isn't code at all. It's benches/budgets.toml:

[core.label_combine]
description = "Label combine / session update"
bench = "garm-core::label_combine"
max_ns = 1_000
 
[policy.evaluate_typical]
description = "Policy evaluation (typical ruleset)"
bench = "garm-policy::evaluate_typical"
max_ns = 10_000
 
[dlp.scan_throughput]
description = "DLP scan throughput"
bench = "garm-dlp::scan"
min_mb_per_s = 500
 
[e2e.added_latency]
description = "Total added latency per tool call (p99)"
bench = "garm-bench::e2e"
max_p99_ms = 5

The end-to-end budget (5 ms added at p99) is a product requirement: the gateway should be invisible next to the tool call it guards. The per-step budgets are far below it on purpose, so the steps together leave most of that 5 ms for I/O and the proxy itself. They're first guesses, to be revisited once there's a real baseline.

Writing budgets down before the code exists changes how reviews go. "Is 4 µs for policy evaluation OK?" stops being a matter of opinion. It's under 10 µs, so yes. The rule that goes with the file: if a PR blows a budget, the PR explains why. Nobody quietly raises the number.

2. Micro-benchmarks with divan

divan is a Rust benchmarking library with very little ceremony. A bench is an attributed function, and args runs it once per input size. Here's the baseline for scanning a tool-call payload:

fn main() {
    divan::main();
}
 
/// Buffer sizes representative of tool-call payloads
/// (small JSON-RPC message up to a large file read).
const SIZES: &[usize] = &[256, 4 * 1024, 64 * 1024];
 
/// Lower bound for a single pass over a tool-call payload: count bytes matching a predicate.
#[divan::bench(args = SIZES)]
fn byte_scan_baseline(bencher: divan::Bencher, size: usize) {
    let payload: Vec<u8> = (0..size)
        .map(|i| u8::try_from(i % 251).expect("i % 251 fits in u8"))
        .collect();
    bencher
        .counter(divan::counter::BytesCount::new(size))
        .bench(|| {
            divan::black_box(&payload)
                .iter()
                .filter(|b| b.is_ascii_alphanumeric())
                .count()
        });
}

Two details are doing the work here. The payload is built outside the bench closure, so only the scan is timed. And BytesCount tells divan how much data each iteration processes, so alongside the time it reports throughput:

╰─ byte_scan_baseline   fastest       │ slowest       │ median        │ mean
   ├─ 256               930.9 ns      │ 6.993 µs      │ 943.4 ns      │ 1.003 µs
   │                    274.9 MB/s    │ 36.6 MB/s     │ 271.3 MB/s    │ 255.1 MB/s
   ├─ 4096              14.99 µs      │ 15.49 µs      │ 15.29 µs      │ 15.25 µs
   │                    273 MB/s      │ 264.2 MB/s    │ 267.7 MB/s    │ 268.5 MB/s
   ╰─ 65536             241.3 µs      │ 456.8 µs      │ 244.8 µs      │ 253.4 µs
                        271.4 MB/s    │ 143.4 MB/s    │ 267.6 MB/s    │ 258.5 MB/s

That throughput line is what makes the number usable. Our DLP budget is in MB/s, not nanoseconds, so the output can be read against the budget directly. Here it says something uncomfortable: a naive scan that checks one byte at a time manages about 270 MB/s on a laptop, well under the 500 MB/s budget. That doesn't mean the budget is impossible. Real detectors use vectorised, memchr-style scanning that handles many bytes per instruction. But we found out before writing the DLP crate, not after, that the straightforward version won't be enough.

cargo bench -p garm-core prints fastest, slowest, median and mean for every size. It's good for local before/after numbers in a PR description, as long as both runs happen on the same machine.

One dependency trick makes the rest of the stack fit together. In the workspace Cargo.toml, we don't depend on divan directly:

[workspace.dependencies]
divan = { package = "codspeed-divan-compat", version = "6.0.0" }

codspeed-divan-compat keeps divan's API. Built normally, it behaves as plain divan. Built with cargo codspeed, the same benchmarks report to CodSpeed instead. One bench file serves local runs and CI, and no code has to change to switch between them.

3. A regression gate that isn't noise

Running benchmarks on shared CI runners and comparing wall-clock times usually goes badly. Noisy neighbours, CPU frequency scaling and different hardware between runs easily cause 10–20% swings. A real 5% regression gets lost in that noise. The usual outcome is people learning to ignore the benchmark job.

CodSpeed avoids this in its simulation mode. It doesn't time anything. It runs each benchmark under Valgrind, counts the instructions executed, and models cache behaviour to estimate a time. The same code produces almost the same count on any runner, so a 2–3% change is a real signal. CodSpeed comments the diff on the PR and fails the check above a threshold; ours is 5%.

The trade-off is that instruction counts are a proxy. They don't fully capture branch misprediction or SIMD throughput. That makes them very good at answering "did this get slower?" and less good at "how fast is this, really?". The local divan numbers answer the second question.

The workflow is short:

jobs:
  codspeed:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
      - run: cargo install cargo-codspeed --version 6.0.0 --locked
      - run: cargo codspeed build --workspace
      - uses: CodSpeedHQ/action@78102a468f96d8e0ccb92d0ec49425753fc42062 # v5.6.0
        with:
          mode: simulation
          run: cargo codspeed run

Two details are worth calling out.

Authenticate with OIDC, not a stored token. Our first run failed with "Unable to retrieve OIDC token for authentication". Without a token, the action falls back to GitHub OIDC: GitHub issues a short-lived token for each run. That needs id-token: write. Grant it on the benchmark job only, not the whole workflow, because the permission isn't tied to a particular service. This turned out better than the token we'd planned: there's no long-lived secret to store or rotate.

Pin actions to commit SHAs. Tags like @v5 can be moved by whoever controls the action's repository. Pinning the full SHA with a version comment means CI runs exactly the code you reviewed, and Dependabot still opens PRs to update the SHA and the comment together.

4. Zero allocations, as a test

On a hot path, heap allocation is the biggest hidden cost. It goes through the allocator, sometimes reaches the kernel, and causes latency spikes that show up at p99. Our rule is that labelling and policy evaluation allocate zero times per call.

A rule like that is only as good as its enforcement, so we made it a test. dhat can replace the global allocator and count every allocation. A small helper in our test-kit crate turns that into an assertion:

/// Installs dhat's counting allocator as the global allocator of the current test binary.
#[macro_export]
macro_rules! install_counting_allocator {
    () => {
        #[global_allocator]
        static GARM_TESTKIT_ALLOC: $crate::alloc::dhat::Alloc = $crate::alloc::dhat::Alloc;
    };
}
 
// dhat allows one profiler per process at a time.
static PROFILER: Mutex<()> = Mutex::new(());
 
pub fn count_allocs<R>(f: impl FnOnce() -> R) -> (R, AllocStats) {
    let _guard = PROFILER.lock().unwrap_or_else(PoisonError::into_inner);
    let _profiler = dhat::Profiler::builder().testing().build();
    let before = dhat::HeapStats::get();
    let out = black_box(f());
    let after = dhat::HeapStats::get();
    let stats = AllocStats {
        blocks: after.total_blocks - before.total_blocks,
        bytes: after.total_bytes - before.total_bytes,
    };
    (out, stats)
}
 
pub fn assert_no_alloc<R>(mut f: impl FnMut() -> R) {
    black_box(f()); // warm-up call, not counted
    let (_, stats) = count_allocs(&mut f);
    assert_eq!(stats, AllocStats { blocks: 0, bytes: 0 }, "expected zero heap allocations");
}

A hot-path crate then has a tests/alloc.rs like this:

garm_testkit::install_counting_allocator!();
 
#[test]
fn combine_does_not_allocate() {
    let (a, b) = (label(), label());
    garm_testkit::alloc::assert_no_alloc(|| a.combine(b));
}

Three details make this reliable rather than flaky:

  • The warm-up call. Code often allocates once on first use, for example a lazily built table. That's fine; allocating on every call is not. assert_no_alloc runs the closure once before it starts counting.
  • black_box. Without it the optimiser can delete work whose result is unused, along with its allocations, and the test passes for the wrong reason. This matters most in release builds, so we run the alloc tests in both debug and --release.
  • One process per test. The counting allocator sees every thread in the process. Under plain cargo test, other tests running on other threads add noise. cargo-nextest runs each test in its own process, which removes the problem. dhat's testing mode also writes no profile file, so tests leave nothing on disk.

The helper has its own tests. A 64-byte Vec must count as exactly one block of 64 bytes, arithmetic must count as zero, the warm-up allocation must be excluded, and a closure that allocates must fail the assertion. A checker that has never been seen to fail can't be trusted.

5. Tail latency with HDR histograms

Averages hide exactly what users notice. A service that answers in 1 ms 99 times and 200 ms once has a mean of about 3 ms and feels broken. Our end-to-end budget is a p99, so we need real percentiles.

An HDR histogram records values in fixed memory, in constant time, with a precision you choose. With 3 significant digits, every reported percentile is within 0.1% of the true value:

const MAX_NS: u64 = 60 * 1_000_000_000;
 
pub struct LatencyRecorder {
    hist: Histogram<u64>,
}
 
impl LatencyRecorder {
    pub fn new() -> Self {
        let hist = Histogram::new_with_bounds(1, MAX_NS, 3).expect("static histogram bounds are valid");
        Self { hist }
    }
 
    /// Records one latency. Values outside 1 ns..=60 s are clamped.
    pub fn record(&mut self, latency: Duration) {
        let ns = u64::try_from(latency.as_nanos()).unwrap_or(u64::MAX);
        self.hist.saturating_record(ns.clamp(1, MAX_NS));
    }
}

The hdrhistogram crate is used with default-features = false. We only need recording and percentiles, not its serialisation and compression features, so we skip those dependencies. A nightly job will drive the proxy with a fake tool server and record every call through this, so slow drift across many small PRs shows up as a trend line before anyone feels it.

6. Profiling, when the numbers say something's wrong

Everything above tells you that something is slow. Profilers tell you why. The one setup step is in the release profile:

[profile.release]
lto = "fat"
codegen-units = 1
debug = "line-tables-only"

line-tables-only keeps enough debug info for profiles to map back to source lines, without the size cost of full debug info. After that, samply works on Windows, macOS and Linux, which matters because our main development machine runs Windows.

What it cost

The tooling isn't free. codspeed-divan-compat brings about 50 transitive crates (clap and regex among them), and dhat about 25 more. For a security product, every dependency is a supply-chain decision, so it's worth stating precisely: all of them are dev-only and none reach the shipped binary. cargo-deny checks licences, advisories and sources on every PR either way.

In practice, the whole harness, including its own tests and docs, came in under 400 changed lines.

The takeaway

The specific tools matter less than the order:

  1. Decide what "fast enough" means, write it down, and make changing it a visible decision.
  2. Make measurements reproducible: instruction counts in CI, the same machine for before/after locally.
  3. Turn performance rules into tests, like zero allocations, so they hold without anyone remembering them.
  4. Measure what users feel: percentiles, end to end.

Set this up before the hot path exists, and every later PR arrives with numbers instead of promises.