Benchmarks

Absolute measurements from uaii_bench v3: multi-provider GEMM, memory bandwidth, attention, session throughput, and Q4 memory footprint.

Workload	Result	Notes
OpenBLAS 256 / 512 / 1024	141 / 284 / 425 GFLOP/s	32 threads on WSL2
ref-tiled 256 / 512 / 1024	7.7 / 11.8 / 15.1 GFLOP/s	Built-in kernel, no vendor BLAS
Bandwidth triad	16.6 GB/s	STREAM triad, 256 MiB working set
Attention e2e	59.7 ms	B1 H8 S512 D64 on the ref GEMM
Session 8×512	2.77 ms	`Session::run`
Q4_0 memory	7.11× · 4.5 MiB	Compression against 32 MiB in f32

Host: WSL2 on an Intel Core i9-14900HX, Release build, with ref and openblas linked. Raw output is committed at benchmarks/results/local_wsl.json.

Reproduce

sudo apt install -y libopenblas-dev libdnnl-dev
TRIALS=21 bash scripts/run_bench_wsl.sh
# or:
./build/benchmarks/uaii_bench --suite all --providers all --trials 21 --json

Suites

gemm: Cycles every linked provider: ref, onednn, and openblas
bandwidth: STREAM copy, scale, add, and triad
attention: QKᵀ, softmax, and AV as one microbenchmark
session, quant: The IR execution stack and the Q4_0 packed path

Scope of these numbers

Everything above is a kernel or session measurement on one host. Three figures are deliberately absent, because they are not measured yet:

Tokens per second on public LLM checkpoints
Comparisons against llama.cpp, ONNX Runtime, or TensorRT
oneDNN throughput, which is only reported when onednn appears in linked_providers

Full methodology is in docs/benchmarks.md in the repository. See also Examples.