Open source · MIT licensed
UAII
Any model. One IR. Any hardware.
UAII compiles Hugging Face decoder-only checkpoints and GGUF blk.* transformers into a single intermediate representation, then executes it on CPU or a native GPU. Weight streaming, a sliding KV cache, and top-k expert paging keep 30B-class models inside a laptop memory budget. Unsupported layouts stop with a precise error instead of producing wrong output.
Scope
What runs today
Hugging Face decoder-only folders and GGUF blk.* transformers are supported paths. The CLI and dashboard select the device automatically: a native GPU when this binary was built against the vendor SDK and a device is present, CPU otherwise. Generation runs on a memory budget you set, so large checkpoints stream from disk rather than exhausting RAM.
HF and GGUF decoders
CLI pull and generate, dashboard chat, in-memory K-quants and IQ4_NL, Mixtral and Qwen MoE expert paging.
30B to 500B MoE
Weight streaming, top-k expert I/O, and a 2048-token KV window bound resident memory. Throughput at that scale is limited by disk, not capacity.
Further architectures
Plugin scaffolds for SSM, RWKV, and vision encoders.
Start here
Benchmarks
Measured on commodity hardware
Absolute GEMM throughput, memory bandwidth, and attention latency from uaii_bench v3 on a WSL2 host with an Intel Core i9-14900HX. Every figure below reproduces with a single command.
425 GFLOP/s
1024³ f32 GEMM, and 284 GFLOP/s at 512³. Median of 21 trials.
15.1 GFLOP/s
Same shape on the built-in kernel, with no vendor BLAS linked.
16.6 GB/s
STREAM triad across a 256 MiB working set.
--suite all
Raw results in benchmarks/results/local_wsl.json.
Pipeline
How a checkpoint becomes tokens
Four stages, one runtime path: ingest, plan, execute, integrate.
Ingest
Hugging Face directories, GGUF, Safetensors, and ONNX, MLX, or PyTorch sidecars compile into UAII IR.
Plan
Shape validation, operator fusion, memory reuse, and a disk-backed plan cache.
Execute
Quantized GEMM, sliding KV cache, top-k expert paging, and verified device selection.
Integrate
CLI, stable C ABI, and Python SDK share identical session semantics.
Capabilities
Built for production workloads
Formats, quantization, backends, and tooling that compose into one runtime.
Hugging Face decoder-only
Hub folders load when layout, dtype, and family math match: Llama, Qwen2/3, Mistral/Mixtral, Gemma, GPT-2. Probe with uaii inspect — status: ok means the graph is wired, not merely that tensors exist. Unknown families and quantized Hub repos fail closed. Vision, SSM, HF MLA, and gpt-oss packing are not loaded.
GGUF blk.* transformers
Capability-based import for llama.cpp-style tensors: dense or MoE with packed banks split per expert, the DeepSeek MLA expand path, and common Q and K quants plus IQ4_NL.
Operator dashboard
A local or self-hosted console for Hub pulls, streaming chat over HF and GGUF models, health checks, benchmarks, an OpenAI-compatible /v1 endpoint, and a request queue.
In-memory quantized GEMM
Q4, Q5, Q8, and K-quants execute without unpacking every weight to f32. IQ4_NL is supported; other IQ variants stop with a conversion hint.
Competitive CPU GEMM
The IGemm interface dispatches to oneDNN or OpenBLAS when linked, and a tiled parallel kernel otherwise. uaii doctor reports the active provider.
GPU backends with verified capabilities
--backend auto resolves to CPU unless native CUDA, Metal, ROCm, or Vulkan support is compiled in and a device responds. CUDA runs decode attention and RoPE on device when enabled, and every host fallback is reported by uaii doctor.
Developer surface
One mental model across CLI demos, C embedders, and Python notebooks, with strict defaults, struct_size ABI compatibility, and Chrome-trace profiling.
| Surface | Entry |
|---|---|
| CLI | generate · chat · run · convert |
| Dashboard | cd dashboard && npm start |
| C API | uaii_capi · 0.3.0 |
| Python | pip install -e bindings/python |
Quick start
From clone to tokens
Build once, then diagnose, pull, and generate. No cloud account required.
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release cmake --build build --config Release --parallel
uaii doctor uaii pull org/model --outdir ./hf-model uaii generate --model ./hf-model --prompt "hi" --preset laptop uaii generate --model model.gguf --prompt "hi"
cd dashboard npm run install:all npm run build && npm start # → http://127.0.0.1:8787
Contribute
Build the runtime with us
UAII is MIT licensed and developed in the open. Loaders, kernels, backends, benchmarks, and documentation all take outside contributions, and every change starts with a GitHub issue so the design is agreed before code is written.
Open an issue
File a bug report or feature request from the issue templates. Include your OS, compiler, CMake version, and the exact command that failed.
Claim the work
Comment on the issue to take it, or pick one labelled good first issue. A maintainer confirms scope before you start.
Build and verify
Fork, branch, then build with CMake and run uaii doctor, ctest, and clang-format before you push.
Send the pull request
Keep it focused, link the issue, explain why the change is needed, and update the docs that describe the behaviour you changed.
Start building on UAII
Build the runtime, pull a Hugging Face decoder or GGUF checkpoint, generate your first tokens, then open the operator console. Begin with the Hugging Face path.