Architecture

Every model compiles to UAII IR. The session plans that graph, then executes it on CPU or on a native GPU when device selection finds one.

Pipeline

Modules

uaii-ir

Graph and tensor types, operator registry, validator, serialization, and plan representation

uaii-runtime

Session lifecycle, DeviceScheduler, sliding KV cache, and memory presets

uaii-kernels

CPU operators, the IGemm interface, and quantized GEMM

uaii-backends

CPU, plus optional CUDA, Metal, Vulkan, WebGPU, and ROCm

uaii-loaders

Hugging Face directories, GGUF including per-expert MoE slices, Safetensors, ONNX, MLX, and PyTorch

uaii-tokenizers

Simple vocabularies, BPE, SentencePiece, and GGUF metadata

uaii-planner

Fusion, memory and storage plans, and the disk plan cache

uaii-storage

Memory mapping, weight streaming, and the expert LRU cache

uaii-capi

The stable C ABI that SDKs build on

Storage-first execution

The CLI and dashboard apply a memory budget by default, so the process pages weights in rather than mapping an entire checkpoint into RAM. This is how 30B-class dense decoders and 500B-class MoE checkpoints run on a laptop: throughput becomes bound by NVMe bandwidth, and capacity stops being the limit.

• Embeddings, lm_head, and early trunk layers stay pinned only while they fit the budget. The planner drops pin_layers, then lm_head, then embed until pin plus two staging buffers fit. GGUF block-quant tensors stay packed in those buffers. Everything else streams through file#tensor references
• The KV cache keeps a resident window, 2048 tokens by default, plus 4 attention sinks. RoPE preserves absolute positions, so long conversations compact the window instead of growing memory
• GGUF packed expert banks are split into per-expert references (#e=N). After routing, only the top-k experts are staged, and the rest remain on disk
• auto selects the first native GPU that reports a real device. A backend built without its vendor SDK does not qualify

C API 0.3.0 and the default Python session do not apply this preset. They run on cpu with kv_window = 0 unless you set the options the ABI exposes. Because paging works per tensor, a dense tensor larger than available RAM can still fail to load.

IR on disk

.uaii.json

Readable JSON for hand-authored graphs and debugging

.uaii

Native binary with the UAIR magic number, aligned to the schema in schemas/uaii_ir.fbs

Full detail is in docs/architecture.md in the repository.