Docs

Architecture

Every model compiles to UAII IR. The session plans that graph, then executes it on CPU or on a native GPU when device selection finds one.

Pipeline

Modules

uaii-ir

Graph and tensor types, operator registry, validator, serialization, and plan representation

uaii-runtime

Session lifecycle, DeviceScheduler, sliding KV cache, and memory presets

uaii-kernels

CPU operators, the IGemm interface, and quantized GEMM

uaii-backends

CPU, plus optional CUDA, Metal, Vulkan, WebGPU, and ROCm

uaii-loaders

Hugging Face directories, GGUF including per-expert MoE slices, Safetensors, ONNX, MLX, and PyTorch

uaii-tokenizers

Simple vocabularies, BPE, SentencePiece, and GGUF metadata

uaii-planner

Fusion, memory and storage plans, and the disk plan cache

uaii-storage

Memory mapping, weight streaming, and the expert LRU cache

uaii-capi

The stable C ABI that SDKs build on

Storage-first execution

CLI and dashboard generation apply the laptop preset unless you pass another one. Weights stay on disk. The session pins a trunk that fits and pages the rest. Phone uses one staging buffer, so the largest streamed tensor has to fit once. Laptop and above use two, so it has to fit twice.

• Final norm stays resident. Embeddings, lm_head, and early layers stay only while pin plus staging still fits. GGUF block-quants stay packed
• KV is a sliding window (2048 on laptop, 512 on phone) plus 4 attention sinks. RoPE keeps absolute positions
• Packed expert banks split into #e=N. Only the routed top-k are staged
• auto picks the first native GPU that reports a real device. Attention and RoPE stay on CPU. Q4_K stays in the CPU quant GEMM

C API 0.4.0 applies the same dial through uaii_resolve_memory_preset. The Python Session.from_path helper does not call that yet, so it stays on cpu with kv_window = 0 unless you pass those options yourself.

IR on disk

.uaii.json

Readable JSON for hand-authored graphs and debugging

.uaii

Native binary with the UAIR magic number, aligned to the schema in schemas/uaii_ir.fbs

Full detail is in docs/architecture.md in the repository.