Features

A modular inference runtime. Hugging Face decoder folders and GGUF blk.* transformers are the supported surface today, and other formats and architectures arrive as plugins.

What UAII is

An execution platform for AI inference rather than a single-model engine. Loaders, operators, backends, schedulers, and storage providers all plug into a common core. The supported paths today are Hugging Face decoder-only folders and GGUF transformers. Layout detection covers Llama-style models, GPT-2, fused QKV, and Mixtral experts. Vision, SSM, and HF MLA layouts are not yet mapped.

Model formats

Format	What you get
Hugging Face dir	Direct support for decoder-only CausalLM families with automatic layout detection, through `uaii pull`, `generate --model ./dir`, and dashboard Hub pulls. Vision, SSM, and HF MLA layouts are excluded.
GGUF	Capability-based import for llama.cpp-style `blk.` decoder tensors, dense or MoE with per-expert `#e=N` slices, including the DeepSeek MLA expand path. Reads `{arch}.` metadata. Common Q and K quants plus IQ4_NL are supported; other IQ variants stop with an error.
Safetensors	Used inside Hugging Face folders. Standalone weight graphs also import when the layout maps to the IR
ONNX	Imports to the IR from a sidecar JSON file, or from proto when enabled
MLX	`config.json` plus `.safetensors`, importing both weights and configuration
PyTorch	`.pt` and `.pth` through an exported `.onnx` or `.uaii.json` sidecar

Quantization & compute

In-memory GGUF block quants: Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q2_K–Q6_K
Pack and unpack helpers for F16, BF16, INT8, INT4, NF4, and MXFP4
Session policy through compute_dtype, either F32 or F16, and keep_quantized_weights
Unrecognised GGUF quantization types stop with an error and a conversion hint rather than filling weights with placeholder values

LLM runtime

Prefill and decode with greedy or temperature, top-k, and top-p sampling, through Session::generate and uaii_session_generate
CLI entry points uaii pull, uaii generate, and uaii chat --jsonl, the warm session the dashboard drives
Sliding-window KV cache, 2048 tokens with 4 attention sinks on the laptop preset, with peak RSS reporting
Memory presets laptop, desktop, and server: pin a trunk that fits, stream packed GGUF weights, and page only routed MoE experts into leftover RAM
Tokenizers for Hugging Face tokenizer JSON, GGUF tokenizer.ggml.* metadata, BPE, optional SentencePiece, and a simple vocabulary for demos

Operator dashboard

The console in dashboard/ runs locally or self-hosted. It provides Hub pulls, streaming chat over GGUF and Hugging Face models, a model library, health checks, benchmarks, a request queue, Bearer authentication for non-loopback binds, and an OpenAI-compatible /v1/chat/completions endpoint. See Operator UI.

Hardware

CPU execution is always fully supported. The CLI and dashboard default to --backend auto, which selects a native GPU only when this binary was built against the vendor SDK and a device is present, and CPU in every other case. uaii doctor reports the recommended: device alongside any host_fallback flags. See Backends.

Runtime tooling

Graph validator with JSON and binary IR formats, .uaii.json and .uaii
Planner covering fusion, memory reuse, storage plans, and a disk plan cache
Weight streaming, top-k expert I/O, and asynchronous CUDA host-to-device overlap when native support is compiled in
Chrome-trace profiler, benchmark CLI, and a versioned plugin operator ABI