Docs
Hugging Face
Run decoder-only Hub folders when the layout, dtype, and tokenizer are in the supported set. Automatic layout detection, a configurable memory budget, and CPU execution unless a native GPU is present.
What loads
A folder with config.json and dense *.safetensors (F16, BF16, or F32, single file or shards) for decoder-only classes such as *ForCausalLM and *LMHeadModel. Tensor names are detected across Llama-style layouts, GPT-2, fused QKV, Mixtral experts, Qwen3 QK-norm, and Gemma residual norms. Probe a directory with uaii inspect before generate — status: ok means the family math is wired (Gemma (1+w) RMSNorm, GPT-2 wpe, Qwen3 QK-norm, sliding window), not merely that tensors were named. Chat uses a documented family equivalent of the shipped Jinja chat_template, not a Jinja interpreter.
Quantized Hub repos (GPTQ, AWQ, bitsandbytes, HQQ, compressed-tensors, MXFP4) are not loaded: convert them to dense BF16/F16 or to GGUF. Kimi / DeepSeek MLA folders are blocked on Hugging Face; run the GGUF instead (uaii generate --model kimi.gguf). gpt-oss is detected and fail-closed until a family handler exists.
Generation defaults to the laptop preset: pin a trunk that fits, page packed GGUF weights, and keep a 2048-token KV window. Laptop keeps two staging buffers, so an unquantized tensor larger than about half the budget fail-closes. Convert Hugging Face BF16 7B and larger to GGUF Q4_K. Phone is a separate preset with one staging buffer. See Memory.
Model sizes
| Scale | What the dial does |
|---|---|
| 0.5B to 8B | Often fits the laptop budget. First generate is a cold load |
| Larger dense GGUF | Pages packed weights. Throughput follows disk and CPU. No device tok/s is published for this path |
| MoE GGUF | Pages the routed experts. Convert Mixtral and Qwen-MoE to Q4_K. BF16 expert banks that exceed the budget fail closed |
| Hugging Face BF16, 7B and up | Fail-closed. Convert to GGUF Q4_K |
Paging works at tensor granularity, so a dense checkpoint with a single FFN tensor larger than available RAM can still fail to load. Mixture-of-experts checkpoints are the reliable path on constrained machines. Adjust the budget with --preset laptop|desktop|server|phone|none, --ram-gb, and --kv-window. Phone success is peak RSS, not tokens per second.
CLI
uaii pull org/model --outdir ./models/hf/org__model uaii inspect ./models/hf/org__model uaii generate --model ./models/hf/org__model --dry-run uaii generate --model ./models/hf/org__model --prompt "Hello" --json uaii generate --model kimi.gguf --prompt "Hello" uaii generate --model ./models/hf/org__model --preset none --prompt "Hello" uaii generate --model ./models/hf/org__model --kv-window 4096 --prompt "Hello" uaii convert ./models/hf/org__model -o model.uaii.json
The device defaults to --backend auto, which uses a native GPU when this binary can drive one and CPU otherwise. Pass --backend cpu to pin the host.
Dashboard
Open Models, choose Pull from Hub or Import HF folder, then switch to Chat. The backend setting defaults to automatic GPU detection. Point UAII_BIN at a current uaii build. An older uaii-dash.json pinned to "backend": "cpu" keeps that value, so reset it to Auto in Settings. Full guide: Operator UI.
Current limitations
- Weight dtypes: F16, BF16, and F32 safetensors. F8 and GPTQ/AWQ/MXFP4 fail closed with a convert hint
- GPTQ, AWQ, and bitsandbytes repos are not loaded. Convert to dense F16/BF16 or to GGUF
- Vision, audio, encoder-decoder, SSM, and RWKV models are not supported. Plugin scaffolds exist for the architectures on the roadmap
- Hugging Face DeepSeek / Kimi MLA checkpoints are not mapped. Use the GGUF expand path
- gpt-oss dense packing is fail-closed (
family=gpt_oss). Convert MXFP4 to BF16 or GGUF; there is no Llama fallback