Hugging Face
Run decoder-only Hub folders when the layout, dtype, and tokenizer are in the supported set. Automatic layout detection, a configurable memory budget, and CPU execution unless a native GPU is present.
What loads
A folder with config.json and dense *.safetensors (F16, BF16, or F32, single file or shards) for decoder-only classes such as *ForCausalLM and *LMHeadModel. Tensor names are detected across Llama-style layouts, GPT-2, fused QKV, Mixtral experts, Qwen3 QK-norm, and Gemma residual norms. Probe a directory with uaii inspect before generate — status: ok means the family math is wired (Gemma (1+w) RMSNorm, GPT-2 wpe, Qwen3 QK-norm, sliding window), not merely that tensors were named. Chat uses a documented family equivalent of the shipped Jinja chat_template, not a Jinja interpreter.
Quantized Hub repos (GPTQ, AWQ, bitsandbytes, HQQ, compressed-tensors, MXFP4) are not loaded: convert them to dense BF16/F16 or to GGUF. Kimi / DeepSeek MLA folders are blocked on Hugging Face; run the GGUF instead (uaii generate --model kimi.gguf). gpt-oss is detected and fail-closed until a family handler exists.
Generation and chat default to a storage-first memory budget sized for a laptop: pin a trunk that still fits (norm, then as many of embed / lm_head / early layers as the 8 GiB floor allows), stream the remainder from disk as packed GGUF bytes, page MoE experts through an LRU filled only with leftover RAM, and slide the KV cache across a 2048-token window with 4 attention sinks. Large checkpoints stay on disk and remain runnable at reduced throughput. A single unquantized tensor larger than about half the budget fail-closes with a convert-to-GGUF message. See docs/huggingface_support.md in the repository for the implementation details.
Model sizes
| Scale | Behaviour on roughly 8 GB of RAM |
|---|---|
| 0.5B to 8B | Fits in memory comfortably, with a cold load on the first generate |
| 30B and above, dense | Runs by paging weights. Bound by disk and CPU rather than capacity |
| 500B-class MoE | Same path: page the routed experts as packed GGUF blocks and bound the KV cache. NVMe traffic dominates latency. Convert Mixtral/Qwen-MoE to GGUF Q4_K for the 8 GB floor; BF16 expert banks that exceed half the budget fail closed |
| Trillion-parameter | Not benchmarked, so no resident-memory figure is published |
Paging works at tensor granularity, so a dense checkpoint with a single FFN tensor larger than available RAM can still fail to load. Mixture-of-experts checkpoints are the reliable path on constrained machines. Adjust the budget with --preset desktop|server|none, --ram-gb, and --kv-window.
CLI
uaii pull org/model --outdir ./models/hf/org__model uaii inspect ./models/hf/org__model uaii generate --model ./models/hf/org__model --dry-run uaii generate --model ./models/hf/org__model --prompt "Hello" --json uaii generate --model kimi.gguf --prompt "Hello" uaii generate --model ./models/hf/org__model --preset none --prompt "Hello" uaii generate --model ./models/hf/org__model --kv-window 4096 --prompt "Hello" uaii convert ./models/hf/org__model -o model.uaii.json
The device defaults to --backend auto, which uses a native GPU when this binary can drive one and CPU otherwise. Pass --backend cpu to pin the host.
Dashboard
Open Models, choose Pull from Hub or Import HF folder, then switch to Chat. The backend setting defaults to automatic GPU detection. Point UAII_BIN at a current uaii build. An older uaii-dash.json pinned to "backend": "cpu" keeps that value, so reset it to Auto in Settings. Full guide: Operator UI.
Current limitations
- Weight dtypes: F16, BF16, and F32 safetensors. F8 and GPTQ/AWQ/MXFP4 fail closed with a convert hint
- GPTQ, AWQ, and bitsandbytes repos are not loaded. Convert to dense F16/BF16 or to GGUF
- Vision, audio, encoder-decoder, SSM, and RWKV models are not supported. Plugin scaffolds exist for the architectures on the roadmap
- Hugging Face DeepSeek / Kimi MLA checkpoints are not mapped. Use the GGUF expand path
- gpt-oss dense packing is fail-closed (
family=gpt_oss). Convert MXFP4 to BF16 or GGUF; there is no Llama fallback