MIT · C++17 · CMake
Universal AI Inference
One runtime for Hugging Face decoder-only folders and GGUF blk.* models. The same paging dial runs on a laptop or a phone. Unsupported layouts stop with an error.
Runtime
What it runs
Models
HF decoders and GGUF
Llama-like, Qwen, Mistral/Mixtral, Gemma, and GPT-2. GGUF adds packed MoE experts and MLA. Vision, SSM, and HF MLA stay out.
Laptop
~8 GiB, KV 2048
Default preset. Pins a trunk that fits, prefetches with two staging buffers, and pages the rest. Shrinks to free RAM only when 8 GiB does not fit.
Phone
~2 GiB, KV 512
One staging buffer, 4 threads. The budget is half of the RAM the OS can still give the process. Q4_K stays on the CPU quant GEMM.
Embed
C API 0.4.0
The C API applies the same dial as the CLI. Phone apps resolve a preset, then cancel, background, or drop KV on a memory warning.
Start
Build, then generate
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release cmake --build build --parallel ./build/libs/uaii-cli/uaii doctor
uaii pull org/model --outdir ./hf-dir uaii generate --model ./hf-dir \ --preset laptop --prompt "hi" uaii generate --model m.q4_k.gguf \ --preset phone --prompt "hi"
cmake -S examples/mobile -B build-android \ -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake \ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-24 cmake --build build-android --target uaii_phone