UAIIRuntime
GitHubQuick start

Quick start

Build the C++ runtime, generate from a Hugging Face or GGUF checkpoint, then integrate through the C API or Python SDK.

Start with a Hugging Face CausalLM folder or a GGUF blk.* model. Generation defaults to --preset laptop, which streams weights and slides the KV cache, and --backend auto, which uses a native GPU when one is available and CPU otherwise. The Hugging Face guide covers the Hub path in detail.

Prerequisites

  • C++17 compiler (MSVC 2019+, Clang 10+, or GCC 9+)
  • CMake 3.20+
  • Ninja (recommended) or your platform generator
  • Optional: CUDA Toolkit, Vulkan SDK, ROCm, oneDNN, OpenBLAS, SentencePiece, Python 3.10+

Build

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DUAII_BUILD_TESTS=ON -DUAII_BUILD_PLUGINS=ON
cmake --build build --config Release --parallel

Windows multi-config generators place binaries under build/Release/. Single-config Ninja builds produce build/libs/uaii-cli/uaii, or uaii.exe on Windows.

First commands

uaii version
uaii doctor

# Hugging Face decoder-only
uaii pull org/model --outdir ./models/hf/org__model
uaii generate --model ./models/hf/org__model --prompt "hi" --max-new-tokens 64 --preset laptop

# Or a GGUF blk.* transformer, same memory preset and device selection
uaii generate --model path/to/model.gguf --prompt "hi" --max-new-tokens 64

# Smoke test the IR stack with no model weights
uaii generate --demo --prompt "hi" --max-new-tokens 4 --json

Hugging Face support requires a current build, so rebuild uaii after pulling. Point the dashboard at that binary with UAII_BIN.

Dashboard (optional UI)

cd dashboard
npm run install:all
npm run build && npm start
# → http://127.0.0.1:8787

# Point at the rebuilt CLI, required for Hugging Face models
UAII_BIN=/path/to/uaii npm start

# Self-host on a LAN or server
UAII_DASH_BIND=0.0.0.0 UAII_DASH_TOKEN=secret npm start

The console provides Hub pulls, streaming chat over HF and GGUF models, health checks, benchmarks, and an OpenAI-compatible /v1 endpoint. The default backend setting is automatic GPU detection. See the operator UI guide.

Run a graph

uaii run examples/ir/toy_mlp.uaii.json \
  --weight-init ones \
  --input x=1,2,3,4 \
  --output y_prob

Python (optional)

python bindings/python/scripts/bundle_native.py --build-dir build
pip install -e bindings/python
python examples/python/load_run_profile.py

If the loader cannot locate the shared library, set UAII_CAPI_PATH to the full path of uaii_capi.dll, .so, or .dylib.

Tests

ctest --test-dir build -C Release --output-on-failure

Next steps

  • Hugging Face — the Hub path, supported model sizes, and memory paging
  • Features — formats, quantization, and runtime capabilities
  • Operator UI — local and self-hosted console
  • CLI — pull, generate, and chat
  • Contribute — open an issue and send your first pull request