Local AI
Glitr can run a small language model and text-to-image models on the device. After the weights are downloaded, chat turns and image prompts stay in-process — they are not sent to a remote LLM API. Local AI is not part of the git mailbox, the recipient crypto cascade, or the live WebRTC path.
This page is the ai sibling library and how whatsup hosts it. Session, poll, and send stay on Network. How the shell sits on the rest of the stack is the Architecture product map.
The product path is closed-source. The ai gallery is the public surface you can press: positive-intentions.github.io/ai.
Product wiring
whatsup links the engines as path Cargo crates from the sibling ai repo:
| Target | Crate | When |
|---|---|---|
| Browser (wasm) | ai-web | Always for wasm32 |
| Desktop GUI | ai-core | Cargo feature desktop |
whatsup-ui has no dependency on ai. It owns the Local AI chrome and domain scaffolding (ChatKind::LocalAi, synthetic contact ids, load/manage dialogs, markdown assistant bubbles). whatsup is the host that calls into ai-web or ai-core and pushes tokens or image data URLs into AppState.
| Host module | Role |
|---|---|
ai_gui.rs | Browser: WebLLM chat + ORT image over WebGPU |
ai_native.rs | Desktop: Candle GGUF chat + Candle Stable Diffusion |
gui.rs | provide_ai_ctx(), route /ai, speed dial, overlays |
The Local AI thread uses fixed ids (AI_CHAT_ID, AI_CONTACT_ID). It is hidden from the main chat list, excluded from mailbox snapshots, and skipped on the TUI live-send path. Image prompts are detected with parse_image_intent (/image …, “draw me …”, and similar phrases); the host unloads chat VRAM when needed, runs the image session, and attaches a PNG data URL in the thread.
Platform availability: desktop and browser GUI builds. Mobile-without-desktop and TUI show a stub (“available in desktop and browser builds”). UI details: UI/UX.
Inference after load is local. First-time model download still uses the network (Hugging Face, MLC, CDNs). That is not the same as a cloud assistant.
ai-core
ai-core is the shared Rust core: curated catalogs, SHA-256-pinned fetch, chat templates, streaming AiSession, parallel ImageSession, and pluggable engines.
| Concern | Behavior |
|---|---|
| Catalog | Static ModelSpec list; list, get, list_for(Platform), default_web |
| Fetch | HttpFetcher pulls Hub resolve/main URLs; progress callbacks |
| Integrity | SHA-256 verify before cache write; mismatch → remove + error |
| Cache | DiskCache::user_cache() under the OS project dir for com.positive-intentions.ai (for example ~/.cache/positive-intentions/ai) |
| Chat | pull / load / generate_user / next_token / cancel_generation / history reset |
| Image | ImageSession + parse_image_intent; engines produce PNG bytes |
| Engines | CandleEngine / CandleImageEngine (feature candle); MockEngine / MockImageEngine for tests and --no-default-features |
Candle is not llama.cpp. Only curated GGUF arches ship in the catalog; arbitrary Hub GGUF files are unsupported. Optional native GPU features (cuda, metal) exist; CI stays on CPU/mock.
ai-web
ai-web is the browser product path. It does not run Candle in WASM for production chat.
| Surface | Stack |
|---|---|
| Chat | WebLLM (@mlc-ai/web-llm@0.2.79 from jsDelivr), preferably a Worker engine; MLC prebuilt catalog (separate ids from native GGUF) |
| Image | ORT-Web WebGPU pipelines (sd_pipeline.js) for SD 1.5 / SDXL-Lightning ONNX packs |
| Prompt | Engine-side Glitr system prompt (system_prompt_for); not shown in the UI |
| Cache | Browser Cache Storage for weights; last model id in localStorage |
WebGPU is required for load. Browser image generation also requires shader-f16. Without an adapter or f16 support, load fails fast instead of hanging. Ubuntu + NVIDIA often needs Chrome launched with Vulkan ANGLE flags so requestAdapter() succeeds; the ai README documents npm run chrome:webgpu.
The host unloads WebLLM before starting an ORT image session so the two do not fight over the same GPU.
Catalogs
Hosts must pick the catalog that matches the surface. Native Candle uses GGUF ids. Browser WebLLM uses MLC ids. Image ids are shared by name across native and browser, with different backends underneath.
Native chat (Candle GGUF)
| Id | Approx. size | Notes |
|---|---|---|
tinyllama-stories-15m | ~19 MB | Tiny; web-default helper for mock/CLI |
smollm2-135m | ~105 MB | Instruct |
qwen3-0.6b-q4 | ~380 MB | Qwen3 |
smollm2-1.7b | ~1.06 GB | Native |
qwen25-1.5b-q4 | ~1.12 GB | Native |
qwen25-3b-q4 | ~2.1 GB | Native |
smollm3-3b-q4 | ~1.92 GB | Native; not in WebLLM |
Browser chat (WebLLM / MLC)
| Id | Notes |
|---|---|
SmolLM2-135M-Instruct-q0f32-MLC | Web default (f32; no shader-f16 required for this entry) |
Qwen2.5-0.5B-Instruct-q4f32_1-MLC | Larger |
Llama-3.2-1B-Instruct-q4f32_1-MLC | ~900 MB |
Qwen2.5-1.5B-Instruct-q4f32_1-MLC | ~1.1 GB; desktop GPU |
SmolLM2-1.7B-Instruct-q4f32_1-MLC | ~1.3 GB; high VRAM |
Qwen2.5-3B-Instruct-q4f32_1-MLC | ~1.9 GB; high VRAM |
Image (OpenRAIL-oriented catalog)
| Id | License | Approx. download | Notes |
| --- | --- | --- |
| sd15 | OpenRAIL-M | ~2 GB | Default. Native: Candle SD 1.5. Browser: Microsoft ONNX via ORT-Web |
| sdxl-lightning | OpenRAIL++-M | ~7 GB native / multi-GB ONNX | High VRAM. Browser: 4-step ONNX. Native uses SDXL-base weights with Lightning-oriented defaults |
Turbo / FLUX [dev] / revenue-capped community packs are excluded on purpose so the closed product path stays on commercial-safer OpenRAIL entries. Hub and MLC model licenses are separate from the library license; they are not part of the AGPL grant on the ai code.
CLI, WASM façade, gallery
| Piece | Role |
|---|---|
| ai-cli | models, pull, chat, complete, image over Candle or mock |
| ai-wasm (root crate) | wasm-bindgen + AiWasmClient JS helper; mock engine only — caller supplies GGUF bytes; not the production browser chat path |
| ai-gallery | Separate Dioxus workspace using whatsup-ui chrome: Local chat, Local image, Benchmark (GUI + TUI) |
Gallery e2e (Playwright) builds with the mock engine so CI never downloads Hub or MLC weights. The Pages build uses WebLLM; models download in the browser on first use.
The office gallery can also dock these engines beside document stories (skills + office::tools). That is a gallery host path, separate from the product Local AI route at /ai. See Drive and Office.
Privacy and network
| Claim | Meaning |
|---|---|
| On-device inference | After weights load, generation runs in the browser or desktop process |
| Not mailbox crypto | Prompts are not cascaded to a peer; the AI thread is not a contact you sync with |
| Not synced | Mailbox snapshots omit the Local AI thread; it is reattached in-session only |
| Network still used | First download of weights/engine from Hub, MLC, or CDNs; “local” ≠ offline-by-default |
| Residual | Large Cache Storage / disk caches stay on the device; clearing models is a product UI action |
The WebLLM system prompt states the hard boundaries: only this thread is visible; no other chats, contacts, or keys. That is product framing for the model, not a sandbox proof.
Runtime and limits
- Browser: Chromium-class browser with a WebGPU adapter; image path needs shader-f16.
- Desktop: RAM/VRAM for chosen GGUF or SD weights; GPU strongly preferred for image.
- Dual catalogs: do not assume a GGUF id works in WebLLM or the reverse.
- Root WASM: mock-only; production browser chat is ai-web + WebLLM.
- License: ai code is AGPL-3.0-only; third-party crates and downloaded models keep their own licenses. A commercial license for proprietary use of the code is offered separately and does not change the public AGPL grant.
Tests
| Layer | What |
|---|---|
| ai-core | Unit tests; CI gates 100% line coverage with and without Candle |
| ai-cli | assert_cmd integration on the mock engine |
| ai-wasm | wasm-pack test --node on the mock façade |
| ai-web | Catalog / system-prompt / image-catalog unit tests |
| e2e | Playwright against the mock gallery build (UI smoke; no real model download) |
Next: Data structures for mailbox record shapes. How the shell mounts Local AI chrome is on UI/UX.