Documentation
How to run the engine, what each device does, and how the pieces fit. Measured numbers are on the wiki.
Guides
| Page | What it covers |
|---|---|
| Strix Halo guide | Run local LLMs on AMD Strix Halo (Ryzen AI MAX+ 395) on Linux: the Radeon 8060S iGPU through HRX, the XDNA 2 NPU, Lemonade Server, and where to start. |
| FAQ | Answers about local LLMs with 1bit engine: how much memory a model needs, which fit in 128 GB, Linux vs Windows, the NPU, Lemonade Server, the license. |
Start
| Page | What it covers |
|---|---|
| 1bit serve | The engine runs inside Lemonade as a backend, the way Lemonade runs llama-server. Lemonade runs 1bit serve, and for a GGUF it starts the llama-server that serve names, one HTTP… |
| Models measured | The models the engine is checked against, with the number each one measured on the Strix Halo box at the current pins. One row is one model file on one route; the standard each… |
| Lemonade | A Strix Halo machine gets its GPU and NPU paths inside Lemonade from the 1bit engine, which runs there as Lemonade's onebit backend; the onebit recipe passes Lemonade's own LLM… |
| Releases | 1bit OS is the Puppy Linux appliance (ResolutePup64, woof-CE on the Ubuntu 26.04 base, with Lemonade and the engine embedded, our MOK-signed kernel, headless by default and a… |
| How we measure | tools/bench.py measures 1bit serve configurations against each other. It is how the serve numbers in these docs are taken, and how a recipe earns its place. |
| Recipes | A setting that makes one model faster on one route can make another slower. For example, on the removed lean ROCm route 1024-token micro-batches sped up one MoE model's prompts by… |
| Hardware gate | ci.yml runs on a GitHub runner with no GPU, so it cannot see an HRX kernel or pin that computes the wrong thing. Eight HRX correctness bugs were found by hand within ten days:… |
Devices
| Page | What it covers |
|---|---|
| HRX | HRX runs every graph node of every model and mode we test on the Strix Halo GPU: 43 of 43 configurations, with 0 nodes on the CPU. That is the 2026-10-20 census (#476, #480,… |
| NPU engine | The NPU engine runs models on the AMD NPU (the XDNA 2 NPU of the Ryzen AI MAX+ 395) on Linux, on our own runtime and kernels. It is in progress: correctness first, speed next. Its… |
| Hybrid | Hybrid serves a .gguf with the prompt processed on one device and the tokens generated on another: a prefill provider computes the prompt's KV cache, and the HRX llama-server… |
| ONNX Runtime | -DONEBIT_ORT=ON links the distribution's ONNX Runtime into 1bit and builds two things on top of it: the 1bit ort-info probe, which reports the pin and the execution providers this… |
| ONNX pack | Design note for deliverable 4. It answers one question: if a model arrives as a plain ONNX graph, what does it take to run it on this NPU without executing a single AMD custom… |
Components
| Page | What it covers |
|---|---|
| Laya router | Laya picks where each request runs: it sorts each conversation into code, prose, short answer or long document (95.5% right), and a measured policy picks the device. The test set… |
| Project Manager | ZAYA1-8B is made for reasoning and calls tools reliably. 1bit serve --pm makes the served model the Project Manager of the engine: it keeps every conversation, answers what it can… |
| Model registry | Step 5 of the port: which Hugging Face models this engine can run, kept current every day. Two numbers are reported, and they are never added together: |
| Architecture gaps | The registry maps an HF architecture only when a pinned backend's own code accepts it. It is generated from the pinned sources; nothing in it is typed in by hand. So "support"… |
| MoE streaming | The goal: run mixture-of-experts models larger than Strix Halo's 128 GB. Only the experts a token routes to are needed. The hot ones stay in RAM, and the rest come from the NVMe… |
| Tokenizers | The engine reads any model's tokenizer.json through Hugging Face tokenizers, the Rust library behind the tokenizers npm and PyPI packages, at the same release as those packages. |
Project
| Page | What it covers |
|---|---|
| How the engine is built | The 1bit engine runs inside AMD's Lemonade Server as its onebit backend, pinned to Lemonade's weekly release (third_party/lemonade, moved by the bump-lemonade workflow;… |
| Linux kernel | The Strix Halo box runs a kernel we build, not the distribution's: everything the engine and the NPU need is pinned by hand, so the next kernel is a step list, not an… |
| NPU leaderboard | One row is one measured run of this engine. A record is only ranked if it names its instrument, its repeat count and its timestamp; see bench/npu-leaderboard/README.md for the… |
| Porting map | This repository is the working 1bit-MONSTER engine, ported without its history. Each step below is one PR (or a short series) that builds and runs on Strix Halo before the next… |
1bit OS
| Page | What it covers |
|---|---|
| What it is | Puppy is v3, and Puppy is 1bit OS. It is ResolutePup64, built with woof-CE on the Ubuntu 26.04 "resolute" base, with our kernel (MOK-signed; 7.4.0-rc1-1bit since cycle 21,… |
| Install 1bit OS | 1bit OS is the Puppy Linux appliance in os/README.md: Lemonade Server with this engine inside, our MOK-signed kernel, headless by default. It runs from the stick it was written… |
| The engine on any Linux | The engine does not need a patched Lemonade. A stock Lemonade discovers it through the llamacpp-hrx recipe it already has: that recipe resolves the name llama-server inside the… |
| Connect a client | 1bit OS runs Lemonade Server, and Lemonade is what a client talks to. On the appliance it listens on 0.0.0.0:13305 — every interface, not just loopback — with an OpenAI-compatible… |
| Why Puppy: the evaluation | The owner's direction (2026-11-01): build 1bit OS v3 on Puppy Linux instead of Ubuntu Server — Puppy for both roles, the live/try stick and the installed system (frugal install on… |
More
| Page | What it covers |
|---|---|
| AMD's drafter families: what they are, what they need, what they measure | AMD publishes two families of speculative-decoding artifacts on Hugging Face, and they are not the same kind of thing. This page is the route decision for both, with the numbers… |
| AXB35-02 (Bosgame BeyondMax): the rear USB-A and Type-C ports | Status: fix merged, mechanism verified on the board, functional plug test still outstanding. engine#823. |
| Self-hosted CI runners on the Strix Halo box | Two runners serve this repository, both on Strix Halo, both installed by scripts/install-ci-runner.sh: |
| Determinism and cross-route log-prob agreement | Two routes serving the same weights are not the same computation. This page states exactly what 1bit serve can promise about repetition and about agreement between devices today,… |
| HeteroMosaic's split prefill, checked on our stack | HeteroMosaic (UIUC + AMD, MICRO 2026, arXiv 2607.12839, UIUC-ChenLab/heteroMosaic, Apache-2.0) is the closest published work to a direction we already have: the XDNA 2 NPU and the… |
| Laya first: the first request on a cold Lemonade | Owner (2026-11-03): Lemonade starts with no model loaded, and "when we send the first request it needs to be our laya router making that decision and picking our specific models… |
| Several models behind one port: 1bit serve --laya-models | --laya picks a device for the one model a serve runs. --laya-models FILE picks the model and its device for every request: a model table names its own classes, the Laya question… |
| Adapters (LoRA): where they live, how to change one without reloading it, and the two traps | The engine has no adapter code of its own. An adapter is a llama.cpp LoRA GGUF, handled entirely inside third_party/llama.cpp (src/llama-adapter.cpp), and the surface is… |
| Model pins: the revision of every upstream model we depend on | The census counts what the engine can run. It does not record which revision of which upstream model we actually run, ship, or were built from — so a vendor can re-publish the… |
| The NPU route's chat template: render the container's own (engine #963 option 2) — implemented | app/forward_serve.cpp renders ChatML for every container it serves, and since #987 it refuses the chat route (400) for a container whose tokenizer has no <|im_start|>/<|im_end|>… |
| NPU plan layer, P4: extracting NpuTarget + NpuModel (engine#803) | Part of engine#799. This is the design for P4: split npu/forward/src/q4nx_forward.cpp along the line between the model's math (portable) and the NPU's runtime + kernels (per… |
| The NPU plan layer (plan.json) | plan.json is the model's stream plan: the decisions the NPU forward used to make by looking at which files happened to exist in --kernels. It sits beside model.q4nx in the model… |
| A drafter on the OGA-pack route: measured as far as the route allows (engine#1371) | The table below is from 2026-11-08, when no OGA pack reached a logit. Since then the 1.9 reader and the b8 kernel families landed, and Qwen2.5-7B-Instruct_rai_1.9.0_npu imports… |
| Before you ask for a merge | Six commands, all host-side, none needing a device or a build. Every one of them answers a question that mergeable=true, a green CI tick and a careful read of your own diff do not. |
| Publishing a model's chat template | A GGUF's tokenizer.chat_template is a Jinja2 program that the llama.cpp-family servers run on every request. It can rewrite the system context, add an instruction, gate behaviour… |
| Zero-copy on the serving paths: what moves, what it costs, what stays | Strix Halo has one DRAM. The CPU, the iGPU (HRX over KFD/GTT) and the XDNA 2 NPU (its buffer objects are host memory mapped through the IOMMU) all read and write it, so a memcpy… |
| The Zyphra family: Zamba2, Zamba v1 and BlackMamba | The three Zyphra architectures the engine carries — zamba2, zamba (v1, with its MoE) and blackmamba (Mamba-1 with alternating MoE) — what their files weigh at the current pin,… |
| Zyphra | Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at |