Porting map#
This repository is the working 1bit-MONSTER engine, ported without its history. Each step below is one PR (or a short series) that builds and runs on Strix Halo before the next one starts.
| Step | Component | Source in 1bit-MONSTER | Done when |
|---|---|---|---|
| 1 | The engine runs inside Lemonade. Lemonade runs 1bit serve as a backend: one model per process behind an OpenAI-compatible API |
tools/unified_server.cpp |
landed (1bit serve): NPU and HRX pass serve_e2e on Strix Halo (Vulkan and ZINC did too, before RFC #213 stage 3 and the core strip removed them); smoke_serve runs in CI. The engine first embedded Lemonade (v11.9.0 plus local recipes); after a review1 it is embedded into Lemonade instead: the vendored copy is removed, and the Lemonade recipe that runs 1bit serve (onebit) lives in our fork 1bit-MONSTER/lemonade (Lemonade) |
| 2 | HRX. llama.cpp with GGML_HRX=ON on AMD's tested pair (it was built with GGML_VULKAN=ON too until RFC #213 stage 3, which removed the engine's Vulkan and ROCm builds) |
1bit-MONSTER/llama.cpp 1bit/hrx-vulkan-patched (AMD's hrx-graph-develop-v2 plus our commits) + 1bit-MONSTER/hrx-system 1bit/main (ROCm/hrx-system at AMD's pin plus our commits), pinned from ROCm/ggml-staging-automation |
landed (HRX): 1bit serve runs GGUF checkpoints on HRX0 (and the CPU from the same build); kept current by bump-hrx.yml. The decode-split flash-attention race (#123, #140) is fixed (llama.cpp 00adc2b, #176) and the kernel is on by default again (#178). Since 2026-10-18 pinned to AMD's e3e6a976 hrx-system line (1bit/e3e6a976-pin-6: our four fixes replayed plus our live-out fix for ROCm/hrx-system#1535) with llama.cpp 8d9ce2c1a (#441; since 2026-10-19 hrx-system 82804197 and llama.cpp 00f2a68de, #458, #461). Measured at the #441 move: test-backend-ops 1141/1141, long-prompt needles 20/20 + 5/5 + 5/5, KLD 0 and identical output on Qwen3-0.6B and ZAYA1-8B, pp512 +4.4% / +7.8%, tg128 +2.5% / +2.8%. The hold in config/hrx-system-hold.config now names that pin, so later AMD hrx-system moves stay manual |
| 3 | NPU engine. The NPU engine, with its kernels built from source | 1bit-MONSTER's NPU engine | in progress (NPU engine): the model reader, Q4NX, the tokenizer and serving landed (#8, #9), and Qwen3-0.6B was served on the NPU through 1bit serve. The XDNA driver is pinned upstream (#12). GGUF on the NPU (#179, #181, #209): 1bit serve -m <gguf> --device npu repacks a GGUF to Q4NX and runs six architectures, correct but not yet fast. The layer kernel and lm head built from source decode Qwen3-0.6B, 1.7B and 4B at cosine similarity 0.9999 or better against an fp64 reference at every tested position, and greedy decode matches the reference exactly; they are closed source and not merged yet |
| 3x | 35B MoE on the NPU (experimental, closed source). Qwen3.6-35B-A3B on the NPU | a closed-source add-on | landed as an add-on (NPU engine): built into 1bit with -DONEBIT_NPU_PRIVATE; parity against the fp64 reference passes (3 positions, argmax 846 / 198 / 3710), and 1bit serve --device npu answers "The capital of France is Paris." Without the add-on, serve says the route is not part of the build |
| + | Linux kernel. The kernel that provides amdxdna and amdgpu, pinned to upstream |
torvalds/linux release tags; config from the Strix Halo kernel of 2026-09-23 |
moved to its own repository (2026-10-04): upstream tags built into Debian packages — v7.4-rc1 today, v7.3-rc4 before it, with amdxdna in-tree — and that repository keeps it current. Installing it on Strix Halo is a separate, deliberate step; 1bit OS (os/) takes a built kernel - the Puppy appliance as woof-CE's kernel package (os/v3/puppy/mk-huge-kernel.sh), the os/mini CI image as KVER= / VMLINUZ= |
| 4 | Laya router. A non-autoregressive scorer that picks where each request runs | src/laya_scorer.cpp, include/laya_scorer.h on backup/laya-and-results-2026-09-22; model at ~/models/laya |
landed, opt-in (Laya router): source NandhaKishorM/laya + the three HF checkpoints pinned, hash-verified fetch, bump-laya.yml (#18; source ca5045a3f1fa and checkpoint revision 7b928d828b7b since #439); the C++ scorer matches the Python reference on the root and typed-decisions/ checkpoints (max logit diff 8.6e-6, same argmax) and 1bit serve --device auto --laya-model <dir> routes each request (#90). Load 2.8 s once, then 0.38 s per decision (was 8.75 s, #91). Since RFC #186 (#187) it classifies each conversation (95.5% on 200 labelled requests) and config/route-policy.json picks the device; the scorer runs on HRX at 15-16 ms a decision (#206, #208). Next: multilingual/ (mmBERT-base) |
| 5 | Every HF model, kept current. The architecture registry and the daily HF census that finds new architectures and ranks the unmapped ones by model count (1bit-MONSTER's registry mapped 2,030 HF arch strings to its own kernels; here the backends' pinned code decides) | src/model_registry*.cpp, Testing/census_*.py and .json, .github/workflows/census-{watch,sweep,autopr}.yml |
landed (Model registry): registry/architectures.json generated from the pinned HRX llama.cpp fork (265 HF architectures at first; ZINC's column left with the core strip); census.yml sweeps HF daily. First sweep: 415,414 text-generation models, mapped 94.88%, checked 64.33% of those with an architecture (reported separately); the 2026-10-17 sweep counts 425,483 text-generation models, 338,650 with an architecture (2,822 architectures), 94.62% mapped and 63.9% checked (#438). All seven checked GGUF architectures pass on HRX since the llama.cpp pin 96f6b89 (#95) |
| 5z | ZAYA1 (Zyphra) on the GPU. An architecture upstream llama.cpp lacks, in our llama.cpp; 1bit serve --device hrx (and auto) runs it |
1bit-MONSTER/llama.cpp 1bit/hrx2-archive (src/models/zaya.cpp) and 1bit-MONSTER's tools/convert_zaya_safetensors_to_gguf.py |
landed (vulkan.md at 0baf286): ported to the pinned fork with a converter; matches transformers in FP32 at 95/96 teacher-forced positions; Q4_K_M decoded at 93 tok/s on the removed Vulkan build, 61.5 on the removed ROCm build and 25.5 on HRX at first; on HRX it now decodes at about 90 tok/s (94.0 tg128, llama.cpp #42, HRX) and passes serve_e2e |
| 6 | ZINC (NVIDIA and more). Upstream zolotukhin/zinc, a Zig GGUF engine |
not in 1bit-MONSTER; was pinned from upstream main |
removed in the core strip (2026-10-04: the engine is HRX + HIP kernels + the NPU; Lemonade's own backends serve other GPUs). It had passed serve_e2e on Strix Halo (Vulkan) and answered on an RTX 5090 (CUDA); the record is zinc.md at 7eaa6e6 |
How the pieces fit#
Lemonade (the host users talk to)
└─ runs, as its onebit backend ─> 1bit serve -m <model> --port <p> one OpenAI-compatible API
├─ NPU model directory ──> the NPU engine, in process
├─ .gguf --device hrx|cpu ──> this build's llama.cpp (ggml-hrx with HIP kernels + the CPU; no Vulkan, ROCm, Metal or CUDA)
├─ Laya ──> picks the device per request (step 4, opt-in)
└─ model registry ──> every HF architecture, refreshed by the daily census (step 5)
Known facts carried over#
- Vulkan is not inside HRX. Until RFC #213 stage 3, one build compiled both ggml backends; it now compiles ggml-hrx only.
- (Previous build, ggml-hrx2.) On zaya1-8b,
Vulkan0decoded at 74.8 tok/s andHRX20at 18.4, because HRX2 fell back to the CPU for 2,088 ops. - Q4NX loaded only on
HRX20, the ggml-hrx2 backend AMD dropped in June. Since the move to AMD's live ggml-hrx (docs/hrx.md) it is not served; the Q4NX kernels are kept on1bit-MONSTER/llama.cpp1bit/hrx2-archivefor a port.
- (Previous build, ggml-hrx2.) On zaya1-8b,