GitHub ↗
← all posts
milestonesnpuhrxvulkanzinc

The first week of 1bit engine

The clean repository started on 2026-09-22. In its first three days, the engine came up on every device it targets, inside Lemonade, with each step verified on real hardware before it merged. These are the milestones, with the numbers and where they come from.

Since 21 October 2026: Vulkan and ZINC are no longer routes of the engine (engine #306); it is HRX with our HIP kernels plus the NPU engine, inside Lemonade. See Every node on the GPU.

Update (2026-10-30): the NPU route now decodes Qwen3-0.6B at 88.6 tok/s in the release gate, and the Linux kernel pin moved to v7.3-rc4 with the IOMMU PerfOpt backports. Current numbers: Models measured; the routes: NPU engine, HRX.

The NPU#

The XDNA 2 NPU runs Qwen3-0.6B end to end through the engine, served inside Lemonade (#8, #9). The XDNA driver is pinned to upstream (#12). The NPU status page tracks where it stands.

HRX and Vulkan in one build, then both at once#

The Radeon iGPU has two devices in one llama.cpp build: HRX0 on AMD's HRX runtime and Vulkan0. HRX follows AMD's live ggml-hrx, pinned to the pair AMD tests and bumped daily (#11); Vulkan has its own pin on upstream llama.cpp's latest release (#28).

  • HRX went from 3-5x slower than Vulkan to close behind it. Its prefill is now 1.3-1.7x Vulkan's, and its decode within 10% on 4-bit files.
  • Our patches make HRX correct on more models (#29): an IQ3_XXS kernel, and op claims that only take the nodes HRX can actually run. test-backend-ops on HRX0 went from about 1150 failures to 791 passing, 0 failing, and Unsloth's sub-4-bit files, which failed on HRX, now come within 1% of Vulkan's perplexity.
  • Prefill on HRX, decode on Vulkan, with zero copies (#30): a whole 8192-token request on Qwen2.5-7B finishes 26% sooner. How it works.

Beyond AMD#

  • NVIDIA, through ZINC's CUDA backend (#14): Qwen3.5-9B at 167-173 tok/s on an RTX 5090, and 295 tok/s through ZINC's Vulkan backend on Strix Halo.
  • Apple Silicon, through Lemonade's mlx recipe and lemon-mlx-engine, verified on an M4 (#10).

Underneath#

  • Tokenizers: Hugging Face tokenizers v0.23.2 behind our C ABI reads any model's tokenizer.json, byte-exact on 18 models (#19).
  • Quantization: on the GPU, Unsloth's Dynamic UD-Q4_K_XL beats Q4_K_M: 17% lower KL divergence for 2% more size. On the NPU, a ternary model converts to 4-bit exactly, and XDNA 2 multiplies int8 by int4 natively (wiki).
  • The Linux kernel is pinned to upstream v7.3-rc4 with amdxdna in-tree (#17), and the Laya router's source and checkpoints are pinned and hash-verified (#18).

Next#

Step 4 is the Laya router, which decides where each request runs. After it: every Hugging Face architecture, kept current by a daily census. The porting map tracks it, and the wiki keeps every measured number with how it was taken.