1bit engine

1bit.MONSTER is now 1bit engine

2026-09-24T00:00:00Z

1bit.MONSTER grew fast: an inference engine, a model zoo, benchmarks, a store, a voice assistant and a lot of experiments, all in one repository with all of its history. The engine that came out of it is now its own project, 1bit engine, and this site is its home.

What changed#

A clean repository. The engine lives in 1bit-MONSTER/engine. It holds verified code only, without the development history. Each piece is ported over once it passes its end-to-end test on Strix Halo, and the porting map shows what has moved and what is next.

Apache-2.0. The old repository was GPL-3.0. The engine is Apache-2.0, with a copyright and license notice on every file, enforced in CI, and every third-party project it builds on credited in its NOTICE.

Inside Lemonade, not around it. The old engine vendored its own copy of Lemonade. Now Lemonade is the server you talk to (its catalog, downloads, router and UI), and it runs the engine as one of its backends, through 1bit serve, the same way it runs llama-server.

Upstream, pinned and kept current. The XDNA driver and XRT, AMD's HRX pair, llama.cpp for Vulkan, ZINC, the Linux kernel, Hugging Face tokenizers and the Laya router are each pinned to upstream, and the pins are bumped automatically, so new models and fixes arrive soon after upstream ships them.

What it runs on#

One 1bit serve drives:

the XDNA 2 NPU, on full ELFs generated in C++;
HRX and Vulkan on the Radeon iGPU, one llama.cpp build with both;
ZINC, which also reaches NVIDIA GPUs through CUDA;
MLX on Apple Silicon.

Measured numbers live on the wiki. The first week of 1bit engine lists what the first three days produced.

The old site#

The 1bit.MONSTER site and its posts stay online in the 1bit.MONSTER archive, and old links land on a page that points at their archived copy. The blog page here lists the old posts worth keeping. 1bit JARVIS, the voice assistant, is not part of the engine for now; it will come back later as its own piece.

Prefill on HRX, decode on Vulkan, one KV cache

2026-09-24T00:00:00Z

On Strix Halo's Radeon iGPU, AMD's HRX runtime prefills a prompt faster than Vulkan, and Vulkan decodes faster than HRX. 1bit serve --device vulkan --prefill-device hrx now uses both on the same request, and the key-value cache never gets copied between them.

The idea#

A request has two phases. Prefill reads the whole prompt at once and is compute-bound; decode makes one token at a time and is bound by memory bandwidth. Our one llama.cpp build carries both GPU backends, HRX0 and Vulkan0, so the engine loads the model on each, runs the prompt prefix on HRX, and hands the rest of the request to Vulkan.

The hand-off is the hard part. Prefill's output is the KV cache: every layer's keys and values for every prompt token. Copying it from one backend to the other would cost the time the split saves.

Zero copies#

So both contexts use one KV cache:

HRX allocates it in its own device memory and exports it as a dma-buf (through HSA).
Vulkan imports the same memory with VK_EXT_external_memory_dma_buf and reads it in place, at 110.6 GB/s against 113.2 GB/s for its own allocations.
The only thing that moves between them is the cache's bookkeeping: which cells hold which positions of which sequence.

We tried sharing host memory first. It worked, but it cost Vulkan half its decode speed; device memory through a dma-buf does not.

Both sides compute the cache's layout independently, so they must agree on it exactly: tensors on 4 KiB boundaries, split into chunks of at most 1 GiB (Vulkan reads garbage from a single buffer past about 4 GiB), and a hash of the layout checked on both sides. HRX's prefill is fastest in whole batches, so only whole ubatches of the prompt go to HRX, and short prompts skip the split: below 1024 tokens (--prefill-min-tokens) it does not pay.

Measured#

llama-server on Strix Halo, prefill and whole request:

Model	Prompt	Vulkan alone	HRX prefill + Vulkan
Qwen2.5-7B Q4_K_M	2048	1541 / 4314 ms	1078 / 3890 ms (-10%)
Qwen2.5-7B Q4_K_M	8192	7347 / 10376 ms	4646 / 7668 ms (-26%)
Qwen3-0.6B Q4_K_M	8192	1289 / 2267 ms	1089 / 2032 ms (-10%)

Decoding on the shared cache runs within 3% of Vulkan with its own cache. The output is the same model: against Vulkan alone, teacher-forced over 64 tokens, the mean KL divergence is 0.0002-0.0006 nats and the top token agrees on 64-65 of 65 positions. For scale, Q4_K_M quantization itself is 0.063 against BF16.

Where it lives#

The patches are in our llama.cpp fork, on top of AMD's tested HRX commit, and are rebased onto every new AMD pair automatically: dma-buf export in ggml-hrx, dma-buf import in ggml-vulkan, the cache sharing in llama, and the prefill hand-off in llama-server. The weights are loaded twice, once per device, so it suits machines with memory to spare; Strix Halo's unified memory is one. The details are in the HRX docs.

The first week of 1bit engine

2026-09-24T00:00:00Z

The clean repository started on 2026-09-22. In its first three days, the engine came up on every device it targets, inside Lemonade, with each step verified on real hardware before it merged. These are the milestones, with the numbers and where they come from.

The NPU, on full ELFs generated in C++#

The XDNA 2 NPU now runs without an xclbin and without per-context kernel files. The engine generates each kernel's full ELF (the design plus its control code) in C++ when the model loads, and XRT opens it directly; for each token, the host submits every layer and the lm head as one runlist (#8, #9).

Qwen3-0.6B decodes at 91 tok/s (11.0 ms per token), and 87 tok/s served through Lemonade.
The XDNA driver and its XRT are pinned to upstream and built privately (#12).
The layer kernel itself is being rebuilt from scratch as our own open kernel ("dx", on IRON and Peano). RMSNorm, RoPE and attention pass their tests so far, correctly rounded to bf16.

HRX and Vulkan in one build, then both at once#

The Radeon iGPU has two devices in one llama.cpp build: HRX0 on AMD's HRX runtime and Vulkan0. HRX follows AMD's live ggml-hrx, pinned to the pair AMD tests and bumped daily (#11); Vulkan has its own pin on upstream llama.cpp's latest release (#28).

HRX went from 3-5x slower than Vulkan to close behind it. Its prefill is now 1.3-1.7x Vulkan's, and its decode within 10% on 4-bit files.
Our patches make HRX correct on more models (#29): an IQ3_XXS kernel, and op claims that only take the nodes HRX can actually run. test-backend-ops on HRX0 went from about 1150 failures to 791 passing, 0 failing, and Unsloth's sub-4-bit files, which failed on HRX, now come within 1% of Vulkan's perplexity.
Prefill on HRX, decode on Vulkan, with zero copies (#30): a whole 8192-token request on Qwen2.5-7B finishes 26% sooner. How it works.

Beyond AMD#

NVIDIA, through ZINC's CUDA backend (#14): Qwen3.5-9B at 167-173 tok/s on an RTX 5090, and 295 tok/s through ZINC's Vulkan backend on Strix Halo.
Apple Silicon, through Lemonade's mlx recipe and lemon-mlx-engine, verified on an M4 (#10).

Underneath#

Tokenizers: Hugging Face tokenizers v0.23.2 behind our C ABI reads any model's tokenizer.json, byte-exact on 18 models (#19).
Quantization: on the GPU, Unsloth's Dynamic UD-Q4_K_XL beats Q4_K_M: 17% lower KL divergence for 2% more size. On the NPU, a ternary model converts to 4-bit exactly, and XDNA 2 multiplies int8 by int4 natively (wiki).
The Linux kernel is pinned to upstream v7.3-rc4 with amdxdna in-tree (#17), and the Laya router's source and checkpoints are pinned and hash-verified (#18).

Next#

Step 4 is the Laya router, which decides where each request runs. After it: every Hugging Face architecture, kept current by a daily census. The porting map tracks it, and the wiki keeps every measured number with how it was taken.