The lean option: ROCmFP4 and ROCmI4#

1bit serve --lean trades accuracy for speed. It runs models in the AMD-focused formats of ROCmFPX (MIT), a llama.cpp fork that upstream llama.cpp cannot read, so the lean route has its own tree: third_party/llama.cpp-rocmfpx, pinned to a commit measured on Strix Halo.

The default route stays Unsloth's UD-Q4_K_XL on upstream llama.cpp (Vulkan (upstream)): it stays far closer to the full model. Use lean when speed matters more than that.

Measured (Strix Halo, Qwen3.8-27B, 2026-09-24)#

Every file quantized from the same BF16; KL divergence and top-token agreement against BF16 over wikitext-2 (40 x 512 tokens); llama-bench pp512 / tg128 in tok/s; MTP chat is decode speed with the MTP head on three prompts (code / prose / short).

• ROCmFP4 decodes 19% faster than UD-Q4_K_XL on Vulkan (14-24% with MTP), from a 16% smaller file, at about 7x the KL divergence.
• ROCmI4 belongs on ROCm. Vulkan has no kernel for it (6 / 4.1). On ROCm, the gfx1151 W4A4 path lifts prompt processing 17% (465 against 397) and leaves decode as it is.
• The ROCmFPX files were quantized without an importance matrix; UD-Q4_K_XL was made with one. An imatrix would narrow the accuracy gap somewhat.

Round 2: with Unsloth's imatrix, and what to use (2026-09-24)#

Round 1's ROCmFPX files had no importance matrix. Rebuilt with Unsloth's own (imatrix_unsloth.gguf, the one UD-Q4_K_XL was made with), next to Unsloth's smaller files:

• For a smaller, faster file, use Unsloth's UD-Q3_K_XL on the default route (no --lean needed): it is smaller, faster and closer to the model than either 4-bit ROCmFPX format, even with the imatrix.
• --lean keeps two jobs: ROCmI4 for the fastest prompt processing (455 tok/s with W4A4 on ROCm), and ROCmFP2 when memory is the limit (8.6 GiB, 21 tok/s, at a large accuracy cost: 75% top-token agreement).
• MTP beats every format change: --mtp on the default file gives 2.4-2.9x (docs/serve.md), more than any quant here.

Two backends at once#

Vulkan0, ROCm0 and HRX0 are one GPU. Decode on both at the same time (UD-Q4_K_XL, tg256): Vulkan alone 12.2 tok/s, ROCm alone 11.9; Vulkan + ROCm together 7.5 + 6.8 = 14.3 (+18% total); two Vulkan processes 6.3 + 6.3 = 12.7 (+4%). A single stream uses about 200 of the bus's roughly 256 GB/s, and two different drivers fill the gap better than two copies of one. Each stream slows down, so it pays for serving several requests at once (one per backend), not for one chat.

Build#

git submodule update --init --depth 1 third_party/llama.cpp-rocmfpx
cmake -B build -G Ninja -DONEBIT_LEAN=ON                        # Vulkan: ROCmFP4
cmake -B build -G Ninja -DONEBIT_LEAN=ON -DONEBIT_LEAN_ROCM=ON  # also ROCm: ROCmI4
cmake --build build

The ROCm build uses TheRock's amdclang++ from ONEBIT_LEAN_ROCM_TOOLCHAIN (default /opt/rocm-therock). If it fails on __ocml_* in the distribution's HIP headers, point that at a TheRock tree whose compiler builds HIP on its own.

Verified (Strix Halo, ROCmFPX fb08d7c, 2026-09-24)#

The ROCm build there used -DONEBIT_LEAN_ROCM_TOOLCHAIN=$HOME/therock100; the default /opt/rocm-therock fails on the __ocml_* headers described above.

Making a lean file#

The lean build carries ROCmFPX's llama-quantize. Start from the BF16 GGUF:

build/lean/llama/bin/llama-quantize model-BF16.gguf model-ROCMFP4.gguf Q4_0_ROCMFP4_STRIX_LEAN
build/lean/llama/bin/llama-quantize model-BF16.gguf model-ROCMI4.gguf  Q4_0_ROCMI4

--imatrix takes an importance matrix, as in upstream. ROCmFPX has more formats (2, 3, 6 and 8 bits); only these two are measured here.

Test#

cmake -B build -DONEBIT_LEAN=ON -DONEBIT_SERVE_TEST_LEAN_GGUF=$HOME/models/model-ROCMFP4.gguf
ctest --test-dir build -R serve_e2e_lean

bump-rocmfpx.yml opens a PR moving the pin to ROCmFPX's latest main once a week; rerun the table above on Strix Halo before merging one.

View this page's source on GitHub ↗