HRX + Vulkan#

One llama.cpp build with two GPU backends, ggml-hrx (AMD's HRX runtime) and ggml-vulkan. Its single llama-server exposes both devices on Strix Halo:

HRX0:    AMD Radeon 8060S Graphics (Node 1) (gfx1151)
Vulkan0: AMD Radeon 8060S Graphics (RADV STRIX_HALO)

1bit serve runs GGUF models on either device with it (1bit serve):

Pinned sources, kept current#

• Where the pair comes from. The two pins are the pair AMD's integration repo, ROCm/ggml-staging-automation, builds and tests together.
• How it stays current. .github/workflows/bump-hrx.yml runs daily. When AMD moves its pair, it:
  • syncs the fork (master to ggml-org, 1bit/hrx-vulkan to AMD's pin);
  • rebases our commits onto AMD's pin and moves 1bit/hrx-vulkan-patched there, after tagging the previous tip patched-<sha> so every commit the engine has pinned stays reachable (a rebase conflict fails the run: rebase by hand);
  • opens a PR here moving both submodules, listing the rebased commits.
• The token it needs. The secret HRX_BUMP_TOKEN is a fine-grained token with Contents read/write on 1bit-MONSTER/llama.cpp and 1bit-MONSTER/engine, and Pull requests read/write on 1bit-MONSTER/engine.
• Validation. CI builds that PR without HRX, so run the checks below on Strix Halo before merging.

Prefill on HRX, decode on Vulkan#

HRX prefills faster than Vulkan and Vulkan decodes faster, on the same GPU. 1bit serve --device vulkan --prefill-device hrx uses both: the HRX build's llama-server decodes on Vulkan0, and a second copy of the model on HRX0 runs the prompt prefix. The two contexts share one KV cache with no copy: HRX owns it in its device memory and exports it as a dma-buf (HSA), Vulkan maps the same memory, and only the KV cell metadata (positions, sequences) moves between them.

• What runs on HRX: the prompt prefix in whole ubatches (HRX prefill halves on a partial one), from --prefill-min-tokens tokens (default 1024; below that the split does not pay). The rest, and all decoding, run on Vulkan.
• Needs: flash attention on both sides (serve passes -fa on), no LoRA, no multimodal. The weights are loaded twice (once per device).
• Layout: the shared region is cut into chunks of at most 1 GiB, each its own dma-buf: Vulkan reads garbage from one buffer past about 4 GiB (a 40960-token Qwen3-0.6B cache), and both sides must compute the same layout.

Measured on Strix Halo, llama-server on Vulkan0, prefill / whole request:

Decode on the shared KV is within 3% of Vulkan with its own KV. Against Vulkan alone, teacher-forced over 64 tokens, the split's mean KL is 0.0002-0.0006 nats (max 0.007) and the top token agrees on 64-65 of 65 positions; Q4_K_M against BF16 is 0.063.

Our patches#

1bit/hrx-vulkan is AMD's commit unchanged; 1bit/hrx-vulkan-patched adds:

• Zero-copy KV sharing (the section above): dma-buf export in ggml-hrx (ggml-hrx-dmabuf.cpp), dma-buf import in ggml-vulkan (ggml-vulkan-dmabuf.inc), llama_kv_share_next / llama_kv_share_from / llama_kv_cells_copy in llama (llama-kv-share.cpp), and ONEBIT_PREFILL_DEVICE in llama-server (server-prefill.cpp).
• IQ3_XXS matmul on HRX. A matrix-vector kernel (kernels/hrx/mul_mat_vec_iq3xxs_f32.loom) and its matcher (common/dispatch-mul-mat-iq3-xxs.cpp), ported from the feat/hrx-port-ae91949 work.
• Honest op claims. ggml-hrx used to claim every op of a declared type, so ggml's scheduler handed it nodes it could not dispatch and the graph failed (unsupported HRX node) with no CPU fallback. It now claims a node only when its dispatcher can execute it, except nodes already in HRX memory (KV cache views), which cannot move. Leaf (NONE) nodes count as covered. AMD's IQ4_NL and IQ4_XS matmul kernels, and GET_ROWS for IQ4_XS and batched IQ3_S, give wrong values, so those nodes are left to the CPU.

Measured on Strix Halo (Qwen3-0.6B, perplexity over 8 x 512 wikitext tokens):

test-backend-ops -b HRX0: 791 OK, 0 failed (on AMD's commit, about 1150 fail). Correct is not fast: the UD files run their IQ4_XS and sub-4-bit layers on the CPU (UD-Q4_K_XL 6303 / 246 tok/s, UD-Q2_K_XL 974 / 76), so they belong on Vulkan0.

1bit-MONSTER/llama.cpp also holds 1bit/hrx2-archive, the previous build (AMD's abandoned ggml-hrx2 plus our Q4NX kernels and the zaya architecture). It is kept for a later port of that work to ggml-hrx.

The Vulkan route has its own upstream llama.cpp pin (latest release), so new architectures do not wait for AMD's pair: see Vulkan (upstream). This build still has a Vulkan backend, which --device vulkan uses when ONEBIT_VULKAN is off.

Build#

git submodule update --init --depth 1 third_party/hrx-system third_party/llama.cpp
cmake -B build -G Ninja -DONEBIT_HRX=ON          # needs TheRock at /opt/rocm-therock (ONEBIT_HRX_TOOLCHAIN)
cmake --build build --target onebit

• One external project. llama.cpp's ggml-hrx builds hrx-system itself (HRX_SOURCE_DIR) with TheRock's amdclang.
• IREE_ROCM_PATH is not passed. It would switch hrx-system to "package" mode, which needs TheRock's aqlprofile-sdk headers, and our /opt/rocm-therock does not ship them. Left unset, hrx-system fetches its pinned HSA/AQL headers.
• The HSA runtime. HRX dlopens it at run time, and the distro's libhsa-runtime64 rejects the HSA_AMD_AGENT_INFO_PM4_EMULATION probe on gfx1151, so HRX then registers no device. CMake finds TheRock's libhsa-runtime64.so.1 (ONEBIT_HRX_LIBHSA), and 1bit serve passes it to llama-server as IREE_HAL_AMDGPU_LIBHSA_PATH, unless it is already set. Without a build path it searches /opt/rocm-therock. To run llama-server or llama-bench by hand, export that variable yourself.

Verified (2026-09-23, llama.cpp f1a0aca, hrx-system 51b1739)#

ctest passed, re-run after the first daily bump (#13). At the time the check was tests/hrx_lemonade_e2e.sh, through the engine's former embedded Lemonade: unsloth/Qwen3-0.6B-GGUF:Q4_0 answered "Paris" on Vulkan0 and on HRX0. The same build now passes tests/serve_e2e.sh on both devices through 1bit serve.

llama-bench, Qwen3-0.6B, pp512 / tg128 tok/s, against the previous build (April llama.cpp with ggml-hrx2 on HRX20):

HRX prefill is now faster than Vulkan's.

Not yet#

• Fast sub-4-bit and IQ4 kernels on HRX. With our patches the UD files are correct on HRX0 but slow, since IQ4_XS, IQ2 and IQ3_S matmuls fall back to the CPU. AMD's IQ4_XS / IQ4_NL kernels need fixing upstream.
• Q4NX on HRX. Our Q4NX kernels lived in ggml-hrx2 and are not in ggml-hrx. They are kept on 1bit/hrx2-archive until they are ported.
• First-request cost. Lemonade's telemetry for the first short HRX chat shows 18 tok/s against llama-bench's 303. It is probably first-use kernel compilation, not steady state, but this is not yet separated.
• CI. CI has no AMD GPU and no TheRock, so ONEBIT_HRX is exercised on Strix Halo only.

View this page's source on GitHub ↗