Zero-copy on the serving paths: what moves, what it costs, what stays#

Strix Halo has one DRAM. The CPU, the iGPU (HRX over KFD/GTT) and the XDNA 2 NPU (its buffer objects are host memory mapped through the IOMMU) all read and write it, so a memcpy between "host", "GPU" and "NPU" buffers on this box is pure overhead — the owner's framing is "we need to be using it, the whole point of having it" (engine#698). This page is the map and the audit: which copies are left on the serving paths, what each one costs per token, and why it is still there.

Every number below was measured on this box, and each names what measured it. Nothing here is an estimate unless it says so. The audit is the deliverable of engine#698's target 4; targets 1-3 are the code work and their status is at the top.

Status#

Target 1 — the hybrid handoff moves no bytes#

The file route writes the provider's sequence state to disk and the decoder reads it back through the page cache, for a handoff that never leaves the box. ONEBIT_KV_SHARE=1 replaces it with dma-buf fds: the decoder exports its KV region, serve passes the descriptors to the provider as inherited fds, the provider prefills into that region, and the decoder adopts the cells with no data movement (llama_kv_cells_adopt). The provider contract, the flag semantics and the fallbacks are in Hybrid; what this page adds is the price.

Measured in the shape that ships (2026-11-06, build at pin d70b67bf0): a separate provider process, the region exported over SCM_RIGHTS, an AddressSanitizer-instrumented provider mapping it and prefilling into it, the decoder adopting the cells. 0.122 / 1.988 / 5.558 ms to adopt 307 / 4 105 / 16 399 cells, with n_read 0 and prompt_n 1 on every arm — the decoder decoded from the adopted cells instead of re-processing the prompt — and 0 ASan reports in the provider. So the handoff is cells bookkeeping and it does grow with the prompt, just not with prompt bytes: 0.12 ms at 307 cells, 5.6 ms at 16 399. The shared figures are on a 4B because that is the model the ASan harness drives (a 30B under the sanitizer prefills for hours); the property being measured — cells, not bytes — does not depend on the model, and the file column beside it is the 8.2-242.9 ms that same handoff costs at those prompt lengths on 30B-A3B.

The provider's own side of that handoff is 40.6 / 78.7 / 150.4 ms to map and first-touch 648 MiB / 1.1 GiB / 2.8 GiB of region — a page fault per page it writes, not a copy, and inflated by the sanitizer; an in-process run of the same adoption step measured 0.048-0.258 ms at 256-2 048 cells. The gate for this target is text identity, not a tolerance: greedy text identical to --device hrx on every arm.

Two properties that make it usable rather than a demo: a decoder that refuses to export now falls back to the state-file route for that request (engine#1203) instead of losing the handoff, and a decoder that cannot export (any backend but ggml-hrx: the CPU, for instance) starts normally and warns once instead of dying at load (fork#212). ONEBIT_KV_SHARE is read as "set and non-empty" on both sides: =0 still enables the route, an unset variable is the opt-out.

What is not proven: the NPU is not yet the provider that writes that region — the provider in the gate above prefills with the CPU. The piece that was missing under it, a device-side K/V writer, now exists in the private tree (engine#753): the NPU writes a prefill's K/V rows straight into the region, both halves of the row landing in the decoder's own tensors at the decoder's offsets and strides, and it is verified byte-for-byte against a CPU reference, including the bytes it does not write (a writer that runs long must fail there). What is left is the provider binary around it, which is the same handoff contract as the CPU provider's.

Two facts from that work belong here because they bound what a device provider can do:

• the region is the decoder's cache, so a provider writes two destinations per layer, not one: cache_k_l<L> and cache_v_l<L> are separate tensors and a token's K|V row is contiguous in the compute but not in the cache (row stride n_embd_k_gqa, K and V at different tensor offsets);
• it must write through its own mapping of the dma-buf. Asking the driver for a mapping of the import is the route that makes every later submission fail (engine#812); the pages are the same either way.

Target 2 — the NPU runtime's copies, measured#

Per launch the runtime quantises one row into the activation BO, flushes it, waits, invalidates the result BO and dequantises row 0 back to host floats. What that cost before this target was worked is what the 2026-10-31 measurements below were taken on — they are the starting point, not the current state:

Each of those was gated byte-identical at token, logprob and head level, on two containers and two rounds, and the narrowed ranges were re-proven on a later driver (6/6 arms). So this target is closed, and the honest residue is not a copy: when the pad tile was measured (2026-10-31) the pack registered 0 kernel roles at M=16, so a decode launch padded one row to M=128 — 8x the work. The M=16 twins exist now (7 ELFs in the 27B pack, 18 in the pack the fleet's serve runs, which is launched with --kernels …/kernels-m16), so what is left there is which tile a decode step selects, which is the NPU runtime's business and not a memory fix.

Two facts from measuring the readback, which are why the sweeps were narrowed rather than deleted:

• bC->sync(XCL_BO_SYNC_BO_FROM_DEVICE) is a real cache sweep (_mm_mfence; clflush per 64 B, then _mm_mfence — disassembled from the shipping runtime), not a copy. It is already paid on every launch.
• A dirty host cache line over the result BO clobbers the NPU's output: poisoning the row before the launch and not writing it back leaves the sentinel to be evicted over real results, which reads exactly like the dropped-write fault in engine#666. That ordering is also why both sweeps are still there: the arms that skip them (NPU_NO_A_FLUSH=1, NPU_NO_C_INVAL=1) emit garbage, so the answer to "when does the cache model need a sync" is "on every launch, over the rows the host touches" — measured, not assumed.

Nothing here needs the file: this is all in-BO traffic.

Target 3 — the ONNX execution provider's boundary#

The EP reads and writes OrtKernelContext in place inside its own host nodes, and copies at the node boundary (DT_CopyTensors, onnx/ep/onebit_ep.cpp:206): per fused QDQ GEMM node, an M x K f32 input and an M x N f32 output — 20 KB + 0.6 MB per node for a 0.6B decode step at M=1. That was the target: bind inputs and outputs to BO-backed OrtValues and remove the boundary copies. It was measured and the boundary is already empty: a profile of the fused path never prints a copy bytes= line at all (1 983 trace lines), because the group is handed the graph's own host tensors and XDNA2 BOs are host memory — binding would be the same bytes to the same place. What a copy could still buy is the readback leg, 0.044 ms of a 4.32 ms call (0.9 %). The largest pure copy left on this route is the cached graph's own Concat (458 KB/position, 41-58 us/position at 4 threads, ~2.1x linear at 4 096 positions) — that is how present is produced, so it needs an op/kernel to write the cache in place, not a binding change. Both numbers come from the ONNX lane's measurements.

The audit: every copy still on the serving paths#

27B Qwen3.5 hybrid on the int8 NPU path, M=128, unless the row says otherwise. "Bytes/token" counts what is read and written per decoded token; a row marked no memcpy is arithmetic over memory rather than a transfer.

Two conclusions the table forces, written down so they are decisions rather than leftovers:

• Rows 1 and 3 were ~59 ms of a 1 330 ms token on the pre-optimisation build (rows 2 and 4 are inside them, so they do not add). That is why target 2 paid off as range narrowing rather than as an architecture change: #988 and #1004 took ~90 % of the traffic out of rows 2 and 3, which is why those rows read as bytes-per-run above now. What is left that pays is the launch count and the M=128 pad — kernel-set work, not a copy fix.
• Row 6 is the biggest single cost and no memory recipe removes it. A GDN recurrence is host arithmetic over 604 MB per token; only moving the recurrence into a kernel changes the number.

Sharing a buffer between the NPU and the iGPU: what works, and the one rule#

Two directions, both measured on this box (engine#753 and engine#812 carry the detail and the harnesses):

One framing correction, because it costs a reader a day otherwise: "XRT cannot bind a foreign dma-buf" is true of XRT and is not our blocker — this engine's NPU lane does not use XRT (npu/rt.h), and its own submission path imports the fd with PRIME_FD_TO_HANDLE and binds a BO that points at the exporter's pages. HeteroMosaic's split prefill, checked on our stack has the comparison.

The rule that follows, and the reason it is written here rather than left to be rediscovered: on a PASID-mode driver, mapping an imported dma-buf through amdxdna invalidates its population and the submit path refaults it through HMM, which an imported range cannot satisfy. Address the exporter's own mapping. The driver patch that removes the invalidation is prepared and not installed (engine#812, and the patch set in engine#1134); until the owner authorises loading it, the exporter-mapping route is the only one that executes.

That is also a contract question, and tests/npu_import_contract.sh (CTest npu_import_contract) is where it is checked instead of remembered. The pinned libamdf advertises AMDF_EXTERNAL_MEMORY_SUPPORT_FLAG_FOREIGN_API with device read/write on the dma-buf-fd entry of its import memory profile (ordinal 1, libamdf/src/xdna/umd/drm/memory_profile.c), i.e. "import a foreign dma-buf and execute on it". The opaque-fd entry in the same profile is cloned from that one and then has the flag cleared, carrying the direct-host provenance instead -- a different advertisement for the same execution, which is why the acceptance below covers both transports and reports them apart. libamdf's import path takes host access through an amdxdna mmap of the import, which is the poisoning path in the table above, so on the shipping module that capability is advertised and refused. The test reads the advertisement from the pin, measures the behaviour with libamdf's own CTS (GPU export -> NPU import -> NPU executes on the imported range, plus registered caller pages as the control), and keys the expected pairing on the loaded module's srcversion against the fingerprint table in third_party/patches/amdxdna/README.md. Run it with ONEBIT_IMPORT_CONTRACT=advertised for the acceptance form: it is red while a module without 0006/0007 is loaded, and green when one that carries them is (measured 2026-11-06 on EE1F36A2B9BA294CE11AD80, the c26 driver: binding 7/7 pass, the array executes registered caller pages, and 8 of 8 imported-backing arms fail at kernel_queue_create).

The advertisement half is a parse of the pin, so it has a device-free test of its own: CTest npu_import_advert (tests/npu_import_contract.sh --self-test) reads two fixtures — an advertising pin and the same pin with the flag dropped from the import class' dma-buf entry — and requires advertised on the first and not on the second. Both bounds it needs are load-bearing and were both wrong in the first version: the ordinal-0 class advertises the same flag, and the import class clears it on the opaque-fd entry it clones, so a whole-file search reports a narrowed pin as advertising (engine#1283's review).

The NPU BO to dma-buf direction still does not exist: the export PRIME_HANDLE_TO_FD returns ENOENT for the handle the runtime produces, so an NPU-owned buffer cannot yet be handed to the GPU. Nothing on the serving paths needs that direction today.

How to re-measure#

• Handoff: the cross-process harness is t1_handoff_lengths.sh + t1_asan_import.py in the zero-copy probe set (it exports, shares over SCM_RIGHTS, runs an ASan provider and adopts). ONEBIT_KV_SHARE=1 ONEBIT_PREFILL_DEVICE=<dev> 1bit serve logs one line per handoff in the engine's own route — shared KV handoff: N cells, X ms, 0 bytes copied. The file route's price is POST /slots/<id>?action=save then ?action=restore, timed; the sizes in the table above come from fitting the state file's size against prompt length.
• NPU runtime: NPU_PHASE_PROF=1 and NPU_PROFILE_TOKEN=1 print the [ph] and [prof] rows quoted above.
• Devices are shared: GPU work through ~/.local/bin/gpu-hold, NPU work through npu-hold, CPU-heavy builds through thermal-run.sh. Kernel internals stay in the private trees; nothing in this page depends on them.

View this page's source on GitHub ↗