The ONNX pack: weights to Q4NX, matmuls to our kernels, no VitisAI#

Design note for deliverable 4. It answers one question: if a model arrives as a plain ONNX graph, what does it take to run it on this NPU without executing a single AMD custom operator?

What a "Ryzen AI pack" is, and why we do not execute one#

The AMD Ryzen AI toolchain ships prebuilt ONNX packs: a graph whose transformer blocks are replaced by nodes in a custom domain (com.amd.*), each one executed by a compiled xclbin through the VitisAI execution provider, with weights pre-quantised into that provider's own layout. It is fast and it is opaque: the pack decides the quantisation, the kernel, the tiling and the schedule, and the operator that runs on the device is a black box the vendor compiles.

We do not execute those operators. Three concrete reasons, in the order they bind:

1. Licence and footprint. The VitisAI EP arrives in the AMD wheel, which the contract forbids us from linking or shipping. A pack that only runs through com.amd.* nodes therefore cannot run in our binary at all.
2. We already own the kernels. npu/ generates and launches the int8 × int8 GEMM that the forward route uses (ONEBIT_NPU lane), with our own quantisation, our own tiling and a layout we can debug with xrt::elf tooling. A vendor pack would replace a measured component with an unmeasured one.
3. The graph must stay a graph. Our plugin EP's value is that ONNX Runtime schedules the graph and our provider takes the matmuls it has a kernel for; everything else stays on the CPU EP. A pack with fused custom nodes removes exactly that composability.

So the pack we design is the same idea, built from parts we control: ONNX graph stays plain, weights become Q4NX on disk, each matmul shape is bound to a kernel we generated, and the EP is the thing that does the binding at run time.

What the graph gives us#

Census of qwen3-0.6b.onnx (from onnx/export_qwen3.py --check, opset 21, ai.onnx domain only, float32, seq dynamic):

254 MatMul nodes, 6 distinct shapes, zero custom domains. The binding is therefore a table lookup on (K, N), which is exactly what ONEBIT_EP_NPU_ELFS="16x1024x2048=<elf>,16x1024x1024=<elf>" already expresses; a shape with no entry returns ORT_NOT_IMPLEMENTED, the EP falls back to the CPU for it, and the session still runs. That is why deliverable 4 can be honest about owning 84 of 254 matmuls rather than claiming the model — 84 is what the two m16 designs the iso-build tree shipped reach (q, and k/v). The design column is not a feature column, and it has grown since that census: with a design for every projection (the five M=32 designs, beside the other xclbins and never in this repository) the float path claims 196 of 254 — all seven projections — and every one of those shapes ran on the array, measured on 2026-11-05 rather than inferred from the table; the numbers and their window are in ONNX Runtime.

The three layers of the pack#

1. Weights: Q4NX on disk, int8 + scales in the kernel's layout. The on-disk format is already Q4NX-family (4-bit blocks with per-block scales, the format npu/q4nx.cpp decodes). At session start we bind a weight tensor by pointer identity and call the runtime's own packB, which quantises each column to int8 with scale = amax/127 and stores the column scales alongside the packed buffer — the same function the forward route calls, so the numerics of a packed weight cannot drift from the weights the forward route would have used. Packing happens once per (tensor, K, N); the EP caches one slot per bound kernel and repacks when the pointer changes.

2. Activations: per-row int8 with amax/127. Every matmul input is quantised per row, on the device's own scale convention, and the dequant multiplies back C = Cm * ascale[row] * wscale[col]. This is the forward route's exact arithmetic; the measured device error on a 1024 × 1024 matmul is max_abs 9.893e-02 of max|ref| 20.414 (rel 4.846e-03) against a 2 % gate, so the int8 path is inside the tolerance the proof asserts.

3. Shapes: one ELF per (M, K, N), generated not fetched. An ELF is compiled for a fixed tiling (_m16 here, 16 rows per launch). The pack step is therefore a coverage problem, not a compatibility problem: for every (K, N) in the census we either have a generated ELF or we do not, and the plan is to generate the missing ones with the existing generator rather than to reshape the model.

What the pack does not do, deliberately#

• KV cache: opt-in, not the default. The default export re-runs the whole sequence per step, so its tokens/s is that graph's rate. --cached exports the same model with the cache as an input and an output (one stacked [56, 1, 8, past, 128] tensor), which decodes a 256-token prompt at 63.7 ms/token against 291.2 — 4.6x, and the gap widens with context. Output matches transformers running the same greedy loop to a mean KL of 1.2e-07 with top-1 identical at every step. What it is still not is a server: no batching, no paged cache, one sequence at a time, so its number is never put next to a serving stack's.
• No fp16, no int4 activations. The kernel is int8 × int8 → int32; the 4-bit-ness of the checkpoint lives on disk and is a memory-bandwidth story, not a compute one. An int4 tensor-core kernel would need a different ELF, and nothing here blocks that later.
• No vendor operators, ever. If a shape has no kernel it goes to the CPU EP. That keeps the failure mode boring: a slower session, never a session that refuses to load because a custom domain is missing.

Where the numbers live#

The measurements live in the tree, with the commands that produced them: onnx/BENCHMARK-PARITY.md holds the CPU rows (ryzen and Strix Halo, five prompt lengths, default and --cached exports) and the NPU row taken with the AMD model_benchmark protocol through onnx/bench_npu.sh; the execution-provider proof (MatMul on the NPU within 0.5 % of the CPU provider, the A8W8 QDQ pattern byte-identical) is on ONNX Runtime. This note is the design; those two pages are the evidence.

A8W8 QDQ, the shape Quark emits#

AMD's Quark ONNX-to-ONNX flow (the 2025 Ryzen AI NPU quantisation article; Linux-only) writes a QDQ graph rather than a custom-op graph: every tensor that used to be float becomes

x_q  --DequantizeLinear(s_x, zp_x)-->  x_dq  \
                                              MatMul --> y  --QuantizeLinear(s_y, zp_y)--> y_q
w_q  --DequantizeLinear(s_w, zp_w)-->  w_dq  /

with int8 activations and int8 weights (A8W8), float32 scales, and the same pattern at A16W8, Xint8 (power-of-two scales), BF16/BFP16, MX4/6/9, MXFP4/MXINT8 and Int4 weight-only, using GPTQ, AdaRound, AdaQuant, SmoothQuant or QuaRot to pick the scales. That is the good news: A8W8 is exactly what our device kernel already computes — int8 × int8 into int32, dequantised by one scale per activation row and one per weight column.

Why it folds without a new kernel#

Our GEMM is C_s32[m][n] = Σ_k a_q[m][k]·w_q[k][n], dequantised as C = C_s32 · s_a[m] · s_w[n]. The QDQ graph dequantises the same product as (s_x·s_w) · C_s32 (per-tensor), or s_x[m]·s_w[n] · C_s32 (per-axis, which is what Quark emits for weights and what SmoothQuant needs for activations). Those are the same expression: our kernel's go_rows already applies Cm·as[m]·gs[n], so a Quark-quantised model maps onto it by feeding the graph's scales instead of deriving them from amax/127. Nothing about the kernel changes; what changes is where the two numbers come from.

Folding the output QuantizeLinear is the same trick one step later: give the readback the scale 1/s_y and it emits int8 directly (y_q = clamp(round(C/s_y) + zp_y)) instead of float. The intermediate float tensor then never exists.

What works today, and what does not#

• Works: the group is ONE node. onebit_ep_qdq_proof builds exactly that pattern with the model editor API (int8 in / int8 out, DequantizeLinear → MatMul → QuantizeLinear) and our EP claims all four nodes as a single fused node, so neither DequantizeLinear nor QuantizeLinear runs and the dequantised float tensors never exist. The int8 activations and the int8 weights go into the kernel as they are; the readback folds both scales and the output scale into one multiply per element (y_q = clamp(round(C_s32 · s_x·s_w/s_y)), ONNX's round-half-to-even), so int8 out comes straight from the integer accumulator. Measured gate, ctest -R onebit_ep_qdq_proof: int8 outputs identical to the CPU EP's, worst |d| = 0 LSB on the 8×16×6 proof graph and also on a 256×1024×1024 case (262144 of 262144 values equal, so the int32-first arithmetic is in fact more exact than the CPU EP's float accumulation, which reaches the same function through two float tensors). Peak RSS growth of the run, measured around the run with the peak reset and malloc_trim between the warm-up and the measurement: the fused side is stable at 248-268 kB, the CPU EP side is 6.1-7.0 MB on 256×1024×1024. Both are ranges on purpose — the CPU EP figure is that process's arena growth, not a tensor size, so it moves run to run (eight runs on this tree: 6156, 6432, 6144, 6136, 6144, 6368, 7004, 6524 kB; a later run of this gate on the same tree measured 7236 kB, which is the same arena churn with a wider tail — read the band as indicative, not as a bound). What does not move is the bottom of that range: 6136 kB is the 1 MB x_dq + 4 MB w_dq + 1 MB y of float32 that no longer get allocated, and what sits above it is the CPU EP's own arena churn. ONEBIT_EP_QDQ=0 is the control: the same binary then claims the MatMul alone, logs a float matmul line, and the fused qdq line never appears.
• Works: per-axis scales, in the two forms the epilogue can fold. s_w[n] (the weight's axis 1, one scale per output column — what Quark emits for weights and what SmoothQuant needs) and s_x[m] (the activation's row axis, one scale per token) are read as vectors instead of scalars and land in the same epilogue: y_q[m][n] = clamp(round(C_s32 · s_x[m]·s_w[n]/s_y)). Measured, same gate as per-tensor: ctest -R onebit_ep_qdq_proof reports worst |d| = 0 LSB for the per-axis case (8 rows × 6 columns of scales, trace sx=8 sw=6) exactly as for the per-tensor one. Both proofs include a refusal: a per-axis scale along K cannot fold into an int8 × int8 epilogue at all (it weights the individual products inside the sum, which is SmoothQuant's job at export time), so the case gates on the EP not claiming the group — claimed=1, qdq_groups=0, the per-node path, and the same int8 output.
• Works: asymmetric zero points, folded into the same epilogue. Writing the shifted product out, Σ_k (a-zx)(w-zw) = Σ_k a·w − zx·Σ_k w − zw·Σ_k a + K·zx·zw, the GEMM already computes the first term, the second is a column sum of the weight (computed once on the first run that needs it and kept for the session — never per step), and the third is a row sum of the activation (one extra add per k while the row is already being read). The output zero point is added after the round-half-to-even, in that order, because rounding after an integer shift would move exact ties to the wrong neighbour. All three are per-tensor or per-axis along the same axes as the scales, and the corrections are exact int64. Measured, ctest -R onebit_ep_qdq_proof: worst |d| = 0 LSB against the CPU EP for zp_x = −40, zp_w = −25, zp_y = −7 with per-tensor scales, and for the fully per-axis case (8 row scales and 8 row zero points, 6 column scales and 6 column zero points, plus zp_y); the EP trace for those runs reads zp_x=8 zp_w=6 zp_y=-7 with a one-off qdq_colsum k=… n=…. A symmetric graph never computes the column sums (its trace reads zp_x=1 zp_w=1 zp_y=0 and no qdq_colsum), which is the point of doing it lazily.
• Not a graph: a zero point along K. The page's last "Not done" item was this one, and it is not unfinished work — a QDQ graph cannot carry a 1-D zero point along K. Two facts, one arithmetic and one checked against the runtime: (1) its term is cheap, not a per-step pass — Σ_k zp_x[k]·w_q[k][n] is a zero-point-weighted column sum of the weight and zp_x is a constant initializer, so it is computed once beside Σ_k w_q[k][n] and kept, unlike the row sums; (2) ONNX's DequantizeLinear says x_scale and x_zero_point must have the same shape, so a 1-D zero point along K only comes beside a 1-D scale along K, which is the un-foldable term above (it weights the products inside the sum — SmoothQuant's job at export time). Built the one way that could carry the term alone, a K-long zero point beside a per-tensor scale. onnx.checker accepts that graph, and the CPU EP's session builds; the node is refused when the CPU EP first prepares it to run (onnxruntime 1.29.0, on the first run(), not at session creation): quantize_linear.cc:79 … x_zero_point must be null or a scalar or 1D tensor or size 1. So the item closes as "cannot occur" rather than "not done": what this epilogue does not fold along the activation's K is what the arithmetic rules out.
• Works: the fused group runs on the NPU, and its int8 output is bit-identical there too. With -DONEBIT_EP_NPU=ON and ONEBIT_EP_NPU_ELFS naming the designs, gemm_q8 writes the graph's int8 activations and int8 weights straight into the kernel's BOs (no packB, no amax/127 — those bytes already carry the graph's scales) and reads the int32 accumulators back into the same epilogue the host path runs, so the two can only differ if the accumulator does. Measured on strixhalo under npu-hold with five designs configured (…/full_i8_O_K2048_N1024.elf, …/full_i8_D_K3072_N1024.elf, and Q_K1024_N2048, KV_K1024_N1024, GU_K1024_N3072 generated from the iso-build design generator, all as M=128 entries): worst |d| = 0 LSB against the CPU EP for o_proj (5×2048×1024, 5120 int8 values), down_proj (5×3072×1024, 5120), k_proj/v_proj (5×1024×1024, 5120), q_proj (5×1024×2048, 10240) and gate_proj/up_proj (5×1024×3072, 15360), with no npu_qdq_fallback line for any of those five shapes in either run's trace — the device ran all of them. That is 196 of the 197 fused groups a real Qwen3-0.6B A8W8 export has (28 layers × 7 projections + lm_head); lm_head (1024×151936) keeps the host kernel because it needs a tiling of its own. Two limits belong with that number. It needs an ELF per (K, N), and a shape that has none is a build rather than a missing capability: o_proj's and down_proj's ship beside the other xclbins, and q/k/v's and gate/up's came out of one -M 128 -K <K> -N <N> -c 4 run of the iso-build generator each (n1_core_i8_v26.py through the IRON/MLIR-AIE aiecc), the artifacts living beside the other xclbins and never in this repository. It is a correctness result, and at model level it is a small speed one — 8.3 % on a quiet box on the current tree, after the same export measured 1.13–1.16× slower on an earlier one; both are measured rather than suspected, and the last row below is the number to quote. Same export, a Release build of this tree, a quiet box (load 1.2–1.4), one thread, a 32-token prompt and 8 generated tokens: the array's arm does 1074 / 1105 ms/token against the host kernel's 947 / 951, i.e. 1.13–1.16× slower than the leg it replaces (and 1.7× behind the A8W8 CPU EP at 625 / 628, 4.8× behind float32 at 226); the 32-token prefill is a wash, 805 / 809 against 843 / 845. Re-measured on the current tree those two arms are 930.5 against 972.9 ms/token, i.e. the array 4.4 % ahead — same window, two runs each, both agreeing on the sign, and the profile now splits a pass into device calls 802.5 + epilogue 19.0 against the host loop's 879.0 with 94–109 ms of per-pass work that neither leg touches. Repeated on a quiet box (load1 1.3–3.3) the same sign is larger and is now the quotable one: 983.4 / 1000.2 against 1073.6 / 1088.8 ms/token, i.e. 8.3 % ahead, per call 4.32 against the host loop's 4.99. Not the code, as far as the repository can say: since those figures, only #1032 (profiling) and #1047 (the re-pack, ~5 % of a token) touch this path, the NPU runtime the EP links against is unchanged and the five ELFs are byte-identical, so the 1.13× loss does not reproduce: the host arm of the window above ran at load1 1.3–1.6, the same load as the 1.2–1.4 that loss was taken at, and there the array is ahead — though that window is not an arm-for-arm repeat of it, since its device arm ran at 2.8–3.3. So the sign is no longer the finding — the magnitude is, and it is small in every window; quote the number only with the session it came from (docs/onnx.md, "Where a pass actually goes"). The kernels are ahead in isolation — 1.54–1.89× per call at a decode step's 36 rows and 3.1× at a 32-row prefill block — and that advantage does not survive the model. These measurements do not, by themselves, say which of the two candidate reasons it is: non-kernel work per call (BO writes, two syncs, a wait, the weight writeback, the shared epilogue); or the host loop being faster on the graph's real operands than on the probe's synthetic ones (docs/onnx.md explains why the second cannot be excluded). Three of those per-call costs were measured separately on the array (weight writeback 0.24 ms, epilogue 0.134 ms, pack 0.06 ms), and the weight path is not worth fixing: keeping every image resident returns 35 ms/pass for 568 MiB (docs/onnx.md carries both). The call's own sections then priced all of them in one window: of a 4.32 ms call the launch plus its wait is 3.90 (90 %, and that section contains the kernel's multiply), the weight pack plus upload 0.25, other 0.13, the readback 0.036, both flushes 0.008 — so every non-multiply thing the leg does per call is 0.42 ms (9.8 %), 0.54 with the 0.12 epilogue it pays outside the call, and no fix aimed at them can recover more. The second candidate turned out to be measurable without the array: dumped from one prefill, the graph's real int8 activations are 62.1 % nonzero against the probe's ~99 %, which makes its host loop 1.42× faster per pass (1098 → 770.6 ms) and cuts the array's kernel advantage from 1.7× to ~1.12×. So the multiply is not where the 1.13× loss of the figure above came from, and that loss does not reproduce on the current tree at any load tried: what is there is the 8.3 % win above, whose per-call cost the probe also prices (4.32 ms/call at model level against its 3.460 for a single shape in the same window, i.e. the EP adds no structural per-call cost). (Those arms, like every number on this page, come from a Release build: the EP targets carry no -O2 of their own, and an unoptimized host loop inflates the device-against-host ratio by ~25×.) docs/onnx.md ("What the device leg is worth at model level") carries the four-arm table, the per-shape accounting and the trace evidence. A decode-side win needs fewer, longer launches. So gemm_q8 routes by rows: below kDeviceMinRows — 16, one constant in onnx/ep/onebit_ep_kernel.h — the host loop runs the multiply and the call never reaches the array, and at or above it the device does. A host-routed call logs npu_qdq_host_rows m=… k=… n=… min=16, so "the device ran this shape" stays "neither npu_qdq_host_rows nor npu_qdq_fallback was written for it". 12 rows is where the margin is smallest — 1.13–1.23× with a quiet box, break-even for k/v with a loaded one — and 16 is the first count that keeps more margin than a busy box adds (1.53–1.64×); at one row the launch costs about 10× the host loop it replaces (docs/onnx.md carries both measurements, the quiet one and the loaded one, with their internal spreads). A decode-side win needs fewer, longer launches rather than a smaller design.

Which row count the gate above is taken at, because the threshold changed it. Those five numbers came from 5-row cases, which are decode width: once gemm_q8 routed by rows they stopped reaching the array at all, so for a while o_proj, down_proj and gate/up had no device gate while this page said they were covered — the same "host gate dressed as a device one" failure this file has recorded before, in its quietest form: the run still printed PASS and worst |d| = 0. Each of those three shapes now has a 32-row device case (quark-o32, quark-d32, quark-gu32), and onebit_ep_qdq_proof asserts both halves — a fused case at or above the threshold must not log npu_qdq_fallback, and one below it must log npu_qdq_host_rows — so the claim is checked in the test rather than in a lane script. The gate needs an ELF for the shape, not a particular design M: 32 rows through an M=128 design pads to 128 and is equally a device run, which is how the 5-row cases reached the array before the threshold existed. Their gate is one LSB rather than zero, and that is arithmetic, not slack: the CPU EP dequantises and accumulates in float32 while the fused path sums exact int32 and rounds once, so at a quantisation boundary the two can round to different neighbours — measured as 0 of 32768 on down_proj and gate/up and 1 of 32768 on o_proj (the same 1-in-5120 the real Quark export showed). Above one LSB is a defect: a wrong weight layout or a truncated K reads as hundreds. - Different kernels, not a configuration: A16W8 needs int16 activations, MX/BFP16 need block scales, and Int4 weight-only is the format our own forward route already stores (Q4NX), so the natural split afterwards is our Q4NX pack for weight-only and the Quark QDQ path for A8W8.

A real Quark export, through the EP#

onnx/quantize_quark_a8w8.py runs AMD's Quark ONNX flow (source tree, release/0.13) on qwen3-0.6b.onnx, calibrating on 32 x 512 tokens of local text (no dataset download). Quark writes the QuantizeLinear/DequantizeLinear pair in the com.microsoft domain, which the EP accepts as the same operator; it also quantises only what it is told to, so --matmul-only keeps attention's QK^T/AV (which have no constant weight) in float.

On a 0.6B A8W8 export with per-channel weights, the graph ORT hands the EP (4169 nodes, ORT_ENABLE_BASIC so the DQ/Q pairs survive) partitions like this:

Measured, single-threaded, 1bit ort-run --basic on the same 5-token prompt (--dump-logits), 16 tokens generated:

Three things that table says, none of them flattering and all of them worth knowing:

1. The fused path is 1.46x the CPU EP on the per-channel export and 0.67x on the per-tensor one. Per-channel DequantizeLinear is expensive on the CPU EP (it broadcasts a vector), so removing the whole DQ -> float MatMul -> Q sequence pays off there; per-tensor DQ is cheap and MLAS's float MatMul is faster than this host int8 kernel. This is a host-kernel measurement: the device leg runs the same epilogue on the array for all seven projection shapes, but only as a correctness result so far, so it is not where the 1.5x-per-projection argument lands either.
2. The quantisation, not the EP, is what costs accuracy. KL 2.2-2.8 nats is my calibration's fault (32 documents, MinMax, no AdaRound/QuaRot/SmoothQuant), and the per-tensor export (6.5 nats) is worse still: one int8 scale per weight matrix is not enough for this model. AMD's own recipe is the one that produces a usable model; this export exists to exercise the EP.
3. The Python probe_dump comparison is the honest correctness gate, not bit-identity. On the per-tensor export the two paths are bit-identical (KL 0.000 between them, 17 of 17 decode steps the same). On the per-channel export the fused int8 kernel and the CPU EP's DQ -> float MatMul -> Q agree exactly on the first 38 projection tensors and then differ on 1 value in 5120 (0.02 %): the accumulator is -3090, the exact value is -3.499998 and rounds to -3, while the CPU EP's float32 accumulation lands on exactly -3.500000 and rounds half-to-even to -4. Both are right for their own arithmetic; the difference then cascades, because every layer requantises to int8 and there is no KV cache to damp it. That is why the fused path's KL is 2.81 and the CPU EP's 2.19 on the same graph, and why the proof's gate is bit-exactness on graphs where the values are not on a boundary.

The EP's debug knob for exactly this work is ONEBIT_EP_QDQ_DEBUG=1: it logs, per graph, why each MatMul was not fused (qdq_refused why=... count=...). Without it "0 groups" is not an answer.

The proof is deliberately opset-21 and domain-only. It pins down the claim (our EP takes the whole group as one node and the arithmetic is exact) for the per-tensor, per-axis and asymmetric forms Quark emits by default, on the two device-relevant shapes as well, and leaves the device leg to its own configuration (ONEBIT_EP_NPU_ELFS, one ELF per shape) and its own evidence.

View this page's source on GitHub ↗