GitHub ↗
Contents

NPU engine leaderboard#

One row is one measured run of this engine. A record is only ranked if it names its instrument, its repeat count and its timestamp; see bench/npu-leaderboard/README.md for the shape. Decode latencies are s/token (lower is better, and they are what a user feels); throughputs are tok/s (higher is better). Rows are not comparable across hosts.

strixhalo#

model route quant metric value settings repeats instrument
Bonsai-2-27B npu PTQ1_0 ternary, t2s 2-bit stream (2.125 bpw) decode 313.0 ms/token device=accel0, port=18099, container=~/models/npu/bonsai2-27b-t2s2 (q4nx_weight_layout=t2s, version 6, sealed SEALED-T2S.elf, M=16), binary=~/wt/bonsai-speed-build2/1bit (2026-11-02, engine 916e90d + the Hadamard SIMD host path), stream_per_token_gb=6.11, tokens=7 1 NPU_PHASE_PROF=1, one run per arm: the serve's own [ph] tok wall= line, median of its 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once
Bonsai-2-27B npu pb16i (int4 codes + int16 magnitudes, 4.5 bpw) decode 410.0 ms/token device=accel0, port=18099, container=~/models/npu/bonsai2-27b-pb16i (q4nx_weight_layout=pb16i, version 4, M=16), binary=~/wt/bonsai-speed-build2/1bit (2026-11-02, engine 916e90d + the Hadamard SIMD host path), stream_per_token_gb=12.93, tokens=7 1 NPU_PHASE_PROF=1, one run per arm: the serve's own [ph] tok wall= line, median of its 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once
Flash-Next (IQ4_XS 4.25 bpw) hrx IQ4_XS prefill pp512 286.63 tok/s ngl=99, fa=1, ub=512, engine=0d0c846 3 llama-bench -p 512, 3 repeats, s.d. 0.94 tok/s
Flash-Next (IQ4_XS 4.25 bpw) hrx IQ4_XS prefill p1024 ub576 269.34 tok/s ngl=99, fa=1, ub=576, engine=0d0c846 1 llama-bench -p 1024, 1 repeat
Flash-Next (IQ4_XS 4.25 bpw) hrx IQ4_XS prefill pp4096 260.59 tok/s ngl=99, fa=1, ub=512, engine=0d0c846 3 llama-bench -p 4096, 3 repeats, s.d. 4.51 tok/s
Qwen3-4B hrx as shipped (1bit-hrx-3092596) decode (eval) 41.48 tok/s parallel=4, kv_unified=true, requests=60 1 llama-server timings in the response body
Qwen3-Coder-30B-A3B-Instruct hrx Q4_K_M decode aggregate 87.73 tok/s parallel=4, requests_in_flight=4, kv_unified=True, n_ctx=8192, per_stream_tok_s=21.94, solo_same_server_tok_s=84.22, engine=38e29a2 + i612-npu-streams (b78868c), llama_cpp=40c4ccb76, model_file=Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf 5 wall clock per request + usage.completion_tokens, n_predict=128, temperature 0, greedy, distinct prompts
Qwen3-Coder-30B-A3B-Instruct hrx Q4_K_M decode 84.33 tok/s parallel=1, requests_in_flight=1, n_ctx=8192, engine=38e29a2 + i612-npu-streams (b78868c), llama_cpp=40c4ccb76, model_file=Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf 5 wall clock per request + usage.completion_tokens, n_predict=128, temperature 0, greedy, distinct prompts
Qwen3-Coder-30B-A3B-Instruct hrx Q4_K_M decode aggregate 63.45 tok/s parallel=2, requests_in_flight=2, kv_unified=True, n_ctx=8192, per_stream_tok_s=31.73, solo_same_server_tok_s=84.33, engine=38e29a2 + i612-npu-streams (b78868c), llama_cpp=40c4ccb76, model_file=Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf 5 wall clock per request + usage.completion_tokens, n_predict=128, temperature 0, greedy, distinct prompts
Qwen3.8-27B-NPU2 npu Q4NX ternary sidecars, sealed single-ELF decode GSM8K accuracy (n=40) 97.5 % device=accel0, engine=1bit t2s2-20261104 (serve-bin), model_dir=~/models/npu/bonsai2-27b-t2s2-serve, container=Qwen3.8-27B-NPU2 (qwen3_5, 64 layers, H=5120, ternary sidecars, sealed single-ELF decode set), tickets=npu-hold -t 25, labels 587-gsm8k-{pilot,0-7,8,16,24,32}, port 18099, sampling=temperature 0, top_k 1, seed 1234, stream=false, max_tokens 512, prompt=Solve this grade-school maths problem. Reason step by step, then write your final answer on the last line as\n#### \n\nProblem: {q}, n=40, split=openai/gsm8k main/test, questions 0-39 in dataset order, accuracy=39/40, truncated=6, truncated_and_wrong=1 1 tools/gsm8k_eval.py against the served route: greedy, answer read from the model's own #### <number> line, gold from the dataset's #### field, both normalised; per-question transcript kept (gold, prediction, finish_reason, tokens, wall)
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact draft hit rate (live loop, k=4, tau=0.6) 78.77 % device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0.6, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 1 tools/mtp_depth_pool.py over the eight in-tree tickets tests/fixtures/mtp_tau06/p*.txt; counts summed over the window's 277 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact draft acceptance, depth 1 (live loop, all chains, tau=0.6) 78.34 % device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0.6, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 1 tools/mtp_depth_pool.py over the eight in-tree tickets tests/fixtures/mtp_tau06/p*.txt; counts summed over the window's 277 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact draft acceptance, depth 1 (live loop, all chains) 75.14 % device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 3 tools/mtp_depth_pool.py over the three in-tree tickets tests/fixtures/mtp_clause55/*.txt; counts summed over the window's 181 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact draft acceptance, depth 2 (live loop, all chains) 47.51 % device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 3 tools/mtp_depth_pool.py over the three in-tree tickets tests/fixtures/mtp_clause55/*.txt; counts summed over the window's 181 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact draft hit rate (live loop, k=4) 46.69 % device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 3 tools/mtp_depth_pool.py over the three in-tree tickets tests/fixtures/mtp_clause55/*.txt; counts summed over the window's 181 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact draft acceptance, depth 3 (live loop, all chains) 34.81 % device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 3 tools/mtp_depth_pool.py over the three in-tree tickets tests/fixtures/mtp_clause55/*.txt; counts summed over the window's 181 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact draft acceptance, depth 4 (live loop, all chains) 29.28 % device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 3 tools/mtp_depth_pool.py over the three in-tree tickets tests/fixtures/mtp_clause55/*.txt; counts summed over the window's 181 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX ternary sidecars, sealed single-ELF decode prefill 19.55 tok/s device=accel0, engine=1bit t2s2-20261104 (serve-bin), model_dir=~/models/npu/bonsai2-27b-t2s2-serve, container=Qwen3.8-27B-NPU2 (qwen3_5, 64 layers, H=5120, ternary sidecars, sealed single-ELF decode set), ticket=npu-hold -t 14 -l 587-27b-row4, port=18099, sampling=temperature 0, top_k 1, seed 1234, stream=false, prompt_tokens=13, prompts=13-token and 37-token prompts, 1 completion token, 3 repeats each, slopes=(t37 - t13) / 24 prompt tokens = 51.1 ms/token 3 wall clock per request at n=1 for two prompt lengths; prefill is the difference over the 24 extra prompt tokens, with the single decode step and the fixed per-request overhead cancelling; t13 0.9537 s +- 0.0038, t37 2.1810 s +- 0.0344
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact draft acceptance, depth 2 (live loop, all chains, tau=0.6) 3.97 % device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0.6, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 1 tools/mtp_depth_pool.py over the eight in-tree tickets tests/fixtures/mtp_tau06/p*.txt; counts summed over the window's 277 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact tokens per pass (live loop, k=4) 2.87 tokens/pass device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 3 tools/mtp_depth_pool.py over the three in-tree tickets tests/fixtures/mtp_clause55/*.txt; counts summed over the window's 181 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact tokens per pass (offline protocol, same positions, k=4) 2.82 tokens/pass device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 3 tools/mtp_depth_pool.py over the three in-tree tickets tests/fixtures/mtp_clause55/*.txt; counts summed over the window's 181 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact tokens per pass (offline protocol, same positions, k=4, tau=0.6) 2.05 tokens/pass device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0.6, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 1 tools/mtp_depth_pool.py over the eight in-tree tickets tests/fixtures/mtp_tau06/p*.txt; counts summed over the window's 277 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact tokens per pass (live loop, k=4, tau=0.6) 1.83 tokens/pass device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0.6, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 1 tools/mtp_depth_pool.py over the eight in-tree tickets tests/fixtures/mtp_tau06/p*.txt; counts summed over the window's 277 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX pb16i (int8 GEMM) + MTP head pb16i-exact draft acceptance, depth 3 (live loop, all chains, tau=0.6) 0.72 % device=accel0, model_dir=bonsai2-27b-pb16i, head=pb16i-exact, k=4, tau=0.6, prompts=8, tokens_per_prompt=64, batch_rows=16, vocab=248320, hidden=post-final-norm 1 tools/mtp_depth_pool.py over the eight in-tree tickets tests/fixtures/mtp_tau06/p*.txt; counts summed over the window's 277 chain iterations
Qwen3.8-27B-NPU2 npu Q4NX ternary sidecars, sealed single-ELF decode decode 0.2902 s/token device=accel0, engine=1bit t2s2-20261104 (serve-bin), model_dir=~/models/npu/bonsai2-27b-t2s2-serve, container=Qwen3.8-27B-NPU2 (qwen3_5, 64 layers, H=5120, ternary sidecars, sealed single-ELF decode set), ticket=npu-hold -t 14 -l 587-27b-row4, port=18099, sampling=temperature 0, top_k 1, seed 1234, stream=false, prompt_tokens=13, batches=n=1, n=5, n=9 completions on the same 13-token prompt, 3 repeats each, slopes=(n5-n1)/4 = 0.2884, (n9-n1)/8 = 0.2920, spread +1.3 % 3 wall clock per request (curl, HTTP 200, usage.completion_tokens read back); decode is the slope between completion lengths, not a single mixed request; n=1 0.9537 s +- 0.0038, n=5 2.1073 s +- 0.0150, n=9 3.2900 s +- 0.0105
Qwen3.8-27B-NPU2 npu Q4NX ternary sidecars, sealed single-ELF decode decode, chain-of-thought answers (per token, prefill included) 0.3857 s/token device=accel0, engine=1bit t2s2-20261104 (serve-bin), model_dir=~/models/npu/bonsai2-27b-t2s2-serve, container=Qwen3.8-27B-NPU2 (qwen3_5, 64 layers, H=5120, ternary sidecars, sealed single-ELF decode set), tickets=npu-hold -t 25, labels 587-gsm8k-{pilot,0-7,8,16,24,32}, port 18099, sampling=temperature 0, top_k 1, seed 1234, stream=false, max_tokens 512, prompt=Solve this grade-school maths problem. Reason step by step, then write your final answer on the last line as\n#### \n\nProblem: {q}, n=40, answer_lengths=106 to 512 completion tokens (mean 331), per_question_ratio=mean 0.3909, median 0.3458, aggregate (total wall / total tokens) 0.3857 1 the same 40 requests as 27b-gsm8k-20261107: per-request wall clock / usage.completion_tokens, summed over the run (13,226 tokens in 5,101.4 s)
Qwen3.8-27B-NPU2 npu Q4NX ternary sidecars, sealed single-ELF decode first token (weight load included) 22.811 s device=accel0, engine=1bit t2s2-20261104 (serve-bin), model_dir=~/models/npu/bonsai2-27b-t2s2-serve, container=Qwen3.8-27B-NPU2 (qwen3_5, 64 layers, H=5120, ternary sidecars, sealed single-ELF decode set), ticket=npu-hold -t 14 -l 587-27b-row4, port=18099, sampling=temperature 0, top_k 1, seed 1234, stream=false, prompt_tokens=13, note=/health answers 4 s after launch, before the container is loaded; the load lands inside the first request 1 wall clock of the first request after the serve answered /health, 13-token prompt, 1 completion token
Qwen3.8-27B-NPU2 npu Q4NX (int8 GEMM) prefill 58.14 s device=accel0, addon=npu-dx 1e89468d4, engine=fix/587-gdn-batch on 800cf6f, batch=5 tokens in one step_batch chunk, submits=350, threads=4, pin=0-3, tokens=1 1 wall clock for one request + /proc//stat cpu_s + npu-trace submit count; same prompt/params as the 145.7 s sequential record
Qwen3.8-27B-NPU2 npu Q4NX (int8 GEMM) decode 63.1 s/token device=accel0, addon=npu-dx 1e89468d4, threads=4, pin=0-3, tokens=8 1 wall clock / tokens, 505 s for 8 greedy tokens
Qwen3.8-27B-NPU2 npu Q4NX (int8 GEMM) decode 67.5 s/token device=accel0, addon=local xrt-free patch, threads=4, tokens=8 1 wall clock / tokens, 540 s for 8 greedy tokens
Qwen3.8-27B-NPU2 npu Q4NX (int8 GEMM) first token (weight pack included) 145.7 s device=accel0, commits=800cf6f, threads=32 1 request wall and driver trace span
Qwen3.8-27B-NPU2 npu Q4NX (int8 GEMM) first token (weight pack included) 435.9 s device=accel0, commits=pre-#622, threads=32 1 driver trace span, gated on ~/npu.lock
  • bonsai2-27b-t2s2-20261102 — 2026-11-02T23:50:00Z; source: ~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 AST entry: three npu-hold tickets (19:47, 19:48, 19:49 AST) on :18099, amdxdna counter 0 in each, spread 311-317 ms/token; P3 (#802) 'after' arm against bonsai2-27b-pb16i-20261102.json: 1.31x per token and 1.42x on the device wait (209 vs 296 ms). Accept is byte-exactness, not a text diff: every decode role matched the pb16i path byte for byte at layer 0 (6/6) and layer 3 (8/8) on three seeds, and greedy text is equal to the pb16i arm's on the prompt both were asked (' Paris.\nThe capital of Germany is'). A three-prompt full-model text window on this pair was NOT run - the 2026-11-06 re-take was blocked by an unrelated serve holding the engine's device lock - so that remains the open acceptance step. Stream is 2.12x fewer bytes (431 images, 14.44 -> 6.82 GB). What would make this wrong: quoting a request wall instead of [ph] tok (a fresh serve adds the ~10.6 s weight-image fill), or comparing across boxes - the pb16i arm is the only valid control.
  • bonsai2-27b-pb16i-20261102 — 2026-11-02T23:50:00Z; source: ~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 AST entry: three npu-hold tickets (19:47, 19:48, 19:49 AST) on :18099, amdxdna counter 0 in each, spread 409-412 ms/token; control arm of the P3 (#802) pair: the same container file, the same binary and the same box one minute from bonsai2-27b-t2s2-20261102.json, so the ratio is a weight-format difference and nothing else. This is the 4.5 bpw path the 2-bit stream replaced; it is not a candidate, it is the denominator. What would make this wrong: measuring it in a different window than the t2s arm (the pair is only valid adjacent).
  • flashnext-pp512-20261027 — 2026-10-27T02:38:52Z; source: ~/lb/572/run-20261027T015408Z/a1-timing.out (strixhalo); gufo reference on the same box is 1679 tok/s; the analysis is in docs/hrx.md
  • flashnext-p1024-ub576-20261027 — 2026-10-27T02:50:00Z; source: ~/lb/572/run-20261027T015408Z/a4-ub576.out (strixhalo); shows the ubatch bound above 512 no longer refuses the run; single repeat, do not read as a tuned number
  • flashnext-pp4096-20261027 — 2026-10-27T02:41:00Z; source: ~/lb/572/run-20261027T015408Z/a1-timing.out (strixhalo)
  • qwen3-4b-hrx-parallel4-20261027 — 2026-10-27T12:36:00Z; source: ~/lb/528/run-20261027T123606Z-none (strixhalo); #528 repro arm: only one slot was used, so this is a single-slot decode rate and the NaN report stayed unreproduced
  • coder30b-hrx-parallel4-20261030 — 2026-10-30T21:16:28Z; source: ~/lb/612-hrx-20261030T211252Z (tools/showcase-612-hrx.sh 2 4 8 on an idle strixhalo, under gpu-hold label 'pi i612'); reproduces ~/lb/612-hrx-20261030T064957Z; 1.04x: four streams only recover the single-stream baseline (91.11 / 1.08x at 06:45Z). llama-server reports all four slots decoding in one batch at 24.88 t/s each, so the batching is real; the aggregate is flat because the model is bandwidth-bound. No --parallel 8 row is published: at 8 streams the rate is non-stationary inside a run (llama-server's own tg per slot goes 14.99 15.12 14.34 12.10 7.09 6.78 5.70 5.64 as the answers grow, then 14.17 for a fresh batch, with die temp 61-71 C and iGPU sclk 694-2127 MHz swinging normally), so an 8-stream number is a function of where in that decay it is sampled: 112.89 / 1.34x at 06:45Z, 78.87 / 0.97x and 68.29 / 0.92x later the same day.
  • coder30b-hrx-solo-20261030 — 2026-10-30T21:16:28Z; source: ~/lb/612-hrx-20261030T211252Z (tools/showcase-612-hrx.sh 2 4 8 on an idle strixhalo, under gpu-hold label 'pi i612'); reproduces ~/lb/612-hrx-20261030T064957Z; One request at a time, the baseline the batched rows are read against. Measured as the solo arm of the --parallel 2/4 servers (84.33 / 84.22, ten requests) and 81.39 on the --parallel 8 server; across both runs today the same load gave 74.03-88.34, i.e. +-8 %, so compare ratios, not absolute tok/s.
  • coder30b-hrx-parallel2-20261030 — 2026-10-30T21:16:28Z; source: ~/lb/612-hrx-20261030T211252Z (tools/showcase-612-hrx.sh 2 4 8 on an idle strixhalo, under gpu-hold label 'pi i612'); reproduces ~/lb/612-hrx-20261030T064957Z; 0.75x: two requests in flight are slower in aggregate than one, and it reproduces (62.96 at 06:45Z, 63.45 here, on different clocks). A 2-row decode batch costs 29.0 ms/token per stream against 11.9 ms for one row; with ONEBIT_HRX_DECODE_SPLIT=0 the same server gives 57.59 from a 45 tok/s solo, so the vectorised decode-split path is the suspect -- not settled.
  • 27b-gsm8k-20261107 — 2026-11-07T13:49:36Z; source: ~/evidence/tmoe-npu/587-gsm8k-20261107/ (chunk-{0,8,16,24,32}.json + merge.py + transcript-40.json), tasks bc2f26ddb / b51b849c3; 97.5 % (39/40) with 6 of the 40 answers cut off by the 512-token budget (finish_reason=length) and exactly 1 of those 6 also wrong (q12: gold 13, predicted 12 from a partial derivation) - so among answers that finished, 34/34 were right. That split is the point of the tool: a truncated answer is not evidence about the model's maths, and a bare '97.5 %' would have hidden it. n=40 of 1319 test questions: +-~5 % at 95 % confidence, so read it as 'about 40 of 40', not as a fourth significant figure. 0 route errors. One relabelling caveat, stated because it is real: the tool's --offset numbered each chunk's rows 0..n-1 at the time, so the transcript's absolute question numbers are derived from each chunk file's own 'questions k-k+7' line (merge.py asserts the result is exactly 0-39); the answers, tokens and wall times are the measured ones and the tool is fixed and pinned by tests/gsm8k_eval_test.py.
  • 27b-mtp-tau06-hit-rate-20261106 — 2026-11-06T22:06:22Z; source: tests/fixtures/mtp_tau06/p0..p7.txt, verbatim extracts of npu-hold tickets tau06 p0..p7 (20261106-164254/165612/170407/172013/172718/173228/174605/175540), one prompt per ticket, tau=0.6; 8 prompts x 64 tokens, k=4, tau=0.6 (the draft route's DEFAULT), one prompt per npu-hold ticket; greedy text byte-identical to plain decode on 8 of 8 prompts, 0 stream mismatches, 0 amdxdna errors during any arm. The window is the union of eight single-prompt tickets, not eight repeats: each prompt was measured once, and prompt 0 was measured twice (16:29 and 16:42) with identical counts 31/35, the only repeat in the set. 230 accepted of 292 drafts. The gate buys confidence per draft: 78.77 % here against 46.69 % at tau=0, on 1.05 drafts per chain against 4.0.
  • 27b-mtp-tau06-depth1-20261106 — 2026-11-06T22:06:22Z; source: tests/fixtures/mtp_tau06/p0..p7.txt, verbatim extracts of npu-hold tickets tau06 p0..p7 (20261106-164254/165612/170407/172013/172718/173228/174605/175540), one prompt per ticket, tau=0.6; 8 prompts x 64 tokens, k=4, tau=0.6 (the draft route's DEFAULT), one prompt per npu-hold ticket; greedy text byte-identical to plain decode on 8 of 8 prompts, 0 stream mismatches, 0 amdxdna errors during any arm. The window is the union of eight single-prompt tickets, not eight repeats: each prompt was measured once, and prompt 0 was measured twice (16:29 and 16:42) with identical counts 31/35, the only repeat in the set. 217/277 chains accepted draft 1.
  • 27b-mtp-live-depth1-20261105 — 2026-11-05T16:07:13Z; source: tests/fixtures/mtp_clause55/{p0,p1-4,p5-7}.txt, verbatim extracts of npu-hold tickets clause55 p0|p1-4|p5-7 (20261105-115131|115551|120713); 136/181 chains. 8 prompts x 64 tokens, k=4, tau=0; greedy text byte-identical to plain decode on 8 of 8 prompts, 0 stream mismatches. The window is the union of three npu-hold tickets, not three repeats of one window. Per-depth rates conditional on reaching the depth (what the 2 pp clause compares): 75.14 / 63.24 / 73.26 / 84.13 % live against 75.14 / 62.69 / 73.17 / 83.05 % offline at the same positions -- deltas +0.00 / +0.55 / +0.09 / +1.08 pp, 4 of 4 PASS. The all-chains view recorded here (each depth over all 181 chains) reads +0.00 / +1.10 / +1.66 / +2.21 pp on the same counts, so the two conditionings disagree at depth 4; every rate in this row is the all-chains one.
  • 27b-mtp-live-depth2-20261105 — 2026-11-05T16:07:13Z; source: tests/fixtures/mtp_clause55/{p0,p1-4,p5-7}.txt, verbatim extracts of npu-hold tickets clause55 p0|p1-4|p5-7 (20261105-115131|115551|120713); 86/181 chains; 86 of 136 reached = 63.24 % conditional, against offline 84/134 = 62.69 % (84/181 = 46.41 % all-chains).
  • 27b-mtp-hit-rate-20261105 — 2026-11-05T16:07:13Z; source: tests/fixtures/mtp_clause55/{p0,p1-4,p5-7}.txt, verbatim extracts of npu-hold tickets clause55 p0|p1-4|p5-7 (20261105-115131|115551|120713); 338 accepted of 724 drafts over 181 chain iterations; the offer rate is what the all-chains per-depth rows above compose with.
  • 27b-mtp-live-depth3-20261105 — 2026-11-05T16:07:13Z; source: tests/fixtures/mtp_clause55/{p0,p1-4,p5-7}.txt, verbatim extracts of npu-hold tickets clause55 p0|p1-4|p5-7 (20261105-115131|115551|120713); 63/181 chains; 63 of 86 reached = 73.26 % conditional, against offline 60/82 = 73.17 % (60/181 = 33.15 % all-chains).
  • 27b-mtp-live-depth4-20261105 — 2026-11-05T16:07:13Z; source: tests/fixtures/mtp_clause55/{p0,p1-4,p5-7}.txt, verbatim extracts of npu-hold tickets clause55 p0|p1-4|p5-7 (20261105-115131|115551|120713); 53/181 chains; 53 of 63 reached = 84.13 % conditional, against offline 49/59 = 83.05 % (49/181 = 27.07 % all-chains). This is the depth where the all-chains view reads +2.21 pp and the conditional one +1.08 pp.
  • 27b-t2s2-prefill-20261106 — 2026-11-06T20:53:40Z; source: ~/evidence/tmoe-npu/587-27b-row-20261106/r2 (this session; per-request JSON kept), task b9e132438; This is the per-op NPU path, not a batched prefill: the plan declares M=128 for prefill but this serve answers one prompt at a time, so the figure is the marginal cost of a prompt token and should not be read as 32-token-chunk throughput. Context: the int8 rung measured 58.14 s for a 5-token chunk (27b-prefill-batched-20261027). HRX prefill on the same box is ~287 tok/s, so the NPU is far behind on this axis - which is why the hybrid answer in engine#587 has always been 'recommended against'.
  • 27b-mtp-tau06-depth2-20261106 — 2026-11-06T22:06:22Z; source: tests/fixtures/mtp_tau06/p0..p7.txt, verbatim extracts of npu-hold tickets tau06 p0..p7 (20261106-164254/165612/170407/172013/172718/173228/174605/175540), one prompt per ticket, tau=0.6; 8 prompts x 64 tokens, k=4, tau=0.6 (the draft route's DEFAULT), one prompt per npu-hold ticket; greedy text byte-identical to plain decode on 8 of 8 prompts, 0 stream mismatches, 0 amdxdna errors during any arm. The window is the union of eight single-prompt tickets, not eight repeats: each prompt was measured once, and prompt 0 was measured twice (16:29 and 16:42) with identical counts 31/35, the only repeat in the set. 11/277 chains accepted draft 2 -- the gate stops most chains after draft 1, which is what this row measures.
  • 27b-mtp-live-tokens-per-pass-20261105 — 2026-11-05T16:07:13Z; source: tests/fixtures/mtp_clause55/{p0,p1-4,p5-7}.txt, verbatim extracts of npu-hold tickets clause55 p0|p1-4|p5-7 (20261105-115131|115551|120713); 1 + accepted/iterations = 1 + 338/181; hit rate 338/724 = 46.69 %. Pooled over the window's iterations, not the unweighted mean of the 8 per-prompt chains (3.14, the figure docs/npu.md quotes) -- they differ because the prompts with short chains take more iterations: 1.80 to 4.33 tokens/pass per prompt.
  • 27b-mtp-offline-tokens-per-pass-20261105 — 2026-11-05T16:07:13Z; source: tests/fixtures/mtp_clause55/{p0,p1-4,p5-7}.txt, verbatim extracts of npu-hold tickets clause55 p0|p1-4|p5-7 (20261105-115131|115551|120713); 1 + 329/181: the offline protocol restricted to the live loop's own chain positions, which is the pair the 2 pp clause compares. Not the published offline bar, which scores every position and gives 3.19 tokens/pass on this pack (engine#805) -- a different position sample, not a better measurement.
  • 27b-mtp-tau06-offline-tokens-per-pass-20261106 — 2026-11-06T22:06:22Z; source: tests/fixtures/mtp_tau06/p0..p7.txt, verbatim extracts of npu-hold tickets tau06 p0..p7 (20261106-164254/165612/170407/172013/172718/173228/174605/175540), one prompt per ticket, tau=0.6; 8 prompts x 64 tokens, k=4, tau=0.6 (the draft route's DEFAULT), one prompt per npu-hold ticket; greedy text byte-identical to plain decode on 8 of 8 prompts, 0 stream mismatches, 0 amdxdna errors during any arm. The window is the union of eight single-prompt tickets, not eight repeats: each prompt was measured once, and prompt 0 was measured twice (16:29 and 16:42) with identical counts 31/35, the only repeat in the set. 1 + 291/277 from the offline chains at the loop's own positions. The offline protocol keeps its own chain gate, so it yields more per pass than the live loop at the same tau; that difference is the loop, not the drafter.
  • 27b-mtp-tau06-tokens-per-pass-20261106 — 2026-11-06T22:06:22Z; source: tests/fixtures/mtp_tau06/p0..p7.txt, verbatim extracts of npu-hold tickets tau06 p0..p7 (20261106-164254/165612/170407/172013/172718/173228/174605/175540), one prompt per ticket, tau=0.6; 8 prompts x 64 tokens, k=4, tau=0.6 (the draft route's DEFAULT), one prompt per npu-hold ticket; greedy text byte-identical to plain decode on 8 of 8 prompts, 0 stream mismatches, 0 amdxdna errors during any arm. The window is the union of eight single-prompt tickets, not eight repeats: each prompt was measured once, and prompt 0 was measured twice (16:29 and 16:42) with identical counts 31/35, the only repeat in the set. 1 + 230/277. Below the 3.2-4.1 break-even measured for this loop, so the default tau is a SLOWDOWN: 0.81x on the prompt-0 pair ticket (2026-11-06 16:29).
  • 27b-mtp-tau06-depth3-20261106 — 2026-11-06T22:06:22Z; source: tests/fixtures/mtp_tau06/p0..p7.txt, verbatim extracts of npu-hold tickets tau06 p0..p7 (20261106-164254/165612/170407/172013/172718/173228/174605/175540), one prompt per ticket, tau=0.6; 8 prompts x 64 tokens, k=4, tau=0.6 (the draft route's DEFAULT), one prompt per npu-hold ticket; greedy text byte-identical to plain decode on 8 of 8 prompts, 0 stream mismatches, 0 amdxdna errors during any arm. The window is the union of eight single-prompt tickets, not eight repeats: each prompt was measured once, and prompt 0 was measured twice (16:29 and 16:42) with identical counts 31/35, the only repeat in the set. 2/277 chains accepted draft 3; depth 4 was never offered by any chain, so it is unmeasured (the tool refuses to score 0/0).
  • 27b-t2s2-decode-20261106 — 2026-11-06T20:53:40Z; source: ~/evidence/tmoe-npu/587-27b-row-20261106/r2 (this session; per-request JSON kept), task bc791346c/b9e132438; 3.45 tok/s. One 13-token prompt, one box, one serve, no batching; the served route reported flm_prefix_cache: no slot shares a prefix on every request, so no prefix cache is inside these numbers. The two slopes agreeing is what makes this a decode figure rather than a prefill+decode mix. The model answers in reasoning_content (content is empty at these budgets), so the token counts, not the text, are what is measured. Earlier rungs on the same container, for context: 63.1 s/token int8 (27b-npudx-20261027) and 1.72 s/token on the per-role PB16i set (engine#697).
  • 27b-decode-cot-20261107 — 2026-11-07T13:49:36Z; source: ~/evidence/tmoe-npu/587-gsm8k-20261107/transcript-40.json; Context sensitivity, measured rather than assumed: the same route is 0.2902 s/token on a 13-token prompt (27b-t2s2-decode-20261106) and 0.3857 s/token over 106-512-token answers, i.e. ~1.33x - so quoting the short-prompt figure as an end-to-end rate would be wrong by a third. This figure also carries each request's prefill (~60-100 prompt tokens at 19.55 tok/s, a few seconds of the ~127 s per question), which is why it is labelled end-to-end per token rather than a kernel-stream rate.
  • 27b-t2s2-first-token-20261106 — 2026-11-06T20:52:41Z; source: ~/evidence/tmoe-npu/587-27b-row-20261106/r2 warm_1 (this session), task b9e132438; Cold only by construction: the serve reports healthy before it loads, so this is load + prefill + one token. Warm 1-token requests on the same prompt are 0.9537 s. Compare the int8 rung's 145.7 s (27b-first-token-after-transpose-20261027) and 435.9 s before the tiled transpose.
  • 27b-prefill-batched-20261027 — 2026-10-27T21:33:22Z; source: ~/lb/587-gbnos.sh run log 587-gbnos-server-20261027T213012Z.log; prompt 'The capital of France is' -> ' Paris'; sequential baseline for the same request was 2298 submits / 145.7 s; device wait is 26 ms per submit, so cost tracks submit count
  • 27b-npudx-20261027 — 2026-10-27T14:40:00Z; source: ~/lb/587-npudx-smoke.sh run log on strixhalo; prompt 'The capital of France is' -> ' Paris.\nThe capital of Germany is'; upstream add-on, no local patch
  • 27b-localaddon-20261027 — 2026-10-27T14:20:00Z; source: ~/lb/587-noxrt-smoke-20261027T101012Z.log; same text as the npu-dx row; kept because it is the pair that established the add-on does not move decode
  • 27b-first-token-after-transpose-20261027 — 2026-10-27T20:15:00Z; source: ~/lb/587-trace-20261027T201250Z/bpf.log (37 507 driver events); same trace, binary built from main 800cf6f (#622): device wait 59.8 s (2 298 x 26 ms), host 83.7 s, one gap above a second
  • 27b-first-token-before-transpose-20261027 — 2026-10-27T17:56:00Z; source: ~/lb/587-trace-20261027T175645Z/bpf.log (37 504 driver events); 362.9 s of this is host time with no ioctl in flight; 220.9 s of it in 120 gaps of more than a second, all before a single 89.13/73.40/52.43 MB weight BO was created

View this page's source on GitHub ↗