The Zyphra family: Zamba2, Zamba v1 and BlackMamba#

The three Zyphra architectures the engine carries — zamba2, zamba (v1, with its MoE) and blackmamba (Mamba-1 with alternating MoE) — what their files weigh at the current pin, which quantization to publish for each, and the two things that will silently go wrong if you convert or quantize them with an older tree.

The architecture work is in Models measured rows and in the fork; this page is the file-level record. Engine issue: #1036 (the Zamba2 sharing work) and #991 (HRX placement classes).

File sizes: a Zamba2 GGUF is now the size of its checkpoint#

Zamba2 checkpoints store the transformer block once per shared block (two of them at 2.7B and 7B) and hold each hybrid layer's MLP difference in a rank-128 adapter pair. Converting used to expand both — the block once per hybrid layer, and each layer's adapter merged into a full FFN matrix — so a 7B BF16 GGUF came out at 20.45 GiB against a 14.82 GB checkpoint. Two changes fixed that, and both are in the pin:

Both forms load from one binary. A GGUF written before these changes keeps taking the merged path — the loader requires ffn_up per layer when the file carries no zamba2.ffn_gate_up_adapter_rank key, and reads the low-rank pair when it does. Nothing published before the change stops working.

The two forms are not bit-identical, and that is expected: the merged file has up @ down computed in f32 at conversion time and rounded to bf16, while the adapter path evaluates the pair at run time. Measured against the merged form's own logits (CPU, 16 × 512):

Both are two to three orders of magnitude below the family bar below, so a quantized arm still has to be re-measured rather than inherited from the merged form — see the next section, where one arm changed sides.

What to publish per model#

The family bar is mean KLD ≤ 0.03 AND same-top-1 ≥ 95 %, and each model gets a sweet spot, a max-quality and a smallest-usable pick. The Zamba2 rows are the adapter form; the others are unchanged.

† 32-chunk run (32 × 512 = 16384 tokens), because the 16-chunk run sat too close to the bar to decide; every other row is a 16-chunk run. Mark the long rows when quoting them.

Why the 7B moves up a tier. The adapter form perturbs top-1 in proportion to how close an arm already sat to the bar, and the two Zamba2 models reacted differently to the same change: the 2.7B's top-1 moved up 0.05 points, the 7B's down 0.52. Q5_K_M was clearing the 95 % bar by 0.025 points on the merged form, so it no longer clears it (94.792 %), and Q6_K — 5829 MiB, still 11-12 % smaller than the merged Q5_K_M — is the smallest arm that does. Q5_K_M is listed above as withdrawn rather than deleted because it is the smaller file for anyone who accepts the lower top-1 deliberately.

Two things that go wrong with an older tree#

1. A quantizer that predates the F16 default will drop the adapter off HRX. The adapter pair stays at F16 by default, because its B contraction has input_size = rank = 128: the float weight path admits that (input_size ≥ 4 && % 4 == 0), the block formats do not (their block layout needs ≥ 256). Quantize the pair with an older llama-quantize and that matmul leaves the device on every hybrid layer — measured on a 2.7B Q5_K_M as 9 of 3051 nodes on the CPU and pp512 463.66 vs 589.47 tok/s (0.79x, -21 %), with no error to tell you. If your tree predates the default, pass it explicitly:

llama-quantize --tensor-type ffn_gate_up_adapter=f16 <in.gguf> <out.gguf> Q6_K

Cost of the F16 pair: +31.8 MiB on an 1804.93 MiB 2.7B file. An explicit --tensor-type still wins; --pure still means pure.

2. Assert the layout before you trust a PPL. A pair written transposed ({rank, n_embd} instead of {n_embd, rank}) is refused by the loader with check_tensor_dims, but only after a conversion; the cheap check is on the container: ffn_up ×2 (not one per hybrid layer), zamba2.ffn_gate_up_adapter_rank = 128, adapters 9 + 9 at 2.7B and 13 + 13 at 7B, and the byte sizes in the table above.

HRX placement, if you are quoting a number#

The adapter path adds two MUL_MATs and an ADD per hybrid layer and all of them are placed on HRX — +27 nodes at 2.7B (9 × 3), with the same off-device residue as the merged form, and GGML_HRX_STRICT=1 exits 0 on both forms at prefill and decode. There is no throughput cost to report: pp512 441.98 → 440.57 and tg64 22.87 → 22.60 on one binary with both forms (0.997 / 0.988).

One caveat for anyone running -ngl 99 today: the token-embedding GET_ROWS is currently not on HRX (1 of 3024 nodes, GET_ROWS x 1, 2 graph splits) after the host-pointer weight import landed — that is base-level and affects both forms equally, tracked as engine #1316, not in this family's changes

View this page's source on GitHub ↗