Kernel readiness: moving the box to a new kernel#

The Strix Halo box runs a kernel we build, not the distribution's: everything the engine and the NPU need is pinned by hand, so the next kernel is a step list, not an investigation. This page is that list, with the facts that go stale marked as facts and the commands that do not.

• Runs today: 7.4.0-rc1-1bit — Linux v7.4-rc1, in use since 2026-11-04 05:45 (/boot/vmlinuz-7.4.0-rc1-1bit); every boot since has loaded it, and the box reboots on each Puppy board cycle, so read uptime -s for the boot you are looking at. PerfOpt is present in it, measured: this box's boot prints amdgpu 0000:c5:00.0: AMD-Vi: PerfOpt armed on IOMMU0. The 7.3 A/B below stays as the measurement that motivated the backports. The compiler is not what this page says — see the "config of record" bullet below.
• Why the backports: with the GPU in its identity domain, PerfOpt is worth real prefill on HRX and costs nothing at decode. Qwen3.8-27B UD-Q4_K_XL on HRX0, same build pin, five repeats:

So the gain is PerfOpt itself, not the compiler (the off arm matches the base), it is prefill only, and greedy output is byte-identical across the three arms. - Out-of-tree modules that must follow every kernel: amdxdna (the NPU driver, NPU engine, "The XDNA driver") and ec_su_axb35 (the box's fan EC). Both live in /lib/modules/<version>/updates/ and both must be rebuilt against the new kernel and signed. Both are also packaged as dkms packages (amdxdna-1bit-dkms, ec-su-axb35-dkms) which do that rebuild on any machine that installs them — scripts/mk-kernel-packages.sh builds the set for deb, rpm and Arch, and The engine on any Linux, "Kernel and NPU driver packages", is the install. The board's other firmware-side quirk is not a module at all: three of its four xHCI controllers park in D3cold, so the rear USB-A and both Type-C ports never enumerate, and the fix that ships for it is a udev rule — written up, with the ACPI evidence and the one-command check, in AXB35-02 (Bosgame BeyondMax): the rear USB-A and Type-C ports.

7.4 on the box: what changed, and what to watch#

v7.4-rc1 is the first tag where the PerfOpt series is upstream (iommu/amd plus the amdgpu identity-domain enable), so nothing is carried: the tree is the plain tag, and the boot still reports AMD-Vi: PerfOpt armed on IOMMU0. Built from the same config with CONFIG_DEBUG_INFO_BTF=y (pahole 1.31, ~5 MiB of BTF in the image), which is also why the bpf-restrict-fs: Failed to load BPF object boot noise is gone — systemd now logs LSM BPF program attached. Compiler is gcc 15.2.0, the same as the 7.3 image; older notes that say clang were wrong.

• Throughput, 7.4 arm only: Qwen3.8-27B UD-Q4_K_XL on HRX0, five repeats, at the current 90/90/105 W fan curve — pp512 239.6, pp2048 234.7, tg128 10.9 tok/s (medians). These do not line up with the 7.3 table above, which was taken on a different machine and power profile; the 7.3 arm of this recipe cannot be taken from inside a 7.4 boot (it needs its own one-shot grub-reboot window), so no 7.3-vs-7.4 delta is claimed. Greedy output is byte-identical run to run, and the NPU known-answer probe is bit-identical against its fp64 reference.
• Watched, not caused by 7.4: the root NVMe logs AMD-Vi: Event logged [IO_PAGE_FAULT] under block I/O (read-only translations, no nvme error / timeout / reset, no filesystem error, smartctl clean) — a device-side read-after-unmap class of noise, monitored rather than normalised.
• Watched, and specifically not a 7.4 regression: a MES REMOVE_QUEUE timeout that ends in a MODE2 GPU reset. The same chain is recorded on the 7.3 kernel four days earlier, and the MES wait budget is unchanged between the two tags; on this box it has appeared with system memory exhausted and several resident GPU clients, which is the regime to keep the box out of.
• The two out-of-tree modules were rebuilt against 7.4 and signed the same way, and the NPU path loads our amdxdna build, not the kernel's own copy — the installed updates/ module wins the resolution (modinfo -n amdxdna), which is the check that proves it after a reboot.

What "ready" means#

1. an upstream tag to move to, and whether the PerfOpt series is native in it (then the backports are dropped, not re-applied);
2. a kernel built from our config on another machine, packaged as .deb;
3. kernel and modules signed with the box's own MOKs;
4. the box boots it, the two out-of-tree modules load, mokutil agrees;
5. the hardware gate passes on it;
6. the weekly package and the 1bit OS image pick it up.

0. Facts first#

git ls-remote --tags https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git | grep 'v7\.4'
# is our series already upstream in that tag?
grep -ril perfopt drivers/iommu/amd/      # a hit = drop the backports

The 2026-10-23 check: no v7.4 tag (v7.3 final was tagged 2026-10-18, the 7.4 merge window was still open, so rc1 lands about two weeks later), no v7.3.y stable either, and perfopt was not in v7.3's drivers/iommu/amd/ — the series is queued for 7.4, so the backports stay until then. Both backport commits (the iommu/amd one and the drm/amdgpu one that enables PerfOpt on iGPUs in the identity domain) applied to v7.3 cleanly, needing no rebase.

1. Build — never on the box#

The box is for running; a build there heats the die that a model is loading on. Build on a quieter machine with the box's config.

git clone --depth 1 --branch <tag> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/wt/k74/<tag>
cd ~/wt/k74/<tag>
cp <box>:/boot/config-$(uname -r of the box) .config      # or the config of record, see below
scripts/config --file .config --set-str LOCALVERSION "-1bit-perfopt"
# only while the tag lacks the series: our patch files are plain `diff --git` (a prose paragraph, then
# the diff), so `git am` refuses them with "Patch format detection failed" - strip to the first diff
# header and apply, then commit so the tree is clean (a dirty tree makes setlocalversion append its own
# '+', and the release string becomes -1bit-perfopt++):
sed -n '/^diff --git /,$p' iommu-amd-perfopt.patch | git apply -
git commit -qam "iommu/amd: the IOMMU PerfOpt backports"
make LLVM=1 olddefconfig
export LLVM=1                    # dpkg-buildpackage re-enters make: export it, do not just pass it
nice -n 19 make LLVM=1 -j12 bindeb-pkg
# and before either .deb is published: drop the one build artefact the headers package carries that nothing reads,
# so the published package stops naming the builder's home path and build tree (engine#1487)
./scripts/prune-kernel-headers-deb.sh ../linux-headers-<rel>_<ver>_amd64.deb ../linux-headers-<rel>_<ver>_amd64.deb

• The config of record is the one the box's running kernel was built from (/boot/config-$(uname -r)): CONFIG_AMD_IOMMU=y, CONFIG_DRM_AMDGPU=m, CONFIG_MODULE_SIG{,_ALL,_SHA256}=y, CONFIG_DEBUG_INFO_NONE=y. It is not CONFIG_CC_IS_CLANG=y. Read off the running kernel on 2026-11-05, that file says CONFIG_CC_IS_GCC=y with CONFIG_GCC_VERSION=150200 and CONFIG_CLANG_VERSION=0; so does /proc/config.gz; and so does the kernel's own version line, Linux version 7.4.0-rc1-1bit (bcloud@strixhalo) (gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, …). The 7.3.0-rc4-1bit-perfopt+ image it replaced reports the same three ways. So the LLVM=1 recipe below is not what built the kernels this box has booted — which is precisely the failure the next bullet warns about. Whether the recipe or the images should change is a kernel-lane decision, not a docs edit, so this bullet records the measurement and stops there. Between the base 7.3.0-rc4-1bit config and -perfopt the only differences were the compiler version and debug info (28 lines, nothing workload-specific).
• bindeb-pkg needs debhelper, libelf-dev and the LLVM tools (llvm-21, lld-21); this recipe builds the kernel with clang, so a gcc-only host silently builds the wrong compiler unless LLVM=1 is exported — and the images this box has booted show that failure rather than the recipe (see the config-of-record bullet above). On a host with the versioned tools only, put /usr/lib/llvm-21/bin on PATH (that is where the unversioned llvm-objcopy the modules step calls lives).
• Artefacts land in the parent directory: linux-image-<version>_<arch>.deb and linux-headers-<version>_<arch>.deb.
• make install on the box also works (that is how the current kernel was installed before the debhelper path existed), but the .deb route is what the weekly package expects and it puts the modules where depmod and the module-signing hook see them.

2. Sign — two keys, on the box#

The box has two machine-owned MOKs enrolled, and they are not interchangeable. Whether Secure Boot itself is on is the owner's setting in firmware (on 2026-10-23 the box booted with it off, after a firmware reset); signing does not depend on it, and skipping it means the next boot with Secure Boot on refuses the kernel.

# kernel: automatic on install (this is what the hook does), or by hand
sudo sbsign --key /var/lib/shim-signed/mok/MOK-Kernel.priv \
            --cert /var/lib/shim-signed/mok/MOK-Kernel.pem \
            --output /boot/vmlinuz-<version> /boot/vmlinuz-<version>
# modules we built ourselves (in-tree modules carry the build-time key already)
sudo kmodsign sha512 /var/lib/shim-signed/mok/MOK.priv \
                       /var/lib/shim-signed/mok/MOK.der updates/amdxdna.ko

• /etc/kernel/postinst.d/z0-mok-sign signs any kernel image the installer did not sign, and does nothing when sbverify --list already shows a signature.
• Trap: a MOK whose EKU is module-signing only (Ubuntu's default style) is rejected by shim for kernel images. That is why there are two keys; signing a vmlinuz with MOK.priv produces a kernel that will not boot under Secure Boot.
• Trap: an unsigned copy left in /boot as vmlinuz-*.unsigned becomes a GRUB menu entry. Keep originals outside /boot (/var/backups/kernels/).
• A BIOS/firmware reset can drop the enrolled MOKs. The fix needs the owner at the console (MokManager), and can be primed over ssh beforehand: mokutil --import /var/lib/shim-signed/mok/MOK-Kernel.der, then the same for MOK.der, then the owner confirms both in MokManager on the next boot.

3. Install — on the box#

sudo dpkg -i linux-image-<version>_<arch>.deb linux-headers-<version>_<arch>.deb
sudo depmod -a <version>
ls /lib/modules/<version>/updates/            # the out-of-tree modules for THIS version go here

Removing a kernel is deleting /boot/*<version>* and /lib/modules/<version> (the .deb route also offers dpkg -r linux-image-<version>). Keep the Canonical fallback kernel installed: it is the recovery path when a new kernel does not reach a console.

4. Boot entry — without changing the default#

update-grub (the installer runs it) adds the new kernel under "Advanced options for Ubuntu". Leave GRUB_DEFAULT pointing at the kernel that is running: the owner chooses when the box moves, and the previous kernel stays as the fallback entry.

5. Reboot — the owner, with a console#

The build and install steps are remote; the reboot is not. Have console access before it (MokManager is a firmware screen), and expect one extra reboot if a MOK has to be enrolled.

6. Verify, in this order#

uname -r                                              # the new version
sudo mokutil --sb-state                               # SecureBoot enabled/disabled (owner's setting)
sudo mokutil --list-enrolled | grep -i 1bit           # both machine MOKs still enrolled
sudo sbverify --list /boot/vmlinuz-$(uname -r)        # "signature 1" + our issuer
for f in /lib/modules/$(uname -r)/updates/*.ko; do modinfo -F signer "$f"; done
sudo dmesg | grep -iE 'amdgpu|iommu|amdxdna' | grep -iE 'fail|error|timeout|fault'   # must be empty

Then the two box-specific gates: the IOMMU PerfOpt line (PerfOpt armed on IOMMU0, or amdgpu.iommu_perfopt=0 when it should be off) and the NPU driver — amdxdna loaded at the pinned version and the NPU visible to xrt-smi (NPU engine).

7. The hardware gate#

A boot is not a proof. scripts/hw-gate.sh (and the hw-gate label / hw-gate.yml) runs the six checks that have caught the real regressions — ops, fa-decode-q8, kat, kld, needle, placement — and it runs them on the box's new kernel, so a kernel that changes numerics, times out on a long prompt or moves a node off HRX fails there rather than in a customer's run. Details: Hardware gate.

8. Packages and the 1bit OS image#

• The weekly release preflight needs a readable kernel image: /boot/vmlinuz-$(uname -r) is root-only on Ubuntu, so pass VMLINUZ=<copy> (Releases, "The release run").
• 1bit OS takes the kernel as woof-CE's own kernel package: os/v3/puppy/mk-huge-kernel.sh writes huge-<kver>.tar.zst (vmlinuz-<kver> plus the module zdrv, both from the signed tree), and os/v3/puppy/build.sh builds the Puppy appliance with it inside, so the appliance boots our kernel with Secure Boot left on instead of asking for it to be switched off (os/v3/puppy/README.md).
• The BusyBox os/mini image is separate and is not 1bit OS (it is the CI/VM target): os/mini/mkimage.sh builds its unified kernel image from the same vmlinuz, and with MOK_KEY + MOK_CERT signs that UKI with the same kernel key.
• Record the kernel version in the release notes: a box that moves kernel is a release-visible change, and the gate result belongs next to it.

Checklist#

• [ ] tag chosen; is the PerfOpt series native (drop the backports) or not (re-apply, and note any rebase)?
• [ ] config of record copied from the box, olddefconfig clean, no prompts
• [ ] bindeb-pkg produced image + headers .deb with the intended LOCALVERSION
• [ ] the headers .deb went through scripts/prune-kernel-headers-deb.sh: no scripts/mod/*.s, and the only files still carrying a /home/... path are the two host tools and upstream's own example text
• [ ] amdxdna and ec_su_axb35 rebuilt against the new /lib/modules/<version>/build
• [ ] kernel signed by MOK-Kernel, modules by MOK, both verified with sbverify/modinfo -F signer
• [ ] fallback kernel and GRUB_DEFAULT untouched
• [ ] owner reboots with console access; MOKs still enrolled
• [ ] hardware gate passes on the new kernel
• [ ] weekly package and OS image built from the new kernel

Rehearsal, 2026-10-23#

No v7.4-rc existed yet, so this was rehearsed twice: against v7.3 (the newest tag) with the backports re-applied, and against linux-next next-20261022, which is the 7.4 merge window's content (Makefile says 7.3.0 + next patches) and is therefore the honest preview of 7.4-rc1. Both were built on a separate 16-core machine with the config of record, LLVM=1 (clang 21.1.8), nice -n 19 make -j12 bindeb-pkg.

Facts#

So the rule the checklist asks for resolves itself: on the 7.4 path the backports are dropped, on a 7.3-based kernel they are re-applied (and needed no rebase).

Builds#

olddefconfig was clean on both. Its config delta is the part to read before adopting a tag: 28 lines on v7.3 (clang version and debug info only), 110 lines on next-20261022 — new clang-21/7.4 symbols (CC_HAS_COUNTED_BY, CC_OPT_*, BPF_CRYPTO, …) plus options our config had that the newer tree no longer offers (ANDROID_BINDER*, ARCH_HAS_DEBUG_WX, ARCH_SUPPORTS_PER_VMA_LOCK). None of them is needed by the engine or the NPU, but the diff is where a silently dropped feature shows up.

The out-of-tree module#

amdxdna (amd/xdna-driver at the engine's submodule pin) builds against both trees, MAKE_RC=0, producing amdxdna.ko — no 7.4 API break in the 7.4 merge-window content. That is the single most likely break in this whole path, and it is not there yet; it is worth re-running the moment v7.4-rc1 is tagged, because next is a moving target.

One trap the driver's own Makefile sets: it exports an empty ARCH into the kernel's configure_kernel.sh probe, and the kernel then stops with scripts/Makefile.clang: add '--target=' option. Pass ARCH=x86_64 on the make line:

make -f drivers/accel/amdxdna/Makefile KERNEL_SRC=<kernel tree> ARCH=x86_64 \
     BUILD_ROOT_DIR=<out> LLVM=1 all

Traps that cost time, in the order they bite#

Not rehearsed here#

Installing, signing, booting and gating on the box (that is the owner's reboot), the fan-EC module (its source lives only on the box), and any throughput re-measurement on the new kernel. Those are steps 2 to 8 above, and the hardware gate is what makes the result more than "it booted".

View this page's source on GitHub ↗