Apply a kernel hotfix to RHCOS with out-of-cluster image mode

Ship a kernel errata to RHCOS nodes before it lands in an OpenShift z-stream, by building a custom layered image off-cluster and pointing a MachineConfig at it with osImageURL.

Reference: Image mode for OpenShift. Upstream projects: machine-config-operator, rpm-ostree, bootc.

Prerequisites

• oc logged in with cluster-admin.
• An entitled RHEL host matching the RHCOS base (same major/minor, EUS repo if applicable) with dnf and podman.
• The cluster pull secret saved as pull-secret.json (needed to pull the RHCOS base image).
• A container registry reachable from the nodes, and push credentials in registry-auth.json.

1. Identify the base image and kernel packages

The RHCOS image of the running release must be the FROM of the custom build.

oc adm release info --image-for rhel-coreos
oc debug node/<worker> -- chroot /host sh -c 'uname -r; rpm -qa "kernel*" | sort'

Typical x86_64 set: kernel, kernel-core, kernel-modules, kernel-modules-core, kernel-modules-extra.

2. Download the fixed RPMs

Set the target kernel version and RHEL release, then download only the subpackages already installed on the nodes.

export KVER=5.14.0-<release>.el9_<y>
export RHEL_REL=9.<y>
mkdir -p build/rpms && cd build/rpms
dnf download --releasever=${RHEL_REL} --repo rhel-9-for-x86_64-baseos-eus-rpms \
  kernel{,-core,-modules,-modules-core,-modules-extra}-${KVER}
sha256sum *.rpm

Compare the checksums with the values on the erratum page.

3. Write the Containerfile

Create build/Containerfile. The bind mount keeps the RPMs out of the image layers.

FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:<rhcos-digest>
RUN --mount=type=bind,source=rpms,target=/tmp/rpms,relabel=shared \
    rpm-ostree override replace /tmp/rpms/*.rpm && \
    rpm-ostree cleanup -m && \
    bootc container lint

4. Build and push the image

Build on an x86_64 host (or add --platform linux/amd64) and record the pushed digest.

cd build
podman build --authfile pull-secret.json -t <registry>/<org>/rhcos-kernel-fix:${KVER} .
podman push --authfile registry-auth.json --digestfile digest.txt \
  <registry>/<org>/rhcos-kernel-fix:${KVER}
podman run --rm <registry>/<org>/rhcos-kernel-fix:${KVER} rpm -q kernel-core
cat digest.txt

5. Grant nodes pull access

For a private repository, add a read-only robot account to the global pull secret.

oc get secret/pull-secret -n openshift-config \
  --template='{{index .data ".dockerconfigjson" | base64decode}}' > global-pull.json
oc registry login --registry=<registry>/<org> \
  --auth-basic='<robot-user>:<robot-token>' --to=global-pull.json
oc set data secret/pull-secret -n openshift-config \
  --from-file=.dockerconfigjson=global-pull.json

6. Apply the MachineConfig

Reference the image by digest, never by tag. Test on a one-node custom pool first, and pause the pool (spec.paused: true) to control reboot timing.

cat <<EOF | oc apply -f -
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-kernel-hotfix
spec:
  osImageURL: <registry>/<org>/rhcos-kernel-fix@sha256:<pushed-digest>
EOF
oc get mcp -w

The MCO cordons, drains and reboots each node in turn. Create one MachineConfig per pool (master, worker, custom pools).

7. Remove before the next upgrade

Nodes on an out-of-cluster image no longer follow the release RHCOS image. Once a z-stream ships the fixed kernel, delete the MachineConfig before upgrading, or rebuild on the new RHCOS digest and update osImageURL. Otherwise override replace can pin or downgrade the kernel.

oc delete mc 99-worker-kernel-hotfix
oc get mcp -w

Verify

oc get mcp worker
oc debug node/<node> -- chroot /host sh -c 'uname -r; rpm-ostree status'

Expected output:

5.14.0-<release>.el9_<y>.x86_64
* ostree-unverified-registry:<registry>/<org>/rhcos-kernel-fix@sha256:<pushed-digest>

A pool stuck Degraded with an image pull error points to the global pull secret (step 5).