# Apply a kernel hotfix to RHCOS with out-of-cluster image mode


Ship a kernel errata to RHCOS nodes before it lands in an OpenShift z-stream, by building a custom layered image off-cluster and pointing a **MachineConfig** at it with `osImageURL`.

Reference: [Image mode for OpenShift](https://docs.redhat.com/en/documentation/openshift_container_platform/latest/html/machine_configuration/mco-coreos-layering). Upstream projects: [machine-config-operator](https://github.com/openshift/machine-config-operator), [rpm-ostree](https://github.com/coreos/rpm-ostree), [bootc](https://github.com/bootc-dev/bootc).

## Prerequisites

- `oc` logged in with **cluster-admin**.
- An entitled RHEL host matching the RHCOS base (same major/minor, EUS repo if applicable) with `dnf` and `podman`.
- The cluster pull secret saved as `pull-secret.json` (needed to pull the RHCOS base image).
- A container registry reachable from the nodes, and push credentials in `registry-auth.json`.

{{< admonition warning >}}
If a z-stream release already ships the fixed kernel, run a regular cluster update instead. It is simpler and fully supported.
{{< /admonition >}}

## 1. Identify the base image and kernel packages

The RHCOS image of the running release must be the `FROM` of the custom build.

```bash
oc adm release info --image-for rhel-coreos
oc debug node/<worker> -- chroot /host sh -c 'uname -r; rpm -qa "kernel*" | sort'
```

Typical x86_64 set: `kernel`, `kernel-core`, `kernel-modules`, `kernel-modules-core`, `kernel-modules-extra`.

## 2. Download the fixed RPMs

Set the target kernel version and RHEL release, then download only the subpackages already installed on the nodes.

```bash
export KVER=5.14.0-<release>.el9_<y>
export RHEL_REL=9.<y>
mkdir -p build/rpms && cd build/rpms
dnf download --releasever=${RHEL_REL} --repo rhel-9-for-x86_64-baseos-eus-rpms \
  kernel{,-core,-modules,-modules-core,-modules-extra}-${KVER}
sha256sum *.rpm
```

Compare the checksums with the values on the erratum page.

{{< admonition note >}}
`rpm-ostree override replace` fails on packages absent from the base image (for example `kernel-devel`). Do not add extra subpackages.
{{< /admonition >}}

## 3. Write the Containerfile

Create `build/Containerfile`. The bind mount keeps the RPMs out of the image layers.

```dockerfile
FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:<rhcos-digest>
RUN --mount=type=bind,source=rpms,target=/tmp/rpms,relabel=shared \
    rpm-ostree override replace /tmp/rpms/*.rpm && \
    rpm-ostree cleanup -m && \
    bootc container lint
```

{{< admonition warning "SELinux build hosts" >}}
Without `relabel=shared`, the mounted directory is unreadable inside the build container on Fedora/RHEL hosts, and the build fails with `error: Handling argument /tmp/rpms/*.rpm: No such file or directory (os error 2)`.
`FROM configs AS final` only applies to on-cluster builds. Out-of-cluster builds use the RHCOS digest from step 1.
{{< /admonition >}}

## 4. Build and push the image

Build on an x86_64 host (or add `--platform linux/amd64`) and record the pushed digest.

```bash
cd build
podman build --authfile pull-secret.json -t <registry>/<org>/rhcos-kernel-fix:${KVER} .
podman push --authfile registry-auth.json --digestfile digest.txt \
  <registry>/<org>/rhcos-kernel-fix:${KVER}
podman run --rm <registry>/<org>/rhcos-kernel-fix:${KVER} rpm -q kernel-core
cat digest.txt
```

## 5. Grant nodes pull access

For a private repository, add a read-only robot account to the global pull secret.

```bash
oc get secret/pull-secret -n openshift-config \
  --template='{{index .data ".dockerconfigjson" | base64decode}}' > global-pull.json
oc registry login --registry=<registry>/<org> \
  --auth-basic='<robot-user>:<robot-token>' --to=global-pull.json
oc set data secret/pull-secret -n openshift-config \
  --from-file=.dockerconfigjson=global-pull.json
```

## 6. Apply the MachineConfig

Reference the image by digest, never by tag. Test on a one-node custom pool first, and pause the pool (`spec.paused: true`) to control reboot timing.

```bash
cat <<EOF | oc apply -f -
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-kernel-hotfix
spec:
  osImageURL: <registry>/<org>/rhcos-kernel-fix@sha256:<pushed-digest>
EOF
oc get mcp -w
```

The MCO cordons, drains and reboots each node in turn. Create one **MachineConfig** per pool (`master`, `worker`, custom pools).

## 7. Remove before the next upgrade

Nodes on an out-of-cluster image no longer follow the release RHCOS image. Once a z-stream ships the fixed kernel, delete the **MachineConfig** before upgrading, or rebuild on the new RHCOS digest and update `osImageURL`. Otherwise `override replace` can pin or downgrade the kernel.

```bash
oc delete mc 99-worker-kernel-hotfix
oc get mcp -w
```

{{< admonition tip "Special pools" >}}
Pools with `kernelType: realtime` or 64k pages need the matching `kernel-rt*` or `kernel-64k*` RPMs (layering RT kernel RPMs is not recommended). Other architectures need a dedicated build.
{{< /admonition >}}

## Verify

```bash
oc get mcp worker
oc debug node/<node> -- chroot /host sh -c 'uname -r; rpm-ostree status'
```

Expected output:

```text
5.14.0-<release>.el9_<y>.x86_64
* ostree-unverified-registry:<registry>/<org>/rhcos-kernel-fix@sha256:<pushed-digest>
```

A pool stuck `Degraded` with an image pull error points to the global pull secret (step 5).