Posts Tags Categories About

Contents

Hosted control planes: TPM2-bound LUKS on agent workers

 included in Openshift
   1655 words   8 minutes 

Lab walkthrough: a Hosted Control Planes (HCP) cluster on the Agent platform with a worker VM on OpenShift Virtualization (persistent vTPM, static IPv4). Two independent customizations are applied from the hub, each in its own NodePool config ConfigMap:

  • LUKS disk encryption: the worker root disk is encrypted with LUKS and unlocked by the TPM 2.0.
  • core password: a console password for the core user, on the installed node and, separately, on the discovery image.

Either one can be used without the other.

Acronyms: HCP = Hosted Control Planes, LUKS = Linux Unified Key Setup, TPM = Trusted Platform Module, MCE = multicluster engine for Kubernetes, CAPI = Cluster API.

Lab repository: github.com/mmayeras/hcp_luks

Every manifest below lives in https://github.com/mmayeras/hcp_luks, with placeholders for all environment values:

  • manifests/: HostedCluster, NodePool, NodePool config ConfigMaps, InfraEnv, NMStateConfig, agent Role.
  • vm/: worker VM and discovery ISO DataVolume.
  • butane/: Butane sources of both MachineConfigs.
  • README.md: placeholder table and the full apply order.

HCP is built on the upstream HyperShift project, the Agent platform on assisted-service, disk encryption on Ignition and Clevis, and the worker VM on KubeVirt.

Topology

  Hub (OpenShift 4.22, MCE 2.17, OpenShift Virtualization, ODF)
  +--------------------------------------------------------------------+
  | ns clusters            HostedCluster hcp-luks, NodePool,           |
  |                        ConfigMap hcp-luks-worker-luks-tpm2         |
  |                        ConfigMap hcp-luks-worker-core-password     |
  | ns clusters-hcp-luks   hosted control plane pods + ignition server |
  | ns <AGENT_NAMESPACE>   InfraEnv, NMStateConfig, Agent,             |
  |                        VM hcp-luks-worker-0 (persistent vTPM)      |
  +--------------------------------------------------------------------+
            discovery ISO -> Agent -> install -> first boot from disk:
            Ignition encrypts root, seals the key in the vTPM

Key point: the NodePool configuration is rendered by the HCP ignition server and applied by Ignition on the first boot from disk. The discovery image never shows it.

Prerequisites

  • MCE with HyperShift and the Agent (assisted installer) components enabled on the hub.
  • OpenShift Virtualization installed, with spec.vmStateStorageClass set in the HyperConverged CR (filesystem volume mode, required by the persistent vTPM).
  • A LoadBalancer provider (MetalLB) for the hosted API server.
  • A NetworkAttachmentDefinition on the machine network (bridge or OVN localnet).
  • oc, virtctl, butane, jq available in $PATH.

The commands below use the agent namespace (InfraEnv, Agent, VM) from a variable:

AGENT_NS=<AGENT_NAMESPACE>

1. LUKS: encrypt the root disk with the TPM

Butane translates boot_device.luks.tpm2 into Ignition storage.luks and storage.filesystems entries:

cat > worker-luks-tpm2.bu <<EOF
variant: openshift
version: 4.18.0
metadata:
  name: 99-worker-luks-tpm2
  labels:
    machineconfiguration.openshift.io/role: worker
boot_device:
  layout: x86_64
  luks:
    tpm2: true
EOF
butane worker-luks-tpm2.bu

Butane 0.23 stops at variant openshift 4.18.0. The output uses Ignition 3.4.0, which newer releases accept. A NodePool reads MachineConfigs from a ConfigMap key named config, in the NodePool namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: hcp-luks-worker-luks-tpm2
  namespace: clusters
data:
  config: |
    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      name: 99-worker-luks-tpm2
      labels:
        machineconfiguration.openshift.io/role: worker
    spec:
      config:
        ignition:
          version: 3.4.0
        storage:
          luks:
            - name: root
              label: luks-root
              device: /dev/disk/by-partlabel/root
              clevis:
                tpm2: true
              wipeVolume: true
          filesystems:
            - device: /dev/mapper/root
              format: xfs
              label: root
              wipeFilesystem: true

tpm2: true seals the key to the TPM without PCR policy. To also bind it to the Secure Boot state, replace the clevis block with custom: {pin: tpm2, config: '{"pcr_bank":"sha256","pcr_ids":"7"}'}. A firmware or Secure Boot change then stops the automatic unlock.

2. Core password: console login on the worker

This item is independent of LUKS. The password is set in two separate places, because the discovery image and the installed node get their Ignition from different sources.

Installed node: NodePool MachineConfig

apiVersion: v1
kind: ConfigMap
metadata:
  name: hcp-luks-worker-core-password
  namespace: clusters
data:
  config: |
    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      name: 99-worker-core-password
      labels:
        machineconfiguration.openshift.io/role: worker
    spec:
      config:
        ignition:
          version: 3.4.0
        passwd:
          users:
            - name: core
              passwordHash: '<CORE_PASSWORD_HASH>'

Generate the hash with openssl passwd -6. The password works on the console only; RHCOS keeps SSH password authentication disabled.

Discovery image: InfraEnv ignitionConfigOverride

The NodePool configuration never reaches the discovery image. A console password there helps when a wrong static IP leaves the host unreachable. The InfraEnv field ignitionConfigOverride is merged into the discovery Ignition. Once the InfraEnv from step 4 exists:

HASH='<CORE_PASSWORD_HASH>'
IGN=$(jq -cn --arg h "$HASH" '{ignition:{version:"3.1.0"},passwd:{users:[{name:"core",passwordHash:$h}]}}')
oc -n "$AGENT_NS" patch infraenv hcp-luks --type merge \
  -p "$(jq -cn --arg i "$IGN" '{spec:{ignitionConfigOverride:$i}}')"

The repository sets the same field directly in 02-infraenv.yaml. Any change regenerates the ISO under a new URL.

3. Reference both configs from the NodePool

Each item is a separate entry in spec.config; removing one entry drops that item only.

apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
  name: hcp-luks-workers
  namespace: clusters
spec:
  clusterName: hcp-luks
  replicas: 1
  arch: amd64
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.22.15-x86_64
  management:
    autoRepair: false
    upgradeType: Replace
    replace:
      strategy: RollingUpdate
      rollingUpdate:
        maxSurge: 0
        maxUnavailable: 1
  platform:
    type: Agent
    agent:
      agentLabelSelector:
        matchLabels:
          hcp-luks.example.com/tpm2: "true"
  config:
    - name: hcp-luks-worker-luks-tpm2
    - name: hcp-luks-worker-core-password
Replace, not InPlace
Ignition encrypts the disk at first boot only. An InPlace NodePool cannot encrypt an existing root disk. With Replace, any change to a referenced ConfigMap (the password included) reprovisions every node of the pool. With a single worker and no spare Agent, the node is removed first.

4. Prepare the agent namespace

The InfraEnv generates the discovery ISO; nmStateConfigLabelSelector embeds the static network configuration. The discovery core password from step 2 is added to it with ignitionConfigOverride:

apiVersion: agent-install.openshift.io/v1beta1
kind: InfraEnv
metadata:
  name: hcp-luks
  namespace: <AGENT_NAMESPACE>
spec:
  pullSecretRef:
    name: hcp-luks-pull-secret
  sshAuthorizedKey: '<SSH_PUBLIC_KEY>'
  cpuArchitecture: x86_64
  agentLabels:
    hcp-luks.example.com/tpm2: "true"
  nmStateConfigLabelSelector:
    matchLabels:
      infraenvs.agent-install.openshift.io: hcp-luks

The NMStateConfig is matched to the host by NIC MAC address; eno1 is only a logical name:

apiVersion: agent-install.openshift.io/v1beta1
kind: NMStateConfig
metadata:
  name: hcp-luks-worker-0
  namespace: <AGENT_NAMESPACE>
  labels:
    infraenvs.agent-install.openshift.io: hcp-luks
spec:
  interfaces:
    - name: eno1
      macAddress: '<WORKER0_MAC>'
  config:
    interfaces:
      - name: eno1
        type: ethernet
        state: up
        mac-address: '<WORKER0_MAC>'
        ipv4:
          enabled: true
          dhcp: false
          address:
            - ip: 192.0.2.50
              prefix-length: 24
        ipv6:
          enabled: false
    dns-resolver:
      config:
        server:
          - 192.0.2.53
    routes:
      config:
        - destination: 0.0.0.0/0
          next-hop-address: 192.0.2.1
          next-hop-interface: eno1
          table-id: 254

The agent namespace also needs the pull secret (a second copy, the HostedCluster reads its own in clusters) and the Role used by the Agent CAPI provider:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: capi-provider-role
  namespace: <AGENT_NAMESPACE>
rules:
  - apiGroups:
      - agent-install.openshift.io
    resources:
      - agents
    verbs:
      - '*'
capi-provider-role is not created by the operator
hcp create cluster agent --render emits this Role; the HyperShift operator only creates the RoleBinding to it. Without the Role, the HostedCluster loops on failed to reconcile Agent RoleBinding: rolebindings.rbac.authorization.k8s.io "capi-provider-role" not found and the control plane namespace stays empty (observed on MCE 2.17.3).

5. Create the HostedCluster

apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  name: hcp-luks
  namespace: clusters
spec:
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.22.15-x86_64
  pullSecret:
    name: hcp-luks-pull-secret
  sshKey:
    name: hcp-luks-ssh-key
  infraID: hcp-luks
  dns:
    baseDomain: <BASE_DOMAIN>
  controllerAvailabilityPolicy: HighlyAvailable
  infrastructureAvailabilityPolicy: SingleReplica
  olmCatalogPlacement: management
  fips: false
  networking:
    networkType: OVNKubernetes
    clusterNetwork:
      - cidr: 10.132.0.0/14
    serviceNetwork:
      - cidr: 172.31.0.0/16
    machineNetwork:
      - cidr: 192.0.2.0/24
  platform:
    type: Agent
    agent:
      agentNamespace: <AGENT_NAMESPACE>
  etcd:
    managementType: Managed
    managed:
      storage:
        type: PersistentVolume
        persistentVolume:
          size: 8Gi
          storageClassName: ocs-storagecluster-ceph-rbd
  services:
    - service: APIServer
      servicePublishingStrategy:
        type: LoadBalancer
    - service: OAuthServer
      servicePublishingStrategy:
        type: Route
    - service: Konnectivity
      servicePublishingStrategy:
        type: Route
    - service: Ignition
      servicePublishingStrategy:
        type: Route

infrastructureAvailabilityPolicy: SingleReplica runs guest infrastructure components (ingress router) with one replica, matching the single worker.

6. Create the worker VM

The MAC address is not known before the VM exists, while the ISO needs it in the NMStateConfig. The VM is therefore created Halted and without macAddress: kubemacpool assigns one and writes it into the VM spec.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: hcp-luks-worker-0
  namespace: <AGENT_NAMESPACE>
spec:
  runStrategy: Halted
  dataVolumeTemplates:
    - metadata:
        name: hcp-luks-worker-0-root
      spec:
        source:
          blank: {}
        storage:
          accessModes:
            - ReadWriteMany
          volumeMode: Block
          resources:
            requests:
              storage: 120Gi
          storageClassName: ocs-storagecluster-ceph-rbd-virtualization
  template:
    metadata:
      labels:
        kubevirt.io/domain: hcp-luks-worker-0
    spec:
      domain:
        cpu:
          cores: 8
        memory:
          guest: 16Gi
        features:
          smm:
            enabled: true
        firmware:
          bootloader:
            efi:
              secureBoot: true
              persistent: true
        devices:
          tpm:
            persistent: true
          disks:
            - name: rootdisk
              bootOrder: 1
              disk:
                bus: virtio
            - name: discovery-iso
              bootOrder: 2
              cdrom:
                bus: sata
          interfaces:
            - name: nic0
              model: virtio
              bridge: {}
      networks:
        - name: nic0
          multus:
            networkName: <NAD_NAMESPACE>/<NAD_NAME>
      volumes:
        - name: rootdisk
          dataVolume:
            name: hcp-luks-worker-0-root
        - name: discovery-iso
          dataVolume:
            name: hcp-luks-discovery-iso
Persistent vTPM and boot order
Without tpm.persistent: true, the vTPM state is lost on VM restart and the sealed LUKS key with it: the root disk never unlocks again. Keep rootdisk first in the boot order: the blank disk falls through to the ISO, and after installation the VM boots the encrypted disk instead of the discovery image.

7. Build the ISO, boot, approve

oc apply -f vm/02-worker-vm.yaml
oc -n "$AGENT_NS" get vm hcp-luks-worker-0 \
  -o jsonpath='{.spec.template.spec.domain.devices.interfaces[0].macAddress}{"\n"}'
# copy the MAC into 03-nmstateconfig.yaml, then
oc apply -k manifests
oc -n "$AGENT_NS" wait infraenv/hcp-luks --for=condition=ImageCreated --timeout=5m
ISO_URL=$(oc -n "$AGENT_NS" get infraenv hcp-luks -o jsonpath='{.status.isoDownloadURL}')
sed "s|<ISO_URL>|${ISO_URL}|" vm/01-iso-datavolume.yaml | oc apply -f -
oc -n "$AGENT_NS" wait dv/hcp-luks-discovery-iso --for=condition=Ready --timeout=15m
virtctl -n "$AGENT_NS" start hcp-luks-worker-0
oc -n "$AGENT_NS" get agents
oc -n "$AGENT_NS" patch agent <AGENT_ID> --type merge -p '{"spec":{"approved":true}}'

The ISO URL carries a token: it is injected at apply time and never stored in the repository. Any change to the InfraEnv or a matching NMStateConfig regenerates the ISO under a new URL.

Verify

Discovery stage, Agent registered and waiting for the NodePool:

oc -n "$AGENT_NS" get agents \
  -o custom-columns=HOST:.spec.hostname,APPROVED:.spec.approved,STATE:.status.debugInfo.state
HOST                APPROVED   STATE
hcp-luks-worker-0   true       known-unbound

The discovery image carries neither the LUKS configuration nor the NodePool password: its root disk is still blank. Both NodePool items apply on the first boot from disk after installation.

LUKS

lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINT
sudo cryptsetup status root
sudo clevis luks list -d /dev/vda4
[core@hcp-luks-worker-0 ~]$ lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINT
sudo cryptsetup status root
sudo clevis luks list -d /dev/vda4
NAME     TYPE  FSTYPE      MOUNTPOINT
loop0    loop  erofs
sr0      rom   iso9660
vda      disk
├─vda1   part
├─vda2   part  vfat
├─vda3   part  ext4        /boot
└─vda4   part  crypto_LUKS
  └─root crypt xfs         /sysroot
/dev/mapper/root is active and is in use.
  type:    LUKS2
  cipher:  aes-xts-plain64
  keysize: 512 [bits]
  key location: keyring
  device:  /dev/vda4
  sector size:  512 [bytes]
  offset:  32768 [512-byte units] (16777216 [bytes])
  size:    250574815 [512-byte units] (128294305280 [bytes])
  mode:    read/write
1: sss '{"t":1,"pins":{"tpm2":[{"hash":"sha256","key":"ecc"}]}}'

Core password

  • Discovery image: virtctl -n "$AGENT_NS" console hcp-luks-worker-0, log in as core with the ignitionConfigOverride password.
  • Installed node:
sudo grep '^core:' /etc/shadow | cut -c1-8
[core@hcp-luks-worker-0 ~]$ sudo grep '^core:' /etc/shadow | cut -c1-8
core:$6$
Updated on 2026-10-22
Read Markdown
 Openshift, Hypershift, Mco, Security, Kubevirt, Nmstate
Back | Home