Posts Tags Categories About
../../images/logo.svg

Reset the OpenShift kubeadmin password

 published on  included in Openshift

Reset the kubeadmin bootstrap password when the original credential is lost and no other cluster-admin identity provider is available.

Prerequisites

  • SSH access to a control plane node (as core, with sudo if required).
  • oc available on the node or copied in with the recovery kubeconfig.
  • htpasswd available to generate the bcrypt hash (install httpd-tools on RHEL, or run from a toolbox/UBI container).
  • A new password of at least 23 characters (required by the bootstrap authenticator).
Warning
This procedure patches the kubeadmin secret in kube-system. Use only for cluster recovery. After restoring access, define an identity provider and create a dedicated cluster-admin user, then remove kubeadmin per Red Hat documentation.

1. SSH to a control plane node

Connect to any healthy control plane node in the cluster.

Read More
 Openshift, Security, Cluster-Ops

Enable LLDP on OpenShift nodes using nmstate

 published on  included in Openshift

This article describes how to enable LLDP (Link Layer Discovery Protocol) on all Ethernet interfaces that are up, using the nmstate operator and a NodeNetworkConfigurationPolicy (NNCP).

Prerequisites: the Kubernetes NMState operator is installed and a NMState instance exists (see Configure OCP network using nmstate operator).

1. Apply the NodeNetworkConfigurationPolicy

The policy below uses nmstate capture to select only Ethernet interfaces that are in state up, then sets lldp.enabled: true on those interfaces. This avoids touching other interface types or down interfaces.

Read More
 Openshift, Nmstate, Lldp

Openshift IPSEC N/S

 published on  included in Openshift
Note
In this POC, we will cover how to configure an IPSEC communication from OCP to a RHEL node

Configuration RHEL node side

Requirements

butane libreswan

dns install -y butane libreswan

Create CA and certs

Note
For simplicty, we’ell use the same certificate with SAN matching both side
mkdir ca certs private
openssl genrsa -out ca/ca.key.pem 2048
openssl req -x509 -new -nodes -key ca/ca.key.pem -sha256 -days 3650 -out ca/ca.crt.pem -subj "/CN=IPsec Test CA/O=MyOrg/OU=MyUnit/L=MyCity/ST=MyState/C=US"
openssl genrsa -out private/hosts.key.pem 2048

cat > openssl.cnf <<EOF
[ req ]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[ req_distinguished_name ]
C = US
ST = MyState
L = MyCity
O = MyOrg
OU = MyUnit
CN = worker-n.example.com

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = worker-n.example.com #CHANGEME
DNS.2 = rhel-remote.example.com #CHANGEME
EOF

openssl req -new -key private/hosts.key.pem -out certs/hosts.csr.pem -config openssl.cnf

openssl x509 -req -in certs/hosts.csr.pem -CA ca/ca.crt.pem -CAkey ca/ca.key.pem -CAcreateserial -out certs/hosts.crt.pem -days 365 -sha256 -extfile <(printf "subjectAltName=DNS:openshift-host.example.com,DNS:rhel-host.example.com")


openssl pkcs12 -export -out certs/hosts.p12 -inkey private/hosts.key.pem -in certs/hosts.crt.pem -certfile ca/ca.crt.pem -name "allnodes"

Import cert on RHEL node

ipsec initnss
ipsec import certs/hosts.p12

# Verify import is successfull with the name provided ; here "allnodes"
certutil -L -d sql:/var/lib/ipsec/nss/
Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

allnodes                                                     u,u,u
IPsec Test CA - MyOrg                                        CT,

IPSEC conf

cat > /etc/ipsec.d/poc.conf << EOF
conn my-host-to-host-vpn
    left=10.8.109.144 #  CAHNGEME This is the RHEL node IP
    leftid=%fromcert
    leftcert="allnodes"    
    right=10.8.51.222 # CHANGEME This is the OCP node IP
    rightid=%fromcert
    authby=rsasig              
    ikev2=yes
    ike=aes256-sha2      
    ikelifetime=8h                     
    esp=aes_gcm256
    auto=start                  
    type=transport 
EOF

Start ipsec.service

systemctl start ipsec.service

Configuration OCP side

Requirements

Note
Install nmstate operator

NNCP

cat > ipsec_nncp.yaml << EOF
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: ipsec-config
spec:
  nodeSelector:
    kubernetes.io/hostname: mmayeras-tqhkk-worker-0-b4ktk
  desiredState:
    interfaces:
    - name: test-ipsec
      type: ipsec
      libreswan:
        left: worker-n.example.com # CHANGEME
        leftid: '%fromcert'
        leftrsasigkey: '%cert'
        leftcert: left_server
        leftmodecfgclient: false
        right: rhel-host.example.com # CHANGEME
        rightid: '%fromcert'
        rightrsasigkey: '%cert'
        rightsubnet: 10.x.x.x/xx # CHANGEME
        ikev2: insist
        type: transport
        esp: aes_gcm256
        ike: aes256-sha2;dh20
EOF

oc apply -f ipsec_nncp.yaml

Machine config for certificate import

cat > 99-ipsec-worker-endpoint-config.bu << EOF
  variant: openshift
  version: 4.18.0
  metadata:
    name: 99-worker-import-certs
    labels:
      machineconfiguration.openshift.io/role: worker
  systemd:
    units:
    - name: ipsec-import.service
      enabled: true
      contents: |
        [Unit]
        Description=Import external certs into ipsec NSS
        Before=ipsec.service

        [Service]
        Type=oneshot
        ExecStart=/usr/local/bin/ipsec-addcert.sh
        RemainAfterExit=false
        StandardOutput=journal

        [Install]
        WantedBy=multi-user.target
  storage:
    files:
    - path: /etc/pki/certs/ca.pem
      mode: 0400
      overwrite: true
      contents:
        local: ca/ca.crt.pem
    - path: /etc/pki/certs/hosts.p12
      mode: 0400
      overwrite: true
      contents:
        local: certs/hosts.p12
    - path: /usr/local/bin/ipsec-addcert.sh
      mode: 0740
      overwrite: true
      contents:
        inline: |
          #!/bin/bash -e
          echo "importing cert to NSS"
          certutil -A -n "CA" -t "CT,C,C" -d /var/lib/ipsec/nss/ -i /etc/pki/certs/ca.pem
          pk12util -W "" -i /etc/pki/certs/hosts.p12 -d /var/lib/ipsec/nss/
          certutil -M -n "allnodes" -t "u,u,u" -d /var/lib/ipsec/nss/
EOF

butane -d . 99-ipsec-worker-endpoint-config.bu -o ./99-ipsec-worker-endpoint-config.yaml

oc apply -f
Warning
Ensure thne name used in the certutil command matches the name used in the openssl pkcs12 -export command used previously
Note
Waint until end of machine config rollout

Check config

Ensure tunnel is active

After rollout OCP side, the tunnel should be up

Read More
 Openshift