Configure OCP network using nmstate operator

mika published on 2023-06-20 included in Openshift

Install operator

Create NS

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  annotations:
    olm.providedAPIs: NMState.v1.nmstate.io
  generateName: openshift-nmstate-
  name: openshift-nmstate-tn6k8
  namespace: openshift-nmstate
spec:
  targetNamespaces:
  - openshift-nmstate

Create operator group

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  annotations:
    olm.providedAPIs: NMState.v1.nmstate.io
  generateName: openshift-nmstate-
  name: openshift-nmstate-tn6k8
  namespace: openshift-nmstate
spec:
  targetNamespaces:
  - openshift-nmstate

Create sub

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    operators.coreos.com/kubernetes-nmstate-operator.openshift-nmstate: ""
  name: kubernetes-nmstate-operator
  namespace: openshift-nmstate
spec:
  channel: stable
  installPlanApproval: Automatic
  name: kubernetes-nmstate-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Create nmstate instance

apiVersion: nmstate.io/v1
kind: NMState
metadata:
  name: nmstate

nmstate resources

NodeNetworkState : Current state of nodes

Configure OCP with letsencrypt certs

mika published on 2023-06-15 included in Openshift

Create wildcard entries in DNS

*.cluster.domain.com

Get acme.sh

git clone https://github.com/acmesh-official/acme.sh.git
cd acme.sh

Get token from CloudFare

Cloudfare > Get CF_Zone_ID CF_Account_ID and create CF_Token with Edit Zone permission

Edit dnsapi/dns_cf.sh with these values

Create certificates

export OCP_API_DOMAIN=$(oc whoami --show-server | cut -f 2 -d ':' | cut -f 3 -d '/' | sed 's/-api././')
export OCP_WILDCARD_DOMAIN=$(oc get ingresscontroller default -n openshift-ingress-operator -o jsonpath='{.status.domain}')
export CERTDIR=$HOME/openshift_certificates
mkdir -p ${CERTDIR}
$ ~/acme.sh/acme.sh --register-account -m your_email_address@example.com

${HOME}/acme.sh/acme.sh --issue --dns dns_cf -d ${OCP_API_DOMAIN} -d *.${OCP_WILDCARD_DOMAIN} --debug
${HOME}/acme.sh/acme.sh --install-cert -d ${OCP_API_DOMAIN} -d *.${OCP_WILDCARD_DOMAIN} --cert-file ${CERTDIR}/cert.pem --key-file ${CERTDIR}/key.pem --fullchain-file ${CERTDIR}/fullchain.pem --ca-file ${CERTDIR}/ca.cer

Patch router

oc -n openshift-ingress create secret tls router-certs --cert=${CERTDIR}/fullchain.pem --key=${CERTDIR}/key.pem
oc -n openshift-ingress-operator patch ingresscontroller default  --type=merge --patch='{"spec": { "defaultCertificate": { "name": "router-certs" }}}'

Patch API cert

oc -n openshift-config create secret tls api-certs --cert=${CERTDIR}/fullchain.pem --key=${CERTDIR}/key.pem
oc patch apiserver cluster --type merge --patch="{\"spec\": {\"servingCerts\": {\"namedCertificates\": [ { \"names\": [  \"$OCP_API_DOMAIN\"  ], \"servingCertificate\": {\"name\": \"api-certs\" }}]}}}"

Force renew if needed

export OCP_API_DOMAIN=$(oc whoami --show-server | cut -f 2 -d ':' | cut -f 3 -d '/' | sed 's/-api././')
export OCP_WILDCARD_DOMAIN=$(oc get ingresscontroller default -n openshift-ingress-operator -o jsonpath='{.status.domain}')
~/acme.sh/acme.sh --renew -d ${OCP_API_DOMAIN} -d *.${OCP_WILDCARD_DOMAIN} --force

Patch ingress to use custom names (if needed)

https://access.redhat.com/solutions/4853401

Qemu customize

mika published on 2023-04-25 included in Linux

Customize img

OCP etcd metrics

mika published on 2023-04-07 included in Openshift

Recommended etcd practices

https://docs.openshift.com/container-platform/4.12/scalability_and_performance/recommended-host-practices.html#recommended-etcd-practices_recommended-host-practices

The histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[2m])) metric reports the round trip time for etcd to finish replicating the client requests between the members. Ensure that it is less than 50 ms.

Metrics to monitor

https://access.redhat.com/articles/6967785#metrics

Monitor Leadership changes:

This is expected as per result of installation/upgrade process or day1/2 operations (as result of Machine Config daemon operations), but we don’t expect to see it happening during normal operations. etcdHighNumberOfLeaderChanges alert can help us to identify that situation. Prometheus query could also be used (sum(rate(etcd_server_leader_changes_seen_total[2m]))). If happening during normal operation, I/O and network metrics can help us to identify the root cause.

Installer-provisioned installation

mika published on 2023-03-24 included in Openshift

IPI

Get openshift installer, openshift cli and pull-secret from https://console.redhat.com/openshift
Create install-config.yaml $ openshift-install create install-config --dir ./cluster

Here is a sample install-config.yaml for vSphere IPI

additionalTrustBundlePolicy: Proxyonly
apiVersion: v1
baseDomain: example.com
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform: {}
  replicas: 3
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3
metadata:
  creationTimestamp: null
  name: mmayeras
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.10.0.0/24
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
platform:
  vsphere:
    apiVIPs:
    - 10.10.0.2
    cluster: your_cluster
    datacenter: your_datacenter
    defaultDatastore: your_datastore
    ingressVIPs:
    - 10.10.0.3
    network: your_network
    password: your_password
    username:  your_username
    vCenter: your_vcenter
publish: External
pullSecret: 'your_pull_secret'
sshKey: your_ssh_pub_key

Backup and Copy the install-config.yaml into the installation dir $ cp install-config.yaml{,.bak} && mv install-config.yaml ./cluster/
Launch the installer $ openshift-install create cluster --dir ./cluster

Openshift docs

mika published on 2023-03-24 included in Openshift

docs.openshift.com

Update Path Lab

Openshift Lifecycle