Ansible Execution Environment Build Factory

mika published on 2024-03-26 included in Openshift

Components overview

Ansible Automation Platform

Red Hat® Ansible® Automation Platform is an end-to-end automation platform to configure systems, deploy software, and orchestrate advanced workflows. It includes resources to create, manage, and scale across the entire enterprise.

Execution Environements

These are container images which include the operating system kernel (Red Hat Enterprise Linux® Universal Base Image), automation engine (ansible-core), programming language (Python), as well as all necessary dependencies. Together, they create an isolated execution environment that can interact with—and run on—almost any IT platform.

OCP etcd backup

mika published on 2023-06-23 included in Openshift

How to schedule etcd backups using cronjob

What you will need :

Namespace
Service Account
Cluster Role
Cluster Role Binding
Extend SA priviliges
Cronjob

Namespace

---
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  annotations:
    openshift.io/description: Openshift Backup Automation Tool
    openshift.io/display-name: Backup ETCD Automation  
  name: ocp-etcd-backup
  finalizers:
  - kubernetes

Service Account

---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: openshift-backup
  namespace: ocp-etcd-backup
  labels:
    app: openshift-backup

Cluster role

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-etcd-backup
rules:
- apiGroups: [""]
  resources:
     - "namespaces"
  verbs: ["get", "list", "create"]

- apiGroups: [""]
  resources:
     - "nodes"
  verbs: ["get", "list"]
- apiGroups: [""]
  resources:
     - "pods"
     - "pods/log"
  verbs: ["get", "list", "create", "delete", "watch"]

Cluster Role Binding

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: openshift-backup
  labels:
    app: openshift-backup
subjects:
  - kind: ServiceAccount
    name: openshift-backup
    namespace: ocp-etcd-backup
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-etcd-backup

Cronjob

---
kind: CronJob
apiVersion: batch/v1
metadata:
  name: openshift-backup
  namespace: ocp-etcd-backup
  labels:
    app: openshift-backup
spec:
  schedule: "56 23 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 5
  jobTemplate:
    metadata:
      labels:
        app: openshift-backup
    spec:
      backoffLimit: 0
      template:
        metadata:
          labels:
            app: openshift-backup
        spec:
          containers:
            - name: backup
              image: "registry.redhat.io/openshift4/ose-cli"
              command:
                - "/bin/bash"
                - "-c"
                - oc get no -l node-role.kubernetes.io/master --no-headers -o name | xargs -I {} --  oc debug {} -- bash -c 'chroot /host sudo -E /usr/local/bin/cluster-backup.sh /home/core/backup/ && chroot /host sudo -E find /home/core/backup/ -type f -mmin +"1" -delete'
          restartPolicy: "Never"
          terminationGracePeriodSeconds: 30
          activeDeadlineSeconds: 500
          dnsPolicy: "ClusterFirst"
          serviceAccountName: "openshift-backup"
          serviceAccount: "openshift-backup"

Info

If you wish to backup to a NFS share, change the command into the backup container to :

Configure OCP network using nmstate operator

mika published on 2023-06-20 included in Openshift

Official documentation

Install operator

Create NS

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  annotations:
    olm.providedAPIs: NMState.v1.nmstate.io
  generateName: openshift-nmstate-
  name: openshift-nmstate-tn6k8
  namespace: openshift-nmstate
spec:
  targetNamespaces:
  - openshift-nmstate

Create operator group

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  annotations:
    olm.providedAPIs: NMState.v1.nmstate.io
  generateName: openshift-nmstate-
  name: openshift-nmstate-tn6k8
  namespace: openshift-nmstate
spec:
  targetNamespaces:
  - openshift-nmstate

Create sub

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    operators.coreos.com/kubernetes-nmstate-operator.openshift-nmstate: ""
  name: kubernetes-nmstate-operator
  namespace: openshift-nmstate
spec:
  channel: stable
  installPlanApproval: Automatic
  name: kubernetes-nmstate-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Create nmstate instance

apiVersion: nmstate.io/v1
kind: NMState
metadata:
  name: nmstate

nmstate resources

NodeNetworkState : Current state of nodes

Configure OCP with letsencrypt certs

mika published on 2023-06-15 included in Openshift

Create wildcard entries in DNS

*.cluster.domain.com

Get acme.sh

git clone https://github.com/acmesh-official/acme.sh.git
cd acme.sh

Get token from CloudFare

Cloudfare > Get CF_Zone_ID CF_Account_ID and create CF_Token with Edit Zone permission

Edit dnsapi/dns_cf.sh with these values

Create certificates

export OCP_API_DOMAIN=$(oc whoami --show-server | cut -f 2 -d ':' | cut -f 3 -d '/' | sed 's/-api././')
export OCP_WILDCARD_DOMAIN=$(oc get ingresscontroller default -n openshift-ingress-operator -o jsonpath='{.status.domain}')
export CERTDIR=$HOME/openshift_certificates
mkdir -p ${CERTDIR}
$ ~/acme.sh/acme.sh --register-account -m your_email_address@example.com

${HOME}/acme.sh/acme.sh --issue --dns dns_cf -d ${OCP_API_DOMAIN} -d *.${OCP_WILDCARD_DOMAIN} --debug
${HOME}/acme.sh/acme.sh --install-cert -d ${OCP_API_DOMAIN} -d *.${OCP_WILDCARD_DOMAIN} --cert-file ${CERTDIR}/cert.pem --key-file ${CERTDIR}/key.pem --fullchain-file ${CERTDIR}/fullchain.pem --ca-file ${CERTDIR}/ca.cer

Patch router

oc -n openshift-ingress create secret tls router-certs --cert=${CERTDIR}/fullchain.pem --key=${CERTDIR}/key.pem
oc -n openshift-ingress-operator patch ingresscontroller default  --type=merge --patch='{"spec": { "defaultCertificate": { "name": "router-certs" }}}'

Patch API cert

oc -n openshift-config create secret tls api-certs --cert=${CERTDIR}/fullchain.pem --key=${CERTDIR}/key.pem
oc patch apiserver cluster --type merge --patch="{\"spec\": {\"servingCerts\": {\"namedCertificates\": [ { \"names\": [  \"$OCP_API_DOMAIN\"  ], \"servingCertificate\": {\"name\": \"api-certs\" }}]}}}"

Force renew if needed

export OCP_API_DOMAIN=$(oc whoami --show-server | cut -f 2 -d ':' | cut -f 3 -d '/' | sed 's/-api././')
export OCP_WILDCARD_DOMAIN=$(oc get ingresscontroller default -n openshift-ingress-operator -o jsonpath='{.status.domain}')
~/acme.sh/acme.sh --renew -d ${OCP_API_DOMAIN} -d *.${OCP_WILDCARD_DOMAIN} --force

Patch ingress to use custom names (if needed)

https://access.redhat.com/solutions/4853401

Qemu customize

mika published on 2023-04-25 included in Linux

Customize img

OCP etcd metrics

mika published on 2023-04-07 included in Openshift

Recommended etcd practices

https://docs.openshift.com/container-platform/4.12/scalability_and_performance/recommended-host-practices.html#recommended-etcd-practices_recommended-host-practices

The histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[2m])) metric reports the round trip time for etcd to finish replicating the client requests between the members. Ensure that it is less than 50 ms.

Metrics to monitor

https://access.redhat.com/articles/6967785#metrics

Monitor Leadership changes:

This is expected as per result of installation/upgrade process or day1/2 operations (as result of Machine Config daemon operations), but we don’t expect to see it happening during normal operations. etcdHighNumberOfLeaderChanges alert can help us to identify that situation. Prometheus query could also be used (sum(rate(etcd_server_leader_changes_seen_total[2m]))). If happening during normal operation, I/O and network metrics can help us to identify the root cause.