User-provisioned installation

mika published on 2023-03-24 included in Openshift

UPI

PXE Config

dnf install -y tftp-server syslinux-tftpboot httpd haproxy
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-6.03.tar.gz
wget https://raw.githubusercontent.com/leoaaraujo/openshift_pxe_boot_menu/main/files/bg-ocp.png -O /var/lib/tftpboot/bios/bg-ocp.png
tar xf syslinux-6.03.tar.gz 
cp syslinux-6.03/bios/core/pxelinux.0 /var/lib/tftpboot/bios/
cp syslinux-6.03/bios/com32/elflink/ldlinux/ldlinux.c32 /var/lib/tftpboot/bios/
cp syslinux-6.03/bios/com32/lib/libcom32.c32 /var/lib/tftpboot/bios/
cp syslinux-6.03/bios/com32/libutil/libutil.c32 /var/lib/tftpboot/bios/
cp syslinux-6.03/bios/memdisk/memdisk /var/lib/tftpboot/bios/
cp syslinux-6.03/bios/com32/modules/poweroff.c32 /var/lib/tftpboot/bios/
cp syslinux-6.03/bios/com32/modules/pxechn.c32 /var/lib/tftpboot/bios/
cp syslinux-6.03/bios/com32/modules/reboot.c32 /var/lib/tftpboot/bios/
cp syslinux-6.03/bios/com32/menu/vesamenu.c32 /var/lib/tftpboot/bios/
 
cp syslinux-6.03/efi64/efi/syslinux.efi /var/lib/tftpboot/efi64/
cp syslinux-6.03/efi64/com32/elflink/ldlinux/ldlinux.e64 /var/lib/tftpboot/efi64/
cp syslinux-6.03/efi64/com32/lib/libcom32.c32 /var/lib/tftpboot/efi64/
cp syslinux-6.03/efi64/com32/libutil/libutil.c32 /var/lib/tftpboot/efi64/
cp syslinux-6.03/bios/memdisk/memdisk /var/lib/tftpboot/efi64/
cp syslinux-6.03/efi64/com32/modules/poweroff.c32 /var/lib/tftpboot/efi64/
cp syslinux-6.03/efi64/com32/modules/pxechn.c32 /var/lib/tftpboot/efi64/
cp syslinux-6.03/efi64/com32/modules/reboot.c32 /var/lib/tftpboot/efi64/
cp syslinux-6.03/efi64/com32/menu/vesamenu.c32 /var/lib/tftpboot/efi64/


# wget https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/latest/rhcos-live-kernel-x86_64  -O /var/www/html/ 
# wget https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/latest/rhcos-live-initramfs.x86_64.img -O /var/www/html/
# wget https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/latest/rhcos-live-rootfs.x86_64.img **-O /var/www/html/
systemctl enable tftp.service httpd --now

/var/lib/tftpboot/efi64/pxelinux.cfg/default

UI vesamenu.c32
MENU BACKGROUND        bg-ocp.png  
MENU COLOR sel         4  #ffffff std 
MENU COLOR title       1  #ffffff  
TIMEOUT 120
PROMPT 0
MENU TITLE OPENSHIFT 4.x INSTALL BARE METAL PXE MENU 
LABEL INSTALL BOOTSTRAP   
  KERNEL http://192.168.0.10:8080/rhcos-live-kernel-x86_64
  APPEND initrd=http://192.168.0.10:8080/rhcos-live-initramfs.x86_64.img coreos.live.rootfs_url=http://192.168.0.10:8080/rhcos-live-rootfs.x86_64.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://192.168.0.10:8080/bootstrap.ign
LABEL INSTALL MASTER     
  KERNEL http://192.168.0.10:8080/rhcos-live-kernel-x86_64
  APPEND initrd=http://192.168.0.10:8080/rhcos-live-initramfs.x86_64.img coreos.live.rootfs_url=http://192.168.0.10:8080/rhcos-live-rootfs.x86_64.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://192.168.0.10:8080/master.ign
LABEL INSTALL WORKER    
  KERNEL http://192.168.0.10:8080/rhcos-live-kernel-x86_64
  APPEND initrd=http://192.168.0.10:8080/rhcos-live-initramfs.x86_64.img coreos.live.rootfs_url=http://192.168.0.10:8080/rhcos-live-rootfs.x86_64.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://192.168.0.10:8080/worker.ign
LABEL INSTALL EL
  KERNEL http://192.168.0.10:8080/el/vmlinuz
  APPEND initrd=http://192.168.0.10:8080/el/initrd.img inst.repo=http://192.168.0.10:8080/el/Packages/ inst.ks=http://192.168.0.10:8080/el/kickstart.cfg 

Create ignition config from install-config.yaml

apiVersion: v1
baseDomain: example.com
metadata:
  name: ocp
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 1
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16

platform:
  none: {}

fips: false

pullSecret: 'pull_secret'
sshKey: 'ssh-rsa'

$ openshift-install create ignition-configs --dir ./cluster

Ceph installation

mika published on 2023-02-15 included in Openshift

Requirements

Red Hat Enterprise Linux 8.4 EUS or later.
Ansible 2.9 or later.
A valid Red Hat subscription with the appropriate entitlements.
Root-level access to all nodes.
An active Red Hat Network (RHN) or service account to access the Red Hat Registry.

Create 3 RHEL 8 virtual machines

ceph1
ceph2
ceph3

Note

Installing Ceph on Virtual Machines is not recommendend for production use

Register servers to RHN

Find and attach Red Hat Ceph Storage pool

$ subscription-manager list --available --matches 'Red Hat Ceph Storage'    
$ subscription-manager attach --pool=POOL_ID

Enable server & extra repos

$subscription-manager repos --disable=*
subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms
subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms
subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms
subscription-manager repos --enable=ansible-2.9-for-rhel-8-x86_64-rpms

Update system

$ dnf update -y

HTPasswd oauth provider

mika published on 2023-02-15

Create htpaswd file

$ htpasswd -c -B -b htpasswd admin adminpass 
$ htpasswd -c -B -b htpasswd developer devpass

Create secret

$ oc create secret generic htpass-secret --from-file=htpasswd -n openshift-config

Patch oauth cluster

$ oc patch oauth/cluster --patch '{"spec":{"identityProviders":[{"name":"htpasswd","mappingMethod":"claim","type":"HTPasswd","htpasswd":{"fileData":{"name":"htpass-secret"}}}]}}' --type=merge

Give admin user cluster-admin role

$ oc adm policy add-cluster-role-to-user cluster-admin admin

NGINX reverse

mika published on 2023-02-15

Use nginx as reverse proxy in front of multiple clusters

$ dnf install -y nginx nginx-mod-stream.x86_64

Add in nginx.conf include /etc/nginx/passthrough.conf;

passthrough.conf

stream {

    map $ssl_preread_server_name $internalport {
	hostnames;
        *.apps.sno1.domain     9441;
        *.apps.sno2.domain      9442;
        api.sno1.domain      6441;
        api.sno2.domain      6442;
    }


    upstream sno2_api {
        server 192.168.0.109:6443 max_fails=3 fail_timeout=10s;
    }
    upstream sno2_ingress {
        server 192.168.0.109:443 max_fails=3 fail_timeout=10s;
    }
    upstream sno1_api {
        server 192.168.0.110:6443 max_fails=3 fail_timeout=10s;
    }
    upstream sno1_ingress {
        server 192.168.0.110:6443 max_fails=3 fail_timeout=10s;
    }

log_format basic '$remote_addr [$time_local] '
                 '$protocol $status $bytes_sent $bytes_received '
                 '$session_time "$upstream_addr" '
                 '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';

    access_log /var/log/nginx/access.log basic;
    error_log  /var/log/nginx/error.log;
    server {
        listen                  443;
        ssl_preread             on;
        proxy_connect_timeout   20s;  # max time to connect to pserver
        proxy_timeout           30s;  # max time between successive reads or writes
        proxy_pass              127.0.0.1:$internalport;
    } 
    server {
        listen                  6443;
        ssl_preread             on;
        proxy_connect_timeout   20s;  # max time to connect to pserver
        proxy_timeout           30s;  # max time between successive reads or writes
        proxy_pass              127.0.0.1:$internalport;
    }
    server {
        listen 9441;
        proxy_pass sno1_ingress;
        proxy_next_upstream on;
    }
    server {
        listen 9442;
        proxy_pass sno2_ingress;
        proxy_next_upstream on;
    }
    server {
        listen 6441;
        proxy_pass sno1_api;
        proxy_next_upstream on;
    }
    server {
        listen 6442;
        proxy_pass sno2_api;
        proxy_next_upstream on;
    }
}

ODF installation

mika published on 2023-02-15 included in Openshift

Install Openshift Data Foundation from Operator Hub

Create a StorageSystem using “Connect an external storage platform” of Red Hat Ceph Storage type

Download ceph-external-cluster-details-exporter.py script and run it on your ceph admin node

$ python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name testrbd --cephfs-data-pool-name cephfs.testfs.data --rgw-endpoint 10.0.0.n:80 --cephfs-filesystem-name testfs

Sample output :

[{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "ceph1=10.0.0.n:6789", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "5dabcb8e-ad19-11ed-a179-005056af8aeb", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "client.healthchecker", "userKey": "********************"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "********"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "testrbd"}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "10.0.0.n", "MonitoringPort": "9283"}}, {"name": "rook-ceph-dashboard-link", "kind": "Secret", "data": {"userID": "ceph-dashboard-link", "userKey": "https://10.0.0.n:8443/"}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "************"}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "***********"}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "*************"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "testfs", "pool": "cephfs.testfs.data"}}, {"name": "ceph-rgw", "kind": "StorageClass", "data": {"endpoint": "10.0.0.n:80", "poolPrefix": "default"}}, {"name": "rgw-admin-ops-user", "kind": "Secret", "data": {"accessKey": "************************", "secretKey": "**********************"}}]

Save the json file and import it in the StorageSystem wizard

rbd pool must be replicated because Erasure-Coded RBD pool(s) are not supported in ODF.

Qemu cloud-init

mika published on 2023-02-14 included in Linux

Create VM from template