Working with Red Hat Enterprise Linux CoreOS Extensions

Red Hat Enterprise Linux CoreOS (RHCOS) is the default operating system for OpenShift. Recently OpenShift added support for RHCOS extensions which allows extending RHCOS in a controlled fashion.

In this article let’s see how to enable a new RHCOS extension. This is only required if you are an OpenShift engineer working on enhancements. As an OpenShift user you won’t need this.

The RHCOS contents are part of the OpenShift release image itself and is referred to as machine-os-content. Consequently adding a new RHCOS extension will require working with machine-os-content.

Let’s take an enhancement scenario and work through the steps.

Scenario

Add a new RHCOS extension — qemu-img to be deployed via OpenShift Machine Config Operator (MCO)

At a high level adding and deploying a new RHCOS extension consists of the following steps:

Let’s go through each of these steps in detail.

Creating new machine-os-content

We need to a select a base machine-os-content image to start with. Let’s say we use the latest 4.7 nightly build image as the base.

The release image details are available from the following link — https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp-dev-preview/latest-4.7/release.txt

Look for the text like this — “Pull From: quay.io/openshift-release-dev/ocp-release”

Pull From: quay.io/openshift-release-dev/ocp-release-nightly@sha256:ecfac365596289b8474e6a0f0bffea996e04b0bdd1379e8fb4776b8e3ef2af1b

Note that this will change as new builds are made available.

You’ll need OpenShift pull secret to download the content. Get it from the following link.

Assuming the pull secret is saved in the file openshift-pull-secret.txt you can extract the machine-os-content from the release payload image using the following command. This is useful if you want to view the contents and understand the layout of RHCOS.

oc image extract -a ~/openshift-pull-secret.txt — path /:/extracted-oscontent quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecfac365596289b8474e6a0f0bffea996e04b0bdd1379e8fb4776b8e3ef2af1b

You can now view the contents under /extracted-oscontent directory. For example you can find the current RHCOS extensions under extensions sub-directory.

The following Dockerfile extends this release image and adds the qemu-img package.

FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecfac365596289b8474e6a0f0bffea996e04b0bdd1379e8fb4776b8e3ef2af1bRUN dnf install -y createrepoRUN dnf install -y — downloadonly — downloaddir=/extensions/qemu-img qemu-imgRUN createrepo /extensionsENTRYPOINT [“bin/bash”]

Build the image.

podman build -t quay.io/bpradipt/extension-test -f Dockerfile .

Note: If the host is not registered via subscription manager then image build will not be able to download the required rpm packages. An alternative mechanism is to explicitly provide yum repos during build.

podman build -v /etc/yum.repos.d/Linux-AppStream.repo:/etc/yum.repos.d/Linux-AppStream.repo -v /etc/yum.repos.d/Linux-BaseOS.repo:/etc/yum.repos.d/Linux-BaseOS.repo -t test -f Dockerfile .

Push the image to the registry.

podman push quay.io/bpradipt/extension-test

Creating new MCO image with extension support

You’ll need to build the machine config operator (MCO) image with changes to add support for the new (qemu-img) extension.

Have a look at the following code for the MCO change to add “qemu-img” extension.

Build the MCO image.

git clone -b qemu-img https://github.com/bpradipt/machine-config-operator
cd machine-config-operator
make image
podman tag localhost/machine-config-operator:latest quay.io/bpradipt/machine-config-operator:latest

Push the image to the registry.

podman push quay.io/bpradipt/machine-config-operator:latest

Creating a new OpenShift release image

The final step is to create a new OpenShift release image and using it for installation and testing the extension.

oc adm release new -n origin --from-release=quay.io/openshift-release-dev/ocp-release-nightly@sha256:d1a968a86cf04a0e621c8ed265d932f365c5a3bc7977a8c0b1c6679d79f3b864 --to-image=docker.io/bpradipt/ocp-release:v4.7.1 machine-config-operator=quay.io/bpradipt/machine-config-operator:latest machine-os-content=quay.io/bpradipt/extension-test@sha256:83aae59f986b376813916807e3d19716e9df8f06b56661fce327c9e27c3e5896

Upgrade OpenShift Cluster with the new release image

For testing, we’ll upgrade an existing cluster with the new image and verify that the extension is getting installed.

oc adm upgrade --force --to-image docker.io/bpradipt/ocp-release:v4.7.1 --allow-explicit-upgrade

Update usually will take 20–30 min.

Watch Update progress by checking the PROGRESSING field.

oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.7.0-0.nightly-2021-01-28-060516 True True 42m Working towards 4.7.0-0.nightly-2021-01-19-083458: 84% complete

Once the upgrade is complete, check if MCO image points to the new image.

oc get pod machine-config-operator-7864794c9f-6jps8  -n openshift-machine-config-operator -o yaml[snip]containerStatuses:
- containerID: cri-o://1030351e1d11fe581757c4a43ae75bbd076e990bf7b96a2289fbf329f93a4d50image: quay.io/bpradipt/machine-config-operator:latest
imageID: quay.io/bpradipt/machine-config-operator@sha256:fc04ffa00e511396643be4bad6e84a3461d64b448720ff8748b8b40cd89363b2
lastState: {}
name: machine-config-operator
ready: true
[snip]

Check if osImageURL points to new machine-os-content

oc -n openshift-machine-config-operator edit configmap/machine-config-osimageurlapiVersion: v1
data:
osImageURL: quay.io/bpradipt/extension-test@sha256:83aae59f986b376813916807e3d19716e9df8f06b56661fce327c9e27c3e5896
releaseVersion: 4.7.0–0.nightly-2021–01–19–083458
kind: ConfigMap
metadata:
annotations:
include.release.openshift.io/ibm-cloud-managed: “true”
include.release.openshift.io/self-managed-high-availability: “true”
include.release.openshift.io/single-node-developer: “true”
creationTimestamp: “2021–01–19T10:12:44Z”
name: machine-config-osimageurl
namespace: openshift-machine-config-operator
resourceVersion: “44890”
selfLink: /api/v1/namespaces/openshift-machine-config-operator/configmaps/machine-config-osimageurl
uid: ff1ed7b1-bf9c-414d-8384–42d66633d0e7

Deploy the new extension

Check the worker nodes to ensure the “qemu-img” extension is not installed. For example here is the o/p from one of the worker nodes in my setup

oc debug node/ip-10–0–147–144.ec2.internal
Creating debug namespace/openshift-debug-node-rvnvx …
Starting pod/ip-10–0–147–144ec2internal-debug …
To use host binaries, run `chroot /host`
Pod IP: 10.0.147.144
If you don’t see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# rpm -qa | grep qemu-img

Now apply the extension by creating a machineconfig.

Following is the sample machineconfig used to apply the extension — 80-extensions.yaml

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 80-worker-extensions
spec:
config:
ignition:
version: 3.1.0
extensions:
- qemu-img
oc create -f 80-extensions.yaml

Watch progress

oc get machineconfigpools

Wait for the worker machineconfigpool to be updated. You should see an o/p like what is shown below. The UPDATING and DEGRADED columns should be FALSE

NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master rendered-master-02910293114b441bd4e08b574159dbeb True False False 3 3 3 0 107m
worker rendered-worker-d17e4a8e84ef124243d8397b79348839 True False False 3 3 3 0 107m

Login to the worker nodes and check for qemu-img extension

oc debug node/ip-10–0–147–144.ec2.internal
Creating debug namespace/openshift-debug-node-wccmn …
Starting pod/ip-10–0–147–144ec2internal-debug …
To use host binaries, run `chroot /host`
Pod IP: 10.0.147.144
If you don’t see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# rpm -qa | grep qemu-img
qemu-img-4.2.0–34.module_el8.3.0+613+9ec9f184.1.x86_64

Happy hacking !!

Reference

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store