Using (customized) Red Hat entitlements / subscriptions in builds for OpenShift 4, and/or custom repositories, like via Red Hat Satellite.
See Red Hat documentation. Which has this per-buildConfig change, as well as options on cluster-wide changes (involving MachineConfig changes)
https://examples.openshift.pub/build/entitled/
Background
OpenShift hosts, by default pass their entitlements through to buildConfigs. It does this by passing in a file: /etc/rhsm-host into the build containers. OpenShift 3 (OCP3) at {{clientName}} used VMs running RHEL 7, entitled to use a particular set of repositories on the {{clientName}} Red Hat Satellite server
The builds used by {{clientName}} used RHEL 7-based images. This allowed builds to easily install additional RPM packages (via the Satellite) without requiring any customization to the entitlementents.
OpenShift 4 (OCP4) at {{clientName}} uses the new Red Hat CoreOS immutable image-based VMs as hosts. These are based on RHEL 8, but using "images" instead of traditional server installs.
The buildConfigs still inherit the OpenShift hosts entitlements, however these differ from the OCP3 entitlements, in that they point to RHEL 8 and Red Hat Universal Base Image 8 (UBI8) subscriptions.
Thus the {{clientName}} builds failed during migration, due to the entitlement/repository missmatch.
Workaround
The workaround used was "Per-Build option 2", below. This supplies entitlement certificates, a custom Red Hat repository list for RHEL 7, and mask /etc/rhsm-host so that the build pods would utilize a different set of entitlements to the hosts.
Note: For each Red Hat major release, different a different set of entitlement certificates and redhat.repo file is required. If this is not done, builds could try to consume packages from RHEL7 and RHEL8 with unexpected/problematic results.
To do this the following k8s objects were required:
secret: etc-pki-entitlement, containing entitlement-key.pem and entitlement.pem.
secret: katello, containing the {{clientName}} Satellite server's identifying certificate.
secret: rhsm-conf, containing custom configuration for /etc/rhsm.conf
configMap: rhsm-mask, basically empty, to mask /etc/rhsm-host in the container
configMap: yum-repos-d, containing the appropriate set of RHEL 7-based repositories.
Changes were then made to the buildConfigs to make use of the above objects.
Additionally, some changes to the Dockerfile were recommended. These removed unnecessary or problematic steps, and reduced the number of layers used.
---------------------------
Per-Build option 1: Dockerfile changes, with entitlement certs copied in
This does not require buildConfig mounts/changes as the required files are pulled into the build via a COPY statement
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
labels:
app: entitled-buildconfig
name: entitled-buildconfig
spec: {}
---
apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
labels:
app: entitled-buildconfig
name: entitled-buildconfig
spec:
runPolicy: "Serial"
triggers:
- type: "ConfigChange"
- type: "ImageChange"
source:
dockerfile: "FROM registry.access.redhat.com/ubi8:latest\nCOPY *.pem /etc/pki/entitlement/entitlement.pem\nCOPY *.pem /etc/pki/entitlement/entitlement-key.pem\nRUN dnf search kernel-devel --showduplicates"
secrets:
- secret:
name: entitlement
strategy:
dockerStrategy:
buildArgs:
- name: "DRIVER_VERSION"
value: "418.87.01"
output:
to:
kind: ImageStreamTag
name: entitled-buildconfig:latestPer-Build options 2: BuildConfig customization:
This does not require Dockerfile changes, as it supplies the required files via volumeMounts in the BuildConfig
Requires the following to be able to override the underlying host's entitlement if they exist
- Custom redhat.repo file that gets mounted over /etc/yum.repos.d
- Custom entitlement identifiers (key and cert) pem files as a secret mounted at /etc/pki/entitlement
- Custom rhsm.conf which gets mounted over /etc/rhsm (and therefore needs to point configure some other locations)
- Katello CA mounted at /etc/rhsm-workaround/ca (see above)
- empty file or emptyDir mount to mask /etc/rhsm-host
myexample
apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
annotations: {}
labels: {}
name: myexample
namespace: mynamespace
spec:
failedBuildsHistoryLimit: 2
nodeSelector: null
output:
to:
kind: ImageStreamTag
name: myexample:latest
resources:
limits:
cpu: 500m
memory: 500Mi
requests:
cpu: 250m
memory: 100Mi
runPolicy: SerialLatestOnly
source:
contextDir: build-pipeline/Dockerfiles
git:
ref: develop
uri: ssh://git@git:2222/repos/myexample.git
sourceSecret:
name: myexample-ssh-secret
type: Git
strategy:
dockerStrategy:
dockerfilePath: myexample.dockerfile
env:
- name: CHECK_INTERVAL
valueFrom:
configMapKeyRef:
key: check-interval
name: environment-config
from:
kind: ImageStreamTag
name: ubi8:latest
volumes:
- mounts:
- destinationPath: /etc/yum.repos.d
name: yum-repos-d
source:
configMap:
defaultMode: 420
name: yum-repos-d
type: ConfigMap
- mounts:
- destinationPath: /etc/pki/entitlement
name: etc-pki-entitlement
source:
secret:
defaultMode: 420
secretName: etc-pki-entitlement
type: Secret
- mounts:
- destinationPath: /etc/rhsm
name: rhsm-conf
source:
secret:
defaultMode: 420
secretName: rhsm-conf
type: Secret
- mounts:
- destinationPath: /etc/rhsm-workaround/ca
name: katello
source:
secret:
defaultMode: 420
secretName: katello
type: Secret
- mounts:
- destinationPath: /etc/rhsm-host
name: rhsm-host
source:
configMap:
defaultMode: 420
name: rhsm-mask
type: ConfigMap
type: Docker
successfulBuildsHistoryLimit: 2
triggers:
- type: ConfigChange