Integrating On Premise Registries with an IBM Cloud Kubernetes Service Cluster

System diagram of a Kubernetes Node Pulling a Container Image from an On-Premise Registry.

This guide will walk through a solution to integrate a On-Premise registry with an IBM Cloud Kubernetes cluster. While the guide focuses on IBM Cloud Kubernetes clusters, a similar strategy can be used in any Kubernetes cluster made up of machines running Ubuntu 18.04 or later using containerd as the container runtime. The solution makes use of coredns DNS proxying and systemd-resolved DNS routing (support for using different DNS servers to resolve different domains). This guide assumes the user has already configured the environment to have IP connectivity to all the On-Premise components.

There’s normally a couple challenges a user faces when trying to integrate an on premise registry (like Artifactory) in a Kubernetes cluster:

1. The DNS entries for the On-Premise registry are only served by On-Premise DNS resolvers. Typically the operating system is not configured out of the box to send DNS traffic to these On-Premise resolvers. Additionally: cloud providers normally have internal DNS resolvers that serve internal domains that the Kubernetes nodes depend on like NTP server addresses, Cloud metadata endpoints, etc. If the machine is configured to send all DNS traffic to the On-Premise resolvers, the internal cloud domains will fail to resolve and stability of the base operating system will be impacted.
2. The X.509 certs utilized by the On-Premise registry are not issued by a Certificate Authority that is trusted by default in the base operating system.

A solution to these problems is a deployment of a Kubernetes Daemonset that will be configured with the following data:

1. The On-Premise DNS domains that map to the On-Prem registry IP Addresses.
2. The Kubernetes Cluster CoreDNS service IP to forward On-Premise DNS domain lookups to.
3. The certificate of the Certificate Authority that issued the On-Premise registry certificates.

The Kubernetes Daemonset will perform the following actions

1. Add the On-Prem registry Certificate Authority to the trust store of the operating system.
2. Add configuration to systemd-resolved to send DNS lookups of the On-Premise registry domains to CoreDNS.
3. Ensure downstream components pick up the configuration change by reloading/restarting the systemd services.

A generic version of the daemonset is shown below:

The variables that need to be filled in are surrounded in ``. The key variables are:

• DNS_SERVICE_IP: The service IP of the coredns service
• SUBDOMAIN_TO_BE_FORWARDED_TO_ON_PREM_RESOLVES: the domain or subdomain that will be forwarded first to CoreDNS which will then forward to the On-Prem resolvers
• CA_CERTS_THAT_ISSUED_ARTIFACTORY_CERTS: Certificate Authority that issued the X.509 certs used in the On-Premise registry
• CA_CERTS_UNIQUE_IDENTIFIER: Unique directory name to store the Certificate Authority for the registry in. Usually the same value as SUBDOMAIN_TO_BE_FORWARDED_TO_ON_PREM_RESOLVES

The additional CoreDNS configuration to send the On-Premise registry domains to the On-Premise DNS resolvers can be added in the CoreDNS configuration file. This configuration needs to be generated by the user based on the topology of their On-Premise network. The CoreDNS forward plugin documentation and IBM Cloud Kubernetes Service Customizing the Cluster DNS provider documentation can be consulted for examples to base the configuration on for ones network.

End to End Example Walkthrough

This section walks through an example of integrating an On-Premise registry that has X509 certs valid for the domain docker-registry.default.svc.cluster.local issued from the Let’s Encrypt Staging Root CA. Note that the Let’s Encrypt Staging Root CA is not safe to use outside of a testing/experimental environment. First, the proper configuration needs to be added into the daemonset. The sample IBM Cloud Kubernetes cluster this example is being executed :

• DNS_SERVICE_IP: 172.21.0.10
• SUBDOMAIN_TO_FORWARD_TO_ON_PREM_RESOLVERS: registry.default.svc.cluster.local
• CA_CERTS_THAT_ISSUED_ARTIFACTORY_CERTS: The Let's Encrypt Staging CA since it issued my registry certs in this example

The full example daemonset with that info filled in is shown below:

The guide assumes the CoreDNS configuration has already been setup and the domain can be resolved by sending the DNS request to the CoreDNS service. This configuration can be validated in the environment by running a DNS lookup for registry.default.svc.cluster.local from a pod running in the cluster and ensuring the domain resolves when the DNS request is sent to the CoreDNS service IP. The following command validates this requirement by executing nslookup on the domain in example pod nslookup-ds-8pstc:

kubectl exec nslookup-ds-8pstc -- nslookup docker-registry.default.svc.cluster.local 172.21.0.10
Server:        172.21.0.10
Address:    172.21.0.10#53

Name:    docker-registry.default.svc.cluster.local
Address: 10.10.5.5

At this point, the daemonset to configure the cluster nodes to be able to pull from the On-Premise registry is ready to be deployed. The deployment is executed by applying the yaml.

kubectl apply -f https://gist.githubusercontent.com/relyt0925/f133de71955dd5fec6dbf2a3cb8c5744/raw/4d3cba78be561ade3afcc13b307edc0dd9e486a1/registry_configurator_ds_filled.yaml

A validation daemonset can now be deployed to verify the system. This daemonset will reference an image hosted in the On-Premise registry to validate that the registry can be used on every node in the cluster. This section assumes the user has already configured the pull secrets to authenticate to the On-Premise registry if applicable. The Pull an Image from a Private Registry Kubernetes documentation provides more details on how to configure registry pull-secrets. The test daemonset for the example environment is shown below along with the command to apply it into the environment:

kubectl apply -f https://gist.githubusercontent.com/relyt0925/68fcacfff8ee05c2414c28c00ec65ea8/raw/f3432006e98f2e391babecfc18e60793ec8251d5/test-private-registry-pull-ds.yaml
daemonset.apps/test-private-registry-pull-ds created

Then the registry integration can be validated by ensuring the pods referencing the On-Premise registry images are in Running state on all nodes. We can also see in the events of the pods the image was pulled from the registry:

kubectl get pods -o wide
NAME                                  READY   STATUS    RESTARTS   AGE     IP              NODE            NOMINATED NODE   READINESS GATES
private-registry-configurator-5rzrc   1/1     Running   0          138m   172.17.89.130    10.240.0.25     <none>           <none>
private-registry-configurator-87nfk   1/1     Running   0          134m   172.17.99.131    10.240.64.10    <none>           <none>
private-registry-configurator-d6nxr   1/1     Running   0          136m   172.17.121.2     10.240.128.17   <none>           <none>
private-registry-configurator-fv7c8   1/1     Running   0          84s    172.17.112.2     10.240.128.22   <none>           <none>
private-registry-configurator-nqt2p   1/1     Running   0          139m   172.17.99.9      10.240.64.12    <none>           <none>
private-registry-configurator-p58vj   1/1     Running   0          138m   172.17.112.196   10.240.0.24     <none>           <none>
private-registry-configurator-qc5cv   1/1     Running   0          138m   172.17.113.131   10.240.128.14   <none>           <none>
test-private-registry-pull-ds-48jn7   1/1     Running   0          139m   172.17.99.11     10.240.64.12    <none>           <none>
test-private-registry-pull-ds-8dzm8   1/1     Running   0          138m   172.17.112.195   10.240.0.24     <none>           <none>
test-private-registry-pull-ds-kvpcm   1/1     Running   0          136m   172.17.121.3     10.240.128.17   <none>           <none>
test-private-registry-pull-ds-kwg42   1/1     Running   0          138m   172.17.113.133   10.240.128.14   <none>           <none>
test-private-registry-pull-ds-s498m   1/1     Running   0          134m   172.17.99.130    10.240.64.10    <none>           <none>
test-private-registry-pull-ds-xh64n   1/1     Running   0          84s    172.17.112.4     10.240.128.22   <none>           <none>
test-private-registry-pull-ds-z8lj6   1/1     Running   0          138m   172.17.89.129    10.240.0.25     <none>           <none>

kubectl describe pod test-private-registry-pull-ds-xh64n
Events:
  Type    Reason     Age   From                  Message
  ----    ------     ----  ----                  -------
  Normal   Pulling                 38s (x2 over 62s)  kubelet, 10.240.128.22  Pulling image "docker-registry.default.svc.cluster.local:5000/openshift/httpd:2.4"
  Normal   Pulled                  27s                kubelet, 10.240.128.22  Successfully pulled image "docker-registry.default.svc.cluster.local:5000/openshift/httpd:2.4" in 11.050472979s
  Normal   Created                 20s                kubelet, 10.240.128.22  Created container test-pull
  Normal   Started                 20s                kubelet, 10.240.128.22  Started container test-pull

Congragulations! At this point the On-Premise registry has been successfully configured for use in the cluster.

Notes

• This is not an official IBM Cloud guide.
• This guide assumes the user has already configured the IBM Cloud Kubernetes cluster to have IP connectivity to all the On-Premise components
• The guide assumes the CoreDNS configuration has already been setup and the domain can be resolved by sending the DNS request to the CoreDNS service.
• The provided daemonset requires connectivity to the UBI repositories. This can be removed if a container image is built that includes the util-linux package installed with microdnf install util-linux -y
• The End To End Example Walkthrough will not work in any user environment with the example parameters used.
• There are different ways to solve this problem. This guide outlines one potential solution.