Using Kubernetes Certificate Signing Requests and RBAC for User Authentication and Authorization
There is a need in Kubernetes environments without a backend identity provider for a form of automated user management. Typically there are two strategies for provisioning user identities without utilizing a backing identity provider: Service Account Tokens and Kubernetes Certificate Signing Requests. This guide will walk through how Kubernetes Certificate Signing Requests can be utilized to distribute certificates that associate a user with a unique identity that can then be assigned access to a Kubernetes cluster with RBAC.
Kubernetes uses well defined fields in X509 certs that are signed by the Kubernetes Cluster CA to associate a given request with a user identity. The Common Name of the certificate maps to the user name of the request. The Organization fields in the certificate defines the group memberships of the user. Using the Kubernetes Certificate Signing Request API: certificates can be requested dynamically and if approved the user of the certificate will be associated with the values of the fields in the certificate. We can illustrate this strategy by provisioning an identity for an SRE and a Development member for a sample organization. The SRE member and Development member will have different levels of access: in our example each will be assigned access to a unique namespace.
The first step of the process is defining the private key and associated certificate signing request of that key that will be sent to the Kubernetes cluster for approval. We will use cfssl to generate these artifacts and the commands are shown below:
cat <<EOF >/tmp/sre-csr.json
{
"CN": "myorg-sre-1",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"O": "myorg-sre"
}
]
}
EOF
cat <<EOF >/tmp/dev-csr.json
{
"CN": "myorg-dev-1",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"O": "myorg-dev"
}
]
}
EOF
rm -rf /tmp/csr-dir
mkdir -p /tmp/csr-dir
set -o pipefail
cd /tmp/csr-dir || exit 1
cfssl genkey /tmp/sre-csr.json | cfssljson -bare sre
cfssl genkey /tmp/dev-csr.json | cfssljson -bare dev
DEV_CSR_CONTENT_64=$(cat dev.csr | base64 -w 0)
SRE_CSR_CONTENT_64=$(cat sre.csr | base64 -w 0)At the end of the automation: there will be a key/certificate signing request pair for both the SRE member and the Development member. With these artifacts generated: the certificate signing requests can be sent to the Kubernetes apiserver:
cat <<EOF >/tmp/kube-csr-dev.yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: myorg-dev-1
spec:
request: $DEV_CSR_CONTENT_64
signerName: kubernetes.io/kube-apiserver-client
usages:
- client auth
EOF
cat <<EOF >/tmp/kube-csr-sre.yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: myorg-sre-1
spec:
request: $SRE_CSR_CONTENT_64
signerName: kubernetes.io/kube-apiserver-client
usages:
- client auth
EOF
kubectl apply -f /tmp/kube-csr-sre.yaml
kubectl apply -f /tmp/kube-csr-dev.yaml
kubectl get csrNAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
myorg-dev-1 8s kubernetes.io/kube-apiserver-client system:admin <none> Pending
myorg-sre-1 9s kubernetes.io/kube-apiserver-client system:admin <none> Pending
When the CertificateSigningRequest instances are applied: the certificate signing requests are now issued to the Kubernetes cluster and are awaiting approval. When approved: a certificate will be issued that is signed by the cluster’s certificate authority which will then issue a cert that the members can utilize in combination with the key tied to the certificate signing request to authenticate to the cluster’s apiserver. The approval is typically performed by a cluster admin and the commands are shown below:
kubectl certificate approve myorg-dev-1
certificatesigningrequest.certificates.k8s.io/myorg-dev-1 approvedkubectl certificate approve myorg-sre-1
certificatesigningrequest.certificates.k8s.io/myorg-sre-1 approved
Once approved the base64 encoded signed certificates can then be fetched with the following command
DEV_CERTIFICATE_64=$(kubectl get csr myorg-dev-1 -o jsonpath='{.status.certificate}')
SRE_CERTIFICATE_64=$(kubectl get csr myorg-sre-1 -o jsonpath='{.status.certificate}')The key content can be gathered from the key files on the local system that generated the key/certificate signing request pair and base64 encoded with the following:
DEV_KEY_CONTENT_64=$(cat /tmp/csr-dir/dev-key.pem | base64 -w 0)
SRE_KEY_CONTENT_64=$(cat /tmp/csr-dir/sre-key.pem | base64 -w 0)Using those values: kubeconfigs can be generated for each identity with the following commands (the kube-apiserver address is a well known address that is input to the automation and can be found by looking at the kubeconfig that was originally used to create the CSRs):
#example apiserver endpoint: https://c100-e.us-
east.containers.cloud.ibm.com:31581
APISERVER_ENDPOINT=https://my-api-server-address:my-api-port
cat <<EOF >/tmp/sre.kubeconfig
apiVersion: v1
clusters:
- cluster:
server: $APISERVER_ENDPOINT
name: mycluster
contexts:
- context:
cluster: mycluster
namespace: default
user: sre/mycluster
name: mycluster/sre
current-context: mycluster/sre
kind: Config
preferences: {}
users:
- name: sre/mycluster
user:
client-certificate-data: $SRE_CERTIFICATE_64
client-key-data: $SRE_KEY_CONTENT_64
EOF
cat <<EOF >/tmp/dev.kubeconfig
apiVersion: v1
clusters:
- cluster:
server: $APISERVER_ENDPOINT
name: mycluster
contexts:
- context:
cluster: mycluster
namespace: default
user: dev/mycluster
name: mycluster/dev
current-context: mycluster/dev
kind: Config
preferences: {}
users:
- name: dev/mycluster
user:
client-certificate-data: $DEV_CERTIFICATE_64
client-key-data: $DEV_KEY_CONTENT_64
EOFWe can confirm we are properly authenticating to the apiserver by issuing a request with the generated kubeconfigs:
kubectl --kubeconfig /tmp/dev.kubeconfig get node
Error from server (Forbidden): nodes is forbidden: User "myorg-dev-1" cannot list resource "nodes" in API group "" at the cluster scopekubectl --kubeconfig /tmp/sre.kubeconfig get node
Error from server (Forbidden): nodes is forbidden: User "myorg-sre-1" cannot list resource "nodes" in API group "" at the cluster scope
At this point we now have the authentication information and can make requests as the appropriate identities: we will now assign access to each group to allow it full access to a unique namespace. Dev will get access to the default namespace and SRE will get access to the kube-system namespace. The associated RBAC to accomplish that is shown below:
cat <<EOF >/tmp/sre-namespace-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: myorg-sre-team
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: myorg-sre-team
subjects:
- kind: Group
name: myorg-sre
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: myorg-sre-team
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- "*"
verbs:
- "*"
EOF
cat <<EOF >/tmp/dev-namespace-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: myorg-dev-team
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: myorg-dev-team
subjects:
- kind: Group
name: myorg-dev
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: myorg-dev-team
namespace: default
rules:
- apiGroups:
- ""
resources:
- "*"
verbs:
- "*"
EOFkubectl apply -f /tmp/dev-namespace-rbac.yaml
rolebinding.rbac.authorization.k8s.io/myorg-dev-team created
role.rbac.authorization.k8s.io/myorg-dev-team createdkubectl apply -f /tmp/sre-namespace-rbac.yaml
rolebinding.rbac.authorization.k8s.io/myorg-sre-team created
role.rbac.authorization.k8s.io/myorg-sre-team created
When that RBAC is created: we can now show that Development can access resources in the default namespace but not kube-system:
kubectl --kubeconfig /tmp/dev.kubeconfig get pods -n default
NAME READY STATUS RESTARTS AGE
prometheus-pushgateway-1597420487-5f6d5c89f9-2m8pt 1/1 Running 0 6h13m
ruby-hello-world-67b66d9d69-m7dtr 1/1 Running 0 6h13mkubectl --kubeconfig /tmp/dev.kubeconfig get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "myorg-dev-1" cannot list resource "pods" in API group "" in the namespace "kube-system"
Additionally, we can show that SRE can access resources in the kube-system namespace but not the default namespace:
kubectl --kubeconfig /tmp/sre.kubeconfig get pods -n kube-systemNAME READY STATUS RESTARTS AGE
ibm-keepalived-watcher-ksp6s 1/1 Running 0 6h5m
ibm-keepalived-watcher-lmb2z 1/1 Running 0 6h4m
ibm-keepalived-watcher-wq6ll 1/1 Running 0 6h5m
ibm-master-proxy-static-10.240.0.12 2/2 Running 0 6h3m
ibm-master-proxy-static-10.240.0.13 2/2 Running 0 6h4m
ibm-master-proxy-static-10.240.0.14 2/2 Running 0 6h4m
ibm-storage-metrics-agent-b44665c75-mxcj4 1/1 Running 0 5h3m
ibm-vpc-block-csi-controller-0 4/4 Running 0 6h18m
ibm-vpc-block-csi-node-km9fv 3/3 Running 0 6h5m
ibm-vpc-block-csi-node-qbk26 3/3 Running 0 6h5m
ibm-vpc-block-csi-node-x4bzh 3/3 Running 0 6h4m
vpn-77577d6bb9-jx28d 1/1 Running 0 5h56mkubectl --kubeconfig /tmp/sre.kubeconfig get pods -n defaultError from server (Forbidden): pods is forbidden: User "myorg-sre-1" cannot list resource "pods" in API group "" in the namespace "default"
Congratulations! At this point you have successfully setup two unique identities using Kubernetes Certificate Signing Requests!
The certificates will have a duration that corresponds to either the configured certificate signing duration in the kube controller manager (default is a year) or to the value specified in the CSR request if applicable. Additionally: if an exposure of that cert happens before the end of the duration: CA rotation can be performed on a cluster to rotate CAs and revoke the certs from being able to authenticate to the APIServer. RBAC can also be revoked for the exposed identity by deleting the role bindings/cluster role bindings that grant authorization to the identities associated with the certificate. Certificates can be renewed by issuing another CSR request to the Kubernetes apiserver at the desired time for renewal.
Fully Automated Steps
Below is a script that automates all the steps walked through above and will result in the creation of the Development and SRE kubeconfig. The following prereqs must be met before running the automation:
Prerequisites:
- Ensure the url for the Kubernetes apiserver is set in variable APISERVER_ENDPOINT
- Ensure an admin kubeconfig is set as the active kubeconfig for the appropriate cluster before running the automation
- Ensure cfssl and cfssljson are downloaded and properly exposed in the system path
