Proxying In Cluster Kube-APIServer Traffic in IBM Cloud Satellite
This guide outlines a strategy to proxying in cluster kube-apiserver traffic in IBM Cloud Satellite through an external TCP proxy. This setup has been utilized in secure environments where there is a requirement for all outbound traffic from a Satellite Location to flow through a dedicated proxy so it can be analyzed by an organizations security team. The solution will make use of /etc/hosts DNS overrides to redirect traffic from going direct to the backend endpoints to instead go to the proxy. The proxy is then responsible for forwarding that TCP stream to the backend after performing the appropriate analysis.
First the destination domains and destination port of the kube-apiserver traffic needs to be identified. This can be gathered with the command shown below:
ibmcloud sat location get --location tyler-hypershift-1
Retrieving location...
OKName: tyler-hypershift-1
ID: c9a383fd0t47h8eusqd0
Created: 2022-04-11 08:58:37 -0500 (2 weeks ago)
Managed From: dal
State: normal
Ready for deployments: yes
Message: R0001: The Satellite location is ready for operations.
Hosts Available: 0
Hosts Total: 5
Host Zones: us-south-1, us-south-2, us-south-3
Provider: -
Provider Region: -
Provider Credentials: no
Public Service Endpoint URL: https://c131-e.us-south.satellite.cloud.ibm.com:31726
Private Service Endpoint URL: -
OpenVPN Server Port: -
Ignition Server Port: 32186
Konnectivity Server Port: 31311
The key section to note is the Public Service Endpoint URL. The 3 domains that will need to be proxied on the hosts for this Satellite Location can be derived. To get the 3 hosts: the -e is simply replaced with -1 -2 -3. That would result in the following 3 endpoints:
https://c131-1.us-south.satellite.cloud.ibm.com:31726
https://c131-2.us-south.satellite.cloud.ibm.com:31726
https://c131-3.us-south.satellite.cloud.ibm.com:31726If the user desires to ultimately proxy to the private endpoints: the -e is replaced with -1–1, -2–1 , -3–1 and adding the domain name label private after the initial label in the domain. In this example: it would result in the following 3 endpoints
https://c131-1-1.private.us-south.satellite.cloud.ibm.com:31726
https://c131-2-1.private.us-south.satellite.cloud.ibm.com:31726
https://c131-3-1.private.us-south.satellite.cloud.ibm.com:31726These are the endpoints that need to be routed through the proxy in order to analyze the in-cluster kube-apiserver traffic. With these identified: the external proxy needs to be setup that will accept this TCP traffic and forward to one of these destinations. The 3 domains provide 3 unique redundant points of ingress and the proxy ideally should forward to all 3 endpoints for redundancy purposes. This will be illustrated by provisioning a separate VM with IP connectivity to the Satellite Hosts and have it run HAProxy with the following configuration:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
maxconn 7000
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
mode tcp
option tcplog
option log-health-checks
log global
option allbackups
balance roundrobin
timeout client 10m
timeout server 10m
timeout connect 10s
timeout client-fin 5s
timeout server-fin 5s
timeout queue 5s
retries 3
frontend masterapiserverfrontend
bind 0.0.0.0:31726
mode tcp
log global
option tcplog
option clitcpka
default_backend masterapiserverbackend
backend masterapiserverbackend
mode tcp
balance roundrobin
log global
server c131-1.us-south.satellite.cloud.ibm.com c131-1.us-south.satellite.cloud.ibm.com:31726
server c131-2.us-south.satellite.cloud.ibm.com c131-2.us-south.satellite.cloud.ibm.com:31726
server c131-3.us-south.satellite.cloud.ibm.com c131-3.us-south.satellite.cloud.ibm.com:31726 root@tyler-test-1:~# ip addr
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:00:3d:0a:4f:b8 brd ff:ff:ff:ff:ff:ff
inet 10.240.64.39/24 brd 10.240.64.255 scope global dynamic ens3
valid_lft 194sec preferred_lft 194sec
inet6 fe80::3dff:fe0a:4fb8/64 scope link
valid_lft forever preferred_lft foreverroot@tyler-test-1:~# curl -k https://10.240.64.39:31726/version
{
"major": "1",
"minor": "22",
"gitVersion": "v1.22.5+5c84e52",
"gitCommit": "ce18cbe56f6e88a8fc0e06366afe113b415ad39b",
"gitTreeState": "clean",
"buildDate": "2022-03-01T18:44:38Z",
"goVersion": "go1.16.12",
"compiler": "gc",
"platform": "linux/amd64"
}
At this point we have now vetted that the proxy is running and properly sending traffic to the backend with the curl request we ran in the example command above.
Now that the proxy is configured: the final step is to redirect the traffic from the hosts for the domains to the proxy box instead of to the direct backend. This can be done by modifying /etc/hosts on the Satellite Hosts to point to the proxy box’s IP. The additions for the example shown in this guide are outlined below:
PROXY_BOX_IP_1 K8S_APISERVER_DOMAIN_1
PROXY_BOX_IP_2 K8S_APISERVER_DOMAIN_2
PROXY_BOX_IP_3 K8S_APISERVER_DOMAIN_3
PROXY_BOX_IP_4 K8S_APISERVER_DOMAIN_3Example)10.240.64.39 c131.us-south.satellite.cloud.ibm.com
10.240.64.39 c131-1.us-south.satellite.cloud.ibm.com
10.240.64.39 c131-2.us-south.satellite.cloud.ibm.com
10.240.64.39 c131-3.us-south.satellite.cloud.ibm.com----------------------[root@tyler-rhcos-1 /]# curl -kv https://c131-1.us-south.satellite.cloud.ibm.com:31726/version
* Trying 10.240.64.39...
* TCP_NODELAY set
* Connected to c131-1.us-south.satellite.cloud.ibm.com (10.240.64.39) port 31726 (#0)
{
"major": "1",
"minor": "22",
"gitVersion": "v1.22.5+5c84e52",
"gitCommit": "ce18cbe56f6e88a8fc0e06366afe113b415ad39b",
"gitTreeState": "clean",
"buildDate": "2022-03-01T18:44:38Z",
"goVersion": "go1.16.12",
"compiler": "gc",
"platform": "linux/amd64"
}
[root@tyler-rhcos-1 /]#
With this addition to /etc/hosts: the same curl command to test that traffic is flowing to the kube-apiserver can be ran from the Satellite Host. The curl output on the IP that is being contacted confirms the Satellite Host is reaching to the proxy box to establish the connection to the kube-apiserver. The proxy box is then able to analyze the traffic and if it is accepted forward to the backend.
One assumption this guide makes is the proxy box listens on the same port as the ultimate backend kube-apiserver port. If the proxy listens on a different port than the backend kube-apiserver port: additional IPTable rules must be added to the OUTPUT chain to properly DNAT the kube-apiserver port to the proxy hosts port on each Satellite Host.
