HA for kubeapi-load-balancer

The standard deployment of Charmed Kubernetes includes a single instance of the kube-api-loadbalancer. For many use cases this is perfectly adequate, but in a production environment you should be keen to eliminate any single point of failure.

The recommended way to provide a failover for the kube-api-loadbalancer on bare metal or MAAS is by using keepalived. This is available as a Juju charm and can be deployed into your Charmed Kubernetes model and configured as follows:

1. Deploy the keepalived charm:

    juju deploy keepalived
   

2. Add the required relations:

    juju add-relation keepalived:juju-info kubeapi-load-balancer:juju-info
    juju add-relation keepalived:lb-sink kubeapi-load-balancer:website
    juju add-relation keepalived:loadbalancer kubernetes-control-plane:loadbalancer
    juju add-relation keepalived:website kubernetes-worker:kube-api-endpoint
   

   This redirects both the Kubernetes master and worker units to point at the keepalived service rather than the api-endpoint directly.

3. Configure the keepalived application. You should substitute a suitable IP address and FQDN in the example below:

    export VIP=10.10.74.250
    export VIP_HOSTNAME=test.example.com
    juju config keepalived virtual_ip=$VIP
    juju config keepalived vip_hostname=$VIP_HOSTNAME
   

   Allocating a VIP and ensuring that it can route to all of the instances is a manual process which depends on your infrastructure. It does require that the VIP be able to route to each instance, and that the VRRP protocol is allowed on the network. While this should be the case on bare metal and MAAS, and can be made to work on OpenStack, it will generally not be possible on public clouds. Thus, it is generally better to in those cases to replace kubeapi-load-balancer with a cloud-provided load balancer with health checks, such as Octavia or ELB.

4. Add both the new hostname and VIP to the API server certificate. This is done by specifying additional SANs:

   juju config kubeapi-load-balancer extra_sans="$VIP $VIP_HOSTNAME"
   juju config kubernetes-control-plane extra_sans="$VIP $VIP_HOSTNAME"
   

5. Wait for the new service to settle. You can check the status of the keepalived application by running:

    juju status keepalived
   

   Once the application reports a ‘ready’ status, continue to the next step.

6. Remove unneeded relations:

    juju remove-relation kubernetes-worker:kube-api-endpoint kubeapi-load-balancer:website
    juju remove-relation kubernetes-control-plane:loadbalancer kubeapi-load-balancer:loadbalancer
   

7. Scale up the kubeapi-load-balancer. You can specify as many units as your situation requires. In this example, we add two additional units for a total of three:

    juju add-unit kubeapi-load-balancer -n 2
   

8. Check for correct functionality by using kubectl and verifying it returns results. You can also check the SANs listed in the certificate returned by the VIP.

    kubectl get pods --all-namespaces
    openssl s_client -connect $VIP:443 | openssl x509 -noout -text
   

Note that the keepalived application is a subordinate charm - it does not require a machine of its own to run on, but rather runs alongside the kubeapi-load-balancer charm. If for any reason you need to view logs or troubleshoot this application, it will be found co-located on the machines running the load balancer.