Kubernetes HA Cluster installation guide

Posted on 4 oktober 2018 | Category :Nieuws | Geen reacties op Kubernetes HA Cluster installation guide

Hardware overview

Hostname Specs IP
k8s-lb01 1 CPU, 1GB RAM, 10GB Disk 192.168.11.10
k8s-lb02 1 CPU, 1GB RAM, 10GB Disk 192.168.11.11
k8s master VIP 192.168.11.12
k8s-master01 2 CPU, 4GB RAM, 30GB disk 192.168.11.13
k8s-master02 2 CPU, 4GB RAM, 30GB disk 192.168.11.14
k8s-master03 2 CPU, 4GB RAM, 30GB disk 192.168.11.15
k8s-node01 2 CPU, 4GB RAM, 30GB disk 192.168.11.16
k8s-node02 2 CPU, 4GB RAM, 30GB disk 192.168.11.17
k8s-node02 2 CPU, 4GB RAM, 30GB disk 192.168.11.18
k8s IP pool 192.168.11.20-30
  • All servers run Ubuntu 18.04.1 LTS.
  • The loadbalancers will run keepalived with HAProxy.

Kubernetes setup:

Version: 1.12.0 Docker Version: 18.06.0 Extra Plugins used:

Create a DNS record that point to 192.168.11.12 (e.g. k8s.domain.com) kubectl and kubeadm will use this address to join/manage the cluster

Install Loadbalancers

k8s-lb01 / k8s-lb02

  • Install the required packages:
sudo apt install linux-headers-$(uname -r)
sudo apt install keepalived
sudo apt install haproxy
  • Setup keepalived configuration (as root):
sudo mkdir -p /etc/keepalived
cat > /etc/keepalived/keepalived.conf <<EOF
global_defs {
   notification_email {
     user@domain.com
   }
   notification_email_from k8s-lb01.omain.ccom
   smtp_server localhost
   smtp_connect_timeout 30
}
vrrp_instance VI_1 {
    state MASTER
    interface ens192
    virtual_router_id 100
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.11.12
    }
}
EOF
  • Dont forget to change auth_pass to something more secure
  • Change interface ens192 to match your interface name
  • On k8s-lb02 change priority to 100
  • On k8s-lb02 change notificationemailfrom to k8s-lb02.domain.com
  • Setup haproxy configuration
cat > /etc/haproxy/haproxy.conf <<EOF
global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    # An alternative list with additional directives can be obtained from
    #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3

defaults
    log global
    mode    http
    option  httplog
    option  dontlognull
    option forwardfor
    option http-server-close
        timeout connect 5000
        timeout client  50000
        timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend http_stats
   bind *:8080
   mode http
   stats uri /haproxy?stats

frontend haproxy_kube
    bind 0.0.0.0:443
    mode tcp
    option tcplog
    timeout client  10800s
    default_backend masters

backend masters
    mode tcp
    option tcplog
    balance leastconn
    timeout server  10800s
    server k8s-master01 192.168.11.13:6443 check
    server k8s-master02 192.168.11.14:6443 check
    server k8s-master03 192.168.11.15:6443 check
EOF
  • Make sure https://192.168.11.12 is return error 503

On all masters and nodes

  • Install docker
wget https://raw.githubusercontent.com/rancher/install-docker/master/18.06.0.sh
sudo sh 18.06.0.sh
swapoff -a

Also remove swap disk from /etc/fstab

  • Install kubelet,kubeadm, kubectl on all servers
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

On k8s-master01

  • Create kubeadm-config.yaml for the first master in the cluster
cat > kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- "k8s.domain.com"
controlPlaneEndpoint: "k8s.domain.com:443"
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://192.168.11.13:2379"
      advertise-client-urls: "https://192.168.11.13:2379"
      listen-peer-urls: "https://192.168.11.13:2380"
      initial-advertise-peer-urls: "https://192.168.11.13:2380"
      initial-cluster: "k8s-master01=https://192.168.11.13:2380"
    serverCertSANs:
      - k8s-master01
      - 192.168.11.13
    peerCertSANs:
      - k8s-master01
      - 192.168.11.13
networking:
    # For flannel use 10.244.0.0/16, calico uses 192.168.0.0/16
    podSubnet: "10.244.0.0/16"
EOF
  • Initialize the kubernetes cluster at k8s-master01 \ Remember that this will use k8s.domain.com to verify the cluster is online, so make sure the loadbalancer are working correctly.\ I made a mistake by using SSL termination on my loadbalancers, but this will result in kubeadm to fail.
kubeadm init --config kubeadm-config.yaml
  • Copy certificates and other required files to the other master nodes (changes username for scp if needed).
scp -r /etc/kubernetes/pki ubuntu@192.168.11.14:./
scp -r /etc/kubernetes/pki ubuntu@192.168.11.15:./
scp -r /etc/kubernetes/admin.conf ubuntu@192.168.11.14:./
scp -r /etc/kubernetes/admin.conf ubuntu@192.168.11.15:./

We need these files later to make the other master nodes join the cluster.

  • Make sure you copy the kubeadm join command (with token), and save it so we make the worker nodes join the cluster.

On k8s-master02

  • Create kubeadm-config.yaml on k8s-master02
cat > kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- "k8s.domain.com"
controlPlaneEndpoint: "k8s.domain.com:443"
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://192.168.11.14:2379"
      advertise-client-urls: "https://192.168.11.14:2379"
      listen-peer-urls: "https://192.168.11.14:2380"
      initial-advertise-peer-urls: "https://192.168.11.14:2380"
      initial-cluster: "k8s-master01=https://192.168.11.13:2380,k8s-master02=https://192.168.11.14:2380"
      initial-cluster-state: existing
    serverCertSANs:
      - k8s-master02
      - 192.168.11.14
    peerCertSANs:
      - k8s-master02
      - 192.168.11.14
networking:
    # For flannel use 10.244.0.0/16, calico uses 192.168.0.0/16
    podSubnet: "10.244.0.0/16"
EOF
  • Copy certificates and other required files (change user if necessary)
USER=ubuntu
mkdir -p /etc/kubernetes/pki/etc
mv /home/${USER}/pki/ca.crt /etc/kubernetes/pki/
mv /home/${USER}/pki/ca.key /etc/kubernetes/pki/
mv /home/${USER}/pki/sa.pub /etc/kubernetes/pki/
mv /home/${USER}/pki/sa.key /etc/kubernetes/pki/
mv /home/${USER}/pki/front-proxy-ca.crt /etc/kubernetes/pki/
mv /home/${USER}/pki/front-proxy-ca.key /etc/kubernetes/pki/
mv /home/${USER}/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/ca.crt
mv /home/${USER}/pki/etcd/ca.key /etc/kubernetes/pki/etcd/ca.key
mv /home/${USER}/admin.conf /etc/kubernetes/admin.conf
  • Run the kubeadm phase commands
kubeadm alpha phase certs all --config kubeadm-config.yaml
kubeadm alpha phase kubelet config write-to-disk --config kubeadm-config.yaml
kubeadm alpha phase kubelet write-env-file --config kubeadm-config.yaml
kubeadm alpha phase kubeconfig kubelet --config kubeadm-config.yaml
systemctl start kubelet
  • Add k8s-master02 to etcd
export CP0_IP=192.168.11.13
export CP0_HOSTNAME=k8s-master01
export CP1_IP=192.168.11.14
export CP1_HOSTNAME=k8s-master02

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP1_HOSTNAME} https://${CP1_IP}:2380
kubeadm alpha phase etcd local --config kubeadm-config.yaml
  • Deploy control plane components and add k8s-master02 as master node
kubeadm alpha phase kubeconfig all --config kubeadm-config.yaml
kubeadm alpha phase controlplane all --config kubeadm-config.yaml
kubeadm alpha phase mark-master --config kubeadm-config.yaml

On k8s-master03

  • Create kubeadm-config.yaml on k8s-master03
cat > kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServerCertSANs:
- "k8s.domain.com"
controlPlaneEndpoint: "k8s.domain.com:443"
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://192.168.11.15:2379"
      advertise-client-urls: "https://192.168.11.15:2379"
      listen-peer-urls: "https://192.168.11.15:2380"
      initial-advertise-peer-urls: "https://192.168.11.15:2380"
      initial-cluster: "k8s-master01=https://192.168.11.13:2380,k8s-master02=https://192.168.11.14:2380,k8s-master03=https://192.168.11.15:2380"
      initial-cluster-state: existing
    serverCertSANs:
      - k8s-master03
      - 192.168.11.15
    peerCertSANs:
      - k8s-master03
      - 192.168.11.15
networking:
    # This CIDR is a calico default. Substitute or remove for your CNI provider.
    podSubnet: "10.244.0.0/16"
EOF
  • Copy certificates and other required files (change user if necessary)
USER=ubuntu
mkdir -p /etc/kubernetes/pki/etc
mv /home/${USER}/pki/ca.crt /etc/kubernetes/pki/
mv /home/${USER}/pki/ca.key /etc/kubernetes/pki/
mv /home/${USER}/pki/sa.pub /etc/kubernetes/pki/
mv /home/${USER}/pki/sa.key /etc/kubernetes/pki/
mv /home/${USER}/pki/front-proxy-ca.crt /etc/kubernetes/pki/
mv /home/${USER}/pki/front-proxy-ca.key /etc/kubernetes/pki/
mv /home/${USER}/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/ca.crt
mv /home/${USER}/pki/etcd/ca.key /etc/kubernetes/pki/etcd/ca.key
mv /home/${USER}/admin.conf /etc/kubernetes/admin.conf
  • Run the kubeadm phase commands
kubeadm alpha phase certs all --config kubeadm-config.yaml
kubeadm alpha phase kubelet config write-to-disk --config kubeadm-config.yaml
kubeadm alpha phase kubelet write-env-file --config kubeadm-config.yaml
kubeadm alpha phase kubeconfig kubelet --config kubeadm-config.yaml
systemctl start kubelet
  • Add k8s-master03 to etcd
export CP0_IP=192.168.11.13
export CP0_HOSTNAME=k8s-master01
export CP2_IP=192.168.11.15
export CP2_HOSTNAME=k8s-master03

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP2_HOSTNAME} https://${CP2_IP}:2380
kubeadm alpha phase etcd local --config kubeadm-config.yaml
  • Deploy control plane components and add k8s-master03 as master node
kubeadm alpha phase kubeconfig all --config kubeadm-config.yaml
kubeadm alpha phase controlplane all --config kubeadm-config.yaml
kubeadm alpha phase mark-master --config kubeadm-config.yaml

On all worker nodes

  • Add the worker node to the cluster (this is the kubeadm join command we saved earlier)
sudo kubeadm join k8s.domain.com:443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

On a system that runs kubectl

  • Install MetalLB
cat > metallb-configmap.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.11.20-192.168.11.30
EOF

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml
kubectl apply -f metallb-configmap.yaml
  • Install kubernetes-ingress
kubectl apply -f https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/master/deployments/common/ns-and-sa.yaml
kubectl apply -f https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/master/deployments/common/default-server-secret.yaml
kubectl apply -f https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/master/deployments/common/nginx-config.yaml
kubectl apply -f https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/master/deployments/rbac/rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/master/deployments/deployment/nginx-ingress.yaml
kubectl apply -f https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/master/deployments/service/loadbalancer.yaml

Troubleshooting

DNS is not working inside pods

When running kubectl get pods -n kube-system I noticed that coredns pods were not running correctly. This appeared to be a problem in Ubuntu 18.04, which has not been fixed yet. \ As a workarround run the following command on all masters and worker nodes

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

After that delete the coredns pods. Dont worry, they will be recreated by the coredns deployment in your kube-system namespace.

Master / worker nodes stay in NotReady state

After some digging I found that the file /etc/cni/net.d/10-flannel.conflist was missing. After creating it, and restarting kubelet the nodes showed up correctly, and flannel components were installed.

cat > /etc/cni/net.d/10-flannel.conflist <<EOF 
{
  "name": "cbr0",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}
EOF
systemctl kubelet restart
» Tags: , , , , , ,

Comments 0

Geef een reactie