kube-cluster

Overview

Initially I used Linode VPS to host my internal project/poc, but beginning of this year I decided to build my own in-house X86 Kubernetes cluster (thanks to the pandemic :)) using X86 SBC.

Hardware

Software

  • CentOS 8
  • Kubernetes 1.18.X
  • Docker CE 19.03.X
  • MetalLB
  • HaProxy

Challenges

The most difficult part was cluster networking and choosing the right CNI. I have started my struggle with

  • Fannel: it work very well between pod-to-pod and with MetalLB but failed to connect outside pod networks e.g. LAN, which I need for external storage and other services.

  • Calico: after struggling with Fannel for few days, I have decided to use calico (which btw not fully supported by MetalLB) and had issue with both outside pod networks and BGP peer conflicts.

  • Cannel: after Calico I move to Cannel (Calico + Fannel) which worked out of the box and I didn’t had any issue so far and one big advantage was it was light weight and support network policy :)

here is how the both extranet/intranet traffic flows

extranet

intranet

Summary

So far things are looking promising except sometime issue with iSCSI persistent volume especially during NAS upgrade/restart due to existing iSCSI mounts/sessions are not un-mounted which causing conflicts and pods are failing with error

Mar 28 20:47:17 k8s-node02 kubelet[1211]: E1028 20:47:17.065398 1211 remote_runtime.go:222] StartContainer
"942a6" from runtime service failed: rpc error: code = Unknown desc = failed to start container
"942a66677": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/
dab15a/volumes/kubernetes.io~iscsi/iscsi-pv-monitoring': mkdir /var/lib/kubelet/pods/dab15aa/volumes/kubernetes.io~iscsi/iscsi-pv-monitoring: file exists

in order to fix this I need to kill all existing iSCSI session and mounts and restart the pods.

$# iscsiadm --mode node --targetname XXXXXX.esdata  --portal X.X.X.X:3260 --logout
$# umount /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/X.X.X.X:3260-XXXX.esdata-lun-0
Stats Value
Total number pods: 58
Total number external service: 1 spawnlab.dev
Total number internal service: 11 
$# kubectl get ingress --all-namespaces -o=custom-columns=Namesapce:.metadata.namespace,Host:.spec.rules[0].host --sort-by=.metadata.namespace
Namesapce        Host
auth             auth.spawnlab.dev
auth             login.spawnlab.dev
budget           budget-api.spawnlab.dev
docs             docs.spawnlab.dev
drone-ci         ci.spawnlab.dev
drone-ci         runner.spawnlab.dev
elastic-system   search.spawnlab.dev
elastic-system   logs.spawnlab.dev
monitoring       prometheus.spawnlab.dev
spawnlab-blog    spawnlab.dev
storage          s3.spawnlab.dev
vault            vault.spawnlab.dev
registry         registry.spawnlab.dev

master

node1

node2

Ref: