One thing we see all the time is people asking what the differences are in resource consumption, between k0s, k3s, and microk8s...
Well, let me show you..
I deployed 3x VMs, each with 1 CPU and 1 GB RAM. This is to represent a constrained node that would often be seen in Edge Compute environments. On each of these i deployed Kubernetes using their default settings. I then ran a simple "free -m" command on each to see how much RAM was being used to "idle" the cluster.
Note, ignore the "free" value as its misleading.. actual free is a combination of free and reclaimable buffer cache.
Whilst MicroK8s installed, i was unable to run ANY MicroKs8s commands...
I contacted Canonical to ask their advise about the above, and was told that by default, MicroK8s installs the Calico network driver and uses dqlite rather then etcd, which is not needed for single-node deployments. They recommended I disable HA (switches from Calico to Flannel and dqlite to etcd) using the command "microk8s disable ha-cluster" and then retry my tests. So i did this..
Resulting memory used dropped by 159MB :
In comparison, a standard docker deployment on the same VM spec uses only 216MB of RAM.
Ok, but this memory usage includes the base OS... and that is correct. Its why the base OS should be running a "minimal" services as possible. My deployment was based off Ubuntu Minimal, and the VM with nothing running on it at all uses 167MB. Again though, this is a real life scenario.
OK, so now we have our Kubernetes Single-Node Clusters up..
Lets try and deploy a simple mysql instance.
I see the pod has started on all 3 platforms,
so how about used memory now.. up to 868MB of 971MB total.
Unfortunately, once the POD started, K0s, K3s, and MicroK8s API servers stopped responding to further API commands, and so i was unable to issue any kubectl commands from this point forward. This effectively rendered the environments unusable.
I wanted to see the same thing on the Docker Node, so deployed mysql using 'docker run mysql' and saw the used memory was only 396MB:
None of the lightweight Kubernetes distributions are suitable to be used on devices that have 1GB of RAM or less.. not if you want to actually deploy any applications on them, thats for sure. I would recommend that lightweight hardware continues to use Docker standalone if containers are needed to be used.
Quick Table to summarise..
|Base OS, no Kubernetes||167MB|
|MicroK8s (no HA)||526MB|
One other thing to note. Kubernetes API server generates a pretty decent amount of Disk I/O and so if you run on CF cards, expect them to to die in a relatively short space of time, and also expect to see a lot of API timeouts occuring (leading to kubernetes instability). At a minimum, you need to run on SSD storage (not even USB stick storage is fast enough).
Now, as an added bonus...
Out of pure interest, i deployed 2x more MicroK8s nodes, and configured them as workers, joining them to a standard MicroK8s cluster node (not the non-HA node), as I wanted to see what memory usage would be for a pure worker node.
Interestingly, the memory usage is substantially lower, at only 238MB. This is impressive, given the cluster is running Calico networking.
Note though, once i enabled the cluster with dns, RBAC, hostpath-storage, and metrics-server (all of which are needed in production), memory usage increased to 335MB.
So, if you absolutely have to use Kubernetes at the edge, on lightweight hardware, you "can" as long as you have a node with at least 2GB of RAM that will operate exclusively as the master node. Clearly this adds a single point of failure, but Kubernetes can be turned to tolerate an extended outage of the master node without taking the cluster and all workloads offline.