Photo by Jainath Ponnala on Unsplash

Understanding kubernetes with raspberrypi

Do you remember the good old days when the fullstack developers were only being defined as a developer who can do backend and frontend?

Well that is over my friend (but I like it 😉).

Here and there nowadays you hear a lot about microservices, cloud environments, clusters, replications and ofcourse Kubernetes(k8s). You hear all these words and you get lost. You don’t know when will you have the time to really start with learning all these or even just understand what each of these mean ?!

Well my friend you are not alone. Couple of weeks ago I was the same. It is around 15 months that at our company we are implementing our softwares as microservices. Microservices have a lot of benefits which is out of the scope of this post. The only thing I can say is that they make the development so much faster but nothing comes without its cons. With microservices architecture one of the many questions which we were always facing was, “How on earth are we going to deploy all these services ?!!”

This is when Kubernetes comes into play. Kubernetes or in short k8s is a container orchestrator which guarantees easy deployment and maintenance. Because it is just not about the deployment.

Consider having 200 😨 or 1000😱 microservices running and you want to make sure if one goes down for any reason it can get restarted automatically and only in really serious cases you get notified about acting on it yourself.

K8s topic is really vast and to me it was something that I really liked to “feel it”. I didn’t want to use Docker Desktop to have k8s in a docker container. I wanted to understand clustering. Setting up nodes and break nodes when the application is running to see with my own eyes that it is real 😁

So that was when raspberry came into play. I ordered 2 raspberrypi 4s with 8GB ram to set them as worker nodes and I already had one 4GB ram raspberry 4 and I configured that as a master node. You can check my dear cluster in the following picture 🤩

My raspberry cluster. masterpi, workerpi0 and workerpi1

Before we continue with the rest, if you are a visual person you can check the following video which is basically the visual presenation of this post. Otherwide please go on :)

So now we need to setup the cluster. What do we need ?!

1. A Linux distribute which raspberry supports the best aaand what that can be 🤔?!! 💡 yup you guessed it, Raspberry Pi OS Lite (32/64). I used 64bit for a reason which I will describe later. The 64bit version is not available in the imager so you need to download it from HERE
2. You need to be able to setup your raspberry headlessly.
3. You need a version of k8s which you can install on your nodes as simple as possible (It is simple in general anyway 😅). I used k3s which is the simpler version of k8s but for us it is enough and it has all the main features of k8s. https://rancher.com/docs/k3s/latest/en/quick-start/
4. An application to deploy on it ofcourse :D

The reason that I picked the 64bit version was that the application which I have written to run on it was in Scala and I had to find a docker image for Scala which could run on arm x86 processors but I couldn’t find one and was not interested to build one so I just used THIS ONE and built the arm64 version of it.

So let’s get started. The first step is pretty clear, just download the image and install it on each node.

btw I have assumed that we are going to use windows for preperation.

Setup each node headlessly

What does it mean headlessly? well it means headlessly like you have no view on raspberry directly. It will not be connected to any monitors or doesn’t have any keyboard connected to it directly.

What you do is that when you installed the OS on each SD card you will not put it yet into the raspberries but you will first do the following.

You create a file and name it “ssh” with no extension or anything. For the content you will only add a space (I don’t know why :D I tried first without the space and it didn’t work so I checked online and I realized that there should be a content in there which doesn’t have any specific meaning! )

Now you need to introduce it also to internet “headlessly”. So you will create another file and name it “wpa_supplicant.conf”. This one have some meaningful content like below.

country=DE
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
scan_ssid=1
ssid="YOUR_WIFI_SSID"
psk="YOUR_WIFI_PASSWORD"
}

and throw these two files directly into the root of the sd cards, just like this.

Btw you can download these two files from here.

https://github.com/farhadnowzari/raspberry-k3s-cluster/raw/master/headless.zip

Now here comes the tricky part. If you connect all of these at once you don’t know which one you have to ssh into. All of them will be similar. You have two options. You either can run all of them together and do an IP scan to find them by their IP addresses using

Advanced IP Scanner - Download Free Network Scanner.

or you can set them one by one. By default they have “raspberrypi” as their name.

Note: here you gonna need PuTTY to ssh into them.

The default credentials are like below:

username: pi
password: raspberry

Congratulations now if you have done everything right so far you can access the raspberries and connect to them.

Now the fun part!

Installing kubernetes on each node

Well the documentation on Rancher website is pretty clear. You can take a look at here and you are good to go.

Quick-Start Guide

BUT not so fast. There are some twists which may happen and we are going to go through it together.

First pick your master node. I chose the one with had less resources because to be fair it doesn’t need much resources.

curl -sfL https://get.k3s.io | sh -

The above command will install your node as the master node.

After doing this, to verify your installation you need to be able to access to kubectl cli but in case that it showed you the following error, you need to do some magic to make it work ^^.

try to run this and see if it works

sudo kubectl get pods

If it gave you “The connection to the server 127.0.0.1:6443 was refused — did you specify the right host or port?”, you will know why when you run the following command

journalctl

This command will show you the system’s loggings. Something like eventvwr on windows.

If you go down enough in the logs you will eventually notice the following lines!

journalctl log, which shows you why the k8s is failing to start on your Raspberry Pi OS Lite

To resolve this, open /boot/cmdline.txt with an editor in terminal as root user.

sudo nano /boot/cmdline.txt

and paste the following at the end of the opened file.

cgroup_memory=1 cgroup_enable=memory

it should look like this (my drawing line skills are the best 😂 don’t judge me though 😅)

Fixing the cgroup_memory issue on booting with raspberry pi

And now save, then run sudo reboot .

After it booted up, try running the sudo kubectl get pods command again. This time it should tell you perhaps that there are no resources which is fine. It means it is now connected to the k8s API server.

Before we continue with the workers, keep your master node PuTTY window open and run the following command to get the master node’s token, we gonna need it later.

sudo cat /var/lib/rancher/k3s/server/node-token

Now let’s continue and set the workers up.

PuTTY into the nodes you have considered as workers and run the following command.

In the following command there are two pleaceholders.

• myserver, which you need to replace with the name of your master node or it’s IP address
• mynodetoken, which you have to replace with the node-token we retrieved from the master node.

curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=mynodetoken sh -

And ofcource you are going to have the same problem as you had with the master node in journalctl so you need to fix it the same way as we did with master node. But after you fix it you will still not be able to receive any information from kubectl commands from worker nodes which I assume is because that they don’t have the server configuration and they don’t need it anyway.

After you fixed the issue with the workers/agents get back to the master node we have work to do :)

First run sudo kubectl get nodes to see if everything is up and running.

When you run this you notice that the workers do not have any labels on them so you can label them with the following command.

sudo kubectl label node YOUR_NODE_NAME_OR_IP node-role.kubernetes.io/worker=worker

Now if you check them out again with sudo kubectl get nodes they are all labeled and have a role.

Check below.

Before 👇

Before labeling the worker nodes

After 👇

After labeling the worker nodes

Yaaay!

Now before we get into deploying our application on our cluster, let’s deploy the kubernetes dashboard.

Kubernetes Dashboard on K3s

So you can access kubenetes to get information about how is it running in three ways.

1. Api Server
2. Its Dashboard
3. Command Line

In my opinion if you are comfortable with command line, it is the most powerful tool to manage your cluster but if you want to have visual statistics, k8s dashboard which you can checkout HERE is also pretty decent and I like it.

The API server is just the heart of the communication and both the dashboard and the cli will communicate with the cluster through the API server.

Now let’s get into installing the dashboard.

First you need to ssh into your master node and as stated in the k8s dashboard documentation after running the following command you have the dashboard installed. So let’s do that.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml

Now by running kubectl proxy you will have a proxy open to port 8001 and you can access the dashboard.

But not so fast we are on a headless raspberry cluster. Whenkubectl proxy opens up port 8001, it opens it up locally so you only can access it from inside the master node and nowhere else. Well this is kinda intended because this dashboard is the configuration of your server. When you are on windows server do you expose IIS to be accessible from outside? nope ^^.

You need to make a ssh tunnel from the machine in which you want to open the dashboard, in our case will be the windows machine we are using to setup this whole thing.

Now you can do this tunneling with PuTTY everytime you connect to your master node or you can do it from ssh command line in windows or on WSL (Window Subsystem for Linux). We will go the easy way, PuTTY :D

Use PuTTY to make a ssh tunnel

Like the above picture, go to Connection->SSH->Tunels then configure your “tunneling” like below.

• Source port: This is the port on your machine (not the cluster) which you want to access the content from.
• Destination: This is a full host address on the cluster, in our case it is like 127.0.0.1:8001 which is the address which k8s dashboard is running on inside the cluster after you run kubectl proxy. Now how?!!?!?!?!

When you go back to the session category and enter your master node info to connect to it, PuTTY will create the configured tunnel for you. -

And now you can access the dashboard from your machine with the following link (Replace the “YOUR_SOURCE_PORT” placeholder)

http://localhost:YOUR_SOURCE_PORT/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login

And you should see this

k8s login page

But it won’t let you in so you gonna need a user and create a token for that user.

Rancher documentation on creating a user is pretty straight forward you can check it HERE but we will go through it together.

Import the following yaml files in your raspberry.

The first one is the ClusterRoleBinding which creates the admin-user role and the second one is the ServiceAccount.

Note: inorder for this to work, note the namespace on each file. Both of the following configs have the same namespace as the dashboard itself.

After importing these files, run the following command to create the role and the user respectively

sudo k3s kubectl create -f dashboard.admin-user-role.yaml -f dashboard.admin-user.yaml

Now we need to create a key for this user. So please run the following command inside your cluster’s master node.

sudo k3s kubectl -n kubernetes-dashboard describe secret admin-user-token | grep '^token'

You should get a “beautiful” response like this

Copy the token and paste it in the dashboard’s login page’s token field (God that was hard to type 😅 ) — Don’t forget to run kubectl proxy first so the dashboard starts.

And here you go :)

Let’s make it display something.

Deploy an application into our lovely Cluster

So if you did not have access to the dashboard you had to import the config files into the cluster and add them with kubectl apply but now that you have access to the dashboard let’s do it from the dashboard it self.

I already have created an application for us to test with. This application has been created for another purpose but I dockerized it and pushed it’s images into dockerhub. You can check it out 👇

• Our server side scala app: https://hub.docker.com/repository/docker/nowzarifarhad/fdn-chat-server
• Our client side vue app: https://hub.docker.com/repository/docker/nowzarifarhad/fdn-chat-client

Now what is this app? nothing it is just a simple video chat application which you can make a room and make video calls with each other.

Now how to deploy.

For deploying into k8s we need deployments, surprised huh?! :D

Let’s deploy the chat server with the following file and we will go through it together.

The above file has two sections

1. Deployment: This section is responsible to pull images, create containers inside pod(s), request hardwares and expose the container ports. It also makes sure that the pods are up and are healthy. Basically this is the dirtiest part and we all appreciate the hard work here ^^.
2. Service: Without services, the pods which created by the deployment cannot service other pods. The service by default is of type ClusterIp but if you wish to make it accessible from the node IP you need to change the type to LoadBalancer. I wouldn’t do that btw because soon we are going to see how we can access the internal services from outside using Ingress.

So now let’s copy and upload this file into k8s dashboard.

Now you will see everything goes “Orange”. It is because the container is being created. It needs to pull the image, create the container and run it eventually. if you go to Pods you can see what exactly is happening.

Creating chat-server pod on workerpi1

And after some time you should see that everything is green :D

Green, green, green k8s dashboard :D

Now let’s also deploy the client.

Please do the same with the following config

There is something different about this one though.

The replicas property. For server it is set to 1 and for the client it is 3. The reason is that this application’s server is not stateless which means if you have more than one replica they can be in different states than each other and if one client connects to one and another client connects to the other one, this chat server cannot connect these two clients with each other. On the other hand the client is stateless. It is just a normal javascript application which runs on the browser.

After waiting long enough you should have all three pods for the client up and running and it looks like the following.

Just take your moment and look at it. It is beautiful. Do you know what is more beautiful than this? The fact that it is showing you which pod is running on which worker. Which means you can detach the power to that node and see how k8s handle the situation 😈

We are not done yet.

Let the services be accessible from outside

So we have a javascript client app which will run on the user’s browser which means it needs to access the server remotely which means the server should also be accessible with an address.

Here we get into Ingress config but with some differences than the documentations on k8s.

See, we are using k3s and if you run k3s version you see this

At the time of this post (For people in the future :D) we are running k3s 1.21.4 and if you have paid enough attention in k3s Rancher documentation you would have seen this

This means that I wasted one day trying to work with normal k8s ingress with nginx-ingress-controller but k3s comes with Traefik.

What is Ingress or IngressController?

• IngressController, either Traefik or NginX controller are deployments on their own which gives k8s a setup for reverse-proxy (They are available as pods).
• Ingress, is the configuration with set of rules to tell the IngressController how the server should react to a specific incoming request with a specific root.

You can check Traefik out from kubect-system namespace with k3s setup (When you have it setup on k8s it can be under traefik namespace).

Traefik inside k3s cluster under kube-system namespace

Now let’s use it, it is doing nothing right now :D

To use Traefik we need to apply IngressRoutes like the config below.

The above file is pretty self-explanatory. The last section is the IngressRoute which is responsible to react to requested urls based on the given rules. So the first route says if the host is masterpi and the relative path is /chat-server then select the chat-server service on port :9000 . The tricky part here are the middlewares. The difference which Traefik v2.x has with it’s former version is that it completely removes annotations . The middlewares which are used here are for removing /chat-server or /chat-client relative paths when the request is forwarded to the service otherwise your service will receive either of these relative paths and will return 404 .

After applying the above file you should be able to access the app.

In case that it didn’t load please check that you are accessing the cluster with the correct domain.

If you are interested on this app’s code too, you can check it out HERE.

Congratulations, you have your cluster with k8s configured and have an application up and running on it.

To make the app stateless we can use a statestore but since we don’t want to make the application to be coupled with a specific statestore we are going to use #dapr which is a post on its own (wait for it ;) ).

Have fun coding 😉

Cheers