Five years of Raspberry Pi Clusters

I’ve been writing about running Docker on Raspberry Pi for 5 years now and things have got a lot easier than when I started back in the day. There’s now no need to patch the kernel, use a bespoke OS or even build Go and Docker from scratch.

My stack of 7 Raspberry PI Model 2s running Docker Swarm (2016)

Since my first blog post and printed article, I noticed that Raspberry Pi clusters were a hot topic.

One of my first tutorials was published in Linux User and Developer

Back then we used “old Swarm” which was arguably more like Kubernetes with swappable orchestration and a remote API that could run containers. Load-balancing wasn’t built-in, and so we used Nginx to do that job.

I built out a special demo using kit from Pimoroni.com. Each LED lit up when a HTTP request came in.

After that, I adapted the code and added in some IoT sensor boards to create a “smart datacenter” and was invited to present the demo at Dockercon 2016:

Docker then released a newer version of Swarm also called “Swarm” and I wrote up these posts:

This is still my most popular video on my YouTube channel.

Now that more and more people were trying out Docker on Raspberry Pi (arm), we had to educate them about not running potentially poisoned images from third-parties and how to port software to arm. I created a Git repository (alexellis/docker-arm) to provide a stack of common software.

I wanted to share with users how to use GPIO for accessing hardware and how to create an IoT doorbell. This was one of my first videos on the topic, a live run-through in one take.

Did you know? I used to run blog.alexellis.io on my RPi3

Then we all started trying to run upstream Kubernetes on our 1GB of RAM RPis with kubeadm. Lucas Käldström did much of the groundwork to port various Kubernetes components and even went as far as to fix some issues in the Go language.

I wrote a re-cap on everything you needed to know including exec format error and various other things. I also put together a solid set of instructions and workarounds for kubeadm on RPi2/3.

Users often ask what a practical use-case is for a cluster. They excel at running distributed web applications, and OpenFaaS is loved by developers for making it easy to build, deploy, monitor and scale APIs.

This blog post was one of the ones that got pinned onto the front page of Hacker News for some time, a great feeling when it happens, but something that only comes every now and then.

The instructions for kubeadm and Raspbian were breaking with every other minor release of Kubernetes, so I moved my original gist into a Git repo to accept PRs and to make the content more accessible.

I have to say that this is the one piece of Intellectual Property (IP) I own which has been plagiarised and passed-off the most.

Shortly after this, Scott Hanselman found my post and we started to collaborate on getting .NET Core to work with OpenFaaS.

This lead to us co-presenting at NDC, London in early 2018. We were practising the demo the night before and the idea was to use Pimoroni Blinkt! LEDs to show which RPi a Pod (workload) was running on.

We wanted the Pod to stop showing an animation and to get rescheduled when we pulled a network cable. For the purposes of the demo it needed to happen within around 30 seconds, but it was taking around 15 minutes to kick-in.

It wasn’t working how we expected, and Scott just said “I’ll phone Kelsey” and Mr Hightower explained to us how to tune the kubelet tolerance flags.

As you can see from the demo, Kelsey’s advice worked out great!

Fast forward and we’re no longer running Docker, or forcing upstream Kubernetes into 1GB of RAM, but running Rancher’s light-weight k3s in as much as 4GB of RAM.

k3s is a game-changer for small devices, but also runs well on regular PCs and cloud. A server takes just 500MB of RAM and each agent only requires 50MB of RAM due to the optimizations that Darren Shepherd was able to make.

I wrote a new Go CLI called k3sup (‘ketchup’) which made building clusters even easier than it was already and brought back some of the UX of the Docker Swarm CLI.

To help combat the issues around the Kubernetes ecosystem and tooling like Helm which wasn’t available for ARM, I started a new project named arkade. arkade makes it easy to install apps whether they use helm charts or kubectl for installation.

k3s, k3sup and arkade are all combined in my latest post which includes installing OpenFaaS and the Kubernetes dashboard.

In late March I put together a webinar with Traefik to show off all the OpenFaaS tooling including k3sup and arkade to create a practical demo. The demo showed how to get a public IP for the RPi cluster, how to integrate with GitHub webhooks and Postgresql.

The latest and most up to date tutorial, with everything set up step by step.

In the webinar you’ll find out how to get a public IP for your IngressController using the inlets-operator.

Take-aways

  • People will always hate

Some people try to reason about whether you should, or should not build a cluster of Raspberry Pis. If you’re asking this question, then don’t do it and don’t ask me to convince you otherwise.

  • It doesn’t have to be expensive

You don’t need special equipment, you don’t even need more than one Raspberry Pi, however I would recommend two or three for the best experience.

  • Know what to expect

Kubernetes clusters are built to run web servers and APIs, not games like you do with your PC. They don’t magically combine the memory of each node into a single super computer, but allow for horizontal scaling, i.e. more replicas of the same thing.

  • Not everything will run on it

Some popular software like Istio, Minio, Linkerd, Flux and SealedSecrets do not run on ARM devices because the maintainers are not incentivised to make them do so. It’s not trivial to port software to ARM and then to support that on an ongoing basis. Companies tend to have little interest since paying customers do not tend to use Raspberry Pis. You have to get ready to hear “no”, and sometimes you’ll be lucky enough to hear “not yet” instead.

  • Things are always moving and getting better

If you compare my opening statement where we had to rebuild kernels from scratch, and even build binaries for Go, in order to build Docker, we live in a completely different world now. We’ve seen classic swarm, new swarm (swarmkit), Kubernetes and now k3s become the platform of choice for clustering on the Raspberry Pi. Where will we be in another 5 years from now? I don’t know, but I suspect things will be better.

  • Have fun and learn

In my opinion, the primary reason to build a cluster is to learn and to explore what can be done. As a secondary gain, the skills that you build can be used for work in DevOps / Cloud Native, but the if that’s all you want out of it, then fire up a few EC2 VMs on AWS.

Recap on projects

Featured: my 24-node uber cluster, chassis by Bitscope.
  • k3sup — build Raspberry Pi clusters with Rancher’s lightweight cut of Kubernetes called k3s
  • arkade — install apps to Kubernetes clusters using an easy CLI with flags and built-in Raspberry Pi support
  • OpenFaaS — easiest way to deploy web services, APIs and functions to your cluster, multi-arch (arm + Intel) support is built-in
  • inlets — a Cloud Native Tunnel you can use to access your RPi or cluster from anywhere, the inlets-operator adds integration into Kubernetes

Want more?

Well all of that should take you some time to watch, read, and to try out, probably less than 5 years. I would recommend working in reverse order from the Traefik webinar back or the homelab tutorial which includes a bill of materials.

Become an Insider via GitHub Sponsors to support my work and to receive regular email updates from me each week on Cloud Native, Kubernetes, OSS and more: https://github.com/sponsors/alexellis

And you’ll find hundreds of blog posts on Docker, Kubernetes, Go and more on my blog over at https://blog.alexellis.io.

CNCF Ambassador. OpenFaaS & Inlets founder — https://www.alexellis.io

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store