I’ve been writing about running Docker on Raspberry Pi for 5 years now and things have got a lot easier than when I started back in the day. There’s now no need to patch the kernel, use a bespoke OS or even build Go and Docker from scratch.
Since my first blog post and printed article, I noticed that Raspberry Pi clusters were a hot topic.
They’ve only got even hotter as the technology got easier to use and the devices became more powerful.
- Docker Article in Linux magazine — March 2016
Back then we used “old Swarm” which was arguably more like Kubernetes with swappable orchestration and a remote API that could run containers. Load-balancing wasn’t built-in, and so we used Nginx to do that job.
I built out a special demo using kit from Pimoroni.com. Each LED lit up when a HTTP request came in.
- Docker load-balanced LED cluster Raspberry Pi — 17 May 2016
After that, I adapted the code and added in some IoT sensor boards to create a “smart datacenter” and was invited to present the demo at Dockercon 2016:
- IoT Dockercon Demo — 28 June 2016
Docker then released a newer version of Swarm also called “Swarm” and I wrote up these posts:
This is still my most popular video on my YouTube channel.
- Live Deep Dive — Docker Swarm Mode on the Pi — Aug 2016
Now that more and more people were trying out Docker on Raspberry Pi (arm), we had to educate them about not running potentially poisoned images from third-parties and how to port software to arm. I created a Git repository (alexellis/docker-arm) to provide a stack of common software.
- 5 things about Docker on Raspberry Pi — Sep 2016
I wanted to share with users how to use GPIO for accessing hardware and how to create an IoT doorbell. This was one of my first videos on the topic, a live run-through in one take.
- Hands-on Docker for Raspberry Pi — Nov 2016
Then we all started trying to run upstream Kubernetes on our 1GB of RAM RPis with kubeadm. Lucas Käldström did much of the groundwork to port various Kubernetes components and even went as far as to fix some issues in the Go language.
I wrote a re-cap on everything you needed to know including exec format error and various other things. I also put together a solid set of instructions and workarounds for kubeadm on RPi2/3.
Users often ask what a practical use-case is for a cluster. They excel at running distributed web applications, and OpenFaaS is loved by developers for making it easy to build, deploy, monitor and scale APIs.
In this post you’ll learn how to deploy a fun Pod to generate ASCII text, from there you can build your own with Python or any other language.
This blog post was one of the ones that got pinned onto the front page of Hacker News for some time, a great feeling when it happens, but something that only comes every now and then.
The instructions for kubeadm and Raspbian were breaking with every other minor release of Kubernetes, so I moved my original gist into a Git repo to accept PRs and to make the content more accessible.
- Build your own bare-metal ARM cluster — Dec 2018
I have to say that this is the one piece of Intellectual Property (IP) I own which has been plagiarised and passed-off the most.
You’ll find dozens of blog posts which are almost identical, even copying my typos. To begin with I found this passing-off of my work frustrating, but now I take it as a vote of confidence.
Shortly after this, Scott Hanselman found my post and we started to collaborate on getting .NET Core to work with OpenFaaS.
This lead to us co-presenting at NDC, London in early 2018. We were practising the demo the night before and the idea was to use Pimoroni Blinkt! LEDs to show which RPi a Pod (workload) was running on.
We wanted the Pod to stop showing an animation and to get rescheduled when we pulled a network cable. For the purposes of the demo it needed to happen within around 30 seconds, but it was taking around 15 minutes to kick-in.
It wasn’t working how we expected, and Scott just said “I’ll phone Kelsey” and Mr Hightower explained to us how to tune the kubelet tolerance flags.
Fast forward and we’re no longer running Docker, or forcing upstream Kubernetes into 1GB of RAM, but running Rancher’s light-weight k3s in as much as 4GB of RAM.
k3s is a game-changer for small devices, but also runs well on regular PCs and cloud. A server takes just 500MB of RAM and each agent only requires 50MB of RAM due to the optimizations that Darren Shepherd was able to make.
- Will it cluster? k3s on your Raspberry Pi — March 2019
I wrote a new Go CLI called k3sup (‘ketchup’) which made building clusters even easier than it was already and brought back some of the UX of the Docker Swarm CLI.
To help combat the issues around the Kubernetes ecosystem and tooling like Helm which wasn’t available for ARM, I started a new project named arkade. arkade makes it easy to install apps whether they use helm charts or kubectl for installation.
k3s, k3sup and arkade are all combined in my latest post which includes installing OpenFaaS and the Kubernetes dashboard.
In late March I put together a webinar with Traefik to show off all the OpenFaaS tooling including k3sup and arkade to create a practical demo. The demo showed how to get a public IP for the RPi cluster, how to integrate with GitHub webhooks and Postgresql.
- People will always hate
Some people try to reason about whether you should, or should not build a cluster of Raspberry Pis. If you’re asking this question, then don’t do it and don’t ask me to convince you otherwise.
- It doesn’t have to be expensive
You don’t need special equipment, you don’t even need more than one Raspberry Pi, however I would recommend two or three for the best experience.
- Know what to expect
Kubernetes clusters are built to run web servers and APIs, not games like you do with your PC. They don’t magically combine the memory of each node into a single super computer, but allow for horizontal scaling, i.e. more replicas of the same thing.
- Not everything will run on it
Some popular software like Istio, Minio, Linkerd, Flux and SealedSecrets do not run on ARM devices because the maintainers are not incentivised to make them do so. It’s not trivial to port software to ARM and then to support that on an ongoing basis. Companies tend to have little interest since paying customers do not tend to use Raspberry Pis. You have to get ready to hear “no”, and sometimes you’ll be lucky enough to hear “not yet” instead.
- Things are always moving and getting better
If you compare my opening statement where we had to rebuild kernels from scratch, and even build binaries for Go, in order to build Docker, we live in a completely different world now. We’ve seen classic swarm, new swarm (swarmkit), Kubernetes and now k3s become the platform of choice for clustering on the Raspberry Pi. Where will we be in another 5 years from now? I don’t know, but I suspect things will be better.
- Have fun and learn
In my opinion, the primary reason to build a cluster is to learn and to explore what can be done. As a secondary gain, the skills that you build can be used for work in DevOps / Cloud Native, but the if that’s all you want out of it, then fire up a few EC2 VMs on AWS.
Recap on projects
- k3sup — build Raspberry Pi clusters with Rancher’s lightweight cut of Kubernetes called k3s
- arkade — install apps to Kubernetes clusters using an easy CLI with flags and built-in Raspberry Pi support
- OpenFaaS — easiest way to deploy web services, APIs and functions to your cluster, multi-arch (arm + Intel) support is built-in
- inlets — a Cloud Native Tunnel you can use to access your RPi or cluster from anywhere, the inlets-operator adds integration into Kubernetes
Well all of that should take you some time to watch, read, and to try out, probably less than 5 years. I would recommend working in reverse order from the Traefik webinar back or the homelab tutorial which includes a bill of materials.
Become an Insider via GitHub Sponsors to support my work and to receive regular email updates from me each week on Cloud Native, Kubernetes, OSS and more: https://github.com/sponsors/alexellis
And you’ll find hundreds of blog posts on Docker, Kubernetes, Go and more on my blog over at https://blog.alexellis.io.