Kubernetes 1.18 broke “kubectl run”, here’s what to do about it

If you’ve been using a recent release of Kubernetes like 1.16, then you may have been used to seeing an odd warning message from kubectl run. Up until recently, the command could be used to generate Deployment objects or YAML from the command line.

With Kubernetes 1.18 the user experience for generating and creating Deployments has been broken in a surprising way. Here’s why, and what you can do about it.

Before we get into the change itself, there are three primary ways to create a Deployment object in Kubernetes:

  • Using the REST API and a client such as client-go (this is what OpenFaaS does)
  • By hand-crafting a YAML file, or more realistically, by copying and pasting from StackOverflow or a known example
  • By kubectl run or kubectl run -o yaml --dry-run

It turns out that I also listed these options in order of complexity (high to low).

For Option 1 — using the REST API is probably one of the most stable ways to create Deployments, but is the most complex option you can find. Up until recently the API had been relatively stable and happy in “extensions/v1beta” until it moved to “apps/v1”, this was a breaking change and meant lots of OSS projects having to migrate their code and YAML files.

I am not sure what’s harder — crafting Kubernetes objects in Go, or in YAML. With Go you get some hints in the IDE, but have to search all over for which package each type exists in. Core? Meta? Apps? Extensions? Maybe that’s what we’re missing with editing manifest files in VSCode — some good old intellisense?

Image for post
Image for post
A snippet of code from the OpenFaaS controller for Kubernetes (faas-netes)

Option 2 is hand-crafting YAML. If you’ve ever used Twitter or watched the #kubernetes hashtag, you’ll see all kinds of folks from newcomers to respected Kubernetes authors and founders bashing on how hard it is to write and grok YAML. Personally I like YAML and find it easy to use in most applications, but I do see where they are coming from. I think what they are really complaining about is the complexity and nested structure of the Kubernetes API schema.

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-1
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80

This example was actually copied from the Kubernetes docs and adjusted to set the replicas to 1/1 instead of 3. I personally would hate to be put on the spot and asked to write this from scratch.

Writing Kubernetes YAML from scratch feels like having to reverse a double-linked list with a pencil on a piece of paper and then being told not to make any mistakes or typos.

Option 3 was until recently a fine alternative to 1 and 2, especially when learning, when wanting succinct instructions to demonstrate another application or controller, or when wanting to generate a sample YAML file for further customisation or for use in a CI/CD pipeline.

To create a Deployment directly into the cluster, with a restart policy, a name, an image and a port, you could do this:

$ kubectl run nginx-1 --image=nginx --port=80 --restart=Always

Even better, if you wanted to mix option 2 and 3, you could output YAML without applying it and then customise it. Let’s see how it compares to the documentation?

$ kubectl run nginx-1 --image=nginx:1.14.2 --port=80 \
--restart=Always -o yaml --dry-run
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
run: nginx-1
name: nginx-1
spec:
replicas: 1
selector:
matchLabels:
run: nginx-1
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
run: nginx-1
spec:
containers:
- image: nginx:1.14.2
name: nginx-1
ports:
- containerPort: 80
resources: {}

Other flags you could set included --replicas and --serviceaccount to name a couple.

It’s not too dissimilar from the documentation, but the reason I’m writing this post is to let you know that this feature has been permanently removed in kubectl 1.18 and later.

I was always a little confused by this error, but the deprecation notice didn’t seem to have a time-line or any other helpful context.

Like you, I just thought “OK so I’ll learn a different CLI command at some point in the near future”

$ kubectl run nginx-1 --image=nginx --port=80 --restart=Alwayskubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.deployment.apps/nginx-1 created

If you download kubectl 1.17 or earlier (I am using 1.15 above) you’ll still be able to create a deployment or generate from from the CLI. As soon as you switch over to Kubernetes 1.18, you’ll get the following instead:

kubectl run nginx-1 --image=nginx --port=80 --restart=Always -o yaml --dry-run
W0512 14:27:13.111424 30104 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: nginx-1
name: nginx-1
spec:
containers:
- image: nginx
name: nginx-1
ports:
- containerPort: 80
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always

That’s a Pod, and it’s not the same as a Deployment. Fortunately you can still specify a service account and a port, but working directly with Pods is not encouraged as first-class citizens when building a production application. Pods are ephemeral, they can move around, get killed, go to OOM and don’t have a stable name (by default) or IP address. Deployments and Services fix this problem.

Pods are more useful for one-shot tasks like running curl to check your API or for adding some messages to a Kafka topic.

We faced this first-hand on the inlets-operator project, when a user told us that the documentation no longer worked.

Image for post
Image for post
You can expose a Pod, if you wish with “kubectl expose pod”.

For our documentation and helm-chart helper message, we wanted a simple instruction that would help users understand how to use the project. You create a Deployment, then expose it as a LoadBalancer Service, we provision a VM on your favourite cloud and then tunnel it back, so you in effect have a fully integrated public IP.

Our documentation pre-k8s 1.17 was:

kubectl run nginx-1 --image=nginx --port=80 --restart=Always
kubectl expose deployment nginx-1 --port=80 --type=LoadBalancer

It’s now:

export DEPLOYMENT=nginx-1(cat<<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: $DEPLOYMENT
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
EOF
) | kubectl apply -f -
kubectl expose deployment nginx-1 --port=80 --type=LoadBalancer

Note to the Kubernetes maintainers, please don’t take kubectl expose away from us next :-)

I am not looking forward to maintaining that amount of text in every tutorial I write, or whenever I need to craft a new Deployment.

I was so unhappy with the resulting documentation, that I’ve moved that over to an embedded file, which people will now need to first download to check, before running into their cluster:

$ kubectl apply -f https://raw.githubusercontent.com/inlets/inlets-operator/master/contrib/nginx-sample-deployment.yaml

Folks, never kubectl apply -f files from the Internet, without checking them over first.

It turns out that there is no equivalent alternative.

Some will be quick to tell you that the new command is kubectl create deployment.

However you’ll notice that all the important flags are missing like --port, --serviceaccount,--replicasand --restart-policy.

kubectl create deployment --help
Create a deployment with the specified name.
Aliases:
deployment, deploy
Examples:
# Create a new deployment named my-dep that runs the busybox image.
kubectl create deployment my-dep --image=busybox
Options:
--allow-missing-template-keys=true: If true, ignore any errors in templates when a field or map key is missing in
the template. Only applies to golang and jsonpath output formats.
--dry-run='none': Must be "none", "server", or "client". If client strategy, only print the object that would be
sent, without sending it. If server strategy, submit server-side request without persisting the resource.
--image=[]: Image name to run.
-o, --output='': Output format. One of:
json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file.
--save-config=false: If true, the configuration of current object will be saved in its annotation. Otherwise, the
annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.
--template='': Template string or path to template file to use when -o=go-template, -o=go-template-file. The
template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].
--validate=true: If true, use a schema to validate the input before sending it
Usage:
kubectl create deployment NAME --image=image [--dry-run=server|client|none] [options]
Use "kubectl options" for a list of global command-line options (applies to all commands).

So our original example will no longer work with kubectl create, and cannot be ported to anything but a block of YAML. If we need a dynamic port, then the alternative is going to be some bash templating which is not cross-platform.

The best we can get to with the new command is this, but then you’re going to have to trawl StackOverflow again, or the Kubernetes docs every time you want a simple Deployment. Why? Because nested YAML schemas are hard to memorise.

$ kubectl create deployment nginx-1 --image=nginx  -o yaml --dry-runW0512 14:38:18.296270   30135 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx-1
name: nginx-1
spec:
replicas: 1
selector:
matchLabels:
app: nginx-1
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: nginx-1
spec:
containers:
- image: nginx
name: nginx
resources: {}

Note the missing port, restart-policy and more.

I am not sure, but sometimes good intentions and dealing with nagging technical debt can result in a worse user-experience despite every hope of improving the world.

Kubernetes PR 68132 seemed to start it all, by wanting kubectl run to work more like the docker run command. Certainly not a bad thing.

My hope was that the maintainers would consider adding the various missing flags to the new kubectl create deployment command, however someone else got here first and was told “if you want it, you send a pull request”.

Image for post
Image for post
Image for post
Image for post

Did you get affected by this change? Comment on the Twitter thread, or the PR above which merged the change.

Making things better is difficult, especially when it breaks long established user-experiences. You do you, Kubernetes authors 💙.

For a complete list of changes in 1.18, see the release notes: https://kubernetes.io/docs/setup/release/notes/#changelog-since-v1170

CNCF Ambassador. OpenFaaS & Inlets founder — https://www.alexellis.io

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store