Developing for Kubernetes with Skaffold
• Jul 19, 2021
There are a growing number of solutions that enable software developers to develop their cloud-native applications on Kubernetes. We've discussed a couple of these solutions in this series already (Tilt and Garden), and in this article we'll take a look at another, Skaffold. Unlike the tools we've already discussed, which have emanated from small, innovative cloud-native start-ups, Skaffold comes from the house of Google. When it comes to cloud-native tooling, Google seems to have a play in just about every category.
The fact that Skaffold originated from Google probably accounts for its relative popularity on GitHub. As I write, it has over 11 000 stars and in excess of 270 contributors, which outstrips the other tools that provide a similar developer experience. This gives the project, which is open source under an Apache Licence, a distinct advantage over its competitive rivals. More contributors bring more capacity to the cause, and the stars help to increase the project's profile. So, is this higher profile warranted? Let's lift the lid on Skaffold.
As a tool with the purpose to aid software developers, it comes as no surprise that Skaffold is designed around a pipeline construct. The pipeline is segmented into stages, some or all of which can be incorporated into a workflow to suit the circumstances. Some of these stages seem very logical (e.g. build, test, deploy), but some feel as if they're an affectation that simply serves to highlight a feature. Let's take an example; the
tag stage is really a detail of the
build stage rather than a workflow stage in itself. This is borne out by the way Skaffold defines a workflow using YAML-based declarative configuration. The important top-level keys in the schema are build, test and deploy, with image tagging details, for example, defined as a sub-key of the build key.
The reason some of these features are given prominence in the documentation concerning stages, is that Skaffold provides the developer with a lot of flexibility in implementing a pipeline, which it achieves with a pluggable system. Images, for example, can be built using a number of different tools, and tagged using a variety of different 'taggers' or policies. To expand on this a little, Skaffold can invoke a build using Docker, Jib, Bazel or Buildpacks, and can tag images using git references, image IDs, environment variables, and so on. This flexibility speaks to the abundant choice in tooling that's available in the cloud-native domain, and the many different ways that workflows can be configured. Kudos to the project for providing the flexibility and choice.
Skaffold, like Tilt and Garden, uses declarative configuration to define the developer's workflow. As is generally the norm with cloud-native configuration, it's stored in YAML format in a file called
skaffold.yaml, which is generally located in the project's root directory. The declarative configuration is consumed by the Skaffold CLI, in order to influence the actions it takes in processing the defined pipeline.
--- apiVersion: skaffold/v2beta16 kind: Config metadata: name: my-app build: artifacts: - image: my-app docker: dockerfile: Dockerfile.build local: useBuildkit: true push: false
In this simple snippet of a
skaffold.yaml file, the
build key declares that an artifact, a Docker image, is to be built from a Dockerfile called .
Dockerfile.build, which will subsequently be tagged
my-app. This assumes a build context exists in the working directory, of course. Running a
skaffold build command triggers an image build, which results in an image with two tags; one with the most recent git tag or git commit (e.g. my-app:ae7926d), and as the image isn't pushed to a container registry, a tag which uses the 64 character ID of the image. Skaffold is big on image immutability, and the latter tag is a content addressable reference to the image in the local cache. If it had been pushed to a registry, the immutable reference would be the image's digest instead. This ensures we only use the image Skaffold has built, and not a nefarious image masquerading as one and the same.
As well as providing flexibility on the method of build that we mentioned earlier (i.e. Docker, Jib and so on), Skaffold can accommodate different locations for performing image builds. They can be performed locally, inside a Kubernetes cluster (using Kaniko), or remotely using (unsurprisingly) Google Cloud Build.
Deploying to Kubernetes
Just as there is a
skaffold build command to build images, there is a
skaffold deploy command to deploy applications that are defined in the
build section of the configuration. Once again, Skaffold provides some flexibility to accommodate the three most popular methods for deploying applications to Kubernetes clusters. It supports good old
kubectl apply, Helm charts, and Kustomize variants and overlays. Skaffold also supports kpt as a deployer, which you can expect to play a bigger role in the future. The
skaffold.yaml file schema has a number of different keys for defining deployment activity for whichever deployer is used, all under the primary
As far as testing goes, Skaffold has an in-built
test stage which sits between the
deploy stages. At present, there are two types of tests that can be conducted; container structure tests, and custom tests, both defined in the 'skaffold.yaml' file. Whenever a build is initiated after a change, any defined tests for specific images are executed before deployment. That is, unless Skaffold is instructed to ignore the defined tests. If tests fail, the pipeline is aborted and no deployment takes place.
So, what type of testing can be achieved? Well, container structure tests are an opinionated approach to testing the content of a container image. They are designed to check for expected filesystem content, image metadata (default user, for example), environment variables, correct command output, and so on. Unfortunately, the GitHub repo that hosts this project suggests that it isn't an officially supported Google project, and that it's currently in maintenance mode. It seems the reason for this is that there isn't sufficient staff to maintain the project. Best to use with caution.
A custom test executes any command the developer wants, and allows for defining dependencies for the test. The test is executed locally, and might be used to run unit tests, scan for image vulnerabilities, or whatever is deemed important prior to deployment.
So far we've discussed the different stages and tools that can be plugged into the pipeline, but we haven't really discussed the overall developer workflow. A
skaffold test, or
skaffold deploy command executes actions for the appropriate stage in question. To get the seamless inner loop experience, a developer needs to use
skaffold dev, which puts Skaffold into a development loop. It watches for source file changes, syncs files to running containers or builds images, runs tests, deploys the services to the target Kubernetes cluster, and then returns to watching for changes. This is considered the main mode of operation for Skaffold.
But the development loop can also be augmented with debugging. The
skaffold debug command behaves much like the
skaffold dev command, but configures the application's container to allow debugging for the language used for the application service. How this works depends heavily on the language in question, but Skaffold supports .NET, Python, Java/JVM, Golang and Node.js at the time of writing.
Skaffold has all the essential ingredients for successful inner loop development in a Kubernetes cluster, and even offers an API in
debug mode for integration with IDEs like Visual Studio Code and Intellij. This further improves the overall developer experience. But, it can also be used in CI/CD pipelines, which is where Skaffold deviates away from what's offered by other competitive tools that tend to have more focus on the inner loop.
By its own admission, Skaffold is an opinionated development environment for developing applications for Kubernetes. Whilst it's open source and provides a degree of flexibility in tool choice, there is an undeniable leaning towards all things Google (Jib, Bazel, Kaniko, Kpt). This might deter some, but for others, having a project supported by a large public cloud provider might be an attractive proposition. Either way, Skaffold sits comfortably alongside the other competing tools in this area of cloud-native developer tooling.
We're always interested to see how cloud-native tools are adopted in the real world. So, let us know if you're a Skaffold fanboi, and share your experiences using it to develop your software applications.
About the author
Puja Abbassi is VP of Product at Giant Swarm. He regularly writes for this blog covering everything from Cluster API and autoscaling, to how we actually make product decisions. A CNCF Ambassador with a passion for martial arts, you can listen to his KubeCon talks, read his KubeCon Europe recommendations or better yet, bump into him at the next KubeCon. Find him on Twitter and say hi! 👋