Configuration management at Giant Swarm: dynamic konfiguration » Giant Swarm

We flipped thousands of application configurations across many management clusters over to a new system with zero issues. This is the second of three posts on configuration management at Giant Swarm, and it's about how we pulled that off.

In part one, we took a historical overview of how our configuration management systems evolved over the years, ending with konfigure, the schema-based system we built to replace the old one.

Swapping the engine mid-flight

Konfigure is capable of describing the exact structure that we inherited from the long gone config-controller with a schema. We decided to call that schema the management-cluster schema, and the 1.0.0 version of it replicates the old system 1-to-1 with two layers:

• defaults: for default/apps/<< app >> value files and templates
• management-cluster: for installations/<< installation >> value files and templates

This is however now all described in a simple YAML instead of being hard coded.

Supporting stages, one of our original motivations, now became incredibly simple. We simply added two new layers between the above two in the 1.1.0 version of the management-cluster schema:

• default-stages: under default/stages/<< stage >>/apps/<< app >> for stage-specific sane defaults shared across all instances of the configuration for each management cluster
• stages: under stages/<< stage >>/apps/<< app >> for customer-specific stage configurations shared across all instances

We chose this setup because the default folder content is shared across all our users. Then each user has a customer-specific configuration repository with potentially lots of management clusters. Each cluster can belong to a given stage. Under the stages folder, customer-specific stage configurations can be set. Then the management cluster declares its stage by setting the stage variable to one of the declared stages.

Note that the stage variable is optional, same for the stage layers and their contents. This can be used to simply fall back to the original, stageless behavior by simply not declaring a stage when rendering the configuration.

Let’s dive a little further to see in more detail how this system works and how it could easily represent any new configuration structures.

Konfigure, the Generalized Configuration System

The schema is a simple YAML file that describes the structure of the configuration repository. It consists of the following main parts:

• variables: names that can take values that will be used to locate the required templates and value files across layers
• layers: highly customizable slices of configurations that can take different shapes that will be rendered and merged together to form the final configuration
• includes: a set of reusable pieces of templates that can be used via dynamic functions in the layer templates

The konfigure repository contains a simple example for generating configuration across multiple stages and clusters. We will walk through that to build up our understanding of the system.

A simple, staged environment example

The schema defines three layers. A layer is a set of value files, templates and patches. It describes the directory structure where each of these files is located. Layers are rendered in order, merged and patched on top of each other to get the final result:

• valuefiles: simple YAML files containing values for the templates
• templates: Go templates, rendered (enhanced with the Sprig Functions library) with a set of the values files
• patches: JSON6902 patches (just like kustomize), applied to the rendered template of the given layer

Variables can be used to control which paths and file names will be picked by the renderer.

We have the following layers here:

• base: common, shared configuration for applications
• stages: stage-specific value files and stage-specific templates and patches for applications
• clusters: cluster-specific value files and cluster-specific application templates and patches

The paths and file names that can be reached within each layer are configured via the following variables:

• stage: the name of the stage
• cluster: the name of the cluster
• konfiguration: the name of the application

Variables can be used in fields for directory and file names as well. They are substituted in the form of << VARIABLE_NAME >> when the engine loads the files before rendering. Each individual path can be marked as optional, meaning that for example you do not have to have stage or cluster-specific overrides for a given application. Doing so makes the engine simply consider the value file, the template or the patches file to be empty. Marking a path as required lets you enforce, for example, that each application defines sane defaults in the base layer, so the render fails when they are missing.

You can give it a try by running the command below at the root of the example folder.

SOPS_AGE_KEY_FILE=example.age konfigure render \
  --schema schema.yaml \
  --dir . \
  --variable "stage=dev" \
  --variable="cluster=cluster-1" \
  --variable="konfiguration=konfiguration-1" \
  --raw

The render command takes the following arguments:

• --schema: points to your schema file that describes the configuration repository
• --dir: points to the root of the configuration repository
• --variable: can be passed multiple times taking arguments in the form of variable_name=value, for the variables defined inside the schema

The command always renders two YAML documents, because there are two kinds of templates you can have: config map and secret. They are functionally the same, the only difference is whether the path contains unencrypted or encrypted data. The first document is always the unencrypted (config map) path, and the second is the encrypted (secret) path. Thus, by default, the rendered config is wrapped into a Kubernetes ConfigMap and a Secret respectively. Passing the --raw flag skips this wrapper and simply prints the rendered configurations as two raw YAML files.

SOPS_AGE_KEY_FILE=example.age konfigure render \
  --schema schema.yaml \
  --dir . \
  --variable "stage=production" \
  --variable="cluster=cluster-1" \
  --variable="konfiguration=konfiguration-1" \
  --raw

Simply changing the stage variable to production yields a different result.

It is possible to define other reusable pieces of templates with the includes section of the schema. Each entry defines a directory that contains such Go templates and a function name that will be dynamically created to use within layer templates to pull these in. Include templates are also subject to variable substitution with the set of variables used at the given layer to render the given template where the include is used. An example of such a shared template for this exercise is the todo.txt file.

Now let’s take a look at the patches feature of layers. They define a file that can contain a set of JSON6902 patches. This is a new feature of the Generalized Configuration System. When YAML documents are merged, it is pretty clever at merging objects, but for example lists are overwritten under the same key from the top document. You can use patches to get around for example this limitation, by appending or prepending to the list, or inserting / replacing new items at a given index. There is a small example of this for konfiguration-1 config map patch for the dev stage in the stages layer.

Getting it all into Kubernetes

The sole purpose of konfigure is to hold the logic to work with the schemas and render configurations. Full local reproducibility was a very strong requirement.

In retrospect, maybe it could have been made even more generic from the schema perspective. For example the layers support unencrypted and encrypted paths and are called config map and secret paths. It’s a little too tied to Kubernetes, and it doesn't need to be. Something for the future to maybe improve upon.

While being a much more generic tool, its main purpose for us was to render configs for Kubernetes. The previous iteration of the tool was strictly bound to Giant Swarm App custom resources. Another strong requirement was that we wanted to get rid of this tie. We wanted to decouple generating configuration from how it is used.

For example, app-admission-controller, our validating and mutating webhook in Giant Swarm App Platform for App CRs, used to have a validation rule that an applied App CR’s referenced ConfigMaps and Secrets exist, and could reject the App CR if not. To some extent, in ancient times this made sense, but it was also an annoying rule when we moved to GitOps and with our goal to decouple the config system from the usage of configs, it was straight in the way.

We decided that finally, even App CRs (while they are still around) should rely on eventual consistency, so to say. Flux HelmReleases do not have such a validation as above. If the HelmRelease cannot reconcile, because a required referenced value file is not present, then it will and should fail at the controller level. If this happens, there are alerting rules in place for a HelmRelease not being reconciled into a Ready state, and thus we get notified about it. We wanted the same for App CRs, so we got rid of annoyingly over-policing the applied resources in the app-admission-controller, finally.

All right, but how do these configurations get into Kubernetes? So konfigure holds the logic and provides a CLI interface to render locally. It also exposes some simple interfaces for the rendering logic in code, so any other tools can later wield the same powers by merely using these entry points. And so we built a Kubernetes controller that does the very simple thing of continuously rendering and enforcing desired state of the configurations the users ask for via custom resources provided by the controller.

It is called konfigure-operator, and is the topic of the next and final episode of this short series of Configuration Management at Giant Swarm.