Configuration management at Giant Swarm: reconciling konfigurations
by Laszlo Uveges on
Table of contents
One broken template in one app's configuration used to be able to stall configuration updates across every management cluster at once. This is the third of three posts on configuration management at Giant Swarm, and it's about the operator we built to fix that.
In part two we took a look at how Giant Swarm built and switched its cluster configuration system to a dynamic, extensible, schema-based system with konfigure. Before that, in part one, we had a historical overview of where this system originated from and how and why it evolved over the years into its current state.
We finished the last post noting that konfigure is merely the brain: a library and CLI for rendering configurations. Getting these configurations into Kubernetes and keeping them up-to-date is a separate responsibility, which we decoupled into a newly built operator, called konfigure-operator.
What the new operator fixed
Let's look at some convenience features we added with the operator and get an overview of the problems we solved by creating konfigure-operator and, with it, separating the concerns of creating and using configuration.
The poison pill problem
In the first article we called this the poison pill problem. With konfigure running as a KRM function inside Flux kustomize-controller, it meant that a single rendering error in any of the dozens of apps in a Flux Kustomization blocked the whole Kustomization in an error state, and nothing else got reconciled. This was even worse when something in our shared, non-customer-specific configuration broke, because then it was capable of poisoning the Kustomization across all management clusters at once.
It also did not help that kustomize and kustomize-controller did not cleanly surface the problem on the Flux Kustomization status. You had to check the logs of kustomize-controller.
It is all in the past now. The konfigure-operator control loop handles each individual iteration entry on its own. If one iteration fails, it will be clearly marked in the .status.failed list which iteration failed and what error message surfaced from the invoked konfigure interface. Clean and simple. The rest of the iterations will still be kept updated. A single failed iteration will, however, also mark the Ready type condition under .status.conditions as status: false, so we can easily alert on such cases.
status:
conditions:
- lastTransitionTime: "2025-03-12T15:06:07Z"
message: 'Attempted revision: c8f73a3b5ad0ddaad337d78f4e49ea8eae49d2a7'
observedGeneration: 4
reason: ReconciliationFailed
status: "False"
type: Ready
failed:
- name: app-operator
message: 'failed to render template from "default/apps/app-operator/configmap-values.yaml.template":
template: main:75:37: executing "main" at <.workloadCluster.ssh.ssoPublicKey>:
map has no entry for key "ssoPublicKey"'
Suspending reconciliation on a single rendered manifest
Sometimes you need to take a single rendered ConfigMap and/or Secret out of reconciliation, whether for testing, debugging, or an emergency fix. Previously, it was possible to label or annotate the given manifest with kustomize.toolkit.fluxcd.io/reconcile: disabled, because Flux managed them.
This worked pretty fine, but developers often forgot to remove the annotation or label, leaving manifests stale until someone noticed. It was a necessary but unhealthy practice.
The label configuration.giantswarm.io/reconcile: disabled was introduced by konfigure-operator. We decided to only allow it as a label - unlike the Flux equivalent - because as a label it is at least searchable. Nevertheless, the operator also keeps track of disabled items under its .status.disabled list, so you can clearly see and act if something is unintentionally going stale.
status:
disabled:
- name: app-operator
kind: ConfigMap
target:
name: app-operator-konfiguration
namespace: giantswarm
The .name field is the key value in the .spec.targets.iterations map. The rest is to identify the actual manifest that carries the above label to disable reconciliation.
Konfigure operator ecosystem
Let’s get a bird's-eye view of the components. The operator is a simple wrapper around konfigure. It provides two Custom Resource Definitions to operate on: Konfiguration and KonfigurationSchema.
The KonfigurationSchema
This is a very simple resource. Setting .spec.raw.remote.url you can point to a remote file that contains the schema. The file is parsed as a Konfiguration Schema as defined in the included konfigure library in the operator.
apiVersion: konfigure.giantswarm.io/v1alpha1
kind: KonfigurationSchema
metadata:
name: management-cluster-configuration
namespace: giantswarm
spec:
raw:
remote:
url: https://raw.githubusercontent.com/giantswarm/konfiguration-schemas/refs/tags/management-cluster-configuration/v1.1.0/schemas/management-cluster-configuration/schema.yaml
Alternatively you can store the contents of the schema under .spec.raw.content. This method is useful for development and debugging, or simply to avoid fetching the schema from, let's say, GitHub all the time.
The Konfiguration
This is the meat and potatoes of the operator. It is used to describe the configurations that you want to render. It is built up from the following blocks:
sources: is used to describe how to fetch the source for the configuration repositorytargets: holds the necessary information on what configurations to reconciledestination: can customize how and where to put the rendered configurationsreconciliation: has settings that affect the control loop of the operator
Sources
This section describes where from and how to fetch the repository. Since we extensively use Flux in Giant Swarm, we primarily added support for fetching the source from a Flux GitRepository source.
spec:
sources:
flux:
gitRepository:
name: "giantswarm-configurations"
namespace: "flux-system"
Targets
This section is the core of the CRD. It is used to describe how (which schema to use to parse the referenced configuration repository) and what (which paths to render from it).
The .schema object currently only supports .reference, which points to a KonfigurationSchema CR on the cluster via its .name and .namespace fields. It will be looked up (or a cached version used, if the CR has not changed) by each individual control loop cycle, and then each individual iteration item will be interpreted with that instance.
The .defaults object holds .variables shared across all .iterations entries.
The .iterations map is looped for each key and its entry object. The .variables defined within are unique for the entry and merged on top of the defaults. This means that an iteration, if necessary, can override defaults.
spec:
targets:
schema:
reference:
name: management-cluster-configuration
namespace: giantswarm
defaults:
variables:
- name: installation
value: gorilla
- name: stage:
value: production
iterations:
app-admission-controller:
variables:
- name: app
value: app-admission-controller
app-operator:
variables:
- name: app
value: app-operator
# ...
Taking the app-operator example from the map, it would translate to the following konfigure render CLI call:
SOPS_AGE_KEY_FILE=example.age konfigure render \
--schema schema.yaml \
--dir . \
--variable "installation=gorilla" \
--variable="stage=production" \
--variable="app=app-operator"
The keys for encrypted data will be automatically set up for SOPS by the controller fetching all secrets from the clusters with the konfigure.giantswarm.io/data: sops-keys label.
Of course, the schema is fetched from the KonfigurationSchema and the source - passed to --dir - is fetched from the .spec.source. Please note that the controller does it all “natively” by invoking the library interfaces of konfigure itself, rather than translating it into raw CLI calls. This is merely to demonstrate how to translate the CR for local testing.
There are some other flags that would need to be set to replicate the exact result of the Konfiguration, for example related to the destination ConfigMap and Secret.
Destination
This section customizes where the rendered ConfigMaps and Secrets will be placed. The .namespace field will obviously be translated to their .metadata.namespace field. The .naming section contains configurations on how to manipulate their .metadata.name fields. Each will by default get the name of the iteration. The .suffix and .prefix field values will be appended and prepended respectively to that name. The .useSeparator field controls whether these should be glued on as is or with a single hyphen character. This is the default behavior.
spec:
destination:
namespace: giantswarm
naming:
suffix: konfiguration
useSeparator: true
Taking the full example so far for app-operator, the controller will create and reconcile the ConfigMap and Secret called app-operator-konfiguration in the giantswarm namespace.
Reconciliation
This section is nothing fancy. It contains some settings to customize the behavior of the control loop. The .suspend flag can be used to freeze reconciliation of the Konfiguration until further update.
spec:
reconciliation:
interval: 1m
retryInterval: 30s
suspend: false
Standing on solid ground
Looking back across these three posts, we have come a long way. The current system unlocked so many improvements for us. Beyond the above improvements, at the larger scale:
- We were able to completely refactor a huge portion of our management cluster deployments (we call them collections) by getting rid of a dozen repositories and moving everything into a single place, because we no longer needed the KRM function approach
- It enabled us to move these collections from App CRs to Flux
HelmReleaseCRs, because of the decoupling of generation and usage - We were able to simply introduce stages into our configuration management by extending the schema with a couple of lines of new YAML declarations
- It opened the way for almost any kind of configuration need that may arise in the future for us and for our customer base, with clear local reproducibility and a clean, Kubernetes-native way of delivering and keeping configurations up-to-date across large, multi-layered systems, across many, many clusters.
Surely, none of these layers will last forever. Sediment is building all the time. There are improvements to be made. The schema and the system could get even more generic. The operator could grow to support new kinds of sources. Something we haven't imagined yet will probably force another rethink in a few years. But for now, we're standing on solid ground. A configuration system that is simple, declarative, extensible, decoupled, and resilient.
Ready to simplify your platform?
See how Giant Swarm helps platform teams run Kubernetes at scale.
Contact us