Giant Swarm Blog | Kubernetes Insights from the team at Giant Swarm

Configuration management at Giant Swarm: reconciling konfigurations » Giant Swarm

Written by Laszlo Uveges | Nov 17, 2026, 11:15:26 AM

One broken template in one app's configuration used to be able to stall configuration updates across every management cluster at once. This is the third of three posts on configuration management at Giant Swarm, and it's about the operator we built to fix that.

In part two we took a look at how Giant Swarm built and switched its cluster configuration system to a dynamic, extensible, schema-based system with konfigure. Before that, in part one, we had a historical overview of where this system originated from and how and why it evolved over the years into its current state.

We finished the last post noting that konfigure is merely the brain: a library and CLI for rendering configurations. Getting these configurations into Kubernetes and keeping them up-to-date is a separate responsibility, which we decoupled into a newly built operator, called konfigure-operator.

What the new operator fixed

Let's look at some convenience features we added with the operator and get an overview of the problems we solved by creating konfigure-operator and, with it, separating the concerns of creating and using configuration.

The poison pill problem

In the first article we called this the poison pill problem. With konfigure running as a KRM function inside Flux kustomize-controller, it meant that a single rendering error in any of the dozens of apps in a Flux Kustomization blocked the whole Kustomization in an error state, and nothing else got reconciled. This was even worse when something in our shared, non-customer-specific configuration broke, because then it was capable of poisoning the Kustomization across all management clusters at once.

It also did not help that kustomize and kustomize-controller did not cleanly surface the problem on the Flux Kustomization status. You had to check the logs of kustomize-controller.

It is all in the past now. The konfigure-operator control loop handles each individual iteration entry on its own. If one iteration fails, it will be clearly marked in the .status.failed list which iteration failed and what error message surfaced from the invoked konfigure interface. Clean and simple. The rest of the iterations will still be kept updated. A single failed iteration will, however, also mark the Ready type condition under .status.conditions as status: false, so we can easily alert on such cases.

status:
  conditions:
    - lastTransitionTime: "2025-03-12T15:06:07Z"
      message: 'Attempted revision: c8f73a3b5ad0ddaad337d78f4e49ea8eae49d2a7'
      observedGeneration: 4
      reason: ReconciliationFailed
      status: "False"
      type: Ready
  failed:
    - name: app-operator
      message: 'failed to render template from "default/apps/app-operator/configmap-values.yaml.template":
      template: main:75:37: executing "main" at <.workloadCluster.ssh.ssoPublicKey>:
      map has no entry for key "ssoPublicKey"'

Suspending reconciliation on a single rendered manifest

Sometimes you need to take a single rendered ConfigMap and/or Secret out of reconciliation, whether for testing, debugging, or an emergency fix. Previously, it was possible to label or annotate the given manifest with kustomize.toolkit.fluxcd.io/reconcile: disabled, because Flux managed them.

This worked pretty fine, but developers often forgot to remove the annotation or label, leaving manifests stale until someone noticed. It was a necessary but unhealthy practice.

The label configuration.giantswarm.io/reconcile: disabled was introduced by konfigure-operator. We decided to only allow it as a label - unlike the Flux equivalent - because as a label it is at least searchable. Nevertheless, the operator also keeps track of disabled items under its .status.disabled list, so you can clearly see and act if something is unintentionally going stale.

status:
  disabled:
  - name: app-operator
    kind: ConfigMap
    target:
      name: app-operator-konfiguration
      namespace: giantswarm

The .name field is the key value in the .spec.targets.iterations map. The rest is to identify the actual manifest that carries the above label to disable reconciliation.

Konfigure operator ecosystem

Let’s get a bird's-eye view of the components. The operator is a simple wrapper around konfigure. It provides two Custom Resource Definitions to operate on: Konfiguration and KonfigurationSchema.

The KonfigurationSchema

This is a very simple resource. Setting .spec.raw.remote.url you can point to a remote file that contains the schema. The file is parsed as a Konfiguration Schema as defined in the included konfigure library in the operator.

apiVersion: konfigure.giantswarm.io/v1alpha1
kind: KonfigurationSchema
metadata:
  name: management-cluster-configuration
  namespace: giantswarm
spec:
  raw:
    remote:
      url: https://raw.githubusercontent.com/giantswarm/konfiguration-schemas/refs/tags/management-cluster-configuration/v1.1.0/schemas/management-cluster-configuration/schema.yaml

Alternatively you can store the contents of the schema under .spec.raw.content. This method is useful for development and debugging, or simply to avoid fetching the schema from, let's say, GitHub all the time.

The Konfiguration

This is the meat and potatoes of the operator. It is used to describe the configurations that you want to render. It is built up from the following blocks:

  • sources: is used to describe how to fetch the source for the configuration repository
  • targets: holds the necessary information on what configurations to reconcile
  • destination: can customize how and where to put the rendered configurations
  • reconciliation: has settings that affect the control loop of the operator

Sources

This section describes where from and how to fetch the repository. Since we extensively use Flux in Giant Swarm, we primarily added support for fetching the source from a Flux GitRepository source.

spec:
  sources:
    flux:
      gitRepository:
        name: "giantswarm-configurations"
        namespace: "flux-system"

Targets

This section is the core of the CRD. It is used to describe how (which schema to use to parse the referenced configuration repository) and what (which paths to render from it).

The .schema object currently only supports .reference, which points to a KonfigurationSchema CR on the cluster via its .name and .namespace fields. It will be looked up (or a cached version used, if the CR has not changed) by each individual control loop cycle, and then each individual iteration item will be interpreted with that instance.

The .defaults object holds .variables shared across all .iterations entries.

The .iterations map is looped for each key and its entry object. The .variables defined within are unique for the entry and merged on top of the defaults. This means that an iteration, if necessary, can override defaults.

spec:
  targets:
    schema:
      reference:
        name: management-cluster-configuration
        namespace: giantswarm
    defaults:
      variables:
        - name: installation
          value: gorilla
        - name: stage:
          value: production
    iterations:
      app-admission-controller:
        variables:
          - name: app
            value: app-admission-controller
      app-operator:
        variables:
          - name: app
            value: app-operator
      # ...

Taking the app-operator example from the map, it would translate to the following konfigure render CLI call:

SOPS_AGE_KEY_FILE=example.age konfigure render \
  --schema schema.yaml \
  --dir . \
  --variable "installation=gorilla" \
  --variable="stage=production" \
  --variable="app=app-operator"

The keys for encrypted data will be automatically set up for SOPS by the controller fetching all secrets from the clusters with the konfigure.giantswarm.io/data: sops-keys label.

Of course, the schema is fetched from the KonfigurationSchema and the source - passed to --dir - is fetched from the .spec.source. Please note that the controller does it all “natively” by invoking the library interfaces of konfigure itself, rather than translating it into raw CLI calls. This is merely to demonstrate how to translate the CR for local testing.

There are some other flags that would need to be set to replicate the exact result of the Konfiguration, for example related to the destination ConfigMap and Secret.

Destination

This section customizes where the rendered ConfigMaps and Secrets will be placed. The .namespace field will obviously be translated to their .metadata.namespace field. The .naming section contains configurations on how to manipulate their .metadata.name fields. Each will by default get the name of the iteration. The .suffix and .prefix field values will be appended and prepended respectively to that name. The .useSeparator field controls whether these should be glued on as is or with a single hyphen character. This is the default behavior.

spec:
  destination:
    namespace: giantswarm
    naming:
      suffix: konfiguration
      useSeparator: true

Taking the full example so far for app-operator, the controller will create and reconcile the ConfigMap and Secret called app-operator-konfiguration in the giantswarm namespace.

Reconciliation

This section is nothing fancy. It contains some settings to customize the behavior of the control loop. The .suspend flag can be used to freeze reconciliation of the Konfiguration until further update.

spec:
  reconciliation:
    interval: 1m
    retryInterval: 30s
    suspend: false

Standing on solid ground

Looking back across these three posts, we have come a long way. The current system unlocked so many improvements for us. Beyond the above improvements, at the larger scale:

  • We were able to completely refactor a huge portion of our management cluster deployments (we call them collections) by getting rid of a dozen repositories and moving everything into a single place, because we no longer needed the KRM function approach
  • It enabled us to move these collections from App CRs to Flux HelmRelease CRs, because of the decoupling of generation and usage
  • We were able to simply introduce stages into our configuration management by extending the schema with a couple of lines of new YAML declarations
  • It opened the way for almost any kind of configuration need that may arise in the future for us and for our customer base, with clear local reproducibility and a clean, Kubernetes-native way of delivering and keeping configurations up-to-date across large, multi-layered systems, across many, many clusters.

Surely, none of these layers will last forever. Sediment is building all the time. There are improvements to be made. The schema and the system could get even more generic. The operator could grow to support new kinds of sources. Something we haven't imagined yet will probably force another rethink in a few years. But for now, we're standing on solid ground. A configuration system that is simple, declarative, extensible, decoupled, and resilient.