technology swamp external-secrets

How my disposable lab became a small infrastructure controller

by Jean-Philippe Evrard (employee post)

I started with a few disposable virtual machines (VM) for testing External Secrets Operator. Then I started to care about the VMs. This is how I ended up managing my local lab with a small infrastructure controller, thanks to Swamp.

The "lab"

My lab is not huge. It is what I use daily to develop External-Secrets Operator.

External Secrets Operator is a Kubernetes controller that syncs secrets from external providers, such as HashiCorp Vault, OpenBao, Vaultwarden, and Bitwarden, into Kubernetes. The project has an e2e framework that covers many cases, but not everything. Next to that, sometimes I need to do manual testing that is not covered by the e2e scenarios. This happens on thorough testing changes for example.

For that, I run on my "lab" (local machine) libvirt (with KVM), and have one VM per provider. I also have an e2e test VM, a disposable VM which creates a kind cluster and runs the command I want, for example make e2e.test or make check-diff.

I created the first provider manually. I thought it was disposable, so I probably would not need to redeploy it.

It did not take long to realise 3 things:

  1. I will have to redeploy it later
  2. I did not want to spend time doing this manually as this was both time consuming and low value work.
  3. LLMs are great to do time consuming low value tasks. They will not complain about writing a state machine/playbook/controller that will deploy my infrastructure. They also type faster than I do!

At the same time, I wanted this setup to be reliable.

I was happy to let an LLM write automation. I was not happy to let it directly manage infrastructure. That distinction matters. I wanted the agent to help build a deterministic machine, not become the machine.

Enter Swamp. Swamp organises infrastructure around models, methods, data, workflows, and extensions. For me, it is a way to ship deterministic automation that both agents and humans can run.

The lab automation, first pass

I spun up an LLM inside a freshly initialised Swamp repo and asked Claude to generate a workflow that would maintain OpenBao on my machine.

It naturally went to search existing TypeScript swamp extensions to manage the VMs, wrote shell scripts for the glue (against recommendations, but I let it slip), then wrote an extension for managing the OpenBao: The configuration/seal/unseal/...

I liked the result enough, it worked. I continued my day job and left the system as is.

The lab automation, second pass

Another day, another issue to triage, another new feature. Came Vaultwarden.

I spun up the LLM again and asked for the same thing, this time with Vaultwarden. Claude picked up the reusable components, adapted the existing models and created new ones (for example: the iso generation). Now the ugly shell scripts became models! A positive improvement :) The workflows were almost a copy of each other. It was very easy to continue this way: efficient, understandable.

So, I continued like that for a few disposable VMs, because it worked. As some people say "If it ain't broke, don't fix it!".

In other words, I started managing my lab with Swamp in the most obvious way: one workflow per service. Each VM was setup by the service workflow. Everything needed to make that machine useful was in the workflow. Self-contained.

What did it look like at that point

With the one workflow per service, my swamp looked like this:

vault-guarantee
gitea-guarantee
vaultwarden-guarantee
eso-e2e-setup

Each workflow guaranteed the full state of one service. For Gitea, that meant preparing the Arch image, creating the cloud-init seed ISO, starting the routed libvirt network, defining the VM, booting it, waiting for SSH, installing packages, generating TLS certificates, rendering app.ini, starting gitea.service, enabling Docker, and configuring nginx stream forwarding.

The upside was readability. If I wanted to know how Gitea was built, I opened gitea-guarantee and read it from top to bottom.

A simplified section looked like this:

jobs:
  - name: prepare
    steps:
      - name: run-prep
        task:
          type: model_method
          modelIdOrName: gitea-image-prep
          methodName: prepare

  - name: network
    dependsOn:
      - job: prepare
        condition:
          type: succeeded
    steps:
      - name: activate-network
        task:
          type: model_method
          modelIdOrName: routed-network
          methodName: start

  - name: vm-define
    dependsOn:
      - job: network
        condition:
          type: succeeded
    steps:
      - name: define-vm
        task:
          type: model_method
          modelIdOrName: gitea-vm
          methodName: define

  - name: ssh-wait
    dependsOn:
      - job: vm-start
        condition:
          type: completed
    steps:
      - name: wait
        task:
          type: model_method
          modelIdOrName: gitea-ssh-wait
          methodName: waitForConnection

At this point, workflows mostly called models directly. Remember, a model is a typed resource definition. I reused or developed models to represent libvirt VMs, SSH hosts, pacman package sets, systemd services, certificate deployments. Those models have methods (operations) such as prepare, start, define, sync, or waitForConnection. The workflow itself is a YAML DAG that connects those method calls. Data is the versioned output that models publish after they run.

It worked well: The result was a simple, readable, explicit, and close enough to Ansible that the mental model was easy. The workflow is both automation and documentation. Every step has a name. Every dependency is visible. When something fails, the failure boundary is obvious.

The model definitions underneath were also direct. A package model installed Gitea on a specific host. A systemd model enabled a specific service. A libvirt VM model contained the XML for that VM.

For example, gitea-pkgs knew the host address and package list:

type: '@adam/cfgmgmt/pacman'
name: gitea-pkgs

globalArguments:
  packages:
    - gitea
  ensure: present
  nodeHost: 192.168.164.12
  nodeUser: admin
  become: true
  becomeUser: root

Where the playbook style started to hurt

The playbook-style design duplicates a lot of structure.

Every service VM needed some variant of this sequence:

prepare image
start network
define VM
start VM
wait for SSH
install base packages
install Docker
configure nginx
apply service-specific setup

At three machines, this is fine. At more than that, it becomes noise.

The same facts also start appearing in too many places. IP addresses, SSH users, image paths, base images, libvirt network names, Docker setup, nginx setup, and package assumptions spread across model files and workflows.

The problem was not that the workflows were bad. They were too responsible. A service workflow owned the VM lifecycle, OS bootstrap, Docker, reverse proxy setup, certificates, and application configuration. That was convenient at first, but most of those layers were not service-specific.

If I changed how Docker should be installed, I had to update every workflow that encoded Docker installation. If the libvirt VM model changed, gitea-guarantee had to care about something that was not really about Gitea. And because each workflow called similar package and service models at different points, the duplication was not only inelegant. It also made the automation less efficient. In other words: I felt copying the model calls was dirty or inelegant. I also did not want to maintain a pile of generated automation after my agent subscription ended.

I realised this was not the right direction for me (I don't judge for other people, YMMV).

My brain started to itch: "There must be a better way!" (Read this with Raymond Hettinger's voice)

The analysis

After reading the workflows side by side, the pattern became obvious: I have a certain "taste" or an "opinion" for them: prepare an Arch linux image, define a libvirt VM, start it, wait for SSH, install Docker to run my software, configure nginx reverse proxy, then finally install the thing I actually cared about. Every VM needed the same substrate setup, the same SSH wait logic, the same Docker setup, and the same reverse proxy wiring.

I also wanted the model output to become the contract between layers.

That repetition pushed me towards a different design: a VM pool model, a capability catalogue, and a planner that turns desired state into ordered work. Like what the internals of certain configuration management tools do.

I rebuilt my lab automation because the first version was easy to read, but annoying to maintain. So I decided to do something about it in my free time :)

The current system

The current system feels less like a set of playbooks and more like a small infrastructure controller with layers:

  • A VM pool model owns the libvirt substrate.
  • A capability catalogue describes reusable setup such as SSH, base Arch configuration, Docker, and nginx stream forwarding.
  • Service workflows still exist, but they only handle the final mile.

That split made the system less immediately obvious, but much easier to evolve.

The VM pool

The VM substrate is owned by one model:

lab-vm-pool

That model uses a custom type:

@evrardjp/libvirt-vm-pool

Instead of creating one libvirt model per VM, I describe all the VMs in one desired-state list:

type: '@evrardjp/libvirt-vm-pool'
name: lab-vm-pool

globalArguments:
  uri: qemu:///system
  vms:
    - name: gitea
      desiredState: reachable
      hostname: gitea
      ipAddress: 192.168.164.12
      prefixLength: 24
      gateway: 192.168.164.1
      nameserver: 192.168.102.1
      sshUser: admin
      sshPubKeyPath: ~/.ssh/id_ed25519.pub
      memoryMiB: 2048
      vcpus: 2
      diskSizeGb: 20
      network: routed
      imagesDir: /var/lib/libvirt/images
      baseImagePath: /var/lib/libvirt/images/arch-cloud-base.qcow2
      baseImageUrl: https://geo.mirror.pkgbuild.com/images/latest/Arch-Linux-x86_64-cloudimg.qcow2
      capabilities:
        - gitea

That file is now the source of truth for the VM substrate.

I still decide which machines exist, which IPs they use, how much CPU and memory they get, which base image they boot from, and what state I want them to reach. I just do not copy libvirt XML and workflow boilerplate for each machine anymore.

The VM pool understands desired states such as absent, defined, running, and reachable.

So I can ask Swamp to plan first:

swamp model method run lab-vm-pool plan --timeout 120

Then I can reconcile the substrate:

swamp model method run lab-vm-pool sync --timeout 1800

For my lab, the plan/sync split is a minor improvement. I can inspect what Swamp intends to do before it mutates the local libvirt host.

Data becomes the contract

The most important change is not that one model manages many VMs. The important change is that the VM pool publishes facts about those VMs as Swamp data.

In the old design, downstream models hardcoded connection details:

nodeHost: 192.168.164.12
nodeUser: admin

In the new design, downstream workflows can consume the VM pool output:

data.latest("lab-vm-pool", "gitea").attributes.ipAddress
data.latest("lab-vm-pool", "gitea").attributes.sshUser

That changes the coupling between layers.

The VM pool owns VM facts. Other models consume those facts. If I change the address of a VM in the pool, I do not want to hunt through unrelated package, systemd, and nginx definitions. The pool should publish the new fact, and the rest of the automation should follow it.

That is the part that made Swamp click for this use case. The workflow is not just a sequence of commands. It can be a data pipeline between infrastructure layers.

Capabilities describe reusable setup

The VM pool knows that a VM needs a capability:

capabilities:
  - gitea

But the pool does not define what gitea means. That lives in another model:

lab-capability-catalog

The catalogue maps capabilities to dependencies and implementation workflows:

capabilities:
  ssh:
    description: Wait for SSH connectivity to the VM.
    requires: []
    implementation:
      type: workflow
      workflowIdOrName: lab-capability-ssh
      inputs: {}

  base-arch:
    description: Apply baseline Arch Linux package/key setup.
    requires:
      - ssh
    implementation:
      type: workflow
      workflowIdOrName: lab-capability-base-arch
      inputs: {}

  docker:
    description: Install and enable Docker runtime.
    requires:
      - base-arch
    implementation:
      type: workflow
      workflowIdOrName: lab-capability-docker
      inputs: {}

  gitea:
    description: Gitea service final-mile setup.
    requires:
      - base-arch
      - docker
      - nginx-stream
    implementation:
      type: workflow
      workflowIdOrName: gitea-guarantee
      inputs: {}

This gives me a small dependency graph. A VM can ask for gitea, and the planner can infer that it also needs the base Arch setup, Docker, nginx stream support, and SSH.

That removes another class of repetition. VMs describe intent. The capability catalogue describes what that intent requires.

Planning into waves

The capability planner combines two inputs:

  1. VM facts from lab-vm-pool.
  2. Capability definitions from lab-capability-catalog.

It produces dependency-ordered waves. Items in the same wave can run independently. Later waves wait for earlier ones.

The workflow that wires this together uses Swamp data directly:

inputs:
  vms: ${{ data.findBySpec("lab-vm-pool", "vm").map(x, x.attributes) }}
  capabilities: ${{ data.findBySpec("lab-capability-catalog", "capability").map(x, x.attributes) }}

This is the part I like. The workflow does not parse the VM pool YAML manually. It does not duplicate catalogue information. It consumes model outputs.

Then another workflow applies a wave by calling the selected implementation workflow for each item:

steps:
  - name: apply-${{ self.item.host }}-${{ self.item.capability }}
    forEach:
      item: item
      in: ${{ inputs.items }}
    task:
      type: workflow
      workflowIdOrName: ${{ self.item.implementation.workflowIdOrName }}
      inputs:
        host: ${{ self.item.host }}
        capability: ${{ self.item.capability }}
        vm: ${{ self.item.vm }}
        implementationInputs: ${{ self.item.implementation.inputs }}

The workflow is generic. The plan decides what it runs.

That is the factory pattern in practice. One piece of orchestration handles many concrete instances because the instance-specific information moved into data.

Generic where possible, explicit where useful

Not every workflow became generic, and I do not want it to.

SSH, Docker, base OS setup, and nginx stream support are generic capabilities. They should take a VM object as input and apply the same logic to any host.

For example, the Docker capability can install packages and enable the service using the VM data it receives:

globalArgs:
  packages:
    - docker
    - docker-compose
  ensure: present
  nodeHost: ${{ inputs.vm.ipAddress }}
  nodeUser: ${{ inputs.vm.sshUser }}
  nodePort: 22
  nodeIdentityFile: ~/.ssh/id_ed25519
  become: true
  becomeUser: root

That replaces separate models like gitea-docker-pkgs, vaultwarden-docker-pkgs, and bao-docker-pkgs.

The service workflows remain explicit. gitea-guarantee still installs Gitea, generates and deploys certificates, renders app.ini, and starts gitea.service. It just no longer owns the VM substrate, SSH bootstrap, Docker installation, or nginx stream setup.

That hybrid feels right.

The generic infrastructure is generic. The application-specific workflow still reads like application-specific automation.

What I still maintain manually

The factory pattern did not remove human input. It moved the input to better places.

I still maintain the VM pool definition. That is where machine names, IP addresses, desired states, CPU, memory, disk size, image paths, SSH users, and requested capabilities live.

I still maintain the capability catalogue. That is where capability names, dependency edges, and workflow implementations live.

I still maintain final-mile service configuration. Gitea still needs its package, config template, certificate paths, and systemd service. OpenBao still needs its own configuration and initialisation flow. Vaultwarden still has service-specific behaviour.

That is fine. Those things are actually service-specific.

The win is that I stopped repeating the common VM lifecycle around every service.

The operational loop

The day-to-day flow now looks roughly like this:

swamp model validate lab-vm-pool --json
swamp model validate lab-capability-catalog --json
swamp model validate lab-capability-plan --json

swamp workflow run lab-plan-capabilities
swamp data get lab-capability-plan current --json

swamp model method run lab-vm-pool sync --timeout 1800 --skip-reports
swamp workflow run lab-apply-capability-plan

I also wrap those commands with mise tasks:

mise run lab:validate
mise run lab:plan
mise run lab:evaluate
mise run lab:sync-substrate
mise run lab:apply

The important property is that I can plan before I apply. I can inspect the VM substrate plan. I can inspect the capability plan. Then I can reconcile.

For a homelab, that may sound heavier than a few shell scripts. For me, the structure pays for itself because the lab is no longer just a few commands. It is a small system with state, dependencies, and drift.

And when a new provider needs its own VM, I can extend this quite easily with my agent. Because the structure is well defined, it's very unlikely that both the LLM and myself mess this up, together.

What I learned

The playbook style is the better starting point.

One guarantee workflow per service is easy to understand. It makes good documentation. It composes nicely from small models. If the lab has only a few machines, I would start there again.

The factory pattern becomes useful when the duplicated parts become more interesting than the service-specific parts. Once most workflows share the same substrate setup, SSH wait, Docker install, and proxy setup, those pieces deserve their own layer.

The trade-off is abstraction. The old design lets me open one workflow and see everything. The new design requires me to understand the relationship between the VM pool, the capability catalogue, the planner, the wave applier, and the final-mile workflows.

I accept that trade-off because each layer now has a cleaner job.

lab-vm-pool owns VM substrate. lab-capability-catalog describes reusable capabilities. lab-capability-plan turns desired capabilities into ordered work. Generic workflows apply common setup. Service workflows handle the final mile.

That is the balance I wanted: readable workflows where the details matter, and reusable infrastructure logic where repetition used to hide the signal.

For this lab, that is the line I wanted to draw. Let playbooks explain the service-specific work. Let models and data carry the shared infrastructure facts.

Your mileage may vary, but that split made my lab much easier to evolve.