Declarative Systems - Nik3sx#

From custom python scripts using ansible_runner module to bash scripts and docker commands, every server provisioning system I have worked on or built up until this point has been procedural based. I wanted to give something new a try as I moved towards declartive systems, so I decided to become more familiar with NIX operating system and using K3s as a lightweight kubernetes system to run containers.

I have always been looking for new ways to do things with my homelab related to devops, and this felt like the next logical step in my journey. Initially I wanted to move to k3s on a debian based system, however I decided to jump all in using NIXOS when I saw I needed tools besides k3s, like restic, go (for development), tmux, etc on my system and nix offered a declarative based system for setting up and configuring these applications and tools. The reason I didn’t opt for something like a golden image with Debian was due to the slow testing process and my lack of infrastructure. I had a hard time with cloud-init testing on my local device and I really didn’t want to spin up a dhcp server for testing out vm builds.

Through this documentation, as a series of posts, I plan to provide my own thought process for developing, and deploying a single host with NIXOS and K3s containing the essential tools for my life. This system is not ideal, however it works towards what I am trying to achieve which is an immutable set of systems (OS, application platform) for deploying a server with a few services.

Below are a few reasons on “why” I wanted to do this:

Declartive vs Procedural Systems#

The primary difference I observed between declarative and procedural systems is where the complexity lives. In procedural systems, like bash scripts or traditional Ansible playbooks, I ended up being responsible for the server reaching a desired state. This often meant I handled the ordering, conditionals, retries, and edge cases when they arose. Over time, these scripts grew defensive logic to account for drift, partial failure, or unexpected initial conditions. That is where I ended up finding myself using python to call ansible.

Declarative systems shift this responsibility. Instead of describing the steps required to reach a state, you describe the state itself and rely on the system to converge toward it. With NixOS, I declare what the operating system should look like. With Kubernetes, I declare what workloads should be running. In both cases, the tooling is responsible for reconciliation.

This approach aligns better with how I want to reason about infrastructure. I am less interested in the exact sequence of steps taken during provisioning and more interested in whether the system currently matches what I have defined. The result is configuration that is easier to reason about at rest, even if the underlying implementation is more complex.

Managing State#

I wanted a way to track the difference between the actual cluster (currently 1 node) state and the state declared in git. By using NIX and K3s I am able to declare the states of both applications installed on the operating system and services running in kubernetes within K3s.

While ansible is good at procedurally setting and running to get to a state, I wanted to try a new way of doing this instead of through ansible playbooks and roles. Using NIX and K3s seemed like a better way to keep the entire system checked in as code moving more towards a true Infrastructure as Code solution.

Where I currently see this NIX fall short is in setting up multiple nodes. Due to this setup being one server there aren’t issues as the complexity is low, and if I needed to deploy 100 of these systems with the application state (databases in this case) separated I could, however for the case of provisioning multiple systems with different tooling and a common layer of tooling I think Ansible would be a better solution (think web frontend servers, application level logic servers, and databse servers, all needing common tools, however all performing separate functions).

Testing#

Where I found the most success was with nix flakes, and using a qemu export. I will dive into this more later, but being able to spin up a vm using qemu made testing simple and quick. I was able to test changes relatively fast which made development easier.

Why do I compare this to Ansible?#

Ansible is what I have used for provisioning users tools, and services in the past. Ansible is a tool I was familiar with and used to both setup from scratch on the proxmox server I had, as well as configure vms once they were online.

How this is setup#

I plan to show as much code as possible in this blog, but most of these pages are setup to track “why and how.”

AI Disclaimer#

I used AI to generate some of the code for this project, however none was used to write these posts. If you think this project is slop and you hate it, please let me know as I would love to hear feedback. The tool is something I designed and created for myself as a way to restore my own systems.