Declarative Systems - Nik3sx on Declarative Systems

Setting up Traefik

Mon, 01 Jan 0001 00:00:00 +0000

Setting up Traefik#

Traefik was one of the more difficult services to setup. Every other service comes nearly configured out of the box with few exceptions. Traefik required quite a bit of careful consideration and configuration when setting up.

Because we have configured a default setting within K3s to disable traefik we are not able to name our configuration for traefik traefik.yaml, hence I have named this file traefik-custom.yaml. The reason for this is that k3s sees the traefik.yaml file, checks it’s own confiugration and sees traefik disabled, so it immedietly rectifies by deleting the file.

Why NIX

Mon, 01 Jan 0001 00:00:00 +0000

Why Nix#

One of the things nix provides is a way to define entire systems and environments as reproducable, declarative code. The same configuration file provides identical results across machines reducing drift and inconsistencies.

Modular System#

Whether a component needs to be added or removed this is all able to be done via code.

Nix Store#

The Nix store is the foundation of how Nix manages software and system state. It is a content-addressed directory where all packages, dependencies, and build outputs are stored in isolated immutable paths.

Setting up K3s on NIX

Mon, 01 Jan 0001 00:00:00 +0000

Setting up K3s on NIX#

Flake.nix#

Below is the flake I am using for this project. You may note I have a couple different outputs, including one configuration, and two packages (images using nixosgenerators). Being able to use QEMU to quickly launch an image from my local pc using the ./result/bin/ folder. The other two outputs are actual qcow and vhd images for use within my other testing environments (virt-manager and aws).

Setting up Vaultwarden

Mon, 01 Jan 0001 00:00:00 +0000

Setting up Vaultwarden#

*Vaultwarden is a lightweight, self-hosted implementation of the Bitwarden server API.

It lets you run your own password manager backend (instead of using Bitwarden’s cloud service), while still using official Bitwarden clients. It is written in Rust and designed to be resource-efficient, making it suitable for small servers or homelab setups.*

I wanted to show a few services, but for the most part after K3s is up and online the services are self explanitory through code. Each service is likely to vary in its own way, but I wanted to present a few.

Root User Provisioning Service

Mon, 01 Jan 0001 00:00:00 +0000

Root User Provisioning Service#

This is one of the more controversial services I have, but I needed a way to restore the system without anything except the ISO and a password.

With this service I am setting up root provisioning via a web link. I will curl the link with a post request and then it will add SSH keys to the root user on the server.

Is this dangerous? Yes, however I understand the risks associated. I also tried to design the service to only run if the system was net new. The service will exit if the done file already exists /var/lib/nik3sx/boot-unlock-service/done.

Setting up Kanboard

Mon, 01 Jan 0001 00:00:00 +0000

Setting up Firefly3#

*Kanboard is a lightweight, self-hosted Kanban-style project management tool.

It lets you organize tasks on boards with columns (e.g., To Do → Doing → Done), focusing on simplicity, minimal dependencies, and low resource usage.*

Setting up Firefly3

Mon, 01 Jan 0001 00:00:00 +0000

Setting up Firefly3#

*Firefly III is a self-hosted personal finance manager.

It helps you track expenses, income, budgets, and accounts, giving you detailed insight into your spending while keeping all financial data under your control.*

Systemd Services

Mon, 01 Jan 0001 00:00:00 +0000

Systemd Services#

Using Systemd seemed like the only choice available for the setup of some of these services. Systemd provided the service management, dependency handling, and monitoring I needed.

Service Management#

Most of these scripts only need to run if the server is unconfigured. Using systemd I can set these scripts to be oneshot services.

serviceConfig = {
 Type = "oneshot"
}

Below is an example service for k3s to ensure the API is online before attempting to apply the manifest files.

Encrypting and Decrypting

Mon, 01 Jan 0001 00:00:00 +0000

Encrypting and decrypting#

One of the features I needed for a full declarative setup was a way to restore data back to the system. If I were not using a separate file system mounted like /var or /lib I would need a way to restore data. With restoration of data I needed a way to encrypt files with Age and then decrypt the files with Age as well.

Below is the Systemd service and Golang program I wrote to solve for this problem. It is named smanager for “secrets manager”.

Luks and /var/lib for Data

Mon, 01 Jan 0001 00:00:00 +0000

Luks and /var/lib#

I am currently running this configuration inside virt-manager for a virtual machine.

I don’t currently have this setup for this use case, however it is something I plan to implement. The reason I don’t have this implemented is due to the way disks are shared within AWS. I can share this as an EBS volume, but there is more work that is needed like getting the disk uuid.

Restoring Data

Mon, 01 Jan 0001 00:00:00 +0000

Restoring Data#

After the smanager program has unlocked the environment variables Restic is able to restore data.

Below are some of the scripts I use to restore data to services.

Every service is a bit different especially with database restores, but below documents the three core services I have.

# Unlock restic backups
/var/lib/nik3sx/smanager/agefile-pass -decrypt -in /var/lib/nik3sx/smanager/env.age -out /var/lib/nik3sx/smanager/env
source /var/lib/nik3sx/smanager/env

restic snapshots

# Restore Vaultwarden
mkdir -p /var/lib/restore_vw
restic restore <snapshot-id> --target /var/lib/restore_vw
cp -r /var/lib/restore_vw/.../vaultwarden/bw-data/* /var/lib/nik3sx/vaultwarden/
kubectl rollout restart deployment vaultwarden


# Restore Kanboard
mkdir -p /var/lib/restore_kb
restic restore <snapshot-id> --target /var/lib/restore_kb
cat /var/lib/restore_kb/.../backup-2026-01-14.sql | kubectl exec -i pods/db-55d74b64-2rqxv -- mariadb -u kb -pkb-secret kanboard


# Restore Firefly_III
mkdir -p /var/lib/restore_f3
restic restore <snapshot-id> --target /var/lib/restore_f3
kubectl create configmap firefly-env --from-env-file=/var/lib/restore_f3/.env -n firefly
kubectl create secret generic firefly-db-env --from-env-file=/var/lib/restore_f3/.db.env -n firefly
cat /var/lib/restore_f3/.../backup-2026-01-14.sql | kubectl exec -i pods/db-55d74b64-2rqxv -- mariadb -u kb -pkb-secret kanboard

Rate Limiting

Mon, 01 Jan 0001 00:00:00 +0000

Rate Limiting#

This was built as a go service to ingest logs and then limit IPs.

Adding PVC#

apiVersion: v1
kind: PersistentVolume
metadata:
 name: traefik-pv
spec:
 capacity:
 storage: 2Gi
 accessModes:
 - ReadWriteOnce
 storageClassName: traefik-local
 local:
 path: /var/lib/nik3sx/traefik
 nodeAffinity:
 required:
 nodeSelectorTerms:
 - matchExpressions:
 - key: kubernetes.io/hostname
 operator: In
 values:
 - l-nodet

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: traefik-pvc
spec:
 accessModes:
 - ReadWriteOnce
 storageClassName: traefik-local
 resources:
 requests:
 storage: 2Gi

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
 name: traefik
 namespace: kube-system
spec:
 chart: https://%{KUBERNETES_API}%/static/charts/traefik-34.2.1+up34.2.0.tgz
 set:
 global.systemDefaultRegistry: ""
 valuesContent: |-

# Needed to add this persistence using the pvc
 persistence:
 enabled: true
 existingClaim: traefik-pvc
 path: /data
# End line addition

 deployment:
 podAnnotations:
 prometheus.io/port: "8082"
 prometheus.io/scrape: "true"
 providers:
 kubernetesIngress:
 publishedService:
 enabled: true
 priorityClassName: "system-cluster-critical"
 image:
 repository: "rancher/mirrored-library-traefik"
 tag: "3.3.6"

This allows us to access the access logs from the host for our go service.

Deploying

Mon, 01 Jan 0001 00:00:00 +0000

Deploying#

My next itch I needed to scratch was the solution to deploy nix hosts. Deploying NIX on the local host is easy with qemu, but I needed a better cloud solution to deploy nik3sx.

Because there was no native solution for deploying NIX images I decided to use a debian base and then a firecracker runtime for NIX.

Declarative Systems - Nik3sx on Declarative Systems

Setting up Traefik

Setting up Traefik#

Why NIX

Why Nix#

Modular System#

Nix Store#

Setting up K3s on NIX

Setting up K3s on NIX#

Flake.nix#

Setting up Vaultwarden

Setting up Vaultwarden#

Root User Provisioning Service

Root User Provisioning Service#

Setting up Kanboard

Setting up Firefly3#

Setting up Firefly3

Setting up Firefly3#

Systemd Services

Systemd Services#

Service Management#

Encrypting and Decrypting

Encrypting and decrypting#

Luks and /var/lib for Data

Luks and /var/lib#

Restoring Data

Restoring Data#

Rate Limiting

Rate Limiting#

Adding PVC#

Deploying

Deploying#

MicroVM#