Setting up Traefik#

Traefik was one of the more difficult services to setup. Every other service comes nearly configured out of the box with few exceptions. Traefik required quite a bit of careful consideration and configuration when setting up.

Because we have configured a default setting within K3s to disable traefik we are not able to name our configuration for traefik traefik.yaml, hence I have named this file traefik-custom.yaml. The reason for this is that k3s sees the traefik.yaml file, checks it’s own confiugration and sees traefik disabled, so it immedietly rectifies by deleting the file.

Below is an almost standard version of the out of the box traefik.yaml configuration with a few exceptions which are logging, entrypoints, and certificates.

traefik-custom.yaml#

Without much thought I decided to move the access.log file to a pvc/pv to allow for local reading of the log file from the host. Something I will need to fix further down the line is the node name for the nodeAffinity for PV. I will need to configure a variable for that value.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: traefik-pv
  namespace: kube-system
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: traefik-local
  local:
    path: /var/lib/nik3sx/traefik-custom
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - l-nodet

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: traefik-pvc
  namespace: kube-system
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: traefik-local
  resources:
    requests:
      storage: 2Gi

---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik-crd
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-crd-34.2.1+up34.2.0.tgz
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-34.2.1+up34.2.0.tgz
  set:
    global.systemDefaultRegistry: ""
  valuesContent: |-
    persistence:
      enabled: true
      existingClaim: traefik-pvc
      path: /data
    deployment:
      hostNetwork: true
      podAnnotations:
        prometheus.io/port: "8082"
        prometheus.io/scrape: "true"
    providers:
      kubernetesIngress:
        publishedService:
          enabled: true
    priorityClassName: "system-cluster-critical"
    image:
      repository: "rancher/mirrored-library-traefik"
      tag: "3.3.6"
    tolerations:
    - key: "CriticalAddonsOnly"
      operator: "Exists"
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "node-role.kubernetes.io/master"
      operator: "Exists"
      effect: "NoSchedule"
    service:
      ipFamilyPolicy: "PreferDualStack"
      spec:
        externalTrafficPolicy: Local
    logs:
      access:
        enabled: true
        format: json
        filePath: /data/access.log
        bufferingSize: 100
        fields:
          defaultMode: keep
          headers:
            defaultMode: keep
    entryPoints:
      web:
        address: ":80"
        forwardedHeaders:
          insecure: true
      websecure:
        address: ":443"
        forwardedHeaders:
          insecure: true
    certificatesResolvers:
      le-staging:
        acme:
          email: lecert@jmoore53.dev
          storage: /data/acme-staging.json
          caServer: https://acme-staging-v02.api.letsencrypt.org/directory
          tlsChallenge: true
      le:
        acme:
          email: jack@jmoore53.com
          storage: /data/acme.json
          tlsChallenge: true

Key Differences#

The following lines within the valuesContent: |- are the key differences between what is standard and what is not.

The reason these are in here are the following:

I wanted a way to access logs in a json format
I wanted to be able to access the sites via 80 and 443 for testing
I wanted to test out certificates quickly using my own certs as well as certs through LetsEncrypt
I needed persistence to the local host system to monitor logging.
I needed source host logging for external requests in.

valuesContent: |-
  persistence:
    enabled: true
    existingClaim: traefik-pvc
    path: /data
  deployment:
    hostNetwork: true
    podAnnotations:
      prometheus.io/port: "8082"
      prometheus.io/scrape: "true"
  providers:
    kubernetesIngress:
      publishedService:
        enabled: true
  priorityClassName: "system-cluster-critical"
  image:
    repository: "rancher/mirrored-library-traefik"
    tag: "3.3.6"
  tolerations:
  - key: "CriticalAddonsOnly"
    operator: "Exists"
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Exists"
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"
  service:
    ipFamilyPolicy: "PreferDualStack"
    spec:
      externalTrafficPolicy: Local
  logs:
    access:
      enabled: true
      format: json
      filePath: /data/access.log
      bufferingSize: 100
      fields:
        defaultMode: keep
        headers:
          defaultMode: keep
  entryPoints:
    web:
      address: ":80"
      forwardedHeaders:
        insecure: true
    websecure:
      address: ":443"
      forwardedHeaders:
        insecure: true
  certificatesResolvers:
    le-staging:
      acme:
        email: lecert@jmoore53.dev
        storage: /data/acme-staging.json
        caServer: https://acme-staging-v02.api.letsencrypt.org/directory
        tlsChallenge: true
    le:
      acme:
        email: jack@jmoore53.com
        storage: /data/acme.json
        tlsChallenge: true```