twenty-four: the platform

2025-10-08 591 words 3 minutes

This is the infrastructure that makes twenty-four possible.

The Cluster

k3s on a mix of hardware that was lying around:

Control plane:

master - Ubuntu 22.04, x86_64, handles scheduling

Workers:

arm1, arm2, arm3 - Fedora Asahi Remix on Apple Silicon (M1/M2 Mac Minis)
lab - Ubuntu 24.04, x86_64, old desktop turned compute node
brain - Ubuntu 24.04, x86_64, currently offline (it’ll be back)

The ARM nodes do most of the work. They’re fast, quiet, and sip power. The x86 box handles anything that doesn’t have ARM builds yet.

Networking: The .home Domain

Everything runs on .home. Local DNS via dnsmasq on my router (192.168.0.1), which forwards .home queries to CoreDNS in the cluster (10.233.64.1).

1
2
3
4
5


home.kirsch.org (router)
  └─> .home → CoreDNS (k8s)
      ├─> calendar.twenty-four.home → 10.233.66.6
      ├─> gym.twenty-four.home → 10.233.66.8
      └─> strava.twenty-four.home → 10.233.66.7

MetalLB provides LoadBalancer IPs in the 10.233.64.x - 10.233.66.x range. Each service gets its own IP, exposed on port 80. No Ingress controller, no reverse proxy complexity. Just HTTP on the LAN.

Works great until I’m not on my LAN. Then it doesn’t work at all. That’s fine - these services are for me, at home.

Storage: Synology NFS

Persistent volumes backed by a Synology NAS (192.168.1.2) via the Synology CSI driver.

AWS S3, of course.

The Build System

Multi-arch is non-negotiable when you’re running arm64 + amd64. Docker Buildx with remote BuildKit instances in the cluster:

BuildKit AMD64 - 10.233.65.3:1234 (runs on lab)
BuildKit ARM64 - 10.233.66.9:1234 (runs on arm1/arm2/arm3)

Every make dbuild pushes to both architectures, creates a multi-arch manifest, and shoves it into the local registry at docker.registry.home:5000.

Standard Makefile pattern (used across all services):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


GITREV := $(shell git describe --tags 2>/dev/null || git rev-parse --short HEAD)
CONTAINER := service:$(GITREV)
REGISTRY := docker.registry.home:5000/

dbuild: Dockerfile
    docker buildx build --builder k8s-builder --platform linux/amd64 . \
        --build-arg VERSION=$(GITREV) --provenance=false \
        -t $(REGISTRY)service:$(GITREV)-amd64 --push
    docker buildx build --builder k8s-builder --platform linux/arm64 . \
        --build-arg VERSION=$(GITREV) --provenance=false \
        -t $(REGISTRY)service:$(GITREV)-arm64 --push
    docker manifest create $(REGISTRY)$(CONTAINER) --amend --insecure \
        $(REGISTRY)service:$(GITREV)-amd64 $(REGISTRY)service:$(GITREV)-arm64
    docker manifest push --insecure $(REGISTRY)$(CONTAINER)

deploy: dbuild
    kubectl apply -f service.yaml -n namespace
    kubectl set image deployment/service service=$(REGISTRY)$(CONTAINER) -n namespace

Git SHA as the version tag. Deploy updates the image and k8s pulls from the local registry. No Docker Hub rate limits, no external dependencies.

Deployment Pattern

Every service follows the same structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


apiVersion: apps/v1
kind: Deployment
metadata:
  name: service
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - arm64  # prefer ARM
      containers:
      - name: service
        image: docker.registry.home:5000/service:latest
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m"
          limits:
            memory: "256Mi"
            cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
  name: service
spec:
  type: LoadBalancer
  ports:
  - port: 80

Node affinity targets ARM where possible. LoadBalancer type gets a MetalLB IP automatically. Health checks at /health, version info at /version, logs at /logs.

It just makes it so much easier.