Deploy: K3s (production scale)

This guide explains the supported deployment path for a Kumiko server app on a production K3s cluster. It is not a generic Kubernetes tutorial and does not cover static Astro sites.

At the end, the app has its own namespace, database and role, Redis, migration init container, Deployment, Service, Ingress, certificate, DNS records, per-app backup schedule, and deploy-scoped CI credentials.

Production operation

Complete this first rollout with another operator. Keep the previous immutable image tag available, test the rollback commands before sending real traffic, and never put credentials in source files or plain Pulumi config.

Before you start

Check every item before changing the cluster:

• The cluster platform and operator layer are deployed.
• You can reach the cluster through WireGuard.
• KUBECONFIG points to a deploy-capable kubeconfig.
• Your infrastructure CLI and Bun are installed.
• You can select the production environments for the cluster platform, operator layer, and application layer.
• The app image builds for every cluster architecture and can be pushed to GHCR.
• The app has bin/main.ts, bin/kumiko.ts, checked-in migrations, and a production Dockerfile.
• All production environment values are known and stored as secrets where appropriate.
• The deployment workflow will publish immutable sha-<short> image tags.
• You have read the rollback section and know the previous working SHA.

Set concrete names for the commands below:

export APP="YOUR_APP_NAME"
export DEPLOYMENT="YOUR_DEPLOYMENT_NAME"
export NAMESPACE="YOUR_NAMESPACE"
export IMAGE="ghcr.io/YOUR_ORG/YOUR_IMAGE"
export DOMAIN="YOUR_APP_DOMAIN"
export KUBECONFIG="$HOME/.kube/kumiko.yaml"

DEPLOYMENT and NAMESPACE are often the same, but not always. Show Pony, for example, uses the show-pony-app deployment and namespace with the show-pony image. Do not assume these names from the repository name.

Understand the deployment layers

A typical K3s environment has three infrastructure layers:

Layer	Responsibility
Cluster platform	Nodes, load balancer, K3s, and network access
Cluster operators	Ingress, certificates, Postgres operator, backups, and monitoring
Application layer	App namespace, database, Redis, ingress, DNS, and backups

Your infrastructure repository may implement these layers with Pulumi, Terraform, Helm, or another tool. Static Astro sites use a different application path from Kumiko server apps; do not use a static-site helper for an app that needs Bun, Postgres, Redis, migrations, or Kumiko health probes.

1. Prove the production image locally

Build the same Dockerfile CI will publish. If it consumes private packages, pass the repository’s documented BuildKit secret or build argument without printing its value:

docker build \
  --file deploy/Dockerfile \
  --tag "${APP}:local-smoke" \
  .

Start an isolated Postgres container:

docker network create "${APP}-smoke"
docker run --detach \
  --name "${APP}-smoke-db" \
  --network "${APP}-smoke" \
  --env POSTGRES_USER=kumiko \
  --env POSTGRES_PASSWORD=kumiko \
  --env POSTGRES_DB=kumiko \
  --health-cmd="pg_isready -U kumiko -d kumiko" \
  --health-interval=2s \
  --health-timeout=3s \
  --health-retries=30 \
  postgres:18.4-alpine

docker run --detach \
  --name "${APP}-smoke-redis" \
  --network "${APP}-smoke" \
  redis:7

Wait until it is healthy:

docker inspect \
  --format '{{.State.Health.Status}}' \
  "${APP}-smoke-db"

Expected output:

healthy

Apply the checked-in schema with the command used by the cluster init container:

docker run --rm \
  --network "${APP}-smoke" \
  --env DATABASE_URL="postgresql://kumiko:kumiko@${APP}-smoke-db:5432/kumiko" \
  "${APP}:local-smoke" \
  bun /app/kumiko.js schema apply

Start the app image with its remaining required environment variables:

export LOCAL_JWT_SECRET="$(openssl rand -base64 32)"
export LOCAL_MASTER_KEY="$(openssl rand -base64 32)"

# Add every other required variable from this app's env schema before boot.
# A production-mode app with PII fields also needs a valid subject-key KMS trio.

docker run --detach \
  --name "${APP}-smoke-app" \
  --network "${APP}-smoke" \
  --publish 13000:3000 \
  --env PORT=3000 \
  --env DATABASE_URL="postgresql://kumiko:kumiko@${APP}-smoke-db:5432/kumiko" \
  --env REDIS_URL="redis://${APP}-smoke-redis:6379" \
  --env JWT_SECRET="$LOCAL_JWT_SECRET" \
  --env KUMIKO_SECRETS_MASTER_KEY_V1="$LOCAL_MASTER_KEY" \
  "${APP}:local-smoke"

curl --fail --silent --show-error http://127.0.0.1:13000/health
curl --fail --silent --show-error http://127.0.0.1:13000/health/ready

Both requests must return success before you continue. Use throwaway local secrets here, not production values.

Clean up after the smoke test. Because command-line environment values may be visible in local process inspection, use only throwaway keys on an isolated developer machine:

docker rm --force "${APP}-smoke-app" "${APP}-smoke-db"
docker rm --force "${APP}-smoke-redis"
docker network rm "${APP}-smoke"
unset LOCAL_JWT_SECRET LOCAL_MASTER_KEY

If this step fails

• /app/kumiko.js missing: fix the Docker build so bin/kumiko.ts is bundled.
• schema apply fails: fix migrations or database permissions before touching K3s.
• /health fails: inspect docker logs "${APP}-smoke-app" for missing native or runtime dependencies.
• /health/ready fails: confirm database and Redis requirements for this app.

2. Add the app to the infrastructure application layer

Open your infrastructure repository’s app-stack configuration. Follow an existing Kumiko app definition with the same domain shape and capabilities. The abstract shape is:

const myApp = defineKumikoAppDeployment({
  name: "YOUR_DEPLOYMENT_NAME",
  image: "ghcr.io/YOUR_ORG/YOUR_IMAGE:latest",
  ghcrPull: { username: ghcrUsername, token: ghcrPullToken },
  baseDomain: "YOUR_APP_DOMAIN",
  // Zone name, zone id and the hcloud provider that manages its records.
  hetznerDnsZone: requireHetznerZone(hetznerZones, "YOUR_ZONE"),
  ingressIp,
  demoAdminEmail: config.requireSecret("yourAppAdminEmail"),
  demoAdminPassword: config.requireSecret("yourAppAdminPassword"),
  demoSysadminEmail: config.requireSecret("yourAppSysadminEmail"),
  demoSysadminPassword: config.requireSecret("yourAppSysadminPassword"),
  k8sProvider,
});

Use the equivalent options in your deployment tool for subject-key KMS, SMTP, S3 storage, metrics, Meilisearch, replicas, or wildcard domains. Do not copy optional settings from another app unless this app needs them.

The application layer should provision an isolated namespace, a dedicated database role, Redis, app secrets, the migration init container, Deployment, Service, Ingress, TLS, DNS, resource limits, rollout RBAC, and a tested backup schedule.

Trust the ingress for client IPs

Set KUMIKO_TRUSTED_PROXY_HOPS=1 on the app container when one ingress controller (for example ingress-nginx) sits in front of it. If more proxies append to X-Forwarded-For before the ingress, count each of them. Without the variable the app sees the ingress as the client, and all users share one rate-limit bucket. Boot logs a warning in that case. Details are in Deploy: Docker.

Set secrets safely

Use secret config for credentials:

# Use the secret-management command provided by your infrastructure repository.
# Example for a Pulumi-based stack:
pulumi stack select prod
pulumi config set --secret yourAppAdminEmail "YOUR_ADMIN_EMAIL"
pulumi config set --secret yourAppAdminPassword "YOUR_STRONG_ADMIN_PASSWORD"
pulumi config set --secret yourAppSysadminEmail "YOUR_SYSADMIN_EMAIL"
pulumi config set --secret yourAppSysadminPassword "YOUR_STRONG_SYSADMIN_PASSWORD"

Use the app’s existing Pulumi config conventions for every additional value. Do not paste secrets into index.ts, commit them to Git, or print them in CI.

What should happen

Nothing changes in the cluster until the infrastructure apply command. For a Pulumi-based stack, typecheck and preview first:

bunx tsc --noEmit
pulumi preview

Review the preview. A new app should create resources in its own namespace. It must not replace another app’s Database, Secret, Ingress, or DNS record.

Stop on destructive changes

If preview proposes deletion or replacement of an existing database, namespace, certificate, or key, stop. Fix resource names or add the required Pulumi alias. Do not approve the preview to “see what happens.”

3. Apply infrastructure

After another operator has reviewed the preview:

pulumi up

Expected result: Pulumi completes without errors and reports the app URL. The first app pod may still wait for an image if CI has not published it yet.

Check the resources:

kubectl get namespace "$NAMESPACE"
kubectl -n cnpg-system get database
kubectl -n "$NAMESPACE" get deployment,service,ingress,certificate
kubectl -n "$NAMESPACE" get pods

The Deployment should contain migrate-db and the app container:

kubectl -n "$NAMESPACE" get deployment "$DEPLOYMENT" \
  -o jsonpath='{.spec.template.spec.initContainers[*].name}{"\n"}{.spec.template.spec.containers[*].name}{"\n"}'

Expected shape:

fetch-db-creds migrate-db
YOUR_DEPLOYMENT_NAME

Some apps have additional init containers. migrate-db must still exist.

4. Enable immutable CI rollouts

The app workflow should:

1. test the app,
2. build and push latest plus sha-<short>, and
3. call the deployment workflow that owns your cluster with image-name.

The essential deploy call is:

jobs:
  deploy:
    needs: build
    uses: YOUR_ORG/YOUR_DEPLOYMENT_REPO/.github/workflows/YOUR_DEPLOY_WORKFLOW.yml@main
    with:
      namespace: YOUR_NAMESPACE
      deployment: YOUR_DEPLOYMENT_NAME
      image-name: YOUR_IMAGE_NAME
    secrets: inherit

image-name is required for a real rollback. The workflow pins both the app container and migrate-db to the same immutable sha-<short> image. Without it, an annotation-only rollout of :latest cannot reliably return to an older image.

If the application repository cannot call the deployment workflow directly, use the repository-dispatch or deployment-bridge mechanism approved by your operations team. Keep the app-to-deployment mapping reviewed and explicit.

What should happen

After a successful main-branch build, inspect the live images:

kubectl -n "$NAMESPACE" get deployment "$DEPLOYMENT" \
  -o jsonpath='{.spec.template.spec.initContainers[?(@.name=="migrate-db")].image}{"\n"}{.spec.template.spec.containers[0].image}{"\n"}'

Both lines must end in the same :sha-<short> tag.

5. Verify the rollout

Wait for Kubernetes:

kubectl -n "$NAMESPACE" rollout status \
  "deployment/${DEPLOYMENT}" \
  --timeout=300s

Check pod state and logs:

kubectl -n "$NAMESPACE" get pods -o wide
kubectl -n "$NAMESPACE" logs \
  "deployment/${DEPLOYMENT}" \
  --container migrate-db \
  --tail=100
kubectl -n "$NAMESPACE" logs \
  "deployment/${DEPLOYMENT}" \
  --container "$DEPLOYMENT" \
  --tail=100

Verify readiness inside the cluster path and then the public route:

kubectl -n "$NAMESPACE" port-forward \
  "service/${DEPLOYMENT}" 18080:80

In another terminal:

curl --fail --silent --show-error http://127.0.0.1:18080/health
curl --fail --silent --show-error http://127.0.0.1:18080/health/ready
curl --fail --silent --show-error "https://${DOMAIN}/health"

Finally verify one real application read and one authenticated write. A green health endpoint proves infrastructure is reachable; it does not prove that the correct encryption keys, tenant routing, or old data are usable.

6. Verify backups

The application layer creates a per-app Velero schedule at 03:30 UTC with a seven-day TTL. PostgreSQL base backups and WAL archiving are handled separately by CloudNativePG and the barman-cloud plugin for point-in-time recovery.

Check the schedule and the latest backup:

kubectl -n velero get schedule "${DEPLOYMENT}-daily"
kubectl -n velero get backups.velero.io \
  --sort-by=.metadata.creationTimestamp

Do not treat resource existence as a restore test. Follow the internal restore runbook for the regular side-by-side recovery drill.

Roll back

Before every production rollout, record the current image and revision:

kubectl -n "$NAMESPACE" get deployment "$DEPLOYMENT" \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n "$NAMESPACE" rollout history "deployment/${DEPLOYMENT}"

A live image ending in :latest is not safely rollback-capable. Fix CI image pinning before relying on these commands.

Roll back one revision

kubectl -n "$NAMESPACE" rollout undo "deployment/${DEPLOYMENT}"
kubectl -n "$NAMESPACE" rollout status \
  "deployment/${DEPLOYMENT}" \
  --timeout=300s

Roll back to a known SHA

export OLD_SHA="YOUR_PREVIOUS_SHORT_SHA"
export OLD_IMAGE="${IMAGE}:sha-${OLD_SHA}"

kubectl -n "$NAMESPACE" set image \
  "deployment/${DEPLOYMENT}" \
  "${DEPLOYMENT}=${OLD_IMAGE}" \
  "migrate-db=${OLD_IMAGE}"

kubectl -n "$NAMESPACE" rollout status \
  "deployment/${DEPLOYMENT}" \
  --timeout=300s

Pin both containers. Running old app code with a different migration image is not a valid rollback.

Database compatibility

Kubernetes image rollback does not reverse an applied database migration. Use expand/contract migrations so the previous app version remains compatible. If a migration is destructive or not backward-compatible, stop and follow a reviewed database recovery plan instead of running rollout undo blindly.

After rollback, repeat the health checks and the same real read/write probe used for deployment verification.

Common problems

ImagePullBackOff

kubectl -n "$NAMESPACE" describe pod -l "app=${DEPLOYMENT}"

Check the GHCR image name, immutable SHA tag, pull secret, and image architecture.

Init:CrashLoopBackOff

Find the pod and read the migration log:

POD=$(kubectl -n "$NAMESPACE" get pod -l "app=${DEPLOYMENT}" \
  -o jsonpath='{.items[0].metadata.name}')
kubectl -n "$NAMESPACE" logs "$POD" --container migrate-db --previous

Typical causes are missing /app/kumiko.js, migration errors, database role permissions, or missing encryption environment for projection rebuilds.

CrashLoopBackOff

kubectl -n "$NAMESPACE" logs \
  "deployment/${DEPLOYMENT}" \
  --container "$DEPLOYMENT" \
  --previous

Compare the error with the app’s env schema. Fix the secret or image; never add a fallback production secret in code.

Rollout times out while old pods still serve

kubectl -n "$NAMESPACE" describe deployment "$DEPLOYMENT"
kubectl -n "$NAMESPACE" get events --sort-by=.lastTimestamp | tail -30

Check readiness failures, ResourceQuota, topology spread, and node capacity. Do not delete healthy old pods until the replacement is understood.

TLS or DNS is not ready

kubectl -n "$NAMESPACE" describe certificate
kubectl -n "$NAMESPACE" describe ingress "$DEPLOYMENT"
dig "$DOMAIN" @1.1.1.1

Confirm the Hetzner DNS zone, DNS record, ClusterIssuer, and ingress IP.

See also

• Docker deployment, build and runtime image details.
• Deployment topology, API and worker lane layout.
• Load encryption keys from a key manager, production KEK custody.
• Rotate encryption keys, planned key changes after deployment.