launchpad/apps/overlays/upc-dev/forte-drop-postgresql/RESTORE.md

forte-drop Postgres — backup & restore runbook

What gets backed up

A CronJob (forte-drop-pg-backup, namespace forte-drop) runs nightly at 02:00 UTC:

1. pg_dump of the drops database → gzip.
2. Upload to UpCloud Managed Object Storage: s3://drops/_pgbackups/forte-drop-<TS>.sql.gz (the _pgbackups/ prefix is collision-proof: app slugs match /^[a-z0-9][a-z0-9-]{0,62}$/ and can never start with _).
3. Retention: dumps older than 30 days are pruned.

S3 creds come from the forte-drop-secrets Secret (S3_ENDPOINT / S3_KEY / S3_SECRET). Postgres creds from forte-drop-pg-creds (pgusername / pgpassword).

Object storage is the durable tier. App data + DB backups both live in UpCloud Managed Object Storage (replicated by UpCloud). The in-cluster Postgres PVC is the live working copy; the nightly dump is the recovery point. The PVC carries Prune=false,Delete=false so ArgoCD never deletes it.

Prerequisites

export KUBECONFIG=~/Downloads/dev-fd-no-svg1_kubeconfig.yaml
# Confirm the namespace + DB pod are up:
kubectl -n forte-drop get pods -l app.kubernetes.io/name=postgresql

List available backups

# Run an ephemeral mc pod with the app's S3 creds:
kubectl -n forte-drop run mc-list --rm -it --restart=Never \
  --image=quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z \
  --overrides='{"spec":{"containers":[{"name":"mc","image":"quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z","command":["sh","-c","mc alias set obj \"$S3_ENDPOINT\" \"$S3_KEY\" \"$S3_SECRET\" >/dev/null && mc ls obj/drops/_pgbackups/"],"envFrom":[{"secretRef":{"name":"forte-drop-secrets"}}]}]}}'

Manually trigger a backup (before risky changes)

kubectl -n forte-drop create job --from=cronjob/forte-drop-pg-backup pg-backup-manual-$(date +%s)
# Watch:
kubectl -n forte-drop get jobs -l app.kubernetes.io/component=backup
kubectl -n forte-drop logs -l app.kubernetes.io/component=backup --tail=40

Restore a dump

Destructive. This overwrites the live drops database. Take a fresh manual backup first (above) and confirm with whoever owns the data before proceeding.

1. Pick the dump to restore

List backups (above), choose forte-drop-<TS>.sql.gz.

2. Run a restore pod that pulls the dump and pipes it into Postgres

DUMP="forte-drop-20260530T020000Z.sql.gz"   # <-- set to the chosen file

kubectl -n forte-drop run pg-restore --rm -it --restart=Never \
  --image=postgres:16-alpine \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "restore",
        "image": "postgres:16-alpine",
        "command": ["sh","-c","set -euo pipefail; \
          apk add --no-cache curl >/dev/null; \
          # download via mc is simpler — use a 2-step instead (see note). \
          echo placeholder"],
        "envFrom": [
          {"secretRef":{"name":"forte-drop-pg-creds"}},
          {"secretRef":{"name":"forte-drop-secrets"}}
        ]
      }]
    }
  }'

Simpler 2-pod approach (recommended — avoids cramming mc + psql in one image):

DUMP="forte-drop-20260530T020000Z.sql.gz"

# (a) Download the dump from object storage to a local file:
kubectl -n forte-drop run mc-get --rm -it --restart=Never \
  --image=quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z \
  --overrides='{"spec":{"containers":[{"name":"mc","image":"quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z","command":["sh","-c","mc alias set obj \"$S3_ENDPOINT\" \"$S3_KEY\" \"$S3_SECRET\" >/dev/null && mc cat obj/drops/_pgbackups/'"$DUMP"'"],"envFrom":[{"secretRef":{"name":"forte-drop-secrets"}}]}]}}' \
  > /tmp/$DUMP

# (b) Pipe it into the live Postgres via the service:
gunzip -c /tmp/$DUMP | kubectl -n forte-drop run pg-restore --rm -i --restart=Never \
  --image=postgres:16-alpine \
  --overrides='{"spec":{"containers":[{"name":"psql","image":"postgres:16-alpine","stdin":true,"command":["sh","-c","PGPASSWORD=\"$pgpassword\" psql -h forte-drop-postgresql.forte-drop.svc -U \"$pgusername\" -d drops"],"env":[{"name":"pgusername","valueFrom":{"secretKeyRef":{"name":"forte-drop-pg-creds","key":"pgusername"}}},{"name":"pgpassword","valueFrom":{"secretKeyRef":{"name":"forte-drop-pg-creds","key":"pgpassword"}}}]}]}}'

The app's schema is created idempotently on boot (CREATE TABLE IF NOT EXISTS + ALTER TABLE ... ADD COLUMN IF NOT EXISTS in src/repo/pg.ts), and pg_dump output includes the data. For a clean restore into a fresh DB this just works. To restore over an existing DB with conflicting rows, drop/recreate the drops database first (coordinate downtime — scale the web Deployment to 0 during the restore so the app isn't writing).

3. Verify

kubectl -n forte-drop run pg-check --rm -it --restart=Never \
  --image=postgres:16-alpine \
  --env="PGPASSWORD=$(kubectl -n forte-drop get secret forte-drop-pg-creds -o jsonpath='{.data.pgpassword}' | base64 -d)" \
  --command -- psql -h forte-drop-postgresql.forte-drop.svc -U drops -d drops \
  -c "SELECT count(*) AS drops FROM drops;" -c "SELECT count(*) AS view_hits FROM view_hits;"

4. Bring the app back

# If you scaled web to 0 for the restore:
kubectl -n forte-drop scale deploy/forte-drop --replicas=2

Object data (uploaded drop files)

Drop files live in s3://drops/<slug>/... in the same managed bucket. They are not part of the pg backup (the dump only holds metadata). Object storage is UpCloud-managed/replicated, so no separate file backup is configured. If a file-level backup is later required, mirror the bucket to a second bucket/region:

mc mirror --overwrite obj/drops/ backup-target/drops-mirror/

(Exclude _pgbackups/ from the app-data mirror if you split them.)

Disaster scenarios

Scenario	Recovery
Postgres pod crash / reschedule	StatefulSet reattaches the PVC; ~1–2 min downtime; no data loss.
PVC lost / corrupted	Recreate StatefulSet, restore latest nightly dump (above). Data since last dump is lost.
Accidental drops table data loss	Restore latest dump; or pg_restore a single table from a dump.
Namespace deleted	PVC has Prune=false,Delete=false; recreate Applications, PVC re-binds, app recovers. Backups in object storage are independent.
Object storage bucket lost	UpCloud-managed (replicated). If the IAM key is rotated, update forte-drop-secrets (re-seal).