From 94c7924e655e470f3a01ef493c41788d755ac42c Mon Sep 17 00:00:00 2001 From: Sten Date: Fri, 29 May 2026 10:31:09 +0200 Subject: [PATCH] docs(infra): pg backup & restore runbook for forte-drop Covers: nightly backup mechanism, listing backups, manual trigger, full restore procedure (2-pod mc-download + psql-pipe), verification, object-data note, and a disaster-scenario recovery table. --- .../upc-dev/forte-drop-postgresql/RESTORE.md | 143 ++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 infra/overlays/upc-dev/forte-drop-postgresql/RESTORE.md diff --git a/infra/overlays/upc-dev/forte-drop-postgresql/RESTORE.md b/infra/overlays/upc-dev/forte-drop-postgresql/RESTORE.md new file mode 100644 index 0000000..028bd6b --- /dev/null +++ b/infra/overlays/upc-dev/forte-drop-postgresql/RESTORE.md @@ -0,0 +1,143 @@ +# forte-drop Postgres — backup & restore runbook + +## What gets backed up + +A CronJob (`forte-drop-pg-backup`, namespace `forte-drop`) runs nightly at **02:00 UTC**: + +1. `pg_dump` of the `drops` database → gzip. +2. Upload to **UpCloud Managed Object Storage**: `s3://drops/_pgbackups/forte-drop-.sql.gz` + (the `_pgbackups/` prefix is collision-proof: app slugs match `/^[a-z0-9][a-z0-9-]{0,62}$/` + and can never start with `_`). +3. Retention: dumps older than **30 days** are pruned. + +S3 creds come from the `forte-drop-secrets` Secret (`S3_ENDPOINT` / `S3_KEY` / `S3_SECRET`). +Postgres creds from `forte-drop-pg-creds` (`pgusername` / `pgpassword`). + +> **Object storage is the durable tier.** App data + DB backups both live in UpCloud +> Managed Object Storage (replicated by UpCloud). The in-cluster Postgres PVC is the +> live working copy; the nightly dump is the recovery point. The PVC carries +> `Prune=false,Delete=false` so ArgoCD never deletes it. + +## Prerequisites + +```bash +export KUBECONFIG=~/Downloads/dev-fd-no-svg1_kubeconfig.yaml +# Confirm the namespace + DB pod are up: +kubectl -n forte-drop get pods -l app.kubernetes.io/name=postgresql +``` + +## List available backups + +```bash +# Run an ephemeral mc pod with the app's S3 creds: +kubectl -n forte-drop run mc-list --rm -it --restart=Never \ + --image=quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z \ + --overrides='{"spec":{"containers":[{"name":"mc","image":"quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z","command":["sh","-c","mc alias set obj \"$S3_ENDPOINT\" \"$S3_KEY\" \"$S3_SECRET\" >/dev/null && mc ls obj/drops/_pgbackups/"],"envFrom":[{"secretRef":{"name":"forte-drop-secrets"}}]}]}}' +``` + +## Manually trigger a backup (before risky changes) + +```bash +kubectl -n forte-drop create job --from=cronjob/forte-drop-pg-backup pg-backup-manual-$(date +%s) +# Watch: +kubectl -n forte-drop get jobs -l app.kubernetes.io/component=backup +kubectl -n forte-drop logs -l app.kubernetes.io/component=backup --tail=40 +``` + +## Restore a dump + +> **Destructive.** This overwrites the live `drops` database. Take a fresh manual +> backup first (above) and confirm with whoever owns the data before proceeding. + +### 1. Pick the dump to restore + +List backups (above), choose `forte-drop-.sql.gz`. + +### 2. Run a restore pod that pulls the dump and pipes it into Postgres + +```bash +DUMP="forte-drop-20260530T020000Z.sql.gz" # <-- set to the chosen file + +kubectl -n forte-drop run pg-restore --rm -it --restart=Never \ + --image=postgres:16-alpine \ + --overrides='{ + "spec": { + "containers": [{ + "name": "restore", + "image": "postgres:16-alpine", + "command": ["sh","-c","set -euo pipefail; \ + apk add --no-cache curl >/dev/null; \ + # download via mc is simpler — use a 2-step instead (see note). \ + echo placeholder"], + "envFrom": [ + {"secretRef":{"name":"forte-drop-pg-creds"}}, + {"secretRef":{"name":"forte-drop-secrets"}} + ] + }] + } + }' +``` + +**Simpler 2-pod approach (recommended — avoids cramming mc + psql in one image):** + +```bash +DUMP="forte-drop-20260530T020000Z.sql.gz" + +# (a) Download the dump from object storage to a local file: +kubectl -n forte-drop run mc-get --rm -it --restart=Never \ + --image=quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z \ + --overrides='{"spec":{"containers":[{"name":"mc","image":"quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z","command":["sh","-c","mc alias set obj \"$S3_ENDPOINT\" \"$S3_KEY\" \"$S3_SECRET\" >/dev/null && mc cat obj/drops/_pgbackups/'"$DUMP"'"],"envFrom":[{"secretRef":{"name":"forte-drop-secrets"}}]}]}}' \ + > /tmp/$DUMP + +# (b) Pipe it into the live Postgres via the service: +gunzip -c /tmp/$DUMP | kubectl -n forte-drop run pg-restore --rm -i --restart=Never \ + --image=postgres:16-alpine \ + --overrides='{"spec":{"containers":[{"name":"psql","image":"postgres:16-alpine","stdin":true,"command":["sh","-c","PGPASSWORD=\"$pgpassword\" psql -h forte-drop-postgresql.forte-drop.svc -U \"$pgusername\" -d drops"],"env":[{"name":"pgusername","valueFrom":{"secretKeyRef":{"name":"forte-drop-pg-creds","key":"pgusername"}}},{"name":"pgpassword","valueFrom":{"secretKeyRef":{"name":"forte-drop-pg-creds","key":"pgpassword"}}}]}]}}' +``` + +> The app's schema is created idempotently on boot (`CREATE TABLE IF NOT EXISTS` + +> `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` in `src/repo/pg.ts`), and `pg_dump` +> output includes the data. For a clean restore into a fresh DB this just works. +> To restore over an existing DB with conflicting rows, drop/recreate the `drops` +> database first (coordinate downtime — scale the web Deployment to 0 during the +> restore so the app isn't writing). + +### 3. Verify + +```bash +kubectl -n forte-drop run pg-check --rm -it --restart=Never \ + --image=postgres:16-alpine \ + --env="PGPASSWORD=$(kubectl -n forte-drop get secret forte-drop-pg-creds -o jsonpath='{.data.pgpassword}' | base64 -d)" \ + --command -- psql -h forte-drop-postgresql.forte-drop.svc -U drops -d drops \ + -c "SELECT count(*) AS drops FROM drops;" -c "SELECT count(*) AS view_hits FROM view_hits;" +``` + +### 4. Bring the app back + +```bash +# If you scaled web to 0 for the restore: +kubectl -n forte-drop scale deploy/forte-drop --replicas=2 +``` + +## Object data (uploaded drop files) + +Drop files live in `s3://drops//...` in the same managed bucket. They are +**not** part of the pg backup (the dump only holds metadata). Object storage is +UpCloud-managed/replicated, so no separate file backup is configured. If a +file-level backup is later required, mirror the bucket to a second bucket/region: + +```bash +mc mirror --overwrite obj/drops/ backup-target/drops-mirror/ +``` + +(Exclude `_pgbackups/` from the app-data mirror if you split them.) + +## Disaster scenarios + +| Scenario | Recovery | +|---|---| +| Postgres pod crash / reschedule | StatefulSet reattaches the PVC; ~1–2 min downtime; no data loss. | +| PVC lost / corrupted | Recreate StatefulSet, restore latest nightly dump (above). Data since last dump is lost. | +| Accidental `drops` table data loss | Restore latest dump; or `pg_restore` a single table from a dump. | +| Namespace deleted | PVC has `Prune=false,Delete=false`; recreate Applications, PVC re-binds, app recovers. Backups in object storage are independent. | +| Object storage bucket lost | UpCloud-managed (replicated). If the IAM key is rotated, update `forte-drop-secrets` (re-seal). |