infra/overlays/upc-dev/forte-drop-postgresql/RESTORE.md

@@ -0,0 +1,143 @@
+# forte-drop Postgres — backup & restore runbook
+
+## What gets backed up
+
+A CronJob (`forte-drop-pg-backup`, namespace `forte-drop`) runs nightly at **02:00 UTC**:
+
+1. `pg_dump` of the `drops` database → gzip.
+2. Upload to **UpCloud Managed Object Storage**: `s3://drops/_pgbackups/forte-drop-<TS>.sql.gz`
+   (the `_pgbackups/` prefix is collision-proof: app slugs match `/^[a-z0-9][a-z0-9-]{0,62}$/`
+   and can never start with `_`).
+3. Retention: dumps older than **30 days** are pruned.
+
+S3 creds come from the `forte-drop-secrets` Secret (`S3_ENDPOINT` / `S3_KEY` / `S3_SECRET`).
+Postgres creds from `forte-drop-pg-creds` (`pgusername` / `pgpassword`).
+
+> **Object storage is the durable tier.** App data + DB backups both live in UpCloud
+> Managed Object Storage (replicated by UpCloud). The in-cluster Postgres PVC is the
+> live working copy; the nightly dump is the recovery point. The PVC carries
+> `Prune=false,Delete=false` so ArgoCD never deletes it.
+
+## Prerequisites
+
+```bash
+export KUBECONFIG=~/Downloads/dev-fd-no-svg1_kubeconfig.yaml
+# Confirm the namespace + DB pod are up:
+kubectl -n forte-drop get pods -l app.kubernetes.io/name=postgresql
+```
+
+## List available backups
+
+```bash
+# Run an ephemeral mc pod with the app's S3 creds:
+kubectl -n forte-drop run mc-list --rm -it --restart=Never \
+  --image=quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z \
+  --overrides='{"spec":{"containers":[{"name":"mc","image":"quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z","command":["sh","-c","mc alias set obj \"$S3_ENDPOINT\" \"$S3_KEY\" \"$S3_SECRET\" >/dev/null && mc ls obj/drops/_pgbackups/"],"envFrom":[{"secretRef":{"name":"forte-drop-secrets"}}]}]}}'
+```
+
+## Manually trigger a backup (before risky changes)
+
+```bash
+kubectl -n forte-drop create job --from=cronjob/forte-drop-pg-backup pg-backup-manual-$(date +%s)
+# Watch:
+kubectl -n forte-drop get jobs -l app.kubernetes.io/component=backup
+kubectl -n forte-drop logs -l app.kubernetes.io/component=backup --tail=40
+```
+
+## Restore a dump
+
+> **Destructive.** This overwrites the live `drops` database. Take a fresh manual
+> backup first (above) and confirm with whoever owns the data before proceeding.
+
+### 1. Pick the dump to restore
+
+List backups (above), choose `forte-drop-<TS>.sql.gz`.
+
+### 2. Run a restore pod that pulls the dump and pipes it into Postgres
+
+```bash
+DUMP="forte-drop-20260530T020000Z.sql.gz"   # <-- set to the chosen file
+
+kubectl -n forte-drop run pg-restore --rm -it --restart=Never \
+  --image=postgres:16-alpine \
+  --overrides='{
+    "spec": {
+      "containers": [{
+        "name": "restore",
+        "image": "postgres:16-alpine",
+        "command": ["sh","-c","set -euo pipefail; \
+          apk add --no-cache curl >/dev/null; \
+          # download via mc is simpler — use a 2-step instead (see note). \
+          echo placeholder"],
+        "envFrom": [
+          {"secretRef":{"name":"forte-drop-pg-creds"}},
+          {"secretRef":{"name":"forte-drop-secrets"}}
+        ]
+      }]
+    }
+  }'
+```
+
+**Simpler 2-pod approach (recommended — avoids cramming mc + psql in one image):**
+
+```bash
+DUMP="forte-drop-20260530T020000Z.sql.gz"
+
+# (a) Download the dump from object storage to a local file:
+kubectl -n forte-drop run mc-get --rm -it --restart=Never \
+  --image=quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z \
+  --overrides='{"spec":{"containers":[{"name":"mc","image":"quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z","command":["sh","-c","mc alias set obj \"$S3_ENDPOINT\" \"$S3_KEY\" \"$S3_SECRET\" >/dev/null && mc cat obj/drops/_pgbackups/'"$DUMP"'"],"envFrom":[{"secretRef":{"name":"forte-drop-secrets"}}]}]}}' \
+  > /tmp/$DUMP
+
+# (b) Pipe it into the live Postgres via the service:
+gunzip -c /tmp/$DUMP | kubectl -n forte-drop run pg-restore --rm -i --restart=Never \
+  --image=postgres:16-alpine \
+  --overrides='{"spec":{"containers":[{"name":"psql","image":"postgres:16-alpine","stdin":true,"command":["sh","-c","PGPASSWORD=\"$pgpassword\" psql -h forte-drop-postgresql.forte-drop.svc -U \"$pgusername\" -d drops"],"env":[{"name":"pgusername","valueFrom":{"secretKeyRef":{"name":"forte-drop-pg-creds","key":"pgusername"}}},{"name":"pgpassword","valueFrom":{"secretKeyRef":{"name":"forte-drop-pg-creds","key":"pgpassword"}}}]}]}}'
+```
+
+> The app's schema is created idempotently on boot (`CREATE TABLE IF NOT EXISTS` +
+> `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` in `src/repo/pg.ts`), and `pg_dump`
+> output includes the data. For a clean restore into a fresh DB this just works.
+> To restore over an existing DB with conflicting rows, drop/recreate the `drops`
+> database first (coordinate downtime — scale the web Deployment to 0 during the
+> restore so the app isn't writing).
+
+### 3. Verify
+
+```bash
+kubectl -n forte-drop run pg-check --rm -it --restart=Never \
+  --image=postgres:16-alpine \
+  --env="PGPASSWORD=$(kubectl -n forte-drop get secret forte-drop-pg-creds -o jsonpath='{.data.pgpassword}' | base64 -d)" \
+  --command -- psql -h forte-drop-postgresql.forte-drop.svc -U drops -d drops \
+  -c "SELECT count(*) AS drops FROM drops;" -c "SELECT count(*) AS view_hits FROM view_hits;"
+```
+
+### 4. Bring the app back
+
+```bash
+# If you scaled web to 0 for the restore:
+kubectl -n forte-drop scale deploy/forte-drop --replicas=2
+```
+
+## Object data (uploaded drop files)
+
+Drop files live in `s3://drops/<slug>/...` in the same managed bucket. They are
+**not** part of the pg backup (the dump only holds metadata). Object storage is
+UpCloud-managed/replicated, so no separate file backup is configured. If a
+file-level backup is later required, mirror the bucket to a second bucket/region:
+
+```bash
+mc mirror --overwrite obj/drops/ backup-target/drops-mirror/
+```
+
+(Exclude `_pgbackups/` from the app-data mirror if you split them.)
+
+## Disaster scenarios
+
+| Scenario | Recovery |
+|---|---|
+| Postgres pod crash / reschedule | StatefulSet reattaches the PVC; ~1–2 min downtime; no data loss. |
+| PVC lost / corrupted | Recreate StatefulSet, restore latest nightly dump (above). Data since last dump is lost. |
+| Accidental `drops` table data loss | Restore latest dump; or `pg_restore` a single table from a dump. |
+| Namespace deleted | PVC has `Prune=false,Delete=false`; recreate Applications, PVC re-binds, app recovers. Backups in object storage are independent. |
+| Object storage bucket lost | UpCloud-managed (replicated). If the IAM key is rotated, update `forte-drop-secrets` (re-seal). |