multi-cloud + mcp

docs/REFERENCE.md

@@ -9,6 +9,7 @@
 - [Kyverno Policies](#kyverno-policies)
 - [Configuration Reference](#configuration-reference)
 - [API Endpoints](#api-endpoints)
+- [Cloud Overlay Pattern](#cloud-overlay-pattern)
 - [Glossary](#glossary)
 
 ---
@@ -92,16 +93,34 @@ launchpad/
 │   ├── sealedsecrets.yaml
 │   ├── secrets.yaml
 │   ├── renovate.yaml
+│   ├── base/                      # ArgoCD Application manifests (Kustomize base)
+│   │   ├── gitea.yaml
+│   │   ├── opencost.yaml
+│   │   ├── traefik-application.yaml
+│   │   ├── keycloak.yaml
+│   │   ├── grafana.yaml
+│   │   └── ...
+│   ├── overlays/
+│   │   └── upc-prod/
+│   │       └── kustomization.yaml # Patches upc-dev → upc-prod valueFile paths
 │   └── values/
-│       ├── argocd-values.yaml
-│       ├── prometheus-values.yaml
-│       ├── grafana-values.yaml
-│       ├── loki-values.yaml
-│       ├── tempo-values.yaml
-│       ├── gitea-values.yaml
-│       ├── gitea-actions-values.yaml
-│       ├── fluent-bit-values.yaml
-│       └── renovate-values.yaml
+│       ├── base/                   # Cloud-agnostic Helm values
+│       │   ├── gitea-values.yaml
+│       │   ├── opencost-values.yaml
+│       │   ├── prometheus-values.yaml
+│       │   └── ...
+│       ├── upc-dev/               # UpCloud dev overlay values
+│       │   ├── traefik-values.yaml
+│       │   ├── keycloak-values.yaml
+│       │   ├── grafana-values.yaml
+│       │   ├── gitea-values.yaml
+│       │   └── opencost-values.yaml
+│       └── upc-prod/              # UpCloud prod overlay values
+│           ├── traefik-values.yaml
+│           ├── keycloak-values.yaml
+│           ├── grafana-values.yaml
+│           ├── gitea-values.yaml
+│           └── opencost-values.yaml
 │
 ├── apps/                          # Business applications
 │   ├── mcp10x.yaml
@@ -135,6 +154,15 @@ launchpad/
 │   ├── mcp10x-credentials-sealed.yaml
 │   └── musicman-credentials.yaml
 │
+├── scripts/                       # Operational helper scripts
+│   ├── gitea-backup.sh            # S3 backup helper (list/download)
+│   ├── gitea-restore.sh
+│   └── backup/                    # Per-cloud backup reference scripts
+│       ├── s3-minio.sh            # S3-compatible (UpCloud, MinIO, Wasabi)
+│       ├── aws-s3.sh              # Native AWS S3
+│       ├── azure-blob.sh          # Azure Blob Storage
+│       └── gcp-gcs.sh             # GCP Cloud Storage
+│
 ├── private/                       # Local-only (Git-ignored)
 │   ├── *.yaml
 │   └── *.sh
@@ -1621,6 +1649,79 @@ POST /loki/api/v1/push
 
 ---
 
+## Cloud Overlay Pattern
+
+### Overview
+
+Cloud-specific configuration (StorageClass, LoadBalancer annotations, pricing models, etc.) lives in per-cloud overlay value files, **not** in `base/`. This means adding a new cloud provider (AKS, EKS, GKE) only requires a new overlay directory — no base changes.
+
+### How It Works
+
+Each ArgoCD Application uses **multi-source Helm values** with two value files:
+
+```yaml
+# infra/base/gitea.yaml (example)
+helm:
+  valueFiles:
+  - $values/infra/values/base/gitea-values.yaml      # [0] cloud-agnostic
+  - $values/infra/values/upc-dev/gitea-values.yaml    # [1] cloud-specific (default: upc-dev)
+```
+
+The `upc-prod` Kustomize overlay patches index `[1]` to swap the cloud-specific file:
+
+```yaml
+# infra/overlays/upc-prod/kustomization.yaml
+- target:
+    kind: Application
+    name: gitea
+  patch: |
+    - op: replace
+      path: /spec/sources/0/helm/valueFiles/1
+      value: $values/infra/values/upc-prod/gitea-values.yaml
+```
+
+### Components Using Cloud Overlays
+
+| Component | Cloud-specific config | Overlay value file |
+|-----------|----------------------|-------------------|
+| **Traefik** | LB annotations, proxy protocol IPs | `traefik-values.yaml` |
+| **Keycloak** | Hostname, TLS settings | `keycloak-values.yaml` |
+| **Grafana** | Hostname, datasource URLs | `grafana-values.yaml` |
+| **Gitea** | StorageClass (persistence + PostgreSQL) | `gitea-values.yaml` |
+| **OpenCost** | Custom pricing model (CPU/RAM/storage rates) | `opencost-values.yaml` |
+
+### Backup CronJob
+
+The `gitea-backup` CronJob uses a generic `s3` alias for `minio/mc`. The actual endpoint and credentials come from the `gitea-backup-s3` Sealed Secret, which is per-cloud. Reference scripts for different cloud providers are in `scripts/backup/`:
+
+| Script | Provider | Tool |
+|--------|----------|------|
+| `s3-minio.sh` | S3-compatible (UpCloud, MinIO, Wasabi) | `minio/mc` |
+| `aws-s3.sh` | AWS S3 | `aws` CLI |
+| `azure-blob.sh` | Azure Blob Storage | `az` CLI |
+| `gcp-gcs.sh` | GCP Cloud Storage | `gsutil` |
+
+### Adding a New Cloud Provider
+
+To add support for a new cloud (e.g., `aks-dev`):
+
+1. **Create overlay value directory**: `infra/values/aks-dev/`
+2. **Add cloud-specific value files** for each component that needs one:
+   - `traefik-values.yaml` — LB annotations, proxy protocol config
+   - `keycloak-values.yaml` — hostname/TLS if different
+   - `grafana-values.yaml` — hostname/datasources if different
+   - `gitea-values.yaml` — `storageClass` for persistence + PostgreSQL
+   - `opencost-values.yaml` — `customPricing` cost model for your cloud
+3. **Create a Kustomize overlay** (if needed): `infra/overlays/aks-prod/kustomization.yaml`
+   - Patch each Application's `valueFiles[1]` to point to `aks-prod/` files
+4. **Create a root Application**: `_app-of-apps-aks-dev.yaml` pointing to the overlay
+5. **Create Sealed Secrets** for the new cluster:
+   - `secrets/aks-dev/` — TLS certs, credentials, backup S3 config
+6. **Update `gitea-backup-s3` secret** with the new cloud's S3-compatible endpoint
+7. **Bootstrap**: `kubectl apply -f _app-of-apps-aks-dev.yaml -n argocd`
+
+---
+
 ## Glossary
 
 ### Terms