9 Commits

Author SHA1 Message Date
ecbb1f8638 pw 2026-04-23 23:00:44 +02:00
424be7ec7e allow login and sync 2026-04-23 22:49:53 +02:00
5afdf00964 session 2026-04-23 21:54:24 +02:00
2781c96d43 tls 2026-04-23 21:50:38 +02:00
a456a11460 db 2026-04-23 21:40:55 +02:00
b5e442d92b policy 2026-04-23 21:25:11 +02:00
2d756295bf backstage resources 2026-04-23 20:52:38 +02:00
026bcb2b31 feature/backstage (#13)
Reviewed-on: #13
Reviewed-by: gitea_admin <admin@forteapps.net>
Co-authored-by: Danijel Simeunovic <danijel.simeunovic@fortedigital.com>
Co-committed-by: Danijel Simeunovic <danijel.simeunovic@fortedigital.com>
2026-04-23 18:45:57 +00:00
aa6775bed2 ns 2026-04-23 14:52:27 +02:00
203 changed files with 2583 additions and 3854 deletions

2
.gitattributes vendored
View File

@@ -1,2 +0,0 @@
# Force LF line endings for shell scripts
*.sh text eol=lf

View File

@@ -1,9 +1,9 @@
# Kubernetes Cluster - GitOps Configuration
> **Kubernetes cluster bootstrapping and GitOps configuration repository** using ArgoCD for multi-cloud Kubernetes (UpCloud, AWS EKS, Azure AKS, GCP GKE)
> **Kubernetes cluster bootstrapping and GitOps configuration repository** using ArgoCD for UpCloud Managed Kubernetes
[![GitOps](https://img.shields.io/badge/GitOps-ArgoCD-blue)](https://argoproj.github.io/cd/)
[![Kubernetes](https://img.shields.io/badge/Kubernetes-Multi--Cloud-orange)]()
[![Kubernetes](https://img.shields.io/badge/Kubernetes-UpCloud-orange)](https://upcloud.com/)
---
@@ -57,7 +57,7 @@ This repository contains the complete GitOps configuration for our Kubernetes cl
### What's Inside
- **Infrastructure Applications**: Traefik, Cert-Manager, Kyverno, Prometheus, Grafana, Loki, Tempo, Sealed Secrets, Homepage (platform dashboard)
- **Infrastructure Applications**: Traefik, Cert-Manager, Kyverno, Prometheus, Grafana, Loki, Tempo, Sealed Secrets
- **Business Applications**: MCP10X, MusicMan, Dot-AI Stack, ArgoCD MCP
- **Policies**: Kyverno security policies for secret management, namespace controls, pod verification
- **Monitoring**: Full observability stack with metrics, logs, traces, and alerting
@@ -84,51 +84,31 @@ This repository contains the complete GitOps configuration for our Kubernetes cl
├── _app-of-apps.yaml # Root ArgoCD Application (App-of-Apps pattern)
├── infra/ # Infrastructure ArgoCD Applications (Kustomize multi-cluster)
│ ├── base/ # Base ArgoCD Application manifests (one dir per component)
│ │ ├── kustomization.yaml # Aggregates all component subdirectories
│ │ ├── traefik-application/
│ │ ├── kustomization.yaml
│ │ │ └── traefik-application.yaml
│ │ ├── keycloak/
│ │ │ ├── kustomization.yaml
│ │ │ └── keycloak.yaml
│ │ ├── grafana/
│ │ ├── prometheus/
│ │ ── ... # Each component in its own subdirectory
│ └── secrets/
│ ├── overlays/ # Per-cluster overrides (Kustomize)
│ │ ── upc-dev/ # UpCloud Dev — includes all base components
│ │ ├── upc-prod/ # UpCloud Prod — all components + patches
│ │ ├── aks-dev/ # Azure AKS Dev — selective components only
│ │ ├── aks-prod/ # Azure AKS Prod
│ │ ├── eks-dev/ # AWS EKS Dev
│ │ ├── eks-prod/ # AWS EKS Prod
│ │ ├── gke-dev/ # GCP GKE Dev
│ │ └── gke-prod/ # GCP GKE Prod
│ ├── base/ # Base ArgoCD Application manifests (EU defaults)
│ │ ├── kustomization.yaml
│ │ ├── traefik-application.yaml
│ │ ├── keycloak.yaml
│ │ ── grafana.yaml
│ │ ├── gitea.yaml
│ │ ├── gitea-actions.yaml
│ │ ├── tempo.yaml
│ │ ├── renovate.yaml
│ │ ├── ... # All other Application manifests
│ │ ── secrets.yaml
├── overlays/ # Per-cluster overrides
│ ├── upc-dev/ # UpCloud Dev cluster (uses base as-is)
│ │ ── upc-prod/ # UpCloud Prod cluster (patches value paths)
│ ├── dashboards/ # Grafana dashboard ConfigMaps
│ └── values/ # Helm value overrides
│ ├── base/ # Shared cloud-agnostic values
│ ├── upc-dev/ # UpCloud Dev (storage, LB, pricing)
── upc-prod/ # UpCloud Prod
│ ├── eks-dev/ # AWS EKS Dev
│ ├── eks-prod/ # AWS EKS Prod
│ ├── aks-dev/ # Azure AKS Dev
│ ├── aks-prod/ # Azure AKS Prod
│ ├── gke-dev/ # GCP GKE Dev
│ └── gke-prod/ # GCP GKE Prod
│ ├── base/ # Shared values (all clusters)
│ ├── upc-dev/ # UpCloud Dev-specific values
── upc-prod/ # UpCloud Prod-specific values
├── apps/ # Business Applications (Kustomize, same pattern as infra)
│ ├── base/ # One subdirectory per app
│ ├── kustomization.yaml
│ ├── musicman/
│ ├── mcp10x/
│ │ ├── dot-ai-stack/
│ │ ├── ts-mcp/
│ │ └── argo-mcp/
│ └── overlays/ # Per-cluster: cherry-pick or include all
│ ├── upc-dev/ # All apps
│ ├── upc-prod/ # All apps + patches
│ └── aks-dev/ # Selective apps only
├── apps/ # Business Applications
│ ├── mcp10x.yaml
│ ├── musicman.yaml
│ ├── dot-ai-stack.yaml
└── argo-mcp.yaml
├── cluster-resources/ # Cluster-wide Kubernetes resources
│ ├── letsencrypt-issuer.yaml
@@ -363,6 +343,7 @@ kubectl patch application myapp -n argocd \
| **Fluent-Bit** | Log shipping | `monitoring` | DaemonSet |
| **OpenCost** | Cost monitoring | `monitoring` | 1 |
| **Renovate** | Dependency updates | `renovate` | CronJob |
| **Trivy** | Vulnerability scanning | `trivy-system` | 1 |
**Full specs**: [Technical Reference - Infrastructure Components](docs/REFERENCE.md#infrastructure-components)
@@ -380,7 +361,7 @@ kubectl patch application myapp -n argocd \
## 📖 Key Concepts
### App-of-Apps Pattern
`_app-of-apps-{cluster}.yaml` is the root Application that manages all other Applications in `infra/`. Each component in `infra/base/` lives in its own subdirectory (e.g., `infra/base/grafana/`). Overlays can either include **all** components (via `../../base`) or **cherry-pick** specific ones (via `../../base/grafana`, `../../base/prometheus`, etc.). Per-cluster patches swap Helm value file paths. Supported clusters: `upc-dev`, `upc-prod`, `eks-dev`, `eks-prod`, `aks-dev`, `aks-prod`, `gke-dev`, `gke-prod`.
`_app-of-apps.yaml` is the root Application that manages all other Applications in `infra/`. Kustomize overlays in `infra/overlays/{upc-dev,upc-prod}/` render the base Applications with per-cluster patches (e.g., swapping value file paths from `upc-dev` to `upc-prod`).
### Multi-Source Pattern
Applications reference both:
@@ -477,14 +458,16 @@ Documentation lives in `docs/`. To update:
## 📝 Notes
### Current Environment
- **Provider**: Multi-cloud (UpCloud, AWS EKS, Azure AKS, GCP GKE)
- **Active clusters**: UpCloud (upc-dev, upc-prod)
- **Provider**: UpCloud Managed Kubernetes
- **Environment**: Production (internal use only)
- **Clusters**: Multi-cluster (upc-dev, upc-prod) via Kustomize overlays
- **Auth**: Disabled for ArgoCD (internal access)
- **Backup**: Gitea daily backup to S3-compatible storage
- **Backup**: None (cluster rebuildable via GitOps)
### Known Limitations
- No automated backups (yet)
- Secret rotation not automated
- Multi-cluster limited to upc-dev and upc-prod environments
- DNS management is manual
**Future improvements**: See [Operations Runbook - Disaster Recovery](docs/OPERATIONS-RUNBOOK.md#disaster-recovery)
@@ -521,7 +504,7 @@ Internal use only. Not for public distribution.
---
**Last Updated**: 2026-04-22
**Last Updated**: 2026-03-16
**Documentation Version**: 1.0.0
**🚀 Ready to get started? Check out the [Documentation Index](docs/README.md)!**

View File

@@ -1,32 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
annotations:
argocd.argoproj.io/sync-wave: "-1"
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: infrastructure-apps
namespace: argocd
labels:
app.kubernetes.io/name: infrastructure-apps
app.kubernetes.io/part-of: platform
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
targetRevision: HEAD
path: infra/overlays/aks-dev
destination:
server: https://kubernetes.default.svc
namespace: default
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

View File

@@ -1,32 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
annotations:
argocd.argoproj.io/sync-wave: "-1"
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: infrastructure-apps
namespace: argocd
labels:
app.kubernetes.io/name: infrastructure-apps
app.kubernetes.io/part-of: platform
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
targetRevision: HEAD
path: infra/overlays/aks-prod
destination:
server: https://kubernetes.default.svc
namespace: default
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

View File

@@ -1,32 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
annotations:
argocd.argoproj.io/sync-wave: "-1"
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: infrastructure-apps
namespace: argocd
labels:
app.kubernetes.io/name: infrastructure-apps
app.kubernetes.io/part-of: platform
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
targetRevision: HEAD
path: infra/overlays/eks-dev
destination:
server: https://kubernetes.default.svc
namespace: default
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

View File

@@ -1,32 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
annotations:
argocd.argoproj.io/sync-wave: "-1"
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: infrastructure-apps
namespace: argocd
labels:
app.kubernetes.io/name: infrastructure-apps
app.kubernetes.io/part-of: platform
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
targetRevision: HEAD
path: infra/overlays/eks-prod
destination:
server: https://kubernetes.default.svc
namespace: default
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

View File

@@ -1,32 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
annotations:
argocd.argoproj.io/sync-wave: "-1"
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: infrastructure-apps
namespace: argocd
labels:
app.kubernetes.io/name: infrastructure-apps
app.kubernetes.io/part-of: platform
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
targetRevision: HEAD
path: infra/overlays/gke-prod
destination:
server: https://kubernetes.default.svc
namespace: default
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

View File

@@ -18,7 +18,7 @@ metadata:
spec:
project: default
source:
repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
repoURL: git@github.com:fortedigital/sturdy-adventure.git
targetRevision: HEAD
path: infra/overlays/upc-prod
destination:

View File

@@ -1,6 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- argo-mcp.yaml
- argocdmcp-auth-oidc-sealed.yaml
- argocd-mcp-credentials.yaml

View File

@@ -37,7 +37,7 @@ spec:
- $values/infra/values/base/dot-ai-stack-values.yaml
- $values/infra/values/upc-dev/dot-ai-stack-values.yaml
- repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
- repoURL: git@github.com:fortedigital/sturdy-adventure.git
targetRevision: HEAD
ref: values

View File

@@ -1,5 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- dot-ai-stack.yaml
- dot-ai-secrets.yaml

View File

@@ -1,8 +1,8 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- dot-ai-stack
- mcp10x
- musicman
- ts-mcp
- argo-mcp
- dot-ai-stack.yaml
- mcp10x.yaml
- musicman.yaml
- ts-mcp.yaml
- argo-mcp.yaml

View File

@@ -1,5 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- mcp10x.yaml
- forte10x-app-credentials-sealed.yaml

View File

@@ -36,8 +36,13 @@ spec:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- CreateNamespace=true
- Validate=true
- ServerSideApply=false
- Replace=false
retry:
limit: 5
backoff:

View File

@@ -1,5 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- musicman.yaml
- musicman-credentials.yaml

View File

@@ -1,3 +1,13 @@
---
# Namespace must be created first (sync-wave: -1)
apiVersion: v1
kind: Namespace
metadata:
name: ts-mcp
annotations:
argocd.argoproj.io/sync-wave: "-1"
---
# ArgoCD Application syncs last (sync-wave: 11)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:

View File

@@ -1,5 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ts-mcp.yaml
- ts-mcp-secrets-sealed.yaml

View File

@@ -1,13 +0,0 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: ts-mcp-secrets
namespace: ts-mcp
spec:
encryptedData:
AZURE_CLIENT_SECRET: AgCWj525+NHkZ8XG97hEe4RS0SDC0QIGDXmEvzSlIqJQ9XVZEeKxVuAYmJ+w/HH7zBXD3qlZISeOPKn3FbMEeRukmYK0d5PsH26tRUMPoMzwWCuQkZIQ83uX9Pz/wMiqW8aZFIxpdEiUgVdanxHSFoDRPC1VlSEtV9B9yN2MgXBID5s0oje5BM9ttc4WVRe6+9pMeaOC6u+YUgcfY7xPLetZfC9nQO4zn4jYhoQXfAddwMzNODvQNGPzIv6PXDXJweTwdmtGaxM6eDdcCJI/30bEV9prA5m6UlgTZ/Qp+onU70KdkBA9gM9tMMVUR6j/2sbWzqMP/rVaFLeUH1PjHv15n4EieWyuDyYEfmZNDFXc7O9RIK6P0jCIE+t3myxK2ZQ7cfXprdOSj94au0qP6leat0UUVoc9CFJHHtrNxXYWl7IYVhwvIQCMSgO2qoAXkdW4wKVJAcbJadJjoL2pWxzjaD4GgnUaAxWBANqZI2lD8CED4VfUVMB0ZUYRS/zvy/eqIGlT8WbzwTYFi3YDZRvAUIknxaWEavIG4x52d0FqTmFYY06W53fGYfBrUjJI54GWYyBpKdZTf7b/AlAN0+kwkk6OqsUWwWDqxR7LVCcPhjSIKd/THp+Tbq9z5TiPIHxOO9V60u51f8IoQrEgQfNov7CEGQZ8B9HUGObjNc5MhujzBJasMhrUcd2Ddk6KWk07B7223p/gIEM+81ZWQYUcc29+U/j1dQyRNZy/TC56ywe5DDBJSoGp
template:
metadata:
name: ts-mcp-secrets
namespace: ts-mcp

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base/musicman

View File

@@ -1,47 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: dbunk-demo
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "12"
labels:
app.kubernetes.io/name: dbunk-demo
app.kubernetes.io/part-of: apps
app.kubernetes.io/managed-by: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
sources:
- repoURL: ssh://git@git.forteapps.net:2222/Forte/forte-helm.git
path: forteapp
targetRevision: HEAD
helm:
valueFiles:
- $values/dbunk-demo/values.yaml
- repoURL: ssh://git@git.forteapps.net:2222/Forte/helm-prod-values.git
targetRevision: HEAD
ref: values
destination:
server: https://kubernetes.default.svc
namespace: dbunk-demo
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- CreateNamespace=true
- Validate=true
- ServerSideApply=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- dbunk-demo.yaml

View File

@@ -1,37 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: forte-drop-mcp
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "1"
notifications.argoproj.io/subscribe.on-sync-succeeded.slack: ""
notifications.argoproj.io/subscribe.on-sync-failed.slack: ""
notifications.argoproj.io/subscribe.on-degraded.slack: ""
labels:
app.kubernetes.io/name: forte-drop-mcp
app.kubernetes.io/part-of: apps
app.kubernetes.io/managed-by: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
sources:
- repoURL: ssh://git@git.forteapps.net:2222/Forte/forte-helm.git
path: forteapp
targetRevision: HEAD
helm:
valueFiles:
- $values/forte-drop-mcp/values.yaml
- repoURL: ssh://git@git.forteapps.net:2222/Forte/helm-prod-values.git
targetRevision: HEAD
ref: values
destination:
server: https://kubernetes.default.svc
namespace: forte-drop
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true

View File

@@ -1,8 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- forte-drop-mcp.yaml
# No keycloak-client config + no auth-oidc Secret for mcp mode. The chart's
# auth.type: mcp auto-registers the MCP client; the sidecar is an RFC 9728
# resource server that validates tokens (no client-secret of its own).
# forte-drop-secrets (shared with web) covers PG + S3 creds.

View File

@@ -1,143 +0,0 @@
# forte-drop Postgres — backup & restore runbook
## What gets backed up
A CronJob (`forte-drop-pg-backup`, namespace `forte-drop`) runs nightly at **02:00 UTC**:
1. `pg_dump` of the `drops` database → gzip.
2. Upload to **UpCloud Managed Object Storage**: `s3://drops/_pgbackups/forte-drop-<TS>.sql.gz`
(the `_pgbackups/` prefix is collision-proof: app slugs match `/^[a-z0-9][a-z0-9-]{0,62}$/`
and can never start with `_`).
3. Retention: dumps older than **30 days** are pruned.
S3 creds come from the `forte-drop-secrets` Secret (`S3_ENDPOINT` / `S3_KEY` / `S3_SECRET`).
Postgres creds from `forte-drop-pg-creds` (`pgusername` / `pgpassword`).
> **Object storage is the durable tier.** App data + DB backups both live in UpCloud
> Managed Object Storage (replicated by UpCloud). The in-cluster Postgres PVC is the
> live working copy; the nightly dump is the recovery point. The PVC carries
> `Prune=false,Delete=false` so ArgoCD never deletes it.
## Prerequisites
```bash
export KUBECONFIG=~/Downloads/dev-fd-no-svg1_kubeconfig.yaml
# Confirm the namespace + DB pod are up:
kubectl -n forte-drop get pods -l app.kubernetes.io/name=postgresql
```
## List available backups
```bash
# Run an ephemeral mc pod with the app's S3 creds:
kubectl -n forte-drop run mc-list --rm -it --restart=Never \
--image=quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z \
--overrides='{"spec":{"containers":[{"name":"mc","image":"quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z","command":["sh","-c","mc alias set obj \"$S3_ENDPOINT\" \"$S3_KEY\" \"$S3_SECRET\" >/dev/null && mc ls obj/drops/_pgbackups/"],"envFrom":[{"secretRef":{"name":"forte-drop-secrets"}}]}]}}'
```
## Manually trigger a backup (before risky changes)
```bash
kubectl -n forte-drop create job --from=cronjob/forte-drop-pg-backup pg-backup-manual-$(date +%s)
# Watch:
kubectl -n forte-drop get jobs -l app.kubernetes.io/component=backup
kubectl -n forte-drop logs -l app.kubernetes.io/component=backup --tail=40
```
## Restore a dump
> **Destructive.** This overwrites the live `drops` database. Take a fresh manual
> backup first (above) and confirm with whoever owns the data before proceeding.
### 1. Pick the dump to restore
List backups (above), choose `forte-drop-<TS>.sql.gz`.
### 2. Run a restore pod that pulls the dump and pipes it into Postgres
```bash
DUMP="forte-drop-20260530T020000Z.sql.gz" # <-- set to the chosen file
kubectl -n forte-drop run pg-restore --rm -it --restart=Never \
--image=postgres:16-alpine \
--overrides='{
"spec": {
"containers": [{
"name": "restore",
"image": "postgres:16-alpine",
"command": ["sh","-c","set -euo pipefail; \
apk add --no-cache curl >/dev/null; \
# download via mc is simpler — use a 2-step instead (see note). \
echo placeholder"],
"envFrom": [
{"secretRef":{"name":"forte-drop-pg-creds"}},
{"secretRef":{"name":"forte-drop-secrets"}}
]
}]
}
}'
```
**Simpler 2-pod approach (recommended — avoids cramming mc + psql in one image):**
```bash
DUMP="forte-drop-20260530T020000Z.sql.gz"
# (a) Download the dump from object storage to a local file:
kubectl -n forte-drop run mc-get --rm -it --restart=Never \
--image=quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z \
--overrides='{"spec":{"containers":[{"name":"mc","image":"quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z","command":["sh","-c","mc alias set obj \"$S3_ENDPOINT\" \"$S3_KEY\" \"$S3_SECRET\" >/dev/null && mc cat obj/drops/_pgbackups/'"$DUMP"'"],"envFrom":[{"secretRef":{"name":"forte-drop-secrets"}}]}]}}' \
> /tmp/$DUMP
# (b) Pipe it into the live Postgres via the service:
gunzip -c /tmp/$DUMP | kubectl -n forte-drop run pg-restore --rm -i --restart=Never \
--image=postgres:16-alpine \
--overrides='{"spec":{"containers":[{"name":"psql","image":"postgres:16-alpine","stdin":true,"command":["sh","-c","PGPASSWORD=\"$pgpassword\" psql -h forte-drop-postgresql.forte-drop.svc -U \"$pgusername\" -d drops"],"env":[{"name":"pgusername","valueFrom":{"secretKeyRef":{"name":"forte-drop-pg-creds","key":"pgusername"}}},{"name":"pgpassword","valueFrom":{"secretKeyRef":{"name":"forte-drop-pg-creds","key":"pgpassword"}}}]}]}}'
```
> The app's schema is created idempotently on boot (`CREATE TABLE IF NOT EXISTS` +
> `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` in `src/repo/pg.ts`), and `pg_dump`
> output includes the data. For a clean restore into a fresh DB this just works.
> To restore over an existing DB with conflicting rows, drop/recreate the `drops`
> database first (coordinate downtime — scale the web Deployment to 0 during the
> restore so the app isn't writing).
### 3. Verify
```bash
kubectl -n forte-drop run pg-check --rm -it --restart=Never \
--image=postgres:16-alpine \
--env="PGPASSWORD=$(kubectl -n forte-drop get secret forte-drop-pg-creds -o jsonpath='{.data.pgpassword}' | base64 -d)" \
--command -- psql -h forte-drop-postgresql.forte-drop.svc -U drops -d drops \
-c "SELECT count(*) AS drops FROM drops;" -c "SELECT count(*) AS view_hits FROM view_hits;"
```
### 4. Bring the app back
```bash
# If you scaled web to 0 for the restore:
kubectl -n forte-drop scale deploy/forte-drop --replicas=2
```
## Object data (uploaded drop files)
Drop files live in `s3://drops/<slug>/...` in the same managed bucket. They are
**not** part of the pg backup (the dump only holds metadata). Object storage is
UpCloud-managed/replicated, so no separate file backup is configured. If a
file-level backup is later required, mirror the bucket to a second bucket/region:
```bash
mc mirror --overwrite obj/drops/ backup-target/drops-mirror/
```
(Exclude `_pgbackups/` from the app-data mirror if you split them.)
## Disaster scenarios
| Scenario | Recovery |
|---|---|
| Postgres pod crash / reschedule | StatefulSet reattaches the PVC; ~12 min downtime; no data loss. |
| PVC lost / corrupted | Recreate StatefulSet, restore latest nightly dump (above). Data since last dump is lost. |
| Accidental `drops` table data loss | Restore latest dump; or `pg_restore` a single table from a dump. |
| Namespace deleted | PVC has `Prune=false,Delete=false`; recreate Applications, PVC re-binds, app recovers. Backups in object storage are independent. |
| Object storage bucket lost | UpCloud-managed (replicated). If the IAM key is rotated, update `forte-drop-secrets` (re-seal). |

View File

@@ -1,40 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: forte-drop-postgresql
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "0"
labels:
app.kubernetes.io/name: forte-drop-postgresql
app.kubernetes.io/part-of: apps
app.kubernetes.io/managed-by: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
targetRevision: HEAD
path: apps/overlays/upc-dev/forte-drop-postgresql/resources
destination:
server: https://kubernetes.default.svc
namespace: forte-drop
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- CreateNamespace=true
- Validate=true
- ServerSideApply=true
ignoreDifferences:
- group: apps
kind: StatefulSet
jsonPointers:
- /spec/volumeClaimTemplates

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- forte-drop-postgresql.yaml

View File

@@ -1,14 +0,0 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: forte-drop-pg-creds
namespace: forte-drop
spec:
encryptedData:
pgpassword: AgBYokuQRCTmPGC8soB7n8W39nmSAZDTV97i77NmdMh5ndaZNtCtXxhMcpM2z8kaGv2tCWIh47dr38a5tGPZ0TsmeEQOF9Rbbtq1fyR7pJwT8S+N2Z2zu354wNWQlltEAVvTvthe2wTer/BpyRofeSZprxihmfNpuHUP8rLsnIXln5tOWDzJ8hnRWoYZFITaihC2qrJj/kFE0Rfdcmzt1tSq3jCB/rWijVaJF9XSh4rzQoqZDiUNjDPUjyERILw59JWU4zf9OKcqNHDnmpXBR4LSjLhd9waN6ElEzO4gGcVaHISKrTwewX1ONwPHDnw6lqkQObyBPx8aUsGzxLkUhNDtvIYDkB4BKWP5Qu4bNcSztIbrxi6l7Lr/DWC9qTbKm1/p83rc6r8VqRMURUyQg8/vBlCHOIUbZ8DM1OfNlMd8gvcSkaxEVIdCDUjguCvE3cGyG4cqv2unllZQ+9417WwLNJecT6x1EL3nQyAlK5c9vUIbcVyaFlbSUcGB7xmPgZrZ6/3RDyOH6Tmew1ssV9gLvdaehscUE0fjnnFnJpczkwdyxIOSNLIkjlWetCKEbhowJbzk05h3M2p6XQQOuNTnsYjAADMGD72GUgAQlY8KXmDELtv09KELcXbeYS4gABPpMrVmvZymq8lqQ13Py8o+cIqbrU5V86WxASTfQ5gMo/ymYabuhTBIapcnaKR1dFCCfu8deh5f2HJ6/1NjdWR+XvEshg+EF5OkTUInukX4vA==
pgusername: AgCs6vyQ8CIv5OneP/jMltIPGdZQbpq/BFmQM1mkBD61Ve+anzve5K0Gkg+zsNfbZf0pOPAXtu4C4aL1Lwv7gqpoe4Hp/UEb/X9uLfJ1b8ZitmM1XsPmmSiCskHjrc2BLkAvfrVIXkHc3LOY2uZ/E5stc6Ss2WFE8/uzzVXW0B8fdEK0criludQ8iwR1gypulEcDNomXgkK/1gmmCWosUcVv4jDMDhqBD+b9WYnBB6J73gUclWVMvYDFdNas2PuoRzu5Twc9TAZrTxN5lvLOXAonOo0YiUbUhEC83sfMWYDT5/9OxqcJhAxtgFe9j83MpCwLSwfeLZm7UsUapWDb60MxPJLGvoGD/ZOhkeYt/YCZYROa57TMslVIL5YU1KCiNWvtRjIqnvdiBxI7MRvPUfAoawS4ktT5PDhTTfrixFbaF95jul2kKBXV+OYB1UNsFhcCgZx9rzYRt4lNmBv4m4HeXIp3EYY8VlGLQ45BVVqjJ4QkISvb7ifQWH1aPMQllj+J3GwW0KJN0dEgsh1LT+C7W5I5mq461NOTF1eih/XRBeuPoLlgApxiGXvFCTx8lji2/JIdOaqcg29hdabSprxa0YMStChi2pbtHhRzAuFCp8mInGt8Q406vu67Y4/51yuwI40YeDVu0lf010TB+/v2Zy3OrNyjlqrD5JNynsLuRl3UhuAKC14Xhg/MiDLvTzfsYE8aog==
template:
metadata:
name: forte-drop-pg-creds
namespace: forte-drop

View File

@@ -1,6 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- postgresql.yaml
- forte-drop-pg-creds-sealed.yaml
- pg-backup-cronjob.yaml

View File

@@ -1,93 +0,0 @@
# Nightly logical backup of the forte-drop Postgres → UpCloud Managed Object Storage.
# Dumps to s3://drops/_pgbackups/ (the `_` prefix is collision-proof: app slugs match
# /^[a-z0-9][a-z0-9-]{0,62}$/ and can never start with `_`). Retains 30 days.
#
# Pod shape: initContainer pg_dump → shared emptyDir → mc upload + retention prune.
# Both images pinned. S3 creds reuse forte-drop-secrets (the app's UpCloud user has
# s3:* on the drops bucket). PG creds from forte-drop-pg-creds.
apiVersion: batch/v1
kind: CronJob
metadata:
name: forte-drop-pg-backup
namespace: forte-drop
labels:
app.kubernetes.io/name: postgresql
app.kubernetes.io/instance: forte-drop
app.kubernetes.io/component: backup
spec:
schedule: "0 2 * * *" # 02:00 UTC daily
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 2
template:
metadata:
labels:
app.kubernetes.io/name: postgresql
app.kubernetes.io/instance: forte-drop
app.kubernetes.io/component: backup
spec:
restartPolicy: Never
securityContext:
runAsNonRoot: true
runAsUser: 65532
fsGroup: 65532
volumes:
- name: work
emptyDir: {}
initContainers:
- name: dump
image: postgres:16-alpine
command:
- sh
- -c
- |
set -euo pipefail
TS=$(date -u +%Y%m%dT%H%M%SZ)
echo "dumping to /work/forte-drop-${TS}.sql.gz"
PGPASSWORD="$PGPASSWORD" pg_dump \
-h forte-drop-postgresql.forte-drop.svc \
-p 5432 -U "$PGUSER" -d drops \
--no-owner --no-privileges \
| gzip -9 > "/work/forte-drop-${TS}.sql.gz"
echo "dump complete: $(ls -lh /work/)"
env:
- name: PGUSER
valueFrom:
secretKeyRef: { name: forte-drop-pg-creds, key: pgusername }
- name: PGPASSWORD
valueFrom:
secretKeyRef: { name: forte-drop-pg-creds, key: pgpassword }
volumeMounts:
- name: work
mountPath: /work
containers:
- name: upload
image: quay.io/minio/mc:RELEASE.2024-11-21T17-21-54Z
command:
- sh
- -c
- |
set -euo pipefail
mc alias set obj "$S3_ENDPOINT" "$S3_KEY" "$S3_SECRET"
mc cp /work/*.sql.gz "obj/${S3_BUCKET}/_pgbackups/"
echo "uploaded. pruning backups older than 30d:"
mc rm --recursive --force --older-than 30d "obj/${S3_BUCKET}/_pgbackups/" || true
echo "backup retention pass complete"
env:
- name: S3_ENDPOINT
valueFrom:
secretKeyRef: { name: forte-drop-secrets, key: S3_ENDPOINT }
- name: S3_BUCKET
value: "drops"
- name: S3_KEY
valueFrom:
secretKeyRef: { name: forte-drop-secrets, key: S3_KEY }
- name: S3_SECRET
valueFrom:
secretKeyRef: { name: forte-drop-secrets, key: S3_SECRET }
volumeMounts:
- name: work
mountPath: /work

View File

@@ -1,105 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: forte-drop-postgresql
namespace: forte-drop
labels:
app.kubernetes.io/name: postgresql
app.kubernetes.io/instance: forte-drop
app.kubernetes.io/component: database
spec:
type: ClusterIP
ports:
- name: tcp-postgresql
port: 5432
targetPort: tcp-postgresql
selector:
app.kubernetes.io/name: postgresql
app.kubernetes.io/instance: forte-drop
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: forte-drop-postgresql
namespace: forte-drop
labels:
app.kubernetes.io/name: postgresql
app.kubernetes.io/instance: forte-drop
app.kubernetes.io/component: database
spec:
serviceName: forte-drop-postgresql
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: postgresql
app.kubernetes.io/instance: forte-drop
template:
metadata:
labels:
app.kubernetes.io/name: postgresql
app.kubernetes.io/instance: forte-drop
app.kubernetes.io/component: database
spec:
containers:
- name: postgresql
image: postgres:16-alpine
# NOTE: no securityContext. The official postgres image's entrypoint must
# start as root to chown a fresh /var/lib/postgresql/data, then drops to
# the postgres user (uid 70 in alpine) via gosu. Forcing runAsNonRoot here
# breaks initdb on a fresh PVC. Matches the vaultwarden-postgresql pattern.
ports:
- name: tcp-postgresql
containerPort: 5432
env:
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: forte-drop-pg-creds
key: pgusername
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: forte-drop-pg-creds
key: pgpassword
- name: POSTGRES_DB
value: drops
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
livenessProbe:
exec:
command:
- sh
- -c
- pg_isready -U "$POSTGRES_USER" -d drops
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
exec:
command:
- sh
- -c
- pg_isready -U "$POSTGRES_USER" -d drops
initialDelaySeconds: 5
periodSeconds: 5
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
volumeClaimTemplates:
- metadata:
name: data
annotations:
argocd.argoproj.io/sync-options: Prune=false,Delete=false
spec:
accessModes:
- ReadWriteOnce
storageClassName: upcloud-block-storage-maxiops
resources:
requests:
storage: 5Gi

View File

@@ -1,24 +0,0 @@
# Keep at least 1 web pod up during voluntary disruptions (node drain, upgrade).
# Pairs with replicaCount: 2 so a drain can evict one pod while the other serves.
#
# Selector verified against live forteapp-chart deployments (mcp10x, argocd-mcp):
# the chart's pod selector is {app.kubernetes.io/instance, app.kubernetes.io/name,
# component: app} where instance/name == the ArgoCD Application (Helm release) name.
# Using all three labels also disambiguates the web pods from the forte-drop-mcp
# deployment that shares the forte-drop namespace (its instance/name == forte-drop-mcp).
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: forte-drop-web
namespace: forte-drop
labels:
app.kubernetes.io/name: forte-drop
app.kubernetes.io/part-of: apps
app.kubernetes.io/managed-by: argocd
spec:
minAvailable: 1
selector:
matchLabels:
app.kubernetes.io/instance: forte-drop
app.kubernetes.io/name: forte-drop
component: app

View File

@@ -1,24 +0,0 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: forte-drop-secrets
namespace: forte-drop
spec:
encryptedData:
BASE_DOMAIN: AgAFybdBryVb2AQuGQC8REXzW0YZlyycJp/KeXnROkW71UjDe4qMAWkWszrJWxZMvAPO/tXmibp7jEol6aB5GKG0k3tswWoprTFXLd9CMR2U9SWR3ZCol4npPXo7uOxhBcNSVt+cDXyejSiFTi6goY2oOtbKAJSF9Nv7Z5ePaqhhFni3ntcmM0S1Ad1l3QR7VvyazHFBXfO0b8Z9NgYsUNbGrXWDwoSAZIv3ly3wx90AXn+dXX5FNPtl9CtyAVhHsl3liwQdhEwS2krZZjj7NiQTCfNXp7BSB9ZETpo9KkoV4AZNy1zupd3HpeXHsyhHjq/JqXIAF3iFU0tZTWjhcwnehYdEU5oduwfLCWym5PYgpiQAGiazpkm1Ss3/PYpZYnR2nWv60b1Pa5i79ZiPNi4GL67AiWoJDw6QxV0Kbzi0AvUkZI1E2PeIJvv1w9NKdMRo49xK8LUx2qSTpWeqRP+1kzklHqclTuNVxiWtR2wUgdoLzvU7p5ETu7kPEmaoE8rYw4dKgQvHlMok2Ky2JsELGBkCiYjUN75T+yNlGs5dzbiwtWOja/r0dJ3ZGBQjcK4/BbTLiMYsrxmJTPPF/2zhCOlFY6cfcRMmc7Mwr68mK9m2rTOJQNjBMDoASiqVMmeSqfRSln7JNb1pAeq4xcz9YJMBJhPy2XNiBvRJK3pGIjVcNST0jSpic1X01NJTy7aFbcniZzYnsKJV61AQb+daGEsB1Ib3GnJ+Rv8+9NfvWg==
PASSWORD_GATE_SECRET: AgDIiAPb7bCpTeORoZXIn7Is/yKT7Qm+qSexgmXuXpe+WaLCJDWM5uOlvxp5tbtK2KD3SNtUDrOJQYfWYBOAPrBMBnSpDr8Ie7/wlJBxXCfROk1TldOU3s5O+5OnFfTDS9rAdcWdZdJ+Bt0aktnpuHJpjdurFca5o8ne5SRwYtGk6mNinCYRcnwMApZ9Y7IxvC9xzOi1QIhoOepMrotRkVJtbrXiclN5cI+P/nU8LaGwM4AEJtO3L8zY5QK30SZ/Tsz20Qo8HHGdIBLhHbhTALrFe8+n+Egda66Vr1DkKgDERW59PeWBd6xHCRGNuAk2ZzUmuiN5DJmWgacAGy9QblMMjhOIGKgiqsd4jx/8wC8FoyxBSstGFajahL2C6oFzCpbp5NiS185znN0ohJXqQzkp5RbcBliSYUZgHC8D1jDGxzcTksD2Hgh2NIAlJOStsfB3fUb6hWIFzxim0lyP9kFM5y92Sf5B1/h6PeHKOi1CMC5RFAEKhoyM4W1W+NxASJeT/NptTehcrjBPxLVKBPh/qgMcuNYNk34EB0asELalYBJ5cxzt31LLiLE/uDxCydXiSk7ACho7LxNffcUUakEwwsJDmjFiCscu83TfZq2vw5/2bgOzUng4XGDBgYwwr/KEHueGX4Keg255R3KqxyiMSKDi4CUEtYShjSsSYwU62rdogOQu20N3HGVn+/CksqKky7GoD958d4PwT5MGJwQhp8EjemBomDxTKNsO+C4moMNZpAEBtPlP65Lc1cfcaLlaFrwP4VtYHLXXBIvMzCC+
PGDATABASE: AgCCkY7AUaoy2V+fRXrEeLODlHOTXWXmWCW6MBWEY1J32QrN0bWUmwDdTxv5KHchbhH8lra9bABNOGfRcz59GIg8fH3fwTWOE8luh9cD50QkphjZPe9eGXc6YzheG0CG5hDnUoiJjnP9/l5cZc6sRxAo7for/bbLa0zja3VMzI+NMhVVRZT01G5R/Aoyf+B/TGm4mYbMyWAIgEEfjl50yIkc4WscfQrRkbxAtBF2qFuS3OL95TYSRULBAt0f3Z2WpCdS2b/pUyHJuuoi8aTo1a9QLFMrUdxVi6ydIKaMcx8xR8DlXQLOOdtIQu4TrgzGX2MgHwrbWuf/SFO5IWB/6/JI3yubo+i88M9LwPKsGeslcquoBG3Ibqlnmw8pcPrXUwBq2BowjAqpGsxdiR5XIi2j8QnpA8dTmFARX9CWKHoXFH5+uxx7SSTPG+izlGtVspsu5xx3F8MZ6eStInCyBimTVrn74+IvMPEBrj7wThO6Bl9EA3POkTRf6AmjuPItTgZK8lfXY/t0qDN5zApeB051+jSGZ0/o4PaY414vCm/+OteuCAI4ooCpDOwG3QD52VtpmoAvGjtuWExrvCHMW+hTIxsNbYJ+2SE1oQy8n8J5kKK+AU/SnYf/VEpbawHrpU48g6sDUHQzzPhoLN41j67qopmfRZAo1r5Tb4n7MnEOF3Yi7aT3lE6JvB3gZUSRqnDYSWObTg==
PGHOST: AgCWDPCQ4P+vPKQXmzNspODUJYVidwQDzFRxY8JDj5JYW1u+2xK8LqE2uWWYCW3YjF+H6yJoH6gu35KqOtjm0E23zMlknwZwkK+L93J/nrnNZHCznoub9jEzaAsTRhytvuIUBPQnFulP/t9Q04RjmR9c14puS7VS/tM57/cLq5zq/BiSYL3Wx2lblE7Xn6k44+8robPP3YkmlOuqziM1li/qevB/jxRV9YFjIpCz6D7WG6Kkx9CA6fzgalMJ5ErUVTJOuLDDty3qKd+9mg4WCLahjyzhPq/rIPUFE77EBDCulnnlMyvOSPSP7pMzkaNR1Ce3cOxOYAZ5mO2u5Y5pXt34V8iPe87N85KHQNZTu1Mu5ELMLdhEYOlO5ZX2x/RsdEuriWz1NW8MC57pEUGE/XrFhDe6x80eHi++qDRat2d04rTBOtlceZMmPVxs8tTa3VaD4PepekSogddLVooscLL7lYIpyR0Q6qOA1NdF2uOpx7hf6QLY1HeD4V8RSVVxV9uK5lUTrnBjtU/aUwnGs0xKxVNdBEKwG5Kh7qdsgyBiEa9V4Y2ElteGHave4J7xzimdASrbgMnlgIfcMuo94eX/0M0BAmpD4bEByMCWbcFgAfNnIOpXLZvoEfZ3DxVysmCzD+0nq+CEatdOjKp11EKbtf3H4PgnKqWDe2p8FNPXhev60CMhFvXS/QYg9fe2YJt/wqcN5MstCej4IuiOJkuFJ0UhZ/F5760xhuPYCUWP1suIVZM=
PGPASSWORD: AgBH32G+EUtc3jzGCA9bf27TCbzgK9xz+r4dqd0QQJL9xHbqgOARGVVaQ88AOkWV5VgYjqc/GFp51jLzVOHxgLdkqO/oCBuX9ajQEoGfq24AxFnaB7fh+Vlc3/N9yhT8lWoxHmHjyMVeX75g/9KvhNaRKBgiWQNHlt1C2FNh1h3U/aMfWVJIENmKKH2A5sxWe5haB7nynZc9r1QXBQKa7XVpuxAFXDHz3j3cFyR5Qflp+ac2APEM1/xbiaZDgkBtBd6dsDoCP56Dr1m91kaRGgbeX6WmRJ/Y89WAp4yt3QVfa8uGL1+DrBBMcfB1nAQKA45eZjPE6zTOEHxgTETCcmXJQiOzttDmBHRkIClOLLipgGDJwMqtgQoEMoJKjXMC0rsRy0NVRmibZa310R3PQjuHrQXxRD9ZAXkYg3opwLKeKi07b/7mvLHr7hU81fkBGnqNm/6heOSAqDZfRdregbBbcI/go72aypn2vQ5R+ozCdfwcp1tGla8FGpkI+zAdBKihp5Yo21VZ83FlIMq2JHF2+tv58C+LFeyqL1nr6BUmGKUQ+lEOnRzGYo1sbO5wBChc6yP3ZbzZYfxfvXAdfDY7vZUsareOC4uyR1wDnIiJgQ4kqmAKf7HulJJatKNgsvbmukj7c6lHLsfFRg5pwLO6iese9TZgtima2wkdcRpHSdt4ycnyHbwrEZ4kepfFlN1pGUl573/3l2cOdzO+WLCqV96P5myL6OOmCTxOaLdSyA==
PGPORT: AgCXyGjhlTcq3ffD+ZZZxe5gvqRLQl2MYGl7SfJDjw4LMdr3iojAeQNdYZIeZN4Bec9thhz1DI/Bc5Iiw5TiB3MAAvl/XtQM/T23JrSlUsfqECuHKQGw5kkgeVCds4j4d/IHFt9yBv4W4+ogcL5TZIGGgMiKa/99T77xgE4OqjeqqC+bpFuLObiKt5sYcqcHl2DDshTy4xeukfdA2yI3N9Tq7zfbLpZUKHWWe1gE+FbA0s+QA1ZCYv9uEtu/E5BWOYxE0u4GKhMDfMpguks0CNnIoRpovY3vmOxzFtqa7RpPoQrQtEN2Xtizu/G8K8p+mZLQ0cf3KX0rwp8tAsVKWrp8FYOy6+2OLYlhmDl1hK+M15czaL60DVBqiRIlQFdj9K+s5KQ0qfsG9m4v2WPSeCLEKwcsyLyq9vzXbDKwcMN2DOMOan8VDo04UZzkfdZa3lztfw7MWIdpvB0+cpgK3mdbbPCqSIQ8UTjeSVBgEz4EPUds81W8gqAgmeVKpUQplgFuLNwCTnCkuMSNiA93D3dgGccXzXiDrIBIcLzxHAWr0XEjISyD/pmNEowbDvXB5yRlNi1ZB5AHsRPurUYs4XtZN5ar+sxyex78tG18OkcBbABatBp3ol28Rs1LEN+HJupzNl4XDUx35/j3t/FPcw9PkcF+hzaQL7aWj4EDEWfjlzz5FEBlU5N6sd078tB79TpasuIk
PGUSER: AgDH9oSzu61NVOUlK7Evr278VuOx7ZxZgdsk+kdJmwpBGX7r5gmPrdomh/mqLYTuLokkanFiTfchRWHc76FvjbA/KxqwCq1qsZbW+dXrRtx/z1wQApKxNUJ7JolwMwP7tHE6QlGzO2mWj6RUROnhKpNybJXVvC3E5sSyz2QWC9hjamQP997RGA9yiiT/OShC7I6drFYR5cRDtpjW7Sy46qhMwlCRppiKh3wOV7qIAa0aPQE3Rfcg2WpK2ugRL1N+SiVnM+wPQwYVLiDaVF40vP40Kari99hIgmhcbjPeGG3kGX5VLww9KGm7iryrW3Yx45L/CCh1arUUpjkK2FGLVKtb3+YmDadnOA/I8Rr5kebhoMc93E3U3+mDfQA3cO/23xgpOJEGRCQwBlN9mazqkdq4zQkb4+nuxsdyQcxYtncgxhfCcZ0mXnbX2aW2kYcxKqa/jNjBcEpGMvos7dq6QzNq2nHrITo15S74M0292CAje2NFvKURA/KZnT26dDw3e5xa74E1nI/tBJEHWrUwRXpPu7naCZ2sZMxQV6ixQMuDakx3YamXZmMwgFO2FZ6ZL9BDDsbV4+JAsNwEaHGIIaTbE28R/xPIcUqcxrQV4ZWmHnJXFGyJ0XXxJ57GGjs7QwvvzAm+9WGYtlSC6H/8rX8uZIQLr3llVbJMuLpIv45i3p0Nkx8jyxGSG3rNQ4l3K0rjly2qZg==
S3_BUCKET: AgBcx2UvWafkVQ4fc+Xuc+gCLm5O5DceYSYBslL1bd8p4hKI/2hJF9z9UoLyAedyyrx8pHizcugkcIccrV1iyfQod2lDrivVJCy3qHz0mrQ6ZCOAk9ApNJTYUn+x/AYZL9EKVj/vfEzSwOTqlri6H53aUrZr8lv01TqLcrZDeG+jv64psydYKmSESkPl0iQ9DOx2sGkowNYbiTjux+ED9yXDMDdzojhMepbuUHhjNr7V7Z2NpX8+pNVz2o8o/oIA0zAJs3+C02018N3cyJ9P9BP2/qR6N31aykyI4P6GU5WlL0CDGKaheYy9ukM5x8SoU87yZBHLUxN0MhrMVKoFswU6CwTG/A8Gjkp84ce2cvlnuXP6JEMlqgSb+O+SeCH8+kwL9A87xmb42OOXSGokS188/2/13b6S733WL81B0RX1X1pOfWybpJzDhgiOC3Pjn18ItVHjKw3FlY6kwk1+P1vbZoRl343avRdyQyJam28A9UvZeTG+ac7drRTDndKYKGP8IWg/A32MEFSiKHsvnRQoYE7KORyzJxp610L1+w46KOUF5VF2afch02Jt4uLcmCOyUb6LdcWln3Jiz+wydJI1HK1RPio5oUT5d5se3K3DCa+GffbPMorHe/SdzqaBH2+uBcQn+wZO7JC1ei7hz4U1xdV41gkG1uq7Xz49ymW6P7G6qUr65MtvD73iMRlnJt/mdkw5UQ==
S3_ENDPOINT: AgDA5n96LeW1spJfmm7f9lcus5Ndv/91HAlhOfv0YwP+9R8rQhKF1sEL9Oi7Fpd4VJCQi9vDbeWYjGOMtsMqsLJivcZK6367rdSqL/YpKax9SkDgU+UG5Bpr5SDv4p5C3/J4+GPkZV+BduJQlNVPycTrdM80VBK8aALCNs7gGxy68v6mMgI90MpTp84ZeWgtJ7ZpTySivpwgycwcezldJTYDDjOvVfbK79trsRzDtjbAZzbDn0M8At4LYAfY7JTqcdCRHSGnzyg+81CX2jjyMOH0K55SsIqPEBdXqvh+VmC+jPyFMzfAN5hA8+otWTXfI7VucxAMZtRUiLU3I4/kIO60OkVjEf/SWrJv3nKb85aVfvRZ9k/A7kJ/ysKTXz2/iSAHU1DHTJlMW7SfhXciotx1p3BWOREF0QAQvrNCURFP+eZogBCIuiP38fD0gHkhG2BZkAY/OVZTr17a3F02kwemAuJ85hgUf0iwFV+/LD6X2OiqM85lnac/3B5Qp7UQf+Enn5RTqd90Pk/7QtAf/zTDmfxiWTkwpKXeQ62xcnHL9dQnUmDDs16eJZd04AJGgT8yrC//2lVOht7u6KuyG/+JXRxLBXAkGkr68lHBBuGmeVZ/t7ChL2Asyfu/uUTl9E1BOUfNvZXxwyLgqueL0Uw/MGLgswBsP6pYNGraMF+K5v7dse8cUxpyqk3s1C1zqoAhkehsCBtTjI+hF90G9Nh6mVSocoH12znt7fgUnwNbpg==
S3_KEY: AgAiciTM2ZNVlU1M1CNNXLkhCEYYbO7q5+Mp/DoC4OHBgIDKVfHurH39Dniuwxe6DcvE3vG2glRyTxQEg/ASLcwa7HBwBAwXe3wl1tRGM5Bp40/Vq935BXpkhcdp2fSoup8lPEbKS8q+L5LOqUlx7jmnkHXbI1tasz63KE8O9RUFDdQ8Gxy3nn/u4xkvibYxwmo60ApLKYgOu/ODPEETrWBcITHAVFUxbA8Kr9X9mPm3VpfrnFcUlxsCFr/zZwE/Y01eWdi8GGafb+apDPKMd7mAsLHFcPIQlpkHVT1M21qwwntZg9yV0RBACNu5BVPUgbmtUOQeWYMXn3FE+NJ7ajfdKAUCcEUV/f4s00b0S7jJTJwOUixDquMKSfu00AwDRCs8UcouikZe110uWnfEF3tVE0xQGF/3ItLni9VugBz7wQv7ACvmwnHmX5ZcjE0hxYcIS7ABWgHOZxgWoRWPao8eNAATipafcVIG1szl5ZMNTmAqHFyp2dlNU3zaiW6fz4q4CU7SrlhsrtqYM788qHvpJvDpFdF/i6oitH9CgpwmdCpH6YbBXxatnkWq9bqjEFcSZGDfDyT+iZaoPwhiOfaEoCyKlZ9RLLaK3E8zFcCDRXHnvnkqPtP/+VG30xz9pIat2EVB1N4b/kVIrr+fIM28mwk0vkC/tU8T55GF5BZr7VaYedHM9DVcQ4OJl7Ctrc9Ki8PXrne8gywyomA0F+YY9lxdDw==
S3_REGION: AgAAHTbNQ3gGnvg67ck2N5zSKKhMwR1j6pi/tZzYMEPK8jSnDTBkLYAt6ZVRtdsO+dG9kjnMsc/xTUMxJspbvQkgLSd9mG5FzJ37rBn+azCCSTDYBKq2ddGK1Yf/9w7MxOgN8aCyD4QCFOzR0EI49GbVYQynxDD5BwYuf7y4t2xCYt5wRGsjyNAmH3202Z90XKUkts8Na1hyD+xrrtrAtNfugyZqKo2WUSsHu82TD+cu5xZI51oQ9w9Mh9LaH3nfn/X5S+t17TjYvI7/c6hOwCVv10OdoaZa+SzqOvy7XxnqAShAYkPqJKfWhjecE1b9c/Cun32X4MRI0GBWyA02z4nR+WBbaVmasicx6hchVn8/wZeKMIl68F5LE7184MKKPNNwqsYslwFhuWq8dEw3TaVvnWgx3+cSMiX15SBwcLtE2UzJp+jjN/dpQ2MM0+6uV1GK4mNso5JAwGpUUUi2i+V1Ng7uXipI+5J9w6K0IMg1puGqFyahby42saSuH6vuPnjSx+2dXQTlgbl2SvCBoCgyOOJs4q5IafEvupAmRCNzx2HHv8/z6CFbSQZITQ3plmyNGLFXjynVw/6Q1PD4z3Pows4uVcYOEPbC0UoaXVgwgdBWa2N44BUhpdbqJUCyKuapLigpjKujG43jmdLzuk6gaPeH7SJTZKr624vs9hrGhzQYKdl6FZOQhmCFRDKXVCUiH2Z2pfwd3oo=
S3_SECRET: AgCcVQ7YtBAGpKBm+rE/hQBHrFlX5O0JO94xkZeoAppA9Tf8YR/PguZRGBWgLdNEJRI8C08lRRCUX3PY68jTySyjamb32iQkslOXGjAfnULeNGoGg05nLY2ZDYCEom6ieL8cc2xfbrV3yHoPQ7yVz9vcLjh1vATxyfdkqMapl8FpvQf0k0Zecmw3rLWE9y6vAn6Gb+/CWTnuhcW/8uDykmjIBTDQQddWshaZi+HosHyDbNxlnGj4U8mie68wytpS+Unp1gIWWE0hvelqO/3OUEEBB1OYMLV2DW8v86HXAE1Ix9jiCpSbyB+UzjOlrE/p4fJpeG4FtUC+/5ibRSxxgQRQYklKFJmdRDYWUnOngjgcT/Ewe41mTrpCUvb+jtir68pYLmVrLoha7S60w1YQHNkDAN2GftOyBjkkt6MtUDNzvNkfnKqKGUWyDSC27yfJdE/9k/4lDxQs0Sp20kIuz66/culBpg/s/oPSNs4SolCqG3GVLlKL775uqwLLuDN3txlPLb+Ex5vZAUapke+rn2zXzJVc1qlPfI/96vSEy6cx58LXdBadmBXn6c4Uy2MDa66EwsxOMXxzGLTd7AGkd5oeQVYfVPdTfGV5zx1AdzQhP3u/DD5FhKeWGDOr21iYB2jNm/P/hw0nFP2pf83W4/jLzPvuth1LF/WLF8cjclnGbcep2Kxrh/Xq0LmufofuVJyEI9/fl6onl5KIa6ZnVBJ8TsQesXJtNEKt9cPHiCvBKfLj5C+a4FlY
template:
metadata:
name: forte-drop-secrets
namespace: forte-drop

View File

@@ -1,38 +0,0 @@
# Labeled config Secret read by the Keycloak Client Registrar. Kyverno clones it
# to the keycloak namespace; a CronJob registers the OIDC client in the forte
# realm and writes the credentials back as forte-drop-oidc-credentials in THIS
# namespace (~2 min). The forte-helm auth sidecar (auth.type: oidc) consumes that
# registrar-created Secret automatically — no manual SealedSecret step needed.
apiVersion: v1
kind: Secret
metadata:
name: keycloak-client-forte-drop
namespace: forte-drop
labels:
keycloak.forteapps.net/client-config: "true"
annotations:
keycloak.forteapps.net/source-namespace: "forte-drop"
stringData:
client.json: |
{
"clientId": "forte-drop",
"name": "Forte Drop (web)",
"enabled": true,
"protocol": "openid-connect",
"clientAuthenticatorType": "client-secret",
"standardFlowEnabled": true,
"directAccessGrantsEnabled": false,
"serviceAccountsEnabled": false,
"publicClient": false,
"redirectUris": ["https://drop-k8s.hackathon.forteapps.net/auth/callback"],
"webOrigins": ["https://drop-k8s.hackathon.forteapps.net"],
"defaultClientScopes": ["openid","email","profile"],
"secret": {
"namespace": "forte-drop",
"name": "forte-drop-oidc-credentials",
"keys": {
"clientId": "client-id",
"clientSecret": "client-secret"
}
}
}

View File

@@ -1,8 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- forte-drop.yaml
- keycloak-client-forte-drop.yaml
- forte-drop-pdb.yaml
- forte-drop-secrets-sealed.yaml

View File

@@ -1,17 +0,0 @@
# Owns the forte-drop namespace shared by the web + mcp deployments and the
# postgres StatefulSet (infra overlay). sync-wave -1 ensures the namespace exists
# before the namespaced Secrets/PDB in this base apply (avoids a first-sync
# "namespaces forte-drop not found" race when the business-apps parent syncs).
# Prune=false so removing this base never cascade-deletes the namespace (and with
# it postgres data + the mcp deployment) — matches the earlier decision to keep
# namespace ownership decoupled from any single workload.
apiVersion: v1
kind: Namespace
metadata:
name: forte-drop
annotations:
argocd.argoproj.io/sync-wave: "-1"
argocd.argoproj.io/sync-options: Prune=false
labels:
app.kubernetes.io/managed-by: argocd
app.kubernetes.io/part-of: apps

View File

@@ -2,13 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
- forte-drop-postgresql
- forte-drop
- forte-drop-mcp
- dbunk-demo
# No patches needed — base apps already default to "upc-dev" value paths
# (upc-dev is the default/base cluster).
# forte-drop (postgres + web + mcp) and dbunk-demo are upc-dev-only apps — they
# have hackathon-domain hardcoded values and must not sync to upc-prod, so they
# live here in the overlay rather than in apps/base/.
# No patches needed — base already has "upc-dev" paths
# upc-dev is the default/base cluster

View File

@@ -1,9 +1,8 @@
#!/bin/zsh
# in case of $'\r': command not found error, run command below first
# sed -i 's/\r$//' ./bootstrap.sh
CLUSTER="${1:?Usage: ./bootstrap.sh <cluster> (upc-dev|upc-prod|aks-dev|aks-prod|eks-dev|eks-prod|gke-dev|gke-prod)}"
CLUSTER="${1:?Usage: ./bootstrap.sh <cluster> (upc-dev|upc-prod)}"
echo "running $0 for cluster: ${CLUSTER}..."
@@ -18,7 +17,7 @@ echo "Bootstrapping cluster: ${clusterName} (${CLUSTER})..."
Bootstrap()
{
ArgoCd
Gitea
# Gitea
}
@@ -28,8 +27,8 @@ Bootstrap()
Gitea()
{
echo "Installing secret..."
kubectl apply -f "private/${CLUSTER}/gitea-repo-main.yaml"
kubectl apply -f "private/${CLUSTER}/main.key"
kubectl apply -f private/gitea-repo-main.yaml
kubectl apply -f private/main.key
}
############################################################
@@ -37,15 +36,10 @@ Gitea()
############################################################
ArgoCd()
{
# Pre-create ConfigMap for repo-server env (must exist before Helm upgrade)
kubectl create namespace argocd --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f cluster-resources/argocd-repo-server-config.yaml
# install argocd
echo "Installing ArgoCD..."
helm upgrade --install argocd argo-cd \
--repo https://argoproj.github.io/argo-helm \
--version "7.8.0" \
--namespace argocd --create-namespace \
--values infra/values/base/argocd-values.yaml \
--values "infra/values/${CLUSTER}/argocd-values.yaml" \
@@ -55,4 +49,4 @@ ArgoCd()
kubectl apply -f "_app-of-apps-${CLUSTER}.yaml" -n argocd
}
Bootstrap
# Bootstrap

View File

@@ -1,83 +0,0 @@
# CronJob: syncs OIDC client secret from registrar-managed
# argocd-oidc-credentials into argocd-secret (oidc.clientSecret key).
# Runs every 2 min. No-ops if source secret doesn't exist yet
# (safe for fresh deploys before Keycloak is up).
apiVersion: v1
kind: ServiceAccount
metadata:
name: argocd-oidc-sync
namespace: argocd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: argocd-oidc-sync
namespace: argocd
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["argocd-oidc-credentials", "argocd-secret"]
verbs: ["get", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: argocd-oidc-sync
namespace: argocd
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: argocd-oidc-sync
subjects:
- kind: ServiceAccount
name: argocd-oidc-sync
namespace: argocd
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: argocd-oidc-sync
namespace: argocd
spec:
schedule: "*/2 * * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 1
template:
spec:
serviceAccountName: argocd-oidc-sync
restartPolicy: Never
containers:
- name: sync
image: bitnami/kubectl:latest
command: ["/bin/sh", "-c"]
args:
- |
set -e
# Exit gracefully if source secret doesn't exist yet
if ! kubectl get secret argocd-oidc-credentials -n argocd >/dev/null 2>&1; then
echo "argocd-oidc-credentials not found — skipping (Keycloak not ready yet)"
exit 0
fi
# Read current OIDC client secret
NEW_SECRET=$(kubectl get secret argocd-oidc-credentials -n argocd \
-o jsonpath='{.data.client-secret}' | base64 -d)
# Read current value in argocd-secret (if any)
CURRENT=$(kubectl get secret argocd-secret -n argocd \
-o jsonpath='{.data.oidc\.clientSecret}' 2>/dev/null | base64 -d || echo "")
# Only patch if changed
if [ "$NEW_SECRET" = "$CURRENT" ]; then
echo "oidc.clientSecret already up to date"
exit 0
fi
kubectl patch secret argocd-secret -n argocd --type merge \
-p "{\"stringData\":{\"oidc.clientSecret\":\"${NEW_SECRET}\"}}"
echo "Patched argocd-secret with oidc.clientSecret"

View File

@@ -1,9 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-repo-server-config
namespace: argocd
data:
# Disable git submodule checkout - submodules (e.g. shared-prompts)
# are not needed for K8s manifest generation
ARGOCD_GIT_MODULES_ENABLED: "false"

View File

@@ -0,0 +1,43 @@
# Self-service Keycloak client config for Backstage.
# Kyverno clones this to the keycloak namespace, where the
# keycloak-client-registrar CronJob processes it and creates
# the backstage-oidc-credentials Secret in the backstage namespace.
apiVersion: v1
kind: Secret
metadata:
name: keycloak-client-backstage
namespace: backstage
labels:
keycloak.forteapps.net/client-config: "true"
stringData:
client.json: |
{
"clientId": "backstage",
"name": "Backstage Developer Portal",
"redirectUris": ["https://backstage.forteapps.net/api/auth/oidc/handler/frame"],
"webOrigins": ["https://backstage.forteapps.net"],
"defaultClientScopes": ["openid", "email", "profile"],
"protocolMappers": [
{
"name": "email_verified",
"protocol": "openid-connect",
"protocolMapper": "oidc-hardcoded-claim-mapper",
"config": {
"claim.name": "email_verified",
"claim.value": "true",
"jsonType.label": "boolean",
"id.token.claim": "true",
"access.token.claim": "true",
"userinfo.token.claim": "true"
}
}
],
"secret": {
"namespace": "backstage",
"name": "backstage-oidc-credentials",
"keys": {
"clientId": "AUTH_OIDC_CLIENT_ID",
"clientSecret": "AUTH_OIDC_CLIENT_SECRET"
}
}
}

View File

@@ -57,17 +57,17 @@ spec:
- sh
- -c
- |
mc alias set s3 "${S3_ENDPOINT}" "${AWS_ACCESS_KEY_ID}" "${AWS_SECRET_ACCESS_KEY}"
mc alias set upcloud "${S3_ENDPOINT}" "${AWS_ACCESS_KEY_ID}" "${AWS_SECRET_ACCESS_KEY}"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
KEY="gitea-dump-${TIMESTAMP}.zip"
echo "Uploading ${KEY}..."
mc cp /backup/gitea-dump.zip "s3/${S3_BUCKET}/${KEY}" && \
mc cp /backup/gitea-dump.zip "upcloud/${S3_BUCKET}/${KEY}" && \
echo "Upload complete."
# Prune backups older than 7 days
echo "Pruning backups older than 7 days..."
mc rm --older-than 7d --force "s3/${S3_BUCKET}/" 2>&1 || true
mc rm --older-than 7d --force "upcloud/${S3_BUCKET}/" 2>&1 || true
echo "Pruning complete."
envFrom:
- secretRef:

View File

@@ -16,6 +16,7 @@ spec:
- resources:
namespaces:
- kube-system
- trivy-system
- monitoring
- argocd
- cert-manager

View File

@@ -1,12 +0,0 @@
# Cluster config reference — values must match the corresponding overlay files.
# Read by bootstrap.sh at install time; NOT auto-propagated to ArgoCD value files.
clusterName: k8s-launchpad # → infra/values/aks-dev/argocd-values.yaml (notifications.context.clusterName)
domain: example.com # → infra/values/base/gitea-values.yaml, renovate-values.yaml, keycloak-values.yaml (subdomains)
argocdDomain: argocd.example.com # → infra/values/aks-dev/argocd-values.yaml (global.domain)
grafanaDomain: grafana.example.com # → infra/values/aks-dev/grafana-values.yaml (ingress.hosts)
keycloakDomain: id.example.com # → infra/values/aks-dev/keycloak-values.yaml (ingress.hostname)
dotaiDomain: kubemcp.example.com # → infra/values/aks-dev/dot-ai-stack-values.yaml (dot-ai.ingress.host) — create if needed
dotaiUiDomain: kubemcpui.example.com # → infra/values/aks-dev/dot-ai-stack-values.yaml (dot-ai-ui.ingress.host) — create if needed
letsencryptEmail: admin@example.com # → cluster-resources/letsencrypt-issuer.yaml (spec.acme.email)
trustedIPs: "10.0.0.0/8,168.63.129.16/32" # → infra/values/aks-dev/traefik-values.yaml (ports.*.trustedIPs) — VNet CIDR + Azure health probe
cloudProvider: azure # → determines overlay directory and cloud-specific LB/storage annotations

View File

@@ -1,12 +0,0 @@
# Cluster config reference — values must match the corresponding overlay files.
# Read by bootstrap.sh at install time; NOT auto-propagated to ArgoCD value files.
clusterName: prod-aks # → infra/values/aks-prod/argocd-values.yaml (notifications.context.clusterName)
domain: example.com # → infra/values/base/gitea-values.yaml, renovate-values.yaml, keycloak-values.yaml (subdomains)
argocdDomain: argocd.example.com # → infra/values/aks-prod/argocd-values.yaml (global.domain)
grafanaDomain: grafana.example.com # → infra/values/aks-prod/grafana-values.yaml (ingress.hosts)
keycloakDomain: id.example.com # → infra/values/aks-prod/keycloak-values.yaml (ingress.hostname)
dotaiDomain: kubemcp.example.com # → infra/values/aks-prod/dot-ai-stack-values.yaml (dot-ai.ingress.host) — create if needed
dotaiUiDomain: kubemcpui.example.com # → infra/values/aks-prod/dot-ai-stack-values.yaml (dot-ai-ui.ingress.host) — create if needed
letsencryptEmail: admin@example.com # → cluster-resources/letsencrypt-issuer.yaml (spec.acme.email)
trustedIPs: "10.0.0.0/8,168.63.129.16/32" # → infra/values/aks-prod/traefik-values.yaml (ports.*.trustedIPs) — VNet CIDR + Azure health probe
cloudProvider: azure # → determines overlay directory and cloud-specific LB/storage annotations

View File

@@ -1,12 +0,0 @@
# Cluster config reference — values must match the corresponding overlay files.
# Read by bootstrap.sh at install time; NOT auto-propagated to ArgoCD value files.
clusterName: dev-eks # → infra/values/eks-dev/argocd-values.yaml (notifications.context.clusterName)
domain: example.com # → infra/values/base/gitea-values.yaml, renovate-values.yaml, keycloak-values.yaml (subdomains)
argocdDomain: argocd.example.com # → infra/values/eks-dev/argocd-values.yaml (global.domain)
grafanaDomain: grafana.example.com # → infra/values/eks-dev/grafana-values.yaml (ingress.hosts)
keycloakDomain: id.example.com # → infra/values/eks-dev/keycloak-values.yaml (ingress.hostname)
dotaiDomain: kubemcp.example.com # → infra/values/eks-dev/dot-ai-stack-values.yaml (dot-ai.ingress.host) — create if needed
dotaiUiDomain: kubemcpui.example.com # → infra/values/eks-dev/dot-ai-stack-values.yaml (dot-ai-ui.ingress.host) — create if needed
letsencryptEmail: admin@example.com # → cluster-resources/letsencrypt-issuer.yaml (spec.acme.email)
trustedIPs: "10.0.0.0/8" # → infra/values/eks-dev/traefik-values.yaml (ports.*.trustedIPs) — VPC CIDR
cloudProvider: eks # → determines overlay directory and cloud-specific LB/storage annotations

View File

@@ -1,12 +0,0 @@
# Cluster config reference — values must match the corresponding overlay files.
# Read by bootstrap.sh at install time; NOT auto-propagated to ArgoCD value files.
clusterName: prod-eks # → infra/values/eks-prod/argocd-values.yaml (notifications.context.clusterName)
domain: example.com # → infra/values/base/gitea-values.yaml, renovate-values.yaml, keycloak-values.yaml (subdomains)
argocdDomain: argocd.example.com # → infra/values/eks-prod/argocd-values.yaml (global.domain)
grafanaDomain: grafana.example.com # → infra/values/eks-prod/grafana-values.yaml (ingress.hosts)
keycloakDomain: id.example.com # → infra/values/eks-prod/keycloak-values.yaml (ingress.hostname)
dotaiDomain: kubemcp.example.com # → infra/values/eks-prod/dot-ai-stack-values.yaml (dot-ai.ingress.host) — create if needed
dotaiUiDomain: kubemcpui.example.com # → infra/values/eks-prod/dot-ai-stack-values.yaml (dot-ai-ui.ingress.host) — create if needed
letsencryptEmail: admin@example.com # → cluster-resources/letsencrypt-issuer.yaml (spec.acme.email)
trustedIPs: "10.0.0.0/8" # → infra/values/eks-prod/traefik-values.yaml (ports.*.trustedIPs) — VPC CIDR
cloudProvider: eks # → determines overlay directory and cloud-specific LB/storage annotations

View File

@@ -1,12 +0,0 @@
# Cluster config reference — values must match the corresponding overlay files.
# Read by bootstrap.sh at install time; NOT auto-propagated to ArgoCD value files.
clusterName: dev-gke # → infra/values/gke-dev/argocd-values.yaml (notifications.context.clusterName)
domain: example.com # → infra/values/base/gitea-values.yaml, renovate-values.yaml, keycloak-values.yaml (subdomains)
argocdDomain: argocd.example.com # → infra/values/gke-dev/argocd-values.yaml (global.domain)
grafanaDomain: grafana.example.com # → infra/values/gke-dev/grafana-values.yaml (ingress.hosts)
keycloakDomain: id.example.com # → infra/values/gke-dev/keycloak-values.yaml (ingress.hostname)
dotaiDomain: kubemcp.example.com # → infra/values/gke-dev/dot-ai-stack-values.yaml (dot-ai.ingress.host) — create if needed
dotaiUiDomain: kubemcpui.example.com # → infra/values/gke-dev/dot-ai-stack-values.yaml (dot-ai-ui.ingress.host) — create if needed
letsencryptEmail: admin@example.com # → cluster-resources/letsencrypt-issuer.yaml (spec.acme.email)
trustedIPs: "10.0.0.0/8,35.191.0.0/16,130.211.0.0/22" # → infra/values/gke-dev/traefik-values.yaml (ports.*.trustedIPs) — subnet + GCP health checks
cloudProvider: gke # → determines overlay directory and cloud-specific LB/storage annotations

View File

@@ -1,12 +0,0 @@
# Cluster config reference — values must match the corresponding overlay files.
# Read by bootstrap.sh at install time; NOT auto-propagated to ArgoCD value files.
clusterName: prod-gke # → infra/values/gke-prod/argocd-values.yaml (notifications.context.clusterName)
domain: example.com # → infra/values/base/gitea-values.yaml, renovate-values.yaml, keycloak-values.yaml (subdomains)
argocdDomain: argocd.example.com # → infra/values/gke-prod/argocd-values.yaml (global.domain)
grafanaDomain: grafana.example.com # → infra/values/gke-prod/grafana-values.yaml (ingress.hosts)
keycloakDomain: id.example.com # → infra/values/gke-prod/keycloak-values.yaml (ingress.hostname)
dotaiDomain: kubemcp.example.com # → infra/values/gke-prod/dot-ai-stack-values.yaml (dot-ai.ingress.host) — create if needed
dotaiUiDomain: kubemcpui.example.com # → infra/values/gke-prod/dot-ai-stack-values.yaml (dot-ai-ui.ingress.host) — create if needed
letsencryptEmail: admin@example.com # → cluster-resources/letsencrypt-issuer.yaml (spec.acme.email)
trustedIPs: "10.0.0.0/8,35.191.0.0/16,130.211.0.0/22" # → infra/values/gke-prod/traefik-values.yaml (ports.*.trustedIPs) — subnet + GCP health checks
cloudProvider: gke # → determines overlay directory and cloud-specific LB/storage annotations

View File

@@ -1,12 +1,10 @@
# Cluster config reference — values must match the corresponding overlay files.
# Read by bootstrap.sh at install time; NOT auto-propagated to ArgoCD value files.
clusterName: dev-fd-no-svg1 # → infra/values/upc-dev/argocd-values.yaml (notifications.context.clusterName)
domain: forteapps.net # → infra/values/base/gitea-values.yaml, renovate-values.yaml, keycloak-values.yaml (subdomains)
argocdDomain: argocd.127.0.0.1.nip.io # → infra/values/upc-dev/argocd-values.yaml (global.domain)
grafanaDomain: grafana.forteapps.net # → infra/values/upc-dev/grafana-values.yaml (ingress.hosts)
keycloakDomain: id.forteapps.net # → infra/values/upc-dev/keycloak-values.yaml (ingress.hostname)
dotaiDomain: kubemcp.forteapps.net # → infra/values/upc-dev/dot-ai-stack-values.yaml (dot-ai.ingress.host)
dotaiUiDomain: kubemcpui.forteapps.net # → infra/values/upc-dev/dot-ai-stack-values.yaml (dot-ai-ui.ingress.host)
letsencryptEmail: danijels@gmail.com # → cluster-resources/letsencrypt-issuer.yaml (spec.acme.email)
trustedIPs: "172.16.1.0/24" # → infra/values/upc-dev/traefik-values.yaml (ports.*.trustedIPs)
cloudProvider: upcloud # → determines overlay directory and cloud-specific LB/storage annotations
clusterName: dev-fd-no-svg1
domain: forteapps.net
argocdDomain: argocd.127.0.0.1.nip.io
grafanaDomain: grafana.forteapps.net
keycloakDomain: id.forteapps.net
dotaiDomain: kubemcp.forteapps.net
dotaiUiDomain: kubemcpui.forteapps.net
letsencryptEmail: danijels@gmail.com
trustedIPs: "172.16.1.0/24"
cloudProvider: upcloud

View File

@@ -1,12 +1,10 @@
# Cluster config reference — values must match the corresponding overlay files.
# Read by bootstrap.sh at install time; NOT auto-propagated to ArgoCD value files.
clusterName: prod-fd-no-svg1 # → infra/values/upc-prod/argocd-values.yaml (notifications.context.clusterName)
domain: fortedigital.com # → infra/values/base/gitea-values.yaml, renovate-values.yaml, keycloak-values.yaml (subdomains)
argocdDomain: argocd.127.0.0.1.nip.io # → infra/values/upc-prod/argocd-values.yaml (global.domain)
grafanaDomain: grafana.fortedigital.com # → infra/values/upc-prod/grafana-values.yaml (ingress.hosts)
keycloakDomain: id.fortedigital.com # → infra/values/upc-prod/keycloak-values.yaml (ingress.hostname)
dotaiDomain: kubemcp.fortedigital.com # → infra/values/upc-prod/dot-ai-stack-values.yaml (dot-ai.ingress.host)
dotaiUiDomain: kubemcpui.fortedigital.com # → infra/values/upc-prod/dot-ai-stack-values.yaml (dot-ai-ui.ingress.host)
letsencryptEmail: danijel.simeunovic@fortedigital.com # → cluster-resources/letsencrypt-issuer.yaml (spec.acme.email)
trustedIPs: "172.16.1.0/24" # → infra/values/upc-prod/traefik-values.yaml (ports.*.trustedIPs)
cloudProvider: upcloud # → determines overlay directory and cloud-specific LB/storage annotations
clusterName: prod-fd-no-svg1
domain: fortedigital.com
argocdDomain: argocd.127.0.0.1.nip.io
grafanaDomain: grafana.fortedigital.com
keycloakDomain: id.fortedigital.com
dotaiDomain: kubemcp.fortedigital.com
dotaiUiDomain: kubemcpui.fortedigital.com
letsencryptEmail: danijel.simeunovic@fortedigital.com
trustedIPs: "172.16.1.0/24"
cloudProvider: upcloud

View File

@@ -1,32 +0,0 @@
{
"$schema": "https://raw.githubusercontent.com/jetify-com/devbox/0.16.0/.schema/devbox.schema.json",
"packages": [
"kubectl@1.33.2",
"kubernetes-helm@3.18.4",
"k9s@0.50.7",
"kubeseal@0.30.0",
"argocd@2.14.11",
"kubecm@0.33.1",
"kubectl-tree@0.4.3",
"kind@0.29.0",
"kustomize@5.7.0",
"kyverno@1.14.3",
"syft@1.29.0",
"grype@0.92.2",
"traefik@3.6.7",
"claude-code@latest",
"go@latest",
"dotnet-sdk@latest",
"opentofu@1.11.6"
],
"shell": {
"init_hook": [
"echo 'Welcome to devbox!' > /dev/null"
],
"scripts": {
"test": [
"echo \"Error: no test specified\" && exit 1"
]
}
}
}

View File

@@ -654,11 +654,21 @@ kubectl create secret generic myapp-credentials \
#### Step 2: Seal the Secret
Get the public certificate (one-time setup):
```bash
# Fetch public cert from cluster
kubeseal --fetch-cert \
--controller-name=sealed-secrets-controller \
--controller-namespace=kube-system \
> pub-cert.pem
```
Seal your secret:
```bash
kubeseal --format=yaml \
--namespace=myapp \
--cert=pub-cert.pem \
< private/myapp-credentials.yaml \
> secrets/myapp-credentials-sealed.yaml
```
@@ -701,7 +711,7 @@ kubectl create secret generic myapp-credentials \
# 2. Seal it
kubeseal --format=yaml \
--namespace=myapp \
--cert=pub-cert.pem \
< private/myapp-credentials.yaml \
> secrets/myapp-credentials-sealed.yaml

View File

@@ -12,11 +12,11 @@
## Overview
This Kubernetes cluster uses a **GitOps approach** powered by **ArgoCD**, where Git repositories serve as the single source of truth for both infrastructure and application deployments. The cluster setup is **cloud-agnostic**, with ready-to-use configurations for **UpCloud**, **AWS EKS**, **Azure AKS**, and **GCP GKE**.
This Kubernetes cluster uses a **GitOps approach** powered by **ArgoCD**, where Git repositories serve as the single source of truth for both infrastructure and application deployments. The cluster is running on **UpCloud Managed Kubernetes** but is designed to be cloud-agnostic.
### Key Characteristics
- **Environment**: Production (internal use only)
- **Cluster Type**: Multi-cloud, multi-cluster via Kustomize overlays (UpCloud, AWS, Azure, GCP)
- **Cluster Type**: Multi-cluster (upc-dev, upc-prod) via Kustomize overlays
- **GitOps Tool**: ArgoCD
- **Deployment Pattern**: App-of-Apps
- **Secret Management**: Sealed Secrets (kubeseal)
@@ -63,7 +63,7 @@ This Kubernetes cluster uses a **GitOps approach** powered by **ArgoCD**, where
┌────────────────────────────────┐
│ Kubernetes Clusters │
│ (UpCloud, AWS, Azure, GCP)
│ (UpCloud: upc-dev, upc-prod)
│ │
│ ┌──────────────────────────┐ │
│ │ ArgoCD │ │
@@ -120,47 +120,46 @@ launchpad/
├── _app-of-apps-upc-prod.yaml # Root ArgoCD Application (upc-prod cluster)
├── infra/ # Infrastructure ArgoCD Applications (Kustomize)
│ ├── base/ # Base Application manifests (one dir per component)
│ │ ├── kustomization.yaml # Aggregates all component subdirectories
│ │ ├── traefik-application/
│ │ ├── kustomization.yaml
│ │ │ └── traefik-application.yaml
│ │ ├── keycloak/
│ │ │ ├── kustomization.yaml
│ │ │ └── keycloak.yaml
│ │ ├── grafana/
│ │ ├── prometheus/
│ │ ── ... # Each component in its own subdirectory
│ └── secrets/
│ ├── overlays/ # Per-cluster Kustomize overrides
│ │ ── upc-dev/ # UpCloud Dev — includes all (resources: ../../base)
│ │ ├── upc-prod/ # UpCloud Prod — all + patches
│ │ ├── aks-dev/ # Azure AKS Dev — selective components
│ │ ├── aks-prod/ # Azure AKS Prod
│ │ ├── eks-dev/ # AWS EKS Dev
│ │ ├── eks-prod/ # AWS EKS Prod
│ │ ├── gke-dev/ # GCP GKE Dev
│ │ └── gke-prod/ # GCP GKE Prod
│ ├── base/ # Base Application manifests (upc-dev defaults)
│ │ ├── kustomization.yaml
│ │ ├── traefik-application.yaml
│ │ ├── keycloak.yaml
│ │ ── grafana.yaml
│ │ ├── gitea.yaml
│ │ ├── gitea-actions.yaml
│ │ ├── tempo.yaml
│ │ ├── renovate.yaml
│ │ ├── ... # All other Application manifests
│ │ ── secrets.yaml
├── overlays/ # Per-cluster overrides
│ ├── upc-dev/ # UpCloud Dev (uses base as-is)
│ │ ── upc-prod/ # UpCloud Prod (patches value paths)
│ ├── dashboards/ # Grafana dashboard ConfigMaps
│ └── values/ # Helm value overrides for infra
│ ├── base/ # Cloud-agnostic shared values
├── upc-{dev,prod}/ # UpCloud: storage class, LB, pricing
├── aws-{dev,prod}/ # AWS: gp3, NLB, CUR pricing
├── aks-{dev,prod}/ # Azure: managed-csi-premium, Standard LB
└── gcp-{dev,prod}/ # GCP: premium-rwo, L4 LB
│ ├── base/ # Shared values (all clusters)
│ ├── traefik-values.yaml
│ ├── keycloak-values.yaml
│ ├── grafana-values.yaml
│ ├── prometheus-values.yaml
│ │ ├── gitea-values.yaml
│ │ └── ...
│ ├── upc-dev/ # upc-dev cluster-specific values
│ │ ├── traefik-values.yaml
│ │ ├── keycloak-values.yaml
│ │ └── grafana-values.yaml
│ └── upc-prod/ # upc-prod cluster-specific values
│ ├── traefik-values.yaml
│ ├── keycloak-values.yaml
│ └── grafana-values.yaml
├── apps/ # Business Application ArgoCD manifests (Kustomize)
│ ├── base/ # One subdirectory per app
│ ├── base/ # Base app manifests
│ │ ├── kustomization.yaml
│ │ ├── musicman/
│ │ ── mcp10x/
│ │ ├── dot-ai-stack/
│ │ ├── ts-mcp/
│ │ └── argo-mcp/
│ │ ├── dot-ai-stack.yaml
│ │ ── ...
│ └── overlays/
│ ├── upc-dev/ # All apps (resources: ../../base)
── upc-prod/ # All apps + patches
│ └── aks-dev/ # Selective apps only
│ ├── upc-dev/ # Uses base as-is
── upc-prod/ # Patches value paths
├── cluster-resources/ # Cluster-wide Kubernetes resources
│ ├── ...
@@ -176,8 +175,6 @@ launchpad/
**Key Points**:
- `_app-of-apps-upc-dev.yaml` and `_app-of-apps-upc-prod.yaml` are the per-cluster root Applications
- Each component in `base/` has its own subdirectory with a `kustomization.yaml`
- Overlays can include **all** components (`resources: [../../base]`) or **cherry-pick** specific ones (`resources: [../../base/grafana, ../../base/prometheus]`)
- Kustomize overlays in `infra/overlays/` render base Applications with per-cluster patches
- Helm values are split: `values/base/` (shared) + `values/upc-dev/` or `values/upc-prod/` (cluster-specific)
- `apps/` follows the same base/overlays pattern for business applications
@@ -290,7 +287,7 @@ app-repository/
### The App-of-Apps Pattern
```
_app-of-apps-{cluster}.yaml (Root, per cluster — e.g. upc-dev, eks-prod, gke-dev)
_app-of-apps-{upc-dev,upc-prod}.yaml (Root, per cluster)
├── infrastructure-apps (manages infra/)
│ ├── cluster-resources-application
@@ -360,30 +357,16 @@ spec:
### Multi-Cluster Pattern
Kustomize overlays enable deploying the same Applications across clusters with different configurations.
Each component in `infra/base/` and `apps/base/` lives in its own subdirectory. Overlays define **which components to include** and optionally **patch** them:
Kustomize overlays enable deploying the same Applications across clusters with different configurations:
```yaml
# Option 1: Include ALL components (full cluster)
# infra/overlays/upc-dev/kustomization.yaml
resources:
- ../../base # Pulls in every component subdirectory
# infra/base/ contains default (upc-dev) Applications
# Helm values are layered: base + cluster-specific
valueFiles:
- $values/infra/values/base/traefik-values.yaml # Shared config
- $values/infra/values/upc-dev/traefik-values.yaml # Cluster-specific
# Option 2: Cherry-pick specific components (lightweight cluster)
# infra/overlays/aks-dev/kustomization.yaml
resources:
- ../../base/traefik-application
- ../../base/grafana
- ../../base/prometheus
- ../../base/loki
# Only listed components are deployed — others are excluded
```
Per-cluster patches swap Helm value file paths:
```yaml
# infra/overlays/upc-prod/kustomization.yaml
# infra/overlays/upc-prod/kustomization.yaml patches the second valueFile
patches:
- target:
kind: Application
@@ -394,15 +377,6 @@ patches:
value: $values/infra/values/upc-prod/traefik-values.yaml
```
Cloud-specific values (storage classes, load balancer annotations, cost model) are isolated in per-cluster value files. Base values are fully cloud-agnostic:
| Cloud | Storage Class | Load Balancer | OpenCost Provider |
|-------|--------------|---------------|-------------------|
| **UpCloud** | `upcloud-block-storage-maxiops` | UpCloud LB (ProxyProtocol v2) | Custom pricing |
| **AWS EKS** | `gp3` (EBS CSI) | NLB (ProxyProtocol v2) | AWS CUR |
| **Azure AKS** | `managed-csi-premium` | Standard LB (`externalTrafficPolicy: Local`) | Azure Billing API |
| **GCP GKE** | `premium-rwo` (PD CSI) | L4 passthrough NLB | GCP Cloud Billing |
**Benefits**:
- Single source of truth for Application definitions
- Cluster-specific values isolated per overlay
@@ -684,6 +658,6 @@ Notifications include:
---
**Last Updated**: 2026-04-22
**Last Updated**: 2026-03-16
**Maintained By**: Platform Team
**Questions?**: Contact #platform-support on Slack

View File

@@ -37,7 +37,7 @@ Bootstrap a new cluster from scratch:
#### Prerequisites
1. **Kubernetes cluster running** (UpCloud, AWS EKS, Azure AKS, GCP GKE, or any K8s cluster)
1. **Kubernetes cluster running** (UpCloud or any K8s cluster)
2. **kubectl configured** with admin access
3. **Repositories cloned** locally
@@ -54,13 +54,11 @@ kubectl get nodes
git clone https://git.forteapps.net/Forte/launchpad
cd launchpad
# 2. Run bootstrap script with cluster target
# Available clusters: upc-dev, upc-prod, eks-dev, eks-prod,
# aks-dev, aks-prod, gke-dev, gke-prod
./bootstrap.sh upc-dev
# 2. Set cluster name (optional)
export CLUSTER_NAME="prod-cluster-01"
# Cluster config is loaded from clusters/<cluster>.yaml
# (cloudProvider, trustedIPs, domain, etc.)
# 3. Run bootstrap script
./bootstrap.sh
```
**What Happens:**
@@ -1264,21 +1262,13 @@ spec:
### Backup Strategy
**Current State**: Gitea daily backups to S3-compatible storage
**Current State**: No automated backups
**What Is Backed Up**:
- ✅ Gitea repositories + database: Daily CronJob (`cluster-resources/gitea-backup-cronjob.yaml`) uploads to S3-compatible storage with 7-day retention
- ✅ Git repositories: Full cluster config recoverable from Git
- ⚠️ Secrets: Sealed secrets in Git; unseal keys need safekeeping
**What Is NOT Backed Up**:
- ❌ Cluster state (recreate via GitOps)
- ❌ Other persistent volumes (Prometheus, Loki, Tempo data)
**Per-cloud backup scripts** (manual restore helpers):
- UpCloud/AWS: `scripts/gitea-backup.sh` / `scripts/gitea-backup-eks.sh` (MinIO CLI, S3-compatible)
- Azure: `scripts/gitea-backup-aks.sh` (Azure CLI + Blob Storage)
- GCP: `scripts/gitea-backup-gke.sh` (gsutil + GCS)
**What Needs Backup**:
- ❌ Cluster state (not backed up - recreate via GitOps)
- ❌ Persistent volumes (currently not critical)
- ✅ Git repositories (Gitea provides backup)
- ⚠️ Secrets (sealed secrets in Git, unseal keys need safekeeping)
### Cluster Rebuild
@@ -1380,9 +1370,6 @@ kubectl get pods -n argocd
```bash
# UpCloud: Upgrade via control panel or CLI
# AWS EKS: eksctl upgrade cluster / AWS Console
# Azure AKS: az aks upgrade / Azure Portal
# GCP GKE: gcloud container clusters upgrade / Cloud Console
# After upgrade, verify cluster
kubectl version
@@ -1520,35 +1507,18 @@ git push
### Multi-Cluster Setup
The repository supports multiple clusters across multiple clouds via Kustomize overlays:
The repository supports multiple clusters via Kustomize overlays:
**Active clusters:**
- **upc-dev** (default): `infra/overlays/upc-dev/` — uses base Applications as-is
- **upc-prod**: `infra/overlays/upc-prod/` — patches value file paths from `upc-dev` to `upc-prod`
**Cloud-ready templates (fill in `clusters/*.yaml` before use):**
- **eks-dev** / **eks-prod**: AWS EKS with NLB, gp3 storage, AWS CUR pricing
- **aks-dev** / **aks-prod**: Azure AKS with Standard LB, managed-csi-premium storage
- **gke-dev** / **gke-prod**: GCP GKE with L4 LB, premium-rwo storage
Each cluster has its own:
- Root app-of-apps: `_app-of-apps-{cluster}.yaml`
- Cluster config: `clusters/{cluster}.yaml` (domain, trustedIPs, cloudProvider)
- Kustomize overlay: `infra/overlays/{cluster}/kustomization.yaml`
- Helm value overrides: `infra/values/{cluster}/` (traefik, gitea, opencost)
- Sealed secrets: `secrets/{cluster}/` (as needed)
- Apps overlay: `apps/overlays/{cluster}/`
- Root app-of-apps file: `_app-of-apps-upc-dev.yaml` / `_app-of-apps-upc-prod.yaml`
- Cluster-specific Helm values: `infra/values/upc-dev/` / `infra/values/upc-prod/`
- Sealed secrets: `secrets/upc-dev/` (others as needed)
- Apps overlay: `apps/overlays/upc-dev/` / `apps/overlays/upc-prod/`
Cloud-specific values handled per-cluster:
| Concern | UpCloud | AWS EKS | Azure AKS | GCP GKE |
|---------|---------|---------|-----------|---------|
| **Storage class** | `upcloud-block-storage-maxiops` | `gp3` | `managed-csi-premium` | `premium-rwo` |
| **Load balancer** | UpCloud LB + ProxyProtocol v2 | NLB + ProxyProtocol v2 | Standard LB + `externalTrafficPolicy: Local` | L4 passthrough NLB |
| **Cost monitoring** | Custom pricing | AWS CUR | Azure Billing API | GCP Cloud Billing |
| **Backup storage** | UpCloud S3-compat | AWS S3 (native) | Azure Blob Storage | GCS |
To add a new cluster, create a new overlay directory (e.g., `infra/overlays/eks-staging/`) with patches that swap the value file paths, and a matching `clusters/eks-staging.yaml`.
To add a new cluster, create a new overlay directory (e.g., `infra/overlays/upc-staging/`) with patches that swap the value file paths.
### Blue-Green Deployments
@@ -1691,6 +1661,6 @@ echo "Remember to delete: $SECRET_FILE"
---
**Last Updated**: 2026-04-22
**Last Updated**: 2026-03-16
**Maintained By**: Platform Team
**Emergency Contact**: #platform-support on Slack

View File

@@ -180,7 +180,7 @@ Reference for:
┌──────────────────────────────────────────────────────────────┐
│ Kubernetes Clusters (UpCloud, AWS, Azure, GCP)
│ Kubernetes Clusters (UpCloud: upc-dev, upc-prod)
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Infrastructure: Traefik, Cert-Manager, Kyverno │ │
│ ├──────────────────────────────────────────────────────┤ │
@@ -194,7 +194,7 @@ Reference for:
### Key Technologies
- **GitOps**: ArgoCD
- **Kubernetes**: Multi-cloud (UpCloud, AWS EKS, Azure AKS, GCP GKE)
- **Kubernetes**: UpCloud Managed Kubernetes (multi-cluster: upc-dev, upc-prod)
- **Ingress**: Traefik v2
- **Certificates**: Cert-Manager + Let's Encrypt
- **Policies**: Kyverno
@@ -299,16 +299,11 @@ docs/
## 🔄 Documentation Versions
**Current Version**: 1.0.0
**Last Updated**: 2026-04-22
**Last Updated**: 2026-03-16
**Maintained By**: Platform Team
### Changelog
- **v1.1.0 (2026-04-22)**: Multi-cloud support
- Cloud-agnostic base values (storage, LB, pricing moved to per-cluster overlays)
- Added AWS EKS, Azure AKS, GCP GKE configurations
- Per-cloud backup scripts
- Updated all documentation
- **v1.0.0 (2026-03-16)**: Initial comprehensive documentation release
- GitOps Architecture guide
- Developer Onboarding guide

View File

@@ -9,7 +9,6 @@
- [Kyverno Policies](#kyverno-policies)
- [Configuration Reference](#configuration-reference)
- [API Endpoints](#api-endpoints)
- [Cloud Overlay Pattern](#cloud-overlay-pattern)
- [Glossary](#glossary)
---
@@ -20,10 +19,9 @@
| Component | Value |
|-----------|-------|
| **Provider** | Multi-cloud (UpCloud, AWS EKS, Azure AKS, GCP GKE) |
| **Environment** | Dev + Production per cloud |
| **Active clusters** | UpCloud (upc-dev, upc-prod) |
| **Cloud-ready templates** | EKS, AKS, GKE (dev + prod each) |
| **Provider** | UpCloud Managed Kubernetes |
| **Environment** | Production (internal use) |
| **Cluster Count** | Multi-cluster (upc-dev, upc-prod) |
| **GitOps Tool** | ArgoCD |
| **Ingress Controller** | Traefik v2 |
| **Certificate Management** | Cert-Manager + Let's Encrypt |
@@ -44,7 +42,7 @@ Internet
[DNS: *.forteapps.net]
[Cloud Load Balancer]
[UpCloud LoadBalancer]
[Traefik Ingress Controller]
@@ -76,59 +74,40 @@ launchpad/
├── _app-of-apps-upc-dev.yaml # Root ArgoCD Application (upc-dev)
├── _app-of-apps-upc-prod.yaml # Root ArgoCD Application (upc-prod)
├── infra/ # Infrastructure applications (Kustomize)
│ ├── base/ # One subdirectory per component
│ ├── kustomization.yaml # Aggregates all component subdirectories
│ ├── traefik-application/
│ │ ├── kustomization.yaml
│ │ └── traefik-application.yaml
│ ├── keycloak/
│ │ ├── kustomization.yaml
│ │ └── keycloak.yaml
│ ├── grafana/
│ ├── prometheus/
│ ├── loki/
│ ├── tempo/
│ ├── gitea/
│ ├── opencost/
│ ├── ... # Each component in own directory
│ └── secrets/
│ ├── overlays/ # Per-cluster: include all or cherry-pick
│ │ ├── upc-dev/ # resources: [../../base] (all components)
│ │ ├── upc-prod/ # resources: [../../base] + patches
│ │ ├── aks-dev/ # resources: [../../base/grafana, ...] (selective)
│ │ └── .../ # 8 clusters total
├── infra/ # Infrastructure applications
│ ├── cluster-resources-application.yaml
│ ├── enterprise-apps.yaml
│ ├── traefik-application.yaml
├── cert-manager-application.yaml
├── kyverno.yaml
│ ├── kyverno-policies.yaml
├── prometheus.yaml
├── grafana.yaml
│ ├── loki.yaml
│ ├── tempo.yaml
│ ├── fluent-bit.yaml
│ ├── trivy.yaml
│ ├── gitea.yaml
│ ├── gitea-actions.yaml
│ ├── sealedsecrets.yaml
── secrets.yaml
│ ├── renovate.yaml
│ └── values/
│ ├── base/ # Cloud-agnostic Helm values
│ ├── gitea-values.yaml
│ ├── opencost-values.yaml
│ ├── prometheus-values.yaml
│ └── ...
│ ├── upc-dev/ # UpCloud dev overlay values
│ ├── traefik-values.yaml
│ ├── keycloak-values.yaml
│ ├── grafana-values.yaml
│ │ ├── gitea-values.yaml
│ │ └── opencost-values.yaml
│ └── upc-prod/ # UpCloud prod overlay values
│ ├── traefik-values.yaml
│ ├── keycloak-values.yaml
│ ├── grafana-values.yaml
│ ├── gitea-values.yaml
│ └── opencost-values.yaml
│ ├── argocd-values.yaml
├── prometheus-values.yaml
├── grafana-values.yaml
├── loki-values.yaml
├── tempo-values.yaml
│ ├── gitea-values.yaml
├── gitea-actions-values.yaml
├── fluent-bit-values.yaml
└── renovate-values.yaml
├── apps/ # Business applications (Kustomize)
│ ├── base/ # One subdirectory per app
│ ├── kustomization.yaml
│ ├── musicman/
│ ├── mcp10x/
│ │ ├── dot-ai-stack/
│ │ ├── ts-mcp/
│ │ └── argo-mcp/
│ └── overlays/ # Per-cluster: include all or cherry-pick
│ ├── upc-dev/
│ ├── upc-prod/
│ └── aks-dev/ # Selective apps only
├── apps/ # Business applications
│ ├── mcp10x.yaml
│ ├── musicman.yaml
│ ├── dot-ai-stack.yaml
└── argo-mcp.yaml
├── cluster-resources/ # Cluster-level resources
│ ├── cert-manager-namespace.yaml
@@ -149,39 +128,12 @@ launchpad/
│ └── auth-sidecar-injector.yaml
├── secrets/ # Application secrets (sealed)
│ ├── base/ # All SealedSecrets (shared across clouds)
│ ├── kustomization.yaml
│ ├── argocd-forte-helm-secret-sealed.yaml
│ ├── argocd-mcp-credentials.yaml
│ ├── argocdmcp-auth-oidc-sealed.yaml
│ ├── dot-ai-secrets.yaml
│ │ ├── forte10x-app-credentials-sealed.yaml
│ │ ├── gitea-backup-s3-sealed.yaml
│ │ ├── gitea-credentials-sealed.yaml
│ │ ├── gitea-runner-token-sealed.yaml
│ │ ├── gitea-smtp-secret-sealed.yaml
│ │ ├── keycloak-credentials-sealed.yaml
│ │ ├── musicman-auth-oidc-sealed.yaml
│ │ ├── musicman-credentials.yaml
│ │ └── renovate-env-sealed.yaml
│ └── overlays/ # Per-cloud overlays (reference base)
│ ├── aks-dev/kustomization.yaml
│ ├── aks-prod/kustomization.yaml
│ ├── eks-dev/kustomization.yaml
│ ├── eks-prod/kustomization.yaml
│ ├── gke-dev/kustomization.yaml
│ ├── gke-prod/kustomization.yaml
│ ├── upc-dev/kustomization.yaml
│ └── upc-prod/kustomization.yaml
├── scripts/ # Operational helper scripts
│ ├── gitea-backup.sh # S3 backup helper (list/download)
│ ├── gitea-restore.sh
│ └── backup/ # Per-cloud backup reference scripts
│ ├── s3-minio.sh # S3-compatible (UpCloud, MinIO, Wasabi)
│ ├── aws-s3.sh # Native AWS S3
│ ├── azure-blob.sh # Azure Blob Storage
│ └── gcp-gcs.sh # GCP Cloud Storage
│ ├── argocd-mcp-credentials.yaml
│ ├── dot-ai-secrets.yaml
│ ├── gitea-credentials-sealed.yaml
│ ├── gitea-runner-token-sealed.yaml
│ ├── mcp10x-credentials-sealed.yaml
└── musicman-credentials.yaml
├── private/ # Local-only (Git-ignored)
│ ├── *.yaml
@@ -656,128 +608,13 @@ retry:
|---------|-------|---------|
| `application.resourceTrackingMethod` | `annotation` | Track resources via annotations |
| `timeout.reconciliation` | `60s` | Reconciliation interval |
| `admin.enabled` | `false` | Admin login disabled (SSO-only) |
| `url` | `https://argocd.forteapps.net` | External URL for ArgoCD UI |
**Git Submodule Disable**: Set via `configs.params` (NOT `repoServer.env` — that causes strategic merge conflicts with chart's `valueFrom` entries):
```yaml
configs:
params:
"reposerver.enable.git.submodule": "false"
```
This writes to `argocd-cmd-params-cm` ConfigMap, which the chart already reads via `valueFrom`. Submodules (e.g., `shared-prompts`) are not needed for K8s manifest generation.
**Break-Glass Admin Access**: Admin login is disabled (`admin.enabled: false`). The admin password remains in `argocd-secret`. To re-enable temporarily:
```bash
# Enable admin login
kubectl patch cm argocd-cm -n argocd -p '{"data":{"admin.enabled":"true"}}'
# Log in as admin, do what's needed, then disable again
kubectl patch cm argocd-cm -n argocd -p '{"data":{"admin.enabled":"false"}}'
```
ArgoCD picks up ConfigMap changes within the reconciliation timeout (60s). Note: ArgoCD will revert this on next sync — this is intentional (temporary access only).
**OIDC Authentication** (Keycloak):
```yaml
configs:
cm:
oidc.config: |
name: Forte SSO
issuer: https://id.forteapps.net/realms/forte
clientID: argocd
clientSecret: $oidc.clientSecret
requestedScopes: ["openid", "email", "profile"]
rbacConfig:
policy.csv: |
g, ArgoCD Admins, role:admin
g, ArgoCD Viewers, role:readonly
# Deny users not in any declared KC group
policy.default: ""
scopes: '[groups]'
```
**Access Control**: Only users in Keycloak groups `ArgoCD Admins` or `ArgoCD Viewers` can access ArgoCD. Users not in either group are denied (empty `policy.default`). Assign users to groups in Keycloak admin console.
- ArgoCD does NOT add `openid` implicitly — must include in `requestedScopes`
- Do NOT add `groups` as a scope — the KC groups mapper emits the claim regardless
- `$oidc.clientSecret` references the `oidc.clientSecret` key in `argocd-secret`
- OIDC secret is synced by CronJob `argocd-oidc-sync` (see `cluster-resources/argocd-oidc-secret-sync.yaml`)
- The CronJob bridges `argocd-oidc-credentials` (from KC registrar) → `argocd-secret` every 2 min
- Safe for fresh deploys: no-ops if source secret doesn't exist yet
**Ingress** (Traefik + TLS):
```yaml
server:
ingress:
enabled: true
ingressClassName: traefik
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
tls: true
extraArgs:
- --insecure
configs:
params:
"server.insecure": true
```
TLS terminates at Traefik; ArgoCD runs in insecure mode behind the proxy.
| `admin.enabled` | `true` | Enable admin account |
| `git.submodule.enabled` | `false` | Disable git submodule checkout — submodules are not needed for manifest generation |
---
## Infrastructure Components
### Homepage (Platform Dashboard)
**Chart**: `jameswynn/homepage`
**Namespace**: `homepage`
**URL**: `https://start.forteapps.net`
Platform dashboard that auto-discovers deployed apps via Kubernetes service annotations.
**Discovery mechanism**: Services annotated with `gethomepage.dev/enabled: "true"` appear in the dashboard. Apps not deployed = annotations absent = not shown. Fully dynamic per environment.
**Annotated services**:
| Service | Namespace | Group | Widget |
|---------|-----------|-------|--------|
| `gitea-http` | `gitea` | DevOps | `gitea` |
| `argocd-server` | `argocd` | DevOps | `argocd` |
| `keycloak` | `keycloak` | Identity | none |
| `grafana` | `monitoring` | Monitoring | `grafana` |
| `karpor-server` | `karpor` | DevOps | none |
**Adding a new app**: Annotate the app's Service in its Helm values:
```yaml
service:
annotations:
gethomepage.dev/enabled: "true"
gethomepage.dev/name: "My App"
gethomepage.dev/description: "What it does"
gethomepage.dev/group: "GroupName"
gethomepage.dev/icon: "icon-name" # https://github.com/walkxcode/dashboard-icons
gethomepage.dev/href: "https://myapp.forteapps.net"
# Optional live widget:
gethomepage.dev/widget.type: "myapp"
gethomepage.dev/widget.url: "https://myapp.forteapps.net"
# gethomepage.dev/widget.key: "{{HOMEPAGE_VAR_MYAPP_TOKEN}}"
```
**Widget API credentials**: Inject via env vars into the Homepage pod:
```yaml
# In homepage-values.yaml per environment
env:
- name: HOMEPAGE_VAR_GRAFANA_TOKEN
valueFrom:
secretKeyRef:
name: homepage-widget-credentials
key: grafana-token
```
Then reference as `gethomepage.dev/widget.key: "{{HOMEPAGE_VAR_GRAFANA_TOKEN}}"`.
**Values files**:
- `infra/values/base/homepage-values.yaml` — RBAC, kubernetes mode, layout
- `infra/values/{env}/homepage-values.yaml` — hostname per environment
---
### Traefik
**Chart**: `traefik/traefik`
@@ -849,10 +686,6 @@ spec:
**Chart**: `sealed-secrets/sealed-secrets-controller`
**Namespace**: `kube-system`
**Directory Structure**: `secrets/base/` contains all SealedSecrets with a `kustomization.yaml`. Per-cloud overlays in `secrets/overlays/<cloud>/` reference the base via Kustomize. The ArgoCD `secrets` Application points to the active overlay (e.g., `secrets/overlays/upc-dev`), and `infra/overlays/upc-prod` patches the path to `secrets/overlays/upc-prod`.
To add cloud-specific secrets, create a new SealedSecret in the overlay directory and add it to the overlay's `kustomization.yaml`.
**Public Certificate**:
```bash
kubeseal --fetch-cert \
@@ -893,15 +726,6 @@ kubeStateMetrics:
- Loki
- Tempo
**Ingress**: Exposed via Traefik at `https://grafana.forteapps.net` with cert-manager TLS.
**OIDC Authentication** (Keycloak):
- Uses `grafana.ini.auth.generic_oauth` with KC `grafana` client
- Secret `grafana-oidc-credentials` synced by KC registrar, loaded via `envFromSecrets`
- SSO-only mode: `auth.disable_login_form: true` + `auth.generic_oauth.auto_login: true`
- Role mapping via JMESPath on `resource_access.grafana.roles` claim (requires KC client role mapper)
- Roles: KC client roles `Admin`/`Editor` map to Grafana roles; default is `Viewer`
### Loki
**Chart**: `grafana/loki-stack`
@@ -1063,52 +887,6 @@ dind:
- Gitea admin panel (`/admin/runners`) — runners show as Online
- Create test workflow in `.gitea/workflows/test.yml` — job executes
### Vaultwarden
**Chart**: `guerzon/vaultwarden`
**Version**: 0.36.4 (app v1.36.0-alpine)
**Namespace**: `vaultwarden`
**Purpose**: Self-hosted Bitwarden-compatible password manager.
**Configuration**:
```yaml
# infra/overlays/upc-dev/vaultwarden/ + infra/values/
domain: "https://bitwarden.forteapps.net"
ingress:
enabled: true
class: "traefik"
tls: true
tlsSecret: vaultwarden-tls
hostname: bitwarden.forteapps.net
additionalAnnotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
database:
type: postgresql
host: vaultwarden-postgresql # StatefulSet in overlay
existingSecret: prod-db-creds
storage:
data: 5Gi (ReadWriteOnce)
attachments: 5Gi (ReadWriteOnce)
```
**TLS**: cert-manager auto-provisions Let's Encrypt certificate via `letsencrypt-prod` ClusterIssuer (same pattern as Gitea, Grafana, etc).
**SSO**: Keycloak OIDC via `forte` realm (client ID: `vaultwarden`). Self-service client config Secret (`keycloak-client-vaultwarden`) triggers registrar to create KC client and sync credentials to `vaultwarden-oidc-credentials`. PKCE enabled.
**Endpoints**:
- Web UI: `https://bitwarden.forteapps.net`
**Database**: Separate ArgoCD Application `vaultwarden-postgresql` (sync-wave `"0"`) deploys PostgreSQL 16 StatefulSet + SealedSecret before Vaultwarden (wave `"1"`). 2Gi PVC. Chart does NOT include a PostgreSQL subchart — must be provisioned separately.
**Secrets**:
- `prod-db-creds` (SealedSecret) — PostgreSQL credentials (`pgusername`, `pgpassword`) + SMTP credentials
- `vaultwarden-oidc-credentials` (registrar-managed) — OIDC client ID + secret
- `vaultwarden-tls` — auto-managed by cert-manager
### AI Code Review (ai-review)
**Type**: Gitea Actions workflow (`.gitea/workflows/ai-review.yaml`)
@@ -1187,29 +965,82 @@ ignore:
- Check Gitea Actions tab for workflow run status and logs
- Monitor Anthropic usage dashboard for token consumption
### Keycloak Browser Flow (IdP Auto-Redirect)
### Backstage / RHDH (Developer Portal)
**File**: `infra/values/base/keycloak-values.yaml` (inside `forte-realm.json`)
**Chart**: `backstage` (RHDH — Red Hat Developer Hub)
**Version**: `5.8.0`
**Namespace**: `backstage`
**Helm Repo**: `https://redhat-developer.github.io/rhdh-chart`
**Image**: `quay.io/rhdh-community/rhdh:next`
The realm uses a custom browser authentication flow (`browser-auto-idp`) that skips the Keycloak login page and redirects directly to the Entra ID identity provider.
**Purpose**: Internal developer portal where teams register and broadcast themselves, their applications, APIs, and systems. Provides a unified catalog, templates, and documentation hub.
**Flow executions**:
**Why RHDH over vanilla Backstage**: Ships 27+ plugins pre-bundled (ArgoCD, Kubernetes, Keycloak, GitHub, GitLab, Jira, SonarQube, Tekton, Jenkins, Quay, and more). Supports dynamic plugin installation at runtime — no image rebuilds needed.
| Priority | Authenticator | Requirement | Purpose |
|----------|--------------|-------------|---------|
| 10 | `auth-cookie` | ALTERNATIVE | Reuse existing session (no redirect) |
| 20 | `identity-provider-redirector` | ALTERNATIVE | Auto-redirect to `forte-entra` IdP |
**Configuration** (`infra/values/base/backstage-values.yaml`):
- OpenShift Route disabled (`route.enabled: false`) — uses Traefik ingress instead
- PostgreSQL subchart enabled for persistence (2Gi)
- SecurityContext configured for vanilla Kubernetes (non-OpenShift)
- Traefik ingress with `websecure` entrypoint
- App title: "Forte Developer Portal"
- Dynamic plugins: loads `dynamic-plugins.default.yaml` (all 27+ bundled plugins)
- Catalog rules: Component, System, API, Resource, Location, Template, Group, User, Domain
**Key fields in realm JSON**:
- `"browserFlow": "browser-auto-idp"` — overrides the default `browser` flow at realm level
- `"authenticationFlows"` — defines the custom flow with its executions
- `"authenticatorConfig"` — sets `defaultProvider: "forte-entra"` on the redirector
**Authentication** (Keycloak OIDC):
- Uses the self-service registrar pattern (see [Keycloak Client Registrar](#keycloak-client-registrar))
- Config Secret: `cluster-resources/backstage-keycloak-client-config.yaml`
- Kyverno clones it → registrar creates `backstage-oidc-credentials` Secret in `backstage` namespace
- Credential keys: `AUTH_OIDC_CLIENT_ID`, `AUTH_OIDC_CLIENT_SECRET` (loaded via `extraEnvVarsSecrets`)
- Redirect URI: `https://backstage.forteapps.net/api/auth/oidc/handler/frame`
- Sign-in resolver: `emailMatchingUserEntityProfileEmail`
**Why custom flow**: The default KC browser flow shows a username/password form with an IdP button. Since all authentication is via Entra ID, the custom flow eliminates this step. The `auth-cookie` execution preserves session reuse so returning users aren't redirected again.
**Catalog Discovery** (Gitea):
- Auto-discovers `catalog-info.yaml` from all repos in the `Forte` organization
- Scans every 30 minutes via the Gitea catalog provider plugin
- Gitea SCM integration configured for URL resolution (`git.forteapps.net`)
**Important**: The `forte-entra` identity provider must exist in Keycloak (currently configured manually in the KC admin console). If the IdP alias changes, update the `defaultProvider` value in the realm JSON.
**Catalog Registration**:
Teams register services by adding a `catalog-info.yaml` to their repo root:
```yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: my-service
description: My service description
annotations:
backstage.io/source-location: url:https://git.forteapps.net/Forte/my-service
spec:
type: service
lifecycle: production
owner: team-name
```
---
Repos with this file are auto-discovered — no manual registration needed.
**Dynamic Plugins**:
Add plugins at runtime via `global.dynamic.plugins` in values — no image rebuild:
```yaml
global:
dynamic:
plugins:
- package: "@scope/my-plugin@1.0.0"
integrity: "sha512-..."
```
**Per-cluster Configuration** (`infra/values/upc-dev/backstage-values.yaml`):
```yaml
global:
host: backstage.forteapps.net
upstream:
backstage:
appConfig:
app:
baseUrl: https://backstage.forteapps.net
backend:
baseUrl: https://backstage.forteapps.net
ingress:
host: backstage.forteapps.net
```
### Keycloak Client Registrar
@@ -1324,33 +1155,6 @@ kubectl get secret keycloak-client-<app> -n keycloak -o jsonpath='{.metadata.ann
**See**: [Developer Guide - Adding a New Keycloak Client](DEVELOPER-GUIDE.md#adding-a-new-keycloak-client)
### Karpor
**Chart**: `karpor` from `https://kusionstack.github.io/charts`
**Version**: 0.7.6 (app v0.6.4)
**Namespace**: `karpor`
**Sync Wave**: 1
**Purpose**: Kubernetes visualization and intelligence tool. Provides cross-cluster resource search, compliance checking, and topology visualization. Gives platform engineers a unified view of all cluster resources and their relationships.
**Architecture** (4 components):
- **Server** — main Karpor API/UI (port 7443)
- **Syncer** — syncs cluster state into the search index
- **ElasticSearch** — search backend for resource indexing
- **etcd** — persistent key-value store (10Gi PVC)
**Configuration** (`infra/values/base/karpor-values.yaml`):
- `namespaceEnabled: false` — ArgoCD manages namespace creation
- Default resource limits tuned for small clusters
- ElasticSearch: 2 CPU / 4Gi memory (the heaviest component)
- AI features available but not enabled (requires `server.ai.authToken` + backend config)
**Access**: Port-forward to reach the UI:
```bash
kubectl port-forward svc/karpor-release-server -n karpor 7443:7443
# Open https://localhost:7443
```
### Renovate
**Chart**: `renovate` (OCI: `ghcr.io/renovatebot/charts`)
@@ -1848,22 +1652,14 @@ Recommended resource allocation:
### Storage Classes
Storage classes are cloud-specific and configured in per-cluster value overrides (`infra/values/{cluster}/gitea-values.yaml`):
| Cloud | Storage Class | Driver |
|-------|--------------|--------|
| **UpCloud** | `upcloud-block-storage-maxiops` | UpCloud CSI |
| **AWS EKS** | `gp3` | EBS CSI |
| **Azure AKS** | `managed-csi-premium` | Azure Disk CSI |
| **GCP GKE** | `premium-rwo` | PD CSI |
Default storage class used: **UpCloud default** (varies by provider)
```yaml
# Example: base values omit storageClass (set in per-cluster overlay)
persistence:
enabled: true
storageClass: "" # Uses default
accessMode: ReadWriteOnce
size: 5Gi
# storageClass set by infra/values/{cluster}/gitea-values.yaml
```
---
@@ -1927,88 +1723,6 @@ POST /loki/api/v1/push
---
## Cloud Overlay Pattern
### Overview
Cloud-specific configuration (StorageClass, LoadBalancer annotations, pricing models, etc.) lives in per-cloud overlay value files, **not** in `base/`. Adding a new cloud provider only requires a new overlay directory — no base changes.
### Supported Clouds
| Cloud | Dev overlay | Prod overlay | StorageClass | LB type |
|-------|-----------|-------------|-------------|---------|
| **UpCloud** | `upc-dev` | `upc-prod` | `upcloud-block-storage-maxiops` | UpCloud LB (proxy protocol v2) |
| **Azure AKS** | `aks-dev` | `aks-prod` | `managed-csi-premium` | Azure LB |
| **AWS EKS** | `eks-dev` | `eks-prod` | `gp3` | AWS NLB (proxy protocol) |
| **GCP GKE** | `gke-dev` | `gke-prod` | `premium-rwo` | GCP NEG |
Bootstrap any cluster with: `./bootstrap.sh <cluster>` (e.g., `./bootstrap.sh aks-dev`)
### How It Works
Each ArgoCD Application uses **multi-source Helm values** with two value files:
```yaml
# infra/base/gitea.yaml (example)
helm:
valueFiles:
- $values/infra/values/base/gitea-values.yaml # [0] cloud-agnostic
- $values/infra/values/upc-dev/gitea-values.yaml # [1] cloud-specific (default: upc-dev)
```
The `upc-prod` Kustomize overlay patches index `[1]` to swap the cloud-specific file:
```yaml
# infra/overlays/upc-prod/kustomization.yaml
- target:
kind: Application
name: gitea
patch: |
- op: replace
path: /spec/sources/0/helm/valueFiles/1
value: $values/infra/values/upc-prod/gitea-values.yaml
```
### Components Using Cloud Overlays
| Component | Cloud-specific config | Overlay value file |
|-----------|----------------------|-------------------|
| **Traefik** | LB annotations, proxy protocol IPs | `traefik-values.yaml` |
| **Keycloak** | Hostname, TLS settings | `keycloak-values.yaml` |
| **Grafana** | Hostname, datasource URLs | `grafana-values.yaml` |
| **Gitea** | StorageClass (persistence + PostgreSQL) | `gitea-values.yaml` |
| **OpenCost** | Custom pricing model (CPU/RAM/storage rates) | `opencost-values.yaml` |
### Backup CronJob
The `gitea-backup` CronJob uses a generic `s3` alias for `minio/mc`. The actual endpoint and credentials come from the `gitea-backup-s3` Sealed Secret, which is per-cloud. Reference scripts for different cloud providers are in `scripts/backup/`:
| Script | Provider | Tool |
|--------|----------|------|
| `s3-minio.sh` | S3-compatible (UpCloud, MinIO, Wasabi) | `minio/mc` |
| `aws-s3.sh` | AWS S3 | `aws` CLI |
| `azure-blob.sh` | Azure Blob Storage | `az` CLI |
| `gcp-gcs.sh` | GCP Cloud Storage | `gsutil` |
### Adding a New Cloud Provider
To add support for a new cloud (e.g., `oci-dev` for Oracle Cloud):
1. **Cluster config**: `clusters/oci-dev.yaml` — clusterName, domain, trustedIPs, cloudProvider
2. **Overlay value files** in `infra/values/oci-dev/`:
- `traefik-values.yaml` — LB annotations, proxy protocol config
- `keycloak-values.yaml` — hostname
- `grafana-values.yaml` — hostname
- `gitea-values.yaml``storageClass` for persistence + PostgreSQL
- `opencost-values.yaml` — pricing model or cloud billing integration
3. **Kustomize overlay**: `infra/overlays/oci-dev/kustomization.yaml` — patch `valueFiles[1]` for each Application
4. **App-of-apps**: `_app-of-apps-oci-dev.yaml` — points to `infra/overlays/oci-dev`
5. **Secrets overlay**: `secrets/overlays/oci-dev/kustomization.yaml` — references `../../base`, add cloud-specific SealedSecrets if needed
6. **Secrets patch**: Add patch to `infra/overlays/oci-dev/kustomization.yaml` to swap secrets path to `secrets/overlays/oci-dev`
7. **Bootstrap**: `./bootstrap.sh oci-dev`
---
## Glossary
### Terms
@@ -2141,6 +1855,6 @@ team: platform
---
**Last Updated**: 2026-04-22
**Last Updated**: 2026-04-16
**Maintained By**: Platform Team
**Version**: 1.0.0

View File

@@ -1,12 +1,12 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: karpor
name: backstage
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "1"
labels:
app.kubernetes.io/name: karpor
app.kubernetes.io/name: backstage
app.kubernetes.io/part-of: developer-portal
app.kubernetes.io/managed-by: argocd
finalizers:
@@ -15,13 +15,14 @@ spec:
project: default
sources:
- repoURL: https://kusionstack.github.io/charts
chart: karpor
targetRevision: "0.7.6"
- repoURL: https://redhat-developer.github.io/rhdh-chart
chart: backstage
targetRevision: "5.8.0"
helm:
releaseName: karpor
releaseName: backstage
valueFiles:
- $values/infra/values/base/karpor-values.yaml
- $values/infra/values/base/backstage-values.yaml
- $values/infra/values/upc-dev/backstage-values.yaml
- repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
targetRevision: HEAD
@@ -29,7 +30,7 @@ spec:
destination:
server: https://kubernetes.default.svc
namespace: karpor
namespace: backstage
syncPolicy:
automated:
@@ -40,9 +41,3 @@ spec:
- CreateNamespace=true
- Validate=true
- ServerSideApply=true
ignoreDifferences:
- group: apps
kind: StatefulSet
jsonPointers:
- /spec/volumeClaimTemplates

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cert-manager-application.yaml

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cluster-resources-application.yaml

View File

@@ -1,42 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: databunker
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "1"
labels:
app.kubernetes.io/name: databunker
app.kubernetes.io/part-of: identity
app.kubernetes.io/managed-by: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
sources:
- repoURL: https://securitybunker.github.io/databunkerpro-setup
chart: databunkerpro
targetRevision: "0.1.0"
helm:
releaseName: databunkerpro
valueFiles:
- $values/infra/values/base/databunker-values.yaml
- repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
targetRevision: HEAD
ref: values
destination:
server: https://kubernetes.default.svc
namespace: databunker
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- CreateNamespace=true
- Validate=true
- ServerSideApply=true

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- databunker.yaml

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- enterprise-apps.yaml

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- fluent-bit.yaml

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gitea-actions.yaml

View File

@@ -22,7 +22,6 @@ spec:
releaseName: gitea
valueFiles:
- $values/infra/values/base/gitea-values.yaml
- $values/infra/values/upc-dev/gitea-values.yaml
- repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
targetRevision: HEAD

View File

@@ -1,8 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gitea.yaml
- gitea-backup-s3-sealed.yaml
- gitea-credentials-sealed.yaml
- gitea-runner-token-sealed.yaml
- gitea-smtp-secret-sealed.yaml

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- grafana-dashboards.yaml

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- grafana.yaml

View File

@@ -1,21 +0,0 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: homepage-services-reader
rules:
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: homepage-services-reader
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: homepage-services-reader
subjects:
- kind: ServiceAccount
name: homepage
namespace: homepage

View File

@@ -1,16 +0,0 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: homepage-widget-credentials
namespace: homepage
spec:
encryptedData:
HOMEPAGE_VAR_GITEA_TOKEN: AgAVN1C931EQpn+sodr3CpjlhORfJVTW8aUr+pGZQb+65Pb8QLGeVGVa7Jv60gDJUX3r+93/jMrEbCOeDL6I4qCz/V35wMCxFZLnXIdkmto0W4MKt6cK8To1/OP7EhQJOGBlSuOFsrwoy+HDtvLIqmyF0nrxhTusm9/NHrw+gCVwSTPhiAX1MCuSOSRWpbXvyNphW8j7aqUaV6ixDt424Fe4alEIShYELcS3EX/VPgsf2p2bhvBRCQOh3LEprkuxSFMuPfCBk06TPTbIN4saNVm0Ke0zW/pxkVNSiIxEnKjOmpPJtacsfWN7du+nQbx276G2qvWrf+iawJVq0Z/SLikA/NUFBL6EjSRfgE3cSOri8sbxsd0AycsFGyp98EM29wE+WOQl52M/lwl02EmCivqkICSO7Jp9pM1ScbmRMa5vcnupsGbVDxhRKLqxhAskt/BXDkRzvHN31gH3YmelES3JuqNMHV0urFxmX2oOX9Pxbtv63csc+zhy1Ui5aoex7TPnLdk7kYLSAE2MSrzT6wHvVhBC5kNnDYVrLehvJrT+eNh0MOLx2wkuJmIOxRAGUyNi5DfDnP6qnvj2aefEymLuOXAIUXH8DbeBtrjsd74HX2hhIfBlPkXvhJR3ks7i5RXjK2/YYHkgJ+nJoW80S9N7ciaRy103g74TNJZt6QzzL5Vb80qZ6yQOD4G081KmTLDmhHjJVIIv9M3nLh2s0IeBV3/Z5qHZmtjN7sSaKAn4MIr5FaH9quhx
HOMEPAGE_VAR_GRAFANA_TOKEN: AgBloBlOlP+R/4VizE1CGpj0wyiwU14BemAnuUpld7OvOGc67dwfDPyponkQXjAZg3UU2cZ70A51WUAuVlAr+25Ktlf/FW2OBqj+1BJOCqMMyu+kv026yjX2aB8dKGzlTxgF8aji+j1mC8vP3vvmgI4Zf2HQAH7uFwLfeo8+QnV5EyhcExSS0xDne+VtOP9jNXbPRayry0DdyRVtaeKAiZacO+45oAJWszWOwmoMTg9FZQkLjER6Q0tyI6NnoNObsFCnh56chZTdzBOYtmPnwld1bP2FjoJDqn8AfRwbPTIj7t0eFP7WLUO7GQKpxVl+pFwJLb5xCOw2+HNtp1BhNCu7icuc0P88IlvwzkbN0lXJbYigVOzyjEo8f/al1DXPM4WaB/Nqmr7Mtt8KTRh2WMVTgiX5jsu25D0rGDvY9gqfBBqswkRhCLsG0v0EN32zXj1/52KYdmB7pk/+2lMwSaGMS11MOenHeU1Z95fGxm9f3EGF0E8xlFr4FowgsNwr+tJQqpM0bT/4mZnaQbGWtKPFizMtsfQFm+rHFcNCrGaOuecslmiIJs8lTm18KlrncsGfxNS64tVXk+LvydU0rwybvpg2rQjEWtAl1IQsaaiz96OAlYxxK1MGxN7KE6F8R4kfnWTZ5Fs1KMmd/DOIVBXyCbqXxk8pbekmaIeNSfv92JNZ0QNJWsBa2vgQ24WI2pb4XiR0BvtLpt3BVlZUcSK92SzUblWmYWVMwHYCJkEeEUV1PhYEmyiN+V/Kq5Qb
template:
metadata:
creationTimestamp: null
name: homepage-widget-credentials
namespace: homepage

View File

@@ -1,43 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: homepage
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "3"
labels:
app.kubernetes.io/name: homepage
app.kubernetes.io/part-of: platform
app.kubernetes.io/managed-by: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
sources:
- repoURL: https://jameswynn.github.io/helm-charts
chart: homepage
targetRevision: "2.1.0"
helm:
releaseName: homepage
valueFiles:
- $values/infra/values/base/homepage-values.yaml
- $values/infra/values/upc-dev/homepage-values.yaml
- repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
targetRevision: HEAD
ref: values
destination:
server: https://kubernetes.default.svc
namespace: homepage
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- CreateNamespace=true
- Validate=true
- ServerSideApply=true

View File

@@ -1,6 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- homepage.yaml
- homepage-widget-credentials-sealed.yaml
- homepage-extra-rbac.yaml

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- karpor.yaml

View File

@@ -15,7 +15,7 @@ spec:
project: default
sources:
- repoURL: registry-1.docker.io/bitnamicharts
- repoURL: https://charts.bitnami.com/bitnami
chart: keycloak
targetRevision: "25.2.0"
helm:
@@ -43,7 +43,7 @@ spec:
- ServerSideApply=true
ignoreDifferences:
- group: apps
kind: StatefulSet
- group: batch
kind: CronJob
jsonPointers:
- /spec/volumeClaimTemplates
- /spec/jobTemplate/spec/template/spec/containers/0/args

View File

@@ -1,5 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- keycloak.yaml
- keycloak-credentials-sealed.yaml

View File

@@ -1,25 +1,25 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- traefik-application
- keycloak
- grafana
- cert-manager-application
- kyverno
- sealedsecrets
- prometheus
- loki
- fluent-bit
- enterprise-apps
- cluster-resources-application
- kyverno-policies
- gitea
- gitea-actions
- opencost
- renovate
- tempo
- grafana-dashboards
- karpor
- databunker
- homepage
- vault
- traefik-application.yaml
- keycloak.yaml
- grafana.yaml
- cert-manager-application.yaml
- kyverno.yaml
- sealedsecrets.yaml
- prometheus.yaml
- loki.yaml
- fluent-bit.yaml
- trivy.yaml
- enterprise-apps.yaml
- cluster-resources-application.yaml
- kyverno-policies.yaml
- secrets.yaml
- gitea.yaml
- gitea-actions.yaml
- opencost.yaml
- renovate.yaml
- tempo.yaml
- grafana-dashboards.yaml
- network-policies-application.yaml
- backstage.yaml

View File

@@ -27,6 +27,7 @@ spec:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- CreateNamespace=true
- Validate=true

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- kyverno-policies.yaml

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- kyverno.yaml

View File

@@ -40,9 +40,3 @@ spec:
- CreateNamespace=true
- Validate=true
- ServerSideApply=true
ignoreDifferences:
- group: apps
kind: StatefulSet
jsonPointers:
- /spec/volumeClaimTemplates

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- loki.yaml

View File

@@ -1,32 +1,33 @@
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
annotations:
argocd.argoproj.io/sync-wave: "-1"
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: infrastructure-apps
name: network-policies
namespace: argocd
labels:
app.kubernetes.io/name: infrastructure-apps
app.kubernetes.io/name: network-policies
app.kubernetes.io/part-of: platform
app.kubernetes.io/managed-by: argocd
annotations:
argocd.argoproj.io/sync-wave: "1"
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
targetRevision: HEAD
path: infra/overlays/gke-dev
path: cluster-resources/network
destination:
server: https://kubernetes.default.svc
namespace: default
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- CreateNamespace=true
- Validate=true
- ServerSideApply=true

View File

@@ -22,9 +22,8 @@ spec:
releaseName: opencost
valueFiles:
- $values/infra/values/base/opencost-values.yaml
- $values/infra/values/upc-dev/opencost-values.yaml
- repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git
- repoURL: git@github.com:fortedigital/sturdy-adventure.git
targetRevision: HEAD
ref: values

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- opencost.yaml

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- prometheus.yaml

View File

@@ -1,5 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- renovate.yaml
- renovate-env-sealed.yaml

View File

@@ -1,5 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- sealedsecrets.yaml
- argocd-forte-helm-secret-sealed.yaml

Some files were not shown because too many files have changed in this diff Show More