client secret bootstrapping

docs/DEVELOPER-GUIDE.md

@@ -9,6 +9,7 @@
 - [Updating an Existing Application](#updating-an-existing-application)
 - [Working with Secrets](#working-with-secrets)
 - [Enabling Authentication for Applications](#enabling-authentication-for-applications)
+- [Adding a New Keycloak Client](#adding-a-new-keycloak-client)
 - [Troubleshooting](#troubleshooting)
 - [Best Practices](#best-practices)
 
@@ -1247,6 +1248,135 @@ kubectl logs -n myapp <pod-name> -c authn
 
 ---
 
+## Adding a New Keycloak Client
+
+When you need an application to authenticate via Keycloak (OIDC), you can add a client definition to the realm config. The secret syncer automatically extracts the Keycloak-generated client secret into a Kubernetes Secret that your application can reference — no manual secret management needed.
+
+### How It Works
+
+1. You define a client in `forte-realm.json` (inside `keycloak-values.yaml`) **without** a `secret` field
+2. Keycloak auto-generates a cryptographically strong secret on first creation
+3. An ArgoCD **PostSync Job** (`keycloak-secret-syncer`) runs after each Keycloak sync:
+   - Authenticates to the Keycloak Admin API
+   - Finds clients with `k8s.secret.sync: "true"` in their attributes
+   - Extracts the auto-generated secret for each client
+   - Creates/updates a K8s Secret in the target namespace with `client-id` and `client-secret` keys
+4. Your application references the syncer-created Secret
+
+### Step 1: Add Client to Realm Config
+
+In `infra/values/keycloak-values.yaml`, add a new entry to the `clients` array in `forte-realm.json`:
+
+```json
+{
+  "clientId": "myapp",
+  "name": "My Application",
+  "enabled": true,
+  "protocol": "openid-connect",
+  "clientAuthenticatorType": "client-secret",
+  "standardFlowEnabled": true,
+  "directAccessGrantsEnabled": false,
+  "publicClient": false,
+  "redirectUris": ["https://myapp.forteapps.net/*"],
+  "webOrigins": ["https://myapp.forteapps.net"],
+  "defaultClientScopes": ["openid", "email", "profile"],
+  "attributes": {
+    "k8s.secret.sync": "true",
+    "k8s.secret.namespace": "myapp",
+    "k8s.secret.name": "myapp-oidc-credentials"
+  }
+}
+```
+
+**Important**:
+- Do **NOT** include a `"secret"` field — Keycloak generates one automatically
+- The `attributes` block tells the syncer where to create the K8s Secret
+- The target namespace must exist before the syncer runs (ArgoCD creates it via `CreateNamespace=true`)
+
+### Step 2: Reference the Secret in Your Application
+
+In your application's Helm values, reference the syncer-created secret:
+
+```yaml
+# In helm-values/myapp/values.yaml (or inline in values file)
+# The secret will have keys: client-id, client-secret
+existingSecret: myapp-oidc-credentials
+key: client-secret
+```
+
+For Gitea-style oauth config:
+```yaml
+oauth:
+- name: "Forte"
+  provider: "openidConnect"
+  existingSecret: myapp-oidc-credentials
+  key: client-secret
+  autoDiscoverUrl: "https://id.forteapps.net/realms/forte/.well-known/openid-configuration"
+```
+
+### Step 3: Commit and Push
+
+```bash
+cd ~/dev/k8s/launchpad
+git add infra/values/keycloak-values.yaml
+git commit -m "Add myapp Keycloak client with auto-sync"
+git push
+```
+
+ArgoCD will:
+1. Sync the Keycloak config (keycloakConfigCli creates the client)
+2. Run the PostSync syncer Job
+3. The syncer creates `myapp-oidc-credentials` in the `myapp` namespace
+
+### Step 4: Verify
+
+```bash
+# Check the syncer job ran successfully
+kubectl get jobs -n keycloak
+kubectl logs -n keycloak job/keycloak-secret-syncer
+
+# Verify the secret was created
+kubectl get secret myapp-oidc-credentials -n myapp -o yaml
+
+# Check the secret has the expected keys
+kubectl get secret myapp-oidc-credentials -n myapp -o jsonpath='{.data.client-id}' | base64 -d
+kubectl get secret myapp-oidc-credentials -n myapp -o jsonpath='{.data.client-secret}' | base64 -d
+```
+
+### Sync Attribute Reference
+
+| Attribute | Required | Description |
+|-----------|----------|-------------|
+| `k8s.secret.sync` | Yes | Set to `"true"` to enable syncing |
+| `k8s.secret.namespace` | Yes | Target K8s namespace for the secret |
+| `k8s.secret.name` | Yes | Name of the K8s Secret to create |
+
+### Retrieving Secrets for External Deployments
+
+The syncer always writes a **central copy** of every synced secret to the `secrets` namespace, in addition to the target namespace. This allows operators to retrieve client credentials for applications deployed outside this cluster:
+
+```bash
+# View the central copy
+kubectl get secret gitea-oidc-credentials -n secrets -o yaml
+
+# Extract the client secret for use elsewhere
+kubectl get secret myapp-oidc-credentials -n secrets \
+  -o jsonpath='{.data.client-secret}' | base64 -d
+```
+
+This is useful when an application runs on a separate cluster or external infrastructure and needs the Keycloak-generated OIDC credentials provisioned manually (e.g., via a SealedSecret on the remote side).
+
+### Syncer Behavior Notes
+
+- The syncer runs as an ArgoCD **PostSync hook** — it executes after all Keycloak resources are healthy
+- `BeforeHookCreation` delete policy ensures old Job is cleaned up before each run
+- If the target namespace doesn't exist, the target write is skipped with a warning (the central copy still happens)
+- A central copy is **always** written to the `secrets` namespace for every synced client
+- The syncer uses the `keycloak-credentials` secret for admin authentication
+- Created secrets have the label `app.kubernetes.io/managed-by: keycloak-secret-syncer`
+
+---
+
 ## Troubleshooting
 
 ### Application Not Deploying
@@ -1579,4 +1709,4 @@ Now that you understand the basics:
 - Docs: [Full documentation index](README.md)
 - Help: Contact platform team
 
-**Last Updated**: 2026-03-16
+**Last Updated**: 2026-04-16

docs/REFERENCE.md

@@ -869,6 +869,65 @@ dind:
 - Gitea admin panel (`/admin/runners`) — runners show as Online
 - Create test workflow in `.gitea/workflows/test.yml` — job executes
 
+### Keycloak Secret Syncer
+
+**Type**: ArgoCD PostSync Job (deployed via Keycloak Helm chart `extraDeploy`)
+**Namespace**: `keycloak`
+
+**Purpose**: Automatically extracts Keycloak-generated client secrets and syncs them into Kubernetes Secrets in target namespaces. Eliminates the need to manually manage OIDC client secrets.
+
+**How It Works**:
+1. Runs as an ArgoCD PostSync hook after Keycloak resources are healthy
+2. Authenticates to Keycloak Admin API using admin credentials from `keycloak-credentials` secret
+3. Queries all clients in the `forte` realm
+4. Filters clients with `k8s.secret.sync: "true"` attribute
+5. For each matching client, retrieves the auto-generated secret via Keycloak Admin API
+6. Creates/updates a K8s Secret in the target namespace (from `k8s.secret.namespace` attribute)
+7. Always writes a central copy to the `secrets` namespace (for external deployment retrieval)
+
+**Resources**:
+- `ServiceAccount`: `keycloak-secret-syncer` (namespace: `keycloak`)
+- `ClusterRole`: `keycloak-secret-syncer` (secrets: get/create/update/patch; namespaces: get/list)
+- `ClusterRoleBinding`: `keycloak-secret-syncer`
+- `Job`: `keycloak-secret-syncer` (PostSync hook)
+
+**Client Attributes** (set in `forte-realm.json`):
+
+| Attribute | Description |
+|-----------|-------------|
+| `k8s.secret.sync` | Set to `"true"` to enable syncing |
+| `k8s.secret.namespace` | Target K8s namespace |
+| `k8s.secret.name` | Name of the K8s Secret |
+
+**Created Secret Format**:
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: <k8s.secret.name>
+  namespace: <k8s.secret.namespace>
+  labels:
+    app.kubernetes.io/managed-by: keycloak-secret-syncer
+type: Opaque
+data:
+  client-id: <base64-encoded client ID>
+  client-secret: <base64-encoded client secret>
+```
+
+**Verification**:
+```bash
+# Check job status
+kubectl get jobs -n keycloak
+
+# View syncer logs
+kubectl logs -n keycloak job/keycloak-secret-syncer
+
+# Verify created secret
+kubectl get secret <name> -n <namespace> -o yaml
+```
+
+**See**: [Developer Guide - Adding a New Keycloak Client](DEVELOPER-GUIDE.md#adding-a-new-keycloak-client)
+
 ### Renovate
 
 **Chart**: `renovate` (OCI: `ghcr.io/renovatebot/charts`)
@@ -1528,6 +1587,6 @@ team: platform
 
 ---
 
-**Last Updated**: 2026-04-14
+**Last Updated**: 2026-04-16
 **Maintained By**: Platform Team
 **Version**: 1.0.0

infra/keycloak.yaml

@@ -40,3 +40,9 @@ spec:
     - CreateNamespace=true
     - Validate=true
     - ServerSideApply=true
+
+  ignoreDifferences:
+  - group: batch
+    kind: Job
+    jsonPointers:
+    - /spec/template/spec/containers/0/args

infra/values/gitea-values.yaml

@@ -69,8 +69,8 @@ gitea:
   oauth:
   - name: "Forte"
     provider: "openidConnect"
-    existingSecret: gitea-credentials
+    existingSecret: gitea-oidc-credentials
-    key: gitea
+    key: client-secret
     autoDiscoverUrl: "https://id.forteapps.net/realms/forte/.well-known/openid-configuration"
     scopes: "openid email profile organization"
     groupClaimName: "groups"

infra/values/keycloak-values.yaml

@@ -72,13 +72,17 @@ keycloakConfigCli:
             "enabled": true,
             "protocol": "openid-connect",
             "clientAuthenticatorType": "client-secret",
-            "secret": "382ed413580cb79d0f54813e5da87007b28fe766a8903d378b9e1c266405a784",
             "standardFlowEnabled": true,
             "directAccessGrantsEnabled": false,
             "publicClient": false,
             "redirectUris": ["https://git.forteapps.net/*"],
             "webOrigins": ["https://git.forteapps.net"],
             "defaultClientScopes": ["openid", "email", "profile"],
+            "attributes": {
+              "k8s.secret.sync": "true",
+              "k8s.secret.namespace": "gitea",
+              "k8s.secret.name": "gitea-oidc-credentials"
+            },
             "protocolMappers": [
               {
                 "name": "email_verified",
@@ -97,3 +101,210 @@ keycloakConfigCli:
           }
         ]
       }
+
+extraDeploy:
+# -- ServiceAccount for the secret syncer Job
+- apiVersion: v1
+  kind: ServiceAccount
+  metadata:
+    name: keycloak-secret-syncer
+    namespace: keycloak
+
+# -- ClusterRole granting access to secrets and namespaces
+- apiVersion: rbac.authorization.k8s.io/v1
+  kind: ClusterRole
+  metadata:
+    name: keycloak-secret-syncer
+  rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "create", "update", "patch"]
+  - apiGroups: [""]
+    resources: ["namespaces"]
+    verbs: ["get", "list"]
+
+# -- ClusterRoleBinding for the syncer ServiceAccount
+- apiVersion: rbac.authorization.k8s.io/v1
+  kind: ClusterRoleBinding
+  metadata:
+    name: keycloak-secret-syncer
+  roleRef:
+    apiGroup: rbac.authorization.k8s.io
+    kind: ClusterRole
+    name: keycloak-secret-syncer
+  subjects:
+  - kind: ServiceAccount
+    name: keycloak-secret-syncer
+    namespace: keycloak
+
+# -- PostSync Job: extracts Keycloak client secrets into K8s Secrets
+- apiVersion: batch/v1
+  kind: Job
+  metadata:
+    name: keycloak-secret-syncer
+    namespace: keycloak
+    annotations:
+      argocd.argoproj.io/hook: PostSync
+      argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
+  spec:
+    backoffLimit: 3
+    template:
+      spec:
+        serviceAccountName: keycloak-secret-syncer
+        restartPolicy: Never
+        containers:
+        - name: syncer
+          image: alpine:3.20
+          command: ["/bin/sh", "-c"]
+          args:
+          - |
+            set -e
+            apk add --no-cache curl jq > /dev/null 2>&1
+
+            KEYCLOAK_URL="http://keycloak:80"
+            REALM="forte"
+
+            # Read admin credentials from the keycloak-credentials secret
+            ADMIN_USER="admin"
+            ADMIN_PASS=$(cat /secrets/admin-password)
+
+            # Authenticate to Keycloak Admin API
+            echo "Authenticating to Keycloak..."
+            TOKEN=$(curl -sf -X POST "${KEYCLOAK_URL}/realms/master/protocol/openid-connect/token" \
+              -d "client_id=admin-cli" \
+              -d "username=${ADMIN_USER}" \
+              -d "password=${ADMIN_PASS}" \
+              -d "grant_type=password" | jq -r '.access_token')
+
+            if [ -z "$TOKEN" ] || [ "$TOKEN" = "null" ]; then
+              echo "ERROR: Failed to authenticate to Keycloak"
+              exit 1
+            fi
+
+            # Get all clients in the realm
+            echo "Fetching clients from realm '${REALM}'..."
+            CLIENTS=$(curl -sf -H "Authorization: Bearer ${TOKEN}" \
+              "${KEYCLOAK_URL}/admin/realms/${REALM}/clients")
+
+            # Filter clients with k8s.secret.sync=true
+            SYNC_CLIENTS=$(echo "$CLIENTS" | jq -c '[.[] | select(.attributes["k8s.secret.sync"] == "true")]')
+            COUNT=$(echo "$SYNC_CLIENTS" | jq 'length')
+            echo "Found ${COUNT} client(s) with sync enabled"
+
+            K8S_API="https://kubernetes.default.svc"
+            SA_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+            CA_CERT="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
+            CENTRAL_NS="secrets"
+
+            # Upsert a K8s Secret: try PUT (update), fall back to POST (create)
+            upsert_secret() {
+              local ns="$1" name="$2" manifest="$3"
+              local code
+              code=$(curl -sf -o /dev/null -w "%{http_code}" \
+                --cacert "$CA_CERT" \
+                -H "Authorization: Bearer ${SA_TOKEN}" \
+                -H "Content-Type: application/json" \
+                -X PUT -d "$manifest" \
+                "${K8S_API}/api/v1/namespaces/${ns}/secrets/${name}")
+              if [ "$code" = "200" ]; then
+                echo "  Updated secret '${ns}/${name}'"
+              elif [ "$code" = "404" ]; then
+                code=$(curl -sf -o /dev/null -w "%{http_code}" \
+                  --cacert "$CA_CERT" \
+                  -H "Authorization: Bearer ${SA_TOKEN}" \
+                  -H "Content-Type: application/json" \
+                  -X POST -d "$manifest" \
+                  "${K8S_API}/api/v1/namespaces/${ns}/secrets")
+                if [ "$code" = "201" ]; then
+                  echo "  Created secret '${ns}/${name}'"
+                else
+                  echo "  ERROR: Failed to create secret '${ns}/${name}' (HTTP ${code})"
+                  return 1
+                fi
+              else
+                echo "  ERROR: Failed to update secret '${ns}/${name}' (HTTP ${code})"
+                return 1
+              fi
+            }
+
+            # Build a Secret JSON manifest
+            build_manifest() {
+              local ns="$1" name="$2" b64_id="$3" b64_secret="$4"
+              cat <<MANIFEST
+            {
+              "apiVersion": "v1",
+              "kind": "Secret",
+              "metadata": {
+                "name": "${name}",
+                "namespace": "${ns}",
+                "labels": {
+                  "app.kubernetes.io/managed-by": "keycloak-secret-syncer"
+                }
+              },
+              "type": "Opaque",
+              "data": {
+                "client-id": "${b64_id}",
+                "client-secret": "${b64_secret}"
+              }
+            }
+            MANIFEST
+            }
+
+            echo "$SYNC_CLIENTS" | jq -c '.[]' | while read -r CLIENT; do
+              CLIENT_ID=$(echo "$CLIENT" | jq -r '.clientId')
+              CLIENT_UUID=$(echo "$CLIENT" | jq -r '.id')
+              TARGET_NS=$(echo "$CLIENT" | jq -r '.attributes["k8s.secret.namespace"]')
+              TARGET_NAME=$(echo "$CLIENT" | jq -r '.attributes["k8s.secret.name"]')
+
+              echo "Processing client '${CLIENT_ID}' -> secret '${TARGET_NS}/${TARGET_NAME}'"
+
+              # Get the client secret from Keycloak
+              SECRET_VALUE=$(curl -sf -H "Authorization: Bearer ${TOKEN}" \
+                "${KEYCLOAK_URL}/admin/realms/${REALM}/clients/${CLIENT_UUID}/client-secret" \
+                | jq -r '.value')
+
+              if [ -z "$SECRET_VALUE" ] || [ "$SECRET_VALUE" = "null" ]; then
+                echo "  WARNING: No secret found for client '${CLIENT_ID}', skipping"
+                continue
+              fi
+
+              B64_CLIENT_ID=$(printf '%s' "$CLIENT_ID" | base64 | tr -d '\n')
+              B64_SECRET=$(printf '%s' "$SECRET_VALUE" | base64 | tr -d '\n')
+
+              # 1. Write to target namespace (if it exists)
+              NS_STATUS=$(curl -sf -o /dev/null -w "%{http_code}" \
+                --cacert "$CA_CERT" \
+                -H "Authorization: Bearer ${SA_TOKEN}" \
+                "${K8S_API}/api/v1/namespaces/${TARGET_NS}")
+
+              if [ "$NS_STATUS" = "200" ]; then
+                MANIFEST=$(build_manifest "$TARGET_NS" "$TARGET_NAME" "$B64_CLIENT_ID" "$B64_SECRET")
+                upsert_secret "$TARGET_NS" "$TARGET_NAME" "$MANIFEST" || exit 1
+              else
+                echo "  WARNING: Namespace '${TARGET_NS}' does not exist, skipping target"
+              fi
+
+              # 2. Always write a central copy to the secrets namespace
+              CENTRAL_MANIFEST=$(build_manifest "$CENTRAL_NS" "$TARGET_NAME" "$B64_CLIENT_ID" "$B64_SECRET")
+              upsert_secret "$CENTRAL_NS" "$TARGET_NAME" "$CENTRAL_MANIFEST" || exit 1
+            done
+
+            echo "Secret sync complete"
+          volumeMounts:
+          - name: keycloak-credentials
+            mountPath: /secrets
+            readOnly: true
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              cpu: 200m
+              memory: 128Mi
+        volumes:
+        - name: keycloak-credentials
+          secret:
+            secretName: keycloak-credentials
+            items:
+            - key: admin-password
+              path: admin-password