launchpad/docs/DEVELOPER-GUIDE.md

Developer Onboarding Guide

Table of Contents

• Getting Started
• Prerequisites
• Local Development Setup
• Understanding the Workflow
• Deploying Your First Application
• Updating an Existing Application
• Working with Secrets
• Enabling Authentication for Applications
• Adding a New Keycloak Client
• Troubleshooting
• Best Practices

---

Getting Started

Welcome! This guide will help you understand how to develop and deploy applications on our Kubernetes cluster using GitOps principles powered by ArgoCD.

What You'll Learn

• How our GitOps architecture works
• How to deploy a new application
• How to update existing applications
• How to manage secrets securely
• Common troubleshooting techniques

Who This Guide Is For

• Developers deploying new applications
• Developers maintaining existing applications
• Team members who need to understand the deployment process

---

Prerequisites

Required Knowledge

• ✅ Basic Git workflow (clone, commit, push, pull)
• ✅ Docker basics (Dockerfile, building images)
• ✅ YAML syntax
• ✅ Basic understanding of Kubernetes concepts (pods, deployments, services)
• ⚠️ Helm knowledge (helpful but not required - templates are provided)

Required Tools

Most developers do NOT need kubectl access to the cluster. You'll primarily work with Git repositories.

If you do need cluster access, install:

1. kubectl - Kubernetes CLI

   # macOS
   brew install kubectl
   
   # Windows
   choco install kubernetes-cli
   
   # Linux
   curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   

2. kubeseal - For sealing secrets

   # macOS
   brew install kubeseal
   
   # Windows
   choco install kubeseal
   
   # Linux
   wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
   tar -xvzf kubeseal-0.24.0-linux-amd64.tar.gz
   sudo mv kubeseal /usr/local/bin/
   

3. Git - Version control

   git --version  # Should already be installed
   

4. Docker - For local development

   # macOS/Windows: Install Docker Desktop
   # Linux: Install Docker Engine
   docker --version
   

Repository Access

You'll need read/write access to these repositories:

1. launchpad (Config repo)

   git clone https://git.forteapps.net/Forte/launchpad.git
   cd launchpad
   

2. helm-prod-values (Values repo)

   git clone https://git.forteapps.net/Forte/helm-prod-values.git
   cd helm-prod-values
   

3. forte-helm (Chart repo - read-only for most developers)

   git clone https://git.forteapps.net/Forte/forte-helm.git
   cd forte-helm
   

Cluster Access (If Needed)

If you need kubectl access, ask the platform team for:

• Kubeconfig file
• Cluster context setup instructions

Save to ~/.kube/config and verify:

kubectl cluster-info
kubectl get nodes

---

Local Development Setup

1. Clone the Repositories

Set up a consistent folder structure:

mkdir -p ~/dev/k8s
cd ~/dev/k8s

# Clone repositories
git clone https://git.forteapps.net/Forte/launchpad.git launchpad
git clone https://git.forteapps.net/Forte/helm-prod-values helm-prod-values
git clone https://git.forteapps.net/Forte/forte-helm forte-helm

# Your folder structure:
# ~/dev/k8s/
# ├── launchpad/           (Config repo)
# ├── helm-prod-values/    (Values repo)
# └── forte-helm/          (Chart repo)

2. Local Development Environment

Most applications use Docker Compose for local development:

# In your application repository
docker-compose up

# Or for frontend applications
npm install
npm run dev

You DO NOT run applications locally on Kubernetes. Use Docker Compose or native tooling (npm, python, etc.).

3. Understanding the Deployment Flow

┌─────────────────────────────────────────────────────────────────┐
│  Step 1: Develop Locally                                        │
│  - Write code in your application repository                    │
│  - Test with Docker Compose or npm/python/etc.                  │
│  - Build Docker image                                            │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 2: CI/CD Pipeline (Automated)                             │
│  - GitHub Actions builds image                                  │
│  - Pushes to container registry (GHCR, Docker Hub)              │
│  - Tags with version (e.g., v2.0.4)                             │
│  - Updates helm-prod-values repository with new tag                  │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 3: GitOps Sync (Automated)                                │
│  - ArgoCD detects change in helm-prod-values                         │
│  - Pulls updated configuration                                  │
│  - Syncs to Kubernetes cluster                                  │
│  - Sends Slack notification on success/failure                  │
└─────────────────────────────────────────────────────────────────┘

Key Insight: You don't deploy directly. You push code, CI/CD builds it, and ArgoCD deploys it.

---

Understanding the Workflow

Three-Repository Pattern

Our setup uses three repositories:

Repository	Purpose	Who Edits	How Often
forte-helm	Helm chart templates (generic, reusable)	Platform engineers	❌ Rarely
helm-prod-values	Application configuration (image tag, env vars)	Developers / CI pipelines	✅ Sometimes
launchpad	ArgoCD Applications (what gets deployed)	Platform / DevOps engineers	✅ Per new app

Example: Deploying "myapp"

Repository: forte-helm (Chart Templates)

# forteapp/templates/deployment.yaml
# Generic template used by ALL apps
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
spec:
  containers:
  - name: app
    image: "{{ .Values.app.image.repository }}:{{ .Values.app.image.tag }}"
    env:
    - name: PORT
      value: {{ .Values.app.port }}

Repository: helm-prod-values (Your App Config)

# myapp/values.yaml
# Your app's specific configuration
app:
  image:
    repository: ghcr.io/fortedigital/myapp
    tag: v1.0.0                    # CI/CD updates this
  port: 3000
  extraEnv:
  - name: API_URL
    value: https://api.example.com

Repository: launchpad (ArgoCD Application)

# apps/myapp.yaml
# Tells ArgoCD to deploy your app
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  sources:
  - repoURL: https://git.forteapps.net/Forte/forte-helm
    path: forteapp
    helm:
      valueFiles:
      - $values/myapp/values.yaml

  - repoURL: git@github.com:fortedigital/helm-prod-values.git
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: myapp

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

---

Deploying Your First Application

Scenario: You've Built a New Application

Let's deploy a new Node.js application called "hello-world".

Step 1: Prepare Your Application Repository

Ensure your app repository has:

1. Dockerfile

   FROM node:18-alpine
   WORKDIR /app
   COPY package*.json ./
   RUN npm ci --only=production
   COPY . .
   EXPOSE 3000
   CMD ["node", "server.js"]
   

2. GitHub Actions Workflow (.github/workflows/deploy.yml)

   name: Build and Deploy
   
   on:
     push:
       branches: [ main ]
   
   jobs:
     build:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v3
   
         - name: Set version
           id: version
           run: echo "VERSION=v$(date +%Y%m%d-%H%M%S)" >> $GITHUB_OUTPUT
   
         - name: Build and push Docker image
           run: |
             echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
             docker build -t ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }} .
             docker push ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }}
   
         - name: Update helm-prod-values
           run: |
             git clone git@github.com:fortedigital/helm-prod-values.git
             cd helm-prod-values
             mkdir -p hello-world
             cat > hello-world/values.yaml <<EOF
             app:
               image:
                 repository: ghcr.io/fortedigital/hello-world
                 tag: ${{ steps.version.outputs.VERSION }}
             EOF
             git add hello-world/values.yaml
             git commit -m "Update hello-world to ${{ steps.version.outputs.VERSION }}"
             git push
   

Step 2: Create Helm Values

Create a folder in helm-prod-values repository:

cd ~/dev/k8s/helm-prod-values
mkdir -p hello-world

Create hello-world/values.yaml:

app:
  image:
    repository: ghcr.io/fortedigital/hello-world
    tag: v1.0.0                    # Will be updated by CI/CD
    containerPort: 3000

  replicaCount: 1

  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi

  extraEnv:
  - name: PORT
    value: "3000"
  - name: NODE_ENV
    value: "production"

  envSecretName: ""                # Optional: reference to secrets

service:
  port: 3000

ingress:
  enabled: true
  host: hello-world.forteapps.net  # Your subdomain

db:
  enabled: false                   # Set to true if you need PostgreSQL

Commit and push:

git add hello-world/values.yaml
git commit -m "Add hello-world application values"
git push

Step 3: Create ArgoCD Application Manifest

In the launchpad repository, create apps/hello-world.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello-world
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: ""
    notifications.argoproj.io/subscribe.on-sync-failed.slack: ""
    notifications.argoproj.io/subscribe.on-degraded.slack: ""
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/part-of: apps
    app.kubernetes.io/managed-by: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io

spec:
  project: default

  sources:
  # Source 1: Helm chart templates
  - repoURL: https://git.forteapps.net/Forte/forte-helm
    path: forteapp
    targetRevision: HEAD
    helm:
      valueFiles:
      - $values/hello-world/values.yaml

  # Source 2: Helm values
  - repoURL: git@github.com:fortedigital/helm-prod-values.git
    targetRevision: HEAD
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: hello-world

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false

    syncOptions:
    - CreateNamespace=true
    - Validate=true
    - ServerSideApply=true

    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas

Commit and push:

cd ~/dev/k8s/launchpad
git add apps/hello-world.yaml
git commit -m "Add hello-world application"
git push

Step 4: Verify Deployment

ArgoCD will automatically detect the new application within 60 seconds.

Option 1: Check Slack

• Watch for sync notifications in your Slack channel
• ✅ "Application hello-world sync succeeded"

Option 2: Check ArgoCD UI (if you have access)

# Port forward to ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Open browser: https://localhost:8080
# Look for "hello-world" application

Option 3: Check with kubectl (if you have access)

# List ArgoCD applications
kubectl get applications -n argocd

# Check application status
kubectl get application hello-world -n argocd

# Verify pods are running
kubectl get pods -n hello-world

Step 5: Access Your Application

Once deployed, access via the configured domain:

# Check if ingress is created
kubectl get ingressroute -n hello-world

# Access application
curl https://hello-world.forteapps.net

⚠️ Note: DNS must be manually configured for new subdomains. Contact the platform team to add DNS records.

---

Updating an Existing Application

Scenario: Deploying a Code Change

You've made changes to your application code and want to deploy them.

Method 1: Automatic (Recommended)

Just push to main branch - CI/CD handles everything:

# In your application repository
git add .
git commit -m "Fix bug in user login"
git push origin main

What Happens Next:

1. ✅ GitHub Actions triggers
2. ✅ Builds new Docker image
3. ✅ Tags with new version (e.g., v20260316-143022)
4. ✅ Pushes to container registry
5. ✅ Updates helm-prod-values/myapp/values.yaml with new tag
6. ✅ ArgoCD detects change
7. ✅ Syncs new version to cluster
8. ✅ Sends Slack notification

Timeline: ~5-10 minutes from push to deployment

Method 2: Manual Image Tag Update

If CI/CD is not set up, manually update the image tag:

cd ~/dev/k8s/helm-prod-values

# Edit your app's values.yaml
vim myapp/values.yaml

# Change:
app:
  image:
    tag: v1.0.0  # Old version
# To:
app:
  image:
    tag: v1.0.1  # New version

# Commit and push
git add myapp/values.yaml
git commit -m "Update myapp to v1.0.1"
git push

ArgoCD will sync within 60 seconds.

Method 3: Configuration Changes

To update environment variables, resources, or other config:

cd ~/dev/k8s/helm-prod-values
vim myapp/values.yaml

Example changes:

app:
  # Increase resources
  resources:
    requests:
      cpu: 200m      # Was 100m
      memory: 256Mi  # Was 128Mi

  # Add new environment variable
  extraEnv:
  - name: API_URL
    value: https://api.example.com
  - name: DEBUG          # NEW
    value: "true"        # NEW

  # Enable HPA
  hpa:
    enabled: true        # Was false
    minReplicas: 2
    maxReplicas: 10

Commit and push:

git add myapp/values.yaml
git commit -m "Increase myapp resources and enable HPA"
git push

Method 4: Application Manifest Changes

To change ArgoCD sync behavior, namespace, or other meta-config:

cd ~/dev/k8s/launchpad
vim apps/myapp.yaml

Example changes:

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: false    # Disable self-healing temporarily

Commit and push:

git add apps/myapp.yaml
git commit -m "Disable self-healing for myapp"
git push

---

Working with Secrets

Understanding Secret Management

NEVER commit plain secrets to Git. We use Sealed Secrets to encrypt secrets before committing.

Creating a New Secret

Step 1: Create Plain Secret Locally

cd ~/dev/k8s/launchpad

# Create secret in private/ folder (Git-ignored)
kubectl create secret generic myapp-credentials \
  --from-literal=API_KEY=your-secret-key-here \
  --from-literal=DB_PASSWORD=super-secret-password \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

DO NOT commit this file! It's in private/ which is Git-ignored.

Step 2: Seal the Secret

Seal your secret:

kubeseal --format=yaml \
  --namespace=myapp \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml

Step 3: Commit Sealed Secret

git add secrets/myapp-credentials-sealed.yaml
git commit -m "Add myapp credentials (sealed)"
git push

Step 4: Reference Secret in Application

Update your helm-prod-values/myapp/values.yaml:

app:
  envSecretName: "myapp-credentials"  # References the SealedSecret

Commit and push:

cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Reference myapp credentials"
git push

Updating a Secret

To update an existing secret:

# 1. Create new version of secret
kubectl create secret generic myapp-credentials \
  --from-literal=API_KEY=new-key-here \
  --from-literal=DB_PASSWORD=new-password \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

# 2. Seal it
kubeseal --format=yaml \
  --namespace=myapp \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml

# 3. Commit sealed version
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Update myapp credentials"
git push

# 4. Restart pods to pick up new secret
kubectl rollout restart deployment myapp -n myapp

Secret Best Practices

✅ DO:

• Store secrets in private/ folder locally
• Always seal secrets before committing
• Delete plain secrets after sealing
• Use meaningful secret names
• Document what each secret contains

❌ DON'T:

• Commit plain secrets to Git
• Share secrets via Slack/email
• Hard-code secrets in code
• Use the same secret across multiple environments
• Store secrets in Docker images

Where Secrets Are Stored

┌─────────────────────────────────────────────────────────────┐
│  Location                │  Content           │  Committed?│
├──────────────────────────┼────────────────────┼────────────┤
│  private/                │  Plain secrets     │  ❌ NO      │
│  secrets/                │  Sealed secrets    │  ✅ YES     │
│  Kubernetes cluster      │  Unsealed secrets  │  N/A       │
└─────────────────────────────────────────────────────────────┘

Sealed Secrets Controller in the cluster decrypts sealed secrets automatically.

---

Enabling Authentication for Applications

The cluster supports automatic authentication sidecar injection for applications via Kyverno policies. This allows you to add authentication to your applications without modifying application code.

How It Works

When you enable authentication in your Helm values, the Kyverno policy automatically:

1. ✅ Injects an authentication sidecar container into your pod
2. ✅ Routes all incoming traffic through the auth sidecar (port 8080)
3. ✅ Validates credentials before forwarding requests to your application
4. ✅ Creates necessary secrets (if they don't exist)
5. ✅ Adds a NetworkPolicy to restrict ingress

Architecture:

Internet → Traefik → Service:8080 → Auth Sidecar:8080 → localhost → Your App:3000
                                         │
                                         ├─ Validates credentials
                                         └─ Forwards if valid

Authentication Modes

Three authentication modes are supported:

1. Token-based: Static tokens (simple, good for service-to-service or internal apps)
2. OIDC: OpenID Connect (full SSO, good for user-facing apps)
3. MCP: OAuth 2.0 for MCP servers via RFC 9728 (Protected Resource Metadata); Keycloak provides native RFC 7591 Dynamic Client Registration (good for MCP tool servers requiring OAuth-based access control)

---

Token-Based Authentication

Step 1: Configure Helm Values

# In helm-prod-values/myapp/values.yaml
auth:
  enabled: true
  type: token                    # Token mode (default)
  tokens:
  - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823
  - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7

Step 2: Generate Token (if needed)

# Generate a secure random token
openssl rand -hex 32

# Or using Python
python3 -c "import secrets; print(secrets.token_hex(32))"

# Example output:
# d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823

Step 3: Deploy Application

Commit and push your changes:

cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Enable token auth for myapp"
git push

ArgoCD will sync, and the Kyverno policy will:

• Inject the auth sidecar container
• Create an auth-tokens Secret with your tokens
• Configure the sidecar to validate against these tokens

Step 4: Access Application

Use your token in the Authorization header:

# Access application with token
curl -H "Authorization: Bearer d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823" \
  https://myapp.forteapps.net/api/data

# Without token (will be rejected)
curl https://myapp.forteapps.net/api/data
# Response: 401 Unauthorized

Advanced: Custom Secret Name

To use a different secret for tokens:

# In Helm values
auth:
  enabled: true
  type: token
  tokens: []                     # Empty - using external secret

# Tokens will be read from custom secret

Then reference it via annotation (configured by Helm chart automatically):

# Helm chart sets this annotation:
policies.forteapps.io/auth-token-secret-name: "myapp-auth-tokens"

Create the secret manually:

kubectl create secret generic myapp-auth-tokens \
  --from-file=tokens=tokens.txt \
  --namespace=myapp

---

OIDC Authentication

OIDC mode integrates with identity providers like Keycloak, Okta, Auth0, Azure AD, etc.

Step 1: Configure Identity Provider

In your identity provider (e.g., Keycloak):

1. Create a new client (e.g., myapp)
2. Set redirect URI: https://myapp.forteapps.net/auth/callback
3. Note the Client ID and Client Secret
4. Note the Authority URL (e.g., https://keycloak.forteapps.net/realms/master)

Step 2: Create OIDC Secret

# Create plain secret
kubectl create secret generic auth-oidc \
  --from-literal=client-secret=your-oidc-client-secret \
  --from-literal=cookie-secret=$(openssl rand -hex 32) \
  --namespace=myapp \
  --dry-run=client -o yaml > private/myapp-auth-oidc.yaml

# Seal it
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  --namespace=myapp \
  < private/myapp-auth-oidc.yaml \
  > secrets/myapp-auth-oidc-sealed.yaml

# Commit sealed secret
cd ~/dev/k8s/launchpad
git add secrets/myapp-auth-oidc-sealed.yaml
git commit -m "Add OIDC secrets for myapp"
git push

# Clean up
rm private/myapp-auth-oidc.yaml

Step 3: Configure Helm Values

# In helm-prod-values/myapp/values.yaml
auth:
  enabled: true
  type: oidc                     # OIDC mode
  oidc:
    authority: https://keycloak.forteapps.net/realms/master
    clientId: myapp
    scopes: "openid,profile,email"
    callbackPath: /auth/callback

Step 4: Deploy Application

cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Enable OIDC auth for myapp"
git push

Step 5: Access Application

When users access https://myapp.forteapps.net:

1. They're redirected to the identity provider login page
2. After successful login, redirected back to /auth/callback
3. Session cookie is set
4. Subsequent requests are authenticated via cookie

User flow:

User → https://myapp.forteapps.net
  ↓
Redirect → https://keycloak.forteapps.net/login
  ↓
Login successful → Redirect with auth code
  ↓
https://myapp.forteapps.net/auth/callback?code=xyz
  ↓
Auth sidecar exchanges code for tokens
  ↓
Sets session cookie
  ↓
Redirects to application → https://myapp.forteapps.net
  ↓
User sees application (authenticated)

---

Accessing Authenticated User Information

The auth sidecar handles all authentication before requests reach your application. Your app never sees unauthenticated traffic — the sidecar returns 401 or redirects to the IdP first.

After successful authentication, the sidecar forwards the request to your application with user identity injected as HTTP headers:

Header	Description	Available in
X-Auth-User	Username or display name	Token, OIDC, MCP
X-Auth-Email	User email address	OIDC
X-Auth-Subject	OIDC sub claim (stable user ID)	OIDC, MCP
X-Auth-Groups	Comma-separated group memberships	OIDC (if scope includes groups)
X-Auth-Token	The validated access token	All modes

Your application reads these headers — no auth library needed:

// Express.js example
app.get('/profile', (req, res) => {
  const user = req.headers['x-auth-user'];
  const email = req.headers['x-auth-email'];
  res.json({ user, email });
});

# Flask example
@app.route('/profile')
def profile():
    user = request.headers.get('X-Auth-User')
    email = request.headers.get('X-Auth-Email')
    return jsonify(user=user, email=email)

Why this is safe: The Kyverno-generated NetworkPolicy restricts ingress to the sidecar port only. Traffic cannot bypass the sidecar to reach the application port directly, so the X-Auth-* headers can be trusted unconditionally.

Key principle: Your application is zero-trust-unaware by design. It reads headers and renders UI. All authentication complexity lives in the sidecar and Kyverno policy.

---

Authentication Configuration Reference

Helm Values Schema

auth:
  enabled: false                 # Enable/disable authentication
  type: token                    # "token", "oidc", or "mcp"

  # Token mode configuration
  tokens: []                     # List of valid bearer tokens
  # - token1
  # - token2

  # OIDC mode configuration
  oidc:
    authority: ""                # OIDC provider URL (required for OIDC)
    clientId: ""                 # OIDC client ID (required for OIDC)
    scopes: "openid,profile,email"  # OIDC scopes (optional)
    callbackPath: /auth/callback    # OAuth callback path (optional)

  # MCP mode configuration (RFC 9728)
  mcp:
    resource: ""                 # Protected resource URL (required for MCP)
    authority: ""                # Authorization server URL (required for MCP)
    scopes: "read,write"         # Supported scopes (optional)

Annotations Set by Helm Chart

When auth.enabled: true, the Helm chart sets these pod annotations:

Token mode:

policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "token"
policies.forteapps.io/auth-token-secret-name: "auth-tokens"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"

OIDC mode:

policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "oidc"
policies.forteapps.io/auth-oidc-authority: "https://keycloak.forteapps.net/realms/master"
policies.forteapps.io/auth-oidc-client-id: "myapp"
policies.forteapps.io/auth-oidc-scopes: "openid,profile,email"
policies.forteapps.io/auth-oidc-callback-path: "/auth/callback"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"

MCP mode (OAuth 2.0 for MCP servers):

policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "mcp"
policies.forteapps.io/auth-mcp-resource: "https://mcp.forteapps.net"
policies.forteapps.io/auth-mcp-authority: "https://keycloak.forteapps.net/realms/master"
policies.forteapps.io/auth-mcp-scopes: "read,write"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"

Sidecar Configuration

The auth sidecar container:

• Image: ghcr.io/fortedigital/auth-sidecar:latest
• Port: 8080
• Resources: 10m CPU / 32Mi memory (requests), 50m CPU / 64Mi memory (limits)
• Health checks: /healthz endpoint
• Security: Read-only root filesystem, no privilege escalation

Advanced: Custom Sidecar Image

To use a different auth sidecar image:

# These annotations can be set in the Helm chart template if needed
policies.forteapps.io/auth-image: "your-registry/your-auth-proxy"
policies.forteapps.io/auth-image-version: "v1.2.3"

---

Authentication Examples

Example 1: Internal API with Token Auth

# helm-prod-values/internal-api/values.yaml
app:
  image:
    repository: ghcr.io/company/internal-api
    tag: v1.0.0

auth:
  enabled: true
  type: token
  tokens:
  - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823  # Service A
  - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7  # Service B

ingress:
  enabled: true
  host: internal-api.forteapps.net

Usage:

# Service A calls API
curl -H "Authorization: Bearer d4f88f..." \
  https://internal-api.forteapps.net/api/endpoint

Example 2: User-Facing App with OIDC

# helm-prod-values/web-app/values.yaml
app:
  image:
    repository: ghcr.io/company/web-app
    tag: v2.1.0

auth:
  enabled: true
  type: oidc
  oidc:
    authority: https://auth.company.com/realms/employees
    clientId: web-app-prod
    scopes: "openid,profile,email,groups"
    callbackPath: /auth/callback

ingress:
  enabled: true
  host: web-app.forteapps.net

With sealed OIDC secret:

# Create and seal secret
kubectl create secret generic auth-oidc \
  --from-literal=client-secret=super-secret-value \
  --from-literal=cookie-secret=$(openssl rand -hex 32) \
  --namespace=web-app \
  --dry-run=client -o yaml | \
  kubeseal --format=yaml --cert=pub-cert.pem --namespace=web-app \
  > secrets/web-app-auth-oidc-sealed.yaml

Example 3: MCP Server with OAuth 2.0

# helm-prod-values/mcp-server/values.yaml
app:
  image:
    repository: ghcr.io/company/mcp-server
    tag: v1.0.0

auth:
  enabled: true
  type: mcp
  mcp:
    resource: https://mcp-server.forteapps.net
    authority: https://auth.company.com/realms/mcp
    scopes: "read,write,admin"

ingress:
  enabled: true
  host: mcp-server.forteapps.net

The MCP auth mode implements RFC 9728 (OAuth 2.0 Protected Resource Metadata) for authorization server discovery. Dynamic Client Registration (RFC 7591) is handled natively by Keycloak; MCP clients discover the authorization server and scopes from the /.well-known/oauth-protected-resource endpoint served by the sidecar and then register directly with Keycloak.

Example 4: Disabling Authentication

# helm-prod-values/public-api/values.yaml
auth:
  enabled: false                 # No authentication

ingress:
  enabled: true
  host: public-api.forteapps.net

---

Troubleshooting Authentication

Issue: 401 Unauthorized (Token Mode)

Check token validity:

# Get auth-tokens secret
kubectl get secret auth-tokens -n myapp -o yaml

# Decode tokens
kubectl get secret auth-tokens -n myapp \
  -o jsonpath='{.data.tokens}' | base64 -d

# Verify your token is in the list

Test with different token:

curl -v -H "Authorization: Bearer YOUR-TOKEN-HERE" \
  https://myapp.forteapps.net/

Issue: OIDC Login Loop

Check OIDC configuration:

# Verify auth-oidc secret exists
kubectl get secret auth-oidc -n myapp

# Check sidecar logs
kubectl logs -n myapp <pod-name> -c authn

# Common issues:
# - Wrong authority URL
# - Wrong client ID
# - Missing client-secret in auth-oidc Secret
# - Redirect URI not configured in identity provider

Verify redirect URI in your identity provider matches:

https://<your-app-domain>/auth/callback

Issue: Auth Sidecar Not Injected

Check pod annotations:

kubectl get pod -n myapp <pod-name> -o yaml | grep policies.forteapps.io

# Should show:
# policies.forteapps.io/auth: "true"

Check Kyverno policy:

kubectl get clusterpolicy inject-auth-sidecar
kubectl describe clusterpolicy inject-auth-sidecar

Check Kyverno logs:

kubectl logs -n kyverno deployment/kyverno | grep inject-auth

Issue: Auth Sidecar Crashes

Check sidecar logs:

kubectl logs -n myapp <pod-name> -c authn

Common causes:

• Missing secret (auth-tokens or auth-oidc)
• Invalid OIDC configuration
• Can't reach OIDC authority URL
• Network policy blocking outbound OIDC requests

---

Authentication Best Practices

✅ DO:

• Use OIDC for user-facing applications
• Use token auth for service-to-service communication
• Rotate tokens and secrets regularly
• Use strong random tokens (32+ bytes)
• Store client secrets in SealedSecrets
• Test authentication before deploying to production
• Document which tokens/users have access

❌ DON'T:

• Share tokens between environments
• Commit tokens to application code
• Use predictable tokens
• Reuse tokens across multiple applications
• Disable authentication on sensitive APIs
• Log tokens or secrets

---

Adding a New Keycloak Client

There are two ways to add an OIDC client, depending on your use case:

Method	Best for	Who edits the infra repo?
Self-service (recommended)	New apps that deploy their own resources	App developer — no infra changes needed
Legacy (realm JSON)	Existing clients already defined in forte-realm.json (e.g., Gitea)	Platform engineer

Both methods are served by the Keycloak Client Registrar CronJob, which runs every 2 minutes.

Self-Service OIDC Client Registration

This is the recommended flow for new applications. Your app deploys a labeled config Secret in its own namespace; the platform handles everything else.

How It Works

1. You deploy a Secret with label keycloak.forteapps.net/client-config: "true" containing a client.json definition
2. A Kyverno ClusterPolicy (keycloak-client-config-cloner) clones it to the keycloak namespace
3. The Client Registrar CronJob picks it up within 2 minutes:
   • Registers (or updates) the client in Keycloak
   • Fetches the auto-generated client secret
   • Creates a credential Secret in your app's namespace
   • Annotates the config Secret with sync status

Step 1: Create the Config Secret

Deploy this Secret in your application's namespace (e.g., as part of your Helm chart or Kustomize overlay):

apiVersion: v1
kind: Secret
metadata:
  name: keycloak-client-myapp
  namespace: myapp
  labels:
    keycloak.forteapps.net/client-config: "true"
stringData:
  client.json: |
    {
      "clientId": "myapp",
      "name": "My Application",
      "redirectUris": ["https://myapp.forteapps.net/*"],
      "webOrigins": ["https://myapp.forteapps.net"],
      "defaultClientScopes": ["openid", "email", "profile"],
      "protocolMappers": [],
      "secret": {
        "namespace": "myapp",
        "name": "myapp-oidc-credentials",
        "keys": { "clientId": "client-id", "clientSecret": "client-secret" }
      }
    }

client.json fields:

Field	Required	Description
clientId	Yes	Keycloak client ID (must be unique in realm)
name	Yes	Display name in Keycloak UI
redirectUris	Yes	Allowed OAuth redirect URLs (supports wildcards like /*)
webOrigins	Yes	Allowed CORS origins
defaultClientScopes	No	OIDC scopes (default: ["openid", "email", "profile"])
protocolMappers	No	Custom claim mappers for tokens (see examples below)
secret.namespace	No	Target namespace for credentials (default: source-namespace annotation value)
secret.name	No	Credential Secret name (default: <clientId>-oidc-credentials)
secret.keys.clientId	No	Key name for client ID (default: client-id)
secret.keys.clientSecret	No	Key name for client secret (default: client-secret)

Protocol Mappers Example:

"protocolMappers": [
  {
    "name": "groups",
    "protocol": "openid-connect",
    "protocolMapper": "oidc-group-membership-mapper",
    "config": {
      "claim.name": "groups",
      "full.path": "false",
      "id.token.claim": "true",
      "access.token.claim": "true",
      "userinfo.token.claim": "true"
    }
  }
]

Step 2: Reference the Credential Secret

In your application's deployment config, reference the credential Secret that the registrar creates:

env:
- name: OIDC_CLIENT_ID
  valueFrom:
    secretKeyRef:
      name: myapp-oidc-credentials
      key: client-id
- name: OIDC_CLIENT_SECRET
  valueFrom:
    secretKeyRef:
      name: myapp-oidc-credentials
      key: client-secret

Step 3: Deploy and Wait

Commit and push your changes. The credential Secret will appear within 2 minutes:

# Watch for the credential Secret to be created
kubectl get secret myapp-oidc-credentials -n myapp -w

# Check registrar logs
kubectl logs -n keycloak job/$(kubectl get jobs -n keycloak --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')

# Check sync status on the config Secret
kubectl get secret keycloak-client-myapp -n keycloak -o jsonpath='{.metadata.annotations}'

Change Detection

The registrar computes a SHA-256 hash of client.json and stores it as an annotation. On subsequent runs, it skips processing if:

• The hash hasn't changed, AND
• The credential Secret already exists in the target namespace

To force a re-sync, update any field in client.json (e.g., add a trailing space to name).

Legacy Method: Realm JSON

Existing clients (like Gitea) are defined directly in forte-realm.json inside keycloak-values.yaml. The registrar syncs their secrets via client attributes.

Step 1: Add Client to Realm Config

In infra/values/base/keycloak-values.yaml, add a new entry to the clients array in forte-realm.json:

{
  "clientId": "myapp",
  "name": "My Application",
  "enabled": true,
  "protocol": "openid-connect",
  "clientAuthenticatorType": "client-secret",
  "standardFlowEnabled": true,
  "directAccessGrantsEnabled": false,
  "publicClient": false,
  "redirectUris": ["https://myapp.forteapps.net/*"],
  "webOrigins": ["https://myapp.forteapps.net"],
  "defaultClientScopes": ["openid", "email", "profile"],
  "attributes": {
    "k8s.secret.sync": "true",
    "k8s.secret.namespace": "myapp",
    "k8s.secret.name": "myapp-oidc-credentials",
    "k8s.secret.client-id-key": "key",
    "k8s.secret.client-secret-key": "secret"
  }
}

Important:

• Do NOT include a "secret" field — Keycloak generates one automatically
• The attributes block tells the registrar where to create the K8s Secret
• Set client-id-key / client-secret-key to match what the consuming app expects (defaults: client-id / client-secret)

Step 2: Reference the Secret in Your Application

existingSecret: myapp-oidc-credentials

Step 3: Commit and Push

cd ~/dev/k8s/launchpad
git add infra/values/base/keycloak-values.yaml
git commit -m "Add myapp Keycloak client with auto-sync"
git push

ArgoCD will sync the Keycloak config, and the registrar CronJob will pick up the new client within 2 minutes.

Legacy Sync Attribute Reference

Attribute	Required	Default	Description
k8s.secret.sync	Yes	—	Set to "true" to enable syncing
k8s.secret.namespace	Yes	—	Target K8s namespace for the secret
k8s.secret.name	Yes	—	Name of the K8s Secret to create
k8s.secret.client-id-key	No	client-id	Field name for the client ID in the K8s Secret
k8s.secret.client-secret-key	No	client-secret	Field name for the client secret in the K8s Secret

Public CLI Client (Device-Code Login)

forte-cli is a shared public client (no secret) with the RFC 8628 device-authorization grant enabled (oauth2.device.authorization.grant.enabled: "true", standardFlowEnabled: false, directAccessGrantsEnabled: false). Downloaded skills and CLI tools that log in through the Auth Sidecar (forte-drop first) use it with <PREFIX>_CLIENT_ID=forte-cli; nothing per-tool needs to be registered in Keycloak.

It must be defined in forte-realm.json (this legacy path): the self-service registrar hardcodes publicClient: false / standardFlowEnabled: true and drops attributes, so a client-config Secret cannot produce a public device-code client. It carries no k8s.secret.sync attribute (the registrar's secret sync skips it) and is listed in the cleanup CronJob's protected clients.

Retrieving Secrets for External Deployments

The registrar always writes a central copy of every synced secret to the secrets namespace, in addition to the target namespace. This allows operators to retrieve client credentials for applications deployed outside this cluster:

# View the central copy
kubectl get secret gitea-oidc-credentials -n secrets -o yaml

# Extract the client secret for use elsewhere
kubectl get secret myapp-oidc-credentials -n secrets \
  -o jsonpath='{.data.client-secret}' | base64 -d

Registrar Behavior Notes

• The registrar runs as a CronJob every 2 minutes (concurrencyPolicy: Forbid)
• If the target namespace doesn't exist, the target write is skipped with a warning (the central copy still happens)
• A central copy is always written to the secrets namespace for every synced client
• The registrar uses the keycloak-credentials secret for admin authentication
• Created secrets have the label app.kubernetes.io/managed-by: keycloak-client-registrar

---

Troubleshooting

Application Not Deploying

Problem: Application stuck in "Syncing" state

Check ArgoCD status:

kubectl get application myapp -n argocd -o yaml

Look for errors in status.conditions.

Common causes:

• ❌ Image doesn't exist or is not accessible
• ❌ Invalid YAML syntax
• ❌ Resource quota exceeded
• ❌ Namespace conflicts
• ❌ Invalid Helm values

Solutions:

# Check image exists
docker pull ghcr.io/fortedigital/myapp:v1.0.0

# Validate YAML syntax
kubectl apply --dry-run=client -f apps/myapp.yaml

# Check ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller | grep myapp

Problem: Pods crashing (CrashLoopBackOff)

Check pod logs:

kubectl get pods -n myapp
kubectl logs -n myapp <pod-name>
kubectl describe pod -n myapp <pod-name>

Common causes:

• ❌ Application error (check logs)
• ❌ Missing environment variables
• ❌ Incorrect port configuration
• ❌ Missing secrets
• ❌ Insufficient resources

Solutions:

# Check environment variables
kubectl exec -n myapp <pod-name> -- env

# Check if secrets exist
kubectl get secrets -n myapp

# Increase resources in helm-prod-values
vim ~/dev/k8s/helm-prod-values/myapp/values.yaml

Problem: Application not accessible via domain

Check ingress:

kubectl get ingressroute -n myapp
kubectl describe ingressroute myapp -n myapp

Common causes:

• ❌ DNS not configured
• ❌ TLS certificate not issued
• ❌ Incorrect domain in values.yaml
• ❌ Traefik not routing correctly

Solutions:

# Check certificate
kubectl get certificate -n myapp

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Verify domain configuration
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep host

# Test with port-forward
kubectl port-forward -n myapp service/myapp 8080:3000
curl http://localhost:8080

Secret Issues

Problem: Secret not found

Check if SealedSecret exists:

kubectl get sealedsecret -n myapp
kubectl get secret -n myapp

Solutions:

# Check if secret is in Git
ls -l secrets/myapp-credentials-sealed.yaml

# Re-apply sealed secret
kubectl apply -f secrets/myapp-credentials-sealed.yaml

# Check sealed-secrets-controller logs
kubectl logs -n kube-system deployment/sealed-secrets-controller

Problem: Secret exists but pods can't access it

Check pod events:

kubectl describe pod -n myapp <pod-name>

Look for: Error: secret "myapp-credentials" not found

Solutions:

# Verify secret name in values.yaml matches actual secret
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep envSecretName
kubectl get secrets -n myapp

# Restart pods
kubectl rollout restart deployment myapp -n myapp

Sync Failures

Problem: ArgoCD shows "Out of Sync"

Manual sync:

# Using kubectl
kubectl patch application myapp -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"syncStrategy":{"hook":{}}}}}'

# Or via ArgoCD UI
# Click "Sync" button in UI

Check what's different:

kubectl get application myapp -n argocd -o yaml

Look at status.sync.comparedTo vs desired state.

Problem: Sync succeeds but application is "Degraded"

Check resource health:

kubectl get application myapp -n argocd -o jsonpath='{.status.resources[*].health}'

Common causes:

• ❌ Pods not ready
• ❌ Deployments not at desired replica count
• ❌ Jobs failed

Solutions:

# Check all resources in namespace
kubectl get all -n myapp

# Check pod events
kubectl get events -n myapp --sort-by='.lastTimestamp'

Getting Help

If you're stuck:

1. Check Slack notifications - Error details are often in sync failure messages
2. Check ArgoCD UI - Visual representation of what's wrong
3. Ask platform team - They have full cluster access and can debug further
4. Check documentation - Operations Runbook has more troubleshooting

---

Best Practices

Development Workflow

✅ DO:

• Develop and test locally with Docker Compose
• Use semantic versioning for releases
• Write descriptive commit messages
• Test changes in a separate namespace first (if possible)
• Monitor Slack for deployment notifications
• Document environment variables and configuration

❌ DON'T:

• Push directly to production without testing
• Use latest tag for Docker images
• Bypass CI/CD for "quick fixes"
• Hard-code configuration values
• Ignore deployment failures

Configuration Management

✅ DO:

• Keep configuration in helm-prod-values repository
• Use environment variables for config
• Document what each value does
• Use reasonable resource limits
• Enable ingress and TLS for public services

❌ DON'T:

• Hard-code config in application code
• Over-allocate resources (wastes money)
• Under-allocate resources (causes crashes)
• Use HTTP for production services

Secret Management

✅ DO:

• Use kubeseal for all secrets
• Store plain secrets in password manager
• Rotate secrets regularly
• Use different secrets per environment
• Document what each secret contains

❌ DON'T:

• Commit plain secrets
• Share secrets in Slack/email
• Reuse secrets across apps
• Log secrets in application code

Git Workflow

✅ DO:

• Use feature branches for changes
• Write clear commit messages
• Use pull requests for review
• Keep commits atomic and focused
• Tag releases in application repos

❌ DON'T:

• Push directly to main without review (for config repos)
• Make multiple unrelated changes in one commit
• Use vague commit messages ("fix", "update")
• Force-push to main branches

---

Quick Reference

Common Commands

# Check application status
kubectl get application myapp -n argocd

# View application details
kubectl describe application myapp -n argocd

# Check pods
kubectl get pods -n myapp

# View pod logs
kubectl logs -n myapp <pod-name>

# Restart deployment
kubectl rollout restart deployment myapp -n myapp

# Port-forward to service
kubectl port-forward -n myapp service/myapp 8080:3000

# Create secret
kubectl create secret generic myapp-credentials \
  --from-literal=KEY=value \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

# Seal secret
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml

Repository Locations

# Config repository
cd ~/dev/k8s/launchpad

# Helm values repository
cd ~/dev/k8s/helm-prod-values

# Helm charts repository
cd ~/dev/k8s/forte-helm

File Paths

# New application manifest
~/dev/k8s/launchpad/apps/myapp.yaml

# Application values
~/dev/k8s/helm-prod-values/myapp/values.yaml

# Sealed secrets
~/dev/k8s/launchpad/secrets/myapp-credentials-sealed.yaml

# Plain secrets (local only)
~/dev/k8s/launchpad/private/myapp-credentials.yaml

---

Next Steps

Now that you understand the basics:

1. ✅ Deploy your first application (follow steps above)
2. 📖 Read the Operations Runbook for common tasks
3. 📖 Review Technical Reference for detailed component docs
4. 📖 Understand GitOps Architecture for the big picture
5. 🚀 Start contributing!

---

Questions?

• Slack: #platform-support
• Docs: Full documentation index
• Help: Contact platform team

Last Updated: 2026-04-16