launchpad/docs/DEVELOPER-GUIDE.md

# Developer Onboarding Guide

## Table of Contents
- [Getting Started](#getting-started)
- [Prerequisites](#prerequisites)
- [Local Development Setup](#local-development-setup)
- [Understanding the Workflow](#understanding-the-workflow)
- [Deploying Your First Application](#deploying-your-first-application)
- [Updating an Existing Application](#updating-an-existing-application)
- [Working with Secrets](#working-with-secrets)
- [Enabling Authentication for Applications](#enabling-authentication-for-applications)
- [Adding a New Keycloak Client](#adding-a-new-keycloak-client)
- [Troubleshooting](#troubleshooting)
- [Best Practices](#best-practices)

---

## Getting Started

Welcome! This guide will help you understand how to develop and deploy applications on our Kubernetes cluster using GitOps principles powered by ArgoCD.

### What You'll Learn
- How our GitOps architecture works
- How to deploy a new application
- How to update existing applications
- How to manage secrets securely
- Common troubleshooting techniques

### Who This Guide Is For
- Developers deploying new applications
- Developers maintaining existing applications
- Team members who need to understand the deployment process

---

## Prerequisites

### Required Knowledge
- ✅ Basic Git workflow (clone, commit, push, pull)
- ✅ Docker basics (Dockerfile, building images)
- ✅ YAML syntax
- ✅ Basic understanding of Kubernetes concepts (pods, deployments, services)
- ⚠️ Helm knowledge (helpful but not required - templates are provided)

### Required Tools

Most developers **do NOT need kubectl access** to the cluster. You'll primarily work with Git repositories.

If you do need cluster access, install:

1. **kubectl** - Kubernetes CLI
   ```bash
   # macOS
   brew install kubectl

   # Windows
   choco install kubernetes-cli

   # Linux
   curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   ```

2. **kubeseal** - For sealing secrets
   ```bash
   # macOS
   brew install kubeseal

   # Windows
   choco install kubeseal

   # Linux
   wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
   tar -xvzf kubeseal-0.24.0-linux-amd64.tar.gz
   sudo mv kubeseal /usr/local/bin/
   ```

3. **Git** - Version control
   ```bash
   git --version  # Should already be installed
   ```

4. **Docker** - For local development
   ```bash
   # macOS/Windows: Install Docker Desktop
   # Linux: Install Docker Engine
   docker --version
   ```

### Repository Access

You'll need read/write access to these repositories:

1. **launchpad** (Config repo)
   ```bash
   git clone https://git.forteapps.net/Forte/launchpad.git
   cd launchpad
   ```

2. **helm-values** (Values repo)
   ```bash
   git clone https://git.forteapps.net/Forte/helm-prod-values.git
   cd helm-values
   ```

3. **forte-helm** (Chart repo - read-only for most developers)
   ```bash
   git clone https://git.forteapps.net/Forte/forte-helm.git
   cd forte-helm
   ```

### Cluster Access (If Needed)

If you need kubectl access, ask the platform team for:
- Kubeconfig file
- Cluster context setup instructions

Save to `~/.kube/config` and verify:
```bash
kubectl cluster-info
kubectl get nodes
```

---

## Local Development Setup

### 1. Clone the Repositories

Set up a consistent folder structure:

```bash
mkdir -p ~/dev/k8s
cd ~/dev/k8s

# Clone repositories
git clone https://git.forteapps.net/Forte/launchpad.git launchpad
git clone https://git.forteapps.net/Forte/helm-prod-values helm-prod-values
git clone https://git.forteapps.net/Forte/forte-helm forte-helm

# Your folder structure:
# ~/dev/k8s/
# ├── launchpad/           (Config repo)
# ├── helm-prod-values/    (Values repo)
# └── forte-helm/          (Chart repo)
```

### 2. Local Development Environment

Most applications use **Docker Compose** for local development:

```bash
# In your application repository
docker-compose up

# Or for frontend applications
npm install
npm run dev
```

**You DO NOT run applications locally on Kubernetes.** Use Docker Compose or native tooling (npm, python, etc.).

### 3. Understanding the Deployment Flow

```
┌─────────────────────────────────────────────────────────────────┐
│  Step 1: Develop Locally                                        │
│  - Write code in your application repository                    │
│  - Test with Docker Compose or npm/python/etc.                  │
│  - Build Docker image                                            │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 2: CI/CD Pipeline (Automated)                             │
│  - GitHub Actions builds image                                  │
│  - Pushes to container registry (GHCR, Docker Hub)              │
│  - Tags with version (e.g., v2.0.4)                             │
│  - Updates helm-values repository with new tag                  │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 3: GitOps Sync (Automated)                                │
│  - ArgoCD detects change in helm-values                         │
│  - Pulls updated configuration                                  │
│  - Syncs to Kubernetes cluster                                  │
│  - Sends Slack notification on success/failure                  │
└─────────────────────────────────────────────────────────────────┘
```

**Key Insight**: You don't deploy directly. You push code, CI/CD builds it, and ArgoCD deploys it.

---

## Understanding the Workflow

### Three-Repository Pattern

Our setup uses three repositories:

| Repository | Purpose | Who Edits | How Often |
|------------|---------|-----------|-----------|
| **forte-helm** | Helm chart templates (generic, reusable) | Platform engineers | ❌ Rarely |
| **helm-values** | Application configuration (image tag, env vars) | Developers / CI pipelines | ✅ Sometimes |
| **launchpad** | ArgoCD Applications (what gets deployed) | Platform / DevOps engineers | ✅ Per new app |

### Example: Deploying "myapp"

#### Repository: `forte-helm` (Chart Templates)
```yaml
# forteapp/templates/deployment.yaml
# Generic template used by ALL apps
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
spec:
  containers:
  - name: app
    image: "{{ .Values.app.image.repository }}:{{ .Values.app.image.tag }}"
    env:
    - name: PORT
      value: {{ .Values.app.port }}
```

#### Repository: `helm-values` (Your App Config)
```yaml
# myapp/values.yaml
# Your app's specific configuration
app:
  image:
    repository: ghcr.io/fortedigital/myapp
    tag: v1.0.0                    # CI/CD updates this
  port: 3000
  extraEnv:
  - name: API_URL
    value: https://api.example.com
```

#### Repository: `launchpad` (ArgoCD Application)
```yaml
# apps/myapp.yaml
# Tells ArgoCD to deploy your app
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  sources:
  - repoURL: https://github.com/fortedigital/forte-helm
    path: forteapp
    helm:
      valueFiles:
      - $values/myapp/values.yaml

  - repoURL: git@github.com:fortedigital/helm-values.git
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: myapp

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
```

---

## Deploying Your First Application

### Scenario: You've Built a New Application

Let's deploy a new Node.js application called "hello-world".

### Step 1: Prepare Your Application Repository

Ensure your app repository has:

1. **Dockerfile**
   ```dockerfile
   FROM node:18-alpine
   WORKDIR /app
   COPY package*.json ./
   RUN npm ci --only=production
   COPY . .
   EXPOSE 3000
   CMD ["node", "server.js"]
   ```

2. **GitHub Actions Workflow** (`.github/workflows/deploy.yml`)
   ```yaml
   name: Build and Deploy

   on:
     push:
       branches: [ main ]

   jobs:
     build:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v3

         - name: Set version
           id: version
           run: echo "VERSION=v$(date +%Y%m%d-%H%M%S)" >> $GITHUB_OUTPUT

         - name: Build and push Docker image
           run: |
             echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
             docker build -t ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }} .
             docker push ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }}

         - name: Update helm-values
           run: |
             git clone git@github.com:fortedigital/helm-values.git
             cd helm-values
             mkdir -p hello-world
             cat > hello-world/values.yaml <<EOF
             app:
               image:
                 repository: ghcr.io/fortedigital/hello-world
                 tag: ${{ steps.version.outputs.VERSION }}
             EOF
             git add hello-world/values.yaml
             git commit -m "Update hello-world to ${{ steps.version.outputs.VERSION }}"
             git push
   ```

### Step 2: Create Helm Values

Create a folder in `helm-values` repository:

```bash
cd ~/dev/k8s/helm-prod-values
mkdir -p hello-world
```

Create `hello-world/values.yaml`:
```yaml
app:
  image:
    repository: ghcr.io/fortedigital/hello-world
    tag: v1.0.0                    # Will be updated by CI/CD
    containerPort: 3000

  replicaCount: 1

  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi

  extraEnv:
  - name: PORT
    value: "3000"
  - name: NODE_ENV
    value: "production"

  envSecretName: ""                # Optional: reference to secrets

service:
  port: 3000

ingress:
  enabled: true
  host: hello-world.forteapps.net  # Your subdomain

db:
  enabled: false                   # Set to true if you need PostgreSQL
```

Commit and push:
```bash
git add hello-world/values.yaml
git commit -m "Add hello-world application values"
git push
```

### Step 3: Create ArgoCD Application Manifest

In the `launchpad` repository, create `apps/hello-world.yaml`:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello-world
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: ""
    notifications.argoproj.io/subscribe.on-sync-failed.slack: ""
    notifications.argoproj.io/subscribe.on-degraded.slack: ""
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/part-of: apps
    app.kubernetes.io/managed-by: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io

spec:
  project: default

  sources:
  # Source 1: Helm chart templates
  - repoURL: https://github.com/fortedigital/forte-helm
    path: forteapp
    targetRevision: HEAD
    helm:
      valueFiles:
      - $values/hello-world/values.yaml

  # Source 2: Helm values
  - repoURL: git@github.com:fortedigital/helm-values.git
    targetRevision: HEAD
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: hello-world

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false

    syncOptions:
    - CreateNamespace=true
    - Validate=true
    - ServerSideApply=true

    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas
```

Commit and push:
```bash
cd ~/dev/k8s/launchpad
git add apps/hello-world.yaml
git commit -m "Add hello-world application"
git push
```

### Step 4: Verify Deployment

ArgoCD will automatically detect the new application within 60 seconds.

**Option 1: Check Slack**
- Watch for sync notifications in your Slack channel
- ✅ "Application hello-world sync succeeded"

**Option 2: Check ArgoCD UI** (if you have access)
```bash
# Port forward to ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Open browser: https://localhost:8080
# Look for "hello-world" application
```

**Option 3: Check with kubectl** (if you have access)
```bash
# List ArgoCD applications
kubectl get applications -n argocd

# Check application status
kubectl get application hello-world -n argocd

# Verify pods are running
kubectl get pods -n hello-world
```

### Step 5: Access Your Application

Once deployed, access via the configured domain:

```bash
# Check if ingress is created
kubectl get ingressroute -n hello-world

# Access application
curl https://hello-world.forteapps.net
```

**⚠️ Note**: DNS must be manually configured for new subdomains. Contact the platform team to add DNS records.

---

## Updating an Existing Application

### Scenario: Deploying a Code Change

You've made changes to your application code and want to deploy them.

### Method 1: Automatic (Recommended)

**Just push to `main` branch** - CI/CD handles everything:

```bash
# In your application repository
git add .
git commit -m "Fix bug in user login"
git push origin main
```

**What Happens Next:**
1. ✅ GitHub Actions triggers
2. ✅ Builds new Docker image
3. ✅ Tags with new version (e.g., `v20260316-143022`)
4. ✅ Pushes to container registry
5. ✅ Updates `helm-values/myapp/values.yaml` with new tag
6. ✅ ArgoCD detects change
7. ✅ Syncs new version to cluster
8. ✅ Sends Slack notification

**Timeline**: ~5-10 minutes from push to deployment

### Method 2: Manual Image Tag Update

If CI/CD is not set up, manually update the image tag:

```bash
cd ~/dev/k8s/helm-prod-values

# Edit your app's values.yaml
vim myapp/values.yaml

# Change:
app:
  image:
    tag: v1.0.0  # Old version
# To:
app:
  image:
    tag: v1.0.1  # New version

# Commit and push
git add myapp/values.yaml
git commit -m "Update myapp to v1.0.1"
git push
```

ArgoCD will sync within 60 seconds.

### Method 3: Configuration Changes

To update environment variables, resources, or other config:

```bash
cd ~/dev/k8s/helm-prod-values
vim myapp/values.yaml
```

Example changes:

```yaml
app:
  # Increase resources
  resources:
    requests:
      cpu: 200m      # Was 100m
      memory: 256Mi  # Was 128Mi

  # Add new environment variable
  extraEnv:
  - name: API_URL
    value: https://api.example.com
  - name: DEBUG          # NEW
    value: "true"        # NEW

  # Enable HPA
  hpa:
    enabled: true        # Was false
    minReplicas: 2
    maxReplicas: 10
```

Commit and push:
```bash
git add myapp/values.yaml
git commit -m "Increase myapp resources and enable HPA"
git push
```

### Method 4: Application Manifest Changes

To change ArgoCD sync behavior, namespace, or other meta-config:

```bash
cd ~/dev/k8s/launchpad
vim apps/myapp.yaml
```

Example changes:

```yaml
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: false    # Disable self-healing temporarily
```

Commit and push:
```bash
git add apps/myapp.yaml
git commit -m "Disable self-healing for myapp"
git push
```

---

## Working with Secrets

### Understanding Secret Management

**NEVER commit plain secrets to Git.** We use **Sealed Secrets** to encrypt secrets before committing.

### Creating a New Secret

#### Step 1: Create Plain Secret Locally

```bash
cd ~/dev/k8s/launchpad

# Create secret in private/ folder (Git-ignored)
kubectl create secret generic myapp-credentials \
  --from-literal=API_KEY=your-secret-key-here \
  --from-literal=DB_PASSWORD=super-secret-password \
  --dry-run=client -o yaml > private/myapp-credentials.yaml
```

**DO NOT commit this file!** It's in `private/` which is Git-ignored.

#### Step 2: Seal the Secret

Get the public certificate (one-time setup):

```bash
# Fetch public cert from cluster
kubeseal --fetch-cert \
  --controller-name=sealed-secrets-controller \
  --controller-namespace=kube-system \
  > pub-cert.pem
```

Seal your secret:

```bash
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml
```

#### Step 3: Commit Sealed Secret

```bash
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Add myapp credentials (sealed)"
git push
```

#### Step 4: Reference Secret in Application

Update your `helm-values/myapp/values.yaml`:

```yaml
app:
  envSecretName: "myapp-credentials"  # References the SealedSecret
```

Commit and push:
```bash
cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Reference myapp credentials"
git push
```

### Updating a Secret

To update an existing secret:

```bash
# 1. Create new version of secret
kubectl create secret generic myapp-credentials \
  --from-literal=API_KEY=new-key-here \
  --from-literal=DB_PASSWORD=new-password \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

# 2. Seal it
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml

# 3. Commit sealed version
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Update myapp credentials"
git push

# 4. Restart pods to pick up new secret
kubectl rollout restart deployment myapp -n myapp
```

### Secret Best Practices

✅ **DO**:
- Store secrets in `private/` folder locally
- Always seal secrets before committing
- Delete plain secrets after sealing
- Use meaningful secret names
- Document what each secret contains

❌ **DON'T**:
- Commit plain secrets to Git
- Share secrets via Slack/email
- Hard-code secrets in code
- Use the same secret across multiple environments
- Store secrets in Docker images

### Where Secrets Are Stored

```
┌─────────────────────────────────────────────────────────────┐
│  Location                │  Content           │  Committed?│
├──────────────────────────┼────────────────────┼────────────┤
│  private/                │  Plain secrets     │  ❌ NO      │
│  secrets/                │  Sealed secrets    │  ✅ YES     │
│  Kubernetes cluster      │  Unsealed secrets  │  N/A       │
└─────────────────────────────────────────────────────────────┘
```

**Sealed Secrets Controller** in the cluster decrypts sealed secrets automatically.

---

## Enabling Authentication for Applications

The cluster supports automatic authentication sidecar injection for applications via Kyverno policies. This allows you to add authentication to your applications without modifying application code.

### How It Works

When you enable authentication in your Helm values, the Kyverno policy automatically:
1. ✅ Injects an authentication sidecar container into your pod
2. ✅ Routes all incoming traffic through the auth sidecar (port 8080)
3. ✅ Validates credentials before forwarding requests to your application
4. ✅ Creates necessary secrets (if they don't exist)
5. ✅ Adds a NetworkPolicy to restrict ingress

**Architecture**:
```
Internet → Traefik → Service:8080 → Auth Sidecar:8080 → localhost → Your App:3000
                                         │
                                         ├─ Validates credentials
                                         └─ Forwards if valid
```

### Authentication Modes

Three authentication modes are supported:
1. **Token-based**: Static tokens (simple, good for service-to-service or internal apps)
2. **OIDC**: OpenID Connect (full SSO, good for user-facing apps)
3. **MCP**: OAuth 2.0 for MCP servers via RFC 9728 / RFC 7591 (good for MCP tool servers requiring OAuth-based access control)

---

### Token-Based Authentication

#### Step 1: Configure Helm Values

```yaml
# In helm-values/myapp/values.yaml
auth:
  enabled: true
  type: token                    # Token mode (default)
  tokens:
  - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823
  - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7
```

#### Step 2: Generate Token (if needed)

```bash
# Generate a secure random token
openssl rand -hex 32

# Or using Python
python3 -c "import secrets; print(secrets.token_hex(32))"

# Example output:
# d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823
```

#### Step 3: Deploy Application

Commit and push your changes:
```bash
cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Enable token auth for myapp"
git push
```

ArgoCD will sync, and the Kyverno policy will:
- Inject the auth sidecar container
- Create an `auth-tokens` Secret with your tokens
- Configure the sidecar to validate against these tokens

#### Step 4: Access Application

Use your token in the `Authorization` header:

```bash
# Access application with token
curl -H "Authorization: Bearer d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823" \
  https://myapp.forteapps.net/api/data

# Without token (will be rejected)
curl https://myapp.forteapps.net/api/data
# Response: 401 Unauthorized
```

#### Advanced: Custom Secret Name

To use a different secret for tokens:

```yaml
# In Helm values
auth:
  enabled: true
  type: token
  tokens: []                     # Empty - using external secret

# Tokens will be read from custom secret
```

Then reference it via annotation (configured by Helm chart automatically):
```yaml
# Helm chart sets this annotation:
policies.forteapps.io/auth-token-secret-name: "myapp-auth-tokens"
```

Create the secret manually:
```bash
kubectl create secret generic myapp-auth-tokens \
  --from-file=tokens=tokens.txt \
  --namespace=myapp
```

---

### OIDC Authentication

OIDC mode integrates with identity providers like Keycloak, Okta, Auth0, Azure AD, etc.

#### Step 1: Configure Identity Provider

In your identity provider (e.g., Keycloak):
1. Create a new client (e.g., `myapp`)
2. Set redirect URI: `https://myapp.forteapps.net/auth/callback`
3. Note the **Client ID** and **Client Secret**
4. Note the **Authority URL** (e.g., `https://keycloak.forteapps.net/realms/master`)

#### Step 2: Create OIDC Secret

```bash
# Create plain secret
kubectl create secret generic auth-oidc \
  --from-literal=client-secret=your-oidc-client-secret \
  --from-literal=cookie-secret=$(openssl rand -hex 32) \
  --namespace=myapp \
  --dry-run=client -o yaml > private/myapp-auth-oidc.yaml

# Seal it
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  --namespace=myapp \
  < private/myapp-auth-oidc.yaml \
  > secrets/myapp-auth-oidc-sealed.yaml

# Commit sealed secret
cd ~/dev/k8s/launchpad
git add secrets/myapp-auth-oidc-sealed.yaml
git commit -m "Add OIDC secrets for myapp"
git push

# Clean up
rm private/myapp-auth-oidc.yaml
```

#### Step 3: Configure Helm Values

```yaml
# In helm-values/myapp/values.yaml
auth:
  enabled: true
  type: oidc                     # OIDC mode
  oidc:
    authority: https://keycloak.forteapps.net/realms/master
    clientId: myapp
    scopes: "openid,profile,email"
    callbackPath: /auth/callback
```

#### Step 4: Deploy Application

```bash
cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Enable OIDC auth for myapp"
git push
```

#### Step 5: Access Application

When users access `https://myapp.forteapps.net`:
1. They're redirected to the identity provider login page
2. After successful login, redirected back to `/auth/callback`
3. Session cookie is set
4. Subsequent requests are authenticated via cookie

**User flow**:
```
User → https://myapp.forteapps.net
  ↓
Redirect → https://keycloak.forteapps.net/login
  ↓
Login successful → Redirect with auth code
  ↓
https://myapp.forteapps.net/auth/callback?code=xyz
  ↓
Auth sidecar exchanges code for tokens
  ↓
Sets session cookie
  ↓
Redirects to application → https://myapp.forteapps.net
  ↓
User sees application (authenticated)
```

---

### Authentication Configuration Reference

#### Helm Values Schema

```yaml
auth:
  enabled: false                 # Enable/disable authentication
  type: token                    # "token", "oidc", or "mcp"

  # Token mode configuration
  tokens: []                     # List of valid bearer tokens
  # - token1
  # - token2

  # OIDC mode configuration
  oidc:
    authority: ""                # OIDC provider URL (required for OIDC)
    clientId: ""                 # OIDC client ID (required for OIDC)
    scopes: "openid,profile,email"  # OIDC scopes (optional)
    callbackPath: /auth/callback    # OAuth callback path (optional)

  # MCP mode configuration (RFC 9728 / RFC 7591)
  mcp:
    resource: ""                 # Protected resource URL (required for MCP)
    authority: ""                # Authorization server URL (required for MCP)
    scopes: "read,write"         # Supported scopes (optional)
```

#### Annotations Set by Helm Chart

When `auth.enabled: true`, the Helm chart sets these pod annotations:

**Token mode**:
```yaml
policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "token"
policies.forteapps.io/auth-token-secret-name: "auth-tokens"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"
```

**OIDC mode**:
```yaml
policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "oidc"
policies.forteapps.io/auth-oidc-authority: "https://keycloak.forteapps.net/realms/master"
policies.forteapps.io/auth-oidc-client-id: "myapp"
policies.forteapps.io/auth-oidc-scopes: "openid,profile,email"
policies.forteapps.io/auth-oidc-callback-path: "/auth/callback"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"
```

**MCP mode** (OAuth 2.0 for MCP servers):
```yaml
policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "mcp"
policies.forteapps.io/auth-mcp-resource: "https://mcp.forteapps.net"
policies.forteapps.io/auth-mcp-authority: "https://keycloak.forteapps.net/realms/master"
policies.forteapps.io/auth-mcp-scopes: "read,write"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"
```

#### Sidecar Configuration

The auth sidecar container:
- **Image**: `ghcr.io/fortedigital/auth-sidecar:latest`
- **Port**: 8080
- **Resources**: 10m CPU / 32Mi memory (requests), 50m CPU / 64Mi memory (limits)
- **Health checks**: `/healthz` endpoint
- **Security**: Read-only root filesystem, no privilege escalation

#### Advanced: Custom Sidecar Image

To use a different auth sidecar image:

```yaml
# These annotations can be set in the Helm chart template if needed
policies.forteapps.io/auth-image: "your-registry/your-auth-proxy"
policies.forteapps.io/auth-image-version: "v1.2.3"
```

---

### Authentication Examples

#### Example 1: Internal API with Token Auth

```yaml
# helm-values/internal-api/values.yaml
app:
  image:
    repository: ghcr.io/company/internal-api
    tag: v1.0.0

auth:
  enabled: true
  type: token
  tokens:
  - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823  # Service A
  - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7  # Service B

ingress:
  enabled: true
  host: internal-api.forteapps.net
```

**Usage**:
```bash
# Service A calls API
curl -H "Authorization: Bearer d4f88f..." \
  https://internal-api.forteapps.net/api/endpoint
```

#### Example 2: User-Facing App with OIDC

```yaml
# helm-values/web-app/values.yaml
app:
  image:
    repository: ghcr.io/company/web-app
    tag: v2.1.0

auth:
  enabled: true
  type: oidc
  oidc:
    authority: https://auth.company.com/realms/employees
    clientId: web-app-prod
    scopes: "openid,profile,email,groups"
    callbackPath: /auth/callback

ingress:
  enabled: true
  host: web-app.forteapps.net
```

**With sealed OIDC secret**:
```bash
# Create and seal secret
kubectl create secret generic auth-oidc \
  --from-literal=client-secret=super-secret-value \
  --from-literal=cookie-secret=$(openssl rand -hex 32) \
  --namespace=web-app \
  --dry-run=client -o yaml | \
  kubeseal --format=yaml --cert=pub-cert.pem --namespace=web-app \
  > secrets/web-app-auth-oidc-sealed.yaml
```

#### Example 3: MCP Server with OAuth 2.0

```yaml
# helm-values/mcp-server/values.yaml
app:
  image:
    repository: ghcr.io/company/mcp-server
    tag: v1.0.0

auth:
  enabled: true
  type: mcp
  mcp:
    resource: https://mcp-server.forteapps.net
    authority: https://auth.company.com/realms/mcp
    scopes: "read,write,admin"

ingress:
  enabled: true
  host: mcp-server.forteapps.net
```

The MCP auth mode implements RFC 9728 (OAuth 2.0 Protected Resource Metadata) for authorization server discovery and RFC 7591 (OAuth 2.0 Dynamic Client Registration) for automatic client registration. MCP clients discover the authorization server and scopes from the `/.well-known/oauth-protected-resource` endpoint served by the sidecar.

#### Example 4: Disabling Authentication

```yaml
# helm-values/public-api/values.yaml
auth:
  enabled: false                 # No authentication

ingress:
  enabled: true
  host: public-api.forteapps.net
```

---

### Troubleshooting Authentication

#### Issue: 401 Unauthorized (Token Mode)

**Check token validity**:
```bash
# Get auth-tokens secret
kubectl get secret auth-tokens -n myapp -o yaml

# Decode tokens
kubectl get secret auth-tokens -n myapp \
  -o jsonpath='{.data.tokens}' | base64 -d

# Verify your token is in the list
```

**Test with different token**:
```bash
curl -v -H "Authorization: Bearer YOUR-TOKEN-HERE" \
  https://myapp.forteapps.net/
```

#### Issue: OIDC Login Loop

**Check OIDC configuration**:
```bash
# Verify auth-oidc secret exists
kubectl get secret auth-oidc -n myapp

# Check sidecar logs
kubectl logs -n myapp <pod-name> -c authn

# Common issues:
# - Wrong authority URL
# - Wrong client ID
# - Missing client-secret in auth-oidc Secret
# - Redirect URI not configured in identity provider
```

**Verify redirect URI** in your identity provider matches:
```
https://<your-app-domain>/auth/callback
```

#### Issue: Auth Sidecar Not Injected

**Check pod annotations**:
```bash
kubectl get pod -n myapp <pod-name> -o yaml | grep policies.forteapps.io

# Should show:
# policies.forteapps.io/auth: "true"
```

**Check Kyverno policy**:
```bash
kubectl get clusterpolicy inject-auth-sidecar
kubectl describe clusterpolicy inject-auth-sidecar
```

**Check Kyverno logs**:
```bash
kubectl logs -n kyverno deployment/kyverno | grep inject-auth
```

#### Issue: Auth Sidecar Crashes

**Check sidecar logs**:
```bash
kubectl logs -n myapp <pod-name> -c authn
```

**Common causes**:
- Missing secret (auth-tokens or auth-oidc)
- Invalid OIDC configuration
- Can't reach OIDC authority URL
- Network policy blocking outbound OIDC requests

---

### Authentication Best Practices

✅ **DO**:
- Use OIDC for user-facing applications
- Use token auth for service-to-service communication
- Rotate tokens and secrets regularly
- Use strong random tokens (32+ bytes)
- Store client secrets in SealedSecrets
- Test authentication before deploying to production
- Document which tokens/users have access

❌ **DON'T**:
- Share tokens between environments
- Commit tokens to application code
- Use predictable tokens
- Reuse tokens across multiple applications
- Disable authentication on sensitive APIs
- Log tokens or secrets

---

## Adding a New Keycloak Client

When you need an application to authenticate via Keycloak (OIDC), you can add a client definition to the realm config. The secret syncer automatically extracts the Keycloak-generated client secret into a Kubernetes Secret that your application can reference — no manual secret management needed.

### How It Works

1. You define a client in `forte-realm.json` (inside `keycloak-values.yaml`) **without** a `secret` field
2. Keycloak auto-generates a cryptographically strong secret on first creation
3. An ArgoCD **PostSync Job** (`keycloak-secret-syncer`) runs after each Keycloak sync:
   - Authenticates to the Keycloak Admin API
   - Finds clients with `k8s.secret.sync: "true"` in their attributes
   - Extracts the auto-generated secret for each client
   - Creates/updates a K8s Secret in the target namespace with `client-id` and `client-secret` keys
4. Your application references the syncer-created Secret

### Step 1: Add Client to Realm Config

In `infra/values/keycloak-values.yaml`, add a new entry to the `clients` array in `forte-realm.json`:

```json
{
  "clientId": "myapp",
  "name": "My Application",
  "enabled": true,
  "protocol": "openid-connect",
  "clientAuthenticatorType": "client-secret",
  "standardFlowEnabled": true,
  "directAccessGrantsEnabled": false,
  "publicClient": false,
  "redirectUris": ["https://myapp.forteapps.net/*"],
  "webOrigins": ["https://myapp.forteapps.net"],
  "defaultClientScopes": ["openid", "email", "profile"],
  "attributes": {
    "k8s.secret.sync": "true",
    "k8s.secret.namespace": "myapp",
    "k8s.secret.name": "myapp-oidc-credentials",
    "k8s.secret.client-id-key": "key",
    "k8s.secret.client-secret-key": "secret"
  }
}
```

**Important**:
- Do **NOT** include a `"secret"` field — Keycloak generates one automatically
- The `attributes` block tells the syncer where to create the K8s Secret
- The target namespace must exist before the syncer runs (ArgoCD creates it via `CreateNamespace=true`)
- Set `client-id-key` / `client-secret-key` to match what the consuming app expects (defaults: `client-id` / `client-secret`)

### Step 2: Reference the Secret in Your Application

In your application's Helm values, reference the syncer-created secret:

```yaml
# In helm-values/myapp/values.yaml (or inline in values file)
# The secret will have keys: client-id, client-secret
existingSecret: myapp-oidc-credentials
key: client-secret
```

For Gitea-style oauth config:
```yaml
oauth:
- name: "Forte"
  provider: "openidConnect"
  existingSecret: myapp-oidc-credentials # Gitea expects "key" and "secret" as fields
  autoDiscoverUrl: "https://id.forteapps.net/realms/forte/.well-known/openid-configuration"
```

### Step 3: Commit and Push

```bash
cd ~/dev/k8s/launchpad
git add infra/values/keycloak-values.yaml
git commit -m "Add myapp Keycloak client with auto-sync"
git push
```

ArgoCD will:
1. Sync the Keycloak config (keycloakConfigCli creates the client)
2. Run the PostSync syncer Job
3. The syncer creates `myapp-oidc-credentials` in the `myapp` namespace

### Step 4: Verify

```bash
# Check the syncer job ran successfully
kubectl get jobs -n keycloak
kubectl logs -n keycloak job/keycloak-secret-syncer

# Verify the secret was created
kubectl get secret myapp-oidc-credentials -n myapp -o yaml

# Check the secret has the expected keys
kubectl get secret myapp-oidc-credentials -n myapp -o jsonpath='{.data.client-id}' | base64 -d
kubectl get secret myapp-oidc-credentials -n myapp -o jsonpath='{.data.client-secret}' | base64 -d
```

### Sync Attribute Reference

| Attribute | Required | Default | Description |
|-----------|----------|---------|-------------|
| `k8s.secret.sync` | Yes | — | Set to `"true"` to enable syncing |
| `k8s.secret.namespace` | Yes | — | Target K8s namespace for the secret |
| `k8s.secret.name` | Yes | — | Name of the K8s Secret to create |
| `k8s.secret.client-id-key` | No | `client-id` | Field name for the client ID in the K8s Secret |
| `k8s.secret.client-secret-key` | No | `client-secret` | Field name for the client secret in the K8s Secret |

**Note on key names:** Different applications expect different field names. For example, the Gitea Helm chart expects `key` and `secret`, while a generic OIDC consumer might expect `client-id` and `client-secret`. Use the optional key attributes to match what the consuming application expects.

### Retrieving Secrets for External Deployments

The syncer always writes a **central copy** of every synced secret to the `secrets` namespace, in addition to the target namespace. This allows operators to retrieve client credentials for applications deployed outside this cluster:

```bash
# View the central copy
kubectl get secret gitea-oidc-credentials -n secrets -o yaml

# Extract the client secret for use elsewhere
kubectl get secret myapp-oidc-credentials -n secrets \
  -o jsonpath='{.data.client-secret}' | base64 -d
```

This is useful when an application runs on a separate cluster or external infrastructure and needs the Keycloak-generated OIDC credentials provisioned manually (e.g., via a SealedSecret on the remote side).

### Syncer Behavior Notes

- The syncer runs as an ArgoCD **PostSync hook** — it executes after all Keycloak resources are healthy
- `BeforeHookCreation` delete policy ensures old Job is cleaned up before each run
- If the target namespace doesn't exist, the target write is skipped with a warning (the central copy still happens)
- A central copy is **always** written to the `secrets` namespace for every synced client
- The syncer uses the `keycloak-credentials` secret for admin authentication
- Created secrets have the label `app.kubernetes.io/managed-by: keycloak-secret-syncer`

---

## Troubleshooting

### Application Not Deploying

#### Problem: Application stuck in "Syncing" state

**Check ArgoCD status:**
```bash
kubectl get application myapp -n argocd -o yaml
```

Look for errors in `status.conditions`.

**Common causes:**
- ❌ Image doesn't exist or is not accessible
- ❌ Invalid YAML syntax
- ❌ Resource quota exceeded
- ❌ Namespace conflicts
- ❌ Invalid Helm values

**Solutions:**
```bash
# Check image exists
docker pull ghcr.io/fortedigital/myapp:v1.0.0

# Validate YAML syntax
kubectl apply --dry-run=client -f apps/myapp.yaml

# Check ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller | grep myapp
```

#### Problem: Pods crashing (CrashLoopBackOff)

**Check pod logs:**
```bash
kubectl get pods -n myapp
kubectl logs -n myapp <pod-name>
kubectl describe pod -n myapp <pod-name>
```

**Common causes:**
- ❌ Application error (check logs)
- ❌ Missing environment variables
- ❌ Incorrect port configuration
- ❌ Missing secrets
- ❌ Insufficient resources

**Solutions:**
```bash
# Check environment variables
kubectl exec -n myapp <pod-name> -- env

# Check if secrets exist
kubectl get secrets -n myapp

# Increase resources in helm-values
vim ~/dev/k8s/helm-prod-values/myapp/values.yaml
```

#### Problem: Application not accessible via domain

**Check ingress:**
```bash
kubectl get ingressroute -n myapp
kubectl describe ingressroute myapp -n myapp
```

**Common causes:**
- ❌ DNS not configured
- ❌ TLS certificate not issued
- ❌ Incorrect domain in values.yaml
- ❌ Traefik not routing correctly

**Solutions:**
```bash
# Check certificate
kubectl get certificate -n myapp

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Verify domain configuration
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep host

# Test with port-forward
kubectl port-forward -n myapp service/myapp 8080:3000
curl http://localhost:8080
```

### Secret Issues

#### Problem: Secret not found

**Check if SealedSecret exists:**
```bash
kubectl get sealedsecret -n myapp
kubectl get secret -n myapp
```

**Solutions:**
```bash
# Check if secret is in Git
ls -l secrets/myapp-credentials-sealed.yaml

# Re-apply sealed secret
kubectl apply -f secrets/myapp-credentials-sealed.yaml

# Check sealed-secrets-controller logs
kubectl logs -n kube-system deployment/sealed-secrets-controller
```

#### Problem: Secret exists but pods can't access it

**Check pod events:**
```bash
kubectl describe pod -n myapp <pod-name>
```

Look for: `Error: secret "myapp-credentials" not found`

**Solutions:**
```bash
# Verify secret name in values.yaml matches actual secret
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep envSecretName
kubectl get secrets -n myapp

# Restart pods
kubectl rollout restart deployment myapp -n myapp
```

### Sync Failures

#### Problem: ArgoCD shows "Out of Sync"

**Manual sync:**
```bash
# Using kubectl
kubectl patch application myapp -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"syncStrategy":{"hook":{}}}}}'

# Or via ArgoCD UI
# Click "Sync" button in UI
```

**Check what's different:**
```bash
kubectl get application myapp -n argocd -o yaml
```

Look at `status.sync.comparedTo` vs desired state.

#### Problem: Sync succeeds but application is "Degraded"

**Check resource health:**
```bash
kubectl get application myapp -n argocd -o jsonpath='{.status.resources[*].health}'
```

**Common causes:**
- ❌ Pods not ready
- ❌ Deployments not at desired replica count
- ❌ Jobs failed

**Solutions:**
```bash
# Check all resources in namespace
kubectl get all -n myapp

# Check pod events
kubectl get events -n myapp --sort-by='.lastTimestamp'
```

### Getting Help

If you're stuck:

1. **Check Slack notifications** - Error details are often in sync failure messages
2. **Check ArgoCD UI** - Visual representation of what's wrong
3. **Ask platform team** - They have full cluster access and can debug further
4. **Check documentation** - [Operations Runbook](OPERATIONS-RUNBOOK.md) has more troubleshooting

---

## Best Practices

### Development Workflow

✅ **DO**:
- Develop and test locally with Docker Compose
- Use semantic versioning for releases
- Write descriptive commit messages
- Test changes in a separate namespace first (if possible)
- Monitor Slack for deployment notifications
- Document environment variables and configuration

❌ **DON'T**:
- Push directly to production without testing
- Use `latest` tag for Docker images
- Bypass CI/CD for "quick fixes"
- Hard-code configuration values
- Ignore deployment failures

### Configuration Management

✅ **DO**:
- Keep configuration in `helm-values` repository
- Use environment variables for config
- Document what each value does
- Use reasonable resource limits
- Enable ingress and TLS for public services

❌ **DON'T**:
- Hard-code config in application code
- Over-allocate resources (wastes money)
- Under-allocate resources (causes crashes)
- Use HTTP for production services

### Secret Management

✅ **DO**:
- Use kubeseal for all secrets
- Store plain secrets in password manager
- Rotate secrets regularly
- Use different secrets per environment
- Document what each secret contains

❌ **DON'T**:
- Commit plain secrets
- Share secrets in Slack/email
- Reuse secrets across apps
- Log secrets in application code

### Git Workflow

✅ **DO**:
- Use feature branches for changes
- Write clear commit messages
- Use pull requests for review
- Keep commits atomic and focused
- Tag releases in application repos

❌ **DON'T**:
- Push directly to `main` without review (for config repos)
- Make multiple unrelated changes in one commit
- Use vague commit messages ("fix", "update")
- Force-push to main branches

---

## Quick Reference

### Common Commands

```bash
# Check application status
kubectl get application myapp -n argocd

# View application details
kubectl describe application myapp -n argocd

# Check pods
kubectl get pods -n myapp

# View pod logs
kubectl logs -n myapp <pod-name>

# Restart deployment
kubectl rollout restart deployment myapp -n myapp

# Port-forward to service
kubectl port-forward -n myapp service/myapp 8080:3000

# Create secret
kubectl create secret generic myapp-credentials \
  --from-literal=KEY=value \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

# Seal secret
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml
```

### Repository Locations

```bash
# Config repository
cd ~/dev/k8s/launchpad

# Helm values repository
cd ~/dev/k8s/helm-prod-values

# Helm charts repository
cd ~/dev/k8s/forte-helm
```

### File Paths

```bash
# New application manifest
~/dev/k8s/launchpad/apps/myapp.yaml

# Application values
~/dev/k8s/helm-prod-values/myapp/values.yaml

# Sealed secrets
~/dev/k8s/launchpad/secrets/myapp-credentials-sealed.yaml

# Plain secrets (local only)
~/dev/k8s/launchpad/private/myapp-credentials.yaml
```

---

## Next Steps

Now that you understand the basics:

1. ✅ Deploy your first application (follow steps above)
2. 📖 Read the [Operations Runbook](OPERATIONS-RUNBOOK.md) for common tasks
3. 📖 Review [Technical Reference](REFERENCE.md) for detailed component docs
4. 📖 Understand [GitOps Architecture](GITOPS-ARCHITECTURE.md) for the big picture
5. 🚀 Start contributing!

---

**Questions?**
- Slack: #platform-support
- Docs: [Full documentation index](README.md)
- Help: Contact platform team

**Last Updated**: 2026-04-16