# Developer Onboarding Guide

## Table of Contents
- [Getting Started](#getting-started)
- [Prerequisites](#prerequisites)
- [Local Development Setup](#local-development-setup)
- [Understanding the Workflow](#understanding-the-workflow)
- [Deploying Your First Application](#deploying-your-first-application)
- [Updating an Existing Application](#updating-an-existing-application)
- [Working with Secrets](#working-with-secrets)
- [Enabling Authentication for Applications](#enabling-authentication-for-applications)
- [Adding a New Keycloak Client](#adding-a-new-keycloak-client)
- [Troubleshooting](#troubleshooting)
- [Best Practices](#best-practices)

---

## Getting Started

Welcome! This guide will help you understand how to develop and deploy applications on our Kubernetes cluster using GitOps principles powered by ArgoCD.

### What You'll Learn
- How our GitOps architecture works
- How to deploy a new application
- How to update existing applications
- How to manage secrets securely
- Common troubleshooting techniques

### Who This Guide Is For
- Developers deploying new applications
- Developers maintaining existing applications
- Team members who need to understand the deployment process

---

## Prerequisites

### Required Knowledge
- ✅ Basic Git workflow (clone, commit, push, pull)
- ✅ Docker basics (Dockerfile, building images)
- ✅ YAML syntax
- ✅ Basic understanding of Kubernetes concepts (pods, deployments, services)
- ⚠️ Helm knowledge (helpful but not required - templates are provided)

### Required Tools

Most developers **do NOT need kubectl access** to the cluster. You'll primarily work with Git repositories.

If you do need cluster access, install:

1. **kubectl** - Kubernetes CLI
   ```bash
   # macOS
   brew install kubectl

   # Windows
   choco install kubernetes-cli

   # Linux
   curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   ```

2. **kubeseal** - For sealing secrets
   ```bash
   # macOS
   brew install kubeseal

   # Windows
   choco install kubeseal

   # Linux
   wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
   tar -xvzf kubeseal-0.24.0-linux-amd64.tar.gz
   sudo mv kubeseal /usr/local/bin/
   ```

3. **Git** - Version control
   ```bash
   git --version  # Should already be installed
   ```

4. **Docker** - For local development
   ```bash
   # macOS/Windows: Install Docker Desktop
   # Linux: Install Docker Engine
   docker --version
   ```

### Repository Access

You'll need read/write access to these repositories:

1. **launchpad** (Config repo)
   ```bash
   git clone https://git.forteapps.net/Forte/launchpad.git
   cd launchpad
   ```

2. **helm-prod-values** (Values repo)
   ```bash
   git clone https://git.forteapps.net/Forte/helm-prod-values.git
   cd helm-prod-values
   ```

3. **forte-helm** (Chart repo - read-only for most developers)
   ```bash
   git clone https://git.forteapps.net/Forte/forte-helm.git
   cd forte-helm
   ```

### Cluster Access (If Needed)

If you need kubectl access, ask the platform team for:
- Kubeconfig file
- Cluster context setup instructions

Save to `~/.kube/config` and verify:
```bash
kubectl cluster-info
kubectl get nodes
```

---

## Local Development Setup

### 1. Clone the Repositories

Set up a consistent folder structure:

```bash
mkdir -p ~/dev/k8s
cd ~/dev/k8s

# Clone repositories
git clone https://git.forteapps.net/Forte/launchpad.git launchpad
git clone https://git.forteapps.net/Forte/helm-prod-values helm-prod-values
git clone https://git.forteapps.net/Forte/forte-helm forte-helm

# Your folder structure:
# ~/dev/k8s/
# ├── launchpad/           (Config repo)
# ├── helm-prod-values/    (Values repo)
# └── forte-helm/          (Chart repo)
```

### 2. Local Development Environment

Most applications use **Docker Compose** for local development:

```bash
# In your application repository
docker-compose up

# Or for frontend applications
npm install
npm run dev
```

**You DO NOT run applications locally on Kubernetes.** Use Docker Compose or native tooling (npm, python, etc.).

### 3. Understanding the Deployment Flow

```
┌─────────────────────────────────────────────────────────────────┐
│  Step 1: Develop Locally                                        │
│  - Write code in your application repository                    │
│  - Test with Docker Compose or npm/python/etc.                  │
│  - Build Docker image                                            │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 2: CI/CD Pipeline (Automated)                             │
│  - GitHub Actions builds image                                  │
│  - Pushes to container registry (GHCR, Docker Hub)              │
│  - Tags with version (e.g., v2.0.4)                             │
│  - Updates helm-prod-values repository with new tag                  │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 3: GitOps Sync (Automated)                                │
│  - ArgoCD detects change in helm-prod-values                         │
│  - Pulls updated configuration                                  │
│  - Syncs to Kubernetes cluster                                  │
│  - Sends Slack notification on success/failure                  │
└─────────────────────────────────────────────────────────────────┘
```

**Key Insight**: You don't deploy directly. You push code, CI/CD builds it, and ArgoCD deploys it.

---

## Understanding the Workflow

### Three-Repository Pattern

Our setup uses three repositories:

| Repository | Purpose | Who Edits | How Often |
|------------|---------|-----------|-----------|
| **forte-helm** | Helm chart templates (generic, reusable) | Platform engineers | ❌ Rarely |
| **helm-prod-values** | Application configuration (image tag, env vars) | Developers / CI pipelines | ✅ Sometimes |
| **launchpad** | ArgoCD Applications (what gets deployed) | Platform / DevOps engineers | ✅ Per new app |

### Example: Deploying "myapp"

#### Repository: `forte-helm` (Chart Templates)
```yaml
# forteapp/templates/deployment.yaml
# Generic template used by ALL apps
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
spec:
  containers:
  - name: app
    image: "{{ .Values.app.image.repository }}:{{ .Values.app.image.tag }}"
    env:
    - name: PORT
      value: {{ .Values.app.port }}
```

#### Repository: `helm-prod-values` (Your App Config)
```yaml
# myapp/values.yaml
# Your app's specific configuration
app:
  image:
    repository: ghcr.io/fortedigital/myapp
    tag: v1.0.0                    # CI/CD updates this
  port: 3000
  extraEnv:
  - name: API_URL
    value: https://api.example.com
```

#### Repository: `launchpad` (ArgoCD Application)
```yaml
# apps/myapp.yaml
# Tells ArgoCD to deploy your app
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  sources:
  - repoURL: https://git.forteapps.net/Forte/forte-helm
    path: forteapp
    helm:
      valueFiles:
      - $values/myapp/values.yaml

  - repoURL: git@github.com:fortedigital/helm-prod-values.git
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: myapp

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
```

---

## Deploying Your First Application

### Scenario: You've Built a New Application

Let's deploy a new Node.js application called "hello-world".

### Step 1: Prepare Your Application Repository

Ensure your app repository has:

1. **Dockerfile**
   ```dockerfile
   FROM node:18-alpine
   WORKDIR /app
   COPY package*.json ./
   RUN npm ci --only=production
   COPY . .
   EXPOSE 3000
   CMD ["node", "server.js"]
   ```

2. **GitHub Actions Workflow** (`.github/workflows/deploy.yml`)
   ```yaml
   name: Build and Deploy

   on:
     push:
       branches: [ main ]

   jobs:
     build:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v3

         - name: Set version
           id: version
           run: echo "VERSION=v$(date +%Y%m%d-%H%M%S)" >> $GITHUB_OUTPUT

         - name: Build and push Docker image
           run: |
             echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
             docker build -t ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }} .
             docker push ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }}

         - name: Update helm-prod-values
           run: |
             git clone git@github.com:fortedigital/helm-prod-values.git
             cd helm-prod-values
             mkdir -p hello-world
             cat > hello-world/values.yaml <<EOF
             app:
               image:
                 repository: ghcr.io/fortedigital/hello-world
                 tag: ${{ steps.version.outputs.VERSION }}
             EOF
             git add hello-world/values.yaml
             git commit -m "Update hello-world to ${{ steps.version.outputs.VERSION }}"
             git push
   ```

### Step 2: Create Helm Values

Create a folder in `helm-prod-values` repository:

```bash
cd ~/dev/k8s/helm-prod-values
mkdir -p hello-world
```

Create `hello-world/values.yaml`:
```yaml
app:
  image:
    repository: ghcr.io/fortedigital/hello-world
    tag: v1.0.0                    # Will be updated by CI/CD
    containerPort: 3000

  replicaCount: 1

  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi

  extraEnv:
  - name: PORT
    value: "3000"
  - name: NODE_ENV
    value: "production"

  envSecretName: ""                # Optional: reference to secrets

service:
  port: 3000

ingress:
  enabled: true
  host: hello-world.forteapps.net  # Your subdomain

db:
  enabled: false                   # Set to true if you need PostgreSQL
```

Commit and push:
```bash
git add hello-world/values.yaml
git commit -m "Add hello-world application values"
git push
```

### Step 3: Create ArgoCD Application Manifest

In the `launchpad` repository, create `apps/hello-world.yaml`:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello-world
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: ""
    notifications.argoproj.io/subscribe.on-sync-failed.slack: ""
    notifications.argoproj.io/subscribe.on-degraded.slack: ""
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/part-of: apps
    app.kubernetes.io/managed-by: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io

spec:
  project: default

  sources:
  # Source 1: Helm chart templates
  - repoURL: https://git.forteapps.net/Forte/forte-helm
    path: forteapp
    targetRevision: HEAD
    helm:
      valueFiles:
      - $values/hello-world/values.yaml

  # Source 2: Helm values
  - repoURL: git@github.com:fortedigital/helm-prod-values.git
    targetRevision: HEAD
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: hello-world

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false

    syncOptions:
    - CreateNamespace=true
    - Validate=true
    - ServerSideApply=true

    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas
```

Commit and push:
```bash
cd ~/dev/k8s/launchpad
git add apps/hello-world.yaml
git commit -m "Add hello-world application"
git push
```

### Step 4: Verify Deployment

ArgoCD will automatically detect the new application within 60 seconds.

**Option 1: Check Slack**
- Watch for sync notifications in your Slack channel
- ✅ "Application hello-world sync succeeded"

**Option 2: Check ArgoCD UI** (if you have access)
```bash
# Port forward to ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Open browser: https://localhost:8080
# Look for "hello-world" application
```

**Option 3: Check with kubectl** (if you have access)
```bash
# List ArgoCD applications
kubectl get applications -n argocd

# Check application status
kubectl get application hello-world -n argocd

# Verify pods are running
kubectl get pods -n hello-world
```

### Step 5: Access Your Application

Once deployed, access via the configured domain:

```bash
# Check if ingress is created
kubectl get ingressroute -n hello-world

# Access application
curl https://hello-world.forteapps.net
```

**⚠️ Note**: DNS must be manually configured for new subdomains. Contact the platform team to add DNS records.

---

## Updating an Existing Application

### Scenario: Deploying a Code Change

You've made changes to your application code and want to deploy them.

### Method 1: Automatic (Recommended)

**Just push to `main` branch** - CI/CD handles everything:

```bash
# In your application repository
git add .
git commit -m "Fix bug in user login"
git push origin main
```

**What Happens Next:**
1. ✅ GitHub Actions triggers
2. ✅ Builds new Docker image
3. ✅ Tags with new version (e.g., `v20260316-143022`)
4. ✅ Pushes to container registry
5. ✅ Updates `helm-prod-values/myapp/values.yaml` with new tag
6. ✅ ArgoCD detects change
7. ✅ Syncs new version to cluster
8. ✅ Sends Slack notification

**Timeline**: ~5-10 minutes from push to deployment

### Method 2: Manual Image Tag Update

If CI/CD is not set up, manually update the image tag:

```bash
cd ~/dev/k8s/helm-prod-values

# Edit your app's values.yaml
vim myapp/values.yaml

# Change:
app:
  image:
    tag: v1.0.0  # Old version
# To:
app:
  image:
    tag: v1.0.1  # New version

# Commit and push
git add myapp/values.yaml
git commit -m "Update myapp to v1.0.1"
git push
```

ArgoCD will sync within 60 seconds.

### Method 3: Configuration Changes

To update environment variables, resources, or other config:

```bash
cd ~/dev/k8s/helm-prod-values
vim myapp/values.yaml
```

Example changes:

```yaml
app:
  # Increase resources
  resources:
    requests:
      cpu: 200m      # Was 100m
      memory: 256Mi  # Was 128Mi

  # Add new environment variable
  extraEnv:
  - name: API_URL
    value: https://api.example.com
  - name: DEBUG          # NEW
    value: "true"        # NEW

  # Enable HPA
  hpa:
    enabled: true        # Was false
    minReplicas: 2
    maxReplicas: 10
```

Commit and push:
```bash
git add myapp/values.yaml
git commit -m "Increase myapp resources and enable HPA"
git push
```

### Method 4: Application Manifest Changes

To change ArgoCD sync behavior, namespace, or other meta-config:

```bash
cd ~/dev/k8s/launchpad
vim apps/myapp.yaml
```

Example changes:

```yaml
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: false    # Disable self-healing temporarily
```

Commit and push:
```bash
git add apps/myapp.yaml
git commit -m "Disable self-healing for myapp"
git push
```

---

## Working with Secrets

### Understanding Secret Management

**NEVER commit plain secrets to Git.** We use **Sealed Secrets** to encrypt secrets before committing.

### Creating a New Secret

#### Step 1: Create Plain Secret Locally

```bash
cd ~/dev/k8s/launchpad

# Create secret in private/ folder (Git-ignored)
kubectl create secret generic myapp-credentials \
  --from-literal=API_KEY=your-secret-key-here \
  --from-literal=DB_PASSWORD=super-secret-password \
  --dry-run=client -o yaml > private/myapp-credentials.yaml
```

**DO NOT commit this file!** It's in `private/` which is Git-ignored.

#### Step 2: Seal the Secret

Seal your secret:

```bash
kubeseal --format=yaml \
  --namespace=myapp \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml
```

#### Step 3: Commit Sealed Secret

```bash
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Add myapp credentials (sealed)"
git push
```

#### Step 4: Reference Secret in Application

Update your `helm-prod-values/myapp/values.yaml`:

```yaml
app:
  envSecretName: "myapp-credentials"  # References the SealedSecret
```

Commit and push:
```bash
cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Reference myapp credentials"
git push
```

### Updating a Secret

To update an existing secret:

```bash
# 1. Create new version of secret
kubectl create secret generic myapp-credentials \
  --from-literal=API_KEY=new-key-here \
  --from-literal=DB_PASSWORD=new-password \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

# 2. Seal it
kubeseal --format=yaml \
  --namespace=myapp \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml

# 3. Commit sealed version
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Update myapp credentials"
git push

# 4. Restart pods to pick up new secret
kubectl rollout restart deployment myapp -n myapp
```

### Secret Best Practices

✅ **DO**:
- Store secrets in `private/` folder locally
- Always seal secrets before committing
- Delete plain secrets after sealing
- Use meaningful secret names
- Document what each secret contains

❌ **DON'T**:
- Commit plain secrets to Git
- Share secrets via Slack/email
- Hard-code secrets in code
- Use the same secret across multiple environments
- Store secrets in Docker images

### Where Secrets Are Stored

```
┌─────────────────────────────────────────────────────────────┐
│  Location                │  Content           │  Committed?│
├──────────────────────────┼────────────────────┼────────────┤
│  private/                │  Plain secrets     │  ❌ NO      │
│  secrets/                │  Sealed secrets    │  ✅ YES     │
│  Kubernetes cluster      │  Unsealed secrets  │  N/A       │
└─────────────────────────────────────────────────────────────┘
```

**Sealed Secrets Controller** in the cluster decrypts sealed secrets automatically.

---

## Enabling Authentication for Applications

The cluster supports automatic authentication sidecar injection for applications via Kyverno policies. This allows you to add authentication to your applications without modifying application code.

### How It Works

When you enable authentication in your Helm values, the Kyverno policy automatically:
1. ✅ Injects an authentication sidecar container into your pod
2. ✅ Routes all incoming traffic through the auth sidecar (port 8080)
3. ✅ Validates credentials before forwarding requests to your application
4. ✅ Creates necessary secrets (if they don't exist)
5. ✅ Adds a NetworkPolicy to restrict ingress

**Architecture**:
```
Internet → Traefik → Service:8080 → Auth Sidecar:8080 → localhost → Your App:3000
                                         │
                                         ├─ Validates credentials
                                         └─ Forwards if valid
```

### Authentication Modes

Three authentication modes are supported:
1. **Token-based**: Static tokens (simple, good for service-to-service or internal apps)
2. **OIDC**: OpenID Connect (full SSO, good for user-facing apps)
3. **MCP**: OAuth 2.0 for MCP servers via RFC 9728 (Protected Resource Metadata); Keycloak provides native RFC 7591 Dynamic Client Registration (good for MCP tool servers requiring OAuth-based access control)

---

### Token-Based Authentication

#### Step 1: Configure Helm Values

```yaml
# In helm-prod-values/myapp/values.yaml
auth:
  enabled: true
  type: token                    # Token mode (default)
  tokens:
  - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823
  - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7
```

#### Step 2: Generate Token (if needed)

```bash
# Generate a secure random token
openssl rand -hex 32

# Or using Python
python3 -c "import secrets; print(secrets.token_hex(32))"

# Example output:
# d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823
```

#### Step 3: Deploy Application

Commit and push your changes:
```bash
cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Enable token auth for myapp"
git push
```

ArgoCD will sync, and the Kyverno policy will:
- Inject the auth sidecar container
- Create an `auth-tokens` Secret with your tokens
- Configure the sidecar to validate against these tokens

#### Step 4: Access Application

Use your token in the `Authorization` header:

```bash
# Access application with token
curl -H "Authorization: Bearer d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823" \
  https://myapp.forteapps.net/api/data

# Without token (will be rejected)
curl https://myapp.forteapps.net/api/data
# Response: 401 Unauthorized
```

#### Advanced: Custom Secret Name

To use a different secret for tokens:

```yaml
# In Helm values
auth:
  enabled: true
  type: token
  tokens: []                     # Empty - using external secret

# Tokens will be read from custom secret
```

Then reference it via annotation (configured by Helm chart automatically):
```yaml
# Helm chart sets this annotation:
policies.forteapps.io/auth-token-secret-name: "myapp-auth-tokens"
```

Create the secret manually:
```bash
kubectl create secret generic myapp-auth-tokens \
  --from-file=tokens=tokens.txt \
  --namespace=myapp
```

---

### OIDC Authentication

OIDC mode integrates with identity providers like Keycloak, Okta, Auth0, Azure AD, etc.

#### Step 1: Configure Identity Provider

In your identity provider (e.g., Keycloak):
1. Create a new client (e.g., `myapp`)
2. Set redirect URI: `https://myapp.forteapps.net/auth/callback`
3. Note the **Client ID** and **Client Secret**
4. Note the **Authority URL** (e.g., `https://keycloak.forteapps.net/realms/master`)

#### Step 2: Create OIDC Secret

```bash
# Create plain secret
kubectl create secret generic auth-oidc \
  --from-literal=client-secret=your-oidc-client-secret \
  --from-literal=cookie-secret=$(openssl rand -hex 32) \
  --namespace=myapp \
  --dry-run=client -o yaml > private/myapp-auth-oidc.yaml

# Seal it
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  --namespace=myapp \
  < private/myapp-auth-oidc.yaml \
  > secrets/myapp-auth-oidc-sealed.yaml

# Commit sealed secret
cd ~/dev/k8s/launchpad
git add secrets/myapp-auth-oidc-sealed.yaml
git commit -m "Add OIDC secrets for myapp"
git push

# Clean up
rm private/myapp-auth-oidc.yaml
```

#### Step 3: Configure Helm Values

```yaml
# In helm-prod-values/myapp/values.yaml
auth:
  enabled: true
  type: oidc                     # OIDC mode
  oidc:
    authority: https://keycloak.forteapps.net/realms/master
    clientId: myapp
    scopes: "openid,profile,email"
    callbackPath: /auth/callback
```

#### Step 4: Deploy Application

```bash
cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Enable OIDC auth for myapp"
git push
```

#### Step 5: Access Application

When users access `https://myapp.forteapps.net`:
1. They're redirected to the identity provider login page
2. After successful login, redirected back to `/auth/callback`
3. Session cookie is set
4. Subsequent requests are authenticated via cookie

**User flow**:
```
User → https://myapp.forteapps.net
  ↓
Redirect → https://keycloak.forteapps.net/login
  ↓
Login successful → Redirect with auth code
  ↓
https://myapp.forteapps.net/auth/callback?code=xyz
  ↓
Auth sidecar exchanges code for tokens
  ↓
Sets session cookie
  ↓
Redirects to application → https://myapp.forteapps.net
  ↓
User sees application (authenticated)
```

---

### Accessing Authenticated User Information

The auth sidecar handles all authentication before requests reach your application. Your app never sees unauthenticated traffic — the sidecar returns 401 or redirects to the IdP first.

After successful authentication, the sidecar forwards the request to your application with user identity injected as HTTP headers:

| Header | Description | Available in |
|--------|-------------|-------------|
| `X-Auth-User` | Username or display name | Token, OIDC, MCP |
| `X-Auth-Email` | User email address | OIDC |
| `X-Auth-Subject` | OIDC `sub` claim (stable user ID) | OIDC, MCP |
| `X-Auth-Groups` | Comma-separated group memberships | OIDC (if scope includes `groups`) |
| `X-Auth-Token` | The validated access token | All modes |

**Your application reads these headers — no auth library needed:**

```javascript
// Express.js example
app.get('/profile', (req, res) => {
  const user = req.headers['x-auth-user'];
  const email = req.headers['x-auth-email'];
  res.json({ user, email });
});
```

```python
# Flask example
@app.route('/profile')
def profile():
    user = request.headers.get('X-Auth-User')
    email = request.headers.get('X-Auth-Email')
    return jsonify(user=user, email=email)
```

**Why this is safe**: The Kyverno-generated NetworkPolicy restricts ingress to the sidecar port only. Traffic cannot bypass the sidecar to reach the application port directly, so the `X-Auth-*` headers can be trusted unconditionally.

**Key principle**: Your application is zero-trust-unaware by design. It reads headers and renders UI. All authentication complexity lives in the sidecar and Kyverno policy.

---

### Authentication Configuration Reference

#### Helm Values Schema

```yaml
auth:
  enabled: false                 # Enable/disable authentication
  type: token                    # "token", "oidc", or "mcp"

  # Token mode configuration
  tokens: []                     # List of valid bearer tokens
  # - token1
  # - token2

  # OIDC mode configuration
  oidc:
    authority: ""                # OIDC provider URL (required for OIDC)
    clientId: ""                 # OIDC client ID (required for OIDC)
    scopes: "openid,profile,email"  # OIDC scopes (optional)
    callbackPath: /auth/callback    # OAuth callback path (optional)

  # MCP mode configuration (RFC 9728)
  mcp:
    resource: ""                 # Protected resource URL (required for MCP)
    authority: ""                # Authorization server URL (required for MCP)
    scopes: "read,write"         # Supported scopes (optional)
```

#### Annotations Set by Helm Chart

When `auth.enabled: true`, the Helm chart sets these pod annotations:

**Token mode**:
```yaml
policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "token"
policies.forteapps.io/auth-token-secret-name: "auth-tokens"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"
```

**OIDC mode**:
```yaml
policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "oidc"
policies.forteapps.io/auth-oidc-authority: "https://keycloak.forteapps.net/realms/master"
policies.forteapps.io/auth-oidc-client-id: "myapp"
policies.forteapps.io/auth-oidc-scopes: "openid,profile,email"
policies.forteapps.io/auth-oidc-callback-path: "/auth/callback"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"
```

**MCP mode** (OAuth 2.0 for MCP servers):
```yaml
policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "mcp"
policies.forteapps.io/auth-mcp-resource: "https://mcp.forteapps.net"
policies.forteapps.io/auth-mcp-authority: "https://keycloak.forteapps.net/realms/master"
policies.forteapps.io/auth-mcp-scopes: "read,write"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"
```

#### Sidecar Configuration

The auth sidecar container:
- **Image**: `ghcr.io/fortedigital/auth-sidecar:latest`
- **Port**: 8080
- **Resources**: 10m CPU / 32Mi memory (requests), 50m CPU / 64Mi memory (limits)
- **Health checks**: `/healthz` endpoint
- **Security**: Read-only root filesystem, no privilege escalation

#### Advanced: Custom Sidecar Image

To use a different auth sidecar image:

```yaml
# These annotations can be set in the Helm chart template if needed
policies.forteapps.io/auth-image: "your-registry/your-auth-proxy"
policies.forteapps.io/auth-image-version: "v1.2.3"
```

---

### Authentication Examples

#### Example 1: Internal API with Token Auth

```yaml
# helm-prod-values/internal-api/values.yaml
app:
  image:
    repository: ghcr.io/company/internal-api
    tag: v1.0.0

auth:
  enabled: true
  type: token
  tokens:
  - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823  # Service A
  - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7  # Service B

ingress:
  enabled: true
  host: internal-api.forteapps.net
```

**Usage**:
```bash
# Service A calls API
curl -H "Authorization: Bearer d4f88f..." \
  https://internal-api.forteapps.net/api/endpoint
```

#### Example 2: User-Facing App with OIDC

```yaml
# helm-prod-values/web-app/values.yaml
app:
  image:
    repository: ghcr.io/company/web-app
    tag: v2.1.0

auth:
  enabled: true
  type: oidc
  oidc:
    authority: https://auth.company.com/realms/employees
    clientId: web-app-prod
    scopes: "openid,profile,email,groups"
    callbackPath: /auth/callback

ingress:
  enabled: true
  host: web-app.forteapps.net
```

**With sealed OIDC secret**:
```bash
# Create and seal secret
kubectl create secret generic auth-oidc \
  --from-literal=client-secret=super-secret-value \
  --from-literal=cookie-secret=$(openssl rand -hex 32) \
  --namespace=web-app \
  --dry-run=client -o yaml | \
  kubeseal --format=yaml --cert=pub-cert.pem --namespace=web-app \
  > secrets/web-app-auth-oidc-sealed.yaml
```

#### Example 3: MCP Server with OAuth 2.0

```yaml
# helm-prod-values/mcp-server/values.yaml
app:
  image:
    repository: ghcr.io/company/mcp-server
    tag: v1.0.0

auth:
  enabled: true
  type: mcp
  mcp:
    resource: https://mcp-server.forteapps.net
    authority: https://auth.company.com/realms/mcp
    scopes: "read,write,admin"

ingress:
  enabled: true
  host: mcp-server.forteapps.net
```

The MCP auth mode implements RFC 9728 (OAuth 2.0 Protected Resource Metadata) for authorization server discovery. Dynamic Client Registration (RFC 7591) is handled natively by Keycloak; MCP clients discover the authorization server and scopes from the `/.well-known/oauth-protected-resource` endpoint served by the sidecar and then register directly with Keycloak.

#### Example 4: Disabling Authentication

```yaml
# helm-prod-values/public-api/values.yaml
auth:
  enabled: false                 # No authentication

ingress:
  enabled: true
  host: public-api.forteapps.net
```

---

### Troubleshooting Authentication

#### Issue: 401 Unauthorized (Token Mode)

**Check token validity**:
```bash
# Get auth-tokens secret
kubectl get secret auth-tokens -n myapp -o yaml

# Decode tokens
kubectl get secret auth-tokens -n myapp \
  -o jsonpath='{.data.tokens}' | base64 -d

# Verify your token is in the list
```

**Test with different token**:
```bash
curl -v -H "Authorization: Bearer YOUR-TOKEN-HERE" \
  https://myapp.forteapps.net/
```

#### Issue: OIDC Login Loop

**Check OIDC configuration**:
```bash
# Verify auth-oidc secret exists
kubectl get secret auth-oidc -n myapp

# Check sidecar logs
kubectl logs -n myapp <pod-name> -c authn

# Common issues:
# - Wrong authority URL
# - Wrong client ID
# - Missing client-secret in auth-oidc Secret
# - Redirect URI not configured in identity provider
```

**Verify redirect URI** in your identity provider matches:
```
https://<your-app-domain>/auth/callback
```

#### Issue: Auth Sidecar Not Injected

**Check pod annotations**:
```bash
kubectl get pod -n myapp <pod-name> -o yaml | grep policies.forteapps.io

# Should show:
# policies.forteapps.io/auth: "true"
```

**Check Kyverno policy**:
```bash
kubectl get clusterpolicy inject-auth-sidecar
kubectl describe clusterpolicy inject-auth-sidecar
```

**Check Kyverno logs**:
```bash
kubectl logs -n kyverno deployment/kyverno | grep inject-auth
```

#### Issue: Auth Sidecar Crashes

**Check sidecar logs**:
```bash
kubectl logs -n myapp <pod-name> -c authn
```

**Common causes**:
- Missing secret (auth-tokens or auth-oidc)
- Invalid OIDC configuration
- Can't reach OIDC authority URL
- Network policy blocking outbound OIDC requests

---

### Authentication Best Practices

✅ **DO**:
- Use OIDC for user-facing applications
- Use token auth for service-to-service communication
- Rotate tokens and secrets regularly
- Use strong random tokens (32+ bytes)
- Store client secrets in SealedSecrets
- Test authentication before deploying to production
- Document which tokens/users have access

❌ **DON'T**:
- Share tokens between environments
- Commit tokens to application code
- Use predictable tokens
- Reuse tokens across multiple applications
- Disable authentication on sensitive APIs
- Log tokens or secrets

---

## Adding a New Keycloak Client

There are two ways to add an OIDC client, depending on your use case:

| Method | Best for | Who edits the infra repo? |
|--------|----------|--------------------------|
| **Self-service** (recommended) | New apps that deploy their own resources | App developer — no infra changes needed |
| **Legacy (realm JSON)** | Existing clients already defined in forte-realm.json (e.g., Gitea) | Platform engineer |

Both methods are served by the **Keycloak Client Registrar** CronJob, which runs every 2 minutes.

### Self-Service OIDC Client Registration

This is the recommended flow for new applications. Your app deploys a labeled config Secret in its own namespace; the platform handles everything else.

#### How It Works

1. You deploy a Secret with label `keycloak.forteapps.net/client-config: "true"` containing a `client.json` definition
2. A **Kyverno ClusterPolicy** (`keycloak-client-config-cloner`) clones it to the `keycloak` namespace
3. The **Client Registrar CronJob** picks it up within 2 minutes:
   - Registers (or updates) the client in Keycloak
   - Fetches the auto-generated client secret
   - Creates a credential Secret in your app's namespace
   - Annotates the config Secret with sync status

#### Step 1: Create the Config Secret

Deploy this Secret in your application's namespace (e.g., as part of your Helm chart or Kustomize overlay):

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: keycloak-client-myapp
  namespace: myapp
  labels:
    keycloak.forteapps.net/client-config: "true"
stringData:
  client.json: |
    {
      "clientId": "myapp",
      "name": "My Application",
      "redirectUris": ["https://myapp.forteapps.net/*"],
      "webOrigins": ["https://myapp.forteapps.net"],
      "defaultClientScopes": ["openid", "email", "profile"],
      "protocolMappers": [],
      "secret": {
        "namespace": "myapp",
        "name": "myapp-oidc-credentials",
        "keys": { "clientId": "client-id", "clientSecret": "client-secret" }
      }
    }
```

**`client.json` fields**:

| Field | Required | Description |
|-------|----------|-------------|
| `clientId` | Yes | Keycloak client ID (must be unique in realm) |
| `name` | Yes | Display name in Keycloak UI |
| `redirectUris` | Yes | Allowed OAuth redirect URLs (supports wildcards like `/*`) |
| `webOrigins` | Yes | Allowed CORS origins |
| `defaultClientScopes` | No | OIDC scopes (default: `["openid", "email", "profile"]`) |
| `protocolMappers` | No | Custom claim mappers for tokens (see examples below) |
| `secret.namespace` | No | Target namespace for credentials (default: `source-namespace` annotation value) |
| `secret.name` | No | Credential Secret name (default: `<clientId>-oidc-credentials`) |
| `secret.keys.clientId` | No | Key name for client ID (default: `client-id`) |
| `secret.keys.clientSecret` | No | Key name for client secret (default: `client-secret`) |

**Protocol Mappers Example**:
```json
"protocolMappers": [
  {
    "name": "groups",
    "protocol": "openid-connect",
    "protocolMapper": "oidc-group-membership-mapper",
    "config": {
      "claim.name": "groups",
      "full.path": "false",
      "id.token.claim": "true",
      "access.token.claim": "true",
      "userinfo.token.claim": "true"
    }
  }
]
```

#### Step 2: Reference the Credential Secret

In your application's deployment config, reference the credential Secret that the registrar creates:

```yaml
env:
- name: OIDC_CLIENT_ID
  valueFrom:
    secretKeyRef:
      name: myapp-oidc-credentials
      key: client-id
- name: OIDC_CLIENT_SECRET
  valueFrom:
    secretKeyRef:
      name: myapp-oidc-credentials
      key: client-secret
```

#### Step 3: Deploy and Wait

Commit and push your changes. The credential Secret will appear within 2 minutes:

```bash
# Watch for the credential Secret to be created
kubectl get secret myapp-oidc-credentials -n myapp -w

# Check registrar logs
kubectl logs -n keycloak job/$(kubectl get jobs -n keycloak --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')

# Check sync status on the config Secret
kubectl get secret keycloak-client-myapp -n keycloak -o jsonpath='{.metadata.annotations}'
```

#### Change Detection

The registrar computes a SHA-256 hash of `client.json` and stores it as an annotation. On subsequent runs, it skips processing if:
- The hash hasn't changed, AND
- The credential Secret already exists in the target namespace

To force a re-sync, update any field in `client.json` (e.g., add a trailing space to `name`).

### Legacy Method: Realm JSON

Existing clients (like Gitea) are defined directly in `forte-realm.json` inside `keycloak-values.yaml`. The registrar syncs their secrets via client attributes.

#### Step 1: Add Client to Realm Config

In `infra/values/base/keycloak-values.yaml`, add a new entry to the `clients` array in `forte-realm.json`:

```json
{
  "clientId": "myapp",
  "name": "My Application",
  "enabled": true,
  "protocol": "openid-connect",
  "clientAuthenticatorType": "client-secret",
  "standardFlowEnabled": true,
  "directAccessGrantsEnabled": false,
  "publicClient": false,
  "redirectUris": ["https://myapp.forteapps.net/*"],
  "webOrigins": ["https://myapp.forteapps.net"],
  "defaultClientScopes": ["openid", "email", "profile"],
  "attributes": {
    "k8s.secret.sync": "true",
    "k8s.secret.namespace": "myapp",
    "k8s.secret.name": "myapp-oidc-credentials",
    "k8s.secret.client-id-key": "key",
    "k8s.secret.client-secret-key": "secret"
  }
}
```

**Important**:
- Do **NOT** include a `"secret"` field — Keycloak generates one automatically
- The `attributes` block tells the registrar where to create the K8s Secret
- Set `client-id-key` / `client-secret-key` to match what the consuming app expects (defaults: `client-id` / `client-secret`)

#### Step 2: Reference the Secret in Your Application

```yaml
existingSecret: myapp-oidc-credentials
```

#### Step 3: Commit and Push

```bash
cd ~/dev/k8s/launchpad
git add infra/values/base/keycloak-values.yaml
git commit -m "Add myapp Keycloak client with auto-sync"
git push
```

ArgoCD will sync the Keycloak config, and the registrar CronJob will pick up the new client within 2 minutes.

#### Legacy Sync Attribute Reference

| Attribute | Required | Default | Description |
|-----------|----------|---------|-------------|
| `k8s.secret.sync` | Yes | — | Set to `"true"` to enable syncing |
| `k8s.secret.namespace` | Yes | — | Target K8s namespace for the secret |
| `k8s.secret.name` | Yes | — | Name of the K8s Secret to create |
| `k8s.secret.client-id-key` | No | `client-id` | Field name for the client ID in the K8s Secret |
| `k8s.secret.client-secret-key` | No | `client-secret` | Field name for the client secret in the K8s Secret |

### Retrieving Secrets for External Deployments

The registrar always writes a **central copy** of every synced secret to the `secrets` namespace, in addition to the target namespace. This allows operators to retrieve client credentials for applications deployed outside this cluster:

```bash
# View the central copy
kubectl get secret gitea-oidc-credentials -n secrets -o yaml

# Extract the client secret for use elsewhere
kubectl get secret myapp-oidc-credentials -n secrets \
  -o jsonpath='{.data.client-secret}' | base64 -d
```

### Registrar Behavior Notes

- The registrar runs as a CronJob every 2 minutes (`concurrencyPolicy: Forbid`)
- If the target namespace doesn't exist, the target write is skipped with a warning (the central copy still happens)
- A central copy is **always** written to the `secrets` namespace for every synced client
- The registrar uses the `keycloak-credentials` secret for admin authentication
- Created secrets have the label `app.kubernetes.io/managed-by: keycloak-client-registrar`

---

## Troubleshooting

### Application Not Deploying

#### Problem: Application stuck in "Syncing" state

**Check ArgoCD status:**
```bash
kubectl get application myapp -n argocd -o yaml
```

Look for errors in `status.conditions`.

**Common causes:**
- ❌ Image doesn't exist or is not accessible
- ❌ Invalid YAML syntax
- ❌ Resource quota exceeded
- ❌ Namespace conflicts
- ❌ Invalid Helm values

**Solutions:**
```bash
# Check image exists
docker pull ghcr.io/fortedigital/myapp:v1.0.0

# Validate YAML syntax
kubectl apply --dry-run=client -f apps/myapp.yaml

# Check ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller | grep myapp
```

#### Problem: Pods crashing (CrashLoopBackOff)

**Check pod logs:**
```bash
kubectl get pods -n myapp
kubectl logs -n myapp <pod-name>
kubectl describe pod -n myapp <pod-name>
```

**Common causes:**
- ❌ Application error (check logs)
- ❌ Missing environment variables
- ❌ Incorrect port configuration
- ❌ Missing secrets
- ❌ Insufficient resources

**Solutions:**
```bash
# Check environment variables
kubectl exec -n myapp <pod-name> -- env

# Check if secrets exist
kubectl get secrets -n myapp

# Increase resources in helm-prod-values
vim ~/dev/k8s/helm-prod-values/myapp/values.yaml
```

#### Problem: Application not accessible via domain

**Check ingress:**
```bash
kubectl get ingressroute -n myapp
kubectl describe ingressroute myapp -n myapp
```

**Common causes:**
- ❌ DNS not configured
- ❌ TLS certificate not issued
- ❌ Incorrect domain in values.yaml
- ❌ Traefik not routing correctly

**Solutions:**
```bash
# Check certificate
kubectl get certificate -n myapp

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Verify domain configuration
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep host

# Test with port-forward
kubectl port-forward -n myapp service/myapp 8080:3000
curl http://localhost:8080
```

### Secret Issues

#### Problem: Secret not found

**Check if SealedSecret exists:**
```bash
kubectl get sealedsecret -n myapp
kubectl get secret -n myapp
```

**Solutions:**
```bash
# Check if secret is in Git
ls -l secrets/myapp-credentials-sealed.yaml

# Re-apply sealed secret
kubectl apply -f secrets/myapp-credentials-sealed.yaml

# Check sealed-secrets-controller logs
kubectl logs -n kube-system deployment/sealed-secrets-controller
```

#### Problem: Secret exists but pods can't access it

**Check pod events:**
```bash
kubectl describe pod -n myapp <pod-name>
```

Look for: `Error: secret "myapp-credentials" not found`

**Solutions:**
```bash
# Verify secret name in values.yaml matches actual secret
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep envSecretName
kubectl get secrets -n myapp

# Restart pods
kubectl rollout restart deployment myapp -n myapp
```

### Sync Failures

#### Problem: ArgoCD shows "Out of Sync"

**Manual sync:**
```bash
# Using kubectl
kubectl patch application myapp -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"syncStrategy":{"hook":{}}}}}'

# Or via ArgoCD UI
# Click "Sync" button in UI
```

**Check what's different:**
```bash
kubectl get application myapp -n argocd -o yaml
```

Look at `status.sync.comparedTo` vs desired state.

#### Problem: Sync succeeds but application is "Degraded"

**Check resource health:**
```bash
kubectl get application myapp -n argocd -o jsonpath='{.status.resources[*].health}'
```

**Common causes:**
- ❌ Pods not ready
- ❌ Deployments not at desired replica count
- ❌ Jobs failed

**Solutions:**
```bash
# Check all resources in namespace
kubectl get all -n myapp

# Check pod events
kubectl get events -n myapp --sort-by='.lastTimestamp'
```

### Getting Help

If you're stuck:

1. **Check Slack notifications** - Error details are often in sync failure messages
2. **Check ArgoCD UI** - Visual representation of what's wrong
3. **Ask platform team** - They have full cluster access and can debug further
4. **Check documentation** - [Operations Runbook](OPERATIONS-RUNBOOK.md) has more troubleshooting

---

## Best Practices

### Development Workflow

✅ **DO**:
- Develop and test locally with Docker Compose
- Use semantic versioning for releases
- Write descriptive commit messages
- Test changes in a separate namespace first (if possible)
- Monitor Slack for deployment notifications
- Document environment variables and configuration

❌ **DON'T**:
- Push directly to production without testing
- Use `latest` tag for Docker images
- Bypass CI/CD for "quick fixes"
- Hard-code configuration values
- Ignore deployment failures

### Configuration Management

✅ **DO**:
- Keep configuration in `helm-prod-values` repository
- Use environment variables for config
- Document what each value does
- Use reasonable resource limits
- Enable ingress and TLS for public services

❌ **DON'T**:
- Hard-code config in application code
- Over-allocate resources (wastes money)
- Under-allocate resources (causes crashes)
- Use HTTP for production services

### Secret Management

✅ **DO**:
- Use kubeseal for all secrets
- Store plain secrets in password manager
- Rotate secrets regularly
- Use different secrets per environment
- Document what each secret contains

❌ **DON'T**:
- Commit plain secrets
- Share secrets in Slack/email
- Reuse secrets across apps
- Log secrets in application code

### Git Workflow

✅ **DO**:
- Use feature branches for changes
- Write clear commit messages
- Use pull requests for review
- Keep commits atomic and focused
- Tag releases in application repos

❌ **DON'T**:
- Push directly to `main` without review (for config repos)
- Make multiple unrelated changes in one commit
- Use vague commit messages ("fix", "update")
- Force-push to main branches

---

## Quick Reference

### Common Commands

```bash
# Check application status
kubectl get application myapp -n argocd

# View application details
kubectl describe application myapp -n argocd

# Check pods
kubectl get pods -n myapp

# View pod logs
kubectl logs -n myapp <pod-name>

# Restart deployment
kubectl rollout restart deployment myapp -n myapp

# Port-forward to service
kubectl port-forward -n myapp service/myapp 8080:3000

# Create secret
kubectl create secret generic myapp-credentials \
  --from-literal=KEY=value \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

# Seal secret
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml
```

### Repository Locations

```bash
# Config repository
cd ~/dev/k8s/launchpad

# Helm values repository
cd ~/dev/k8s/helm-prod-values

# Helm charts repository
cd ~/dev/k8s/forte-helm
```

### File Paths

```bash
# New application manifest
~/dev/k8s/launchpad/apps/myapp.yaml

# Application values
~/dev/k8s/helm-prod-values/myapp/values.yaml

# Sealed secrets
~/dev/k8s/launchpad/secrets/myapp-credentials-sealed.yaml

# Plain secrets (local only)
~/dev/k8s/launchpad/private/myapp-credentials.yaml
```

---

## Next Steps

Now that you understand the basics:

1. ✅ Deploy your first application (follow steps above)
2. 📖 Read the [Operations Runbook](OPERATIONS-RUNBOOK.md) for common tasks
3. 📖 Review [Technical Reference](REFERENCE.md) for detailed component docs
4. 📖 Understand [GitOps Architecture](GITOPS-ARCHITECTURE.md) for the big picture
5. 🚀 Start contributing!

---

**Questions?**
- Slack: #platform-support
- Docs: [Full documentation index](README.md)
- Help: Contact platform team

**Last Updated**: 2026-04-16