# Developer Onboarding Guide

## Table of Contents
- [Getting Started](#getting-started)
- [Prerequisites](#prerequisites)
- [Local Development Setup](#local-development-setup)
- [Understanding the Workflow](#understanding-the-workflow)
- [Deploying Your First Application](#deploying-your-first-application)
- [Updating an Existing Application](#updating-an-existing-application)
- [Working with Secrets](#working-with-secrets)
- [Enabling Authentication for Applications](#enabling-authentication-for-applications)
- [Adding a New Keycloak Client](#adding-a-new-keycloak-client)
- [Troubleshooting](#troubleshooting)
- [Best Practices](#best-practices)

---

## Getting Started

Welcome! This guide will help you understand how to develop and deploy applications on our Kubernetes cluster using GitOps principles powered by ArgoCD.

### What You'll Learn
- How our GitOps architecture works
- How to deploy a new application
- How to update existing applications
- How to manage secrets securely
- Common troubleshooting techniques

### Who This Guide Is For
- Developers deploying new applications
- Developers maintaining existing applications
- Team members who need to understand the deployment process

---

## Prerequisites

### Required Knowledge
- ✅ Basic Git workflow (clone, commit, push, pull)
- ✅ Docker basics (Dockerfile, building images)
- ✅ YAML syntax
- ✅ Basic understanding of Kubernetes concepts (pods, deployments, services)
- ⚠️ Helm knowledge (helpful but not required - templates are provided)

### Required Tools

Most developers **do NOT need kubectl access** to the cluster. You'll primarily work with Git repositories.

If you do need cluster access, install:

1. **kubectl** - Kubernetes CLI
   ```bash
   # macOS
   brew install kubectl

   # Windows
   choco install kubernetes-cli

   # Linux
   curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   ```

2. **vault** CLI - For managing secrets in HashiCorp Vault
   ```bash
   # macOS
   brew install hashicorp/tap/vault

   # Windows
   choco install vault

   # Linux
   # See https://developer.hashicorp.com/vault/install
   ```

3. **Git** - Version control
   ```bash
   git --version  # Should already be installed
   ```

4. **Docker** - For local development
   ```bash
   # macOS/Windows: Install Docker Desktop
   # Linux: Install Docker Engine
   docker --version
   ```

### Repository Access

You'll need read/write access to these repositories:

1. **launchpad** (Config repo)
   ```bash
   git clone https://git.forteapps.net/Forte/launchpad.git
   cd launchpad
   ```

2. **helm-prod-values** (Values repo)
   ```bash
   git clone https://git.forteapps.net/Forte/helm-prod-values.git
   cd helm-prod-values
   ```

3. **forte-helm** (Chart repo - read-only for most developers)
   ```bash
   git clone https://git.forteapps.net/Forte/forte-helm.git
   cd forte-helm
   ```

### Cluster Access (If Needed)

If you need kubectl access, ask the platform team for:
- Kubeconfig file
- Cluster context setup instructions

Save to `~/.kube/config` and verify:
```bash
kubectl cluster-info
kubectl get nodes
```

---

## Local Development Setup

### 1. Clone the Repositories

Set up a consistent folder structure:

```bash
mkdir -p ~/dev/k8s
cd ~/dev/k8s

# Clone repositories
git clone https://git.forteapps.net/Forte/launchpad.git launchpad
git clone https://git.forteapps.net/Forte/helm-prod-values helm-prod-values
git clone https://git.forteapps.net/Forte/forte-helm forte-helm

# Your folder structure:
# ~/dev/k8s/
# ├── launchpad/           (Config repo)
# ├── helm-prod-values/    (Values repo)
# └── forte-helm/          (Chart repo)
```

### 2. Local Development Environment

Most applications use **Docker Compose** for local development:

```bash
# In your application repository
docker-compose up

# Or for frontend applications
npm install
npm run dev
```

**You DO NOT run applications locally on Kubernetes.** Use Docker Compose or native tooling (npm, python, etc.).

### 3. Understanding the Deployment Flow

```
┌─────────────────────────────────────────────────────────────────┐
│  Step 1: Develop Locally                                        │
│  - Write code in your application repository                    │
│  - Test with Docker Compose or npm/python/etc.                  │
│  - Build Docker image                                            │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 2: CI/CD Pipeline (Automated)                             │
│  - GitHub Actions builds image                                  │
│  - Pushes to container registry (GHCR, Docker Hub)              │
│  - Tags with version (e.g., v2.0.4)                             │
│  - Updates helm-prod-values repository with new tag                  │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 3: GitOps Sync (Automated)                                │
│  - ArgoCD detects change in helm-prod-values                         │
│  - Pulls updated configuration                                  │
│  - Syncs to Kubernetes cluster                                  │
│  - Sends Slack notification on success/failure                  │
└─────────────────────────────────────────────────────────────────┘
```

**Key Insight**: You don't deploy directly. You push code, CI/CD builds it, and ArgoCD deploys it.

---

## Understanding the Workflow

### Three-Repository Pattern

Our setup uses three repositories:

| Repository | Purpose | Who Edits | How Often |
|------------|---------|-----------|-----------|
| **forte-helm** | Helm chart templates (generic, reusable) | Platform engineers | ❌ Rarely |
| **helm-prod-values** | Application configuration (image tag, env vars) | Developers / CI pipelines | ✅ Sometimes |
| **launchpad** | ArgoCD Applications (what gets deployed) | Platform / DevOps engineers | ✅ Per new app |

### Example: Deploying "myapp"

#### Repository: `forte-helm` (Chart Templates)
```yaml
# forteapp/templates/deployment.yaml
# Generic template used by ALL apps
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
spec:
  containers:
  - name: app
    image: "{{ .Values.app.image.repository }}:{{ .Values.app.image.tag }}"
    env:
    - name: PORT
      value: {{ .Values.app.port }}
```

#### Repository: `helm-prod-values` (Your App Config)
```yaml
# myapp/values.yaml
# Your app's specific configuration
app:
  image:
    repository: ghcr.io/fortedigital/myapp
    tag: v1.0.0                    # CI/CD updates this
  port: 3000
  extraEnv:
  - name: API_URL
    value: https://api.example.com
```

#### Repository: `launchpad` (ArgoCD Application)
```yaml
# apps/myapp.yaml
# Tells ArgoCD to deploy your app
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  sources:
  - repoURL: https://git.forteapps.net/Forte/forte-helm
    path: forteapp
    helm:
      valueFiles:
      - $values/myapp/values.yaml

  - repoURL: git@github.com:fortedigital/helm-prod-values.git
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: myapp

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
```

---

## Deploying Your First Application

### Scenario: You've Built a New Application

Let's deploy a new Node.js application called "hello-world".

### Step 1: Prepare Your Application Repository

Ensure your app repository has:

1. **Dockerfile**
   ```dockerfile
   FROM node:18-alpine
   WORKDIR /app
   COPY package*.json ./
   RUN npm ci --only=production
   COPY . .
   EXPOSE 3000
   CMD ["node", "server.js"]
   ```

2. **GitHub Actions Workflow** (`.github/workflows/deploy.yml`)
   ```yaml
   name: Build and Deploy

   on:
     push:
       branches: [ main ]

   jobs:
     build:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v3

         - name: Set version
           id: version
           run: echo "VERSION=v$(date +%Y%m%d-%H%M%S)" >> $GITHUB_OUTPUT

         - name: Build and push Docker image
           run: |
             echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
             docker build -t ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }} .
             docker push ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }}

         - name: Update helm-prod-values
           run: |
             git clone git@github.com:fortedigital/helm-prod-values.git
             cd helm-prod-values
             mkdir -p hello-world
             cat > hello-world/values.yaml <<EOF
             app:
               image:
                 repository: ghcr.io/fortedigital/hello-world
                 tag: ${{ steps.version.outputs.VERSION }}
             EOF
             git add hello-world/values.yaml
             git commit -m "Update hello-world to ${{ steps.version.outputs.VERSION }}"
             git push
   ```

### Step 2: Create Helm Values

Create a folder in `helm-prod-values` repository:

```bash
cd ~/dev/k8s/helm-prod-values
mkdir -p hello-world
```

Create `hello-world/values.yaml`:
```yaml
app:
  image:
    repository: ghcr.io/fortedigital/hello-world
    tag: v1.0.0                    # Will be updated by CI/CD
    containerPort: 3000

  replicaCount: 1

  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi

  extraEnv:
  - name: PORT
    value: "3000"
  - name: NODE_ENV
    value: "production"

  envSecretName: ""                # Optional: reference to secrets

service:
  port: 3000

ingress:
  enabled: true
  host: hello-world.forteapps.net  # Your subdomain

db:
  enabled: false                   # Set to true if you need PostgreSQL
```

Commit and push:
```bash
git add hello-world/values.yaml
git commit -m "Add hello-world application values"
git push
```

### Step 3: Create ArgoCD Application Manifest

In the `launchpad` repository, create `apps/hello-world.yaml`:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello-world
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: ""
    notifications.argoproj.io/subscribe.on-sync-failed.slack: ""
    notifications.argoproj.io/subscribe.on-degraded.slack: ""
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/part-of: apps
    app.kubernetes.io/managed-by: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io

spec:
  project: default

  sources:
  # Source 1: Helm chart templates
  - repoURL: https://git.forteapps.net/Forte/forte-helm
    path: forteapp
    targetRevision: HEAD
    helm:
      valueFiles:
      - $values/hello-world/values.yaml

  # Source 2: Helm values
  - repoURL: git@github.com:fortedigital/helm-prod-values.git
    targetRevision: HEAD
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: hello-world

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false

    syncOptions:
    - CreateNamespace=true
    - Validate=true
    - ServerSideApply=true

    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas
```

Commit and push:
```bash
cd ~/dev/k8s/launchpad
git add apps/hello-world.yaml
git commit -m "Add hello-world application"
git push
```

### Step 4: Verify Deployment

ArgoCD will automatically detect the new application within 60 seconds.

**Option 1: Check Slack**
- Watch for sync notifications in your Slack channel
- ✅ "Application hello-world sync succeeded"

**Option 2: Check ArgoCD UI** (if you have access)
```bash
# Port forward to ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Open browser: https://localhost:8080
# Look for "hello-world" application
```

**Option 3: Check with kubectl** (if you have access)
```bash
# List ArgoCD applications
kubectl get applications -n argocd

# Check application status
kubectl get application hello-world -n argocd

# Verify pods are running
kubectl get pods -n hello-world
```

### Step 5: Access Your Application

Once deployed, access via the configured domain:

```bash
# Check if ingress is created
kubectl get ingressroute -n hello-world

# Access application
curl https://hello-world.forteapps.net
```

**⚠️ Note**: DNS must be manually configured for new subdomains. Contact the platform team to add DNS records.

---

## Updating an Existing Application

### Scenario: Deploying a Code Change

You've made changes to your application code and want to deploy them.

### Method 1: Automatic (Recommended)

**Just push to `main` branch** - CI/CD handles everything:

```bash
# In your application repository
git add .
git commit -m "Fix bug in user login"
git push origin main
```

**What Happens Next:**
1. ✅ GitHub Actions triggers
2. ✅ Builds new Docker image
3. ✅ Tags with new version (e.g., `v20260316-143022`)
4. ✅ Pushes to container registry
5. ✅ Updates `helm-prod-values/myapp/values.yaml` with new tag
6. ✅ ArgoCD detects change
7. ✅ Syncs new version to cluster
8. ✅ Sends Slack notification

**Timeline**: ~5-10 minutes from push to deployment

### Method 2: Manual Image Tag Update

If CI/CD is not set up, manually update the image tag:

```bash
cd ~/dev/k8s/helm-prod-values

# Edit your app's values.yaml
vim myapp/values.yaml

# Change:
app:
  image:
    tag: v1.0.0  # Old version
# To:
app:
  image:
    tag: v1.0.1  # New version

# Commit and push
git add myapp/values.yaml
git commit -m "Update myapp to v1.0.1"
git push
```

ArgoCD will sync within 60 seconds.

### Method 3: Configuration Changes

To update environment variables, resources, or other config:

```bash
cd ~/dev/k8s/helm-prod-values
vim myapp/values.yaml
```

Example changes:

```yaml
app:
  # Increase resources
  resources:
    requests:
      cpu: 200m      # Was 100m
      memory: 256Mi  # Was 128Mi

  # Add new environment variable
  extraEnv:
  - name: API_URL
    value: https://api.example.com
  - name: DEBUG          # NEW
    value: "true"        # NEW

  # Enable HPA
  hpa:
    enabled: true        # Was false
    minReplicas: 2
    maxReplicas: 10
```

Commit and push:
```bash
git add myapp/values.yaml
git commit -m "Increase myapp resources and enable HPA"
git push
```

### Method 4: Application Manifest Changes

To change ArgoCD sync behavior, namespace, or other meta-config:

```bash
cd ~/dev/k8s/launchpad
vim apps/myapp.yaml
```

Example changes:

```yaml
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: false    # Disable self-healing temporarily
```

Commit and push:
```bash
git add apps/myapp.yaml
git commit -m "Disable self-healing for myapp"
git push
```

---

## Working with Secrets

### Understanding Secret Management

Secrets are managed via **HashiCorp Vault** and synced to Kubernetes by the **Vault Secrets Operator (VSO)**. See [Vault Secrets Operator Reference](vault-secrets-operator.md) for full details.

**NEVER commit plain secret values to Git.** Only VaultStaticSecret CRD manifests are committed.

### Creating a New Secret

#### Step 1: Write Secret to Vault

```bash
vault kv put kv/myapp/myapp-credentials \
  API_KEY=your-secret-key-here \
  DB_PASSWORD=super-secret-password
```

#### Step 2: Create VaultStaticSecret CRD

Create a YAML file (e.g., `apps/base/myapp/myapp-credentials-vault.yaml`):

```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: myapp-credentials
  namespace: myapp
spec:
  type: kv-v2
  mount: kv
  path: myapp/myapp-credentials
  destination:
    name: myapp-credentials
    create: true
  refreshAfter: 30s
  vaultAuthRef: vault-auth
```

#### Step 3: Add VaultAuth (if new namespace)

If this is a new namespace, also create a `vault-auth.yaml` with a ServiceAccount and VaultAuth CRD. See [VSO Reference](vault-secrets-operator.md#vaultauth) for template.

#### Step 4: Commit and Push

```bash
git add apps/base/myapp/myapp-credentials-vault.yaml
git commit -m "Add myapp credentials (VSO)"
git push
```

ArgoCD syncs the CRD, VSO creates the K8s Secret.

#### Step 5: Reference Secret in Application

Update your `helm-prod-values/myapp/values.yaml`:

```yaml
app:
  envSecretName: "myapp-credentials"  # VSO creates this K8s Secret
```

### Updating / Rotating a Secret

**No git commit needed** — just update in Vault:

```bash
vault kv put kv/myapp/myapp-credentials \
  API_KEY=new-key-here \
  DB_PASSWORD=new-password
```

VSO picks up changes within 30 seconds. Restart pods if they don't watch for secret updates:

```bash
kubectl rollout restart deployment myapp -n myapp
```

### Secret Best Practices

- Write secrets to Vault via UI or CLI — never commit values to Git
- Use meaningful secret names matching the KV path convention: `kv/{namespace}/{secret-name}`
- Document what each secret contains
- Use Vault's versioning for audit trail

### Where Secrets Are Stored

```
┌──────────────────────────────────────────────────────────────────┐
│  Location                  │  Content                │  In Git? │
├────────────────────────────┼─────────────────────────┼──────────┤
│  Vault KV (kv/{ns}/{name}) │  Secret values          │  ❌ NO   │
│  VaultStaticSecret CRD     │  Sync config (no values)│  ✅ YES  │
│  Kubernetes cluster        │  K8s Secret (synced)    │  N/A     │
└──────────────────────────────────────────────────────────────────┘
```

**Vault Secrets Operator** syncs secrets from Vault to K8s automatically (30s refresh).

---

## Enabling Authentication for Applications

The cluster supports automatic authentication sidecar injection for applications via Kyverno policies. This allows you to add authentication to your applications without modifying application code.

### How It Works

When you enable authentication in your Helm values, the Kyverno policy automatically:
1. ✅ Injects an authentication sidecar container into your pod
2. ✅ Routes all incoming traffic through the auth sidecar (port 8080)
3. ✅ Validates credentials before forwarding requests to your application
4. ✅ Creates necessary secrets (if they don't exist)
5. ✅ Adds a NetworkPolicy to restrict ingress

**Architecture**:
```
Internet → Traefik → Service:8080 → Auth Sidecar:8080 → localhost → Your App:3000
                                         │
                                         ├─ Validates credentials
                                         └─ Forwards if valid
```

### Authentication Modes

Three authentication modes are supported:
1. **Token-based**: Static tokens (simple, good for service-to-service or internal apps)
2. **OIDC**: OpenID Connect (full SSO, good for user-facing apps)
3. **MCP**: OAuth 2.0 for MCP servers via RFC 9728 / RFC 7591 (good for MCP tool servers requiring OAuth-based access control)

---

### Token-Based Authentication

#### Step 1: Configure Helm Values

```yaml
# In helm-prod-values/myapp/values.yaml
auth:
  enabled: true
  type: token                    # Token mode (default)
  tokens:
  - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823
  - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7
```

#### Step 2: Generate Token (if needed)

```bash
# Generate a secure random token
openssl rand -hex 32

# Or using Python
python3 -c "import secrets; print(secrets.token_hex(32))"

# Example output:
# d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823
```

#### Step 3: Deploy Application

Commit and push your changes:
```bash
cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Enable token auth for myapp"
git push
```

ArgoCD will sync, and the Kyverno policy will:
- Inject the auth sidecar container
- Create an `auth-tokens` Secret with your tokens
- Configure the sidecar to validate against these tokens

#### Step 4: Access Application

Use your token in the `Authorization` header:

```bash
# Access application with token
curl -H "Authorization: Bearer d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823" \
  https://myapp.forteapps.net/api/data

# Without token (will be rejected)
curl https://myapp.forteapps.net/api/data
# Response: 401 Unauthorized
```

#### Advanced: Custom Secret Name

To use a different secret for tokens:

```yaml
# In Helm values
auth:
  enabled: true
  type: token
  tokens: []                     # Empty - using external secret

# Tokens will be read from custom secret
```

Then reference it via annotation (configured by Helm chart automatically):
```yaml
# Helm chart sets this annotation:
policies.forteapps.io/auth-token-secret-name: "myapp-auth-tokens"
```

Create the secret manually:
```bash
kubectl create secret generic myapp-auth-tokens \
  --from-file=tokens=tokens.txt \
  --namespace=myapp
```

---

### OIDC Authentication

OIDC mode integrates with identity providers like Keycloak, Okta, Auth0, Azure AD, etc.

#### Step 1: Configure Identity Provider

In your identity provider (e.g., Keycloak):
1. Create a new client (e.g., `myapp`)
2. Set redirect URI: `https://myapp.forteapps.net/auth/callback`
3. Note the **Client ID** and **Client Secret**
4. Note the **Authority URL** (e.g., `https://keycloak.forteapps.net/realms/master`)

#### Step 2: Create OIDC Secret

```bash
# Write OIDC secret to Vault
vault kv put kv/myapp/auth-oidc \
  client-secret=your-oidc-client-secret \
  cookie-secret=$(openssl rand -hex 32)

# Create VaultStaticSecret CRD (see docs/vault-secrets-operator.md for template)
# Add to apps/base/myapp/auth-oidc-vault.yaml and commit
```

#### Step 3: Configure Helm Values

```yaml
# In helm-prod-values/myapp/values.yaml
auth:
  enabled: true
  type: oidc                     # OIDC mode
  oidc:
    authority: https://keycloak.forteapps.net/realms/master
    clientId: myapp
    scopes: "openid,profile,email"
    callbackPath: /auth/callback
```

#### Step 4: Deploy Application

```bash
cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Enable OIDC auth for myapp"
git push
```

#### Step 5: Access Application

When users access `https://myapp.forteapps.net`:
1. They're redirected to the identity provider login page
2. After successful login, redirected back to `/auth/callback`
3. Session cookie is set
4. Subsequent requests are authenticated via cookie

**User flow**:
```
User → https://myapp.forteapps.net
  ↓
Redirect → https://keycloak.forteapps.net/login
  ↓
Login successful → Redirect with auth code
  ↓
https://myapp.forteapps.net/auth/callback?code=xyz
  ↓
Auth sidecar exchanges code for tokens
  ↓
Sets session cookie
  ↓
Redirects to application → https://myapp.forteapps.net
  ↓
User sees application (authenticated)
```

---

### Accessing Authenticated User Information

The auth sidecar handles all authentication before requests reach your application. Your app never sees unauthenticated traffic — the sidecar returns 401 or redirects to the IdP first.

After successful authentication, the sidecar forwards the request to your application with user identity injected as HTTP headers:

| Header | Description | Available in |
|--------|-------------|-------------|
| `X-Auth-User` | Username or display name | Token, OIDC, MCP |
| `X-Auth-Email` | User email address | OIDC |
| `X-Auth-Subject` | OIDC `sub` claim (stable user ID) | OIDC, MCP |
| `X-Auth-Groups` | Comma-separated group memberships | OIDC (if scope includes `groups`) |
| `X-Auth-Token` | The validated access token | All modes |

**Your application reads these headers — no auth library needed:**

```javascript
// Express.js example
app.get('/profile', (req, res) => {
  const user = req.headers['x-auth-user'];
  const email = req.headers['x-auth-email'];
  res.json({ user, email });
});
```

```python
# Flask example
@app.route('/profile')
def profile():
    user = request.headers.get('X-Auth-User')
    email = request.headers.get('X-Auth-Email')
    return jsonify(user=user, email=email)
```

**Why this is safe**: The Kyverno-generated NetworkPolicy restricts ingress to the sidecar port only. Traffic cannot bypass the sidecar to reach the application port directly, so the `X-Auth-*` headers can be trusted unconditionally.

**Key principle**: Your application is zero-trust-unaware by design. It reads headers and renders UI. All authentication complexity lives in the sidecar and Kyverno policy.

---

### Authentication Configuration Reference

#### Helm Values Schema

```yaml
auth:
  enabled: false                 # Enable/disable authentication
  type: token                    # "token", "oidc", or "mcp"

  # Token mode configuration
  tokens: []                     # List of valid bearer tokens
  # - token1
  # - token2

  # OIDC mode configuration
  oidc:
    authority: ""                # OIDC provider URL (required for OIDC)
    clientId: ""                 # OIDC client ID (required for OIDC)
    scopes: "openid,profile,email"  # OIDC scopes (optional)
    callbackPath: /auth/callback    # OAuth callback path (optional)

  # MCP mode configuration (RFC 9728 / RFC 7591)
  mcp:
    resource: ""                 # Protected resource URL (required for MCP)
    authority: ""                # Authorization server URL (required for MCP)
    scopes: "read,write"         # Supported scopes (optional)
```

#### Annotations Set by Helm Chart

When `auth.enabled: true`, the Helm chart sets these pod annotations:

**Token mode**:
```yaml
policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "token"
policies.forteapps.io/auth-token-secret-name: "auth-tokens"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"
```

**OIDC mode**:
```yaml
policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "oidc"
policies.forteapps.io/auth-oidc-authority: "https://keycloak.forteapps.net/realms/master"
policies.forteapps.io/auth-oidc-client-id: "myapp"
policies.forteapps.io/auth-oidc-scopes: "openid,profile,email"
policies.forteapps.io/auth-oidc-callback-path: "/auth/callback"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"
```

**MCP mode** (OAuth 2.0 for MCP servers):
```yaml
policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "mcp"
policies.forteapps.io/auth-mcp-resource: "https://mcp.forteapps.net"
policies.forteapps.io/auth-mcp-authority: "https://keycloak.forteapps.net/realms/master"
policies.forteapps.io/auth-mcp-scopes: "read,write"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"
```

#### Sidecar Configuration

The auth sidecar container:
- **Image**: `ghcr.io/fortedigital/auth-sidecar:latest`
- **Port**: 8080
- **Resources**: 10m CPU / 32Mi memory (requests), 50m CPU / 64Mi memory (limits)
- **Health checks**: `/healthz` endpoint
- **Security**: Read-only root filesystem, no privilege escalation

#### Advanced: Custom Sidecar Image

To use a different auth sidecar image:

```yaml
# These annotations can be set in the Helm chart template if needed
policies.forteapps.io/auth-image: "your-registry/your-auth-proxy"
policies.forteapps.io/auth-image-version: "v1.2.3"
```

---

### Authentication Examples

#### Example 1: Internal API with Token Auth

```yaml
# helm-prod-values/internal-api/values.yaml
app:
  image:
    repository: ghcr.io/company/internal-api
    tag: v1.0.0

auth:
  enabled: true
  type: token
  tokens:
  - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823  # Service A
  - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7  # Service B

ingress:
  enabled: true
  host: internal-api.forteapps.net
```

**Usage**:
```bash
# Service A calls API
curl -H "Authorization: Bearer d4f88f..." \
  https://internal-api.forteapps.net/api/endpoint
```

#### Example 2: User-Facing App with OIDC

```yaml
# helm-prod-values/web-app/values.yaml
app:
  image:
    repository: ghcr.io/company/web-app
    tag: v2.1.0

auth:
  enabled: true
  type: oidc
  oidc:
    authority: https://auth.company.com/realms/employees
    clientId: web-app-prod
    scopes: "openid,profile,email,groups"
    callbackPath: /auth/callback

ingress:
  enabled: true
  host: web-app.forteapps.net
```

**With Vault OIDC secret**:
```bash
# Write OIDC secret to Vault
vault kv put kv/web-app/auth-oidc \
  client-secret=super-secret-value \
  cookie-secret=$(openssl rand -hex 32)
# Then create VaultStaticSecret CRD — see docs/vault-secrets-operator.md
```

#### Example 3: MCP Server with OAuth 2.0

```yaml
# helm-prod-values/mcp-server/values.yaml
app:
  image:
    repository: ghcr.io/company/mcp-server
    tag: v1.0.0

auth:
  enabled: true
  type: mcp
  mcp:
    resource: https://mcp-server.forteapps.net
    authority: https://auth.company.com/realms/mcp
    scopes: "read,write,admin"

ingress:
  enabled: true
  host: mcp-server.forteapps.net
```

The MCP auth mode implements RFC 9728 (OAuth 2.0 Protected Resource Metadata) for authorization server discovery and RFC 7591 (OAuth 2.0 Dynamic Client Registration) for automatic client registration. MCP clients discover the authorization server and scopes from the `/.well-known/oauth-protected-resource` endpoint served by the sidecar.

#### Example 4: Disabling Authentication

```yaml
# helm-prod-values/public-api/values.yaml
auth:
  enabled: false                 # No authentication

ingress:
  enabled: true
  host: public-api.forteapps.net
```

---

### Troubleshooting Authentication

#### Issue: 401 Unauthorized (Token Mode)

**Check token validity**:
```bash
# Get auth-tokens secret
kubectl get secret auth-tokens -n myapp -o yaml

# Decode tokens
kubectl get secret auth-tokens -n myapp \
  -o jsonpath='{.data.tokens}' | base64 -d

# Verify your token is in the list
```

**Test with different token**:
```bash
curl -v -H "Authorization: Bearer YOUR-TOKEN-HERE" \
  https://myapp.forteapps.net/
```

#### Issue: OIDC Login Loop

**Check OIDC configuration**:
```bash
# Verify auth-oidc secret exists
kubectl get secret auth-oidc -n myapp

# Check sidecar logs
kubectl logs -n myapp <pod-name> -c authn

# Common issues:
# - Wrong authority URL
# - Wrong client ID
# - Missing client-secret in auth-oidc Secret
# - Redirect URI not configured in identity provider
```

**Verify redirect URI** in your identity provider matches:
```
https://<your-app-domain>/auth/callback
```

#### Issue: Auth Sidecar Not Injected

**Check pod annotations**:
```bash
kubectl get pod -n myapp <pod-name> -o yaml | grep policies.forteapps.io

# Should show:
# policies.forteapps.io/auth: "true"
```

**Check Kyverno policy**:
```bash
kubectl get clusterpolicy inject-auth-sidecar
kubectl describe clusterpolicy inject-auth-sidecar
```

**Check Kyverno logs**:
```bash
kubectl logs -n kyverno deployment/kyverno | grep inject-auth
```

#### Issue: Auth Sidecar Crashes

**Check sidecar logs**:
```bash
kubectl logs -n myapp <pod-name> -c authn
```

**Common causes**:
- Missing secret (auth-tokens or auth-oidc)
- Invalid OIDC configuration
- Can't reach OIDC authority URL
- Network policy blocking outbound OIDC requests

---

### Authentication Best Practices

✅ **DO**:
- Use OIDC for user-facing applications
- Use token auth for service-to-service communication
- Rotate tokens and secrets regularly
- Use strong random tokens (32+ bytes)
- Store client secrets in Vault
- Test authentication before deploying to production
- Document which tokens/users have access

❌ **DON'T**:
- Share tokens between environments
- Commit tokens to application code
- Use predictable tokens
- Reuse tokens across multiple applications
- Disable authentication on sensitive APIs
- Log tokens or secrets

---

## Adding a New Keycloak Client

There are two ways to add an OIDC client, depending on your use case:

| Method | Best for | Who edits the infra repo? |
|--------|----------|--------------------------|
| **Self-service** (recommended) | New apps that deploy their own resources | App developer — no infra changes needed |
| **Legacy (realm JSON)** | Existing clients already defined in forte-realm.json (e.g., Gitea) | Platform engineer |

Both methods are served by the **Keycloak Client Registrar** CronJob, which runs every 2 minutes.

### Self-Service OIDC Client Registration

This is the recommended flow for new applications. Your app deploys a labeled config Secret in its own namespace; the platform handles everything else.

#### How It Works

1. You deploy a Secret with label `keycloak.forteapps.net/client-config: "true"` containing a `client.json` definition
2. A **Kyverno ClusterPolicy** (`keycloak-client-config-cloner`) clones it to the `keycloak` namespace
3. The **Client Registrar CronJob** picks it up within 2 minutes:
   - Registers (or updates) the client in Keycloak
   - Fetches the auto-generated client secret
   - Creates a credential Secret in your app's namespace
   - Annotates the config Secret with sync status

#### Step 1: Create the Config Secret

Deploy this Secret in your application's namespace (e.g., as part of your Helm chart or Kustomize overlay):

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: keycloak-client-myapp
  namespace: myapp
  labels:
    keycloak.forteapps.net/client-config: "true"
stringData:
  client.json: |
    {
      "clientId": "myapp",
      "name": "My Application",
      "redirectUris": ["https://myapp.forteapps.net/*"],
      "webOrigins": ["https://myapp.forteapps.net"],
      "defaultClientScopes": ["openid", "email", "profile"],
      "protocolMappers": [],
      "secret": {
        "namespace": "myapp",
        "name": "myapp-oidc-credentials",
        "keys": { "clientId": "client-id", "clientSecret": "client-secret" }
      }
    }
```

**`client.json` fields**:

| Field | Required | Description |
|-------|----------|-------------|
| `clientId` | Yes | Keycloak client ID |
| `name` | Yes | Display name in Keycloak |
| `redirectUris` | Yes | Allowed redirect URIs |
| `webOrigins` | Yes | Allowed web origins (CORS) |
| `defaultClientScopes` | No | Scopes (default: `["openid", "email", "profile"]`) |
| `protocolMappers` | No | Custom claim mappers (default: `[]`) |
| `secret.namespace` | No | Namespace for the credential Secret (default: source namespace) |
| `secret.name` | No | Name of the credential Secret (default: `<clientId>-oidc-credentials`) |
| `secret.keys.clientId` | No | Key name for client ID in credential Secret (default: `client-id`) |
| `secret.keys.clientSecret` | No | Key name for client secret in credential Secret (default: `client-secret`) |

#### Step 2: Reference the Credential Secret

In your application's deployment config, reference the credential Secret that the registrar creates:

```yaml
env:
- name: OIDC_CLIENT_ID
  valueFrom:
    secretKeyRef:
      name: myapp-oidc-credentials
      key: client-id
- name: OIDC_CLIENT_SECRET
  valueFrom:
    secretKeyRef:
      name: myapp-oidc-credentials
      key: client-secret
```

#### Step 3: Deploy and Wait

Commit and push your changes. The credential Secret will appear within 2 minutes:

```bash
# Watch for the credential Secret to be created
kubectl get secret myapp-oidc-credentials -n myapp -w

# Check registrar logs
kubectl logs -n keycloak job/$(kubectl get jobs -n keycloak --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')

# Check sync status on the config Secret
kubectl get secret keycloak-client-myapp -n keycloak -o jsonpath='{.metadata.annotations}'
```

#### Change Detection

The registrar computes a SHA-256 hash of `client.json` and stores it as an annotation. On subsequent runs, it skips processing if:
- The hash hasn't changed, AND
- The credential Secret already exists in the target namespace

To force a re-sync, update any field in `client.json` (e.g., add a trailing space to `name`).

### Legacy Method: Realm JSON

Existing clients (like Gitea) are defined directly in `forte-realm.json` inside `keycloak-values.yaml`. The registrar syncs their secrets via client attributes.

#### Step 1: Add Client to Realm Config

In `infra/values/base/keycloak-values.yaml`, add a new entry to the `clients` array in `forte-realm.json`:

```json
{
  "clientId": "myapp",
  "name": "My Application",
  "enabled": true,
  "protocol": "openid-connect",
  "clientAuthenticatorType": "client-secret",
  "standardFlowEnabled": true,
  "directAccessGrantsEnabled": false,
  "publicClient": false,
  "redirectUris": ["https://myapp.forteapps.net/*"],
  "webOrigins": ["https://myapp.forteapps.net"],
  "defaultClientScopes": ["openid", "email", "profile"],
  "attributes": {
    "k8s.secret.sync": "true",
    "k8s.secret.namespace": "myapp",
    "k8s.secret.name": "myapp-oidc-credentials",
    "k8s.secret.client-id-key": "key",
    "k8s.secret.client-secret-key": "secret"
  }
}
```

**Important**:
- Do **NOT** include a `"secret"` field — Keycloak generates one automatically
- The `attributes` block tells the registrar where to create the K8s Secret
- Set `client-id-key` / `client-secret-key` to match what the consuming app expects (defaults: `client-id` / `client-secret`)

#### Step 2: Reference the Secret in Your Application

```yaml
existingSecret: myapp-oidc-credentials
```

#### Step 3: Commit and Push

```bash
cd ~/dev/k8s/launchpad
git add infra/values/base/keycloak-values.yaml
git commit -m "Add myapp Keycloak client with auto-sync"
git push
```

ArgoCD will sync the Keycloak config, and the registrar CronJob will pick up the new client within 2 minutes.

#### Legacy Sync Attribute Reference

| Attribute | Required | Default | Description |
|-----------|----------|---------|-------------|
| `k8s.secret.sync` | Yes | — | Set to `"true"` to enable syncing |
| `k8s.secret.namespace` | Yes | — | Target K8s namespace for the secret |
| `k8s.secret.name` | Yes | — | Name of the K8s Secret to create |
| `k8s.secret.client-id-key` | No | `client-id` | Field name for the client ID in the K8s Secret |
| `k8s.secret.client-secret-key` | No | `client-secret` | Field name for the client secret in the K8s Secret |

### Retrieving Secrets for External Deployments

The registrar always writes a **central copy** of every synced secret to the `secrets` namespace, in addition to the target namespace. This allows operators to retrieve client credentials for applications deployed outside this cluster:

```bash
# View the central copy
kubectl get secret gitea-oidc-credentials -n secrets -o yaml

# Extract the client secret for use elsewhere
kubectl get secret myapp-oidc-credentials -n secrets \
  -o jsonpath='{.data.client-secret}' | base64 -d
```

### Registrar Behavior Notes

- The registrar runs as a CronJob every 2 minutes (`concurrencyPolicy: Forbid`)
- If the target namespace doesn't exist, the target write is skipped with a warning (the central copy still happens)
- A central copy is **always** written to the `secrets` namespace for every synced client
- The registrar uses the `keycloak-credentials` secret for admin authentication
- Created secrets have the label `app.kubernetes.io/managed-by: keycloak-client-registrar`

---

## Troubleshooting

### Application Not Deploying

#### Problem: Application stuck in "Syncing" state

**Check ArgoCD status:**
```bash
kubectl get application myapp -n argocd -o yaml
```

Look for errors in `status.conditions`.

**Common causes:**
- ❌ Image doesn't exist or is not accessible
- ❌ Invalid YAML syntax
- ❌ Resource quota exceeded
- ❌ Namespace conflicts
- ❌ Invalid Helm values

**Solutions:**
```bash
# Check image exists
docker pull ghcr.io/fortedigital/myapp:v1.0.0

# Validate YAML syntax
kubectl apply --dry-run=client -f apps/myapp.yaml

# Check ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller | grep myapp
```

#### Problem: Pods crashing (CrashLoopBackOff)

**Check pod logs:**
```bash
kubectl get pods -n myapp
kubectl logs -n myapp <pod-name>
kubectl describe pod -n myapp <pod-name>
```

**Common causes:**
- ❌ Application error (check logs)
- ❌ Missing environment variables
- ❌ Incorrect port configuration
- ❌ Missing secrets
- ❌ Insufficient resources

**Solutions:**
```bash
# Check environment variables
kubectl exec -n myapp <pod-name> -- env

# Check if secrets exist
kubectl get secrets -n myapp

# Increase resources in helm-prod-values
vim ~/dev/k8s/helm-prod-values/myapp/values.yaml
```

#### Problem: Application not accessible via domain

**Check ingress:**
```bash
kubectl get ingressroute -n myapp
kubectl describe ingressroute myapp -n myapp
```

**Common causes:**
- ❌ DNS not configured
- ❌ TLS certificate not issued
- ❌ Incorrect domain in values.yaml
- ❌ Traefik not routing correctly

**Solutions:**
```bash
# Check certificate
kubectl get certificate -n myapp

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Verify domain configuration
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep host

# Test with port-forward
kubectl port-forward -n myapp service/myapp 8080:3000
curl http://localhost:8080
```

### Secret Issues

#### Problem: Secret not found

**Check VSO sync status:**
```bash
kubectl get vaultstaticsecret -n myapp
kubectl get secret -n myapp
```

**Solutions:**
```bash
# Check VaultAuth is authenticated
kubectl get vaultauth -n myapp

# Check VaultStaticSecret events
kubectl describe vaultstaticsecret myapp-credentials -n myapp

# Verify secret exists in Vault
vault kv get kv/myapp/myapp-credentials
```

#### Problem: Secret exists but pods can't access it

**Check pod events:**
```bash
kubectl describe pod -n myapp <pod-name>
```

Look for: `Error: secret "myapp-credentials" not found`

**Solutions:**
```bash
# Verify secret name in values.yaml matches actual secret
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep envSecretName
kubectl get secrets -n myapp

# Restart pods
kubectl rollout restart deployment myapp -n myapp
```

### Sync Failures

#### Problem: ArgoCD shows "Out of Sync"

**Manual sync:**
```bash
# Using kubectl
kubectl patch application myapp -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"syncStrategy":{"hook":{}}}}}'

# Or via ArgoCD UI
# Click "Sync" button in UI
```

**Check what's different:**
```bash
kubectl get application myapp -n argocd -o yaml
```

Look at `status.sync.comparedTo` vs desired state.

#### Problem: Sync succeeds but application is "Degraded"

**Check resource health:**
```bash
kubectl get application myapp -n argocd -o jsonpath='{.status.resources[*].health}'
```

**Common causes:**
- ❌ Pods not ready
- ❌ Deployments not at desired replica count
- ❌ Jobs failed

**Solutions:**
```bash
# Check all resources in namespace
kubectl get all -n myapp

# Check pod events
kubectl get events -n myapp --sort-by='.lastTimestamp'
```

### Getting Help

If you're stuck:

1. **Check Slack notifications** - Error details are often in sync failure messages
2. **Check ArgoCD UI** - Visual representation of what's wrong
3. **Ask platform team** - They have full cluster access and can debug further
4. **Check documentation** - [Operations Runbook](OPERATIONS-RUNBOOK.md) has more troubleshooting

---

## Best Practices

### Development Workflow

✅ **DO**:
- Develop and test locally with Docker Compose
- Use semantic versioning for releases
- Write descriptive commit messages
- Test changes in a separate namespace first (if possible)
- Monitor Slack for deployment notifications
- Document environment variables and configuration

❌ **DON'T**:
- Push directly to production without testing
- Use `latest` tag for Docker images
- Bypass CI/CD for "quick fixes"
- Hard-code configuration values
- Ignore deployment failures

### Configuration Management

✅ **DO**:
- Keep configuration in `helm-prod-values` repository
- Use environment variables for config
- Document what each value does
- Use reasonable resource limits
- Enable ingress and TLS for public services

❌ **DON'T**:
- Hard-code config in application code
- Over-allocate resources (wastes money)
- Under-allocate resources (causes crashes)
- Use HTTP for production services

### Secret Management

✅ **DO**:
- Use Vault for all secrets (see docs/vault-secrets-operator.md)
- Store plain secrets in password manager
- Rotate secrets regularly
- Use different secrets per environment
- Document what each secret contains

❌ **DON'T**:
- Commit plain secrets
- Share secrets in Slack/email
- Reuse secrets across apps
- Log secrets in application code

### Git Workflow

✅ **DO**:
- Use feature branches for changes
- Write clear commit messages
- Use pull requests for review
- Keep commits atomic and focused
- Tag releases in application repos

❌ **DON'T**:
- Push directly to `main` without review (for config repos)
- Make multiple unrelated changes in one commit
- Use vague commit messages ("fix", "update")
- Force-push to main branches

---

## Quick Reference

### Common Commands

```bash
# Check application status
kubectl get application myapp -n argocd

# View application details
kubectl describe application myapp -n argocd

# Check pods
kubectl get pods -n myapp

# View pod logs
kubectl logs -n myapp <pod-name>

# Restart deployment
kubectl rollout restart deployment myapp -n myapp

# Port-forward to service
kubectl port-forward -n myapp service/myapp 8080:3000

# Write secret to Vault
vault kv put kv/myapp/myapp-credentials KEY=value
# Create VaultStaticSecret CRD — see docs/vault-secrets-operator.md
```

### Repository Locations

```bash
# Config repository
cd ~/dev/k8s/launchpad

# Helm values repository
cd ~/dev/k8s/helm-prod-values

# Helm charts repository
cd ~/dev/k8s/forte-helm
```

### File Paths

```bash
# New application manifest
~/dev/k8s/launchpad/apps/myapp.yaml

# Application values
~/dev/k8s/helm-prod-values/myapp/values.yaml

# Sealed secrets
~/dev/k8s/launchpad/secrets/myapp-credentials-sealed.yaml

# Plain secrets (local only)
~/dev/k8s/launchpad/private/myapp-credentials.yaml
```

---

## Next Steps

Now that you understand the basics:

1. ✅ Deploy your first application (follow steps above)
2. 📖 Read the [Operations Runbook](OPERATIONS-RUNBOOK.md) for common tasks
3. 📖 Review [Technical Reference](REFERENCE.md) for detailed component docs
4. 📖 Understand [GitOps Architecture](GITOPS-ARCHITECTURE.md) for the big picture
5. 🚀 Start contributing!

---

**Questions?**
- Slack: #platform-support
- Docs: [Full documentation index](README.md)
- Help: Contact platform team

**Last Updated**: 2026-04-16