launchpad/docs/DEVELOPER-GUIDE.md

# Developer Onboarding Guide

## Table of Contents
- [Getting Started](#getting-started)
- [Prerequisites](#prerequisites)
- [Local Development Setup](#local-development-setup)
- [Understanding the Workflow](#understanding-the-workflow)
- [Deploying Your First Application](#deploying-your-first-application)
- [Updating an Existing Application](#updating-an-existing-application)
- [Working with Secrets](#working-with-secrets)
- [Enabling Authentication for Applications](#enabling-authentication-for-applications)
- [Troubleshooting](#troubleshooting)
- [Best Practices](#best-practices)

---

## Getting Started

Welcome! This guide will help you understand how to develop and deploy applications on our Kubernetes cluster using GitOps principles powered by ArgoCD.

### What You'll Learn
- How our GitOps architecture works
- How to deploy a new application
- How to update existing applications
- How to manage secrets securely
- Common troubleshooting techniques

### Who This Guide Is For
- Developers deploying new applications
- Developers maintaining existing applications
- Team members who need to understand the deployment process

---

## Prerequisites

### Required Knowledge
- ✅ Basic Git workflow (clone, commit, push, pull)
- ✅ Docker basics (Dockerfile, building images)
- ✅ YAML syntax
- ✅ Basic understanding of Kubernetes concepts (pods, deployments, services)
- ⚠️ Helm knowledge (helpful but not required - templates are provided)

### Required Tools

Most developers **do NOT need kubectl access** to the cluster. You'll primarily work with Git repositories.

If you do need cluster access, install:

1. **kubectl** - Kubernetes CLI
   ```bash
   # macOS
   brew install kubectl

   # Windows
   choco install kubernetes-cli

   # Linux
   curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   ```

2. **kubeseal** - For sealing secrets
   ```bash
   # macOS
   brew install kubeseal

   # Windows
   choco install kubeseal

   # Linux
   wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
   tar -xvzf kubeseal-0.24.0-linux-amd64.tar.gz
   sudo mv kubeseal /usr/local/bin/
   ```

3. **Git** - Version control
   ```bash
   git --version  # Should already be installed
   ```

4. **Docker** - For local development
   ```bash
   # macOS/Windows: Install Docker Desktop
   # Linux: Install Docker Engine
   docker --version
   ```

### Repository Access

You'll need read/write access to these repositories:

1. **sturdy-adventure** (Config repo)
   ```bash
   git clone https://github.com/fortedigital/sturdy-adventure.git
   cd sturdy-adventure
   ```

2. **helm-values** (Values repo)
   ```bash
   git clone git@github.com:fortedigital/helm-values.git
   cd helm-values
   ```

3. **forte-helm** (Chart repo - read-only for most developers)
   ```bash
   git clone https://github.com/snothub/forte-helm.git
   cd forte-helm
   ```

### Cluster Access (If Needed)

If you need kubectl access, ask the platform team for:
- Kubeconfig file
- Cluster context setup instructions

Save to `~/.kube/config` and verify:
```bash
kubectl cluster-info
kubectl get nodes
```

---

## Local Development Setup

### 1. Clone the Repositories

Set up a consistent folder structure:

```bash
mkdir -p ~/dev/k8s
cd ~/dev/k8s

# Clone repositories
git clone https://github.com/fortedigital/sturdy-adventure.git launchpad
git clone git@github.com:fortedigital/helm-values.git helm-prod-values
git clone https://github.com/snothub/forte-helm.git forte-helm

# Your folder structure:
# ~/dev/k8s/
# ├── launchpad/           (Config repo)
# ├── helm-prod-values/    (Values repo)
# └── forte-helm/          (Chart repo)
```

### 2. Local Development Environment

Most applications use **Docker Compose** for local development:

```bash
# In your application repository
docker-compose up

# Or for frontend applications
npm install
npm run dev
```

**You DO NOT run applications locally on Kubernetes.** Use Docker Compose or native tooling (npm, python, etc.).

### 3. Understanding the Deployment Flow

```
┌─────────────────────────────────────────────────────────────────┐
│  Step 1: Develop Locally                                        │
│  - Write code in your application repository                    │
│  - Test with Docker Compose or npm/python/etc.                  │
│  - Build Docker image                                            │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 2: CI/CD Pipeline (Automated)                             │
│  - GitHub Actions builds image                                  │
│  - Pushes to container registry (GHCR, Docker Hub)              │
│  - Tags with version (e.g., v2.0.4)                             │
│  - Updates helm-values repository with new tag                  │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 3: GitOps Sync (Automated)                                │
│  - ArgoCD detects change in helm-values                         │
│  - Pulls updated configuration                                  │
│  - Syncs to Kubernetes cluster                                  │
│  - Sends Slack notification on success/failure                  │
└─────────────────────────────────────────────────────────────────┘
```

**Key Insight**: You don't deploy directly. You push code, CI/CD builds it, and ArgoCD deploys it.

---

## Understanding the Workflow

### Three-Repository Pattern

Our setup uses three repositories:

| Repository | Purpose | You Edit |
|------------|---------|----------|
| **forte-helm** | Helm chart templates (generic, reusable) | ❌ Rarely |
| **helm-values** | Application configuration (image tag, env vars) | ✅ Sometimes |
| **sturdy-adventure** | ArgoCD Applications (what gets deployed) | ✅ Yes (for new apps) |

### Example: Deploying "myapp"

#### Repository: `forte-helm` (Chart Templates)
```yaml
# forteapp/templates/deployment.yaml
# Generic template used by ALL apps
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
spec:
  containers:
  - name: app
    image: "{{ .Values.app.image.repository }}:{{ .Values.app.image.tag }}"
    env:
    - name: PORT
      value: {{ .Values.app.port }}
```

#### Repository: `helm-values` (Your App Config)
```yaml
# myapp/values.yaml
# Your app's specific configuration
app:
  image:
    repository: ghcr.io/fortedigital/myapp
    tag: v1.0.0                    # CI/CD updates this
  port: 3000
  extraEnv:
  - name: API_URL
    value: https://api.example.com
```

#### Repository: `sturdy-adventure` (ArgoCD Application)
```yaml
# apps/myapp.yaml
# Tells ArgoCD to deploy your app
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  sources:
  - repoURL: https://github.com/snothub/forte-helm
    path: forteapp
    helm:
      valueFiles:
      - $values/myapp/values.yaml

  - repoURL: git@github.com:fortedigital/helm-values.git
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: myapp

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
```

---

## Deploying Your First Application

### Scenario: You've Built a New Application

Let's deploy a new Node.js application called "hello-world".

### Step 1: Prepare Your Application Repository

Ensure your app repository has:

1. **Dockerfile**
   ```dockerfile
   FROM node:18-alpine
   WORKDIR /app
   COPY package*.json ./
   RUN npm ci --only=production
   COPY . .
   EXPOSE 3000
   CMD ["node", "server.js"]
   ```

2. **GitHub Actions Workflow** (`.github/workflows/deploy.yml`)
   ```yaml
   name: Build and Deploy

   on:
     push:
       branches: [ main ]

   jobs:
     build:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v3

         - name: Set version
           id: version
           run: echo "VERSION=v$(date +%Y%m%d-%H%M%S)" >> $GITHUB_OUTPUT

         - name: Build and push Docker image
           run: |
             echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
             docker build -t ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }} .
             docker push ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }}

         - name: Update helm-values
           run: |
             git clone git@github.com:fortedigital/helm-values.git
             cd helm-values
             mkdir -p hello-world
             cat > hello-world/values.yaml <<EOF
             app:
               image:
                 repository: ghcr.io/fortedigital/hello-world
                 tag: ${{ steps.version.outputs.VERSION }}
             EOF
             git add hello-world/values.yaml
             git commit -m "Update hello-world to ${{ steps.version.outputs.VERSION }}"
             git push
   ```

### Step 2: Create Helm Values

Create a folder in `helm-values` repository:

```bash
cd ~/dev/k8s/helm-prod-values
mkdir -p hello-world
```

Create `hello-world/values.yaml`:
```yaml
app:
  image:
    repository: ghcr.io/fortedigital/hello-world
    tag: v1.0.0                    # Will be updated by CI/CD
    containerPort: 3000

  replicaCount: 1

  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi

  extraEnv:
  - name: PORT
    value: "3000"
  - name: NODE_ENV
    value: "production"

  envSecretName: ""                # Optional: reference to secrets

service:
  port: 3000

ingress:
  enabled: true
  host: hello-world.forteapps.net  # Your subdomain

db:
  enabled: false                   # Set to true if you need PostgreSQL
```

Commit and push:
```bash
git add hello-world/values.yaml
git commit -m "Add hello-world application values"
git push
```

### Step 3: Create ArgoCD Application Manifest

In the `sturdy-adventure` repository, create `apps/hello-world.yaml`:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello-world
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: ""
    notifications.argoproj.io/subscribe.on-sync-failed.slack: ""
    notifications.argoproj.io/subscribe.on-degraded.slack: ""
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/part-of: apps
    app.kubernetes.io/managed-by: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io

spec:
  project: default

  sources:
  # Source 1: Helm chart templates
  - repoURL: https://github.com/snothub/forte-helm
    path: forteapp
    targetRevision: HEAD
    helm:
      valueFiles:
      - $values/hello-world/values.yaml

  # Source 2: Helm values
  - repoURL: git@github.com:fortedigital/helm-values.git
    targetRevision: HEAD
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: hello-world

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false

    syncOptions:
    - CreateNamespace=true
    - Validate=true
    - ServerSideApply=true

    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas
```

Commit and push:
```bash
cd ~/dev/k8s/launchpad
git add apps/hello-world.yaml
git commit -m "Add hello-world application"
git push
```

### Step 4: Verify Deployment

ArgoCD will automatically detect the new application within 60 seconds.

**Option 1: Check Slack**
- Watch for sync notifications in your Slack channel
- ✅ "Application hello-world sync succeeded"

**Option 2: Check ArgoCD UI** (if you have access)
```bash
# Port forward to ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Open browser: https://localhost:8080
# Look for "hello-world" application
```

**Option 3: Check with kubectl** (if you have access)
```bash
# List ArgoCD applications
kubectl get applications -n argocd

# Check application status
kubectl get application hello-world -n argocd

# Verify pods are running
kubectl get pods -n hello-world
```

### Step 5: Access Your Application

Once deployed, access via the configured domain:

```bash
# Check if ingress is created
kubectl get ingressroute -n hello-world

# Access application
curl https://hello-world.forteapps.net
```

**⚠️ Note**: DNS must be manually configured for new subdomains. Contact the platform team to add DNS records.

---

## Updating an Existing Application

### Scenario: Deploying a Code Change

You've made changes to your application code and want to deploy them.

### Method 1: Automatic (Recommended)

**Just push to `main` branch** - CI/CD handles everything:

```bash
# In your application repository
git add .
git commit -m "Fix bug in user login"
git push origin main
```

**What Happens Next:**
1. ✅ GitHub Actions triggers
2. ✅ Builds new Docker image
3. ✅ Tags with new version (e.g., `v20260316-143022`)
4. ✅ Pushes to container registry
5. ✅ Updates `helm-values/myapp/values.yaml` with new tag
6. ✅ ArgoCD detects change
7. ✅ Syncs new version to cluster
8. ✅ Sends Slack notification

**Timeline**: ~5-10 minutes from push to deployment

### Method 2: Manual Image Tag Update

If CI/CD is not set up, manually update the image tag:

```bash
cd ~/dev/k8s/helm-prod-values

# Edit your app's values.yaml
vim myapp/values.yaml

# Change:
app:
  image:
    tag: v1.0.0  # Old version
# To:
app:
  image:
    tag: v1.0.1  # New version

# Commit and push
git add myapp/values.yaml
git commit -m "Update myapp to v1.0.1"
git push
```

ArgoCD will sync within 60 seconds.

### Method 3: Configuration Changes

To update environment variables, resources, or other config:

```bash
cd ~/dev/k8s/helm-prod-values
vim myapp/values.yaml
```

Example changes:

```yaml
app:
  # Increase resources
  resources:
    requests:
      cpu: 200m      # Was 100m
      memory: 256Mi  # Was 128Mi

  # Add new environment variable
  extraEnv:
  - name: API_URL
    value: https://api.example.com
  - name: DEBUG          # NEW
    value: "true"        # NEW

  # Enable HPA
  hpa:
    enabled: true        # Was false
    minReplicas: 2
    maxReplicas: 10
```

Commit and push:
```bash
git add myapp/values.yaml
git commit -m "Increase myapp resources and enable HPA"
git push
```

### Method 4: Application Manifest Changes

To change ArgoCD sync behavior, namespace, or other meta-config:

```bash
cd ~/dev/k8s/launchpad
vim apps/myapp.yaml
```

Example changes:

```yaml
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: false    # Disable self-healing temporarily
```

Commit and push:
```bash
git add apps/myapp.yaml
git commit -m "Disable self-healing for myapp"
git push
```

---

## Working with Secrets

### Understanding Secret Management

**NEVER commit plain secrets to Git.** We use **Sealed Secrets** to encrypt secrets before committing.

### Creating a New Secret

#### Step 1: Create Plain Secret Locally

```bash
cd ~/dev/k8s/launchpad

# Create secret in private/ folder (Git-ignored)
kubectl create secret generic myapp-credentials \
  --from-literal=API_KEY=your-secret-key-here \
  --from-literal=DB_PASSWORD=super-secret-password \
  --dry-run=client -o yaml > private/myapp-credentials.yaml
```

**DO NOT commit this file!** It's in `private/` which is Git-ignored.

#### Step 2: Seal the Secret

Get the public certificate (one-time setup):

```bash
# Fetch public cert from cluster
kubeseal --fetch-cert \
  --controller-name=sealed-secrets-controller \
  --controller-namespace=kube-system \
  > pub-cert.pem
```

Seal your secret:

```bash
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml
```

#### Step 3: Commit Sealed Secret

```bash
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Add myapp credentials (sealed)"
git push
```

#### Step 4: Reference Secret in Application

Update your `helm-values/myapp/values.yaml`:

```yaml
app:
  envSecretName: "myapp-credentials"  # References the SealedSecret
```

Commit and push:
```bash
cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Reference myapp credentials"
git push
```

### Updating a Secret

To update an existing secret:

```bash
# 1. Create new version of secret
kubectl create secret generic myapp-credentials \
  --from-literal=API_KEY=new-key-here \
  --from-literal=DB_PASSWORD=new-password \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

# 2. Seal it
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml

# 3. Commit sealed version
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Update myapp credentials"
git push

# 4. Restart pods to pick up new secret
kubectl rollout restart deployment myapp -n myapp
```

### Secret Best Practices

✅ **DO**:
- Store secrets in `private/` folder locally
- Always seal secrets before committing
- Delete plain secrets after sealing
- Use meaningful secret names
- Document what each secret contains

❌ **DON'T**:
- Commit plain secrets to Git
- Share secrets via Slack/email
- Hard-code secrets in code
- Use the same secret across multiple environments
- Store secrets in Docker images

### Where Secrets Are Stored

```
┌─────────────────────────────────────────────────────────────┐
│  Location                │  Content           │  Committed?│
├──────────────────────────┼────────────────────┼────────────┤
│  private/                │  Plain secrets     │  ❌ NO      │
│  secrets/                │  Sealed secrets    │  ✅ YES     │
│  Kubernetes cluster      │  Unsealed secrets  │  N/A       │
└─────────────────────────────────────────────────────────────┘
```

**Sealed Secrets Controller** in the cluster decrypts sealed secrets automatically.

---

## Enabling Authentication for Applications

The cluster supports automatic authentication sidecar injection for applications via Kyverno policies. This allows you to add authentication to your applications without modifying application code.

### How It Works

When you enable authentication in your Helm values, the Kyverno policy automatically:
1. ✅ Injects an authentication sidecar container into your pod
2. ✅ Routes all incoming traffic through the auth sidecar (port 8080)
3. ✅ Validates credentials before forwarding requests to your application
4. ✅ Creates necessary secrets (if they don't exist)
5. ✅ Adds a NetworkPolicy to restrict ingress

**Architecture**:
```
Internet → Traefik → Service:8080 → Auth Sidecar:8080 → localhost → Your App:3000
                                         │
                                         ├─ Validates credentials
                                         └─ Forwards if valid
```

### Authentication Modes

Two authentication modes are supported:
1. **Token-based**: Static tokens (simple, good for service-to-service or internal apps)
2. **OIDC**: OpenID Connect (full SSO, good for user-facing apps)

---

### Token-Based Authentication

#### Step 1: Configure Helm Values

```yaml
# In helm-values/myapp/values.yaml
auth:
  enabled: true
  type: token                    # Token mode (default)
  tokens:
  - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823
  - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7
```

#### Step 2: Generate Token (if needed)

```bash
# Generate a secure random token
openssl rand -hex 32

# Or using Python
python3 -c "import secrets; print(secrets.token_hex(32))"

# Example output:
# d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823
```

#### Step 3: Deploy Application

Commit and push your changes:
```bash
cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Enable token auth for myapp"
git push
```

ArgoCD will sync, and the Kyverno policy will:
- Inject the auth sidecar container
- Create an `auth-tokens` Secret with your tokens
- Configure the sidecar to validate against these tokens

#### Step 4: Access Application

Use your token in the `Authorization` header:

```bash
# Access application with token
curl -H "Authorization: Bearer d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823" \
  https://myapp.forteapps.net/api/data

# Without token (will be rejected)
curl https://myapp.forteapps.net/api/data
# Response: 401 Unauthorized
```

#### Advanced: Custom Secret Name

To use a different secret for tokens:

```yaml
# In Helm values
auth:
  enabled: true
  type: token
  tokens: []                     # Empty - using external secret

# Tokens will be read from custom secret
```

Then reference it via annotation (configured by Helm chart automatically):
```yaml
# Helm chart sets this annotation:
policies.forteapps.io/auth-token-secret-name: "myapp-auth-tokens"
```

Create the secret manually:
```bash
kubectl create secret generic myapp-auth-tokens \
  --from-file=tokens=tokens.txt \
  --namespace=myapp
```

---

### OIDC Authentication

OIDC mode integrates with identity providers like Keycloak, Okta, Auth0, Azure AD, etc.

#### Step 1: Configure Identity Provider

In your identity provider (e.g., Keycloak):
1. Create a new client (e.g., `myapp`)
2. Set redirect URI: `https://myapp.forteapps.net/auth/callback`
3. Note the **Client ID** and **Client Secret**
4. Note the **Authority URL** (e.g., `https://keycloak.forteapps.net/realms/master`)

#### Step 2: Create OIDC Secret

```bash
# Create plain secret
kubectl create secret generic auth-oidc \
  --from-literal=client-secret=your-oidc-client-secret \
  --from-literal=cookie-secret=$(openssl rand -hex 32) \
  --namespace=myapp \
  --dry-run=client -o yaml > private/myapp-auth-oidc.yaml

# Seal it
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  --namespace=myapp \
  < private/myapp-auth-oidc.yaml \
  > secrets/myapp-auth-oidc-sealed.yaml

# Commit sealed secret
cd ~/dev/k8s/launchpad
git add secrets/myapp-auth-oidc-sealed.yaml
git commit -m "Add OIDC secrets for myapp"
git push

# Clean up
rm private/myapp-auth-oidc.yaml
```

#### Step 3: Configure Helm Values

```yaml
# In helm-values/myapp/values.yaml
auth:
  enabled: true
  type: oidc                     # OIDC mode
  oidc:
    authority: https://keycloak.forteapps.net/realms/master
    clientId: myapp
    scopes: "openid,profile,email"
    callbackPath: /auth/callback
```

#### Step 4: Deploy Application

```bash
cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Enable OIDC auth for myapp"
git push
```

#### Step 5: Access Application

When users access `https://myapp.forteapps.net`:
1. They're redirected to the identity provider login page
2. After successful login, redirected back to `/auth/callback`
3. Session cookie is set
4. Subsequent requests are authenticated via cookie

**User flow**:
```
User → https://myapp.forteapps.net
  ↓
Redirect → https://keycloak.forteapps.net/login
  ↓
Login successful → Redirect with auth code
  ↓
https://myapp.forteapps.net/auth/callback?code=xyz
  ↓
Auth sidecar exchanges code for tokens
  ↓
Sets session cookie
  ↓
Redirects to application → https://myapp.forteapps.net
  ↓
User sees application (authenticated)
```

---

### Authentication Configuration Reference

#### Helm Values Schema

```yaml
auth:
  enabled: false                 # Enable/disable authentication
  type: token                    # "token" or "oidc"

  # Token mode configuration
  tokens: []                     # List of valid bearer tokens
  # - token1
  # - token2

  # OIDC mode configuration
  oidc:
    authority: ""                # OIDC provider URL (required for OIDC)
    clientId: ""                 # OIDC client ID (required for OIDC)
    scopes: "openid,profile,email"  # OIDC scopes (optional)
    callbackPath: /auth/callback    # OAuth callback path (optional)
```

#### Annotations Set by Helm Chart

When `auth.enabled: true`, the Helm chart sets these pod annotations:

**Token mode**:
```yaml
policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "token"
policies.forteapps.io/auth-token-secret-name: "auth-tokens"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"
```

**OIDC mode**:
```yaml
policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "oidc"
policies.forteapps.io/auth-oidc-authority: "https://keycloak.forteapps.net/realms/master"
policies.forteapps.io/auth-oidc-client-id: "myapp"
policies.forteapps.io/auth-oidc-scopes: "openid,profile,email"
policies.forteapps.io/auth-oidc-callback-path: "/auth/callback"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"
```

#### Sidecar Configuration

The auth sidecar container:
- **Image**: `ghcr.io/snothub/stunning-memory:latest`
- **Port**: 8080
- **Resources**: 10m CPU / 32Mi memory (requests), 50m CPU / 64Mi memory (limits)
- **Health checks**: `/healthz` endpoint
- **Security**: Read-only root filesystem, no privilege escalation

#### Advanced: Custom Sidecar Image

To use a different auth sidecar image:

```yaml
# These annotations can be set in the Helm chart template if needed
policies.forteapps.io/auth-image: "your-registry/your-auth-proxy"
policies.forteapps.io/auth-image-version: "v1.2.3"
```

---

### Authentication Examples

#### Example 1: Internal API with Token Auth

```yaml
# helm-values/internal-api/values.yaml
app:
  image:
    repository: ghcr.io/company/internal-api
    tag: v1.0.0

auth:
  enabled: true
  type: token
  tokens:
  - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823  # Service A
  - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7  # Service B

ingress:
  enabled: true
  host: internal-api.forteapps.net
```

**Usage**:
```bash
# Service A calls API
curl -H "Authorization: Bearer d4f88f..." \
  https://internal-api.forteapps.net/api/endpoint
```

#### Example 2: User-Facing App with OIDC

```yaml
# helm-values/web-app/values.yaml
app:
  image:
    repository: ghcr.io/company/web-app
    tag: v2.1.0

auth:
  enabled: true
  type: oidc
  oidc:
    authority: https://auth.company.com/realms/employees
    clientId: web-app-prod
    scopes: "openid,profile,email,groups"
    callbackPath: /auth/callback

ingress:
  enabled: true
  host: web-app.forteapps.net
```

**With sealed OIDC secret**:
```bash
# Create and seal secret
kubectl create secret generic auth-oidc \
  --from-literal=client-secret=super-secret-value \
  --from-literal=cookie-secret=$(openssl rand -hex 32) \
  --namespace=web-app \
  --dry-run=client -o yaml | \
  kubeseal --format=yaml --cert=pub-cert.pem --namespace=web-app \
  > secrets/web-app-auth-oidc-sealed.yaml
```

#### Example 3: Disabling Authentication

```yaml
# helm-values/public-api/values.yaml
auth:
  enabled: false                 # No authentication

ingress:
  enabled: true
  host: public-api.forteapps.net
```

---

### Troubleshooting Authentication

#### Issue: 401 Unauthorized (Token Mode)

**Check token validity**:
```bash
# Get auth-tokens secret
kubectl get secret auth-tokens -n myapp -o yaml

# Decode tokens
kubectl get secret auth-tokens -n myapp \
  -o jsonpath='{.data.tokens}' | base64 -d

# Verify your token is in the list
```

**Test with different token**:
```bash
curl -v -H "Authorization: Bearer YOUR-TOKEN-HERE" \
  https://myapp.forteapps.net/
```

#### Issue: OIDC Login Loop

**Check OIDC configuration**:
```bash
# Verify auth-oidc secret exists
kubectl get secret auth-oidc -n myapp

# Check sidecar logs
kubectl logs -n myapp <pod-name> -c authn

# Common issues:
# - Wrong authority URL
# - Wrong client ID
# - Missing client-secret in auth-oidc Secret
# - Redirect URI not configured in identity provider
```

**Verify redirect URI** in your identity provider matches:
```
https://<your-app-domain>/auth/callback
```

#### Issue: Auth Sidecar Not Injected

**Check pod annotations**:
```bash
kubectl get pod -n myapp <pod-name> -o yaml | grep policies.forteapps.io

# Should show:
# policies.forteapps.io/auth: "true"
```

**Check Kyverno policy**:
```bash
kubectl get clusterpolicy inject-auth-sidecar
kubectl describe clusterpolicy inject-auth-sidecar
```

**Check Kyverno logs**:
```bash
kubectl logs -n kyverno deployment/kyverno | grep inject-auth
```

#### Issue: Auth Sidecar Crashes

**Check sidecar logs**:
```bash
kubectl logs -n myapp <pod-name> -c authn
```

**Common causes**:
- Missing secret (auth-tokens or auth-oidc)
- Invalid OIDC configuration
- Can't reach OIDC authority URL
- Network policy blocking outbound OIDC requests

---

### Authentication Best Practices

✅ **DO**:
- Use OIDC for user-facing applications
- Use token auth for service-to-service communication
- Rotate tokens and secrets regularly
- Use strong random tokens (32+ bytes)
- Store client secrets in SealedSecrets
- Test authentication before deploying to production
- Document which tokens/users have access

❌ **DON'T**:
- Share tokens between environments
- Commit tokens to application code
- Use predictable tokens
- Reuse tokens across multiple applications
- Disable authentication on sensitive APIs
- Log tokens or secrets

---

## Troubleshooting

### Application Not Deploying

#### Problem: Application stuck in "Syncing" state

**Check ArgoCD status:**
```bash
kubectl get application myapp -n argocd -o yaml
```

Look for errors in `status.conditions`.

**Common causes:**
- ❌ Image doesn't exist or is not accessible
- ❌ Invalid YAML syntax
- ❌ Resource quota exceeded
- ❌ Namespace conflicts
- ❌ Invalid Helm values

**Solutions:**
```bash
# Check image exists
docker pull ghcr.io/fortedigital/myapp:v1.0.0

# Validate YAML syntax
kubectl apply --dry-run=client -f apps/myapp.yaml

# Check ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller | grep myapp
```

#### Problem: Pods crashing (CrashLoopBackOff)

**Check pod logs:**
```bash
kubectl get pods -n myapp
kubectl logs -n myapp <pod-name>
kubectl describe pod -n myapp <pod-name>
```

**Common causes:**
- ❌ Application error (check logs)
- ❌ Missing environment variables
- ❌ Incorrect port configuration
- ❌ Missing secrets
- ❌ Insufficient resources

**Solutions:**
```bash
# Check environment variables
kubectl exec -n myapp <pod-name> -- env

# Check if secrets exist
kubectl get secrets -n myapp

# Increase resources in helm-values
vim ~/dev/k8s/helm-prod-values/myapp/values.yaml
```

#### Problem: Application not accessible via domain

**Check ingress:**
```bash
kubectl get ingressroute -n myapp
kubectl describe ingressroute myapp -n myapp
```

**Common causes:**
- ❌ DNS not configured
- ❌ TLS certificate not issued
- ❌ Incorrect domain in values.yaml
- ❌ Traefik not routing correctly

**Solutions:**
```bash
# Check certificate
kubectl get certificate -n myapp

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Verify domain configuration
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep host

# Test with port-forward
kubectl port-forward -n myapp service/myapp 8080:3000
curl http://localhost:8080
```

### Secret Issues

#### Problem: Secret not found

**Check if SealedSecret exists:**
```bash
kubectl get sealedsecret -n myapp
kubectl get secret -n myapp
```

**Solutions:**
```bash
# Check if secret is in Git
ls -l secrets/myapp-credentials-sealed.yaml

# Re-apply sealed secret
kubectl apply -f secrets/myapp-credentials-sealed.yaml

# Check sealed-secrets-controller logs
kubectl logs -n kube-system deployment/sealed-secrets-controller
```

#### Problem: Secret exists but pods can't access it

**Check pod events:**
```bash
kubectl describe pod -n myapp <pod-name>
```

Look for: `Error: secret "myapp-credentials" not found`

**Solutions:**
```bash
# Verify secret name in values.yaml matches actual secret
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep envSecretName
kubectl get secrets -n myapp

# Restart pods
kubectl rollout restart deployment myapp -n myapp
```

### Sync Failures

#### Problem: ArgoCD shows "Out of Sync"

**Manual sync:**
```bash
# Using kubectl
kubectl patch application myapp -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"syncStrategy":{"hook":{}}}}}'

# Or via ArgoCD UI
# Click "Sync" button in UI
```

**Check what's different:**
```bash
kubectl get application myapp -n argocd -o yaml
```

Look at `status.sync.comparedTo` vs desired state.

#### Problem: Sync succeeds but application is "Degraded"

**Check resource health:**
```bash
kubectl get application myapp -n argocd -o jsonpath='{.status.resources[*].health}'
```

**Common causes:**
- ❌ Pods not ready
- ❌ Deployments not at desired replica count
- ❌ Jobs failed

**Solutions:**
```bash
# Check all resources in namespace
kubectl get all -n myapp

# Check pod events
kubectl get events -n myapp --sort-by='.lastTimestamp'
```

### Getting Help

If you're stuck:

1. **Check Slack notifications** - Error details are often in sync failure messages
2. **Check ArgoCD UI** - Visual representation of what's wrong
3. **Ask platform team** - They have full cluster access and can debug further
4. **Check documentation** - [Operations Runbook](OPERATIONS-RUNBOOK.md) has more troubleshooting

---

## Best Practices

### Development Workflow

✅ **DO**:
- Develop and test locally with Docker Compose
- Use semantic versioning for releases
- Write descriptive commit messages
- Test changes in a separate namespace first (if possible)
- Monitor Slack for deployment notifications
- Document environment variables and configuration

❌ **DON'T**:
- Push directly to production without testing
- Use `latest` tag for Docker images
- Bypass CI/CD for "quick fixes"
- Hard-code configuration values
- Ignore deployment failures

### Configuration Management

✅ **DO**:
- Keep configuration in `helm-values` repository
- Use environment variables for config
- Document what each value does
- Use reasonable resource limits
- Enable ingress and TLS for public services

❌ **DON'T**:
- Hard-code config in application code
- Over-allocate resources (wastes money)
- Under-allocate resources (causes crashes)
- Use HTTP for production services

### Secret Management

✅ **DO**:
- Use kubeseal for all secrets
- Store plain secrets in password manager
- Rotate secrets regularly
- Use different secrets per environment
- Document what each secret contains

❌ **DON'T**:
- Commit plain secrets
- Share secrets in Slack/email
- Reuse secrets across apps
- Log secrets in application code

### Git Workflow

✅ **DO**:
- Use feature branches for changes
- Write clear commit messages
- Use pull requests for review
- Keep commits atomic and focused
- Tag releases in application repos

❌ **DON'T**:
- Push directly to `main` without review (for config repos)
- Make multiple unrelated changes in one commit
- Use vague commit messages ("fix", "update")
- Force-push to main branches

---

## Quick Reference

### Common Commands

```bash
# Check application status
kubectl get application myapp -n argocd

# View application details
kubectl describe application myapp -n argocd

# Check pods
kubectl get pods -n myapp

# View pod logs
kubectl logs -n myapp <pod-name>

# Restart deployment
kubectl rollout restart deployment myapp -n myapp

# Port-forward to service
kubectl port-forward -n myapp service/myapp 8080:3000

# Create secret
kubectl create secret generic myapp-credentials \
  --from-literal=KEY=value \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

# Seal secret
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml
```

### Repository Locations

```bash
# Config repository
cd ~/dev/k8s/launchpad

# Helm values repository
cd ~/dev/k8s/helm-prod-values

# Helm charts repository
cd ~/dev/k8s/forte-helm
```

### File Paths

```bash
# New application manifest
~/dev/k8s/launchpad/apps/myapp.yaml

# Application values
~/dev/k8s/helm-prod-values/myapp/values.yaml

# Sealed secrets
~/dev/k8s/launchpad/secrets/myapp-credentials-sealed.yaml

# Plain secrets (local only)
~/dev/k8s/launchpad/private/myapp-credentials.yaml
```

---

## Next Steps

Now that you understand the basics:

1. ✅ Deploy your first application (follow steps above)
2. 📖 Read the [Operations Runbook](OPERATIONS-RUNBOOK.md) for common tasks
3. 📖 Review [Technical Reference](REFERENCE.md) for detailed component docs
4. 📖 Understand [GitOps Architecture](GITOPS-ARCHITECTURE.md) for the big picture
5. 🚀 Start contributing!

---

**Questions?**
- Slack: #platform-support
- Docs: [Full documentation index](README.md)
- Help: Contact platform team

**Last Updated**: 2026-03-16