Forte/launchpad

Fork 0

Files

Danijel Simeunovic 7aff19ccab fix

2026-03-16 11:45:43 +01:00

38 KiB

Raw Blame History

Developer Onboarding Guide

Getting Started
Prerequisites
Local Development Setup
Understanding the Workflow
Deploying Your First Application
Updating an Existing Application
Working with Secrets
Enabling Authentication for Applications
Troubleshooting
Best Practices

Getting Started

Welcome! This guide will help you understand how to develop and deploy applications on our Kubernetes cluster using GitOps principles powered by ArgoCD.

What You'll Learn

How our GitOps architecture works
How to deploy a new application
How to update existing applications
How to manage secrets securely
Common troubleshooting techniques

Who This Guide Is For

Developers deploying new applications
Developers maintaining existing applications
Team members who need to understand the deployment process

Prerequisites

Required Knowledge

✅ Basic Git workflow (clone, commit, push, pull)
✅ Docker basics (Dockerfile, building images)
✅ YAML syntax
✅ Basic understanding of Kubernetes concepts (pods, deployments, services)
⚠️ Helm knowledge (helpful but not required - templates are provided)

Required Tools

Most developers do NOT need kubectl access to the cluster. You'll primarily work with Git repositories.

If you do need cluster access, install:

kubectl - Kubernetes CLI

# macOS
brew install kubectl

# Windows
choco install kubernetes-cli

# Linux
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

kubeseal - For sealing secrets

# macOS
brew install kubeseal

# Windows
choco install kubeseal

# Linux
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
tar -xvzf kubeseal-0.24.0-linux-amd64.tar.gz
sudo mv kubeseal /usr/local/bin/

Git - Version control

git --version  # Should already be installed

Docker - For local development

# macOS/Windows: Install Docker Desktop
# Linux: Install Docker Engine
docker --version

Repository Access

You'll need read/write access to these repositories:

sturdy-adventure (Config repo)

git clone https://github.com/fortedigital/sturdy-adventure.git
cd sturdy-adventure

helm-values (Values repo)

git clone git@github.com:fortedigital/helm-values.git
cd helm-values

forte-helm (Chart repo - read-only for most developers)

git clone https://github.com/snothub/forte-helm.git
cd forte-helm

Cluster Access (If Needed)

If you need kubectl access, ask the platform team for:

Kubeconfig file
Cluster context setup instructions

Save to ~/.kube/config and verify:

kubectl cluster-info
kubectl get nodes

Local Development Setup

1. Clone the Repositories

Set up a consistent folder structure:

mkdir -p ~/dev/k8s
cd ~/dev/k8s

# Clone repositories
git clone https://github.com/fortedigital/sturdy-adventure.git launchpad
git clone git@github.com:fortedigital/helm-values.git helm-prod-values
git clone https://github.com/snothub/forte-helm.git forte-helm

# Your folder structure:
# ~/dev/k8s/
# ├── launchpad/           (Config repo)
# ├── helm-prod-values/    (Values repo)
# └── forte-helm/          (Chart repo)

2. Local Development Environment

Most applications use Docker Compose for local development:

# In your application repository
docker-compose up

# Or for frontend applications
npm install
npm run dev

You DO NOT run applications locally on Kubernetes. Use Docker Compose or native tooling (npm, python, etc.).

3. Understanding the Deployment Flow

┌─────────────────────────────────────────────────────────────────┐
│  Step 1: Develop Locally                                        │
│  - Write code in your application repository                    │
│  - Test with Docker Compose or npm/python/etc.                  │
│  - Build Docker image                                            │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 2: CI/CD Pipeline (Automated)                             │
│  - GitHub Actions builds image                                  │
│  - Pushes to container registry (GHCR, Docker Hub)              │
│  - Tags with version (e.g., v2.0.4)                             │
│  - Updates helm-values repository with new tag                  │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 3: GitOps Sync (Automated)                                │
│  - ArgoCD detects change in helm-values                         │
│  - Pulls updated configuration                                  │
│  - Syncs to Kubernetes cluster                                  │
│  - Sends Slack notification on success/failure                  │
└─────────────────────────────────────────────────────────────────┘

Key Insight: You don't deploy directly. You push code, CI/CD builds it, and ArgoCD deploys it.

Understanding the Workflow

Three-Repository Pattern

Our setup uses three repositories:

Repository	Purpose	You Edit
forte-helm	Helm chart templates (generic, reusable)	❌ Rarely
helm-values	Application configuration (image tag, env vars)	✅ Sometimes
sturdy-adventure	ArgoCD Applications (what gets deployed)	✅ Yes (for new apps)

Example: Deploying "myapp"

Repository: `forte-helm` (Chart Templates)

# forteapp/templates/deployment.yaml
# Generic template used by ALL apps
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
spec:
  containers:
  - name: app
    image: "{{ .Values.app.image.repository }}:{{ .Values.app.image.tag }}"
    env:
    - name: PORT
      value: {{ .Values.app.port }}

Repository: `helm-values` (Your App Config)

# myapp/values.yaml
# Your app's specific configuration
app:
  image:
    repository: ghcr.io/fortedigital/myapp
    tag: v1.0.0                    # CI/CD updates this
  port: 3000
  extraEnv:
  - name: API_URL
    value: https://api.example.com

Repository: `sturdy-adventure` (ArgoCD Application)

# apps/myapp.yaml
# Tells ArgoCD to deploy your app
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  sources:
  - repoURL: https://github.com/snothub/forte-helm
    path: forteapp
    helm:
      valueFiles:
      - $values/myapp/values.yaml

  - repoURL: git@github.com:fortedigital/helm-values.git
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: myapp

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Deploying Your First Application

Scenario: You've Built a New Application

Let's deploy a new Node.js application called "hello-world".

Step 1: Prepare Your Application Repository

Ensure your app repository has:

Dockerfile

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]

GitHub Actions Workflow (.github/workflows/deploy.yml)

name: Build and Deploy

on:
  push:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set version
        id: version
        run: echo "VERSION=v$(date +%Y%m%d-%H%M%S)" >> $GITHUB_OUTPUT

      - name: Build and push Docker image
        run: |
          echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
          docker build -t ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }} .
          docker push ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }}

      - name: Update helm-values
        run: |
          git clone git@github.com:fortedigital/helm-values.git
          cd helm-values
          mkdir -p hello-world
          cat > hello-world/values.yaml <<EOF
          app:
            image:
              repository: ghcr.io/fortedigital/hello-world
              tag: ${{ steps.version.outputs.VERSION }}
          EOF
          git add hello-world/values.yaml
          git commit -m "Update hello-world to ${{ steps.version.outputs.VERSION }}"
          git push

Step 2: Create Helm Values

Create a folder in helm-values repository:

cd ~/dev/k8s/helm-prod-values
mkdir -p hello-world

Create hello-world/values.yaml:

app:
  image:
    repository: ghcr.io/fortedigital/hello-world
    tag: v1.0.0                    # Will be updated by CI/CD
    containerPort: 3000

  replicaCount: 1

  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi

  extraEnv:
  - name: PORT
    value: "3000"
  - name: NODE_ENV
    value: "production"

  envSecretName: ""                # Optional: reference to secrets

service:
  port: 3000

ingress:
  enabled: true
  host: hello-world.forteapps.net  # Your subdomain

db:
  enabled: false                   # Set to true if you need PostgreSQL

Commit and push:

git add hello-world/values.yaml
git commit -m "Add hello-world application values"
git push

Step 3: Create ArgoCD Application Manifest

In the sturdy-adventure repository, create apps/hello-world.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello-world
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: ""
    notifications.argoproj.io/subscribe.on-sync-failed.slack: ""
    notifications.argoproj.io/subscribe.on-degraded.slack: ""
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/part-of: apps
    app.kubernetes.io/managed-by: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io

spec:
  project: default

  sources:
  # Source 1: Helm chart templates
  - repoURL: https://github.com/snothub/forte-helm
    path: forteapp
    targetRevision: HEAD
    helm:
      valueFiles:
      - $values/hello-world/values.yaml

  # Source 2: Helm values
  - repoURL: git@github.com:fortedigital/helm-values.git
    targetRevision: HEAD
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: hello-world

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false

    syncOptions:
    - CreateNamespace=true
    - Validate=true
    - ServerSideApply=true

    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas

Commit and push:

cd ~/dev/k8s/launchpad
git add apps/hello-world.yaml
git commit -m "Add hello-world application"
git push

Step 4: Verify Deployment

ArgoCD will automatically detect the new application within 60 seconds.

Option 1: Check Slack

Watch for sync notifications in your Slack channel
✅ "Application hello-world sync succeeded"

Option 2: Check ArgoCD UI (if you have access)

# Port forward to ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Open browser: https://localhost:8080
# Look for "hello-world" application

Option 3: Check with kubectl (if you have access)

# List ArgoCD applications
kubectl get applications -n argocd

# Check application status
kubectl get application hello-world -n argocd

# Verify pods are running
kubectl get pods -n hello-world

Step 5: Access Your Application

Once deployed, access via the configured domain:

# Check if ingress is created
kubectl get ingressroute -n hello-world

# Access application
curl https://hello-world.forteapps.net

⚠️ Note: DNS must be manually configured for new subdomains. Contact the platform team to add DNS records.

Updating an Existing Application

Scenario: Deploying a Code Change

You've made changes to your application code and want to deploy them.

Method 1: Automatic (Recommended)

Just push to main branch - CI/CD handles everything:

# In your application repository
git add .
git commit -m "Fix bug in user login"
git push origin main

What Happens Next:

✅ GitHub Actions triggers
✅ Builds new Docker image
✅ Tags with new version (e.g., v20260316-143022)
✅ Pushes to container registry
✅ Updates helm-values/myapp/values.yaml with new tag
✅ ArgoCD detects change
✅ Syncs new version to cluster
✅ Sends Slack notification

Timeline: ~5-10 minutes from push to deployment

Method 2: Manual Image Tag Update

If CI/CD is not set up, manually update the image tag:

cd ~/dev/k8s/helm-prod-values

# Edit your app's values.yaml
vim myapp/values.yaml

# Change:
app:
  image:
    tag: v1.0.0  # Old version
# To:
app:
  image:
    tag: v1.0.1  # New version

# Commit and push
git add myapp/values.yaml
git commit -m "Update myapp to v1.0.1"
git push

ArgoCD will sync within 60 seconds.

Method 3: Configuration Changes

To update environment variables, resources, or other config:

cd ~/dev/k8s/helm-prod-values
vim myapp/values.yaml

Example changes:

app:
  # Increase resources
  resources:
    requests:
      cpu: 200m      # Was 100m
      memory: 256Mi  # Was 128Mi

  # Add new environment variable
  extraEnv:
  - name: API_URL
    value: https://api.example.com
  - name: DEBUG          # NEW
    value: "true"        # NEW

  # Enable HPA
  hpa:
    enabled: true        # Was false
    minReplicas: 2
    maxReplicas: 10

Commit and push:

git add myapp/values.yaml
git commit -m "Increase myapp resources and enable HPA"
git push

Method 4: Application Manifest Changes

To change ArgoCD sync behavior, namespace, or other meta-config:

cd ~/dev/k8s/launchpad
vim apps/myapp.yaml

Example changes:

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: false    # Disable self-healing temporarily

Commit and push:

git add apps/myapp.yaml
git commit -m "Disable self-healing for myapp"
git push

Working with Secrets

Understanding Secret Management

NEVER commit plain secrets to Git. We use Sealed Secrets to encrypt secrets before committing.

Creating a New Secret

Step 1: Create Plain Secret Locally

cd ~/dev/k8s/launchpad

# Create secret in private/ folder (Git-ignored)
kubectl create secret generic myapp-credentials \
  --from-literal=API_KEY=your-secret-key-here \
  --from-literal=DB_PASSWORD=super-secret-password \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

DO NOT commit this file! It's in private/ which is Git-ignored.

Step 2: Seal the Secret

Get the public certificate (one-time setup):

# Fetch public cert from cluster
kubeseal --fetch-cert \
  --controller-name=sealed-secrets-controller \
  --controller-namespace=kube-system \
  > pub-cert.pem

Seal your secret:

kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml

Step 3: Commit Sealed Secret

git add secrets/myapp-credentials-sealed.yaml
git commit -m "Add myapp credentials (sealed)"
git push

Step 4: Reference Secret in Application

Update your helm-values/myapp/values.yaml:

app:
  envSecretName: "myapp-credentials"  # References the SealedSecret

Commit and push:

cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Reference myapp credentials"
git push

Updating a Secret

To update an existing secret:

# 1. Create new version of secret
kubectl create secret generic myapp-credentials \
  --from-literal=API_KEY=new-key-here \
  --from-literal=DB_PASSWORD=new-password \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

# 2. Seal it
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml

# 3. Commit sealed version
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Update myapp credentials"
git push

# 4. Restart pods to pick up new secret
kubectl rollout restart deployment myapp -n myapp

Secret Best Practices

✅ DO:

Store secrets in private/ folder locally
Always seal secrets before committing
Delete plain secrets after sealing
Use meaningful secret names
Document what each secret contains

❌ DON'T:

Commit plain secrets to Git
Share secrets via Slack/email
Hard-code secrets in code
Use the same secret across multiple environments
Store secrets in Docker images

Where Secrets Are Stored

┌─────────────────────────────────────────────────────────────┐
│  Location                │  Content           │  Committed?│
├──────────────────────────┼────────────────────┼────────────┤
│  private/                │  Plain secrets     │  ❌ NO      │
│  secrets/                │  Sealed secrets    │  ✅ YES     │
│  Kubernetes cluster      │  Unsealed secrets  │  N/A       │
└─────────────────────────────────────────────────────────────┘

Sealed Secrets Controller in the cluster decrypts sealed secrets automatically.

Enabling Authentication for Applications

The cluster supports automatic authentication sidecar injection for applications via Kyverno policies. This allows you to add authentication to your applications without modifying application code.

How It Works

When you enable authentication in your Helm values, the Kyverno policy automatically:

✅ Injects an authentication sidecar container into your pod
✅ Routes all incoming traffic through the auth sidecar (port 8080)
✅ Validates credentials before forwarding requests to your application
✅ Creates necessary secrets (if they don't exist)
✅ Adds a NetworkPolicy to restrict ingress

Architecture:

Internet → Traefik → Service:8080 → Auth Sidecar:8080 → localhost → Your App:3000
                                         │
                                         ├─ Validates credentials
                                         └─ Forwards if valid

Authentication Modes

Two authentication modes are supported:

Token-based: Static tokens (simple, good for service-to-service or internal apps)
OIDC: OpenID Connect (full SSO, good for user-facing apps)

Token-Based Authentication

Step 1: Configure Helm Values

# In helm-values/myapp/values.yaml
auth:
  enabled: true
  type: token                    # Token mode (default)
  tokens:
  - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823
  - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7

Step 2: Generate Token (if needed)

# Generate a secure random token
openssl rand -hex 32

# Or using Python
python3 -c "import secrets; print(secrets.token_hex(32))"

# Example output:
# d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823

Step 3: Deploy Application

Commit and push your changes:

cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Enable token auth for myapp"
git push

ArgoCD will sync, and the Kyverno policy will:

Inject the auth sidecar container
Create an auth-tokens Secret with your tokens
Configure the sidecar to validate against these tokens

Step 4: Access Application

Use your token in the Authorization header:

# Access application with token
curl -H "Authorization: Bearer d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823" \
  https://myapp.forteapps.net/api/data

# Without token (will be rejected)
curl https://myapp.forteapps.net/api/data
# Response: 401 Unauthorized

Advanced: Custom Secret Name

To use a different secret for tokens:

# In Helm values
auth:
  enabled: true
  type: token
  tokens: []                     # Empty - using external secret

# Tokens will be read from custom secret

Then reference it via annotation (configured by Helm chart automatically):

# Helm chart sets this annotation:
policies.forteapps.io/auth-token-secret-name: "myapp-auth-tokens"

Create the secret manually:

kubectl create secret generic myapp-auth-tokens \
  --from-file=tokens=tokens.txt \
  --namespace=myapp

OIDC Authentication

OIDC mode integrates with identity providers like Keycloak, Okta, Auth0, Azure AD, etc.

Step 1: Configure Identity Provider

In your identity provider (e.g., Keycloak):

Create a new client (e.g., myapp)
Set redirect URI: https://myapp.forteapps.net/auth/callback
Note the Client ID and Client Secret
Note the Authority URL (e.g., https://keycloak.forteapps.net/realms/master)

Step 2: Create OIDC Secret

# Create plain secret
kubectl create secret generic auth-oidc \
  --from-literal=client-secret=your-oidc-client-secret \
  --from-literal=cookie-secret=$(openssl rand -hex 32) \
  --namespace=myapp \
  --dry-run=client -o yaml > private/myapp-auth-oidc.yaml

# Seal it
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  --namespace=myapp \
  < private/myapp-auth-oidc.yaml \
  > secrets/myapp-auth-oidc-sealed.yaml

# Commit sealed secret
cd ~/dev/k8s/launchpad
git add secrets/myapp-auth-oidc-sealed.yaml
git commit -m "Add OIDC secrets for myapp"
git push

# Clean up
rm private/myapp-auth-oidc.yaml

Step 3: Configure Helm Values

# In helm-values/myapp/values.yaml
auth:
  enabled: true
  type: oidc                     # OIDC mode
  oidc:
    authority: https://keycloak.forteapps.net/realms/master
    clientId: myapp
    scopes: "openid,profile,email"
    callbackPath: /auth/callback

Step 4: Deploy Application

cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Enable OIDC auth for myapp"
git push

Step 5: Access Application

When users access https://myapp.forteapps.net:

They're redirected to the identity provider login page
After successful login, redirected back to /auth/callback
Session cookie is set
Subsequent requests are authenticated via cookie

User flow:

User → https://myapp.forteapps.net
  ↓
Redirect → https://keycloak.forteapps.net/login
  ↓
Login successful → Redirect with auth code
  ↓
https://myapp.forteapps.net/auth/callback?code=xyz
  ↓
Auth sidecar exchanges code for tokens
  ↓
Sets session cookie
  ↓
Redirects to application → https://myapp.forteapps.net
  ↓
User sees application (authenticated)

Authentication Configuration Reference

Helm Values Schema

auth:
  enabled: false                 # Enable/disable authentication
  type: token                    # "token" or "oidc"

  # Token mode configuration
  tokens: []                     # List of valid bearer tokens
  # - token1
  # - token2

  # OIDC mode configuration
  oidc:
    authority: ""                # OIDC provider URL (required for OIDC)
    clientId: ""                 # OIDC client ID (required for OIDC)
    scopes: "openid,profile,email"  # OIDC scopes (optional)
    callbackPath: /auth/callback    # OAuth callback path (optional)

Annotations Set by Helm Chart

When auth.enabled: true, the Helm chart sets these pod annotations:

Token mode:

policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "token"
policies.forteapps.io/auth-token-secret-name: "auth-tokens"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"

OIDC mode:

policies.forteapps.io/auth: "true"
policies.forteapps.io/auth-type: "oidc"
policies.forteapps.io/auth-oidc-authority: "https://keycloak.forteapps.net/realms/master"
policies.forteapps.io/auth-oidc-client-id: "myapp"
policies.forteapps.io/auth-oidc-scopes: "openid,profile,email"
policies.forteapps.io/auth-oidc-callback-path: "/auth/callback"
policies.forteapps.io/auth-upstream-url: "http://localhost:3000"

Sidecar Configuration

The auth sidecar container:

Image: ghcr.io/snothub/stunning-memory:latest
Port: 8080
Resources: 10m CPU / 32Mi memory (requests), 50m CPU / 64Mi memory (limits)
Health checks: /healthz endpoint
Security: Read-only root filesystem, no privilege escalation

Advanced: Custom Sidecar Image

To use a different auth sidecar image:

# These annotations can be set in the Helm chart template if needed
policies.forteapps.io/auth-image: "your-registry/your-auth-proxy"
policies.forteapps.io/auth-image-version: "v1.2.3"

Authentication Examples

Example 1: Internal API with Token Auth

# helm-values/internal-api/values.yaml
app:
  image:
    repository: ghcr.io/company/internal-api
    tag: v1.0.0

auth:
  enabled: true
  type: token
  tokens:
  - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823  # Service A
  - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7  # Service B

ingress:
  enabled: true
  host: internal-api.forteapps.net

Usage:

# Service A calls API
curl -H "Authorization: Bearer d4f88f..." \
  https://internal-api.forteapps.net/api/endpoint

Example 2: User-Facing App with OIDC

# helm-values/web-app/values.yaml
app:
  image:
    repository: ghcr.io/company/web-app
    tag: v2.1.0

auth:
  enabled: true
  type: oidc
  oidc:
    authority: https://auth.company.com/realms/employees
    clientId: web-app-prod
    scopes: "openid,profile,email,groups"
    callbackPath: /auth/callback

ingress:
  enabled: true
  host: web-app.forteapps.net

With sealed OIDC secret:

# Create and seal secret
kubectl create secret generic auth-oidc \
  --from-literal=client-secret=super-secret-value \
  --from-literal=cookie-secret=$(openssl rand -hex 32) \
  --namespace=web-app \
  --dry-run=client -o yaml | \
  kubeseal --format=yaml --cert=pub-cert.pem --namespace=web-app \
  > secrets/web-app-auth-oidc-sealed.yaml

Example 3: Disabling Authentication

# helm-values/public-api/values.yaml
auth:
  enabled: false                 # No authentication

ingress:
  enabled: true
  host: public-api.forteapps.net

Troubleshooting Authentication

Issue: 401 Unauthorized (Token Mode)

Check token validity:

# Get auth-tokens secret
kubectl get secret auth-tokens -n myapp -o yaml

# Decode tokens
kubectl get secret auth-tokens -n myapp \
  -o jsonpath='{.data.tokens}' | base64 -d

# Verify your token is in the list

Test with different token:

curl -v -H "Authorization: Bearer YOUR-TOKEN-HERE" \
  https://myapp.forteapps.net/

Check OIDC configuration:

# Verify auth-oidc secret exists
kubectl get secret auth-oidc -n myapp

# Check sidecar logs
kubectl logs -n myapp <pod-name> -c authn

# Common issues:
# - Wrong authority URL
# - Wrong client ID
# - Missing client-secret in auth-oidc Secret
# - Redirect URI not configured in identity provider

Verify redirect URI in your identity provider matches:

https://<your-app-domain>/auth/callback

Issue: Auth Sidecar Not Injected

Check pod annotations:

kubectl get pod -n myapp <pod-name> -o yaml | grep policies.forteapps.io

# Should show:
# policies.forteapps.io/auth: "true"

Check Kyverno policy:

kubectl get clusterpolicy inject-auth-sidecar
kubectl describe clusterpolicy inject-auth-sidecar

Check Kyverno logs:

kubectl logs -n kyverno deployment/kyverno | grep inject-auth

Issue: Auth Sidecar Crashes

Check sidecar logs:

kubectl logs -n myapp <pod-name> -c authn

Common causes:

Missing secret (auth-tokens or auth-oidc)
Invalid OIDC configuration
Can't reach OIDC authority URL
Network policy blocking outbound OIDC requests

Authentication Best Practices

✅ DO:

Use OIDC for user-facing applications
Use token auth for service-to-service communication
Rotate tokens and secrets regularly
Use strong random tokens (32+ bytes)
Store client secrets in SealedSecrets
Test authentication before deploying to production
Document which tokens/users have access

❌ DON'T:

Share tokens between environments
Commit tokens to application code
Use predictable tokens
Reuse tokens across multiple applications
Disable authentication on sensitive APIs
Log tokens or secrets

Troubleshooting

Application Not Deploying

Problem: Application stuck in "Syncing" state

Check ArgoCD status:

kubectl get application myapp -n argocd -o yaml

Look for errors in status.conditions.

Common causes:

❌ Image doesn't exist or is not accessible
❌ Invalid YAML syntax
❌ Resource quota exceeded
❌ Namespace conflicts
❌ Invalid Helm values

Solutions:

# Check image exists
docker pull ghcr.io/fortedigital/myapp:v1.0.0

# Validate YAML syntax
kubectl apply --dry-run=client -f apps/myapp.yaml

# Check ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller | grep myapp

Problem: Pods crashing (CrashLoopBackOff)

Check pod logs:

kubectl get pods -n myapp
kubectl logs -n myapp <pod-name>
kubectl describe pod -n myapp <pod-name>

Common causes:

❌ Application error (check logs)
❌ Missing environment variables
❌ Incorrect port configuration
❌ Missing secrets
❌ Insufficient resources

Solutions:

# Check environment variables
kubectl exec -n myapp <pod-name> -- env

# Check if secrets exist
kubectl get secrets -n myapp

# Increase resources in helm-values
vim ~/dev/k8s/helm-prod-values/myapp/values.yaml

Problem: Application not accessible via domain

Check ingress:

kubectl get ingressroute -n myapp
kubectl describe ingressroute myapp -n myapp

Common causes:

❌ DNS not configured
❌ TLS certificate not issued
❌ Incorrect domain in values.yaml
❌ Traefik not routing correctly

Solutions:

# Check certificate
kubectl get certificate -n myapp

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Verify domain configuration
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep host

# Test with port-forward
kubectl port-forward -n myapp service/myapp 8080:3000
curl http://localhost:8080

Secret Issues

Problem: Secret not found

Check if SealedSecret exists:

kubectl get sealedsecret -n myapp
kubectl get secret -n myapp

Solutions:

# Check if secret is in Git
ls -l secrets/myapp-credentials-sealed.yaml

# Re-apply sealed secret
kubectl apply -f secrets/myapp-credentials-sealed.yaml

# Check sealed-secrets-controller logs
kubectl logs -n kube-system deployment/sealed-secrets-controller

Problem: Secret exists but pods can't access it

Check pod events:

kubectl describe pod -n myapp <pod-name>

Look for: Error: secret "myapp-credentials" not found

Solutions:

# Verify secret name in values.yaml matches actual secret
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep envSecretName
kubectl get secrets -n myapp

# Restart pods
kubectl rollout restart deployment myapp -n myapp

Sync Failures

Problem: ArgoCD shows "Out of Sync"

Manual sync:

# Using kubectl
kubectl patch application myapp -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"syncStrategy":{"hook":{}}}}}'

# Or via ArgoCD UI
# Click "Sync" button in UI

Check what's different:

kubectl get application myapp -n argocd -o yaml

Look at status.sync.comparedTo vs desired state.

Problem: Sync succeeds but application is "Degraded"

Check resource health:

kubectl get application myapp -n argocd -o jsonpath='{.status.resources[*].health}'

Common causes:

❌ Pods not ready
❌ Deployments not at desired replica count
❌ Jobs failed

Solutions:

# Check all resources in namespace
kubectl get all -n myapp

# Check pod events
kubectl get events -n myapp --sort-by='.lastTimestamp'

Getting Help

If you're stuck:

Check Slack notifications - Error details are often in sync failure messages
Check ArgoCD UI - Visual representation of what's wrong
Ask platform team - They have full cluster access and can debug further
Check documentation - Operations Runbook has more troubleshooting

Best Practices

Development Workflow

✅ DO:

Develop and test locally with Docker Compose
Use semantic versioning for releases
Write descriptive commit messages
Test changes in a separate namespace first (if possible)
Monitor Slack for deployment notifications
Document environment variables and configuration

❌ DON'T:

Push directly to production without testing
Use latest tag for Docker images
Bypass CI/CD for "quick fixes"
Hard-code configuration values
Ignore deployment failures

Configuration Management

✅ DO:

Keep configuration in helm-values repository
Use environment variables for config
Document what each value does
Use reasonable resource limits
Enable ingress and TLS for public services

❌ DON'T:

Hard-code config in application code
Over-allocate resources (wastes money)
Under-allocate resources (causes crashes)
Use HTTP for production services

Secret Management

✅ DO:

Use kubeseal for all secrets
Store plain secrets in password manager
Rotate secrets regularly
Use different secrets per environment
Document what each secret contains

❌ DON'T:

Commit plain secrets
Share secrets in Slack/email
Reuse secrets across apps
Log secrets in application code

Git Workflow

✅ DO:

Use feature branches for changes
Write clear commit messages
Use pull requests for review
Keep commits atomic and focused
Tag releases in application repos

❌ DON'T:

Push directly to main without review (for config repos)
Make multiple unrelated changes in one commit
Use vague commit messages ("fix", "update")
Force-push to main branches

Quick Reference

Common Commands

# Check application status
kubectl get application myapp -n argocd

# View application details
kubectl describe application myapp -n argocd

# Check pods
kubectl get pods -n myapp

# View pod logs
kubectl logs -n myapp <pod-name>

# Restart deployment
kubectl rollout restart deployment myapp -n myapp

# Port-forward to service
kubectl port-forward -n myapp service/myapp 8080:3000

# Create secret
kubectl create secret generic myapp-credentials \
  --from-literal=KEY=value \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

# Seal secret
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml

Repository Locations

# Config repository
cd ~/dev/k8s/launchpad

# Helm values repository
cd ~/dev/k8s/helm-prod-values

# Helm charts repository
cd ~/dev/k8s/forte-helm

File Paths

# New application manifest
~/dev/k8s/launchpad/apps/myapp.yaml

# Application values
~/dev/k8s/helm-prod-values/myapp/values.yaml

# Sealed secrets
~/dev/k8s/launchpad/secrets/myapp-credentials-sealed.yaml

# Plain secrets (local only)
~/dev/k8s/launchpad/private/myapp-credentials.yaml

Next Steps

Now that you understand the basics:

✅ Deploy your first application (follow steps above)
📖 Read the Operations Runbook for common tasks
📖 Review Technical Reference for detailed component docs
📖 Understand GitOps Architecture for the big picture
🚀 Start contributing!

Questions?

Slack: #platform-support
Docs: Full documentation index
Help: Contact platform team

Last Updated: 2026-03-16

38 KiB Raw Blame History

Developer Onboarding Guide

Table of Contents

Getting Started

What You'll Learn

Who This Guide Is For

Prerequisites

Required Knowledge

Required Tools

Repository Access

Cluster Access (If Needed)

Local Development Setup

1. Clone the Repositories

2. Local Development Environment

3. Understanding the Deployment Flow

Understanding the Workflow

Three-Repository Pattern

Example: Deploying "myapp"

Repository: forte-helm (Chart Templates)

Repository: helm-values (Your App Config)

Repository: sturdy-adventure (ArgoCD Application)

Deploying Your First Application

Scenario: You've Built a New Application

Step 1: Prepare Your Application Repository

Step 2: Create Helm Values

Step 3: Create ArgoCD Application Manifest

Step 4: Verify Deployment

Step 5: Access Your Application

Updating an Existing Application

Scenario: Deploying a Code Change

Method 1: Automatic (Recommended)

Method 2: Manual Image Tag Update

Method 3: Configuration Changes

Method 4: Application Manifest Changes

Working with Secrets

Understanding Secret Management

Creating a New Secret

Step 1: Create Plain Secret Locally

Step 2: Seal the Secret

Step 3: Commit Sealed Secret

Step 4: Reference Secret in Application

Updating a Secret

Secret Best Practices

Where Secrets Are Stored

Enabling Authentication for Applications

How It Works

Authentication Modes

Token-Based Authentication

Step 1: Configure Helm Values

Step 2: Generate Token (if needed)

Step 3: Deploy Application

Step 4: Access Application

Advanced: Custom Secret Name

OIDC Authentication

Step 1: Configure Identity Provider

Step 2: Create OIDC Secret

Step 3: Configure Helm Values

Step 4: Deploy Application

Step 5: Access Application

Authentication Configuration Reference

Helm Values Schema

Annotations Set by Helm Chart

Sidecar Configuration

Advanced: Custom Sidecar Image

Authentication Examples

Example 1: Internal API with Token Auth

Example 2: User-Facing App with OIDC

Example 3: Disabling Authentication

Troubleshooting Authentication

Issue: 401 Unauthorized (Token Mode)

Issue: OIDC Login Loop

Issue: Auth Sidecar Not Injected

Issue: Auth Sidecar Crashes

Authentication Best Practices

Troubleshooting

Application Not Deploying

Problem: Application stuck in "Syncing" state

Problem: Pods crashing (CrashLoopBackOff)

Problem: Application not accessible via domain

Secret Issues

38 KiB

Raw Blame History

Repository: `forte-helm` (Chart Templates)

Repository: `helm-values` (Your App Config)

Repository: `sturdy-adventure` (ArgoCD Application)