launchpad/docs/DEVELOPER-GUIDE.md

Developer Onboarding Guide

Table of Contents

• Getting Started
• Prerequisites
• Local Development Setup
• Understanding the Workflow
• Deploying Your First Application
• Updating an Existing Application
• Working with Secrets
• Troubleshooting
• Best Practices

---

Getting Started

Welcome! This guide will help you understand how to develop and deploy applications on our Kubernetes cluster using GitOps principles powered by ArgoCD.

What You'll Learn

• How our GitOps architecture works
• How to deploy a new application
• How to update existing applications
• How to manage secrets securely
• Common troubleshooting techniques

Who This Guide Is For

• Developers deploying new applications
• Developers maintaining existing applications
• Team members who need to understand the deployment process

---

Prerequisites

Required Knowledge

• ✅ Basic Git workflow (clone, commit, push, pull)
• ✅ Docker basics (Dockerfile, building images)
• ✅ YAML syntax
• ✅ Basic understanding of Kubernetes concepts (pods, deployments, services)
• ⚠️ Helm knowledge (helpful but not required - templates are provided)

Required Tools

Most developers do NOT need kubectl access to the cluster. You'll primarily work with Git repositories.

If you do need cluster access, install:

1. kubectl - Kubernetes CLI

   # macOS
   brew install kubectl
   
   # Windows
   choco install kubernetes-cli
   
   # Linux
   curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   

2. kubeseal - For sealing secrets

   # macOS
   brew install kubeseal
   
   # Windows
   choco install kubeseal
   
   # Linux
   wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
   tar -xvzf kubeseal-0.24.0-linux-amd64.tar.gz
   sudo mv kubeseal /usr/local/bin/
   

3. Git - Version control

   git --version  # Should already be installed
   

4. Docker - For local development

   # macOS/Windows: Install Docker Desktop
   # Linux: Install Docker Engine
   docker --version
   

Repository Access

You'll need read/write access to these repositories:

1. sturdy-adventure (Config repo)

   git clone https://github.com/snothub/sturdy-adventure.git
   cd sturdy-adventure
   

2. helm-values (Values repo)

   git clone git@github.com:fortedigital/helm-values.git
   cd helm-values
   

3. forte-helm (Chart repo - read-only for most developers)

   git clone https://github.com/snothub/forte-helm.git
   cd forte-helm
   

Cluster Access (If Needed)

If you need kubectl access, ask the platform team for:

• Kubeconfig file
• Cluster context setup instructions

Save to ~/.kube/config and verify:

kubectl cluster-info
kubectl get nodes

---

Local Development Setup

1. Clone the Repositories

Set up a consistent folder structure:

mkdir -p ~/dev/k8s
cd ~/dev/k8s

# Clone repositories
git clone https://github.com/snothub/sturdy-adventure.git launchpad
git clone git@github.com:fortedigital/helm-values.git helm-prod-values
git clone https://github.com/snothub/forte-helm.git forte-helm

# Your folder structure:
# ~/dev/k8s/
# ├── launchpad/           (Config repo)
# ├── helm-prod-values/    (Values repo)
# └── forte-helm/          (Chart repo)

2. Local Development Environment

Most applications use Docker Compose for local development:

# In your application repository
docker-compose up

# Or for frontend applications
npm install
npm run dev

You DO NOT run applications locally on Kubernetes. Use Docker Compose or native tooling (npm, python, etc.).

3. Understanding the Deployment Flow

┌─────────────────────────────────────────────────────────────────┐
│  Step 1: Develop Locally                                        │
│  - Write code in your application repository                    │
│  - Test with Docker Compose or npm/python/etc.                  │
│  - Build Docker image                                            │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 2: CI/CD Pipeline (Automated)                             │
│  - GitHub Actions builds image                                  │
│  - Pushes to container registry (GHCR, Docker Hub)              │
│  - Tags with version (e.g., v2.0.4)                             │
│  - Updates helm-values repository with new tag                  │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Step 3: GitOps Sync (Automated)                                │
│  - ArgoCD detects change in helm-values                         │
│  - Pulls updated configuration                                  │
│  - Syncs to Kubernetes cluster                                  │
│  - Sends Slack notification on success/failure                  │
└─────────────────────────────────────────────────────────────────┘

Key Insight: You don't deploy directly. You push code, CI/CD builds it, and ArgoCD deploys it.

---

Understanding the Workflow

Three-Repository Pattern

Our setup uses three repositories:

Repository	Purpose	You Edit
forte-helm	Helm chart templates (generic, reusable)	❌ Rarely
helm-values	Application configuration (image tag, env vars)	✅ Sometimes
sturdy-adventure	ArgoCD Applications (what gets deployed)	✅ Yes (for new apps)

Example: Deploying "myapp"

Repository: forte-helm (Chart Templates)

# forteapp/templates/deployment.yaml
# Generic template used by ALL apps
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
spec:
  containers:
  - name: app
    image: "{{ .Values.app.image.repository }}:{{ .Values.app.image.tag }}"
    env:
    - name: PORT
      value: {{ .Values.app.port }}

Repository: helm-values (Your App Config)

# myapp/values.yaml
# Your app's specific configuration
app:
  image:
    repository: ghcr.io/fortedigital/myapp
    tag: v1.0.0                    # CI/CD updates this
  port: 3000
  extraEnv:
  - name: API_URL
    value: https://api.example.com

Repository: sturdy-adventure (ArgoCD Application)

# apps/myapp.yaml
# Tells ArgoCD to deploy your app
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  sources:
  - repoURL: https://github.com/snothub/forte-helm
    path: forteapp
    helm:
      valueFiles:
      - $values/myapp/values.yaml

  - repoURL: git@github.com:fortedigital/helm-values.git
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: myapp

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

---

Deploying Your First Application

Scenario: You've Built a New Application

Let's deploy a new Node.js application called "hello-world".

Step 1: Prepare Your Application Repository

Ensure your app repository has:

1. Dockerfile

   FROM node:18-alpine
   WORKDIR /app
   COPY package*.json ./
   RUN npm ci --only=production
   COPY . .
   EXPOSE 3000
   CMD ["node", "server.js"]
   

2. GitHub Actions Workflow (.github/workflows/deploy.yml)

   name: Build and Deploy
   
   on:
     push:
       branches: [ main ]
   
   jobs:
     build:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v3
   
         - name: Set version
           id: version
           run: echo "VERSION=v$(date +%Y%m%d-%H%M%S)" >> $GITHUB_OUTPUT
   
         - name: Build and push Docker image
           run: |
             echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
             docker build -t ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }} .
             docker push ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }}
   
         - name: Update helm-values
           run: |
             git clone git@github.com:fortedigital/helm-values.git
             cd helm-values
             mkdir -p hello-world
             cat > hello-world/values.yaml <<EOF
             app:
               image:
                 repository: ghcr.io/fortedigital/hello-world
                 tag: ${{ steps.version.outputs.VERSION }}
             EOF
             git add hello-world/values.yaml
             git commit -m "Update hello-world to ${{ steps.version.outputs.VERSION }}"
             git push
   

Step 2: Create Helm Values

Create a folder in helm-values repository:

cd ~/dev/k8s/helm-prod-values
mkdir -p hello-world

Create hello-world/values.yaml:

app:
  image:
    repository: ghcr.io/fortedigital/hello-world
    tag: v1.0.0                    # Will be updated by CI/CD
    containerPort: 3000

  replicaCount: 1

  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi

  extraEnv:
  - name: PORT
    value: "3000"
  - name: NODE_ENV
    value: "production"

  envSecretName: ""                # Optional: reference to secrets

service:
  port: 3000

ingress:
  enabled: true
  host: hello-world.forteapps.net  # Your subdomain

db:
  enabled: false                   # Set to true if you need PostgreSQL

Commit and push:

git add hello-world/values.yaml
git commit -m "Add hello-world application values"
git push

Step 3: Create ArgoCD Application Manifest

In the sturdy-adventure repository, create apps/hello-world.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello-world
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: ""
    notifications.argoproj.io/subscribe.on-sync-failed.slack: ""
    notifications.argoproj.io/subscribe.on-degraded.slack: ""
  labels:
    app.kubernetes.io/name: hello-world
    app.kubernetes.io/part-of: apps
    app.kubernetes.io/managed-by: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io

spec:
  project: default

  sources:
  # Source 1: Helm chart templates
  - repoURL: https://github.com/snothub/forte-helm
    path: forteapp
    targetRevision: HEAD
    helm:
      valueFiles:
      - $values/hello-world/values.yaml

  # Source 2: Helm values
  - repoURL: git@github.com:fortedigital/helm-values.git
    targetRevision: HEAD
    ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: hello-world

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false

    syncOptions:
    - CreateNamespace=true
    - Validate=true
    - ServerSideApply=true

    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas

Commit and push:

cd ~/dev/k8s/launchpad
git add apps/hello-world.yaml
git commit -m "Add hello-world application"
git push

Step 4: Verify Deployment

ArgoCD will automatically detect the new application within 60 seconds.

Option 1: Check Slack

• Watch for sync notifications in your Slack channel
• ✅ "Application hello-world sync succeeded"

Option 2: Check ArgoCD UI (if you have access)

# Port forward to ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Open browser: https://localhost:8080
# Look for "hello-world" application

Option 3: Check with kubectl (if you have access)

# List ArgoCD applications
kubectl get applications -n argocd

# Check application status
kubectl get application hello-world -n argocd

# Verify pods are running
kubectl get pods -n hello-world

Step 5: Access Your Application

Once deployed, access via the configured domain:

# Check if ingress is created
kubectl get ingressroute -n hello-world

# Access application
curl https://hello-world.forteapps.net

⚠️ Note: DNS must be manually configured for new subdomains. Contact the platform team to add DNS records.

---

Updating an Existing Application

Scenario: Deploying a Code Change

You've made changes to your application code and want to deploy them.

Method 1: Automatic (Recommended)

Just push to main branch - CI/CD handles everything:

# In your application repository
git add .
git commit -m "Fix bug in user login"
git push origin main

What Happens Next:

1. ✅ GitHub Actions triggers
2. ✅ Builds new Docker image
3. ✅ Tags with new version (e.g., v20260316-143022)
4. ✅ Pushes to container registry
5. ✅ Updates helm-values/myapp/values.yaml with new tag
6. ✅ ArgoCD detects change
7. ✅ Syncs new version to cluster
8. ✅ Sends Slack notification

Timeline: ~5-10 minutes from push to deployment

Method 2: Manual Image Tag Update

If CI/CD is not set up, manually update the image tag:

cd ~/dev/k8s/helm-prod-values

# Edit your app's values.yaml
vim myapp/values.yaml

# Change:
app:
  image:
    tag: v1.0.0  # Old version
# To:
app:
  image:
    tag: v1.0.1  # New version

# Commit and push
git add myapp/values.yaml
git commit -m "Update myapp to v1.0.1"
git push

ArgoCD will sync within 60 seconds.

Method 3: Configuration Changes

To update environment variables, resources, or other config:

cd ~/dev/k8s/helm-prod-values
vim myapp/values.yaml

Example changes:

app:
  # Increase resources
  resources:
    requests:
      cpu: 200m      # Was 100m
      memory: 256Mi  # Was 128Mi

  # Add new environment variable
  extraEnv:
  - name: API_URL
    value: https://api.example.com
  - name: DEBUG          # NEW
    value: "true"        # NEW

  # Enable HPA
  hpa:
    enabled: true        # Was false
    minReplicas: 2
    maxReplicas: 10

Commit and push:

git add myapp/values.yaml
git commit -m "Increase myapp resources and enable HPA"
git push

Method 4: Application Manifest Changes

To change ArgoCD sync behavior, namespace, or other meta-config:

cd ~/dev/k8s/launchpad
vim apps/myapp.yaml

Example changes:

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: false    # Disable self-healing temporarily

Commit and push:

git add apps/myapp.yaml
git commit -m "Disable self-healing for myapp"
git push

---

Working with Secrets

Understanding Secret Management

NEVER commit plain secrets to Git. We use Sealed Secrets to encrypt secrets before committing.

Creating a New Secret

Step 1: Create Plain Secret Locally

cd ~/dev/k8s/launchpad

# Create secret in private/ folder (Git-ignored)
kubectl create secret generic myapp-credentials \
  --from-literal=API_KEY=your-secret-key-here \
  --from-literal=DB_PASSWORD=super-secret-password \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

DO NOT commit this file! It's in private/ which is Git-ignored.

Step 2: Seal the Secret

Get the public certificate (one-time setup):

# Fetch public cert from cluster
kubeseal --fetch-cert \
  --controller-name=sealed-secrets-controller \
  --controller-namespace=kube-system \
  > pub-cert.pem

Seal your secret:

kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml

Step 3: Commit Sealed Secret

git add secrets/myapp-credentials-sealed.yaml
git commit -m "Add myapp credentials (sealed)"
git push

Step 4: Reference Secret in Application

Update your helm-values/myapp/values.yaml:

app:
  envSecretName: "myapp-credentials"  # References the SealedSecret

Commit and push:

cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Reference myapp credentials"
git push

Updating a Secret

To update an existing secret:

# 1. Create new version of secret
kubectl create secret generic myapp-credentials \
  --from-literal=API_KEY=new-key-here \
  --from-literal=DB_PASSWORD=new-password \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

# 2. Seal it
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml

# 3. Commit sealed version
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Update myapp credentials"
git push

# 4. Restart pods to pick up new secret
kubectl rollout restart deployment myapp -n myapp

Secret Best Practices

✅ DO:

• Store secrets in private/ folder locally
• Always seal secrets before committing
• Delete plain secrets after sealing
• Use meaningful secret names
• Document what each secret contains

❌ DON'T:

• Commit plain secrets to Git
• Share secrets via Slack/email
• Hard-code secrets in code
• Use the same secret across multiple environments
• Store secrets in Docker images

Where Secrets Are Stored

┌─────────────────────────────────────────────────────────────┐
│  Location                │  Content           │  Committed?│
├──────────────────────────┼────────────────────┼────────────┤
│  private/                │  Plain secrets     │  ❌ NO      │
│  secrets/                │  Sealed secrets    │  ✅ YES     │
│  Kubernetes cluster      │  Unsealed secrets  │  N/A       │
└─────────────────────────────────────────────────────────────┘

Sealed Secrets Controller in the cluster decrypts sealed secrets automatically.

---

Troubleshooting

Application Not Deploying

Problem: Application stuck in "Syncing" state

Check ArgoCD status:

kubectl get application myapp -n argocd -o yaml

Look for errors in status.conditions.

Common causes:

• ❌ Image doesn't exist or is not accessible
• ❌ Invalid YAML syntax
• ❌ Resource quota exceeded
• ❌ Namespace conflicts
• ❌ Invalid Helm values

Solutions:

# Check image exists
docker pull ghcr.io/fortedigital/myapp:v1.0.0

# Validate YAML syntax
kubectl apply --dry-run=client -f apps/myapp.yaml

# Check ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller | grep myapp

Problem: Pods crashing (CrashLoopBackOff)

Check pod logs:

kubectl get pods -n myapp
kubectl logs -n myapp <pod-name>
kubectl describe pod -n myapp <pod-name>

Common causes:

• ❌ Application error (check logs)
• ❌ Missing environment variables
• ❌ Incorrect port configuration
• ❌ Missing secrets
• ❌ Insufficient resources

Solutions:

# Check environment variables
kubectl exec -n myapp <pod-name> -- env

# Check if secrets exist
kubectl get secrets -n myapp

# Increase resources in helm-values
vim ~/dev/k8s/helm-prod-values/myapp/values.yaml

Problem: Application not accessible via domain

Check ingress:

kubectl get ingressroute -n myapp
kubectl describe ingressroute myapp -n myapp

Common causes:

• ❌ DNS not configured
• ❌ TLS certificate not issued
• ❌ Incorrect domain in values.yaml
• ❌ Traefik not routing correctly

Solutions:

# Check certificate
kubectl get certificate -n myapp

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Verify domain configuration
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep host

# Test with port-forward
kubectl port-forward -n myapp service/myapp 8080:3000
curl http://localhost:8080

Secret Issues

Problem: Secret not found

Check if SealedSecret exists:

kubectl get sealedsecret -n myapp
kubectl get secret -n myapp

Solutions:

# Check if secret is in Git
ls -l secrets/myapp-credentials-sealed.yaml

# Re-apply sealed secret
kubectl apply -f secrets/myapp-credentials-sealed.yaml

# Check sealed-secrets-controller logs
kubectl logs -n kube-system deployment/sealed-secrets-controller

Problem: Secret exists but pods can't access it

Check pod events:

kubectl describe pod -n myapp <pod-name>

Look for: Error: secret "myapp-credentials" not found

Solutions:

# Verify secret name in values.yaml matches actual secret
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep envSecretName
kubectl get secrets -n myapp

# Restart pods
kubectl rollout restart deployment myapp -n myapp

Sync Failures

Problem: ArgoCD shows "Out of Sync"

Manual sync:

# Using kubectl
kubectl patch application myapp -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"syncStrategy":{"hook":{}}}}}'

# Or via ArgoCD UI
# Click "Sync" button in UI

Check what's different:

kubectl get application myapp -n argocd -o yaml

Look at status.sync.comparedTo vs desired state.

Problem: Sync succeeds but application is "Degraded"

Check resource health:

kubectl get application myapp -n argocd -o jsonpath='{.status.resources[*].health}'

Common causes:

• ❌ Pods not ready
• ❌ Deployments not at desired replica count
• ❌ Jobs failed

Solutions:

# Check all resources in namespace
kubectl get all -n myapp

# Check pod events
kubectl get events -n myapp --sort-by='.lastTimestamp'

Getting Help

If you're stuck:

1. Check Slack notifications - Error details are often in sync failure messages
2. Check ArgoCD UI - Visual representation of what's wrong
3. Ask platform team - They have full cluster access and can debug further
4. Check documentation - Operations Runbook has more troubleshooting

---

Best Practices

Development Workflow

✅ DO:

• Develop and test locally with Docker Compose
• Use semantic versioning for releases
• Write descriptive commit messages
• Test changes in a separate namespace first (if possible)
• Monitor Slack for deployment notifications
• Document environment variables and configuration

❌ DON'T:

• Push directly to production without testing
• Use latest tag for Docker images
• Bypass CI/CD for "quick fixes"
• Hard-code configuration values
• Ignore deployment failures

Configuration Management

✅ DO:

• Keep configuration in helm-values repository
• Use environment variables for config
• Document what each value does
• Use reasonable resource limits
• Enable ingress and TLS for public services

❌ DON'T:

• Hard-code config in application code
• Over-allocate resources (wastes money)
• Under-allocate resources (causes crashes)
• Use HTTP for production services

Secret Management

✅ DO:

• Use kubeseal for all secrets
• Store plain secrets in password manager
• Rotate secrets regularly
• Use different secrets per environment
• Document what each secret contains

❌ DON'T:

• Commit plain secrets
• Share secrets in Slack/email
• Reuse secrets across apps
• Log secrets in application code

Git Workflow

✅ DO:

• Use feature branches for changes
• Write clear commit messages
• Use pull requests for review
• Keep commits atomic and focused
• Tag releases in application repos

❌ DON'T:

• Push directly to main without review (for config repos)
• Make multiple unrelated changes in one commit
• Use vague commit messages ("fix", "update")
• Force-push to main branches

---

Quick Reference

Common Commands

# Check application status
kubectl get application myapp -n argocd

# View application details
kubectl describe application myapp -n argocd

# Check pods
kubectl get pods -n myapp

# View pod logs
kubectl logs -n myapp <pod-name>

# Restart deployment
kubectl rollout restart deployment myapp -n myapp

# Port-forward to service
kubectl port-forward -n myapp service/myapp 8080:3000

# Create secret
kubectl create secret generic myapp-credentials \
  --from-literal=KEY=value \
  --dry-run=client -o yaml > private/myapp-credentials.yaml

# Seal secret
kubeseal --format=yaml \
  --cert=pub-cert.pem \
  < private/myapp-credentials.yaml \
  > secrets/myapp-credentials-sealed.yaml

Repository Locations

# Config repository
cd ~/dev/k8s/launchpad

# Helm values repository
cd ~/dev/k8s/helm-prod-values

# Helm charts repository
cd ~/dev/k8s/forte-helm

File Paths

# New application manifest
~/dev/k8s/launchpad/apps/myapp.yaml

# Application values
~/dev/k8s/helm-prod-values/myapp/values.yaml

# Sealed secrets
~/dev/k8s/launchpad/secrets/myapp-credentials-sealed.yaml

# Plain secrets (local only)
~/dev/k8s/launchpad/private/myapp-credentials.yaml

---

Next Steps

Now that you understand the basics:

1. ✅ Deploy your first application (follow steps above)
2. 📖 Read the Operations Runbook for common tasks
3. 📖 Review Technical Reference for detailed component docs
4. 📖 Understand GitOps Architecture for the big picture
5. 🚀 Start contributing!

---

Questions?

• Slack: #platform-support
• Docs: Full documentation index
• Help: Contact platform team

Last Updated: 2026-03-16