# Developer Onboarding Guide ## Table of Contents - [Getting Started](#getting-started) - [Prerequisites](#prerequisites) - [Local Development Setup](#local-development-setup) - [Understanding the Workflow](#understanding-the-workflow) - [Deploying Your First Application](#deploying-your-first-application) - [Updating an Existing Application](#updating-an-existing-application) - [Working with Secrets](#working-with-secrets) - [Troubleshooting](#troubleshooting) - [Best Practices](#best-practices) --- ## Getting Started Welcome! This guide will help you understand how to develop and deploy applications on our Kubernetes cluster using GitOps principles powered by ArgoCD. ### What You'll Learn - How our GitOps architecture works - How to deploy a new application - How to update existing applications - How to manage secrets securely - Common troubleshooting techniques ### Who This Guide Is For - Developers deploying new applications - Developers maintaining existing applications - Team members who need to understand the deployment process --- ## Prerequisites ### Required Knowledge - ✅ Basic Git workflow (clone, commit, push, pull) - ✅ Docker basics (Dockerfile, building images) - ✅ YAML syntax - ✅ Basic understanding of Kubernetes concepts (pods, deployments, services) - ⚠️ Helm knowledge (helpful but not required - templates are provided) ### Required Tools Most developers **do NOT need kubectl access** to the cluster. You'll primarily work with Git repositories. If you do need cluster access, install: 1. **kubectl** - Kubernetes CLI ```bash # macOS brew install kubectl # Windows choco install kubernetes-cli # Linux curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" ``` 2. **kubeseal** - For sealing secrets ```bash # macOS brew install kubeseal # Windows choco install kubeseal # Linux wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz tar -xvzf kubeseal-0.24.0-linux-amd64.tar.gz sudo mv kubeseal /usr/local/bin/ ``` 3. **Git** - Version control ```bash git --version # Should already be installed ``` 4. **Docker** - For local development ```bash # macOS/Windows: Install Docker Desktop # Linux: Install Docker Engine docker --version ``` ### Repository Access You'll need read/write access to these repositories: 1. **sturdy-adventure** (Config repo) ```bash git clone https://github.com/snothub/sturdy-adventure.git cd sturdy-adventure ``` 2. **helm-values** (Values repo) ```bash git clone git@github.com:fortedigital/helm-values.git cd helm-values ``` 3. **forte-helm** (Chart repo - read-only for most developers) ```bash git clone https://github.com/snothub/forte-helm.git cd forte-helm ``` ### Cluster Access (If Needed) If you need kubectl access, ask the platform team for: - Kubeconfig file - Cluster context setup instructions Save to `~/.kube/config` and verify: ```bash kubectl cluster-info kubectl get nodes ``` --- ## Local Development Setup ### 1. Clone the Repositories Set up a consistent folder structure: ```bash mkdir -p ~/dev/k8s cd ~/dev/k8s # Clone repositories git clone https://github.com/snothub/sturdy-adventure.git launchpad git clone git@github.com:fortedigital/helm-values.git helm-prod-values git clone https://github.com/snothub/forte-helm.git forte-helm # Your folder structure: # ~/dev/k8s/ # ├── launchpad/ (Config repo) # ├── helm-prod-values/ (Values repo) # └── forte-helm/ (Chart repo) ``` ### 2. Local Development Environment Most applications use **Docker Compose** for local development: ```bash # In your application repository docker-compose up # Or for frontend applications npm install npm run dev ``` **You DO NOT run applications locally on Kubernetes.** Use Docker Compose or native tooling (npm, python, etc.). ### 3. Understanding the Deployment Flow ``` ┌─────────────────────────────────────────────────────────────────┐ │ Step 1: Develop Locally │ │ - Write code in your application repository │ │ - Test with Docker Compose or npm/python/etc. │ │ - Build Docker image │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Step 2: CI/CD Pipeline (Automated) │ │ - GitHub Actions builds image │ │ - Pushes to container registry (GHCR, Docker Hub) │ │ - Tags with version (e.g., v2.0.4) │ │ - Updates helm-values repository with new tag │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Step 3: GitOps Sync (Automated) │ │ - ArgoCD detects change in helm-values │ │ - Pulls updated configuration │ │ - Syncs to Kubernetes cluster │ │ - Sends Slack notification on success/failure │ └─────────────────────────────────────────────────────────────────┘ ``` **Key Insight**: You don't deploy directly. You push code, CI/CD builds it, and ArgoCD deploys it. --- ## Understanding the Workflow ### Three-Repository Pattern Our setup uses three repositories: | Repository | Purpose | You Edit | |------------|---------|----------| | **forte-helm** | Helm chart templates (generic, reusable) | ❌ Rarely | | **helm-values** | Application configuration (image tag, env vars) | ✅ Sometimes | | **sturdy-adventure** | ArgoCD Applications (what gets deployed) | ✅ Yes (for new apps) | ### Example: Deploying "myapp" #### Repository: `forte-helm` (Chart Templates) ```yaml # forteapp/templates/deployment.yaml # Generic template used by ALL apps apiVersion: apps/v1 kind: Deployment metadata: name: {{ .Values.app.name }} spec: containers: - name: app image: "{{ .Values.app.image.repository }}:{{ .Values.app.image.tag }}" env: - name: PORT value: {{ .Values.app.port }} ``` #### Repository: `helm-values` (Your App Config) ```yaml # myapp/values.yaml # Your app's specific configuration app: image: repository: ghcr.io/fortedigital/myapp tag: v1.0.0 # CI/CD updates this port: 3000 extraEnv: - name: API_URL value: https://api.example.com ``` #### Repository: `sturdy-adventure` (ArgoCD Application) ```yaml # apps/myapp.yaml # Tells ArgoCD to deploy your app apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: myapp namespace: argocd spec: sources: - repoURL: https://github.com/snothub/forte-helm path: forteapp helm: valueFiles: - $values/myapp/values.yaml - repoURL: git@github.com:fortedigital/helm-values.git ref: values destination: server: https://kubernetes.default.svc namespace: myapp syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ``` --- ## Deploying Your First Application ### Scenario: You've Built a New Application Let's deploy a new Node.js application called "hello-world". ### Step 1: Prepare Your Application Repository Ensure your app repository has: 1. **Dockerfile** ```dockerfile FROM node:18-alpine WORKDIR /app COPY package*.json ./ RUN npm ci --only=production COPY . . EXPOSE 3000 CMD ["node", "server.js"] ``` 2. **GitHub Actions Workflow** (`.github/workflows/deploy.yml`) ```yaml name: Build and Deploy on: push: branches: [ main ] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set version id: version run: echo "VERSION=v$(date +%Y%m%d-%H%M%S)" >> $GITHUB_OUTPUT - name: Build and push Docker image run: | echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin docker build -t ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }} . docker push ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }} - name: Update helm-values run: | git clone git@github.com:fortedigital/helm-values.git cd helm-values mkdir -p hello-world cat > hello-world/values.yaml < private/myapp-credentials.yaml ``` **DO NOT commit this file!** It's in `private/` which is Git-ignored. #### Step 2: Seal the Secret Get the public certificate (one-time setup): ```bash # Fetch public cert from cluster kubeseal --fetch-cert \ --controller-name=sealed-secrets-controller \ --controller-namespace=kube-system \ > pub-cert.pem ``` Seal your secret: ```bash kubeseal --format=yaml \ --cert=pub-cert.pem \ < private/myapp-credentials.yaml \ > secrets/myapp-credentials-sealed.yaml ``` #### Step 3: Commit Sealed Secret ```bash git add secrets/myapp-credentials-sealed.yaml git commit -m "Add myapp credentials (sealed)" git push ``` #### Step 4: Reference Secret in Application Update your `helm-values/myapp/values.yaml`: ```yaml app: envSecretName: "myapp-credentials" # References the SealedSecret ``` Commit and push: ```bash cd ~/dev/k8s/helm-prod-values git add myapp/values.yaml git commit -m "Reference myapp credentials" git push ``` ### Updating a Secret To update an existing secret: ```bash # 1. Create new version of secret kubectl create secret generic myapp-credentials \ --from-literal=API_KEY=new-key-here \ --from-literal=DB_PASSWORD=new-password \ --dry-run=client -o yaml > private/myapp-credentials.yaml # 2. Seal it kubeseal --format=yaml \ --cert=pub-cert.pem \ < private/myapp-credentials.yaml \ > secrets/myapp-credentials-sealed.yaml # 3. Commit sealed version git add secrets/myapp-credentials-sealed.yaml git commit -m "Update myapp credentials" git push # 4. Restart pods to pick up new secret kubectl rollout restart deployment myapp -n myapp ``` ### Secret Best Practices ✅ **DO**: - Store secrets in `private/` folder locally - Always seal secrets before committing - Delete plain secrets after sealing - Use meaningful secret names - Document what each secret contains ❌ **DON'T**: - Commit plain secrets to Git - Share secrets via Slack/email - Hard-code secrets in code - Use the same secret across multiple environments - Store secrets in Docker images ### Where Secrets Are Stored ``` ┌─────────────────────────────────────────────────────────────┐ │ Location │ Content │ Committed?│ ├──────────────────────────┼────────────────────┼────────────┤ │ private/ │ Plain secrets │ ❌ NO │ │ secrets/ │ Sealed secrets │ ✅ YES │ │ Kubernetes cluster │ Unsealed secrets │ N/A │ └─────────────────────────────────────────────────────────────┘ ``` **Sealed Secrets Controller** in the cluster decrypts sealed secrets automatically. --- ## Troubleshooting ### Application Not Deploying #### Problem: Application stuck in "Syncing" state **Check ArgoCD status:** ```bash kubectl get application myapp -n argocd -o yaml ``` Look for errors in `status.conditions`. **Common causes:** - ❌ Image doesn't exist or is not accessible - ❌ Invalid YAML syntax - ❌ Resource quota exceeded - ❌ Namespace conflicts - ❌ Invalid Helm values **Solutions:** ```bash # Check image exists docker pull ghcr.io/fortedigital/myapp:v1.0.0 # Validate YAML syntax kubectl apply --dry-run=client -f apps/myapp.yaml # Check ArgoCD logs kubectl logs -n argocd deployment/argocd-application-controller | grep myapp ``` #### Problem: Pods crashing (CrashLoopBackOff) **Check pod logs:** ```bash kubectl get pods -n myapp kubectl logs -n myapp kubectl describe pod -n myapp ``` **Common causes:** - ❌ Application error (check logs) - ❌ Missing environment variables - ❌ Incorrect port configuration - ❌ Missing secrets - ❌ Insufficient resources **Solutions:** ```bash # Check environment variables kubectl exec -n myapp -- env # Check if secrets exist kubectl get secrets -n myapp # Increase resources in helm-values vim ~/dev/k8s/helm-prod-values/myapp/values.yaml ``` #### Problem: Application not accessible via domain **Check ingress:** ```bash kubectl get ingressroute -n myapp kubectl describe ingressroute myapp -n myapp ``` **Common causes:** - ❌ DNS not configured - ❌ TLS certificate not issued - ❌ Incorrect domain in values.yaml - ❌ Traefik not routing correctly **Solutions:** ```bash # Check certificate kubectl get certificate -n myapp # Check cert-manager logs kubectl logs -n cert-manager deployment/cert-manager # Verify domain configuration cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep host # Test with port-forward kubectl port-forward -n myapp service/myapp 8080:3000 curl http://localhost:8080 ``` ### Secret Issues #### Problem: Secret not found **Check if SealedSecret exists:** ```bash kubectl get sealedsecret -n myapp kubectl get secret -n myapp ``` **Solutions:** ```bash # Check if secret is in Git ls -l secrets/myapp-credentials-sealed.yaml # Re-apply sealed secret kubectl apply -f secrets/myapp-credentials-sealed.yaml # Check sealed-secrets-controller logs kubectl logs -n kube-system deployment/sealed-secrets-controller ``` #### Problem: Secret exists but pods can't access it **Check pod events:** ```bash kubectl describe pod -n myapp ``` Look for: `Error: secret "myapp-credentials" not found` **Solutions:** ```bash # Verify secret name in values.yaml matches actual secret cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep envSecretName kubectl get secrets -n myapp # Restart pods kubectl rollout restart deployment myapp -n myapp ``` ### Sync Failures #### Problem: ArgoCD shows "Out of Sync" **Manual sync:** ```bash # Using kubectl kubectl patch application myapp -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"syncStrategy":{"hook":{}}}}}' # Or via ArgoCD UI # Click "Sync" button in UI ``` **Check what's different:** ```bash kubectl get application myapp -n argocd -o yaml ``` Look at `status.sync.comparedTo` vs desired state. #### Problem: Sync succeeds but application is "Degraded" **Check resource health:** ```bash kubectl get application myapp -n argocd -o jsonpath='{.status.resources[*].health}' ``` **Common causes:** - ❌ Pods not ready - ❌ Deployments not at desired replica count - ❌ Jobs failed **Solutions:** ```bash # Check all resources in namespace kubectl get all -n myapp # Check pod events kubectl get events -n myapp --sort-by='.lastTimestamp' ``` ### Getting Help If you're stuck: 1. **Check Slack notifications** - Error details are often in sync failure messages 2. **Check ArgoCD UI** - Visual representation of what's wrong 3. **Ask platform team** - They have full cluster access and can debug further 4. **Check documentation** - [Operations Runbook](OPERATIONS-RUNBOOK.md) has more troubleshooting --- ## Best Practices ### Development Workflow ✅ **DO**: - Develop and test locally with Docker Compose - Use semantic versioning for releases - Write descriptive commit messages - Test changes in a separate namespace first (if possible) - Monitor Slack for deployment notifications - Document environment variables and configuration ❌ **DON'T**: - Push directly to production without testing - Use `latest` tag for Docker images - Bypass CI/CD for "quick fixes" - Hard-code configuration values - Ignore deployment failures ### Configuration Management ✅ **DO**: - Keep configuration in `helm-values` repository - Use environment variables for config - Document what each value does - Use reasonable resource limits - Enable ingress and TLS for public services ❌ **DON'T**: - Hard-code config in application code - Over-allocate resources (wastes money) - Under-allocate resources (causes crashes) - Use HTTP for production services ### Secret Management ✅ **DO**: - Use kubeseal for all secrets - Store plain secrets in password manager - Rotate secrets regularly - Use different secrets per environment - Document what each secret contains ❌ **DON'T**: - Commit plain secrets - Share secrets in Slack/email - Reuse secrets across apps - Log secrets in application code ### Git Workflow ✅ **DO**: - Use feature branches for changes - Write clear commit messages - Use pull requests for review - Keep commits atomic and focused - Tag releases in application repos ❌ **DON'T**: - Push directly to `main` without review (for config repos) - Make multiple unrelated changes in one commit - Use vague commit messages ("fix", "update") - Force-push to main branches --- ## Quick Reference ### Common Commands ```bash # Check application status kubectl get application myapp -n argocd # View application details kubectl describe application myapp -n argocd # Check pods kubectl get pods -n myapp # View pod logs kubectl logs -n myapp # Restart deployment kubectl rollout restart deployment myapp -n myapp # Port-forward to service kubectl port-forward -n myapp service/myapp 8080:3000 # Create secret kubectl create secret generic myapp-credentials \ --from-literal=KEY=value \ --dry-run=client -o yaml > private/myapp-credentials.yaml # Seal secret kubeseal --format=yaml \ --cert=pub-cert.pem \ < private/myapp-credentials.yaml \ > secrets/myapp-credentials-sealed.yaml ``` ### Repository Locations ```bash # Config repository cd ~/dev/k8s/launchpad # Helm values repository cd ~/dev/k8s/helm-prod-values # Helm charts repository cd ~/dev/k8s/forte-helm ``` ### File Paths ```bash # New application manifest ~/dev/k8s/launchpad/apps/myapp.yaml # Application values ~/dev/k8s/helm-prod-values/myapp/values.yaml # Sealed secrets ~/dev/k8s/launchpad/secrets/myapp-credentials-sealed.yaml # Plain secrets (local only) ~/dev/k8s/launchpad/private/myapp-credentials.yaml ``` --- ## Next Steps Now that you understand the basics: 1. ✅ Deploy your first application (follow steps above) 2. 📖 Read the [Operations Runbook](OPERATIONS-RUNBOOK.md) for common tasks 3. 📖 Review [Technical Reference](REFERENCE.md) for detailed component docs 4. 📖 Understand [GitOps Architecture](GITOPS-ARCHITECTURE.md) for the big picture 5. 🚀 Start contributing! --- **Questions?** - Slack: #platform-support - Docs: [Full documentation index](README.md) - Help: Contact platform team **Last Updated**: 2026-03-16