1090 lines
26 KiB
Markdown
1090 lines
26 KiB
Markdown
# Developer Onboarding Guide
|
|
|
|
## Table of Contents
|
|
- [Getting Started](#getting-started)
|
|
- [Prerequisites](#prerequisites)
|
|
- [Local Development Setup](#local-development-setup)
|
|
- [Understanding the Workflow](#understanding-the-workflow)
|
|
- [Deploying Your First Application](#deploying-your-first-application)
|
|
- [Updating an Existing Application](#updating-an-existing-application)
|
|
- [Working with Secrets](#working-with-secrets)
|
|
- [Troubleshooting](#troubleshooting)
|
|
- [Best Practices](#best-practices)
|
|
|
|
---
|
|
|
|
## Getting Started
|
|
|
|
Welcome! This guide will help you understand how to develop and deploy applications on our Kubernetes cluster using GitOps principles powered by ArgoCD.
|
|
|
|
### What You'll Learn
|
|
- How our GitOps architecture works
|
|
- How to deploy a new application
|
|
- How to update existing applications
|
|
- How to manage secrets securely
|
|
- Common troubleshooting techniques
|
|
|
|
### Who This Guide Is For
|
|
- Developers deploying new applications
|
|
- Developers maintaining existing applications
|
|
- Team members who need to understand the deployment process
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
### Required Knowledge
|
|
- ✅ Basic Git workflow (clone, commit, push, pull)
|
|
- ✅ Docker basics (Dockerfile, building images)
|
|
- ✅ YAML syntax
|
|
- ✅ Basic understanding of Kubernetes concepts (pods, deployments, services)
|
|
- ⚠️ Helm knowledge (helpful but not required - templates are provided)
|
|
|
|
### Required Tools
|
|
|
|
Most developers **do NOT need kubectl access** to the cluster. You'll primarily work with Git repositories.
|
|
|
|
If you do need cluster access, install:
|
|
|
|
1. **kubectl** - Kubernetes CLI
|
|
```bash
|
|
# macOS
|
|
brew install kubectl
|
|
|
|
# Windows
|
|
choco install kubernetes-cli
|
|
|
|
# Linux
|
|
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
|
|
```
|
|
|
|
2. **kubeseal** - For sealing secrets
|
|
```bash
|
|
# macOS
|
|
brew install kubeseal
|
|
|
|
# Windows
|
|
choco install kubeseal
|
|
|
|
# Linux
|
|
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
|
|
tar -xvzf kubeseal-0.24.0-linux-amd64.tar.gz
|
|
sudo mv kubeseal /usr/local/bin/
|
|
```
|
|
|
|
3. **Git** - Version control
|
|
```bash
|
|
git --version # Should already be installed
|
|
```
|
|
|
|
4. **Docker** - For local development
|
|
```bash
|
|
# macOS/Windows: Install Docker Desktop
|
|
# Linux: Install Docker Engine
|
|
docker --version
|
|
```
|
|
|
|
### Repository Access
|
|
|
|
You'll need read/write access to these repositories:
|
|
|
|
1. **sturdy-adventure** (Config repo)
|
|
```bash
|
|
git clone https://github.com/snothub/sturdy-adventure.git
|
|
cd sturdy-adventure
|
|
```
|
|
|
|
2. **helm-values** (Values repo)
|
|
```bash
|
|
git clone git@github.com:fortedigital/helm-values.git
|
|
cd helm-values
|
|
```
|
|
|
|
3. **forte-helm** (Chart repo - read-only for most developers)
|
|
```bash
|
|
git clone https://github.com/snothub/forte-helm.git
|
|
cd forte-helm
|
|
```
|
|
|
|
### Cluster Access (If Needed)
|
|
|
|
If you need kubectl access, ask the platform team for:
|
|
- Kubeconfig file
|
|
- Cluster context setup instructions
|
|
|
|
Save to `~/.kube/config` and verify:
|
|
```bash
|
|
kubectl cluster-info
|
|
kubectl get nodes
|
|
```
|
|
|
|
---
|
|
|
|
## Local Development Setup
|
|
|
|
### 1. Clone the Repositories
|
|
|
|
Set up a consistent folder structure:
|
|
|
|
```bash
|
|
mkdir -p ~/dev/k8s
|
|
cd ~/dev/k8s
|
|
|
|
# Clone repositories
|
|
git clone https://github.com/snothub/sturdy-adventure.git launchpad
|
|
git clone git@github.com:fortedigital/helm-values.git helm-prod-values
|
|
git clone https://github.com/snothub/forte-helm.git forte-helm
|
|
|
|
# Your folder structure:
|
|
# ~/dev/k8s/
|
|
# ├── launchpad/ (Config repo)
|
|
# ├── helm-prod-values/ (Values repo)
|
|
# └── forte-helm/ (Chart repo)
|
|
```
|
|
|
|
### 2. Local Development Environment
|
|
|
|
Most applications use **Docker Compose** for local development:
|
|
|
|
```bash
|
|
# In your application repository
|
|
docker-compose up
|
|
|
|
# Or for frontend applications
|
|
npm install
|
|
npm run dev
|
|
```
|
|
|
|
**You DO NOT run applications locally on Kubernetes.** Use Docker Compose or native tooling (npm, python, etc.).
|
|
|
|
### 3. Understanding the Deployment Flow
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Step 1: Develop Locally │
|
|
│ - Write code in your application repository │
|
|
│ - Test with Docker Compose or npm/python/etc. │
|
|
│ - Build Docker image │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Step 2: CI/CD Pipeline (Automated) │
|
|
│ - GitHub Actions builds image │
|
|
│ - Pushes to container registry (GHCR, Docker Hub) │
|
|
│ - Tags with version (e.g., v2.0.4) │
|
|
│ - Updates helm-values repository with new tag │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Step 3: GitOps Sync (Automated) │
|
|
│ - ArgoCD detects change in helm-values │
|
|
│ - Pulls updated configuration │
|
|
│ - Syncs to Kubernetes cluster │
|
|
│ - Sends Slack notification on success/failure │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**Key Insight**: You don't deploy directly. You push code, CI/CD builds it, and ArgoCD deploys it.
|
|
|
|
---
|
|
|
|
## Understanding the Workflow
|
|
|
|
### Three-Repository Pattern
|
|
|
|
Our setup uses three repositories:
|
|
|
|
| Repository | Purpose | You Edit |
|
|
|------------|---------|----------|
|
|
| **forte-helm** | Helm chart templates (generic, reusable) | ❌ Rarely |
|
|
| **helm-values** | Application configuration (image tag, env vars) | ✅ Sometimes |
|
|
| **sturdy-adventure** | ArgoCD Applications (what gets deployed) | ✅ Yes (for new apps) |
|
|
|
|
### Example: Deploying "myapp"
|
|
|
|
#### Repository: `forte-helm` (Chart Templates)
|
|
```yaml
|
|
# forteapp/templates/deployment.yaml
|
|
# Generic template used by ALL apps
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: {{ .Values.app.name }}
|
|
spec:
|
|
containers:
|
|
- name: app
|
|
image: "{{ .Values.app.image.repository }}:{{ .Values.app.image.tag }}"
|
|
env:
|
|
- name: PORT
|
|
value: {{ .Values.app.port }}
|
|
```
|
|
|
|
#### Repository: `helm-values` (Your App Config)
|
|
```yaml
|
|
# myapp/values.yaml
|
|
# Your app's specific configuration
|
|
app:
|
|
image:
|
|
repository: ghcr.io/fortedigital/myapp
|
|
tag: v1.0.0 # CI/CD updates this
|
|
port: 3000
|
|
extraEnv:
|
|
- name: API_URL
|
|
value: https://api.example.com
|
|
```
|
|
|
|
#### Repository: `sturdy-adventure` (ArgoCD Application)
|
|
```yaml
|
|
# apps/myapp.yaml
|
|
# Tells ArgoCD to deploy your app
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: myapp
|
|
namespace: argocd
|
|
spec:
|
|
sources:
|
|
- repoURL: https://github.com/snothub/forte-helm
|
|
path: forteapp
|
|
helm:
|
|
valueFiles:
|
|
- $values/myapp/values.yaml
|
|
|
|
- repoURL: git@github.com:fortedigital/helm-values.git
|
|
ref: values
|
|
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: myapp
|
|
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
```
|
|
|
|
---
|
|
|
|
## Deploying Your First Application
|
|
|
|
### Scenario: You've Built a New Application
|
|
|
|
Let's deploy a new Node.js application called "hello-world".
|
|
|
|
### Step 1: Prepare Your Application Repository
|
|
|
|
Ensure your app repository has:
|
|
|
|
1. **Dockerfile**
|
|
```dockerfile
|
|
FROM node:18-alpine
|
|
WORKDIR /app
|
|
COPY package*.json ./
|
|
RUN npm ci --only=production
|
|
COPY . .
|
|
EXPOSE 3000
|
|
CMD ["node", "server.js"]
|
|
```
|
|
|
|
2. **GitHub Actions Workflow** (`.github/workflows/deploy.yml`)
|
|
```yaml
|
|
name: Build and Deploy
|
|
|
|
on:
|
|
push:
|
|
branches: [ main ]
|
|
|
|
jobs:
|
|
build:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v3
|
|
|
|
- name: Set version
|
|
id: version
|
|
run: echo "VERSION=v$(date +%Y%m%d-%H%M%S)" >> $GITHUB_OUTPUT
|
|
|
|
- name: Build and push Docker image
|
|
run: |
|
|
echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
|
|
docker build -t ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }} .
|
|
docker push ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }}
|
|
|
|
- name: Update helm-values
|
|
run: |
|
|
git clone git@github.com:fortedigital/helm-values.git
|
|
cd helm-values
|
|
mkdir -p hello-world
|
|
cat > hello-world/values.yaml <<EOF
|
|
app:
|
|
image:
|
|
repository: ghcr.io/fortedigital/hello-world
|
|
tag: ${{ steps.version.outputs.VERSION }}
|
|
EOF
|
|
git add hello-world/values.yaml
|
|
git commit -m "Update hello-world to ${{ steps.version.outputs.VERSION }}"
|
|
git push
|
|
```
|
|
|
|
### Step 2: Create Helm Values
|
|
|
|
Create a folder in `helm-values` repository:
|
|
|
|
```bash
|
|
cd ~/dev/k8s/helm-prod-values
|
|
mkdir -p hello-world
|
|
```
|
|
|
|
Create `hello-world/values.yaml`:
|
|
```yaml
|
|
app:
|
|
image:
|
|
repository: ghcr.io/fortedigital/hello-world
|
|
tag: v1.0.0 # Will be updated by CI/CD
|
|
containerPort: 3000
|
|
|
|
replicaCount: 1
|
|
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 128Mi
|
|
limits:
|
|
cpu: 500m
|
|
memory: 512Mi
|
|
|
|
extraEnv:
|
|
- name: PORT
|
|
value: "3000"
|
|
- name: NODE_ENV
|
|
value: "production"
|
|
|
|
envSecretName: "" # Optional: reference to secrets
|
|
|
|
service:
|
|
port: 3000
|
|
|
|
ingress:
|
|
enabled: true
|
|
host: hello-world.forteapps.net # Your subdomain
|
|
|
|
db:
|
|
enabled: false # Set to true if you need PostgreSQL
|
|
```
|
|
|
|
Commit and push:
|
|
```bash
|
|
git add hello-world/values.yaml
|
|
git commit -m "Add hello-world application values"
|
|
git push
|
|
```
|
|
|
|
### Step 3: Create ArgoCD Application Manifest
|
|
|
|
In the `sturdy-adventure` repository, create `apps/hello-world.yaml`:
|
|
|
|
```yaml
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: hello-world
|
|
namespace: argocd
|
|
annotations:
|
|
argocd.argoproj.io/sync-wave: "1"
|
|
notifications.argoproj.io/subscribe.on-sync-succeeded.slack: ""
|
|
notifications.argoproj.io/subscribe.on-sync-failed.slack: ""
|
|
notifications.argoproj.io/subscribe.on-degraded.slack: ""
|
|
labels:
|
|
app.kubernetes.io/name: hello-world
|
|
app.kubernetes.io/part-of: apps
|
|
app.kubernetes.io/managed-by: argocd
|
|
finalizers:
|
|
- resources-finalizer.argocd.argoproj.io
|
|
|
|
spec:
|
|
project: default
|
|
|
|
sources:
|
|
# Source 1: Helm chart templates
|
|
- repoURL: https://github.com/snothub/forte-helm
|
|
path: forteapp
|
|
targetRevision: HEAD
|
|
helm:
|
|
valueFiles:
|
|
- $values/hello-world/values.yaml
|
|
|
|
# Source 2: Helm values
|
|
- repoURL: git@github.com:fortedigital/helm-values.git
|
|
targetRevision: HEAD
|
|
ref: values
|
|
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: hello-world
|
|
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
allowEmpty: false
|
|
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
- Validate=true
|
|
- ServerSideApply=true
|
|
|
|
retry:
|
|
limit: 5
|
|
backoff:
|
|
duration: 5s
|
|
factor: 2
|
|
maxDuration: 3m
|
|
|
|
ignoreDifferences:
|
|
- group: apps
|
|
kind: Deployment
|
|
jsonPointers:
|
|
- /spec/replicas
|
|
```
|
|
|
|
Commit and push:
|
|
```bash
|
|
cd ~/dev/k8s/launchpad
|
|
git add apps/hello-world.yaml
|
|
git commit -m "Add hello-world application"
|
|
git push
|
|
```
|
|
|
|
### Step 4: Verify Deployment
|
|
|
|
ArgoCD will automatically detect the new application within 60 seconds.
|
|
|
|
**Option 1: Check Slack**
|
|
- Watch for sync notifications in your Slack channel
|
|
- ✅ "Application hello-world sync succeeded"
|
|
|
|
**Option 2: Check ArgoCD UI** (if you have access)
|
|
```bash
|
|
# Port forward to ArgoCD UI
|
|
kubectl port-forward svc/argocd-server -n argocd 8080:443
|
|
|
|
# Open browser: https://localhost:8080
|
|
# Look for "hello-world" application
|
|
```
|
|
|
|
**Option 3: Check with kubectl** (if you have access)
|
|
```bash
|
|
# List ArgoCD applications
|
|
kubectl get applications -n argocd
|
|
|
|
# Check application status
|
|
kubectl get application hello-world -n argocd
|
|
|
|
# Verify pods are running
|
|
kubectl get pods -n hello-world
|
|
```
|
|
|
|
### Step 5: Access Your Application
|
|
|
|
Once deployed, access via the configured domain:
|
|
|
|
```bash
|
|
# Check if ingress is created
|
|
kubectl get ingressroute -n hello-world
|
|
|
|
# Access application
|
|
curl https://hello-world.forteapps.net
|
|
```
|
|
|
|
**⚠️ Note**: DNS must be manually configured for new subdomains. Contact the platform team to add DNS records.
|
|
|
|
---
|
|
|
|
## Updating an Existing Application
|
|
|
|
### Scenario: Deploying a Code Change
|
|
|
|
You've made changes to your application code and want to deploy them.
|
|
|
|
### Method 1: Automatic (Recommended)
|
|
|
|
**Just push to `main` branch** - CI/CD handles everything:
|
|
|
|
```bash
|
|
# In your application repository
|
|
git add .
|
|
git commit -m "Fix bug in user login"
|
|
git push origin main
|
|
```
|
|
|
|
**What Happens Next:**
|
|
1. ✅ GitHub Actions triggers
|
|
2. ✅ Builds new Docker image
|
|
3. ✅ Tags with new version (e.g., `v20260316-143022`)
|
|
4. ✅ Pushes to container registry
|
|
5. ✅ Updates `helm-values/myapp/values.yaml` with new tag
|
|
6. ✅ ArgoCD detects change
|
|
7. ✅ Syncs new version to cluster
|
|
8. ✅ Sends Slack notification
|
|
|
|
**Timeline**: ~5-10 minutes from push to deployment
|
|
|
|
### Method 2: Manual Image Tag Update
|
|
|
|
If CI/CD is not set up, manually update the image tag:
|
|
|
|
```bash
|
|
cd ~/dev/k8s/helm-prod-values
|
|
|
|
# Edit your app's values.yaml
|
|
vim myapp/values.yaml
|
|
|
|
# Change:
|
|
app:
|
|
image:
|
|
tag: v1.0.0 # Old version
|
|
# To:
|
|
app:
|
|
image:
|
|
tag: v1.0.1 # New version
|
|
|
|
# Commit and push
|
|
git add myapp/values.yaml
|
|
git commit -m "Update myapp to v1.0.1"
|
|
git push
|
|
```
|
|
|
|
ArgoCD will sync within 60 seconds.
|
|
|
|
### Method 3: Configuration Changes
|
|
|
|
To update environment variables, resources, or other config:
|
|
|
|
```bash
|
|
cd ~/dev/k8s/helm-prod-values
|
|
vim myapp/values.yaml
|
|
```
|
|
|
|
Example changes:
|
|
|
|
```yaml
|
|
app:
|
|
# Increase resources
|
|
resources:
|
|
requests:
|
|
cpu: 200m # Was 100m
|
|
memory: 256Mi # Was 128Mi
|
|
|
|
# Add new environment variable
|
|
extraEnv:
|
|
- name: API_URL
|
|
value: https://api.example.com
|
|
- name: DEBUG # NEW
|
|
value: "true" # NEW
|
|
|
|
# Enable HPA
|
|
hpa:
|
|
enabled: true # Was false
|
|
minReplicas: 2
|
|
maxReplicas: 10
|
|
```
|
|
|
|
Commit and push:
|
|
```bash
|
|
git add myapp/values.yaml
|
|
git commit -m "Increase myapp resources and enable HPA"
|
|
git push
|
|
```
|
|
|
|
### Method 4: Application Manifest Changes
|
|
|
|
To change ArgoCD sync behavior, namespace, or other meta-config:
|
|
|
|
```bash
|
|
cd ~/dev/k8s/launchpad
|
|
vim apps/myapp.yaml
|
|
```
|
|
|
|
Example changes:
|
|
|
|
```yaml
|
|
spec:
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: false # Disable self-healing temporarily
|
|
```
|
|
|
|
Commit and push:
|
|
```bash
|
|
git add apps/myapp.yaml
|
|
git commit -m "Disable self-healing for myapp"
|
|
git push
|
|
```
|
|
|
|
---
|
|
|
|
## Working with Secrets
|
|
|
|
### Understanding Secret Management
|
|
|
|
**NEVER commit plain secrets to Git.** We use **Sealed Secrets** to encrypt secrets before committing.
|
|
|
|
### Creating a New Secret
|
|
|
|
#### Step 1: Create Plain Secret Locally
|
|
|
|
```bash
|
|
cd ~/dev/k8s/launchpad
|
|
|
|
# Create secret in private/ folder (Git-ignored)
|
|
kubectl create secret generic myapp-credentials \
|
|
--from-literal=API_KEY=your-secret-key-here \
|
|
--from-literal=DB_PASSWORD=super-secret-password \
|
|
--dry-run=client -o yaml > private/myapp-credentials.yaml
|
|
```
|
|
|
|
**DO NOT commit this file!** It's in `private/` which is Git-ignored.
|
|
|
|
#### Step 2: Seal the Secret
|
|
|
|
Get the public certificate (one-time setup):
|
|
|
|
```bash
|
|
# Fetch public cert from cluster
|
|
kubeseal --fetch-cert \
|
|
--controller-name=sealed-secrets-controller \
|
|
--controller-namespace=kube-system \
|
|
> pub-cert.pem
|
|
```
|
|
|
|
Seal your secret:
|
|
|
|
```bash
|
|
kubeseal --format=yaml \
|
|
--cert=pub-cert.pem \
|
|
< private/myapp-credentials.yaml \
|
|
> secrets/myapp-credentials-sealed.yaml
|
|
```
|
|
|
|
#### Step 3: Commit Sealed Secret
|
|
|
|
```bash
|
|
git add secrets/myapp-credentials-sealed.yaml
|
|
git commit -m "Add myapp credentials (sealed)"
|
|
git push
|
|
```
|
|
|
|
#### Step 4: Reference Secret in Application
|
|
|
|
Update your `helm-values/myapp/values.yaml`:
|
|
|
|
```yaml
|
|
app:
|
|
envSecretName: "myapp-credentials" # References the SealedSecret
|
|
```
|
|
|
|
Commit and push:
|
|
```bash
|
|
cd ~/dev/k8s/helm-prod-values
|
|
git add myapp/values.yaml
|
|
git commit -m "Reference myapp credentials"
|
|
git push
|
|
```
|
|
|
|
### Updating a Secret
|
|
|
|
To update an existing secret:
|
|
|
|
```bash
|
|
# 1. Create new version of secret
|
|
kubectl create secret generic myapp-credentials \
|
|
--from-literal=API_KEY=new-key-here \
|
|
--from-literal=DB_PASSWORD=new-password \
|
|
--dry-run=client -o yaml > private/myapp-credentials.yaml
|
|
|
|
# 2. Seal it
|
|
kubeseal --format=yaml \
|
|
--cert=pub-cert.pem \
|
|
< private/myapp-credentials.yaml \
|
|
> secrets/myapp-credentials-sealed.yaml
|
|
|
|
# 3. Commit sealed version
|
|
git add secrets/myapp-credentials-sealed.yaml
|
|
git commit -m "Update myapp credentials"
|
|
git push
|
|
|
|
# 4. Restart pods to pick up new secret
|
|
kubectl rollout restart deployment myapp -n myapp
|
|
```
|
|
|
|
### Secret Best Practices
|
|
|
|
✅ **DO**:
|
|
- Store secrets in `private/` folder locally
|
|
- Always seal secrets before committing
|
|
- Delete plain secrets after sealing
|
|
- Use meaningful secret names
|
|
- Document what each secret contains
|
|
|
|
❌ **DON'T**:
|
|
- Commit plain secrets to Git
|
|
- Share secrets via Slack/email
|
|
- Hard-code secrets in code
|
|
- Use the same secret across multiple environments
|
|
- Store secrets in Docker images
|
|
|
|
### Where Secrets Are Stored
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Location │ Content │ Committed?│
|
|
├──────────────────────────┼────────────────────┼────────────┤
|
|
│ private/ │ Plain secrets │ ❌ NO │
|
|
│ secrets/ │ Sealed secrets │ ✅ YES │
|
|
│ Kubernetes cluster │ Unsealed secrets │ N/A │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**Sealed Secrets Controller** in the cluster decrypts sealed secrets automatically.
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Application Not Deploying
|
|
|
|
#### Problem: Application stuck in "Syncing" state
|
|
|
|
**Check ArgoCD status:**
|
|
```bash
|
|
kubectl get application myapp -n argocd -o yaml
|
|
```
|
|
|
|
Look for errors in `status.conditions`.
|
|
|
|
**Common causes:**
|
|
- ❌ Image doesn't exist or is not accessible
|
|
- ❌ Invalid YAML syntax
|
|
- ❌ Resource quota exceeded
|
|
- ❌ Namespace conflicts
|
|
- ❌ Invalid Helm values
|
|
|
|
**Solutions:**
|
|
```bash
|
|
# Check image exists
|
|
docker pull ghcr.io/fortedigital/myapp:v1.0.0
|
|
|
|
# Validate YAML syntax
|
|
kubectl apply --dry-run=client -f apps/myapp.yaml
|
|
|
|
# Check ArgoCD logs
|
|
kubectl logs -n argocd deployment/argocd-application-controller | grep myapp
|
|
```
|
|
|
|
#### Problem: Pods crashing (CrashLoopBackOff)
|
|
|
|
**Check pod logs:**
|
|
```bash
|
|
kubectl get pods -n myapp
|
|
kubectl logs -n myapp <pod-name>
|
|
kubectl describe pod -n myapp <pod-name>
|
|
```
|
|
|
|
**Common causes:**
|
|
- ❌ Application error (check logs)
|
|
- ❌ Missing environment variables
|
|
- ❌ Incorrect port configuration
|
|
- ❌ Missing secrets
|
|
- ❌ Insufficient resources
|
|
|
|
**Solutions:**
|
|
```bash
|
|
# Check environment variables
|
|
kubectl exec -n myapp <pod-name> -- env
|
|
|
|
# Check if secrets exist
|
|
kubectl get secrets -n myapp
|
|
|
|
# Increase resources in helm-values
|
|
vim ~/dev/k8s/helm-prod-values/myapp/values.yaml
|
|
```
|
|
|
|
#### Problem: Application not accessible via domain
|
|
|
|
**Check ingress:**
|
|
```bash
|
|
kubectl get ingressroute -n myapp
|
|
kubectl describe ingressroute myapp -n myapp
|
|
```
|
|
|
|
**Common causes:**
|
|
- ❌ DNS not configured
|
|
- ❌ TLS certificate not issued
|
|
- ❌ Incorrect domain in values.yaml
|
|
- ❌ Traefik not routing correctly
|
|
|
|
**Solutions:**
|
|
```bash
|
|
# Check certificate
|
|
kubectl get certificate -n myapp
|
|
|
|
# Check cert-manager logs
|
|
kubectl logs -n cert-manager deployment/cert-manager
|
|
|
|
# Verify domain configuration
|
|
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep host
|
|
|
|
# Test with port-forward
|
|
kubectl port-forward -n myapp service/myapp 8080:3000
|
|
curl http://localhost:8080
|
|
```
|
|
|
|
### Secret Issues
|
|
|
|
#### Problem: Secret not found
|
|
|
|
**Check if SealedSecret exists:**
|
|
```bash
|
|
kubectl get sealedsecret -n myapp
|
|
kubectl get secret -n myapp
|
|
```
|
|
|
|
**Solutions:**
|
|
```bash
|
|
# Check if secret is in Git
|
|
ls -l secrets/myapp-credentials-sealed.yaml
|
|
|
|
# Re-apply sealed secret
|
|
kubectl apply -f secrets/myapp-credentials-sealed.yaml
|
|
|
|
# Check sealed-secrets-controller logs
|
|
kubectl logs -n kube-system deployment/sealed-secrets-controller
|
|
```
|
|
|
|
#### Problem: Secret exists but pods can't access it
|
|
|
|
**Check pod events:**
|
|
```bash
|
|
kubectl describe pod -n myapp <pod-name>
|
|
```
|
|
|
|
Look for: `Error: secret "myapp-credentials" not found`
|
|
|
|
**Solutions:**
|
|
```bash
|
|
# Verify secret name in values.yaml matches actual secret
|
|
cat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep envSecretName
|
|
kubectl get secrets -n myapp
|
|
|
|
# Restart pods
|
|
kubectl rollout restart deployment myapp -n myapp
|
|
```
|
|
|
|
### Sync Failures
|
|
|
|
#### Problem: ArgoCD shows "Out of Sync"
|
|
|
|
**Manual sync:**
|
|
```bash
|
|
# Using kubectl
|
|
kubectl patch application myapp -n argocd --type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"syncStrategy":{"hook":{}}}}}'
|
|
|
|
# Or via ArgoCD UI
|
|
# Click "Sync" button in UI
|
|
```
|
|
|
|
**Check what's different:**
|
|
```bash
|
|
kubectl get application myapp -n argocd -o yaml
|
|
```
|
|
|
|
Look at `status.sync.comparedTo` vs desired state.
|
|
|
|
#### Problem: Sync succeeds but application is "Degraded"
|
|
|
|
**Check resource health:**
|
|
```bash
|
|
kubectl get application myapp -n argocd -o jsonpath='{.status.resources[*].health}'
|
|
```
|
|
|
|
**Common causes:**
|
|
- ❌ Pods not ready
|
|
- ❌ Deployments not at desired replica count
|
|
- ❌ Jobs failed
|
|
|
|
**Solutions:**
|
|
```bash
|
|
# Check all resources in namespace
|
|
kubectl get all -n myapp
|
|
|
|
# Check pod events
|
|
kubectl get events -n myapp --sort-by='.lastTimestamp'
|
|
```
|
|
|
|
### Getting Help
|
|
|
|
If you're stuck:
|
|
|
|
1. **Check Slack notifications** - Error details are often in sync failure messages
|
|
2. **Check ArgoCD UI** - Visual representation of what's wrong
|
|
3. **Ask platform team** - They have full cluster access and can debug further
|
|
4. **Check documentation** - [Operations Runbook](OPERATIONS-RUNBOOK.md) has more troubleshooting
|
|
|
|
---
|
|
|
|
## Best Practices
|
|
|
|
### Development Workflow
|
|
|
|
✅ **DO**:
|
|
- Develop and test locally with Docker Compose
|
|
- Use semantic versioning for releases
|
|
- Write descriptive commit messages
|
|
- Test changes in a separate namespace first (if possible)
|
|
- Monitor Slack for deployment notifications
|
|
- Document environment variables and configuration
|
|
|
|
❌ **DON'T**:
|
|
- Push directly to production without testing
|
|
- Use `latest` tag for Docker images
|
|
- Bypass CI/CD for "quick fixes"
|
|
- Hard-code configuration values
|
|
- Ignore deployment failures
|
|
|
|
### Configuration Management
|
|
|
|
✅ **DO**:
|
|
- Keep configuration in `helm-values` repository
|
|
- Use environment variables for config
|
|
- Document what each value does
|
|
- Use reasonable resource limits
|
|
- Enable ingress and TLS for public services
|
|
|
|
❌ **DON'T**:
|
|
- Hard-code config in application code
|
|
- Over-allocate resources (wastes money)
|
|
- Under-allocate resources (causes crashes)
|
|
- Use HTTP for production services
|
|
|
|
### Secret Management
|
|
|
|
✅ **DO**:
|
|
- Use kubeseal for all secrets
|
|
- Store plain secrets in password manager
|
|
- Rotate secrets regularly
|
|
- Use different secrets per environment
|
|
- Document what each secret contains
|
|
|
|
❌ **DON'T**:
|
|
- Commit plain secrets
|
|
- Share secrets in Slack/email
|
|
- Reuse secrets across apps
|
|
- Log secrets in application code
|
|
|
|
### Git Workflow
|
|
|
|
✅ **DO**:
|
|
- Use feature branches for changes
|
|
- Write clear commit messages
|
|
- Use pull requests for review
|
|
- Keep commits atomic and focused
|
|
- Tag releases in application repos
|
|
|
|
❌ **DON'T**:
|
|
- Push directly to `main` without review (for config repos)
|
|
- Make multiple unrelated changes in one commit
|
|
- Use vague commit messages ("fix", "update")
|
|
- Force-push to main branches
|
|
|
|
---
|
|
|
|
## Quick Reference
|
|
|
|
### Common Commands
|
|
|
|
```bash
|
|
# Check application status
|
|
kubectl get application myapp -n argocd
|
|
|
|
# View application details
|
|
kubectl describe application myapp -n argocd
|
|
|
|
# Check pods
|
|
kubectl get pods -n myapp
|
|
|
|
# View pod logs
|
|
kubectl logs -n myapp <pod-name>
|
|
|
|
# Restart deployment
|
|
kubectl rollout restart deployment myapp -n myapp
|
|
|
|
# Port-forward to service
|
|
kubectl port-forward -n myapp service/myapp 8080:3000
|
|
|
|
# Create secret
|
|
kubectl create secret generic myapp-credentials \
|
|
--from-literal=KEY=value \
|
|
--dry-run=client -o yaml > private/myapp-credentials.yaml
|
|
|
|
# Seal secret
|
|
kubeseal --format=yaml \
|
|
--cert=pub-cert.pem \
|
|
< private/myapp-credentials.yaml \
|
|
> secrets/myapp-credentials-sealed.yaml
|
|
```
|
|
|
|
### Repository Locations
|
|
|
|
```bash
|
|
# Config repository
|
|
cd ~/dev/k8s/launchpad
|
|
|
|
# Helm values repository
|
|
cd ~/dev/k8s/helm-prod-values
|
|
|
|
# Helm charts repository
|
|
cd ~/dev/k8s/forte-helm
|
|
```
|
|
|
|
### File Paths
|
|
|
|
```bash
|
|
# New application manifest
|
|
~/dev/k8s/launchpad/apps/myapp.yaml
|
|
|
|
# Application values
|
|
~/dev/k8s/helm-prod-values/myapp/values.yaml
|
|
|
|
# Sealed secrets
|
|
~/dev/k8s/launchpad/secrets/myapp-credentials-sealed.yaml
|
|
|
|
# Plain secrets (local only)
|
|
~/dev/k8s/launchpad/private/myapp-credentials.yaml
|
|
```
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
Now that you understand the basics:
|
|
|
|
1. ✅ Deploy your first application (follow steps above)
|
|
2. 📖 Read the [Operations Runbook](OPERATIONS-RUNBOOK.md) for common tasks
|
|
3. 📖 Review [Technical Reference](REFERENCE.md) for detailed component docs
|
|
4. 📖 Understand [GitOps Architecture](GITOPS-ARCHITECTURE.md) for the big picture
|
|
5. 🚀 Start contributing!
|
|
|
|
---
|
|
|
|
**Questions?**
|
|
- Slack: #platform-support
|
|
- Docs: [Full documentation index](README.md)
|
|
- Help: Contact platform team
|
|
|
|
**Last Updated**: 2026-03-16
|