Kubernetes Cluster - GitOps Configuration
Kubernetes cluster bootstrapping and GitOps configuration repository using ArgoCD for multi-cloud Kubernetes (UpCloud, AWS EKS, Azure AKS, GCP GKE)
📚 Complete Documentation
New developers and operators: Please refer to our comprehensive documentation for detailed guides and references:
🎯 START HERE: Documentation Index
| Document | Description | Audience |
|---|---|---|
| GitOps Architecture | System architecture, repository structure, GitOps workflows, security model | Everyone (start here) |
| Developer Guide | Local setup, deploying apps, managing secrets, troubleshooting | Developers |
| Operations Runbook | Cluster bootstrap, day-to-day operations, incident response, maintenance | Platform Engineers, SREs |
| Technical Reference | Component specs, Helm charts, ArgoCD config, Kyverno policies, API docs | Everyone (reference) |
🚀 Quick Start
For New Developers
# 1. Clone repositories
git clone https://git.forteapps.net/Forte/launchpad.git
git clone ssh://git@git.forteapps.net:2222/Forte/helm-prod-values.git
# 2. Read the guides
# - Start: docs/GITOPS-ARCHITECTURE.md
# - Follow: docs/DEVELOPER-GUIDE.md
# 3. Deploy your first app (see Developer Guide)
For Operators
# 1. Bootstrap new cluster
./bootstrap.sh
# 2. Verify deployment
kubectl get applications -n argocd
kubectl get pods --all-namespaces
# 3. Read Operations Runbook for day-to-day tasks
📋 Overview
This repository contains the complete GitOps configuration for our Kubernetes cluster, using the App-of-Apps pattern with ArgoCD.
What's Inside
- Infrastructure Applications: Traefik, Cert-Manager, Kyverno, Prometheus, Grafana, Loki, Tempo, Sealed Secrets
- Business Applications: MCP10X, MusicMan, Dot-AI Stack, ArgoCD MCP
- Policies: Kyverno security policies for secret management, namespace controls, pod verification
- Monitoring: Full observability stack with metrics, logs, traces, and alerting
- Secrets: Sealed Secrets for secure Git storage
Key Features
✅ GitOps-Native: Git is the single source of truth ✅ Auto-Sync: Changes automatically deployed (60s reconciliation) ✅ Self-Healing: Manual cluster changes are reverted ✅ Multi-Source: Separate chart templates from configuration ✅ Policy Enforcement: Kyverno ensures security and compliance ✅ Authentication: Automatic sidecar injection (token & OIDC support) ✅ TLS Everywhere: Automatic Let's Encrypt certificates ✅ Full Observability: Prometheus, Grafana, Loki, Tempo integration
🗂️ Repository Structure
.
├── bootstrap.sh # Cluster initialization script
├── _app-of-apps.yaml # Root ArgoCD Application (App-of-Apps pattern)
│
├── infra/ # Infrastructure ArgoCD Applications (Kustomize multi-cluster)
│ ├── base/ # Base ArgoCD Application manifests (EU defaults)
│ │ ├── kustomization.yaml
│ │ ├── traefik-application.yaml
│ │ ├── keycloak.yaml
│ │ ├── grafana.yaml
│ │ ├── gitea.yaml
│ │ ├── gitea-actions.yaml
│ │ ├── tempo.yaml
│ │ ├── renovate.yaml
│ │ ├── ... # All other Application manifests
│ │ └── secrets.yaml
│ ├── overlays/ # Per-cluster overrides (Kustomize)
│ │ ├── upc-dev/ # UpCloud Dev (uses base as-is)
│ │ ├── upc-prod/ # UpCloud Prod (patches value paths)
│ │ ├── eks-dev/ # AWS EKS Dev
│ │ ├── eks-prod/ # AWS EKS Prod
│ │ ├── aks-dev/ # Azure AKS Dev
│ │ ├── aks-prod/ # Azure AKS Prod
│ │ ├── gke-dev/ # GCP GKE Dev
│ │ └── gke-prod/ # GCP GKE Prod
│ ├── dashboards/ # Grafana dashboard ConfigMaps
│ └── values/ # Helm value overrides
│ ├── base/ # Shared cloud-agnostic values
│ ├── upc-dev/ # UpCloud Dev (storage, LB, pricing)
│ ├── upc-prod/ # UpCloud Prod
│ ├── eks-dev/ # AWS EKS Dev
│ ├── eks-prod/ # AWS EKS Prod
│ ├── aks-dev/ # Azure AKS Dev
│ ├── aks-prod/ # Azure AKS Prod
│ ├── gke-dev/ # GCP GKE Dev
│ └── gke-prod/ # GCP GKE Prod
│
├── apps/ # Business Applications
│ ├── mcp10x.yaml
│ ├── musicman.yaml
│ ├── dot-ai-stack.yaml
│ └── argo-mcp.yaml
│
├── cluster-resources/ # Cluster-wide Kubernetes resources
│ ├── letsencrypt-issuer.yaml
│ ├── kyverno-config.yaml
│ ├── *-sealed.yaml # Sealed secrets
│ └── policies/ # Kyverno policies
│ ├── secret-cloner.yaml
│ ├── default-ns-blocker.yaml
│ ├── bare-pod-cleaner.yaml
│ └── auth-sidecar-injector.yaml
│
├── secrets/ # Application secrets (sealed)
│ └── *-credentials-sealed.yaml
│
├── private/ # Local-only files (Git-ignored)
│ └── *.yaml # Unsealed secrets (never committed)
│
└── docs/ # 📚 Comprehensive documentation
├── README.md # Documentation index
├── GITOPS-ARCHITECTURE.md # Architecture guide
├── DEVELOPER-GUIDE.md # Developer onboarding
├── OPERATIONS-RUNBOOK.md # Operations procedures
└── REFERENCE.md # Technical reference
See GitOps Architecture - Repository Structure for detailed explanation.
🏗️ Architecture
Three-Repository Pattern
| Repository | Purpose | Who Edits | How Often |
|---|---|---|---|
| launchpad (this repo) | ArgoCD Applications, cluster resources | Platform / DevOps engineers | ✅ Often |
| forte-helm | Generic Helm chart templates | Platform engineers | ❌ Rarely |
| helm-prod-values | App-specific configuration & versions | Developers / CI pipelines | ✅ Sometimes |
GitOps Workflow
Developer commits code → CI/CD builds image → Updates helm-prod-values → ArgoCD syncs → Deployed to cluster
Learn more: GitOps Architecture - GitOps Workflow
🔧 Common Tasks
Deploy a New Application
See detailed guide: Developer Guide - Deploying Your First Application
Quick version:
- Create
apps/myapp.yaml(ArgoCD Application manifest) - Create
helm-prod-values/myapp/values.yaml(configuration) - Create sealed secrets if needed
- Commit and push - ArgoCD auto-syncs!
Update an Existing Application
See detailed guide: Developer Guide - Updating an Existing Application
Quick version:
- Update code: Push to app repo → CI/CD updates image tag in helm-prod-values
- Update config: Edit
helm-prod-values/myapp/values.yaml→ commit → push
Manage Secrets
See detailed guide: Developer Guide - Working with Secrets
# Create plain secret
kubectl create secret generic myapp-creds \
--from-literal=KEY=value \
--dry-run=client -o yaml > private/myapp-creds.yaml
# Seal it
kubeseal --format=yaml --cert=pub-cert.pem \
< private/myapp-creds.yaml > secrets/myapp-creds-sealed.yaml
# Commit sealed version
git add secrets/myapp-creds-sealed.yaml
git commit -m "Add myapp credentials"
git push
Enable Authentication
See detailed guide: Developer Guide - Enabling Authentication
Quick version:
# In helm-prod-values/myapp/values.yaml
# Token-based auth (simple)
auth:
enabled: true
type: token
tokens:
- your-secret-token-here
# OIDC auth (SSO)
auth:
enabled: true
type: oidc
oidc:
authority: https://auth.example.com/realms/master
clientId: myapp
Then create OIDC secret (if using OIDC):
kubectl create secret generic auth-oidc \
--from-literal=client-secret=your-oidc-secret \
--from-literal=cookie-secret=$(openssl rand -hex 32) \
--namespace=myapp | \
kubeseal --format=yaml --cert=pub-cert.pem --namespace=myapp | \
kubectl apply -f -
Bootstrap Cluster
See detailed guide: Operations Runbook - Cluster Bootstrap
# Initialize new cluster
./bootstrap.sh
# Verify
kubectl get applications -n argocd
kubectl get pods --all-namespaces
🛠️ Quick Reference
Monitor Applications
# List all ArgoCD applications
kubectl get applications -n argocd
# Watch sync status
kubectl get applications -n argocd -w
# Check specific application
kubectl describe application myapp -n argocd
# View application logs
kubectl logs -n myapp <pod-name>
Access UIs
# ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443
# Access: https://localhost:8080 (no auth required)
# Grafana
kubectl port-forward -n monitoring svc/grafana 3000:80
# Access: http://localhost:3000
# Prometheus
kubectl port-forward -n monitoring svc/prometheus-server 9090:80
# Access: http://localhost:9090
Troubleshooting
# Check pod status
kubectl get pods -n myapp
# View pod logs
kubectl logs -n myapp <pod-name>
# Check pod events
kubectl describe pod -n myapp <pod-name>
# Check ArgoCD sync errors
kubectl describe application myapp -n argocd
# Force sync
kubectl patch application myapp -n argocd \
--type merge -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'
Full troubleshooting guide: Developer Guide - Troubleshooting
🔐 Security
Secret Management
- ✅ Sealed Secrets for Git storage
- ✅ Kyverno auto-clones secrets to namespaces
- ❌ Never commit plain secrets
Network Security
- ✅ All traffic TLS-encrypted (Let's Encrypt)
- ✅ HTTP → HTTPS redirect
- ✅ Traefik IngressRoute per application
Policy Enforcement
- ✅ Kyverno policies for security
- ✅ Default namespace blocked
- ✅ Bare pods not allowed
- ✅ Optional authentication sidecar injection
Learn more: GitOps Architecture - Security Model
📊 Infrastructure Components
| Component | Purpose | Namespace | Replicas |
|---|---|---|---|
| ArgoCD | GitOps controller | argocd |
1 |
| Traefik | Ingress controller | traefik |
2 |
| Cert-Manager | TLS certificates | cert-manager |
1 |
| Kyverno | Policy engine | kyverno |
1 |
| Sealed Secrets | Secret encryption | kube-system |
1 |
| Prometheus | Metrics | monitoring |
1 |
| Grafana | Dashboards | monitoring |
1 |
| Loki | Logs | monitoring |
1 |
| Tempo | Distributed tracing | monitoring |
1 |
| Fluent-Bit | Log shipping | monitoring |
DaemonSet |
| OpenCost | Cost monitoring | monitoring |
1 |
| Renovate | Dependency updates | renovate |
CronJob |
Full specs: Technical Reference - Infrastructure Components
🌐 Domains & Networking
- Local development:
*.127.0.0.1.nip.io - Production:
*.forteapps.net - DNS: Manual configuration (contact platform team)
- TLS: Automatic via Let's Encrypt
📖 Key Concepts
App-of-Apps Pattern
_app-of-apps-{cluster}.yaml is the root Application that manages all other Applications in infra/. Kustomize overlays in infra/overlays/{cluster}/ render the base Applications with per-cluster patches (e.g., swapping value file paths). Supported clusters: upc-dev, upc-prod, eks-dev, eks-prod, aks-dev, aks-prod, gke-dev, gke-prod.
Multi-Source Pattern
Applications reference both:
- Helm charts from
forte-helm(templates) - Values from
helm-prod-values(configuration)
This separates reusable templates from environment-specific config.
Sync Waves
Applications deploy in order using argocd.argoproj.io/sync-wave:
- Wave
-1: Namespaces - Wave
0: Kyverno (policies) - Wave
1: Infrastructure - Wave
2+: Applications
Auto-Sync & Self-Heal
- Auto-Sync: ArgoCD automatically deploys Git changes (60s polling)
- Self-Heal: Manual cluster changes are reverted to match Git
- Prune: Deleted resources in Git are removed from cluster
Learn more: GitOps Architecture - GitOps Workflow
⚙️ Configuration
ArgoCD Settings
- Reconciliation: Every 60 seconds
- Sync timeout: 5 minutes per application
- Retry policy: 5 attempts with exponential backoff
- Authentication: Disabled (internal use only)
Application Defaults
- Auto-sync: Enabled
- Self-heal: Enabled
- Prune: Enabled
- Validation: Server-side validation enabled
- Server-side apply: Enabled
Full configuration: Technical Reference - ArgoCD Configuration
🆘 Getting Help
Documentation
- Start here: Documentation Index
- For development: Developer Guide
- For operations: Operations Runbook
- For reference: Technical Reference
Support
- Slack: #platform-support
- Issues: Contact platform team
- Emergencies: Escalate via Slack
Common Questions
| Question | Answer |
|---|---|
| How do I deploy an app? | Developer Guide - Deploying Your First Application |
| How do I manage secrets? | Developer Guide - Working with Secrets |
| App won't sync? | Developer Guide - Troubleshooting |
| How do I bootstrap a cluster? | Operations Runbook - Cluster Bootstrap |
| Where are the logs? | Operations Runbook - Monitoring & Alerting |
🤝 Contributing
Adding a New Application
- Read Developer Guide - Deploying Your First Application
- Create ArgoCD Application manifest in
apps/ - Create Helm values in
helm-prod-values/ - Create sealed secrets if needed
- Commit and push - ArgoCD handles the rest!
Modifying Infrastructure
- Read Operations Runbook
- Update relevant files in
infra/orcluster-resources/ - Test changes in isolated namespace if possible
- Commit and push
- Monitor sync status in Slack/ArgoCD UI
Updating Documentation
Documentation lives in docs/. To update:
- Edit relevant markdown file
- Update "Last Updated" date
- Submit PR or push directly
- Notify team of significant changes
📝 Notes
Current Environment
- Provider: Multi-cloud (UpCloud, AWS EKS, Azure AKS, GCP GKE)
- Active clusters: UpCloud (upc-dev, upc-prod)
- Environment: Production (internal use only)
- Auth: Disabled for ArgoCD (internal access)
- Backup: Gitea daily backup to S3-compatible storage
Known Limitations
- Secret rotation not automated
- DNS management is manual
Future improvements: See Operations Runbook - Disaster Recovery
📚 Additional Resources
External Documentation
- ArgoCD Documentation
- Kyverno Documentation
- Traefik Documentation
- Cert-Manager Documentation
- Grafana Tempo Documentation
- Sealed Secrets
Related Repositories
- forte-helm - Helm chart templates
- helm-prod-values - Application values
📄 License
Internal use only. Not for public distribution.
👥 Maintainers
Platform Team
- Contact: #platform-support on Slack
- Issues: Create issue in repository or contact team directly
Last Updated: 2026-04-22 Documentation Version: 1.0.0
🚀 Ready to get started? Check out the Documentation Index!