1 line
208 KiB
JSON
1 line
208 KiB
JSON
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"],"fields":{"title":{"boost":1000.0},"text":{"boost":1.0},"tags":{"boost":1000000.0}}},"docs":[{"location":"","title":"Kubernetes Cluster Documentation","text":"<p>Welcome to the comprehensive documentation for our Kubernetes cluster GitOps setup. This documentation covers architecture, development workflows, operations, and technical references.</p>"},{"location":"#documentation-index","title":"\ud83d\udcda Documentation Index","text":""},{"location":"#1-gitops-architecture-repository-guide","title":"1. GitOps Architecture & Repository Guide","text":"<p>Start here to understand the system</p> <p>Learn about: - Overall architecture and design decisions - Repository structure and relationships - GitOps workflow and deployment patterns - CI/CD pipeline integration - Security model and best practices</p> <p>Best for: Understanding how everything fits together, architectural decisions, and the big picture.</p>"},{"location":"#2-developer-onboarding-guide","title":"2. Developer Onboarding Guide","text":"<p>For developers deploying and maintaining applications</p> <p>Learn how to: - Set up your local development environment - Deploy your first application - Update existing applications - Manage secrets securely - Troubleshoot common issues - Follow development best practices</p> <p>Best for: New developers joining the team, deploying applications, day-to-day development workflows.</p>"},{"location":"#3-operations-runbook","title":"3. Operations Runbook","text":"<p>For platform engineers and operators</p> <p>Learn how to: - Bootstrap a new cluster - Monitor and maintain applications - Manage infrastructure components - Handle secrets and credentials - Troubleshoot production issues - Perform disaster recovery - Execute maintenance procedures</p> <p>Best for: Platform team members, SRE tasks, incident response, cluster maintenance.</p>"},{"location":"#4-technical-reference","title":"4. Technical Reference","text":"<p>Detailed technical specifications</p> <p>Reference for: - Component specifications and versions - Helm chart templates and values - ArgoCD configuration options - Kyverno policy definitions - API endpoints and interfaces - Configuration schemas - Complete glossary</p> <p>Best for: Looking up specific configuration options, understanding component details, API references.</p>"},{"location":"#quick-start","title":"\ud83d\ude80 Quick Start","text":""},{"location":"#for-new-developers","title":"For New Developers","text":"<ol> <li>Read GitOps Architecture to understand the system</li> <li>Follow Developer Guide - Prerequisites to set up your environment</li> <li>Deploy your first application using Deploying Your First Application</li> </ol>"},{"location":"#for-platform-engineers","title":"For Platform Engineers","text":"<ol> <li>Understand the architecture in GitOps Architecture</li> <li>Learn cluster bootstrap in Operations Runbook - Cluster Bootstrap</li> <li>Review Day-to-Day Operations procedures</li> </ol>"},{"location":"#for-troubleshooting","title":"For Troubleshooting","text":"<ol> <li>Check Developer Guide - Troubleshooting for common developer issues</li> <li>Check Operations Runbook - Troubleshooting for operational issues</li> <li>Consult Technical Reference for configuration details</li> </ol>"},{"location":"#documentation-map","title":"\ud83d\uddfa\ufe0f Documentation Map","text":"<pre><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 GITOPS ARCHITECTURE \u2502\n\u2502 (System Overview, Repositories, Workflows, Security) \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502\n \u25bc \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 DEVELOPER GUIDE \u2502 \u2502 OPERATIONS RUNBOOK \u2502\n \u2502 (Development) \u2502 \u2502 (Operations) \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 TECHNICAL REFERENCE\u2502\n \u2502 (Specifications) \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n</code></pre>"},{"location":"#reading-paths","title":"\ud83d\udcd6 Reading Paths","text":""},{"location":"#path-1-new-developer-no-k8s-experience","title":"Path 1: New Developer (No K8s Experience)","text":"<ol> <li>GitOps Architecture - Overview</li> <li>GitOps Architecture - GitOps Workflow</li> <li>Developer Guide - Understanding the Workflow</li> <li>Developer Guide - Deploying Your First Application</li> <li>Developer Guide - Troubleshooting</li> </ol>"},{"location":"#path-2-experienced-developer-has-k8s-experience","title":"Path 2: Experienced Developer (Has K8s Experience)","text":"<ol> <li>GitOps Architecture - Repository Structure</li> <li>Developer Guide - Local Development Setup</li> <li>Developer Guide - Deploying Your First Application</li> <li>Technical Reference - Helm Chart Reference</li> </ol>"},{"location":"#path-3-platform-engineer-sre","title":"Path 3: Platform Engineer / SRE","text":"<ol> <li>GitOps Architecture (entire document)</li> <li>Operations Runbook - Cluster Bootstrap</li> <li>Operations Runbook - Day-to-Day Operations</li> <li>Operations Runbook - Troubleshooting</li> <li>Technical Reference (as needed)</li> </ol>"},{"location":"#path-4-quick-reference","title":"Path 4: Quick Reference","text":"<ol> <li>Developer Guide - Quick Reference</li> <li>Technical Reference - Configuration Reference</li> <li>Technical Reference - Glossary</li> </ol>"},{"location":"#finding-information","title":"\ud83d\udd0d Finding Information","text":""},{"location":"#how-do-i","title":"How do I...?","text":"Task Documentation Deploy a new application Developer Guide - Deploying Your First Application Update an existing application Developer Guide - Updating an Existing Application Create and seal secrets Developer Guide - Working with Secrets Troubleshoot deployment issues Developer Guide - Troubleshooting Bootstrap a new cluster Operations Runbook - Cluster Bootstrap Scale an application Operations Runbook - Scaling Applications Roll back a deployment Operations Runbook - Rolling Back Deployments Manage monitoring Operations Runbook - Monitoring & Alerting Understand ArgoCD config Technical Reference - ArgoCD Configuration Look up Helm values Technical Reference - Helm Chart Reference Find component versions Technical Reference - Version Matrix"},{"location":"#system-overview","title":"\ud83d\udcca System Overview","text":""},{"location":"#cluster-architecture","title":"Cluster Architecture","text":"<pre><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 GitHub Repositories \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 Config \u2502 \u2502 Charts \u2502 \u2502 Values \u2502 \u2502\n\u2502 \u2502 (ArgoCD) \u2502 \u2502 (Templates)\u2502 \u2502 (Environment Config) \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 ArgoCD (GitOps Engine) \u2502\n\u2502 Sync every 60 seconds \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Kubernetes Clusters (UpCloud: upc-dev, upc-prod) \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 Infrastructure: Traefik, Cert-Manager, Kyverno \u2502 \u2502\n\u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502\n\u2502 \u2502 Monitoring: Prometheus, Grafana, Loki, Fluent-Bit \u2502 \u2502\n\u2502 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524 \u2502\n\u2502 \u2502 Applications: mcp10x, musicman, dot-ai-stack \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n</code></pre>"},{"location":"#key-technologies","title":"Key Technologies","text":"<ul> <li>GitOps: ArgoCD</li> <li>Kubernetes: UpCloud Managed Kubernetes (multi-cluster: upc-dev, upc-prod)</li> <li>Ingress: Traefik v2</li> <li>Certificates: Cert-Manager + Let's Encrypt</li> <li>Policies: Kyverno</li> <li>Secrets: Sealed Secrets</li> <li>Monitoring: Prometheus + Grafana</li> <li>Logging: Loki + Fluent-Bit</li> </ul>"},{"location":"#common-tasks","title":"\ud83d\udee0\ufe0f Common Tasks","text":""},{"location":"#development-tasks","title":"Development Tasks","text":"<pre><code># Deploy new application\ncd ~/dev/k8s/launchpad\n# Create apps/myapp.yaml and helm-prod-values/myapp/values.yaml\ngit add apps/myapp.yaml\ngit commit -m \"Add myapp\"\ngit push\n\n# Update application\ncd ~/dev/k8s/helm-prod-values\nvim myapp/values.yaml\ngit commit -am \"Update myapp config\"\ngit push\n\n# Create secret\nkubeseal --format=yaml --cert=pub-cert.pem \\\n < private/secret.yaml > secrets/secret-sealed.yaml\ngit add secrets/secret-sealed.yaml\ngit push\n</code></pre>"},{"location":"#operations-tasks","title":"Operations Tasks","text":"<pre><code># Check application status\nkubectl get applications -n argocd\n\n# View application details\nkubectl describe application myapp -n argocd\n\n# Force sync\nkubectl patch application myapp -n argocd \\\n --type merge -p '{\"metadata\":{\"annotations\":{\"argocd.argoproj.io/refresh\":\"hard\"}}}'\n\n# Check pod logs\nkubectl logs -n myapp <pod-name>\n\n# Restart deployment\nkubectl rollout restart deployment myapp -n myapp\n</code></pre>"},{"location":"#getting-help","title":"\ud83c\udd98 Getting Help","text":""},{"location":"#documentation-search-order","title":"Documentation Search Order","text":"<ol> <li>Quick Reference: Developer Guide - Quick Reference</li> <li>Troubleshooting: Developer Guide - Troubleshooting or Operations Runbook - Troubleshooting</li> <li>Technical Details: Technical Reference</li> <li>Architecture Context: GitOps Architecture</li> </ol>"},{"location":"#support-channels","title":"Support Channels","text":"<ul> <li>Slack: #platform-support</li> <li>Issues: Platform team</li> <li>Emergencies: Escalate via Slack</li> </ul>"},{"location":"#document-maintenance","title":"\ud83d\udcdd Document Maintenance","text":""},{"location":"#updating-documentation","title":"Updating Documentation","text":"<p>If you find: - Outdated information - Missing procedures - Errors or typos - Areas needing clarification</p> <p>Please: 1. Create an issue or PR in the repository 2. Notify the platform team 3. Update the relevant documentation file</p>"},{"location":"#documentation-structure","title":"Documentation Structure","text":"<pre><code>docs/\n\u251c\u2500\u2500 README.md # This file (index)\n\u251c\u2500\u2500 GITOPS-ARCHITECTURE.md # Architecture overview\n\u251c\u2500\u2500 DEVELOPER-GUIDE.md # Developer workflows\n\u251c\u2500\u2500 OPERATIONS-RUNBOOK.md # Operations procedures\n\u2514\u2500\u2500 REFERENCE.md # Technical specifications\n</code></pre>"},{"location":"#documentation-versions","title":"\ud83d\udd04 Documentation Versions","text":"<p>Current Version: 1.0.0 Last Updated: 2026-03-16 Maintained By: Platform Team</p>"},{"location":"#changelog","title":"Changelog","text":"<ul> <li>v1.0.0 (2026-03-16): Initial comprehensive documentation release</li> <li>GitOps Architecture guide</li> <li>Developer Onboarding guide</li> <li>Operations Runbook</li> <li>Technical Reference</li> <li>Documentation index</li> </ul>"},{"location":"#next-steps","title":"\ud83c\udfaf Next Steps","text":"<p>Choose your path:</p> <ul> <li>\ud83d\udc68\u200d\ud83d\udcbb New Developer? Start with Developer Guide</li> <li>\ud83d\udd27 Platform Engineer? Read Operations Runbook</li> <li>\ud83c\udfd7\ufe0f Architect? Explore GitOps Architecture</li> <li>\ud83d\udd0d Need Details? Check Technical Reference</li> </ul> <p>Welcome to the team! \ud83d\ude80</p>"},{"location":"DEVELOPER-GUIDE/","title":"Developer Onboarding Guide","text":""},{"location":"DEVELOPER-GUIDE/#table-of-contents","title":"Table of Contents","text":"<ul> <li>Getting Started</li> <li>Prerequisites</li> <li>Local Development Setup</li> <li>Understanding the Workflow</li> <li>Deploying Your First Application</li> <li>Updating an Existing Application</li> <li>Working with Secrets</li> <li>Enabling Authentication for Applications</li> <li>Adding a New Keycloak Client</li> <li>Troubleshooting</li> <li>Documentation</li> <li>Best Practices</li> </ul>"},{"location":"DEVELOPER-GUIDE/#getting-started","title":"Getting Started","text":"<p>Welcome! This guide will help you understand how to develop and deploy applications on our Kubernetes cluster using GitOps principles powered by ArgoCD.</p>"},{"location":"DEVELOPER-GUIDE/#what-youll-learn","title":"What You'll Learn","text":"<ul> <li>How our GitOps architecture works</li> <li>How to deploy a new application</li> <li>How to update existing applications</li> <li>How to manage secrets securely</li> <li>Common troubleshooting techniques</li> </ul>"},{"location":"DEVELOPER-GUIDE/#who-this-guide-is-for","title":"Who This Guide Is For","text":"<ul> <li>Developers deploying new applications</li> <li>Developers maintaining existing applications</li> <li>Team members who need to understand the deployment process</li> </ul>"},{"location":"DEVELOPER-GUIDE/#prerequisites","title":"Prerequisites","text":""},{"location":"DEVELOPER-GUIDE/#required-knowledge","title":"Required Knowledge","text":"<ul> <li>\u2705 Basic Git workflow (clone, commit, push, pull)</li> <li>\u2705 Docker basics (Dockerfile, building images)</li> <li>\u2705 YAML syntax</li> <li>\u2705 Basic understanding of Kubernetes concepts (pods, deployments, services)</li> <li>\u26a0\ufe0f Helm knowledge (helpful but not required - templates are provided)</li> </ul>"},{"location":"DEVELOPER-GUIDE/#required-tools","title":"Required Tools","text":"<p>Most developers do NOT need kubectl access to the cluster. You'll primarily work with Git repositories.</p> <p>If you do need cluster access, install:</p> <ol> <li> <p>kubectl - Kubernetes CLI <pre><code># macOS\nbrew install kubectl\n\n# Windows\nchoco install kubernetes-cli\n\n# Linux\ncurl -LO \"https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl\"\n</code></pre></p> </li> <li> <p>kubeseal - For sealing secrets <pre><code># macOS\nbrew install kubeseal\n\n# Windows\nchoco install kubeseal\n\n# Linux\nwget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz\ntar -xvzf kubeseal-0.24.0-linux-amd64.tar.gz\nsudo mv kubeseal /usr/local/bin/\n</code></pre></p> </li> <li> <p>Git - Version control <pre><code>git --version # Should already be installed\n</code></pre></p> </li> <li> <p>Docker - For local development <pre><code># macOS/Windows: Install Docker Desktop\n# Linux: Install Docker Engine\ndocker --version\n</code></pre></p> </li> </ol>"},{"location":"DEVELOPER-GUIDE/#repository-access","title":"Repository Access","text":"<p>You'll need read/write access to these repositories:</p> <ol> <li> <p>launchpad (Config repo) <pre><code>git clone https://git.forteapps.net/Forte/launchpad.git\ncd launchpad\n</code></pre></p> </li> <li> <p>helm-values (Values repo) <pre><code>git clone https://git.forteapps.net/Forte/helm-prod-values.git\ncd helm-values\n</code></pre></p> </li> <li> <p>forte-helm (Chart repo - read-only for most developers) <pre><code>git clone https://git.forteapps.net/Forte/forte-helm.git\ncd forte-helm\n</code></pre></p> </li> </ol>"},{"location":"DEVELOPER-GUIDE/#cluster-access-if-needed","title":"Cluster Access (If Needed)","text":"<p>If you need kubectl access, ask the platform team for: - Kubeconfig file - Cluster context setup instructions</p> <p>Save to <code>~/.kube/config</code> and verify: <pre><code>kubectl cluster-info\nkubectl get nodes\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#local-development-setup","title":"Local Development Setup","text":""},{"location":"DEVELOPER-GUIDE/#1-clone-the-repositories","title":"1. Clone the Repositories","text":"<p>Set up a consistent folder structure:</p> <pre><code>mkdir -p ~/dev/k8s\ncd ~/dev/k8s\n\n# Clone repositories\ngit clone https://git.forteapps.net/Forte/launchpad.git launchpad\ngit clone https://git.forteapps.net/Forte/helm-prod-values helm-prod-values\ngit clone https://git.forteapps.net/Forte/forte-helm forte-helm\n\n# Your folder structure:\n# ~/dev/k8s/\n# \u251c\u2500\u2500 launchpad/ (Config repo)\n# \u251c\u2500\u2500 helm-prod-values/ (Values repo)\n# \u2514\u2500\u2500 forte-helm/ (Chart repo)\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#2-local-development-environment","title":"2. Local Development Environment","text":"<p>Most applications use Docker Compose for local development:</p> <pre><code># In your application repository\ndocker-compose up\n\n# Or for frontend applications\nnpm install\nnpm run dev\n</code></pre> <p>You DO NOT run applications locally on Kubernetes. Use Docker Compose or native tooling (npm, python, etc.).</p>"},{"location":"DEVELOPER-GUIDE/#3-understanding-the-deployment-flow","title":"3. Understanding the Deployment Flow","text":"<pre><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Step 1: Develop Locally \u2502\n\u2502 - Write code in your application repository \u2502\n\u2502 - Test with Docker Compose or npm/python/etc. \u2502\n\u2502 - Build Docker image \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Step 2: CI/CD Pipeline (Automated) \u2502\n\u2502 - GitHub Actions builds image \u2502\n\u2502 - Pushes to container registry (GHCR, Docker Hub) \u2502\n\u2502 - Tags with version (e.g., v2.0.4) \u2502\n\u2502 - Updates helm-values repository with new tag \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Step 3: GitOps Sync (Automated) \u2502\n\u2502 - ArgoCD detects change in helm-values \u2502\n\u2502 - Pulls updated configuration \u2502\n\u2502 - Syncs to Kubernetes cluster \u2502\n\u2502 - Sends Slack notification on success/failure \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n</code></pre> <p>Key Insight: You don't deploy directly. You push code, CI/CD builds it, and ArgoCD deploys it.</p>"},{"location":"DEVELOPER-GUIDE/#understanding-the-workflow","title":"Understanding the Workflow","text":""},{"location":"DEVELOPER-GUIDE/#three-repository-pattern","title":"Three-Repository Pattern","text":"<p>Our setup uses three repositories:</p> Repository Purpose Who Edits How Often forte-helm Helm chart templates (generic, reusable) Platform engineers \u274c Rarely helm-values Application configuration (image tag, env vars) Developers / CI pipelines \u2705 Sometimes launchpad ArgoCD Applications (what gets deployed) Platform / DevOps engineers \u2705 Per new app"},{"location":"DEVELOPER-GUIDE/#example-deploying-myapp","title":"Example: Deploying \"myapp\"","text":""},{"location":"DEVELOPER-GUIDE/#repository-forte-helm-chart-templates","title":"Repository: <code>forte-helm</code> (Chart Templates)","text":"<pre><code># forteapp/templates/deployment.yaml\n# Generic template used by ALL apps\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: {{ .Values.app.name }}\nspec:\n containers:\n - name: app\n image: \"{{ .Values.app.image.repository }}:{{ .Values.app.image.tag }}\"\n env:\n - name: PORT\n value: {{ .Values.app.port }}\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#repository-helm-values-your-app-config","title":"Repository: <code>helm-values</code> (Your App Config)","text":"<pre><code># myapp/values.yaml\n# Your app's specific configuration\napp:\n image:\n repository: ghcr.io/fortedigital/myapp\n tag: v1.0.0 # CI/CD updates this\n port: 3000\n extraEnv:\n - name: API_URL\n value: https://api.example.com\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#repository-launchpad-argocd-application","title":"Repository: <code>launchpad</code> (ArgoCD Application)","text":"<pre><code># apps/myapp.yaml\n# Tells ArgoCD to deploy your app\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n name: myapp\n namespace: argocd\nspec:\n sources:\n - repoURL: https://github.com/fortedigital/forte-helm\n path: forteapp\n helm:\n valueFiles:\n - $values/myapp/values.yaml\n\n - repoURL: git@github.com:fortedigital/helm-values.git\n ref: values\n\n destination:\n server: https://kubernetes.default.svc\n namespace: myapp\n\n syncPolicy:\n automated:\n prune: true\n selfHeal: true\n syncOptions:\n - CreateNamespace=true\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#deploying-your-first-application","title":"Deploying Your First Application","text":""},{"location":"DEVELOPER-GUIDE/#scenario-youve-built-a-new-application","title":"Scenario: You've Built a New Application","text":"<p>Let's deploy a new Node.js application called \"hello-world\".</p>"},{"location":"DEVELOPER-GUIDE/#step-1-prepare-your-application-repository","title":"Step 1: Prepare Your Application Repository","text":"<p>Ensure your app repository has:</p> <ol> <li> <p>Dockerfile <pre><code>FROM node:18-alpine\nWORKDIR /app\nCOPY package*.json ./\nRUN npm ci --only=production\nCOPY . .\nEXPOSE 3000\nCMD [\"node\", \"server.js\"]\n</code></pre></p> </li> <li> <p>GitHub Actions Workflow (<code>.github/workflows/deploy.yml</code>) <pre><code>name: Build and Deploy\n\non:\n push:\n branches: [ main ]\n\njobs:\n build:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v3\n\n - name: Set version\n id: version\n run: echo \"VERSION=v$(date +%Y%m%d-%H%M%S)\" >> $GITHUB_OUTPUT\n\n - name: Build and push Docker image\n run: |\n echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin\n docker build -t ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }} .\n docker push ghcr.io/fortedigital/hello-world:${{ steps.version.outputs.VERSION }}\n\n - name: Update helm-values\n run: |\n git clone git@github.com:fortedigital/helm-values.git\n cd helm-values\n mkdir -p hello-world\n cat > hello-world/values.yaml <<EOF\n app:\n image:\n repository: ghcr.io/fortedigital/hello-world\n tag: ${{ steps.version.outputs.VERSION }}\n EOF\n git add hello-world/values.yaml\n git commit -m \"Update hello-world to ${{ steps.version.outputs.VERSION }}\"\n git push\n</code></pre></p> </li> </ol>"},{"location":"DEVELOPER-GUIDE/#step-2-create-helm-values","title":"Step 2: Create Helm Values","text":"<p>Create a folder in <code>helm-values</code> repository:</p> <pre><code>cd ~/dev/k8s/helm-prod-values\nmkdir -p hello-world\n</code></pre> <p>Create <code>hello-world/values.yaml</code>: <pre><code>app:\n image:\n repository: ghcr.io/fortedigital/hello-world\n tag: v1.0.0 # Will be updated by CI/CD\n containerPort: 3000\n\n replicaCount: 1\n\n resources:\n requests:\n cpu: 100m\n memory: 128Mi\n limits:\n cpu: 500m\n memory: 512Mi\n\n extraEnv:\n - name: PORT\n value: \"3000\"\n - name: NODE_ENV\n value: \"production\"\n\n envSecretName: \"\" # Optional: reference to secrets\n\nservice:\n port: 3000\n\ningress:\n enabled: true\n host: hello-world.forteapps.net # Your subdomain\n\ndb:\n enabled: false # Set to true if you need PostgreSQL\n</code></pre></p> <p>Commit and push: <pre><code>git add hello-world/values.yaml\ngit commit -m \"Add hello-world application values\"\ngit push\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#step-3-create-argocd-application-manifest","title":"Step 3: Create ArgoCD Application Manifest","text":"<p>In the <code>launchpad</code> repository, create <code>apps/hello-world.yaml</code>:</p> <pre><code>apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n name: hello-world\n namespace: argocd\n annotations:\n argocd.argoproj.io/sync-wave: \"1\"\n notifications.argoproj.io/subscribe.on-sync-succeeded.slack: \"\"\n notifications.argoproj.io/subscribe.on-sync-failed.slack: \"\"\n notifications.argoproj.io/subscribe.on-degraded.slack: \"\"\n labels:\n app.kubernetes.io/name: hello-world\n app.kubernetes.io/part-of: apps\n app.kubernetes.io/managed-by: argocd\n finalizers:\n - resources-finalizer.argocd.argoproj.io\n\nspec:\n project: default\n\n sources:\n # Source 1: Helm chart templates\n - repoURL: https://github.com/fortedigital/forte-helm\n path: forteapp\n targetRevision: HEAD\n helm:\n valueFiles:\n - $values/hello-world/values.yaml\n\n # Source 2: Helm values\n - repoURL: git@github.com:fortedigital/helm-values.git\n targetRevision: HEAD\n ref: values\n\n destination:\n server: https://kubernetes.default.svc\n namespace: hello-world\n\n syncPolicy:\n automated:\n prune: true\n selfHeal: true\n allowEmpty: false\n\n syncOptions:\n - CreateNamespace=true\n - Validate=true\n - ServerSideApply=true\n\n retry:\n limit: 5\n backoff:\n duration: 5s\n factor: 2\n maxDuration: 3m\n\n ignoreDifferences:\n - group: apps\n kind: Deployment\n jsonPointers:\n - /spec/replicas\n</code></pre> <p>Commit and push: <pre><code>cd ~/dev/k8s/launchpad\ngit add apps/hello-world.yaml\ngit commit -m \"Add hello-world application\"\ngit push\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#step-4-verify-deployment","title":"Step 4: Verify Deployment","text":"<p>ArgoCD will automatically detect the new application within 60 seconds.</p> <p>Option 1: Check Slack - Watch for sync notifications in your Slack channel - \u2705 \"Application hello-world sync succeeded\"</p> <p>Option 2: Check ArgoCD UI (if you have access) <pre><code># Port forward to ArgoCD UI\nkubectl port-forward svc/argocd-server -n argocd 8080:443\n\n# Open browser: https://localhost:8080\n# Look for \"hello-world\" application\n</code></pre></p> <p>Option 3: Check with kubectl (if you have access) <pre><code># List ArgoCD applications\nkubectl get applications -n argocd\n\n# Check application status\nkubectl get application hello-world -n argocd\n\n# Verify pods are running\nkubectl get pods -n hello-world\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#step-5-access-your-application","title":"Step 5: Access Your Application","text":"<p>Once deployed, access via the configured domain:</p> <pre><code># Check if ingress is created\nkubectl get ingressroute -n hello-world\n\n# Access application\ncurl https://hello-world.forteapps.net\n</code></pre> <p>\u26a0\ufe0f Note: DNS must be manually configured for new subdomains. Contact the platform team to add DNS records.</p>"},{"location":"DEVELOPER-GUIDE/#updating-an-existing-application","title":"Updating an Existing Application","text":""},{"location":"DEVELOPER-GUIDE/#scenario-deploying-a-code-change","title":"Scenario: Deploying a Code Change","text":"<p>You've made changes to your application code and want to deploy them.</p>"},{"location":"DEVELOPER-GUIDE/#method-1-automatic-recommended","title":"Method 1: Automatic (Recommended)","text":"<p>Just push to <code>main</code> branch - CI/CD handles everything:</p> <pre><code># In your application repository\ngit add .\ngit commit -m \"Fix bug in user login\"\ngit push origin main\n</code></pre> <p>What Happens Next: 1. \u2705 GitHub Actions triggers 2. \u2705 Builds new Docker image 3. \u2705 Tags with new version (e.g., <code>v20260316-143022</code>) 4. \u2705 Pushes to container registry 5. \u2705 Updates <code>helm-values/myapp/values.yaml</code> with new tag 6. \u2705 ArgoCD detects change 7. \u2705 Syncs new version to cluster 8. \u2705 Sends Slack notification</p> <p>Timeline: ~5-10 minutes from push to deployment</p>"},{"location":"DEVELOPER-GUIDE/#method-2-manual-image-tag-update","title":"Method 2: Manual Image Tag Update","text":"<p>If CI/CD is not set up, manually update the image tag:</p> <pre><code>cd ~/dev/k8s/helm-prod-values\n\n# Edit your app's values.yaml\nvim myapp/values.yaml\n\n# Change:\napp:\n image:\n tag: v1.0.0 # Old version\n# To:\napp:\n image:\n tag: v1.0.1 # New version\n\n# Commit and push\ngit add myapp/values.yaml\ngit commit -m \"Update myapp to v1.0.1\"\ngit push\n</code></pre> <p>ArgoCD will sync within 60 seconds.</p>"},{"location":"DEVELOPER-GUIDE/#method-3-configuration-changes","title":"Method 3: Configuration Changes","text":"<p>To update environment variables, resources, or other config:</p> <pre><code>cd ~/dev/k8s/helm-prod-values\nvim myapp/values.yaml\n</code></pre> <p>Example changes:</p> <pre><code>app:\n # Increase resources\n resources:\n requests:\n cpu: 200m # Was 100m\n memory: 256Mi # Was 128Mi\n\n # Add new environment variable\n extraEnv:\n - name: API_URL\n value: https://api.example.com\n - name: DEBUG # NEW\n value: \"true\" # NEW\n\n # Enable HPA\n hpa:\n enabled: true # Was false\n minReplicas: 2\n maxReplicas: 10\n</code></pre> <p>Commit and push: <pre><code>git add myapp/values.yaml\ngit commit -m \"Increase myapp resources and enable HPA\"\ngit push\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#method-4-application-manifest-changes","title":"Method 4: Application Manifest Changes","text":"<p>To change ArgoCD sync behavior, namespace, or other meta-config:</p> <pre><code>cd ~/dev/k8s/launchpad\nvim apps/myapp.yaml\n</code></pre> <p>Example changes:</p> <pre><code>spec:\n syncPolicy:\n automated:\n prune: true\n selfHeal: false # Disable self-healing temporarily\n</code></pre> <p>Commit and push: <pre><code>git add apps/myapp.yaml\ngit commit -m \"Disable self-healing for myapp\"\ngit push\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#working-with-secrets","title":"Working with Secrets","text":""},{"location":"DEVELOPER-GUIDE/#understanding-secret-management","title":"Understanding Secret Management","text":"<p>NEVER commit plain secrets to Git. We use Sealed Secrets to encrypt secrets before committing.</p>"},{"location":"DEVELOPER-GUIDE/#creating-a-new-secret","title":"Creating a New Secret","text":""},{"location":"DEVELOPER-GUIDE/#step-1-create-plain-secret-locally","title":"Step 1: Create Plain Secret Locally","text":"<pre><code>cd ~/dev/k8s/launchpad\n\n# Create secret in private/ folder (Git-ignored)\nkubectl create secret generic myapp-credentials \\\n --from-literal=API_KEY=your-secret-key-here \\\n --from-literal=DB_PASSWORD=super-secret-password \\\n --dry-run=client -o yaml > private/myapp-credentials.yaml\n</code></pre> <p>DO NOT commit this file! It's in <code>private/</code> which is Git-ignored.</p>"},{"location":"DEVELOPER-GUIDE/#step-2-seal-the-secret","title":"Step 2: Seal the Secret","text":"<p>Get the public certificate (one-time setup):</p> <pre><code># Fetch public cert from cluster\nkubeseal --fetch-cert \\\n --controller-name=sealed-secrets-controller \\\n --controller-namespace=kube-system \\\n > pub-cert.pem\n</code></pre> <p>Seal your secret:</p> <pre><code>kubeseal --format=yaml \\\n --cert=pub-cert.pem \\\n < private/myapp-credentials.yaml \\\n > secrets/myapp-credentials-sealed.yaml\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#step-3-commit-sealed-secret","title":"Step 3: Commit Sealed Secret","text":"<pre><code>git add secrets/myapp-credentials-sealed.yaml\ngit commit -m \"Add myapp credentials (sealed)\"\ngit push\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#step-4-reference-secret-in-application","title":"Step 4: Reference Secret in Application","text":"<p>Update your <code>helm-values/myapp/values.yaml</code>:</p> <pre><code>app:\n envSecretName: \"myapp-credentials\" # References the SealedSecret\n</code></pre> <p>Commit and push: <pre><code>cd ~/dev/k8s/helm-prod-values\ngit add myapp/values.yaml\ngit commit -m \"Reference myapp credentials\"\ngit push\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#updating-a-secret","title":"Updating a Secret","text":"<p>To update an existing secret:</p> <pre><code># 1. Create new version of secret\nkubectl create secret generic myapp-credentials \\\n --from-literal=API_KEY=new-key-here \\\n --from-literal=DB_PASSWORD=new-password \\\n --dry-run=client -o yaml > private/myapp-credentials.yaml\n\n# 2. Seal it\nkubeseal --format=yaml \\\n --cert=pub-cert.pem \\\n < private/myapp-credentials.yaml \\\n > secrets/myapp-credentials-sealed.yaml\n\n# 3. Commit sealed version\ngit add secrets/myapp-credentials-sealed.yaml\ngit commit -m \"Update myapp credentials\"\ngit push\n\n# 4. Restart pods to pick up new secret\nkubectl rollout restart deployment myapp -n myapp\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#secret-best-practices","title":"Secret Best Practices","text":"<p>\u2705 DO: - Store secrets in <code>private/</code> folder locally - Always seal secrets before committing - Delete plain secrets after sealing - Use meaningful secret names - Document what each secret contains</p> <p>\u274c DON'T: - Commit plain secrets to Git - Share secrets via Slack/email - Hard-code secrets in code - Use the same secret across multiple environments - Store secrets in Docker images</p>"},{"location":"DEVELOPER-GUIDE/#where-secrets-are-stored","title":"Where Secrets Are Stored","text":"<pre><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Location \u2502 Content \u2502 Committed?\u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 private/ \u2502 Plain secrets \u2502 \u274c NO \u2502\n\u2502 secrets/ \u2502 Sealed secrets \u2502 \u2705 YES \u2502\n\u2502 Kubernetes cluster \u2502 Unsealed secrets \u2502 N/A \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n</code></pre> <p>Sealed Secrets Controller in the cluster decrypts sealed secrets automatically.</p>"},{"location":"DEVELOPER-GUIDE/#enabling-authentication-for-applications","title":"Enabling Authentication for Applications","text":"<p>The cluster supports automatic authentication sidecar injection for applications via Kyverno policies. This allows you to add authentication to your applications without modifying application code.</p>"},{"location":"DEVELOPER-GUIDE/#how-it-works","title":"How It Works","text":"<p>When you enable authentication in your Helm values, the Kyverno policy automatically: 1. \u2705 Injects an authentication sidecar container into your pod 2. \u2705 Routes all incoming traffic through the auth sidecar (port 8080) 3. \u2705 Validates credentials before forwarding requests to your application 4. \u2705 Creates necessary secrets (if they don't exist) 5. \u2705 Adds a NetworkPolicy to restrict ingress</p> <p>Architecture: <pre><code>Internet \u2192 Traefik \u2192 Service:8080 \u2192 Auth Sidecar:8080 \u2192 localhost \u2192 Your App:3000\n \u2502\n \u251c\u2500 Validates credentials\n \u2514\u2500 Forwards if valid\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#authentication-modes","title":"Authentication Modes","text":"<p>Three authentication modes are supported: 1. Token-based: Static tokens (simple, good for service-to-service or internal apps) 2. OIDC: OpenID Connect (full SSO, good for user-facing apps) 3. MCP: OAuth 2.0 for MCP servers via RFC 9728 / RFC 7591 (good for MCP tool servers requiring OAuth-based access control)</p>"},{"location":"DEVELOPER-GUIDE/#token-based-authentication","title":"Token-Based Authentication","text":""},{"location":"DEVELOPER-GUIDE/#step-1-configure-helm-values","title":"Step 1: Configure Helm Values","text":"<pre><code># In helm-values/myapp/values.yaml\nauth:\n enabled: true\n type: token # Token mode (default)\n tokens:\n - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823\n - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#step-2-generate-token-if-needed","title":"Step 2: Generate Token (if needed)","text":"<pre><code># Generate a secure random token\nopenssl rand -hex 32\n\n# Or using Python\npython3 -c \"import secrets; print(secrets.token_hex(32))\"\n\n# Example output:\n# d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#step-3-deploy-application","title":"Step 3: Deploy Application","text":"<p>Commit and push your changes: <pre><code>cd ~/dev/k8s/helm-prod-values\ngit add myapp/values.yaml\ngit commit -m \"Enable token auth for myapp\"\ngit push\n</code></pre></p> <p>ArgoCD will sync, and the Kyverno policy will: - Inject the auth sidecar container - Create an <code>auth-tokens</code> Secret with your tokens - Configure the sidecar to validate against these tokens</p>"},{"location":"DEVELOPER-GUIDE/#step-4-access-application","title":"Step 4: Access Application","text":"<p>Use your token in the <code>Authorization</code> header:</p> <pre><code># Access application with token\ncurl -H \"Authorization: Bearer d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823\" \\\n https://myapp.forteapps.net/api/data\n\n# Without token (will be rejected)\ncurl https://myapp.forteapps.net/api/data\n# Response: 401 Unauthorized\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#advanced-custom-secret-name","title":"Advanced: Custom Secret Name","text":"<p>To use a different secret for tokens:</p> <pre><code># In Helm values\nauth:\n enabled: true\n type: token\n tokens: [] # Empty - using external secret\n\n# Tokens will be read from custom secret\n</code></pre> <p>Then reference it via annotation (configured by Helm chart automatically): <pre><code># Helm chart sets this annotation:\npolicies.forteapps.io/auth-token-secret-name: \"myapp-auth-tokens\"\n</code></pre></p> <p>Create the secret manually: <pre><code>kubectl create secret generic myapp-auth-tokens \\\n --from-file=tokens=tokens.txt \\\n --namespace=myapp\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#oidc-authentication","title":"OIDC Authentication","text":"<p>OIDC mode integrates with identity providers like Keycloak, Okta, Auth0, Azure AD, etc.</p>"},{"location":"DEVELOPER-GUIDE/#step-1-configure-identity-provider","title":"Step 1: Configure Identity Provider","text":"<p>In your identity provider (e.g., Keycloak): 1. Create a new client (e.g., <code>myapp</code>) 2. Set redirect URI: <code>https://myapp.forteapps.net/auth/callback</code> 3. Note the Client ID and Client Secret 4. Note the Authority URL (e.g., <code>https://keycloak.forteapps.net/realms/master</code>)</p>"},{"location":"DEVELOPER-GUIDE/#step-2-create-oidc-secret","title":"Step 2: Create OIDC Secret","text":"<pre><code># Create plain secret\nkubectl create secret generic auth-oidc \\\n --from-literal=client-secret=your-oidc-client-secret \\\n --from-literal=cookie-secret=$(openssl rand -hex 32) \\\n --namespace=myapp \\\n --dry-run=client -o yaml > private/myapp-auth-oidc.yaml\n\n# Seal it\nkubeseal --format=yaml \\\n --cert=pub-cert.pem \\\n --namespace=myapp \\\n < private/myapp-auth-oidc.yaml \\\n > secrets/myapp-auth-oidc-sealed.yaml\n\n# Commit sealed secret\ncd ~/dev/k8s/launchpad\ngit add secrets/myapp-auth-oidc-sealed.yaml\ngit commit -m \"Add OIDC secrets for myapp\"\ngit push\n\n# Clean up\nrm private/myapp-auth-oidc.yaml\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#step-3-configure-helm-values","title":"Step 3: Configure Helm Values","text":"<pre><code># In helm-values/myapp/values.yaml\nauth:\n enabled: true\n type: oidc # OIDC mode\n oidc:\n authority: https://keycloak.forteapps.net/realms/master\n clientId: myapp\n scopes: \"openid,profile,email\"\n callbackPath: /auth/callback\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#step-4-deploy-application","title":"Step 4: Deploy Application","text":"<pre><code>cd ~/dev/k8s/helm-prod-values\ngit add myapp/values.yaml\ngit commit -m \"Enable OIDC auth for myapp\"\ngit push\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#step-5-access-application","title":"Step 5: Access Application","text":"<p>When users access <code>https://myapp.forteapps.net</code>: 1. They're redirected to the identity provider login page 2. After successful login, redirected back to <code>/auth/callback</code> 3. Session cookie is set 4. Subsequent requests are authenticated via cookie</p> <p>User flow: <pre><code>User \u2192 https://myapp.forteapps.net\n \u2193\nRedirect \u2192 https://keycloak.forteapps.net/login\n \u2193\nLogin successful \u2192 Redirect with auth code\n \u2193\nhttps://myapp.forteapps.net/auth/callback?code=xyz\n \u2193\nAuth sidecar exchanges code for tokens\n \u2193\nSets session cookie\n \u2193\nRedirects to application \u2192 https://myapp.forteapps.net\n \u2193\nUser sees application (authenticated)\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#authentication-configuration-reference","title":"Authentication Configuration Reference","text":""},{"location":"DEVELOPER-GUIDE/#helm-values-schema","title":"Helm Values Schema","text":"<pre><code>auth:\n enabled: false # Enable/disable authentication\n type: token # \"token\", \"oidc\", or \"mcp\"\n\n # Token mode configuration\n tokens: [] # List of valid bearer tokens\n # - token1\n # - token2\n\n # OIDC mode configuration\n oidc:\n authority: \"\" # OIDC provider URL (required for OIDC)\n clientId: \"\" # OIDC client ID (required for OIDC)\n scopes: \"openid,profile,email\" # OIDC scopes (optional)\n callbackPath: /auth/callback # OAuth callback path (optional)\n\n # MCP mode configuration (RFC 9728 / RFC 7591)\n mcp:\n resource: \"\" # Protected resource URL (required for MCP)\n authority: \"\" # Authorization server URL (required for MCP)\n scopes: \"read,write\" # Supported scopes (optional)\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#annotations-set-by-helm-chart","title":"Annotations Set by Helm Chart","text":"<p>When <code>auth.enabled: true</code>, the Helm chart sets these pod annotations:</p> <p>Token mode: <pre><code>policies.forteapps.io/auth: \"true\"\npolicies.forteapps.io/auth-type: \"token\"\npolicies.forteapps.io/auth-token-secret-name: \"auth-tokens\"\npolicies.forteapps.io/auth-upstream-url: \"http://localhost:3000\"\n</code></pre></p> <p>OIDC mode: <pre><code>policies.forteapps.io/auth: \"true\"\npolicies.forteapps.io/auth-type: \"oidc\"\npolicies.forteapps.io/auth-oidc-authority: \"https://keycloak.forteapps.net/realms/master\"\npolicies.forteapps.io/auth-oidc-client-id: \"myapp\"\npolicies.forteapps.io/auth-oidc-scopes: \"openid,profile,email\"\npolicies.forteapps.io/auth-oidc-callback-path: \"/auth/callback\"\npolicies.forteapps.io/auth-upstream-url: \"http://localhost:3000\"\n</code></pre></p> <p>MCP mode (OAuth 2.0 for MCP servers): <pre><code>policies.forteapps.io/auth: \"true\"\npolicies.forteapps.io/auth-type: \"mcp\"\npolicies.forteapps.io/auth-mcp-resource: \"https://mcp.forteapps.net\"\npolicies.forteapps.io/auth-mcp-authority: \"https://keycloak.forteapps.net/realms/master\"\npolicies.forteapps.io/auth-mcp-scopes: \"read,write\"\npolicies.forteapps.io/auth-upstream-url: \"http://localhost:3000\"\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#sidecar-configuration","title":"Sidecar Configuration","text":"<p>The auth sidecar container: - Image: <code>ghcr.io/fortedigital/auth-sidecar:latest</code> - Port: 8080 - Resources: 10m CPU / 32Mi memory (requests), 50m CPU / 64Mi memory (limits) - Health checks: <code>/healthz</code> endpoint - Security: Read-only root filesystem, no privilege escalation</p>"},{"location":"DEVELOPER-GUIDE/#advanced-custom-sidecar-image","title":"Advanced: Custom Sidecar Image","text":"<p>To use a different auth sidecar image:</p> <pre><code># These annotations can be set in the Helm chart template if needed\npolicies.forteapps.io/auth-image: \"your-registry/your-auth-proxy\"\npolicies.forteapps.io/auth-image-version: \"v1.2.3\"\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#authentication-examples","title":"Authentication Examples","text":""},{"location":"DEVELOPER-GUIDE/#example-1-internal-api-with-token-auth","title":"Example 1: Internal API with Token Auth","text":"<pre><code># helm-values/internal-api/values.yaml\napp:\n image:\n repository: ghcr.io/company/internal-api\n tag: v1.0.0\n\nauth:\n enabled: true\n type: token\n tokens:\n - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823 # Service A\n - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7 # Service B\n\ningress:\n enabled: true\n host: internal-api.forteapps.net\n</code></pre> <p>Usage: <pre><code># Service A calls API\ncurl -H \"Authorization: Bearer d4f88f...\" \\\n https://internal-api.forteapps.net/api/endpoint\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#example-2-user-facing-app-with-oidc","title":"Example 2: User-Facing App with OIDC","text":"<pre><code># helm-values/web-app/values.yaml\napp:\n image:\n repository: ghcr.io/company/web-app\n tag: v2.1.0\n\nauth:\n enabled: true\n type: oidc\n oidc:\n authority: https://auth.company.com/realms/employees\n clientId: web-app-prod\n scopes: \"openid,profile,email,groups\"\n callbackPath: /auth/callback\n\ningress:\n enabled: true\n host: web-app.forteapps.net\n</code></pre> <p>With sealed OIDC secret: <pre><code># Create and seal secret\nkubectl create secret generic auth-oidc \\\n --from-literal=client-secret=super-secret-value \\\n --from-literal=cookie-secret=$(openssl rand -hex 32) \\\n --namespace=web-app \\\n --dry-run=client -o yaml | \\\n kubeseal --format=yaml --cert=pub-cert.pem --namespace=web-app \\\n > secrets/web-app-auth-oidc-sealed.yaml\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#example-3-mcp-server-with-oauth-20","title":"Example 3: MCP Server with OAuth 2.0","text":"<pre><code># helm-values/mcp-server/values.yaml\napp:\n image:\n repository: ghcr.io/company/mcp-server\n tag: v1.0.0\n\nauth:\n enabled: true\n type: mcp\n mcp:\n resource: https://mcp-server.forteapps.net\n authority: https://auth.company.com/realms/mcp\n scopes: \"read,write,admin\"\n\ningress:\n enabled: true\n host: mcp-server.forteapps.net\n</code></pre> <p>The MCP auth mode implements RFC 9728 (OAuth 2.0 Protected Resource Metadata) for authorization server discovery and RFC 7591 (OAuth 2.0 Dynamic Client Registration) for automatic client registration. MCP clients discover the authorization server and scopes from the <code>/.well-known/oauth-protected-resource</code> endpoint served by the sidecar.</p>"},{"location":"DEVELOPER-GUIDE/#example-4-disabling-authentication","title":"Example 4: Disabling Authentication","text":"<pre><code># helm-values/public-api/values.yaml\nauth:\n enabled: false # No authentication\n\ningress:\n enabled: true\n host: public-api.forteapps.net\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#troubleshooting-authentication","title":"Troubleshooting Authentication","text":""},{"location":"DEVELOPER-GUIDE/#issue-401-unauthorized-token-mode","title":"Issue: 401 Unauthorized (Token Mode)","text":"<p>Check token validity: <pre><code># Get auth-tokens secret\nkubectl get secret auth-tokens -n myapp -o yaml\n\n# Decode tokens\nkubectl get secret auth-tokens -n myapp \\\n -o jsonpath='{.data.tokens}' | base64 -d\n\n# Verify your token is in the list\n</code></pre></p> <p>Test with different token: <pre><code>curl -v -H \"Authorization: Bearer YOUR-TOKEN-HERE\" \\\n https://myapp.forteapps.net/\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#issue-oidc-login-loop","title":"Issue: OIDC Login Loop","text":"<p>Check OIDC configuration: <pre><code># Verify auth-oidc secret exists\nkubectl get secret auth-oidc -n myapp\n\n# Check sidecar logs\nkubectl logs -n myapp <pod-name> -c authn\n\n# Common issues:\n# - Wrong authority URL\n# - Wrong client ID\n# - Missing client-secret in auth-oidc Secret\n# - Redirect URI not configured in identity provider\n</code></pre></p> <p>Verify redirect URI in your identity provider matches: <pre><code>https://<your-app-domain>/auth/callback\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#issue-auth-sidecar-not-injected","title":"Issue: Auth Sidecar Not Injected","text":"<p>Check pod annotations: <pre><code>kubectl get pod -n myapp <pod-name> -o yaml | grep policies.forteapps.io\n\n# Should show:\n# policies.forteapps.io/auth: \"true\"\n</code></pre></p> <p>Check Kyverno policy: <pre><code>kubectl get clusterpolicy inject-auth-sidecar\nkubectl describe clusterpolicy inject-auth-sidecar\n</code></pre></p> <p>Check Kyverno logs: <pre><code>kubectl logs -n kyverno deployment/kyverno | grep inject-auth\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#issue-auth-sidecar-crashes","title":"Issue: Auth Sidecar Crashes","text":"<p>Check sidecar logs: <pre><code>kubectl logs -n myapp <pod-name> -c authn\n</code></pre></p> <p>Common causes: - Missing secret (auth-tokens or auth-oidc) - Invalid OIDC configuration - Can't reach OIDC authority URL - Network policy blocking outbound OIDC requests</p>"},{"location":"DEVELOPER-GUIDE/#authentication-best-practices","title":"Authentication Best Practices","text":"<p>\u2705 DO: - Use OIDC for user-facing applications - Use token auth for service-to-service communication - Rotate tokens and secrets regularly - Use strong random tokens (32+ bytes) - Store client secrets in SealedSecrets - Test authentication before deploying to production - Document which tokens/users have access</p> <p>\u274c DON'T: - Share tokens between environments - Commit tokens to application code - Use predictable tokens - Reuse tokens across multiple applications - Disable authentication on sensitive APIs - Log tokens or secrets</p>"},{"location":"DEVELOPER-GUIDE/#adding-a-new-keycloak-client","title":"Adding a New Keycloak Client","text":"<p>There are two ways to add an OIDC client, depending on your use case:</p> Method Best for Who edits the infra repo? Self-service (recommended) New apps that deploy their own resources App developer \u2014 no infra changes needed Legacy (realm JSON) Existing clients already defined in forte-realm.json (e.g., Gitea) Platform engineer <p>Both methods are served by the Keycloak Client Registrar CronJob, which runs every 2 minutes.</p>"},{"location":"DEVELOPER-GUIDE/#self-service-oidc-client-registration","title":"Self-Service OIDC Client Registration","text":"<p>This is the recommended flow for new applications. Your app deploys a labeled config Secret in its own namespace; the platform handles everything else.</p>"},{"location":"DEVELOPER-GUIDE/#how-it-works_1","title":"How It Works","text":"<ol> <li>You deploy a Secret with label <code>keycloak.forteapps.net/client-config: \"true\"</code> containing a <code>client.json</code> definition</li> <li>A Kyverno ClusterPolicy (<code>keycloak-client-config-cloner</code>) clones it to the <code>keycloak</code> namespace</li> <li>The Client Registrar CronJob picks it up within 2 minutes:</li> <li>Registers (or updates) the client in Keycloak</li> <li>Fetches the auto-generated client secret</li> <li>Creates a credential Secret in your app's namespace</li> <li>Annotates the config Secret with sync status</li> </ol>"},{"location":"DEVELOPER-GUIDE/#step-1-create-the-config-secret","title":"Step 1: Create the Config Secret","text":"<p>Deploy this Secret in your application's namespace (e.g., as part of your Helm chart or Kustomize overlay):</p> <pre><code>apiVersion: v1\nkind: Secret\nmetadata:\n name: keycloak-client-myapp\n namespace: myapp\n labels:\n keycloak.forteapps.net/client-config: \"true\"\nstringData:\n client.json: |\n {\n \"clientId\": \"myapp\",\n \"name\": \"My Application\",\n \"redirectUris\": [\"https://myapp.forteapps.net/*\"],\n \"webOrigins\": [\"https://myapp.forteapps.net\"],\n \"defaultClientScopes\": [\"openid\", \"email\", \"profile\"],\n \"protocolMappers\": [],\n \"secret\": {\n \"namespace\": \"myapp\",\n \"name\": \"myapp-oidc-credentials\",\n \"keys\": { \"clientId\": \"client-id\", \"clientSecret\": \"client-secret\" }\n }\n }\n</code></pre> <p><code>client.json</code> fields:</p> Field Required Description <code>clientId</code> Yes Keycloak client ID <code>name</code> Yes Display name in Keycloak <code>redirectUris</code> Yes Allowed redirect URIs <code>webOrigins</code> Yes Allowed web origins (CORS) <code>defaultClientScopes</code> No Scopes (default: <code>[\"openid\", \"email\", \"profile\"]</code>) <code>protocolMappers</code> No Custom claim mappers (default: <code>[]</code>) <code>secret.namespace</code> No Namespace for the credential Secret (default: source namespace) <code>secret.name</code> No Name of the credential Secret (default: <code><clientId>-oidc-credentials</code>) <code>secret.keys.clientId</code> No Key name for client ID in credential Secret (default: <code>client-id</code>) <code>secret.keys.clientSecret</code> No Key name for client secret in credential Secret (default: <code>client-secret</code>)"},{"location":"DEVELOPER-GUIDE/#step-2-reference-the-credential-secret","title":"Step 2: Reference the Credential Secret","text":"<p>In your application's deployment config, reference the credential Secret that the registrar creates:</p> <pre><code>env:\n- name: OIDC_CLIENT_ID\n valueFrom:\n secretKeyRef:\n name: myapp-oidc-credentials\n key: client-id\n- name: OIDC_CLIENT_SECRET\n valueFrom:\n secretKeyRef:\n name: myapp-oidc-credentials\n key: client-secret\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#step-3-deploy-and-wait","title":"Step 3: Deploy and Wait","text":"<p>Commit and push your changes. The credential Secret will appear within 2 minutes:</p> <pre><code># Watch for the credential Secret to be created\nkubectl get secret myapp-oidc-credentials -n myapp -w\n\n# Check registrar logs\nkubectl logs -n keycloak job/$(kubectl get jobs -n keycloak --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')\n\n# Check sync status on the config Secret\nkubectl get secret keycloak-client-myapp -n keycloak -o jsonpath='{.metadata.annotations}'\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#change-detection","title":"Change Detection","text":"<p>The registrar computes a SHA-256 hash of <code>client.json</code> and stores it as an annotation. On subsequent runs, it skips processing if: - The hash hasn't changed, AND - The credential Secret already exists in the target namespace</p> <p>To force a re-sync, update any field in <code>client.json</code> (e.g., add a trailing space to <code>name</code>).</p>"},{"location":"DEVELOPER-GUIDE/#legacy-method-realm-json","title":"Legacy Method: Realm JSON","text":"<p>Existing clients (like Gitea) are defined directly in <code>forte-realm.json</code> inside <code>keycloak-values.yaml</code>. The registrar syncs their secrets via client attributes.</p>"},{"location":"DEVELOPER-GUIDE/#step-1-add-client-to-realm-config","title":"Step 1: Add Client to Realm Config","text":"<p>In <code>infra/values/base/keycloak-values.yaml</code>, add a new entry to the <code>clients</code> array in <code>forte-realm.json</code>:</p> <pre><code>{\n \"clientId\": \"myapp\",\n \"name\": \"My Application\",\n \"enabled\": true,\n \"protocol\": \"openid-connect\",\n \"clientAuthenticatorType\": \"client-secret\",\n \"standardFlowEnabled\": true,\n \"directAccessGrantsEnabled\": false,\n \"publicClient\": false,\n \"redirectUris\": [\"https://myapp.forteapps.net/*\"],\n \"webOrigins\": [\"https://myapp.forteapps.net\"],\n \"defaultClientScopes\": [\"openid\", \"email\", \"profile\"],\n \"attributes\": {\n \"k8s.secret.sync\": \"true\",\n \"k8s.secret.namespace\": \"myapp\",\n \"k8s.secret.name\": \"myapp-oidc-credentials\",\n \"k8s.secret.client-id-key\": \"key\",\n \"k8s.secret.client-secret-key\": \"secret\"\n }\n}\n</code></pre> <p>Important: - Do NOT include a <code>\"secret\"</code> field \u2014 Keycloak generates one automatically - The <code>attributes</code> block tells the registrar where to create the K8s Secret - Set <code>client-id-key</code> / <code>client-secret-key</code> to match what the consuming app expects (defaults: <code>client-id</code> / <code>client-secret</code>)</p>"},{"location":"DEVELOPER-GUIDE/#step-2-reference-the-secret-in-your-application","title":"Step 2: Reference the Secret in Your Application","text":"<pre><code>existingSecret: myapp-oidc-credentials\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#step-3-commit-and-push","title":"Step 3: Commit and Push","text":"<pre><code>cd ~/dev/k8s/launchpad\ngit add infra/values/base/keycloak-values.yaml\ngit commit -m \"Add myapp Keycloak client with auto-sync\"\ngit push\n</code></pre> <p>ArgoCD will sync the Keycloak config, and the registrar CronJob will pick up the new client within 2 minutes.</p>"},{"location":"DEVELOPER-GUIDE/#legacy-sync-attribute-reference","title":"Legacy Sync Attribute Reference","text":"Attribute Required Default Description <code>k8s.secret.sync</code> Yes \u2014 Set to <code>\"true\"</code> to enable syncing <code>k8s.secret.namespace</code> Yes \u2014 Target K8s namespace for the secret <code>k8s.secret.name</code> Yes \u2014 Name of the K8s Secret to create <code>k8s.secret.client-id-key</code> No <code>client-id</code> Field name for the client ID in the K8s Secret <code>k8s.secret.client-secret-key</code> No <code>client-secret</code> Field name for the client secret in the K8s Secret"},{"location":"DEVELOPER-GUIDE/#retrieving-secrets-for-external-deployments","title":"Retrieving Secrets for External Deployments","text":"<p>The registrar always writes a central copy of every synced secret to the <code>secrets</code> namespace, in addition to the target namespace. This allows operators to retrieve client credentials for applications deployed outside this cluster:</p> <pre><code># View the central copy\nkubectl get secret gitea-oidc-credentials -n secrets -o yaml\n\n# Extract the client secret for use elsewhere\nkubectl get secret myapp-oidc-credentials -n secrets \\\n -o jsonpath='{.data.client-secret}' | base64 -d\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#registrar-behavior-notes","title":"Registrar Behavior Notes","text":"<ul> <li>The registrar runs as a CronJob every 2 minutes (<code>concurrencyPolicy: Forbid</code>)</li> <li>If the target namespace doesn't exist, the target write is skipped with a warning (the central copy still happens)</li> <li>A central copy is always written to the <code>secrets</code> namespace for every synced client</li> <li>The registrar uses the <code>keycloak-credentials</code> secret for admin authentication</li> <li>Created secrets have the label <code>app.kubernetes.io/managed-by: keycloak-client-registrar</code></li> </ul>"},{"location":"DEVELOPER-GUIDE/#troubleshooting","title":"Troubleshooting","text":""},{"location":"DEVELOPER-GUIDE/#application-not-deploying","title":"Application Not Deploying","text":""},{"location":"DEVELOPER-GUIDE/#problem-application-stuck-in-syncing-state","title":"Problem: Application stuck in \"Syncing\" state","text":"<p>Check ArgoCD status: <pre><code>kubectl get application myapp -n argocd -o yaml\n</code></pre></p> <p>Look for errors in <code>status.conditions</code>.</p> <p>Common causes: - \u274c Image doesn't exist or is not accessible - \u274c Invalid YAML syntax - \u274c Resource quota exceeded - \u274c Namespace conflicts - \u274c Invalid Helm values</p> <p>Solutions: <pre><code># Check image exists\ndocker pull ghcr.io/fortedigital/myapp:v1.0.0\n\n# Validate YAML syntax\nkubectl apply --dry-run=client -f apps/myapp.yaml\n\n# Check ArgoCD logs\nkubectl logs -n argocd deployment/argocd-application-controller | grep myapp\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#problem-pods-crashing-crashloopbackoff","title":"Problem: Pods crashing (CrashLoopBackOff)","text":"<p>Check pod logs: <pre><code>kubectl get pods -n myapp\nkubectl logs -n myapp <pod-name>\nkubectl describe pod -n myapp <pod-name>\n</code></pre></p> <p>Common causes: - \u274c Application error (check logs) - \u274c Missing environment variables - \u274c Incorrect port configuration - \u274c Missing secrets - \u274c Insufficient resources</p> <p>Solutions: <pre><code># Check environment variables\nkubectl exec -n myapp <pod-name> -- env\n\n# Check if secrets exist\nkubectl get secrets -n myapp\n\n# Increase resources in helm-values\nvim ~/dev/k8s/helm-prod-values/myapp/values.yaml\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#problem-application-not-accessible-via-domain","title":"Problem: Application not accessible via domain","text":"<p>Check ingress: <pre><code>kubectl get ingressroute -n myapp\nkubectl describe ingressroute myapp -n myapp\n</code></pre></p> <p>Common causes: - \u274c DNS not configured - \u274c TLS certificate not issued - \u274c Incorrect domain in values.yaml - \u274c Traefik not routing correctly</p> <p>Solutions: <pre><code># Check certificate\nkubectl get certificate -n myapp\n\n# Check cert-manager logs\nkubectl logs -n cert-manager deployment/cert-manager\n\n# Verify domain configuration\ncat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep host\n\n# Test with port-forward\nkubectl port-forward -n myapp service/myapp 8080:3000\ncurl http://localhost:8080\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#secret-issues","title":"Secret Issues","text":""},{"location":"DEVELOPER-GUIDE/#problem-secret-not-found","title":"Problem: Secret not found","text":"<p>Check if SealedSecret exists: <pre><code>kubectl get sealedsecret -n myapp\nkubectl get secret -n myapp\n</code></pre></p> <p>Solutions: <pre><code># Check if secret is in Git\nls -l secrets/myapp-credentials-sealed.yaml\n\n# Re-apply sealed secret\nkubectl apply -f secrets/myapp-credentials-sealed.yaml\n\n# Check sealed-secrets-controller logs\nkubectl logs -n kube-system deployment/sealed-secrets-controller\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#problem-secret-exists-but-pods-cant-access-it","title":"Problem: Secret exists but pods can't access it","text":"<p>Check pod events: <pre><code>kubectl describe pod -n myapp <pod-name>\n</code></pre></p> <p>Look for: <code>Error: secret \"myapp-credentials\" not found</code></p> <p>Solutions: <pre><code># Verify secret name in values.yaml matches actual secret\ncat ~/dev/k8s/helm-prod-values/myapp/values.yaml | grep envSecretName\nkubectl get secrets -n myapp\n\n# Restart pods\nkubectl rollout restart deployment myapp -n myapp\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#sync-failures","title":"Sync Failures","text":""},{"location":"DEVELOPER-GUIDE/#problem-argocd-shows-out-of-sync","title":"Problem: ArgoCD shows \"Out of Sync\"","text":"<p>Manual sync: <pre><code># Using kubectl\nkubectl patch application myapp -n argocd --type merge -p '{\"operation\":{\"initiatedBy\":{\"username\":\"admin\"},\"sync\":{\"syncStrategy\":{\"hook\":{}}}}}'\n\n# Or via ArgoCD UI\n# Click \"Sync\" button in UI\n</code></pre></p> <p>Check what's different: <pre><code>kubectl get application myapp -n argocd -o yaml\n</code></pre></p> <p>Look at <code>status.sync.comparedTo</code> vs desired state.</p>"},{"location":"DEVELOPER-GUIDE/#problem-sync-succeeds-but-application-is-degraded","title":"Problem: Sync succeeds but application is \"Degraded\"","text":"<p>Check resource health: <pre><code>kubectl get application myapp -n argocd -o jsonpath='{.status.resources[*].health}'\n</code></pre></p> <p>Common causes: - \u274c Pods not ready - \u274c Deployments not at desired replica count - \u274c Jobs failed</p> <p>Solutions: <pre><code># Check all resources in namespace\nkubectl get all -n myapp\n\n# Check pod events\nkubectl get events -n myapp --sort-by='.lastTimestamp'\n</code></pre></p>"},{"location":"DEVELOPER-GUIDE/#getting-help","title":"Getting Help","text":"<p>If you're stuck:</p> <ol> <li>Check Slack notifications - Error details are often in sync failure messages</li> <li>Check ArgoCD UI - Visual representation of what's wrong</li> <li>Ask platform team - They have full cluster access and can debug further</li> <li>Check documentation - Operations Runbook has more troubleshooting</li> </ol>"},{"location":"DEVELOPER-GUIDE/#documentation","title":"Documentation","text":"<p>This repository's documentation is built with MkDocs using the Material theme and published automatically to Gitea Pages.</p>"},{"location":"DEVELOPER-GUIDE/#viewing-the-docs","title":"Viewing the Docs","text":"<p>The live documentation site is available at:</p> <p>https://git.forteapps.net/Forte/launchpad/pages/</p>"},{"location":"DEVELOPER-GUIDE/#editing-documentation","title":"Editing Documentation","text":"<p>All documentation source files live in the <code>docs/</code> directory as Markdown. To make changes:</p> <ol> <li>Edit the relevant <code>.md</code> file in <code>docs/</code></li> <li>Commit and push to <code>main</code></li> <li>The Gitea Actions workflow automatically rebuilds and deploys the site</li> </ol>"},{"location":"DEVELOPER-GUIDE/#local-preview","title":"Local Preview","text":"<p>To preview documentation changes locally before pushing:</p> <pre><code># Install dependencies (one-time)\npip install mkdocs mkdocs-material\n\n# Start the local dev server\nmkdocs serve\n</code></pre> <p>Then open <code>http://127.0.0.1:8000</code> in your browser. The server live-reloads on file changes.</p>"},{"location":"DEVELOPER-GUIDE/#how-it-works_2","title":"How It Works","text":"<ul> <li>Workflow: <code>.gitea/workflows/docs.yaml</code> triggers on pushes to <code>main</code> that change <code>docs/**</code>, <code>mkdocs.yml</code>, <code>Dockerfile.docs</code>, or <code>nginx.conf</code></li> <li>Build: Installs MkDocs + Material theme, runs <code>mkdocs build</code></li> <li>Deploy: Force-pushes the built <code>site/</code> directory to the <code>gitea-pages</code> branch</li> <li>Serve: Gitea Pages serves the static site from the <code>gitea-pages</code> branch</li> </ul>"},{"location":"DEVELOPER-GUIDE/#best-practices","title":"Best Practices","text":""},{"location":"DEVELOPER-GUIDE/#development-workflow","title":"Development Workflow","text":"<p>\u2705 DO: - Develop and test locally with Docker Compose - Use semantic versioning for releases - Write descriptive commit messages - Test changes in a separate namespace first (if possible) - Monitor Slack for deployment notifications - Document environment variables and configuration</p> <p>\u274c DON'T: - Push directly to production without testing - Use <code>latest</code> tag for Docker images - Bypass CI/CD for \"quick fixes\" - Hard-code configuration values - Ignore deployment failures</p>"},{"location":"DEVELOPER-GUIDE/#configuration-management","title":"Configuration Management","text":"<p>\u2705 DO: - Keep configuration in <code>helm-values</code> repository - Use environment variables for config - Document what each value does - Use reasonable resource limits - Enable ingress and TLS for public services</p> <p>\u274c DON'T: - Hard-code config in application code - Over-allocate resources (wastes money) - Under-allocate resources (causes crashes) - Use HTTP for production services</p>"},{"location":"DEVELOPER-GUIDE/#secret-management","title":"Secret Management","text":"<p>\u2705 DO: - Use kubeseal for all secrets - Store plain secrets in password manager - Rotate secrets regularly - Use different secrets per environment - Document what each secret contains</p> <p>\u274c DON'T: - Commit plain secrets - Share secrets in Slack/email - Reuse secrets across apps - Log secrets in application code</p>"},{"location":"DEVELOPER-GUIDE/#git-workflow","title":"Git Workflow","text":"<p>\u2705 DO: - Use feature branches for changes - Write clear commit messages - Use pull requests for review - Keep commits atomic and focused - Tag releases in application repos</p> <p>\u274c DON'T: - Push directly to <code>main</code> without review (for config repos) - Make multiple unrelated changes in one commit - Use vague commit messages (\"fix\", \"update\") - Force-push to main branches</p>"},{"location":"DEVELOPER-GUIDE/#quick-reference","title":"Quick Reference","text":""},{"location":"DEVELOPER-GUIDE/#common-commands","title":"Common Commands","text":"<pre><code># Check application status\nkubectl get application myapp -n argocd\n\n# View application details\nkubectl describe application myapp -n argocd\n\n# Check pods\nkubectl get pods -n myapp\n\n# View pod logs\nkubectl logs -n myapp <pod-name>\n\n# Restart deployment\nkubectl rollout restart deployment myapp -n myapp\n\n# Port-forward to service\nkubectl port-forward -n myapp service/myapp 8080:3000\n\n# Create secret\nkubectl create secret generic myapp-credentials \\\n --from-literal=KEY=value \\\n --dry-run=client -o yaml > private/myapp-credentials.yaml\n\n# Seal secret\nkubeseal --format=yaml \\\n --cert=pub-cert.pem \\\n < private/myapp-credentials.yaml \\\n > secrets/myapp-credentials-sealed.yaml\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#repository-locations","title":"Repository Locations","text":"<pre><code># Config repository\ncd ~/dev/k8s/launchpad\n\n# Helm values repository\ncd ~/dev/k8s/helm-prod-values\n\n# Helm charts repository\ncd ~/dev/k8s/forte-helm\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#file-paths","title":"File Paths","text":"<pre><code># New application manifest\n~/dev/k8s/launchpad/apps/myapp.yaml\n\n# Application values\n~/dev/k8s/helm-prod-values/myapp/values.yaml\n\n# Sealed secrets\n~/dev/k8s/launchpad/secrets/myapp-credentials-sealed.yaml\n\n# Plain secrets (local only)\n~/dev/k8s/launchpad/private/myapp-credentials.yaml\n</code></pre>"},{"location":"DEVELOPER-GUIDE/#next-steps","title":"Next Steps","text":"<p>Now that you understand the basics:</p> <ol> <li>\u2705 Deploy your first application (follow steps above)</li> <li>\ud83d\udcd6 Read the Operations Runbook for common tasks</li> <li>\ud83d\udcd6 Review Technical Reference for detailed component docs</li> <li>\ud83d\udcd6 Understand GitOps Architecture for the big picture</li> <li>\ud83d\ude80 Start contributing!</li> </ol> <p>Questions? - Slack: #platform-support - Docs: Full documentation index - Help: Contact platform team</p> <p>Last Updated: 2026-04-16</p>"},{"location":"GITOPS-ARCHITECTURE/","title":"GitOps Architecture & Repository Guide","text":""},{"location":"GITOPS-ARCHITECTURE/#table-of-contents","title":"Table of Contents","text":"<ul> <li>Overview</li> <li>Architecture Diagram</li> <li>Repository Structure</li> <li>GitOps Workflow</li> <li>CI/CD Pipeline</li> <li>Security Model</li> </ul>"},{"location":"GITOPS-ARCHITECTURE/#overview","title":"Overview","text":"<p>This Kubernetes cluster uses a GitOps approach powered by ArgoCD, where Git repositories serve as the single source of truth for both infrastructure and application deployments. The cluster is running on UpCloud Managed Kubernetes but is designed to be cloud-agnostic.</p>"},{"location":"GITOPS-ARCHITECTURE/#key-characteristics","title":"Key Characteristics","text":"<ul> <li>Environment: Production (internal use only)</li> <li>Cluster Type: Multi-cluster (upc-dev, upc-prod) via Kustomize overlays</li> <li>GitOps Tool: ArgoCD</li> <li>Deployment Pattern: App-of-Apps</li> <li>Secret Management: Sealed Secrets (kubeseal)</li> <li>Ingress: Traefik with Let's Encrypt TLS</li> <li>Monitoring: Prometheus + Grafana + Loki + Tempo + Fluent-Bit</li> <li>Policy Engine: Kyverno</li> <li>Notifications: Slack integration for sync status</li> </ul>"},{"location":"GITOPS-ARCHITECTURE/#architecture-diagram","title":"Architecture Diagram","text":"<pre><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Developer Workflow \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Application Code \u2502 \u2502 Helm Charts \u2502 \u2502 Helm Values \u2502\n\u2502 Repositories \u2502\u2500\u2500\u2500\u2500\u2500\u2500\u2502 Repository \u2502\u2500\u2500\u2500\u2500\u2500\u2500\u2502 Repository \u2502\n\u2502 (Source Code) \u2502 \u2502 (Templates) \u2502 \u2502 (Config/Env) \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502 \u2502\n \u2502 \u2502 \u2502\n GitHub Actions \u2502 \u2502\n Build & Push Image \u2502 \u2502\n \u2502 \u2502 \u2502\n \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba Update image tag \u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n in helm-values \u2502\n \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Config Repository \u2502\n \u2502 (ArgoCD Applications) \u2502\n \u2502 git.forteapps.net/Forte/ \u2502\n \u2502 launchpad \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u2502\n ArgoCD monitors & syncs\n \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Kubernetes Clusters \u2502\n \u2502 (UpCloud: upc-dev, upc-prod) \u2502\n \u2502 \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 ArgoCD \u2502 \u2502\n \u2502 \u2502 (GitOps Controller) \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2502 \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 Infrastructure Layer \u2502 \u2502\n \u2502 \u2502 - Traefik (Ingress) \u2502 \u2502\n \u2502 \u2502 - Cert-Manager (TLS) \u2502 \u2502\n \u2502 \u2502 - Kyverno (Policies) \u2502 \u2502\n \u2502 \u2502 - Sealed Secrets \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2502 \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 Monitoring Stack \u2502 \u2502\n \u2502 \u2502 - Prometheus \u2502 \u2502\n \u2502 \u2502 - Grafana \u2502 \u2502\n \u2502 \u2502 - Loki \u2502 \u2502\n \u2502 \u2502 - Tempo \u2502 \u2502\n \u2502 \u2502 - Fluent-Bit \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2502 \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 Application Layer \u2502 \u2502\n \u2502 \u2502 - mcp10x \u2502 \u2502\n \u2502 \u2502 - musicman \u2502 \u2502\n \u2502 \u2502 - dot-ai-stack \u2502 \u2502\n \u2502 \u2502 - argo-mcp \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Slack Channel \u2502\n \u2502 (Notifications) \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n</code></pre>"},{"location":"GITOPS-ARCHITECTURE/#repository-structure","title":"Repository Structure","text":""},{"location":"GITOPS-ARCHITECTURE/#1-config-repository-current-repo","title":"1. Config Repository (Current Repo)","text":"<p>Repository: <code>https://git.forteapps.net/Forte/launchpad</code> Purpose: GitOps configuration - ArgoCD Applications and cluster resources Location: <code>C:\\dev\\k8s\\launchpad</code></p> <pre><code>launchpad/\n\u251c\u2500\u2500 bootstrap.sh # Cluster initialization script\n\u251c\u2500\u2500 _app-of-apps-upc-dev.yaml # Root ArgoCD Application (upc-dev cluster)\n\u251c\u2500\u2500 _app-of-apps-upc-prod.yaml # Root ArgoCD Application (upc-prod cluster)\n\u2502\n\u251c\u2500\u2500 infra/ # Infrastructure ArgoCD Applications (Kustomize)\n\u2502 \u251c\u2500\u2500 base/ # Base Application manifests (upc-dev defaults)\n\u2502 \u2502 \u251c\u2500\u2500 kustomization.yaml\n\u2502 \u2502 \u251c\u2500\u2500 traefik-application.yaml\n\u2502 \u2502 \u251c\u2500\u2500 keycloak.yaml\n\u2502 \u2502 \u251c\u2500\u2500 grafana.yaml\n\u2502 \u2502 \u251c\u2500\u2500 gitea.yaml\n\u2502 \u2502 \u251c\u2500\u2500 gitea-actions.yaml\n\u2502 \u2502 \u251c\u2500\u2500 tempo.yaml\n\u2502 \u2502 \u251c\u2500\u2500 renovate.yaml\n\u2502 \u2502 \u251c\u2500\u2500 ... # All other Application manifests\n\u2502 \u2502 \u2514\u2500\u2500 secrets.yaml\n\u2502 \u251c\u2500\u2500 overlays/ # Per-cluster overrides\n\u2502 \u2502 \u251c\u2500\u2500 upc-dev/ # UpCloud Dev (uses base as-is)\n\u2502 \u2502 \u2514\u2500\u2500 upc-prod/ # UpCloud Prod (patches value paths)\n\u2502 \u251c\u2500\u2500 dashboards/ # Grafana dashboard ConfigMaps\n\u2502 \u2514\u2500\u2500 values/ # Helm value overrides for infra\n\u2502 \u251c\u2500\u2500 base/ # Shared values (all clusters)\n\u2502 \u2502 \u251c\u2500\u2500 traefik-values.yaml\n\u2502 \u2502 \u251c\u2500\u2500 keycloak-values.yaml\n\u2502 \u2502 \u251c\u2500\u2500 grafana-values.yaml\n\u2502 \u2502 \u251c\u2500\u2500 prometheus-values.yaml\n\u2502 \u2502 \u251c\u2500\u2500 gitea-values.yaml\n\u2502 \u2502 \u2514\u2500\u2500 ...\n\u2502 \u251c\u2500\u2500 upc-dev/ # upc-dev cluster-specific values\n\u2502 \u2502 \u251c\u2500\u2500 traefik-values.yaml\n\u2502 \u2502 \u251c\u2500\u2500 keycloak-values.yaml\n\u2502 \u2502 \u2514\u2500\u2500 grafana-values.yaml\n\u2502 \u2514\u2500\u2500 upc-prod/ # upc-prod cluster-specific values\n\u2502 \u251c\u2500\u2500 traefik-values.yaml\n\u2502 \u251c\u2500\u2500 keycloak-values.yaml\n\u2502 \u2514\u2500\u2500 grafana-values.yaml\n\u2502\n\u251c\u2500\u2500 apps/ # Business Application ArgoCD manifests (Kustomize)\n\u2502 \u251c\u2500\u2500 base/ # Base app manifests\n\u2502 \u2502 \u251c\u2500\u2500 kustomization.yaml\n\u2502 \u2502 \u251c\u2500\u2500 dot-ai-stack.yaml\n\u2502 \u2502 \u2514\u2500\u2500 ...\n\u2502 \u2514\u2500\u2500 overlays/\n\u2502 \u251c\u2500\u2500 upc-dev/ # Uses base as-is\n\u2502 \u2514\u2500\u2500 upc-prod/ # Patches value paths\n\u2502\n\u251c\u2500\u2500 cluster-resources/ # Cluster-wide Kubernetes resources\n\u2502 \u251c\u2500\u2500 ...\n\u2502 \u2514\u2500\u2500 policies/ # Kyverno policies\n\u2502\n\u251c\u2500\u2500 secrets/ # Application secrets (sealed, per-cluster)\n\u2502 \u2514\u2500\u2500 upc-dev/ # Secrets for upc-dev cluster\n\u2502\n\u251c\u2500\u2500 private/ # Local-only files (NOT in Git)\n\u2502\n\u2514\u2500\u2500 docs/ # Documentation\n</code></pre> <p>Key Points: - <code>_app-of-apps-upc-dev.yaml</code> and <code>_app-of-apps-upc-prod.yaml</code> are the per-cluster root Applications - Kustomize overlays in <code>infra/overlays/</code> render base Applications with per-cluster patches - Helm values are split: <code>values/base/</code> (shared) + <code>values/upc-dev/</code> or <code>values/upc-prod/</code> (cluster-specific) - <code>apps/</code> follows the same base/overlays pattern for business applications - Changes pushed to this repo trigger automatic syncs in ArgoCD - <code>private/</code> folder contains local-only files (Git-ignored)</p>"},{"location":"GITOPS-ARCHITECTURE/#2-helm-charts-repository","title":"2. Helm Charts Repository","text":"<p>Repository: <code>https://github.com/fortedigital/forte-helm</code> Purpose: Reusable Helm chart templates for Forte applications Location: <code>C:\\dev\\k8s\\forte-helm</code></p> <pre><code>forte-helm/\n\u2514\u2500\u2500 forteapp/ # Generic Forte application chart\n \u251c\u2500\u2500 Chart.yaml # Chart metadata (v0.1.0)\n \u251c\u2500\u2500 values.yaml # Default values (base template)\n \u251c\u2500\u2500 templates/\n \u2502 \u251c\u2500\u2500 _helpers.tpl # Template helpers\n \u2502 \u251c\u2500\u2500 namespace.yaml\n \u2502 \u251c\u2500\u2500 deployment.yaml # Main app deployment\n \u2502 \u251c\u2500\u2500 service.yaml\n \u2502 \u251c\u2500\u2500 ingressroute.yaml # Traefik IngressRoute\n \u2502 \u251c\u2500\u2500 certificate.yaml # Cert-Manager Certificate\n \u2502 \u251c\u2500\u2500 configmap.yaml\n \u2502 \u251c\u2500\u2500 secret-auth-tokens.yaml\n \u2502 \u251c\u2500\u2500 hpa.yaml # Horizontal Pod Autoscaler\n \u2502 \u251c\u2500\u2500 database-statefulset.yaml # Optional PostgreSQL DB\n \u2502 \u2514\u2500\u2500 database-service.yaml\n \u2514\u2500\u2500 README.md\n</code></pre> <p>Key Points: - Single generic chart (<code>forteapp</code>) used by all Forte applications - Supports optional PostgreSQL database (StatefulSet) - Configurable authentication (token-based or OIDC) - Traefik IngressRoute with automatic TLS via Cert-Manager - Designed for microservices with similar patterns</p>"},{"location":"GITOPS-ARCHITECTURE/#3-helm-values-repository","title":"3. Helm Values Repository","text":"<p>Repository: <code>git@github.com:fortedigital/helm-values.git</code> Purpose: Environment-specific configuration for each application Location: <code>C:\\dev\\k8s\\helm-prod-values</code></p> <pre><code>helm-prod-values/\n\u251c\u2500\u2500 mcp10x/\n\u2502 \u2514\u2500\u2500 values.yaml # MCP 10X configuration\n\u251c\u2500\u2500 musicman/\n\u2502 \u2514\u2500\u2500 values.yaml # Music Man configuration\n\u251c\u2500\u2500 mcpcoder/\n\u2502 \u2514\u2500\u2500 values.yaml # MCP Coder configuration\n\u2514\u2500\u2500 argocd-mcp/\n \u2514\u2500\u2500 values.yaml # ArgoCD MCP configuration\n</code></pre> <p>Key Points: - Each app has its own folder with <code>values.yaml</code> - Contains environment-specific settings (image tags, env vars, resources, etc.) - Referenced by ArgoCD Applications using multi-source pattern - Image tags are updated here by CI/CD pipelines - Secrets are referenced by name (actual secrets stored as SealedSecrets)</p> <p>Example (<code>mcp10x/values.yaml</code>): <pre><code>app:\n image:\n repository: ghcr.io/fortedigital/10x\n tag: 2.0.4 # Updated by CI/CD\n extraEnv:\n - name: PORT\n value: \"3000\"\n envSecretName: \"app-credentials\" # References SealedSecret\n\ningress:\n enabled: true\n host: mcp10x.forteapps.net # Public domain\n</code></pre></p>"},{"location":"GITOPS-ARCHITECTURE/#4-application-source-code-repositories","title":"4. Application Source Code Repositories","text":"<p>Purpose: Application source code with CI/CD pipelines Examples: Various private repositories</p> <p>Typical Structure: <pre><code>app-repository/\n\u251c\u2500\u2500 src/ # Application source code\n\u251c\u2500\u2500 Dockerfile # Container build definition\n\u251c\u2500\u2500 .github/\n\u2502 \u2514\u2500\u2500 workflows/\n\u2502 \u2514\u2500\u2500 build-and-deploy.yml # GitHub Actions workflow\n\u2514\u2500\u2500 package.json / requirements.txt # Dependencies\n</code></pre></p> <p>CI/CD Workflow (GitHub Actions): 1. Trigger on push to <code>main</code> branch 2. Build Docker image 3. Tag with version (e.g., <code>v2.0.4</code>) 4. Push to container registry (GHCR, Docker Hub, etc.) 5. Update image tag in <code>helm-values</code> repository 6. ArgoCD detects change and syncs automatically</p>"},{"location":"GITOPS-ARCHITECTURE/#gitops-workflow","title":"GitOps Workflow","text":""},{"location":"GITOPS-ARCHITECTURE/#the-app-of-apps-pattern","title":"The App-of-Apps Pattern","text":"<pre><code>_app-of-apps-{upc-dev,upc-prod}.yaml (Root, per cluster)\n \u2502\n \u251c\u2500\u2500 infrastructure-apps (manages infra/)\n \u2502 \u251c\u2500\u2500 cluster-resources-application\n \u2502 \u251c\u2500\u2500 traefik-application\n \u2502 \u251c\u2500\u2500 cert-manager-application\n \u2502 \u251c\u2500\u2500 kyverno\n \u2502 \u251c\u2500\u2500 prometheus\n \u2502 \u251c\u2500\u2500 grafana\n \u2502 \u251c\u2500\u2500 tempo\n \u2502 \u2514\u2500\u2500 ... (other infra apps)\n \u2502\n \u2514\u2500\u2500 enterprise-apps (manages apps/)\n \u251c\u2500\u2500 mcp10x\n \u251c\u2500\u2500 musicman\n \u251c\u2500\u2500 dot-ai-stack\n \u2514\u2500\u2500 argo-mcp\n</code></pre> <p>How It Works: 1. Bootstrap script installs ArgoCD and applies <code>_app-of-apps-upc-dev.yaml</code> (or <code>upc-prod</code>) 2. ArgoCD creates the root Application which monitors the appropriate <code>infra/overlays/</code> folder 3. Kustomize renders base Applications with cluster-specific patches 4. <code>enterprise-apps</code> Application monitors the cluster's <code>apps/overlays/</code> folder 5. ArgoCD continuously syncs (every 60s) and auto-heals drift</p>"},{"location":"GITOPS-ARCHITECTURE/#sync-waves-ordering","title":"Sync Waves & Ordering","text":"<p>Applications deploy in order using <code>argocd.argoproj.io/sync-wave</code> annotations:</p> <pre><code>Wave -1: Namespaces (created first)\nWave 0: Kyverno (policies ready before resources)\nWave 1: Cluster resources, infrastructure apps\nWave 2+: Business applications\n</code></pre> <p>Example: <pre><code>metadata:\n annotations:\n argocd.argoproj.io/sync-wave: \"1\"\n</code></pre></p>"},{"location":"GITOPS-ARCHITECTURE/#multi-source-pattern","title":"Multi-Source Pattern","text":"<p>Applications like <code>mcp10x</code> and <code>musicman</code> use multiple sources:</p> <pre><code>spec:\n sources:\n - repoURL: https://github.com/fortedigital/forte-helm\n path: forteapp # Helm chart templates\n helm:\n valueFiles:\n - $values/mcp10x/values.yaml # Reference to second source\n\n - repoURL: git@github.com:fortedigital/helm-values.git\n targetRevision: HEAD\n ref: values # Named reference\n</code></pre> <p>Benefits: - Chart templates separated from configuration - Single chart reused across all apps - Easy to update all apps by changing the chart - Environment-specific values isolated in separate repo</p>"},{"location":"GITOPS-ARCHITECTURE/#multi-cluster-pattern","title":"Multi-Cluster Pattern","text":"<p>Kustomize overlays enable deploying the same Applications across clusters with different configurations:</p> <pre><code># infra/base/ contains default (upc-dev) Applications\n# Helm values are layered: base + cluster-specific\nvalueFiles:\n- $values/infra/values/base/traefik-values.yaml # Shared config\n- $values/infra/values/upc-dev/traefik-values.yaml # Cluster-specific\n\n# infra/overlays/upc-prod/kustomization.yaml patches the second valueFile\npatches:\n- target:\n kind: Application\n name: traefik\n patch: |\n - op: replace\n path: /spec/sources/0/helm/valueFiles/1\n value: $values/infra/values/upc-prod/traefik-values.yaml\n</code></pre> <p>Benefits: - Single source of truth for Application definitions - Cluster-specific values isolated per overlay - Easy to add new clusters by creating a new overlay - Base values shared across all clusters reduce duplication</p>"},{"location":"GITOPS-ARCHITECTURE/#cicd-pipeline","title":"CI/CD Pipeline","text":""},{"location":"GITOPS-ARCHITECTURE/#continuous-integration","title":"Continuous Integration","text":"<p>Application Repositories contain GitHub Actions workflows:</p> <pre><code>name: Build and Deploy\n\non:\n push:\n branches: [ main ]\n\njobs:\n build:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v3\n\n - name: Build Docker image\n run: docker build -t ghcr.io/fortedigital/app:$VERSION .\n\n - name: Push to registry\n run: docker push ghcr.io/fortedigital/app:$VERSION\n\n - name: Update Helm values\n run: |\n git clone git@github.com:fortedigital/helm-values.git\n cd helm-values/app\n sed -i \"s/tag: .*/tag: $VERSION/\" values.yaml\n git commit -am \"Update app to $VERSION\"\n git push\n</code></pre>"},{"location":"GITOPS-ARCHITECTURE/#continuous-deployment","title":"Continuous Deployment","text":"<p>ArgoCD automatically syncs when changes are detected:</p> <ol> <li>Config Repo Change:</li> <li>Developer updates <code>apps/myapp.yaml</code></li> <li>Pushes to <code>launchpad</code> repo</li> <li>ArgoCD detects change (60s reconciliation)</li> <li> <p>Syncs application to cluster</p> </li> <li> <p>Helm Values Change:</p> </li> <li>CI/CD updates <code>helm-values/myapp/values.yaml</code></li> <li>ArgoCD detects change</li> <li>Pulls new Helm chart with updated values</li> <li> <p>Applies to cluster</p> </li> <li> <p>Sync Policy: <pre><code>syncPolicy:\n automated:\n prune: true # Remove deleted resources\n selfHeal: true # Revert manual changes\n retry:\n limit: 5 # Retry up to 5 times\n backoff:\n duration: 5s\n maxDuration: 3m\n</code></pre></p> </li> </ol>"},{"location":"GITOPS-ARCHITECTURE/#deployment-validation","title":"Deployment Validation","text":"<p>Before applying, ArgoCD: - \u2705 Validates YAML syntax - \u2705 Checks Kubernetes schema - \u2705 Runs server-side dry-run - \u2705 Verifies resource quotas - \u2705 Applies Kyverno policies</p> <p>After applying: - \u2705 Waits for resources to become healthy - \u2705 Sends Slack notification (success/failure) - \u2705 Tracks sync status in UI</p>"},{"location":"GITOPS-ARCHITECTURE/#security-model","title":"Security Model","text":""},{"location":"GITOPS-ARCHITECTURE/#secret-management","title":"Secret Management","text":"<p>Sealed Secrets encrypt secrets for safe Git storage:</p> <pre><code># Developer creates plain secret locally\nkubectl create secret generic app-creds \\\n --from-literal=API_KEY=secret123 \\\n --dry-run=client -o yaml > private/app-creds.yaml\n\n# Seal the secret using kubeseal\nkubeseal --format=yaml \\\n --cert=pub-cert.pem \\\n < private/app-creds.yaml \\\n > secrets/app-creds-sealed.yaml\n\n# Commit sealed secret to Git\ngit add secrets/app-creds-sealed.yaml\ngit commit -m \"Add app credentials\"\n</code></pre> <p>Storage: - \u2705 Sealed secrets committed to Git - \u274c Plain secrets kept in <code>private/</code> (Git-ignored) or discarded - \u26a0\ufe0f Secret rotation process not yet established</p>"},{"location":"GITOPS-ARCHITECTURE/#kyverno-policies","title":"Kyverno Policies","text":"<p>Policy Engine enforces security rules:</p> <ol> <li> <p>Secret Cloning: Automatically clones secrets to new namespaces <pre><code># cluster-resources/policies/secret-cloner.yaml\n# Secrets labeled \"allowedToBeCloned: true\" are synced\n</code></pre></p> </li> <li> <p>Default Namespace Blocker: Prevents use of <code>default</code> namespace</p> </li> <li>Bare Pod Cleaner: Removes pods without controllers (Deployments/StatefulSets)</li> <li>Deployment Verifier: Ensures pods have proper controllers</li> <li>Auth Sidecar Injector: Injects authentication proxy based on annotations</li> </ol>"},{"location":"GITOPS-ARCHITECTURE/#repository-access","title":"Repository Access","text":"<p>Private Repository Credentials stored as SealedSecrets:</p> <pre><code># cluster-resources/forte10x-repo-credentials-sealed.yaml\n</code></pre> <p>ArgoCD uses these to access private Helm values repositories.</p>"},{"location":"GITOPS-ARCHITECTURE/#network-security","title":"Network Security","text":"<p>Traefik Ingress with TLS: - All HTTP traffic redirects to HTTPS - Let's Encrypt automatic certificate renewal - Cert-Manager manages certificate lifecycle - Per-application IngressRoutes with dedicated certificates</p>"},{"location":"GITOPS-ARCHITECTURE/#authentication","title":"Authentication","text":"<p>Application-Level Auth (optional): - Token-based authentication (static tokens) - OIDC integration (Keycloak, Okta, etc.) - Auth sidecar injected via Kyverno policy - Tokens stored in SealedSecrets</p> <p>Example: <pre><code># In deployment.yaml template\nannotations:\n policies.forteapps.io/auth: \"true\"\n policies.forteapps.io/auth-token-secret-name: \"app-tokens\"\n</code></pre></p>"},{"location":"GITOPS-ARCHITECTURE/#monitoring-observability","title":"Monitoring & Observability","text":""},{"location":"GITOPS-ARCHITECTURE/#stack-components","title":"Stack Components","text":"<ol> <li>Prometheus: Metrics collection and storage</li> <li>Grafana: Metrics visualization and dashboards</li> <li>Loki: Log aggregation</li> <li>Tempo: Distributed tracing (OTLP)</li> <li>Fluent-Bit: Log shipping from pods to Loki</li> <li>Trivy: Container vulnerability scanning</li> </ol>"},{"location":"GITOPS-ARCHITECTURE/#slack-notifications","title":"Slack Notifications","text":"<p>All ArgoCD applications send notifications to shared Slack channel:</p> <pre><code>metadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.slack: \"\"\n notifications.argoproj.io/subscribe.on-sync-failed.slack: \"\"\n notifications.argoproj.io/subscribe.on-degraded.slack: \"\"\n</code></pre> <p>Notifications include: - \u2705 Sync succeeded - \u274c Sync failed - \u26a0\ufe0f Application degraded</p>"},{"location":"GITOPS-ARCHITECTURE/#disaster-recovery","title":"Disaster Recovery","text":""},{"location":"GITOPS-ARCHITECTURE/#cluster-rebuild","title":"Cluster Rebuild","text":"<p>Current State: No backup routines exist yet. Cluster can be rebuilt from Git.</p> <p>Rebuild Process: 1. Provision new Kubernetes cluster 2. Clone <code>launchpad</code> repository 3. Run <code>./bootstrap.sh</code> 4. ArgoCD installs and syncs all applications 5. Manually recreate unsealed secrets and seal them</p> <p>Data Loss: - Currently: Data loss is acceptable (internal use) - Future: One stateful application may require backup strategy</p>"},{"location":"GITOPS-ARCHITECTURE/#gitops-advantages-for-dr","title":"GitOps Advantages for DR","text":"<p>\u2705 Infrastructure as Code: Entire cluster defined in Git \u2705 Reproducible: Cluster can be rebuilt identically \u2705 Auditable: All changes tracked in Git history \u2705 Rollback: Easy to revert to previous Git commit \u2705 Multi-Cluster: Same config can deploy to multiple clusters</p>"},{"location":"GITOPS-ARCHITECTURE/#best-practices","title":"Best Practices","text":""},{"location":"GITOPS-ARCHITECTURE/#repository-organization","title":"Repository Organization","text":"<p>\u2705 DO: - Separate infrastructure (<code>infra/</code>) from applications (<code>apps/</code>) - Use sync waves to control deployment order - Keep secrets in <code>private/</code> folder (Git-ignored) - Commit only sealed secrets to Git - Use multi-source pattern for chart/values separation</p> <p>\u274c DON'T: - Commit plain secrets to Git - Mix infrastructure and application configs - Hard-code environment-specific values in charts - Manually modify resources in cluster (use Git)</p>"},{"location":"GITOPS-ARCHITECTURE/#gitops-workflow_1","title":"GitOps Workflow","text":"<p>\u2705 DO: - All changes through Git (single source of truth) - Use PR reviews for production changes - Test changes in isolated namespaces first - Monitor ArgoCD sync status - Respond to Slack notifications</p> <p>\u274c DON'T: - Use <code>kubectl apply</code> directly (breaks GitOps) - Ignore sync failures - Bypass ArgoCD for \"quick fixes\" - Edit resources in place (<code>kubectl edit</code>)</p>"},{"location":"GITOPS-ARCHITECTURE/#application-development","title":"Application Development","text":"<p>\u2705 DO: - Follow the <code>forteapp</code> chart pattern - Use semantic versioning for image tags - Update helm-values via CI/CD - Test locally with Docker Compose - Document environment variables</p> <p>\u274c DON'T: - Use <code>latest</code> image tag - Hard-code configuration in code - Skip local testing - Deploy untested images to production</p>"},{"location":"GITOPS-ARCHITECTURE/#next-steps","title":"Next Steps","text":"<p>\ud83d\udcd6 Continue to: - Developer Guide - Learn how to deploy and manage applications - Operations Runbook - Common operational tasks - Technical Reference - Detailed component documentation</p> <p>Last Updated: 2026-03-16 Maintained By: Platform Team Questions?: Contact #platform-support on Slack</p>"},{"location":"OPERATIONS-RUNBOOK/","title":"Operations Runbook","text":""},{"location":"OPERATIONS-RUNBOOK/#table-of-contents","title":"Table of Contents","text":"<ul> <li>Overview</li> <li>Cluster Bootstrap</li> <li>Initial Cluster Setup</li> <li>ArgoCD Repository Access Setup</li> <li>Day-to-Day Operations</li> <li>Application Management</li> <li>Secret Management</li> <li>Monitoring & Alerting</li> <li>Troubleshooting</li> <li>Disaster Recovery</li> <li>Maintenance Procedures</li> </ul>"},{"location":"OPERATIONS-RUNBOOK/#overview","title":"Overview","text":"<p>This runbook provides operational procedures for maintaining the Kubernetes cluster and managing applications. It's intended for platform engineers and operators with full cluster access.</p>"},{"location":"OPERATIONS-RUNBOOK/#operator-prerequisites","title":"Operator Prerequisites","text":"<ul> <li>\u2705 Full kubectl access to cluster</li> <li>\u2705 Write access to all Git repositories</li> <li>\u2705 ArgoCD UI access</li> <li>\u2705 Slack notifications configured</li> <li>\u2705 Understanding of Kubernetes concepts</li> </ul>"},{"location":"OPERATIONS-RUNBOOK/#cluster-bootstrap","title":"Cluster Bootstrap","text":""},{"location":"OPERATIONS-RUNBOOK/#initial-cluster-setup","title":"Initial Cluster Setup","text":"<p>Bootstrap a new cluster from scratch:</p>"},{"location":"OPERATIONS-RUNBOOK/#prerequisites","title":"Prerequisites","text":"<ol> <li>Kubernetes cluster running (UpCloud or any K8s cluster)</li> <li>kubectl configured with admin access</li> <li>Repositories cloned locally</li> </ol> <pre><code># Verify cluster access\nkubectl cluster-info\nkubectl get nodes\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#bootstrap-procedure","title":"Bootstrap Procedure","text":"<pre><code># 1. Clone config repository\ngit clone https://git.forteapps.net/Forte/launchpad\ncd launchpad\n\n# 2. Set cluster name (optional)\nexport CLUSTER_NAME=\"prod-cluster-01\"\n\n# 3. Run bootstrap script\n./bootstrap.sh\n</code></pre> <p>What Happens: 1. \u2705 Installs ArgoCD via Helm 2. \u2705 Configures ArgoCD with custom values 3. \u2705 Applies root App-of-Apps manifest 4. \u2705 ArgoCD automatically syncs all applications 5. \u2705 Infrastructure and apps deploy in waves</p>"},{"location":"OPERATIONS-RUNBOOK/#verify-bootstrap","title":"Verify Bootstrap","text":"<pre><code># Wait for ArgoCD to be ready\nkubectl wait --for=condition=available --timeout=300s \\\n deployment/argocd-server -n argocd\n\n# Check ArgoCD applications\nkubectl get applications -n argocd\n\n# Expected output: infrastructure-apps, enterprise-apps, and all child apps\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#post-bootstrap-steps","title":"Post-Bootstrap Steps","text":"<ol> <li>Configure DNS for ingress domains:</li> <li><code>argocd.127.0.0.1.nip.io</code> (local dev)</li> <li> <p><code>*.forteapps.net</code> (production)</p> </li> <li> <p>Verify Let's Encrypt certificates: <pre><code>kubectl get certificate --all-namespaces\nkubectl get clusterissuer\n</code></pre></p> </li> <li> <p>Check Kyverno policies: <pre><code>kubectl get clusterpolicy\n</code></pre></p> </li> <li> <p>Verify monitoring stack: <pre><code>kubectl get pods -n monitoring\n</code></pre></p> </li> <li> <p>Test Slack notifications by triggering a sync</p> </li> </ol>"},{"location":"OPERATIONS-RUNBOOK/#argocd-repository-access-setup","title":"ArgoCD Repository Access Setup","text":"<p>ArgoCD needs SSH access to private Git repositories to pull manifests and Helm values. This section covers setting up deploy keys for GitHub repositories.</p>"},{"location":"OPERATIONS-RUNBOOK/#why-deploy-keys","title":"Why Deploy Keys?","text":"<ul> <li>Read-only access: Deploy keys provide secure, read-only access to repositories</li> <li>No user credentials: No need to share personal SSH keys or tokens</li> <li>Repository-specific: Each repository gets its own key for better security</li> <li>Revocable: Easy to revoke access without affecting other repositories</li> </ul>"},{"location":"OPERATIONS-RUNBOOK/#prerequisites_1","title":"Prerequisites","text":"<ul> <li>kubectl access to the cluster</li> <li>Write access to the GitHub repository</li> <li>ArgoCD installed and running</li> </ul>"},{"location":"OPERATIONS-RUNBOOK/#setup-procedure","title":"Setup Procedure","text":"<p>Step 1: Generate SSH Key Pair</p> <p>Generate a dedicated SSH key for ArgoCD without a passphrase (required for automated access):</p> <pre><code># Generate ED25519 key (recommended - smaller and more secure)\nssh-keygen -t ed25519 -C \"argocd-deploy-key-launchpad\" -f argocd-deploy-key -N \"\"\n\n# Or RSA key if ED25519 is not supported\nssh-keygen -t rsa -b 4096 -C \"argocd-deploy-key-launchpad\" -f argocd-deploy-key -N \"\"\n</code></pre> <p>This creates two files: - <code>argocd-deploy-key</code> - Private key (keep secret) - <code>argocd-deploy-key.pub</code> - Public key (add to GitHub)</p> <p>Step 2: Add Public Key to GitHub</p> <ol> <li> <p>Copy the public key: <pre><code>cat argocd-deploy-key.pub\n</code></pre></p> </li> <li> <p>Go to GitHub repository settings:</p> </li> <li>Navigate to: <code>https://git.forteapps.net/Forte/launchpad/settings/keys</code></li> <li> <p>Or: Repository \u2192 Settings \u2192 Deploy keys</p> </li> <li> <p>Click \"Add deploy key\"</p> </li> <li>Title: <code>ArgoCD Production Cluster</code></li> <li>Key: Paste the public key content</li> <li>\u2610 Allow write access (leave unchecked - read-only is sufficient)</li> <li> <p>Click \"Add key\"</p> </li> <li> <p>Repeat for the <code>helm-values</code> repository if it's private: <pre><code># Generate separate key for helm-values repo\nssh-keygen -t ed25519 -C \"argocd-deploy-key-helm-values\" -f argocd-helm-values-key -N \"\"\n\n# Add to: https://github.com/fortedigital/helm-values/settings/keys\n</code></pre></p> </li> </ol> <p>Step 3: Create Kubernetes Secret</p> <p>Add the private key to ArgoCD as a repository secret:</p> <p>Save the following file in private/ (gitignored) folder as secret.yaml <pre><code> apiVersion: v1\n kind: Secret\n metadata:\n name: forte-helm-repo\n namespace: argocd\n labels:\n argocd.argoproj.io/secret-type: repository\n stringData:\n type: git\n url: ssh://git@git.forteapps.net:2222/Forte/forte-helm.git\n sshPrivateKey: |\n <paste your private key here>\n project: default\n</code></pre> Seal the secret using <code>kubeseal</code> command <pre><code>kubeseal --format=yaml \\\n --namespace=argocd \\\n < private/secret.yaml \\\n > secrets/forte-helm-repo-secret-sealed.yaml\n</code></pre></p> <p>Step 4: Register Repository in ArgoCD</p> <p>Check in secrets/forte-helm-repo-secret-sealed.yaml and let Argo sync and create the secret.</p> <p>Step 5: Verify Repository Access</p> <pre><code># Check if repository is connected\nkubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=repository\n\n# Verify connection in ArgoCD UI\n# Settings \u2192 Repositories \u2192 Should show \"Successful\" status\n\n# Test by creating an application\nkubectl apply -f _app-of-apps-upc-dev.yaml # or _app-of-apps-upc-prod.yaml\n\n# Check application sync status\nkubectl get applications -n argocd\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#testing-repository-access","title":"Testing Repository Access","text":"<p>Create a test application to verify SSH access:</p> <pre><code>cat > /tmp/test-repo-access.yaml <<EOF\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n name: test-repo-access\n namespace: argocd\nspec:\n project: default\n source:\n repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git\n targetRevision: main\n path: cluster-resources\n destination:\n server: https://kubernetes.default.svc\n namespace: default\n syncPolicy:\n automated: null # Manual sync for testing\nEOF\n\nkubectl apply -f /tmp/test-repo-access.yaml\n\n# Check if ArgoCD can access the repository\nkubectl describe application test-repo-access -n argocd\n\n# Look for sync status - should show repository contents\nkubectl get application test-repo-access -n argocd -o jsonpath='{.status.sync.status}'\n\n# Clean up test application\nkubectl delete application test-repo-access -n argocd\nrm /tmp/test-repo-access.yaml\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#security-best-practices","title":"Security Best Practices","text":"<ol> <li> <p>Secure Private Keys <pre><code># Store private key securely and delete local copy\n# Option 1: Store in password manager (recommended)\n# Option 2: Backup to encrypted storage\n\n# Delete local private key after adding to Kubernetes\nshred -u argocd-deploy-key\n\n# Or on Windows\n# Remove-Item -Path argocd-deploy-key -Force\n</code></pre></p> </li> <li> <p>Rotate Keys Regularly <pre><code># Generate new key\nssh-keygen -t ed25519 -C \"argocd-deploy-key-$(date +%Y%m)\" -f argocd-new-key -N \"\"\n\n# Add new public key to GitHub (keep old key for now)\n\n# Update Kubernetes secret\nkubectl create secret generic repo-launchpad \\\n --from-file=sshPrivateKey=argocd-new-key \\\n --namespace=argocd \\\n --dry-run=client -o yaml | kubectl apply -f -\n\n# Test access, then remove old deploy key from GitHub\n\n# Clean up\nshred -u argocd-new-key\n</code></pre></p> </li> <li> <p>Audit Repository Access <pre><code># List all repository secrets\nkubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=repository\n\n# Review deploy keys in GitHub\n# Visit: https://git.forteapps.net/Forte/launchpad/settings/keys\n</code></pre></p> </li> <li> <p>Use Different Keys per Repository</p> </li> <li>Don't reuse the same deploy key across repositories</li> <li>If one key is compromised, only one repository is affected</li> <li>Easier to track and audit access</li> </ol>"},{"location":"OPERATIONS-RUNBOOK/#troubleshooting-repository-access","title":"Troubleshooting Repository Access","text":"<p>Issue: \"permission denied (publickey)\"</p> <pre><code># Check if secret exists\nkubectl get secret repo-launchpad -n argocd\n\n# Verify secret has correct label\nkubectl get secret repo-launchpad -n argocd -o yaml | grep argocd.argoproj.io/secret-type\n\n# Check ArgoCD application controller logs\nkubectl logs -n argocd deployment/argocd-application-controller | grep -i \"permission denied\"\n\n# Verify deploy key is added to GitHub\n# Visit: https://git.forteapps.net/Forte/launchpad/settings/keys\n</code></pre> <p>Issue: \"Host key verification failed\"</p> <pre><code># Add GitHub to known_hosts\nkubectl exec -n argocd deployment/argocd-repo-server -- \\\n ssh-keyscan github.com >> ~/.ssh/known_hosts\n\n# Or disable strict host key checking (less secure)\nkubectl patch secret repo-launchpad -n argocd \\\n --type merge \\\n -p '{\"stringData\":{\"insecure\":\"true\"}}'\n</code></pre> <p>Issue: Repository shows as \"Unknown\" status</p> <pre><code># Check repository server logs\nkubectl logs -n argocd deployment/argocd-repo-server\n\n# Refresh repository connection\nkubectl delete secret repo-launchpad -n argocd\n# Recreate secret (see Step 3 above)\n\n# Restart ArgoCD components\nkubectl rollout restart deployment argocd-repo-server -n argocd\nkubectl rollout restart deployment argocd-application-controller -n argocd\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#multiple-repository-setup","title":"Multiple Repository Setup","text":"<p>For the three-repository pattern (launchpad, forte-helm, helm-values):</p> <pre><code># 1. launchpad (main config repo)\nssh-keygen -t ed25519 -C \"argocd-launchpad\" -f key-sturdy -N \"\"\n# Add key-sturdy.pub to: https://git.forteapps.net/Forte/launchpad/settings/keys\n\n# 2. helm-values (private values repo)\nssh-keygen -t ed25519 -C \"argocd-helm-values\" -f key-helm-values -N \"\"\n# Add key-helm-values.pub to: https://github.com/fortedigital/helm-values/settings/keys\n\n# 3. forte-helm (private helm charts repo)\n\n# Create secrets\nkubectl create secret generic repo-launchpad \\\n --from-file=sshPrivateKey=key-sturdy \\\n --namespace=argocd --dry-run=client -o yaml | \\\n kubectl label --local -f - argocd.argoproj.io/secret-type=repository --dry-run=client -o yaml | \\\n kubectl apply -f -\n\nkubectl create secret generic repo-helm-values \\\n --from-file=sshPrivateKey=key-helm-values \\\n --namespace=argocd --dry-run=client -o yaml | \\\n kubectl label --local -f - argocd.argoproj.io/secret-type=repository --dry-run=client -o yaml | \\\n kubectl apply -f -\n\n# Clean up keys\nshred -u key-sturdy key-helm-values\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#converting-https-to-ssh","title":"Converting HTTPS to SSH","text":"<p>If you're currently using HTTPS and want to switch to SSH:</p> <pre><code># 1. Generate and add deploy key (see steps above)\n\n# 2. Update all Application manifests\n# Change from:\n# repoURL: https://git.forteapps.net/Forte/launchpad\n# To:\n# repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git\n\n# 3. Update and commit\nfind . -name \"*.yaml\" -type f -exec sed -i 's|https://github.com/fortedigital/|git@github.com:fortedigital/|g' {} +\n\ngit add .\ngit commit -m \"Switch from HTTPS to SSH for repository access\"\ngit push\n\n# 4. ArgoCD will automatically re-sync with new SSH URLs\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#day-to-day-operations","title":"Day-to-Day Operations","text":""},{"location":"OPERATIONS-RUNBOOK/#monitoring-argocd-sync-status","title":"Monitoring ArgoCD Sync Status","text":""},{"location":"OPERATIONS-RUNBOOK/#via-slack","title":"Via Slack","text":"<p>All applications send notifications to shared Slack channel: - \u2705 <code>on-sync-succeeded</code> - Deployment succeeded - \u274c <code>on-sync-failed</code> - Deployment failed - \u26a0\ufe0f <code>on-degraded</code> - Application unhealthy</p>"},{"location":"OPERATIONS-RUNBOOK/#via-cli","title":"Via CLI","text":"<pre><code># List all applications\nkubectl get applications -n argocd\n\n# Watch application status\nkubectl get applications -n argocd -w\n\n# Get detailed status\nkubectl describe application myapp -n argocd\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#via-argocd-ui","title":"Via ArgoCD UI","text":"<pre><code># Port forward to UI\nkubectl port-forward svc/argocd-server -n argocd 8080:443\n\n# Access: https://localhost:8080\n# No login required (insecure mode for internal use)\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#checking-application-health","title":"Checking Application Health","text":"<pre><code># Quick health check for all apps\nkubectl get applications -n argocd \\\n -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status\n\n# Expected output:\n# NAME SYNC HEALTH\n# infrastructure-apps Synced Healthy\n# enterprise-apps Synced Healthy\n# mcp10x Synced Healthy\n# musicman Synced Healthy\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#manual-sync","title":"Manual Sync","text":"<p>Force sync an application:</p> <pre><code># Trigger sync\nkubectl patch application myapp -n argocd \\\n --type merge \\\n -p '{\"metadata\":{\"annotations\":{\"argocd.argoproj.io/refresh\":\"hard\"}}}'\n\n# Or via ArgoCD CLI (if installed)\nargocd app sync myapp\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#pausing-auto-sync","title":"Pausing Auto-Sync","text":"<p>Temporarily disable automatic syncing:</p> <pre><code># Edit application\nkubectl edit application myapp -n argocd\n\n# Set automated to null\nspec:\n syncPolicy:\n automated: null # Disable auto-sync\n\n# Re-enable later\nspec:\n syncPolicy:\n automated:\n prune: true\n selfHeal: true\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#application-management","title":"Application Management","text":""},{"location":"OPERATIONS-RUNBOOK/#deploying-a-new-application","title":"Deploying a New Application","text":"<p>See Developer Guide for detailed steps.</p> <p>Quick checklist: - [ ] Create <code>helm-values/myapp/values.yaml</code> - [ ] Create <code>apps/myapp.yaml</code> in config repo - [ ] Create SealedSecret if needed - [ ] Commit and push changes - [ ] Verify sync in Slack/ArgoCD - [ ] Configure DNS for domain - [ ] Test application accessibility</p>"},{"location":"OPERATIONS-RUNBOOK/#removing-an-application","title":"Removing an Application","text":""},{"location":"OPERATIONS-RUNBOOK/#safe-removal-procedure","title":"Safe Removal Procedure","text":"<pre><code># 1. Delete ArgoCD Application (with cascade)\nkubectl delete application myapp -n argocd\n\n# This will:\n# - Remove application from ArgoCD\n# - Delete all Kubernetes resources (cascade)\n# - Remove namespace\n\n# 2. Clean up Git repositories\ncd ~/dev/k8s/launchpad\ngit rm apps/myapp.yaml\ngit commit -m \"Remove myapp application\"\ngit push\n\ncd ~/dev/k8s/helm-prod-values\ngit rm -r myapp/\ngit commit -m \"Remove myapp values\"\ngit push\n\n# 3. Remove sealed secrets (if any)\ncd ~/dev/k8s/launchpad\ngit rm secrets/myapp-credentials-sealed.yaml\ngit commit -m \"Remove myapp secrets\"\ngit push\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#removal-without-cascade","title":"Removal Without Cascade","text":"<p>To remove from ArgoCD but keep resources running:</p> <pre><code># Delete application with no cascade\nkubectl patch application myapp -n argocd \\\n -p '{\"metadata\":{\"finalizers\":[]}}' --type merge\nkubectl delete application myapp -n argocd\n\n# Resources remain in cluster but are no longer managed\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#scaling-applications","title":"Scaling Applications","text":""},{"location":"OPERATIONS-RUNBOOK/#manual-scaling","title":"Manual Scaling","text":"<pre><code># Scale deployment directly\nkubectl scale deployment myapp -n myapp --replicas=3\n\n# Note: If selfHeal is enabled, this will be reverted\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#gitops-scaling","title":"GitOps Scaling","text":"<p>Update <code>helm-values/myapp/values.yaml</code>:</p> <pre><code>app:\n replicaCount: 3 # Change from 1 to 3\n</code></pre> <p>Commit and push - ArgoCD will sync.</p>"},{"location":"OPERATIONS-RUNBOOK/#auto-scaling-hpa","title":"Auto-Scaling (HPA)","text":"<p>Enable Horizontal Pod Autoscaler:</p> <pre><code># In helm-values/myapp/values.yaml\napp:\n hpa:\n enabled: true\n minReplicas: 2\n maxReplicas: 10\n targetCPUUtilizationPercentage: 70\n</code></pre> <p>Note: Remove <code>replicaCount</code> from ArgoCD ignore list if using HPA:</p> <pre><code># In apps/myapp.yaml\nignoreDifferences:\n- group: apps\n kind: Deployment\n jsonPointers:\n - /spec/replicas # Remove this line\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#rolling-back-deployments","title":"Rolling Back Deployments","text":""},{"location":"OPERATIONS-RUNBOOK/#option-1-git-revert","title":"Option 1: Git Revert","text":"<pre><code># Find the commit before the bad change\ncd ~/dev/k8s/helm-prod-values\ngit log --oneline myapp/values.yaml\n\n# Revert to previous version\ngit revert <commit-hash>\ngit push\n\n# ArgoCD will sync the rollback\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#option-2-manual-rollback","title":"Option 2: Manual Rollback","text":"<pre><code># Rollback to previous revision\nkubectl rollout undo deployment myapp -n myapp\n\n# Note: This will be reverted by ArgoCD selfHeal\n# Make permanent by updating Git\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#option-3-change-image-tag","title":"Option 3: Change Image Tag","text":"<pre><code># Edit helm-values\ncd ~/dev/k8s/helm-prod-values\nvim myapp/values.yaml\n\n# Change image tag to previous version\napp:\n image:\n tag: v1.0.0 # Roll back from v1.0.1\n\n# Commit and push\ngit add myapp/values.yaml\ngit commit -m \"Rollback myapp to v1.0.0\"\ngit push\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#resource-updates","title":"Resource Updates","text":""},{"location":"OPERATIONS-RUNBOOK/#update-resource-limits","title":"Update Resource Limits","text":"<pre><code># In helm-values/myapp/values.yaml\napp:\n resources:\n requests:\n cpu: 200m # Increased from 100m\n memory: 512Mi # Increased from 256Mi\n limits:\n cpu: 1000m\n memory: 2Gi\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#enable-database","title":"Enable Database","text":"<pre><code># In helm-values/myapp/values.yaml\ndb:\n enabled: true\n persistence:\n size: 10Gi # Increase storage\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#secret-management","title":"Secret Management","text":""},{"location":"OPERATIONS-RUNBOOK/#creating-secrets","title":"Creating Secrets","text":""},{"location":"OPERATIONS-RUNBOOK/#step-1-get-public-certificate","title":"Step 1: Get Public Certificate","text":"<pre><code># Fetch sealed-secrets public cert (one-time)\nkubeseal --fetch-cert \\\n --controller-name=sealed-secrets-controller \\\n --controller-namespace=kube-system \\\n > pub-cert.pem\n\n# Save this certificate for future use\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#step-2-create-plain-secret","title":"Step 2: Create Plain Secret","text":"<pre><code># Method 1: From literal values\nkubectl create secret generic myapp-credentials \\\n --from-literal=API_KEY=secret123 \\\n --from-literal=DB_PASSWORD=pass456 \\\n --namespace=myapp \\\n --dry-run=client -o yaml > private/myapp-credentials.yaml\n\n# Method 2: From file\nkubectl create secret generic myapp-credentials \\\n --from-file=.env \\\n --namespace=myapp \\\n --dry-run=client -o yaml > private/myapp-credentials.yaml\n\n# Method 3: From multiple files\nkubectl create secret generic myapp-credentials \\\n --from-file=api-key.txt \\\n --from-file=db-password.txt \\\n --namespace=myapp \\\n --dry-run=client -o yaml > private/myapp-credentials.yaml\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#step-3-seal-secret","title":"Step 3: Seal Secret","text":"<pre><code>kubeseal --format=yaml \\\n --cert=pub-cert.pem \\\n --namespace=myapp \\\n < private/myapp-credentials.yaml \\\n > secrets/myapp-credentials-sealed.yaml\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#step-4-commit-sealed-secret","title":"Step 4: Commit Sealed Secret","text":"<pre><code>git add secrets/myapp-credentials-sealed.yaml\ngit commit -m \"Add myapp credentials\"\ngit push\n\n# Delete plain secret\nrm private/myapp-credentials.yaml\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#updating-secrets","title":"Updating Secrets","text":"<pre><code># 1. Create new version\nkubectl create secret generic myapp-credentials \\\n --from-literal=API_KEY=new-secret-key \\\n --from-literal=DB_PASSWORD=new-password \\\n --namespace=myapp \\\n --dry-run=client -o yaml > private/myapp-credentials.yaml\n\n# 2. Seal it\nkubeseal --format=yaml \\\n --cert=pub-cert.pem \\\n --namespace=myapp \\\n < private/myapp-credentials.yaml \\\n > secrets/myapp-credentials-sealed.yaml\n\n# 3. Commit\ngit add secrets/myapp-credentials-sealed.yaml\ngit commit -m \"Update myapp credentials\"\ngit push\n\n# 4. Restart pods to pick up new secret\nkubectl rollout restart deployment myapp -n myapp\n\n# 5. Delete plain secret\nrm private/myapp-credentials.yaml\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#viewing-secrets-unsealed","title":"Viewing Secrets (Unsealed)","text":"<pre><code># List secrets in namespace\nkubectl get secrets -n myapp\n\n# Describe secret (doesn't show values)\nkubectl describe secret myapp-credentials -n myapp\n\n# View secret values (base64 encoded)\nkubectl get secret myapp-credentials -n myapp -o yaml\n\n# Decode secret value\nkubectl get secret myapp-credentials -n myapp \\\n -o jsonpath='{.data.API_KEY}' | base64 -d\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#secret-cloning-kyverno","title":"Secret Cloning (Kyverno)","text":"<p>Secrets labeled <code>allowedToBeCloned: \"true\"</code> in the <code>secrets</code> namespace are automatically cloned to new namespaces.</p> <pre><code># Example: secrets-namespace.yaml\napiVersion: v1\nkind: Secret\nmetadata:\n name: shared-credentials\n namespace: secrets\n labels:\n allowedToBeCloned: \"true\"\ntype: Opaque\ndata:\n API_KEY: <base64-encoded-value>\n</code></pre> <p>When a new namespace is created, Kyverno automatically copies this secret.</p>"},{"location":"OPERATIONS-RUNBOOK/#authentication-secrets","title":"Authentication Secrets","text":"<p>Applications using the authentication sidecar require specific secrets depending on the auth mode.</p>"},{"location":"OPERATIONS-RUNBOOK/#token-mode-secrets","title":"Token Mode Secrets","text":"<p>Token-based auth uses an <code>auth-tokens</code> Secret:</p> <pre><code># Method 1: From Helm values (automatic)\n# Tokens specified in values.yaml are automatically created\n\n# Method 2: Manual creation\nkubectl create secret generic auth-tokens \\\n --from-literal=tokens=\"token1\ntoken2\ntoken3\" \\\n --namespace=myapp\n\n# Method 3: From file\necho \"d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823\" > tokens.txt\necho \"8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7\" >> tokens.txt\nkubectl create secret generic auth-tokens \\\n --from-file=tokens=tokens.txt \\\n --namespace=myapp\nrm tokens.txt\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#oidc-mode-secrets","title":"OIDC Mode Secrets","text":"<p>OIDC auth requires an <code>auth-oidc</code> Secret with two keys:</p> <pre><code># Generate secrets\nCLIENT_SECRET=\"your-oidc-client-secret-from-provider\"\nCOOKIE_SECRET=$(openssl rand -hex 32)\n\n# Create plain secret\nkubectl create secret generic auth-oidc \\\n --from-literal=client-secret=$CLIENT_SECRET \\\n --from-literal=cookie-secret=$COOKIE_SECRET \\\n --namespace=myapp \\\n --dry-run=client -o yaml > private/myapp-auth-oidc.yaml\n\n# Seal it\nkubeseal --format=yaml \\\n --cert=pub-cert.pem \\\n --namespace=myapp \\\n < private/myapp-auth-oidc.yaml \\\n > secrets/myapp-auth-oidc-sealed.yaml\n\n# Apply sealed secret\nkubectl apply -f secrets/myapp-auth-oidc-sealed.yaml\n\n# Commit to Git\ngit add secrets/myapp-auth-oidc-sealed.yaml\ngit commit -m \"Add OIDC secrets for myapp\"\ngit push\n\n# Clean up\nrm private/myapp-auth-oidc.yaml\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#rotating-authentication-secrets","title":"Rotating Authentication Secrets","text":"<p>Token Rotation:</p> <pre><code># Generate new token\nNEW_TOKEN=$(openssl rand -hex 32)\n\n# Get current tokens\nkubectl get secret auth-tokens -n myapp -o yaml > /tmp/tokens.yaml\n\n# Edit tokens (add new, optionally remove old)\n# Then re-seal and apply\n\n# Restart pods to use new tokens\nkubectl rollout restart deployment myapp -n myapp\n</code></pre> <p>OIDC Secret Rotation:</p> <pre><code># Rotate cookie secret (safe - invalidates existing sessions)\nNEW_COOKIE_SECRET=$(openssl rand -hex 32)\n\n# Recreate secret\nkubectl create secret generic auth-oidc \\\n --from-literal=client-secret=$CLIENT_SECRET \\\n --from-literal=cookie-secret=$NEW_COOKIE_SECRET \\\n --namespace=myapp \\\n --dry-run=client -o yaml | \\\n kubeseal --format=yaml --cert=pub-cert.pem --namespace=myapp | \\\n kubectl apply -f -\n\n# Restart to pick up new secret\nkubectl rollout restart deployment myapp -n myapp\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#viewing-authentication-secrets","title":"Viewing Authentication Secrets","text":"<pre><code># List auth-related secrets\nkubectl get secrets -n myapp | grep auth\n\n# View token secret (tokens are in plain text in the Secret)\nkubectl get secret auth-tokens -n myapp -o jsonpath='{.data.tokens}' | base64 -d\n\n# View OIDC secret keys (values are base64 encoded)\nkubectl get secret auth-oidc -n myapp -o jsonpath='{.data.client-secret}' | base64 -d\nkubectl get secret auth-oidc -n myapp -o jsonpath='{.data.cookie-secret}' | base64 -d\n</code></pre> <p>See: Developer Guide - Enabling Authentication for complete authentication setup guide.</p>"},{"location":"OPERATIONS-RUNBOOK/#monitoring-alerting","title":"Monitoring & Alerting","text":""},{"location":"OPERATIONS-RUNBOOK/#prometheus-metrics","title":"Prometheus Metrics","text":"<pre><code># Port forward to Prometheus\nkubectl port-forward -n monitoring svc/prometheus-server 9090:80\n\n# Access: http://localhost:9090\n</code></pre> <p>Common Queries: <pre><code># CPU usage per pod\nsum(rate(container_cpu_usage_seconds_total[5m])) by (pod)\n\n# Memory usage per pod\nsum(container_memory_usage_bytes) by (pod)\n\n# Request rate per service\nrate(http_requests_total[5m])\n</code></pre></p>"},{"location":"OPERATIONS-RUNBOOK/#grafana-dashboards","title":"Grafana Dashboards","text":"<pre><code># Port forward to Grafana\nkubectl port-forward -n monitoring svc/grafana 3000:80\n\n# Access: http://localhost:3000\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#loki-logs","title":"Loki Logs","text":"<pre><code># Port forward to Loki\nkubectl port-forward -n monitoring svc/loki 3100:3100\n\n# Query logs\ncurl -G -s 'http://localhost:3100/loki/api/v1/query_range' \\\n --data-urlencode 'query={namespace=\"myapp\"}' \\\n --data-urlencode 'start=1h' | jq\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#tempo-traces","title":"Tempo Traces","text":"<pre><code># Port forward to Tempo query API\nkubectl port-forward -n monitoring svc/tempo 3200:3200\n\n# Access: http://localhost:3200\n</code></pre> <p>Query traces via Grafana: 1. Open Grafana \u2192 Explore 2. Select Tempo datasource 3. Use TraceQL or search by service name</p> <p>Verify Traefik is sending traces: <pre><code># Check Traefik logs for OTLP export errors\nkubectl logs -n traefik-system -l app.kubernetes.io/name=traefik | grep -i \"traces export\"\n\n# Check Tempo is receiving data\nkubectl logs -n monitoring -l app.kubernetes.io/name=tempo | grep \"receiver\"\n</code></pre></p> <p>Trace-to-log correlation: - Click a trace span in Grafana \u2192 linked Loki logs appear (by namespace, pod, container) - Trace-to-metrics links to Prometheus by service name</p>"},{"location":"OPERATIONS-RUNBOOK/#fluent-bit-log-shipping","title":"Fluent-Bit Log Shipping","text":"<p>Verify Fluent-Bit is shipping logs:</p> <pre><code># Check Fluent-Bit pods\nkubectl get pods -n monitoring | grep fluent-bit\n\n# Check logs\nkubectl logs -n monitoring daemonset/fluent-bit\n\n# Verify Loki is receiving logs\nkubectl logs -n monitoring deployment/loki | grep \"POST /loki/api/v1/push\"\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#trivy-vulnerability-scanning","title":"Trivy Vulnerability Scanning","text":"<pre><code># Check Trivy scan results\nkubectl get vulnerabilityreports --all-namespaces\n\n# View report for specific pod\nkubectl describe vulnerabilityreport -n myapp <report-name>\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#slack-notifications","title":"Slack Notifications","text":"<p>All applications have Slack notifications enabled:</p> <pre><code>metadata:\n annotations:\n notifications.argoproj.io/subscribe.on-sync-succeeded.slack: \"\"\n notifications.argoproj.io/subscribe.on-sync-failed.slack: \"\"\n notifications.argoproj.io/subscribe.on-degraded.slack: \"\"\n</code></pre> <p>Test Notification: <pre><code># Trigger a sync to test\nkubectl patch application myapp -n argocd \\\n --type merge \\\n -p '{\"metadata\":{\"annotations\":{\"argocd.argoproj.io/refresh\":\"hard\"}}}'\n</code></pre></p>"},{"location":"OPERATIONS-RUNBOOK/#troubleshooting","title":"Troubleshooting","text":""},{"location":"OPERATIONS-RUNBOOK/#application-wont-sync","title":"Application Won't Sync","text":""},{"location":"OPERATIONS-RUNBOOK/#check-application-status","title":"Check Application Status","text":"<pre><code>kubectl describe application myapp -n argocd\n</code></pre> <p>Look for errors in: - <code>Status.Conditions</code> - <code>Status.OperationState</code></p>"},{"location":"OPERATIONS-RUNBOOK/#common-issues","title":"Common Issues","text":"<p>Issue 1: Image Pull Error <pre><code># Error: ErrImagePull, ImagePullBackOff\n\n# Check if image exists\ndocker pull ghcr.io/fortedigital/myapp:v1.0.0\n\n# Check image pull secrets\nkubectl get secrets -n myapp | grep regcred\n\n# Check pod events\nkubectl describe pod -n myapp <pod-name>\n</code></pre></p> <p>Issue 2: Invalid YAML <pre><code># Error: unable to decode manifest\n\n# Validate YAML locally\nkubectl apply --dry-run=client -f apps/myapp.yaml\n\n# Check ArgoCD application controller logs\nkubectl logs -n argocd deployment/argocd-application-controller | grep myapp\n</code></pre></p> <p>Issue 3: Resource Quota Exceeded <pre><code># Error: exceeded quota\n\n# Check namespace quotas\nkubectl get resourcequota -n myapp\nkubectl describe resourcequota -n myapp\n\n# Increase quota or reduce resource requests\n</code></pre></p>"},{"location":"OPERATIONS-RUNBOOK/#pod-crashes","title":"Pod Crashes","text":""},{"location":"OPERATIONS-RUNBOOK/#crashloopbackoff","title":"CrashLoopBackOff","text":"<pre><code># Check pod status\nkubectl get pods -n myapp\n\n# View logs\nkubectl logs -n myapp <pod-name>\nkubectl logs -n myapp <pod-name> --previous # Previous container\n\n# Check events\nkubectl describe pod -n myapp <pod-name>\n</code></pre> <p>Common Causes: - Application error (check logs) - Missing environment variables - Wrong port configuration - Missing secrets - Insufficient memory/CPU</p>"},{"location":"OPERATIONS-RUNBOOK/#imagepullbackoff","title":"ImagePullBackOff","text":"<pre><code># Check image name\nkubectl get deployment myapp -n myapp -o yaml | grep image\n\n# Verify credentials\nkubectl get secret -n myapp\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#pending","title":"Pending","text":"<pre><code># Check why pod is pending\nkubectl describe pod -n myapp <pod-name>\n\n# Common reasons:\n# - Insufficient resources on nodes\n# - PVC not bound\n# - Node selector doesn't match\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#ingress-tls-issues","title":"Ingress / TLS Issues","text":""},{"location":"OPERATIONS-RUNBOOK/#application-not-accessible","title":"Application Not Accessible","text":"<pre><code># Check IngressRoute\nkubectl get ingressroute -n myapp\nkubectl describe ingressroute myapp -n myapp\n\n# Check Traefik\nkubectl get pods -n traefik\nkubectl logs -n traefik deployment/traefik\n\n# Test with port-forward\nkubectl port-forward -n myapp service/myapp 8080:3000\ncurl http://localhost:8080\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#certificate-issues","title":"Certificate Issues","text":"<pre><code># Check certificates\nkubectl get certificate -n myapp\nkubectl describe certificate myapp-tls -n myapp\n\n# Check cert-manager\nkubectl get clusterissuer\nkubectl logs -n cert-manager deployment/cert-manager\n\n# Check Let's Encrypt challenges\nkubectl get challenges --all-namespaces\n</code></pre> <p>Manual Certificate Renewal: <pre><code># Delete and recreate certificate\nkubectl delete certificate myapp-tls -n myapp\n\n# Certificate will be automatically recreated\n</code></pre></p>"},{"location":"OPERATIONS-RUNBOOK/#database-issues","title":"Database Issues","text":""},{"location":"OPERATIONS-RUNBOOK/#postgresql-wont-start","title":"PostgreSQL Won't Start","text":"<pre><code># Check StatefulSet\nkubectl get statefulset -n myapp\nkubectl describe statefulset postgres -n myapp\n\n# Check PVC\nkubectl get pvc -n myapp\nkubectl describe pvc -n myapp\n\n# Check logs\nkubectl logs -n myapp postgres-0\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#data-persistence","title":"Data Persistence","text":"<pre><code># Verify PVC is bound\nkubectl get pvc -n myapp\n\n# Check storage class\nkubectl get storageclass\n\n# Resize PVC (if supported)\nkubectl edit pvc postgres-data-postgres-0 -n myapp\n# Change: storage: 10Gi (from 5Gi)\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#kyverno-policy-issues","title":"Kyverno Policy Issues","text":""},{"location":"OPERATIONS-RUNBOOK/#policy-violations","title":"Policy Violations","text":"<pre><code># List policies\nkubectl get clusterpolicy\n\n# Check policy reports\nkubectl get policyreport --all-namespaces\n\n# View specific policy\nkubectl describe clusterpolicy secret-cloner\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#secret-not-cloned","title":"Secret Not Cloned","text":"<pre><code># Check if secret has label\nkubectl get secret -n secrets --show-labels\n\n# Check Kyverno logs\nkubectl logs -n kyverno deployment/kyverno\n\n# Manually trigger by recreating namespace\nkubectl delete ns test-ns\nkubectl create ns test-ns\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#argocd-issues","title":"ArgoCD Issues","text":""},{"location":"OPERATIONS-RUNBOOK/#argocd-ui-not-accessible","title":"ArgoCD UI Not Accessible","text":"<pre><code># Check ArgoCD pods\nkubectl get pods -n argocd\n\n# Restart ArgoCD server\nkubectl rollout restart deployment argocd-server -n argocd\n\n# Port forward\nkubectl port-forward svc/argocd-server -n argocd 8080:443\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#sync-takes-too-long","title":"Sync Takes Too Long","text":"<pre><code># Check application controller logs\nkubectl logs -n argocd deployment/argocd-application-controller\n\n# Increase timeout (in apps/myapp.yaml)\nspec:\n syncPolicy:\n retry:\n backoff:\n maxDuration: 5m # Increase from 3m\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#disaster-recovery","title":"Disaster Recovery","text":""},{"location":"OPERATIONS-RUNBOOK/#backup-strategy","title":"Backup Strategy","text":"<p>Current State: No automated backups</p> <p>What Needs Backup: - \u274c Cluster state (not backed up - recreate via GitOps) - \u274c Persistent volumes (currently not critical) - \u2705 Git repositories (GitHub provides backup) - \u26a0\ufe0f Secrets (sealed secrets in Git, unseal keys need safekeeping)</p>"},{"location":"OPERATIONS-RUNBOOK/#cluster-rebuild","title":"Cluster Rebuild","text":"<p>Scenario: Complete cluster failure</p> <pre><code># 1. Provision new Kubernetes cluster\n\n# 2. Configure kubectl\nkubectl config use-context new-cluster\nkubectl cluster-info\n\n# 3. Bootstrap cluster\ncd ~/dev/k8s/launchpad\n./bootstrap.sh\n\n# 4. Wait for ArgoCD to sync all applications\nkubectl get applications -n argocd -w\n\n# 5. Recreate any unsealed secrets (from password manager)\n# 6. Configure DNS for new cluster IPs\n# 7. Verify all applications are healthy\n</code></pre> <p>Time Estimate: 30-60 minutes</p> <p>Data Loss: - Ephemeral data: Lost - Database data: Lost (no backups currently) - Configuration: No loss (in Git)</p>"},{"location":"OPERATIONS-RUNBOOK/#future-backup-plan","title":"Future Backup Plan","text":"<p>Recommended:</p> <ol> <li> <p>Velero for cluster backups <pre><code>helm install velero vmware-tanzu/velero \\\n --namespace velero \\\n --create-namespace \\\n --set configuration.provider=aws \\\n --set configuration.backupStorageLocation[0].bucket=cluster-backups\n</code></pre></p> </li> <li> <p>PostgreSQL backups via CronJob <pre><code># pg-backup-cronjob.yaml\nkind: CronJob\nspec:\n schedule: \"0 2 * * *\" # Daily at 2am\n jobTemplate:\n spec:\n template:\n spec:\n containers:\n - name: pg-dump\n image: postgres:16-alpine\n command:\n - /bin/sh\n - -c\n - pg_dump -U $DB_USER -d $DB_NAME > /backup/dump-$(date +%Y%m%d).sql\n</code></pre></p> </li> <li> <p>Sealed Secrets private key backup <pre><code># Backup sealed-secrets controller private key\nkubectl get secret -n kube-system sealed-secrets-key \\\n -o yaml > sealed-secrets-key-backup.yaml\n\n# Store in secure location (password manager, vault)\n</code></pre></p> </li> </ol>"},{"location":"OPERATIONS-RUNBOOK/#maintenance-procedures","title":"Maintenance Procedures","text":""},{"location":"OPERATIONS-RUNBOOK/#upgrading-argocd","title":"Upgrading ArgoCD","text":"<pre><code># Check current version\nkubectl get deployment argocd-server -n argocd \\\n -o jsonpath='{.spec.template.spec.containers[0].image}'\n\n# Update version in values\nvim infra/values/base/argocd-values.yaml\n\n# Or upgrade via Helm directly\nhelm upgrade argocd argo-cd \\\n --repo https://argoproj.github.io/argo-helm \\\n --namespace argocd \\\n --values infra/values/base/argocd-values.yaml \\\n --version 6.0.0 # New version\n\n# Verify\nkubectl get pods -n argocd\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#upgrading-kubernetes-version","title":"Upgrading Kubernetes Version","text":"<pre><code># UpCloud: Upgrade via control panel or CLI\n\n# After upgrade, verify cluster\nkubectl version\nkubectl get nodes\n\n# Check for deprecated APIs\nkubectl api-resources\n\n# Update any deprecated resources in Git\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#rotating-tls-certificates","title":"Rotating TLS Certificates","text":"<p>Let's Encrypt certificates auto-renew, but if manual rotation is needed:</p> <pre><code># Delete certificate to force renewal\nkubectl delete certificate myapp-tls -n myapp\n\n# Cert-manager will automatically recreate\nkubectl get certificate -n myapp -w\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#cleaning-up-old-resources","title":"Cleaning Up Old Resources","text":"<pre><code># List all namespaces\nkubectl get namespaces\n\n# Remove unused namespaces\nkubectl delete namespace old-app\n\n# Clean up ArgoCD applications\nkubectl get applications -n argocd\nkubectl delete application old-app -n argocd\n\n# Clean up old Docker images (on nodes)\n# SSH to nodes and run:\ndocker image prune -a --filter \"until=720h\" # 30 days\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#dns-management","title":"DNS Management","text":"<p>Adding New Subdomain:</p> <ol> <li> <p>Add DNS A record pointing to Traefik LoadBalancer IP <pre><code># Get LoadBalancer IP\nkubectl get svc -n traefik traefik -o jsonpath='{.status.loadBalancer.ingress[0].ip}'\n</code></pre></p> </li> <li> <p>Add to DNS provider: <pre><code>myapp.forteapps.net A <LoadBalancer-IP>\n</code></pre></p> </li> <li> <p>Verify DNS propagation: <pre><code>nslookup myapp.forteapps.net\ndig myapp.forteapps.net\n</code></pre></p> </li> </ol>"},{"location":"OPERATIONS-RUNBOOK/#monitoring-resource-usage","title":"Monitoring Resource Usage","text":"<pre><code># Node resource usage\nkubectl top nodes\n\n# Pod resource usage\nkubectl top pods --all-namespaces\n\n# Identify resource hogs\nkubectl top pods --all-namespaces --sort-by=memory\nkubectl top pods --all-namespaces --sort-by=cpu\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#advanced-operations","title":"Advanced Operations","text":""},{"location":"OPERATIONS-RUNBOOK/#adding-a-new-infrastructure-component","title":"Adding a New Infrastructure Component","text":"<p>Example: Adding Redis</p> <pre><code># 1. Create application manifest in base/\ncat > infra/base/redis-application.yaml <<EOF\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n name: redis\n namespace: argocd\n annotations:\n argocd.argoproj.io/sync-wave: \"1\"\nspec:\n project: default\n sources:\n - repoURL: https://charts.bitnami.com/bitnami\n chart: redis\n targetRevision: 18.0.0\n helm:\n releaseName: redis\n valueFiles:\n - \\$values/infra/values/base/redis-values.yaml\n - repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git\n targetRevision: HEAD\n ref: values\n destination:\n server: https://kubernetes.default.svc\n namespace: redis\n syncPolicy:\n automated:\n prune: true\n selfHeal: true\n syncOptions:\n - CreateNamespace=true\nEOF\n\n# 2. Add to base kustomization\n# Edit infra/base/kustomization.yaml and add: - redis-application.yaml\n\n# 3. Create base values file\ncat > infra/values/base/redis-values.yaml <<EOF\nauth:\n enabled: true\nEOF\n\n# 4. Commit and push\ngit add infra/base/redis-application.yaml infra/values/base/redis-values.yaml infra/base/kustomization.yaml\ngit commit -m \"Add Redis infrastructure component\"\ngit push\n\n# 5. ArgoCD will auto-sync within 60 seconds\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#multi-cluster-setup","title":"Multi-Cluster Setup","text":"<p>The repository supports multiple clusters via Kustomize overlays:</p> <ul> <li>upc-dev (default): <code>infra/overlays/upc-dev/</code> \u2014 uses base Applications as-is</li> <li>upc-prod: <code>infra/overlays/upc-prod/</code> \u2014 patches value file paths from <code>upc-dev</code> to <code>upc-prod</code></li> </ul> <p>Each cluster has its own: - Root app-of-apps file: <code>_app-of-apps-upc-dev.yaml</code> / <code>_app-of-apps-upc-prod.yaml</code> - Cluster-specific Helm values: <code>infra/values/upc-dev/</code> / <code>infra/values/upc-prod/</code> - Sealed secrets: <code>secrets/upc-dev/</code> (others as needed) - Apps overlay: <code>apps/overlays/upc-dev/</code> / <code>apps/overlays/upc-prod/</code></p> <p>To add a new cluster, create a new overlay directory (e.g., <code>infra/overlays/upc-staging/</code>) with patches that swap the value file paths.</p>"},{"location":"OPERATIONS-RUNBOOK/#blue-green-deployments","title":"Blue-Green Deployments","text":"<pre><code># Deploy blue version\nhelm install myapp-blue forteapp \\\n --set app.image.tag=v1.0.0\n\n# Deploy green version\nhelm install myapp-green forteapp \\\n --set app.image.tag=v2.0.0\n\n# Switch traffic via IngressRoute\nkubectl patch ingressroute myapp -n myapp --type merge \\\n -p '{\"spec\":{\"routes\":[{\"services\":[{\"name\":\"myapp-green\"}]}]}}'\n\n# Remove blue deployment after validation\nhelm uninstall myapp-blue\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#emergency-procedures","title":"Emergency Procedures","text":""},{"location":"OPERATIONS-RUNBOOK/#emergency-rollback","title":"Emergency Rollback","text":"<pre><code># Immediate rollback\nkubectl rollout undo deployment myapp -n myapp\n\n# Update Git to make permanent\ncd ~/dev/k8s/helm-prod-values\ngit revert HEAD\ngit push\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#emergency-scale-down","title":"Emergency Scale Down","text":"<pre><code># Scale to zero (maintenance mode)\nkubectl scale deployment myapp -n myapp --replicas=0\n\n# Update Git\nvim helm-values/myapp/values.yaml\n# Set replicaCount: 0\ngit commit -am \"Scale down myapp for maintenance\"\ngit push\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#emergency-application-removal","title":"Emergency Application Removal","text":"<pre><code># Remove application but keep data\nkubectl patch application myapp -n argocd \\\n -p '{\"metadata\":{\"finalizers\":[]}}' --type merge\nkubectl delete application myapp -n argocd\n\n# Resources remain in cluster\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#useful-scripts","title":"Useful Scripts","text":""},{"location":"OPERATIONS-RUNBOOK/#sync-all-applications","title":"Sync All Applications","text":"<pre><code>#!/bin/bash\n# sync-all.sh\nfor app in $(kubectl get applications -n argocd -o name); do\n kubectl patch $app -n argocd \\\n --type merge \\\n -p '{\"metadata\":{\"annotations\":{\"argocd.argoproj.io/refresh\":\"hard\"}}}'\ndone\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#check-all-applications-health","title":"Check All Applications Health","text":"<pre><code>#!/bin/bash\n# health-check.sh\nkubectl get applications -n argocd \\\n -o custom-columns=\\\nNAME:.metadata.name,\\\nSYNC:.status.sync.status,\\\nHEALTH:.status.health.status,\\\nMESSAGE:.status.health.message\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#seal-secret-helper","title":"Seal Secret Helper","text":"<pre><code>#!/bin/bash\n# seal-secret.sh\nNAMESPACE=${1:-default}\nSECRET_FILE=${2:-private/secret.yaml}\nOUTPUT_FILE=${3:-secrets/secret-sealed.yaml}\n\nkubeseal --format=yaml \\\n --cert=pub-cert.pem \\\n --namespace=$NAMESPACE \\\n < $SECRET_FILE \\\n > $OUTPUT_FILE\n\necho \"Sealed secret created: $OUTPUT_FILE\"\necho \"Remember to delete: $SECRET_FILE\"\n</code></pre>"},{"location":"OPERATIONS-RUNBOOK/#checklist-templates","title":"Checklist Templates","text":""},{"location":"OPERATIONS-RUNBOOK/#new-application-deployment-checklist","title":"New Application Deployment Checklist","text":"<ul> <li>[ ] Application code repository created</li> <li>[ ] Dockerfile created and tested</li> <li>[ ] GitHub Actions workflow configured</li> <li>[ ] Helm values created in <code>helm-prod-values/</code></li> <li>[ ] ArgoCD application manifest created in <code>apps/</code></li> <li>[ ] Secrets created and sealed</li> <li>[ ] DNS record added for domain</li> <li>[ ] Application synced successfully</li> <li>[ ] Health check passed</li> <li>[ ] Slack notification received</li> <li>[ ] Application accessible via domain</li> <li>[ ] Monitoring configured</li> <li>[ ] Documentation updated</li> </ul>"},{"location":"OPERATIONS-RUNBOOK/#incident-response-checklist","title":"Incident Response Checklist","text":"<ul> <li>[ ] Incident identified (Slack alert, monitoring)</li> <li>[ ] Severity assessed</li> <li>[ ] Incident channel created</li> <li>[ ] Initial investigation (logs, metrics, events)</li> <li>[ ] Root cause identified</li> <li>[ ] Mitigation applied</li> <li>[ ] Verification of fix</li> <li>[ ] Post-mortem scheduled</li> <li>[ ] Documentation updated</li> </ul> <p>Last Updated: 2026-03-16 Maintained By: Platform Team Emergency Contact: #platform-support on Slack</p>"},{"location":"REFERENCE/","title":"Technical Reference","text":""},{"location":"REFERENCE/#table-of-contents","title":"Table of Contents","text":"<ul> <li>Architecture Components</li> <li>Repository Reference</li> <li>Helm Chart Reference</li> <li>ArgoCD Configuration</li> <li>Infrastructure Components</li> <li>Kyverno Policies</li> <li>Configuration Reference</li> <li>API Endpoints</li> <li>Glossary</li> </ul>"},{"location":"REFERENCE/#architecture-components","title":"Architecture Components","text":""},{"location":"REFERENCE/#cluster-specifications","title":"Cluster Specifications","text":"Component Value Provider UpCloud Managed Kubernetes Environment Production (internal use) Cluster Count Multi-cluster (upc-dev, upc-prod) GitOps Tool ArgoCD Ingress Controller Traefik v2 Certificate Management Cert-Manager + Let's Encrypt Policy Engine Kyverno Secret Management Sealed Secrets (Bitnami) Monitoring Prometheus + Grafana Logging Loki + Fluent-Bit Tracing Tempo (OTLP) Container Scanning Trivy Version Control Gitea"},{"location":"REFERENCE/#network-architecture","title":"Network Architecture","text":"<pre><code>Internet\n \u2502\n \u25bc\n[DNS: *.forteapps.net]\n \u2502\n \u25bc\n[UpCloud LoadBalancer]\n \u2502\n \u25bc\n[Traefik Ingress Controller]\n \u2502\n \u251c\u2500\u2500\u25ba IngressRoute (TLS termination via Cert-Manager)\n \u2502\n \u251c\u2500\u2500\u25ba Service (ClusterIP)\n \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u25ba Pod (Application Container)\n \u2502\n \u2514\u2500\u2500\u25ba Service (Database - ClusterIP)\n \u2502\n \u2514\u2500\u2500\u25ba StatefulSet (PostgreSQL)\n</code></pre>"},{"location":"REFERENCE/#repository-reference","title":"Repository Reference","text":""},{"location":"REFERENCE/#config-repository-launchpad","title":"Config Repository: <code>launchpad</code>","text":"<p>URL: <code>https://git.forteapps.net/Forte/launchpad</code></p>"},{"location":"REFERENCE/#directory-structure","title":"Directory Structure","text":"<pre><code>launchpad/\n\u251c\u2500\u2500 bootstrap.sh # Cluster initialization script\n\u251c\u2500\u2500 _app-of-apps-upc-dev.yaml # Root ArgoCD Application (upc-dev)\n\u251c\u2500\u2500 _app-of-apps-upc-prod.yaml # Root ArgoCD Application (upc-prod)\n\u2502\n\u251c\u2500\u2500 infra/ # Infrastructure applications\n\u2502 \u251c\u2500\u2500 cluster-resources-application.yaml\n\u2502 \u251c\u2500\u2500 enterprise-apps.yaml\n\u2502 \u251c\u2500\u2500 traefik-application.yaml\n\u2502 \u251c\u2500\u2500 cert-manager-application.yaml\n\u2502 \u251c\u2500\u2500 kyverno.yaml\n\u2502 \u251c\u2500\u2500 kyverno-policies.yaml\n\u2502 \u251c\u2500\u2500 prometheus.yaml\n\u2502 \u251c\u2500\u2500 grafana.yaml\n\u2502 \u251c\u2500\u2500 loki.yaml\n\u2502 \u251c\u2500\u2500 tempo.yaml\n\u2502 \u251c\u2500\u2500 fluent-bit.yaml\n\u2502 \u251c\u2500\u2500 trivy.yaml\n\u2502 \u251c\u2500\u2500 gitea.yaml\n\u2502 \u251c\u2500\u2500 gitea-actions.yaml\n\u2502 \u251c\u2500\u2500 sealedsecrets.yaml\n\u2502 \u251c\u2500\u2500 secrets.yaml\n\u2502 \u251c\u2500\u2500 renovate.yaml\n\u2502 \u2514\u2500\u2500 values/\n\u2502 \u251c\u2500\u2500 argocd-values.yaml\n\u2502 \u251c\u2500\u2500 prometheus-values.yaml\n\u2502 \u251c\u2500\u2500 grafana-values.yaml\n\u2502 \u251c\u2500\u2500 loki-values.yaml\n\u2502 \u251c\u2500\u2500 tempo-values.yaml\n\u2502 \u251c\u2500\u2500 gitea-values.yaml\n\u2502 \u251c\u2500\u2500 gitea-actions-values.yaml\n\u2502 \u251c\u2500\u2500 fluent-bit-values.yaml\n\u2502 \u2514\u2500\u2500 renovate-values.yaml\n\u2502\n\u251c\u2500\u2500 apps/ # Business applications\n\u2502 \u251c\u2500\u2500 mcp10x.yaml\n\u2502 \u251c\u2500\u2500 musicman.yaml\n\u2502 \u251c\u2500\u2500 dot-ai-stack.yaml\n\u2502 \u2514\u2500\u2500 argo-mcp.yaml\n\u2502\n\u251c\u2500\u2500 cluster-resources/ # Cluster-level resources\n\u2502 \u251c\u2500\u2500 cert-manager-namespace.yaml\n\u2502 \u251c\u2500\u2500 secrets-namespace.yaml\n\u2502 \u251c\u2500\u2500 letsencrypt-issuer.yaml\n\u2502 \u251c\u2500\u2500 kyverno-config.yaml\n\u2502 \u251c\u2500\u2500 argocd-notifications-secret-sealed.yaml\n\u2502 \u251c\u2500\u2500 forte10x-repo-credentials-sealed.yaml\n\u2502 \u251c\u2500\u2500 mcp10x-repo-credentials-sealed.yaml\n\u2502 \u2514\u2500\u2500 policies/\n\u2502 \u251c\u2500\u2500 deployment-verifier.yaml\n\u2502 \u251c\u2500\u2500 label-checker.yaml\n\u2502 \u251c\u2500\u2500 bare-pod-cleaner.yaml\n\u2502 \u251c\u2500\u2500 replicaset-cleaner.yaml\n\u2502 \u251c\u2500\u2500 default-ns-blocker.yaml\n\u2502 \u251c\u2500\u2500 secret-cloner.yaml\n\u2502 \u251c\u2500\u2500 keycloak-client-cloner.yaml\n\u2502 \u2514\u2500\u2500 auth-sidecar-injector.yaml\n\u2502\n\u251c\u2500\u2500 secrets/ # Application secrets (sealed)\n\u2502 \u251c\u2500\u2500 argocd-mcp-credentials.yaml\n\u2502 \u251c\u2500\u2500 dot-ai-secrets.yaml\n\u2502 \u251c\u2500\u2500 gitea-credentials-sealed.yaml\n\u2502 \u251c\u2500\u2500 gitea-runner-token-sealed.yaml\n\u2502 \u251c\u2500\u2500 mcp10x-credentials-sealed.yaml\n\u2502 \u2514\u2500\u2500 musicman-credentials.yaml\n\u2502\n\u251c\u2500\u2500 private/ # Local-only (Git-ignored)\n\u2502 \u251c\u2500\u2500 *.yaml\n\u2502 \u2514\u2500\u2500 *.sh\n\u2502\n\u2514\u2500\u2500 docs/ # Documentation\n \u251c\u2500\u2500 GITOPS-ARCHITECTURE.md\n \u251c\u2500\u2500 DEVELOPER-GUIDE.md\n \u251c\u2500\u2500 OPERATIONS-RUNBOOK.md\n \u2514\u2500\u2500 REFERENCE.md\n</code></pre>"},{"location":"REFERENCE/#key-files","title":"Key Files","text":"<p><code>bootstrap.sh</code> <pre><code>#!/bin/zsh\n# Initializes cluster with ArgoCD\n\nArgoCd() {\n helm upgrade --install argocd argo-cd \\\n --repo https://argoproj.github.io/argo-helm \\\n --namespace argocd --create-namespace \\\n --values infra/values/base/argocd-values.yaml \\\n --set notifications.context.clusterName=\"$CLUSTER_NAME\" \\\n --timeout 60s --atomic\n\n kubectl apply -f _app-of-apps-upc-dev.yaml -n argocd # or _app-of-apps-upc-prod.yaml\n}\n</code></pre></p> <p><code>_app-of-apps-upc-dev.yaml</code> / <code>_app-of-apps-upc-prod.yaml</code> <pre><code>apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n name: infrastructure-apps\n namespace: argocd\nspec:\n project: default\n source:\n repoURL: ssh://git@git.forteapps.net:2222/Forte/launchpad.git\n path: infra\n destination:\n server: https://kubernetes.default.svc\n namespace: default\n syncPolicy:\n automated:\n prune: true\n selfHeal: true\n</code></pre></p>"},{"location":"REFERENCE/#helm-charts-repository-forte-helm","title":"Helm Charts Repository: <code>forte-helm</code>","text":"<p>URL: <code>https://github.com/fortedigital/forte-helm</code></p>"},{"location":"REFERENCE/#chart-forteapp","title":"Chart: <code>forteapp</code>","text":"<p>Version: 0.1.0 App Version: 1.0.0 Type: application</p>"},{"location":"REFERENCE/#templates","title":"Templates","text":"Template Purpose <code>_helpers.tpl</code> Template helper functions <code>namespace.yaml</code> Namespace resource <code>deployment.yaml</code> Main application Deployment <code>service.yaml</code> ClusterIP Service <code>ingressroute.yaml</code> Traefik IngressRoute <code>certificate.yaml</code> Cert-Manager Certificate <code>configmap.yaml</code> Application ConfigMap <code>secret-auth-tokens.yaml</code> Authentication tokens <code>hpa.yaml</code> Horizontal Pod Autoscaler <code>database-statefulset.yaml</code> Optional PostgreSQL StatefulSet <code>database-service.yaml</code> PostgreSQL Service"},{"location":"REFERENCE/#default-values-schema","title":"Default Values Schema","text":"<pre><code>app:\n image:\n repository: \"\" # Required\n tag: \"\" # Required\n pullPolicy: IfNotPresent\n containerPort: 3000\n\n replicaCount: 1\n\n resources:\n requests:\n cpu: 100m\n memory: 128Mi\n limits:\n cpu: 500m\n memory: 512Mi\n\n hpa:\n enabled: false\n minReplicas: 2\n maxReplicas: 10\n targetCPUUtilizationPercentage: 70\n\n extraEnv: []\n # - name: KEY\n # value: \"value\"\n\n envSecretName: \"\" # Reference to Secret\n nodeEnv: production\n\ndb:\n enabled: false\n name: postgres\n image:\n repository: postgres\n tag: \"16-alpine\"\n\n service:\n type: ClusterIP\n port: 5432\n targetPort: 5432\n\n persistence:\n enabled: true\n storageClass: \"\"\n accessMode: ReadWriteOnce\n size: 5Gi\n\n resources:\n requests:\n memory: \"256Mi\"\n cpu: \"250m\"\n limits:\n memory: \"1Gi\"\n cpu: \"1000m\"\n\n extraEnv: []\n envSecretName: \"\"\n\n livenessProbe:\n exec:\n command:\n - pg_isready\n - -U\n - db_user\n - -d\n - db_name\n initialDelaySeconds: 30\n periodSeconds: 10\n\n readinessProbe:\n exec:\n command:\n - pg_isready\n - -U\n - db_user\n - -d\n - db_name\n initialDelaySeconds: 5\n periodSeconds: 5\n\nservice:\n type: ClusterIP\n port: 3000\n\ningress:\n enabled: false\n host: \"\"\n entrypoint: websecure\n tls:\n enabled: true\n secretName: \"\"\n clusterIssuer: letsencrypt-prod\n\nauth:\n enabled: false # Enable authentication sidecar injection\n type: token # Authentication mode: \"token\" or \"oidc\"\n\n # Token-based authentication configuration\n tokens: [] # List of valid bearer tokens (hex strings, 32+ bytes recommended)\n # - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823\n # - 8803f621acc3898df1d7a8f514bc3602551a0681a8f747bd4e43c3c5849d57a7\n\n # OIDC authentication configuration\n oidc:\n authority: \"\" # OIDC provider URL (e.g., https://auth.example.com/realms/master)\n clientId: \"\" # OIDC client ID registered with provider\n scopes: \"openid,profile,email\" # OAuth scopes (comma-separated)\n callbackPath: /auth/callback # OAuth callback path (default: /auth/callback)\n # Note: Client secret must be in 'auth-oidc' Secret (client-secret key)\n # Cookie secret must be in 'auth-oidc' Secret (cookie-secret key)\n\nconfigmap: [] # Application ConfigMap key-value pairs\n# KEY: value\n# DB_HOST: postgres\n# DB_PORT: \"5432\"\n</code></pre>"},{"location":"REFERENCE/#helm-values-repository-helm-values","title":"Helm Values Repository: <code>helm-values</code>","text":"<p>URL: <code>https://github.com/fortedigital/helm-values.git</code></p>"},{"location":"REFERENCE/#structure","title":"Structure","text":"<pre><code>helm-values/\n\u251c\u2500\u2500 mcp10x/\n\u2502 \u2514\u2500\u2500 values.yaml\n\u251c\u2500\u2500 musicman/\n\u2502 \u2514\u2500\u2500 values.yaml\n\u251c\u2500\u2500 mcpcoder/\n\u2502 \u2514\u2500\u2500 values.yaml\n\u2514\u2500\u2500 argocd-mcp/\n \u2514\u2500\u2500 values.yaml\n</code></pre>"},{"location":"REFERENCE/#example-mcp10xvaluesyaml","title":"Example: <code>mcp10x/values.yaml</code>","text":"<pre><code>app:\n image:\n repository: ghcr.io/fortedigital/10x\n tag: 2.0.4 # Updated by CI/CD\n\n extraEnv:\n - name: PORT\n value: \"3000\"\n - name: SKILLS_DIR\n value: \"/app/skills\"\n - name: FLOWCASE_ENDPOINT\n value: \"https://forte.cvpartner.com/api/\"\n\n envSecretName: \"app-credentials\"\n\nauth:\n enabled: false\n tokens:\n - d4f88f6d9292c10cc3e21c4aad56d2be485db532b54fe961d738e1137d247823\n\ningress:\n enabled: true\n host: mcp10x.forteapps.net\n</code></pre>"},{"location":"REFERENCE/#helm-chart-reference","title":"Helm Chart Reference","text":""},{"location":"REFERENCE/#template-functions","title":"Template Functions","text":""},{"location":"REFERENCE/#forteappfullname","title":"<code>forteapp.fullname</code>","text":"<pre><code>{{ include \"forteapp.fullname\" . }}\n# Output: <release-name>\n</code></pre>"},{"location":"REFERENCE/#forteapplabels","title":"<code>forteapp.labels</code>","text":"<pre><code>{{ include \"forteapp.labels\" . }}\n# Output:\n# app.kubernetes.io/name: forteapp\n# app.kubernetes.io/instance: <release-name>\n# app.kubernetes.io/version: <chart-version>\n# app.kubernetes.io/managed-by: Helm\n</code></pre>"},{"location":"REFERENCE/#forteappselectorlabels","title":"<code>forteapp.selectorLabels</code>","text":"<pre><code>{{ include \"forteapp.selectorLabels\" . }}\n# Output:\n# app.kubernetes.io/name: forteapp\n# app.kubernetes.io/instance: <release-name>\n</code></pre>"},{"location":"REFERENCE/#deployment-specification","title":"Deployment Specification","text":"<pre><code>apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: {{ include \"forteapp.fullname\" . }}\n labels:\n {{- include \"forteapp.labels\" . | nindent 4 }}\nspec:\n replicas: {{ .Values.app.replicaCount }}\n selector:\n matchLabels:\n {{- include \"forteapp.selectorLabels\" . | nindent 6 }}\n template:\n metadata:\n annotations:\n policies.forteapps.io/auth: {{ .Values.auth.enabled | quote }}\n labels:\n {{- include \"forteapp.selectorLabels\" . | nindent 8 }}\n spec:\n containers:\n - name: app\n image: \"{{ .Values.app.image.repository }}:{{ .Values.app.image.tag }}\"\n imagePullPolicy: {{ .Values.app.image.pullPolicy }}\n ports:\n - name: http\n containerPort: {{ .Values.app.image.containerPort }}\n env:\n - name: NODE_ENV\n value: {{ .Values.app.nodeEnv | quote }}\n {{- with .Values.app.extraEnv }}\n {{- toYaml . | nindent 8 }}\n {{- end }}\n {{- if .Values.app.envSecretName }}\n envFrom:\n - secretRef:\n name: {{ .Values.app.envSecretName }}\n {{- end }}\n resources:\n {{- toYaml .Values.app.resources | nindent 10 }}\n securityContext:\n readOnlyRootFilesystem: true\n allowPrivilegeEscalation: false\n</code></pre>"},{"location":"REFERENCE/#ingressroute-specification","title":"IngressRoute Specification","text":"<pre><code>apiVersion: traefik.io/v1alpha1\nkind: IngressRoute\nmetadata:\n name: {{ include \"forteapp.fullname\" . }}\nspec:\n entryPoints:\n - {{ .Values.ingress.entrypoint }}\n routes:\n - match: Host(`{{ .Values.ingress.host }}`)\n kind: Rule\n services:\n - name: {{ include \"forteapp.fullname\" . }}\n port: {{ .Values.service.port }}\n {{- if .Values.ingress.tls.enabled }}\n tls:\n secretName: {{ default .Release.Name .Values.ingress.tls.secretName }}-tls\n {{- end }}\n</code></pre>"},{"location":"REFERENCE/#certificate-specification","title":"Certificate Specification","text":"<pre><code>apiVersion: cert-manager.io/v1\nkind: Certificate\nmetadata:\n name: {{ include \"forteapp.fullname\" . }}-tls\nspec:\n secretName: {{ default .Release.Name .Values.ingress.tls.secretName }}-tls\n issuerRef:\n name: {{ .Values.ingress.tls.clusterIssuer }}\n kind: ClusterIssuer\n dnsNames:\n - {{ .Values.ingress.host }}\n</code></pre>"},{"location":"REFERENCE/#argocd-configuration","title":"ArgoCD Configuration","text":""},{"location":"REFERENCE/#application-manifest-schema","title":"Application Manifest Schema","text":"<pre><code>apiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n name: <app-name>\n namespace: argocd\n annotations:\n argocd.argoproj.io/sync-wave: \"1\"\n notifications.argoproj.io/subscribe.on-sync-succeeded.slack: \"\"\n notifications.argoproj.io/subscribe.on-sync-failed.slack: \"\"\n notifications.argoproj.io/subscribe.on-degraded.slack: \"\"\n labels:\n app.kubernetes.io/name: <app-name>\n app.kubernetes.io/part-of: apps\n app.kubernetes.io/managed-by: argocd\n finalizers:\n - resources-finalizer.argocd.argoproj.io\n\nspec:\n project: default\n\n # Multi-source configuration\n sources:\n - repoURL: https://github.com/fortedigital/forte-helm\n path: forteapp\n targetRevision: HEAD\n helm:\n valueFiles:\n - $values/<app-name>/values.yaml\n\n - repoURL: git@github.com:fortedigital/helm-values.git\n targetRevision: HEAD\n ref: values\n\n destination:\n server: https://kubernetes.default.svc\n namespace: <app-name>\n\n syncPolicy:\n automated:\n prune: true\n selfHeal: true\n allowEmpty: false\n\n syncOptions:\n - CreateNamespace=true\n - Validate=true\n - ServerSideApply=true\n - Replace=false\n\n retry:\n limit: 5\n backoff:\n duration: 5s\n factor: 2\n maxDuration: 3m\n\n ignoreDifferences:\n - group: apps\n kind: Deployment\n jsonPointers:\n - /spec/replicas\n</code></pre>"},{"location":"REFERENCE/#sync-waves","title":"Sync Waves","text":"Wave Components Purpose <code>-1</code> Namespaces Create namespaces first <code>0</code> Kyverno Install policy engine <code>1</code> Cluster resources, infrastructure Base infrastructure <code>2+</code> Applications Business applications"},{"location":"REFERENCE/#sync-options","title":"Sync Options","text":"Option Description <code>CreateNamespace=true</code> Automatically create target namespace <code>Validate=true</code> Validate resources before applying <code>ServerSideApply=true</code> Use server-side apply (safer) <code>Replace=false</code> Don't use kubectl replace <code>Prune=true</code> Delete resources not in Git"},{"location":"REFERENCE/#retry-policy","title":"Retry Policy","text":"<pre><code>retry:\n limit: 5 # Max retry attempts\n backoff:\n duration: 5s # Initial backoff\n factor: 2 # Exponential factor\n maxDuration: 3m # Max backoff time\n</code></pre> <p>Retry Schedule: 1. 5 seconds 2. 10 seconds 3. 20 seconds 4. 40 seconds 5. 80 seconds (capped at 3 minutes)</p>"},{"location":"REFERENCE/#infrastructure-components","title":"Infrastructure Components","text":""},{"location":"REFERENCE/#traefik","title":"Traefik","text":"<p>Chart: <code>traefik/traefik</code> Version: Latest Namespace: <code>traefik</code></p> <p>Configuration: <pre><code># infra/base/traefik-application.yaml\nreplicas: 2\n\nservice:\n type: LoadBalancer\n\ningressRoute:\n dashboard:\n enabled: false\n\nports:\n web:\n redirectTo: websecure # HTTP \u2192 HTTPS redirect\n websecure:\n tls:\n enabled: true\n</code></pre></p> <p>Endpoints: - HTTP: <code>:80</code> \u2192 Redirects to HTTPS - HTTPS: <code>:443</code></p>"},{"location":"REFERENCE/#cert-manager","title":"Cert-Manager","text":"<p>Chart: <code>jetstack/cert-manager</code> Namespace: <code>cert-manager</code></p> <p>ClusterIssuer: <pre><code>apiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt-prod\nspec:\n acme:\n server: https://acme-v02.api.letsencrypt.org/directory\n email: admin@forteapps.net\n privateKeySecretRef:\n name: letsencrypt-prod-key\n solvers:\n - http01:\n ingress:\n class: traefik\n</code></pre></p>"},{"location":"REFERENCE/#kyverno","title":"Kyverno","text":"<p>Chart: <code>kyverno/kyverno</code> Namespace: <code>kyverno</code></p> <p>Policies: - Secret cloner - Default namespace blocker - Bare pod cleaner - ReplicaSet cleaner - Deployment verifier - Auth sidecar injector</p>"},{"location":"REFERENCE/#sealed-secrets","title":"Sealed Secrets","text":"<p>Chart: <code>sealed-secrets/sealed-secrets-controller</code> Namespace: <code>kube-system</code></p> <p>Public Certificate: <pre><code>kubeseal --fetch-cert \\\n --controller-name=sealed-secrets-controller \\\n --controller-namespace=kube-system \\\n > pub-cert.pem\n</code></pre></p>"},{"location":"REFERENCE/#prometheus","title":"Prometheus","text":"<p>Chart: <code>prometheus-community/prometheus</code> Namespace: <code>monitoring</code></p> <p>Configuration: <pre><code>server:\n persistentVolume:\n enabled: true\n size: 10Gi\n\nalertmanager:\n enabled: false\n\nnodeExporter:\n enabled: true\n\nkubeStateMetrics:\n enabled: true\n</code></pre></p>"},{"location":"REFERENCE/#grafana","title":"Grafana","text":"<p>Chart: <code>grafana/grafana</code> Namespace: <code>monitoring</code></p> <p>Datasources: - Prometheus - Loki - Tempo</p>"},{"location":"REFERENCE/#loki","title":"Loki","text":"<p>Chart: <code>grafana/loki-stack</code> Namespace: <code>monitoring</code></p> <p>Configuration: <pre><code>loki:\n persistence:\n enabled: true\n size: 10Gi\n\npromtail:\n enabled: false # Using Fluent-Bit instead\n</code></pre></p>"},{"location":"REFERENCE/#tempo","title":"Tempo","text":"<p>Chart: <code>grafana/tempo</code> Version: 1.24.4 Namespace: <code>monitoring</code></p> <p>Purpose: Distributed tracing backend receiving OTLP traces from Traefik and other instrumented services.</p> <p>Configuration: <pre><code>tempo:\n storage:\n trace:\n backend: local\n local:\n path: /var/tempo/traces\n receivers:\n otlp:\n protocols:\n grpc:\n endpoint: \"0.0.0.0:4317\"\n http:\n endpoint: \"0.0.0.0:4318\"\n\npersistence:\n enabled: true\n size: 10Gi\n</code></pre></p> <p>Endpoints: - gRPC OTLP receiver: <code>:4317</code> - HTTP OTLP receiver: <code>:4318</code> - Query API: <code>:3200</code></p> <p>Grafana Integration: - Trace-to-logs correlation with Loki (by namespace, pod, container) - Trace-to-metrics correlation with Prometheus (by service name) - Service graph and node graph visualization</p>"},{"location":"REFERENCE/#fluent-bit","title":"Fluent-Bit","text":"<p>Chart: <code>fluent/fluent-bit</code> Namespace: <code>monitoring</code></p> <p>Output: Loki</p>"},{"location":"REFERENCE/#gitea","title":"Gitea","text":"<p>Chart: <code>gitea/gitea</code> Version: 12.5.0 (app v1.25.4) Namespace: <code>gitea</code></p> <p>Purpose: Self-hosted Git repository hosting with pull requests, issues, CI/CD (Gitea Actions), container registry, and package registry.</p> <p>Configuration: <pre><code># infra/base/gitea.yaml + infra/values/base/gitea-values.yaml\ningress:\n host: git.forteapps.net\n tls: cert-manager (letsencrypt-prod)\n\ngitea:\n admin:\n existingSecret: gitea-credentials\n config:\n service:\n DISABLE_REGISTRATION: true\n ALLOW_ONLY_EXTERNAL_REGISTRATION: true\n actions:\n ENABLED: true\n packages:\n ENABLED: true\n metrics:\n ENABLED: true\n\npostgresql:\n enabled: true\n persistence: 8Gi (upcloud-block-storage-maxiops)\n</code></pre></p> <p>Authentication: Keycloak OIDC via <code>forte</code> realm (client ID: <code>gitea</code>). Protocol mapper: <code>email_verified</code> hardcoded claim (<code>true</code>, boolean) on ID token, Access token, and Userinfo.</p> <p>Endpoints: - Web UI: <code>https://git.forteapps.net</code> - SSH: port 22 (ClusterIP) - Metrics: <code>/metrics</code> (Prometheus scrape)</p> <p>Secrets: <code>gitea-credentials</code> (SealedSecret) containing <code>admin-password</code>, <code>postgres-password</code>, <code>secret</code> (OIDC client secret)</p>"},{"location":"REFERENCE/#gitea-actions-runners","title":"Gitea Actions Runners","text":"<p>Chart: <code>actions</code> (from <code>https://dl.gitea.com/charts</code>) Namespace: <code>gitea</code> Sync Wave: 2 (deploys after Gitea)</p> <p>Purpose: Act runners execute Gitea Actions CI/CD workflows. Deployed as a StatefulSet with a Docker-in-Docker sidecar for container-based job execution.</p> <p>Configuration: <pre><code># infra/base/gitea-actions.yaml + infra/values/base/gitea-actions-values.yaml\nreplicaCount: 3\n\nrunner:\n labels:\n - \"ubuntu-latest:docker://node:20-bookworm\"\n - \"ubuntu-22.04:docker://node:20-bookworm\"\n existingSecret: gitea-runner-token\n\ngitea:\n instance:\n url: http://gitea-http.gitea.svc.cluster.local:3000\n\ndind:\n enabled: true # Docker-in-Docker sidecar (privileged)\n</code></pre></p> <p>Resources:</p> Container CPU Request Memory Request CPU Limit Memory Limit Runner 250m 256Mi 1 1Gi DinD sidecar 250m 256Mi 1 1Gi <p>Secrets: <code>gitea-runner-token</code> (SealedSecret) containing <code>token</code> (instance-level runner registration token from <code>/admin/runners</code>)</p> <p>Setup Steps: 1. Get runner registration token from Gitea admin panel (<code>/admin/runners</code>) 2. Fill in <code>private/gitea-runner-token.yaml</code> with the token 3. Seal: <code>kubeseal --format yaml < private/gitea-runner-token.yaml > secrets/gitea-runner-token-sealed.yaml</code> 4. Commit and push \u2014 ArgoCD deploys runners automatically</p> <p>Verification: - <code>kubectl get statefulset -n gitea</code> \u2014 3/3 runners ready - Gitea admin panel (<code>/admin/runners</code>) \u2014 runners show as Online - Create test workflow in <code>.gitea/workflows/test.yml</code> \u2014 job executes</p>"},{"location":"REFERENCE/#keycloak-client-registrar","title":"Keycloak Client Registrar","text":"<p>Type: CronJob (deployed via Keycloak Helm chart <code>extraDeploy</code>) Namespace: <code>keycloak</code> Schedule: <code>*/2 * * * *</code> (every 2 minutes)</p> <p>Purpose: Handles two responsibilities: 1. Legacy sync \u2014 extracts secrets from Keycloak clients with <code>k8s.secret.sync: \"true\"</code> attribute (same as former PostSync syncer) 2. Self-service registration \u2014 processes config Secrets (cloned by Kyverno) to register new OIDC clients and sync their credentials</p> <p>How It Works:</p> <p>Legacy path (existing clients like Gitea): 1. Authenticates to Keycloak Admin API using admin credentials from <code>keycloak-credentials</code> secret 2. Queries all clients in the <code>forte</code> realm 3. Filters clients with <code>k8s.secret.sync: \"true\"</code> attribute 4. For each matching client, retrieves the auto-generated secret via Keycloak Admin API 5. Creates/updates a K8s Secret in the target namespace (from <code>k8s.secret.namespace</code> attribute) 6. Always writes a central copy to the <code>secrets</code> namespace</p> <p>Self-service path (new clients): 1. Lists Secrets in <code>keycloak</code> namespace with label <code>keycloak.forteapps.net/client-config=true</code> 2. For each config Secret, parses <code>client.json</code> and computes a config hash 3. Skips if hash matches annotation and credential Secret already exists 4. Creates or updates the Keycloak client via Admin API 5. Fetches the generated client secret 6. Upserts credential Secret in target namespace + central <code>secrets</code> namespace 7. Annotates config Secret with sync status, config hash, and timestamp</p> <p>Resources: - <code>ServiceAccount</code>: <code>keycloak-client-registrar</code> (namespace: <code>keycloak</code>) - <code>ClusterRole</code>: <code>keycloak-client-registrar</code> (secrets: get/list/create/update/patch; namespaces: get/list) - <code>ClusterRoleBinding</code>: <code>keycloak-client-registrar</code> - <code>CronJob</code>: <code>keycloak-client-registrar</code></p> <p>Kyverno Policy: <code>keycloak-client-config-cloner</code> \u2014 clones labeled Secrets from app namespaces to <code>keycloak</code> namespace (see Kyverno Policies)</p> <p>Legacy Client Attributes (set in <code>forte-realm.json</code>):</p> Attribute Required Default Description <code>k8s.secret.sync</code> Yes \u2014 Set to <code>\"true\"</code> to enable syncing <code>k8s.secret.namespace</code> Yes \u2014 Target K8s namespace <code>k8s.secret.name</code> Yes \u2014 Name of the K8s Secret <code>k8s.secret.client-id-key</code> No <code>client-id</code> Field name for client ID in the Secret <code>k8s.secret.client-secret-key</code> No <code>client-secret</code> Field name for client secret in the Secret <p>Self-Service Config Secret Schema: <pre><code>apiVersion: v1\nkind: Secret\nmetadata:\n name: keycloak-client-<app>\n namespace: <app-namespace>\n labels:\n keycloak.forteapps.net/client-config: \"true\"\nstringData:\n client.json: |\n {\n \"clientId\": \"<app>\",\n \"name\": \"<App Name>\",\n \"redirectUris\": [\"https://<app>.forteapps.net/*\"],\n \"webOrigins\": [\"https://<app>.forteapps.net\"],\n \"defaultClientScopes\": [\"openid\", \"email\", \"profile\"],\n \"protocolMappers\": [],\n \"secret\": {\n \"namespace\": \"<app-namespace>\",\n \"name\": \"<app>-oidc-credentials\",\n \"keys\": { \"clientId\": \"client-id\", \"clientSecret\": \"client-secret\" }\n }\n }\n</code></pre></p> <p>Created Credential Secret Format: <pre><code>apiVersion: v1\nkind: Secret\nmetadata:\n name: <target-name>\n namespace: <target-namespace>\n labels:\n app.kubernetes.io/managed-by: keycloak-client-registrar\ntype: Opaque\ndata:\n <client-id-key>: <base64-encoded client ID>\n <client-secret-key>: <base64-encoded client secret>\n</code></pre></p> <p>Config Secret Annotations (set by registrar):</p> Annotation Description <code>keycloak.forteapps.net/config-hash</code> SHA-256 hash of client.json for change detection <code>keycloak.forteapps.net/sync-status</code> <code>synced</code> or <code>error</code> <code>keycloak.forteapps.net/last-sync</code> ISO 8601 timestamp of last successful sync <p>Verification: <pre><code># Check CronJob status\nkubectl get cronjobs -n keycloak\n\n# View latest registrar logs\nkubectl logs -n keycloak job/$(kubectl get jobs -n keycloak --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')\n\n# Verify created secret\nkubectl get secret <name> -n <namespace> -o yaml\n\n# Check config Secret annotations (self-service)\nkubectl get secret keycloak-client-<app> -n keycloak -o jsonpath='{.metadata.annotations}'\n</code></pre></p> <p>See: Developer Guide - Adding a New Keycloak Client</p>"},{"location":"REFERENCE/#renovate","title":"Renovate","text":"<p>Chart: <code>renovate</code> (OCI: <code>ghcr.io/renovatebot/charts</code>) Version: 46.109.0 (app v43.113.0) Namespace: <code>renovate</code> Sync Wave: 2</p> <p>Purpose: Automated dependency update bot. Runs as a CronJob that scans Gitea repositories for outdated dependencies and creates pull requests with updates.</p> <p>Configuration: <pre><code># infra/base/renovate.yaml + infra/values/base/renovate-values.yaml\ncronjob:\n schedule: \"@daily\"\n concurrencyPolicy: Forbid\n\nrenovate:\n config:\n platform: gitea\n endpoint: https://git.forteapps.net\n autodiscover: true\n gitAuthor: \"Renovate Bot <renovate@forteapps.net>\"\n packageRules:\n - matchRepositories: [\"**/10x\"]\n assignees: [\"edvard.unsvag\"]\n reviewers: [\"edvard.unsvag\"]\n - matchRepositories: [\"**/auth-sidecar\"]\n assignees: [\"danijel.simeunovic\"]\n reviewers: [\"danijel.simeunovic\"]\n - matchRepositories: [\"**/forte-helm\"]\n assignees: [\"danijel.simeunovic\"]\n reviewers: [\"danijel.simeunovic\"]\n\nresources:\n requests: { cpu: 500m, memory: 1Gi }\n limits: { cpu: \"2\", memory: 4Gi }\n</code></pre></p> <p>Note: Assignees and reviewers are only applied at PR creation time. Existing PRs must be closed and recreated for new assignment rules to take effect.</p> <p>Secrets: <code>renovate-env</code> (SealedSecret in <code>secrets</code> namespace, cloned by Kyverno) containing: - <code>RENOVATE_TOKEN</code> \u2014 Gitea PAT with repo write + issue write permissions - <code>RENOVATE_GITHUB_COM_TOKEN</code> \u2014 GitHub PAT (public_repo read-only) for changelog fetching</p> <p>Setup Steps: 1. Fill in <code>private/renovate-env.yaml</code> with tokens 2. Seal: <code>kubeseal --format yaml < private/renovate-env.yaml > secrets/renovate-env-sealed.yaml</code> 3. Commit and push \u2014 ArgoCD deploys the CronJob, Kyverno clones the secret</p> <p>Verification: - <code>kubectl get cronjob -n renovate</code> \u2014 CronJob exists - <code>kubectl create job --from=cronjob/renovate renovate-test -n renovate</code> \u2014 manual trigger - <code>kubectl logs -n renovate job/renovate-test</code> \u2014 check logs</p>"},{"location":"REFERENCE/#gitea-pages","title":"Gitea Pages","text":"<p>Purpose: Hosts the MkDocs documentation site for this repository.</p> <p>How It Works: - A Gitea Actions workflow (<code>.gitea/workflows/docs.yaml</code>) builds MkDocs on push to <code>main</code> - The built site is force-pushed to the <code>gitea-pages</code> branch - Gitea serves the static site from that branch</p> <p>URL: <code>https://git.forteapps.net/Forte/launchpad/pages/</code></p> <p>Configuration: - Gitea server config: <code>ENABLE_GITEA_PAGES: true</code> (in gitea-values.yaml) - MkDocs config: <code>mkdocs.yml</code> (repo root) - Source files: <code>docs/</code> directory - Theme: Material for MkDocs</p> <p>Trigger Paths: - <code>docs/**</code> - <code>mkdocs.yml</code> - <code>Dockerfile.docs</code> - <code>nginx.conf</code></p>"},{"location":"REFERENCE/#kyverno-policies","title":"Kyverno Policies","text":""},{"location":"REFERENCE/#secret-cloner","title":"Secret Cloner","text":"<p>File: <code>cluster-resources/policies/secret-cloner.yaml</code></p> <p>Purpose: Automatically clone secrets from <code>secrets</code> namespace to new namespaces</p> <pre><code>apiVersion: kyverno.io/v1\nkind: ClusterPolicy\nmetadata:\n name: sync-secret-with-multi-clone\nspec:\n rules:\n - name: clone-secret\n match:\n any:\n - resources:\n kinds:\n - Namespace\n generate:\n apiVersion: v1\n kind: Secret\n name: \"{{ request.object.metadata.name }}\"\n namespace: \"{{ request.object.metadata.name }}\"\n synchronize: true\n clone:\n namespace: secrets\n name: shared-credentials\n</code></pre> <p>Label Requirement: Secrets must have <code>allowedToBeCloned: \"true\"</code></p>"},{"location":"REFERENCE/#keycloak-client-config-cloner","title":"Keycloak Client Config Cloner","text":"<p>File: <code>cluster-resources/policies/keycloak-client-cloner.yaml</code></p> <p>Purpose: Clones Secrets labeled <code>keycloak.forteapps.net/client-config: \"true\"</code> from app namespaces to the <code>keycloak</code> namespace. This allows apps to declare their OIDC client configuration in their own namespace, which the Keycloak Client Registrar then processes.</p> <p>Trigger: Any Secret with label <code>keycloak.forteapps.net/client-config: \"true\"</code> created outside the <code>keycloak</code> namespace.</p> <p>Behavior: - Generates a copy of the Secret in the <code>keycloak</code> namespace with the same name - Adds source tracking annotations (<code>keycloak.forteapps.net/source-namespace</code>, <code>keycloak.forteapps.net/source-name</code>) - <code>synchronize: true</code> \u2014 changes to the source Secret are reflected in the clone</p>"},{"location":"REFERENCE/#default-namespace-blocker","title":"Default Namespace Blocker","text":"<p>File: <code>cluster-resources/policies/default-ns-blocker.yaml</code></p> <p>Purpose: Prevent resources from being created in <code>default</code> namespace</p> <pre><code>apiVersion: kyverno.io/v1\nkind: ClusterPolicy\nmetadata:\n name: disallow-default-namespace\nspec:\n validationFailureAction: enforce\n rules:\n - name: validate-namespace\n match:\n any:\n - resources:\n kinds:\n - Pod\n - Deployment\n - Service\n validate:\n message: \"Using 'default' namespace is not allowed\"\n pattern:\n metadata:\n namespace: \"!default\"\n</code></pre>"},{"location":"REFERENCE/#bare-pod-cleaner","title":"Bare Pod Cleaner","text":"<p>File: <code>cluster-resources/policies/bare-pod-cleaner.yaml</code></p> <p>Purpose: Delete pods without ownerReferences (not managed by Deployment/StatefulSet)</p> <pre><code>apiVersion: kyverno.io/v1\nkind: ClusterPolicy\nmetadata:\n name: cleanup-bare-pods\nspec:\n rules:\n - name: delete-bare-pod\n match:\n any:\n - resources:\n kinds:\n - Pod\n preconditions:\n all:\n - key: \"{{ request.object.metadata.ownerReferences[] || '' }}\"\n operator: Equals\n value: \"\"\n validate:\n message: \"Bare pods (without controllers) are not allowed\"\n deny: {}\n</code></pre>"},{"location":"REFERENCE/#auth-sidecar-injector","title":"Auth Sidecar Injector","text":"<p>File: <code>cluster-resources/policies/auth-sidecar-injector.yaml</code></p> <p>Purpose: Automatically inject authentication sidecar into pods with authentication enabled</p> <p>Rules: 6 rules in the policy 1. <code>generate-auth-tokens-secret</code> - Creates Secret for token mode 2. <code>generate-auth-oidc-secret</code> - Creates Secret for OIDC mode 3. <code>inject-sidecar-token</code> - Injects auth sidecar for token mode 4. <code>inject-sidecar-oidc</code> - Injects auth sidecar for OIDC mode 5. <code>inject-sidecar-mcp</code> - Injects auth sidecar for MCP OAuth mode (RFC 9728 / RFC 7591) 6. <code>generate-auth-network-policy</code> - Creates NetworkPolicy to restrict ingress</p>"},{"location":"REFERENCE/#trigger-annotation","title":"Trigger Annotation","text":"<pre><code>policies.forteapps.io/auth: \"true\"\n</code></pre>"},{"location":"REFERENCE/#authentication-modes","title":"Authentication Modes","text":"<p>Token Mode (default): <pre><code># Annotations\npolicies.forteapps.io/auth: \"true\"\npolicies.forteapps.io/auth-type: \"token\"\npolicies.forteapps.io/auth-token-secret-name: \"auth-tokens\"\npolicies.forteapps.io/auth-upstream-url: \"http://localhost:3000\"\n\n# Optional customization\npolicies.forteapps.io/auth-image: \"ghcr.io/fortedigital/auth-sidecar\"\npolicies.forteapps.io/auth-image-version: \"latest\"\n</code></pre></p> <p>OIDC Mode: <pre><code># Annotations (required)\npolicies.forteapps.io/auth: \"true\"\npolicies.forteapps.io/auth-type: \"oidc\"\npolicies.forteapps.io/auth-oidc-authority: \"https://auth.example.com/realms/master\"\npolicies.forteapps.io/auth-oidc-client-id: \"myapp\"\n\n# Optional annotations\npolicies.forteapps.io/auth-oidc-callback-path: \"/auth/callback\"\npolicies.forteapps.io/auth-oidc-scopes: \"openid,profile,email\"\npolicies.forteapps.io/auth-upstream-url: \"http://localhost:3000\"\npolicies.forteapps.io/auth-image: \"ghcr.io/fortedigital/auth-sidecar\"\npolicies.forteapps.io/auth-image-version: \"latest\"\n</code></pre></p> <p>MCP Mode (OAuth 2.0 for MCP servers, implements RFC 9728 / RFC 7591): <pre><code># Annotations (required)\npolicies.forteapps.io/auth: \"true\"\npolicies.forteapps.io/auth-type: \"mcp\"\npolicies.forteapps.io/auth-mcp-resource: \"https://mcp.example.com\"\npolicies.forteapps.io/auth-mcp-authority: \"https://auth.example.com\"\n\n# Optional annotations\npolicies.forteapps.io/auth-mcp-scopes: \"read,write\"\npolicies.forteapps.io/auth-upstream-url: \"http://localhost:3000\"\npolicies.forteapps.io/auth-log-level: \"info\"\npolicies.forteapps.io/auth-image: \"ghcr.io/fortedigital/auth-sidecar\"\npolicies.forteapps.io/auth-image-version: \"latest\"\n</code></pre></p>"},{"location":"REFERENCE/#sidecar-container-specification","title":"Sidecar Container Specification","text":"<p>Token Mode: <pre><code>name: authn\nimage: ghcr.io/fortedigital/auth-sidecar:latest\nports:\n- containerPort: 8080\n name: auth\n protocol: TCP\nenv:\n- name: AUTH_MODE\n value: \"token\"\n- name: AUTH_LISTEN_ADDR\n value: \":8080\"\n- name: AUTH_UPSTREAM_URL\n value: \"http://localhost:3000\"\n- name: AUTH_TOKEN_FILE\n value: \"/etc/auth/tokens\"\nvolumeMounts:\n- name: auth-tokens\n mountPath: /etc/auth\n readOnly: true\nresources:\n requests:\n cpu: 10m\n memory: 32Mi\n limits:\n cpu: 50m\n memory: 64Mi\nsecurityContext:\n allowPrivilegeEscalation: false\n readOnlyRootFilesystem: true\n capabilities:\n drop: [ALL]\n</code></pre></p> <p>OIDC Mode: <pre><code>name: authn\nimage: ghcr.io/fortedigital/auth-sidecar:latest\nports:\n- containerPort: 8080\n name: auth\n protocol: TCP\nenv:\n- name: AUTH_MODE\n value: \"oidc\"\n- name: AUTH_LISTEN_ADDR\n value: \":8080\"\n- name: AUTH_UPSTREAM_URL\n value: \"http://localhost:3000\"\n- name: AUTH_OIDC_AUTHORITY\n value: \"https://auth.example.com/realms/master\"\n- name: AUTH_OIDC_CLIENT_ID\n value: \"myapp\"\n- name: AUTH_OIDC_CALLBACK_PATH\n value: \"/auth/callback\"\n- name: AUTH_OIDC_SCOPES\n value: \"openid,profile,email\"\n- name: AUTH_OIDC_COOKIE_SECRET\n valueFrom:\n secretKeyRef:\n name: auth-oidc\n key: cookie-secret\n- name: AUTH_OIDC_CLIENT_SECRET\n valueFrom:\n secretKeyRef:\n name: auth-oidc\n key: client-secret\nresources:\n requests:\n cpu: 10m\n memory: 32Mi\n limits:\n cpu: 50m\n memory: 64Mi\nsecurityContext:\n allowPrivilegeEscalation: false\n readOnlyRootFilesystem: true\n capabilities:\n drop: [ALL]\n</code></pre></p> <p>MCP Mode: <pre><code>name: authn\nimage: ghcr.io/fortedigital/auth-sidecar:latest\nports:\n- containerPort: 8080\n name: auth\n protocol: TCP\nenv:\n- name: AUTH_MODE\n value: \"mcp\"\n- name: AUTH_LISTEN_ADDR\n value: \":8080\"\n- name: AUTH_LOG_LEVEL\n value: \"info\"\n- name: AUTH_UPSTREAM_URL\n value: \"http://localhost:3000\"\n- name: AUTH_MCP_RESOURCE\n value: \"https://mcp.example.com\"\n- name: AUTH_MCP_AUTHORIZATION_SERVERS\n value: \"https://auth.example.com\"\n- name: AUTH_MCP_SCOPES_SUPPORTED\n value: \"read,write\"\nresources:\n requests:\n cpu: 10m\n memory: 32Mi\n limits:\n cpu: 50m\n memory: 64Mi\nsecurityContext:\n allowPrivilegeEscalation: false\n readOnlyRootFilesystem: true\n capabilities:\n drop: [ALL]\n</code></pre></p>"},{"location":"REFERENCE/#generated-resources","title":"Generated Resources","text":"<p>Secret (Token Mode): <pre><code>apiVersion: v1\nkind: Secret\nmetadata:\n name: auth-tokens\n namespace: <app-namespace>\n labels:\n app.kubernetes.io/managed-by: kyverno\n app.kubernetes.io/created-by: inject-auth-sidecar\ntype: Opaque\ndata: {} # Populated by Helm chart\n</code></pre></p> <p>Secret (OIDC Mode): <pre><code>apiVersion: v1\nkind: Secret\nmetadata:\n name: auth-oidc\n namespace: <app-namespace>\n labels:\n app.kubernetes.io/managed-by: kyverno\n app.kubernetes.io/created-by: inject-auth-sidecar\ntype: Opaque\ndata:\n client-secret: <base64>\n cookie-secret: <base64>\n</code></pre></p> <p>NetworkPolicy: <pre><code>apiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n name: <pod-name>-auth-ingress\n namespace: <app-namespace>\n labels:\n app.kubernetes.io/managed-by: kyverno\n app.kubernetes.io/created-by: inject-auth-sidecar\nspec:\n podSelector:\n matchLabels: <pod-labels>\n policyTypes:\n - Ingress\n ingress:\n - ports:\n - port: 8080\n protocol: TCP\n</code></pre></p>"},{"location":"REFERENCE/#excluded-namespaces","title":"Excluded Namespaces","text":"<p>The policy does NOT apply to: - <code>kube-system</code> - <code>kyverno</code> - <code>argocd</code> - <code>cert-manager</code> - <code>monitoring</code></p>"},{"location":"REFERENCE/#health-checks","title":"Health Checks","text":"<pre><code>readinessProbe:\n httpGet:\n path: /healthz\n port: 8080\n initialDelaySeconds: 2\n periodSeconds: 5\n\nlivenessProbe:\n httpGet:\n path: /healthz\n port: 8080\n initialDelaySeconds: 5\n periodSeconds: 10\n</code></pre>"},{"location":"REFERENCE/#request-flow","title":"Request Flow","text":"<pre><code>External Request \u2192 Traefik\n \u2193\nService (port 8080)\n \u2193\nPod: Auth Sidecar (port 8080)\n \u251c\u2500 Validate credentials\n \u2502 \u2022 Token mode: Check Bearer token\n \u2502 \u2022 OIDC mode: Validate session or redirect to IdP\n \u2502 \u2022 MCP mode: OAuth 2.0 via RFC 9728 discovery / RFC 7591 dynamic registration\n \u2193\nForward to Application (localhost:3000)\n \u2193\nApplication processes request\n</code></pre> <p>See: Developer Guide - Enabling Authentication for usage examples.</p>"},{"location":"REFERENCE/#configuration-reference","title":"Configuration Reference","text":""},{"location":"REFERENCE/#environment-variables","title":"Environment Variables","text":"<p>Common environment variables used across applications:</p> Variable Purpose Example <code>NODE_ENV</code> Node.js environment <code>production</code> <code>PORT</code> Application port <code>3000</code> <code>DB_HOST</code> Database host <code>postgres</code> <code>DB_PORT</code> Database port <code>5432</code> <code>DB_USER</code> Database user <code>app_user</code> <code>DB_NAME</code> Database name <code>app_db</code> <code>DB_PASSWORD</code> Database password From secret <code>API_KEY</code> External API key From secret"},{"location":"REFERENCE/#resource-limits","title":"Resource Limits","text":"<p>Recommended resource allocation:</p> Application Type CPU Request Memory Request CPU Limit Memory Limit Lightweight API 100m 128Mi 500m 512Mi Standard Web App 200m 256Mi 1000m 1Gi Heavy Processing 500m 512Mi 2000m 2Gi Database 250m 256Mi 1000m 1Gi"},{"location":"REFERENCE/#storage-classes","title":"Storage Classes","text":"<p>Default storage class used: UpCloud default (varies by provider)</p> <pre><code>persistence:\n enabled: true\n storageClass: \"\" # Uses default\n accessMode: ReadWriteOnce\n size: 5Gi\n</code></pre>"},{"location":"REFERENCE/#api-endpoints","title":"API Endpoints","text":""},{"location":"REFERENCE/#argocd-api","title":"ArgoCD API","text":"<pre><code># Server\nhttps://argocd.127.0.0.1.nip.io\n\n# Applications endpoint\nGET /api/v1/applications\n\n# Application details\nGET /api/v1/applications/{name}\n\n# Sync application\nPOST /api/v1/applications/{name}/sync\n</code></pre>"},{"location":"REFERENCE/#prometheus-api","title":"Prometheus API","text":"<pre><code># Query endpoint\nGET /api/v1/query?query={promql}\n\n# Query range\nGET /api/v1/query_range?query={promql}&start={time}&end={time}&step={duration}\n\n# Metrics\nGET /api/v1/label/__name__/values\n</code></pre>"},{"location":"REFERENCE/#tempo-api","title":"Tempo API","text":"<pre><code># Search traces\nGET /api/search?q={traceql}\n\n# Get trace by ID\nGET /api/traces/{traceID}\n\n# Service tag values\nGET /api/v2/search/tag/resource.service.name/values\n</code></pre>"},{"location":"REFERENCE/#loki-api","title":"Loki API","text":"<pre><code># Query logs\nGET /loki/api/v1/query?query={logql}\n\n# Query range\nGET /loki/api/v1/query_range?query={logql}&start={time}&end={time}\n\n# Push logs\nPOST /loki/api/v1/push\n</code></pre>"},{"location":"REFERENCE/#glossary","title":"Glossary","text":""},{"location":"REFERENCE/#terms","title":"Terms","text":"<p>App-of-Apps: ArgoCD pattern where a parent Application manages child Applications</p> <p>GitOps: Operations approach where Git is the single source of truth</p> <p>IngressRoute: Traefik CRD for routing external traffic to services</p> <p>Multi-Source: ArgoCD feature allowing multiple Git sources per Application</p> <p>SealedSecret: Encrypted secret that can be safely stored in Git</p> <p>Sync Wave: Ordered deployment using annotations</p> <p>Self-Heal: ArgoCD automatically reverts manual cluster changes</p> <p>Prune: Automatically delete resources removed from Git</p>"},{"location":"REFERENCE/#annotations-reference","title":"Annotations Reference","text":""},{"location":"REFERENCE/#argocd-annotations","title":"ArgoCD Annotations","text":"<pre><code># Sync wave (deployment order)\nargocd.argoproj.io/sync-wave: \"1\"\n\n# Refresh application\nargocd.argoproj.io/refresh: \"hard\"\n\n# Compare options\nargocd.argoproj.io/compare-options: IgnoreExtraneous\n\n# Sync options per resource\nargocd.argoproj.io/sync-options: Prune=false\n</code></pre>"},{"location":"REFERENCE/#kyverno-annotations","title":"Kyverno Annotations","text":"<pre><code># Exclude from policy\npolicies.kyverno.io/exclude: \"true\"\n\n# Severity\npolicies.kyverno.io/severity: high\n</code></pre>"},{"location":"REFERENCE/#custom-annotations","title":"Custom Annotations","text":"<pre><code># Authentication enabled\npolicies.forteapps.io/auth: \"true\"\n\n# OIDC configuration\npolicies.forteapps.io/auth-oidc-authority: \"https://...\"\npolicies.forteapps.io/auth-oidc-client-id: \"client-id\"\n</code></pre>"},{"location":"REFERENCE/#labels-reference","title":"Labels Reference","text":""},{"location":"REFERENCE/#standard-labels","title":"Standard Labels","text":"<pre><code># Application name\napp.kubernetes.io/name: myapp\n\n# Application instance\napp.kubernetes.io/instance: myapp\n\n# Application version\napp.kubernetes.io/version: \"1.0.0\"\n\n# Component type\napp.kubernetes.io/component: frontend\n\n# Part of larger application\napp.kubernetes.io/part-of: ecommerce\n\n# Managed by\napp.kubernetes.io/managed-by: argocd\n</code></pre>"},{"location":"REFERENCE/#custom-labels","title":"Custom Labels","text":"<pre><code># Allow secret cloning\nallowedToBeCloned: \"true\"\n\n# Environment\nenvironment: production\n\n# Team ownership\nteam: platform\n</code></pre>"},{"location":"REFERENCE/#version-matrix","title":"Version Matrix","text":""},{"location":"REFERENCE/#component-versions","title":"Component Versions","text":"Component Version Chart Version ArgoCD 2.9.0+ Latest Traefik 2.10.0+ Latest Cert-Manager 1.13.0+ Latest Kyverno 1.10.0+ Latest Sealed Secrets 0.24.0+ Latest Prometheus 2.47.0+ Latest Grafana 10.0.0+ Latest Loki 2.9.0+ Latest Tempo 2.6.0+ 1.24.4 Fluent-Bit 2.1.0+ Latest Gitea 1.25.4 12.5.0 Gitea Act Runner Latest Latest Renovate v43.113.0 46.109.0 PostgreSQL 16-alpine N/A Trivy Latest Latest"},{"location":"REFERENCE/#kubernetes-compatibility","title":"Kubernetes Compatibility","text":"<ul> <li>Minimum: 1.24+</li> <li>Tested: 1.28+</li> <li>Recommended: Latest stable</li> </ul> <p>Last Updated: 2026-04-16 Maintained By: Platform Team Version: 1.0.0</p>"}]} |