tofu+tools

docs/OPERATIONS-RUNBOOK.md

@@ -2,6 +2,12 @@
 
 ## Table of Contents
 - [Overview](#overview)
+- [Infrastructure Provisioning (OpenTofu)](#infrastructure-provisioning-opentofu)
+  - [Prerequisites](#provisioning-prerequisites)
+  - [Provisioning a Cluster](#provisioning-a-cluster)
+  - [Tearing Down a Cluster](#tearing-down-a-cluster)
+  - [Retrieving Kubeconfig](#retrieving-kubeconfig)
+  - [Platform Credentials](#platform-credentials)
 - [Cluster Bootstrap](#cluster-bootstrap)
   - [Initial Cluster Setup](#initial-cluster-setup)
   - [ArgoCD Repository Access Setup](#argocd-repository-access-setup)
@@ -29,6 +35,120 @@ This runbook provides operational procedures for maintaining the Kubernetes clus
 
 ---
 
+## Infrastructure Provisioning (OpenTofu)
+
+The `.tofu/` directory contains multi-cloud Kubernetes infrastructure-as-code using [OpenTofu](https://opentofu.org/). It provisions clusters on four cloud platforms (AKS, EKS, GKE, UpCloud), each with three environment tiers: **dev**, **prod**, and **workload**.
+
+### Provisioning Prerequisites {#provisioning-prerequisites}
+
+- **OpenTofu** (`tofu`) installed
+- **kubectl** installed
+- **helm** installed
+- **yq** (optional — loads cluster config from `clusters/<cluster>.yaml`)
+- Platform CLI tools:
+  - **AKS**: `az` (Azure CLI)
+  - **EKS**: `aws` (AWS CLI)
+  - **GKE**: `gcloud` (Google Cloud SDK)
+  - **UPC**: `upctl` (UpCloud CLI)
+
+### Provisioning a Cluster
+
+```bash
+# Navigate to the scripts directory
+cd .tofu/scripts
+
+# 1. Copy and fill in credentials for your platform
+cp ../configs/aks.env.example ../configs/aks.env
+# Edit ../configs/aks.env with your credentials
+
+# 2. Provision cluster (interactive — prompts before applying)
+./setup-cluster.sh aks-dev
+
+# 3. Dry-run only (plan without applying)
+./setup-cluster.sh aks-dev --plan
+
+# 4. Non-interactive (skip confirmations)
+./setup-cluster.sh aks-dev --auto
+```
+
+**Cluster name format**: `<platform>-<env>` — e.g., `aks-dev`, `eks-prod`, `gke-workload`, `upc-dev`
+
+**What `setup-cluster.sh` does**:
+1. Validates cluster name, extracts platform and environment
+2. Checks prerequisites (tofu, kubectl, helm)
+3. Loads credentials from `configs/<platform>.env`
+4. Optionally loads cluster config from `clusters/<cluster>.yaml` (via yq)
+5. Runs `tofu init` → `tofu plan` → prompts → `tofu apply`
+6. Fetches and caches kubeconfig to `private/<cluster>/kubeconfig`
+7. Waits for all nodes to reach Ready state (300s timeout)
+8. Outputs next steps: `export KUBECONFIG` + `./bootstrap.sh`
+
+### Tearing Down a Cluster
+
+```bash
+# Destroy cluster infrastructure
+./teardown-cluster.sh aks-dev
+
+# Equivalent to:
+./setup-cluster.sh aks-dev --destroy
+```
+
+### Retrieving Kubeconfig
+
+```bash
+# Get kubeconfig for an existing cluster (uses cache or platform CLI)
+./get-kubeconfig.sh aks-dev
+
+# Cached kubeconfigs stored in: private/<cluster>/kubeconfig
+```
+
+Platform-specific retrieval fallbacks:
+- **AKS**: `az aks get-credentials`
+- **EKS**: `aws eks update-kubeconfig`
+- **GKE**: `gcloud container clusters get-credentials`
+- **UPC**: `upctl kubernetes config`
+
+### Platform Credentials
+
+Each platform has a `configs/<platform>.env.example` template. Copy to `.env` and populate:
+
+| Platform | Required Variables | Optional |
+|----------|--------------------|----------|
+| **AKS** | `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID` | `ARM_RESOURCE_GROUP` (defaults to cluster name) |
+| **EKS** | `AWS_PROFILE` (default: "default"), `AWS_REGION` (default: "eu-west-1") | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` |
+| **GKE** | `GCP_PROJECT_ID`, `GCP_REGION` (default: "europe-west4") | `GOOGLE_APPLICATION_CREDENTIALS` (SA JSON path) |
+| **UPC** | `UPCLOUD_TOKEN` | `UPCLOUD_CLUSTER_ID` (set after creation) |
+
+> **Note**: `.env` files are git-ignored. Never commit credentials.
+
+### End-to-End Workflow
+
+Full cluster lifecycle: provision → bootstrap → operate → teardown:
+
+```bash
+# 1. Provision infrastructure
+cd .tofu/scripts
+./setup-cluster.sh aks-dev
+
+# 2. Export kubeconfig (printed by setup-cluster.sh)
+export KUBECONFIG=$(pwd)/../../private/aks-dev/kubeconfig
+
+# 3. Bootstrap GitOps (ArgoCD + App-of-Apps)
+cd ../..
+./bootstrap.sh aks-dev
+
+# 4. Verify
+kubectl get applications -n argocd
+
+# ... operate cluster ...
+
+# 5. Teardown when done
+cd .tofu/scripts
+./teardown-cluster.sh aks-dev
+```
+
+---
+
 ## Cluster Bootstrap
 
 ### Initial Cluster Setup
@@ -37,7 +157,7 @@ Bootstrap a new cluster from scratch:
 
 #### Prerequisites
 
-1. **Kubernetes cluster running** (UpCloud, AWS EKS, Azure AKS, GCP GKE, or any K8s cluster)
+1. **Kubernetes cluster running** (provisioned via `.tofu/scripts/setup-cluster.sh` or manually on UpCloud, AWS EKS, Azure AKS, GCP GKE)
 2. **kubectl configured** with admin access
 3. **Repositories cloned** locally
 
@@ -1286,14 +1406,17 @@ spec:
 
 ```bash
 # 1. Provision new Kubernetes cluster
+cd .tofu/scripts
+./setup-cluster.sh upc-dev          # or aks-dev, eks-prod, etc.
+export KUBECONFIG=$(pwd)/../../private/upc-dev/kubeconfig
 
-# 2. Configure kubectl
-kubectl config use-context new-cluster
+# 2. Verify cluster access
 kubectl cluster-info
+kubectl get nodes
 
 # 3. Bootstrap cluster
-cd ~/dev/k8s/launchpad
-./bootstrap.sh
+cd ../..
+./bootstrap.sh upc-dev
 
 # 4. Wait for ArgoCD to sync all applications
 kubectl get applications -n argocd -w