ssh access

docs/OPERATIONS-RUNBOOK.md

@@ -3,6 +3,8 @@
 ## Table of Contents
 - [Overview](#overview)
 - [Cluster Bootstrap](#cluster-bootstrap)
+  - [Initial Cluster Setup](#initial-cluster-setup)
+  - [ArgoCD Repository Access Setup](#argocd-repository-access-setup)
 - [Day-to-Day Operations](#day-to-day-operations)
 - [Application Management](#application-management)
 - [Secret Management](#secret-management)
@@ -103,6 +105,345 @@ kubectl get applications -n argocd
 
 5. **Test Slack notifications** by triggering a sync
 
+### ArgoCD Repository Access Setup
+
+ArgoCD needs SSH access to private Git repositories to pull manifests and Helm values. This section covers setting up deploy keys for GitHub repositories.
+
+#### Why Deploy Keys?
+
+- **Read-only access**: Deploy keys provide secure, read-only access to repositories
+- **No user credentials**: No need to share personal SSH keys or tokens
+- **Repository-specific**: Each repository gets its own key for better security
+- **Revocable**: Easy to revoke access without affecting other repositories
+
+#### Prerequisites
+
+- kubectl access to the cluster
+- Write access to the GitHub repository
+- ArgoCD installed and running
+
+#### Setup Procedure
+
+**Step 1: Generate SSH Key Pair**
+
+Generate a dedicated SSH key for ArgoCD without a passphrase (required for automated access):
+
+```bash
+# Generate ED25519 key (recommended - smaller and more secure)
+ssh-keygen -t ed25519 -C "argocd-deploy-key-sturdy-adventure" -f argocd-deploy-key -N ""
+
+# Or RSA key if ED25519 is not supported
+ssh-keygen -t rsa -b 4096 -C "argocd-deploy-key-sturdy-adventure" -f argocd-deploy-key -N ""
+```
+
+This creates two files:
+- `argocd-deploy-key` - Private key (keep secret)
+- `argocd-deploy-key.pub` - Public key (add to GitHub)
+
+**Step 2: Add Public Key to GitHub**
+
+1. Copy the public key:
+   ```bash
+   cat argocd-deploy-key.pub
+   ```
+
+2. Go to GitHub repository settings:
+   - Navigate to: `https://github.com/fortedigital/sturdy-adventure/settings/keys`
+   - Or: Repository → Settings → Deploy keys
+
+3. Click **"Add deploy key"**
+   - Title: `ArgoCD Production Cluster`
+   - Key: Paste the public key content
+   - ☐ Allow write access (leave unchecked - read-only is sufficient)
+   - Click **"Add key"**
+
+4. Repeat for the `helm-values` repository if it's private:
+   ```bash
+   # Generate separate key for helm-values repo
+   ssh-keygen -t ed25519 -C "argocd-deploy-key-helm-values" -f argocd-helm-values-key -N ""
+
+   # Add to: https://github.com/fortedigital/helm-values/settings/keys
+   ```
+
+**Step 3: Create Kubernetes Secret**
+
+Add the private key to ArgoCD as a repository secret:
+
+```bash
+# Create secret for sturdy-adventure repository
+kubectl create secret generic repo-sturdy-adventure \
+  --from-file=sshPrivateKey=argocd-deploy-key \
+  --namespace=argocd \
+  --dry-run=client -o yaml | kubectl apply -f -
+
+# Label it for ArgoCD to recognize
+kubectl label secret repo-sturdy-adventure \
+  -n argocd \
+  argocd.argoproj.io/secret-type=repository
+
+# Add repository annotations
+kubectl annotate secret repo-sturdy-adventure \
+  -n argocd \
+  managed-by=argocd.argoproj.io
+```
+
+Alternatively, create a complete repository secret with all metadata:
+
+```bash
+kubectl apply -f - <<EOF
+apiVersion: v1
+kind: Secret
+metadata:
+  name: repo-sturdy-adventure
+  namespace: argocd
+  labels:
+    argocd.argoproj.io/secret-type: repository
+  annotations:
+    managed-by: argocd.argoproj.io
+type: Opaque
+stringData:
+  type: git
+  url: git@github.com:fortedigital/sturdy-adventure.git
+  sshPrivateKey: |
+$(cat argocd-deploy-key | sed 's/^/    /')
+EOF
+```
+
+**Step 4: Register Repository in ArgoCD**
+
+Add the repository to ArgoCD's configuration:
+
+```bash
+# Via kubectl (recommended for GitOps)
+kubectl apply -f - <<EOF
+apiVersion: v1
+kind: Secret
+metadata:
+  name: repo-sturdy-adventure
+  namespace: argocd
+  labels:
+    argocd.argoproj.io/secret-type: repository
+type: Opaque
+stringData:
+  type: git
+  url: git@github.com:fortedigital/sturdy-adventure.git
+  sshPrivateKey: |
+$(cat argocd-deploy-key | sed 's/^/    /')
+  insecure: "false"
+  enableLfs: "false"
+EOF
+
+# Or via ArgoCD UI
+# 1. Open ArgoCD UI: kubectl port-forward svc/argocd-server -n argocd 8080:443
+# 2. Navigate to: Settings → Repositories → Connect Repo
+# 3. Connection Method: Via SSH
+# 4. Repository URL: git@github.com:fortedigital/sturdy-adventure.git
+# 5. SSH private key: Paste private key content
+# 6. Click "Connect"
+```
+
+**Step 5: Verify Repository Access**
+
+```bash
+# Check if repository is connected
+kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=repository
+
+# Verify connection in ArgoCD UI
+# Settings → Repositories → Should show "Successful" status
+
+# Test by creating an application
+kubectl apply -f _app-of-apps.yaml
+
+# Check application sync status
+kubectl get applications -n argocd
+```
+
+#### Testing Repository Access
+
+Create a test application to verify SSH access:
+
+```bash
+cat > /tmp/test-repo-access.yaml <<EOF
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: test-repo-access
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: git@github.com:fortedigital/sturdy-adventure.git
+    targetRevision: main
+    path: cluster-resources
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: default
+  syncPolicy:
+    automated: null  # Manual sync for testing
+EOF
+
+kubectl apply -f /tmp/test-repo-access.yaml
+
+# Check if ArgoCD can access the repository
+kubectl describe application test-repo-access -n argocd
+
+# Look for sync status - should show repository contents
+kubectl get application test-repo-access -n argocd -o jsonpath='{.status.sync.status}'
+
+# Clean up test application
+kubectl delete application test-repo-access -n argocd
+rm /tmp/test-repo-access.yaml
+```
+
+#### Security Best Practices
+
+1. **Secure Private Keys**
+   ```bash
+   # Store private key securely and delete local copy
+   # Option 1: Store in password manager (recommended)
+   # Option 2: Backup to encrypted storage
+
+   # Delete local private key after adding to Kubernetes
+   shred -u argocd-deploy-key
+
+   # Or on Windows
+   # Remove-Item -Path argocd-deploy-key -Force
+   ```
+
+2. **Rotate Keys Regularly**
+   ```bash
+   # Generate new key
+   ssh-keygen -t ed25519 -C "argocd-deploy-key-$(date +%Y%m)" -f argocd-new-key -N ""
+
+   # Add new public key to GitHub (keep old key for now)
+
+   # Update Kubernetes secret
+   kubectl create secret generic repo-sturdy-adventure \
+     --from-file=sshPrivateKey=argocd-new-key \
+     --namespace=argocd \
+     --dry-run=client -o yaml | kubectl apply -f -
+
+   # Test access, then remove old deploy key from GitHub
+
+   # Clean up
+   shred -u argocd-new-key
+   ```
+
+3. **Audit Repository Access**
+   ```bash
+   # List all repository secrets
+   kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=repository
+
+   # Review deploy keys in GitHub
+   # Visit: https://github.com/fortedigital/sturdy-adventure/settings/keys
+   ```
+
+4. **Use Different Keys per Repository**
+   - Don't reuse the same deploy key across repositories
+   - If one key is compromised, only one repository is affected
+   - Easier to track and audit access
+
+#### Troubleshooting Repository Access
+
+**Issue: "permission denied (publickey)"**
+
+```bash
+# Check if secret exists
+kubectl get secret repo-sturdy-adventure -n argocd
+
+# Verify secret has correct label
+kubectl get secret repo-sturdy-adventure -n argocd -o yaml | grep argocd.argoproj.io/secret-type
+
+# Check ArgoCD application controller logs
+kubectl logs -n argocd deployment/argocd-application-controller | grep -i "permission denied"
+
+# Verify deploy key is added to GitHub
+# Visit: https://github.com/fortedigital/sturdy-adventure/settings/keys
+```
+
+**Issue: "Host key verification failed"**
+
+```bash
+# Add GitHub to known_hosts
+kubectl exec -n argocd deployment/argocd-repo-server -- \
+  ssh-keyscan github.com >> ~/.ssh/known_hosts
+
+# Or disable strict host key checking (less secure)
+kubectl patch secret repo-sturdy-adventure -n argocd \
+  --type merge \
+  -p '{"stringData":{"insecure":"true"}}'
+```
+
+**Issue: Repository shows as "Unknown" status**
+
+```bash
+# Check repository server logs
+kubectl logs -n argocd deployment/argocd-repo-server
+
+# Refresh repository connection
+kubectl delete secret repo-sturdy-adventure -n argocd
+# Recreate secret (see Step 3 above)
+
+# Restart ArgoCD components
+kubectl rollout restart deployment argocd-repo-server -n argocd
+kubectl rollout restart deployment argocd-application-controller -n argocd
+```
+
+#### Multiple Repository Setup
+
+For the three-repository pattern (sturdy-adventure, forte-helm, helm-values):
+
+```bash
+# 1. sturdy-adventure (main config repo)
+ssh-keygen -t ed25519 -C "argocd-sturdy-adventure" -f key-sturdy -N ""
+# Add key-sturdy.pub to: https://github.com/fortedigital/sturdy-adventure/settings/keys
+
+# 2. helm-values (private values repo)
+ssh-keygen -t ed25519 -C "argocd-helm-values" -f key-helm-values -N ""
+# Add key-helm-values.pub to: https://github.com/fortedigital/helm-values/settings/keys
+
+# 3. forte-helm is public - no key needed (use HTTPS)
+
+# Create secrets
+kubectl create secret generic repo-sturdy-adventure \
+  --from-file=sshPrivateKey=key-sturdy \
+  --namespace=argocd --dry-run=client -o yaml | \
+  kubectl label --local -f - argocd.argoproj.io/secret-type=repository --dry-run=client -o yaml | \
+  kubectl apply -f -
+
+kubectl create secret generic repo-helm-values \
+  --from-file=sshPrivateKey=key-helm-values \
+  --namespace=argocd --dry-run=client -o yaml | \
+  kubectl label --local -f - argocd.argoproj.io/secret-type=repository --dry-run=client -o yaml | \
+  kubectl apply -f -
+
+# Clean up keys
+shred -u key-sturdy key-helm-values
+```
+
+#### Converting HTTPS to SSH
+
+If you're currently using HTTPS and want to switch to SSH:
+
+```bash
+# 1. Generate and add deploy key (see steps above)
+
+# 2. Update all Application manifests
+# Change from:
+#   repoURL: https://github.com/fortedigital/sturdy-adventure.git
+# To:
+#   repoURL: git@github.com:fortedigital/sturdy-adventure.git
+
+# 3. Update and commit
+find . -name "*.yaml" -type f -exec sed -i 's|https://github.com/fortedigital/|git@github.com:fortedigital/|g' {} +
+
+git add .
+git commit -m "Switch from HTTPS to SSH for repository access"
+git push
+
+# 4. ArgoCD will automatically re-sync with new SSH URLs
+```
+
 ---
 
 ## Day-to-Day Operations