Forte/launchpad
6
0
Fork 0
Code Issues Pull Requests Actions Packages Wiki Activity

ssh access

Browse Source
This commit is contained in:
Danijel Simeunovic
2026-03-16 11:54:32 +01:00
parent 7aff19ccab
commit fae0826400
1 changed files with 341 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

341
docs/OPERATIONS-RUNBOOK.md
View File

@@ -3,6 +3,8 @@
## Table of Contents ## Table of Contents
- [Overview](#overview) - [Overview](#overview)
- [Cluster Bootstrap](#cluster-bootstrap) - [Cluster Bootstrap](#cluster-bootstrap)
- [Initial Cluster Setup](#initial-cluster-setup)
- [ArgoCD Repository Access Setup](#argocd-repository-access-setup)
- [Day-to-Day Operations](#day-to-day-operations) - [Day-to-Day Operations](#day-to-day-operations)
- [Application Management](#application-management) - [Application Management](#application-management)
- [Secret Management](#secret-management) - [Secret Management](#secret-management)
@@ -103,6 +105,345 @@ kubectl get applications -n argocd
5. **Test Slack notifications** by triggering a sync 5. **Test Slack notifications** by triggering a sync
### ArgoCD Repository Access Setup
ArgoCD needs SSH access to private Git repositories to pull manifests and Helm values. This section covers setting up deploy keys for GitHub repositories.
#### Why Deploy Keys?
- **Read-only access**: Deploy keys provide secure, read-only access to repositories
- **No user credentials**: No need to share personal SSH keys or tokens
- **Repository-specific**: Each repository gets its own key for better security
- **Revocable**: Easy to revoke access without affecting other repositories
#### Prerequisites
- kubectl access to the cluster
- Write access to the GitHub repository
- ArgoCD installed and running
#### Setup Procedure
**Step 1: Generate SSH Key Pair**
Generate a dedicated SSH key for ArgoCD without a passphrase (required for automated access):
```bash
# Generate ED25519 key (recommended - smaller and more secure)
ssh-keygen -t ed25519 -C "argocd-deploy-key-sturdy-adventure" -f argocd-deploy-key -N ""
# Or RSA key if ED25519 is not supported
ssh-keygen -t rsa -b 4096 -C "argocd-deploy-key-sturdy-adventure" -f argocd-deploy-key -N ""
```
This creates two files:
- `argocd-deploy-key` - Private key (keep secret)
- `argocd-deploy-key.pub` - Public key (add to GitHub)
**Step 2: Add Public Key to GitHub**
1. Copy the public key:
```bash
cat argocd-deploy-key.pub
```
2. Go to GitHub repository settings:
- Navigate to: `https://github.com/fortedigital/sturdy-adventure/settings/keys`
- Or: Repository → Settings → Deploy keys
3. Click **"Add deploy key"**
- Title: `ArgoCD Production Cluster`
- Key: Paste the public key content
- ☐ Allow write access (leave unchecked - read-only is sufficient)
- Click **"Add key"**
4. Repeat for the `helm-values` repository if it's private:
```bash
# Generate separate key for helm-values repo
ssh-keygen -t ed25519 -C "argocd-deploy-key-helm-values" -f argocd-helm-values-key -N ""
# Add to: https://github.com/fortedigital/helm-values/settings/keys
```
**Step 3: Create Kubernetes Secret**
Add the private key to ArgoCD as a repository secret:
```bash
# Create secret for sturdy-adventure repository
kubectl create secret generic repo-sturdy-adventure \
--from-file=sshPrivateKey=argocd-deploy-key \
--namespace=argocd \
--dry-run=client -o yaml | kubectl apply -f -
# Label it for ArgoCD to recognize
kubectl label secret repo-sturdy-adventure \
-n argocd \
argocd.argoproj.io/secret-type=repository
# Add repository annotations
kubectl annotate secret repo-sturdy-adventure \
-n argocd \
managed-by=argocd.argoproj.io
```
Alternatively, create a complete repository secret with all metadata:
```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: repo-sturdy-adventure
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
annotations:
managed-by: argocd.argoproj.io
type: Opaque
stringData:
type: git
url: git@github.com:fortedigital/sturdy-adventure.git
sshPrivateKey: |
$(cat argocd-deploy-key | sed 's/^/ /')
EOF
```
**Step 4: Register Repository in ArgoCD**
Add the repository to ArgoCD's configuration:
```bash
# Via kubectl (recommended for GitOps)
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: repo-sturdy-adventure
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
type: git
url: git@github.com:fortedigital/sturdy-adventure.git
sshPrivateKey: |
$(cat argocd-deploy-key | sed 's/^/ /')
insecure: "false"
enableLfs: "false"
EOF
# Or via ArgoCD UI
# 1. Open ArgoCD UI: kubectl port-forward svc/argocd-server -n argocd 8080:443
# 2. Navigate to: Settings → Repositories → Connect Repo
# 3. Connection Method: Via SSH
# 4. Repository URL: git@github.com:fortedigital/sturdy-adventure.git
# 5. SSH private key: Paste private key content
# 6. Click "Connect"
```
**Step 5: Verify Repository Access**
```bash
# Check if repository is connected
kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=repository
# Verify connection in ArgoCD UI
# Settings → Repositories → Should show "Successful" status
# Test by creating an application
kubectl apply -f _app-of-apps.yaml
# Check application sync status
kubectl get applications -n argocd
```
#### Testing Repository Access
Create a test application to verify SSH access:
```bash
cat > /tmp/test-repo-access.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: test-repo-access
namespace: argocd
spec:
project: default
source:
repoURL: git@github.com:fortedigital/sturdy-adventure.git
targetRevision: main
path: cluster-resources
destination:
server: https://kubernetes.default.svc
namespace: default
syncPolicy:
automated: null # Manual sync for testing
EOF
kubectl apply -f /tmp/test-repo-access.yaml
# Check if ArgoCD can access the repository
kubectl describe application test-repo-access -n argocd
# Look for sync status - should show repository contents
kubectl get application test-repo-access -n argocd -o jsonpath='{.status.sync.status}'
# Clean up test application
kubectl delete application test-repo-access -n argocd
rm /tmp/test-repo-access.yaml
```
#### Security Best Practices
1. **Secure Private Keys**
```bash
# Store private key securely and delete local copy
# Option 1: Store in password manager (recommended)
# Option 2: Backup to encrypted storage
# Delete local private key after adding to Kubernetes
shred -u argocd-deploy-key
# Or on Windows
# Remove-Item -Path argocd-deploy-key -Force
```
2. **Rotate Keys Regularly**
```bash
# Generate new key
ssh-keygen -t ed25519 -C "argocd-deploy-key-$(date +%Y%m)" -f argocd-new-key -N ""
# Add new public key to GitHub (keep old key for now)
# Update Kubernetes secret
kubectl create secret generic repo-sturdy-adventure \
--from-file=sshPrivateKey=argocd-new-key \
--namespace=argocd \
--dry-run=client -o yaml | kubectl apply -f -
# Test access, then remove old deploy key from GitHub
# Clean up
shred -u argocd-new-key
```
3. **Audit Repository Access**
```bash
# List all repository secrets
kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=repository
# Review deploy keys in GitHub
# Visit: https://github.com/fortedigital/sturdy-adventure/settings/keys
```
4. **Use Different Keys per Repository**
- Don't reuse the same deploy key across repositories
- If one key is compromised, only one repository is affected
- Easier to track and audit access
#### Troubleshooting Repository Access
**Issue: "permission denied (publickey)"**
```bash
# Check if secret exists
kubectl get secret repo-sturdy-adventure -n argocd
# Verify secret has correct label
kubectl get secret repo-sturdy-adventure -n argocd -o yaml | grep argocd.argoproj.io/secret-type
# Check ArgoCD application controller logs
kubectl logs -n argocd deployment/argocd-application-controller | grep -i "permission denied"
# Verify deploy key is added to GitHub
# Visit: https://github.com/fortedigital/sturdy-adventure/settings/keys
```
**Issue: "Host key verification failed"**
```bash
# Add GitHub to known_hosts
kubectl exec -n argocd deployment/argocd-repo-server -- \
ssh-keyscan github.com >> ~/.ssh/known_hosts
# Or disable strict host key checking (less secure)
kubectl patch secret repo-sturdy-adventure -n argocd \
--type merge \
-p '{"stringData":{"insecure":"true"}}'
```
**Issue: Repository shows as "Unknown" status**
```bash
# Check repository server logs
kubectl logs -n argocd deployment/argocd-repo-server
# Refresh repository connection
kubectl delete secret repo-sturdy-adventure -n argocd
# Recreate secret (see Step 3 above)
# Restart ArgoCD components
kubectl rollout restart deployment argocd-repo-server -n argocd
kubectl rollout restart deployment argocd-application-controller -n argocd
```
#### Multiple Repository Setup
For the three-repository pattern (sturdy-adventure, forte-helm, helm-values):
```bash
# 1. sturdy-adventure (main config repo)
ssh-keygen -t ed25519 -C "argocd-sturdy-adventure" -f key-sturdy -N ""
# Add key-sturdy.pub to: https://github.com/fortedigital/sturdy-adventure/settings/keys
# 2. helm-values (private values repo)
ssh-keygen -t ed25519 -C "argocd-helm-values" -f key-helm-values -N ""
# Add key-helm-values.pub to: https://github.com/fortedigital/helm-values/settings/keys
# 3. forte-helm is public - no key needed (use HTTPS)
# Create secrets
kubectl create secret generic repo-sturdy-adventure \
--from-file=sshPrivateKey=key-sturdy \
--namespace=argocd --dry-run=client -o yaml | \
kubectl label --local -f - argocd.argoproj.io/secret-type=repository --dry-run=client -o yaml | \
kubectl apply -f -
kubectl create secret generic repo-helm-values \
--from-file=sshPrivateKey=key-helm-values \
--namespace=argocd --dry-run=client -o yaml | \
kubectl label --local -f - argocd.argoproj.io/secret-type=repository --dry-run=client -o yaml | \
kubectl apply -f -
# Clean up keys
shred -u key-sturdy key-helm-values
```
#### Converting HTTPS to SSH
If you're currently using HTTPS and want to switch to SSH:
```bash
# 1. Generate and add deploy key (see steps above)
# 2. Update all Application manifests
# Change from:
# repoURL: https://github.com/fortedigital/sturdy-adventure.git
# To:
# repoURL: git@github.com:fortedigital/sturdy-adventure.git
# 3. Update and commit
find . -name "*.yaml" -type f -exec sed -i 's|https://github.com/fortedigital/|git@github.com:fortedigital/|g' {} +
git add .
git commit -m "Switch from HTTPS to SSH for repository access"
git push
# 4. ArgoCD will automatically re-sync with new SSH URLs
```
--- ---
## Day-to-Day Operations ## Day-to-Day Operations