ssh access
This commit is contained in:
Danijel Simeunovic
@@ -3,6 +3,8 @@
|
||||
## Table of Contents
|
||||
- [Overview](#overview)
|
||||
- [Cluster Bootstrap](#cluster-bootstrap)
|
||||
- [Initial Cluster Setup](#initial-cluster-setup)
|
||||
- [ArgoCD Repository Access Setup](#argocd-repository-access-setup)
|
||||
- [Day-to-Day Operations](#day-to-day-operations)
|
||||
- [Application Management](#application-management)
|
||||
- [Secret Management](#secret-management)
|
||||
@@ -103,6 +105,345 @@ kubectl get applications -n argocd
|
||||
|
||||
5. **Test Slack notifications** by triggering a sync
|
||||
|
||||
### ArgoCD Repository Access Setup
|
||||
|
||||
ArgoCD needs SSH access to private Git repositories to pull manifests and Helm values. This section covers setting up deploy keys for GitHub repositories.
|
||||
|
||||
#### Why Deploy Keys?
|
||||
|
||||
- **Read-only access**: Deploy keys provide secure, read-only access to repositories
|
||||
- **No user credentials**: No need to share personal SSH keys or tokens
|
||||
- **Repository-specific**: Each repository gets its own key for better security
|
||||
- **Revocable**: Easy to revoke access without affecting other repositories
|
||||
|
||||
#### Prerequisites
|
||||
|
||||
- kubectl access to the cluster
|
||||
- Write access to the GitHub repository
|
||||
- ArgoCD installed and running
|
||||
|
||||
#### Setup Procedure
|
||||
|
||||
**Step 1: Generate SSH Key Pair**
|
||||
|
||||
Generate a dedicated SSH key for ArgoCD without a passphrase (required for automated access):
|
||||
|
||||
```bash
|
||||
# Generate ED25519 key (recommended - smaller and more secure)
|
||||
ssh-keygen -t ed25519 -C "argocd-deploy-key-sturdy-adventure" -f argocd-deploy-key -N ""
|
||||
|
||||
# Or RSA key if ED25519 is not supported
|
||||
ssh-keygen -t rsa -b 4096 -C "argocd-deploy-key-sturdy-adventure" -f argocd-deploy-key -N ""
|
||||
```
|
||||
|
||||
This creates two files:
|
||||
- `argocd-deploy-key` - Private key (keep secret)
|
||||
- `argocd-deploy-key.pub` - Public key (add to GitHub)
|
||||
|
||||
**Step 2: Add Public Key to GitHub**
|
||||
|
||||
1. Copy the public key:
|
||||
```bash
|
||||
cat argocd-deploy-key.pub
|
||||
```
|
||||
|
||||
2. Go to GitHub repository settings:
|
||||
- Navigate to: `https://github.com/fortedigital/sturdy-adventure/settings/keys`
|
||||
- Or: Repository → Settings → Deploy keys
|
||||
|
||||
3. Click **"Add deploy key"**
|
||||
- Title: `ArgoCD Production Cluster`
|
||||
- Key: Paste the public key content
|
||||
- ☐ Allow write access (leave unchecked - read-only is sufficient)
|
||||
- Click **"Add key"**
|
||||
|
||||
4. Repeat for the `helm-values` repository if it's private:
|
||||
```bash
|
||||
# Generate separate key for helm-values repo
|
||||
ssh-keygen -t ed25519 -C "argocd-deploy-key-helm-values" -f argocd-helm-values-key -N ""
|
||||
|
||||
# Add to: https://github.com/fortedigital/helm-values/settings/keys
|
||||
```
|
||||
|
||||
**Step 3: Create Kubernetes Secret**
|
||||
|
||||
Add the private key to ArgoCD as a repository secret:
|
||||
|
||||
```bash
|
||||
# Create secret for sturdy-adventure repository
|
||||
kubectl create secret generic repo-sturdy-adventure \
|
||||
--from-file=sshPrivateKey=argocd-deploy-key \
|
||||
--namespace=argocd \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
# Label it for ArgoCD to recognize
|
||||
kubectl label secret repo-sturdy-adventure \
|
||||
-n argocd \
|
||||
argocd.argoproj.io/secret-type=repository
|
||||
|
||||
# Add repository annotations
|
||||
kubectl annotate secret repo-sturdy-adventure \
|
||||
-n argocd \
|
||||
managed-by=argocd.argoproj.io
|
||||
```
|
||||
|
||||
Alternatively, create a complete repository secret with all metadata:
|
||||
|
||||
```bash
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: repo-sturdy-adventure
|
||||
namespace: argocd
|
||||
labels:
|
||||
argocd.argoproj.io/secret-type: repository
|
||||
annotations:
|
||||
managed-by: argocd.argoproj.io
|
||||
type: Opaque
|
||||
stringData:
|
||||
type: git
|
||||
url: git@github.com:fortedigital/sturdy-adventure.git
|
||||
sshPrivateKey: |
|
||||
$(cat argocd-deploy-key | sed 's/^/ /')
|
||||
EOF
|
||||
```
|
||||
|
||||
**Step 4: Register Repository in ArgoCD**
|
||||
|
||||
Add the repository to ArgoCD's configuration:
|
||||
|
||||
```bash
|
||||
# Via kubectl (recommended for GitOps)
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: repo-sturdy-adventure
|
||||
namespace: argocd
|
||||
labels:
|
||||
argocd.argoproj.io/secret-type: repository
|
||||
type: Opaque
|
||||
stringData:
|
||||
type: git
|
||||
url: git@github.com:fortedigital/sturdy-adventure.git
|
||||
sshPrivateKey: |
|
||||
$(cat argocd-deploy-key | sed 's/^/ /')
|
||||
insecure: "false"
|
||||
enableLfs: "false"
|
||||
EOF
|
||||
|
||||
# Or via ArgoCD UI
|
||||
# 1. Open ArgoCD UI: kubectl port-forward svc/argocd-server -n argocd 8080:443
|
||||
# 2. Navigate to: Settings → Repositories → Connect Repo
|
||||
# 3. Connection Method: Via SSH
|
||||
# 4. Repository URL: git@github.com:fortedigital/sturdy-adventure.git
|
||||
# 5. SSH private key: Paste private key content
|
||||
# 6. Click "Connect"
|
||||
```
|
||||
|
||||
**Step 5: Verify Repository Access**
|
||||
|
||||
```bash
|
||||
# Check if repository is connected
|
||||
kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=repository
|
||||
|
||||
# Verify connection in ArgoCD UI
|
||||
# Settings → Repositories → Should show "Successful" status
|
||||
|
||||
# Test by creating an application
|
||||
kubectl apply -f _app-of-apps.yaml
|
||||
|
||||
# Check application sync status
|
||||
kubectl get applications -n argocd
|
||||
```
|
||||
|
||||
#### Testing Repository Access
|
||||
|
||||
Create a test application to verify SSH access:
|
||||
|
||||
```bash
|
||||
cat > /tmp/test-repo-access.yaml <<EOF
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: test-repo-access
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: default
|
||||
source:
|
||||
repoURL: git@github.com:fortedigital/sturdy-adventure.git
|
||||
targetRevision: main
|
||||
path: cluster-resources
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: default
|
||||
syncPolicy:
|
||||
automated: null # Manual sync for testing
|
||||
EOF
|
||||
|
||||
kubectl apply -f /tmp/test-repo-access.yaml
|
||||
|
||||
# Check if ArgoCD can access the repository
|
||||
kubectl describe application test-repo-access -n argocd
|
||||
|
||||
# Look for sync status - should show repository contents
|
||||
kubectl get application test-repo-access -n argocd -o jsonpath='{.status.sync.status}'
|
||||
|
||||
# Clean up test application
|
||||
kubectl delete application test-repo-access -n argocd
|
||||
rm /tmp/test-repo-access.yaml
|
||||
```
|
||||
|
||||
#### Security Best Practices
|
||||
|
||||
1. **Secure Private Keys**
|
||||
```bash
|
||||
# Store private key securely and delete local copy
|
||||
# Option 1: Store in password manager (recommended)
|
||||
# Option 2: Backup to encrypted storage
|
||||
|
||||
# Delete local private key after adding to Kubernetes
|
||||
shred -u argocd-deploy-key
|
||||
|
||||
# Or on Windows
|
||||
# Remove-Item -Path argocd-deploy-key -Force
|
||||
```
|
||||
|
||||
2. **Rotate Keys Regularly**
|
||||
```bash
|
||||
# Generate new key
|
||||
ssh-keygen -t ed25519 -C "argocd-deploy-key-$(date +%Y%m)" -f argocd-new-key -N ""
|
||||
|
||||
# Add new public key to GitHub (keep old key for now)
|
||||
|
||||
# Update Kubernetes secret
|
||||
kubectl create secret generic repo-sturdy-adventure \
|
||||
--from-file=sshPrivateKey=argocd-new-key \
|
||||
--namespace=argocd \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
# Test access, then remove old deploy key from GitHub
|
||||
|
||||
# Clean up
|
||||
shred -u argocd-new-key
|
||||
```
|
||||
|
||||
3. **Audit Repository Access**
|
||||
```bash
|
||||
# List all repository secrets
|
||||
kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=repository
|
||||
|
||||
# Review deploy keys in GitHub
|
||||
# Visit: https://github.com/fortedigital/sturdy-adventure/settings/keys
|
||||
```
|
||||
|
||||
4. **Use Different Keys per Repository**
|
||||
- Don't reuse the same deploy key across repositories
|
||||
- If one key is compromised, only one repository is affected
|
||||
- Easier to track and audit access
|
||||
|
||||
#### Troubleshooting Repository Access
|
||||
|
||||
**Issue: "permission denied (publickey)"**
|
||||
|
||||
```bash
|
||||
# Check if secret exists
|
||||
kubectl get secret repo-sturdy-adventure -n argocd
|
||||
|
||||
# Verify secret has correct label
|
||||
kubectl get secret repo-sturdy-adventure -n argocd -o yaml | grep argocd.argoproj.io/secret-type
|
||||
|
||||
# Check ArgoCD application controller logs
|
||||
kubectl logs -n argocd deployment/argocd-application-controller | grep -i "permission denied"
|
||||
|
||||
# Verify deploy key is added to GitHub
|
||||
# Visit: https://github.com/fortedigital/sturdy-adventure/settings/keys
|
||||
```
|
||||
|
||||
**Issue: "Host key verification failed"**
|
||||
|
||||
```bash
|
||||
# Add GitHub to known_hosts
|
||||
kubectl exec -n argocd deployment/argocd-repo-server -- \
|
||||
ssh-keyscan github.com >> ~/.ssh/known_hosts
|
||||
|
||||
# Or disable strict host key checking (less secure)
|
||||
kubectl patch secret repo-sturdy-adventure -n argocd \
|
||||
--type merge \
|
||||
-p '{"stringData":{"insecure":"true"}}'
|
||||
```
|
||||
|
||||
**Issue: Repository shows as "Unknown" status**
|
||||
|
||||
```bash
|
||||
# Check repository server logs
|
||||
kubectl logs -n argocd deployment/argocd-repo-server
|
||||
|
||||
# Refresh repository connection
|
||||
kubectl delete secret repo-sturdy-adventure -n argocd
|
||||
# Recreate secret (see Step 3 above)
|
||||
|
||||
# Restart ArgoCD components
|
||||
kubectl rollout restart deployment argocd-repo-server -n argocd
|
||||
kubectl rollout restart deployment argocd-application-controller -n argocd
|
||||
```
|
||||
|
||||
#### Multiple Repository Setup
|
||||
|
||||
For the three-repository pattern (sturdy-adventure, forte-helm, helm-values):
|
||||
|
||||
```bash
|
||||
# 1. sturdy-adventure (main config repo)
|
||||
ssh-keygen -t ed25519 -C "argocd-sturdy-adventure" -f key-sturdy -N ""
|
||||
# Add key-sturdy.pub to: https://github.com/fortedigital/sturdy-adventure/settings/keys
|
||||
|
||||
# 2. helm-values (private values repo)
|
||||
ssh-keygen -t ed25519 -C "argocd-helm-values" -f key-helm-values -N ""
|
||||
# Add key-helm-values.pub to: https://github.com/fortedigital/helm-values/settings/keys
|
||||
|
||||
# 3. forte-helm is public - no key needed (use HTTPS)
|
||||
|
||||
# Create secrets
|
||||
kubectl create secret generic repo-sturdy-adventure \
|
||||
--from-file=sshPrivateKey=key-sturdy \
|
||||
--namespace=argocd --dry-run=client -o yaml | \
|
||||
kubectl label --local -f - argocd.argoproj.io/secret-type=repository --dry-run=client -o yaml | \
|
||||
kubectl apply -f -
|
||||
|
||||
kubectl create secret generic repo-helm-values \
|
||||
--from-file=sshPrivateKey=key-helm-values \
|
||||
--namespace=argocd --dry-run=client -o yaml | \
|
||||
kubectl label --local -f - argocd.argoproj.io/secret-type=repository --dry-run=client -o yaml | \
|
||||
kubectl apply -f -
|
||||
|
||||
# Clean up keys
|
||||
shred -u key-sturdy key-helm-values
|
||||
```
|
||||
|
||||
#### Converting HTTPS to SSH
|
||||
|
||||
If you're currently using HTTPS and want to switch to SSH:
|
||||
|
||||
```bash
|
||||
# 1. Generate and add deploy key (see steps above)
|
||||
|
||||
# 2. Update all Application manifests
|
||||
# Change from:
|
||||
# repoURL: https://github.com/fortedigital/sturdy-adventure.git
|
||||
# To:
|
||||
# repoURL: git@github.com:fortedigital/sturdy-adventure.git
|
||||
|
||||
# 3. Update and commit
|
||||
find . -name "*.yaml" -type f -exec sed -i 's|https://github.com/fortedigital/|git@github.com:fortedigital/|g' {} +
|
||||
|
||||
git add .
|
||||
git commit -m "Switch from HTTPS to SSH for repository access"
|
||||
git push
|
||||
|
||||
# 4. ArgoCD will automatically re-sync with new SSH URLs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Day-to-Day Operations
|
||||
|
||||
Reference in New Issue
Block a user