vault migration
This commit is contained in:
@@ -60,18 +60,16 @@ If you do need cluster access, install:
|
||||
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
|
||||
```
|
||||
|
||||
2. **kubeseal** - For sealing secrets
|
||||
2. **vault** CLI - For managing secrets in HashiCorp Vault
|
||||
```bash
|
||||
# macOS
|
||||
brew install kubeseal
|
||||
brew install hashicorp/tap/vault
|
||||
|
||||
# Windows
|
||||
choco install kubeseal
|
||||
choco install vault
|
||||
|
||||
# Linux
|
||||
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
|
||||
tar -xvzf kubeseal-0.24.0-linux-amd64.tar.gz
|
||||
sudo mv kubeseal /usr/local/bin/
|
||||
# See https://developer.hashicorp.com/vault/install
|
||||
```
|
||||
|
||||
3. **Git** - Version control
|
||||
@@ -634,115 +632,100 @@ git push
|
||||
|
||||
### Understanding Secret Management
|
||||
|
||||
**NEVER commit plain secrets to Git.** We use **Sealed Secrets** to encrypt secrets before committing.
|
||||
Secrets are managed via **HashiCorp Vault** and synced to Kubernetes by the **Vault Secrets Operator (VSO)**. See [Vault Secrets Operator Reference](vault-secrets-operator.md) for full details.
|
||||
|
||||
**NEVER commit plain secret values to Git.** Only VaultStaticSecret CRD manifests are committed.
|
||||
|
||||
### Creating a New Secret
|
||||
|
||||
#### Step 1: Create Plain Secret Locally
|
||||
#### Step 1: Write Secret to Vault
|
||||
|
||||
```bash
|
||||
cd ~/dev/k8s/launchpad
|
||||
|
||||
# Create secret in private/ folder (Git-ignored)
|
||||
kubectl create secret generic myapp-credentials \
|
||||
--from-literal=API_KEY=your-secret-key-here \
|
||||
--from-literal=DB_PASSWORD=super-secret-password \
|
||||
--dry-run=client -o yaml > private/myapp-credentials.yaml
|
||||
vault kv put kv/myapp/myapp-credentials \
|
||||
API_KEY=your-secret-key-here \
|
||||
DB_PASSWORD=super-secret-password
|
||||
```
|
||||
|
||||
**DO NOT commit this file!** It's in `private/` which is Git-ignored.
|
||||
#### Step 2: Create VaultStaticSecret CRD
|
||||
|
||||
#### Step 2: Seal the Secret
|
||||
Create a YAML file (e.g., `apps/base/myapp/myapp-credentials-vault.yaml`):
|
||||
|
||||
Seal your secret:
|
||||
|
||||
```bash
|
||||
kubeseal --format=yaml \
|
||||
--namespace=myapp \
|
||||
< private/myapp-credentials.yaml \
|
||||
> secrets/myapp-credentials-sealed.yaml
|
||||
```yaml
|
||||
apiVersion: secrets.hashicorp.com/v1beta1
|
||||
kind: VaultStaticSecret
|
||||
metadata:
|
||||
name: myapp-credentials
|
||||
namespace: myapp
|
||||
spec:
|
||||
type: kv-v2
|
||||
mount: kv
|
||||
path: myapp/myapp-credentials
|
||||
destination:
|
||||
name: myapp-credentials
|
||||
create: true
|
||||
refreshAfter: 30s
|
||||
vaultAuthRef: vault-auth
|
||||
```
|
||||
|
||||
#### Step 3: Commit Sealed Secret
|
||||
#### Step 3: Add VaultAuth (if new namespace)
|
||||
|
||||
If this is a new namespace, also create a `vault-auth.yaml` with a ServiceAccount and VaultAuth CRD. See [VSO Reference](vault-secrets-operator.md#vaultauth) for template.
|
||||
|
||||
#### Step 4: Commit and Push
|
||||
|
||||
```bash
|
||||
git add secrets/myapp-credentials-sealed.yaml
|
||||
git commit -m "Add myapp credentials (sealed)"
|
||||
git add apps/base/myapp/myapp-credentials-vault.yaml
|
||||
git commit -m "Add myapp credentials (VSO)"
|
||||
git push
|
||||
```
|
||||
|
||||
#### Step 4: Reference Secret in Application
|
||||
ArgoCD syncs the CRD, VSO creates the K8s Secret.
|
||||
|
||||
#### Step 5: Reference Secret in Application
|
||||
|
||||
Update your `helm-prod-values/myapp/values.yaml`:
|
||||
|
||||
```yaml
|
||||
app:
|
||||
envSecretName: "myapp-credentials" # References the SealedSecret
|
||||
envSecretName: "myapp-credentials" # VSO creates this K8s Secret
|
||||
```
|
||||
|
||||
Commit and push:
|
||||
### Updating / Rotating a Secret
|
||||
|
||||
**No git commit needed** — just update in Vault:
|
||||
|
||||
```bash
|
||||
cd ~/dev/k8s/helm-prod-values
|
||||
git add myapp/values.yaml
|
||||
git commit -m "Reference myapp credentials"
|
||||
git push
|
||||
vault kv put kv/myapp/myapp-credentials \
|
||||
API_KEY=new-key-here \
|
||||
DB_PASSWORD=new-password
|
||||
```
|
||||
|
||||
### Updating a Secret
|
||||
|
||||
To update an existing secret:
|
||||
VSO picks up changes within 30 seconds. Restart pods if they don't watch for secret updates:
|
||||
|
||||
```bash
|
||||
# 1. Create new version of secret
|
||||
kubectl create secret generic myapp-credentials \
|
||||
--from-literal=API_KEY=new-key-here \
|
||||
--from-literal=DB_PASSWORD=new-password \
|
||||
--dry-run=client -o yaml > private/myapp-credentials.yaml
|
||||
|
||||
# 2. Seal it
|
||||
kubeseal --format=yaml \
|
||||
--namespace=myapp \
|
||||
< private/myapp-credentials.yaml \
|
||||
> secrets/myapp-credentials-sealed.yaml
|
||||
|
||||
# 3. Commit sealed version
|
||||
git add secrets/myapp-credentials-sealed.yaml
|
||||
git commit -m "Update myapp credentials"
|
||||
git push
|
||||
|
||||
# 4. Restart pods to pick up new secret
|
||||
kubectl rollout restart deployment myapp -n myapp
|
||||
```
|
||||
|
||||
### Secret Best Practices
|
||||
|
||||
✅ **DO**:
|
||||
- Store secrets in `private/` folder locally
|
||||
- Always seal secrets before committing
|
||||
- Delete plain secrets after sealing
|
||||
- Use meaningful secret names
|
||||
- Write secrets to Vault via UI or CLI — never commit values to Git
|
||||
- Use meaningful secret names matching the KV path convention: `kv/{namespace}/{secret-name}`
|
||||
- Document what each secret contains
|
||||
|
||||
❌ **DON'T**:
|
||||
- Commit plain secrets to Git
|
||||
- Share secrets via Slack/email
|
||||
- Hard-code secrets in code
|
||||
- Use the same secret across multiple environments
|
||||
- Store secrets in Docker images
|
||||
- Use Vault's versioning for audit trail
|
||||
|
||||
### Where Secrets Are Stored
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Location │ Content │ Committed?│
|
||||
├──────────────────────────┼────────────────────┼────────────┤
|
||||
│ private/ │ Plain secrets │ ❌ NO │
|
||||
│ secrets/ │ Sealed secrets │ ✅ YES │
|
||||
│ Kubernetes cluster │ Unsealed secrets │ N/A │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ Location │ Content │ In Git? │
|
||||
├────────────────────────────┼─────────────────────────┼──────────┤
|
||||
│ Vault KV (kv/{ns}/{name}) │ Secret values │ ❌ NO │
|
||||
│ VaultStaticSecret CRD │ Sync config (no values)│ ✅ YES │
|
||||
│ Kubernetes cluster │ K8s Secret (synced) │ N/A │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Sealed Secrets Controller** in the cluster decrypts sealed secrets automatically.
|
||||
**Vault Secrets Operator** syncs secrets from Vault to K8s automatically (30s refresh).
|
||||
|
||||
---
|
||||
|
||||
@@ -876,28 +859,13 @@ In your identity provider (e.g., Keycloak):
|
||||
#### Step 2: Create OIDC Secret
|
||||
|
||||
```bash
|
||||
# Create plain secret
|
||||
kubectl create secret generic auth-oidc \
|
||||
--from-literal=client-secret=your-oidc-client-secret \
|
||||
--from-literal=cookie-secret=$(openssl rand -hex 32) \
|
||||
--namespace=myapp \
|
||||
--dry-run=client -o yaml > private/myapp-auth-oidc.yaml
|
||||
# Write OIDC secret to Vault
|
||||
vault kv put kv/myapp/auth-oidc \
|
||||
client-secret=your-oidc-client-secret \
|
||||
cookie-secret=$(openssl rand -hex 32)
|
||||
|
||||
# Seal it
|
||||
kubeseal --format=yaml \
|
||||
--cert=pub-cert.pem \
|
||||
--namespace=myapp \
|
||||
< private/myapp-auth-oidc.yaml \
|
||||
> secrets/myapp-auth-oidc-sealed.yaml
|
||||
|
||||
# Commit sealed secret
|
||||
cd ~/dev/k8s/launchpad
|
||||
git add secrets/myapp-auth-oidc-sealed.yaml
|
||||
git commit -m "Add OIDC secrets for myapp"
|
||||
git push
|
||||
|
||||
# Clean up
|
||||
rm private/myapp-auth-oidc.yaml
|
||||
# Create VaultStaticSecret CRD (see docs/vault-secrets-operator.md for template)
|
||||
# Add to apps/base/myapp/auth-oidc-vault.yaml and commit
|
||||
```
|
||||
|
||||
#### Step 3: Configure Helm Values
|
||||
@@ -1127,16 +1095,13 @@ ingress:
|
||||
host: web-app.forteapps.net
|
||||
```
|
||||
|
||||
**With sealed OIDC secret**:
|
||||
**With Vault OIDC secret**:
|
||||
```bash
|
||||
# Create and seal secret
|
||||
kubectl create secret generic auth-oidc \
|
||||
--from-literal=client-secret=super-secret-value \
|
||||
--from-literal=cookie-secret=$(openssl rand -hex 32) \
|
||||
--namespace=web-app \
|
||||
--dry-run=client -o yaml | \
|
||||
kubeseal --format=yaml --cert=pub-cert.pem --namespace=web-app \
|
||||
> secrets/web-app-auth-oidc-sealed.yaml
|
||||
# Write OIDC secret to Vault
|
||||
vault kv put kv/web-app/auth-oidc \
|
||||
client-secret=super-secret-value \
|
||||
cookie-secret=$(openssl rand -hex 32)
|
||||
# Then create VaultStaticSecret CRD — see docs/vault-secrets-operator.md
|
||||
```
|
||||
|
||||
#### Example 3: MCP Server with OAuth 2.0
|
||||
@@ -1264,7 +1229,7 @@ kubectl logs -n myapp <pod-name> -c authn
|
||||
- Use token auth for service-to-service communication
|
||||
- Rotate tokens and secrets regularly
|
||||
- Use strong random tokens (32+ bytes)
|
||||
- Store client secrets in SealedSecrets
|
||||
- Store client secrets in Vault
|
||||
- Test authentication before deploying to production
|
||||
- Document which tokens/users have access
|
||||
|
||||
@@ -1568,22 +1533,22 @@ curl http://localhost:8080
|
||||
|
||||
#### Problem: Secret not found
|
||||
|
||||
**Check if SealedSecret exists:**
|
||||
**Check VSO sync status:**
|
||||
```bash
|
||||
kubectl get sealedsecret -n myapp
|
||||
kubectl get vaultstaticsecret -n myapp
|
||||
kubectl get secret -n myapp
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
```bash
|
||||
# Check if secret is in Git
|
||||
ls -l secrets/myapp-credentials-sealed.yaml
|
||||
# Check VaultAuth is authenticated
|
||||
kubectl get vaultauth -n myapp
|
||||
|
||||
# Re-apply sealed secret
|
||||
kubectl apply -f secrets/myapp-credentials-sealed.yaml
|
||||
# Check VaultStaticSecret events
|
||||
kubectl describe vaultstaticsecret myapp-credentials -n myapp
|
||||
|
||||
# Check sealed-secrets-controller logs
|
||||
kubectl logs -n kube-system deployment/sealed-secrets-controller
|
||||
# Verify secret exists in Vault
|
||||
vault kv get kv/myapp/myapp-credentials
|
||||
```
|
||||
|
||||
#### Problem: Secret exists but pods can't access it
|
||||
@@ -1694,7 +1659,7 @@ If you're stuck:
|
||||
### Secret Management
|
||||
|
||||
✅ **DO**:
|
||||
- Use kubeseal for all secrets
|
||||
- Use Vault for all secrets (see docs/vault-secrets-operator.md)
|
||||
- Store plain secrets in password manager
|
||||
- Rotate secrets regularly
|
||||
- Use different secrets per environment
|
||||
@@ -1746,16 +1711,9 @@ kubectl rollout restart deployment myapp -n myapp
|
||||
# Port-forward to service
|
||||
kubectl port-forward -n myapp service/myapp 8080:3000
|
||||
|
||||
# Create secret
|
||||
kubectl create secret generic myapp-credentials \
|
||||
--from-literal=KEY=value \
|
||||
--dry-run=client -o yaml > private/myapp-credentials.yaml
|
||||
|
||||
# Seal secret
|
||||
kubeseal --format=yaml \
|
||||
--cert=pub-cert.pem \
|
||||
< private/myapp-credentials.yaml \
|
||||
> secrets/myapp-credentials-sealed.yaml
|
||||
# Write secret to Vault
|
||||
vault kv put kv/myapp/myapp-credentials KEY=value
|
||||
# Create VaultStaticSecret CRD — see docs/vault-secrets-operator.md
|
||||
```
|
||||
|
||||
### Repository Locations
|
||||
|
||||
Reference in New Issue
Block a user