Forte/launchpad
6
0
Fork 0
Code Issues Pull Requests Actions Packages Wiki Activity

vault migration

Browse Source
This commit is contained in:
Danijel Simeunovic
2026-04-30 22:38:33 +02:00
parent 2e09a2d404
commit 73376a0a7d
49 changed files with 1103 additions and 272 deletions
Download Patch File Download Diff File Expand all files Collapse all files

208
docs/DEVELOPER-GUIDE.md
View File

@@ -60,18 +60,16 @@ If you do need cluster access, install:
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
```
2. **kubeseal** - For sealing secrets
2. **vault** CLI - For managing secrets in HashiCorp Vault
```bash
# macOS
brew install kubeseal
brew install hashicorp/tap/vault
# Windows
choco install kubeseal
choco install vault
# Linux
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
tar -xvzf kubeseal-0.24.0-linux-amd64.tar.gz
sudo mv kubeseal /usr/local/bin/
# See https://developer.hashicorp.com/vault/install
```
3. **Git** - Version control
@@ -634,115 +632,100 @@ git push
### Understanding Secret Management
**NEVER commit plain secrets to Git.** We use **Sealed Secrets** to encrypt secrets before committing.
Secrets are managed via **HashiCorp Vault** and synced to Kubernetes by the **Vault Secrets Operator (VSO)**. See [Vault Secrets Operator Reference](vault-secrets-operator.md) for full details.
**NEVER commit plain secret values to Git.** Only VaultStaticSecret CRD manifests are committed.
### Creating a New Secret
#### Step 1: Create Plain Secret Locally
#### Step 1: Write Secret to Vault
```bash
cd ~/dev/k8s/launchpad
# Create secret in private/ folder (Git-ignored)
kubectl create secret generic myapp-credentials \
--from-literal=API_KEY=your-secret-key-here \
--from-literal=DB_PASSWORD=super-secret-password \
--dry-run=client -o yaml > private/myapp-credentials.yaml
vault kv put kv/myapp/myapp-credentials \
API_KEY=your-secret-key-here \
DB_PASSWORD=super-secret-password
```
**DO NOT commit this file!** It's in `private/` which is Git-ignored.
#### Step 2: Create VaultStaticSecret CRD
#### Step 2: Seal the Secret
Create a YAML file (e.g., `apps/base/myapp/myapp-credentials-vault.yaml`):
Seal your secret:
```bash
kubeseal --format=yaml \
--namespace=myapp \
< private/myapp-credentials.yaml \
> secrets/myapp-credentials-sealed.yaml
```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: myapp-credentials
namespace: myapp
spec:
type: kv-v2
mount: kv
path: myapp/myapp-credentials
destination:
name: myapp-credentials
create: true
refreshAfter: 30s
vaultAuthRef: vault-auth
```
#### Step 3: Commit Sealed Secret
#### Step 3: Add VaultAuth (if new namespace)
If this is a new namespace, also create a `vault-auth.yaml` with a ServiceAccount and VaultAuth CRD. See [VSO Reference](vault-secrets-operator.md#vaultauth) for template.
#### Step 4: Commit and Push
```bash
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Add myapp credentials (sealed)"
git add apps/base/myapp/myapp-credentials-vault.yaml
git commit -m "Add myapp credentials (VSO)"
git push
```
#### Step 4: Reference Secret in Application
ArgoCD syncs the CRD, VSO creates the K8s Secret.
#### Step 5: Reference Secret in Application
Update your `helm-prod-values/myapp/values.yaml`:
```yaml
app:
envSecretName: "myapp-credentials" # References the SealedSecret
envSecretName: "myapp-credentials" # VSO creates this K8s Secret
```
Commit and push:
### Updating / Rotating a Secret
**No git commit needed** — just update in Vault:
```bash
cd ~/dev/k8s/helm-prod-values
git add myapp/values.yaml
git commit -m "Reference myapp credentials"
git push
vault kv put kv/myapp/myapp-credentials \
API_KEY=new-key-here \
DB_PASSWORD=new-password
```
### Updating a Secret
To update an existing secret:
VSO picks up changes within 30 seconds. Restart pods if they don't watch for secret updates:
```bash
# 1. Create new version of secret
kubectl create secret generic myapp-credentials \
--from-literal=API_KEY=new-key-here \
--from-literal=DB_PASSWORD=new-password \
--dry-run=client -o yaml > private/myapp-credentials.yaml
# 2. Seal it
kubeseal --format=yaml \
--namespace=myapp \
< private/myapp-credentials.yaml \
> secrets/myapp-credentials-sealed.yaml
# 3. Commit sealed version
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Update myapp credentials"
git push
# 4. Restart pods to pick up new secret
kubectl rollout restart deployment myapp -n myapp
```
### Secret Best Practices
✅ **DO**:
- Store secrets in `private/` folder locally
- Always seal secrets before committing
- Delete plain secrets after sealing
- Use meaningful secret names
- Write secrets to Vault via UI or CLI — never commit values to Git
- Use meaningful secret names matching the KV path convention: `kv/{namespace}/{secret-name}`
- Document what each secret contains
❌ **DON'T**:
- Commit plain secrets to Git
- Share secrets via Slack/email
- Hard-code secrets in code
- Use the same secret across multiple environments
- Store secrets in Docker images
- Use Vault's versioning for audit trail
### Where Secrets Are Stored
```
┌─────────────────────────────────────────────────────────────┐
│ Location │ Content │ Committed?│
├──────────────────────────┼────────────────────┼────────────┤
│ private/ │ Plain secrets │ ❌ NO │
│ secrets/ │ Sealed secrets │ ✅ YES │
│ Kubernetes cluster │ Unsealed secrets │ N/A │
└─────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────┐
│ Location │ Content │ In Git? │
├────────────────────────────┼─────────────────────────┼──────────┤
│ Vault KV (kv/{ns}/{name}) │ Secret values │ ❌ NO │
│ VaultStaticSecret CRD │ Sync config (no values)│ ✅ YES │
│ Kubernetes cluster │ K8s Secret (synced) │ N/A │
└──────────────────────────────────────────────────────────────────┘
```
**Sealed Secrets Controller** in the cluster decrypts sealed secrets automatically.
**Vault Secrets Operator** syncs secrets from Vault to K8s automatically (30s refresh).
---
@@ -876,28 +859,13 @@ In your identity provider (e.g., Keycloak):
#### Step 2: Create OIDC Secret
```bash
# Create plain secret
kubectl create secret generic auth-oidc \
--from-literal=client-secret=your-oidc-client-secret \
--from-literal=cookie-secret=$(openssl rand -hex 32) \
--namespace=myapp \
--dry-run=client -o yaml > private/myapp-auth-oidc.yaml
# Write OIDC secret to Vault
vault kv put kv/myapp/auth-oidc \
client-secret=your-oidc-client-secret \
cookie-secret=$(openssl rand -hex 32)
# Seal it
kubeseal --format=yaml \
--cert=pub-cert.pem \
--namespace=myapp \
< private/myapp-auth-oidc.yaml \
> secrets/myapp-auth-oidc-sealed.yaml
# Commit sealed secret
cd ~/dev/k8s/launchpad
git add secrets/myapp-auth-oidc-sealed.yaml
git commit -m "Add OIDC secrets for myapp"
git push
# Clean up
rm private/myapp-auth-oidc.yaml
# Create VaultStaticSecret CRD (see docs/vault-secrets-operator.md for template)
# Add to apps/base/myapp/auth-oidc-vault.yaml and commit
```
#### Step 3: Configure Helm Values
@@ -1127,16 +1095,13 @@ ingress:
host: web-app.forteapps.net
```
**With sealed OIDC secret**:
**With Vault OIDC secret**:
```bash
# Create and seal secret
kubectl create secret generic auth-oidc \
--from-literal=client-secret=super-secret-value \
--from-literal=cookie-secret=$(openssl rand -hex 32) \
--namespace=web-app \
--dry-run=client -o yaml | \
kubeseal --format=yaml --cert=pub-cert.pem --namespace=web-app \
> secrets/web-app-auth-oidc-sealed.yaml
# Write OIDC secret to Vault
vault kv put kv/web-app/auth-oidc \
client-secret=super-secret-value \
cookie-secret=$(openssl rand -hex 32)
# Then create VaultStaticSecret CRD — see docs/vault-secrets-operator.md
```
#### Example 3: MCP Server with OAuth 2.0
@@ -1264,7 +1229,7 @@ kubectl logs -n myapp <pod-name> -c authn
- Use token auth for service-to-service communication
- Rotate tokens and secrets regularly
- Use strong random tokens (32+ bytes)
- Store client secrets in SealedSecrets
- Store client secrets in Vault
- Test authentication before deploying to production
- Document which tokens/users have access
@@ -1568,22 +1533,22 @@ curl http://localhost:8080
#### Problem: Secret not found
**Check if SealedSecret exists:**
**Check VSO sync status:**
```bash
kubectl get sealedsecret -n myapp
kubectl get vaultstaticsecret -n myapp
kubectl get secret -n myapp
```
**Solutions:**
```bash
# Check if secret is in Git
ls -l secrets/myapp-credentials-sealed.yaml
# Check VaultAuth is authenticated
kubectl get vaultauth -n myapp
# Re-apply sealed secret
kubectl apply -f secrets/myapp-credentials-sealed.yaml
# Check VaultStaticSecret events
kubectl describe vaultstaticsecret myapp-credentials -n myapp
# Check sealed-secrets-controller logs
kubectl logs -n kube-system deployment/sealed-secrets-controller
# Verify secret exists in Vault
vault kv get kv/myapp/myapp-credentials
```
#### Problem: Secret exists but pods can't access it
@@ -1694,7 +1659,7 @@ If you're stuck:
### Secret Management
✅ **DO**:
- Use kubeseal for all secrets
- Use Vault for all secrets (see docs/vault-secrets-operator.md)
- Store plain secrets in password manager
- Rotate secrets regularly
- Use different secrets per environment
@@ -1746,16 +1711,9 @@ kubectl rollout restart deployment myapp -n myapp
# Port-forward to service
kubectl port-forward -n myapp service/myapp 8080:3000
# Create secret
kubectl create secret generic myapp-credentials \
--from-literal=KEY=value \
--dry-run=client -o yaml > private/myapp-credentials.yaml
# Seal secret
kubeseal --format=yaml \
--cert=pub-cert.pem \
< private/myapp-credentials.yaml \
> secrets/myapp-credentials-sealed.yaml
# Write secret to Vault
vault kv put kv/myapp/myapp-credentials KEY=value
# Create VaultStaticSecret CRD — see docs/vault-secrets-operator.md
```
### Repository Locations

174
docs/OPERATIONS-RUNBOOK.md
View File

@@ -188,13 +188,15 @@ Save the following file in private/ (gitignored) folder as secret.yaml
<paste your private key here>
project: default
```
Seal the secret using `kubeseal` command
Write the secret to Vault:
```bash
kubeseal --format=yaml \
--namespace=argocd \
< private/secret.yaml \
> secrets/forte-helm-repo-secret-sealed.yaml
vault kv put kv/argocd/forte-helm-repo \
type=git \
url=ssh://git@git.forteapps.net:2222/Forte/forte-helm.git \
sshPrivateKey="$(cat private/ssh-key)" \
project=default
```
Then create a VaultStaticSecret CRD with `argocd.argoproj.io/secret-type: repository` label.
**Step 4: Register Repository in ArgoCD**
@@ -499,7 +501,7 @@ See [Developer Guide](DEVELOPER-GUIDE.md#deploying-your-first-application) for d
**Quick checklist:**
- [ ] Create `helm-prod-values/myapp/values.yaml`
- [ ] Create `apps/myapp.yaml` in config repo
- [ ] Create SealedSecret if needed
- [ ] Write secrets to Vault and create VaultStaticSecret CRD if needed
- [ ] Commit and push changes
- [ ] Verify sync in Slack/ArgoCD
- [ ] Configure DNS for domain
@@ -670,92 +672,61 @@ db:
## Secret Management
Secrets are managed via **HashiCorp Vault** and synced to Kubernetes by the **Vault Secrets Operator (VSO)**. See [Vault Secrets Operator Reference](vault-secrets-operator.md) for full details.
### Creating Secrets
#### Step 1: Get Public Certificate
#### Step 1: Write to Vault
```bash
# Fetch sealed-secrets public cert (one-time)
kubeseal --fetch-cert \
--controller-name=sealed-secrets-controller \
--controller-namespace=kube-system \
> pub-cert.pem
# Save this certificate for future use
# From literal values
vault kv put kv/myapp/myapp-credentials \
API_KEY=secret123 \
DB_PASSWORD=pass456
```
#### Step 2: Create Plain Secret
#### Step 2: Create VaultStaticSecret CRD
```bash
# Method 1: From literal values
kubectl create secret generic myapp-credentials \
--from-literal=API_KEY=secret123 \
--from-literal=DB_PASSWORD=pass456 \
--namespace=myapp \
--dry-run=client -o yaml > private/myapp-credentials.yaml
# Method 2: From file
kubectl create secret generic myapp-credentials \
--from-file=.env \
--namespace=myapp \
--dry-run=client -o yaml > private/myapp-credentials.yaml
# Method 3: From multiple files
kubectl create secret generic myapp-credentials \
--from-file=api-key.txt \
--from-file=db-password.txt \
--namespace=myapp \
--dry-run=client -o yaml > private/myapp-credentials.yaml
```yaml
# apps/base/myapp/myapp-credentials-vault.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: myapp-credentials
namespace: myapp
spec:
type: kv-v2
mount: kv
path: myapp/myapp-credentials
destination:
name: myapp-credentials
create: true
refreshAfter: 30s
vaultAuthRef: vault-auth
```
#### Step 3: Seal Secret
#### Step 3: Commit CRD
```bash
kubeseal --format=yaml \
--cert=pub-cert.pem \
--namespace=myapp \
< private/myapp-credentials.yaml \
> secrets/myapp-credentials-sealed.yaml
```
#### Step 4: Commit Sealed Secret
```bash
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Add myapp credentials"
git add apps/base/myapp/myapp-credentials-vault.yaml
git commit -m "Add myapp credentials (VSO)"
git push
# Delete plain secret
rm private/myapp-credentials.yaml
```
### Updating Secrets
ArgoCD syncs the CRD, VSO creates the K8s Secret automatically.
### Updating / Rotating Secrets
**No git commit needed** — just update in Vault:
```bash
# 1. Create new version
kubectl create secret generic myapp-credentials \
--from-literal=API_KEY=new-secret-key \
--from-literal=DB_PASSWORD=new-password \
--namespace=myapp \
--dry-run=client -o yaml > private/myapp-credentials.yaml
vault kv put kv/myapp/myapp-credentials \
API_KEY=new-secret-key \
DB_PASSWORD=new-password
# 2. Seal it
kubeseal --format=yaml \
--cert=pub-cert.pem \
--namespace=myapp \
< private/myapp-credentials.yaml \
> secrets/myapp-credentials-sealed.yaml
# 3. Commit
git add secrets/myapp-credentials-sealed.yaml
git commit -m "Update myapp credentials"
git push
# 4. Restart pods to pick up new secret
# VSO picks up changes within 30 seconds
# Restart pods if needed
kubectl rollout restart deployment myapp -n myapp
# 5. Delete plain secret
rm private/myapp-credentials.yaml
```
### Viewing Secrets (Unsealed)
@@ -832,30 +803,13 @@ OIDC auth requires an `auth-oidc` Secret with two keys:
CLIENT_SECRET="your-oidc-client-secret-from-provider"
COOKIE_SECRET=$(openssl rand -hex 32)
# Create plain secret
kubectl create secret generic auth-oidc \
--from-literal=client-secret=$CLIENT_SECRET \
--from-literal=cookie-secret=$COOKIE_SECRET \
--namespace=myapp \
--dry-run=client -o yaml > private/myapp-auth-oidc.yaml
# Write to Vault
vault kv put kv/myapp/auth-oidc \
client-secret=$CLIENT_SECRET \
cookie-secret=$COOKIE_SECRET
# Seal it
kubeseal --format=yaml \
--cert=pub-cert.pem \
--namespace=myapp \
< private/myapp-auth-oidc.yaml \
> secrets/myapp-auth-oidc-sealed.yaml
# Apply sealed secret
kubectl apply -f secrets/myapp-auth-oidc-sealed.yaml
# Commit to Git
git add secrets/myapp-auth-oidc-sealed.yaml
git commit -m "Add OIDC secrets for myapp"
git push
# Clean up
rm private/myapp-auth-oidc.yaml
# Create VaultStaticSecret CRD (one-time) and commit
# See docs/vault-secrets-operator.md for CRD template
```
#### Rotating Authentication Secrets
@@ -882,16 +836,12 @@ kubectl rollout restart deployment myapp -n myapp
# Rotate cookie secret (safe - invalidates existing sessions)
NEW_COOKIE_SECRET=$(openssl rand -hex 32)
# Recreate secret
kubectl create secret generic auth-oidc \
--from-literal=client-secret=$CLIENT_SECRET \
--from-literal=cookie-secret=$NEW_COOKIE_SECRET \
--namespace=myapp \
--dry-run=client -o yaml | \
kubeseal --format=yaml --cert=pub-cert.pem --namespace=myapp | \
kubectl apply -f -
# Update in Vault — no git commit needed
vault kv put kv/myapp/auth-oidc \
client-secret=$CLIENT_SECRET \
cookie-secret=$NEW_COOKIE_SECRET
# Restart to pick up new secret
# VSO picks up within 30s. Restart pods to use new secret:
kubectl rollout restart deployment myapp -n myapp
```
@@ -1342,13 +1292,11 @@ kubectl get applications -n argocd -w
- pg_dump -U $DB_USER -d $DB_NAME > /backup/dump-$(date +%Y%m%d).sql
```
3. **Sealed Secrets private key backup**
3. **Vault backup**
```bash
# Backup sealed-secrets controller private key
kubectl get secret -n kube-system sealed-secrets-key \
-o yaml > sealed-secrets-key-backup.yaml
# Store in secure location (password manager, vault)
# Vault data is stored on PVC — ensure PVC snapshots are configured
# For disaster recovery, maintain Vault unseal keys in a secure location
# All secrets can be re-seeded from source if needed
```
---
@@ -1668,7 +1616,7 @@ echo "Remember to delete: $SECRET_FILE"
- [ ] Gitea Actions workflow configured
- [ ] Helm values created in `helm-prod-values/`
- [ ] ArgoCD application manifest created in `apps/`
- [ ] Secrets created and sealed
- [ ] Secrets written to Vault and VaultStaticSecret CRD created
- [ ] DNS record added for domain
- [ ] Application synced successfully
- [ ] Health check passed

206
docs/vault-secrets-operator.md Normal file
View File

@@ -0,0 +1,206 @@
# Vault Secrets Operator (VSO) Reference
## Overview
The platform uses HashiCorp Vault Secrets Operator (VSO) to sync secrets from Vault KV v2 to native Kubernetes Secrets. This replaces the previous SealedSecrets workflow.
**Key benefit**: Secret values can be rotated via Vault UI/CLI without a git commit. Only new VaultStaticSecret CRDs need to be committed.
## Architecture
```
Vault (KV v2) VSO K8s Secret
kv/{namespace}/{name} --> VaultStaticSecret CRD --> Secret in namespace
(polls every 30s)
```
- **Vault**: Standalone instance in `vault` namespace, KV v2 at `kv/`
- **VSO**: Deployed in `vault-secrets-operator-system` namespace via ArgoCD
- **Auth**: Kubernetes auth method — each namespace has its own ServiceAccount + VaultAuth CRD
## KV Path Convention
```
kv/{namespace}/{secret-name}
```
Examples:
- `kv/homepage/homepage-widget-credentials`
- `kv/argocd/forte-helm-repo`
- `kv/gitea/gitea-smtp-secret`
- `kv/keycloak/keycloak-credentials`
## Vault Policy Structure
Each namespace gets a read-only policy:
```hcl
# Policy: ns-{namespace}
path "kv/data/{namespace}/*" {
capabilities = ["read"]
}
path "kv/metadata/{namespace}/*" {
capabilities = ["read", "list"]
}
```
## Kubernetes Auth Roles
Each namespace has a bound ServiceAccount:
```
Role: ns-{namespace}
bound_service_account_names: vault-auth-{namespace}
bound_service_account_namespaces: {namespace}
policies: ns-{namespace}
audience: vault
ttl: 1h
```
## CRD Reference
### VaultAuth
Per-namespace auth binding. One per namespace.
```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: vault-auth
namespace: {namespace}
spec:
method: kubernetes
mount: kubernetes
kubernetes:
role: ns-{namespace}
serviceAccount: vault-auth-{namespace}
audiences:
- vault
```
Each VaultAuth requires a corresponding ServiceAccount:
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-auth-{namespace}
namespace: {namespace}
```
### VaultStaticSecret
One per secret. Syncs a Vault KV path to a K8s Secret.
```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: {secret-name}
namespace: {namespace}
spec:
type: kv-v2
mount: kv
path: {namespace}/{secret-name}
destination:
name: {secret-name} # K8s Secret name (must match what apps expect)
create: true
type: Opaque # Optional, defaults to Opaque
labels: # Optional, for secrets that need labels
some-label: "value"
refreshAfter: 30s
vaultAuthRef: vault-auth
```
## Special Labels
Some secrets require specific labels for correct operation:
| Secret | Label | Purpose |
|--------|-------|---------|
| `renovate-env` | `allowedToBeCloned: "true"` | Kyverno secret-cloner policy |
| `gitea-smtp-secret` | `allowedToBeCloned: "true"` | Kyverno secret-cloner policy |
| `forte-helm-repo` | `argocd.argoproj.io/secret-type: repository` | ArgoCD repository recognition |
| `forte10x-repo-creds` | `argocd.argoproj.io/secret-type: repository` | ArgoCD repository recognition |
| `mcp10x-repo-creds` | `argocd.argoproj.io/secret-type: repository` | ArgoCD repository recognition |
These are set in `destination.labels` of the VaultStaticSecret CRD.
## Namespaces & Secrets Map
| Namespace | Secrets |
|-----------|---------|
| `homepage` | homepage-widget-credentials |
| `renovate` | renovate-env |
| `gitea` | gitea-credentials, gitea-backup-s3, gitea-smtp-secret, gitea-runner-token |
| `keycloak` | keycloak-credentials, microsoft-idp-credentials (overlay) |
| `argocd` | forte-helm-repo, forte10x-repo-creds, mcp10x-repo-creds, argocd-notifications-secret |
| `mcp10x` | app-credentials |
| `ts-mcp` | ts-mcp-secrets |
| `argocd-mcp` | auth-oidc, argocd-mcp-credentials |
| `dot-ai` | dot-ai-secrets |
| `music-man` | musicman-credentials |
## Common Operations
### Add a new secret
1. Write to Vault:
```bash
vault kv put kv/{namespace}/{secret-name} key1=val1 key2=val2
```
2. Create VaultStaticSecret YAML (see template above)
3. Add to kustomization.yaml in the appropriate directory
4. Commit and push — ArgoCD syncs the CRD, VSO creates the K8s Secret
### Rotate a secret value
No git commit needed:
```bash
vault kv put kv/{namespace}/{secret-name} key1=new-val1 key2=new-val2
```
VSO picks up changes within 30 seconds.
### Check sync status
```bash
# VaultAuth status
kubectl get vaultauth -n {namespace}
# VaultStaticSecret status
kubectl get vaultstaticsecret -n {namespace}
# Verify K8s Secret exists with correct keys
kubectl get secret {name} -n {namespace} -o jsonpath='{.data}' | jq
```
### Troubleshooting
1. **VaultAuth not authenticating**: Check ServiceAccount exists, Vault role matches SA name/namespace
2. **VaultStaticSecret not syncing**: Check `kubectl describe vaultstaticsecret {name} -n {ns}` for events
3. **Secret missing keys**: Verify Vault KV path has all expected keys: `vault kv get kv/{ns}/{name}`
4. **Permission denied**: Verify Vault policy allows read on `kv/data/{ns}/*`
## File Locations
| Type | Location |
|------|----------|
| VSO ArgoCD Application | `infra/base/vault-secrets-operator/` |
| VSO Helm values | `infra/values/base/vault-secrets-operator-values.yaml` |
| Vault policies script | `scripts/vault-setup-policies.sh` |
| Seed script | `scripts/seed-vault-from-cluster.sh` |
| VaultAuth + VaultStaticSecret | Alongside ArgoCD Application in each component directory |
## Setup Scripts
```bash
# Create all Vault policies and auth roles
./scripts/vault-setup-policies.sh
# Seed Vault KV from existing K8s Secrets
./scripts/seed-vault-from-cluster.sh
```