vault migration
This commit is contained in:
206
docs/vault-secrets-operator.md
Normal file
206
docs/vault-secrets-operator.md
Normal file
@@ -0,0 +1,206 @@
|
||||
# Vault Secrets Operator (VSO) Reference
|
||||
|
||||
## Overview
|
||||
|
||||
The platform uses HashiCorp Vault Secrets Operator (VSO) to sync secrets from Vault KV v2 to native Kubernetes Secrets. This replaces the previous SealedSecrets workflow.
|
||||
|
||||
**Key benefit**: Secret values can be rotated via Vault UI/CLI without a git commit. Only new VaultStaticSecret CRDs need to be committed.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Vault (KV v2) VSO K8s Secret
|
||||
kv/{namespace}/{name} --> VaultStaticSecret CRD --> Secret in namespace
|
||||
(polls every 30s)
|
||||
```
|
||||
|
||||
- **Vault**: Standalone instance in `vault` namespace, KV v2 at `kv/`
|
||||
- **VSO**: Deployed in `vault-secrets-operator-system` namespace via ArgoCD
|
||||
- **Auth**: Kubernetes auth method — each namespace has its own ServiceAccount + VaultAuth CRD
|
||||
|
||||
## KV Path Convention
|
||||
|
||||
```
|
||||
kv/{namespace}/{secret-name}
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `kv/homepage/homepage-widget-credentials`
|
||||
- `kv/argocd/forte-helm-repo`
|
||||
- `kv/gitea/gitea-smtp-secret`
|
||||
- `kv/keycloak/keycloak-credentials`
|
||||
|
||||
## Vault Policy Structure
|
||||
|
||||
Each namespace gets a read-only policy:
|
||||
|
||||
```hcl
|
||||
# Policy: ns-{namespace}
|
||||
path "kv/data/{namespace}/*" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
path "kv/metadata/{namespace}/*" {
|
||||
capabilities = ["read", "list"]
|
||||
}
|
||||
```
|
||||
|
||||
## Kubernetes Auth Roles
|
||||
|
||||
Each namespace has a bound ServiceAccount:
|
||||
|
||||
```
|
||||
Role: ns-{namespace}
|
||||
bound_service_account_names: vault-auth-{namespace}
|
||||
bound_service_account_namespaces: {namespace}
|
||||
policies: ns-{namespace}
|
||||
audience: vault
|
||||
ttl: 1h
|
||||
```
|
||||
|
||||
## CRD Reference
|
||||
|
||||
### VaultAuth
|
||||
|
||||
Per-namespace auth binding. One per namespace.
|
||||
|
||||
```yaml
|
||||
apiVersion: secrets.hashicorp.com/v1beta1
|
||||
kind: VaultAuth
|
||||
metadata:
|
||||
name: vault-auth
|
||||
namespace: {namespace}
|
||||
spec:
|
||||
method: kubernetes
|
||||
mount: kubernetes
|
||||
kubernetes:
|
||||
role: ns-{namespace}
|
||||
serviceAccount: vault-auth-{namespace}
|
||||
audiences:
|
||||
- vault
|
||||
```
|
||||
|
||||
Each VaultAuth requires a corresponding ServiceAccount:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: vault-auth-{namespace}
|
||||
namespace: {namespace}
|
||||
```
|
||||
|
||||
### VaultStaticSecret
|
||||
|
||||
One per secret. Syncs a Vault KV path to a K8s Secret.
|
||||
|
||||
```yaml
|
||||
apiVersion: secrets.hashicorp.com/v1beta1
|
||||
kind: VaultStaticSecret
|
||||
metadata:
|
||||
name: {secret-name}
|
||||
namespace: {namespace}
|
||||
spec:
|
||||
type: kv-v2
|
||||
mount: kv
|
||||
path: {namespace}/{secret-name}
|
||||
destination:
|
||||
name: {secret-name} # K8s Secret name (must match what apps expect)
|
||||
create: true
|
||||
type: Opaque # Optional, defaults to Opaque
|
||||
labels: # Optional, for secrets that need labels
|
||||
some-label: "value"
|
||||
refreshAfter: 30s
|
||||
vaultAuthRef: vault-auth
|
||||
```
|
||||
|
||||
## Special Labels
|
||||
|
||||
Some secrets require specific labels for correct operation:
|
||||
|
||||
| Secret | Label | Purpose |
|
||||
|--------|-------|---------|
|
||||
| `renovate-env` | `allowedToBeCloned: "true"` | Kyverno secret-cloner policy |
|
||||
| `gitea-smtp-secret` | `allowedToBeCloned: "true"` | Kyverno secret-cloner policy |
|
||||
| `forte-helm-repo` | `argocd.argoproj.io/secret-type: repository` | ArgoCD repository recognition |
|
||||
| `forte10x-repo-creds` | `argocd.argoproj.io/secret-type: repository` | ArgoCD repository recognition |
|
||||
| `mcp10x-repo-creds` | `argocd.argoproj.io/secret-type: repository` | ArgoCD repository recognition |
|
||||
|
||||
These are set in `destination.labels` of the VaultStaticSecret CRD.
|
||||
|
||||
## Namespaces & Secrets Map
|
||||
|
||||
| Namespace | Secrets |
|
||||
|-----------|---------|
|
||||
| `homepage` | homepage-widget-credentials |
|
||||
| `renovate` | renovate-env |
|
||||
| `gitea` | gitea-credentials, gitea-backup-s3, gitea-smtp-secret, gitea-runner-token |
|
||||
| `keycloak` | keycloak-credentials, microsoft-idp-credentials (overlay) |
|
||||
| `argocd` | forte-helm-repo, forte10x-repo-creds, mcp10x-repo-creds, argocd-notifications-secret |
|
||||
| `mcp10x` | app-credentials |
|
||||
| `ts-mcp` | ts-mcp-secrets |
|
||||
| `argocd-mcp` | auth-oidc, argocd-mcp-credentials |
|
||||
| `dot-ai` | dot-ai-secrets |
|
||||
| `music-man` | musicman-credentials |
|
||||
|
||||
## Common Operations
|
||||
|
||||
### Add a new secret
|
||||
|
||||
1. Write to Vault:
|
||||
```bash
|
||||
vault kv put kv/{namespace}/{secret-name} key1=val1 key2=val2
|
||||
```
|
||||
|
||||
2. Create VaultStaticSecret YAML (see template above)
|
||||
|
||||
3. Add to kustomization.yaml in the appropriate directory
|
||||
|
||||
4. Commit and push — ArgoCD syncs the CRD, VSO creates the K8s Secret
|
||||
|
||||
### Rotate a secret value
|
||||
|
||||
No git commit needed:
|
||||
```bash
|
||||
vault kv put kv/{namespace}/{secret-name} key1=new-val1 key2=new-val2
|
||||
```
|
||||
VSO picks up changes within 30 seconds.
|
||||
|
||||
### Check sync status
|
||||
|
||||
```bash
|
||||
# VaultAuth status
|
||||
kubectl get vaultauth -n {namespace}
|
||||
|
||||
# VaultStaticSecret status
|
||||
kubectl get vaultstaticsecret -n {namespace}
|
||||
|
||||
# Verify K8s Secret exists with correct keys
|
||||
kubectl get secret {name} -n {namespace} -o jsonpath='{.data}' | jq
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
1. **VaultAuth not authenticating**: Check ServiceAccount exists, Vault role matches SA name/namespace
|
||||
2. **VaultStaticSecret not syncing**: Check `kubectl describe vaultstaticsecret {name} -n {ns}` for events
|
||||
3. **Secret missing keys**: Verify Vault KV path has all expected keys: `vault kv get kv/{ns}/{name}`
|
||||
4. **Permission denied**: Verify Vault policy allows read on `kv/data/{ns}/*`
|
||||
|
||||
## File Locations
|
||||
|
||||
| Type | Location |
|
||||
|------|----------|
|
||||
| VSO ArgoCD Application | `infra/base/vault-secrets-operator/` |
|
||||
| VSO Helm values | `infra/values/base/vault-secrets-operator-values.yaml` |
|
||||
| Vault policies script | `scripts/vault-setup-policies.sh` |
|
||||
| Seed script | `scripts/seed-vault-from-cluster.sh` |
|
||||
| VaultAuth + VaultStaticSecret | Alongside ArgoCD Application in each component directory |
|
||||
|
||||
## Setup Scripts
|
||||
|
||||
```bash
|
||||
# Create all Vault policies and auth roles
|
||||
./scripts/vault-setup-policies.sh
|
||||
|
||||
# Seed Vault KV from existing K8s Secrets
|
||||
./scripts/seed-vault-from-cluster.sh
|
||||
```
|
||||
Reference in New Issue
Block a user