launchpad/docs/vault-secrets-operator.md

# Vault Secrets Operator (VSO) Reference

## Overview

The platform uses HashiCorp Vault Secrets Operator (VSO) to sync secrets from Vault KV v2 to native Kubernetes Secrets. This replaces the previous SealedSecrets workflow.

**Key benefit**: Secret values can be rotated via Vault UI/CLI without a git commit. Only new VaultStaticSecret CRDs need to be committed.

## Architecture

```
Vault (KV v2)                VSO                        K8s Secret
kv/{namespace}/{name}  -->  VaultStaticSecret CRD  -->  Secret in namespace
                            (polls every 30s)
```

- **Vault**: Standalone instance in `vault` namespace, KV v2 at `kv/`
- **VSO**: Deployed in `vault-secrets-operator-system` namespace via ArgoCD
- **Auth**: Kubernetes auth method — each namespace has its own ServiceAccount + VaultAuth CRD

## KV Path Convention

```
kv/{namespace}/{secret-name}
```

Examples:
- `kv/homepage/homepage-widget-credentials`
- `kv/argocd/forte-helm-repo`
- `kv/gitea/gitea-smtp-secret`
- `kv/keycloak/keycloak-credentials`

## Vault Policy Structure

Each namespace gets a read-only policy:

```hcl
# Policy: ns-{namespace}
path "kv/data/{namespace}/*" {
  capabilities = ["read"]
}
path "kv/metadata/{namespace}/*" {
  capabilities = ["read", "list"]
}
```

## Kubernetes Auth Roles

Each namespace has a bound ServiceAccount:

```
Role: ns-{namespace}
  bound_service_account_names: vault-auth-{namespace}
  bound_service_account_namespaces: {namespace}
  policies: ns-{namespace}
  audience: vault
  ttl: 1h
```

## CRD Reference

### VaultAuth

Per-namespace auth binding. One per namespace.

```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
  namespace: {namespace}
spec:
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: ns-{namespace}
    serviceAccount: vault-auth-{namespace}
    audiences:
    - vault
```

Each VaultAuth requires a corresponding ServiceAccount:

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth-{namespace}
  namespace: {namespace}
```

### VaultStaticSecret

One per secret. Syncs a Vault KV path to a K8s Secret.

```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: {secret-name}
  namespace: {namespace}
spec:
  type: kv-v2
  mount: kv
  path: {namespace}/{secret-name}
  destination:
    name: {secret-name}        # K8s Secret name (must match what apps expect)
    create: true
    type: Opaque               # Optional, defaults to Opaque
    labels:                    # Optional, for secrets that need labels
      some-label: "value"
  refreshAfter: 30s
  vaultAuthRef: vault-auth
```

## Special Labels

Some secrets require specific labels for correct operation:

| Secret | Label | Purpose |
|--------|-------|---------|
| `renovate-env` | `allowedToBeCloned: "true"` | Kyverno secret-cloner policy |
| `gitea-smtp-secret` | `allowedToBeCloned: "true"` | Kyverno secret-cloner policy |
| `forte-helm-repo` | `argocd.argoproj.io/secret-type: repository` | ArgoCD repository recognition |
| `forte10x-repo-creds` | `argocd.argoproj.io/secret-type: repository` | ArgoCD repository recognition |
| `mcp10x-repo-creds` | `argocd.argoproj.io/secret-type: repository` | ArgoCD repository recognition |

These are set in `destination.labels` of the VaultStaticSecret CRD.

## Namespaces & Secrets Map

| Namespace | Secrets |
|-----------|---------|
| `homepage` | homepage-widget-credentials |
| `renovate` | renovate-env |
| `gitea` | gitea-credentials, gitea-backup-s3, gitea-smtp-secret, gitea-runner-token |
| `keycloak` | keycloak-credentials, microsoft-idp-credentials (overlay) |
| `argocd` | forte-helm-repo, forte10x-repo-creds, mcp10x-repo-creds, argocd-notifications-secret |
| `mcp10x` | app-credentials |
| `ts-mcp` | ts-mcp-secrets |
| `argocd-mcp` | auth-oidc, argocd-mcp-credentials |
| `dot-ai` | dot-ai-secrets |
| `music-man` | musicman-credentials |

## Common Operations

### Add a new secret

1. Write to Vault:
   ```bash
   vault kv put kv/{namespace}/{secret-name} key1=val1 key2=val2
   ```

2. Create VaultStaticSecret YAML (see template above)

3. Add to kustomization.yaml in the appropriate directory

4. Commit and push — ArgoCD syncs the CRD, VSO creates the K8s Secret

### Rotate a secret value

No git commit needed:
```bash
vault kv put kv/{namespace}/{secret-name} key1=new-val1 key2=new-val2
```
VSO picks up changes within 30 seconds.

### Check sync status

```bash
# VaultAuth status
kubectl get vaultauth -n {namespace}

# VaultStaticSecret status
kubectl get vaultstaticsecret -n {namespace}

# Verify K8s Secret exists with correct keys
kubectl get secret {name} -n {namespace} -o jsonpath='{.data}' | jq
```

### Troubleshooting

1. **VaultAuth not authenticating**: Check ServiceAccount exists, Vault role matches SA name/namespace
2. **VaultStaticSecret not syncing**: Check `kubectl describe vaultstaticsecret {name} -n {ns}` for events
3. **Secret missing keys**: Verify Vault KV path has all expected keys: `vault kv get kv/{ns}/{name}`
4. **Permission denied**: Verify Vault policy allows read on `kv/data/{ns}/*`

## File Locations

| Type | Location |
|------|----------|
| VSO ArgoCD Application | `infra/base/vault-secrets-operator/` |
| VSO Helm values | `infra/values/base/vault-secrets-operator-values.yaml` |
| Vault policies script | `scripts/vault-setup-policies.sh` |
| Seed script | `scripts/seed-vault-from-cluster.sh` |
| VaultAuth + VaultStaticSecret | Alongside ArgoCD Application in each component directory |

## Setup Scripts

```bash
# Create all Vault policies and auth roles
./scripts/vault-setup-policies.sh

# Seed Vault KV from existing K8s Secrets
./scripts/seed-vault-from-cluster.sh
```