launchpad/docs/vault-secrets-operator.md

Vault Secrets Operator (VSO) Reference

Overview

The platform uses HashiCorp Vault Secrets Operator (VSO) to sync secrets from Vault KV v2 to native Kubernetes Secrets. This replaces the previous SealedSecrets workflow.

Key benefit: Secret values can be rotated via Vault UI/CLI without a git commit. Only new VaultStaticSecret CRDs need to be committed.

Architecture

Vault (KV v2)                VSO                        K8s Secret
kv/{namespace}/{name}  -->  VaultStaticSecret CRD  -->  Secret in namespace
                            (polls every 30s)

• Vault: Standalone instance in vault namespace, KV v2 at kv/
• VSO: Deployed in vault-secrets-operator-system namespace via ArgoCD
• Auth: Kubernetes auth method — each namespace has its own ServiceAccount + VaultAuth CRD

KV Path Convention

kv/{namespace}/{secret-name}

Examples:

• kv/homepage/homepage-widget-credentials
• kv/argocd/forte-helm-repo
• kv/gitea/gitea-smtp-secret
• kv/keycloak/keycloak-credentials

Vault Policy Structure

Each namespace gets a read-only policy:

# Policy: ns-{namespace}
path "kv/data/{namespace}/*" {
  capabilities = ["read"]
}
path "kv/metadata/{namespace}/*" {
  capabilities = ["read", "list"]
}

Kubernetes Auth Roles

Each namespace has a bound ServiceAccount:

Role: ns-{namespace}
  bound_service_account_names: vault-auth-{namespace}
  bound_service_account_namespaces: {namespace}
  policies: ns-{namespace}
  audience: vault
  ttl: 1h

CRD Reference

VaultAuth

Per-namespace auth binding. One per namespace.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
  namespace: {namespace}
spec:
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: ns-{namespace}
    serviceAccount: vault-auth-{namespace}
    audiences:
    - vault

Each VaultAuth requires a corresponding ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth-{namespace}
  namespace: {namespace}

VaultStaticSecret

One per secret. Syncs a Vault KV path to a K8s Secret.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: {secret-name}
  namespace: {namespace}
spec:
  type: kv-v2
  mount: kv
  path: {namespace}/{secret-name}
  destination:
    name: {secret-name}        # K8s Secret name (must match what apps expect)
    create: true
    type: Opaque               # Optional, defaults to Opaque
    labels:                    # Optional, for secrets that need labels
      some-label: "value"
  refreshAfter: 30s
  vaultAuthRef: vault-auth

Special Labels

Some secrets require specific labels for correct operation:

Secret	Label	Purpose
renovate-env	allowedToBeCloned: "true"	Kyverno secret-cloner policy
gitea-smtp-secret	allowedToBeCloned: "true"	Kyverno secret-cloner policy
forte-helm-repo	argocd.argoproj.io/secret-type: repository	ArgoCD repository recognition
forte10x-repo-creds	argocd.argoproj.io/secret-type: repository	ArgoCD repository recognition
mcp10x-repo-creds	argocd.argoproj.io/secret-type: repository	ArgoCD repository recognition

These are set in destination.labels of the VaultStaticSecret CRD.

Namespaces & Secrets Map

Namespace	Secrets
homepage	homepage-widget-credentials
renovate	renovate-env
gitea	gitea-credentials, gitea-backup-s3, gitea-smtp-secret, gitea-runner-token
keycloak	keycloak-credentials, microsoft-idp-credentials (overlay)
argocd	forte-helm-repo, forte10x-repo-creds, mcp10x-repo-creds, argocd-notifications-secret
mcp10x	app-credentials
ts-mcp	ts-mcp-secrets
argocd-mcp	auth-oidc, argocd-mcp-credentials
dot-ai	dot-ai-secrets
music-man	musicman-credentials

Common Operations

Add a new secret

1. Write to Vault:

   vault kv put kv/{namespace}/{secret-name} key1=val1 key2=val2
   

2. Create VaultStaticSecret YAML (see template above)

3. Add to kustomization.yaml in the appropriate directory

4. Commit and push — ArgoCD syncs the CRD, VSO creates the K8s Secret

Rotate a secret value

No git commit needed:

vault kv put kv/{namespace}/{secret-name} key1=new-val1 key2=new-val2

VSO picks up changes within 30 seconds.

Check sync status

# VaultAuth status
kubectl get vaultauth -n {namespace}

# VaultStaticSecret status
kubectl get vaultstaticsecret -n {namespace}

# Verify K8s Secret exists with correct keys
kubectl get secret {name} -n {namespace} -o jsonpath='{.data}' | jq

Troubleshooting

1. VaultAuth not authenticating: Check ServiceAccount exists, Vault role matches SA name/namespace
2. VaultStaticSecret not syncing: Check kubectl describe vaultstaticsecret {name} -n {ns} for events
3. Secret missing keys: Verify Vault KV path has all expected keys: vault kv get kv/{ns}/{name}
4. Permission denied: Verify Vault policy allows read on kv/data/{ns}/*

File Locations

Type	Location
VSO ArgoCD Application	infra/base/vault-secrets-operator/
VSO Helm values	infra/values/base/vault-secrets-operator-values.yaml
Vault policies script	scripts/vault-setup-policies.sh
Seed script	scripts/seed-vault-from-cluster.sh
VaultAuth + VaultStaticSecret	Alongside ArgoCD Application in each component directory

Setup Scripts

# Create all Vault policies and auth roles
./scripts/vault-setup-policies.sh

# Seed Vault KV from existing K8s Secrets
./scripts/seed-vault-from-cluster.sh