Secrets Management

Vault, KMS, sealed secrets — keeping passwords and API keys out of Git and out of trouble.

The Secret Sprawl Problem

Almost every backend has secrets:
• Database passwords
• API keys for third parties
• Encryption keys
• Service-to-service auth tokens
• OAuth client secrets
• TLS private keys

Without discipline, these end up in:
• Source code (committed to Git, leaked forever)
• .env files (one engineer's laptop is a single point of compromise)
• Slack messages, emails, tickets
• Hardcoded in CI configs
• Hardcoded in Dockerfiles

A 2026 study found about 1 in 6 GitHub repositories contained a leaked secret somewhere in their history.

The cost is real:
• Stolen credentials → breaches
• Compromised CI → supply chain attacks
• Key rotation impossible (you don't know everywhere it's used)
• Audit failures (compliance frameworks all require secret management)

Proper secrets management isn't optional. It's foundational.

The Three-Layer Architecture

A complete secrets strategy has three layers:

   ┌──────────────────────────────────────────┐
   │  1. STORAGE (the vault)                   │
   │  - HashiCorp Vault                        │
   │  - AWS Secrets Manager                    │
   │  - GCP Secret Manager                     │
   │  - Azure Key Vault                        │
   │  Secrets live here. Encrypted at rest.    │
   └──────────────────────────────────────────┘
                       │
   ┌───────────────────┴────────────────────┐
   │  2. DELIVERY (how secrets reach apps)    │
   │  - Environment variables (at start)      │
   │  - Files on disk (mounted)               │
   │  - SDK calls (fetched at runtime)        │
   │  - Sidecar injectors (Vault Agent)       │
   └──────────────────────────────────────────┘
                       │
   ┌───────────────────┴────────────────────┐
   │  3. ROTATION (changing secrets)          │
   │  - Time-based (every 90 days)            │
   │  - Event-based (employee leaves)         │
   │  - Automated where possible              │
   └──────────────────────────────────────────┘

Get all three layers right and you're 95% of the way there. Many teams do storage but skip rotation; that's a bandaid.

Cloud-Native Secret Stores

Each cloud has its own:

AWS Secrets Manager:

aws secretsmanager create-secret \
  --name production/myapp/db \
  --secret-string '{"username":"app","password":"super-secret"}'

aws secretsmanager get-secret-value --secret-id production/myapp/db

Built-in features:
• Automatic rotation (RDS passwords, Redshift, etc.)
• IAM-based access control
• Audit logs (CloudTrail)
• Cross-region replication

Cost: $0.40/secret/month + $0.05/10k API calls.

Cheaper alternative: AWS Systems Manager Parameter Store. Free for standard parameters. Less feature-rich but adequate for most config:

aws ssm put-parameter --name /myapp/api-key --value "secret" --type SecureString

GCP Secret Manager:

echo -n "secret-value" | gcloud secrets create my-secret --data-file=-
gcloud secrets versions access latest --secret=my-secret

Pattern: integrate IAM with the secret store. Your EC2 instance / Lambda / Cloud Run service has a role granting read access to specific secrets. No keys in code; the cloud's IAM provides identity.

HashiCorp Vault

Vault is the most powerful secret manager — and the most complex. Worth using when:
• You're multi-cloud
• You need dynamic secrets (Vault generates short-lived DB credentials per request)
• You need a unified secret platform across many tools
• Your scale or compliance requirements justify the operational cost

Core concepts:

Secret engines — different types of secrets
• kv — static key-value secrets (passwords, API keys)
• database — dynamic database credentials, generated per request
• pki — issue TLS certificates
• aws — generate short-lived AWS credentials
• transit — encryption-as-a-service

Auth methods — how clients authenticate to Vault
• Kubernetes service account
• AWS IAM
• OIDC
• AppRole (machine-to-machine)

Dynamic database credentials are powerful:

1. App requests DB creds from Vault
2. Vault creates a NEW user in the DB with a random password
3. Vault returns those creds with a 1-hour lease
4. App uses them
5. After 1 hour, Vault deletes the user

Compromised creds become useless within an hour. No long-lived DB passwords anywhere.

Vault's downside: it's a critical piece of infrastructure to operate. HA, backup, unsealing, audit logging. Many small teams adopt Vault, struggle to operate it, and fall back to cloud-native. That's fine — the cloud-native solutions cover 80% of cases.

Kubernetes Secrets Strategy

K8s has a built-in Secret object, but it's just base64-encoded — not encrypted. Three patterns for real secret management in K8s:

1. External Secrets Operator (ESO) — pulls from external store

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: ClusterSecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: production/myapp/db
        property: password

ESO syncs from AWS Secrets Manager / Vault / GCP Secret Manager into K8s Secrets. Apps see normal Secrets; the source of truth is external.

2. Sealed Secrets — encrypted secrets in Git
A Bitnami project. You encrypt secrets with the cluster's public key; the encrypted blobs are safe to commit. The Sealed Secrets controller in the cluster decrypts them.

3. SOPS — encrypted YAML files
Mozilla SOPS encrypts only VALUES in YAML, leaving keys readable. Used heavily in Helmfile and ArgoCD setups.

For most teams using K8s in 2026: External Secrets Operator + cloud-native secret store. It's the right balance of power and operational simplicity.

Rotation — The Hardest Part

Storing secrets is the easy part. Changing them safely is hard.

Why rotation matters:
• Limits damage from leaks (compromised key only useful for 90 days)
• Compliance requires it (PCI-DSS, SOC 2)
• Forces you to TEST that rotation works

Two approaches:

1. Atomic swap — rotate at a boundary

1. Generate new credential
2. Update secret store
3. Restart all consumers (they pick up new value)
4. Invalidate old credential

Causes a brief disruption. Fine for low-criticality systems.

2. Dual credential window — both old and new work for a while

1. Generate new credential, store as v2
2. Add v2 to the active set; v1 still works
3. Roll consumers to use v2
4. After all confirmed on v2, invalidate v1

Zero downtime. More complex.

Test your rotation. If you've never actually rotated production credentials, your rotation is a fantasy. Plan a rotation drill quarterly.

The next lesson covers cloud security posture — taking secrets management as the foundation and building broader cloud security on top.