Guide

Secrets management explained

A secret is any credential that grants access if leaked: database passwords, third-party API keys, TLS private keys, OAuth client secrets, and blockchain signing keys. Config values like feature flags or public RPC URLs are not secrets — they can live in source control. Secrets cannot. Every major breach story eventually traces to a credential stored in git, baked into a Docker image, or printed in a log line. Secrets management is the discipline of storing, injecting, rotating, and auditing those credentials so developers can ship without becoming the weak link. This guide covers what counts as a secret, storage options from env vars to vaults, runtime injection patterns, rotation, least privilege, CI/CD hygiene, and how secrets relate to JWT signing keys and OAuth client credentials.

Secrets vs configuration: know the difference

Configuration describes how your app behaves. Secrets prove who your app is when it talks to other systems. Mixing them causes two failure modes: developers commit secrets because they look like config, or teams over-engineer vault infrastructure for values that could be public.

Secrets — database passwords, Stripe secret keys, HMAC signing salts, SSH private keys, encryption master keys, refresh-token pepper values, hot-wallet private keys for server-side signing.
Config (non-secret) — service URLs, log levels, pool sizes, public API endpoints, feature-flag names, AdSense publisher IDs visible in HTML.
Borderline — internal service hostnames in a zero-trust mesh may be config; the mTLS client certificate is a secret. When in doubt, treat it as a secret until threat modeling says otherwise.

The rule of thumb: if an attacker reading the value could impersonate your service or read customer data, it is a secret.

Common anti-patterns (and why they keep happening)

Most credential leaks are boring, not sophisticated:

Committed .env files — git history is forever; even after deletion, bots scan public repos within minutes.
Hardcoded fallbacks — process.env.KEY || 'dev-key-123' ships to production when env injection fails silently.
Logging the secret — debug lines that dump request headers, connection strings, or JWT payloads.
Shared long-lived keys — one AWS root-style key used by five microservices; rotation becomes impossible.
Secrets in client bundles — embedding private API keys in React builds where anyone can read network traffic or source maps.
Images with baked-in creds — Docker layers retain env values; registry access becomes a secret store.

Prevention is cultural and technical: pre-commit secret scanners, code review checklists, and architecture that makes the right path easier than the shortcut.

Storage options: from env vars to dedicated vaults

Choose storage by blast radius, team size, and compliance requirements:

Environment variables (small teams, single host)

The platform injects secrets at process start — systemd unit files, Docker Compose secrets, Kubernetes Secret objects mounted as env vars. Works well for a handful of credentials on one or two servers. Weaknesses: no audit trail, rotation requires redeploy, easy to dump via /proc or crash reports, and developers copy values into Slack.

Managed secret managers (production default)

Cloud-native services — AWS Secrets Manager, GCP Secret Manager, Azure Key Vault — store encrypted blobs with IAM-scoped access, automatic rotation hooks for RDS passwords, and CloudTrail-style audit logs. HashiCorp Vault adds dynamic credentials: your app requests a Postgres user valid for one hour, Vault creates it, returns the password, and revokes on lease expiry. That pattern pairs well with connection pooling because short-lived creds limit exposure if a pool worker is compromised.

Hardware security modules (HSM) and cloud KMS

For high-value signing keys — payment HSMs, certificate authorities, blockchain treasury keys — private key material never leaves tamper-resistant hardware. AWS KMS, Google Cloud KMS, and Azure Managed HSM expose sign/decrypt APIs; your application holds a reference ID, not the raw key bytes. Latency is higher but theft requires compromising the HSM policy layer, not reading a file.

Runtime injection: fetch late, cache carefully

The secure pattern is pull at startup, never at build time:

Container or VM boots with an identity (IAM role, Kubernetes service account).
Application calls the secret manager API using that identity — no static bootstrap password in the image.
Secrets load into memory once; avoid writing them to disk unless the library requires a temp file (and delete immediately).
On rotation signal or TTL expiry, refresh in memory and reconnect pools gracefully.

For local development, use a .env.local file gitignored by default, or tools like direnv / 1Password CLI that inject without persisting in the repo. Never share production secrets into developer laptops — use scoped staging credentials instead.

Serverless functions complicate this: cold starts must fetch secrets quickly. Cache in the execution environment between invocations, but respect rotation by checking a version field or TTL. See serverless architecture for cold-start trade-offs.

Rotation, expiry, and break-glass access

A secret that never rotates is a time bomb. Rotation policy depends on sensitivity:

Database passwords — 30–90 days, or dynamic per-session via Vault.
Third-party API keys — vendor-dependent; prefer scoped sub-keys with independent expiry.
TLS certificates — automate with Let's Encrypt or ACME; monitor 30-day expiry alerts.
JWT signing keys — support key IDs (kid) in headers so you can rotate without invalidating every outstanding token instantly; publish a JWKS endpoint with overlapping valid keys during transition.
OAuth client secrets — rotate on personnel changes; use PKCE for public clients so no long-lived secret ships to mobile apps.

Run rotation drills: can you swap the DB password at 2 p.m. without a outage? Document break-glass procedures — emergency admin credentials stored offline, used only with dual control, and audited after every access.

Least privilege and scoped credentials

Every secret should do the minimum possible:

Database users — app role with SELECT/INSERT on one schema, not SUPERUSER.
Cloud IAM — policy scoped to one S3 prefix or one Secrets Manager ARN, not * on the account.
API keys — read-only vs write scopes; IP allowlists where vendors support them.
Service-to-service — mTLS or signed workload identity instead of a shared static bearer token passed between twelve services.

When a microservice is compromised, scoped credentials limit lateral movement — the same principle behind SSRF egress controls that block metadata endpoints from reading cloud IAM tokens.

CI/CD and supply chain: secrets in pipelines

Build pipelines need secrets too — deploy keys, container registry passwords, Slack webhooks. Rules:

Store in the CI platform's encrypted secret store (GitHub Actions secrets, GitLab CI variables marked protected/masked).
Never echo secrets in job logs; mask known patterns in output.
Fork PRs from untrusted contributors must not receive secret context — use pull_request_target carefully or require maintainer approval.
Sign artifacts (Sigstore, cosign) so production deploys verify provenance, not just possession of a deploy key.
Separate staging and production secret namespaces; a staging leak must not unlock production databases.

Pair pipeline hygiene with CI/CD best practices — immutable artifacts, environment promotion gates, and rollback paths that do not require re-entering credentials manually.

Observability without leaking

You need to know when secrets are accessed without logging the values:

Audit logs from the secret manager (who fetched which secret, when).
Alert on anomalous access patterns — spike in reads, access from new IP ranges.
Structured logs that redact known secret fields (authorization, password, api_key).
Error trackers configured to scrub PII and credentials before upload.

If debugging requires seeing a token, use one-time reveal in a secure admin UI, not permanent log retention.

Common mistakes

One vault for everything — production and dev share a namespace; a dev script deletes the prod DB password.
Rotation without dual-write — swap the DB password before all app instances reload; half the fleet loses connectivity.
Secrets in frontend env vars — NEXT_PUBLIC_STRIPE_SECRET is oxymoronic; only publishable keys belong in the browser.
Ignoring supply-chain tokens — npm publish tokens and PyPI credentials are as valuable as production DB passwords.
No offboarding checklist — departed employee's personal API token still works because nobody revoked it.

Production checklist

Inventory every secret: owner, rotation schedule, blast radius if leaked.
Remove secrets from git history (rotate after cleanup — history scans persist).
Enable pre-commit and CI secret scanning (gitleaks, trufflehog, GitHub push protection).
Move production secrets to a managed vault with IAM-scoped access and audit logs.
Inject at runtime via workload identity — no secrets in Docker layers or JS bundles.
Scope credentials to minimum permissions; separate staging and production namespaces.
Automate TLS and database password rotation with tested rollback.
Document break-glass access, dual control, and post-incident revocation steps.
Redact secrets in logs, metrics labels, and error reports.
Run a quarterly drill: rotate one critical secret without downtime.

Key takeaways

Secrets grant access — treat them differently from public config and never commit them to source control.
Pull at runtime via workload identity; do not bake credentials into images or frontend bundles.
Managed vaults add encryption, audit trails, and rotation hooks that env files alone cannot provide.
Least privilege limits blast radius when one service or laptop is compromised.
Rotation and monitoring turn a leaked key from a permanent backdoor into a contained incident.