The API Key That Cost $50,000
A developer committed an AWS key to GitHub. Within minutes, bots found it. Within hours, someone spun up $50,000 worth of crypto mining instances. The key had been in the code "temporarily" for local testing.
Secrets in code is the gift that keeps on giving—to attackers. Exposed credentials are one of the top causes of cloud breaches. GitHub reports finding millions of exposed secrets every year.
This guide covers practical secrets management for startups—how to handle API keys, passwords, and credentials without ending up in the news.
What Counts as a Secret?
Definitely Secrets:
- API keys and tokens
- Database passwords
- AWS/GCP/Azure credentials
- Private keys and certificates
- OAuth client secrets
- Encryption keys
- Third-party service credentials
Also Secrets (Often Forgotten):
- Connection strings
- Webhook URLs with tokens
- SSH keys
- JWT signing keys
- Slack/Discord webhooks
- SMTP credentials
- License keys
If losing it would let someone access something they shouldn't, it's a secret. If you'd be uncomfortable sharing it publicly, it's a secret. When in doubt, treat it as a secret.
Where Secrets Should NOT Be
Never Store Secrets In:
- Source Code — Even "temporarily." Git history is forever.
- Environment Variables (in code) — .env files committed to repos
- Configuration Files — config.json, settings.yaml in repos
- Documentation — README files, wikis, Confluence pages
- Log Files — Application logs, debug output
- Error Messages — Exception messages exposed to users
- Email — Sharing credentials via email
- Slack/Chat — Sharing in chat tools (searchable!)
- Spreadsheets — Password lists in Google Sheets
Removing a secret from the latest commit doesn't help—it's still in git history. If you've committed a secret, rotate it immediately. Assume it's compromised. You cannot reliably remove it from history, especially if it's been pushed.
Where Secrets SHOULD Be
Secrets Management Solutions
Choosing the Right Solution
- Single Cloud? — Use the native secrets manager (AWS Secrets Manager, etc.)
- Multi-Cloud? — Consider HashiCorp Vault or Doppler
- Simple Needs? — Cloud-native + .env files (not committed) may suffice
- Team Sharing? — 1Password Teams/Business for shared credentials
Secrets Management Best Practices
1. Use Environment Variables (Properly)
- Never commit .env files — Add to .gitignore immediately
- Inject at runtime — CI/CD, container orchestration, or secrets manager
- Separate by environment — Different secrets for dev, staging, production
- Document required variables — .env.example with dummy values
2. Rotate Credentials Regularly
- Automate rotation — AWS Secrets Manager can rotate RDS passwords automatically
- Short-lived credentials — Use IAM roles, temporary tokens where possible
- Rotation schedule — 90 days maximum for long-lived credentials
- Rotation after exposure — Immediately rotate any potentially exposed secret
3. Implement Least Privilege
- Separate credentials per service — Don't share database passwords across apps
- Scope permissions — API keys should only have required permissions
- Environment isolation — Prod credentials can't access dev and vice versa
4. Audit and Monitor
- Log access — Know who/what accessed each secret
- Alert on anomalies — Unusual access patterns, failed attempts
- Regular audits — Review what secrets exist and who has access
Developers should never need to see or handle production credentials directly. Secrets should flow from a secrets manager to applications automatically, with access logged and monitored.
Preventing Secret Exposure
Pre-Commit Scanning
- git-secrets — AWS-maintained tool to prevent committing secrets
- detect-secrets (Yelp) — Broad secret detection with baseline support
- pre-commit hooks — Block commits containing secrets before they happen
- IDE plugins — Real-time detection in your editor
CI/CD Scanning
- GitGuardian — Scans repositories for exposed secrets
- GitHub Secret Scanning — Built-in scanning for known secret formats
- Gitleaks — Open source secret scanner for CI pipelines
Multiple layers of protection: pre-commit hooks catch before commit, CI/CD scanning catches in pull requests, repository scanning catches historical issues. Use all three.
Handling Secret Exposure
If a Secret is Exposed
Step 1: Rotate Immediately
Generate new credential. Update all systems using it. This is the priority.
Step 2: Revoke Old Credential
After new credential is deployed, revoke the exposed one.
Step 3: Assess Impact
Check logs. Was the credential used maliciously? What could have been accessed?
Step 4: Remove from History
If still in repo, use BFG or git-filter-branch. But assume it's already seen.
Step 5: Post-Mortem
How did this happen? What process changes prevent recurrence?
Common Secrets Mistakes
Mistake 1: "It's Just for Development"
"Temporary" secrets in code become permanent. Development credentials end up in production. Treat all secrets with the same care, regardless of environment.
Mistake 2: Overly Permissive Credentials
An API key with admin access is more dangerous when exposed than one with read-only access. Create credentials with minimum necessary permissions.
Mistake 3: No Rotation Plan
Credentials created three years ago are still active? That's three years of potential exposure. Implement rotation before you need it.
Mistake 4: Sharing Via Insecure Channels
Slack messages and emails are searchable and logged. Use a password manager or secrets manager to share credentials securely.
Quick Start: Your First Week
Day 1: Audit Current State
Search your repos for common secret patterns (password=, api_key=, etc.). Check .env files.
Day 2: Add .gitignore Rules
Ensure .env, *.pem, *.key, and similar files are gitignored across all repos.
Day 3: Install Pre-Commit Hooks
Set up git-secrets or detect-secrets to prevent future commits of secrets.
Day 4-5: Implement Secrets Manager
Choose a solution (cloud-native is fine). Migrate at least production secrets.
Day 6-7: Rotate Existing Credentials
For any secrets that may have been exposed in code, rotate immediately.
Next Steps
Secrets management isn't optional—it's fundamental. Every exposed credential is a potential breach. Every shared password is an audit failure. Every secret in code is a ticking time bomb.
Start with prevention: pre-commit hooks and .gitignore rules cost nothing and catch most issues. Add a secrets manager for production. Implement rotation. Build the habit of treating secrets as the sensitive assets they are.
Managing credentials? vCISO Lite helps you track secrets management practices, document your controls, and demonstrate secure credential handling to auditors and customers.