AWS Secrets Management: From Hardcoded Credentials to Automatic Rotation

Every application needs credentials to function — database passwords, API keys, TLS certificates, OAuth tokens. How those credentials are stored and delivered to the application is one of the most consequential security decisions in a cloud architecture, and it is also one of the most commonly deferred. The result is a landscape where hardcoded credentials in source code and credentials passed as plaintext environment variables remain the norm, even in organizations with mature security programs in other areas.

This guide covers the four patterns, ranks them by risk, and provides the concrete implementation details for getting to the right answer.

The four patterns, ranked by risk

Hardcoded in source code is the worst possible option — not because it is the easiest for an attacker to exploit in isolation, but because it is permanent. A credential committed to a git repository exists in the commit history indefinitely, even if the file is later deleted. If the repository is ever made public, cloned by a third party, or accessed by a compromised developer account, every credential that has ever been hardcoded in its history is exposed. Toyota’s five-year exposure (covered elsewhere on this blog) is the canonical example.

Environment variables set at deploy time move the credential out of source code but introduce a different exposure surface. CI/CD pipelines log environment variables during build and deploy steps. Container runtimes expose them via docker inspect or the /proc/1/environ endpoint on misconfigured instances. Kubernetes Secrets, which are often the backing store for container environment variables, are base64-encoded but not encrypted at rest by default. The credential is still a static value that must be rotated manually and tracked outside the system that uses it.

AWS Systems Manager Parameter Store with SecureString is a meaningful improvement. Parameters are stored encrypted using KMS, retrieved at runtime via ssm:GetParameter, and access is controlled by IAM. Cost is low — standard parameters are free, advanced parameters (larger values, higher throughput) are $0.05 per parameter per month. The limitation is that Parameter Store has no native rotation support. You can implement rotation externally, but it requires custom automation. Use Parameter Store for configuration values that change infrequently and do not require automatic rotation — feature flags, endpoint URLs, non-rotating API keys.

AWS Secrets Manager is the correct choice for any credential that requires rotation or for any secret whose access pattern warrants audit logging at the individual retrieval level. Cost is $0.40 per secret per month plus $0.05 per 10,000 API calls — meaningfully higher than Parameter Store but justified for database credentials, OAuth client secrets, and any credential that must be rotated on a schedule. Secrets Manager includes built-in rotation support for RDS, Redshift, DocumentDB, and other AWS services, and a Lambda-based rotation framework for custom integrations.

How automatic rotation works

Secrets Manager rotation operates through a four-phase Lambda function that AWS invokes on the configured schedule (or immediately on demand via rotate-secret):

createSecret: The Lambda generates a new credential value and stores it as a new version of the secret with the staging label AWSPENDING. The existing version retains the label AWSCURRENT — callers continue receiving the old value while the new one is being prepared.

setSecret: The Lambda applies the new credential to the target service. For an RDS database, this means calling the database’s user management API to update the password for the application user.

testSecret: The Lambda verifies the new credential works — typically by opening a connection to the database using the AWSPENDING version and executing a test query.

finishSecret: The Lambda moves the AWSCURRENT label to the new version and removes AWSPENDING. Subsequent calls to GetSecretValue return the new credential. The previous version retains AWSPREVIOUS for a brief grace period, allowing in-flight connections using the old credential to complete.

For RDS, Redshift, and DocumentDB, AWS provides pre-built Lambda rotation functions. For third-party APIs — payment processors, SaaS platforms, external data providers — you implement the four phases in a custom Lambda, and Secrets Manager handles the invocation schedule and version labeling.

Cross-account secret sharing and VPC endpoints

Share a secret across accounts by adding a resource-based policy to the secret that allows secretsmanager:GetSecretValue for the target account’s IAM principals. The consuming account’s Lambda or EC2 role calls GetSecretValue using the full secret ARN from the owning account.

For workloads that access Secrets Manager from within a VPC, provision a VPC interface endpoint (com.amazonaws.region.secretsmanager). This routes GetSecretValue calls over AWS’s private network — the EC2 instance or Lambda function never makes an internet-bound HTTPS call to retrieve a credential. Add an endpoint policy restricting which secrets can be retrieved through the endpoint, and set a bucket-policy-style deny on the secret itself requiring aws:sourceVpce to match your endpoint ID.

IaC patterns that avoid exposing secret values

In CDK, reference a secret by ARN without embedding its value in the stack:

const dbPassword = secretsmanager.Secret.fromSecretNameV2(
  this, 'DBPassword', 'prod/myapp/db-password'
);
dbInstance.addRotationSingleUser(dbPassword);

In Terraform, use aws_secretsmanager_secret_version data sources and pass the secret ARN (not the value) to resources that support native Secrets Manager integration. Never output secret values from Terraform — they are stored in state files in plaintext.

Scanning for leaked secrets

Add gitleaks to your pre-commit hooks and CI pipeline:

- name: Scan for secrets
  run: gitleaks detect --source . --redact --exit-code 1

Run trufflehog git file://. --only-verified as a retrospective audit against existing repositories to surface any credentials that were committed historically and may still be valid. For GitHub-hosted repositories, enable push protection at the organization level — it blocks the push server-side even if the developer bypassed local hooks.

Monitor secretsmanager:GetSecretValue in CloudTrail. Alert on retrievals from unexpected IAM principals, from IP addresses outside your known egress ranges, or at unusual hours. A secret retrieved by an identity that has never previously accessed it is a signal worth investigating.

Takeaway

The credential storage decision is made once per secret and tends to persist indefinitely. Starting with Secrets Manager and rotation built in costs a few dollars per month per secret — less than the hourly rate of the engineer who will eventually handle a credential rotation incident caused by a static secret. The operational investment in setting up rotation correctly the first time pays back on the first scheduled rotation cycle, when no human intervention is required and no downtime occurs.