TL;DR: Stop relying on insecure base64-encoded Kubernetes secrets that complicate SOC2 compliance and cause secrets sprawl. This guide demonstrates how to architect zero-touch, dynamic secret rotation by integrating AWS Secrets Manager with the External Secrets Operator (ESO). You will learn how to deploy ESO via Helm and securely authenticate it using AWS IAM Roles for Service Accounts (IRSA) via OIDC to eliminate static credentials entirely.

⚡ Key Takeaways

• Replace insecure, base64-encoded native Kubernetes secrets with dynamic secret injection to eliminate secrets sprawl and simplify SOC2 compliance audits.
• Choose the External Secrets Operator (ESO) over the Secrets Store CSI driver to allow applications to consume secrets via standard envFrom declarations without requiring code changes to read from file paths.
• Deploy ESO using Helm with --set installCRDs=true to automatically provision required Custom Resource Definitions like SecretStore and ExternalSecret.
• Authenticate ESO to AWS Secrets Manager using IAM Roles for Service Accounts (IRSA) and OIDC to completely eliminate the anti-pattern of storing static AWS IAM access keys in your cluster.

Native Kubernetes secrets are inherently insecure by default. If your deployment pipelines involve checking base64-encoded strings into version control, injecting environment variables via CI/CD platforms, or relying on statically defined Secret manifests, your architecture is actively suffering from secrets sprawl.

The fundamental problem with native Kubernetes secrets is that they are merely encoded, not encrypted. Anyone with kubectl get secret privileges can trivially expose your database credentials, API keys, and third-party tokens.

The Base64 Illusion and the Cost of Secrets Sprawl

When you create a standard Kubernetes Secret, you are likely using a manifest that looks something like this:

apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
type: Opaque
data:
  DB_PASSWORD: c3VwZXJzZWNyZXRwYXNzd29yZA== # "supersecretpassword"

To experienced engineers, this is a well-known risk. Base64 is an encoding mechanism designed for data transport, not a cryptographic protection mechanism.

Security Note: Even if you enable Encryption at Rest in etcd, this only protects the data on the physical disks of the control plane. An attacker—or a compromised pod with access to the Kubernetes API—can still read the decrypted secret in plain text.

Secrets sprawl occurs when these static files are copied across developers' local environments, staging clusters, and Git repositories. When a developer leaves the company, or when an API key is inadvertently logged to Datadog or CloudWatch, tracking down every instance of that static secret becomes an incident response nightmare.

The SOC2 Nightmare and Rotation Friction

When preparing for SOC2 or ISO-27001 compliance, auditors will inevitably ask for evidence of your secret rotation policies. They require proof that your database passwords and third-party API keys are rotated on a strict schedule (e.g., every 30 to 90 days) without causing application downtime.

If you use static .env files or native Kubernetes secrets injected via CI/CD, secret rotation becomes a friction-laden, manual process:

1. Manually change the password in the database (e.g., AWS RDS).
2. Update the CI/CD pipeline secrets (GitHub Actions, GitLab CI, etc.).
3. Distribute the new credentials to developers for their local .env files.
4. Run kubectl apply -f new-secret.yaml.
5. Run kubectl rollout restart deployment/my-api.

The result: Inevitable downtime and mismatched credentials during the rotation window.

The frustration peaks when a manual rotation fails, bringing down an entire microservice simply because a single pipeline job failed to inject the updated secret. We need a system where Kubernetes acts as a consumer of secrets, not the source of truth.

The Solution: External Secrets Operator (ESO) vs. Secrets Store CSI

To achieve zero-touch, automated secret rotation, we must decouple secret storage from secret consumption.

There are two primary paradigms for securely managing secrets in AWS EKS:

1. Secrets Store CSI Driver: Mounts secrets as file volumes directly inside the pod.
2. External Secrets Operator (ESO): Fetches secrets from external APIs (like AWS Secrets Manager) and dynamically generates native Kubernetes Secret objects.

While the CSI driver is highly secure, it often requires rewriting application code to read credentials from file paths instead of environment variables. ESO is vastly superior for most standard architectures because it bridges the gap: it securely fetches the secret from AWS and creates a native Secret that your application can consume via standard envFrom declarations without code changes.

Let's install the External Secrets Operator (ESO) using Helm:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
    --set installCRDs=true \
    --set webhook.port=9443

By setting installCRDs=true, Helm provisions the necessary Custom Resource Definitions (CRDs): SecretStore, ClusterSecretStore, and ExternalSecret.

Architecting Least Privilege with AWS IRSA

Before ESO can pull secrets from AWS Secrets Manager, it needs authorization. The anti-pattern here is to create an AWS IAM User, generate long-lived Access Keys, and store those keys in Kubernetes. This simply replaces one static secret with an even more powerful static secret.

Instead, we use IAM Roles for Service Accounts (IRSA). IRSA leverages OpenID Connect (OIDC) to securely map a Kubernetes ServiceAccount directly to an AWS IAM Role.

First, we define an IAM Trust Policy that allows the ESO ServiceAccount in our EKS cluster to assume the IAM Role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53OSOMETHING"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53OSOMETHING:sub": "system:serviceaccount:external-secrets:external-secrets-sa"
        }
      }
    }
  ]
}

Next, we attach a permissions policy restricting this role to only read the specific secrets it needs from AWS Secrets Manager:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:production/db/*"
    }
  ]
}

When our team handles DevOps and Cloud Deployment configurations for enterprise clients, properly scoping these IAM policies is always the first critical step to ensure a compromised namespace cannot read the entire organization's AWS Secrets Manager vault.

Bootstrapping the ClusterSecretStore

With ESO installed and IRSA configured, we must instruct the operator on how to connect to AWS. We achieve this by creating a ClusterSecretStore. This is a cluster-scoped resource, meaning any namespace can reference it (if you prefer strict namespace isolation, use a SecretStore instead).

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

Notice the clean authentication block. Because we are using the jwt authentication method linked to our ServiceAccount, there are absolutely zero AWS credentials hardcoded in this configuration. The ESO controller uses the injected AWS web identity token to seamlessly authenticate against the AWS API.

Fetching and Synchronizing the ExternalSecret

Now for the core mechanism. We want to pull a JSON payload from AWS Secrets Manager (e.g., production/db/credentials) and map it into a native Kubernetes Secret.

We create an ExternalSecret manifest, which serves as the instruction manual for ESO.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials-es
  namespace: production-apps
spec:
  refreshInterval: "1h" # The critical rotation interval
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials-native # Name of the standard K8s Secret it will create
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: production/db/credentials

Production Tip: The refreshInterval is where the magic happens. Every hour, the ESO reconciliation loop queries AWS Secrets Manager. If the secret in AWS has changed, ESO immediately updates the native db-credentials-native secret in Kubernetes.

You can now map this dynamically generated secret into your application deployment exactly as you would with a static secret:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nodejs-backend
  namespace: production-apps
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: api
        image: my-company/nodejs-backend:v2.4.1
        envFrom:
        - secretRef:
            name: db-credentials-native # This is managed by ESO!

Automating Rotation on the AWS Backend

The Kubernetes side is now fully dynamic. However, to satisfy SOC2 compliance completely, we must automate the rotation of the actual secret value within AWS.

AWS Secrets Manager supports native rotation schedules powered by AWS Lambda. When the schedule triggers, the Lambda function connects to your RDS instance, generates a new randomized password, updates the database, and stores the new password back in Secrets Manager.

You can configure this via the AWS CLI by attaching a rotation schedule to your secret:

aws secretsmanager rotate-secret \
    --secret-id production/db/credentials \
    --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:RotateRDSPassword \
    --rotation-rules AutomaticallyAfterDays=30

The Full Lifecycle in Action

1. Day 30: AWS Secrets Manager triggers the Lambda function.
2. The Lambda updates the RDS password and saves it to Secrets Manager.
3. Within 1 Hour: ESO's refreshInterval is triggered. It fetches the new JSON payload from AWS.
4. ESO instantly updates the db-credentials-native Kubernetes Secret.

Trade-offs and Production Safeguards

While this architecture effectively eliminates secrets sprawl, engineers must be aware of the architectural trade-offs introduced.

1. The Pod Restart Problem

When ESO updates the underlying Kubernetes Secret, running pods do not automatically restart. Environment variables are injected only at pod boot time. If your database password rotates, your existing pods will eventually drop their connection and fail to reconnect, causing downtime.

To solve this, you must run a controller like Reloader by Stakater. Reloader watches for changes to ConfigMaps and Secrets and automatically performs a rolling restart of any associated Deployments.

# Add this annotation to your Deployment
metadata:
  annotations:
    secret.reloader.stakater.com/reload: "db-credentials-native"

2. AWS API Rate Limiting and Costs

AWS Secrets Manager charges per 10,000 API calls. If you set your refreshInterval to 1m (one minute) across 500 ExternalSecret resources, you are generating 720,000 API calls a day. This will lead to unnecessary AWS billing and potential throttling (ThrottlingException).

A 1-hour or 4-hour refreshInterval is generally sufficient for compliance while keeping AWS API calls minimal.

3. Fallback and Resilience

What happens if AWS Secrets Manager goes down or the IAM trust relationship breaks? By setting creationPolicy: Owner in the ExternalSecret, ESO ensures that if the secret is successfully pulled once, it will maintain the cached native Secret in the cluster even if the AWS API becomes temporarily unreachable. Your applications will boot safely using the last known good secret.

# Checking the status of your ExternalSecret integration
$ kubectl get externalsecrets -n production-apps

NAME                STORE                 REFRESH INTERVAL   STATUS         READY
db-credentials-es   aws-secrets-manager   1h                 SecretSynced   True

By implementing the External Secrets Operator with AWS Secrets Manager, you effectively remove human interaction from the credential lifecycle. Developers no longer need access to production database passwords, CI/CD pipelines no longer transport sensitive tokens, and SOC2 auditors receive verifiable proof of continuous, zero-touch rotation.

Work With Us

Need help building this in production? SoftwareCrafting is a full-stack dev agency — we ship React, Next.js, Node.js, React Native & Flutter apps for global clients. Or simply want to book a free architecture review to assess your current setup? We are here to help.

Get a Free Consultation →

Frequently Asked Questions

Why are native Kubernetes secrets considered insecure by default?

Native Kubernetes secrets are merely base64-encoded strings, not cryptographically encrypted data. Anyone with kubectl get secret privileges or access to a compromised pod can easily decode and read your sensitive credentials in plain text. Even if you enable etcd encryption at rest, the data remains exposed at the Kubernetes API level.

What is the difference between External Secrets Operator (ESO) and the Secrets Store CSI Driver?

The Secrets Store CSI Driver mounts external secrets as file volumes directly inside your pods, which often requires rewriting application code to read from file paths instead of environment variables. In contrast, ESO fetches secrets from external APIs like AWS Secrets Manager and dynamically generates native Kubernetes Secret objects. This allows applications to consume secrets natively via standard envFrom declarations without any code changes.

How do I authenticate the External Secrets Operator with AWS without using static IAM access keys?

You should use IAM Roles for Service Accounts (IRSA) to securely map a Kubernetes ServiceAccount directly to an AWS IAM Role. By leveraging OpenID Connect (OIDC), ESO can assume the necessary IAM Role to fetch credentials from AWS Secrets Manager without ever storing long-lived, static AWS access keys in your cluster.

How does integrating AWS Secrets Manager with Kubernetes help achieve SOC2 compliance?

SOC2 and ISO-27001 auditors require proof of strict, automated secret rotation policies without causing application downtime. By decoupling secret storage from consumption using AWS Secrets Manager and ESO, you can implement zero-touch, automated rotation schedules. This eliminates the friction, human error, and potential outages associated with manually updating Kubernetes manifests and CI/CD pipelines.

Our team is struggling to implement zero-touch secret rotation in EKS. Can SoftwareCrafting help architect this?

Yes, SoftwareCrafting specializes in designing and implementing secure, SOC2-compliant DevOps architectures for Kubernetes environments. Our experts can help you seamlessly integrate AWS Secrets Manager, configure the External Secrets Operator, and establish least-privilege IAM policies using IRSA to eliminate secrets sprawl entirely.

Does SoftwareCrafting provide security audits for existing Kubernetes deployments suffering from secrets sprawl?

Absolutely. SoftwareCrafting offers comprehensive security and infrastructure audits to identify vulnerabilities like static base64 secrets, misconfigured CI/CD pipelines, and excessive IAM privileges. We provide actionable remediation plans to modernize your secret management and prepare your infrastructure for rigorous compliance frameworks.

📎 Full Code on GitHub Gist: The complete kubernetes-secret.yaml from this post is available as a standalone GitHub Gist — copy, fork, or embed it directly.