Kubernetes Secrets: How We Leaked API Keys (And Fixed It)

Found our AWS access keys in a public GitHub repo last month. Fun times. Here’s how it happened and what we actually did to prevent it.

How we leaked them

Developer needed to add an API key to a Kubernetes service. Did this:

# deployment.yaml - DON'T DO THIS
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  template:
    spec:
      containers:
      - name: api
        env:
        - name: AWS_ACCESS_KEY
          value: "AKIAIOSFODNN7EXAMPLE"  # 🚨 Plaintext in git!
        - name: AWS_SECRET_KEY
          value: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

Committed it. Pushed it. Public repo. Keys were active for 6 hours before AWS alerted us to unusual activity.

Someone spun up 50 EC2 instances mining cryptocurrency. $2,400 bill.

The “obvious” fix that doesn’t work

“Just use Kubernetes Secrets!”

apiVersion: v1
kind: Secret
metadata:
  name: aws-creds
type: Opaque
data:
  access-key: QUTJQVBMRERKWVVXNUZXM0lG  # Base64 encoded
  secret-key: NVBuREJaeE5wRUxhMkJKSGxNMlpmajNx

# deployment.yaml
env:
- name: AWS_ACCESS_KEY
  valueFrom:
    secretKeyRef:
      name: aws-creds
      key: access-key

Better, right? Nope. Kubernetes Secrets are just base64 encoded, not encrypted. Anyone with kubectl access can decode them:

kubectl get secret aws-creds -o yaml
# Copy the base64 string
echo "QUTJQVBMRERKWVVXNUZXM0lG" | base64 -d
# AKIAIOSFODNN7EXAMPLE

Also, the Secret itself is still committed to git. Just base64 encoded instead of plaintext. Security theater.

What actually works: External secret management

We moved to AWS Secrets Manager with External Secrets Operator.

# First, store secret in AWS Secrets Manager (CLI or Console)
aws secretsmanager create-secret \
  --name prod/aws-creds \
  --secret-string '{"access_key":"AKIA...","secret_key":"wJal..."}'

# Install External Secrets Operator in cluster
helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets-system \
  --create-namespace

# Create SecretStore (points to AWS Secrets Manager)
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa  # Uses IAM role via IRSA

# Create ExternalSecret (syncs from AWS)
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: aws-creds
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: aws-creds  # Creates this Kubernetes Secret
  data:
  - secretKey: access-key
    remoteRef:
      key: prod/aws-creds
      property: access_key
  - secretKey: secret-key
    remoteRef:
      key: prod/aws-creds
      property: secret_key

Now:

Secrets live in AWS Secrets Manager (encrypted at rest)
External Secrets Operator syncs them to K8s
Nothing sensitive in git
Secrets can be rotated in AWS and automatically update in K8s

Security infrastructure diagram

The IRSA setup (critical)

External Secrets needs permission to read from Secrets Manager. Don’t hardcode AWS credentials. Use IAM Roles for Service Accounts (IRSA).

# Create IAM policy
cat > secrets-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret"
    ],
    "Resource": "arn:aws:secretsmanager:us-east-1:*:secret:prod/*"
  }]
}
EOF

aws iam create-policy \
  --policy-name ExternalSecretsPolicy \
  --policy-document file://secrets-policy.json

# Create IAM role for service account
eksctl create iamserviceaccount \
  --name external-secrets-sa \
  --namespace external-secrets-system \
  --cluster production-cluster \
  --attach-policy-arn arn:aws:iam::123456789:policy/ExternalSecretsPolicy \
  --approve

Now the External Secrets operator uses the IAM role, no credentials needed.

Secret rotation

With AWS Secrets Manager, rotation is automated:

aws secretsmanager rotate-secret \
  --secret-id prod/aws-creds \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789:function:RotateSecrets \
  --rotation-rules AutomaticallyAfterDays=30

External Secrets Operator detects the change (checks every hour) and updates the K8s Secret. Pods pick up the new secret automatically via environment variable refresh or volume mount reload.

Encrypting Secrets at rest in etcd

Even with External Secrets, K8s stores the synced Secrets in etcd. By default, etcd doesn’t encrypt.

Enable encryption at rest:

# encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: <base64-32-byte-key>
    - identity: {}  # Fallback for unencrypted data

# Generate encryption key
head -c 32 /dev/urandom | base64

# Apply to API server
kube-apiserver \
  --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \
  ...

Now secrets in etcd are encrypted with AES-CBC. Even if someone dumps etcd, they can’t read secrets.

What we don’t commit to git

# .gitignore for K8s repos
*.secret.yaml
secrets/
credentials/
.env*
kubeconfig
*.key
*.pem

Our CI/CD applies secrets from AWS Secrets Manager directly:

# GitHub Actions
- name: Sync secrets to K8s
  run: |
    aws secretsmanager get-secret-value \
      --secret-id prod/db-password \
      --query SecretString \
      --output text | \
    kubectl create secret generic db-creds \
      --from-literal=password=$(cat) \
      --dry-run=client -o yaml | \
    kubectl apply -f -

No secrets in GitHub Actions env vars either. Uses OIDC to authenticate to AWS.

Sealed Secrets (alternative approach)

If you can’t use external secret management, Sealed Secrets is next best:

# Install Sealed Secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml

# Install kubeseal CLI
brew install kubeseal

# Create sealed secret (encrypted, safe to commit)
kubectl create secret generic mysecret \
  --from-literal=password=mypassword \
  --dry-run=client -o yaml | \
kubeseal -o yaml > sealed-secret.yaml

# Commit sealed-secret.yaml to git
git add sealed-secret.yaml

sealed-secret.yaml is encrypted with the cluster’s public key. Only the Sealed Secrets controller (running in-cluster) can decrypt it.

We use this for non-critical secrets like webhook URLs. Critical secrets (DB passwords, API keys) still go to AWS Secrets Manager.

The actual security model

Developers never see production secrets - They’re stored in AWS Secrets Manager with restricted IAM access
Secrets sync automatically - External Secrets Operator pulls them into K8s
Secrets encrypted at rest - Both in AWS (KMS) and K8s (encryption config)
Secrets encrypted in transit - TLS between everything
Secrets audited - CloudTrail logs every access to AWS Secrets Manager
Secrets rotated - Automatically every 30 days

We also use:

RBAC to limit who can read Secrets in K8s
Pod Security Standards to prevent privilege escalation
OPA Gatekeeper policies to block plaintext secrets in manifests

What actually stopped the leaks

Not technology. Code review policies.

We added a pre-commit hook:

# .git/hooks/pre-commit
#!/bin/bash

if git diff --cached | grep -iE "password|secret|key.*=|token" | grep -v "secretKeyRef"; then
  echo "❌ Possible secret detected in commit!"
  echo "Use Kubernetes Secrets or External Secrets instead."
  exit 1
fi

And a CI check:

# GitHub Actions
- name: Scan for secrets
  uses: trufflesecurity/trufflehog@main
  with:
    path: ./
    base: main
    head: HEAD

TruffleHog scans commits for patterns that look like secrets (AWS keys, JWT tokens, private keys). Fails the build if it finds any.

We also enabled GitHub’s secret scanning and got alerts when we pushed known patterns.

The $2,400 lesson

AWS detected unusual activity within hours. Their abuse team emailed us. We:

Rotated all potentially exposed credentials
Killed the rogue EC2 instances
Enabled CloudTrail logging everywhere
Set up billing alerts
Requested a bill adjustment (AWS forgave $1,800 as a one-time courtesy)

Total actual cost: $600 + a week of security hardening work.

Could’ve been worse. We learned about External Secrets, IRSA, secret encryption, and the importance of secret scanning before someone used our keys to access customer data instead of just mining crypto.

The real fix wasn’t technical. It was making secrets management a team priority and building guardrails that prevent accidents before they happen.

How we leaked them#

The “obvious” fix that doesn’t work#

What actually works: External secret management#

The IRSA setup (critical)#

Secret rotation#

Encrypting Secrets at rest in etcd#

What we don’t commit to git#

Sealed Secrets (alternative approach)#

The actual security model#

What actually stopped the leaks#

The $2,400 lesson#