Skip to content

SOPS

Introduction

So to avoid that, I need some solution, at least they can not access my secret env, which contains API keys, login credentials, etc. Because in K8S, a secret is just encoded base64, not encrypted.

Install locally to generate encrypted file

You can read detail in here https://github.com/getsops/sops

So, I expected that you already have sops and age-keygen available.

1. Generate an Age Key Pair (if you haven’t already)

This creates your private key file (key.txt). This file must be present in the project root when you build the Docker image. It is ignored by .gitignore so it won't be committed to Git.

age-keygen -o key.txt
# You will need the public key (from the output or key.txt.pub) for the next step.

Expected output:

Public key: age1dk67a27dugyy964g38s462mjjn32aewzr4a2fy3yuwcd6wmvufrsg0n7ef

It also generated a private key file (key.txt), which you should keep secure.

2. Encrypt dotenv secret

Create this file .sops.yaml

# This file tells sops how to encrypt/decrypt files.
creation_rules:
  - path_regex: \.env\.enc$ # This regex now correctly matches files ending with .env.enc
    # Your age public key
    age: "age1dk67a27dugyy964g38s462mjjn32aewzr4a2fy3yuwcd6wmvufrsg0n7ef"

Run the following command:

sops --encrypt --age age1dk67a27dugyy964g38s462mjjn32aewzr4a2fy3yuwcd6wmvufrsg0n7ef .env > .env.enc

Expected output of the file .env.enc:

{
 "data": "ENC[AES256_GCM,data:adasdasdasdasd,type:str]",
 "sops": {
  "age": [
   {
    "recipient": "age1dk67a27dugyy964g38s462mjjn32aewzr4a2fy3yuwcd6wmvufrsg0n7ef",
    "enc": "-----BEGIN AGE ENCRYPTED FILE-----\somethingsecrethereLOLLLn-----END AGE ENCRYPTED FILE-----\n"
   }
  ],
  "lastmodified": "2026-01-30T04:31:13Z",
  "mac": "ENC[AES256_GCM,data:aaasddsadadasda==,type:str]",
  "version": "3.11.0"
 }
}

You can test to decrypt it with:

export AGE_KEY_FILE=key.txt # Set environment variable
sops --decrypt --input-type json --output-type binary .env.enc > env.decrypted

And check the output by read content of env.decrypted file.

3. Create secret and build Dockerfile for it

Create secret from file: with this, you don’t care about people with access can decode the secret xD

# The key is: encrypted.env. Value is content of .env.enc
kubectl create secret generic my-secret --from-file=encrypted.env=.env.enc

Dockerfile for that shit, you will notice that I did removed shell from final image. So in that scenario those with k8s cluster access won’t be able to enter into container to decode the secret xD

But you need to keep key.txt and build image manually, it is not possible to put key.txt into git repo and use pipeline for CICD in there.

# Stage 1: Builder
FROM python:3.12-alpine AS builder

WORKDIR /app
RUN apk add --no-cache build-base && python -m pip install --no-cache-dir pip==25.3

RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

COPY requirements.txt .
RUN python -m pip install --no-cache-dir -r requirements.txt

# Stage 2: Runner
FROM python:3.12-alpine

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PATH="/opt/venv/bin:$PATH"

# Upgrade pip
RUN python -m pip install --no-cache-dir pip==25.3

# Install sops and jq, then remove shell-related packages
# Note: --no-check-certificate is used due to corporate proxy limitations
RUN apk add --no-cache wget jq && \
    SOPS_VERSION="v3.11.0" && \
    wget "https://github.com/getsops/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux.amd64" --no-check-certificate -O /usr/local/bin/sops && \
    chmod +x /usr/local/bin/sops && \
    apk del wget

# Create user (still need shadow temporarily)
RUN apk add --no-cache shadow && \
    groupadd -g 1000 appuser && \
    useradd -u 1000 -r -g appuser -d /app appuser && \
    apk del shadow

WORKDIR /app

# Copy virtual environment
COPY --from=builder /opt/venv /opt/venv

# Copy source code and entrypoint
COPY . .
COPY entrypoint.py /app/entrypoint.py
RUN chmod +x /app/entrypoint.py

# Copy age key
RUN mkdir -p /etc/.yolo
COPY key.txt /etc/.yolo/key.txt
RUN chown -R appuser:appuser /app /etc/.yolo

# Remove shells (bash, sh, ash)
RUN rm -f /bin/sh /bin/ash /bin/bash /usr/bin/sh 2>/dev/null || true

USER appuser

EXPOSE 8501

# Use exec form to avoid shell
CMD ["python3", "/app/entrypoint.py"]

And important for runtime decryption entrypoint.py. I create this script to decrypt the secret at runtime then start the application.

#!/usr/bin/env python3
import os
import subprocess
import sys

ENCRYPTED_SECRET_PATH = "/etc/secrets/encrypted.env"
AGE_KEY_FILE = "/etc/.yolo/key.txt"
DECRYPTED_ENV_PATH = "/app/.env"

def main():
    print("Attempting to decrypt secret...")

    # Check files exist
    if not os.path.isfile(ENCRYPTED_SECRET_PATH):
        print(f"ERROR: Encrypted secret file not found at {ENCRYPTED_SECRET_PATH}", file=sys.stderr)
        sys.exit(1)

    if not os.path.isfile(AGE_KEY_FILE):
        print(f"ERROR: Age private key file not found at {AGE_KEY_FILE}", file=sys.stderr)
        sys.exit(1)

    # Read age key
    with open(AGE_KEY_FILE, 'r') as f:
        age_key = ''.join(line for line in f if not line.startswith('#')).strip()

    if not age_key:
        print("ERROR: Age private key file is empty or could not be read.", file=sys.stderr)
        sys.exit(1)

    # Set SOPS_AGE_KEY environment variable
    env = os.environ.copy()
    env['SOPS_AGE_KEY'] = age_key

    # Detect format
    with open(ENCRYPTED_SECRET_PATH, 'r') as f:
        first_char = f.read(1)

    # Decrypt
    if first_char == '{':
        print("Detected JSON format (binary encrypted)")
        cmd = ['/usr/local/bin/sops', '--decrypt', '--input-type', 'json', '--output-type', 'binary', ENCRYPTED_SECRET_PATH]
    else:
        print("Detected dotenv format")
        cmd = ['/usr/local/bin/sops', '--decrypt', ENCRYPTED_SECRET_PATH]

    with open(DECRYPTED_ENV_PATH, 'wb') as out:
        result = subprocess.run(cmd, env=env, stdout=out, stderr=subprocess.PIPE)

    if result.returncode != 0:
        print(f"ERROR: Decryption failed: {result.stderr.decode()}", file=sys.stderr)
        sys.exit(1)

    # print(f"Secret decrypted successfully to {DECRYPTED_ENV_PATH}")

    # Show first 1 lines (for debugging)
    # with open(DECRYPTED_ENV_PATH, 'r') as f:
    #     for i, line in enumerate(f):
    #         if i >= 1:
    #             break
    #         print(f"Line {i+1}: {line.rstrip()}")

    # Start Streamlit
    print("Starting Streamlit application...")
    os.execvp('streamlit', ['streamlit', 'run', 'report.py', '--server.address=0.0.0.0'])

if __name__ == '__main__':
    main()

4. Deploy to k8s

Here is my sample deployment. I create service so I know which user for securityContext and yeah Sonarqube is annoying, it forces me to put resource request and limit, haha

apiVersion: apps/v1
kind: Deployment
metadata:
  name: reporting-dashboard
  namespace: dashboard-analytics
  labels:
    app: reporting-dashboard
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reporting-dashboard
  template:
    metadata:
      labels:
        app: reporting-dashboard
    spec:
      # non-root (appuser - 1000) defined in Dockerfile
      automountServiceAccountToken: false
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        runAsNonRoot: true
      imagePullSecrets:
      - name: secret-for-pull-image-here
      containers:
      - name: dashboard
        securityContext:
          allowPrivilegeEscalation: false
        image: your-image-here:v1.2.3
        imagePullPolicy: Always
        ports:
        - containerPort: 8501
          name: http
        # New: Mount encrypted secret and age key for runtime decryption
        volumeMounts:
        - name: encrypted-secret
          mountPath: /etc/secrets/encrypted.env
          subPath: encrypted.env # Mounts the 'encrypted.env' key from the secret as a file
          readOnly: true
        resources:
          requests:
            cpu: "200m"
            memory: "256Mi"
          limits:
            cpu: "1000m"
            memory: "1Gi"
            ephemeral-storage: "5Gi"
      volumes:
      # New: Define volumes for the encrypted secret
      - name: encrypted-secret
        secret:
          secretName: my-secret
          items:
          - key: encrypted.env
            path: encrypted.env

That’s all, I think it’s the best solution for me in this case!

Conclusion

Trade-off: Only able to save you from lazy attackers. Not from determined attackers.

  • Risk: if anyone has access to the image registry or pull image, they can get the key.txt and decrypt the secret. So make sure your image registry is secure. In fact, you’ll still be fucked if someone has access to your k8s cluster. Can’t hide the image pull secret.
  • How to mitigate: Make sure your image registry is secure, and only authorized users can pull images.

Something fun I learned:

  • sops for encryption/decryption
  • age for key management
  • Just remove fucking shell from the container. A little hard to debug, but I’m capable of doing that via logging; I don’t need to use the shell in the container for debugging. I think it’s a good practice for security. And yeah, I know how the application works since I’m the only developer in that project.
  • SonarQube is annoying but worth it; it forces me to put resource requests and limits if I don’t want to see a lot of warnings.

I’m using this solution because it just matched for my case. It’s simple, secure, and easy to maintain.

a System Admin knows how to echo "Hello World"