Configuration Management in Kubernetes

What are ConfigMaps?

ConfigMaps are Kubernetes objects that store non-sensitive configuration data in key-value pairs. They decouple environment-specific configuration from container images.

Environment Variables: Inject configuration as env vars
Configuration Files: Mount as files in volumes
Command Arguments: Pass as container command args

What are Secrets?

Secrets are similar to ConfigMaps but designed to hold sensitive information such as passwords, OAuth tokens, SSH keys, and TLS certificates.

Base64 Encoded: Data is encoded (not encrypted)
Size Limited: Maximum 1MB per Secret
Types: Opaque, TLS, Docker Registry, Service Account

Feature	ConfigMap	Secret
Purpose	Non-sensitive configuration	Sensitive data
Data Format	Plain text	Base64 encoded
Encryption at Rest	Optional	Recommended
RBAC Support	Yes	Yes (more restrictive)
Size Limit	1MB	1MB
Update Behavior	Can be updated	Can be updated

Best Practice

Always use ConfigMaps for non-sensitive data and Secrets for sensitive data. Never store sensitive information like passwords or API keys in ConfigMaps, even if they're "internal" to your cluster.

Working with ConfigMaps

Creating ConfigMaps

From Literal Values

kubectl create configmap app-config \
  --from-literal=database_url=postgres://localhost:5432/mydb \
  --from-literal=cache_size=100 \
  --from-literal=log_level=debug

From File

# Create from a properties file
kubectl create configmap app-config --from-file=app.properties

# Create from multiple files
kubectl create configmap app-config \
  --from-file=config/ \
  --from-file=app.properties

ConfigMap Manifest

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: default
data:
  # Simple key-value pairs
  database_url: "postgres://localhost:5432/mydb"
  cache_size: "100"
  log_level: "debug"
  
  # Multi-line file content
  app.properties: |
    server.port=8080
    server.host=0.0.0.0
    debug.enabled=true
    
  nginx.conf: |
    server {
      listen 80;
      server_name example.com;
      
      location / {
        proxy_pass http://backend:8080;
      }
    }

Using ConfigMaps in Pods

As Environment Variables

apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    env:
    # Individual keys
    - name: DATABASE_URL
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: database_url
    - name: LOG_LEVEL
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: log_level
    
    # All keys from ConfigMap
    envFrom:
    - configMapRef:
        name: app-config
        prefix: APP_  # Optional prefix for env vars

As Volume Mounts

apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: config-volume
      mountPath: /etc/config
      readOnly: true
    
    # Mount specific keys to specific paths
    - name: nginx-config
      mountPath: /etc/nginx/nginx.conf
      subPath: nginx.conf
      
  volumes:
  - name: config-volume
    configMap:
      name: app-config
      # Optional: specify permissions
      defaultMode: 0644
      
  - name: nginx-config
    configMap:
      name: app-config
      items:
      - key: nginx.conf
        path: nginx.conf
        mode: 0644

ConfigMap Updates

When a ConfigMap is updated:

Environment variables do NOT update automatically
Volume-mounted files update within ~1 minute (kubelet sync period)
SubPath mounts do NOT update automatically
Pods must be restarted to pick up env var changes

Working with Secrets

Types of Secrets

Opaque Secrets

Default secret type for arbitrary user-defined data

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=  # base64 encoded
  password: MWYyZDFlMmU2N2Rm  # base64 encoded

TLS Secrets

For storing TLS certificates and keys

kubectl create secret tls tls-secret \
  --cert=path/to/tls.crt \
  --key=path/to/tls.key

Docker Registry Secrets

For authenticating with private Docker registries

kubectl create secret docker-registry regcred \
  --docker-server=myregistry.io \
  --docker-username=user \
  --docker-password=pass \
  --docker-email=user@example.com

Service Account Token

Automatically created for service accounts

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
secrets:
- name: my-sa-token

Creating and Managing Secrets

Create Secret from Literals

# Create secret with literal values
kubectl create secret generic db-secret \
  --from-literal=username=dbuser \
  --from-literal=password='S3cur3P@ssw0rd'

# Create from files
kubectl create secret generic ssh-key-secret \
  --from-file=ssh-privatekey=/path/to/.ssh/id_rsa \
  --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub

Secret Manifest with stringData

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
# stringData allows plain text (converted to base64 automatically)
stringData:
  username: admin
  password: 'S3cur3P@ssw0rd'
  config.yaml: |
    apiUrl: https://api.example.com
    apiKey: abc123xyz789
    timeout: 30

Using Secrets in Pods

apiVersion: v1
kind: Pod
metadata:
  name: secret-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    env:
    # As environment variables
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: db-secret
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-secret
          key: password
    
    volumeMounts:
    # As files in a volume
    - name: secret-volume
      mountPath: "/etc/secrets"
      readOnly: true
    - name: ssh-keys
      mountPath: "/root/.ssh"
      readOnly: true
      
  volumes:
  - name: secret-volume
    secret:
      secretName: mysecret
      defaultMode: 0400  # Read-only for owner
  
  - name: ssh-keys
    secret:
      secretName: ssh-key-secret
      items:
      - key: ssh-privatekey
        path: id_rsa
        mode: 0600
      - key: ssh-publickey
        path: id_rsa.pub
        mode: 0644
  
  # For pulling images from private registry
  imagePullSecrets:
  - name: regcred

Security Warning

Remember that Secrets are only base64 encoded, NOT encrypted by default!

Enable encryption at rest in etcd
Use RBAC to limit Secret access
Consider using external secret management systems
Never commit Secrets to version control

Usage Patterns & Best Practices

Common Patterns

Application Configuration Pattern

# ConfigMap for app configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  application.yaml: |
    server:
      port: 8080
      host: 0.0.0.0
    
    features:
      cache: enabled
      rateLimit: 100
      
    logging:
      level: info
      format: json
---
# Secret for sensitive data
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  database.yaml: |
    host: db.example.com
    port: 5432
    username: appuser
    password: 'V3ryS3cur3P@ss'
    
  api-keys.yaml: |
    stripe: sk_live_xxxxx
    sendgrid: SG.xxxxx
    aws_access_key: AKIAXXXXX
    aws_secret_key: xxxxx
---
# Deployment using both
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: app
        image: myapp:latest
        volumeMounts:
        - name: config
          mountPath: /app/config
        - name: secrets
          mountPath: /app/secrets
          
      volumes:
      - name: config
        configMap:
          name: app-config
      - name: secrets
        secret:
          secretName: app-secrets
          defaultMode: 0400

Environment-Specific Configuration

# Base ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-base
data:
  APP_NAME: "MyApplication"
  LOG_FORMAT: "json"
---
# Development environment
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-dev
data:
  ENVIRONMENT: "development"
  API_URL: "https://api-dev.example.com"
  DEBUG: "true"
  LOG_LEVEL: "debug"
---
# Production environment
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-prod
data:
  ENVIRONMENT: "production"
  API_URL: "https://api.example.com"
  DEBUG: "false"
  LOG_LEVEL: "warn"
---
# Pod that combines configurations
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: myapp:latest
    envFrom:
    # Load base config first
    - configMapRef:
        name: app-config-base
    # Then environment-specific (can override base)
    - configMapRef:
        name: app-config-prod  # or app-config-dev

Advanced Patterns

Dynamic Configuration Reload

# Deployment with config reloader sidecar
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-with-reloader
spec:
  template:
    spec:
      containers:
      - name: app
        image: myapp:latest
        volumeMounts:
        - name: config
          mountPath: /etc/config
      
      # Sidecar to watch for config changes
      - name: config-reloader
        image: jimmidyson/configmap-reload:v0.5.0
        args:
        - --volume-dir=/etc/config
        - --webhook-url=http://localhost:8080/reload
        volumeMounts:
        - name: config
          mountPath: /etc/config
          
      volumes:
      - name: config
        configMap:
          name: app-config

Immutable ConfigMaps/Secrets

# Immutable ConfigMap (Kubernetes 1.21+)
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-v1
immutable: true  # Cannot be updated once created
data:
  config.yaml: |
    version: v1.0.0
    features:
      newFeature: enabled
---
# Immutable Secret
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets-v1
immutable: true
type: Opaque
data:
  apiKey: YXBpLWtleS12YWx1ZQ==

Best Practices

Version your configs: Use versioned names (app-config-v1, app-config-v2)
Use namespaces: Organize configs by namespace for isolation
Validate configs: Test configuration changes in staging first
Document structure: Add comments explaining configuration options
Keep configs small: Split large configs into multiple ConfigMaps
Use tools: Consider Helm, Kustomize for config management

Security Best Practices

Encryption at Rest

Enable etcd encryption for Secrets

RBAC Policies

Limit Secret access with RBAC

External Secrets

Use HashiCorp Vault, AWS Secrets Manager

Rotation

Regularly rotate sensitive credentials

RBAC for Secrets

Restrict Secret Access

# Role to read specific secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["app-secrets", "db-secrets"]
  verbs: ["get", "list"]
---
# RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-secrets
  namespace: production
subjects:
- kind: ServiceAccount
  name: app-service-account
  namespace: production
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

Sealed Secrets Pattern

Using Sealed Secrets

# Install sealed-secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml

# Create a secret
echo -n mypassword | kubectl create secret generic mysecret \
  --dry-run=client \
  --from-file=password=/dev/stdin \
  -o yaml > mysecret.yaml

# Seal the secret
kubeseal -f mysecret.yaml -w mysealedsecret.yaml

# The sealed secret can be stored in Git
cat mysealedsecret.yaml

External Secrets Operator

# SecretStore for AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-credentials
            key: access-key
          secretAccessKeySecretRef:
            name: aws-credentials
            key: secret-key
---
# ExternalSecret that syncs from AWS
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: aws-secret-store
    kind: SecretStore
  target:
    name: app-secrets
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: prod/database/password

Security Anti-Patterns to Avoid

❌ Storing secrets in ConfigMaps
❌ Hardcoding secrets in container images
❌ Using secrets in Pod commands/args (visible in ps)
❌ Logging secret values
❌ Committing secrets to version control
❌ Using default service account tokens unnecessarily

Hands-On Practice

Exercise 1: Multi-Environment Application

Deploy an application with different configurations for dev and prod environments.

Requirements:

Create base configuration shared across environments
Create environment-specific configurations
Store database credentials securely
Mount configuration files in the application
Implement configuration hot-reload capability

# base-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-base-config
data:
  APP_NAME: "MyApp"
  CACHE_TTL: "3600"
  MAX_CONNECTIONS: "100"
---
# dev-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-dev-config
data:
  ENVIRONMENT: "development"
  API_URL: "http://api-dev.local"
  DEBUG: "true"
  LOG_LEVEL: "debug"
---
# prod-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-prod-config
data:
  ENVIRONMENT: "production"
  API_URL: "https://api.example.com"
  DEBUG: "false"
  LOG_LEVEL: "info"
---
# secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:
  username: dbadmin
  password: "Sup3rS3cur3!"
  connection-string: "postgresql://dbadmin:Sup3rS3cur3!@db:5432/myapp"
---
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
    spec:
      containers:
      - name: app
        image: myapp:latest
        envFrom:
        - configMapRef:
            name: app-base-config
        - configMapRef:
            name: app-prod-config  # Change to app-dev-config for dev
        env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
        volumeMounts:
        - name: config-volume
          mountPath: /app/config
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
      volumes:
      - name: config-volume
        configMap:
          name: app-prod-config

Exercise 2: TLS Certificate Management

Configure HTTPS for your application using TLS certificates stored as Secrets.

Tasks:

Generate self-signed TLS certificates
Create a TLS Secret
Configure an Ingress with TLS
Mount certificates in a pod

# Generate self-signed certificate
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout tls.key -out tls.crt \
  -subj "/CN=myapp.example.com/O=myapp"

# Create TLS secret
kubectl create secret tls myapp-tls \
  --cert=tls.crt \
  --key=tls.key

# ingress-tls.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
spec:
  tls:
  - hosts:
    - myapp.example.com
    secretName: myapp-tls
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-service
            port:
              number: 80
---
# Pod mounting TLS certificates
apiVersion: v1
kind: Pod
metadata:
  name: tls-app
spec:
  containers:
  - name: app
    image: nginx:alpine
    volumeMounts:
    - name: tls-certs
      mountPath: /etc/tls
      readOnly: true
    - name: nginx-config
      mountPath: /etc/nginx/conf.d
  volumes:
  - name: tls-certs
    secret:
      secretName: myapp-tls
      items:
      - key: tls.crt
        path: server.crt
        mode: 0644
      - key: tls.key
        path: server.key
        mode: 0600
  - name: nginx-config
    configMap:
      name: nginx-ssl-config

Challenge: Implement Secret Rotation

Design a system for rotating database credentials without downtime.

Requirements:

Create versioned secrets
Implement graceful credential rotation
Use init containers for validation
Monitor secret usage
Automate the rotation process

ConfigMaps & Secrets

Configuration Management in Kubernetes

What are ConfigMaps?

What are Secrets?

Best Practice

Working with ConfigMaps

Creating ConfigMaps

Using ConfigMaps in Pods

ConfigMap Updates

Working with Secrets

Types of Secrets

Opaque Secrets

TLS Secrets

Docker Registry Secrets

Service Account Token

Creating and Managing Secrets

Security Warning

Usage Patterns & Best Practices

Common Patterns

Advanced Patterns

Best Practices

Security Best Practices

Encryption at Rest

RBAC Policies

External Secrets

Rotation

RBAC for Secrets

Sealed Secrets Pattern

Security Anti-Patterns to Avoid

Hands-On Practice

Exercise 1: Multi-Environment Application

Requirements:

Exercise 2: TLS Certificate Management

Tasks:

Challenge: Implement Secret Rotation

Requirements:

Additional Resources