Kubernetes Service Mesh - Advanced Traffic Management

Service Mesh Architecture

Core Components

Data Plane

Proxy sidecars handling actual network traffic

Control Plane

Configuration and policy management

Observability

Metrics, logs, and distributed tracing

Security

mTLS, authentication, and authorization

Why Service Mesh?

Traffic Management

Advanced routing, load balancing, circuit breaking, and retries without application changes.

# Traffic splitting example
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
spec:
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 75
    - destination:
        host: reviews
        subset: v2
      weight: 25

Security

Automatic mTLS, fine-grained access control, and encryption at rest.

# Authorization policy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: httpbin
spec:
  selector:
    matchLabels:
      app: httpbin
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/default/sa/sleep"]
    to:
    - operation:
        methods: ["GET"]
        paths: ["/info*"]

Istio Service Mesh

Installation and Setup

# Download and install Istio
curl -L https://istio.io/downloadIstio | sh -
cd istio-*
export PATH=$PWD/bin:$PATH

# Install Istio with demo profile
istioctl install --set profile=demo -y

# Enable sidecar injection
kubectl label namespace default istio-injection=enabled

# Verify installation
kubectl get pods -n istio-system
istioctl verify-install

Core Features

Traffic Management

# Destination Rule
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: productpage
spec:
  host: productpage
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 100
        h2UpgradePolicy: UPGRADE
    loadBalancer:
      simple: LEAST_REQUEST
  subsets:
  - name: v1
    labels:
      version: v1
    trafficPolicy:
      connectionPool:
        tcp:
          maxConnections: 10

Circuit Breaking

# Circuit breaker configuration
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutiveErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100
      minHealthPercent: 0

Advanced Patterns

Canary Deployment

# Canary deployment with Istio
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: helloworld
spec:
  hosts:
  - helloworld
  http:
  - match:
    - headers:
        canary:
          exact: "true"
    route:
    - destination:
        host: helloworld
        subset: v2
  - route:
    - destination:
        host: helloworld
        subset: v1
      weight: 90
    - destination:
        host: helloworld
        subset: v2
      weight: 10

Request Routing

# Content-based routing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        cookie:
          regex: "^(.*?;)?(user=jason)(;.*)?$"
    route:
    - destination:
        host: reviews
        subset: v3
  - match:
    - queryParams:
        test:
          exact: "true"
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v1

Linkerd Service Mesh

Installation

# Install Linkerd CLI
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

# Add to PATH
export PATH=$PATH:$HOME/.linkerd2/bin

# Validate cluster
linkerd check --pre

# Install control plane
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -

# Verify installation
linkerd check

# Install viz extension
linkerd viz install | kubectl apply -f -

Linkerd Features

Automatic mTLS

# Inject Linkerd proxy
kubectl get deploy -o yaml | \
  linkerd inject - | \
  kubectl apply -f -

# Check mTLS status
linkerd viz edges deployment

# View TLS certificates
linkerd identity cert

Traffic Splitting

# Traffic split with SMI
apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
  name: web-svc-split
spec:
  service: web-svc
  backends:
  - service: web-svc-v1
    weight: 500
  - service: web-svc-v2
    weight: 500

Service Profiles

# Service profile for webapp
apiVersion: linkerd.io/v1alpha2
kind: ServiceProfile
metadata:
  name: webapp.default.svc.cluster.local
  namespace: default
spec:
  routes:
  - name: GET
    condition:
      method: GET
      pathRegex: "/"
    responseClasses:
    - condition:
        status:
          min: 200
          max: 299
      isFailure: false
  - name: POST
    condition:
      method: POST
      pathRegex: "/api"
    timeout: 30s
    retryBudget:
      retryRatio: 0.2
      minRetriesPerSecond: 10
      ttl: 10s
  retryBudget:
    retryRatio: 0.2
    minRetriesPerSecond: 10
    ttl: 10s

Traffic Management Patterns

Load Balancing Strategies

Round Robin

# Istio round robin
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-service
spec:
  host: my-service
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN

Consistent Hash

# Session affinity
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-service
spec:
  host: my-service
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpCookie:
          name: session-cookie
          ttl: 3600s

Resilience Patterns

Retry Configuration

# Retry policy
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - timeout: 10s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: gateway-error,connect-failure,refused-stream
      retryRemoteLocalities: true
    route:
    - destination:
        host: ratings
        subset: v1

Fault Injection

# Inject faults for testing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - fault:
      delay:
        percentage:
          value: 10
        fixedDelay: 5s
      abort:
        percentage:
          value: 10
        httpStatus: 500
    route:
    - destination:
        host: ratings
        subset: v1

Rate Limiting

# Envoy rate limiting
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: rate-limit
spec:
  workloadSelector:
    labels:
      app: productpage
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.local_ratelimit
        typed_config:
          "@type": type.googleapis.com/udpa.type.v1.TypedStruct
          type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
          value:
            stat_prefix: http_local_rate_limiter
            token_bucket:
              max_tokens: 100
              tokens_per_fill: 100
              fill_interval: 60s
            filter_enabled:
              runtime_key: local_rate_limit_enabled
              default_value:
                numerator: 100
                denominator: HUNDRED
            filter_enforced:
              runtime_key: local_rate_limit_enforced
              default_value:
                numerator: 100
                denominator: HUNDRED

Observability

Distributed Tracing

# Enable tracing in Istio
apiVersion: v1
data:
  mesh: |
    defaultConfig:
      proxyStatsMatcher:
        inclusionRegexps:
        - ".*outlier_detection.*"
        - ".*osconfig.*"
      tracing:
        zipkin:
          address: zipkin.istio-system:9411
        sampling: 100.0
kind: ConfigMap
metadata:
  name: istio
  namespace: istio-system

# Deploy Jaeger
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.14/samples/addons/jaeger.yaml

# Access Jaeger UI
istioctl dashboard jaeger

Metrics and Monitoring

# Prometheus metrics
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: istio-system
spec:
  selector:
    app: prometheus
  ports:
  - name: http
    port: 9090

# Custom metrics
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: custom-metrics
spec:
  metrics:
  - providers:
    - name: prometheus
    dimensions:
      request_protocol: request.protocol | "unknown"
      response_code: response.code | 200
    name: request_total
    unit: REQUEST
    value: "1"

Service Mesh Dashboard

# Install Kiali
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.14/samples/addons/kiali.yaml

# Access Kiali dashboard
istioctl dashboard kiali

# Grafana dashboards
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.14/samples/addons/grafana.yaml
istioctl dashboard grafana

Service Mesh Security

Mutual TLS (mTLS)

Strict mTLS

# Enable strict mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

# Per-namespace mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT

Certificate Management

# External certificate
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: external-mtls
spec:
  selector:
    matchLabels:
      app: external-service
  mtls:
    mode: STRICT
    # Use external certs
    caCertificates: /etc/ssl/certs/ca-cert.pem

Authorization Policies

# RBAC with JWT
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-auth
  namespace: default
spec:
  selector:
    matchLabels:
      app: httpbin
  jwtRules:
  - issuer: "testing@secure.istio.io"
    jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.14/security/tools/jwt/samples/jwks.json"

---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: require-jwt
  namespace: default
spec:
  selector:
    matchLabels:
      app: httpbin
  action: ALLOW
  rules:
  - from:
    - source:
        requestPrincipals: ["testing@secure.istio.io/testing@secure.istio.io"]
    when:
    - key: request.auth.claims[groups]
      values: ["group1"]

⚠️ Security Best Practices

Always enable mTLS in production
Use namespace isolation
Implement defense in depth
Regular certificate rotation
Audit all authorization policies

Service Mesh Comparison

Feature	Istio	Linkerd	Consul Connect	AWS App Mesh
Architecture	Envoy-based	Rust micro-proxy	Envoy/Native	Envoy-based
Performance	High overhead	Low overhead	Medium overhead	Medium overhead
Complexity	High	Low	Medium	Medium
Multi-cluster	Excellent	Good	Excellent	Good (AWS)
Protocol Support	HTTP, gRPC, TCP	HTTP, gRPC	HTTP, gRPC, TCP	HTTP, gRPC, TCP
mTLS	Automatic	Automatic	Automatic	Manual config
Observability	Comprehensive	Built-in dashboard	Good	CloudWatch/X-Ray
Resource Usage	~500MB per sidecar	~50MB per sidecar	~100MB per sidecar	~200MB per sidecar

When to Use Each

Istio

Complex traffic management needs
Multi-cluster deployments
Advanced security requirements
Rich observability features

Linkerd

Simple, lightweight mesh
Low resource overhead critical
Getting started with service mesh
Focus on reliability

Consul Connect

Multi-cloud deployments
Existing Consul users
Service discovery focus
VM and container workloads

AWS App Mesh

AWS-native deployments
EKS integration
Managed service preference
CloudWatch integration

Migration Strategies

Adopting Service Mesh

Gradual Rollout Strategy

# 1. Start with observability only
kubectl label namespace dev istio-injection=enabled

# 2. Enable permissive mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: PERMISSIVE

# 3. Gradually add traffic management
# 4. Enable strict mTLS when ready
# 5. Implement authorization policies

Common Challenges

Migration Pitfalls

Performance Impact: Test sidecar overhead in staging
Debugging Complexity: Prepare team with training
Configuration Drift: Use GitOps for consistency
Protocol Support: Verify all protocols work
Egress Traffic: Plan external service access

← Back to Kubernetes Overview