Kubernetes Networking: Service Mesh and Ingress Controller Guide
Kubernetes networking complexity grows exponentially as microservices scale. While basic Services and DNS handle internal communication, production environments demand sophisticated traffic management, security policies, and observability. Service mesh and ingress controllers solve these challenges by providing advanced networking capabilities that traditional Kubernetes networking cannot address alone.
This guide explores practical implementation of service mesh architecture, ingress controllers, and network policies for production-ready Kubernetes deployments.
Understanding Kubernetes Network Architecture Layers
Kubernetes networking operates across multiple abstraction layers, each serving specific purposes in microservices communication.
Core Networking Components
The foundation consists of Pods, Services, and DNS. Each Pod receives a unique IP address within the cluster CIDR range, while Services provide stable endpoints for Pod groups. The cluster DNS (CoreDNS) enables service discovery through predictable naming conventions.
Network Policies control traffic flow between Pods using label selectors. By default, Kubernetes allows all Pod-to-Pod communication, making network policies essential for security.
CNI (Container Network Interface) plugins like Calico, Flannel, or Cilium handle the underlying network implementation, providing different features and performance characteristics.
Service Types and Use Cases
Kubernetes offers four Service types:
- ClusterIP: Internal cluster communication (default) - NodePort: Exposes services on node IP addresses - LoadBalancer: Cloud provider integration for external access - ExternalName: DNS-based service mapping
Most production workloads combine ClusterIP services for internal communication with ingress controllers for external traffic management.
Implementing Ingress Controllers for External Traffic
Ingress controllers manage external HTTP/HTTPS traffic routing to cluster services. Unlike LoadBalancer services that operate at Layer 4, ingress controllers provide Layer 7 routing with advanced features.
NGINX Ingress Controller Implementation
Here's a complete NGINX ingress setup with TLS termination:
`yaml
ingress-controller.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-ingress-controller namespace: ingress-nginx spec: replicas: 2 selector: matchLabels: app: nginx-ingress template: metadata: labels: app: nginx-ingress spec: containers: - name: nginx-ingress-controller image: k8s.gcr.io/ingress-nginx/controller:v1.8.1 ports: - containerPort: 80 - containerPort: 443 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace args: - /nginx-ingress-controller - --election-id=ingress-controller-leader - --controller-class=k8s.io/ingress-nginx - --configmap=$(POD_NAMESPACE)/nginx-configuration --- apiVersion: v1 kind: Service metadata: name: ingress-nginx-controller namespace: ingress-nginx spec: type: LoadBalancer ports: - name: http port: 80 targetPort: 80 - name: https port: 443 targetPort: 443 selector: app: nginx-ingress --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / nginx.ingress.kubernetes.io/ssl-redirect: "true" cert-manager.io/cluster-issuer: "letsencrypt-prod" spec: tls: - hosts: - api.example.com - web.example.com secretName: tls-secret rules: - host: api.example.com http: paths: - path: /api pathType: Prefix backend: service: name: api-service port: number: 8080 - host: web.example.com http: paths: - path: / pathType: Prefix backend: service: name: web-service port: number: 80Advanced Ingress Configuration
This configuration demonstrates several production features:
TLS Termination: Automatic certificate provisioning using cert-manager with Let's Encrypt integration. The cert-manager.io/cluster-issuer annotation triggers certificate creation.
Path-based Routing: Different URL paths route to appropriate backend services. The /api path forwards to the API service while root paths serve the web application.
SSL Redirect: Forces HTTPS connections for security compliance. The ssl-redirect annotation automatically redirects HTTP traffic to HTTPS.
High Availability: Multiple controller replicas ensure ingress availability during node failures or updates.
Service Mesh Architecture with Istio
Service mesh provides advanced traffic management, security, and observability for microservices communication. Istio leads the service mesh ecosystem with comprehensive features for production deployments.
Istio Installation and Configuration
`bash
Install Istio CLI
curl -L https://istio.io/downloadIstio | sh - export PATH=$PWD/istio-1.19.0/bin:$PATHInstall Istio with configuration profile
istioctl install --set values.defaultRevision=defaultEnable sidecar injection for namespace
kubectl label namespace production istio-injection=enabledTraffic Management with Virtual Services
`yaml
traffic-management.yaml
apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata: name: product-service namespace: production spec: hosts: - product-service http: - match: - headers: canary: exact: "true" route: - destination: host: product-service subset: canary weight: 100 - route: - destination: host: product-service subset: stable weight: 90 - destination: host: product-service subset: canary weight: 10 --- apiVersion: networking.istio.io/v1beta1 kind: DestinationRule metadata: name: product-service namespace: production spec: host: product-service trafficPolicy: connectionPool: tcp: maxConnections: 100 http: http1MaxPendingRequests: 50 maxRequestsPerConnection: 10 circuitBreaker: consecutiveErrors: 3 interval: 30s baseEjectionTime: 30s subsets: - name: stable labels: version: v1 trafficPolicy: portLevelSettings: - port: number: 8080 connectionPool: tcp: maxConnections: 50 - name: canary labels: version: v2 trafficPolicy: portLevelSettings: - port: number: 8080 connectionPool: tcp: maxConnections: 25Service Mesh Security Policies
`yaml
security-policies.yaml
apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: production spec: mtls: mode: STRICT --- apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: product-service-policy namespace: production spec: selector: matchLabels: app: product-service rules: - from: - source: principals: ["cluster.local/ns/production/sa/api-gateway"] - to: - operation: methods: ["GET", "POST"] paths: ["/api/products/*"] - when: - key: request.headers[user-role] values: ["admin", "user"] --- apiVersion: networking.istio.io/v1beta1 kind: ServiceEntry metadata: name: external-database namespace: production spec: hosts: - database.external.com ports: - number: 5432 name: postgres protocol: TCP location: MESH_EXTERNAL resolution: DNSNetwork Policy Implementation for Microsegmentation
Network policies provide microsegmentation capabilities, controlling traffic flow between Pods using Kubernetes-native resources.
Comprehensive Network Policy Example
`yaml
network-policies.yaml
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: web-tier-policy namespace: production spec: podSelector: matchLabels: tier: web policyTypes: - Ingress - Egress ingress: - from: - namespaceSelector: matchLabels: name: ingress-nginx - podSelector: matchLabels: app: nginx-ingress ports: - protocol: TCP port: 8080 egress: - to: - podSelector: matchLabels: tier: api ports: - protocol: TCP port: 8080 - to: - namespaceSelector: matchLabels: name: kube-system - podSelector: matchLabels: k8s-app: kube-dns ports: - protocol: UDP port: 53 --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: api-tier-policy namespace: production spec: podSelector: matchLabels: tier: api policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: tier: web ports: - protocol: TCP port: 8080 egress: - to: - podSelector: matchLabels: tier: database ports: - protocol: TCP port: 5432 - to: [] ports: - protocol: TCP port: 443 - protocol: UDP port: 53Network Policy Best Practices
Default Deny Strategy: Implement default deny policies for all namespaces, then explicitly allow required traffic. This zero-trust approach minimizes attack surface.
Granular Selectors: Use specific label selectors rather than broad namespace-level policies. This enables fine-grained control over service communication.
DNS Resolution: Always allow DNS traffic to kube-system namespace for service discovery functionality.
Monitoring Integration: Combine network policies with monitoring tools like Falco or Cilium Hubble for policy violation detection.
Production Deployment Strategies
Service Mesh Adoption Patterns
Progressive Rollout: Start with non-critical services to validate service mesh behavior. Gradually expand to critical workloads after establishing operational confidence.
Sidecar Resource Management: Configure appropriate CPU and memory limits for Envoy sidecars. Monitor resource consumption patterns to optimize cluster resource utilization.
Control Plane Scaling: Deploy Istio control plane components across multiple availability zones with adequate resource allocation for high-throughput environments.
Observability and Monitoring
Service mesh provides comprehensive telemetry data for traffic analysis, security monitoring, and performance optimization.
Distributed Tracing: Integrate with Jaeger or Zipkin for request flow visualization across microservices boundaries.
Metrics Collection: Configure Prometheus integration for service mesh metrics collection and alerting on SLA violations.
Access Logging: Enable detailed access logs for security analysis and compliance requirements.
Troubleshooting Common Issues
Ingress Controller Problems
Certificate Issues: Verify cert-manager RBAC permissions and DNS propagation for domain validation. Check certificate status using kubectl describe certificate.
Backend Connectivity: Ensure service endpoints exist and are healthy. Use kubectl get endpoints to verify service-to-pod mapping.
Configuration Syntax: Validate ingress annotations syntax and controller-specific requirements. Different controllers have varying annotation formats.
Service Mesh Debugging
Sidecar Injection Failures: Check namespace labels and admission controller status. Verify webhook configurations are properly installed.
mTLS Authentication Issues: Use istioctl proxy-config to inspect certificate distribution and validation policies.
Traffic Routing Problems: Analyze virtual service configurations and destination rule subset definitions for conflicts or misconfigurations.
Network Policy Troubleshooting
Policy Conflicts: Multiple network policies with overlapping selectors combine additively. Review all applicable policies for unintended restrictions.
CNI Compatibility: Ensure your CNI plugin supports network policies. Not all CNI implementations provide policy enforcement capabilities.
Testing Connectivity: Use network debugging tools like kubectl exec with curl or netcat to test connectivity between pods.
Performance Optimization
Ingress Controller Tuning
Optimize NGINX ingress controller for high-throughput scenarios:
- Increase worker processes and connections - Configure appropriate buffer sizes - Enable HTTP/2 and connection keep-alive - Implement rate limiting for DDoS protection
Service Mesh Performance
Envoy Configuration: Tune Envoy proxy settings for your workload patterns. Adjust connection pool sizes, circuit breaker thresholds, and timeout values.
Control Plane Optimization: Scale Istiod replicas based on service count and configuration change frequency. Monitor control plane resource utilization.
Telemetry Sampling: Configure appropriate tracing sample rates to balance observability with performance overhead.
Next Steps and Advanced Topics
Master these networking concepts through hands-on practice in development environments before production deployment. Consider exploring advanced topics:
- Multi-cluster Service Mesh: Implement cross-cluster communication with Istio multi-primary configurations - GitOps Integration: Automate network policy and service mesh configuration using ArgoCD or Flux - Custom Resources: Develop custom network policies using tools like Cilium's CNP (Cilium Network Policy) - Service Mesh Alternatives: Evaluate Linkerd, Consul Connect, or AWS App Mesh for specific use cases
Production Kubernetes networking requires careful planning, gradual implementation, and continuous monitoring. Start with basic ingress controllers and network policies, then progressively adopt service mesh capabilities as your microservices architecture matures.
Regular security audits, performance testing, and disaster recovery planning ensure robust networking infrastructure that scales with your organization's growth.