Service Discovery and Load Balancing

By Khimananda Oli | Last reviewed: August 2026

Modern distributed systems fail when clients cannot reliably locate healthy backend instances during scaling events or deployments. Effective service discovery and load balancing solve this by decoupling network endpoints from application logic, ensuring traffic always routes to available targets. This guide covers the architectural patterns, DNS mechanisms, and proxy configurations you need to build resilient infrastructure, building on foundational concepts like those in my Kubernetes basics deployment guide.

How does service discovery and load balancing differ from static DNS?

Traditional DNS relies on static A records that map hostnames to fixed IP addresses. In cloud-native environments where containers are ephemeral and IPs change frequently, static records become stale within seconds of a scaling event. Service discovery replaces this rigidity with a dynamic control plane that updates routing tables automatically.

The critical distinction lies in the feedback loop. Static DNS is fire-and-forget; the client assumes the returned IP is valid until TTL expiration. Dynamic discovery integrates health checks directly into the routing decision. When integrating this with infrastructure automation, tools discussed in Infrastructure as Code with Terraform can provision the underlying networking primitives, but the runtime discovery logic must be handled by platform-native components like CoreDNS, Consul, or Envoy.

Client-side vs server-side discovery

• Client-side: The service queries a registry (e.g., Eureka, etcd) directly and selects an instance. Common in Java/Spring ecosystems. Adds complexity to every client library.
• Server-side: The client sends requests to a known endpoint (VIP or DNS name), and an intermediary (NGINX, AWS ALB, Kubernetes Service) handles lookup and forwarding. Preferred for polyglot architectures since clients remain dumb.

How do you configure NGINX for dynamic upstream discovery?

NGINX Open Source caches DNS lookups at startup by default, which defeats dynamic discovery. To enable true runtime resolution, you must use variables in the proxy_pass directive, forcing NGINX to re-resolve on each request according to the resolver's TTL. This pattern is essential when running Laravel deployments on Ubuntu VPS with NGINX behind auto-scaling groups.

# /etc/nginx/conf.d/dynamic-upstream.conf
resolver 10.96.0.10 valid=5s ipv6=off;  # K8s CoreDNS or VPC DNS
set $backend_service "http://api-backend.namespace.svc.cluster.local:8080";

upstream api_pool {
    zone api_pool 64k;       # Shared memory for worker consistency
    least_conn;              # Better than round-robin for variable latency
}

server {
    listen 80;
    server_name api.example.com;

    location / {
        # Variable usage forces runtime resolution
        proxy_pass $backend_service;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        
        # Health-aware retries
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
        proxy_connect_timeout 2s;
    }
}

Note the zone directive: without shared memory, each NGINX worker maintains its own connection pool and DNS cache, leading to uneven distribution. The valid=5s parameter overrides any TTL from the DNS provider, giving you explicit control over propagation delay during blue-green deployments.

What are the trade-offs between Kubernetes Services and Ingress controllers?

Kubernetes provides built-in service discovery through ClusterIP Services and kube-proxy (or eBPF/Cilium). However, Layer 7 routing, TLS termination, and path-based splitting require an Ingress controller. Understanding where each layer operates prevents misconfiguration.

In practice, most production clusters use both: Ingress for external entry points with TLS and path routing, and ClusterIP Services for internal east-west traffic. If you're adopting GitOps for these configurations, GitOps with ArgoCD ensures your Ingress and Service manifests stay synchronized with application code changes.

How do health checks prevent cascading failures in load balancers?

A load balancer without active health checks is just a random failure generator. Passive checks (relying on HTTP 5xx responses) react too slowly; by the time errors surface, users have already experienced degraded performance. Active probes remove unhealthy targets before they receive traffic.

1. Define meaningful probe endpoints: Never use / or /health that returns 200 unconditionally. Create /readyz that verifies database connectivity, cache availability, and downstream dependencies.
2. Tune intervals conservatively: Start with 10s interval, 3s timeout, 3 consecutive failures for removal, and 2 successes for reinstatement. Aggressive settings cause flapping during GC pauses or brief network blips.
3. Separate liveness from readiness: Liveness determines if a container should restart; readiness determines if it should receive traffic. Conflating them causes unnecessary restarts during transient dependency outages.
4. Implement graceful shutdown: When SIGTERM arrives, immediately fail readiness probes, wait for in-flight requests to drain (typically 30s), then exit. This prevents 502s during rolling updates.

# Kubernetes readiness probe example
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3
  successThreshold: 2

When should you use service mesh instead of traditional load balancing?

Service meshes (Istio, Linkerd, Cilium) embed service discovery and load balancing into sidecar proxies, enabling mTLS, observability, and fine-grained traffic policies without application changes. However, they add operational complexity and latency overhead (~2-5ms per hop).

For teams in Nepal managing compliance-sensitive workloads (banking, healthtech), service mesh provides audit-friendly mTLS and policy-as-code. But for most web applications, traditional Ingress controllers with proper health checks deliver sufficient reliability at lower cognitive cost. Monitor your actual failure modes before investing in mesh infrastructure; premature abstraction slows delivery more than it prevents incidents.

Implementing Resilient Service Discovery and Load Balancing

Reliable service discovery and load balancing depend on three pillars: accurate health signaling, conservative tuning of discovery intervals, and layered redundancy between L4 and L7. Start with server-side discovery via Kubernetes Services or managed cloud LBs, add Ingress for HTTP concerns, and reserve service mesh for genuine zero-trust or observability gaps. Audit your configuration quarterly—stale resolver settings and missing readiness probes are the most common silent failures I encounter in production reviews. If your team needs help designing or validating this architecture, reach out to discuss your specific infrastructure requirements.