Welcome back to Systems Architecture. In our previous lecture, we placed Nginx in front of a single backend server as a reverse proxy. But what happens when your application explodes in popularity and a single backend machine can no longer keep up with incoming traffic?

Today, we dive into Load Balancing—the art and science of distributing network traffic across a pool of backend servers to achieve high availability, fault tolerance, and horizontal scalability.

1. The Scaling Dilemma: Vertical vs. Horizontal

When a single server reaches 100% CPU or RAM utilization, you have two options to scale:

1
2
3
4
5
6

VERTICAL SCALING (Scale Up)               HORIZONTAL SCALING (Scale Out)
+------------------------+                +--------+  +--------+  +--------+
| BIGGER SERVER          |                | Node 1 |  | Node 2 |  | Node 3 |
| 128 Cores, 512GB RAM   |                | 4 Cores|  | 4 Cores|  | 4 Cores|
+------------------------+                +--------+  +--------+  +--------+
(Single point of failure, expensive)      (Requires a Load Balancer)

Key Takeaway: Horizontal scaling is the foundation of modern cloud architecture. However, horizontal scaling requires a Load Balancer to sit in front of the pool and decide which server gets which request.

2. Where Does a Load Balancer Sit?

A Load Balancer (LB) sits between the clients (or reverse proxies) and the pool of worker instances:

1
2
3
4
5
6
7
8
9

[ Public Clients ]
        |
        v
[ External Load Balancer ] (e.g., AWS ALB / HAProxy)
        |
        +-------------------+-------------------+
        |                   |                   |
        v                   v                   v
 [ Backend App #1 ]  [ Backend App #2 ]  [ Backend App #3 ]

Is a Reverse Proxy the same as a Load Balancer?

• Reverse Proxy: Primarily focused on acting on behalf of servers (TLS termination, header rewrite, caching, path routing).
• Load Balancer: Primarily focused on traffic distribution across multiple targets.
• Can one tool do both?: Absolutely! Tools like Nginx, HAProxy, and Traefik excel as both reverse proxies AND load balancers simultaneously.

3. Layer 4 vs. Layer 7 Load Balancing

Load balancers operate at different layers of the OSI model. Choosing between Layer 4 (L4) and Layer 7 (L7) is one of the most critical design choices in system architecture.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

OSI LAYER 4 (Transport Layer - TCP/UDP)
Client TCP Packet [SRC: 203.0.113.5:1234 -> DST: 1.2.3.4:443]
      |
      v  L4 LB rewrites IP header (NAT / IPVS / eBPF)
Server TCP Packet [SRC: 203.0.113.5:1234 -> DST: 10.0.0.12:8000]
(LB does NOT inspect HTTP path, cookies, or payload)

-------------------------------------------------------------------------------

OSI LAYER 7 (Application Layer - HTTP/HTTPS)
Client Request: GET /api/v1/checkout HTTP/1.1  [Cookie: session=xyz]
      |
      v  L7 LB decrypts TLS, parses HTTP headers, cookies, URL path
Forwarded to: Checkout Service Pool (10.0.0.12:8000)

Detailed Comparison: L4 vs. L7

4. Load Balancing Algorithms

How does the load balancer choose which backend server gets the next request?

1
2
3
4

1. ROUND-ROBIN          2. LEAST-CONNECTIONS          3. CONSISTENT HASHING
    Request 1 -> S1          Request -> S2 (Only 2 active)    Hash(user_id=42) -> S3
    Request 2 -> S2          (S1 has 15, S3 has 8)            (Always routes user 42 to S3)
    Request 3 -> S3

1. Round-Robin & Weighted Round-Robin

• Standard Round-Robin: Requests are distributed sequentially across servers ($S_1 \rightarrow S_2 \rightarrow S_3 \rightarrow S_1$). Assumes all servers have equal capacity and processing times.
• Weighted Round-Robin: Assigns weights to servers based on hardware spec ($S_1$: Weight 3, $S_2$: Weight 1). $S_1$ gets 3 requests for every 1 sent to $S_2$.

2. Least Connections

Directs incoming requests to the server with the lowest number of active open connections. Ideal for long-running requests (e.g., file processing or WebSocket streams) where connection durations vary wildly.

3. IP-Hash & Consistent Hashing

• IP-Hash: Hashes the client IP address (hash(Client_IP) % N_servers). Ensures a specific client consistently hits the same backend.
• Consistent Hashing: Hashes request keys onto a virtual ring. When adding or removing a server, only $\frac{1}{N}$ of keys need to be remapped, rather than re-shuffling the entire key space. Essential for distributed caching clusters (Memcached/Redis).

5. Health Checking & Failover Mechanics

What happens if Backend Server 2 crashes or suffers a hardware kernel panic?

1
2
3
4

[ Load Balancer ] --- (1. Active Health Probe GET /health) ---> [ Server 1: 200 OK ]
                  --- (2. Active Health Probe GET /health) ---> [ Server 2: TIMEOUT ] ❌
                                                                       |
                  [ LB Marks Server 2 DOWN & Evicts from Pool ] <------+

Active vs. Passive Health Checks

• Active Health Check: The load balancer periodically (e.g., every 5s) sends explicit probe requests (GET /health) to each backend. If a server fails 3 consecutive checks (returns 5xx or times out), it is marked Unhealthy and evicted from the pool.
• Passive Health Check: The LB monitors actual user traffic. If a backend fails 3 real client requests in a row, the LB temporarily suspends traffic to that instance.

Connection Draining (Graceful Shutdown)

When deploying new code or terminating an auto-scaled instance, you don’t abruptly kill the server. The LB enables Connection Draining:

1. LB stops sending new incoming requests to the target server.
2. LB allows existing active requests/connections a grace period (e.g., 30–60 seconds) to complete.
3. Once active connections drop to zero, the server is safely shut down.

6. Session Persistence: Sticky Sessions vs. Stateless Architecture

If a user logs in on Server 1, their session data is created. What happens when their next request lands on Server 2?

1
2
3
4
5
6
7
8
9

STATEFUL (Sticky Sessions - Anti-Pattern at Scale)
[ Client ] -> [ Load Balancer ] -> (Cookie says: Server 1) -> [ Server 1 (Holds Session in RAM) ]
Problem: If Server 1 dies, user session is lost! Unbalanced load across pool.

-------------------------------------------------------------------------------

STATELESS ARCHITECTURE (Recommended Production Pattern)
[ Client ] -> [ Load Balancer ] -> (Any Server 1, 2, or 3) -> Reads Session from [ Central Redis Cluster ]
Benefit: Any server can fail at any time without logging users out!

Sticky Sessions (Session Persistence)

The load balancer inspects a cookie or IP hash to bind a client to a specific backend server.

• Drawbacks: Causes unequal server load (one heavy user can overload one server), breaks autoscaling, and user sessions are wiped if that specific server crashes.

The Stateless Architecture Solution

Store session state outside the application servers in a fast, distributed memory store like Redis or use self-contained JWT (JSON Web Tokens):

• Backend app servers hold zero user state in local RAM.
• Any request can hit any backend server interchangeably.
• Servers can be added or destroyed dynamically without affecting connected users.

7. Global Load Balancing: Anycast, DNS LB & CDNs

At enterprise scale, load balancing isn’t just inside a single datacenter; it spans the planet.

1
2

[ User in Tokyo ] -------> [ Anycast IP / DNS Geo-Routing ] -------> [ Tokyo Datacenter ]
[ User in Frankfurt ] ---> [ Anycast IP / DNS Geo-Routing ] -------> [ Frankfurt Datacenter ]

• DNS Round-Robin / Geo-DNS: DNS server returns different IP addresses based on the client’s geographic origin (resolving example.com to Tokyo IP for Asian users).
• Anycast BGP Routing: Multiple datacenters across the globe advertise the exact same public IP address via Border Gateway Protocol (BGP). Internet routers automatically route client packets to the topologically closest datacenter!
• CDN vs. Load Balancer: A CDN (Cloudflare, Fastly) sits in front of your load balancer. It caches static content (images, JS, CSS) at edge locations worldwide, absorbing up to 90% of traffic before it ever touches your primary load balancer.

Classroom Summary & Key Takeaways

Let’s review the main concepts:

• Scaling: Horizontal scaling out-classes vertical scaling in resilience and cost.
• L4 vs L7: L4 is blistering fast packet routing; L7 is smart HTTP path/cookie routing.
• Algorithms: Use Least Connections for variable workloads, Consistent Hashing for caching.
• Stateless Architecture: Never rely on sticky sessions in production—offload state to Redis or use JWTs.
• High Availability: Combine active health checks with connection draining for zero-downtime deploys.

Next lecture: API Gateways, Resilience Patterns (Circuit Breakers & Retries), and Distributed Tracing. Class dismissed!