System Design on Noman Chowdhury Munna | Blog

Load Balancing Mastery: L4 vs L7, Algorithms, Session Persistence & Distributed Scaling

Sat, 22 Aug 2026 00:00:00 +0000

Welcome back to Systems Architecture. In our previous lecture, we placed Nginx in front of a single backend server as a reverse proxy. But what happens when your application explodes in popularity and a single backend machine can no longer keep up with incoming traffic?

Today, we dive into Load Balancing—the art and science of distributing network traffic across a pool of backend servers to achieve high availability, fault tolerance, and horizontal scalability.

1. The Scaling Dilemma: Vertical vs. Horizontal

When a single server reaches 100% CPU or RAM utilization, you have two options to scale:

1
2
3
4
5
6


VERTICAL SCALING (Scale Up) HORIZONTAL SCALING (Scale Out)
+------------------------+ +--------+ +--------+ +--------+
| BIGGER SERVER | | Node 1 | | Node 2 | | Node 3 |
| 128 Cores, 512GB RAM | | 4 Cores| | 4 Cores| | 4 Cores|
+------------------------+ +--------+ +--------+ +--------+
(Single point of failure, expensive) (Requires a Load Balancer)

Dimension	Vertical Scaling (Scale Up)	Horizontal Scaling (Scale Out)
Strategy	Add more CPU, RAM, or NVMe storage to your existing server.	Add more cheap commodity server instances to a pool.
Cost Curve	Non-linear / Exponentially expensive at high specs.	Linear cost curve (pay only for nodes added).
Hardware Limit	You eventually hit motherboard & CPU socket physical ceilings.	Virtually unlimited (scale to thousands of nodes).
Downtime	Upgrading RAM/CPU requires shutting down the machine.	Zero downtime (dynamically add/remove nodes).
Fault Tolerance	Zero. If the big server hardware fails, everything goes down.	High. If Node 1 crashes, Nodes 2 and 3 take the load.

Key Takeaway: Horizontal scaling is the foundation of modern cloud architecture. However, horizontal scaling requires a Load Balancer to sit in front of the pool and decide which server gets which request.

2. Where Does a Load Balancer Sit?

A Load Balancer (LB) sits between the clients (or reverse proxies) and the pool of worker instances:

1
2
3
4
5
6
7
8
9


[ Public Clients ]
 |
 v
[ External Load Balancer ] (e.g., AWS ALB / HAProxy)
 |
 +-------------------+-------------------+
 | | |
 v v v
 [ Backend App #1 ] [ Backend App #2 ] [ Backend App #3 ]

Is a Reverse Proxy the same as a Load Balancer?

Reverse Proxy: Primarily focused on acting on behalf of servers (TLS termination, header rewrite, caching, path routing).
Load Balancer: Primarily focused on traffic distribution across multiple targets.
Can one tool do both?: Absolutely! Tools like Nginx, HAProxy, and Traefik excel as both reverse proxies AND load balancers simultaneously.

3. Layer 4 vs. Layer 7 Load Balancing

Load balancers operate at different layers of the OSI model. Choosing between Layer 4 (L4) and Layer 7 (L7) is one of the most critical design choices in system architecture.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


OSI LAYER 4 (Transport Layer - TCP/UDP)
Client TCP Packet [SRC: 203.0.113.5:1234 -> DST: 1.2.3.4:443]
 |
 v L4 LB rewrites IP header (NAT / IPVS / eBPF)
Server TCP Packet [SRC: 203.0.113.5:1234 -> DST: 10.0.0.12:8000]
(LB does NOT inspect HTTP path, cookies, or payload)

-------------------------------------------------------------------------------

OSI LAYER 7 (Application Layer - HTTP/HTTPS)
Client Request: GET /api/v1/checkout HTTP/1.1 [Cookie: session=xyz]
 |
 v L7 LB decrypts TLS, parses HTTP headers, cookies, URL path
Forwarded to: Checkout Service Pool (10.0.0.12:8000)

Detailed Comparison: L4 vs. L7

Feature	Layer 4 Load Balancer (Transport)	Layer 7 Load Balancer (Application)
OSI Layer	Layer 4 (TCP / UDP).	Layer 7 (HTTP / HTTPS / gRPC / WebSockets).
Payload Visibility	Blind to application data (no HTTP headers, cookies, or URL paths).	Full visibility into HTTP headers, cookies, paths, and JSON bodies.
Performance / Throughput	Extremely fast (Millions of RPS). Minimal CPU overhead (packet routing in kernel).	Moderate (Hundreds of thousands of RPS). Requires CPU to parse HTTP/TLS.
Routing Capability	IP address & TCP/UDP port only.	Path-based (`/api`), Host-based (`api.com`), Header/Cookie routing.
TLS Handling	SSL Passthrough (no decryption).	TLS Termination (decryption & re-encryption).
Examples	AWS NLB, IPVS, HAProxy (TCP mode), F5 BIG-IP.	AWS ALB, Nginx, HAProxy (HTTP mode), Envoy.

4. Load Balancing Algorithms

How does the load balancer choose which backend server gets the next request?

1
2
3
4


1. ROUND-ROBIN 2. LEAST-CONNECTIONS 3. CONSISTENT HASHING
 Request 1 -> S1 Request -> S2 (Only 2 active) Hash(user_id=42) -> S3
 Request 2 -> S2 (S1 has 15, S3 has 8) (Always routes user 42 to S3)
 Request 3 -> S3

1. Round-Robin & Weighted Round-Robin

Standard Round-Robin: Requests are distributed sequentially across servers ($S_1 \rightarrow S_2 \rightarrow S_3 \rightarrow S_1$). Assumes all servers have equal capacity and processing times.
Weighted Round-Robin: Assigns weights to servers based on hardware spec ($S_1$: Weight 3, $S_2$: Weight 1). $S_1$ gets 3 requests for every 1 sent to $S_2$.

2. Least Connections

Directs incoming requests to the server with the lowest number of active open connections. Ideal for long-running requests (e.g., file processing or WebSocket streams) where connection durations vary wildly.

3. IP-Hash & Consistent Hashing

IP-Hash: Hashes the client IP address (hash(Client_IP) % N_servers). Ensures a specific client consistently hits the same backend.
Consistent Hashing: Hashes request keys onto a virtual ring. When adding or removing a server, only $\frac{1}{N}$ of keys need to be remapped, rather than re-shuffling the entire key space. Essential for distributed caching clusters (Memcached/Redis).

5. Health Checking & Failover Mechanics

What happens if Backend Server 2 crashes or suffers a hardware kernel panic?

1
2
3
4


[ Load Balancer ] --- (1. Active Health Probe GET /health) ---> [ Server 1: 200 OK ]
 --- (2. Active Health Probe GET /health) ---> [ Server 2: TIMEOUT ] ❌
 |
 [ LB Marks Server 2 DOWN & Evicts from Pool ] <------+

Active vs. Passive Health Checks

Active Health Check: The load balancer periodically (e.g., every 5s) sends explicit probe requests (GET /health) to each backend. If a server fails 3 consecutive checks (returns 5xx or times out), it is marked Unhealthy and evicted from the pool.
Passive Health Check: The LB monitors actual user traffic. If a backend fails 3 real client requests in a row, the LB temporarily suspends traffic to that instance.

Connection Draining (Graceful Shutdown)

When deploying new code or terminating an auto-scaled instance, you don’t abruptly kill the server. The LB enables Connection Draining:

LB stops sending new incoming requests to the target server.
LB allows existing active requests/connections a grace period (e.g., 30–60 seconds) to complete.
Once active connections drop to zero, the server is safely shut down.

6. Session Persistence: Sticky Sessions vs. Stateless Architecture

If a user logs in on Server 1, their session data is created. What happens when their next request lands on Server 2?

1
2
3
4
5
6
7
8
9


STATEFUL (Sticky Sessions - Anti-Pattern at Scale)
[ Client ] -> [ Load Balancer ] -> (Cookie says: Server 1) -> [ Server 1 (Holds Session in RAM) ]
Problem: If Server 1 dies, user session is lost! Unbalanced load across pool.

-------------------------------------------------------------------------------

STATELESS ARCHITECTURE (Recommended Production Pattern)
[ Client ] -> [ Load Balancer ] -> (Any Server 1, 2, or 3) -> Reads Session from [ Central Redis Cluster ]
Benefit: Any server can fail at any time without logging users out!

Sticky Sessions (Session Persistence)

The load balancer inspects a cookie or IP hash to bind a client to a specific backend server.

Drawbacks: Causes unequal server load (one heavy user can overload one server), breaks autoscaling, and user sessions are wiped if that specific server crashes.

The Stateless Architecture Solution

Store session state outside the application servers in a fast, distributed memory store like Redis or use self-contained JWT (JSON Web Tokens):

Backend app servers hold zero user state in local RAM.
Any request can hit any backend server interchangeably.
Servers can be added or destroyed dynamically without affecting connected users.

7. Global Load Balancing: Anycast, DNS LB & CDNs

At enterprise scale, load balancing isn’t just inside a single datacenter; it spans the planet.

1
2


[ User in Tokyo ] -------> [ Anycast IP / DNS Geo-Routing ] -------> [ Tokyo Datacenter ]
[ User in Frankfurt ] ---> [ Anycast IP / DNS Geo-Routing ] -------> [ Frankfurt Datacenter ]

DNS Round-Robin / Geo-DNS: DNS server returns different IP addresses based on the client’s geographic origin (resolving example.com to Tokyo IP for Asian users).
Anycast BGP Routing: Multiple datacenters across the globe advertise the exact same public IP address via Border Gateway Protocol (BGP). Internet routers automatically route client packets to the topologically closest datacenter!
CDN vs. Load Balancer: A CDN (Cloudflare, Fastly) sits in front of your load balancer. It caches static content (images, JS, CSS) at edge locations worldwide, absorbing up to 90% of traffic before it ever touches your primary load balancer.

Classroom Summary & Key Takeaways

Let’s review the main concepts:

Scaling: Horizontal scaling out-classes vertical scaling in resilience and cost.
L4 vs L7: L4 is blistering fast packet routing; L7 is smart HTTP path/cookie routing.
Algorithms: Use Least Connections for variable workloads, Consistent Hashing for caching.
Stateless Architecture: Never rely on sticky sessions in production—offload state to Redis or use JWTs.
High Availability: Combine active health checks with connection draining for zero-downtime deploys.

Next lecture: API Gateways, Resilience Patterns (Circuit Breakers & Retries), and Distributed Tracing. Class dismissed!

Production Web Systems Architecture & Real-World System Design Interview Scenarios

Mon, 24 Aug 2026 00:00:00 +0000

Welcome back, everyone. Today is our capstone lecture in Systems Architecture.

Over the past four lectures, we dissected individual components: sockets, web servers, reverse proxies, load balancers, and API gateways. Now comes the moment where true senior engineers stand out from juniors: Putting everything together into a unified, resilient, enterprise-grade production architecture.

Today, we will map out the complete modern web request pipeline from edge to database, trace how infrastructure evolves from a $5 VPS to a multi-region microservices cluster, and solve 35 tough real-world system design interview scenarios.

1. The Unified Web Architecture Blueprint

Where does every component actually sit in a production network? Here is the complete end-to-end blueprint:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


[ PUBLIC INTERNET ]
 |
 v
+-----------------------------------------------------------------------+
| 1. CLOUD EDGE / WAF & CDN (Cloudflare / AWS CloudFront) |
| - DDoS Protection, Web Application Firewall (WAF) |
| - Global Edge Caching (Images, JS, CSS, Static HTML) |
+-----------------------------------------------------------------------+
 |
 v (Public Anycast IP)
+-----------------------------------------------------------------------+
| 2. GLOBAL LAYER 4 LOAD BALANCER (AWS NLB / Anycast BGP) |
| - High-throughput TCP/UDP packet routing to healthy datacenters |
+-----------------------------------------------------------------------+
 |
 v (Private Subnet VPC)
+-----------------------------------------------------------------------+
| 3. LAYER 7 LOAD BALANCER / KUBERNETES INGRESS (Nginx / Envoy / ALB) |
| - TLS Termination & Path Routing |
+-----------------------------------------------------------------------+
 |
 v
+-----------------------------------------------------------------------+
| 4. API GATEWAY (Kong / APISix) |
| - JWT Authentication, Rate Limiting, API Key Quotas |
+-----------------------------------------------------------------------+
 |
 v
+-----------------------------------------------------------------------+
| 5. SERVICE MESH (Istio / Linkerd) & MICROSERVICES |
| - Sidecar Proxy mTLS, Circuit Breakers, Tracing (`X-Request-ID`) |
+-----------------------------------------------------------------------+
 |
 v
+-----------------------------------------------------------------------+
| 6. DATA & STATE TIER (PostgreSQL Primary/Replica, Redis Cluster) |
+-----------------------------------------------------------------------+

The Component Distinction Matrix

Component	Primary OSI Layer	Main Purpose	Can Nginx do this?
Web Server	Layer 7	Serves static assets off disk; handles raw HTTP protocols.	Yes (Native core)
Reverse Proxy	Layer 7	Hides backend IPs; handles TLS termination & header rewriting.	Yes (Native `proxy_pass`)
Load Balancer	Layer 4 or 7	Distributes incoming traffic across a pool of backend servers.	Yes (Native `upstream` block)
API Gateway	Layer 7	Edge authentication, rate limiting, request transformation, BFF.	Yes (With Lua/OpenResty/Modules)
Ingress Controller	Layer 7	Entry point for Kubernetes clusters mapping HTTP routes to K8s Services.	Yes (Nginx Ingress Controller)
Service Mesh	Layer 4 & 7	East-West (service-to-service) internal mTLS, telemetry & retries.	No (Handled by Envoy / Istio)

2. Evolutionary Architecture: Monolith to Microservices

You should never start a project with microservices, Kubernetes, and an API gateway on Day 1. That is over-engineering. Architecture must evolve as your user base and engineering team grow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


STAGE 1: Single VPS Monolith STAGE 2: Single VPS Docker Containers
[ Internet ] -> [ Nginx ] -> [ FastAPI ] [ Internet ] -> [ Nginx ] -> [ Next.js Container ]
 -> [ FastAPI Container ]

---------------------------------------------------------------------------------------------------

STAGE 3: Multi-VPS Load Balanced STAGE 4: Production Microservices Cluster
[ Internet ] [ Internet ] -> [ Cloudflare WAF/CDN ]
 | |
[ AWS ALB Load Balancer ] [ AWS NLB (L4) ]
 | |
 +--+--+ [ Nginx Ingress / Gateway ]
 | | |
[VPS 1] [VPS 2] (Stateless Monoliths) [ Istio Service Mesh ]
 | | [ 20 Microservices ] -> [ Redis / Postgres ]
 +--+--+
 v
 [ Central Redis & Postgres DB ]

When to introduce each component?

Reverse Proxy (Nginx): Day 1. Always put Nginx in front of any backend app.
Load Balancer: When a single VPS hits 70% CPU during peak hours or when you require 99.9% uptime (fault tolerance against server hardware failure).
API Gateway: When you split your monolith into 3+ independent microservices, or when mobile and web teams require separate client APIs (BFF pattern).

3. Real-World System Design & Operational Scenarios

Let’s work through 10 high-frequency system design and troubleshooting scenarios that senior engineers encounter in technical interviews and production post-mortems.

Scenario A: Zero-Downtime Deployments behind a Load Balancer

Question: How do you deploy a new backend version without dropping active user requests?

Answer: Use Blue-Green Deployment or Canary Release:

1
2
3
4
5
6
7


BLUE-GREEN DEPLOYMENT
[ Load Balancer ] ---> [ Blue Pool (v1.0 - Active 100%) ]
 [ Green Pool (v2.0 - Standby 0%) ]

Step 1: Deploy v2.0 to Green Pool & run health checks.
Step 2: Switch LB target group from Blue to Green instantly.
Step 3: Allow connection draining on Blue, then decommission v1.0.

Scenario B: Diagnosing 502 Bad Gateway vs. 504 Gateway Timeout

Question: Users report intermittent errors. How do you distinguish a 502 from a 504?

502 Bad Gateway: Nginx could not reach the backend socket (backend process crashed, out of memory OOM, or bound to wrong port).
- Fix: Check backend service status (systemctl status fastapi), verify socket permissions, inspect OS kernel OOM killer logs (dmesg -T).
504 Gateway Timeout: Nginx reached the backend, but the backend didn’t answer before proxy_read_timeout expired.
- Fix: Backend is suffering from slow DB queries, thread pool exhaustion, or external API blocking calls. Inspect database slow query logs and application trace spans.

Scenario C: Preventing Retry Storms & Duplicate Payments

Question: A user clicks “Pay $100”. Network stutters, the gateway times out, and the mobile app automatically retries 3 times. How do you prevent charging the user $300?

Answer: Enforce Idempotency Keys:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


[ Client ] -> Header: X-Idempotency-Key: 9f8a-1234 -> [ API Gateway / Payment Service ]
 |
 [ Check Redis for Key 9f8a-1234 ]
 |
 +----------------------+----------------------+
 | |
 [ Key Exists in Redis? ] [ New Key? ]
 | |
 Return Saved Payment Result Process Payment & Save
 (No double charge!) Result to Redis (TTL 24h)

Scenario D: Protecting `/login` from Brute-Force Attacks

Question: How do you protect authentication endpoints from automated password-guessing bots?

Answer: Implement Distributed Rate Limiting + Fail2ban at the Gateway:

Nginx / Gateway Rate Limit: Restrict /login requests to 5 req/min per IP using a Leaky Bucket directive (limit_req_zone).
Redis Distributed Counter: Track failed login attempts per username across all gateway nodes. If failed_attempts > 5 in 15 minutes, lock the account temporarily and trigger CAPTCHA verification.

Scenario E: What Happens When the Gateway or Load Balancer Dies?

Question: If the Load Balancer is the single entry point, how do you prevent it from being a Single Point of Failure (SPOF)?

Answer:

Cloud Environments: Use managed load balancers (AWS ALB/NLB) which run across multiple Availability Zones (AZs) with auto-healing managed control planes.
Bare-Metal / VPS: Run pairs of load balancers using VRRP / Keepalived with a shared Virtual IP (VIP), or use BGP Anycast routing across multiple datacenters.

Classroom Conclusion & Final Remarks

Congratulations, everyone! You have completed the Systems Architecture series.

Let’s summarize the journey we took together:

Level 0: Mastered Sockets, Ports, DNS, and Web vs Application Servers.
Level 1 & 2: Configured Nginx Reverse Proxies, TLS Termination, Header Forwarding, and Debugged 502/504 errors.
Level 3 & 4: Built Load Balancing strategies (L4 vs L7), Consistent Hashing, and Stateless Architectures with Redis.
Level 5 & 6: Designed API Gateways with Edge Auth, Token Bucket Rate Limiting, Circuit Breakers, and Distributed Tracing.
Level 7 & 8: Assembled End-to-End Enterprise Production Blueprints, Blue-Green deployments, and Idempotency patterns.

You now possess the theoretical foundation and practical blueprint required to design, build, and debug high-performance production web systems.

Go build incredible, resilient systems. Class dismissed!