Patterns of Distributed Systems | Network Communication & Request Handling

Patterns of Distributed Systems

Based on: Catalog of Patterns of Distributed Systems by Unmesh Joshi (published on martinfowler.com, Nov 2023)

---

Table of Contents

1. Group F — Network Communication & Request Handling
   • Single-Socket Channel
   • Request Pipeline
   • Request Batch
   • Singular Update Queue
   • Request Waiting List
   • Idempotent Receiver

---

Group F — Network Communication & Request Handling

Reliable data structures and leader election aren't enough. Nodes must communicate efficiently and correctly over unreliable networks.

---

F1. Single-Socket Channel

One TCP connection per peer pair to preserve message order.

Problem

When a node sends multiple messages to another node using separate connections (or UDP), messages can arrive out of order. For many distributed protocols (like log replication), ordering is critical: entry 5 must be applied before entry 6.

Solution

Maintain one persistent TCP connection between each pair of communicating nodes. TCP guarantees in-order delivery within a single connection, so messages always arrive in the order sent.

How it works

Node A ◄═══ single TCP socket ═══► Node B
                                     (reused for all messages between A and B)

Message ordering:
  A sends: [msg1, msg2, msg3]
  B receives: [msg1, msg2, msg3]  ← guaranteed by TCP ✓

Connection lifecycle:

• Connection established at startup or on first message
• Reused indefinitely (with heartbeats to detect dead connections)
• On disconnect, reconnect and resume from the right position (using log indices)

Real-world analogy

A dedicated phone line between two offices used for all communications, rather than dialing a new number for each call. Everything said on that line is heard in order.

Example systems

ZooKeeper (single connection per client), Raft implementations, etcd

Trade-offs

Pro	Con
Guaranteed message ordering at no extra cost	Head-of-line blocking: a slow message delays all subsequent messages
Simple — no sequence numbering needed	Connection management complexity (reconnection, multiplexing)
Efficient (reuse TCP handshake cost)	Single socket limits throughput if messages are large

Related patterns

→ Request Pipeline (send multiple requests without waiting for responses) → HeartBeat (keeps the socket alive) → Singular Update Queue (processes messages from the socket serially)

---

F2. Request Pipeline

Don't wait for acknowledgement before sending the next request.

Problem

Using a Single-Socket Channel, if you send a request and wait for a response before sending the next request, your throughput is limited by the round-trip latency. If latency is 10ms and requests take 1ms to process, you can only do 100 requests/second even if the server could handle 10,000.

Solution

Pipeline multiple requests: send them one after another without waiting for the previous response. The server processes them in order (FIFO) and returns responses in the same order.

How it works

Without pipelining:
  Send Req1 → wait → Recv Resp1 → Send Req2 → wait → Recv Resp2
  Time = 2 × RTT per operation

With pipelining:
  Send Req1 → Send Req2 → Send Req3 → ...
                     → Recv Resp1 → Recv Resp2 → Recv Resp3
  Time ≈ 1 × RTT for many operations

The sender keeps an in-flight window (e.g., up to 100 outstanding requests). If the window fills up, it pauses until acknowledgements arrive.

Real-world analogy

An assembly line vs. building one product at a time. You start work on the next unit before the first is finished — the stages overlap.

Example systems

Redis pipelining, HTTP/2 multiplexing, Raft leader-to-follower replication

Trade-offs

Pro	Con
Dramatically improves throughput	Errors are harder to handle (which request failed?)
Reduces effective latency under load	Window sizing is tricky — too large can overwhelm receiver
Utilizes network bandwidth efficiently	Out-of-order responses (if using multiplexed channels) require correlation IDs

Related patterns

→ Single-Socket Channel (provides the ordered channel) → Request Batch (complementary throughput optimization) → Request Waiting List (tracks in-flight requests)

---

F3. Request Batch

Bundle multiple small requests into one large request.

Problem

Sending many small requests incurs high overhead: each request has network framing, serialization, and processing overhead. When throughput is critical (e.g., log replication), the overhead per message can dominate actual work time.

Solution

Accumulate multiple pending requests and send them as a single batch. The receiver processes the batch together and sends a single response for the batch.

How it works

Without batching (N small requests):
  [req1] → network → [resp1]
  [req2] → network → [resp2]
  [req3] → network → [resp3]
  Cost: N × (serialization + network overhead)

With batching:
  [req1 + req2 + req3] → network → [resp1 + resp2 + resp3]
  Cost: 1 × (serialization + network overhead) + 3 × processing

Batching strategies:

• Size-based: batch when accumulated size reaches N bytes
• Time-based: batch every T milliseconds (e.g., 5ms)
• Hybrid: batch when size OR time threshold is reached

Real-world analogy

A courier service: instead of sending 50 individual packages across the city one by one, fill a van and deliver them all in one trip.

Example systems

Kafka (producer batching), database write batching, Raft heartbeat piggybacking

Trade-offs

Pro	Con
Major throughput improvements	Increases latency for individual requests (must wait for batch to fill)
Reduces per-request overhead	More complex error handling (partial batch failures)
Efficient network utilization	Buffering introduces memory overhead

Related patterns

→ Request Pipeline (send batches in a pipeline) → Singular Update Queue (where batches are assembled) → Write-Ahead Log (batch writes to the log)

---

F4. Singular Update Queue

Process state changes sequentially using a single worker thread.

Problem

Multiple threads or network handlers updating shared state concurrently require locks, condition variables, and careful synchronization — a recipe for deadlocks, race conditions, and hard-to-test code.

Solution

Route all state-changing operations through a single queue processed by one dedicated thread. This thread processes one request at a time, eliminating all concurrency issues around shared state. Callers submit tasks to the queue and optionally wait for a result via a future/promise.

How it works

Thread 1 ─────► ┌─────────────────────┐
Thread 2 ─────► │    Request Queue    │ ◄──── Single Worker Thread
Thread 3 ─────► │  [req1, req2, req3] │          (processes serially)
                └─────────────────────┘

Network I/O and request parsing are still multi-threaded. Only state mutation is serialized through the queue.

This is the actor model in miniature.

Real-world analogy

A restaurant kitchen has one expediter who calls out orders one at a time to the kitchen staff. No two orders are processed simultaneously — each is completed before the next is started. Waiters (multiple threads) can still take orders concurrently.

Example systems

etcd (single Raft goroutine), ZooKeeper, Actor frameworks (Akka), Node.js event loop

Trade-offs

Pro	Con
Eliminates concurrency bugs on shared state	Single thread is a throughput ceiling
Dramatically simplified code — no locks needed	Queue can become a bottleneck under high load
Easy to reason about state transitions	Slow operations in the queue block everything

Related patterns

→ Request Waiting List (tracks async responses from the queue) → Request Batch (queue can batch work) → Single-Socket Channel (feeds the queue)

---

F5. Request Waiting List

Track in-flight requests waiting for responses from the cluster.

Problem

A server receives a client request and must wait for acknowledgements from multiple cluster nodes before it can respond. The server is asynchronous — it doesn't block waiting — so it must remember which client is waiting for which cluster responses.

Solution

Maintain a waiting list (a map) of in-flight client requests. Each entry stores:

• The original client request
• A callback or future to invoke when the required conditions are met (e.g., majority acks)
• A timeout

When cluster node responses arrive, match them to waiting list entries, check if the success criterion is met, and either respond to the client or trigger retry logic.

How it works

Client sends write request → Server creates WaitEntry {
  requestId: "abc123",
  requiredAcks: 3,  (majority of 5)
  receivedAcks: [],
  timeout: 5000ms,
  clientCallback: fn
}
Server stores WaitEntry in map["abc123"]

Follower1 acks → map["abc123"].receivedAcks.push(1) → count=1
Follower2 acks → map["abc123"].receivedAcks.push(2) → count=2
Follower3 acks → map["abc123"].receivedAcks.push(3) → count=3 = majority
  → clientCallback("SUCCESS") → respond to client
  → remove from waiting list

Real-world analogy

A restaurant manager tracks each table's order. When the kitchen finishes each dish, the manager checks if the whole table's order is complete. Only when all dishes are ready does the server bring them out.

Example systems

Raft leader implementations, ZooKeeper request processors, custom quorum write handlers

Trade-offs

Pro	Con
Non-blocking — server doesn't stall	Must handle timeouts explicitly
Clean separation of request receipt and response	Memory grows with in-flight requests
Natural place to implement retry and timeout logic	Correlating responses with requests requires unique IDs

Related patterns

→ Singular Update Queue (submits tasks that eventually respond via the waiting list) → Request Pipeline (multiple in-flight requests simultaneously) → Idempotent Receiver (handle retries when waiting list times out)

---

F6. Idempotent Receiver

Make servers safe to retry: process the same request twice = process it once.

Problem

Networks are unreliable. A client sends a request, the server processes it, but the response is lost. The client times out and retries. The server processes the request again — potentially double-charging a credit card, double-sending an email, or creating duplicate records.

Solution

Assign each request a unique client ID + request ID pair. The server tracks which requests it has already processed (in a deduplication store, usually persisted). On receiving a request, check if its ID has been seen before:

• If yes → return the cached response (don't re-execute)
• If no → process and store the result

How it works

Client:
  requestId = UUID()  ← unique per request attempt
  Send: {clientId: "client-1", requestId: "req-abc", data: "charge $100"}

Server:
  Check dedup_store["client-1"]["req-abc"]
  → Not found → process → charge $100 → response = "OK, charge ID=XYZ"
  → Store dedup_store["client-1"]["req-abc"] = "OK, charge ID=XYZ"
  → Return "OK, charge ID=XYZ"

Client retry (response was lost):
  Send again: {clientId: "client-1", requestId: "req-abc", data: "charge $100"}
  Server:
  → Found in dedup_store → return "OK, charge ID=XYZ" (no re-execution)

The dedup store has a TTL — old entries can be cleaned up once the client is done retrying.

Real-world analogy

A bank check with a unique check number. If you lose the confirmation that your check was cashed and submit it again, the bank recognizes the duplicate check number and rejects the second submission.

Example systems

AWS API Gateway (idempotency keys), Stripe (idempotency keys), Kafka exactly-once semantics, database UPSERT with unique constraints

Trade-offs

Pro	Con
Safe retries — clients can always retry on timeout	Requires persistent deduplication storage
Simplifies client logic (just retry on any failure)	Dedup store must be consulted on every request (overhead)
Enables exactly-once semantics	Old dedup entries must be cleaned up eventually

Related patterns

→ Request Waiting List (server retries internally; idempotency handles external retries) → Write-Ahead Log (dedup results are persisted here) → Generation Clock (can be used to detect outdated requests)

---