Module 14: Fault Tolerance & Self-Healing Infrastructure

PREREQUISITE STATEMENT: Read this module after completing Module 13 (Edge Gateways). While Edge API Gateways control incoming traffic rates, internal network splits and database locks will still occur. This module teaches you how to design self-healing backend codebases that survive downstream failures without propagating outages.

---

1. Introduction: The Cascading Failure Mechanism

In a monolithic architecture, if a database query slows down, the thread pool blocks but stays within a single system process. In a distributed microservice architecture, calling another service over the network is an unreliable operation. If Service C (e.g., a payment gateway) experiences a latency spike, the upstream Service B (e.g., an order service) blocks its worker threads while waiting for Service C to respond. In turn, Service A (e.g., the API Gateway) blocks its sockets waiting for Service B, causing a cascading failure that can crash the entire system:

[ Client ] ---> [ API Gateway (Blocked) ] ---> [ Order Service (Thread Exhausted) ] ---> [ Payment Service (Hangs) ]

To build a fault-tolerant system, you must design for failure. Your software must isolate dependencies, fail fast when downstream services are unhealthy, and degrade gracefully to preserve core functionality.

---

2. Resiliency Patterns

To prevent cascading system failures, architects rely on four primary resiliency patterns:

                  [ Distributed Resiliency Patterns ]
                                   |
         +-------------------+-----+-------------------+
         |                   |                         |
  [ Circuit Breaker ]   [ Bulkheads ]         [ Exponential Backoff ]
  - Fail fast early    - Isolate pools       - Back off retries
  - Protect downstream - Prevent contagion   - Add random jitter

A. The Circuit Breaker Pattern

Inspired by electrical circuit breakers, this pattern prevents a service from repeatedly calling a downstream dependency that is highly likely to fail. The circuit breaker operates as a state machine with three primary states:

stateDiagram-v2
    [*] --> Closed : System Normal
    Closed --> Open : Failure Rate > Threshold
    Open --> HalfOpen : Cooldown Period Expired
    HalfOpen --> Closed : Trial Requests Succeed
    HalfOpen --> Open : Trial Request Fails

1. Closed State: Normal operation. Requests flow through to the downstream service. The circuit breaker monitors success/failure rates over a rolling time window (e.g., 100 requests).
2. Open State: When the failure rate exceeds a configured threshold (e.g., 50% of requests fail), the breaker trips. Subsequent requests fail fast immediately, returning a fallback value or error response without calling the downstream service, saving network resources.
3. Half-Open State: After a cooldown period (e.g., 30 seconds), the breaker enters Half-Open. It permits a limited number of trial requests to pass. If they succeed, the breaker resets to Closed. If any fail, it trips back to Open, restarting the cooldown timer.

---

B. Bulkheads

Named after the partition walls in ship hulls that prevent a single hull breach from sinking the entire vessel, the Bulkhead pattern isolates resources into dedicated, bounded pools:

           [ Unisolated Architecture ]
           Shared Thread Pool ---> [ Calling Service A ]
                              ---> [ Calling Service B (Blocked / Saturated) ]
                              
           [ Bulkhead Isolated Architecture ]
           Thread Pool A (10 Threads) ---> [ Service A (Healthy) ]
           Thread Pool B (10 Threads) ---> [ Service B (Saturated / Blocked) ]

• Thread Pool Bulkheads: Assign a dedicated pool of worker threads to each downstream dependency. If Service B hangs, it can only saturate its own thread pool (e.g. 10 threads), leaving Thread Pool A fully available to service requests for Service A.
• Semaphore Bulkheads: Limit the maximum number of concurrent requests allowed to a service. If the limit is reached, incoming requests are rejected immediately, preventing resource saturation.

---

C. Retries with Exponential Backoff and Jitter

When a network call fails due to a transient blip, retrying immediately can overload the recovering server, causing a retry storm.

Architects implement Exponential Backoff to increase the wait time between retries, combined with Jitter (randomness) to prevent synchronized retry waves:

// TypeScript Implementation of Retry with Exponential Backoff and Full Jitter
async function retryWithBackoff<T>(
  operation: () => Promise<T>,
  maxAttempts: number = 3,
  baseDelayMs: number = 100,
  maxDelayMs: number = 3000
): Promise<T> {
  let attempt = 0;
  
  while (true) {
    try {
      return await operation();
    } catch (error) {
      attempt++;
      if (attempt >= maxAttempts) {
        throw new Error(`Operation failed after ${attempt} attempts: ${error}`);
      }
      
      // Calculate exponential backoff delay: base * 2^attempt
      const exponentialDelay = baseDelayMs * Math.pow(2, attempt);
      const boundedDelay = Math.min(maxDelayMs, exponentialDelay);
      
      // Add Full Jitter: random delay between 0 and boundedDelay
      const jitterDelay = Math.random() * boundedDelay;
      
      console.log(`Attempt ${attempt} failed. Retrying in ${Math.round(jitterDelay)}ms...`);
      await new Promise((resolve) => setTimeout(resolve, jitterDelay));
    }
  }
}

---

D. Timeouts

Never allow a network request to wait indefinitely. Every remote call must define:

• Connect Timeout: The maximum time allowed to establish a TCP connection with the target server (typically 1–2 seconds).
• Read Timeout: The maximum time allowed for the target server to respond once the connection is established (typically 2–5 seconds, depending on the operation).

---

3. Service Mesh & Proxy-Level Resiliency

Historically, resiliency logic was implemented inside application libraries (e.g. Netflix Hystrix, Polly, Resilience4j). In modern architectures, these concerns are offloaded to Envoy Sidecar Proxies operating within a Service Mesh (such as Istio or Linkerd):

[ App Container ] <--- localhost ---> [ Envoy Proxy (Sidecar) ] <=== Network ===> [ Remote Envoy ]

Envoy intercepts all incoming and outgoing network traffic, applying timeouts, retries, and circuit breakers transparently without requiring application code changes.

Example: Istio VirtualService Retry Policy

Below is an Istio configuration file defining timeout and retry policies for a payment service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: payment-service-route
spec:
  hosts:
    - payment-service
  http:
    - route:
        - destination:
            host: payment-service
      timeout: 3.000s
      retries:
        attempts: 3
        perTryTimeout: 1.000s
        retryOn: "5xx,connect-failure,refused-stream"

---

4. Documentation Standard: High-Availability & Disaster Recovery Runbook

An enterprise-grade High-Availability & Disaster Recovery (HA/DR) Runbook defines the recovery objectives and procedures for critical system outages:

1. Key Recovery Objectives

• Recovery Time Objective (RTO): The maximum tolerable duration of service downtime before restoration (Target: $< 15\text{ minutes}$ for Tier-1 services).
• Recovery Point Objective (RPO): The maximum tolerable age of data that can be lost due to an outage (Target: $< 1\text{ minute}$ for transactions).

2. Triage & Incident Resolution Playbook

Trigger Event	Severity	Detection Metric	Automated Reaction	Manual Intervention Step
Downstream Payment API Timeout	P2	HTTP 504 errors on /checkout $> 5%$ over 1 min.	Circuit breaker trips to Open state; returns cached checkout or queue-for-later status.	Notify payment gateway partner support team; monitor outbox database table size.
Primary Database Node Crash	P1	Postgres connection health check failure.	Sentinel failover promotes standby replica database node to primary; routes traffic to new IP.	Verify replication lag of promoted primary database node; trigger data integrity check.
Redis Cache CPU Saturation	P2	Redis container CPU utilization $> 90%$.	Circuit breaker disables cache-aside updates; queries fall back to secondary read DB replicas.	Analyze Redis command log for unindexed search queries; scale Redis cluster size.

---

5. Hands-on Architecture Challenge

Scenario Description

A microservice architecture suffers from cascading failures. When a downstream dependency fails or slows down, the upstream service blocks its connection thread pool waiting for replies, exhausting local server resources. You must model a complete Circuit Breaker state machine.

Your Goal:

1. Define the three states: Closed, Open, and HalfOpen.
2. Connect them with appropriate state transition triggers:
   • Closed $\rightarrow$ Open (Trigger: Failure Rate > Threshold).
   • Open $\rightarrow$ HalfOpen (Trigger: Cooldown Timeout Expired).
   • HalfOpen $\rightarrow$ Closed (Trigger: Trial Requests Succeed).
   • HalfOpen $\rightarrow$ Open (Trigger: Trial Request Fails).
3. Model this state logic using the diagram editor's stateDiagram-v2 syntax.

---

6. Practice Challenge Template

Use this template in your sandbox to model the circuit breaker state machine:

stateDiagram-v2
    [*] --> Closed : Start Normal
    
    state Closed {
        [*] --> MonitorFailures
        MonitorFailures --> MonitorFailures : Request Succeeds
    }

    state Open {
        [*] --> RejectRequests
        RejectRequests --> RejectRequests : Fail Fast Return Fallback
    }

    state HalfOpen {
        [*] --> SendTrialRequests
        SendTrialRequests --> SendTrialRequests : Trial succeeds
    }

    Closed --> Open : Failure Rate > Threshold (Trip Circuit)
    Open --> HalfOpen : Cooldown Duration Expired (Cooldown Timeout)
    HalfOpen --> Closed : Trial Requests Succeed (Reset Circuit)
    HalfOpen --> Open : Trial Request Fails (Re-trip Circuit)

---

NEXT MODULE BRIDGE: Designing fault-tolerant system boundaries protects your microservices during runtime, but the migration paths for brownfield legacy monoliths present different operational constraints. Proceed to Module 15: Environmental Assessment (Greenfield vs. Brownfield) to discover how to safely migrate monolithic systems using the Strangler Fig and Anti-Corruption Layer patterns.