Building Resilient APIs: Circuit Breakers, Retries, and Rate Limiting in Production

APIs fail. Networks timeout. Services go down. In production, the question isn’t if something will fail, but when — and how your system responds. Resilient APIs don’t just survive failures; they degrade gracefully, recover automatically, and protect downstream services from cascading outages.

This post covers the patterns I use to build APIs that handle failure gracefully: circuit breakers, retry strategies with exponential backoff, rate limiting, and fallback mechanisms.

The Problem: Cascading Failures

Without resilience patterns, a single slow service can bring down your entire system:

Service A → Service B → Service C (slow)
                ↓
           Thread pool exhausted
                ↓
           Service A also fails
                ↓
           Cascading outage

The solution is to fail fast, limit retries, and isolate failures.

Circuit Breaker Pattern

A circuit breaker prevents cascading failures by stopping calls to a failing service:

type CircuitState = 'CLOSED' | 'OPEN' | 'HALF_OPEN';

interface CircuitBreakerConfig {
  failureThreshold: number;      // Failures before opening
  successThreshold: number;      // Successes to close from half-open
  timeout: number;               // Time in open state before half-open (ms)
  resetTimeout: number;          // Time to wait before retrying in half-open
}

class CircuitBreaker {
  private state: CircuitState = 'CLOSED';
  private failures: number = 0;
  private successes: number = 0;
  private lastFailureTime: number = 0;

  constructor(
    private name: string,
    private config: CircuitBreakerConfig
  ) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'OPEN') {
      if (Date.now() - this.lastFailureTime >= this.config.timeout) {
        this.state = 'HALF_OPEN';
        this.successes = 0;
      } else {
        throw new Error(`Circuit breaker [${this.name}] is OPEN`);
      }
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess(): void {
    this.failures = 0;

    if (this.state === 'HALF_OPEN') {
      this.successes++;
      if (this.successes >= this.config.successThreshold) {
        this.state = 'CLOSED';
        this.successes = 0;
      }
    }
  }

  private onFailure(): void {
    this.failures++;
    this.lastFailureTime = Date.now();

    if (this.state === 'HALF_OPEN') {
      this.state = 'OPEN';
    } else if (this.failures >= this.config.failureThreshold) {
      this.state = 'OPEN';
    }
  }

  getState(): CircuitState {
    return this.state;
  }

  getStats() {
    return {
      name: this.name,
      state: this.state,
      failures: this.failures,
      successes: this.successes,
    };
  }
}

// Usage
const paymentServiceBreaker = new CircuitBreaker('payment-service', {
  failureThreshold: 5,
  successThreshold: 3,
  timeout: 30000,
  resetTimeout: 5000,
});

async function processPayment(payment: Payment): Promise<PaymentResult> {
  return paymentServiceBreaker.execute(async () => {
    const response = await fetch('https://payment-api.example.com/charge', {
      method: 'POST',
      body: JSON.stringify(payment),
    });

    if (!response.ok) {
      throw new Error(`Payment failed: ${response.status}`);
    }

    return response.json();
  });
}

Circuit Breaker States

State	Behavior
CLOSED	Normal operation, requests pass through
OPEN	Requests fail immediately without calling the service
HALF_OPEN	Limited requests pass through to test if service recovered

Retry with Exponential Backoff

Not all failures are permanent. Network blips and temporary overloads often resolve with a retry. But mindless retries make problems worse — you need exponential backoff with jitter:

interface RetryConfig {
  maxAttempts: number;
  baseDelay: number;        // Initial delay in ms
  maxDelay: number;         // Maximum delay in ms
  jitter: boolean;          // Add randomness to prevent thundering herd
  retryableErrors: string[]; // Error messages/patterns that should trigger retry
}

class RetryManager {
  constructor(private config: RetryConfig) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    let lastError: Error | null = null;

    for (let attempt = 1; attempt <= this.config.maxAttempts; attempt++) {
      try {
        return await fn();
      } catch (error) {
        lastError = error as Error;

        if (!this.shouldRetry(error as Error, attempt)) {
          throw error;
        }

        const delay = this.calculateDelay(attempt);
        await this.sleep(delay);
      }
    }

    throw lastError;
  }

  private shouldRetry(error: Error, attempt: number): boolean {
    if (attempt >= this.config.maxAttempts) return false;

    const errorMessage = error.message.toLowerCase();
    return this.config.retryableErrors.some(
      retryable => errorMessage.includes(retryable.toLowerCase())
    );
  }

  private calculateDelay(attempt: number): number {
    // Exponential backoff: baseDelay * 2^(attempt-1)
    const exponentialDelay = this.config.baseDelay * Math.pow(2, attempt - 1);

    // Cap at max delay
    let delay = Math.min(exponentialDelay, this.config.maxDelay);

    // Add jitter (random 50-100% of delay)
    if (this.config.jitter) {
      delay = delay * (0.5 + Math.random() * 0.5);
    }

    return Math.floor(delay);
  }

  private sleep(ms: number): Promise<void> {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

// Usage
const retryManager = new RetryManager({
  maxAttempts: 3,
  baseDelay: 100,
  maxDelay: 5000,
  jitter: true,
  retryableErrors: ['ECONNRESET', 'ETIMEDOUT', '503', '429'],
});

async function fetchWithRetry(url: string): Promise<Response> {
  return retryManager.execute(() => fetch(url));
}

Why Jitter Matters

Without jitter, all retrying clients hit the server simultaneously:

Without jitter:
Client A ─────X─────R─────R─────✓
Client B ─────X─────R─────R─────✓
Client C ─────X─────R─────R─────✓
             ↑ All retry at the same time → Thundering herd

With jitter:
Client A ─────X──────R─────R─────✓
Client B ─────X────R──────R──────✓
Client C ─────X───────R─────R────✓
             ↑ Retries spread out

Rate Limiting

Rate limiting protects your API from abuse and ensures fair resource distribution:

interface RateLimitConfig {
  windowMs: number;       // Time window in milliseconds
  maxRequests: number;    // Maximum requests per window
  keyGenerator?: (req: Request) => string; // Custom key (default: IP)
}

interface RateLimitStore {
  count: number;
  resetTime: number;
}

class RateLimiter {
  private store: Map<string, RateLimitStore> = new Map();

  constructor(private config: RateLimitConfig) {
    // Clean up expired entries periodically
    setInterval(() => this.cleanup(), config.windowMs);
  }

  middleware() {
    return async (req: Request, res: Response, next: () => Promise<void>) => {
      const key = this.config.keyGenerator
        ? this.config.keyGenerator(req)
        : this.getClientIP(req);

      const result = this.checkLimit(key);

      // Set rate limit headers
      res.setHeader('X-RateLimit-Limit', this.config.maxRequests);
      res.setHeader('X-RateLimit-Remaining', Math.max(0, this.config.maxRequests - result.count));
      res.setHeader('X-RateLimit-Reset', result.resetTime);

      if (result.limited) {
        res.status(429).json({
          error: 'Too Many Requests',
          retryAfter: Math.ceil((result.resetTime - Date.now()) / 1000),
        });
        return;
      }

      await next();
    };
  }

  private checkLimit(key: string): { count: number; resetTime: number; limited: boolean } {
    const now = Date.now();
    const entry = this.store.get(key);

    if (!entry || now >= entry.resetTime) {
      // New window
      const resetTime = now + this.config.windowMs;
      this.store.set(key, { count: 1, resetTime });
      return { count: 1, resetTime, limited: false };
    }

    // Existing window
    entry.count++;
    this.store.set(key, entry);

    return {
      count: entry.count,
      resetTime: entry.resetTime,
      limited: entry.count > this.config.maxRequests,
    };
  }

  private getClientIP(req: Request): string {
    const forwarded = req.headers.get('x-forwarded-for');
    if (forwarded) {
      return forwarded.split(',')[0].trim();
    }
    return 'unknown';
  }

  private cleanup(): void {
    const now = Date.now();
    for (const [key, entry] of this.store.entries()) {
      if (now >= entry.resetTime) {
        this.store.delete(key);
      }
    }
  }
}

// Usage with Express-like framework
const apiLimiter = new RateLimiter({
  windowMs: 60 * 1000,  // 1 minute
  maxRequests: 100,
});

const authLimiter = new RateLimiter({
  windowMs: 15 * 60 * 1000,  // 15 minutes
  maxRequests: 5,
  keyGenerator: (req) => `auth:${req.body.email}`,
});

Rate Limiting Algorithms

Algorithm	Pros	Cons
Fixed Window	Simple, memory-efficient	Burst at window edges
Sliding Window	Smoother distribution	More memory
Token Bucket	Allows burst up to bucket size	More complex
Leaky Bucket	Constant output rate	Can’t handle bursts

Combining Patterns: The Resilient Client

Here’s a production-ready HTTP client combining all patterns:

interface ResilientClientConfig {
  baseURL: string;
  timeout: number;
  retry: RetryConfig;
  circuitBreaker: CircuitBreakerConfig;
  rateLimit?: RateLimitConfig;
}

class ResilientHttpClient {
  private circuitBreaker: CircuitBreaker;
  private retryManager: RetryManager;
  private rateLimiter: RateLimiter | null = null;

  constructor(private config: ResilientClientConfig) {
    this.circuitBreaker = new CircuitBreaker(
      config.baseURL,
      config.circuitBreaker
    );

    this.retryManager = new RetryManager(config.retry);

    if (config.rateLimit) {
      this.rateLimiter = new RateLimiter(config.rateLimit);
    }
  }

  async get<T>(path: string, options?: RequestInit): Promise<T> {
    return this.request<T>('GET', path, options);
  }

  async post<T>(path: string, body: unknown, options?: RequestInit): Promise<T> {
    return this.request<T>('POST', path, {
      ...options,
      body: JSON.stringify(body),
      headers: {
        'Content-Type': 'application/json',
        ...options?.headers,
      },
    });
  }

  private async request<T>(
    method: string,
    path: string,
    options?: RequestInit
  ): Promise<T> {
    return this.circuitBreaker.execute(async () => {
      return this.retryManager.execute(async () => {
        const controller = new AbortController();
        const timeoutId = setTimeout(
          () => controller.abort(),
          this.config.timeout
        );

        try {
          const response = await fetch(`${this.config.baseURL}${path}`, {
            method,
            signal: controller.signal,
            ...options,
          });

          clearTimeout(timeoutId);

          if (!response.ok) {
            const error = new Error(`HTTP ${response.status}`);
            (error as any).status = response.status;
            throw error;
          }

          return response.json();
        } catch (error) {
          clearTimeout(timeoutId);
          throw error;
        }
      });
    });
  }

  getHealth() {
    return {
      circuitBreaker: this.circuitBreaker.getStats(),
    };
  }
}

// Production configuration
const apiClient = new ResilientHttpClient({
  baseURL: 'https://api.example.com',
  timeout: 5000,
  retry: {
    maxAttempts: 3,
    baseDelay: 100,
    maxDelay: 2000,
    jitter: true,
    retryableErrors: ['ECONNRESET', 'ETIMEDOUT', '502', '503', '504'],
  },
  circuitBreaker: {
    failureThreshold: 5,
    successThreshold: 3,
    timeout: 30000,
    resetTimeout: 5000,
  },
});

Fallback Strategies

When a service is unavailable, provide degraded functionality:

interface CacheEntry<T> {
  data: T;
  timestamp: number;
  ttl: number;
}

class FallbackService<T> {
  private cache: Map<string, CacheEntry<T>> = new Map();

  constructor(
    private primary: () => Promise<T>,
    private fallback: () => Promise<T>,
    private staleWhileRevalidate: boolean = true,
    private staleTTL: number = 300000 // 5 minutes
  ) {}

  async get(key: string): Promise<T> {
    try {
      const data = await this.primary();
      this.cache.set(key, {
        data,
        timestamp: Date.now(),
        ttl: this.staleTTL,
      });
      return data;
    } catch (error) {
      // Try stale cache first
      const cached = this.cache.get(key);
      if (cached && Date.now() - cached.timestamp < cached.ttl) {
        if (this.staleWhileRevalidate) {
          // Return stale data and revalidate in background
          this.revalidateInBackground(key);
        }
        return cached.data;
      }

      // No valid cache, use fallback
      return this.fallback();
    }
  }

  private revalidateInBackground(key: string): void {
    this.primary()
      .then(data => {
        this.cache.set(key, {
          data,
          timestamp: Date.now(),
          ttl: this.staleTTL,
        });
      })
      .catch(() => {
        // Silently fail, we already returned stale data
      });
  }
}

// Usage: Product recommendations with fallback
const recommendationsService = new FallbackService(
  async () => {
    const response = await fetch('https://ml-service.example.com/recommend');
    return response.json();
  },
  async () => {
    // Fallback: return popular products
    return getDefaultRecommendations();
  },
  true,
  60000
);

Monitoring and Observability

Resilient systems need visibility. Track these metrics:

interface ResilienceMetrics {
  circuitBreakerState: CircuitState;
  circuitBreakerFailures: number;
  retryAttempts: number;
  rateLimitedRequests: number;
  fallbackInvocations: number;
  averageLatency: number;
}

class ResilienceMonitor {
  private metrics: ResilienceMetrics = {
    circuitBreakerState: 'CLOSED',
    circuitBreakerFailures: 0,
    retryAttempts: 0,
    rateLimitedRequests: 0,
    fallbackInvocations: 0,
    averageLatency: 0,
  };

  private latencySamples: number[] = [];

  recordLatency(ms: number): void {
    this.latencySamples.push(ms);
    if (this.latencySamples.length > 100) {
      this.latencySamples.shift();
    }
    this.metrics.averageLatency =
      this.latencySamples.reduce((a, b) => a + b, 0) / this.latencySamples.length;
  }

  recordRetry(): void {
    this.metrics.retryAttempts++;
  }

  recordRateLimit(): void {
    this.metrics.rateLimitedRequests++;
  }

  recordFallback(): void {
    this.metrics.fallbackInvocations++;
  }

  updateCircuitBreaker(state: CircuitState, failures: number): void {
    this.metrics.circuitBreakerState = state;
    this.metrics.circuitBreakerFailures = failures;
  }

  getMetrics(): ResilienceMetrics {
    return { ...this.metrics };
  }

  healthCheck(): { healthy: boolean; issues: string[] } {
    const issues: string[] = [];

    if (this.metrics.circuitBreakerState === 'OPEN') {
      issues.push('Circuit breaker is OPEN');
    }

    if (this.metrics.averageLatency > 1000) {
      issues.push(`High latency: ${Math.round(this.metrics.averageLatency)}ms`);
    }

    if (this.metrics.retryAttempts > 100) {
      issues.push(`High retry count: ${this.metrics.retryAttempts}`);
    }

    return {
      healthy: issues.length === 0,
      issues,
    };
  }
}

Production Checklist

Before deploying to production, ensure:

• Circuit breakers configured for all external dependencies
• Retry with exponential backoff for transient failures
• Rate limiting on public endpoints
• Timeouts on all network calls (no infinite waits)
• Fallback strategies for critical services
• Monitoring and alerting on resilience metrics
• Load testing to validate configuration values
• Documentation of failure modes and recovery procedures

Conclusion

Resilient APIs are built on three pillars: fail fast (circuit breakers), retry smart (exponential backoff with jitter), and protect resources (rate limiting). These patterns work together to create systems that degrade gracefully under pressure and recover automatically when conditions improve.

Start with timeouts and circuit breakers — they prevent cascading failures. Add retries for transient errors. Implement rate limiting to protect against abuse. Finally, add fallback strategies for critical paths. The combination creates a system that’s reliable by design, not by accident.