Prompt Caching: Reduce LLM Costs by 90% with Smart Context Management

Every LLM API call costs money. When you’re sending the same system prompt, documentation, or codebase context with every request, you’re paying for the same tokens over and over. Prompt caching changes this equation entirely, offering up to 90% cost reduction on cached portions of your requests.

For teams building AI-powered features, understanding prompt caching is essential for scaling affordably. This guide covers how caching works, which providers support it, and practical implementation patterns.

The Problem: Redundant Token Processing

How LLM Billing Works

Most LLM providers charge per token for both input and output:

Provider	Input Cost (per 1M tokens)	Output Cost (per 1M tokens)
Claude 3.5 Sonnet	$3.00	$15.00
GPT-4o	$2.50	$10.00
Gemini 1.5 Pro	$1.25	$5.00

When your application sends a 50,000-token system prompt (documentation, codebase context, examples) with every request, you’re paying $0.15+ just for that prompt—every single time.

The Scaling Problem

Consider a chatbot handling 100,000 daily requests with a 20,000-token shared context:

Daily input tokens: 100,000 requests × 20,000 tokens = 2 billion tokens
Daily cost (Claude 3.5 Sonnet): 2,000 × $3.00 = $6,000/day
Monthly cost: ~$180,000

With prompt caching at 90% discount on cached tokens:

Cached tokens (90%): 1.8 billion tokens at $0.30/1M = $540
Uncached tokens (10%): 200 million tokens at $3.00/1M = $600
Daily cost: $1,140
Monthly cost: ~$34,200

Savings: $145,800/month (81% reduction)

How Prompt Caching Works

The Technical Mechanism

Prompt caching leverages the fact that LLMs process tokens through multiple transformer layers. When the same prefix appears across requests, providers can:

1. Cache the Key-Value (KV) pairs from transformer attention layers
2. Skip re-computation for cached portions
3. Only process new tokens added after the cached prefix

This is similar to how browsers cache static assets, but for neural network intermediate states.

Cache Lifecycle

Request 1: [System Prompt] [Documentation] [User Query]
           ↓
           Cache created for prefix tokens
           ↓
Request 2: [System Prompt] [Documentation] [NEW Query]
           ↓
           Cache HIT → Process only new query
           ↓
Request 3: [System Prompt] [Documentation] [Another Query]
           ↓
           Cache HIT → Continued savings

Cache Invalidation

Caches expire based on provider rules:

Provider	Cache TTL
Anthropic	5 minutes (refreshed on hit)
OpenAI	5-60 minutes depending on use
Google	Varies by tier

Any change to the cached prefix invalidates the cache entirely.

Provider Comparison

Anthropic (Claude)

Anthropic offers the most mature prompt caching implementation with explicit cache control:

Pricing:

• Cache write: $3.75 per 1M tokens (Claude 3.5 Sonnet)
• Cache read: $0.30 per 1M tokens (90% discount)
• Minimum: 1,024 tokens for caching

Implementation:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": long_system_prompt,
                    "cache_control": {"type": "ephemeral"}
                },
                {
                    "type": "text", 
                    "text": user_query
                }
            ]
        }
    ]
)

Cache Write vs Read:

• First request with new prefix: Pays cache write price
• Subsequent requests: Pays cache read price (90% cheaper)

OpenAI (GPT)

OpenAI implements automatic caching for GPT-4o and GPT-4 Turbo:

Pricing:

• Cached tokens: 50% discount on input
• No explicit cache control API
• Automatic when prefix matches

Best Practices:

from openai import OpenAI

client = OpenAI()

# Keep system message consistent across requests
system_message = """You are a helpful coding assistant with access to the following documentation:
[Large documentation block that stays constant]
"""

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": system_message},
        {"role": "user", "content": user_query}
    ]
)

Google (Gemini)

Google offers context caching with explicit controls:

Pricing:

• Cache storage: $1.00 per 1M tokens per hour
• Cached retrieval: $0.01875 per 1M tokens
• Minimum: 32,768 tokens

Implementation:

from google.genai import types
from google import genai

client = genai.Client()

# Create cached content
cache = client.caches.create(
    model="gemini-1.5-pro",
    config=types.CreateCachedContentConfig(
        contents=[
            types.Content(
                role="user",
                parts=[types.Part.from_text(long_context)]
            )
        ]
    )
)

# Use cached content
response = client.models.generate_content(
    model="gemini-1.5-pro",
    contents=user_query,
    config=types.GenerateContentConfig(
        cached_content=cache.name
    )
)

When to Use Prompt Caching

Ideal Use Cases

1. RAG Applications with Large Context

Retrieval-Augmented Generation often includes extensive documentation:

# RAG with caching
def rag_query(query: str, retrieved_docs: list[str]):
    context = "\n\n".join(retrieved_docs)
    
    return client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": f"Context:\n{context}\n\nQuery: {query}",
                    "cache_control": {"type": "ephemeral"}
                }
            ]
        }]
    )

For more on RAG architectures, see my guide on building RAG applications.

2. Coding Assistants with Codebase Context

Tools like Claude Code send entire file contents:

# Coding assistant with file context
def code_query(query: str, files: dict[str, str]):
    file_context = "\n".join(
        f"--- {path} ---\n{content}" 
        for path, content in files.items()
    )
    
    system_prompt = f"""You are a coding assistant. Here is the relevant code:
    
{file_context}
"""
    
    return client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=2048,
        system=[
            {
                "type": "text",
                "text": system_prompt,
                "cache_control": {"type": "ephemeral"}
            }
        ],
        messages=[{"role": "user", "content": query}]
    )

Learn more about AI code assistants in my comprehensive guide to AI code assistants 2026.

3. Multi-Turn Conversations

Chat applications with conversation history:

# Conversation with cached history
def chat_with_history(message: str, history: list[dict]):
    # Cache the conversation history
    cached_messages = []
    for msg in history[:-1]:  # All but last
        cached_messages.append({
            "role": msg["role"],
            "content": [
                {
                    "type": "text",
                    "text": msg["content"],
                    "cache_control": {"type": "ephemeral"}
                }
            ]
        })
    
    # Add new message
    cached_messages.append({
        "role": "user",
        "content": message
    })
    
    return client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=cached_messages
    )

4. Document Analysis

Processing large documents repeatedly:

# Document analysis with caching
def analyze_document(document: str, query: str):
    return client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": f"Document:\n{document}",
                    "cache_control": {"type": "ephemeral"}
                },
                {
                    "type": "text",
                    "text": f"Analysis request: {query}"
                }
            ]
        }]
    )

When NOT to Use Caching

1. Single-Request Workflows

If you’re making one-off requests without repeated context, caching adds overhead without benefit.

2. Highly Variable Contexts

When your prefix changes frequently (e.g., different system prompts per user), cache hit rates will be low.

3. Short Prompts

Below the minimum token threshold (1,024 for Anthropic), caching won’t activate.

Optimization Strategies

1. Structure for Cache Hits

Place stable content at the start of prompts:

# BAD: Variable content first
prompt = f"User: {username}\nContext: {shared_docs}\nQuery: {query}"

# GOOD: Stable content first
prompt = f"Context: {shared_docs}\nUser: {username}\nQuery: {query}"

2. Batch Similar Requests

Group requests that share context:

# Process multiple queries with same context
def batch_queries(queries: list[str], context: str):
    results = []
    for query in queries:
        # Cache persists across batch
        result = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=1024,
            messages=[{
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": context,
                        "cache_control": {"type": "ephemeral"}
                    },
                    {
                        "type": "text",
                        "text": query
                    }
                ]
            }]
        )
        results.append(result)
    return results

3. Monitor Cache Performance

Track your cache hit rate:

import time

def query_with_cache_metrics(query: str, context: str):
    start = time.time()
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": context,
                    "cache_control": {"type": "ephemeral"}
                },
                {
                    "type": "text",
                    "text": query
                }
            ]
        }]
    )
    
    # Extract cache metrics from response
    usage = response.usage
    cache_read = getattr(usage, 'cache_read_input_tokens', 0)
    cache_write = getattr(usage, 'cache_creation_input_tokens', 0)
    
    print(f"Cache read: {cache_read} tokens")
    print(f"Cache write: {cache_write} tokens")
    
    return response

4. Use Smaller Models for Cacheable Tasks

Combine caching with model selection:

def smart_query(query: str, context: str, complexity: str = "simple"):
    # Use cheaper model with caching for simple tasks
    model = "claude-3-5-haiku-20241022" if complexity == "simple" else "claude-3-5-sonnet-20241022"
    
    return client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": context,
                    "cache_control": {"type": "ephemeral"}
                },
                {
                    "type": "text",
                    "text": query
                }
            ]
        }]
    )

Real-World Cost Analysis

Case Study: AI Code Assistant

A development team building an AI coding assistant saw these results after implementing prompt caching:

Metric	Before Caching	After Caching	Change
Monthly API cost	$12,400	$2,100	-83%
Avg cost per query	$0.08	$0.014	-82%
Cache hit rate	N/A	94%	—
Latency (P50)	2.3s	1.1s	-52%

The latency improvement comes from skipping transformer computation on cached portions.

Case Study: Document Q&A System

A legal tech company processing contracts:

Metric	Before Caching	After Caching	Change
Monthly API cost	$8,700	$1,900	-78%
Tokens processed	2.9B	2.9B (cached)	Same volume
Effective input cost	$3.00/1M	$0.65/1M	-78%

Integration Patterns

LangChain Integration

from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate

llm = ChatAnthropic(
    model="claude-3-5-sonnet-20241022",
    cache=True  # Enable caching
)

prompt = ChatPromptTemplate.from_messages([
    ("system", "{context}"),
    ("user", "{query}")
])

chain = prompt | llm

Streaming with Caching

def stream_with_cache(query: str, context: str):
    with client.messages.stream(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": context,
                    "cache_control": {"type": "ephemeral"}
                },
                {
                    "type": "text",
                    "text": query
                }
            ]
        }]
    ) as stream:
        for text in stream.text_stream:
            yield text

Monitoring and Debugging

Cache Performance Dashboard

Track these metrics:

metrics = {
    "cache_hit_rate": cache_hits / total_requests,
    "cache_read_tokens": total_cache_read,
    "cache_write_tokens": total_cache_write,
    "cost_saved": calculate_savings(cache_read, input_price),
    "avg_latency_improvement": latency_before - latency_after
}

Common Issues

1. Low Cache Hit Rate

Cause: Variable prefix content Solution: Move stable content to the beginning

2. Cache Not Activating

Cause: Below minimum token threshold Solution: Add more context or combine prompts

3. Unexpected Cache Invalidation

Cause: Subtle changes in cached content Solution: Normalize whitespace, remove timestamps

Future of Prompt Caching

Emerging Trends

1. Persistent Caches

Some providers are exploring longer-lived caches (hours to days) for frequently used contexts.

2. Cross-User Caching

Shared caches for common system prompts across different users (privacy-preserving).

3. Automatic Cache Optimization

AI systems that automatically restructure prompts for better cache efficiency.

Best Practices Summary

Practice	Impact
Place stable content first	Higher cache hit rate
Batch similar requests	Better cache utilization
Monitor cache metrics	Identify optimization opportunities
Use appropriate model size	Maximize cost savings
Keep prompts consistent	Avoid invalidation

Conclusion

Prompt caching is a game-changer for production LLM applications. By caching the expensive computation of repeated context, you can achieve up to 90% cost reduction while improving latency.

Key Takeaways:

• Structure prompts with stable content at the beginning for cache hits
• Monitor cache metrics to optimize your implementation
• Combine caching with model selection for maximum savings
• Use caching for RAG, coding assistants, and document analysis workflows
• Each provider has different pricing and implementation details

Next Steps:

• Explore building RAG applications for context-heavy AI systems
• Learn about production-ready LLM integration architecture for scaling
• Check out AI code assistants 2026 for tools that leverage caching

---

Implementing prompt caching in your LLM application? Connect with me on Twitter to discuss optimization strategies!