Prompt caching promises a 90% discount. Most API calls pay full price anyway.

LLM & Model Guides · By Caleb Sakala · April 30, 2026

OpenAI's documentation says prompt caching can cut input token costs by up to 90% and latency by up to 80%. Anthropic quotes similar numbers. These figures are real, measured, and documented. They are also the best-case scenario for a perfectly structured prompt that stays identical across thousands of requests. The typical implementation hits the cache on fewer than half of its API calls. Some hit it on none.

The gap between the advertised savings and what shows up on the invoice comes down to three things: how tokens are ordered in the prompt, how each provider's cache expiration works, and whether the prompt structure accidentally invalidates the cache on every single request.

Token ordering is the entire game

Every major LLM provider uses prefix matching for prompt caching. The cache stores the computed key-value tensors for a specific sequence of tokens. When a new request arrives, the system compares tokens from the beginning of the prompt, left to right, until it finds a mismatch. Everything before the mismatch gets loaded from cache. Everything after gets recomputed.

This means the position of content in the prompt determines the cache hit rate. Static content goes first. Dynamic content goes last.

A prompt structure that caches well:

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},   # 2,000 tokens, identical every request
    {"role": "user", "content": FEW_SHOT_EXAMPLES},  # 500 tokens, identical every request
    {"role": "user", "content": user_query}           # varies per request
]

A prompt structure that never caches:

messages = [
    {
        "role": "system",
        "content": f"Current time: {datetime.now()}. Session: {session_id}. {SYSTEM_PROMPT}"
    },
    {"role": "user", "content": user_query}
]

The only difference is a timestamp and session ID prepended to the system prompt. Those 15 tokens change on every request, the prefix match fails at token position 3, and the entire 2,000-token system prompt gets recomputed from scratch. No cache hit. Full price.

Anthropic charges you extra to write to the cache

This is the part most guides skip. On Anthropic's API, cached input tokens cost 90% less to read. Writing to the cache costs 25% more than standard input pricing. If a prompt gets cached once and read a hundred times, the math is overwhelming. If the prompt changes frequently and the cache entry expires before a second request reads it, you pay 25% extra on every call with zero benefit.

Anthropic's cache has a default TTL of 5 minutes, extendable to 1 hour with the cache_control parameter. For low-traffic endpoints processing fewer than one request per five minutes on a given prefix, prompt caching on Anthropic is more expensive than skipping it entirely.

Three providers handle prompt caching differently enough to matter

OpenAI enables prompt caching automatically on all gpt-4o and newer models with zero code changes. Cached tokens get a 50% discount (not 90%). The cache requires a minimum of 1,024 tokens and persists for 5 to 10 minutes of inactivity.

# OpenAI: nothing to configure
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages
)
cached = response.usage.prompt_tokens_details.cached_tokens
print(f"Cached: {cached} of {response.usage.prompt_tokens} input tokens")

Anthropic requires explicit cache_control markers. The tradeoff: deterministic control over what gets cached, but a 25% write surcharge and a minimum block size of 1,024 tokens (2,048 for Claude 3.5 Haiku).

response = anthropic_client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": SYSTEM_PROMPT,
        "cache_control": {"type": "ephemeral"}
    }],
    messages=[{"role": "user", "content": user_query}]
)
print(f"Cache read: {response.usage.cache_read_input_tokens}")
print(f"Cache write: {response.usage.cache_creation_input_tokens}")

Google's Gemini API creates the cache as a separate object with a configurable TTL, then references it in subsequent requests. Cached tokens get a 75% discount, but there is a per-token-per-hour storage cost. Minimum cached content is 4,096 tokens for Gemini 1.5 Flash and 2,048 for Gemini 1.5 Pro.

Choosing between them depends on the workload. High-traffic automated pipelines benefit from OpenAI's zero-configuration approach even with its smaller 50% discount. Batch processing jobs with predictable prompt structures justify Anthropic's implementation cost because the 90% read discount compounds quickly at volume. Long document analysis over extended sessions fits Google's explicit cache with its configurable TTL, since the same large document can be referenced across many requests over hours without re-uploading.

These patterns break your cache on every request

The most common cache killers are dynamic values at the start of the system prompt. Timestamps, request IDs, user-specific context, and per-request tool definitions all invalidate the prefix before it reaches the static content.

A less obvious cache breaker: a changing tool list. If the available tools for an agent change between requests because different users have different permissions, or because tools are loaded conditionally, the prefix changes and the cache misses. LangChain's default agent configuration loads tools dynamically from a registry on each invocation. Every call sends a slightly different tool list, which means the prefix match fails before it reaches the static system prompt. Chase Agents avoids this because tool declarations are fixed at the automation step level. Each step defines its tool schema once during construction, so the prefix stays identical across every run of that step.

Four rules that keep the cache hit rate above 90%:

1. System prompt goes first, user message goes last.
2. No timestamps, session IDs, or request metadata anywhere in the system prompt.
3. Tool definitions must be identical and in the same order on every request.
4. Few-shot examples belong in the system prompt, not in per-request user messages.

Check if your cache is working in five lines

Log the cache fields from every API response for 24 hours and compute the ratio.

def log_cache_stats(response, provider="openai"):
    if provider == "openai":
        total = response.usage.prompt_tokens
        cached = response.usage.prompt_tokens_details.cached_tokens
    elif provider == "anthropic":
        total = response.usage.input_tokens
        cached = response.usage.cache_read_input_tokens
    hit_rate = (cached / total * 100) if total > 0 else 0
    print(f"Cache hit rate: {hit_rate:.1f}% ({cached}/{total} tokens)")

If the hit rate sits below 50% after a full day of traffic, the prompt structure needs to change before caching delivers any value. If it sits at zero, check for dynamic content at the beginning of the system prompt. That single fix typically moves the hit rate from 0% to above 85% without any other changes.

In Chase Agents, the cache hit diagnostic is part of step-level observability. Each automation step logs cache read and cache write token counts separately, so identifying the cache-breaking step in a multi-step workflow takes seconds instead of the manual grep through raw API logs that most custom orchestration setups require.

The next frontier for prompt caching is semantic caching, where responses get reused across similar (not identical) prompts. Redis and GPTCache already offer this at the application layer. But until major providers add native semantic caching to their APIs, the difference between 90% savings and 0% savings is the first 100 tokens of the system prompt.