As of May 2026. In late March 2026, the math on Gemini API costs at MemX did not add up. The MemX pipeline had swapped one stage from Gemini 2.5 Flash-Lite to Gemini 2.5 Flash, expecting the higher input cost ($0.30/M vs $0.10/M) to be offset by Flash's context caching (cached input at $0.03/M, a 10x discount). Same workload, same prompts, similar net cost. The bill came in 1.83x higher.
Hunting for the cache hit rate surfaced this buried in the API response: `"Failed to create Gemini cache: Cached content is too small. total_token_count=781, min_total_token_count=1024."` The MemX Stage 2 prompts were 781 tokens. The cache requires 1,024. MemX had been paying full price on every request while believing caching was active. The API never returned an HTTP error. It silently no-opped.
Quick takeaways. Editor's note updated June 2026: Google has raised the documented caching floors since this post first shipped on April 30, 2026, so the figures below reflect the live docs rather than the April numbers. Per the current ai.google.dev docs (June 2026) Gemini context caching needs 2,048 tokens for 2.5 Flash and 2.5 Pro, and 4,096 for the newer 3.5 Flash and 3.1 Pro Preview; the enforced minimum that errored at MemX was 1,024. Below the floor, explicit caching returns an error string inside the 200 response and implicit caching no-ops with nothing at all. Anthropic does the same (silent no-op; Sonnet 4.5/4.6 at 1,024, Opus 4.5/4.6/4.7 at 4,096, Haiku 3.5 at 2,048, Haiku 4.5 at 4,096). OpenAI returns `cached_tokens: 0`. Fix: instrument the response, not the request. Treat cache-hit-rate as an SLO. Claude Code runs 92% as their floor with 81% cost reduction.
What Google actually requires
Gemini context caching, both explicit (you create a named cache) and implicit (Google caches automatically on 2.5 and newer), enforces a minimum prompt size. Per the current ai.google.dev docs as of June 2026: 2,048 tokens for Gemini 2.5 Flash and Gemini 2.5 Pro, and 4,096 tokens for the newer Gemini 3.5 Flash and Gemini 3.1 Pro Preview. These doc-quoted floors have moved over time; Google's May 8, 2025 implicit caching announcement listed 1,024 for 2.5 Flash and 2,048 for 2.5 Pro, and the enforced minimum that errored at MemX in March 2026 was 1,024 (`min_total_token_count=1024` in the API string). The numbers diverge across Google's own surfaces and across releases, so always trust the live docs page for the specific API you call.
The advertised discount is real and large: cached input at $0.03/M for Flash vs $0.30/M standard. Google raised the Gemini 2.5 implicit-caching discount to 90% in October 2025 (the legacy 75% rate now applies only to Gemini 2.0). $0.03 / $0.30 = 90% off, which matches. The catch is that the discount only applies above the minimum threshold. Most production prompts that nobody bothered to pad sit under it.
The silent-failure pattern
Three failure modes live in the same trap.
- Explicit caching throws the error string above (`Cached content is too small`). The string sits inside the response payload, not in the HTTP status. Check only status codes and you never see it. The cache create call "succeeds" and no cache exists.
- Implicit caching no-ops without an error. Google AI Developer Forum threads report the practical implicit-cache threshold sometimes runs higher than the published 1,024 tokens, with community testing showing hits often skip below the ~6,000 token mark on 2.5 Flash. The bill is your only signal.
- Cache busters quietly wipe valid caches too. A timestamp injected into a system prompt, non-deterministic JSON key ordering in tool schemas, or mutating tool params mid-session invalidates the entire prefix. No error. No warning. A higher invoice arrives instead.
How OpenAI and Anthropic compare
| Provider | Minimum cache size | What happens below the minimum |
|---|---|---|
| Google Gemini 2.5 Flash and 2.5 Pro | 2,048 tokens (ai.google.dev, June 2026) | Explicit returns error string inside 200 response. Implicit no-ops silently. Enforced minimum has historically been as low as 1,024; real-world implicit floor often higher than docs. |
| Google Gemini 3.5 Flash and 3.1 Pro Preview | 4,096 tokens (ai.google.dev) | Doc-quoted floors diverge across Google's surfaces and releases; trust the live docs for your API. |
| OpenAI (GPT-4o family, o-series) | 1,024 tokens (grows in 128-token increments) | Response returns `cached_tokens: 0`. Visible if you check. Launched Oct 1, 2024 with 50% cached discount; some current models go up to about 90%. |
| Anthropic Claude Sonnet 4.5/4.6 | 1,024 tokens | Silent no-op. Read `cache_creation_input_tokens` and `cache_read_input_tokens`. Cache writes cost 25% extra; cache reads cost 10% of base. |
| Anthropic Claude Opus 4.5/4.6/4.7 and Haiku 4.5 | 4,096 tokens | Silent no-op. Same response fields apply. |
| Anthropic Claude Haiku 3.5 | 2,048 tokens | Silent no-op. |
The cost shape that materialised
Gemini 2.5 Flash with caching enabled but failing silently cost MemX 1.83x what Flash-Lite cost on the same workload. Not the slim premium expected. Not the small markup that caching was supposed to neutralise. 83% more, paid daily, for nothing.
The shape generalises. ProjectDiscovery published a case study where adding proper prompt caching cut their LLM spend by 59% on launch, climbing to 66% after optimisation and 70% over their last ten measured days. Their gains were real because they fixed the caching. If yours silently no-ops, you sit on the same potential savings and pay them daily in the wrong direction.
Three fixes that actually worked
- Promote, do not pad. Skip the filler whitespace; move durable artifacts to the top of the prompt. Tool schemas, JSON output examples, persona instructions, few-shot exemplars: all of it is stable across requests and belongs above the dynamic user input. Push the prefix past 1,024 tokens deterministically. This clears the minimum and satisfies prefix stability in one move.
- If you cannot get to 1,024 tokens, switch the stage to Flash-Lite. At $0.10/M input, Flash-Lite costs 3x less than uncached Flash. No scenario justifies paying full-price Flash on a sub-1,024-token prompt; the only reason to use Flash there is the cache, and the cache is dead.
- Instrument the response, not the request. Log `usage_metadata.cached_content_token_count` on every Gemini call. Log `cached_tokens` on every OpenAI call. Log `cache_read_input_tokens` and `cache_creation_input_tokens` on every Anthropic call. Alert when the rolling 1-hour cache-hit ratio drops below your target. The Claude Code team holds 92% as their floor with 81% cost reduction; anything below opens an incident.
The bigger lesson: observability is your contract
The bug class here is "API returns 200, bill goes up." Status codes hide it. Response latency hides it. Accuracy metrics hide it. The only way to catch it is to treat cache-hit rate as an SLO and instrument every response to verify caching actually happens.
Borrow the discipline from Claude Code: 92% hit rate as a floor; below that, page somebody. One engineer documented a timestamp injection inside a system prompt that produced a 9% spike in cache misses and traced it back to a single line by reading response telemetry. Without that telemetry the spike stays invisible and the bill runs higher.
The cache is silent. The bill is loud. Make the cache observable. Every LLM provider lets the cache fail silently; only one team has to make the failure visible, and that team is yours. Treat cache-hit rate as an SLO, alert on regressions, and stop paying the full-price tax on prompts you thought were cached.
Key takeaway. Gemini, OpenAI, and Anthropic all silently no-op caching below their minimum prompt sizes. Push your stable prefix above the threshold deterministically, instrument the response payload (not the request), and run cache-hit rate as a production SLO with a 92% floor. The fix takes one afternoon. The bill it pays back is monthly.
01What is the minimum prompt size to use Gemini context caching?
As of June 2026 per Google's live ai.google.dev docs: 2,048 tokens for Gemini 2.5 Flash and Gemini 2.5 Pro, and 4,096 tokens for the newer Gemini 3.5 Flash and Gemini 3.1 Pro Preview. (The enforced minimum has historically been as low as 1,024 on 2.5 Flash.) Below those thresholds, explicit caching returns the error string `Cached content is too small` inside the 200 response payload, and implicit caching no-ops with no error at all.
02Why doesn't Gemini caching work on my short prompts?
Because Gemini enforces a minimum prompt size (per ai.google.dev today, 2,048 tokens on 2.5 Flash and 2.5 Pro, 4,096 on the newer 3.5 Flash and 3.1 Pro Preview; the enforced floor on 2.5 Flash has historically been as low as 1,024). Many production prompts sit under that floor, and there the cache silently fails. Either restructure to push the stable prefix above the threshold, or switch to Flash-Lite at $0.10/M input ($0.01/M cached) where the absolute cost is low enough that the cache-miss penalty barely matters.
03How do I check if my LLM prompt cache is actually working?
Log the response metadata. Gemini: `usage_metadata.cached_content_token_count`. OpenAI: `cached_tokens`. Anthropic: `cache_read_input_tokens` and `cache_creation_input_tokens`. If those fields read zero when you expected hits, your cache is failing. Treat cache-hit rate as an SLO; Claude Code holds 92% as their floor.
04Does OpenAI prompt caching also fail silently on short prompts?
Yes, but more visibly than Gemini. OpenAI returns `cached_tokens: 0` when caching does not apply (minimum 1,024 tokens, grows in 128-token increments). The field sits right there in the response; read it. Anthropic no-ops silently with no error.
05How much can prompt caching save in LLM costs?
ProjectDiscovery published a case study showing 59% cost reduction at launch, 66% after optimisation, and 70% over their last ten days. Anthropic prices cache reads at roughly 10% of base input (cache writes cost 25% extra). Gemini cached input on Flash runs $0.03/M vs $0.30/M standard. OpenAI launched at 50% off on cached input in October 2024, with some current models reaching about 90%.
06What invalidates a Gemini context cache?
Any change to the cached prefix: system prompt, tool schemas, few-shot examples. Common silent cache-busters include timestamps in system prompts, non-deterministic JSON key ordering in tool definitions, and mutating tool parameters mid-session. Each invalidates the whole cache without an error. Keep the prefix byte-stable.
07Where can I monitor my Gemini cache-hit rate in production?
Read `usage_metadata.cached_content_token_count` on every Gemini call, ship it to your metrics pipeline (Prometheus, Datadog, BigQuery), and chart cache-hit ratio per route. Page on a 15-minute drop below your floor. Track the ratio per system-prompt version so you can attribute regressions to the deploy that caused them.
