Semantic Caching for LLM Apps: Cutting Cost on Repeated Queries
Users ask the same questions in slightly different words, and you pay full price every time. Semantic caching answers the repeats for almost nothing.
Key takeaways
- A semantic cache embeds each incoming query, finds the nearest stored query in a vector store, and returns its cached answer when similarity clears a threshold.
- It pays off on repetitive, impersonal traffic like support FAQs.
- Tune the threshold on labeled query pairs and never cache answers that depend on user identity or time.
On this page
Semantic caching cuts LLM cost on repeated queries by keying the cache on meaning instead of exact text: embed the incoming query, find the closest stored query in a vector store, and return its cached answer when cosine similarity clears a threshold you tuned on labeled pairs. It pays off on repetitive, impersonal traffic and is dangerous on anything personalized.
Verified against
OpenAI text-embedding-3-small (1,536 dimensions, 8,192-token input), Redis vector search, pgvector, against the OpenAI embeddings guide, redis.io and the pgvector README. Checked October 2026.
The pattern shows up on almost every support assistant. A large share of traffic is the same handful of questions asked slightly differently: "how do I reset my password," "I can't log in and need a new password," "forgot password help." Exact-match caching catches none of these because the strings differ. Every rephrasing hits the model at full token cost for an answer you've already generated a thousand times.
The mechanism
Every cache entry stores three things: the original query text, its embedding vector, and the model's response. On a new query you embed it, run a nearest-neighbor search against stored embeddings, and check whether the closest match is within a similarity threshold.
from openai import OpenAI # embeddings; generation can be any provider
import numpy as np
client = OpenAI()
def embed(text: str) -> np.ndarray:
v = client.embeddings.create(
model="text-embedding-3-small",
input=text,
).data[0].embedding
return np.array(v, dtype=np.float32)
def cosine(a: np.ndarray, b: np.ndarray) -> float:
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
In production you don't loop over vectors in Python, you use a vector store. Redis with vector search is the easy pick if you already run Redis, but pgvector or Qdrant work the same way. The lookup:
def lookup(query: str, threshold: float = 0.92):
qv = embed(query)
hit = redis_vector_knn(qv, k=1) # returns (stored_query, response, score)
if hit and hit.score >= threshold:
return hit.response
return None
This is a different thing from provider-side prompt caching, which discounts a repeated prompt prefix but still runs a generation. Our guide to prompt caching in production LLM apps covers that layer; the two stack well, since semantic caching skips the call entirely on a hit and prompt caching cheapens the misses.
The threshold is everything
That 0.92 is the single most important number in the system, and it's a knife's edge. Set it too low and the cache returns a plausible but wrong answer. "How do I cancel my subscription" and "how do I change my subscription" share almost every word and can score very close on cosine similarity, yet they need completely different answers. Return the cancel answer to someone asking to upgrade and you've created a support ticket, not saved one.
Set it too high, say 0.98, and you barely get any hits, because natural rephrasings rarely land that close. The way to pick it: build a labeled set of a few hundred query pairs marked "same intent" or "different intent" from your own logs, then sweep the threshold to maximize correct hits while keeping wrong hits near zero. Don't guess this, and don't copy ours. Embedding geometry differs by domain and by model.
# threshold sweep: pick the value that keeps false hits under 1%
for t in [0.90, 0.92, 0.94, 0.96]:
tp = sum(1 for p in pairs if p.same and cosine(embed(p.a), embed(p.b)) >= t)
fp = sum(1 for p in pairs if not p.same and cosine(embed(p.a), embed(p.b)) >= t)
print(f"threshold {t}: correct hits {tp}, wrong hits {fp}")
Re-run the sweep whenever you change embedding model. A threshold tuned on one model means nothing on another, and switching models also invalidates every stored vector, so plan a cache flush with the migration.
What to never cache
Anything personalized or time-sensitive must skip the cache entirely, or you'll serve one user's data to another. "What's my order status" embeds almost identically across every user, so a semantic cache would happily return the previous user's order. Route those queries around the cache with a simple classifier on the query, plus a hard rule: if the answer depends on user identity or current time, no caching. Ever.
Set a TTL on everything else, too. A cached answer about pricing or a feature that changed last week is a wrong answer with good latency.
The numbers
Embeddings are cheap. OpenAI's embeddings guide lists text-embedding-3-small at about 62,500 pages per dollar (roughly 800 tokens a page), which works out to about $0.02 per million tokens as of October 2026. A 20-token query costs a tiny fraction of a cent to embed, while a full generation with a long system prompt costs orders of magnitude more. So a cache miss adds negligible cost, and a hit saves the entire generation.
Hit rate is what decides whether it's worth it, and it depends entirely on how clustered your traffic is. Support and documentation Q&A cluster hard and can see a meaningful share of hits. A general-purpose assistant sees far more varied queries, and a single-digit hit rate there may not justify the extra lookup latency or the wrong-answer risk. Measure on a week of real traffic before you build anything: embed historical queries, simulate the cache at your chosen threshold, and count.
Latency is a side benefit. A hit costs one embedding call plus one vector search, typically tens of milliseconds, against seconds for a full generation. Users notice. For the wider set of levers beyond caching, see our notes on LLM cost optimization in production.
Questions people ask
What similarity threshold should a semantic cache use?
There is no universal value. Build a labeled set of a few hundred query pairs from your own traffic, marked same intent or different intent, and sweep thresholds to find the lowest one that keeps wrong hits under about 1%. Sweeping 0.90 to 0.96 is a reasonable starting range for cosine similarity, and you must re-tune after any embedding model change.
Is semantic caching the same as prompt caching?
No. Prompt caching is a provider feature that bills a repeated prompt prefix at a discount, but the model still generates a fresh answer. Semantic caching is your own layer that matches a new query to a previously answered one and returns the stored answer without calling the model at all. They solve different costs and work well together.
What should never go in a semantic cache?
Anything whose correct answer depends on who is asking or when. Order status, account balances, "my" anything, and questions about live data embed almost identically across users, so a cache will serve one user's answer to another. Route those around the cache with a classifier and a hard rule, and put a TTL on everything else so stale answers expire.
The call we'd make
Semantic caching pays off when your traffic is repetitive and impersonal: support FAQs, documentation Q&A, product questions. Measure your hit rate on real traffic before committing, because below roughly 15% the operational complexity and the wrong-answer risk usually aren't worth it. Tune the threshold against a labeled pair set rather than picking a round number, keep anything personalized or time-bound out of the cache with a hard rule, and monitor for wrong hits in production by sampling cache returns. Get the threshold right and it's the cheapest cost cut available on an LLM bill. Get it wrong and you're shipping confident wrong answers at scale.
Sources
Cite this article
Kiril Urbonas (2026). Semantic Caching for LLM Apps: Cutting Cost on Repeated Queries. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/semantic-caching-for-llm-apps
[Semantic Caching for LLM Apps: Cutting Cost on Repeated Queries](https://www.devopsness.com/blog/semantic-caching-for-llm-apps) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)
More from AI
Explore more articles in this category
Production AI Pipelines: Building End-to-End ML Systems
We've shipped three end-to-end ML systems. The pieces that look obvious in slides and turn out to be the actual work.
Multi-Agent Systems Explained
A practical look at when multiple coordinating AI agents actually help, the orchestration patterns that work, and where they fail.
Guardrails for Production LLMs: Input and Output Filtering That Holds
A support bot that recites its system prompt and drafts refunds it isn't authorized to give has two holes. Two layers of guardrails, one on input, one on output, close both.
You might have missed
Evergreen posts worth revisiting.