Users kept asking the same questions in slightly different words, and we paid full price every time. Semantic caching cut our LLM bill by a third.
Semantic Caching for LLM Apps: Cutting Cost on Repeated Queries
Our support assistant's monthly bill crossed a number that got a finance email forwarded to me. Digging into the logs, a huge share of traffic was the same handful of questions asked slightly differently: "how do I reset my password," "I can't log in and need a new password," "forgot password help." Exact-match caching caught none of these because the strings differed. Every rephrasing hit the model at full token cost for an answer we'd already generated a thousand times.
Semantic caching fixes this by keying the cache on meaning instead of exact text. You embed the incoming query, look for a stored query whose embedding is close enough, and if you find one, return the cached answer without calling the model at all.
Every cache entry stores three things: the original query text, its embedding vector, and the model's response. On a new query you embed it, run a nearest-neighbor search against stored embeddings, and check whether the closest match is within a similarity threshold.
from openai import OpenAI # embeddings; generation can be any provider
import numpy as np
client = OpenAI()
def embed(text: str) -> np.ndarray:
v = client.embeddings.create(
model="text-embedding-3-small",
input=text,
).data[0].embedding
return np.array(v, dtype=np.float32)
def cosine(a: np.ndarray, b: np.ndarray) -> float:
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
In production you don't loop over vectors in Python, you use a vector store. We use Redis with its vector search module because we already ran Redis, but pgvector or Qdrant work the same way. The lookup:
def lookup(query: str, threshold: float = 0.92):
qv = embed(query)
hit = redis_vector_knn(qv, k=1) # returns (stored_query, response, score)
if hit and hit.score >= threshold:
return hit.response
return None
That 0.92 is the single most important number in the system, and it's a knife's edge. Set it too low and the cache returns a plausible-but-wrong answer. "How do I cancel my subscription" and "how do I change my subscription" sit around 0.90 cosine similarity with text-embedding-3-small, and they need completely different answers. Return the cancel answer to someone asking to upgrade and you've created a support ticket, not saved one.
Set it too high, say 0.98, and you barely get any hits, because natural rephrasings rarely land that close. We landed on 0.94 after building a labeled set of 200 query pairs marked "same intent" or "different intent" and sweeping the threshold to maximize correct hits while keeping wrong hits near zero. Don't guess this. Measure it against your own traffic, because embedding geometry differs by domain.
# threshold sweep: pick the value that keeps false hits under 1%
for t in [0.90, 0.92, 0.94, 0.96]:
tp = sum(1 for p in pairs if p.same and cosine(embed(p.a), embed(p.b)) >= t)
fp = sum(1 for p in pairs if not p.same and cosine(embed(p.a), embed(p.b)) >= t)
print(f"threshold {t}: correct hits {tp}, wrong hits {fp}")
Anything personalized or time-sensitive must skip the cache entirely, or you'll serve one user's data to another. "What's my order status" embeds almost identically across every user, so a semantic cache would happily return the previous user's order. We route those queries around the cache with a simple classifier on the query, plus a hard rule: if the answer depends on user identity or current time, no caching. Ever.
An embedding call on text-embedding-3-small costs a tiny fraction of a full generation, roughly $0.00002 per query at current pricing versus a few cents for a Sonnet-class generation with a long system prompt. So even on a cache miss you've added negligible cost, and on a hit you've saved the entire generation.
Our hit rate settled at 31% on support traffic, which is high because support questions cluster hard. On our general-purpose assistant it was only 8%, because those queries are far more varied. The support bill dropped about 34% month over month. The general assistant barely moved, and honestly the added latency of the embedding lookup wasn't worth 8%, so we turned semantic caching off there and kept it on support only.
Latency is a nice side effect. A cache hit returns in about 40ms (embed plus vector search) versus 1.5 to 4 seconds for a full generation. Users notice.
Semantic caching pays off when your traffic is repetitive and impersonal, support FAQs, documentation Q&A, product questions. Measure your hit rate on real traffic before committing, because below roughly 15% the operational complexity and the wrong-answer risk aren't worth it. Tune the threshold against a labeled pair set rather than picking a round number, keep anything personalized or time-bound out of the cache with a hard rule, and monitor for wrong hits in production by sampling cache returns. Get the threshold right and it's the cheapest third you'll ever cut off an LLM bill. Get it wrong and you're shipping confident wrong answers at scale.
