# Semantic Caching for LLM Apps: Cutting Cost on Repeated Queries

URL: https://www.devopsness.com/blog/semantic-caching-for-llm-apps
Author: Kiril Urbonas
Published: 2026-07-01
Updated: 2026-10-31
Categories: AI
Tags: LLM, Caching, Cost Optimization, Production, AI

Summary: Users ask the same questions in slightly different words, and you pay full price every time. Semantic caching answers the repeats for almost nothing.

---
# Semantic Caching for LLM Apps: Cutting Cost on Repeated Queries

Semantic caching cuts LLM cost on repeated queries by keying the cache on meaning instead of exact text: embed the incoming query, find the closest stored query in a vector store, and return its cached answer when cosine similarity clears a threshold you tuned on labeled pairs. It pays off on repetitive, impersonal traffic and is dangerous on anything personalized.

> [!VERIFIED]
> OpenAI `text-embedding-3-small` (1,536 dimensions, 8,192-token input), Redis vector search, pgvector, against the OpenAI embeddings guide, redis.io and the pgvector README. Checked October 2026.

The pattern shows up on almost every support assistant. A large share of traffic is the same handful of questions asked slightly differently: "how do I reset my password," "I can't log in and need a new password," "forgot password help." Exact-match caching catches none of these because the strings differ. Every rephrasing hits the model at full token cost for an answer you've already generated a thousand times.

## The mechanism

Every cache entry stores three things: the original query text, its embedding vector, and the model's response. On a new query you embed it, run a nearest-neighbor search against stored embeddings, and check whether the closest match is within a similarity threshold.

```python
# semantic_cache.py
from openai import OpenAI  # embeddings; generation can be any provider
import numpy as np

client = OpenAI()

def embed(text: str) -> np.ndarray:
    v = client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
    ).data[0].embedding
    return np.array(v, dtype=np.float32)

def cosine(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
```

In production you don't loop over vectors in Python, you use a vector store. Redis with vector search is the easy pick if you already run Redis, but pgvector or Qdrant work the same way. The lookup:

```python
def lookup(query: str, threshold: float = 0.92):
    qv = embed(query)
    hit = redis_vector_knn(qv, k=1)   # returns (stored_query, response, score)
    if hit and hit.score >= threshold:
        return hit.response
    return None
```

This is a different thing from provider-side prompt caching, which discounts a repeated prompt prefix but still runs a generation. Our guide to [prompt caching in production LLM apps](/blog/prompt-caching-production-llm-apps) covers that layer; the two stack well, since semantic caching skips the call entirely on a hit and prompt caching cheapens the misses.

## The threshold is everything

That `0.92` is the single most important number in the system, and it's a knife's edge. Set it too low and the cache returns a plausible but wrong answer. "How do I cancel my subscription" and "how do I change my subscription" share almost every word and can score very close on cosine similarity, yet they need completely different answers. Return the cancel answer to someone asking to upgrade and you've created a support ticket, not saved one.

Set it too high, say 0.98, and you barely get any hits, because natural rephrasings rarely land that close. The way to pick it: build a labeled set of a few hundred query pairs marked "same intent" or "different intent" from your own logs, then sweep the threshold to maximize correct hits while keeping wrong hits near zero. Don't guess this, and don't copy ours. Embedding geometry differs by domain and by model.

```python
# threshold sweep: pick the value that keeps false hits under 1%
for t in [0.90, 0.92, 0.94, 0.96]:
    tp = sum(1 for p in pairs if p.same and cosine(embed(p.a), embed(p.b)) >= t)
    fp = sum(1 for p in pairs if not p.same and cosine(embed(p.a), embed(p.b)) >= t)
    print(f"threshold {t}: correct hits {tp}, wrong hits {fp}")
```

Re-run the sweep whenever you change embedding model. A threshold tuned on one model means nothing on another, and switching models also invalidates every stored vector, so plan a cache flush with the migration.

## What to never cache

Anything personalized or time-sensitive must skip the cache entirely, or you'll serve one user's data to another. "What's my order status" embeds almost identically across every user, so a semantic cache would happily return the previous user's order. Route those queries around the cache with a simple classifier on the query, plus a hard rule: if the answer depends on user identity or current time, no caching. Ever.

Set a TTL on everything else, too. A cached answer about pricing or a feature that changed last week is a wrong answer with good latency.

## The numbers

Embeddings are cheap. OpenAI's embeddings guide lists `text-embedding-3-small` at about 62,500 pages per dollar (roughly 800 tokens a page), which works out to about $0.02 per million tokens as of October 2026. A 20-token query costs a tiny fraction of a cent to embed, while a full generation with a long system prompt costs orders of magnitude more. So a cache miss adds negligible cost, and a hit saves the entire generation.

Hit rate is what decides whether it's worth it, and it depends entirely on how clustered your traffic is. Support and documentation Q&A cluster hard and can see a meaningful share of hits. A general-purpose assistant sees far more varied queries, and a single-digit hit rate there may not justify the extra lookup latency or the wrong-answer risk. Measure on a week of real traffic before you build anything: embed historical queries, simulate the cache at your chosen threshold, and count.

Latency is a side benefit. A hit costs one embedding call plus one vector search, typically tens of milliseconds, against seconds for a full generation. Users notice. For the wider set of levers beyond caching, see our notes on [LLM cost optimization in production](/blog/llm-cost-optimization-in-production).

## Questions people ask

### What similarity threshold should a semantic cache use?

There is no universal value. Build a labeled set of a few hundred query pairs from your own traffic, marked same intent or different intent, and sweep thresholds to find the lowest one that keeps wrong hits under about 1%. Sweeping 0.90 to 0.96 is a reasonable starting range for cosine similarity, and you must re-tune after any embedding model change.

### Is semantic caching the same as prompt caching?

No. Prompt caching is a provider feature that bills a repeated prompt prefix at a discount, but the model still generates a fresh answer. Semantic caching is your own layer that matches a new query to a previously answered one and returns the stored answer without calling the model at all. They solve different costs and work well together.

### What should never go in a semantic cache?

Anything whose correct answer depends on who is asking or when. Order status, account balances, "my" anything, and questions about live data embed almost identically across users, so a cache will serve one user's answer to another. Route those around the cache with a classifier and a hard rule, and put a TTL on everything else so stale answers expire.

## The call we'd make

Semantic caching pays off when your traffic is repetitive and impersonal: support FAQs, documentation Q&A, product questions. Measure your hit rate on real traffic before committing, because below roughly 15% the operational complexity and the wrong-answer risk usually aren't worth it. Tune the threshold against a labeled pair set rather than picking a round number, keep anything personalized or time-bound out of the cache with a hard rule, and monitor for wrong hits in production by sampling cache returns. Get the threshold right and it's the cheapest cost cut available on an LLM bill. Get it wrong and you're shipping confident wrong answers at scale.

## Sources

- [OpenAI API Docs: Vector embeddings guide](https://developers.openai.com/api/docs/guides/embeddings)
- [Redis Docs: Vector search](https://redis.io/docs/latest/develop/ai/search-and-query/vectors/)
- [pgvector: Open-source vector similarity search for Postgres](https://github.com/pgvector/pgvector)

