The top model on the MTEB leaderboard made our search worse and our bill bigger. Here's how we actually picked an embedding model for a real RAG system.
Choosing an Embedding Model: Dimensions, Cost, and MTEB Reality
We picked our first embedding model the lazy way: opened the MTEB leaderboard, took the model at the top, shipped it. Retrieval quality on our actual support docs was mediocre, latency was up, and the vector index had ballooned because the model output 1,536-dimension vectors for a corpus where a 384-dim model would have done fine. The leaderboard number was real. It just had almost nothing to do with our problem.
The MTEB leaderboard ranks models on an average across dozens of tasks: classification, clustering, retrieval, reranking, across many domains and languages. Your RAG system is one task (retrieval), in one domain (yours), usually in one language. A model that wins the overall average can lose badly on retrieval for technical documentation, because its edge came from clustering tweets or classifying sentiment.
Look at the retrieval sub-score, not the headline average, and specifically the retrieval datasets closest to your domain. Even then, treat it as a shortlist, not an answer. The only benchmark that counts is your own documents with your own queries.
You do not need a fancy harness. Take 50 real queries from your logs, label the document that should come back for each, and measure recall@5 for each candidate model. This took us an afternoon and completely changed the decision.
def recall_at_k(model, queries, k=5):
hits = 0
for q in queries:
q_vec = model.encode(q["text"])
top = index.search(q_vec, k)
if q["expected_doc_id"] in [r.id for r in top]:
hits += 1
return hits / len(queries)
for name in ["bge-small-en-v1.5", "text-embedding-3-small",
"text-embedding-3-large", "e5-base-v2"]:
print(name, recall_at_k(load(name), queries))
On our data, bge-small-en-v1.5 (384 dims, runs locally) scored recall@5 of 0.86. text-embedding-3-large (3,072 dims, paid API) scored 0.88. Two points of recall for eight times the dimensions and an API bill. That's not a trade we'd make.
Every dimension is bytes in the index and math on every query. A million chunks at 1,536 dims in float32 is about 6 GB in memory before overhead; at 384 dims it's 1.5 GB. That difference decides whether the index fits on a cheaper instance and how fast nearest-neighbor search runs.
Newer API models support dimension truncation (Matryoshka embeddings), where you can ask for fewer dimensions and keep most of the quality:
# request 512 dims instead of the full 3072
resp = client.embeddings.create(
model="text-embedding-3-large",
input=chunk,
dimensions=512,
)
We tested 512-dim truncated vectors and lost less than one point of recall versus the full 3,072. If you're set on a large model, truncate aggressively and re-measure. Most people leave dimensions at the default and pay for precision their retrieval never uses.
There's the embedding cost (per token, at ingestion and per query) and the storage/serving cost (index size, forever). The API model charged us per query embedding, which sounds tiny until you multiply by traffic. A local model like bge runs on a CPU or a small GPU you already have, so per-query cost is effectively zero after the box is up. For our volume, roughly 400,000 queries a month, the local model saved a few hundred dollars a month and cut p95 query latency because we skipped a network round trip to the embedding API.
The catch with local models is you own the serving. If you don't want to run inference, the managed API is worth the money, especially at low volume where the box would sit idle.
One more thing the leaderboard hides: some models are trained for asymmetric search (short query, long document) and expect a prefix. The e5 and bge families want query: and passage: prefixes, and forgetting them quietly tanks recall.
q_vec = model.encode("query: how do I rotate an API key")
d_vec = model.encode("passage: To rotate a key, open Settings...")
We lost half a day to bad results before realizing we'd embedded queries without the prefix. If your model's card mentions prefixes, use them, and make sure ingestion and query time agree.
Ignore the MTEB headline number. Pull the retrieval sub-scores to build a shortlist of three or four models, then run a 50-query recall@5 eval on your own data, which is the only measurement that predicts production. Default to a small local model like bge-small unless the eval shows a real gap, because the dimensions you save are storage and latency you keep forever. If you go with a large API model, truncate the dimensions and re-measure. And check for prefix requirements before you blame the model. We shipped 384 dims running locally and never looked back at the leaderboard.
