Most agents that "need memory" actually need a smaller context window and a database. Here's how we cut a support agent's token bill by 60 percent by deleting memory.
Agent Memory: Short-Term, Long-Term, and When You Need Neither
A team handed me an agent that was spending 18,000 tokens per turn on a customer support flow. The "memory system" was a vector store stuffed with every past message, re-embedded and re-retrieved on every single turn. It was slow, it hallucinated old order numbers into new conversations, and it cost about $0.40 per resolved ticket. The fix wasn't a better memory system. It was less memory.
When someone says an agent "needs memory," they usually mean one of three unrelated things, and conflating them is where the architecture goes wrong.
Short-term memory is the current conversation. It lives in the context window. You don't need a database for it, you need to manage the message list and trim it. Long-term memory is facts that should survive across sessions: this user prefers metric units, that account is on the enterprise plan. Then there's the third case, which is really just data retrieval, where the agent needs to look something up (an order status, a doc) that was never "remembered" at all.
The support agent had jammed all three into one vector store, and the retrieval case was poisoning the conversation case.
For an active conversation, keep the raw messages. Embedding your own recent turns and retrieving them back is a common and expensive mistake. The model already has them. What you want is a trim strategy when the conversation gets long:
def build_messages(history, system, max_tokens=8000):
kept, running = [], count_tokens(system)
for msg in reversed(history):
t = count_tokens(msg["content"])
if running + t > max_tokens:
break
kept.append(msg)
running += t
kept.reverse()
# if we dropped older turns, prepend a rolling summary
if len(kept) < len(history):
summary = summarize(history[: len(history) - len(kept)])
kept.insert(0, {"role": "system", "content": f"Earlier context: {summary}"})
return [{"role": "system", "content": system}] + kept
Recency-first trimming plus a rolling summary of what got dropped covers the vast majority of chat agents. No vector store, no embedding calls, deterministic behavior.
The real long-term facts about a user are almost always structured and small. Preferences, plan tier, known account IDs. That's a row in Postgres keyed by user, not a fuzzy nearest-neighbor lookup.
CREATE TABLE user_memory (
user_id text PRIMARY KEY,
facts jsonb NOT NULL DEFAULT '{}',
updated_at timestamptz NOT NULL DEFAULT now()
);
-- write a fact
UPDATE user_memory
SET facts = facts || '{"units": "metric", "plan": "enterprise"}'::jsonb,
updated_at = now()
WHERE user_id = 'u_8812';
You load that one row at conversation start and inject it into the system prompt. It's exact, it's cheap, and it can't retrieve a stranger's data because the key is the user. Reach for embeddings only when the "memory" is genuinely unstructured and large, like recalling relevant snippets from months of prior conversations. That's rare, and it's a real feature decision, not a default.
The support agent's core job was "answer questions about this order." The order data lives in the orders service. That's a tool call, not memory:
tools = [{
"name": "get_order",
"description": "Fetch current status and line items for an order id",
"input_schema": {"type": "object", "properties": {
"order_id": {"type": "string"}}, "required": ["order_id"]},
}]
Once we let the agent call get_order on demand and stopped embedding the entire order history, the stale-order-number hallucinations vanished. The model wasn't confused anymore, it was asking the source of truth. Tokens per turn dropped from 18,000 to about 6,500, and cost fell roughly 60 percent.
Vector-store memory feels sophisticated, so it gets added early and questioned late. But every retrieved chunk is a chance to inject wrong or stale context, and retrieval quality on your own conversation history is usually mediocre. Each memory layer you add is a new failure mode: stale writes, wrong-user leakage, retrieval that surfaces the wrong turn.
Default to no memory. Manage the context window with recency trimming and a rolling summary. When facts must persist across sessions, put them in a keyed table, not embeddings. Use a vector store only when you're recalling large unstructured history and you've measured that retrieval actually helps. For anything the agent can look up live, give it a tool and let it fetch. The best memory system is often the one you didn't build.
