Skip to main content
Agent Memory: Short-Term, Long-Term, and When You Need Neither

Agent Memory: Short-Term, Long-Term, and When You Need Neither

KU
· Founder & Chief Editor
• 7 min read•Updated •42 views

Most agents that "need memory" actually need a smaller context window and a database. Here's how we cut a support agent's token bill by deleting memory.

Key takeaways

  • Most agents need less memory than they get.
  • Handle short-term memory by trimming the message list and summarizing older turns, store long-term facts in a keyed database row per user, and fetch live data such as order status with a tool call.
  • Use a vector store only for large, unstructured history where retrieval measurably helps.

Most agents need less memory than they have. Short-term memory is the context window: trim it and summarize what you drop. Long-term memory is a keyed database row per user. Anything the agent can look up live, like an order status, is a tool call, not memory. A vector store is the exception, not the default.

Verified against

Claude API context window, tool use and memory tool docs (platform.claude.com), and PostgreSQL 18 (18.6) JSON types and functions docs. Checked October 2026.

We learned this on a customer support agent that was spending a five-figure number of tokens on every turn. Its "memory system" was a vector store stuffed with every past message, re-embedded and re-retrieved on every single turn. It was slow, it hallucinated old order numbers into new conversations, and its cost per resolved ticket was hard to justify. The fix wasn't a better memory system. It was less memory.

The three things people call memory

When someone says an agent "needs memory," they usually mean one of three unrelated things, and conflating them is where the architecture goes wrong.

Short-term memory is the current conversation. It lives in the context window. You don't need a database for it, you need to manage the message list and trim it. Long-term memory is facts that should survive across sessions: this user prefers metric units, that account is on the enterprise plan. Then there's the third case, which is really just data retrieval, where the agent needs to look something up (an order status, a doc) that was never "remembered" at all.

The support agent had jammed all three into one vector store, and the retrieval case was poisoning the conversation case.

Short-term: manage the window, don't embed it

For an active conversation, keep the raw messages. Embedding your own recent turns and retrieving them back is a common and expensive mistake. The model already has them. Bigger windows don't remove the need to trim: Anthropic's context window docs note that accuracy and recall degrade as token count grows ("context rot"), so curating what goes in matters as much as how much fits. What you want is a trim strategy when the conversation gets long:

agent/context.py
def build_messages(history, system, max_tokens=8000):
    kept, running = [], count_tokens(system)
    for msg in reversed(history):
        t = count_tokens(msg["content"])
        if running + t > max_tokens:
            break
        kept.append(msg)
        running += t
    kept.reverse()
    # if we dropped older turns, prepend a rolling summary
    if len(kept) < len(history):
        summary = summarize(history[: len(history) - len(kept)])
        kept.insert(0, {"role": "system", "content": f"Earlier context: {summary}"})
    return [{"role": "system", "content": system}] + kept

Recency-first trimming plus a rolling summary of what got dropped covers the vast majority of chat agents. No vector store, no embedding calls, deterministic behavior. If you are on a provider that offers server-side compaction (Anthropic's API has it in beta as of October 2026), that is the same idea run for you, and worth trying before you write your own summarizer. Our notes on token budgeting for long-context prompts cover how to size max_tokens here.

Long-term: a keyed table beats a vector store

The real long-term facts about a user are almost always structured and small. Preferences, plan tier, known account IDs. That's a row in Postgres keyed by user, not a fuzzy nearest-neighbor lookup.

db/user_memory.sql
CREATE TABLE user_memory (
  user_id     text PRIMARY KEY,
  facts       jsonb NOT NULL DEFAULT '{}',
  updated_at  timestamptz NOT NULL DEFAULT now()
);

-- write a fact (jsonb || merges keys, right side wins)
UPDATE user_memory
SET facts = facts || '{"units": "metric", "plan": "enterprise"}'::jsonb,
    updated_at = now()
WHERE user_id = 'u_8812';

You load that one row at conversation start and inject it into the system prompt. It's exact, it's cheap, and it can't retrieve a stranger's data because the key is the user. Reach for embeddings only when the "memory" is genuinely unstructured and large, like recalling relevant snippets from months of prior conversations. That's rare, and it's a real feature decision, not a default.

When you need neither

The support agent's core job was "answer questions about this order." The order data lives in the orders service. That's a tool call, not memory:

agent/tools.py
tools = [{
  "name": "get_order",
  "description": "Fetch current status and line items for an order id",
  "input_schema": {"type": "object", "properties": {
      "order_id": {"type": "string"}}, "required": ["order_id"]},
}]

Once we let the agent call get_order on demand and stopped embedding the entire order history, the stale-order-number hallucinations vanished. The model wasn't confused anymore, it was asking the source of truth. Tokens per turn dropped to roughly a third of what they had been, and the bill followed.

The trap of "just add memory"

Vector-store memory feels sophisticated, so it gets added early and questioned late. But every retrieved chunk is a chance to inject wrong or stale context, and retrieval quality on your own conversation history is usually mediocre. Each memory layer you add is a new failure mode: stale writes, wrong-user leakage, retrieval that surfaces the wrong turn.

If you do add retrieval, measure it like any other retrieval system. The offline harness we use for RAG evaluation works for memory recall too: a fixed set of past conversations, the facts a correct answer needs, and a score for whether retrieval surfaced them. If recall on that set is poor, the memory layer is adding noise, not context.

Questions people ask

What is the difference between short-term and long-term memory in AI agents?

Short-term memory is the current conversation held in the model's context window, managed by trimming and summarizing the message list. Long-term memory is information that survives across sessions, such as a user's preferences or plan tier. The first needs no storage beyond your message history; the second usually fits in one database row keyed by user ID, loaded at the start of each session.

Do AI agents need a vector database for memory?

Usually not. Structured facts about a user belong in a keyed table, and live business data belongs behind a tool call. A vector store earns its place only when the agent must recall large amounts of unstructured history, such as relevant passages from months of prior conversations, and you have measured that retrieval improves answers. Added by default, it mostly injects stale or irrelevant context.

How do you stop an agent's context window from growing forever?

Keep the most recent turns verbatim, drop older ones once you pass a token budget, and replace what you dropped with a short rolling summary. Every turn adds input tokens, so without trimming, cost and latency grow with conversation length. Provider features such as server-side compaction do the same job automatically and are worth testing before writing your own.

The call we'd make

Default to no memory. Manage the context window with recency trimming and a rolling summary. When facts must persist across sessions, put them in a keyed table, not embeddings. Use a vector store only when you're recalling large unstructured history and you've measured that retrieval actually helps. For anything the agent can look up live, give it a tool and let it fetch. The best memory system is often the one you didn't build.

Sources

Explore topics:LLMAI
React
Share this post
Cite this article
Plain text

Kiril Urbonas (2026). Agent Memory: Short-Term, Long-Term, and When You Need Neither. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/agent-memory-short-term-long-term

Markdown

[Agent Memory: Short-Term, Long-Term, and When You Need Neither](https://www.devopsness.com/blog/agent-memory-short-term-long-term) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)

KU

Kiril Urbonas

Founder & Chief Editor

604 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.