A practitioner's guide to tracing, cost tracking, and evaluating LLM apps in production with Langfuse, Helicone, Arize Phoenix, and LangSmith.
Best LLM Observability Tools in 2026
The first time an LLM feature breaks in production, you reach for your normal observability stack and find it useless. Your APM shows a 200 response and a 900ms latency. The request succeeded. The model also invented a refund policy that does not exist, and you have no trace of why. The cost on that call was fine, but multiply it across a retry loop nobody noticed and you burned $400 overnight. Traditional monitoring watches the plumbing. LLM observability watches what came out of the pipe.
A normal request is deterministic enough that a status code and a latency number tell most of the story. An LLM call is not. The same prompt returns different text on every run, quality degrades silently when a provider ships a new model version, and a single user action can fan out into a chain of five model calls, three tool invocations, and a vector search. Watching that requires a different set of primitives.
Token and cost tracking: Spend is per-token and varies by model, so a small prompt change can double your bill. You need cost attributed per request, per user, and per feature, not a monthly invoice you reverse-engineer later.
Quality and drift: There is no exception when output gets worse. A prompt that scored well last month can regress after a model update. You need to sample outputs and score them over time.
Tracing multi-step chains: Agents and RAG pipelines are nested. When the final answer is wrong, you need to see which retrieval returned garbage or which tool call failed, not just the last message.
Prompt versioning: Prompts are code. When quality drops, the first question is "what changed in the prompt," and you cannot answer that without version history tied to traces.
Evals and latency: Offline test sets and online scoring, via heuristics, an LLM-as-judge, or human review, turn "it feels worse" into a number. And time to first token, streaming duration, and time lost in retrieval steps all matter separately, since a single latency figure hides all of it.
The short checklist: automatic tracing of nested calls, per-request cost, prompt management, an evals framework, dataset support for regression testing, and a way to get data in without rewriting your app. Self-hosting matters if your prompts or user data cannot leave your infrastructure. And the ingestion path matters more than people expect. A tool you instrument by hand gets adopted slowly; a proxy or an auto-instrumenting SDK gets adopted the same afternoon.
Langfuse is the open-source anchor of this list. It is self-hostable, and it covers tracing, prompt management, evals, and datasets in one place. You wrap your calls with its SDK or use its OpenTelemetry endpoint, and nested spans show up as a proper trace tree. It is the default when you want one tool that does most jobs with the option to run it on your own infrastructure.
from langfuse.openai import openai # drop-in wrapper
response = openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Summarize this ticket."}],
name="ticket-summary", # names the trace
metadata={"user_id": "u_8821"}, # attaches attributes for filtering
)
That import swap is the whole integration for the simple case. Cost, latency, token counts, and the prompt land in Langfuse automatically, and you add spans around retrieval or tool steps when you need the full chain.
Helicone takes the proxy route, and that is its whole appeal. You change your base URL and add a header, and every call flows through Helicone with cost and latency logged. No SDK, no code inside your request path. It is the fastest way to get cost tracking and basic logging, especially across a polyglot codebase where instrumenting each service is painful.
curl https://oai.helicone.ai/v1/chat/completions \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Helicone-Auth: Bearer $HELICONE_API_KEY" \
-H "Helicone-Property-Feature: ticket-summary" \
-d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}'
The tradeoff is that a proxy sits in your request path, so you are trusting its uptime and adding a hop. For deep multi-step tracing and rich evals it is lighter than Langfuse, though it has grown well beyond pure logging.
Arize Phoenix comes from the ML observability world, and it shows. Phoenix is the open-source, OpenTelemetry-native piece, strong on evals, embeddings analysis, and drift detection. If your problem is "is quality regressing and why," Phoenix leans harder into evaluation and dataset workflows than the cost-first tools. The commercial Arize platform extends this to production monitoring at scale. Reach for it when evals and drift are the center of gravity.
LangSmith is the LangChain-native option. If you already build with LangChain or LangGraph, tracing is close to free: set a few environment variables and every chain run is captured with no extra instrumentation. It has solid evals, prompt management, and a good debugging UI. The catch is gravity toward the LangChain ecosystem. It works outside it, but the smoothest experience assumes you are already inside.
Datadog, OpenTelemetry, and OpenLLMetry matter if you would rather not add a separate tool. Datadog has LLM Observability that fits model calls into the APM view you already watch, compelling when you want one pane of glass. OpenTelemetry is standardizing GenAI semantic conventions, so LLM spans increasingly look like any other span, and OpenLLMetry (from Traceloop) is an open-source SDK that auto-instruments common LLM and vector libraries and exports OTel spans to whatever backend you run, Langfuse and Datadog included. The pattern to notice: OpenTelemetry is becoming the shared wire format, so an OTel-compatible tool keeps you from getting locked in.
Cost-focused: You mostly need to know what you are spending and where. Start with Helicone for the near-zero-effort proxy, or Datadog if you already live there.
Eval-focused: Your risk is silent quality regression. Arize Phoenix or LangSmith give you the strongest evaluation and dataset tooling.
Self-hosted: Prompts or user data cannot leave your infrastructure. Langfuse is the default, with Phoenix as the eval-heavy alternative, both open-source and OTel-friendly.
One tool for most jobs: Langfuse covers tracing, prompts, cost, and evals in a single self-hostable package, which is why it wins the general case.
For most teams shipping LLM features, start with Langfuse. It is open-source, self-hostable, and covers the widest span of what you will actually need without committing you to one model provider or framework. Wire ingestion through OpenTelemetry or OpenLLMetry so your spans stay portable. If your only near-term problem is cost visibility, put Helicone in front first and add deeper tracing later. If you are eval-heavy or drowning in drift questions, lead with Arize Phoenix. And if your whole stack is LangChain, LangSmith is the path of least resistance. The mistake is treating this as a job for your existing APM alone. It is a real category now, and the teams that instrument it early catch the $400 retry loop before finance does.
For the broader stack these tools sit on top of, see our guide to the best LLM APIs and AI infrastructure. And if you want the traditional side of the house, here is our take on the best APM and observability tools.
