Skip to main content
Building Production-Ready AI Applications with LangChain and Docker

Building Production-Ready AI Applications with LangChain and Docker

KU
· Founder & Chief Editor
• 8 min read•Updated •

We deploy LangChain apps in Docker on Kubernetes. The patterns that work, the LangChain-specific gotchas, and what we'd build differently next time.

Deploying LangChain Apps in Production with Docker

LangChain is the framework most teams reach for when building LLM-powered apps. We have several LangChain-based services in production, all containerized and running on Kubernetes. The framework helps with prototyping; production introduces specific challenges. This post is the practical version of getting from "it works on my machine" to "it serves real customers reliably."

Why containerize LangChain apps

Containers solve the standard set of problems:

  • Reproducible runtime (Python version, dependencies, system libs)
  • Isolation between services on the same host
  • Standard deployment pipeline (same as our other services)
  • Resource limits

For Python apps with heavy dependencies (LangChain itself plus the various integrations: openai, anthropic, pinecone, qdrant, etc.), the dependency tree is messy. Pinning everything in requirements.txt and baking it into a container is the only sane way to ship.

The Dockerfile

A working Dockerfile for a LangChain service:

Dockerfile
# Stage 1: build with toolchain for compiling C extensions
FROM python:3.12-slim AS build
WORKDIR /src
RUN apt-get update && apt-get install -y --no-install-recommends \
    gcc g++ libffi-dev && \
    rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

# Stage 2: runtime
FROM python:3.12-slim AS runtime
WORKDIR /app
COPY --from=build /root/.local /root/.local
COPY . .
ENV PATH=/root/.local/bin:$PATH
ENV PYTHONUNBUFFERED=1
USER 1000:1000
CMD ["uvicorn", "myapp:app", "--host", "0.0.0.0", "--port", "8000"]

Multi-stage build keeps the runtime image small. With LangChain + OpenAI + a vector DB client + FastAPI, the image is ~450MB. Larger than I'd like but manageable.

PYTHONUNBUFFERED=1 ensures logs flush immediately. Without it, logs sometimes don't appear until the container exits — bad for debugging.

requirements.txt: pin everything

LangChain moves fast. Breaking changes happen between minor versions. Loose pins (langchain>=0.1.0) cause "it worked yesterday" surprises.

We pin everything:

code
langchain==0.3.7
langchain-openai==0.3.0
langchain-anthropic==0.3.1
openai==1.40.6
anthropic==0.32.0
fastapi==0.115.5
uvicorn==0.32.0
pydantic==2.10.0
qdrant-client==1.12.1

When we want to upgrade, it's an explicit PR with version bumps and any code changes needed. Not "oh we picked up a new version randomly."

We use pip-compile (from pip-tools) to generate locked files from a higher-level requirements.in. This handles transitive dependencies cleanly.

Configuration: env, not code

LangChain apps need lots of config: API keys, model names, endpoints, vector DB credentials, etc. All of these come from environment variables, not hardcoded:

Python
import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    model=os.environ.get("LLM_MODEL", "gpt-4o-mini"),
    temperature=float(os.environ.get("LLM_TEMPERATURE", "0")),
)

For development, a .env file (loaded via python-dotenv in dev only). For production, env vars come from Kubernetes Secrets (populated by ESO from AWS Secrets Manager).

The pattern: code reads env vars; deployment provides them. The same image runs in dev, staging, and prod with different env values.

Health checks

LangChain apps need a health endpoint. Standard:

Python
@app.get("/health")
async def health():
    # Light check: process is up and able to respond
    return {"status": "ok"}

@app.get("/ready")
async def ready():
    # Deeper check: dependencies are reachable
    try:
        # cheap call to verify LLM provider is reachable
        await llm.ainvoke("ok")
        return {"status": "ready"}
    except Exception as e:
        return Response(status_code=503, content=str(e))

/health is fast and cheap; used by Kubernetes liveness probe. /ready is more expensive; used by Kubernetes readiness probe (and called less often).

The split matters. Liveness probes that hit a slow /ready cause unnecessary pod restarts. A typical loop: a slow LLM call makes the liveness probe time out, kubelet restarts the pod, repeat. Keep liveness cheap.

Timeout discipline

LangChain calls can take a long time. Without timeouts, requests can hang indefinitely. We set timeouts at multiple layers:

Python
llm = ChatOpenAI(timeout=30)  # 30s timeout per LLM call

# At the FastAPI route level
@app.post("/chat")
async def chat(req: ChatRequest):
    async with asyncio.timeout(60):  # 60s total request budget
        return await chain.ainvoke(req.input)

Plus Kubernetes readiness probe timeouts, ingress timeouts, etc. The shortest timeout in the chain wins.

A specific gotcha: the OpenAI Python SDK's default timeout is high (10 minutes for some operations). Always set explicit timeouts.

Streaming responses

For chat-style features, users want incremental responses (the LLM's output streams in). LangChain supports streaming; FastAPI does too.

Python
@app.post("/chat")
async def chat(req: ChatRequest):
    async def generate():
        async for chunk in chain.astream(req.input):
            yield f"data: {json.dumps({'text': chunk.content})}\n\n"
    return StreamingResponse(generate(), media_type="text/event-stream")

Server-Sent Events (SSE) is the simplest streaming protocol. WebSockets work too but are heavier.

The connection lifecycle: keep streaming until the LLM is done; close cleanly. Cancel handling matters — if the client disconnects, we should stop the LLM call (saves cost).

Logging and tracing

Standard structured logging. Per-call:

Python
logger.info("llm_call", extra={
    "model": "gpt-4o-mini",
    "input_tokens": response.usage.prompt_tokens,
    "output_tokens": response.usage.completion_tokens,
    "duration_ms": elapsed * 1000,
    "request_id": request_id,
    "user_id": user_id,
})

Plus OpenTelemetry tracing for distributed traces. We use the opentelemetry-instrumentation-langchain package which adds spans for every LangChain operation.

The tracing visibility is essential. When a user reports a slow response, we can see exactly which LLM call took how long, what was retrieved, where the latency went.

Deployment on Kubernetes

The Kubernetes Deployment:

YAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chatbot
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: our-registry/chatbot:v1.42.0
        resources:
          requests:
            cpu: 250m
            memory: 512Mi
          limits:
            cpu: 1000m
            memory: 1Gi
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /ready
            port: 8000
          periodSeconds: 10
        env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: llm-secrets
              key: openai-api-key

Standard pattern. Resources set conservatively at 250m / 512Mi — most LangChain apps don't need much CPU per replica because the work is I/O-bound (waiting for LLM responses).

Specific LangChain gotchas

Things that have bitten us:

Async vs sync mixing. LangChain has both sync (invoke) and async (ainvoke) variants. Mixing them in one chain causes thread-pool starvation. Pick async; stay async throughout.

Memory growth in long chains. Some LangChain memory implementations grow unbounded in long conversations. We cap conversation length explicitly and summarize older messages.

Default verbose logging. verbose=True on a chain logs the full prompt and response. Useful in development; disastrous in production (PII, secrets, tokens).

Output parsers that break on edge cases. LangChain's structured output parsers expect specific formats. When the LLM returns slightly off-format responses, parsers crash. We wrap parsers in try-except + fallback.

Tool/function calling cost. Each tool the model can call adds tokens to every prompt. With 20 tools defined, every call costs more than expected. We limit tools to ~5 per chain when possible.

Caching to save money

LangChain has cache abstractions (InMemoryCache, RedisCache, SQLiteCache). For production, Redis is the right one:

Python
from langchain.globals import set_llm_cache
from langchain_community.cache import RedisCache

set_llm_cache(RedisCache(redis_=redis_client))

The cache is keyed on (model, prompt). Identical (model, prompt) → cached response. Useful for:

  • Repeated identical queries from different users
  • Prompts that include large stable contexts (the cache keys on the full prompt; small input changes invalidate)

Hit rate for our chatbot service: ~8%. Saves ~$80/month at our scale.

Versioning

LangChain itself is moving fast. Versioning matters:

  • Pin LangChain version in requirements.txt
  • When upgrading, run regression tests (we have a small set of "expected" inputs/outputs we check)
  • Major LangChain version bumps are their own deploy event, not bundled with feature changes

What we'd do differently next time

Looking back:

Used LangChain LCEL (Expression Language) from the start, not the older Chain APIs. LCEL is the future direction; older patterns are being deprecated. Migrating mid-project is annoying.

Built our own narrow abstractions earlier. LangChain provides building blocks; for our specific use cases, a thin wrapper over the OpenAI/Anthropic SDKs would have been less framework-coupling. We use LangChain for some things; pure SDK for others.

Logged the raw prompts from day one. Debugging quality issues without seeing the actual prompt sent to the LLM is hard. We log prompt now; should have from the start.

Set up evals before iterating. We added the regression test suite later. Should have been on day one — much faster iteration with eval safety net.

What I'd tell a team starting

Pin every dependency. LangChain breaks between minor versions; upgrades should be explicit.

Health check splits matter. Cheap liveness, deeper readiness.

Timeouts everywhere. LLM calls can hang; without timeouts, your app does too.

Log the prompt + response. Not always, but for debugging investigations.

Async throughout. Don't mix sync and async in one chain.

Cache where it makes sense. Redis-backed cache for repeated queries.

Build evals before iterating. Each prompt change should run against a regression suite.

LangChain is one of those frameworks where the "getting started" experience is good and the "running it in production" experience is harder. The patterns above are the operational discipline that makes it work. With them, LangChain in production is mostly fine. Without them, you get the everyday LLM-app problems amplified by framework complexity.

Explore topics:DockerLLMKubernetesAI
React
Share this post
Cite this article
Plain text

Kiril Urbonas (2024). Building Production-Ready AI Applications with LangChain and Docker. DevOpsNess. Updated 2026-11-20. https://www.devopsness.com/blog/production-ai-applications-langchain-docker

Markdown

[Building Production-Ready AI Applications with LangChain and Docker](https://www.devopsness.com/blog/production-ai-applications-langchain-docker) (Kiril Urbonas, DevOpsNess, updated 2026-11-20)

KU

Kiril Urbonas

Founder & Chief Editor

608 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.