Skip to main content
_d
devops/ness

Learn MLOps and LLMOps: RAG, Evals, Serving and LLM Observability in Production

Learn production LLM engineering in four stages: build a small RAG app, measure whether its answers are right, serve and route models reliably, then monitor cost, latency and failures. Each step links a guide with runnable code and the metrics we track.

Python is assumed. Model and price examples are dated in each post because they change monthly.

  1. Stage 1: Foundations

    Embeddings and a first retrieval-augmented app.

    1. 1What are embeddings?Understand vector similarity.
    2. 2Build your first RAG app in PythonIndex documents and answer questions over them.
    3. 3RAG chunking strategiesPick a chunking method by document type.
    4. 4RAG vs fine-tuningChoose the right approach for the problem.

  2. Stage 2: Quality

    Prove the answers are right before users see them.

    1. 5Debugging RAG retrievalFind why retrieval returns the wrong chunks.
    2. 6An offline RAG evaluation harnessScore retrieval with recall and MRR.
    3. 7LLM evals in CIBlock prompt regressions in pull requests.
    4. 8Production RAG reliabilityGround and cite answers.

  3. Stage 3: Serving

    Run and route models reliably.

    1. 9Ollama vs vLLMPick a local or production inference server.
    2. 10Multi-provider LLM gatewaysRoute and fail over between providers.
    3. 11Model registry vs MLflow trackingVersion models and experiments.

  4. Stage 4: Production

    Cost, safety and observability.

    1. 12LLM cost optimization in productionCut token spend without losing quality.
    2. 13Prompt cachingReuse prefixes to cut latency and cost.
    3. 14Guardrails for production LLMsFilter inputs and outputs.
    4. 15Best LLM observability toolsMonitor cost, latency and failures.

Keep these open

Every guide on this path names the versions and docs it was checked against and lists its sources at the end. Browse all llm guides or another learning path.