Ollama vs vLLM — Which to Use for Serving LLMs

People keep framing this as a fight, and it isn't one. Ollama and vLLM solve different problems and mostly hand off to each other. I've watched teams reach for vLLM to run a model on a laptop and fight CUDA for an afternoon, and watched teams push 500 concurrent users through Ollama and wonder why latency fell off a cliff. Pick the wrong side of the split and you feel it fast.

Here's the split in one line. Ollama: get a model running locally with almost no ceremony. vLLM: squeeze maximum throughput out of GPUs when many users hit the same endpoint.

Ollama: the laptop-friendly path #

Ollama is built for developer experience. You install it, you pull a model, you run it. It wraps llama.cpp underneath, ships GGUF quantized weights, and manages the model files, prompt templates, and a local API server for you. It runs on CPU, on Apple Metal, and on a single consumer GPU without you thinking about drivers.

bash.bash

# Pull and chat with a quantized model
ollama run llama3.1:8b

# Or hit its OpenAI-compatible endpoint
curl http://localhost:11434/v1/chat/completions \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Summarize this log line"}]
  }'

That's the whole story for a prototype. Models are quantized by default (4-bit is common), so an 8B model fits comfortably on a 16GB MacBook and a 70B model runs, slowly, on a machine with enough RAM. The tradeoff is concurrency. Ollama handles one or a few requests at a time well, but it does not do the aggressive batching that keeps a GPU busy under real load. Fire twenty simultaneous requests at it and they largely queue.

vLLM: built for throughput #

vLLM is the opposite optimization. It assumes you have server GPUs and a lot of traffic, and it is engineered to serve that traffic efficiently. Two ideas do most of the work.

PagedAttention: manages the KV cache like an operating system manages virtual memory, in pages instead of one contiguous block per request. That kills the memory fragmentation that normally wastes GPU RAM, so you fit far more concurrent sequences on the same card.

Continuous batching: instead of waiting for a batch to fill and finish together, vLLM adds and removes requests from the running batch token by token. A short request doesn't get stuck behind a long one. This is where the throughput gap comes from. Under concurrency vLLM routinely does multiples of what a naive server manages on identical hardware.

It also supports tensor parallelism to shard a model across multiple GPUs, so you can serve a 70B model across four cards. And the server speaks the OpenAI API, so your existing client code barely changes.

bash.bash

# Serve a model with an OpenAI-compatible API
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 1 \
  --max-model-len 8192

python.python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Classify this ticket"}],
)
print(resp.choices[0].message.content)

The cost is setup and hardware. vLLM wants a proper NVIDIA GPU (or AMD ROCm), matching CUDA drivers, and enough VRAM to hold the model plus KV cache. It is not something you casually run on a laptop, and it is not trying to be.

Throughput and concurrency #

This is the real dividing line. For a single user waiting on one reply, both feel fine, and Ollama's simpler stack often wins on time-to-first-token. The picture flips with concurrent users. vLLM's batching keeps the GPU saturated and total tokens-per-second climbs as you add load, up to the card's limit. Ollama's throughput stays roughly flat because it isn't batching aggressively, so per-user latency degrades as concurrency rises.

Put simply: one user, either works. Many users on one GPU, vLLM is the tool built for that job.

Hardware and quantization #

Hardware need is the other honest difference. Ollama meets you where you are: a laptop, a Mac, a CPU-only box, a single gaming GPU. vLLM expects datacenter or workstation GPUs and rewards more of them. If you're picking rented hardware for the vLLM side, our GPU cloud comparison walks through the tradeoffs.

On quantization, both support it but with different defaults. Ollama is quantized-first through GGUF, which is how it fits big models on small machines. vLLM historically favored full or half precision for maximum quality and speed, and has since added quantized formats like AWQ, GPTQ, and FP8. So you can run quantized on vLLM, but quantization is Ollama's whole reason for fitting on your laptop, and an optimization on vLLM.

Both speak OpenAI #

Worth calling out because it makes migration cheap. Both expose an OpenAI-compatible /v1/chat/completions endpoint. You can prototype against Ollama locally, then point the same client at a vLLM server in staging by changing a base URL. That interoperability is a big part of why the two live together so comfortably in one workflow.

The alternatives #

The landscape is wider than these two. TGI (Hugging Face Text Generation Inference) is a close vLLM competitor for production serving. SGLang is a newer high-throughput server, strong on structured output and complex prompting. llama.cpp is the engine under Ollama if you want the raw layer. LM Studio is a desktop GUI in Ollama's local-first niche for people who'd rather not touch a terminal. None of them change the core split; they just give you more points on the same spectrum. For the broader map, see our roundup of the best LLM APIs and AI infrastructure.

The decision #

Reach for Ollama when you're prototyping, building a local dev loop, running a single-user or low-traffic internal tool, working offline, or on hardware without a server GPU. Reach for vLLM when you're serving many concurrent users, need high tokens-per-second per dollar, are running on real GPUs, or want tensor parallelism for large models behind a stable OpenAI-compatible API.

A pattern that works well: build and iterate against Ollama on the laptop, then deploy the same model through vLLM once you have real traffic. Same API, different engine.

The call we'd make #

If it runs on your machine and one person uses it at a time, use Ollama. Don't overthink it. The DX is the point, and you'll ship faster.

If it's a service other people or systems call, and you care about cost per token under load, use vLLM. The setup tax is real but you pay it once, and PagedAttention plus continuous batching save far more in GPU hours than you spent configuring it. The mistake to avoid is one tool for both jobs. Prototype on Ollama, serve on vLLM, and let each do what it's good at.

#Inference

Ollama vs vLLM — Which to Use for Serving LLMs