# Self-Hosted LLMs vs OpenAI API: A Cost-vs-Latency Analysis After 6 Months

URL: https://www.devopsness.com/blog/self-hosted-llms-vs-openai-api-a-cost-vs-latency-analysis-after-6-months-2026-04-11
Author: Kiril Urbonas
Published: 2026-04-11
Updated: 2026-04-11
Categories: AI
Tags: Python, LLM, GPT

Summary: We ran the same workload on both for half a year. The break-even point isn't where most blog posts say it is — and the latency story has more nuance than throughput-per-dollar charts admit.

---
# Self-Hosted LLMs vs OpenAI API: A Cost-vs-Latency Analysis After 6 Months

For the last six months we've run the same workload — internal-facing assistant for ~2,400 employees — against both OpenAI's API and a self-hosted Llama 3 70B Instruct deployment. Not as a science fair, but as production. Here are the numbers, and the calls we'd make differently next time.

## The Workload

- **~190k requests/day** across business hours (≈ 5.5k/hour peak)
- **Median input**: 2,800 tokens (RAG-augmented chat)
- **Median output**: 380 tokens
- **Quality bar**: internal eval set of 800 prompts; Llama 70B and GPT-4o-mini scored within 2 percentage points of each other
- **Latency target**: p95 < 4 seconds for full response

## The Two Stacks

### Stack A: OpenAI API

- `gpt-4o-mini` for 90% of traffic
- `gpt-4o` for 10% (escalation queries)
- A small caching layer in Redis (semantic cache hit rate: 22%)

### Stack B: Self-Hosted

- 4× `g6e.12xlarge` (4× L40S each = 16 GPUs total) on EKS
- vLLM serving `Meta-Llama-3-70B-Instruct` quantized to AWQ-INT4
- Triton-based router with continuous batching, max-batch-size 64
- Same Redis semantic cache in front

## Cost Math (Actual Bills)

### OpenAI API path — $ per million tokens

| | Input | Output |
|---|---:|---:|
| gpt-4o-mini | $0.15 | $0.60 |
| gpt-4o (10% escalations) | $2.50 | $10.00 |

Effective blended rate at our mix: **$0.42/M input + $1.74/M output**.

Daily token volume:
- Input: 190k × 2,800 ≈ 532M tokens
- Output: 190k × 380 ≈ 72M tokens

**Daily API cost**: 532 × 0.42 + 72 × 1.74 ≈ **$348/day → ~$10,400/mo**

After 22% cache hit rate: **~$8,100/mo**.

### Self-hosted path — $ per month

- 4× `g6e.12xlarge` reserved 1-year, no upfront: $2,815/mo each = **$11,260/mo** in compute
- EKS control plane + load balancer + EBS: ~$280/mo
- Engineer time to maintain (averaged): **$3,500/mo** (we measured this — see below)

**Total self-hosted: ~$15,040/mo.**

After 22% cache hit rate the savings on compute are **zero** because the GPUs are reserved 24/7. Cache hits just give us idle headroom for traffic spikes.

## So Self-Hosted Was More Expensive?

For *us*, yes — by ~$7k/mo. But the math flips quickly with three changes:

1. **Higher utilization**: We sized for peak. If we ran another workload on the same GPUs (we now do — see below), the marginal cost of the second workload is near zero.
2. **Higher token volume**: Above ~12M output tokens/day, self-hosted starts winning even at our utilization.
3. **Lower-quality acceptable**: Llama 3 8B at 4-bit on much smaller GPUs is dramatically cheaper. Our quality bar required 70B.

We now run a second internal workload (PR review assistant) on the same cluster. Effective GPU utilization went from 41% to 78%. The cost-per-request is back below OpenAI for the combined load.

## Latency: Where It Got Interesting

Throughput-per-dollar is the headline number every comparison post focuses on. The latency picture is messier and arguably more important for user experience.

### Time-to-First-Token (TTFT)

| | p50 | p95 | p99 |
|---|---:|---:|---:|
| OpenAI gpt-4o-mini | 380ms | 720ms | 1.4s |
| Self-hosted (idle batch) | 95ms | 180ms | 290ms |
| Self-hosted (full batch) | 240ms | 480ms | 870ms |

**Self-hosted was ~3× faster on TTFT** because we control the network hop and the batch scheduling. For chat UX, TTFT is what users feel as "is it working?"

### Tokens-Per-Second After First Token

| | p50 | p95 |
|---|---:|---:|
| OpenAI gpt-4o-mini | 92 t/s | 41 t/s |
| Self-hosted (idle batch) | 78 t/s | 52 t/s |
| Self-hosted (full batch) | 38 t/s | 19 t/s |

OpenAI was generally faster *per token* but with more variance. Self-hosted under load was slower per token but predictable.

### Tail Latency Stability

This is where self-hosted really shone. p99 on OpenAI varied across the day; we'd see occasional 8–15 second responses that we couldn't explain or escalate. Self-hosted p99 was 870ms and rock-stable because the queue was ours.

For a customer-facing product where you SLA on p99, this matters a lot.

## The "Hidden" Operational Cost

The $3,500/mo "engineer time" line above is not a guess. We tracked it. Six months in:

- **vLLM upgrades**: 2 hours/month (release cadence is fast; sometimes a pin is required)
- **GPU node failures**: 1 incident/quarter, ~4 hours each
- **Quantization re-validation when a new model lands**: 6–8 hours per model
- **Cluster scaling tweaks** (batch size, KV cache config): 4 hours/month
- **On-call response to GPU-related pages**: ~2 incidents/month, 30 min each

That's roughly **15 hours/month at $230/hr fully loaded = $3,450/mo**.

## What We'd Do Differently

### Don't Self-Host for "Maybe Later" Workloads

We were briefly considering self-hosting for a single workload that was projected to be our main use case. It wasn't. It became one of three. If we'd committed GPU capacity to a single workload we'd have wasted most of it.

**Rule we apply now**: don't reserve GPUs unless you have **at least two** workloads that can share them, *and* a third one in your roadmap.

### Use OpenAI for Everything Until You Have Numbers

Months 1–3 we burned ~$8k/mo on the API. That bill funded the eval framework and gave us real production traffic data. With that data the GPU sizing decision was straightforward.

If we'd jumped to self-hosting in month 1 we'd have built for the wrong shape of workload.

### Build the Eval Framework Before the Infra

Our internal eval set caught two regressions on the self-hosted side that would have shipped to users otherwise (prompt caching bug and a tokenizer mismatch). We run the eval against both stacks every deploy.

```python
# Sketch of the eval harness
def run_eval(stack, suite):
    results = []
    for prompt in suite:
        completion = stack.complete(prompt.input)
        score = grade(prompt.expected, completion, llm_judge="gpt-4o")
        results.append({"prompt_id": prompt.id, "score": score})
    return summarize(results)
```

### Always Front Both With a Cache

Semantic cache hit rate of 22% pays for itself either way. It's a no-brainer.

## When Self-Hosted Wins

You should self-host if **at least two** of the following hold:

1. You have **multiple workloads** that can share GPUs.
2. Your **quality bar** is met by an open-weights model in a size you can afford to serve.
3. You need **predictable tail latency** (p99 SLAs).
4. You have **regulatory constraints** that make outbound API calls hard.
5. Your **token volume** is high enough that the API bill clearly exceeds GPU + ops cost.

## When OpenAI Wins

If most of these are true:

1. You have **one workload** with bursty traffic.
2. Your **quality requirements** demand frontier models.
3. You'd rather spend engineer time on product than on serving infra.
4. You're **early** and the workload shape isn't stable yet.

## Best Practices For Either

- **Cache aggressively** with semantic similarity, not exact match.
- **Stream responses to users**; don't wait for completion.
- **Track $/request and tokens/request per route**, not just totals.
- **Keep an evaluation suite** versioned with your prompt code.
- **Plan for fallback**: self-hosted should fall back to a paid API on incident; paid API should have a degraded mode if quotas exhaust.

The right answer depends on numbers we can't predict for you. Run both for a quarter. The decision will be obvious from the data.