Self-Hosted LLMs vs OpenAI API: A Cost-vs-Latency Analysis After 6 Months

For the last six months we've run the same workload — internal-facing assistant for ~2,400 employees — against both OpenAI's API and a self-hosted Llama 3 70B Instruct deployment. Not as a science fair, but as production. Here are the numbers, and the calls we'd make differently next time.

The Workload

• ~190k requests/day across business hours (≈ 5.5k/hour peak)
• Median input: 2,800 tokens (RAG-augmented chat)
• Median output: 380 tokens
• Quality bar: internal eval set of 800 prompts; Llama 70B and GPT-4o-mini scored within 2 percentage points of each other
• Latency target: p95 < 4 seconds for full response

The Two Stacks

Stack A: OpenAI API

• gpt-4o-mini for 90% of traffic
• gpt-4o for 10% (escalation queries)
• A small caching layer in Redis (semantic cache hit rate: 22%)

Stack B: Self-Hosted

• 4× g6e.12xlarge (4× L40S each = 16 GPUs total) on EKS
• vLLM serving Meta-Llama-3-70B-Instruct quantized to AWQ-INT4
• Triton-based router with continuous batching, max-batch-size 64
• Same Redis semantic cache in front

Cost Math (Actual Bills)

OpenAI API path — $ per million tokens

Effective blended rate at our mix: $0.42/M input + $1.74/M output.

Daily token volume:

• Input: 190k × 2,800 ≈ 532M tokens
• Output: 190k × 380 ≈ 72M tokens

Daily API cost: 532 × 0.42 + 72 × 1.74 ≈ $348/day → ~$10,400/mo

After 22% cache hit rate: ~$8,100/mo.

Self-hosted path — $ per month

• 4× g6e.12xlarge reserved 1-year, no upfront: $2,815/mo each = $11,260/mo in compute
• EKS control plane + load balancer + EBS: ~$280/mo
• Engineer time to maintain (averaged): $3,500/mo (we measured this — see below)

Total self-hosted: ~$15,040/mo.

After 22% cache hit rate the savings on compute are zero because the GPUs are reserved 24/7. Cache hits just give us idle headroom for traffic spikes.

So Self-Hosted Was More Expensive?

For us, yes — by ~$7k/mo. But the math flips quickly with three changes:

1. Higher utilization: We sized for peak. If we ran another workload on the same GPUs (we now do — see below), the marginal cost of the second workload is near zero.
2. Higher token volume: Above ~12M output tokens/day, self-hosted starts winning even at our utilization.
3. Lower-quality acceptable: Llama 3 8B at 4-bit on much smaller GPUs is dramatically cheaper. Our quality bar required 70B.

We now run a second internal workload (PR review assistant) on the same cluster. Effective GPU utilization went from 41% to 78%. The cost-per-request is back below OpenAI for the combined load.

Latency: Where It Got Interesting

Throughput-per-dollar is the headline number every comparison post focuses on. The latency picture is messier and arguably more important for user experience.

Time-to-First-Token (TTFT)

Self-hosted was ~3× faster on TTFT because we control the network hop and the batch scheduling. For chat UX, TTFT is what users feel as "is it working?"

Tokens-Per-Second After First Token

OpenAI was generally faster per token but with more variance. Self-hosted under load was slower per token but predictable.

Tail Latency Stability

This is where self-hosted really shone. p99 on OpenAI varied across the day; we'd see occasional 8–15 second responses that we couldn't explain or escalate. Self-hosted p99 was 870ms and rock-stable because the queue was ours.

For a customer-facing product where you SLA on p99, this matters a lot.

The "Hidden" Operational Cost

The $3,500/mo "engineer time" line above is not a guess. We tracked it. Six months in:

• vLLM upgrades: 2 hours/month (release cadence is fast; sometimes a pin is required)
• GPU node failures: 1 incident/quarter, ~4 hours each
• Quantization re-validation when a new model lands: 6–8 hours per model
• Cluster scaling tweaks (batch size, KV cache config): 4 hours/month
• On-call response to GPU-related pages: ~2 incidents/month, 30 min each

That's roughly 15 hours/month at $230/hr fully loaded = $3,450/mo.

What We'd Do Differently

Don't Self-Host for "Maybe Later" Workloads

We were briefly considering self-hosting for a single workload that was projected to be our main use case. It wasn't. It became one of three. If we'd committed GPU capacity to a single workload we'd have wasted most of it.

Rule we apply now: don't reserve GPUs unless you have at least two workloads that can share them, and a third one in your roadmap.

Use OpenAI for Everything Until You Have Numbers

Months 1–3 we burned ~$8k/mo on the API. That bill funded the eval framework and gave us real production traffic data. With that data the GPU sizing decision was straightforward.

If we'd jumped to self-hosting in month 1 we'd have built for the wrong shape of workload.

Build the Eval Framework Before the Infra

Our internal eval set caught two regressions on the self-hosted side that would have shipped to users otherwise (prompt caching bug and a tokenizer mismatch). We run the eval against both stacks every deploy.

# Sketch of the eval harness
def run_eval(stack, suite):
    results = []
    for prompt in suite:
        completion = stack.complete(prompt.input)
        score = grade(prompt.expected, completion, llm_judge="gpt-4o")
        results.append({"prompt_id": prompt.id, "score": score})
    return summarize(results)

Always Front Both With a Cache

Semantic cache hit rate of 22% pays for itself either way. It's a no-brainer.

When Self-Hosted Wins

You should self-host if at least two of the following hold:

1. You have multiple workloads that can share GPUs.
2. Your quality bar is met by an open-weights model in a size you can afford to serve.
3. You need predictable tail latency (p99 SLAs).
4. You have regulatory constraints that make outbound API calls hard.
5. Your token volume is high enough that the API bill clearly exceeds GPU + ops cost.

When OpenAI Wins

If most of these are true:

1. You have one workload with bursty traffic.
2. Your quality requirements demand frontier models.
3. You'd rather spend engineer time on product than on serving infra.
4. You're early and the workload shape isn't stable yet.

Best Practices For Either

• Cache aggressively with semantic similarity, not exact match.
• Stream responses to users; don't wait for completion.
• Track $/request and tokens/request per route, not just totals.
• Keep an evaluation suite versioned with your prompt code.
• Plan for fallback: self-hosted should fall back to a paid API on incident; paid API should have a degraded mode if quotas exhaust.

The right answer depends on numbers we can't predict for you. Run both for a quarter. The decision will be obvious from the data.