A practitioner's guide to picking a GPU cloud for training and inference, where hourly rates for the same H100 can differ by 3x.
The same H100 can cost you $2/hour on one provider and $6/hour on another. Multiply that across a week-long training run or a fleet of inference nodes and the difference is a car payment versus a mortgage. GPU cloud choice is one of the few infrastructure decisions where getting it wrong shows up directly on the invoice, and where getting it right takes an afternoon of reading rate cards instead of a migration.
This is a survey of where to rent GPUs in 2026 and how to decide. Rates move constantly, so treat every number here as a tier, not a quote. Verify current rates before you commit.
A GPU-hour is not a commodity. The headline rate bundles the silicon, the interconnect around it, the storage attached to it, the network out of it, and the amount of hand-holding the provider does. A hyperscaler charges a premium for compliance, global regions, and a support contract. A marketplace like Vast.ai charges almost nothing because you are renting someone's idle rig with no SLA. Everything in between is a trade of price against reliability.
RunPod: Developer-friendly, fast to start, strong at both pods (rent-a-container) and serverless GPU endpoints. Community Cloud tier is cheap because it runs on vetted third-party hosts. Good default for indie work and small inference.
Lambda Labs: Positioned for ML teams. Clean on-demand pricing, decent H100/H200 availability, and a reserved-cluster business for serious training. Less serverless focus, more "give me a box that works."
CoreWeave: Built for scale. Kubernetes-native, InfiniBand-connected clusters, the place you go when you need 256 H100s wired together for a real training job. Reserved and contract-heavy; not aimed at hobbyists.
Vast.ai: A marketplace. You bid on other people's hardware. Cheapest per hour by a wide margin, but availability and reliability vary by host. Great for interruptible, fault-tolerant work.
Together AI: Leans toward serverless inference and fine-tuning APIs rather than raw boxes. You hit an endpoint; they run the GPUs. Convenient when you do not want to manage anything.
AWS / GCP / Azure: The hyperscalers. Most expensive on-demand, but unmatched for regions, compliance, spot markets, and gluing GPUs to the rest of your stack (data, IAM, VPC). AWS p5 (H100) and g6 (L40S) instances, GCP A3, Azure ND-series.
GPU type: H200 and H100 for large training and high-throughput inference; A100 (still everywhere) for mid-size work; L40S and L4 for cost-efficient inference and smaller models. Match the GPU to memory footprint first, then to throughput.
Pricing model: On-demand for flexibility, reserved/committed for steady load (30-60% cheaper), spot/community/interruptible for anything that can checkpoint and restart. Most teams mix all three.
Availability: The cheapest H100 you cannot actually provision is worth zero. Capacity for top-tier GPUs is regional and spiky. Marketplaces show it live; hyperscalers make you request quota.
Egress: The classic hidden bill. Hyperscalers charge for data leaving the network. RunPod, Lambda and the specialists are far more generous. If you move large datasets or model weights out often, egress can rival compute.
Storage: Persistent volumes, network storage, and object storage each price differently. A pod that loses its disk on restart forces you to re-download weights every cold start.
Networking for multi-GPU: For anything past a single node, interconnect is everything. InfiniBand or NVLink between nodes (CoreWeave, reserved Lambda) versus plain Ethernet decides whether your distributed job scales or stalls on gradient sync.
Cold-start: For serverless, how long from request to first token when scaled to zero. Seconds of cold start are fine for batch, fatal for interactive.
They want opposite things. Training wants big, tightly-networked, long-lived clusters where you buy reserved capacity and keep the GPUs saturated. Interruptions are tolerable if you checkpoint. Inference wants elasticity: scale to zero when idle, spike on traffic, keep latency low. That is where serverless GPU shines. If you are choosing a serving stack, Ollama vs vLLM covers the throughput side of that decision.
RunPod Serverless, Together, Modal, and similar let you pay per second of actual compute instead of per rented hour. You ship a container, they autoscale it, you pay for requests. The tradeoff is cold starts and less control. For bursty or low-volume inference this is dramatically cheaper than a box idling at 5% utilization. For steady high load, a dedicated instance wins.
- H100 on-demand: ~$2-3/hr on RunPod/Lambda/Vast, ~$4-7/hr on hyperscalers.
- A100 80GB: ~$1-2/hr specialists, more on hyperscalers.
- L40S / L4: sub-$1/hr, the value pick for inference.
- Reserved commitments: cut on-demand by roughly a third to a half.
- Community/spot: cheapest available, no reliability guarantee.
RunPod's CLI gets you a GPU container in one command:
# Install and authenticate
pip install runpod
runpod config # paste your API key
# Launch an H100 pod from a PyTorch image
runpod pod create \
--name train-run \
--gpuType "NVIDIA H100 80GB HBM3" \
--gpuCount 1 \
--imageName runpod/pytorch:2.4.0-py3.11-cuda12.4 \
--volumeSize 100 \
--ports "8888/http,22/tcp"
# List running pods and grab connection details
runpod pod list
For serving, most providers just take a Docker image. A vLLM inference server locally or on a rented box:
docker run --gpus all -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 8192
That container exposes an OpenAI-compatible endpoint, which is the same interface most managed providers give you. It makes swapping between a self-hosted GPU and a hosted API a config change rather than a rewrite.
Egress is the big one. After that: storage that bills whether the GPU runs or not, minimum billing increments, idle pods you forgot to stop, cold-start compute you pay for on every scale-up, and support tiers gated behind enterprise contracts. Marketplaces add a reliability tax paid in failed jobs rather than dollars. Budget 10-20% over the compute line for the rest.
Hobbyist fine-tuning: RunPod Community or Vast.ai. Rent an A100 or single H100 by the hour, checkpoint to persistent storage, accept the occasional interruption. You want cheap and disposable.
Startup inference: RunPod Serverless or Together for bursty traffic; a dedicated Lambda or RunPod L40S box once load is steady and predictable. Watch egress if you serve large responses.
Enterprise training: CoreWeave or reserved Lambda for InfiniBand clusters, or a hyperscaler if you are already deep in their ecosystem and need the compliance and data locality. Commit to reserved capacity and keep it saturated.
Start on RunPod. It covers the widest range of needs, the pricing is honest, and you can go from a single hobby pod to serverless inference without changing vendors. Reach for Vast.ai when you want the absolute floor on price and can tolerate flakiness. Move to CoreWeave or reserved Lambda only when you have a multi-node training job that genuinely needs the interconnect. Save the hyperscalers for when compliance, region, or an existing data gravity forces the issue, because you will pay for the privilege.
Match the GPU to the memory footprint, put anything interruptible on spot, keep egress in view, and stop your idle pods. For the broader stack this fits into, see our guide to the best LLM APIs and AI infrastructure.
