Quick answer
What is the best GPU for LLM inference?
The best GPU for LLM inference is the cheapest one whose VRAM holds the model plus about 20%: an RTX 4090 (24 GB) for 7–8B models, an L40S (48 GB) for 32B in FP8 or 70B in 4-bit, an H200 (141 GB) for 70B in FP8, and 8× H200 for 671B DeepSeek. As of 29 October 2026, GPUwithCrypto rents the H200 at $3.07/hour, paid in crypto.
Which GPU should I use for each model size?
VRAM needed = weights × 1.2 (see the VRAM guide). Our price and the lowest self-serve on-demand price per GPU-hour that we found among the providers we checked on 29 October 2026.
| Model size (example) | VRAM needed | Best GPU | Why | GPUwithCrypto | Lowest elsewhere (per GPU) |
|---|---|---|---|---|---|
| 7–8B in BF16 (Llama 3.1 8B, Mistral 7B) | 17.5–19.2 GB | RTX 4090 24GB | Holds BF16 weights plus a 32K-token cache (4.3 GB on Llama 3.1 8B); FP8 Tensor Cores | $0.28/hour | $0.33 (Salad) |
| 7–8B with a 128K-token context | 16 GB + 17.2 GB cache = 33.2 GB | L40S 48GB | One 128K-token BF16 cache alone needs 17.2 GB | $0.82/hour | $0.97 (Massed Compute) |
| 30–32B in FP8 (Qwen2.5 32B) | 39 GB | L40S 48GB | 48 GB with FP8 Tensor Cores (733 TFLOPS dense FP8 peak) | $0.82/hour | $0.97 (Massed Compute) |
| 30–32B in BF16 | 78 GB | A100 80GB | Entry 80 GB GPU in our range; no FP8 Tensor Cores, so serve BF16 or 4-bit | $0.92/hour | $1.09 (Thunder Compute) |
| 70–72B in 4-bit | 42–43.6 GB | L40S 48GB | Entry GPU above 44 GB in our range | $0.82/hour | $0.97 (Massed Compute) |
| 70–72B in FP8 (Llama 3.1 70B, Qwen2.5 72B) | 84–87.2 GB | H200 141GB | One GPU instead of two H100s; 54–57 GB left for KV cache; 4,800 GB/s | $3.07/hour | $3.62 (Massed Compute) |
| gpt-oss-120b (MXFP4) | 78.3 GB | H100 80GB | Its model card targets a single 80 GB GPU; FP8 Tensor Cores, 3,350 GB/s (SXM) | $2.32/hour | $2.73 (Massed Compute) |
| DeepSeek V4 Flash (284B MoE) | 191.5 GB | 2× H200 | Its FP4/FP8 weights take 159.6 GB; SGLang documents 4× H200 | $6.14/hour | $3.62 (Massed Compute) |
| 405B in FP8 (Llama 3.1 405B) | 486 GB | 4× H200 | 564 GB on four GPUs; 8× H100 would need twice the GPU count | $12.28/hour | $3.62 (Massed Compute) |
| 671B–1.6T MoE (DeepSeek-V3/R1 FP8, V4 Pro) | 805–1,037.6 GB | 8× H200 | 1,128 GB on one host; 8× H100 holds only 640 GB | $24.56/hour | $3.62 (Massed Compute) |
Prices checked on
Per GPU-hour, USD, taxes excluded. Spot and marketplace prices are not counted: they can be lower, but the GPU can be taken back.
Why does VRAM matter more than TFLOPS for inference?
VRAM decides whether the model runs at all, and memory bandwidth sets the speed limit for generating text. Each new token reads every active weight from memory once, so at small batch sizes the GPU waits on memory, not on arithmetic. Compute matters more for long prompts (prefill) and for large batches of users.
That is why an H200 and an H100 have the same peak FP8 compute (1,979 TFLOPS dense, NVIDIA figures) but the H200 holds 76% more memory and moves it 43% faster: 4,800 GB/s against 3,350 GB/s. Bandwidth and TFLOPS here are NVIDIA's peak figures, not measurements of ours.
Which GPU fits which model? (ranked by VRAM)
The seven GPUs we rent, listed by VRAM per GPU, largest first, because VRAM decides which model fits. Bandwidth is the second criterion, the price per hour the third.
-
1
NVIDIA B200
Most VRAM180 GB of HBM3e at 8,000 GB/s (per-GPU share of NVIDIA's 8-GPU HGX figures), FP8 and FP4 Tensor Cores. Holds a 70B model in BF16 (168 GB) on one GPU.
-
141 GB of HBM3e at 4,800 GB/s. Holds a 70B model in FP8 (84 GB) with 57 GB left for the KV cache. Eight of them hold DeepSeek-V3 or V4 Pro.
-
80 GB of HBM3 at 3,350 GB/s (SXM) or 2,000 GB/s (PCIe), FP8 Tensor Cores. Holds gpt-oss-120b, or a 32B model in BF16.
-
80 GB of HBM2e at 2,039 GB/s (SXM) or 1,935 GB/s (PCIe). No FP8 Tensor Cores: serve BF16 or 4-bit.
-
48 GB of GDDR6 at 864 GB/s, FP8 Tensor Cores, no NVLink. Holds 32B in FP8 or 70B in 4-bit.
-
32 GB of GDDR7 at 1,792 GB/s, FP8 and FP4 Tensor Cores. Holds a 12B model in BF16 (28.8 GB) or a 32B model in 4-bit (19.5 GB) with room for a long context.
-
24 GB of GDDR6X at 1,008 GB/s, FP8 Tensor Cores. Holds a 7–8B model in BF16.
Disclosure: this page is published by GPUwithCrypto. Competitor details come from their public websites at the date shown above and may have changed since.
Is one H200 better value than two H100s?
For a 70B model in FP8, yes: one H200 holds it, while two H100s must split it and exchange data on every layer. According to GPUwithCrypto's price check of 29 October 2026, the median on-demand H200 across 13 providers was $5.29 per GPU-hour, against $4.16 for an H100 across 16 providers. The H200 costs 27% more per hour for 76% more VRAM, so it is 28% cheaper per GB of memory.
At our prices the choice is $3.07/hour for one H200 against $4.64/hour for two H100s. Full comparison: H100 vs H200.
What about spot and marketplace GPUs?
They cost less and can disappear mid-job. The cheapest H100 offer on Vast.ai's marketplace was $1.73 per hour on 29 October 2026 (Vast.ai public price feed), below the lowest on-demand price we found. For an inference server that must stay up, an interruption means downtime. Instances are on-demand, not spot: nobody can take your GPU back while your balance covers it, and the instance runs until you stop it. When to use each: what is a spot GPU.
LLM inference GPU questions
Which GPU do I need to run Llama 70B?
One H200 in FP8 (84 GB needed, 141 GB available), one L40S in 4-bit (42 GB needed, 48 GB available) or one B200 in BF16 (168 GB needed, 180 GB available). Two H100s also hold the FP8 version, at 160 GB combined.
What is the best GPU for DeepSeek V4?
The H200. DeepSeek V4 Flash's weights take 159.6 GB, so it needs at least two H200s, and the SGLang cookbook documents four. V4 Pro's 864.7 GB need eight H200s. Commands are in the DeepSeek V4 guide.
Is the RTX 4090 good for LLM inference?
For 7–8B models, yes: 24 GB holds Llama 3.1 8B in BF16 (19.2 GB with overhead) and it is the entry GPU of our range, with 1,008 GB/s of bandwidth. Above about 10B parameters in BF16 it runs out of memory.
Is the A100 still good for inference in 2026?
For BF16 and 4-bit models up to 80 GB, yes, and it is the entry 80 GB GPU in our range. It has no FP8 Tensor Cores, so FP8 checkpoints run better on an L40S, H100 or H200. See H100 vs A100.
Can I rent these GPUs with crypto?
Yes. GPUwithCrypto takes BTC, USDT, USDC, ETH, SOL, LTC from a $25 top-up, with no ID and no card. Instances are billed per minute at the hourly rate while they run, from the GPU-hours you bought for that GPU. Stopping an instance stops the charge at once.
Sources (14)
- NVIDIA H200 specifications — NVIDIA (read 2026-10-29)
- NVIDIA H100 specifications — NVIDIA (read 2026-10-29)
- NVIDIA A100 specifications — NVIDIA (read 2026-10-29)
- NVIDIA L40S specifications — NVIDIA (read 2026-10-29)
- NVIDIA HGX platform (B200) — NVIDIA (read 2026-10-29)
- NVIDIA Ada GPU architecture whitepaper (RTX 4090) — NVIDIA (read 2026-10-29)
- NVIDIA RTX Blackwell architecture whitepaper (RTX 5090) — NVIDIA (read 2026-10-29)
- Transformer Math 101 — EleutherAI (read 2026-10-29)
- gpt-oss-120b model card — OpenAI / Hugging Face (read 2026-10-29)
- DeepSeek-V4 cookbook — SGLang (read 2026-10-29)
- Massed Compute pricing vm.massedcompute.com/pricing — Massed Compute (read 2026-10-29)
- SaladCloud price changes, November 2026 blog.salad.com/saladcloud-price-changes-november-2026 — Salad (read 2026-10-29)
- Thunder Compute pricing thundercompute.com/pricing — Thunder Compute (read 2026-10-29)
- Vast.ai public GPU price feed — Vast.ai (read 2026-10-29)