How much VRAM do I need to run or fine-tune an LLM?

How do I calculate the VRAM a model needs?

Multiply the parameter count by the bytes each parameter takes, then add 20%. Worked example, Llama 3.1 70B in FP8: 70 billion × 1 byte = 70 GB of weights, and 70 × 1.2 = 84 GB. That is more than one 80 GB H100 and less than one 141 GB H200, so the smallest fit is one H200. Which GPU serves which model fastest for the money: best GPU for LLM inference.

The 20% covers the KV cache and activations at a modest context length and batch size. The rule and the bytes per parameter come from EleutherAI's Transformer Math 101. It is a sizing rule, not a guarantee: long prompts, many parallel users and framework overhead push past it. A total within 5% of a card's VRAM, such as Qwen2.5 32B in BF16 (78 GB) on an 80 GB card, runs with short prompts only; take the next size up for long ones.

How much VRAM does a long context add?

A long context adds the KV cache: two vectors (key and value) per layer, per KV head, per token. For Llama 3.1 70B (80 layers, 8 KV heads of 128 values, per the Llama 3 paper) in BF16: 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes per token. One 128K-token conversation (131,072 tokens) therefore needs 42.9 GB.

Add that to the FP8 weights: 70 + 42.9 = 112.9 GB, still one H200. Four such conversations at once need 70 + 4 × 42.9 = 241.6 GB, so two H200s. An FP8 KV cache halves the cache figures. DeepSeek models compress their attention cache, so this formula does not apply to them.

How much VRAM do I need to fine-tune an LLM?

Full fine-tuning needs 16 to 18 bytes per parameter before activations. EleutherAI counts 2 bytes of weights, 12 of optimizer state (with the FP32 copy) and 2 of gradients; Hugging Face counts 6 + 8 + 4 = 18. LoRA freezes the base model, so it needs about 2 bytes per parameter for a BF16 base; the LoRA paper reports a 3× cut in GPU memory against full fine-tuning. Our × 1.2 planning margin on LoRA is our own convention. QLoRA loads the base in 4-bit: its authors fine-tuned a 65B model on one 48 GB GPU, 48 ÷ 65 = 0.74 GB per billion parameters, rounded to 0.75. Each method applied to a 70B model: best GPU to fine-tune Llama 70B.

Why do GPU counts jump from 2 to 4 to 8?

Tensor parallelism splits every layer's attention heads evenly across the GPUs. vLLM refuses to start when the head count is not divisible by the number of GPUs. Llama 3.1 70B has 64 attention heads, so 2, 4 or 8 GPUs work and 3 do not. Three 80 GB H100s (240 GB) hold a 168 GB model on paper but are not a usable layout; four are. Instances have 1 to 8 GPUs depending on the host. Before you start, the dashboard shows each host’s country, GPU count, form factor (SXM or PCIe) and interconnect.

How much does the right GPU cost per hour?

Precision changes the bill more than any other choice. Llama 3.1 70B needs one B200 in BF16 but one L40S in 4-bit. According to GPUwithCrypto's price check of 29 October 2026, the lowest self-serve on-demand prices for those two GPUs were $5.43 and $0.97 per hour: 4-bit cuts the hourly cost by 82%. At our prices the pair costs $4.61/hour and $0.82/hour. Check 4-bit output on your own prompts before you commit. Every GPU and block of hours is on the pricing page.