Before choosing a GPU, work out how much memory your model needs. Get this wrong and the job either won’t start or you pay for GPUs you don’t use. The maths is simpler than it looks.
Running a model (inference)
Each parameter takes a fixed amount of memory depending on precision:
- 16-bit (FP16 / BF16): 2 bytes per parameter. Best quality.
- 8-bit (FP8 / INT8): 1 byte. Very close to 16-bit quality for most models.
- 4-bit: half a byte. Smallest, with some loss in quality.
Then add about 20% for the attention (KV) cache and runtime. More users at once, or very long context windows, need more.
| Model size | 16-bit | 8-bit | 4-bit | A setup that fits (8-bit) |
|---|---|---|---|---|
| 8B | ≈ 21 GB | ≈ 12 GB | ≈ 7 GB | 1× L4 or 1× L40S |
| 32B | ≈ 79 GB | ≈ 40 GB | ≈ 21 GB | 1× L40S |
| 70B | ≈ 170 GB | ≈ 86 GB | ≈ 44 GB | 1× RTX PRO 6000 or 1× H200 |
| 405B | ≈ 974 GB | ≈ 488 GB | ≈ 245 GB | 4× H200 or 2× B300 |
Fine-tuning with LoRA
LoRA keeps the base model frozen in 16-bit and trains small adapter layers. You need roughly the 16-bit model size plus about a third:
A 70B model needs about 190 GB: four H100s, two H200s or one B300.
Full fine-tuning and pre-training
Training every weight with the Adam optimiser in mixed precision needs about 16 bytes per parameter (weights, gradients and optimiser state), plus room for activations:
| Model size | Memory for full training | A setup that fits |
|---|---|---|
| 8B | ≈ 154 GB | 4× H100, or 1× B200 |
| 70B | ≈ 1,344 GB | 8× B300 (one server), or 3 servers of 8× H100 |
For pre-training from scratch, memory is only the minimum: real runs use dozens to thousands of GPUs to finish in reasonable time.
Three tips to fit on fewer GPUs
- Quantise for serving. 8-bit halves memory with little quality loss.
- Pick a bigger-memory GPU instead of more GPUs. One H200 is simpler than two H100s.
- Use LoRA instead of full fine-tuning unless you truly need to retrain every weight.
These are planning estimates. Context length, batch size and your framework change the real numbers, so our engineers check your setup before you commit. Or let the GPU advisor do the maths for you.