Workloads by GPU Class

Estimate the GPU memory your work needs, then request the smallest GPU class it fits within, among the classes your workspace can use. See Choosing a Class and Workspace Class Grants. Not every card or slice listed here backs its class in every quarter.

Estimating GPU Memory¶

A model's weights take a fixed amount of GPU memory for each billion parameters, set by their precision.

Inference needs the weights plus a cache that grows with the length of the context. Training also needs gradients, optimizer state, and the activations for one batch. Full fine-tuning with the Adam optimizer takes about 16 bytes per parameter before activations, so a 650M-parameter model needs about 10 GB. LoRA (low-rank adaptation) freezes the base weights and trains small adapter matrices, and QLoRA does the same on a base quantized to 4 bits, so both need far less.

The memory figures in the examples are approximate peaks. They vary with batch size, sequence length and LoRA rank. Measure a short trial run with nvidia-smi or torch.cuda.memory_summary() before sizing a long one.

Differences Within a Class¶

The cards and slices behind one class differ in memory, speed and supported number formats. A medium session can run on a 20 GB slice or a 24 GB card, and a large session on a 40 GB slice or a 48 GB card. A run that fits only the larger device fails with a CUDA out-of-memory error whenever the session lands on the smaller one. Size work for the smallest device listed for the class.

The Turing cards, the RTX 2080 Ti in small and the Titan RTX in medium, have neither BF16 nor FlashAttention 2. Code that requires either fails or runs slower on those cards. FP16 mixed precision runs on every device listed. See GPU Hardware & CUDA.

xsmall¶

xsmall suits encoder fine-tuning, QLoRA of models up to about 3 billion parameters, and small vision and graph models.

A 3B model at FP16 needs about 6.5 GB for inference, more than an xsmall slice has. At 8-bit precision it needs about 3.5 GB.

xsmall Hardware¶

small¶

small suits QLoRA of 7B to 9B language models, larger vision models, and image-generation LoRA training.

A QLoRA run of an 8B or 9B model is close to the 10 GB of the H100 slice, the smallest small device.

small Hardware¶

medium¶

medium suits LoRA of 7B and 8B models without quantization, INT8 LoRA of 13B and 14B models, and vision-language models.

A 14B model at BF16 needs about 28 GB for its weights alone, more than any medium device has.

medium Hardware¶

large¶

large suits QLoRA of 30B to 34B models, video generation, and protein structure work.

QLoRA of a 70B model, such as Llama 3.1 70B, needs about 38 to 42 GB. It fits the 48 GB devices, but not reliably the 40 GB slice. A 32B model at BF16 needs about 66 GB for its weights, more than any large device has.

large Hardware¶

xlarge¶

xlarge suits fine-tuning 70B models, long-context work, and pretraining experiments.

A 70B model at BF16 needs about 140 GB for its weights, more than any single card has. Work across several GPUs is a separate request; see Multiple GPUs.

xlarge Hardware¶

Report a problem with this page