Home · GPU servers for LLM inference
Workload guide
GPU servers for LLM inference
Inference is a serving problem, not a training problem: many requests, strict latency, and a model that must fit in accelerator memory once — then throughput is a question of how requests are batched and spread. This guide sizes GPU nodes for that job from the range we supply.
Serving scales across nodes, not inside them
Training rewards the biggest node; serving rewards the right number of nodes. Each serving node holds the model (or a shard of it) and answers requests independently, so capacity grows by adding nodes and the failure of one node costs a fraction of capacity rather than the whole service. That is why inference estates are usually built from 2-GPU and 4-GPU 2U servers rather than the 8-GPU training block.
The exception is the very large model that does not fit on one or two accelerators. Then the node must hold the shard group — four or eight accelerators working as one serving unit — and the sizing conversation starts from model size and quantisation, not from request rate.
Latency is a configuration property
Time-to-first-token depends on accelerator memory bandwidth and how much of the node the request shares; throughput depends on batching. Both are set by the accelerator class and the memory per card, which is why the inquiry for an inference node asks for the model, its size as deployed, and the concurrency you must hold — the three numbers that decide the card and the count.
Where the range fits
The Dell GPU configurations in the PowerEdge line cover the mainstream serving node; the wider GPU page covers the 3U and 4U systems for shard-group serving. If the deployment is still being designed, send the model and the traffic expectation and the desk will size the node before pricing it.
Operating a serving fleet
Serving fleets are operated on latency percentiles and cost per thousand requests, and the hardware choices in this guide are what move those numbers. Two operational habits protect them. Keep the node configuration uniform across the fleet, so any node can take any model's traffic and capacity planning stays arithmetic. And hold the accelerator memory headroom honestly: a model upgrade that no longer fits the fleet's cards is a hardware project, and it should be seen coming a quarter ahead, not discovered at deployment.
Growth in serving is rarely smooth — a successful feature doubles traffic in a week. Estates that cope keep the expansion path warm: a known node configuration, a supplier who has quoted it before, and rack and power headroom reserved in the room plan. The time to arrange all three is while the service is small.
Redundancy is a node-count decision
Serving promises are kept by spare capacity: enough nodes that one failure, or one maintenance window, leaves the latency targets intact. That headroom is a hardware line — an extra node in the fleet — and it is cheaper bought with the fleet than improvised after the first incident writes the requirement in red ink. State the availability target in the inquiry and the fleet size follows from it.
Questions buyers ask
Should inference and training share hardware?
They can, on a schedule, but the sizing pulls opposite ways. Most estates separate them once serving traffic matters.
What does quantisation change?
A quantised model fits in less accelerator memory, which can halve the node size. State the deployed model size, not the headline parameter count.
Price a configuration like this
Send the model name, your target configuration and the delivery destination. We reply with a configuration-matched quote and the checks to run before shipment.