Choosing a GPU server for AI workloads

Decide what the node is for, before anything else

AI work divides into two different purchases. Training builds models: it runs for days, synchronises constantly between accelerators, and rewards the largest, most tightly connected node you can feed with power. Inference serves models: it answers many independent requests, cares about latency as much as throughput, and rewards the right number of medium nodes over a few giants. A server that is perfect for one is a mediocre purchase for the other, which is why the first question in any serious GPU conversation is not 'which card' but 'which job'.

Mixed estates exist — a research group trains by night and serves by day — but even there, the honest approach is to size for the heavier job and accept the compromise in writing. What does not work is buying 'a GPU server' in the abstract: the abstraction hides exactly the parameters that decide whether the purchase succeeds. Start with the workload, the framework, and the largest model involved. Everything else in this article hangs off those three facts.

Accelerator memory is the constraint that bites first

A model must fit. Parameters, gradients and optimiser state for training — or the deployed weights and the request cache for serving — live in accelerator memory, and when they do not fit, the choices are sharding across more cards or a smaller model, both of which are architectural decisions with price tags. This is why accelerator selection starts from memory capacity per card and works outward to count, rather than starting from a benchmark chart.

It is also why the deployed form of the model matters more than its headline. A quantised model served at half precision occupies a fraction of the memory of its training form; buyers who state the deployed size buy the node they need, and buyers who state the parameter count often buy the node they fear. Bring the real number: the framework, the model, and the precision it will actually run at.

Node topology: eight, four, or two at a time

Inside a node, accelerators synchronise over a high-speed interconnect; across nodes, they synchronise over the network, more slowly. Training therefore concentrates: the eight-GPU node is the standard building block because frameworks and recipes are tuned for it, and multi-node training is planned in whole blocks. Our own range's listed example of the shape is a 3U system with eight GPUs on dual EPYC hosts — the extra chassis height buys the power delivery and cooling that eight loaded cards demand.

Inference inverts the logic. Serving capacity grows by adding nodes, each holding the model and answering independently, so the estate is usually built from two- and four-GPU nodes in 2U chassis. The exception is the very large served model, which forces a shard group of four or eight cards acting as one unit. If your model fits on two cards, buy serving nodes; if it needs eight to exist at all, you are buying training-class hardware for serving duty, and the budget should know why.

The host is not a bystander

Accelerators starve quietly. A GPU node whose host processors, memory and local storage cannot feed the cards at rate will post disappointing utilisation numbers that no benchmark predicted, because benchmarks feed cards perfectly. The host side of the configuration — processor class, memory capacity, NVMe staging for the dataset — deserves the same care as the cards themselves, especially in training nodes where the data pipeline runs continuously.

This is the practical reason GPU servers are quoted as configurations rather than selected from a shelf. The card gets the headline; the host decides whether the headline was true. When you compare two GPU quotes, compare the whole node: cards, host, memory, storage, power supplies. A cheaper quote that quietly thins the host is not cheaper at all.

Power and cooling are purchasing decisions

A fully loaded GPU node is the heaviest continuous electrical load in most buildings, and it arrives as heat that must leave the room. Before any configuration is priced, the delivery site needs three numbers: the per-rack power budget, the cooling capacity in the room, and the electrical work outstanding to support both. More GPU projects stall on facilities than on hardware, and they stall late — after the exciting decisions — which is the expensive order to discover things in.

Chassis height is part of this story. The 3U and 4U GPU systems in the range exist because some card counts and power designs do not fit honestly into 2U. Choosing the slimmer chassis against the power design saves rack units and spends reliability; the quote should make the trade visible rather than hide it.

What a good GPU quote answers

A good GPU server quote reads like the workload brief it answers. It names the accelerator model and count, the host configuration, memory, local storage, power supplies and rails — and it states the assumptions it was priced under: the framework, the model, the node role. If any of those is missing, the quote is a price for a badge, and badges do not train models.

The working habit this site recommends for every purchase applies with double force here: send the workload with the inquiry, ask for the full configuration in writing, and ask for serial numbers before shipment. GPU servers are where configuration discipline pays for itself fastest.

A note on buying ahead of the roadmap

GPU roadmaps tempt buyers into waiting — the next accelerator generation is always months away and always faster. The arithmetic that disciplines the temptation is utilisation: a cluster bought today trains models for the whole waiting period, and the models it trains are the ones the organisation needed this year. Estates that buy on need and refresh on a planned cycle out-produce estates that perpetually wait for the better card, because compute in hand compounds and roadmaps do not.

The exception is a purchase measured in weeks from a confirmed generational step, with a workload that specifically needs the new memory capacity. Everything else is the roadmap fallacy — and a supply desk that talks you out of a premature purchase is worth more than one that never does.