When AI teams start competing for GPU access, the obvious response is to add capacity. 

Projects are waiting, existing infrastructure looks busy, and another hardware order can appear to be the shortest route to getting development moving. 

But while a queue demonstrates demand, it does not necessarily demonstrate a shortage. 

A harder question for technology leaders to answer is how much useful work the GPUs they already pay for are delivering — and whether the constraint preventing more work from being completed sits with the accelerator at all. 

As enterprise AI expectations transition from experimentation to production, this difference will become vital. Before procurement becomes the answer to rising demand, organisations need a better picture of the capacity they already have. 

First, Find the Real Bottleneck 

GPU performance cannot be understood in isolation from the infrastructure surrounding it. 

Speaking at Tech Show London, Seif Mazigh of Crusoe described the wider system around the accelerator. 

“Once you move to the data centre layer, every workload or deployment will depend on three big bottlenecks: compute, storage, and networking,” said Mazigh. 

He also explained how one of those constraints can leave expensive compute waiting for another part of the system. 

“If you're training or if you're doing any sort of AI workloads, you would have to take that data, training data, ingest it," Mazigh added. "Or in the case of trying to fine-tune or post-train, the GPU will need to wait for that data to be loaded in memory.” 

This is important because what presents itself as insufficient GPU capacity may be a problem elsewhere in the infrastructure. 

Mazigh also cautioned against assuming every AI workload needs the latest accelerator generation. 

“If you talk to any AI lab, they want the best of the best yesterday," he stated. "But when you're inferring, once the model is built, done, exported or distilled, sometimes you can actually run it on very cheap GPUs — not cheap because they're bad; like 10 years ago they were the best.” 

That does not mean every inference workload should be moved to older hardware. It means the newest accelerator should not automatically be treated as the default requirement for every stage of AI deployment. 

The objective is to avoid paying for capabilities a particular workload does not need. 

Mazigh described one of the metrics Crusoe monitors this way: 

“Most of the metrics we track usually are how much of the GPU time we're paying for is being used productively to train or to infer.” 

That word — productively — is important. 

NVIDIA's Data Center GPU Manager guidance has illustrated why a headline utilisation figure can be misleading. Its profiling tools look separately at compute activity, memory activity, and data moving across connections such as PCIe and NVLink. 

Crucially, NVIDIA has noted that GPU threads waiting for memory can still register as active. It has recommended comparing multiple measurements with application throughput rather than using a single metric to decide whether a workload is compute- or memory-bound. 

For infrastructure teams, the implication is straightforward. A GPU reporting high activity should not automatically become evidence for another GPU purchase. The first question is what the workload is waiting for. 

Not Every AI Workload Needs the Same Infrastructure 

That diagnosis also requires organisations to stop treating “AI workload” as a useful sizing category on its own. 

Training, fine-tuning, experimentation, batch inference, and real-time inference have different requirements. Model size, memory demand, latency, concurrency, and the amount of data moving through the system can all change what appropriate infrastructure looks like. 

Jeff Fan, Senior Solutions Architect at DigitalOcean, made that point during the Cloud & AI Infrastructure programme at Tech Show London. 

He stated that there was “no universal configuration that you can just copy and paste." 

"But what is most important is starting from knowing your workload," he continued. 

Fan left the audience with this piece of advice: 

“Analyse the input token versus output token ratios and also check your system setup. How much memory you have, what model you are using, is there any kernels that you can do optimisation?” 

As explored in our Week 1 analysis of where AI workloads should run, infrastructure decisions should follow the characteristics and constraints of the workload rather than the other way around. The same principle applies when deciding what accelerator capacity it needs. 

Fan also presented a production inference case study involving DigitalOcean, AMD, and Character.AI at Tech Show London. 

“Let me say that again. We just use the same AMD GPU. We didn't add additional GPU on top of it. We just tweaked how we architect the GPU, and that 45% performance just improved," said Fan describing one stage of the optimisation. 

For Fan, “How you split your GPU matters more than how many GPU you have.” 

The result belongs to that workload, architecture, and test environment. It should not be interpreted as evidence that enterprises can universally gain 45% more capacity by reconfiguring their GPU estates. 

It does demonstrate why architecture should be investigated before hardware count is treated as the only lever available. 

Measure Productive GPU Time, Not Just Activity 

The question then becomes how organisations define whether existing capacity is being used well. 

Infrastructure utilisation tells an organisation whether a resource is active. It does not, by itself, establish whether that activity is producing the required service at an acceptable cost. 

The FinOps Foundation similarly recommends looking beyond utilisation alone when managing generative AI infrastructure, connecting infrastructure consumption with cost, performance, and the requirements of individual workloads. 

The industry is beginning to express that movement in different ways. 

During the Shaping Europe's Digital Future discussion at Tech Show London, an NVIDIA speaker contrasted traditional infrastructure metrics with the output produced by AI infrastructure: 

“In the past, we might have been looking at the number of chips in there or the power usage, the wattage right? And maybe the gigahertz is inside of the Computing nodes," the speaker from NVIDIA stated. "And now we see, we're moving away completely from that because an AI Factory produces tokens.” 

The speaker went on to discuss tokens per watt, tokens per dollar, and revenue per token. 

For an enterprise, tokens will not always be the right business measure. Depending on the application, useful output might instead mean cost per request, training run, completed task, or successful transaction. 

What matters is what the compute delivers. 

Software Can Release Capacity Before Hardware Does 

Fan's case is not the only evidence that software architecture can change the amount of useful work available from existing hardware. 

Research into LLM serving provides another example. 

The PagedAttention research underpinning vLLM examined the memory used by the KV cache — the working memory an LLM uses to retain information about tokens it has already processed while generating a response. 

The researchers found that existing approaches could waste significant amounts of that memory by reserving capacity that requests ultimately did not use. PagedAttention instead divided the KV cache into smaller blocks that could be allocated as required, allowing more requests to fit into available GPU memory. 

In the researchers' evaluations, vLLM delivered two to four times higher serving throughput at a similar level of latency than the comparison systems tested. The gains were more pronounced for longer sequences, larger models, and more complex decoding methods. 

Again, this is not evidence that deploying PagedAttention will make every enterprise GPU estate two to four times more productive. The useful finding is narrower: memory and serving architecture can materially alter the throughput delivered by the same underlying hardware. 

The paper has also illustrated why diagnosis is integral to the process. In one configuration with relatively abundant GPU memory and short sequences, vLLM's advantage narrowed because the workload became compute-bound rather than memory-bound. 

There is no single optimisation for every bottleneck. 

Inference Has More Than One Optimisation Lever 

Memory management is only one option. 

Google Cloud's guidance on inference optimisation has identified techniques including quantisation, batching, and more efficient memory management. It has also distinguished between the prefill stage, where the model processes the initial prompt, and decode, where output tokens are generated sequentially. The two stages place different demands on infrastructure. 

Quantisation can reduce a model's memory requirements. Batching can allow multiple requests to be processed more efficiently. Better management of the KV cache can make more effective use of available GPU memory. 

The appropriate combination depends on what the workload is trying to achieve, and which constraint has actually been measured. 

That is why optimisation needs to precede procurement rather than follow it. 

The Spare Capacity May Belong to Another Team 

The next constraint may be organisational rather than technical. 

Expensive accelerator capacity can become fragmented when individual teams reserve infrastructure around their own expected peaks. One group can consequently be waiting for GPUs while capacity assigned elsewhere is underused. 

The FinOps Foundation has recommended approaches including shared GPU pools, workload right-sizing, quotas, and prioritisation to improve the economics of GenAI infrastructure. It also emphasised collaboration between engineering, finance, and business stakeholders when evaluating cost and performance. 

Pooling will not remove genuine capacity constraints, and critical production workloads still need appropriate guarantees. But it changes the question from “does this team need another GPU?” to “does the organisation need another GPU?” 

Those are not necessarily the same thing. 

Five Questions Before Requesting More GPU Capacity 

A stronger procurement process therefore starts with evidence. 

Before additional accelerator capacity is approved, the organisation should be able to answer: 

• Which workload is constrained? Is it training, experimentation, fine-tuning, batch inference, or a production service? 

• What service level does it need to meet? Establish the required latency, throughput, concurrency, or completion time. 

• Where is the measured bottleneck? Is compute genuinely saturated, or is performance being constrained by memory, storage, networking, data movement, or orchestration? 

• What does useful output currently cost? Connect infrastructure consumption to a relevant unit such as a token, request, training run, completed task, or business outcome. 

• What has already been optimised? Record what has been tested across model choice, memory, batching, serving architecture, scheduling, pooling, and hardware right-sizing. 

That does not create an argument against buying GPUs. It creates a better argument for buying them when additional capacity really is required. 

If a workload has been classified, the bottleneck measured, the existing environment optimised, and spare capacity pooled or scheduled — and demand still exceeds what the infrastructure can deliver — additional accelerators may be exactly the right investment. 

The difference is that procurement is now responding to a measured constraint rather than a visible queue. 

For enterprise AI infrastructure, the order of operations should be turning towards this sequence: classify, measure, optimise, pool and schedule — then procure against the constraint that remains.