Agents per megawatt: the new unit inference is going to be priced in

Artificial Analysis launched a benchmark today called AA-AgentPerf. It measures how many AI agents a piece of hardware can run at once, per megawatt of power it draws, while still hitting latency targets. NVIDIA's Blackwell GB300 rack came out roughly twenty times ahead of the previous Hopper generation. I want to explain what that number actually means, because the unit itself, agents-per-megawatt, is more interesting than the result.

If you have been reading about AI hardware for the last two years, the number that gets quoted is tokens per second. How fast can the model produce output. AgentPerf is built on the premise that tokens per second is the wrong unit for what data centres are now being asked to do. Here is the shift, and why it matters.

What tokens-per-second actually measures

The classic inference workload is one conversation at a time. You send a prompt, the model generates a response, you wait for it to finish, you read it. Tokens per second tells you how fast that one stream comes out. Higher is better. It is the speedometer reading for a single request.

This was the right unit when the binding constraint on a chatbot was "is the reply fast enough that the user does not get bored". Most benchmarks before now optimised for it. Vendors quoted it on their spec sheets.

The agentic workload is structurally different. An agent is not one conversation. It is a loop: the model reads a task, calls a tool, reads the result, thinks, calls another tool, and so on for dozens or hundreds of steps. A coding agent that fixes a bug might take 200 turns before it finishes. And critically, a real deployment runs many of these loops at the same time — hundreds or thousands of independent agents, each on their own task, each holding their own state.

That changes what the hardware is actually doing. It is not racing through one long generation. It is juggling many short, stateful, parallel requests, each of which needs to come back fast enough that the agent does not stall. The question stops being "how fast is one stream" and becomes "how many streams can you keep alive at once without any of them missing their deadline".

What AgentPerf measures instead

AgentPerf replays real agent trajectories, long coding sessions with up to 200 turns and over 100,000 tokens of context, and asks: how many of these can you run concurrently on this hardware, while every single one still meets two latency targets?

The two targets are:

• TTFT (time to first token) — how long the agent waits before the model starts replying to each turn. If this drifts, the agent feels frozen.
• Output token rate — how fast each reply comes out once it starts. If this drops below the threshold, the agent has technically failed the SLO (service level objective, the latency contract the system promises to hit).

An agent that misses either target does not count. You cannot game the benchmark by cramming more agents on at the cost of any of them stalling. Raw throughput, the old tokens-per-second number, is not what wins. Sustained concurrency under a latency contract is what wins.

Then the number of successful concurrent agents gets divided by the system's power draw, in megawatts. That gives you agents per megawatt.

Why the denominator is megawatts

This is the part I think is genuinely clever, and it took me a minute to see why it matters.

If you are a hyperscaler building out inference capacity in 2026, you are not running out of money to buy GPUs. You are running out of power. Data centre sites have a fixed megawatt allocation from the grid — sometimes hundreds of megawatts, but a hard ceiling. New allocations take years and increasingly do not exist. The constraint on how much AI you can serve from a given building is the size of the power line coming in.

That makes "how many agents can I run per megawatt I am allowed to draw" the actual procurement question. It is not how many GPUs fit in the rack. It is how much useful agent-work you extract from each watt the utility lets you pull.

That is the GB300 NVL72 result on DeepSeek V4 Pro at the baseline SLO. The H200 Hopper system on the same workload comes in at 2,594. The ratio is the roughly 20x figure NVIDIA quoted.

Where the 20x actually comes from

It would be easy to read 20x as "Blackwell silicon is twenty times better than Hopper silicon". That is not quite right, and the gap matters.

The GB300 NVL72 is not a server. It is a rack-scale unit: 72 GPUs wired together over NVLink into a single shared-memory domain. From the workload's point of view, the whole rack behaves more like one very large accelerator than like 72 separate ones. Memory can be pooled across the GPUs. Traffic between them moves at NVLink speeds, which are roughly an order of magnitude faster than the InfiniBand or PCIe links that connect Hopper servers to each other.

For agentic workloads — where many parallel requests are constantly reading and writing shared state, KV caches, model weights — this matters enormously. The bottleneck on a Hopper cluster is often not the GPU itself but the bandwidth between GPUs sitting in different boxes. NVLink at rack scale collapses that bottleneck.

So the 20x is partly newer silicon and partly the rack-scale co-design. The second part is what a Hopper customer cannot replicate by buying more Hopper. The architecture is the moat.

Where the metaphor breaks

I have been describing agents-per-megawatt as if it were a clean, universal unit. It is not, and it is worth being honest about the edges.

The number is measured on one reference model (DeepSeek V4 Pro, chosen because it is open-weight and anyone can run it), with one workload (long-context coding agents), under one SLO tier. Change the model, the task mix, or the latency targets, and the ratios will move. Short-context agents will look different from long-context ones. Reasoning-heavy workloads will look different again.

The metric also structurally favours dense, rack-scale deployments. If you are running inference distributed across regions, or on smaller edge boxes, you will never hit the agents-per-megawatt numbers a 72-GPU NVLink rack produces. The benchmark accurately describes the efficiency frontier; it also describes a deployment shape that only hyperscalers and large clouds can actually build.

And the independence question is real. Artificial Analysis is a commercial firm. NVIDIA is co-promoting the results. The private-test-set design (workloads not published, to stop vendors overfitting) is a genuine anti-gaming mechanism borrowed from ML evaluation. But I would want to see AMD MI355X, Google TPU, and an independent cloud running Blackwell publish their own numbers before treating any single ratio as settled.

What to watch

Two things. First, whether agents-per-megawatt sticks as the unit buyers actually procure on. If hyperscalers start writing it into RFPs, every hardware vendor will be forced to optimise against it, and the benchmark will shape the next generation of silicon. Second, whether the rack-scale gap closes. AMD and the TPU teams know exactly what NVLink-at-rack-scale buys NVIDIA. The interesting question is not whether Blackwell leads today. It is whether the architectural advantage compounds or gets matched.

Footnotes and links

Further reading

• Artificial Analysis methodology page: https://artificialanalysis.ai
• DeepSeek V4 Pro model card and weights (the reference workload used by the benchmark at launch)
• NVIDIA GB300 NVL72 architecture brief (for the rack-scale NVLink design)

¶