AI datacenters may need to reduce demand quickly without simply postponing users' requests. The query-routing study examines whether changing how requests are allocated among language-model instances can provide that flexibility.
A model of serving and power
The framework connects arrival rates and sequence-length distributions with token iteration time, request residence time, serving capacity and power consumption. It uses those relationships to identify feasible routing allocations and their associated demand behavior.
This is a serving-system proposal. It does not create electricity or make different models interchangeable in quality. A real implementation would need to state which instances can serve each request and which latency and output requirements must be preserved.
Hardware measurements and simulation
The authors validate the analytical framework across four GPU platforms and models ranging from 1B to 70B parameters. They report a 2.8% mean absolute percentage error when estimating achievable power flexibility.
The larger datacenter result comes from a real-trace-driven simulation of 10,000 H100 GPUs. That distinction is important: using a real request trace does not turn a simulated fleet into an observed utility-grid deployment.
How much flexibility was reported?
The simulation exhibits flexibility up to 16% of total power consumption, with 90% of the desired reduction achieved in approximately ten seconds. These figures describe the tested workload and assumptions. They are not a promise that every AI facility can reduce demand by the same amount.
A useful planning question
The paper suggests evaluating routing as part of datacenter orchestration, alongside the model and hardware choices already being made. For an operator, a useful next step would be to test the framework against a representative workload and explicit service constraints. Power reduction, response quality and latency need to be assessed together before treating flexibility as a dependable operational resource.
Comments