CloudCons: A Comprehensive End-to-End Benchmark for... | AI Research

CloudCons: A Comprehensive End-to-End Benchmark for Cloud Resource Consolidation
Cloud data centers often suffer from low resource utilization because they over-provision hardware to ensure service reliability. To solve this, many systems use a "forecast-then-optimize" approach, where they predict future demand and then consolidate workloads onto fewer servers. While new time series foundation models have shown promise in making better predictions, there has been no standardized way to test if these predictions actually lead to better real-world consolidation decisions. CloudCons is a new benchmark designed to bridge this gap by evaluating how well different forecasting models perform within the specific, practical context of managing cloud resources.

A Multi-Cloud Evaluation Framework

CloudCons moves beyond simple prediction error metrics by creating an end-to-end simulation environment. The researchers built high-quality datasets using real-world workload traces from Huawei Cloud, Microsoft Azure, and Google Borg. These datasets capture a wide range of service behaviors, from predictable daily cycles to sudden, unpredictable bursts of activity and high-frequency noise. By using this diverse data, the benchmark allows researchers to see how different models handle the complex, non-stationary environments typical of modern cloud infrastructure.

Testing Decision Utility

A core goal of this benchmark is to determine if better forecasting accuracy actually results in better consolidation decisions. The researchers evaluated a wide array of models, including traditional statistical methods, deep learning architectures, and the latest time series foundation models. The framework tests these models through a two-stage process: first, the model predicts future resource demand; second, an optimization algorithm uses those predictions to decide how to pack virtual machines onto physical servers. This allows the benchmark to measure performance across five key dimensions: prediction error, resource efficiency, load balance, service reliability, and uncertainty quantification.

Surprising Findings on Foundation Models

The study reveals a critical insight: while foundation models often achieve superior forecasting accuracy compared to traditional methods, this does not always translate into better decision-making. High accuracy in a vacuum does not guarantee that a model will effectively minimize the number of active servers while maintaining service reliability. The researchers found that the misalignment between standard prediction metrics and the actual goals of resource consolidation is a significant hurdle.

Balancing Efficiency and Reliability

The benchmark highlights that the selection of "predictive quantiles"—the specific statistical thresholds used to forecast demand—acts as a vital lever for cloud operators. By systematically analyzing these quantiles, the researchers provide actionable guidelines for balancing the trade-off between resource efficiency and service reliability. This suggests that for real-world deployment, simply choosing the most "accurate" model is less important than calibrating the model’s output to meet the specific risk and efficiency requirements of the data center.