Back to AI Research

AI Research

CFM-Bench: A Unified Multi-Domain, Multi-Task Bench... | AI Research

Key Takeaways

  • CFM-Bench: A Unified Multi-Domain, Multi-Task Benchmark for Channel Foundation Models Channel foundation models (CFMs) are emerging as a powerful way to impr...
  • Channel foundation models (CFMs) are developing rapidly, with recent studies reporting benefits from pretraining across downstream wireless tasks.
  • Yet CFMs are commonly evaluated in model-specific pipelines with different data, radio configurations, partitions, adaptation procedures, task definitions, and metrics.
  • Reported comparisons therefore tend to show that pretraining improves over supervised training from scratch within one pipeline, but neither rank CFMs nor compare them fairly with task-specific models.
  • We release CFM-Bench, a unified multi-domain, multi-task benchmark designed to address this gap.
Paper AbstractExpand

Channel foundation models (CFMs) are developing rapidly, with recent studies reporting benefits from pretraining across downstream wireless tasks. Yet CFMs are commonly evaluated in model-specific pipelines with different data, radio configurations, partitions, adaptation procedures, task definitions, and metrics. Reported comparisons therefore tend to show that pretraining improves over supervised training from scratch within one pipeline, but neither rank CFMs nor compare them fairly with task-specific models. We release CFM-Bench, a unified multi-domain, multi-task benchmark designed to address this gap. It curates six channel configurations spanning 3GPP statistical simulation, two independent ray-tracing pipelines, industrial and aerial measurements, and synchronized vehicular multimodal simulation. Official partitions isolate complete trajectories, measurement sessions, vehicle links, simulation realizations, or buffered spatial regions. CFM-Bench does not prescribe an external pretraining corpus or strategy; no benchmark split may be used for foundation-model pretraining, and the official training split is reserved exclusively for downstream fine-tuning. The benchmark additionally requires disclosure of all data used during model development and prohibits training-stage use of official test units. Six task groups are organized along three CFM application dimensions: physical-layer (PHY) channel intelligence, radio-access-network (RAN) decision intelligence, and integrated sensing and communication (ISAC). They cover CSI feedback, frequency and temporal channel extrapolation, propagation-state classification, current- and future-beam prediction, and single-frame and temporal localization. CFM-Bench provides a common substrate for comparing the transferability of channel representations across models, domains, and tasks.

CFM-Bench: A Unified Multi-Domain, Multi-Task Benchmark for Channel Foundation Models
Channel foundation models (CFMs) are emerging as a powerful way to improve wireless network performance by learning reusable representations of radio channels. However, the field currently lacks a standardized way to evaluate these models. Researchers typically test their models on private, custom-built datasets with different rules and metrics, making it impossible to fairly compare one model against another. CFM-Bench addresses this by providing a unified, multi-domain benchmark that allows for consistent, reproducible evaluation of channel foundation models across a wide variety of wireless tasks and environments.

A Unified Testing Ground

The core purpose of CFM-Bench is to move the industry away from isolated, "model-specific" evaluation pipelines. It provides a common substrate for testing how well a model’s learned representations transfer across different domains. The benchmark includes six distinct channel configurations, ranging from 3GPP statistical simulations and ray-tracing models to real-world industrial and aerial measurements, as well as synchronized vehicular multimodal data. By using these diverse sources, the benchmark ensures that models are tested for robustness against different propagation physics, hardware effects, and environmental contexts.

Strict Rules for Fair Comparison

To prevent the "leakage" of data—where a model accidentally learns from the test set during its training phase—CFM-Bench enforces strict partitioning rules. It isolates data based on complete trajectories, measurement sessions, or vehicle links, ensuring that the training and testing data are truly independent. The benchmark does not allow any of its official splits to be used for the initial pretraining of a foundation model; the official training data is reserved exclusively for downstream fine-tuning. Furthermore, developers are required to disclose all data used during their model's development, and any model that has been exposed to official test units during development is disqualified from being considered a compliant result.

Diverse Tasks and Applications

CFM-Bench organizes its evaluation into six task groups that span three key application areas: physical-layer (PHY) channel intelligence, radio-access-network (RAN) decision intelligence, and integrated sensing and communication (ISAC). These tasks cover a broad spectrum of real-world requirements, including:

  • CSI Feedback: Reconstructing channel state information.

  • Extrapolation: Predicting channel behavior across different frequencies or time intervals.

  • Classification: Identifying propagation states like Line-of-Sight (LoS) or Non-Line-of-Sight (NLoS).

  • Beam Prediction: Forecasting current and future beam directions.

  • Localization: Determining the position of devices using single-frame or temporal data.
    By providing these standardized tasks and metrics, the benchmark allows researchers to see which models perform best under matched, transparent conditions, rather than relying on disparate, self-reported results.

Comments (0)

No comments yet

Be the first to share your thoughts!