Back to AI Research

AI Research

HCRMap: Pressure-Aware Hot-Expert Residency Mapping... | AI Research

Key Takeaways

  • HCRMap is a research framework designed to optimize the performance of Mixture-of-Experts (MoE) large language models when running on 3.5D multi-chiplet hard...
  • On 3.5D multi-chiplet systems, this skew not only causes compute imbalance but also amplifies pressure on communication, memory bandwidth, I/O, and execution queues.
  • Therefore, the core problem is not simply to reduce token movement, but to dynamically place and reuse hot expert replicas across different memory tiers.
  • This paper proposes HCRMap, a hot expert residency mapping framework for pressure-aware expert replica management in 3.5D MoE inference.
  • Based on expert hotness, weight loading cost, migration overhead, and runtime resource pressure, HCRMap dynamically determines which experts should be promoted, retained, demoted, or evicted.
Paper AbstractExpand

Mixture-of-Experts (MoE) large language models (LLM) activate only a small number of experts during inference, but token routing introduces persistent expert hotness skew: a small set of hot experts continuously receives most tokens, while the remaining experts are lightly loaded. On 3.5D multi-chiplet systems, this skew not only causes compute imbalance but also amplifies pressure on communication, memory bandwidth, I/O, and execution queues. Therefore, the core problem is not simply to reduce token movement, but to dynamically place and reuse hot expert replicas across different memory tiers. This paper proposes HCRMap, a hot expert residency mapping framework for pressure-aware expert replica management in 3.5D MoE inference. Based on expert hotness, weight loading cost, migration overhead, and runtime resource pressure, HCRMap dynamically determines which experts should be promoted, retained, demoted, or evicted. It then maps routed token groups to suitable resident replicas, thereby jointly mitigating communication, memory, and queue bottlenecks. Experimental results show that HCRMap reduces end-to-end latency by 43.6% and 43.0% over Hydra in the prefill and decode stages, respectively; by 34.5% and 33.1% over MoEntwine; and by 46.7% and 46.0% over PIMoE.

HCRMap is a research framework designed to optimize the performance of Mixture-of-Experts (MoE) large language models when running on 3.5D multi-chiplet hardware. In these systems, a small number of "hot" experts receive the majority of incoming data, creating bottlenecks that go beyond simple compute imbalances. HCRMap addresses this by dynamically managing where expert weights are stored across different memory tiers—such as SRAM, HBM, and DRAM—to reduce latency and improve resource efficiency.

The Challenge of Hot Experts

In MoE models, token routing causes a persistent skew where some experts are used constantly while others remain idle. On 3.5D chiplet systems, this creates a complex bottleneck. Because these systems use a hierarchical memory structure, simply moving data to the nearest location isn't always effective. If too many tokens are routed to the same expert, they queue up, causing delays. Conversely, if expert weights are moved too frequently between memory tiers to save space, the system wastes time and bandwidth on migration. Existing methods often fail because they treat this as a simple communication or replication problem, ignoring the physical constraints of memory banks, shared I/O, and link congestion.

How HCRMap Works

HCRMap introduces a two-timescale control system to manage expert residency:

  • The Slow Loop: This controller makes strategic decisions about whether to promote, retain, demote, or evict expert weights across the memory hierarchy. It considers factors like the "hotness" of an expert, the cost of moving weights, and current resource pressure. By allowing "warm" experts to be demoted to an intermediate tier rather than being evicted to far-away DRAM, the system avoids the performance cliff caused by limited near-tier capacity.

  • The Fast Loop: This controller handles real-time token assignment. It maps incoming token groups to the most efficient resident replica based on current queueing, path congestion, and memory bank pressure.
    By separating these two loops, HCRMap keeps the overall layout stable while allowing the system to react instantly to traffic spikes.

Performance Results

The framework was evaluated using the LEGOSim simulator against several existing baselines, including Hydra, MoEntwine, and PIMoE. HCRMap demonstrated significant improvements in end-to-end latency across both the prefill and decode stages of inference. Specifically, it reduced latency by 43.6% and 43.0% compared to Hydra, 34.5% and 33.1% compared to MoEntwine, and 46.7% and 46.0% compared to PIMoE. These results highlight the effectiveness of managing expert residency as a pressure-aware, multi-level resource problem rather than a simple caching or replication task.

Key Considerations

HCRMap is specifically tailored for 3.5D chiplet architectures, which provide a unique multi-level memory hierarchy. The framework is designed to be stable, using techniques like hysteresis and bounded updates to prevent "churn," where constant migrations would otherwise degrade performance. It treats resident copies of experts as scarce, valuable resources that must be budgeted carefully. By balancing the trade-offs between memory capacity, migration overhead, and link contention, HCRMap ensures that the system remains responsive even under heavy, skewed workloads.

Comments (0)

No comments yet

Be the first to share your thoughts!