Google DeepMind Unveils Decoupled DiLoCo for Resilient AI Training

Google DeepMind has introduced Decoupled DiLoCo, a distributed training architecture designed to overcome the fragility and bandwidth constraints inherent in training frontier-scale AI models. By moving away from traditional synchronous training methods, the system utilizes asynchronous, fault-isolated "islands" of compute to enable large language model pre-training across geographically distant data centers without the performance bottlenecks caused by individual hardware failures.

Overcoming Synchronization Bottlenecks

Traditional Data-Parallel training relies on a blocking synchronization process known as AllReduce, where every accelerator must wait for the slowest device to complete its gradient update. This requirement makes large-scale training brittle, as a single chip failure or slowdown can stall the entire network. Furthermore, standard approaches demand massive inter-datacenter bandwidth—approximately 198 Gbps across eight data centers—which exceeds the capabilities of most wide-area networking (WAN) infrastructure.
Decoupled DiLoCo addresses these issues by building on Google’s Pathways and DiLoCo systems. It divides training into separate clusters of accelerators called learner units. Each unit performs multiple local gradient steps independently before sharing a compressed signal with an outer optimizer. This asynchronous approach eliminates the need for constant, blocking communication, reducing required inter-datacenter bandwidth to just 0.84 Gbps.

Resilience and Self-Healing Capabilities

A significant technical achievement of Decoupled DiLoCo is its ability to maintain high performance despite hardware instability. Through the use of chaos engineering, researchers deliberately introduced artificial failures into the system to test its robustness. The architecture demonstrated "self-healing" properties, allowing the system to continue training after the loss of entire learner units and seamlessly reintegrate them once they returned online.
In simulations involving 1.2 million chips under high failure rates, Decoupled DiLoCo achieved a goodput of 88%, significantly outperforming the 27% goodput recorded by standard Data-Parallel methods. This resilience does not come at the cost of model quality; in experiments using Gemma 4 models, the architecture achieved an average ML benchmark accuracy of 64.1%, nearly identical to the 64.4% accuracy of the conventional baseline.

Production-Scale Validation and Hardware Flexibility

The research team validated the architecture by training a 12 billion parameter model across four U.S. regions using standard commercial internet infrastructure. By folding communication into computation, the system performed more than 20 times faster than conventional synchronization methods. This validation confirms that global-scale training is possible without requiring custom, high-speed network infrastructure.
Additionally, the asynchronous nature of the learner units allows for the use of heterogeneous hardware. The team successfully mixed TPU v6e and TPU v5p chips in a single training job without degrading performance. This capability extends the operational life of older hardware and provides a solution for organizations managing the logistical challenges of transitioning between different generations of compute infrastructure.