Carbon-Aware Scheduling for Large-Scale Model Training

Abstract

Training large machine learning models now consumes gigawatt-hours of electricity per run, yet the carbon intensity of that electricity varies by a factor of five or more depending on when and where the computation is placed. We present CarbonShift, a carbon-aware scheduler for distributed training workloads that exploits both temporal and spatial flexibility: jobs are paused, resumed, and migrated across geographically distributed GPU clusters in response to short-term grid carbon-intensity forecasts. Unlike prior work that treats training jobs as indivisible batch units, CarbonShift models the checkpoint-restart cost of each job explicitly and co-optimises carbon savings against deadline slack and cross-region data-egress overheads. We evaluate the system on a 168-hour trace of real carbon-intensity data from four European grid zones, replaying training workloads ranging from 350M to 13B parameters on a testbed of 128 A100 GPUs. CarbonShift reduces operational carbon emissions by 31.4% on average (up to 58% for deadline-tolerant jobs) while increasing mean completion time by only 6.2%. We further show that a simple forecast-error-aware hedging policy recovers most of the savings of a perfect-information oracle. Our results suggest that carbon-aware scheduling is a practical, immediately deployable lever for reducing the footprint of large-scale AI, complementary to hardware efficiency gains and model-level optimisations.