Carbon-aware batch scheduling against hard deadlines

Shifting batch work to a lower-carbon hour sounds simple until a deadline is attached. A nightly data pipeline has to be ready before analysts arrive, a model retraining run has to finish before the weekly release, and a CI job has to complete before a customer-facing deployment window. Carbon-aware scheduling is therefore a constraint-satisfaction problem: the scheduler moves work into cleaner periods, but only within the slack that deadlines, service levels and dependent jobs allow.

The data required is more than a carbon signal. The scheduler needs a forecast of grid intensity with a known horizon and error, a runtime estimate for each job with its own uncertainty, a view of queue priorities and dependencies, and knowledge of which regions and clusters have capacity for the delayed work. Where the job runs matters as much as when: moving a job to another region changes the grid intensity, the PUE and sometimes the data transfer cost. A local carbon signal without a regional forecast is only half the input.

Typical failures are scheduling failures that surface as reliability incidents. The scheduler pushes a job past its deadline and it is rerun at the worst possible time, the forecast is treated as certain and the cleaner window never materialises, only whole jobs can move so short pipelines gain nothing, or the shift is made without checking egress and data residency rules. Savings are then reported without a baseline, and nobody can tell whether the change reduced emissions or simply moved them.

Evaluate scheduling tools on deadline modelling, forecast quality, queue integration, workload eligibility rules and an audit trail that records why each job moved.