Energy efficiency benchmarks for AI models and hardware

Benchmark claims about AI energy efficiency are easy to publish and hard to compare. A vendor reports that one accelerator finishes a training job in fewer megawatt-hours, a customer reports emissions per token, and a research group reports a carbon score for a model, and the three numbers often cannot be placed side by side because they use different functional units, boundaries and measurement methods. Procurement teams buying AI hardware or cloud capacity, ML platform teams choosing a serving configuration, and sustainability teams checking a supplier claim all need a benchmark they can interrogate.

The method exists but has to be applied deliberately. MLPerf Power defines power measurement techniques for MLPerf benchmarks, with approved power analyzers, result formats and metrics, and covers wall-powered systems first. The SCI specification from the Green Software Foundation turns measured energy and embodied emissions into a rate per functional unit, such as per training run, per inference or per token, and requires the boundary, functional unit and methodology to be disclosed. Both depend on hardware-level measurement rather than configuration estimates, and both separate elimination from offsets.

Failures usually come from mixing levels. A lab benchmark run on a dedicated node is compared with production telemetry from a shared cluster, idle power is left out, the functional unit changes between comparisons, or the software stack, precision and batch size are not disclosed. Embodied emissions are sometimes omitted entirely. The result is a ranking that looks precise and cannot be reproduced.

Evaluate benchmark methods and tools on functional unit definition, power measurement, boundary and embodied emissions, disclosed workload configuration, cross-vendor comparability and reporting that survives verification.