Bench
What "measured" means here.
Every number on the leaderboard and the picker came from the same open-source benchmark harness this project ships, not a spec sheet. This page is the honest account: how energy actually gets measured, what rides along with every submitted run so a number can be checked rather than trusted, and where the current limits are.
Three numbers for one energy figure
Every run records GPU energy three independent ways, on purpose, because they disagree in informative ways. Measured on 100-item runs, integrating the raw power samples overestimates the hardware counter by 2.6% to 5.9% -- not a rounding error, and it does not correlate with how spiky the workload looks.
The GPU's own cumulative energy counter (NVML), read at the
start and end of a run. Continuous, hardware-integrated. This is
the figure every joules_per_* metric prefers.
Power sampled at 5 Hz and integrated over the run. The fallback for hosts without a counter -- retained, and its drift from the counter is recorded per run rather than absorbed silently.
A smart-plug kWh counter for the whole machine -- CPU, RAM, fans, PSU losses included. Coarse (~0.01 kWh steps) and a cross-check only, never the source of a headline figure.
Fields that ride along with every run
A submitted run carries more than its headline metric -- these fields exist so a number can be checked instead of taken on faith:
- measurement_tier
- A, B, or C -- see "Measurement tiers" below. Never averaged across tiers for the same config; the leaderboard and picker always report them side by side.
- energy_source
-
counterorintegrated-- which of the two measurements above the run's energy figures actually came from, since the two differ by several percent. - counter_vs_integration_pct_diff
- How far the integrated figure drifted from the counter, when both were available for that run. Kept visible rather than assumed away.
- thermal_throttle_pct / power_cap_throttle_pct
- Share of the run spent throttled by temperature versus pinned at the configured power cap -- a run that throttled on heat is a different kind of result than one that simply ran at its cap.
- streaming_used
-
Whether latency was measured over a real token stream.
falsemeans every latency figure for that run is withheld rather than mixed with a placeholder. - is_canary
- A task carried for instrument value rather than ranking signal (it's saturated -- most models score near ceiling on it) and excluded from graded comparisons for that reason.
The headline metric, and why it isn't joules per token
Joules per token is the intuitive number, and it's actively misleading across task types: it flatters a model that generates cheap tokens and gets the answer wrong, and it punishes a task that thinks hard and answers briefly. Measured on the same model, 100 items each, counter-sourced energy:
| Task | J / token | J / correct answer |
|---|---|---|
| gsm8k | 7.3 | 2339 |
| mmlu | 8.1 | 115 |
Joules per token says these two runs are within 11% of each other.
Joules per correct answer says one is 20× more efficient. That's
why joules_per_correct_answer is
the headline everywhere on this site, and
joules_per_token is kept only for
comparing models on the same task, where the distortion is
at least constant.
The model picker's ranking also draws on a per-run cost model, fit once per run over that run's own items:
E_i = e_fixed_j + alpha × prompt_tokens_i + beta × completion_tokens_iUnlike the headline metric, these three coefficients are a property of the model on that hardware, not of the benchmark task -- which is what makes them usable for a workload the benchmark never ran.
Correct answers per joule, per watt, and net of idle
Joules per correct answer stays the headline everywhere on this site.
These are literature-comparable inversions of the same measurement,
computed from the exact formulas the harness's own
metrics/derived.py module defines -- this page and the
harness quote the same source, so they can't drift apart.
| Metric | Formula | Meaning |
|---|---|---|
| ipj | n_correct / total_joules | Correct answers per joule -- the inverse of the headline metric, in the units the wider literature reports. |
| accuracy per watt | accuracy / mean_gpu_w | Accuracy per watt of mean GPU draw. Time-free by construction: a slower run at equal power and equal accuracy scores identically. |
| net joules | total_joules − loaded_idle_w × duration_s | Gross joules minus the node's own idle draw with the model resident. Computed only when that baseline exists -- never estimated, and never a fallback to an idle-with-nothing-loaded reading. |
Every one of these renders as "—" rather than a guess whenever an input is missing or a denominator is zero. Net joules is the one figure allowed to go negative -- idle draw over a very short run against a noisy baseline can exceed gross energy -- and it's shown as measured rather than clamped to zero, since clamping would hide the measurement issue instead of surfacing it.
Confidence intervals, and what a repeated run measures
Every scored run carries two 95% intervals, computed once from that run's own items: a bootstrap interval on joules per correct answer (per-item energy and correctness resampled jointly, 10,000 times, seeded for reproducibility) and a Wilson interval on accuracy, which needs no energy reading at all. Both are withheld -- never estimated -- on a run that scored too little to compute them from.
When a config has been run more than once, the leaderboard and picker show the mean and sample standard deviation across those runs (mean ± σ (n)) in place of a single pooled figure -- never a fabricated spread from just one run.
One caveat worth reading before comparing that σ to a number from elsewhere: this harness runs every repeat at temperature 0.0, so a same-seed repeat is expected to reproduce the same output. Its spread mostly reflects measurement noise -- telemetry sampling, thermal drift, background load -- not the model behaving differently between runs. A spread reported at a nonzero temperature includes real sampling variance from stochastic decoding on top of that; the two aren't directly comparable even when both get called "standard deviation."
Measurement tiers
Not every submitter has the same instrumentation on hand. Rather than require one setup, the harness accepts three tiers and never blends them into one number for the same config:
Tier A
A calibrated external power analyzer. Defined in the schema; nothing in this project produces it yet.
Tier B
GPU telemetry plus a metered wall outlet -- a whole-machine cross-check on top of the GPU counter.
Tier C
GPU telemetry only, no wall meter -- the ~25-minute onboarding path anyone can run against their own engine.
Run it yourself
Every row above is someone's own box, not ours. Adding one is the Tier C path from above, run through the on-box controller — no smart plug, no separate install of this harness:
pip install async-energy-controller
async-energy-controller bench quick # ~25 minutes, Tier C, no smart plug
async-energy-controller bench submit <bundle.json> # by hand, if you weren't opted in yet bench quick runs this project's benchmark harness
against your existing engine install and writes a submission bundle.
Opt in once (bench opt-in) and every future run submits
on its own; skip it and hand a bundle to bench submit
whenever you're ready. Full walkthrough, including what a submission
does and doesn't share, in the
controller's own README.
Known limits
Read this before quoting any number:
- Accuracy figures aren't leaderboard-comparable. Our prompt format and few-shot selection differ from published protocol. Valid for comparing models against each other on our fixed workload, not as a claim against a published benchmark.
- Some GPUs run power-limited, not compute-limited. Where that's true, differences in joules per token are largely time differences at a near-constant power cap -- "efficiency" there partly means "speed".
- Confidence intervals exist per run; a repeat campaign at scale mostly doesn't yet. Every scored run carries a within-run bootstrap interval on joules per correct answer and a Wilson interval on accuracy -- instrument variance, from that run's own items. Cross-repeat mean ± σ only shows for the handful of configs that happen to have more than one run behind them; a dedicated repeat campaign is queued but hasn't run yet.
- The wall-plug counter is coarse. It reads in ~0.01 kWh steps, so runs under a few minutes are GPU-telemetry only -- the wall-energy field stays empty rather than reporting a false zero.
- Idle power is measured, and net-of-idle is now derived where possible. Every headline energy figure is still gross draw for the run. A net-of-idle figure (above) exists only when the submitting node's own "model resident, idle" baseline is on file -- never a guess, and never derived from an idle-with-nothing-loaded reading instead.
- Single GPU, no concurrency. Every run targets one GPU and issues requests sequentially -- multi-GPU and concurrent-request numbers aren't represented here.
License
Community submissions accepted through the public program are licensed CC BY 4.0 -- reuse freely, with attribution. Seed rows (this project's own reference matrix) carry the same license.