History / version Q07SQR93 · saved
Cite this version
@software{constellaration_2026, author = {ScienceHub editor}, title = {ConStellaration: leaderboard}, version = {Q07SQR93}, year = {2026}, publisher = {ScienceHub}, url = {https://sciencehub.sh/sciencehub-team/constellaration/notebooks/leaderboard/history/ver_01M5Z4D71V47Q5WPM9Q07SQR93} }
editor, S. (2026). ConStellaration: leaderboard (Version Q07SQR93) [Computer software]. ScienceHub. https://sciencehub.sh/sciencehub-team/constellaration/notebooks/leaderboard/history/ver_01M5Z4D71V47Q5WPM9Q07SQR93cff-version: 1.2.0 message: "If you use this version, cite it as below." title: "ConStellaration: leaderboard" type: software authors: - family-names: "editor" given-names: "ScienceHub" version: "Q07SQR93" date-released: "2026-10-27" url: "https://sciencehub.sh/sciencehub-team/constellaration/notebooks/leaderboard/history/ver_01M5Z4D71V47Q5WPM9Q07SQR93"
This address names exactly this version. The project page cites all versions.
leaderboard.py
22 cells changed against the version before: +328 −5 lines.
ChangedScript header+6−1
Added: # /// script
Added: # requires-python = ">=3.11"
Added: # dependencies = ["marimo>=0.25.1", "numpy", "pandas", "pyarrow", "matplotlib"]
Added: # ///
Added:
import marimo
__generated_with = "0.25.1"
Removed: app = marimo.App()
Added: app = marimo.App(width="medium", app_title="The ConStellaration leaderboard, scored again")
ChangedSetup cell+8−1
Removed: import marimo as mo
Added: import numpy as np
Added: import pandas as pd
Added:
Added: ENTRIES = "data/entries.parquet"
Added: RESCORED = "data/rescored.parquet"
Added: TOLERANCE = 0.01
Added: FLUX_COMPRESSION_BOUND = 0.9
Added: TITLES = {"geometrical": "Geometrical", "simple_to_build": "Simple-to-build QI", "mhd_stable": "MHD-stable QI"}
RemovedCell−3
Removed: mo.md("""
Removed: # leaderboard
Removed: """)
AddedFunction hypervolume+15
Added: def hypervolume(gradient_scale_length, aspect_ratio, reference=(1.0, 20.0)) -> float:
Added: """Area dominated by the points in the plane of (−L∇B, A), up to the reference point.
Added:
Added: The MHD-stable score of constellaration 0.2.6 for feasible points; equal
Added: to pymoo's Hypervolume to 1e-12 on all 58 leaderboard submissions.
Added: """
Added: x = -np.asarray(gradient_scale_length, dtype=float)
Added: y = np.asarray(aspect_ratio, dtype=float)
Added: keep = (x < reference[0]) & (y < reference[1])
Added: x, y = x[keep], y[keep]
Added: order = np.argsort(x, kind="stable")
Added: x, y = x[order], y[order]
Added: edges = np.append(x, reference[0])
Added: lowest = np.minimum.accumulate(y) if len(y) else y
Added: return float(np.sum(np.diff(edges) * (reference[1] - lowest)))
AddedFunction load_entries+4
Added: def load_entries(path: str = ENTRIES) -> pd.DataFrame:
Added: """The leaderboard as recorded on 25 October 2026: one row per single-objective
Added: submission and one per point of each MHD-stable submission."""
Added: return pd.read_parquet(path)
AddedFunction mhd_stable_scores+23
Added: def mhd_stable_scores(entries: pd.DataFrame, tolerance: float = TOLERANCE, top: int = 10) -> pd.DataFrame:
Added: """The `top` MHD-stable submissions by hypervolume when only points within `tolerance` count.
Added:
Added: All submissions are ranked; a resubmission with the same user and
Added: recorded score counts once. "recorded rank" is the order of the
Added: recorded scores.
Added: """
Added: points = entries[entries["problem"] == "mhd_stable"]
Added: rows = []
Added: for (user, submitted), group in points.groupby(["user", "submitted"]):
Added: kept = group[group["feasibility"] <= tolerance]
Added: rows.append({
Added: "user": user,
Added: "submitted": submitted[:10],
Added: "points": len(group),
Added: "recorded score": group["entry_score"].iloc[0],
Added: "score at this tolerance": hypervolume(kept["gradient_scale_length"], kept["aspect_ratio"]),
Added: "points in the last tenth of the band": float(np.mean((group["feasibility"] > 0.9 * TOLERANCE) & (group["feasibility"] <= TOLERANCE))),
Added: })
Added: table = pd.DataFrame(rows).sort_values("recorded score", ascending=False).drop_duplicates(["user", "recorded score"])
Added: table.insert(0, "recorded rank", range(1, len(table) + 1))
Added: table.insert(0, "rank at this tolerance", table["score at this tolerance"].rank(ascending=False, method="min").astype(int))
Added: return table.sort_values(["rank at this tolerance", "recorded rank"]).head(top).reset_index(drop=True)
AddedCell+2
Added: import marimo as mo
Added: import matplotlib.pyplot as plt
AddedCell+19
Added: mo.md(r"""
Added: # The ConStellaration leaderboard, scored again
Added:
Added: The ConStellaration leaderboard on Hugging Face ranks stellarator boundaries
Added: with Proxima Fusion's evaluator, constellaration 0.2.6
Added: ([Cadena et al. 2025](https://arxiv.org/abs/2506.19583)). By 25 October
Added: 2026 it held 174 evaluated submissions. The best entries had passed the
Added: paper's baselines on all three problems: a maximum elongation of 1.18
Added: against 1.27 on the geometrical problem, a normalised gradient scale
Added: length of 13.4 against 8.6 on the simple-to-build problem, and a
Added: hypervolume of 249 against 130 on the MHD-stable problem.
Added:
Added: We check whether the recorded scores reproduce on another machine, how
Added: much of each leading score depends on the 1 % tolerance the evaluator
Added: grants every constraint, and how the leading MHD-stable fronts change
Added: when the flux-compression proxy is averaged the way Proxima corrected it
Added: in version 0.3.0. The forward-model runs were made with `scripts/rescore.py`
Added: of this project; this notebook only reads their results.
Added: """)
AddedCell+2
Added: entries = load_entries()
Added: rescored = pd.read_parquet(RESCORED)
AddedCell+12
Added: mo.md(r"""
Added: ## The recorded scores reproduce
Added:
Added: We evaluated the paper's baselines (the first submissions of the paper's
Added: first author), the three best single-objective entries per problem, and a
Added: random sample of points from the best MHD-stable front, in a fresh
Added: environment built like the leaderboard's (Python 3.10, constellaration
Added: 0.2.6, booz-xform 0.0.9 compiled from source) on an x86-64 server. The
Added: reference boundary and the paper's baselines also ran in ScienceHub's
Added: kernel image (Python 3.13, booz-xform 0.1.0), which the evaluate
Added: notebook of this project uses.
Added: """)
AddedCell+33
Added: _checked = rescored[rescored["recorded_feasibility"].notna() & rescored["flux_compression_pest"].isna() & (rescored["run"] == "leaderboard environment")]
Added: _single = _checked[_checked["problem"] != "mhd_stable"]
Added: _multi = _checked[_checked["problem"] == "mhd_stable"]
Added: _table = pd.DataFrame({
Added: "entry": _single["user"] + ", " + _single["submitted"].str[:10],
Added: "problem": _single["problem"].map(TITLES),
Added: "recorded score": _single["recorded_entry_score"],
Added: "our score": _single["score"],
Added: "recorded feasibility": _single["recorded_feasibility"],
Added: "our feasibility": _single["feasibility"],
Added: })
Added: _gap_single = float((_single["score"] - _single["recorded_entry_score"]).abs().max())
Added: _gap_points = float((_multi["feasibility"] - _multi["recorded_feasibility"]).abs().max())
Added: _gap_objective = float((_multi["minimum_normalized_magnetic_gradient_scale_length"] - _multi["recorded_gradient_scale_length"]).abs().max())
Added: _metrics = list(rescored.loc[:, "feasibility":"flux_compression_in_regions_of_bad_curvature"].columns.drop(["flux_compression_pest", "flux_compression_vmec"]))
Added: _pairs = rescored[rescored["run"] == "kernel image"].merge(
Added: rescored[rescored["run"] == "leaderboard environment"], on=["user", "submitted", "problem", "index"], suffixes=("", "_leaderboard")
Added: )
Added: _kernel = _pairs[_metrics].to_numpy(dtype=float)
Added: _leaderboard = _pairs[[f"{_name}_leaderboard" for _name in _metrics]].to_numpy(dtype=float)
Added: assert np.array_equal(np.isnan(_kernel), np.isnan(_leaderboard))
Added: _values = int(np.sum(~np.isnan(_kernel)))
Added: _gap_kernel = float(np.nanmax(np.abs(_kernel - _leaderboard)))
Added: mo.vstack([
Added: mo.ui.table(_table, selection=None),
Added: mo.md(
Added: f"Largest difference in a single-objective score: **{_gap_single:.1e}**. "
Added: f"Over {len(_multi)} MHD-stable points, the largest difference in feasibility is **{_gap_points:.1e}** "
Added: f"and in L∇B **{_gap_objective:.1e}**. "
Added: f"For the {len(_pairs)} boundaries evaluated in both environments, the largest difference over "
Added: f"{_values} metric values, feasibility and score included, is **{_gap_kernel:g}**."
Added: ),
Added: ])
AddedCell+20
Added: mo.md(r"""
Added: The evaluator is deterministic: VMEC++ runs single-threaded, and the
Added: same boundary gives the same numbers on another machine to the precision
Added: shown. The kernel image, with Python 3.13, booz-xform 0.1.0, simsopt
Added: 1.9.0, jax 0.5.3 and scipy 1.17.1 instead of the leaderboard's versions,
Added: gives the same metrics bit for bit. A score on
Added: the leaderboard can therefore be checked by anyone with a minute of CPU
Added: time per boundary and about 3 GiB of memory.
Added:
Added: ## How much of each score lies in the tolerance
Added:
Added: A boundary counts as feasible when every normalised constraint violation
Added: is at most 0.01. Optimisers use this margin: the best entry on each
Added: single-objective problem sits at feasibility 0.0090 (geometrical) and
Added: 0.0097 (simple-to-build), and on the
Added: ten leading MHD-stable fronts between 17 % and 62 % of the points lie in
Added: the last tenth of the band, above 0.009. The table ranks the 55 distinct
Added: MHD-stable submissions (58 before removing resubmissions) by the hypervolume of their points within the
Added: tolerance you choose and shows the ten best.
Added: """)
AddedCell+2
Added: tolerance = mo.ui.slider(0.0, 0.01, step=0.0005, value=0.001, label="Tolerance", show_value=True, full_width=True)
Added: tolerance
AddedCell+2
Added: _table = mhd_stable_scores(entries, tolerance.value)
Added: mo.ui.table(_table.round({"recorded score": 2, "score at this tolerance": 2, "points in the last tenth of the band": 2}), selection=None)
AddedCell+15
Added: _grid = np.linspace(0, TOLERANCE, 41)
Added: _leaders = pd.concat([mhd_stable_scores(entries, TOLERANCE, top=4), mhd_stable_scores(entries, 0.001, top=1)])
Added: _fig, _ax = plt.subplots(figsize=(7, 3.6))
Added: _points = entries[entries["problem"] == "mhd_stable"]
Added: for (_, _row), _color in zip(_leaders.iterrows(), ("#1f2328", "#0969da", "#9a6700", "#57606a", "#cf222e")):
Added: _group = _points[(_points["user"] == _row["user"]) & (_points["submitted"].str[:10] == _row["submitted"]) & (_points["entry_score"] == _row["recorded score"])]
Added: _curve = [hypervolume(*_group.loc[_group["feasibility"] <= _t, ["gradient_scale_length", "aspect_ratio"]].to_numpy().T) for _t in _grid]
Added: _ax.plot(_grid, _curve, color=_color, lw=1.6, label=f"{_row['user']} ({_row['recorded score']:.1f})")
Added: _ax.set_xlabel("tolerance on the normalised constraint violation")
Added: _ax.set_ylabel("hypervolume")
Added: _ax.legend(frameon=False, fontsize=8, loc="lower right")
Added: for _side in ("top", "right"):
Added: _ax.spines[_side].set_visible(False)
Added: _fig.tight_layout()
Added: _fig
AddedCell+23
Added: mo.md(r"""
Added: At a tolerance of 0.001 the four leading hypervolumes fall from 243–249
Added: to 167–186: between a quarter and a third of each leading score is
Added: earned by points that break a constraint by more than 0.1 %. Most of
Added: their points sit in the upper part of the band: 80 % of Eppy's 688
Added: points and 86 % of Wenyang Li's 783 lie above 0.005. Other fronts lose
Added: less. The front canyon-ant submitted on 9 September, 16th on the
Added: leaderboard with 43 points, keeps 94 % of its hypervolume and scores
Added: 192.5, the highest at this tolerance. The front phanerozoic submitted on
Added: 15 September, 15th, keeps 93 % with only 15 of its 668 points and scores
Added: 192.2. Wenyang Li follows with 185.8 and the recorded leader, Eppy, with
Added: 184.1; at a tolerance of 0 Wenyang Li leads with 185.7 ahead of Eppy
Added: with 183.9. A front scores well at a tight tolerance when its few points
Added: with margin are spread along it. The leaderboard rewards only the
Added: official tolerance, so the order at 0.001 shows how much of each score
Added: rests on the band, not which method would win a benchmark with a
Added: tighter one. (The third and fourth recorded submissions tie at tight
Added: tolerances because 507 of the fourth's 512 points also appear in the
Added: third.) The tolerance is part of the benchmark's definition, so none of
Added: this changes the leaderboard. It does mean that a comparison which cares
Added: about designs with margin should report the score at a tighter
Added: tolerance beside the official one.
Added: """)
AddedCell+20
Added: mo.md(r"""
Added: ## The flux-compression correction
Added:
Added: The MHD-stable problem bounds a geometric proxy for turbulent transport:
Added: the flux-surface average, at ρ = 0.7, of $|\nabla r|^2$ where the
Added: curvature drift is bad and zero elsewhere. Version 0.2.6 averages with the VMEC Jacobian; in
Added: version 0.3.0 Proxima changed the weight to the field-aligned (PEST)
Added: Jacobian, calling the old one a bug
Added: ([constellaration#108](https://github.com/proximafusion/constellaration/issues/108),
Added: [#109](https://github.com/proximafusion/constellaration/pull/109)). The
Added: leaderboard stays on 0.2.6.
Added:
Added: To see what the correction alone changes, we solved each boundary's
Added: equilibrium once with 0.2.6 and computed the proxy with both weights from
Added: that same equilibrium. A point that was feasible under 0.2.6 stays
Added: feasible exactly when the corrected proxy is at most 0.9 × 1.01; no other
Added: constraint is affected. The two leading fronts were evaluated in full
Added: (688 and 783 points). The third and fourth recorded fronts, 1,364 and 512
Added: points, were evaluated at 120 points each, drawn at random.
Added: """)
AddedCell+14
Added: flux = rescored[rescored["run"] == "flux compression only"].copy()
Added: _solved = flux["error"].isna()
Added: # The VMEC-weighted value recomputed here must be the one 0.2.6 reports.
Added: assert np.allclose(
Added: flux.loc[_solved, "flux_compression_vmec"].astype(float),
Added: flux.loc[_solved, "flux_compression_in_regions_of_bad_curvature"].astype(float),
Added: rtol=1e-12,
Added: )
Added: # A failed equilibrium counts as not feasible and stays in the point counts.
Added: flux["feasible_after_correction"] = (
Added: _solved
Added: & (flux["recorded_feasibility"] <= TOLERANCE)
Added: & (flux["flux_compression_pest"].astype(float) <= FLUX_COMPRESSION_BOUND * (1 + TOLERANCE))
Added: )
AddedCell+14
Added: mo.stop(flux.empty, mo.md("The flux-compression runs are not in `data/rescored.parquet` yet."))
Added: _fig, _ax = plt.subplots(figsize=(5.5, 4.5))
Added: _ax.scatter(flux["flux_compression_vmec"], flux["flux_compression_pest"], s=6, color="#1f2328", alpha=0.6, lw=0)
Added: _limit = FLUX_COMPRESSION_BOUND * (1 + TOLERANCE)
Added: _ax.axhline(_limit, color="#cf222e", lw=0.8, ls="--")
Added: _ax.axvline(_limit, color="#cf222e", lw=0.8, ls="--")
Added: _lo, _hi = float(flux[["flux_compression_vmec", "flux_compression_pest"]].min().min()), float(flux[["flux_compression_vmec", "flux_compression_pest"]].max().max())
Added: _ax.plot([_lo, _hi], [_lo, _hi], color="#57606a", lw=0.8)
Added: _ax.set_xlabel("flux compression, VMEC weight (0.2.6)")
Added: _ax.set_ylabel("flux compression, PEST weight (0.3.0)")
Added: for _side in ("top", "right"):
Added: _ax.spines[_side].set_visible(False)
Added: _fig.tight_layout()
Added: _fig
AddedCell+27
Added: mo.stop(flux.empty)
Added: _points = entries[entries["problem"] == "mhd_stable"]
Added: _rows = []
Added: for (_user, _submitted), _group in flux.groupby(["user", "submitted"]):
Added: _all = _points[(_points["user"] == _user) & (_points["submitted"] == _submitted)]
Added: _complete = len(_group) == len(_all)
Added: _n = len(_group)
Added: _share = float(_group["feasible_after_correction"].mean())
Added: # Wilson score interval, 95 %.
Added: _z = 1.96
Added: _centre = (_share + _z**2 / (2 * _n)) / (1 + _z**2 / _n)
Added: _half_width = _z * np.sqrt(_share * (1 - _share) / _n + _z**2 / (4 * _n**2)) / (1 + _z**2 / _n)
Added: _row = {
Added: "user": _user,
Added: "submitted": _submitted[:10],
Added: "points evaluated": f"{_n} of {len(_all)}",
Added: "failed runs": int(_group["error"].notna().sum()),
Added: "still feasible": f"{_share:.0%}" + ("" if _complete else f" (95 % interval {_centre - _half_width:.0%} to {_centre + _half_width:.0%})"),
Added: "recorded score": round(float(_all["entry_score"].iloc[0]), 2),
Added: }
Added: _ok = _group[_group["feasible_after_correction"]]
Added: _recorded = _all.set_index("index").loc[_ok["index"]]
Added: _score = hypervolume(_recorded["gradient_scale_length"], _recorded["aspect_ratio"])
Added: # Adding points never lowers a hypervolume, so a sample's score is a lower bound.
Added: _row["score after correction"] = f"{_score:.2f}" if _complete else f"at least {_score:.2f}"
Added: _rows.append(_row)
Added: mo.ui.table(pd.DataFrame(_rows).sort_values("recorded score", ascending=False), selection=None)
AddedCell+33
Added: mo.md(r"""
Added: The corrected weight raises the proxy for most points. The median ratio
Added: of the PEST-weighted to the VMEC-weighted value is 1.09 on Eppy's front
Added: and 1.10 on Wenyang Li's, against 1.04 on phanerozoic's and 1.03 on
Added: mmdeuss's; over all 1,711 points it lies between 0.96 and 1.20. Every point
Added: of the two leading fronts was feasible under 0.2.6 and every equilibrium
Added: solved, but only 181 of Eppy's 688 points (26 %) and 140 of Wenyang Li's
Added: 783 (18 %) stay within the bound. Their scores fall from 249.14 to
Added: 208.83 and from 248.23 to 208.05, so the two keep their order, 0.78
Added: apart instead of 0.91.
Added:
Added: The sampled fronts lose fewer points: 47 of the 120 sampled from
Added: phanerozoic's front stay within the bound (39 %, 95 % interval
Added: 31–48 %), and 58 of the 120 from mmdeuss's (48 %, 40–57 %). Their
Added: corrected scores are not known, but the surviving sample points alone
Added: score 220.47 and 223.84, and adding the rest of a front can only raise
Added: that. Both lower bounds are above the corrected
Added: scores of the first two fronts. Under the corrected proxy, the third and
Added: fourth recorded fronts would therefore both rank above the first and
Added: second; the samples do not settle the order between those two, and
Added: fronts below the fourth were not evaluated, so we cannot say which
Added: submission would lead.
Added:
Added: This is a statement about the benchmark, not about the designs. The
Added: proxy is a geometric stand-in for turbulent transport, not a transport
Added: calculation, and each submission was optimised against the 0.2.6
Added: version of it, which the leaderboard still uses. A front optimised
Added: against the corrected proxy could win back much of what it loses here.
Added: What the result shows is that the order at the top of the MHD-stable
Added: leaderboard depends on how one constraint is averaged, and that the
Added: correction raises the proxy more on the two leading fronts than on the
Added: next two.
Added: """)
AddedCell+34
Added: mo.md(r"""
Added: ## Limits and references
Added:
Added: **What was run.** Scores in the first section come from a fresh
Added: installation of constellaration 0.2.6 on Python 3.10 with booz-xform
Added: 0.0.9 built from source, in the Docker image `python:3.10-slim`, on a
Added: Hetzner CPX42 server; the flux-compression runs use the same environment
Added: with the Boozer transform and the QI residual switched off, which the
Added: proxy does not depend on. The tolerance analysis needs no new runs: it
Added: uses the per-point feasibility the leaderboard recorded.
Added:
Added: **What was not run.** The third and fourth fronts are sampled, so their
Added: share of surviving points carries a 95 % Wilson interval and their
Added: corrected score is a lower bound. A boundary whose equilibrium failed
Added: would count as not feasible; none of the 1,711 runs failed. The
Added: corrected score keeps every other metric from 0.2.6 and so isolates the
Added: change of weight; it is not an evaluation with constellaration 0.3.0. A
Added: front that loses points under the correction could recover some
Added: hypervolume by re-optimisation, which we did not attempt.
Added:
Added: **Data.** `data/entries.parquet` is derived from the results dataset of
Added: the leaderboard Space,
Added: [proxima-fusion/constellaration-bench-results](https://huggingface.co/datasets/proxima-fusion/constellaration-bench-results)
Added: (178 rows, 174 of them evaluated submissions from users other than
Added: `test`, last modified 25 October 2026, no licence stated). It
Added: keeps the recorded objectives, feasibilities and scores with the
Added: submitters' public names and dates, and none of the submitted boundaries.
Added: `data/rescored.parquet` holds our own evaluations; `build_data.py` writes
Added: both files.
Added:
Added: Cadena, S. A. et al. (2025). ConStellaration: A dataset of QI-like
Added: stellarator plasma boundaries and optimization benchmarks. *NeurIPS 2025
Added: Datasets and Benchmarks Track*. [arXiv:2506.19583](https://arxiv.org/abs/2506.19583).
Added: """)