{"title":"leaderboard","code":"# /// script\n# requires-python = \">=3.11\"\n# dependencies = [\"marimo>=0.25.1\", \"numpy\", \"pandas\", \"pyarrow\", \"matplotlib\"]\n# ///\n\nimport marimo\n\n__generated_with = \"0.25.1\"\napp = marimo.App(width=\"medium\", app_title=\"The ConStellaration leaderboard, scored again\")\n\nwith app.setup:\n import numpy as np\n import pandas as pd\n\n ENTRIES = \"data/entries.parquet\"\n RESCORED = \"data/rescored.parquet\"\n TOLERANCE = 0.01\n FLUX_COMPRESSION_BOUND = 0.9\n TITLES = {\"geometrical\": \"Geometrical\", \"simple_to_build\": \"Simple-to-build QI\", \"mhd_stable\": \"MHD-stable QI\"}\n\n\n@app.function\ndef hypervolume(gradient_scale_length, aspect_ratio, reference=(1.0, 20.0)) -> float:\n \"\"\"Area dominated by the points in the plane of (−L∇B, A), up to the reference point.\n\n The MHD-stable score of constellaration 0.2.6 for feasible points; equal\n to pymoo's Hypervolume to 1e-12 on all 58 leaderboard submissions.\n \"\"\"\n x = -np.asarray(gradient_scale_length, dtype=float)\n y = np.asarray(aspect_ratio, dtype=float)\n keep = (x < reference[0]) & (y < reference[1])\n x, y = x[keep], y[keep]\n order = np.argsort(x, kind=\"stable\")\n x, y = x[order], y[order]\n edges = np.append(x, reference[0])\n lowest = np.minimum.accumulate(y) if len(y) else y\n return float(np.sum(np.diff(edges) * (reference[1] - lowest)))\n\n\n@app.function\ndef load_entries(path: str = ENTRIES) -> pd.DataFrame:\n \"\"\"The leaderboard as recorded on 25 October 2026: one row per single-objective\n submission and one per point of each MHD-stable submission.\"\"\"\n return pd.read_parquet(path)\n\n\n@app.function\ndef mhd_stable_scores(entries: pd.DataFrame, tolerance: float = TOLERANCE, top: int = 10) -> pd.DataFrame:\n \"\"\"The `top` MHD-stable submissions by hypervolume when only points within `tolerance` count.\n\n All submissions are ranked; a resubmission with the same user and\n recorded score counts once. \"recorded rank\" is the order of the\n recorded scores.\n \"\"\"\n points = entries[entries[\"problem\"] == \"mhd_stable\"]\n rows = []\n for (user, submitted), group in points.groupby([\"user\", \"submitted\"]):\n kept = group[group[\"feasibility\"] <= tolerance]\n rows.append({\n \"user\": user,\n \"submitted\": submitted[:10],\n \"points\": len(group),\n \"recorded score\": group[\"entry_score\"].iloc[0],\n \"score at this tolerance\": hypervolume(kept[\"gradient_scale_length\"], kept[\"aspect_ratio\"]),\n \"points in the last tenth of the band\": float(np.mean((group[\"feasibility\"] > 0.9 * TOLERANCE) & (group[\"feasibility\"] <= TOLERANCE))),\n })\n table = pd.DataFrame(rows).sort_values(\"recorded score\", ascending=False).drop_duplicates([\"user\", \"recorded score\"])\n table.insert(0, \"recorded rank\", range(1, len(table) + 1))\n table.insert(0, \"rank at this tolerance\", table[\"score at this tolerance\"].rank(ascending=False, method=\"min\").astype(int))\n return table.sort_values([\"rank at this tolerance\", \"recorded rank\"]).head(top).reset_index(drop=True)\n\n\n@app.cell\ndef _():\n import marimo as mo\n import matplotlib.pyplot as plt\n\n return mo, plt\n\n\n@app.cell(hide_code=True)\ndef _(mo):\n mo.md(r\"\"\"\n # The ConStellaration leaderboard, scored again\n\n The ConStellaration leaderboard on Hugging Face ranks stellarator boundaries\n with Proxima Fusion's evaluator, constellaration 0.2.6\n ([Cadena et al. 2025](https://arxiv.org/abs/2506.19583)). By 25 October\n 2026 it held 174 evaluated submissions. The best entries had passed the\n paper's baselines on all three problems: a maximum elongation of 1.18\n against 1.27 on the geometrical problem, a normalised gradient scale\n length of 13.4 against 8.6 on the simple-to-build problem, and a\n hypervolume of 249 against 130 on the MHD-stable problem.\n\n We check whether the recorded scores reproduce on another machine, how\n much of each leading score depends on the 1 % tolerance the evaluator\n grants every constraint, and how the leading MHD-stable fronts change\n when the flux-compression proxy is averaged the way Proxima corrected it\n in version 0.3.0. The forward-model runs were made with `scripts/rescore.py`\n of this project; this notebook only reads their results.\n \"\"\")\n return\n\n\n@app.cell\ndef _():\n entries = load_entries()\n rescored = pd.read_parquet(RESCORED)\n return entries, rescored\n\n\n@app.cell(hide_code=True)\ndef _(mo):\n mo.md(r\"\"\"\n ## The recorded scores reproduce\n\n We evaluated the paper's baselines (the first submissions of the paper's\n first author), the three best single-objective entries per problem, and a\n random sample of points from the best MHD-stable front, in a fresh\n environment built like the leaderboard's (Python 3.10, constellaration\n 0.2.6, booz-xform 0.0.9 compiled from source) on an x86-64 server. The\n reference boundary and the paper's baselines also ran in ScienceHub's\n kernel image (Python 3.13, booz-xform 0.1.0), which the evaluate\n notebook of this project uses.\n \"\"\")\n return\n\n\n@app.cell\ndef _(mo, rescored):\n _checked = rescored[rescored[\"recorded_feasibility\"].notna() & rescored[\"flux_compression_pest\"].isna() & (rescored[\"run\"] == \"leaderboard environment\")]\n _single = _checked[_checked[\"problem\"] != \"mhd_stable\"]\n _multi = _checked[_checked[\"problem\"] == \"mhd_stable\"]\n _table = pd.DataFrame({\n \"entry\": _single[\"user\"] + \", \" + _single[\"submitted\"].str[:10],\n \"problem\": _single[\"problem\"].map(TITLES),\n \"recorded score\": _single[\"recorded_entry_score\"],\n \"our score\": _single[\"score\"],\n \"recorded feasibility\": _single[\"recorded_feasibility\"],\n \"our feasibility\": _single[\"feasibility\"],\n })\n _gap_single = float((_single[\"score\"] - _single[\"recorded_entry_score\"]).abs().max())\n _gap_points = float((_multi[\"feasibility\"] - _multi[\"recorded_feasibility\"]).abs().max())\n _gap_objective = float((_multi[\"minimum_normalized_magnetic_gradient_scale_length\"] - _multi[\"recorded_gradient_scale_length\"]).abs().max())\n _metrics = list(rescored.loc[:, \"feasibility\":\"flux_compression_in_regions_of_bad_curvature\"].columns.drop([\"flux_compression_pest\", \"flux_compression_vmec\"]))\n _pairs = rescored[rescored[\"run\"] == \"kernel image\"].merge(\n rescored[rescored[\"run\"] == \"leaderboard environment\"], on=[\"user\", \"submitted\", \"problem\", \"index\"], suffixes=(\"\", \"_leaderboard\")\n )\n _kernel = _pairs[_metrics].to_numpy(dtype=float)\n _leaderboard = _pairs[[f\"{_name}_leaderboard\" for _name in _metrics]].to_numpy(dtype=float)\n assert np.array_equal(np.isnan(_kernel), np.isnan(_leaderboard))\n _values = int(np.sum(~np.isnan(_kernel)))\n _gap_kernel = float(np.nanmax(np.abs(_kernel - _leaderboard)))\n mo.vstack([\n mo.ui.table(_table, selection=None),\n mo.md(\n f\"Largest difference in a single-objective score: **{_gap_single:.1e}**. \"\n f\"Over {len(_multi)} MHD-stable points, the largest difference in feasibility is **{_gap_points:.1e}** \"\n f\"and in L∇B **{_gap_objective:.1e}**. \"\n f\"For the {len(_pairs)} boundaries evaluated in both environments, the largest difference over \"\n f\"{_values} metric values, feasibility and score included, is **{_gap_kernel:g}**.\"\n ),\n ])\n return\n\n\n@app.cell(hide_code=True)\ndef _(mo):\n mo.md(r\"\"\"\n The evaluator is deterministic: VMEC++ runs single-threaded, and the\n same boundary gives the same numbers on another machine to the precision\n shown. The kernel image, with Python 3.13, booz-xform 0.1.0, simsopt\n 1.9.0, jax 0.5.3 and scipy 1.17.1 instead of the leaderboard's versions,\n gives the same metrics bit for bit. A score on\n the leaderboard can therefore be checked by anyone with a minute of CPU\n time per boundary and about 3 GiB of memory.\n\n ## How much of each score lies in the tolerance\n\n A boundary counts as feasible when every normalised constraint violation\n is at most 0.01. Optimisers use this margin: the best entry on each\n single-objective problem sits at feasibility 0.0090 (geometrical) and\n 0.0097 (simple-to-build), and on the\n ten leading MHD-stable fronts between 17 % and 62 % of the points lie in\n the last tenth of the band, above 0.009. The table ranks the 55 distinct\n MHD-stable submissions (58 before removing resubmissions) by the hypervolume of their points within the\n tolerance you choose and shows the ten best.\n \"\"\")\n return\n\n\n@app.cell\ndef _(mo):\n tolerance = mo.ui.slider(0.0, 0.01, step=0.0005, value=0.001, label=\"Tolerance\", show_value=True, full_width=True)\n tolerance\n return (tolerance,)\n\n\n@app.cell\ndef _(entries, mo, tolerance):\n _table = mhd_stable_scores(entries, tolerance.value)\n mo.ui.table(_table.round({\"recorded score\": 2, \"score at this tolerance\": 2, \"points in the last tenth of the band\": 2}), selection=None)\n return\n\n\n@app.cell\ndef _(entries, mo, plt):\n _grid = np.linspace(0, TOLERANCE, 41)\n _leaders = pd.concat([mhd_stable_scores(entries, TOLERANCE, top=4), mhd_stable_scores(entries, 0.001, top=1)])\n _fig, _ax = plt.subplots(figsize=(7, 3.6))\n _points = entries[entries[\"problem\"] == \"mhd_stable\"]\n for (_, _row), _color in zip(_leaders.iterrows(), (\"#1f2328\", \"#0969da\", \"#9a6700\", \"#57606a\", \"#cf222e\")):\n _group = _points[(_points[\"user\"] == _row[\"user\"]) & (_points[\"submitted\"].str[:10] == _row[\"submitted\"]) & (_points[\"entry_score\"] == _row[\"recorded score\"])]\n _curve = [hypervolume(*_group.loc[_group[\"feasibility\"] <= _t, [\"gradient_scale_length\", \"aspect_ratio\"]].to_numpy().T) for _t in _grid]\n _ax.plot(_grid, _curve, color=_color, lw=1.6, label=f\"{_row['user']} ({_row['recorded score']:.1f})\")\n _ax.set_xlabel(\"tolerance on the normalised constraint violation\")\n _ax.set_ylabel(\"hypervolume\")\n _ax.legend(frameon=False, fontsize=8, loc=\"lower right\")\n for _side in (\"top\", \"right\"):\n _ax.spines[_side].set_visible(False)\n _fig.tight_layout()\n _fig\n return\n\n\n@app.cell(hide_code=True)\ndef _(mo):\n mo.md(r\"\"\"\n At a tolerance of 0.001 the four leading hypervolumes fall from 243–249\n to 167–186: between a quarter and a third of each leading score is\n earned by points that break a constraint by more than 0.1 %. Most of\n their points sit in the upper part of the band: 80 % of Eppy's 688\n points and 86 % of Wenyang Li's 783 lie above 0.005. Other fronts lose\n less. The front canyon-ant submitted on 9 September, 16th on the\n leaderboard with 43 points, keeps 94 % of its hypervolume and scores\n 192.5, the highest at this tolerance. The front phanerozoic submitted on\n 15 September, 15th, keeps 93 % with only 15 of its 668 points and scores\n 192.2. Wenyang Li follows with 185.8 and the recorded leader, Eppy, with\n 184.1; at a tolerance of 0 Wenyang Li leads with 185.7 ahead of Eppy\n with 183.9. A front scores well at a tight tolerance when its few points\n with margin are spread along it. The leaderboard rewards only the\n official tolerance, so the order at 0.001 shows how much of each score\n rests on the band, not which method would win a benchmark with a\n tighter one. (The third and fourth recorded submissions tie at tight\n tolerances because 507 of the fourth's 512 points also appear in the\n third.) The tolerance is part of the benchmark's definition, so none of\n this changes the leaderboard. It does mean that a comparison which cares\n about designs with margin should report the score at a tighter\n tolerance beside the official one.\n \"\"\")\n return\n\n\n@app.cell(hide_code=True)\ndef _(mo):\n mo.md(r\"\"\"\n ## The flux-compression correction\n\n The MHD-stable problem bounds a geometric proxy for turbulent transport:\n the flux-surface average, at ρ = 0.7, of $|\\nabla r|^2$ where the\n curvature drift is bad and zero elsewhere. Version 0.2.6 averages with the VMEC Jacobian; in\n version 0.3.0 Proxima changed the weight to the field-aligned (PEST)\n Jacobian, calling the old one a bug\n ([constellaration#108](https://github.com/proximafusion/constellaration/issues/108),\n [#109](https://github.com/proximafusion/constellaration/pull/109)). The\n leaderboard stays on 0.2.6.\n\n To see what the correction alone changes, we solved each boundary's\n equilibrium once with 0.2.6 and computed the proxy with both weights from\n that same equilibrium. A point that was feasible under 0.2.6 stays\n feasible exactly when the corrected proxy is at most 0.9 × 1.01; no other\n constraint is affected. The two leading fronts were evaluated in full\n (688 and 783 points). The third and fourth recorded fronts, 1,364 and 512\n points, were evaluated at 120 points each, drawn at random.\n \"\"\")\n return\n\n\n@app.cell\ndef _(rescored):\n flux = rescored[rescored[\"run\"] == \"flux compression only\"].copy()\n _solved = flux[\"error\"].isna()\n # The VMEC-weighted value recomputed here must be the one 0.2.6 reports.\n assert np.allclose(\n flux.loc[_solved, \"flux_compression_vmec\"].astype(float),\n flux.loc[_solved, \"flux_compression_in_regions_of_bad_curvature\"].astype(float),\n rtol=1e-12,\n )\n # A failed equilibrium counts as not feasible and stays in the point counts.\n flux[\"feasible_after_correction\"] = (\n _solved\n & (flux[\"recorded_feasibility\"] <= TOLERANCE)\n & (flux[\"flux_compression_pest\"].astype(float) <= FLUX_COMPRESSION_BOUND * (1 + TOLERANCE))\n )\n return (flux,)\n\n\n@app.cell\ndef _(flux, mo, plt):\n mo.stop(flux.empty, mo.md(\"The flux-compression runs are not in `data/rescored.parquet` yet.\"))\n _fig, _ax = plt.subplots(figsize=(5.5, 4.5))\n _ax.scatter(flux[\"flux_compression_vmec\"], flux[\"flux_compression_pest\"], s=6, color=\"#1f2328\", alpha=0.6, lw=0)\n _limit = FLUX_COMPRESSION_BOUND * (1 + TOLERANCE)\n _ax.axhline(_limit, color=\"#cf222e\", lw=0.8, ls=\"--\")\n _ax.axvline(_limit, color=\"#cf222e\", lw=0.8, ls=\"--\")\n _lo, _hi = float(flux[[\"flux_compression_vmec\", \"flux_compression_pest\"]].min().min()), float(flux[[\"flux_compression_vmec\", \"flux_compression_pest\"]].max().max())\n _ax.plot([_lo, _hi], [_lo, _hi], color=\"#57606a\", lw=0.8)\n _ax.set_xlabel(\"flux compression, VMEC weight (0.2.6)\")\n _ax.set_ylabel(\"flux compression, PEST weight (0.3.0)\")\n for _side in (\"top\", \"right\"):\n _ax.spines[_side].set_visible(False)\n _fig.tight_layout()\n _fig\n return\n\n\n@app.cell\ndef _(entries, flux, mo):\n mo.stop(flux.empty)\n _points = entries[entries[\"problem\"] == \"mhd_stable\"]\n _rows = []\n for (_user, _submitted), _group in flux.groupby([\"user\", \"submitted\"]):\n _all = _points[(_points[\"user\"] == _user) & (_points[\"submitted\"] == _submitted)]\n _complete = len(_group) == len(_all)\n _n = len(_group)\n _share = float(_group[\"feasible_after_correction\"].mean())\n # Wilson score interval, 95 %.\n _z = 1.96\n _centre = (_share + _z**2 / (2 * _n)) / (1 + _z**2 / _n)\n _half_width = _z * np.sqrt(_share * (1 - _share) / _n + _z**2 / (4 * _n**2)) / (1 + _z**2 / _n)\n _row = {\n \"user\": _user,\n \"submitted\": _submitted[:10],\n \"points evaluated\": f\"{_n} of {len(_all)}\",\n \"failed runs\": int(_group[\"error\"].notna().sum()),\n \"still feasible\": f\"{_share:.0%}\" + (\"\" if _complete else f\" (95 % interval {_centre - _half_width:.0%} to {_centre + _half_width:.0%})\"),\n \"recorded score\": round(float(_all[\"entry_score\"].iloc[0]), 2),\n }\n _ok = _group[_group[\"feasible_after_correction\"]]\n _recorded = _all.set_index(\"index\").loc[_ok[\"index\"]]\n _score = hypervolume(_recorded[\"gradient_scale_length\"], _recorded[\"aspect_ratio\"])\n # Adding points never lowers a hypervolume, so a sample's score is a lower bound.\n _row[\"score after correction\"] = f\"{_score:.2f}\" if _complete else f\"at least {_score:.2f}\"\n _rows.append(_row)\n mo.ui.table(pd.DataFrame(_rows).sort_values(\"recorded score\", ascending=False), selection=None)\n return\n\n\n@app.cell(hide_code=True)\ndef _(mo):\n mo.md(r\"\"\"\n The corrected weight raises the proxy for most points. The median ratio\n of the PEST-weighted to the VMEC-weighted value is 1.09 on Eppy's front\n and 1.10 on Wenyang Li's, against 1.04 on phanerozoic's and 1.03 on\n mmdeuss's; over all 1,711 points it lies between 0.96 and 1.20. Every point\n of the two leading fronts was feasible under 0.2.6 and every equilibrium\n solved, but only 181 of Eppy's 688 points (26 %) and 140 of Wenyang Li's\n 783 (18 %) stay within the bound. Their scores fall from 249.14 to\n 208.83 and from 248.23 to 208.05, so the two keep their order, 0.78\n apart instead of 0.91.\n\n The sampled fronts lose fewer points: 47 of the 120 sampled from\n phanerozoic's front stay within the bound (39 %, 95 % interval\n 31–48 %), and 58 of the 120 from mmdeuss's (48 %, 40–57 %). Their\n corrected scores are not known, but the surviving sample points alone\n score 220.47 and 223.84, and adding the rest of a front can only raise\n that. Both lower bounds are above the corrected\n scores of the first two fronts. Under the corrected proxy, the third and\n fourth recorded fronts would therefore both rank above the first and\n second; the samples do not settle the order between those two, and\n fronts below the fourth were not evaluated, so we cannot say which\n submission would lead.\n\n This is a statement about the benchmark, not about the designs. The\n proxy is a geometric stand-in for turbulent transport, not a transport\n calculation, and each submission was optimised against the 0.2.6\n version of it, which the leaderboard still uses. A front optimised\n against the corrected proxy could win back much of what it loses here.\n What the result shows is that the order at the top of the MHD-stable\n leaderboard depends on how one constraint is averaged, and that the\n correction raises the proxy more on the two leading fronts than on the\n next two.\n \"\"\")\n return\n\n\n@app.cell(hide_code=True)\ndef _(mo):\n mo.md(r\"\"\"\n ## Limits and references\n\n **What was run.** Scores in the first section come from a fresh\n installation of constellaration 0.2.6 on Python 3.10 with booz-xform\n 0.0.9 built from source, in the Docker image `python:3.10-slim`, on a\n Hetzner CPX42 server; the flux-compression runs use the same environment\n with the Boozer transform and the QI residual switched off, which the\n proxy does not depend on. The tolerance analysis needs no new runs: it\n uses the per-point feasibility the leaderboard recorded.\n\n **What was not run.** The third and fourth fronts are sampled, so their\n share of surviving points carries a 95 % Wilson interval and their\n corrected score is a lower bound. A boundary whose equilibrium failed\n would count as not feasible; none of the 1,711 runs failed. The\n corrected score keeps every other metric from 0.2.6 and so isolates the\n change of weight; it is not an evaluation with constellaration 0.3.0. A\n front that loses points under the correction could recover some\n hypervolume by re-optimisation, which we did not attempt.\n\n **Data.** `data/entries.parquet` is derived from the results dataset of\n the leaderboard Space,\n [proxima-fusion/constellaration-bench-results](https://huggingface.co/datasets/proxima-fusion/constellaration-bench-results)\n (178 rows, 174 of them evaluated submissions from users other than\n `test`, last modified 25 October 2026, no licence stated). It\n keeps the recorded objectives, feasibilities and scores with the\n submitters' public names and dates, and none of the submitted boundaries.\n `data/rescored.parquet` holds our own evaluations; `build_data.py` writes\n both files.\n\n Cadena, S. A. et al. (2025). ConStellaration: A dataset of QI-like\n stellarator plasma boundaries and optimization benchmarks. *NeurIPS 2025\n Datasets and Benchmarks Track*. [arXiv:2506.19583](https://arxiv.org/abs/2506.19583).\n \"\"\")\n return\n\n\nif __name__ == \"__main__\":\n app.run()\n","revision":"2026-10-27T16:53:23.225Z","appConfig":{"width":"medium","app_title":"The ConStellaration leaderboard, scored again"},"firstPaint":{"notebook":{"cells":[{"code":"import numpy as np\nimport pandas as pd\n\nENTRIES = \"data/entries.parquet\"\nRESCORED = \"data/rescored.parquet\"\nTOLERANCE = 0.01\nFLUX_COMPRESSION_BOUND = 0.9\nTITLES = {\"geometrical\": \"Geometrical\", \"simple_to_build\": \"Simple-to-build QI\", \"mhd_stable\": \"MHD-stable QI\"}","code_hash":"d2b4e64b7ac567cae2422e837f8b6867","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"setup","name":"setup"},{"code":"def hypervolume(gradient_scale_length, aspect_ratio, reference=(1.0, 20.0)) -> float:\n \"\"\"Area dominated by the points in the plane of (−L∇B, A), up to the reference point.\n\n The MHD-stable score of constellaration 0.2.6 for feasible points; equal\n to pymoo's Hypervolume to 1e-12 on all 58 leaderboard submissions.\n \"\"\"\n x = -np.asarray(gradient_scale_length, dtype=float)\n y = np.asarray(aspect_ratio, dtype=float)\n keep = (x < reference[0]) & (y < reference[1])\n x, y = x[keep], y[keep]\n order = np.argsort(x, kind=\"stable\")\n x, y = x[order], y[order]\n edges = np.append(x, reference[0])\n lowest = np.minimum.accumulate(y) if len(y) else y\n return float(np.sum(np.diff(edges) * (reference[1] - lowest)))","code_hash":"b8fc5ad884246580801cd9d1dbf77257","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"Hbol","name":"*hypervolume"},{"code":"def load_entries(path: str = ENTRIES) -> pd.DataFrame:\n \"\"\"The leaderboard as recorded on 25 October 2026: one row per single-objective\n submission and one per point of each MHD-stable submission.\"\"\"\n return pd.read_parquet(path)","code_hash":"a733fc98730dee67a47c59ce9ade8c90","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"MJUe","name":"*load_entries"},{"code":"def mhd_stable_scores(entries: pd.DataFrame, tolerance: float = TOLERANCE, top: int = 10) -> pd.DataFrame:\n \"\"\"The `top` MHD-stable submissions by hypervolume when only points within `tolerance` count.\n\n All submissions are ranked; a resubmission with the same user and\n recorded score counts once. \"recorded rank\" is the order of the\n recorded scores.\n \"\"\"\n points = entries[entries[\"problem\"] == \"mhd_stable\"]\n rows = []\n for (user, submitted), group in points.groupby([\"user\", \"submitted\"]):\n kept = group[group[\"feasibility\"] <= tolerance]\n rows.append({\n \"user\": user,\n \"submitted\": submitted[:10],\n \"points\": len(group),\n \"recorded score\": group[\"entry_score\"].iloc[0],\n \"score at this tolerance\": hypervolume(kept[\"gradient_scale_length\"], kept[\"aspect_ratio\"]),\n \"points in the last tenth of the band\": float(np.mean((group[\"feasibility\"] > 0.9 * TOLERANCE) & (group[\"feasibility\"] <= TOLERANCE))),\n })\n table = pd.DataFrame(rows).sort_values(\"recorded score\", ascending=False).drop_duplicates([\"user\", \"recorded score\"])\n table.insert(0, \"recorded rank\", range(1, len(table) + 1))\n table.insert(0, \"rank at this tolerance\", table[\"score at this tolerance\"].rank(ascending=False, method=\"min\").astype(int))\n return table.sort_values([\"rank at this tolerance\", \"recorded rank\"]).head(top).reset_index(drop=True)","code_hash":"bbb1544eba6a2e8399173269a80d7a16","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"vblA","name":"*mhd_stable_scores"},{"code":"import marimo as mo\nimport matplotlib.pyplot as plt","code_hash":"7802eb41d8b52c22105f8bbe08d60476","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"bkHC","name":"_"},{"code":"mo.md(r\"\"\"\n# The ConStellaration leaderboard, scored again\n\nThe ConStellaration leaderboard on Hugging Face ranks stellarator boundaries\nwith Proxima Fusion's evaluator, constellaration 0.2.6\n([Cadena et al. 2025](https://arxiv.org/abs/2506.19583)). By 25 October\n2026 it held 174 evaluated submissions. The best entries had passed the\npaper's baselines on all three problems: a maximum elongation of 1.18\nagainst 1.27 on the geometrical problem, a normalised gradient scale\nlength of 13.4 against 8.6 on the simple-to-build problem, and a\nhypervolume of 249 against 130 on the MHD-stable problem.\n\nWe check whether the recorded scores reproduce on another machine, how\nmuch of each leading score depends on the 1 % tolerance the evaluator\ngrants every constraint, and how the leading MHD-stable fronts change\nwhen the flux-compression proxy is averaged the way Proxima corrected it\nin version 0.3.0. The forward-model runs were made with `scripts/rescore.py`\nof this project; this notebook only reads their results.\n\"\"\")","code_hash":"4110e55500ae21febdd38c06dff49d7b","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":true},"id":"lEQa","name":"_"},{"code":"entries = load_entries()\nrescored = pd.read_parquet(RESCORED)","code_hash":"4082a175cd617b8c1fed199273d0cd0b","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"PKri","name":"_"},{"code":"mo.md(r\"\"\"\n## The recorded scores reproduce\n\nWe evaluated the paper's baselines (the first submissions of the paper's\nfirst author), the three best single-objective entries per problem, and a\nrandom sample of points from the best MHD-stable front, in a fresh\nenvironment built like the leaderboard's (Python 3.10, constellaration\n0.2.6, booz-xform 0.0.9 compiled from source) on an x86-64 server. The\nreference boundary and the paper's baselines also ran in ScienceHub's\nkernel image (Python 3.13, booz-xform 0.1.0), which the evaluate\nnotebook of this project uses.\n\"\"\")","code_hash":"a5ee761a0ad424752a33fff279fbd7e0","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":true},"id":"Xref","name":"_"},{"code":"_checked = rescored[rescored[\"recorded_feasibility\"].notna() & rescored[\"flux_compression_pest\"].isna() & (rescored[\"run\"] == \"leaderboard environment\")]\n_single = _checked[_checked[\"problem\"] != \"mhd_stable\"]\n_multi = _checked[_checked[\"problem\"] == \"mhd_stable\"]\n_table = pd.DataFrame({\n \"entry\": _single[\"user\"] + \", \" + _single[\"submitted\"].str[:10],\n \"problem\": _single[\"problem\"].map(TITLES),\n \"recorded score\": _single[\"recorded_entry_score\"],\n \"our score\": _single[\"score\"],\n \"recorded feasibility\": _single[\"recorded_feasibility\"],\n \"our feasibility\": _single[\"feasibility\"],\n})\n_gap_single = float((_single[\"score\"] - _single[\"recorded_entry_score\"]).abs().max())\n_gap_points = float((_multi[\"feasibility\"] - _multi[\"recorded_feasibility\"]).abs().max())\n_gap_objective = float((_multi[\"minimum_normalized_magnetic_gradient_scale_length\"] - _multi[\"recorded_gradient_scale_length\"]).abs().max())\n_metrics = list(rescored.loc[:, \"feasibility\":\"flux_compression_in_regions_of_bad_curvature\"].columns.drop([\"flux_compression_pest\", \"flux_compression_vmec\"]))\n_pairs = rescored[rescored[\"run\"] == \"kernel image\"].merge(\n rescored[rescored[\"run\"] == \"leaderboard environment\"], on=[\"user\", \"submitted\", \"problem\", \"index\"], suffixes=(\"\", \"_leaderboard\")\n)\n_kernel = _pairs[_metrics].to_numpy(dtype=float)\n_leaderboard = _pairs[[f\"{_name}_leaderboard\" for _name in _metrics]].to_numpy(dtype=float)\nassert np.array_equal(np.isnan(_kernel), np.isnan(_leaderboard))\n_values = int(np.sum(~np.isnan(_kernel)))\n_gap_kernel = float(np.nanmax(np.abs(_kernel - _leaderboard)))\nmo.vstack([\n mo.ui.table(_table, selection=None),\n mo.md(\n f\"Largest difference in a single-objective score: **{_gap_single:.1e}**. \"\n f\"Over {len(_multi)} MHD-stable points, the largest difference in feasibility is **{_gap_points:.1e}** \"\n f\"and in L∇B **{_gap_objective:.1e}**. \"\n f\"For the {len(_pairs)} boundaries evaluated in both environments, the largest difference over \"\n f\"{_values} metric values, feasibility and score included, is **{_gap_kernel:g}**.\"\n ),\n])","code_hash":"e6e9c3199ebb5cad0ed5ae275e316608","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"SFPL","name":"_"},{"code":"mo.md(r\"\"\"\nThe evaluator is deterministic: VMEC++ runs single-threaded, and the\nsame boundary gives the same numbers on another machine to the precision\nshown. The kernel image, with Python 3.13, booz-xform 0.1.0, simsopt\n1.9.0, jax 0.5.3 and scipy 1.17.1 instead of the leaderboard's versions,\ngives the same metrics bit for bit. A score on\nthe leaderboard can therefore be checked by anyone with a minute of CPU\ntime per boundary and about 3 GiB of memory.\n\n## How much of each score lies in the tolerance\n\nA boundary counts as feasible when every normalised constraint violation\nis at most 0.01. Optimisers use this margin: the best entry on each\nsingle-objective problem sits at feasibility 0.0090 (geometrical) and\n0.0097 (simple-to-build), and on the\nten leading MHD-stable fronts between 17 % and 62 % of the points lie in\nthe last tenth of the band, above 0.009. The table ranks the 55 distinct\nMHD-stable submissions (58 before removing resubmissions) by the hypervolume of their points within the\ntolerance you choose and shows the ten best.\n\"\"\")","code_hash":"7b0d2d608fb6cc6564b6cbf1ba795955","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":true},"id":"BYtC","name":"_"},{"code":"tolerance = mo.ui.slider(0.0, 0.01, step=0.0005, value=0.001, label=\"Tolerance\", show_value=True, full_width=True)\ntolerance","code_hash":"ffde1efcc8f9ec83f57c29d3a714a609","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"RGSE","name":"_"},{"code":"_table = mhd_stable_scores(entries, tolerance.value)\nmo.ui.table(_table.round({\"recorded score\": 2, \"score at this tolerance\": 2, \"points in the last tenth of the band\": 2}), selection=None)","code_hash":"4ec204f71cd12fab0c17e42b6447aa3f","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"Kclp","name":"_"},{"code":"_grid = np.linspace(0, TOLERANCE, 41)\n_leaders = pd.concat([mhd_stable_scores(entries, TOLERANCE, top=4), mhd_stable_scores(entries, 0.001, top=1)])\n_fig, _ax = plt.subplots(figsize=(7, 3.6))\n_points = entries[entries[\"problem\"] == \"mhd_stable\"]\nfor (_, _row), _color in zip(_leaders.iterrows(), (\"#1f2328\", \"#0969da\", \"#9a6700\", \"#57606a\", \"#cf222e\")):\n _group = _points[(_points[\"user\"] == _row[\"user\"]) & (_points[\"submitted\"].str[:10] == _row[\"submitted\"]) & (_points[\"entry_score\"] == _row[\"recorded score\"])]\n _curve = [hypervolume(*_group.loc[_group[\"feasibility\"] <= _t, [\"gradient_scale_length\", \"aspect_ratio\"]].to_numpy().T) for _t in _grid]\n _ax.plot(_grid, _curve, color=_color, lw=1.6, label=f\"{_row['user']} ({_row['recorded score']:.1f})\")\n_ax.set_xlabel(\"tolerance on the normalised constraint violation\")\n_ax.set_ylabel(\"hypervolume\")\n_ax.legend(frameon=False, fontsize=8, loc=\"lower right\")\nfor _side in (\"top\", \"right\"):\n _ax.spines[_side].set_visible(False)\n_fig.tight_layout()\n_fig","code_hash":"f2151e5ec60c40a5f6a4e883a7c3f67a","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"emfo","name":"_"},{"code":"mo.md(r\"\"\"\nAt a tolerance of 0.001 the four leading hypervolumes fall from 243–249\nto 167–186: between a quarter and a third of each leading score is\nearned by points that break a constraint by more than 0.1 %. Most of\ntheir points sit in the upper part of the band: 80 % of Eppy's 688\npoints and 86 % of Wenyang Li's 783 lie above 0.005. Other fronts lose\nless. The front canyon-ant submitted on 9 September, 16th on the\nleaderboard with 43 points, keeps 94 % of its hypervolume and scores\n192.5, the highest at this tolerance. The front phanerozoic submitted on\n15 September, 15th, keeps 93 % with only 15 of its 668 points and scores\n192.2. Wenyang Li follows with 185.8 and the recorded leader, Eppy, with\n184.1; at a tolerance of 0 Wenyang Li leads with 185.7 ahead of Eppy\nwith 183.9. A front scores well at a tight tolerance when its few points\nwith margin are spread along it. The leaderboard rewards only the\nofficial tolerance, so the order at 0.001 shows how much of each score\nrests on the band, not which method would win a benchmark with a\ntighter one. (The third and fourth recorded submissions tie at tight\ntolerances because 507 of the fourth's 512 points also appear in the\nthird.) The tolerance is part of the benchmark's definition, so none of\nthis changes the leaderboard. It does mean that a comparison which cares\nabout designs with margin should report the score at a tighter\ntolerance beside the official one.\n\"\"\")","code_hash":"a9ef7be4b35182d4bba35b55b8e0d3ae","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":true},"id":"Hstk","name":"_"},{"code":"mo.md(r\"\"\"\n## The flux-compression correction\n\nThe MHD-stable problem bounds a geometric proxy for turbulent transport:\nthe flux-surface average, at ρ = 0.7, of $|\\nabla r|^2$ where the\ncurvature drift is bad and zero elsewhere. Version 0.2.6 averages with the VMEC Jacobian; in\nversion 0.3.0 Proxima changed the weight to the field-aligned (PEST)\nJacobian, calling the old one a bug\n([constellaration#108](https://github.com/proximafusion/constellaration/issues/108),\n[#109](https://github.com/proximafusion/constellaration/pull/109)). The\nleaderboard stays on 0.2.6.\n\nTo see what the correction alone changes, we solved each boundary's\nequilibrium once with 0.2.6 and computed the proxy with both weights from\nthat same equilibrium. A point that was feasible under 0.2.6 stays\nfeasible exactly when the corrected proxy is at most 0.9 × 1.01; no other\nconstraint is affected. The two leading fronts were evaluated in full\n(688 and 783 points). The third and fourth recorded fronts, 1,364 and 512\npoints, were evaluated at 120 points each, drawn at random.\n\"\"\")","code_hash":"a93d5542e752800fd43b0c4760ebe56b","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":true},"id":"nWHF","name":"_"},{"code":"flux = rescored[rescored[\"run\"] == \"flux compression only\"].copy()\n_solved = flux[\"error\"].isna()\n# The VMEC-weighted value recomputed here must be the one 0.2.6 reports.\nassert np.allclose(\n flux.loc[_solved, \"flux_compression_vmec\"].astype(float),\n flux.loc[_solved, \"flux_compression_in_regions_of_bad_curvature\"].astype(float),\n rtol=1e-12,\n)\n# A failed equilibrium counts as not feasible and stays in the point counts.\nflux[\"feasible_after_correction\"] = (\n _solved\n & (flux[\"recorded_feasibility\"] <= TOLERANCE)\n & (flux[\"flux_compression_pest\"].astype(float) <= FLUX_COMPRESSION_BOUND * (1 + TOLERANCE))\n)","code_hash":"4df4457e3c5ce8ad47e95e9b1fff950e","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"iLit","name":"_"},{"code":"mo.stop(flux.empty, mo.md(\"The flux-compression runs are not in `data/rescored.parquet` yet.\"))\n_fig, _ax = plt.subplots(figsize=(5.5, 4.5))\n_ax.scatter(flux[\"flux_compression_vmec\"], flux[\"flux_compression_pest\"], s=6, color=\"#1f2328\", alpha=0.6, lw=0)\n_limit = FLUX_COMPRESSION_BOUND * (1 + TOLERANCE)\n_ax.axhline(_limit, color=\"#cf222e\", lw=0.8, ls=\"--\")\n_ax.axvline(_limit, color=\"#cf222e\", lw=0.8, ls=\"--\")\n_lo, _hi = float(flux[[\"flux_compression_vmec\", \"flux_compression_pest\"]].min().min()), float(flux[[\"flux_compression_vmec\", \"flux_compression_pest\"]].max().max())\n_ax.plot([_lo, _hi], [_lo, _hi], color=\"#57606a\", lw=0.8)\n_ax.set_xlabel(\"flux compression, VMEC weight (0.2.6)\")\n_ax.set_ylabel(\"flux compression, PEST weight (0.3.0)\")\nfor _side in (\"top\", \"right\"):\n _ax.spines[_side].set_visible(False)\n_fig.tight_layout()\n_fig","code_hash":"9f5a99ea9575c727e673e33f855a7bde","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"ZHCJ","name":"_"},{"code":"mo.stop(flux.empty)\n_points = entries[entries[\"problem\"] == \"mhd_stable\"]\n_rows = []\nfor (_user, _submitted), _group in flux.groupby([\"user\", \"submitted\"]):\n _all = _points[(_points[\"user\"] == _user) & (_points[\"submitted\"] == _submitted)]\n _complete = len(_group) == len(_all)\n _n = len(_group)\n _share = float(_group[\"feasible_after_correction\"].mean())\n # Wilson score interval, 95 %.\n _z = 1.96\n _centre = (_share + _z**2 / (2 * _n)) / (1 + _z**2 / _n)\n _half_width = _z * np.sqrt(_share * (1 - _share) / _n + _z**2 / (4 * _n**2)) / (1 + _z**2 / _n)\n _row = {\n \"user\": _user,\n \"submitted\": _submitted[:10],\n \"points evaluated\": f\"{_n} of {len(_all)}\",\n \"failed runs\": int(_group[\"error\"].notna().sum()),\n \"still feasible\": f\"{_share:.0%}\" + (\"\" if _complete else f\" (95 % interval {_centre - _half_width:.0%} to {_centre + _half_width:.0%})\"),\n \"recorded score\": round(float(_all[\"entry_score\"].iloc[0]), 2),\n }\n _ok = _group[_group[\"feasible_after_correction\"]]\n _recorded = _all.set_index(\"index\").loc[_ok[\"index\"]]\n _score = hypervolume(_recorded[\"gradient_scale_length\"], _recorded[\"aspect_ratio\"])\n # Adding points never lowers a hypervolume, so a sample's score is a lower bound.\n _row[\"score after correction\"] = f\"{_score:.2f}\" if _complete else f\"at least {_score:.2f}\"\n _rows.append(_row)\nmo.ui.table(pd.DataFrame(_rows).sort_values(\"recorded score\", ascending=False), selection=None)","code_hash":"cc77ec99c6f064ed7483a014dd2f842d","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":false},"id":"ROlb","name":"_"},{"code":"mo.md(r\"\"\"\nThe corrected weight raises the proxy for most points. The median ratio\nof the PEST-weighted to the VMEC-weighted value is 1.09 on Eppy's front\nand 1.10 on Wenyang Li's, against 1.04 on phanerozoic's and 1.03 on\nmmdeuss's; over all 1,711 points it lies between 0.96 and 1.20. Every point\nof the two leading fronts was feasible under 0.2.6 and every equilibrium\nsolved, but only 181 of Eppy's 688 points (26 %) and 140 of Wenyang Li's\n783 (18 %) stay within the bound. Their scores fall from 249.14 to\n208.83 and from 248.23 to 208.05, so the two keep their order, 0.78\napart instead of 0.91.\n\nThe sampled fronts lose fewer points: 47 of the 120 sampled from\nphanerozoic's front stay within the bound (39 %, 95 % interval\n31–48 %), and 58 of the 120 from mmdeuss's (48 %, 40–57 %). Their\ncorrected scores are not known, but the surviving sample points alone\nscore 220.47 and 223.84, and adding the rest of a front can only raise\nthat. Both lower bounds are above the corrected\nscores of the first two fronts. Under the corrected proxy, the third and\nfourth recorded fronts would therefore both rank above the first and\nsecond; the samples do not settle the order between those two, and\nfronts below the fourth were not evaluated, so we cannot say which\nsubmission would lead.\n\nThis is a statement about the benchmark, not about the designs. The\nproxy is a geometric stand-in for turbulent transport, not a transport\ncalculation, and each submission was optimised against the 0.2.6\nversion of it, which the leaderboard still uses. A front optimised\nagainst the corrected proxy could win back much of what it loses here.\nWhat the result shows is that the order at the top of the MHD-stable\nleaderboard depends on how one constraint is averaged, and that the\ncorrection raises the proxy more on the two leading fronts than on the\nnext two.\n\"\"\")","code_hash":"c2a3302f87fac6f3e7f9717d07105740","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":true},"id":"qnkX","name":"_"},{"code":"mo.md(r\"\"\"\n## Limits and references\n\n**What was run.** Scores in the first section come from a fresh\ninstallation of constellaration 0.2.6 on Python 3.10 with booz-xform\n0.0.9 built from source, in the Docker image `python:3.10-slim`, on a\nHetzner CPX42 server; the flux-compression runs use the same environment\nwith the Boozer transform and the QI residual switched off, which the\nproxy does not depend on. The tolerance analysis needs no new runs: it\nuses the per-point feasibility the leaderboard recorded.\n\n**What was not run.** The third and fourth fronts are sampled, so their\nshare of surviving points carries a 95 % Wilson interval and their\ncorrected score is a lower bound. A boundary whose equilibrium failed\nwould count as not feasible; none of the 1,711 runs failed. The\ncorrected score keeps every other metric from 0.2.6 and so isolates the\nchange of weight; it is not an evaluation with constellaration 0.3.0. A\nfront that loses points under the correction could recover some\nhypervolume by re-optimisation, which we did not attempt.\n\n**Data.** `data/entries.parquet` is derived from the results dataset of\nthe leaderboard Space,\n[proxima-fusion/constellaration-bench-results](https://huggingface.co/datasets/proxima-fusion/constellaration-bench-results)\n(178 rows, 174 of them evaluated submissions from users other than\n`test`, last modified 25 October 2026, no licence stated). It\nkeeps the recorded objectives, feasibilities and scores with the\nsubmitters' public names and dates, and none of the submitted boundaries.\n`data/rescored.parquet` holds our own evaluations; `build_data.py` writes\nboth files.\n\nCadena, S. A. et al. (2025). ConStellaration: A dataset of QI-like\nstellarator plasma boundaries and optimization benchmarks. *NeurIPS 2025\nDatasets and Benchmarks Track*. [arXiv:2506.19583](https://arxiv.org/abs/2506.19583).\n\"\"\")","code_hash":"3f742dabcf0a42dc34ff74307b63c730","config":{"column":null,"disabled":false,"expand_output":false,"hide_code":true},"id":"TqIu","name":"_"}],"metadata":{"marimo_version":"0.25.2"},"version":"1"},"session":{"cells":[{"code_hash":"d2b4e64b7ac567cae2422e837f8b6867","console":[],"id":"setup","outputs":[{"data":{"text/plain":""},"type":"data"}]},{"code_hash":"b8fc5ad884246580801cd9d1dbf77257","console":[],"id":"Hbol","outputs":[{"data":{"text/plain":""},"type":"data"}]},{"code_hash":"a733fc98730dee67a47c59ce9ade8c90","console":[],"id":"MJUe","outputs":[{"data":{"text/plain":""},"type":"data"}]},{"code_hash":"bbb1544eba6a2e8399173269a80d7a16","console":[],"id":"vblA","outputs":[{"data":{"text/plain":""},"type":"data"}]},{"code_hash":"7802eb41d8b52c22105f8bbe08d60476","console":[],"id":"bkHC","outputs":[{"data":{"text/plain":""},"type":"data"}]},{"code_hash":"4110e55500ae21febdd38c06dff49d7b","console":[],"id":"lEQa","outputs":[{"data":{"text/markdown":"

The ConStellaration leaderboard, scored again

\nThe ConStellaration leaderboard on Hugging Face ranks stellarator boundaries\nwith Proxima Fusion's evaluator, constellaration 0.2.6\n(Cadena et al. 2025). By 25 October\n2026 it held 174 evaluated submissions. The best entries had passed the\npaper's baselines on all three problems: a maximum elongation of 1.18\nagainst 1.27 on the geometrical problem, a normalised gradient scale\nlength of 13.4 against 8.6 on the simple-to-build problem, and a\nhypervolume of 249 against 130 on the MHD-stable problem.\nWe check whether the recorded scores reproduce on another machine, how\nmuch of each leading score depends on the 1 % tolerance the evaluator\ngrants every constraint, and how the leading MHD-stable fronts change\nwhen the flux-compression proxy is averaged the way Proxima corrected it\nin version 0.3.0. The forward-model runs were made with scripts/rescore.py\nof this project; this notebook only reads their results."},"type":"data"}]},{"code_hash":"4082a175cd617b8c1fed199273d0cd0b","console":[],"id":"PKri","outputs":[{"data":{"text/plain":""},"type":"data"}]},{"code_hash":"a5ee761a0ad424752a33fff279fbd7e0","console":[],"id":"Xref","outputs":[{"data":{"text/markdown":"

The recorded scores reproduce

\nWe evaluated the paper's baselines (the first submissions of the paper's\nfirst author), the three best single-objective entries per problem, and a\nrandom sample of points from the best MHD-stable front, in a fresh\nenvironment built like the leaderboard's (Python 3.10, constellaration\n0.2.6, booz-xform 0.0.9 compiled from source) on an x86-64 server. The\nreference boundary and the paper's baselines also ran in ScienceHub's\nkernel image (Python 3.13, booz-xform 0.1.0), which the evaluate\nnotebook of this project uses."},"type":"data"}]},{"code_hash":"e6e9c3199ebb5cad0ed5ae275e316608","console":[],"id":"SFPL","outputs":[{"data":{"text/html":"
Largest difference in a single-objective score: 1.1e-16. Over 28 MHD-stable points, the largest difference in feasibility is 0.0e+00 and in L∇B 0.0e+00. For the 8 boundaries evaluated in both environments, the largest difference over 110 metric values, feasibility and score included, is 0.
"},"type":"data"}]},{"code_hash":"7b0d2d608fb6cc6564b6cbf1ba795955","console":[],"id":"BYtC","outputs":[{"data":{"text/markdown":"The evaluator is deterministic: VMEC++ runs single-threaded, and the\nsame boundary gives the same numbers on another machine to the precision\nshown. The kernel image, with Python 3.13, booz-xform 0.1.0, simsopt\n1.9.0, jax 0.5.3 and scipy 1.17.1 instead of the leaderboard's versions,\ngives the same metrics bit for bit. A score on\nthe leaderboard can therefore be checked by anyone with a minute of CPU\ntime per boundary and about 3 GiB of memory.\n

How much of each score lies in the tolerance

\nA boundary counts as feasible when every normalised constraint violation\nis at most 0.01. Optimisers use this margin: the best entry on each\nsingle-objective problem sits at feasibility 0.0090 (geometrical) and\n0.0097 (simple-to-build), and on the\nten leading MHD-stable fronts between 17 % and 62 % of the points lie in\nthe last tenth of the band, above 0.009. The table ranks the 55 distinct\nMHD-stable submissions (58 before removing resubmissions) by the hypervolume of their points within the\ntolerance you choose and shows the ten best."},"type":"data"}]},{"code_hash":"ffde1efcc8f9ec83f57c29d3a714a609","console":[],"id":"RGSE","outputs":[{"data":{"text/html":""},"type":"data"}]},{"code_hash":"4ec204f71cd12fab0c17e42b6447aa3f","console":[],"id":"Kclp","outputs":[{"data":{"text/html":""},"type":"data"}]},{"code_hash":"f2151e5ec60c40a5f6a4e883a7c3f67a","console":[],"id":"emfo","outputs":[{"data":{"application/vnd.marimo+mimebundle":"{\"image/png\":\"https://ncct56n7w75x7sivrb3lxfl4n4.notebooks.sciencehubusercontent.com/_outputs/eyJuYW1lc3BhY2VJZCI6Im5zLXdicjZya2tkOWZqMG1neGUiLCJwcm9qZWN0SWQiOiJwcm9qLWs4ams4ZnQ4dDAzamc1NnEiLCJub3RlYm9va0lkIjoibmItem50ZXJnM3ZxYjUzNWI2MSIsImhhc2giOiI2ZGU2ZDBlZjE2OTFjYTVkYzUyY2I4ZjU4YjJjMjIzNSIsInNoYXJlZCI6dHJ1ZX0.0.Jp0AtjeNYt4Oa_3rYhvBNv8Nljtx9taQiBEonvo-BbY\",\"__metadata__\":{\"image/png\":{\"width\":690,\"height\":350}}}"},"type":"data"}]},{"code_hash":"a9ef7be4b35182d4bba35b55b8e0d3ae","console":[],"id":"Hstk","outputs":[{"data":{"text/markdown":"At a tolerance of 0.001 the four leading hypervolumes fall from 243–249\nto 167–186: between a quarter and a third of each leading score is\nearned by points that break a constraint by more than 0.1 %. Most of\ntheir points sit in the upper part of the band: 80 % of Eppy's 688\npoints and 86 % of Wenyang Li's 783 lie above 0.005. Other fronts lose\nless. The front canyon-ant submitted on 9 September, 16th on the\nleaderboard with 43 points, keeps 94 % of its hypervolume and scores\n192.5, the highest at this tolerance. The front phanerozoic submitted on\n15 September, 15th, keeps 93 % with only 15 of its 668 points and scores\n192.2. Wenyang Li follows with 185.8 and the recorded leader, Eppy, with\n184.1; at a tolerance of 0 Wenyang Li leads with 185.7 ahead of Eppy\nwith 183.9. A front scores well at a tight tolerance when its few points\nwith margin are spread along it. The leaderboard rewards only the\nofficial tolerance, so the order at 0.001 shows how much of each score\nrests on the band, not which method would win a benchmark with a\ntighter one. (The third and fourth recorded submissions tie at tight\ntolerances because 507 of the fourth's 512 points also appear in the\nthird.) The tolerance is part of the benchmark's definition, so none of\nthis changes the leaderboard. It does mean that a comparison which cares\nabout designs with margin should report the score at a tighter\ntolerance beside the official one."},"type":"data"}]},{"code_hash":"a93d5542e752800fd43b0c4760ebe56b","console":[],"id":"nWHF","outputs":[{"data":{"text/markdown":"

The flux-compression correction

\nThe MHD-stable problem bounds a geometric proxy for turbulent transport:\nthe flux-surface average, at ρ = 0.7, of ||(|\\nabla r|^2||) where the\ncurvature drift is bad and zero elsewhere. Version 0.2.6 averages with the VMEC Jacobian; in\nversion 0.3.0 Proxima changed the weight to the field-aligned (PEST)\nJacobian, calling the old one a bug\n(constellaration#108,\n#109). The\nleaderboard stays on 0.2.6.\nTo see what the correction alone changes, we solved each boundary's\nequilibrium once with 0.2.6 and computed the proxy with both weights from\nthat same equilibrium. A point that was feasible under 0.2.6 stays\nfeasible exactly when the corrected proxy is at most 0.9 × 1.01; no other\nconstraint is affected. The two leading fronts were evaluated in full\n(688 and 783 points). The third and fourth recorded fronts, 1,364 and 512\npoints, were evaluated at 120 points each, drawn at random."},"type":"data"}]},{"code_hash":"4df4457e3c5ce8ad47e95e9b1fff950e","console":[],"id":"iLit","outputs":[{"data":{"text/plain":""},"type":"data"}]},{"code_hash":"9f5a99ea9575c727e673e33f855a7bde","console":[],"id":"ZHCJ","outputs":[{"data":{"application/vnd.marimo+mimebundle":"{\"image/png\":\"https://ncct56n7w75x7sivrb3lxfl4n4.notebooks.sciencehubusercontent.com/_outputs/eyJuYW1lc3BhY2VJZCI6Im5zLXdicjZya2tkOWZqMG1neGUiLCJwcm9qZWN0SWQiOiJwcm9qLWs4ams4ZnQ4dDAzamc1NnEiLCJub3RlYm9va0lkIjoibmItem50ZXJnM3ZxYjUzNWI2MSIsImhhc2giOiJhNmIwNTI4NTQxYmM0OGZmNjRjMWQxZWU4ZGFlNmY4YiIsInNoYXJlZCI6dHJ1ZX0.0.P748UiSZ73ISzRZS0JR_87C2IJDCLFxl-57F-ot44QQ\",\"__metadata__\":{\"image/png\":{\"width\":540,\"height\":440}}}"},"type":"data"}]},{"code_hash":"cc77ec99c6f064ed7483a014dd2f842d","console":[],"id":"ROlb","outputs":[{"data":{"text/html":""},"type":"data"}]},{"code_hash":"c2a3302f87fac6f3e7f9717d07105740","console":[],"id":"qnkX","outputs":[{"data":{"text/markdown":"The corrected weight raises the proxy for most points. The median ratio\nof the PEST-weighted to the VMEC-weighted value is 1.09 on Eppy's front\nand 1.10 on Wenyang Li's, against 1.04 on phanerozoic's and 1.03 on\nmmdeuss's; over all 1,711 points it lies between 0.96 and 1.20. Every point\nof the two leading fronts was feasible under 0.2.6 and every equilibrium\nsolved, but only 181 of Eppy's 688 points (26 %) and 140 of Wenyang Li's\n783 (18 %) stay within the bound. Their scores fall from 249.14 to\n208.83 and from 248.23 to 208.05, so the two keep their order, 0.78\napart instead of 0.91.\nThe sampled fronts lose fewer points: 47 of the 120 sampled from\nphanerozoic's front stay within the bound (39 %, 95 % interval\n31–48 %), and 58 of the 120 from mmdeuss's (48 %, 40–57 %). Their\ncorrected scores are not known, but the surviving sample points alone\nscore 220.47 and 223.84, and adding the rest of a front can only raise\nthat. Both lower bounds are above the corrected\nscores of the first two fronts. Under the corrected proxy, the third and\nfourth recorded fronts would therefore both rank above the first and\nsecond; the samples do not settle the order between those two, and\nfronts below the fourth were not evaluated, so we cannot say which\nsubmission would lead.\nThis is a statement about the benchmark, not about the designs. The\nproxy is a geometric stand-in for turbulent transport, not a transport\ncalculation, and each submission was optimised against the 0.2.6\nversion of it, which the leaderboard still uses. A front optimised\nagainst the corrected proxy could win back much of what it loses here.\nWhat the result shows is that the order at the top of the MHD-stable\nleaderboard depends on how one constraint is averaged, and that the\ncorrection raises the proxy more on the two leading fronts than on the\nnext two."},"type":"data"}]},{"code_hash":"3f742dabcf0a42dc34ff74307b63c730","console":[],"id":"TqIu","outputs":[{"data":{"text/markdown":"

Limits and references

\nWhat was run. Scores in the first section come from a fresh\ninstallation of constellaration 0.2.6 on Python 3.10 with booz-xform\n0.0.9 built from source, in the Docker image python:3.10-slim, on a\nHetzner CPX42 server; the flux-compression runs use the same environment\nwith the Boozer transform and the QI residual switched off, which the\nproxy does not depend on. The tolerance analysis needs no new runs: it\nuses the per-point feasibility the leaderboard recorded.\nWhat was not run. The third and fourth fronts are sampled, so their\nshare of surviving points carries a 95 % Wilson interval and their\ncorrected score is a lower bound. A boundary whose equilibrium failed\nwould count as not feasible; none of the 1,711 runs failed. The\ncorrected score keeps every other metric from 0.2.6 and so isolates the\nchange of weight; it is not an evaluation with constellaration 0.3.0. A\nfront that loses points under the correction could recover some\nhypervolume by re-optimisation, which we did not attempt.\nData. data/entries.parquet is derived from the results dataset of\nthe leaderboard Space,\nproxima-fusion/constellaration-bench-results\n(178 rows, 174 of them evaluated submissions from users other than\ntest, last modified 25 October 2026, no licence stated). It\nkeeps the recorded objectives, feasibilities and scores with the\nsubmitters' public names and dates, and none of the submitted boundaries.\ndata/rescored.parquet holds our own evaluations; build_data.py writes\nboth files.\nCadena, S. A. et al. (2025). ConStellaration: A dataset of QI-like\nstellarator plasma boundaries and optimization benchmarks. NeurIPS 2025\nDatasets and Benchmarks Track. arXiv:2506.19583."},"type":"data"}]}],"metadata":{"marimo_version":"0.25.2","script_metadata_hash":null},"version":"1"},"appConfig":{"app_title":"The ConStellaration leaderboard, scored again","sql_output":"auto","width":"medium"},"layout":null,"renderedOn":"browser","formulas":{"||(|\\nabla r|^2||)":"∣∇r∣2"}},"files":[{"path":"build_data.py","size":7726,"url":"/sciencehub-team/constellaration/notebooks/leaderboard/raw/build_data.py"},{"path":"data/entries.parquet","size":517778,"url":"/sciencehub-team/constellaration/notebooks/leaderboard/raw/data/entries.parquet"},{"path":"data/rescored.parquet","size":267846,"url":"/sciencehub-team/constellaration/notebooks/leaderboard/raw/data/rescored.parquet"},{"path":"README.md","size":2001,"url":"/sciencehub-team/constellaration/notebooks/leaderboard/raw/README.md"},{"path":"scripts/rescore.py","size":11022,"url":"/sciencehub-team/constellaration/notebooks/leaderboard/raw/scripts/rescore.py"},{"path":"UPSTREAM-LICENSE","size":1109,"url":"/sciencehub-team/constellaration/notebooks/leaderboard/raw/UPSTREAM-LICENSE"}]}