EMA-GNN — Structural GNN for Crystal Stability Prediction

Bar chart comparing F1, MAE and R2 for six IS2E direct-prediction models on Matbench Discovery, with EMA-GNN highlighted
F1 and MAE across the six IS2E direct-prediction models on the Matbench Discovery leaderboard. EMA-GNN ranks 3rd by F1 (0.566, within 0.003 of the top two) and 1st by MAE (0.084 eV/atom) in this cohort.
Matbench Discovery — live leaderboard 6-seed ensemble · RTX 4070 Ti 3rd/6 IS2E · best MAE & R² in cohort

Built a GNoME-inspired structural GNN — independent hyperparameter search (converging to the paper's own configuration) plus an EMA-weighted ensemble — and submitted it to the community-maintained Matbench Discovery leaderboard via PR #387 (merged 2026-08-12), where it currently ranks 1st by MAE (0.084 eV/atom) and 3rd by F1 (0.566) among the six models sharing the same unrelaxed-structure prediction task, on the 256,963-structure WBM test set; along the way, corrected three scoring bugs that were silently distorting the metrics, and diagnosed the model's remaining error ceiling using a pretrained interatomic potential (MACE-MP-0) to pre-relax test inputs — which improved both energy accuracy (MAE 0.084 → 0.079 eV/atom) and correlation (R² 0.387 → 0.499) while leaving classification performance flat, isolating the gap to a training-data domain mismatch rather than a modeling shortfall.

GitHub

Parity plot of predicted vs true formation energy on the WBM test set, colored by stability classification outcome
Predicted vs. true formation energy per atom on the WBM test set, colored by TP/FP/TN/FN. Recall (0.818) substantially exceeds precision (0.433) — the model finds most genuinely stable materials but over-predicts stability, the expected signature of a direct energy predictor evaluated on unrelaxed inputs. F1 here is 0.559 (local scoring, vs. 0.566 on the leaderboard — 26 missing predictions counted differently, same model).
False positive rate rising with number of distinct elements in the structure
False-positive rate vs. number of elements in the structure. FP rate climbs from binaries to quinaries, consistent with the Materials Project 2018 training set being thinner on higher-order chemical systems — one of three diagnosed causes of the classification ceiling, alongside the relaxed/unrelaxed domain gap and confidently-wrong high-energy false positives.