Companion to Stepurko, N. & Laulainen, J. (2026), Out-of-training properties in frozen interatomic potential embeddings are decodable and ordered by physical scale — accepted at the AI4Sci and AI4Mat workshops, NeurIPS 2026. The paper is released in November 2026.

When is a probe score trustworthy under limited data?

An empirical study of how many labelled materials a probe on a frozen machine-learned interatomic potential needs before its score can be trusted: how far it sits below what more data would give, how much it moves with the test set, and how much of it the chemical formula alone explains.

All the code is public. The pipeline and result tables behind every number on this page, with the commands to re-measure them.Natalija-Stepurko/mlip-probe-data-requirements More of my work. Other studies and what I am working on now.natalija-stepurko.com

What would my score mean?

Pick a target and two sizes. Everything below is how these numbers were measured.

target

features

training materials 5,000

5020,000

test materials 200

2010,000

expected score0.823

shortfall0.061

spread p10–p900.128

total error0.098

how far to the ceiling

0.823 of 0.885 reached (R²)

what the error is made of

shortfall spread — shares of squared error, which is how the two combine

The score and the spread are both read off the measured curves. The total error combines them as √(shortfall² + sd²). The bar splits that total by squared contribution. One adjustment goes into the total: the spread was measured on a probe trained on 20,000 materials (6,427 for bulk modulus). A probe trained on less sits further from its ceiling and scatters more, so the spread is scaled up by (1−score)/(1−ceiling) first. This treats the 20,000-material probe as if it were at its ceiling, so the scaled spread errs high.

Three questions behind every small-data probe score

How much data do you need before a comparison between materials representations can be trusted? A score from a small dataset can mislead in three separate ways, and this page measures each one on frozen ORB-v3 and UMA-S embeddings over nine Materials Project properties.

Is the model trained on enough materials? A probe trained on few materials scores below what the same probe reaches with more data. That gap is the shortfall, and training size sets it.
Would another test set give the same score? A small test set gives a score that moves when the test set is redrawn. That movement is the spread, and test size sets it.
Does the embedding add anything beyond composition? A good score can come from the chemical formula alone. Composition-only Magpie features, run on the same draws, show how much of each score the embedding adds.

What the measurements show

Stability and sufficiency are different. Band gap gives a reproducible score from 500 training materials, but its score keeps rising until 7,000, where it comes within 0.05 of the full-data score.
There is no universal minimum dataset size. At 5,000 training materials the shortfall runs from 0.017 on formation energy to 0.128 on crystal system.
A strong representation can lose to composition when data is scarce. Composition alone is ahead of both embeddings on density up to 500 training materials, and neither embedding resolvably beats it on magnetic density.
The limiting error switches with dataset size. At 200 materials a 70/30 split is best; from 500 up the shortfall dominates and the best split moves toward training.
The usual 80/20 split is close to the best one here. From 500 materials up it is within 0.004 of the lowest total error at every size measured; at 200 materials 70/30 is better, and at 1,000, 2,000, 10,000 and 20,000 90/10 is very slightly better.

The two errors in one paragraph, and what this page measures

A probe trained on a few hundred materials gets a score. That score carries two errors that are usually conflated. The shortfall is how far it sits below what the same probe reaches on more data; it is set by training size and always points down. The spread is how far it moves when the split is redrawn; it is set by test size. This page measures the two separately, on frozen ORB-v3 and UMA-S embeddings over nine Materials Project properties, with composition-only Magpie features run on the same draws as a baseline, and uses the result to decide how much data is enough, where to put the split, whether cross-validation is worth it, and from how many materials an embedding's score says more than the formula.

What this page measures.

The size of each error, in R² or macro-F1, for frozen ORB-v3 and UMA-S embeddings on nine Materials Project targets: formation energy, energy above hull, band gap, density, magnetic density, bulk modulus, crystal system, metal / insulator and magnetic / not.
Whether cross-validation's advantage survives measurement at the few hundred to few thousand materials an experimental dataset actually has.
Whether the answer depends on which model produced the embedding.
How much of each score composition alone accounts for, and from how many materials the embedding's lead over it is resolvable.

Why two models. Shortfall varies five- to ninefold across targets at a fixed training size, so a threshold derived from one property would be wrong for the next, and one derived from one model might be wrong for another. ORB-v3 and UMA-S are architecturally opposite: a non-equivariant force field, and an equivariant multi-task model with a spherical-harmonic backbone. Agreement between them makes the thresholds a property of the task.

Why a composition baseline. Magpie (Ward et al., 2016) turns a formula into 132 statistics of its elements' properties and sees no structure. Run through the same probe on the same draws, it marks how much of each score the formula alone supplies, and the embedding's contribution is the difference.

Where the ceilings come from. Every score on this page is measured against the full-corpus score: what the same probe and embedding reach on an 80/20 split of all 154,875 materials (12,853 for bulk modulus). Those numbers are taken from Stepurko, N. & Laulainen, J. (2026), Out-of-training properties in frozen interatomic potential embeddings are decodable and ordered by physical scale, accepted at the AI4Sci and AI4Mat workshops, NeurIPS 2026. Magpie's ceilings are the same manuscript's composition-baseline control: the same probe on the Magpie features, full 80/20 split.

How it was measured

Everything here is measured on 154,875 Materials Project crystals, each represented by one frozen ORB-v3 or UMA-S embedding vector or by its 132 Magpie composition features, with XGBoost as the probe. The three matrices share one row order, so at every size the frozen test set and each training draw are the same materials for all three arms. Every score is compared against the full-corpus score for that target and feature set, from the manuscript cited at the top of the page.

Nine targets carry the curves. Six regressions: formation energy per atom, energy above hull, band gap, density, magnetic density, and bulk modulus — the last on the 12,853-material elastic subset, the rest on the full corpus. Three classifications: crystal system (seven classes), metal / insulator, and magnetic / not.

Training grid: 50, 100, 200, 500, 1,000 and 2,000 at 30 draws per size, extended to 5,000, 7,000, 10,000 and 20,000 at 10, 7, 7 and 6 draws. Test grid: 20 to 10,000 at 200 draws per size, plus the full 20,000-material test set once. Bulk modulus stops at 5,000 on both axes, with 12, 6, 3 and 1 training draws at 500, 1,000, 2,000 and 5,000.

What each number on this page is built from

quantity	built from
shortfall (training-size curves)	At each training size, many separate training sets drawn from the pool, no material in more than one of them. Each trains a fresh probe, scored on the same frozen 20,000 test materials. A point on the curve is the median over those training sets; its band is their 10th-to-90th-percentile range.
spread (test-size curves)	One probe, trained once on 20,000 materials. Its predictions on the frozen test set are re-scored on 200 random subsets of that set at each test size; only the test set changes.
ceiling (full-corpus score)	From Stepurko & Laulainen (2026), the paper this page accompanies: the same probe trained on 80 % of all 154,875 materials and scored on the other 20 %, one split per target and feature set.
split fraction	Computed from the two curves above; no probe is trained for it.
cross-validation	The only direct comparison of one split, five averaged splits and 5-fold cross-validation, on 20 separate small datasets per combination.

So a point on the main curves is neither one split nor five: it is many independent training sets, all scored on one large fixed test set. That is what lets the shortfall and the spread be measured separately.

Why so many draws. Each point needs to be precise enough to read a threshold of 0.05 off it. Taking 5 of the 30 draws at each training size up to 2,000, the median lands 0.005 from the 30-draw value on a typical point, and 0.015 or more one time in ten; at 50 training materials, 0.056 or more one time in ten. A spread estimated from 5 draws is off by about a quarter of its value.

Worked through step by step, on band gap — what each axis looks like in numbers

Training size — read the spread

Band gap, XGBoost, ORB backbone, 30 draws per size up to 2,000, then 10, 7, 7 and 6 at 5,000, 7,000, 10,000 and 20,000:

N train	median R²	worst draw	best draw
50	0.271	-0.036	0.401
100	0.438	0.307	0.519
200	0.549	0.446	0.602
500	0.669	0.621	0.684
1,000	0.728	0.718	0.749
2,000	0.778	0.738	0.782
5,000	0.823	0.818	0.833
7,000	0.836	0.830	0.839
10,000	0.848	0.844	0.854
20,000	0.862	0.860	0.871

At N = 50 the answer ranges from −0.04 to 0.40 on luck alone — a negative R² means worse than predicting the average. At N = 500 thirty draws agree to within 0.07.

Test size — do it again, separately

Train once, then score on 20, 50, 100 … randomly drawn test materials, 200 draws each, and on all 20,000 once. Same target and probe as above, with training now held fixed:

N test	median R²	worst draw	best draw
20	0.890	-0.277	0.988
50	0.883	0.562	0.958
100	0.878	0.585	0.941
200	0.876	0.530	0.927
500	0.872	0.687	0.916
1,000	0.870	0.764	0.904
2,000	0.872	0.794	0.894
5,000	0.865	0.830	0.888
10,000	0.864	0.844	0.878
20,000	0.864	0.864	0.864

The median barely moves — 0.890 to 0.864 — while the worst draw climbs from −0.28 to 0.84 at 10,000. At 20 test materials a single run could report anything from −0.28 to 0.99 for the same model.

Two thresholds, because two different things go wrong

The two axes fail differently, and the tables above show it. Too little training data pushes the score down — the median climbs 0.271 → 0.862. The number stays valid, just a lower bound.

Too little test data barely biases the score; it blurs it. The median holds near 0.87 while the worst draw moves by 1.1. So a training shortfall understates your embedding, and a test shortfall means the number simply will not reproduce — which is why they are reported as two numbers, a spread and a shortfall.

Known from the literature

Test-set size sets the variance of a score; training-set size sets its bias (Hastie, Tibshirani & Friedman, 2009, ch. 7).
At around 100 samples a cross-validated score carries error bars of roughly ±10 % (Varoquaux, 2018).
Learning curves rise monotonically with training size and rarely saturate at the sizes a study can afford (Cortes et al., 1994; Xu et al., 2023).
Cross-validation estimates how well the procedure does on average at that training size, not how well the one model you fitted will do (Bates, Hastie & Tibshirani, 2023).
The k fold scores from one cross-validation are not independent — every pair of folds shares most of its training data — so no formula can turn their scatter into a correct error bar in general; any such estimate is biased (Bengio & Grandvalet, 2004). That is why this page measures interval coverage directly.

1 · Is the model trained on enough materials?

Training size sets the shortfall: how far a score sits below the same probe's full-corpus score.

ORB-v3UMA-SMagpie, composition onlydashed: that arm's full-corpus scoreband: p10–p90 across draws

show

formation energy / atom

energy above hull

band gap

density

magnetic density

bulk modulus

crystal system

metal / insulator

magnetic / not

The point where a score stabilises and the point where it stops rising are different. Most targets' bands collapse between 100 and 500 training materials; energy above hull needs 2,000. But band gap under XGBoost has a spread below 0.05 from n = 500, where it scores 0.669, and keeps climbing to 0.778 at n = 2,000 against a full-corpus value of 0.885. A score can be perfectly reproducible and still well short of what the embedding can do.

Magpie, the composition-only arm, sits below both embeddings from 100 training materials on six targets. On density it is ahead of both up to 500 training materials and level at 1,000; on bulk modulus the three arms are not separable until about 1,000; on magnetic density it is level with or ahead of UMA-S and ahead of ORB-v3 at every size from 500. The composition section measures those differences on paired draws.

Full data — every training size, both embeddings and Magpie

target	features	n train	draws	median	worst draw	best draw	spread	full-corpus	shortfall
formation energy / atom	ORB	50	30	0.630	0.476	0.740	0.138	0.987	0.357
		100	30	0.739	0.648	0.801	0.093	0.987	0.248
		200	30	0.825	0.730	0.846	0.066	0.987	0.162
		500	30	0.890	0.868	0.909	0.023	0.987	0.097
		1,000	30	0.926	0.897	0.937	0.018	0.987	0.061
		2,000	30	0.948	0.942	0.953	0.008	0.987	0.039
		5,000	10	0.968	0.963	0.969	0.004	0.987	0.019
		7,000	7	0.972	0.971	0.973	0.002	0.987	0.015
		10,000	7	0.975	0.974	0.977	0.003	0.987	0.012
		20,000	6	0.980	0.979	0.981	0.001	0.987	0.007
formation energy / atom	UMA	50	30	0.716	0.638	0.827	0.130	0.990	0.274
		100	30	0.803	0.738	0.871	0.083	0.990	0.187
		200	30	0.874	0.800	0.896	0.055	0.990	0.116
		500	30	0.919	0.902	0.930	0.018	0.990	0.071
		1,000	30	0.946	0.931	0.953	0.011	0.990	0.044
		2,000	30	0.963	0.955	0.967	0.007	0.990	0.027
		5,000	10	0.976	0.975	0.978	0.002	0.990	0.014
		7,000	7	0.980	0.978	0.980	0.002	0.990	0.010
		10,000	7	0.982	0.981	0.983	0.002	0.990	0.008
		20,000	6	0.985	0.985	0.986	0.002	0.990	0.004
formation energy / atom	Magpie	50	30	0.643	0.430	0.712	0.175	0.890	0.248
		100	30	0.702	0.623	0.756	0.084	0.890	0.188
		200	30	0.747	0.691	0.773	0.051	0.890	0.143
		500	30	0.791	0.737	0.804	0.036	0.890	0.099
		1,000	30	0.813	0.800	0.821	0.014	0.890	0.077
		2,000	30	0.829	0.815	0.834	0.014	0.890	0.061
		5,000	10	0.845	0.842	0.850	0.006	0.890	0.045
		7,000	7	0.853	0.849	0.858	0.007	0.890	0.038
		10,000	7	0.858	0.852	0.860	0.007	0.890	0.033
		20,000	6	0.866	0.865	0.868	0.003	0.890	0.024
energy above hull	ORB	50	30	0.136	-1.262	0.419	0.855	0.945	0.810
		100	30	0.385	0.084	0.593	0.345	0.945	0.560
		200	30	0.554	-0.322	0.667	0.525	0.945	0.391
		500	30	0.675	0.550	0.751	0.111	0.945	0.270
		1,000	30	0.763	0.643	0.807	0.086	0.945	0.182
		2,000	30	0.818	0.775	0.846	0.043	0.945	0.127
		5,000	10	0.874	0.863	0.881	0.014	0.945	0.072
		7,000	7	0.888	0.878	0.892	0.013	0.945	0.057
		10,000	7	0.902	0.895	0.903	0.007	0.945	0.043
		20,000	6	0.918	0.913	0.922	0.007	0.945	0.027
energy above hull	UMA	50	30	0.194	-0.698	0.575	0.714	0.959	0.765
		100	30	0.500	-0.370	0.718	0.566	0.959	0.459
		200	30	0.612	0.347	0.738	0.283	0.959	0.347
		500	30	0.765	0.579	0.814	0.106	0.959	0.194
		1,000	30	0.834	0.717	0.866	0.078	0.959	0.125
		2,000	30	0.871	0.843	0.892	0.032	0.959	0.088
		5,000	10	0.916	0.906	0.922	0.013	0.959	0.043
		7,000	7	0.925	0.914	0.929	0.014	0.959	0.034
		10,000	7	0.934	0.930	0.936	0.007	0.959	0.025
		20,000	6	0.945	0.943	0.947	0.004	0.959	0.014
energy above hull	Magpie	50	30	-0.351	-5.579	-0.041	3.014	0.296	0.647
		100	30	-0.323	-1.918	-0.034	1.185	0.296	0.619
		200	30	-0.158	-2.240	-0.000	1.117	0.296	0.454
		500	30	-0.121	-0.631	-0.010	0.372	0.296	0.417
		1,000	30	-0.021	-0.150	0.046	0.146	0.296	0.317
		2,000	30	0.045	-0.021	0.106	0.077	0.296	0.251
		5,000	10	0.137	0.067	0.166	0.075	0.296	0.159
		7,000	7	0.134	0.132	0.188	0.056	0.296	0.162
		10,000	7	0.180	0.154	0.201	0.039	0.296	0.116
		20,000	6	0.232	0.221	0.239	0.017	0.296	0.064
band gap	ORB	50	30	0.271	-0.036	0.401	0.263	0.885	0.614
		100	30	0.438	0.307	0.519	0.133	0.885	0.447
		200	30	0.549	0.446	0.602	0.089	0.885	0.336
		500	30	0.669	0.621	0.684	0.042	0.885	0.216
		1,000	30	0.728	0.718	0.749	0.018	0.885	0.157
		2,000	30	0.778	0.738	0.782	0.021	0.885	0.107
		5,000	10	0.823	0.818	0.833	0.010	0.885	0.061
		7,000	7	0.836	0.830	0.839	0.008	0.885	0.049
		10,000	7	0.848	0.844	0.854	0.009	0.885	0.037
		20,000	6	0.862	0.860	0.871	0.011	0.885	0.022
band gap	UMA	50	30	0.403	0.244	0.548	0.170	0.906	0.503
		100	30	0.560	0.417	0.612	0.112	0.906	0.346
		200	30	0.632	0.559	0.670	0.067	0.906	0.274
		500	30	0.720	0.693	0.734	0.030	0.906	0.185
		1,000	30	0.768	0.754	0.783	0.019	0.906	0.138
		2,000	30	0.810	0.790	0.818	0.014	0.906	0.096
		5,000	10	0.849	0.845	0.856	0.008	0.906	0.057
		7,000	7	0.858	0.856	0.864	0.008	0.906	0.047
		10,000	7	0.869	0.866	0.879	0.011	0.906	0.037
		20,000	6	0.883	0.881	0.890	0.010	0.906	0.023
band gap	Magpie	50	30	0.198	-0.099	0.291	0.225	0.698	0.500
		100	30	0.260	0.027	0.324	0.159	0.698	0.438
		200	30	0.325	0.273	0.361	0.067	0.698	0.373
		500	30	0.389	0.333	0.421	0.046	0.698	0.309
		1,000	30	0.450	0.418	0.478	0.038	0.698	0.248
		2,000	30	0.505	0.483	0.530	0.027	0.698	0.193
		5,000	10	0.576	0.563	0.588	0.017	0.698	0.122
		7,000	7	0.599	0.585	0.603	0.017	0.698	0.099
		10,000	7	0.621	0.608	0.628	0.019	0.698	0.077
		20,000	6	0.655	0.653	0.664	0.011	0.698	0.043
density	ORB	50	30	0.308	0.190	0.448	0.169	0.947	0.638
		100	30	0.460	0.259	0.545	0.169	0.947	0.487
		200	30	0.605	0.516	0.659	0.092	0.947	0.341
		500	30	0.707	0.656	0.743	0.055	0.947	0.240
		1,000	30	0.775	0.748	0.807	0.035	0.947	0.171
		2,000	30	0.829	0.814	0.846	0.020	0.947	0.118
		5,000	10	0.884	0.879	0.890	0.009	0.947	0.062
		7,000	7	0.897	0.895	0.899	0.003	0.947	0.050
		10,000	7	0.911	0.909	0.913	0.003	0.947	0.036
		20,000	6	0.927	0.925	0.930	0.004	0.947	0.020
density	UMA	50	30	0.371	-0.032	0.449	0.289	0.940	0.569
		100	30	0.483	0.336	0.567	0.155	0.940	0.457
		200	30	0.602	0.564	0.656	0.067	0.940	0.338
		500	30	0.725	0.671	0.755	0.049	0.940	0.215
		1,000	30	0.782	0.762	0.806	0.027	0.940	0.158
		2,000	30	0.828	0.811	0.843	0.018	0.940	0.112
		5,000	10	0.877	0.871	0.880	0.007	0.940	0.063
		7,000	7	0.890	0.886	0.893	0.005	0.940	0.050
		10,000	7	0.903	0.900	0.905	0.004	0.940	0.037
		20,000	6	0.918	0.918	0.922	0.004	0.940	0.022
density	Magpie	50	30	0.588	0.329	0.669	0.252	0.888	0.300
		100	30	0.647	0.402	0.705	0.172	0.888	0.241
		200	30	0.721	0.636	0.743	0.073	0.888	0.167
		500	30	0.763	0.719	0.784	0.046	0.888	0.126
		1,000	30	0.794	0.761	0.809	0.031	0.888	0.095
		2,000	30	0.815	0.788	0.829	0.024	0.888	0.074
		5,000	10	0.835	0.828	0.839	0.009	0.888	0.053
		7,000	7	0.843	0.837	0.848	0.012	0.888	0.046
		10,000	7	0.852	0.844	0.859	0.012	0.888	0.037
		20,000	6	0.860	0.854	0.866	0.012	0.888	0.029
magnetic density	ORB	50	30	0.059	-0.148	0.249	0.235	0.813	0.753
		100	30	0.206	-0.041	0.349	0.268	0.813	0.606
		200	30	0.350	0.187	0.441	0.165	0.813	0.462
		500	30	0.507	0.441	0.555	0.059	0.813	0.306
		1,000	30	0.578	0.532	0.603	0.043	0.813	0.235
		2,000	30	0.639	0.618	0.672	0.031	0.813	0.173
		5,000	10	0.699	0.689	0.707	0.017	0.813	0.114
		7,000	7	0.722	0.708	0.726	0.019	0.813	0.091
		10,000	7	0.734	0.732	0.742	0.011	0.813	0.079
		20,000	6	0.763	0.759	0.768	0.011	0.813	0.050
magnetic density	UMA	50	30	0.235	-0.029	0.376	0.265	0.831	0.596
		100	30	0.369	0.151	0.428	0.169	0.831	0.463
		200	30	0.459	0.370	0.522	0.091	0.831	0.372
		500	30	0.563	0.532	0.609	0.049	0.831	0.268
		1,000	30	0.620	0.594	0.656	0.037	0.831	0.212
		2,000	30	0.676	0.657	0.695	0.024	0.831	0.155
		5,000	10	0.729	0.720	0.736	0.013	0.831	0.103
		7,000	7	0.736	0.731	0.748	0.014	0.831	0.095
		10,000	7	0.758	0.751	0.767	0.013	0.831	0.073
		20,000	6	0.782	0.776	0.787	0.011	0.831	0.050
magnetic density	Magpie	50	30	0.119	-0.424	0.344	0.420	0.816	0.697
		100	30	0.297	-0.036	0.410	0.295	0.816	0.519
		200	30	0.428	0.323	0.548	0.160	0.816	0.388
		500	30	0.573	0.400	0.616	0.116	0.816	0.243
		1,000	30	0.638	0.579	0.675	0.064	0.816	0.178
		2,000	30	0.690	0.667	0.711	0.027	0.816	0.127
		5,000	10	0.737	0.727	0.754	0.024	0.816	0.080
		7,000	7	0.750	0.737	0.758	0.016	0.816	0.067
		10,000	7	0.761	0.755	0.771	0.012	0.816	0.055
		20,000	6	0.783	0.774	0.788	0.012	0.816	0.034
bulk modulus	ORB	50	30	0.520	0.334	0.607	0.162	0.936	0.416
		100	30	0.613	0.450	0.680	0.120	0.936	0.322
		200	30	0.701	0.648	0.749	0.071	0.936	0.234
		500	12	0.798	0.775	0.823	0.036	0.936	0.137
		1,000	6	0.857	0.847	0.863	0.015	0.936	0.079
		2,000	3	0.893	0.886	0.895	0.012	0.936	0.043
		5,000	1	0.919	0.919	0.919	—	0.936	0.017
bulk modulus	UMA	50	30	0.577	0.335	0.665	0.185	0.939	0.362
		100	30	0.665	0.450	0.737	0.133	0.939	0.274
		200	30	0.744	0.690	0.792	0.065	0.939	0.195
		500	12	0.826	0.807	0.840	0.028	0.939	0.113
		1,000	6	0.868	0.860	0.874	0.013	0.939	0.071
		2,000	3	0.905	0.900	0.906	0.008	0.939	0.034
		5,000	1	0.926	0.926	0.926	—	0.939	0.013
bulk modulus	Magpie	50	30	0.575	0.231	0.684	0.254	0.882	0.307
		100	30	0.652	0.472	0.713	0.134	0.882	0.230
		200	30	0.726	0.662	0.786	0.064	0.882	0.156
		500	12	0.798	0.754	0.809	0.041	0.882	0.084
		1,000	6	0.823	0.814	0.828	0.012	0.882	0.059
		2,000	3	0.850	0.844	0.851	0.009	0.882	0.032
		5,000	1	0.870	0.870	0.870	—	0.882	0.013
crystal system	ORB	50	30	0.185	0.142	0.227	0.050	0.604	0.419
		100	30	0.223	0.200	0.272	0.051	0.604	0.381
		200	30	0.272	0.233	0.316	0.052	0.604	0.332
		500	30	0.331	0.301	0.351	0.032	0.604	0.273
		1,000	30	0.372	0.354	0.394	0.027	0.604	0.232
		2,000	30	0.412	0.397	0.428	0.018	0.604	0.192
		5,000	10	0.472	0.457	0.476	0.015	0.604	0.132
		7,000	7	0.492	0.487	0.494	0.007	0.604	0.112
		10,000	7	0.511	0.504	0.519	0.015	0.604	0.093
		20,000	6	0.554	0.553	0.557	0.004	0.604	0.050
crystal system	UMA	50	30	0.251	0.206	0.288	0.048	0.663	0.412
		100	30	0.293	0.264	0.350	0.048	0.663	0.370
		200	30	0.337	0.300	0.366	0.041	0.663	0.327
		500	30	0.389	0.376	0.419	0.026	0.663	0.274
		1,000	30	0.440	0.421	0.450	0.022	0.663	0.224
		2,000	30	0.481	0.468	0.488	0.014	0.663	0.183
		5,000	10	0.539	0.527	0.546	0.013	0.663	0.124
		7,000	7	0.556	0.548	0.560	0.010	0.663	0.108
		10,000	7	0.577	0.574	0.580	0.005	0.663	0.086
		20,000	6	0.617	0.612	0.619	0.007	0.663	0.046
crystal system	Magpie	50	30	0.203	0.170	0.226	0.036	0.520	0.317
		100	30	0.222	0.198	0.264	0.045	0.520	0.298
		200	30	0.259	0.232	0.289	0.032	0.520	0.262
		500	30	0.293	0.262	0.310	0.025	0.520	0.227
		1,000	30	0.327	0.311	0.339	0.019	0.520	0.193
		2,000	30	0.357	0.347	0.367	0.014	0.520	0.163
		5,000	10	0.407	0.399	0.412	0.009	0.520	0.114
		7,000	7	0.424	0.419	0.431	0.012	0.520	0.097
		10,000	7	0.438	0.435	0.444	0.008	0.520	0.082
		20,000	6	0.474	0.471	0.476	0.005	0.520	0.046
metal / insulator	ORB	50	30	0.727	0.673	0.769	0.073	0.908	0.181
		100	30	0.787	0.748	0.817	0.039	0.908	0.121
		200	30	0.816	0.797	0.832	0.022	0.908	0.092
		500	30	0.849	0.835	0.860	0.016	0.908	0.059
		1,000	30	0.863	0.852	0.870	0.009	0.908	0.045
		2,000	30	0.872	0.867	0.878	0.007	0.908	0.036
		5,000	10	0.885	0.883	0.888	0.004	0.908	0.023
		7,000	7	0.887	0.885	0.890	0.005	0.908	0.021
		10,000	7	0.892	0.890	0.893	0.002	0.908	0.016
		20,000	6	0.897	0.895	0.899	0.004	0.908	0.011
metal / insulator	UMA	50	30	0.792	0.735	0.833	0.063	0.914	0.122
		100	30	0.819	0.785	0.843	0.037	0.914	0.095
		200	30	0.840	0.825	0.854	0.016	0.914	0.073
		500	30	0.858	0.848	0.868	0.013	0.914	0.055
		1,000	30	0.870	0.862	0.877	0.009	0.914	0.044
		2,000	30	0.880	0.874	0.885	0.006	0.914	0.034
		5,000	10	0.891	0.887	0.895	0.006	0.914	0.023
		7,000	7	0.895	0.891	0.897	0.005	0.914	0.019
		10,000	7	0.898	0.897	0.900	0.003	0.914	0.015
		20,000	6	0.905	0.904	0.908	0.004	0.914	0.008
metal / insulator	Magpie	50	30	0.725	0.675	0.754	0.050	0.845	0.120
		100	30	0.743	0.698	0.767	0.042	0.845	0.102
		200	30	0.757	0.729	0.770	0.030	0.845	0.088
		500	30	0.771	0.760	0.786	0.014	0.845	0.075
		1,000	30	0.784	0.776	0.794	0.011	0.845	0.061
		2,000	30	0.796	0.784	0.801	0.010	0.845	0.049
		5,000	10	0.815	0.810	0.819	0.009	0.845	0.031
		7,000	7	0.820	0.818	0.824	0.005	0.845	0.025
		10,000	7	0.825	0.823	0.829	0.005	0.845	0.021
		20,000	6	0.835	0.834	0.836	0.002	0.845	0.010
magnetic / not	ORB	50	30	0.647	0.579	0.701	0.082	0.888	0.241
		100	30	0.698	0.654	0.737	0.049	0.888	0.190
		200	30	0.741	0.713	0.763	0.037	0.888	0.147
		500	30	0.789	0.768	0.802	0.020	0.888	0.100
		1,000	30	0.812	0.803	0.825	0.012	0.888	0.076
		2,000	30	0.833	0.825	0.839	0.009	0.888	0.055
		5,000	10	0.854	0.851	0.855	0.003	0.888	0.034
		7,000	7	0.862	0.859	0.863	0.003	0.888	0.026
		10,000	7	0.867	0.863	0.869	0.005	0.888	0.021
		20,000	6	0.877	0.875	0.878	0.002	0.888	0.011
magnetic / not	UMA	50	30	0.701	0.584	0.754	0.096	0.903	0.202
		100	30	0.743	0.692	0.782	0.059	0.903	0.161
		200	30	0.785	0.756	0.808	0.032	0.903	0.118
		500	30	0.828	0.814	0.841	0.021	0.903	0.075
		1,000	30	0.846	0.838	0.856	0.012	0.903	0.057
		2,000	30	0.862	0.856	0.867	0.007	0.903	0.041
		5,000	10	0.877	0.875	0.879	0.004	0.903	0.026
		7,000	7	0.883	0.878	0.884	0.005	0.903	0.021
		10,000	7	0.887	0.885	0.890	0.003	0.903	0.016
		20,000	6	0.894	0.894	0.895	0.001	0.903	0.009
magnetic / not	Magpie	50	30	0.651	0.514	0.705	0.095	0.860	0.209
		100	30	0.684	0.632	0.723	0.055	0.860	0.176
		200	30	0.721	0.691	0.739	0.033	0.860	0.138
		500	30	0.756	0.741	0.767	0.019	0.860	0.104
		1,000	30	0.775	0.763	0.787	0.015	0.860	0.085
		2,000	30	0.797	0.789	0.808	0.009	0.860	0.063
		5,000	10	0.820	0.815	0.828	0.008	0.860	0.040
		7,000	7	0.828	0.827	0.830	0.002	0.860	0.032
		10,000	7	0.835	0.831	0.836	0.005	0.860	0.025
		20,000	6	0.844	0.842	0.847	0.005	0.860	0.015

One number of materials cannot describe nine targets

Shortfall at 5,000 training materials, per target, for both embeddings and for Magpie. Each bar is measured against that arm's own full-corpus score.

ORB-v3UMA-SMagpie, composition only

The same number of materials is 0.017 short on one target and 0.128 short on another. At 5,000 training materials formation energy is essentially at its ceiling, while crystal system is still 0.13 of macro-F1 below it — a span of about 7.7×. ORB-v3 and UMA-S agree to within 0.029 on every target at this size, and within 0.01 on seven of the nine. The largest gaps are on energy above hull and magnetic density, where ORB-v3 sits 0.03 and 0.01 further below its ceiling than UMA-S does. Both embeddings agree on which targets need more data. Magpie shows a narrower span, 5.2×, from 0.031 on metal / insulator to 0.159 on energy above hull. Both spans leave out bulk modulus, which has one draw at this size; with it, Magpie's span is 12.6× and the embeddings' 8.6×. Magpie sits further below its own ceiling than the embeddings do on five targets and closer to it on density, magnetic density, bulk modulus and crystal system.

training materials	5,000	7,000	10,000	20,000
span, hardest / easiest target	7.7×	8.6×	8.7×	8.8×

At one training size the targets sit far apart. At 5,000 the shortfall is 0.017 on formation energy and 0.128 on crystal system — about 7.7× larger — and adding data widens the span between hardest and easiest target, to 8.8× at 20,000. The shortfall is therefore reported per target alongside the score.

Training floor: 5,000 materials

The shortfall is how far a score sits below the same probe's full-corpus score. It falls at every step of the grid:

training materials	median shortfall	range across targets	materials at 80/20
50	0.487	0.151–0.787	62
100	0.386	0.108–0.535	125
200	0.317	0.083–0.417	250
500	0.214	0.057–0.287	625
1,000	0.150	0.045–0.228	1,250
2,000	0.105	0.033–0.187	2,500
5,000	0.058	0.017–0.128	6,250
7,000	0.047	0.013–0.110	8,750
10,000	0.035	0.010–0.090	12,500
20,000	0.021	0.006–0.050	25,000

Median and range over the eight targets measured at every size; bulk modulus is left out because its corpus stops the curve at 5,000. The highlighted row is the floor.

There is no knee. Up to 2,000 the median target is more than 0.10 short, so a score there is mainly a lower bound. From 5,000 to 20,000 each step buys 0.011–0.014 of shortfall and costs progressively more data — 40 % more materials for the first step, then 43 %, then 100 %. The floor sits at 5,000, where the median target is 0.058 short; it is set by how much data it is reasonable to demand.

ORB-v3 and UMA-S agree to within 0.029 at every size, and within 0.01 on all targets except energy above hull, where the gap is 0.013–0.029, and magnetic density at 5,000, where it is 0.011; both rank the targets in nearly the same order (rank correlation 0.88–0.93), so which targets need more data is a property of the task and not of either one.

Raising the floor does not fix the target that needs fixing — shortfall per target at 5,000 to 20,000

target	5,000	7,000	10,000	20,000
bulk modulus	0.015	—	—	—
formation energy / atom	0.017	0.013	0.010	0.006
metal / insulator	0.023	0.020	0.016	0.010
magnetic / not	0.030	0.024	0.019	0.010
energy above hull	0.057	0.046	0.034	0.021
band gap	0.059	0.048	0.037	0.023
density	0.063	0.050	0.036	0.021
magnetic density	0.108	0.093	0.076	0.050
crystal system	0.128	0.110	0.090	0.048
span, hardest / easiest	7.7×	8.6×	8.7×	8.8×

Bulk modulus stops at 5,000 because its corpus holds 12,853 materials, so a training set of 7,000 cannot be drawn alongside a disjoint reference set. It is left out of the pooled mean above for the same reason: including it would change what the column averages partway along the row.

The span across targets widens with training data. Crystal system at 20,000 is still 0.048 short — worse than formation energy is at 5,000, on four times the data. A floor set high enough to bring crystal system within 0.05 of its ceiling would be 20,000, the largest size measured, and still leave the easy targets paying for it.

2 · Would another test set give the same score?

Test size sets the spread: how far a score moves when the test set is redrawn.

The probe is fitted once on 20,000 materials (6,427 on bulk modulus); only the materials it is scored on change.

ORB-v3UMA-SMagpie, composition onlydashed: that arm's full-corpus scoreband: p10–p90 across draws

formation energy / atom

energy above hull

band gap

density

magnetic density

bulk modulus

crystal system

metal / insulator

magnetic / not

Test size moves the spread and little else. Across the grid the p10–p90 spread falls from 0.239 at 20 test materials to 0.010 at 5,000 — a factor of 24 — while from 200 test materials up the median moves by no more than 0.021. At 200 test materials a re-split moves a typical score by about ±0.040; at 500, ±0.024; at 1,000, ±0.016. This is the axis that decides whether a number reproduces, and the spread at a given test size depends on the property being predicted: at 200 test materials ORB-v3's spread runs from 0.018 on formation energy to 0.189 on magnetic density.

It is not independent of how well the probe scores. Magpie's spread on the regression targets is wider than the embeddings' at every test size up to 500 — on formation energy 0.127 against 0.018 for ORB-v3 at 200 test materials — and on formation energy the ratio tracks 1 − R²: a probe further from a perfect score scatters more. The calculator applies that rescaling before combining spread with shortfall.

Full data — every test size, both embeddings and Magpie

Every N on the grid, both embeddings and the composition baseline. The highlighted row is where the spread first falls below 0.05 and stays there. Worst and best draw are the extremes across that row’s draws. The model is fitted once on 20,000 materials (6,427 for bulk modulus) and only its predictions are resampled, so the shortfall moves within a curve only with the median score, by at most 0.021 from 200 test materials up.

target	features	n test	draws	median	worst draw	best draw	spread	full-corpus	shortfall
formation energy / atom	ORB	20	200	0.984	0.854	0.997	0.040	0.987	0.003
		50	200	0.985	0.906	0.996	0.037	0.987	0.002
		100	200	0.984	0.925	0.993	0.025	0.987	0.003
		200	200	0.983	0.947	0.991	0.018	0.987	0.004
		500	200	0.982	0.952	0.990	0.014	0.987	0.005
		1,000	200	0.981	0.960	0.987	0.012	0.987	0.006
		2,000	200	0.981	0.968	0.985	0.007	0.987	0.006
		5,000	200	0.980	0.975	0.984	0.005	0.987	0.007
		10,000	200	0.980	0.976	0.983	0.003	0.987	0.007
		20,000	1	0.980	0.980	0.980	—	0.987	0.007
formation energy / atom	UMA	20	200	0.989	0.931	0.998	0.026	0.990	0.001
		50	200	0.988	0.952	0.997	0.018	0.990	0.001
		100	200	0.988	0.953	0.995	0.016	0.990	0.002
		200	200	0.988	0.956	0.994	0.012	0.990	0.002
		500	200	0.987	0.953	0.992	0.011	0.990	0.003
		1,000	200	0.987	0.967	0.991	0.010	0.990	0.003
		2,000	200	0.987	0.976	0.990	0.006	0.990	0.003
		5,000	200	0.986	0.981	0.989	0.004	0.990	0.003
		10,000	200	0.986	0.983	0.988	0.003	0.990	0.004
		20,000	1	0.986	0.986	0.986	—	0.990	0.004
formation energy / atom	Magpie	20	200	0.935	0.084	0.991	0.420	0.890	-0.045
		50	200	0.907	0.351	0.979	0.242	0.890	-0.017
		100	200	0.894	0.642	0.972	0.167	0.890	-0.004
		200	200	0.878	0.710	0.956	0.127	0.890	0.012
		500	200	0.876	0.685	0.942	0.092	0.890	0.015
		1,000	200	0.871	0.778	0.927	0.069	0.890	0.019
		2,000	200	0.867	0.803	0.908	0.049	0.890	0.023
		5,000	200	0.866	0.830	0.899	0.028	0.890	0.025
		10,000	200	0.866	0.848	0.883	0.015	0.890	0.024
		20,000	1	0.866	0.866	0.866	—	0.890	0.024
energy above hull	ORB	20	200	0.816	-2.864	0.998	1.019	0.945	0.129
		50	200	0.903	-0.847	0.993	0.506	0.945	0.042
		100	200	0.903	0.341	0.988	0.259	0.945	0.042
		200	200	0.917	0.641	0.982	0.158	0.945	0.028
		500	200	0.920	0.718	0.965	0.091	0.945	0.025
		1,000	200	0.919	0.858	0.960	0.049	0.945	0.026
		2,000	200	0.920	0.878	0.944	0.036	0.945	0.025
		5,000	200	0.919	0.895	0.940	0.022	0.945	0.026
		10,000	200	0.919	0.905	0.931	0.012	0.945	0.026
		20,000	1	0.920	0.920	0.920	—	0.945	0.025
energy above hull	UMA	20	200	0.868	-3.158	0.998	0.956	0.959	0.091
		50	200	0.939	-0.094	0.995	0.376	0.959	0.020
		100	200	0.940	0.546	0.986	0.187	0.959	0.019
		200	200	0.941	0.682	0.987	0.116	0.959	0.019
		500	200	0.944	0.803	0.973	0.069	0.959	0.015
		1,000	200	0.944	0.897	0.972	0.037	0.959	0.015
		2,000	200	0.943	0.914	0.963	0.026	0.959	0.016
		5,000	200	0.943	0.926	0.957	0.016	0.959	0.016
		10,000	200	0.944	0.933	0.953	0.009	0.959	0.015
		20,000	1	0.944	0.944	0.944	—	0.959	0.015
energy above hull	Magpie	20	200	0.060	-65.350	0.866	16.326	0.296	0.236
		50	200	0.147	-4.953	0.854	1.934	0.296	0.150
		100	200	0.144	-2.076	0.712	0.968	0.296	0.152
		200	200	0.218	-0.391	0.739	0.484	0.296	0.079
		500	200	0.202	-0.119	0.560	0.349	0.296	0.094
		1,000	200	0.208	-0.010	0.467	0.210	0.296	0.088
		2,000	200	0.216	0.053	0.370	0.153	0.296	0.080
		5,000	200	0.219	0.125	0.322	0.080	0.296	0.077
		10,000	200	0.220	0.165	0.287	0.048	0.296	0.076
		20,000	1	0.220	0.220	0.220	—	0.296	0.076
band gap	ORB	20	200	0.890	-0.277	0.988	0.394	0.885	-0.005
		50	200	0.883	0.562	0.958	0.162	0.885	0.002
		100	200	0.878	0.585	0.941	0.146	0.885	0.007
		200	200	0.876	0.530	0.927	0.128	0.885	0.009
		500	200	0.872	0.687	0.916	0.081	0.885	0.013
		1,000	200	0.870	0.764	0.904	0.076	0.885	0.014
		2,000	200	0.872	0.794	0.894	0.052	0.885	0.013
		5,000	200	0.865	0.830	0.888	0.031	0.885	0.020
		10,000	200	0.864	0.844	0.878	0.017	0.885	0.021
		20,000	1	0.864	0.864	0.864	—	0.885	0.021
band gap	UMA	20	200	0.919	0.020	0.991	0.315	0.906	-0.013
		50	200	0.902	0.610	0.966	0.158	0.906	0.004
		100	200	0.899	0.646	0.961	0.127	0.906	0.007
		200	200	0.899	0.517	0.936	0.124	0.906	0.007
		500	200	0.893	0.690	0.934	0.086	0.906	0.013
		1,000	200	0.891	0.786	0.933	0.081	0.906	0.015
		2,000	200	0.893	0.805	0.912	0.056	0.906	0.013
		5,000	200	0.887	0.847	0.907	0.033	0.906	0.019
		10,000	200	0.883	0.866	0.899	0.018	0.906	0.023
		20,000	1	0.883	0.883	0.883	—	0.906	0.023
band gap	Magpie	20	200	0.669	-1.860	0.945	0.865	0.698	0.029
		50	200	0.644	0.195	0.867	0.308	0.698	0.054
		100	200	0.653	0.306	0.829	0.230	0.698	0.045
		200	200	0.657	0.379	0.787	0.173	0.698	0.041
		500	200	0.659	0.522	0.749	0.102	0.698	0.039
		1,000	200	0.653	0.567	0.724	0.081	0.698	0.045
		2,000	200	0.656	0.605	0.704	0.054	0.698	0.042
		5,000	200	0.651	0.619	0.683	0.028	0.698	0.047
		10,000	200	0.653	0.633	0.673	0.018	0.698	0.045
		20,000	1	0.653	0.653	0.653	—	0.698	0.045
density	ORB	20	200	0.931	0.699	0.987	0.130	0.947	0.016
		50	200	0.932	0.711	0.982	0.079	0.947	0.015
		100	200	0.929	0.810	0.965	0.056	0.947	0.017
		200	200	0.930	0.879	0.960	0.037	0.947	0.016
		500	200	0.928	0.896	0.954	0.026	0.947	0.018
		1,000	200	0.929	0.904	0.944	0.018	0.947	0.018
		2,000	200	0.930	0.915	0.941	0.013	0.947	0.017
		5,000	200	0.928	0.918	0.936	0.007	0.947	0.018
		10,000	200	0.929	0.925	0.934	0.004	0.947	0.018
		20,000	1	0.929	0.929	0.929	—	0.947	0.018
density	UMA	20	200	0.919	0.627	0.990	0.157	0.940	0.021
		50	200	0.930	0.731	0.977	0.096	0.940	0.010
		100	200	0.928	0.830	0.969	0.061	0.940	0.012
		200	200	0.924	0.860	0.961	0.046	0.940	0.016
		500	200	0.922	0.896	0.945	0.026	0.940	0.018
		1,000	200	0.922	0.893	0.938	0.020	0.940	0.018
		2,000	200	0.922	0.908	0.936	0.014	0.940	0.018
		5,000	200	0.922	0.913	0.930	0.007	0.940	0.018
		10,000	200	0.922	0.917	0.927	0.005	0.940	0.018
		20,000	1	0.922	0.922	0.922	—	0.940	0.018
density	Magpie	20	200	0.940	-1.297	0.992	0.743	0.888	-0.051
		50	200	0.895	0.066	0.980	0.353	0.888	-0.007
		100	200	0.872	0.329	0.982	0.269	0.888	0.017
		200	200	0.876	0.615	0.968	0.152	0.888	0.012
		500	200	0.868	0.738	0.943	0.097	0.888	0.021
		1,000	200	0.860	0.773	0.928	0.076	0.888	0.028
		2,000	200	0.859	0.815	0.916	0.047	0.888	0.030
		5,000	200	0.858	0.827	0.896	0.030	0.888	0.030
		10,000	200	0.859	0.844	0.877	0.017	0.888	0.030
		20,000	1	0.859	0.859	0.859	—	0.888	0.029
magnetic density	ORB	20	200	0.797	-20.586	0.980	4.177	0.813	0.016
		50	200	0.809	-0.391	0.964	0.526	0.813	0.004
		100	200	0.770	0.325	0.909	0.295	0.813	0.042
		200	200	0.768	0.390	0.890	0.189	0.813	0.044
		500	200	0.766	0.615	0.865	0.117	0.813	0.047
		1,000	200	0.767	0.658	0.840	0.080	0.813	0.045
		2,000	200	0.763	0.700	0.822	0.059	0.813	0.049
		5,000	200	0.764	0.729	0.792	0.032	0.813	0.049
		10,000	200	0.764	0.742	0.782	0.019	0.813	0.049
		20,000	1	0.763	0.763	0.763	—	0.813	0.049
magnetic density	UMA	20	200	0.804	-20.237	0.976	4.103	0.831	0.028
		50	200	0.802	-0.483	0.975	0.487	0.831	0.030
		100	200	0.775	0.316	0.921	0.305	0.831	0.057
		200	200	0.783	0.379	0.911	0.190	0.831	0.049
		500	200	0.784	0.606	0.870	0.118	0.831	0.048
		1,000	200	0.780	0.683	0.845	0.081	0.831	0.052
		2,000	200	0.778	0.704	0.830	0.059	0.831	0.054
		5,000	200	0.776	0.739	0.808	0.035	0.831	0.055
		10,000	200	0.776	0.758	0.794	0.019	0.831	0.055
		20,000	1	0.777	0.777	0.777	—	0.831	0.055
magnetic density	Magpie	20	200	0.816	-19.436	0.989	5.382	0.816	0.001
		50	200	0.813	-1.171	0.979	0.651	0.816	0.003
		100	200	0.785	0.218	0.929	0.349	0.816	0.031
		200	200	0.800	0.495	0.923	0.199	0.816	0.017
		500	200	0.783	0.551	0.879	0.135	0.816	0.034
		1,000	200	0.781	0.664	0.867	0.087	0.816	0.035
		2,000	200	0.779	0.688	0.832	0.067	0.816	0.037
		5,000	200	0.779	0.737	0.812	0.035	0.816	0.038
		10,000	200	0.778	0.755	0.798	0.021	0.816	0.038
		20,000	1	0.778	0.778	0.778	—	0.816	0.038
bulk modulus	ORB	20	200	0.950	0.266	0.991	0.239	0.936	-0.014
		50	200	0.938	0.643	0.983	0.120	0.936	-0.003
		100	200	0.937	0.740	0.979	0.112	0.936	-0.002
		200	200	0.934	0.801	0.970	0.081	0.936	0.002
		500	200	0.930	0.843	0.955	0.048	0.936	0.005
		1,000	200	0.927	0.891	0.954	0.033	0.936	0.009
		2,000	200	0.925	0.904	0.944	0.021	0.936	0.011
		5,000	200	0.925	0.920	0.934	0.008	0.936	0.011
bulk modulus	UMA	20	200	0.949	0.015	0.990	0.261	0.939	-0.010
		50	200	0.950	0.648	0.985	0.115	0.939	-0.011
		100	200	0.941	0.746	0.973	0.102	0.939	-0.001
		200	200	0.940	0.809	0.974	0.075	0.939	-0.001
		500	200	0.938	0.867	0.960	0.045	0.939	0.002
		1,000	200	0.933	0.896	0.954	0.031	0.939	0.007
		2,000	200	0.930	0.911	0.949	0.020	0.939	0.009
		5,000	200	0.930	0.925	0.940	0.007	0.939	0.009
bulk modulus	Magpie	20	200	0.914	-0.032	0.990	0.311	0.882	-0.032
		50	200	0.896	0.656	0.972	0.165	0.882	-0.014
		100	200	0.882	0.590	0.967	0.156	0.882	-0.000
		200	200	0.884	0.691	0.946	0.120	0.882	-0.001
		500	200	0.878	0.793	0.924	0.069	0.882	0.004
		1,000	200	0.878	0.826	0.917	0.048	0.882	0.004
		2,000	200	0.876	0.832	0.907	0.030	0.882	0.006
		5,000	200	0.876	0.867	0.889	0.012	0.882	0.006
crystal system	ORB	20	200	0.489	0.155	0.790	0.341	0.604	0.115
		50	200	0.537	0.342	0.776	0.215	0.604	0.067
		100	200	0.535	0.351	0.665	0.146	0.604	0.069
		200	200	0.550	0.442	0.656	0.099	0.604	0.054
		500	200	0.553	0.486	0.633	0.064	0.604	0.051
		1,000	200	0.553	0.510	0.603	0.043	0.604	0.051
		2,000	200	0.555	0.521	0.598	0.029	0.604	0.049
		5,000	200	0.555	0.537	0.573	0.017	0.604	0.050
		10,000	200	0.555	0.543	0.565	0.010	0.604	0.049
		20,000	1	0.555	0.555	0.555	—	0.604	0.049
crystal system	UMA	20	200	0.556	0.236	0.871	0.345	0.663	0.108
		50	200	0.606	0.318	0.806	0.211	0.663	0.058
		100	200	0.607	0.460	0.725	0.141	0.663	0.056
		200	200	0.612	0.524	0.710	0.092	0.663	0.051
		500	200	0.617	0.555	0.686	0.059	0.663	0.046
		1,000	200	0.618	0.585	0.651	0.037	0.663	0.046
		2,000	200	0.617	0.583	0.651	0.030	0.663	0.047
		5,000	200	0.618	0.599	0.637	0.017	0.663	0.045
		10,000	200	0.618	0.606	0.629	0.010	0.663	0.045
		20,000	1	0.618	0.618	0.618	—	0.663	0.045
crystal system	Magpie	20	200	0.393	0.057	0.700	0.324	0.520	0.127
		50	200	0.463	0.262	0.659	0.204	0.520	0.057
		100	200	0.462	0.288	0.607	0.145	0.520	0.058
		200	200	0.470	0.382	0.573	0.101	0.520	0.050
		500	200	0.480	0.417	0.529	0.062	0.520	0.040
		1,000	200	0.474	0.440	0.509	0.040	0.520	0.046
		2,000	200	0.475	0.442	0.510	0.031	0.520	0.045
		5,000	200	0.477	0.460	0.495	0.018	0.520	0.043
		10,000	200	0.477	0.467	0.486	0.010	0.520	0.044
		20,000	1	0.477	0.477	0.477	—	0.520	0.044
metal / insulator	ORB	20	200	0.897	0.697	1.000	0.178	0.908	0.011
		50	200	0.897	0.745	0.980	0.113	0.908	0.011
		100	200	0.900	0.830	0.970	0.072	0.908	0.008
		200	200	0.895	0.840	0.945	0.054	0.908	0.013
		500	200	0.897	0.853	0.924	0.032	0.908	0.011
		1,000	200	0.899	0.872	0.921	0.023	0.908	0.009
		2,000	200	0.898	0.880	0.916	0.016	0.908	0.010
		5,000	200	0.898	0.887	0.908	0.009	0.908	0.010
		10,000	200	0.898	0.892	0.903	0.006	0.908	0.010
		20,000	1	0.898	0.898	0.898	—	0.908	0.010
metal / insulator	UMA	20	200	0.899	0.567	1.000	0.176	0.914	0.014
		50	200	0.902	0.731	0.980	0.115	0.914	0.012
		100	200	0.908	0.847	0.970	0.072	0.914	0.005
		200	200	0.906	0.842	0.955	0.055	0.914	0.008
		500	200	0.904	0.872	0.928	0.030	0.914	0.010
		1,000	200	0.907	0.883	0.929	0.023	0.914	0.007
		2,000	200	0.906	0.882	0.925	0.017	0.914	0.008
		5,000	200	0.906	0.898	0.917	0.008	0.914	0.008
		10,000	200	0.906	0.900	0.911	0.006	0.914	0.008
		20,000	1	0.906	0.906	0.906	—	0.914	0.008
metal / insulator	Magpie	20	200	0.840	0.520	1.000	0.225	0.845	0.005
		50	200	0.830	0.672	0.936	0.130	0.845	0.015
		100	200	0.835	0.729	0.930	0.094	0.845	0.011
		200	200	0.834	0.752	0.894	0.069	0.845	0.011
		500	200	0.831	0.789	0.873	0.041	0.845	0.015
		1,000	200	0.834	0.805	0.861	0.029	0.845	0.012
		2,000	200	0.833	0.811	0.850	0.020	0.845	0.013
		5,000	200	0.835	0.821	0.848	0.012	0.845	0.011
		10,000	200	0.834	0.827	0.841	0.007	0.845	0.011
		20,000	1	0.834	0.834	0.834	—	0.845	0.012
magnetic / not	ORB	20	200	0.850	0.600	1.000	0.186	0.888	0.038
		50	200	0.879	0.699	1.000	0.135	0.888	0.009
		100	200	0.880	0.799	0.960	0.083	0.888	0.009
		200	200	0.879	0.805	0.930	0.061	0.888	0.009
		500	200	0.878	0.836	0.920	0.038	0.888	0.010
		1,000	200	0.877	0.844	0.905	0.027	0.888	0.011
		2,000	200	0.878	0.858	0.895	0.018	0.888	0.010
		5,000	200	0.878	0.868	0.890	0.010	0.888	0.011
		10,000	200	0.877	0.871	0.882	0.006	0.888	0.011
		20,000	1	0.877	0.877	0.877	—	0.888	0.011
magnetic / not	UMA	20	200	0.896	0.670	1.000	0.168	0.903	0.007
		50	200	0.899	0.773	1.000	0.116	0.903	0.004
		100	200	0.899	0.819	0.960	0.074	0.903	0.004
		200	200	0.895	0.805	0.945	0.059	0.903	0.009
		500	200	0.896	0.858	0.928	0.036	0.903	0.007
		1,000	200	0.895	0.868	0.916	0.025	0.903	0.009
		2,000	200	0.896	0.876	0.915	0.017	0.903	0.008
		5,000	200	0.895	0.886	0.905	0.009	0.903	0.008
		10,000	200	0.895	0.887	0.901	0.006	0.903	0.008
		20,000	1	0.895	0.895	0.895	—	0.903	0.008
magnetic / not	Magpie	20	200	0.847	0.549	1.000	0.210	0.860	0.013
		50	200	0.839	0.657	0.959	0.136	0.860	0.021
		100	200	0.847	0.748	0.940	0.094	0.860	0.013
		200	200	0.847	0.751	0.929	0.067	0.860	0.013
		500	200	0.847	0.794	0.891	0.041	0.860	0.013
		1,000	200	0.845	0.817	0.869	0.028	0.860	0.015
		2,000	200	0.845	0.823	0.863	0.020	0.860	0.015
		5,000	200	0.845	0.832	0.856	0.012	0.860	0.015
		10,000	200	0.845	0.838	0.853	0.007	0.860	0.015
		20,000	1	0.845	0.845	0.845	—	0.860	0.015

Test size moves the spread and little else. Across the grid the spread falls from 0.239 at 20 test materials to 0.010 at 5,000 — a factor of 24 — while the shortfall down each curve moves by no more than 0.021 from 200 test materials upward — for band gap under XGBoost, 0.394 to 0.031 of spread against a shortfall that moves by at most 0.021. The two axes are separable. Reaching ±0.016 on a score takes about 1,000 test materials; 500 gives ±0.024 and 200 gives ±0.040.

Test floor: 500 materials

The spread is how far a score moves when the test set is re-drawn: the p10–p90 width over 200 re-drawn test sets, scored with one probe trained on 20,000 materials.

test materials	median spread	range across targets	± on one score
20	0.250	0.033–4.140	±0.125
50	0.125	0.028–0.507	±0.063
100	0.107	0.020–0.300	±0.053
200	0.078	0.015–0.189	±0.039
500	0.046	0.012–0.117	±0.023
1,000	0.032	0.011–0.080	±0.016
2,000	0.020	0.007–0.059	±0.010
5,000	0.009	0.005–0.034	±0.005
10,000	0.008	0.003–0.019	±0.004

Median and range over every target measured at that size; bulk modulus stops at 5,000, and the 20,000 point is the whole test set, a single score with no spread. The highlighted row is the floor.

500 is the smallest size on the grid at which the median target's spread is under 0.05, about ±0.023 on one score. At 200 it is 0.078 (±0.039), and the hardest target, magnetic density, moves by 0.189. Like the shortfall, the spread has no knee: it more than halves from 200 to 1,000 and keeps falling. At an 80/20 split the training floor already brings 1,250 test materials, above the test floor, so the test floor matters for smaller datasets and for other splits.

Both floors together

Two floors, one per axis: a training size read off the shortfall, and a test size read off the spread. XGBoost, each target the mean of ORB-v3 and UMA-S.

training floor5,000

median shortfall there0.058

test floor500

median spread there0.046 (±0.023)

materials at 80/206,250

median across targetsband: middle half of targetsdashed: spread of 0.05dotted: the floor

training floor

test floor

The floors are 5,000 training and 500 test materials. At 80/20 the training floor takes 6,250 materials in total, which puts 1,250 in the test set.

Neither curve has a natural cut. Both floors were chosen after seeing the curves, as the smallest sizes at which the median target's shortfall is small enough to report and its spread is under 0.05. They are judgements about what is reasonable to ask for.

A score below the training floor is not worthless — it is a lower bound, and should be reported as one, with its per-target shortfall beside it. Below the test floor it is noisier, and should be reported with its spread. Buying 0.011 of shortfall for another 2,500 materials is a trade for the reader to make.

Floors for one property — the training and test size each property needs on its own

Both floors describe the median target, and the targets differ by an order of magnitude on both axes. Anyone interested in a single property should set the floors from its own row:

target	within 0.05 of its ceiling from (training materials, ORB-v3 / UMA-S)	spread at most 0.05 from (test materials, ORB-v3 / UMA-S)
formation energy / atom	2,000 / 1,000	20 / 20
energy above hull	10,000 / 5,000	1,000 / 1,000
band gap	7,000 / 7,000	5,000 / 5,000
density	7,000 / 7,000	200 / 200
magnetic density	20,000† / 20,000†	5,000 / 5,000
bulk modulus	2,000 / 2,000	500 / 500
crystal system	20,000† / 20,000†	1,000 / 1,000
metal / insulator	1,000 / 1,000	500 / 500
magnetic / not	5,000 / 2,000	500 / 500

† reached only at the largest training size measured for that target. For formation energy alone, 2,000 training materials (1,000 with UMA-S) bring the score within 0.05 of its ceiling, and 20 test materials already hold its spread at or below 0.05. Band gap needs 7,000 training and 5,000 test materials. Crystal system and magnetic density reach their ceilings only at 20,000, the largest size measured, and magnetic density also needs 5,000 test materials.

Where questions 1 and 2 meet: the split

Total error against the train/test fraction, at six dataset sizes from 200 to 10,000; the full-data tables below add 20,000.

With a fixed number of materials, every split has to decide how many go into the test set, and that choice moves two errors in opposite directions. Fewer materials for training means the probe scores lower than it would with more data: the shortfall, read off the training-size curve. More materials for testing means the score is measured more precisely: the spread, read off the test-size curve. For each split the page reads both off the curves and combines them into one total error; the lowest point on the plot below is the best split for that dataset size.

How the two are combined. The shortfall is a fixed offset: the score sits that far below its ceiling every time. The spread is random: a re-split moves the score up or down. A fixed error and a random one combine as √(shortfall² + sd²), where sd is the random error expressed as a standard deviation. The spread on this page is the width between the 10th and 90th percentile of the score across draws, which for a normal distribution is 2.5631 standard deviations; the page estimates it as exactly that, so sd = spread ÷ 2.5631 recovers the standard deviation. One adjustment comes first: a probe that scores lower scatters more, so the spread is scaled up by (1 − score)/(1 − ceiling), as in the calculator.

dataset size

target

Full data — total error at every split fraction, and which fraction won in each cell

N	split	train	test	score	shortfall	spread	total error
200	90/10	180	20	0.608	0.309	1.052	0.506
	80/20	160	40	0.592	0.319	0.727	0.394
	70/30	140	60	0.575	0.330	0.607	0.384
	60/40	120	80	0.554	0.343	0.566	0.390
	50/50	100	100	0.530	0.358	0.543	0.406
500	90/10	450	50	0.729	0.204	0.393	0.282
	80/20	400	100	0.713	0.219	0.317	0.267
	70/30	350	150	0.696	0.237	0.281	0.279
	60/40	300	200	0.675	0.257	0.253	0.295
	50/50	250	250	0.651	0.279	0.245	0.311
1,000	90/10	900	100	0.791	0.140	0.236	0.190
	80/20	800	200	0.784	0.150	0.176	0.194
	70/30	700	300	0.776	0.162	0.154	0.197
	60/40	600	400	0.761	0.175	0.137	0.205
	50/50	500	500	0.745	0.190	0.124	0.215
2,000	90/10	1,800	200	0.826	0.098	0.139	0.133
	80/20	1,600	400	0.820	0.105	0.101	0.134
	70/30	1,400	600	0.813	0.112	0.085	0.138
	60/40	1,200	800	0.806	0.121	0.076	0.142
	50/50	1,000	1,000	0.797	0.131	0.071	0.148
5,000	90/10	4,500	500	0.873	0.055	0.060	0.072
	80/20	4,000	1,000	0.868	0.060	0.045	0.070
	70/30	3,500	1,500	0.860	0.066	0.039	0.076
	60/40	3,000	2,000	0.851	0.073	0.035	0.081
	50/50	2,500	2,500	0.841	0.082	0.033	0.088
10,000	90/10	9,000	1,000	0.888	0.040	0.040	0.042
	80/20	8,000	2,000	0.887	0.044	0.031	0.046
	70/30	7,000	3,000	0.885	0.048	0.025	0.050
	60/40	6,000	4,000	0.881	0.053	0.022	0.056
	50/50	5,000	5,000	0.877	0.050	0.015	0.052
20,000	90/10	18,000	2,000	0.895	0.023	0.027	0.024
	80/20	16,000	4,000	0.894	0.026	0.018	0.026
	70/30	14,000	6,000	0.892	0.029	0.014	0.029
	60/40	12,000	8,000	0.891	0.032	0.012	0.033
	50/50	10,000	10,000	0.889	0.036	0.010	0.037

N	80/20 beats 50/50	80/20 is best	70/30 is best	90/10 is best
200	9 of 18	1 of 18	9 of 18	1 of 18
500	15 of 18	9 of 18	2 of 18	4 of 18
1,000	16 of 18	9 of 18	4 of 18	3 of 18
2,000	16 of 18	8 of 18	2 of 18	6 of 18
5,000	18 of 18	9 of 18	0 of 18	9 of 18
10,000	16 of 16	11 of 16	0 of 16	5 of 16
20,000	16 of 16	8 of 16	0 of 16	8 of 16

Shortfall costs more than spread at every dataset size from 500 up, so the minimum sits on the training side of an even split. 80/20 beats 50/50 in 15 of 18 cells at 500 materials, 16 at 1,000 and 2,000, and every cell from 5,000 up. 80/20 has the lowest median total error at 500 and 5,000; at 1,000 and 2,000 90/10 edges it by 0.004 and 0.001 while 80/20 wins more cells, 9 to 3 and 8 to 6; at 10,000 and 20,000 90/10 is lower by 0.003 and 0.002 and the cells split 11 to 5 for 80/20 and 8 to 8. At 200 materials the median minimum is 70/30 and 80/20 beats the even split in only 9 of 18 — but 200 is far below the floor this page settles on.

3 · Does the embedding add anything beyond composition?

Magpie is run once: the same probe on 132 composition-only features, on the same frozen test set and the same training draws as both embeddings. Each panel subtracts that one Magpie score from the ORB-v3 score and from the UMA-S score on the same draw, so the two lines share a baseline and differ only in the embedding.

ORB-v3 score, less MagpieUMA-S score, less Magpieband: p10–p90 of the paired differencezero: the formula does as well

formation energy / atom

energy above hull

band gap

density

magnetic density

bulk modulus

crystal system

metal / insulator

magnetic / not

Whether an embedding says more than the formula depends on the target and on the training size. On 6 of the nine targets both embeddings beat composition on at least 90 % of paired draws from 500 training materials or fewer. At 20,000 training materials, the embedding's score exceeds Magpie's by between 0.033 (magnetic / not, macro-F1) and 0.712 (energy above hull, R²). On density composition alone is ahead of both embeddings up to 500 training materials and the sign is not resolved until 2,000. On magnetic density neither embedding ever resolvably beats it: ORB-v3 is behind from 500 to 20,000, and UMA-S is behind from 2,000 to 10,000 and level at the other sizes.

One dot per embedding, target and training size. Filled in the arm's colour: that arm ahead on at least 90 % of the paired draws. Filled in the Magpie colour: composition ahead on at least 90 %. Hollow: no clear winner; the embedding wins some draws and composition wins others. Hover a dot for its numbers.

target	ORB-v3			UMA-S			largest training size
target	beats Magpie from	Magpie ahead at	difference at largest size	beats Magpie from	Magpie ahead at	difference at largest size	largest training size
formation energy / atom	200	—	+0.114	100	—	+0.120	20,000
energy above hull	50	—	+0.686	50	—	+0.712	20,000
band gap	100	—	+0.207	50	—	+0.227	20,000
density	2,000	50–500	+0.067	2,000	50–500	+0.059	20,000
magnetic density	never	500–20,000	-0.018	never	2,000–10,000	-0.001	20,000
bulk modulus	1,000	—	+0.042	500	—	+0.055	2,000
crystal system	500	—	+0.080	50	—	+0.143	20,000
metal / insulator	100	—	+0.062	50	—	+0.071	20,000
magnetic / not	200	—	+0.033	50	—	+0.049	20,000

Beats Magpie from: the smallest training size from which the embedding scores higher than Magpie on at least 90 % of paired draws, at that size and every larger one. Magpie ahead at: the training sizes where Magpie scores higher on at least 90 % of draws; at 7,000 and 10,000 training materials, which have seven draws, that means six of seven. Difference at the largest size: embedding score minus Magpie score, median over draws, in R² or macro-F1. Bulk modulus stops at 2,000 because its corpus cannot supply three separate training sets beyond that.

A score can clear both floors and still say nothing the formula did not. At 5,000 training materials, the floor, magnetic density under ORB-v3 reproduces to about ±0.009 and composition alone scores higher on every draw. The shortfall and the spread describe the score; neither says whether the embedding earned it. That is a third quantity, set by the target and the training size, and it belongs beside the other two.

Full data — embedding minus Magpie at every training size, both embeddings

Median, p10 and p90 of the paired difference over the draws at that size, and the share of draws the embedding won. Highlighted rows have p10 above zero: the embedding ahead on at least 90 % of draws.

target	embedding	n train	draws	median	p10	p90	embedding won
formation energy / atom	ORB	50	30	-0.014	-0.091	+0.078	47 %
		100	30	+0.036	-0.018	+0.108	73 %
		200	30	+0.064	+0.036	+0.106	100 %
		500	30	+0.103	+0.081	+0.119	100 %
		1,000	30	+0.113	+0.106	+0.123	100 %
		2,000	30	+0.119	+0.114	+0.130	100 %
		5,000	10	+0.121	+0.118	+0.125	100 %
		7,000	7	+0.120	+0.116	+0.122	100 %
		10,000	7	+0.118	+0.116	+0.121	100 %
		20,000	6	+0.114	+0.113	+0.115	100 %
energy above hull	ORB	50	30	+0.466	+0.115	+1.428	97 %
		100	30	+0.731	+0.400	+1.396	100 %
		200	30	+0.743	+0.422	+1.172	100 %
		500	30	+0.810	+0.670	+1.002	100 %
		1,000	30	+0.776	+0.696	+0.904	100 %
		2,000	30	+0.773	+0.722	+0.820	100 %
		5,000	10	+0.734	+0.713	+0.776	100 %
		7,000	7	+0.754	+0.709	+0.757	100 %
		10,000	7	+0.722	+0.708	+0.742	100 %
		20,000	6	+0.686	+0.683	+0.693	100 %
band gap	ORB	50	30	+0.069	-0.058	+0.250	80 %
		100	30	+0.178	+0.126	+0.281	100 %
		200	30	+0.219	+0.194	+0.251	100 %
		500	30	+0.273	+0.249	+0.301	100 %
		1,000	30	+0.278	+0.267	+0.298	100 %
		2,000	30	+0.271	+0.258	+0.279	100 %
		5,000	10	+0.247	+0.243	+0.254	100 %
		7,000	7	+0.235	+0.232	+0.245	100 %
		10,000	7	+0.226	+0.220	+0.235	100 %
		20,000	6	+0.207	+0.206	+0.211	100 %
density	ORB	50	30	-0.258	-0.376	-0.067	3 %
		100	30	-0.179	-0.298	-0.047	3 %
		200	30	-0.114	-0.176	-0.061	0 %
		500	30	-0.057	-0.077	-0.023	0 %
		1,000	30	-0.014	-0.034	+0.005	13 %
		2,000	30	+0.015	+0.007	+0.029	100 %
		5,000	10	+0.049	+0.045	+0.055	100 %
		7,000	7	+0.054	+0.050	+0.060	100 %
		10,000	7	+0.058	+0.054	+0.064	100 %
		20,000	6	+0.067	+0.063	+0.072	100 %
magnetic density	ORB	50	30	-0.040	-0.201	+0.097	37 %
		100	30	-0.108	-0.221	+0.047	17 %
		200	30	-0.081	-0.198	+0.008	13 %
		500	30	-0.064	-0.098	-0.018	7 %
		1,000	30	-0.062	-0.088	-0.039	0 %
		2,000	30	-0.049	-0.066	-0.039	0 %
		5,000	10	-0.039	-0.046	-0.030	0 %
		7,000	7	-0.029	-0.037	-0.027	0 %
		10,000	7	-0.027	-0.029	-0.022	0 %
		20,000	6	-0.018	-0.024	-0.015	0 %
bulk modulus	ORB	50	30	-0.035	-0.170	+0.088	30 %
		100	30	-0.030	-0.070	+0.021	17 %
		200	30	-0.035	-0.069	+0.025	23 %
		500	12	+0.009	-0.004	+0.025	83 %
		1,000	6	+0.032	+0.026	+0.042	100 %
		2,000	3	+0.042	+0.042	+0.044	100 %
crystal system	ORB	50	30	-0.017	-0.038	+0.006	13 %
		100	30	+0.005	-0.014	+0.019	57 %
		200	30	+0.010	-0.006	+0.031	70 %
		500	30	+0.041	+0.029	+0.049	100 %
		1,000	30	+0.048	+0.040	+0.057	100 %
		2,000	30	+0.055	+0.045	+0.065	100 %
		5,000	10	+0.063	+0.058	+0.069	100 %
		7,000	7	+0.068	+0.063	+0.072	100 %
		10,000	7	+0.075	+0.068	+0.077	100 %
		20,000	6	+0.080	+0.078	+0.084	100 %
metal / insulator	ORB	50	30	+0.011	-0.042	+0.032	67 %
		100	30	+0.048	+0.027	+0.070	97 %
		200	30	+0.060	+0.047	+0.080	100 %
		500	30	+0.079	+0.069	+0.085	100 %
		1,000	30	+0.078	+0.070	+0.085	100 %
		2,000	30	+0.076	+0.071	+0.080	100 %
		5,000	10	+0.071	+0.066	+0.075	100 %
		7,000	7	+0.068	+0.064	+0.070	100 %
		10,000	7	+0.067	+0.063	+0.068	100 %
		20,000	6	+0.062	+0.060	+0.063	100 %
magnetic / not	ORB	50	30	-0.007	-0.047	+0.054	43 %
		100	30	+0.018	-0.008	+0.048	73 %
		200	30	+0.021	+0.006	+0.038	97 %
		500	30	+0.033	+0.019	+0.042	100 %
		1,000	30	+0.038	+0.030	+0.044	100 %
		2,000	30	+0.037	+0.029	+0.041	100 %
		5,000	10	+0.034	+0.032	+0.035	100 %
		7,000	7	+0.033	+0.032	+0.034	100 %
		10,000	7	+0.032	+0.031	+0.036	100 %
		20,000	6	+0.033	+0.030	+0.035	100 %
formation energy / atom	UMA	50	30	+0.091	-0.008	+0.206	87 %
		100	30	+0.098	+0.052	+0.155	97 %
		200	30	+0.125	+0.094	+0.156	100 %
		500	30	+0.130	+0.112	+0.150	100 %
		1,000	30	+0.133	+0.125	+0.144	100 %
		2,000	30	+0.132	+0.127	+0.143	100 %
		5,000	10	+0.131	+0.128	+0.134	100 %
		7,000	7	+0.127	+0.124	+0.130	100 %
		10,000	7	+0.124	+0.122	+0.129	100 %
		20,000	6	+0.120	+0.119	+0.120	100 %
energy above hull	UMA	50	30	+0.444	+0.080	+1.727	93 %
		100	30	+0.829	+0.494	+1.532	97 %
		200	30	+0.826	+0.503	+1.260	100 %
		500	30	+0.893	+0.784	+1.084	100 %
		1,000	30	+0.833	+0.783	+0.969	100 %
		2,000	30	+0.824	+0.783	+0.873	100 %
		5,000	10	+0.776	+0.761	+0.816	100 %
		7,000	7	+0.790	+0.746	+0.795	100 %
		10,000	7	+0.751	+0.742	+0.776	100 %
		20,000	6	+0.712	+0.707	+0.722	100 %
band gap	UMA	50	30	+0.228	+0.127	+0.348	100 %
		100	30	+0.309	+0.220	+0.399	100 %
		200	30	+0.303	+0.273	+0.339	100 %
		500	30	+0.331	+0.310	+0.342	100 %
		1,000	30	+0.319	+0.301	+0.336	100 %
		2,000	30	+0.304	+0.291	+0.315	100 %
		5,000	10	+0.273	+0.268	+0.283	100 %
		7,000	7	+0.262	+0.254	+0.268	100 %
		10,000	7	+0.251	+0.243	+0.256	100 %
		20,000	6	+0.227	+0.226	+0.233	100 %
density	UMA	50	30	-0.205	-0.344	-0.029	10 %
		100	30	-0.166	-0.286	-0.025	7 %
		200	30	-0.103	-0.140	-0.053	0 %
		500	30	-0.040	-0.070	-0.015	3 %
		1,000	30	-0.009	-0.024	+0.002	23 %
		2,000	30	+0.015	+0.003	+0.023	100 %
		5,000	10	+0.042	+0.039	+0.045	100 %
		7,000	7	+0.048	+0.043	+0.051	100 %
		10,000	7	+0.051	+0.045	+0.055	100 %
		20,000	6	+0.059	+0.056	+0.063	100 %
magnetic density	UMA	50	30	+0.106	-0.060	+0.345	80 %
		100	30	+0.038	-0.037	+0.219	73 %
		200	30	+0.033	-0.053	+0.111	67 %
		500	30	-0.006	-0.030	+0.043	43 %
		1,000	30	-0.016	-0.043	+0.012	27 %
		2,000	30	-0.013	-0.026	-0.003	7 %
		5,000	10	-0.009	-0.018	-0.000	10 %
		7,000	7	-0.011	-0.017	-0.004	0 %
		10,000	7	-0.003	-0.007	-0.000	14 %
		20,000	6	-0.001	-0.005	+0.004	50 %
bulk modulus	UMA	50	30	-0.013	-0.078	+0.132	47 %
		100	30	+0.019	-0.051	+0.057	63 %
		200	30	+0.017	-0.028	+0.055	67 %
		500	12	+0.032	+0.023	+0.043	100 %
		1,000	6	+0.042	+0.038	+0.056	100 %
		2,000	3	+0.055	+0.054	+0.055	100 %
crystal system	UMA	50	30	+0.048	+0.028	+0.066	97 %
		100	30	+0.072	+0.051	+0.093	100 %
		200	30	+0.075	+0.063	+0.089	100 %
		500	30	+0.099	+0.089	+0.114	100 %
		1,000	30	+0.113	+0.100	+0.119	100 %
		2,000	30	+0.123	+0.115	+0.128	100 %
		5,000	10	+0.131	+0.127	+0.139	100 %
		7,000	7	+0.131	+0.126	+0.138	100 %
		10,000	7	+0.138	+0.136	+0.143	100 %
		20,000	6	+0.143	+0.138	+0.146	100 %
metal / insulator	UMA	50	30	+0.065	+0.025	+0.098	97 %
		100	30	+0.081	+0.060	+0.109	100 %
		200	30	+0.085	+0.072	+0.102	100 %
		500	30	+0.089	+0.079	+0.093	100 %
		1,000	30	+0.086	+0.078	+0.092	100 %
		2,000	30	+0.084	+0.079	+0.089	100 %
		5,000	10	+0.077	+0.070	+0.081	100 %
		7,000	7	+0.074	+0.072	+0.076	100 %
		10,000	7	+0.074	+0.070	+0.076	100 %
		20,000	6	+0.071	+0.068	+0.072	100 %
magnetic / not	UMA	50	30	+0.049	+0.007	+0.084	93 %
		100	30	+0.059	+0.025	+0.088	97 %
		200	30	+0.065	+0.050	+0.087	100 %
		500	30	+0.071	+0.061	+0.084	100 %
		1,000	30	+0.072	+0.064	+0.080	100 %
		2,000	30	+0.066	+0.060	+0.069	100 %
		5,000	10	+0.057	+0.054	+0.059	100 %
		7,000	7	+0.054	+0.052	+0.056	100 %
		10,000	7	+0.053	+0.051	+0.056	100 %
		20,000	6	+0.049	+0.048	+0.053	100 %

4 · A methodological check: reusing the data

The cross-validation comparison has not been run to completion yet; its results will appear here.

Given a small dataset, which way of measuring a probe's score gives the most reliable estimate?

With a small dataset, measuring a probe creates a trade-off. Materials kept back for testing cannot be used for training, but a very small test set gives a noisy score. Three common ways of making that estimate are compared.

method	procedure	models trained
single 80/20 split	train on 80 % of the materials, score on the other 20 %, once	1
five 80/20 splits, averaged	repeat five random 80/20 splits and average the five scores	5
5-fold cross-validation	cut the materials into five equal parts; train on four and predict the fifth, rotating until every material has been predicted once; score all predictions together	5

In all three methods, each individual model is trained on 80 % of the small dataset. They differ only in how the held-out materials are used to score it. In 5-fold cross-validation every material is scored exactly once, by a model that was not trained on it.

How the methods were evaluated

Someone who has only a small dataset cannot know how well their probe really performs; that is what the estimate is for. Here the much larger corpus makes it possible to approximate. A frozen test set of 20,000 materials (6,426 for bulk modulus) is kept out of everything. For each small dataset of n materials, a probe is trained on all n and scored on those 20,000. Because that test set is large, this score is used as the reference score for the small dataset. It is a different number from the paper's full-corpus score used elsewhere on this page, which belongs to a probe trained on all 154,875 materials.

Each of the three methods then estimates the score from the n materials alone, and its estimate is compared with the reference score. The comparison covers 90 combinations — two embeddings, nine properties, five dataset sizes from 100 to 2,000 materials — with 20 independent datasets for each (fewer for bulk modulus above 500).

Important caveat. The reference probe is trained on all n materials, while every model inside the three methods is trained on 0.8n. The difference from the reference therefore includes both the error of the method and a small effect of the smaller training set; all three methods share that effect, so it lowers all three estimates slightly.

Accuracy of the reported score

The typical error is the root-mean-square difference between a method's reported score and the reference score over the 20 datasets of a size, as a median over properties and embeddings (R² or macro-F1). The lowest in each column is in bold.

method	dataset of 100	dataset of 200	dataset of 500	dataset of 1,000	dataset of 2,000
single 80/20 split	0.178	0.093	0.053	0.045	0.028
five 80/20 splits, averaged	0.085	0.079	0.034	0.025	0.017
5-fold cross-validation	0.082	0.065	0.030	0.023	0.015

5-fold cross-validation gives the score closest to the reference at every dataset size. Averaging five random splits closes most of the gap; a single split is much noisier, with 1.6 to 2 times the error of either alternative from 500 materials up.

The chart below compares cross-validation with the five-split average directly: each bar is the share by which cross-validation's error is smaller than the five-split average's, averaged over the nine properties at one dataset size.

How much smaller cross-validation's error is than that of five averaged 80/20 splits

Reliability of the 95 % intervals

Each method also reports a 95 % interval around its score. An interval that is as reliable as it claims should contain the reference score on 95 % of datasets.

method	interval contains the reference score	interval contains the expected score at 80 % of the data
single 80/20 split	88 %	90 %
five 80/20 splits, averaged	85 %	87 %
5-fold cross-validation	81 %	88 %

Share of datasets, averaged over all 90 combinations. The second column checks the caveat above: it compares each interval with the expected score of a probe trained on 80 % of the dataset, which is what every method actually trains.

None of the methods reaches 95 %. Comparing with the expected score at 80 % of the data lifts all three, to 87–90 %, so the training-size mismatch explains only part of the shortfall. The chart below follows the two more accurate methods across dataset sizes: each pair of dots is one size, averaged over the nine properties, with the five-split average in grey and cross-validation in its embedding's colour.

How often each method's 95 % interval contains the reference score

five averaged 80/20 splits5-fold cross-validationdashed: 0.95, what a 95 % interval promises

Every dot sits below the dashed 0.95 line. Cross-validation's interval is the narrowest of the three, and it is the less reliable of the pair at 100, 200 and 2,000 materials on both embeddings and at 1,000 on UMA-S. Cross-validation is therefore the most accurate point estimate and the least reliable interval. For how much a score moves, quote the spread measured on this page for the test size in hand.

By property and dataset size

CV closer than five splits in84 %

median error cut, n ≤ 1,00016.8 %

cells90

embedding

show

Full data — band gap, five averaged splits and cross-validation at every dataset size

Shortfall is the distance from the full-corpus score at the training size each arm uses, 0.8n.

emb	n	draws	five-split error	CV RMSE	reduction	five-split coverage	CV cov.	shortfall
ORB	100	20	0.2290	0.1539	+33 %	0.75	0.70	0.501
ORB	200	20	0.0892	0.0613	+31 %	0.85	0.85	0.371
ORB	500	20	0.0352	0.0312	+11 %	0.95	0.85	0.245
ORB	1,000	20	0.0298	0.0368	-23 %	0.95	0.80	0.176
ORB	2,000	20	0.0194	0.0137	+29 %	0.85	0.95	0.123
UMA	100	20	0.2124	0.1133	+47 %	0.95	0.90	0.397
UMA	200	20	0.0911	0.0778	+15 %	1.00	0.80	0.297
UMA	500	20	0.0360	0.0299	+17 %	0.80	0.90	0.207
UMA	1,000	20	0.0316	0.0287	+9 %	0.90	0.85	0.153
UMA	2,000	20	0.0157	0.0101	+36 %	0.90	1.00	0.109

CV has the lower RMSE in 9 of 10 cells; its coverage is below 0.90 in 6 of the 10, and the five-split average's in 4. Draws are 20 at every n. The shortfall column is the same quantity for both arms, since both train on 0.8n.

Main conclusion. For limited datasets, 5-fold cross-validation is the best of the three procedures tested for estimating the score itself, and its advantage is largest when the dataset is very small. Five averaged splits come close; a single split is substantially noisier and should not be used. None of the reported 95 % intervals reaches its stated coverage, cross-validation's least of all, so they should be read with caution, and the comparison carries the training-size mismatch described above.

More on the cross-validation comparison — what it does not change, and its limits

What cross-validation does not change

Cross-validation makes the estimate of a small-data probe's score more accurate. It does not bring that score closer to what the same embedding reaches on the full corpus, because each of its models still trains on 80 % of the data. For an embedding comparison, that distance is usually the larger error.

Band gap, ORB-v3. Bias is the shortfall from the full-corpus score at 80 % of the data; noise is the spread of each method's error across datasets; total combines the two.

materials	bias	noise: 1 split	5 splits	CV	total: 1 split	5 splits	CV
100	0.501	0.562	0.154	0.141	0.753	0.524	0.520
200	0.371	0.096	0.080	0.051	0.384	0.380	0.375
500	0.245	0.073	0.032	0.027	0.256	0.247	0.247
1,000	0.176	0.067	0.026	0.035	0.188	0.178	0.179
2,000	0.123	0.020	0.015	0.011	0.125	0.124	0.124

At 2,000 materials cross-validation cuts the noise from 0.020 (one split) to 0.011, and the total from 0.125 to 0.124, because the bias of 0.123 dominates.

what this does not settle

Each combination rests on 20 datasets, so its own coverage figure is uncertain by about ±5 percentage points; the averages over 90 combinations are firmer.
Only the XGBoost probe, only 5 folds, and only datasets of 100 to 2,000 materials were tested.
Bulk modulus has fewer datasets above 500 materials (12, 6 and 3 at 500, 1,000 and 2,000) because its pool runs out.

Conclusion

Is the model trained on enough materials? Use a floor of 5,000 training materials. Targets differ five- to ninefold, so for a single property read its own row of the floors table.
Would another test set give the same score? Use a floor of 500 test materials and an 80/20 split; at 80/20 the training floor already brings 1,250 test materials. On small datasets, score with 5-fold cross-validation or five averaged splits, never a single split, and do not quote their 95 % intervals as they stand.
Does the embedding add anything beyond composition? Report the gain over composition per target. The embeddings beat Magpie from 500 training materials or fewer on 6 targets, from 2,000 on density, and never on magnetic density.

References

Bates, S., Hastie, T., Tibshirani, R. (2023). Cross-validation: what does it estimate and how well does it do it? Journal of the American Statistical Association. arXiv:2104.00673.
Varoquaux, G. (2018). Cross-validation failure: small sample sizes lead to large error bars. NeuroImage 180, 68–77. arXiv:1706.07581.
Bengio, Y., Grandvalet, Y. (2004). No unbiased estimator of the variance of k-fold cross-validation. Journal of Machine Learning Research 5, 1089–1105.
Xu, P., Ji, X., Li, M., Lu, W. (2023). Small data machine learning in materials science. npj Computational Materials 9, 42.
Ward, L., Agrawal, A., Choudhary, A., Wolverton, C. (2016). A general-purpose machine learning framework for predicting properties of inorganic materials. npj Computational Materials 2, 16028. The Magpie feature set.
Stepurko, N., Laulainen, J. (2026). Out-of-training properties in frozen interatomic potential embeddings are decodable and ordered by physical scale. Accepted, AI4Sci and AI4Mat workshops, NeurIPS 2026; to be released in November 2026. Source of the full-corpus scores used here as ceilings.
Hastie, T., Tibshirani, R., Friedman, J. (2009). The Elements of Statistical Learning, 2nd ed., ch. 7, Model assessment and selection. Springer.
Cortes, C., Jackel, L. D., Solla, S. A., Vapnik, V., Denker, J. S. (1994). Learning curves: asymptotic values and rate of convergence. Advances in Neural Information Processing Systems 6, 327–334.