Biomolecular emulators · solution scattering · experimental design
BioEmu-1 [1] predicts the set of shapes a protein adopts in solution from its sequence. I asked where these predicted ensembles disagree with measurements, and which new measurements would test and improve such a model. As a benchmark, I compared raw BioEmu-1 ensembles with small-angle X-ray scattering (SAXS) profiles from SASBDB [4], using the ensembles and computed curves that the PeptoneBench authors published [3]. The clearest systematic error ran opposite to my expectation: for disordered proteins the ensembles are too extended. The obstacle for choosing new experiments is prediction: the size of the error on a protein could not be predicted before measuring it. The proposed campaign therefore has two arms, a controlled replication of known discrepancies and a prospective test of the extension bias.
SAXS reports the overall size and shape of a protein in solution, averaged over every conformation it adopts, so it tests an ensemble as a whole. BioEmu-1 was trained on predicted structures, molecular-dynamics simulations and measured stabilities [1]. BioEmu conformers have served as a pool for SAXS ensemble selection on a few dozen proteins, where the unweighted pool did not reproduce the data [2], and PeptoneBench ranks generators by their fit after reweighting [3]. This study asks where the raw model fails by protein class and what that implies for the next measurements.
This project re-analyses BioEmu-1 ensembles sampled by the PeptoneBench authors; it does not retrain the model. The archive records the BioEmu code version but not the checkpoint, so for 51 proteins I also sampled the current checkpoint, bioemu-v1.2, and compared the two (Does the current checkpoint show the same bias?).
Four expectations were written down, each with its reasoning, before any score was computed.
| expectation | reasoning | outcome |
|---|---|---|
| E1 Raw χ² is worse for disordered than for folded proteins. | The model was trained mostly on folded structures and simulations of them. | Supported |
| E2 Among folded proteins, raw χ² rises with chain length. | More domains, more ways to misplace them. | Not supported |
| E3 Most entries reach χ² ≈ 1 while keeping φ > 0.3. | The right conformers are usually present, only mis-weighted. | Not supported |
| E4 Disordered ensembles are too compact (Rg below experiment). | The known over-compaction of models trained on folded data. | Opposite direction observed |
E1 median χ² 8.9 for disordered against 2.3 for folded proteins; one-sided Mann–Whitney p < 0.001.
E2 Spearman ρ = +0.09 (p = 0.16, n = 246). The data show a step: folded chains of up to 100 residues meet χ² ≤ 2 in 67% of cases (n = 15), longer ones in 45% (n = 231), with no trend above that.
E3 27% of entries reach χ² ≈ 1 at all; 12% do so keeping φ > 0.3.
E4 median Rg ratio 1.21 (95% interval 1.12–1.28), two-sided Wilcoxon on the log ratio p < 0.001; 76% of disordered entries above 1.
Each raw ensemble is the uniform average of its conformer curves, scaled to the profile with one factor and one constant background. Of the 399 profiles, 146 meet χ² ≤ 2 (37%, 95% interval 32%–41%). The median raw χ² is 8.9 for disordered and 2.3 for folded proteins (4.3 for partly disordered). Under NRMSD, a deviation of ln I that does not weight points by their reported errors, the class medians are similar: 0.030 folded, 0.034 partly disordered and 0.031 disordered, with overlapping intervals. A poor reduced χ² can reflect both the disagreement with the curve and the precision assigned to the measurement. The size error is examined separately below.
What it shows. Each point is one of the 399 profiles. Left: reduced χ² of the uniform BioEmu-1 ensemble, log scale, with the class median (bar and value) and the χ² = 2 threshold (dashed). Right: NRMSD of the same fits, with the class median and its 95% bootstrap interval.
How to read it. Under χ² the disordered median is several times the folded one; under NRMSD the three medians lie within each other's intervals.
| class | n | median raw χ² | quartiles | χ² ≤ 2 (95%) | median NRMSD (95%) | target fit not reached |
|---|---|---|---|---|---|---|
| folded | 246 | 2.3 | 1.3–7.7 | 46% (40%–53%) | 0.030 (0.028–0.033) | 19% |
| partly disordered | 112 | 4.3 | 2.0–13.7 | 24% (17%–32%) | 0.034 (0.029–0.037) | 16% |
| disordered | 41 | 8.9 | 4.6–25.1 | 12% (2%–22%) | 0.031 (0.025–0.040) | 12% |
NRMSD: root-mean-square deviation of ln I between the scaled model and the data over the points with I > 0 and I ≥ 3σ, divided by the range of ln I of the data over those points. Intervals are percentile bootstraps over entries (2,000 resamples).
What it shows. Raw reduced χ² of every profile against chain length, coloured by class. Among folded proteins the rank correlation is weak (ρ = +0.09); chains of up to 100 residues meet χ² ≤ 2 more often (67% against 45%).
One protein per class, the one nearest its class median of raw χ², shows the fits on measured curves.
What it shows. Top: every measured point with its error band and three computed curves scaled to it, BioEmu-1 raw (solid), reweighted (dotted) and AlphaFold2 (dashed), with their χ² above each panel. Middle: residuals. Bottom: the ab initio envelope reconstructed from the measured curve, with the Cα trace of a BioEmu-1 conformer docked inside.
How to read it. In the disordered column the raw ensemble is larger than the protein in solution (Rg 78 Å against 72 Å measured), and reweighting to χ² ≤ 2 moves it to 75 Å, toward the measured value.
Residuals. (Imodel − Iexp)/σ against q, with ±3 guides. A residual that wanders outside ±3 over a range of q is a shape error at that length scale (about 1/q); a flat band within ±3 is a fit at the noise level.
Reweighted curve. The reweighted curve is taken where the four kinds are read, at the first point along the reweighting path with χ² ≤ 2 (φ 0.96 for SASDH84, φ 0.90 for SASDM65, φ 0.84 for SASDLK9). φ is the effective sample fraction the weights keep, 1 for uniform weights. The χ² minimum of each path is 1.15 at φ 0.021, 0.99 at φ 0.080, 0.93 at φ 0.085 (folded, partly disordered, disordered); a minimum at small φ rests on a few conformers and overfits.
Envelope. DENSS [8] reconstructions from the measured curve alone (ten maps, aligned and averaged; Dmax from the SASBDB record), drawn as filled projections of two density levels of the averaged map: the isosurface enclosing the protein's expected volume at 1.7 ų per Da (darker) and the one enclosing the volume DENSS assigned to the particle (lighter). The conformer is the highest-weight one at the χ² minimum of the path, docked into the protein-volume level by principal axes; the share of its Cα atoms inside that level is given under each panel and, for both levels, in the table. The drawn conformer has breaks in the sampled chain (consecutive Cα more than 4.2 Å apart): 5 in SASDM65, 24 in SASDLK9.
How typical the examples are. Each is the entry nearest its class median of raw χ². SASDLK9 is not typical in size: its size ratio (1.09) is below the disordered median (1.21), so this column understates the class-wide excess. It is 475 residues long, and the drawn conformer has 24 chain breaks.
What the envelope is. An illustration from an ensemble-averaged measurement, not evidence about any single conformation; for a disordered protein one conformer cannot fill it.
| class | entry | residues | kind | χ² raw | χ² reweighted | χ² AlphaFold2 | Rg measured (Å) | Rg raw | Rg reweighted | Cα inside, protein / particle level | chain breaks | DENSS χ² | resolution (Å) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| folded | SASDH84 | 135 | modestly reweightable | 2.33 | 1.91 (φ 0.96) | 2.3 | 21.2 | 19.8 | 20.0 | 89% / 99% | 0 | 1.21 | 25 |
| partly disordered | SASDM65 | 292 | modestly reweightable | 4.34 | 1.97 (φ 0.90) | 28.9 | 35.9 | 37.3 | 36.2 | 60% / 86% | 5 | 3.70 | 32 |
| disordered | SASDLK9 | 475 | modestly reweightable | 8.89 | 1.80 (φ 0.84) | 115.6 | 71.7 | 78.3 | 75.2 | 13% / 64% | 24 | 1.48 | 64 |
The radius of gyration, Rg, is the root-mean-square distance of a protein's mass from its centre, a single number for overall size; SAXS measures it from the lowest-angle part of the curve. For disordered proteins the median ensemble size is 21% above experiment from the scattering curves and 35% above from the conformer coordinates.
What it shows. Left: ensemble Rg over measured Rg for every profile, from the computed and measured curves, by class; bars are the median and its 95% bootstrap interval. Right: for the 41 disordered and 40 folded entries whose conformer coordinates were analysed, the same ratio from the curves (circles) and from the Cα coordinates of the conformers (diamonds).
How to read it. Above the dashed line the model's ensemble is larger than the protein in solution. The expectation was over-compaction (below the line); the disordered median lies above it.
E4 expected the usual failure of models trained on folded data, over-compaction of disordered chains. The data show the opposite: the median ratio is 1.21 (95% interval 1.12–1.28, two-sided Wilcoxon p < 0.001, n = 41), and 76% of disordered entries lie above 1; folded proteins sit at 0.98. The size discrepancy is also present in coordinate-derived radii: the Cα radius of the sampled conformers is 1.35 times the measured value (1.20–1.47). Cα, full-atom and SAXS-derived radii are different observables, however: for folded proteins the Cα ratio is 0.91 where the curve gives 0.95, an offset because the curve includes side chains and a hydration layer that the Cα radius leaves out. The 35% is therefore not to be read as an exact physical excess; it shows that the extension is present in the conformers before any curve is computed.
| disordered entries | n | median ratio (95%) | Wilcoxon p, log ratio |
|---|---|---|---|
| all clean disordered entries | 41 | 1.21 (1.12–1.28) | < 0.001 |
| five most extreme ratios dropped | 36 | 1.19 (1.07–1.24) | < 0.001 |
| near-duplicates collapsed (5-mer Jaccard ≥ 0.5) | 37 | 1.21 (1.12–1.30) | < 0.001 |
| Cα coordinates of the conformers | 41 | 1.35 (1.20–1.47) | < 0.001 |
Within the class the ratio grows with chain length (Spearman ρ = +0.35, p = 0.03, n = 41). The 439 profiles contain no identical sequences; 38 pairs are near-duplicates (about 93% identity or closer), and collapsing each cluster leaves the raw-fit share at 36%.
What it shows. Radius of gyration of the BioEmu-1 ensemble curve against the Guinier radius of the measured profile, one point per profile, log axes.
| entry | entry | 5-mer Jaccard | containment |
|---|---|---|---|
| SASDA37 | SASDA47 | 0.89 | 0.98 |
| SASDA37 | SASDA57 | 0.89 | 0.98 |
| SASDA47 | SASDA57 | 1.00 | 1.00 |
| SASDAE6 | SASDAR3 | 0.70 | 0.98 |
| SASDAQ2 | SASDBN2 | 0.95 | 1.00 |
| SASDAQ2 | SASDBP2 | 0.95 | 1.00 |
| SASDAQ2 | SASDRK7 | 0.95 | 1.00 |
| SASDAQ2 | SASDRQ7 | 0.83 | 1.00 |
| SASDAY4 | SASDAZ4 | 0.65 | 0.98 |
| SASDB26 | SASDB76 | 0.61 | 0.97 |
| SASDB43 | SASDB53 | 0.68 | 1.00 |
| SASDBN2 | SASDBP2 | 1.00 | 1.00 |
| SASDBN2 | SASDRK7 | 0.90 | 0.95 |
| SASDBN2 | SASDRQ7 | 0.79 | 0.95 |
| SASDBP2 | SASDRK7 | 0.90 | 0.95 |
| SASDBP2 | SASDRQ7 | 0.79 | 0.95 |
| SASDBW2 | SASDBX2 | 0.54 | 1.00 |
| SASDC26 | SASDC36 | 0.53 | 1.00 |
| SASDC35 | SASDJZ6 | 0.59 | 0.92 |
| SASDCD3 | SASDJM5 | 0.66 | 0.95 |
| SASDET2 | SASDUL9 | 0.67 | 0.93 |
| SASDF53 | SASDRD9 | 0.52 | 0.91 |
| SASDH65 | SASDH95 | 0.67 | 0.98 |
| SASDJF6 | SASDJH6 | 0.52 | 0.87 |
| SASDJG6 | SASDJH6 | 0.77 | 1.00 |
| SASDJN5 | SASDJP5 | 0.74 | 1.00 |
| SASDLS4 | SASDLT4 | 0.50 | 0.88 |
| SASDM43 | SASDM53 | 0.52 | 1.00 |
| SASDM55 | SASDM65 | 0.59 | 1.00 |
| SASDNK8 | SASDNL8 | 0.51 | 0.70 |
| SASDPC7 | SASDPD7 | 0.68 | 0.92 |
| SASDQC9 | SASDQD9 | 0.71 | 1.00 |
| SASDQF8 | SASDQJ8 | 0.72 | 0.98 |
| SASDR33 | SASDSV9 | 0.53 | 0.99 |
| SASDRK7 | SASDRQ7 | 0.79 | 0.95 |
| SASDRW9 | SASDRX9 | 0.55 | 1.00 |
| SASDSP6 | SASDSS6 | 0.70 | 0.88 |
| SASDUW6 | SASDUX6 | 0.65 | 0.94 |
The archive's ensembles came from a BioEmu checkpoint that the archive does not record. I re-sampled 51 proteins, every disordered profile that passes the quality checks and has a BioEmu-1 ensemble (41), with 9 folded proteins and one partly disordered protein as controls, using the current checkpoint, bioemu-v1.2, 100 samples each, on a free Colab GPU, and kept every sample. The two sets of conformers are compared like for like: the radius of gyration of the Cα atoms against the measured Guinier radius, and computed curves from the same back-mapping to all atoms (PULCHRA [19]) and the same Pepsi-SAXS run [6] for both, scored as the raw fits above. Two expectations were fixed before the re-sample was analysed.
| expectation, fixed before the re-sample was analysed | reasoning | outcome |
|---|---|---|
| E12. v1.2's median Cα-Rg ratio over the disordered proteins, all samples, remains above 1.1. | The archive's excess size is in the conformers, and disordered chains are nearly absent from BioEmu-1's public training data; I have no record that a later checkpoint added disordered training data. | met median 1.16, 0.95–1.51; on the point estimate only |
| E13. v1.2's median raw χ² over all 51 proteins, all samples, is lower than the archive's on the same proteins through the same path. | A later checkpoint released by the developers is expected to fit at least as well. | not met 25.5 against 6.1 |
For the 41 disordered proteins the median ratio of v1.2 is 1.16 (95% interval 0.95–1.51), against 1.35 (1.20–1.47) for the archive's ensembles of the same proteins. Protein by protein, v1.2's ensemble is the smaller for 24 of 41 (median ratio v1.2 over archive 0.95, 0.85–1.02; two-sided Wilcoxon p 0.07). The wide interval comes from chain length. For the 16 chains of up to 100 residues the v1.2 median is 0.89 (0.68–1.13), 10 of them below the measured size, against 1.18 for the archive; for the 25 longer chains (102–475 residues) it is 1.57 (1.01–1.69), against 1.43. The ratio rises with length (Spearman ρ = +0.51, p < 0.001), more steeply than the archive's (ρ = +0.33). Folded proteins sit at 0.86 against 0.94 (n = 9).
What it shows. Each point is one of the 51 proteins: the Cα radius of gyration of its ensemble (root mean square over all samples) divided by the measured Guinier radius, for the archive's ensemble (horizontal) and the v1.2 re-sample (vertical), coloured by class, log axes. Open circles: the 6 proteins for which BioEmu's filter rejects every v1.2 sample.
How to read it. Points below the diagonal are proteins whose v1.2 ensemble is smaller than the archive's; the dashed lines mark the measured size. A Cα radius leaves out side chains and the hydration layer, so folded proteins sit below 1 on both axes.
With all samples, v1.2 fits the profiles worse than the archive's ensembles do through the same path: median raw χ² 25.5 against 6.1 over the 51 proteins (difference of the medians 19.3, 95% interval 7.3–43.3; Wilcoxon p < 0.001), lower for only 12 of 51. The gap sits mostly with the disordered proteins (45.0 against 9.0); for folded proteins it is 4.5 against 2.2 (p 0.10). The archive's conformers through this path stay close to the archive's own published curves (median 6.1 against 5.7), so the path does not create the gap.
Most of v1.2's raw samples for disordered proteins fail BioEmu's own physical checks. BioEmu's sampling code removes, by default, any sample with consecutive Cα atoms more than 4.5 Å apart, a peptide C–N distance above 2.0 Å, or two atoms of residues more than two apart closer than 1 Å (a clash). Applied to every sample, these checks reject a median of 90% of v1.2's samples for the disordered proteins (mean 78%), mostly for clashes (71% of samples on average), and 40% of the archive's conformers of the same proteins, mostly for chain geometry; for folded proteins 13% and 8%. The share rises with chain length (Spearman ρ = +0.73, p < 0.001): above 100 residues the median is 97%, and for 6 disordered proteins every v1.2 sample fails: SASDEC3 (178), SASDJK6 (209), SASDF34 (271), SASDBZ6 (404), SASDLT4 (410), SASDLK9 (475) residues.
With the filter on both sides, the comparison keeps the 32 proteins with at least 10 passing conformers on each side (22 of the 41 disordered). On them the χ² gap narrows: 6.1 for v1.2 against 5.2 for the archive (Wilcoxon p 0.0057), where all samples of the same proteins give 8.6 against 5.0. Their sizes barely move: the disordered median ratio is 0.92 against 1.04 with the filter and 0.94 against 1.08 without. These proteins are mostly the shorter chains, so the filtered comparison says little about the long chains that carry the excess size.
What this means. The size error for disordered proteins is not specific to the archive's checkpoint: with bioemu-v1.2 the median ratio is still above 1.1, though its interval (0.95–1.51) is wide, and the ratio climbs with chain length more steeply than the archive's, so long chains remain too extended (median 1.57, interval above 1). The current checkpoint's raw samples for disordered proteins are mostly unphysical by BioEmu's own checks, and its filter leaves almost none for chains above 100 residues. This rests on one run of 100 samples per protein.
| class | n | v1.2 median ratio (95%) | archive median ratio (95%) | v1.2 over archive, median (95%) | v1.2 smaller | Wilcoxon p |
|---|---|---|---|---|---|---|
| all samples | ||||||
| disordered | 41 | 1.16 (0.95–1.51) | 1.35 (1.20–1.47) | 0.95 (0.85–1.02) | 24 of 41 | 0.07 |
| folded | 9 | 0.86 (0.77–1.08) | 0.94 (0.77–1.07) | 0.98 (0.91–1.01) | 7 of 9 | 0.05 |
| partly disordered (SASDEE9) | 1 | 1.11 | 1.24 | – | – | – |
| all samples, proteins of the filtered comparison | ||||||
| disordered | 22 | 0.94 (0.84–1.05) | 1.08 (0.90–1.30) | 0.85 (0.71–0.94) | 18 of 22 | < 0.001 |
| folded | 9 | 0.86 (0.77–1.08) | 0.94 (0.77–1.07) | 0.98 (0.91–1.01) | 7 of 9 | 0.05 |
| partly disordered (SASDEE9) | 1 | 1.11 | 1.24 | – | – | – |
| filter on both sides | ||||||
| disordered | 22 | 0.92 (0.78–1.14) | 1.04 (0.91–1.30) | 0.81 (0.75–0.90) | 19 of 22 | < 0.001 |
| folded | 9 | 0.87 (0.76–1.07) | 0.93 (0.77–1.08) | 0.98 (0.91–1.00) | 6 of 9 | 0.05 |
| partly disordered (SASDEE9) | 1 | 1.02 | 1.22 | – | – | – |
Ratio: root-mean-square Cα radius of gyration of the ensemble over the Guinier radius of the measured profile. Intervals: percentile bootstrap over proteins (2,000 resamples). The partly disordered protein is listed on its own. Filtered comparison: proteins with at least 10 passing conformers on both sides.
| proteins | n | v1.2 median χ² (95%) | archive, same path (95%) | archive, published curves | v1.2 over archive, median (95%) | v1.2 lower | Wilcoxon p |
|---|---|---|---|---|---|---|---|
| all samples | |||||||
| all proteins | 51 | 25.5 (12.7–54.0) | 6.1 (4.9–12.3) | 5.7 | 2.58 (1.87–4.21) | 12 of 51 | < 0.001 |
| disordered | 41 | 45.0 (23.6–62.7) | 9.0 (5.6–14.4) | 8.9 | 3.34 (1.90–6.53) | 8 of 41 | < 0.001 |
| folded | 9 | 4.5 (2.5–9.1) | 2.2 (1.1–5.4) | 2.0 | 2.52 (0.82–4.18) | 3 of 9 | 0.10 |
| all samples, proteins of the filtered comparison | |||||||
| all proteins | 32 | 8.6 (4.6–23.6) | 5.0 (3.2–9.4) | 4.6 | 2.55 (1.42–4.21) | 9 of 32 | < 0.001 |
| disordered | 22 | 20.0 (4.9–64.7) | 5.9 (4.2–14.4) | 5.7 | 3.66 (1.69–8.07) | 5 of 22 | < 0.001 |
| folded | 9 | 4.5 (2.5–9.1) | 2.2 (1.1–5.4) | 2.0 | 2.52 (0.82–4.18) | 3 of 9 | 0.10 |
| filter on both sides | |||||||
| all proteins | 32 | 6.1 (4.3–11.4) | 5.2 (2.7–9.5) | 4.7 | 2.26 (0.99–3.07) | 11 of 32 | 0.0057 |
| disordered | 22 | 8.5 (4.6–37.2) | 6.2 (3.9–15.5) | 6.0 | 2.46 (0.94–4.29) | 7 of 22 | 0.01 |
| folded | 9 | 4.1 (2.3–6.6) | 2.1 (1.2–5.3) | 2.1 | 1.53 (0.85–3.12) | 3 of 9 | 0.13 |
Reduced χ² of the uniform average of the conformer curves, with one scale factor and one constant background. Same path: the archive's conformers rebuilt from their Cα trace and passed to Pepsi-SAXS exactly as the v1.2 samples were; published curves: the archive's own. The v1.2 samples were recovered from the notebook's raw output with BioEmu's own conversion; repeating the notebook's filtered conversion reproduces its files exactly.
| class | conformers | proteins | median share rejected | mean | failing Cα–Cα | failing C–N | clashing | every sample rejected |
|---|---|---|---|---|---|---|---|---|
| folded | v1.2 | 9 | 13% | 17% | 2% | 3% | 14% | 0 |
| folded | archive | 9 | 8% | 11% | 4% | 6% | 5% | 0 |
| partly disordered | v1.2 | 1 | 37% | 37% | 16% | 17% | 23% | 0 |
| partly disordered | archive | 1 | 32% | 32% | 22% | 26% | 6% | 0 |
| disordered | v1.2 | 41 | 90% | 78% | 24% | 29% | 71% | 6 |
| disordered | archive | 41 | 40% | 48% | 32% | 39% | 17% | 2 |
The three failure columns are mean shares of samples over the proteins of the class; a sample can fail more than one check.
Reweighting keeps every conformer fixed and adjusts only how often each one occurs, until the averaged curve matches the profile. Maximum-entropy reweighting [7] does this while keeping the weights as close to uniform as the data allow; relaxing that constraint step by step traces a path from the raw ensemble towards a full fit. The effective sample fraction φ, the Kish effective sample size divided by the number of conformers, measures how much of the original ensemble still contributes after reweighting: at φ = 1 every conformer contributes equally; at φ = 0.8 the weights are still spread out; at φ = 0.1 a small subset of conformers dominates the ensemble. The four kinds are read where the path first reaches χ² ≤ 2: raw fit (already there), modestly reweightable (reached with φ ≥ 0.5), strongly reweightable (reached only with φ < 0.5, which requires substantial population changes) and target fit not reached (by the tested reweighting procedure).
What it shows. Left: the 399 profiles by kind. Right: χ² against φ along the path of SASDBT2, the strongly reweightable entry with the median φ of its kind; the open circle marks where χ² first falls to 2.
Of the 399 profiles, 146 fit without reweighting. Reweighting brings 183 more to χ² ≤ 2, and the useful distinction is between the two reweightable kinds. For the 77 modestly reweightable profiles the ensemble fits the measurement without a drastic change of populations. For the 106 strongly reweightable profiles a satisfactory fit needs the weight concentrated on a smaller set of conformers (φ < 0.5). For the remaining 70 the tested procedure does not reach the target.
What reweighting tests. Whether BioEmu-1's disagreement with a profile can be explained by wrong populations of conformations it already samples, without conformations it does not produce. A fit after reweighting shows that such populations exist; it does not show that they are the true physical probabilities.
Two cautions. Reweightability is not information gain: SAXS is low-dimensional, and distinct ensembles can give the same curve, so a reweighted fit does not identify the true populations. Not reaching the target does not establish that compatible conformations are absent from BioEmu's distribution; it describes this finite ensemble of 100 conformers, this forward model and this procedure.
| kind | definition | φ threshold 0.3 | 0.5 (used) | 0.7 |
|---|---|---|---|---|
| raw fit | raw χ² ≤ 2 | 146 | 146 | 146 |
| modestly reweightable | χ² ≤ 2 reached while keeping φ ≥ threshold | 107 | 77 | 55 |
| strongly reweightable | χ² ≤ 2 reached only with φ below the threshold | 76 | 106 | 128 |
| target fit not reached | χ² ≤ 2 not reached by the tested reweighting procedure | 70 | 70 | 70 |
The threshold moves entries between the two reweightable kinds only; the raw-fit and target-not-reached counts do not depend on it.
What it shows. Left: χ² against φ for every profile, coloured by class. Right: the share of each class in the four kinds.
An error map needs the measurement. Choosing a new experiment needs the error predicted before the measurement exists. I fitted predictors of the raw error (log₁₀ χ²) from quantities known before a profile exists: chain length, disorder score and class, sequence composition, the Rg expected from length by scaling laws [9], and the model's own ensemble Rg and conformer spread. Evaluation is 5-fold cross-validation grouped by sequence cluster, repeated five times. I then asked which entries each acquisition policy would pick from held-out folds and scored the picks by the high-discrepancy target yield: the share that are strongly reweightable or whose target fit is not reached. The yield is a proxy for where the model is wrong, not a measure of training value.
What it shows. A: observed against predicted log₁₀ raw χ² on held-out folds (ridge regression), coloured by class. B: high-discrepancy target yield among the top 10 picks; error bars are the sd over five repeats, and for random the sd of a single random pick of 10. The oracle ranking uses the measured SAXS and is shown as a ceiling; it is not a deployable policy.
How to read it. In A a predictive model would spread the points along the diagonal; here they form a vertical band. In B a policy is useful when it stands clear of the random bar and its spread.
The best model explains 5% of held-out variation in the raw error (R² = 0.05; class means alone 0.04), and the tree classifier of the high-discrepancy kinds reaches an AUC of 0.59. Random picks of ten give a yield of 0.44, with a spread of ±0.15 for any single pick. Ranking by predicted error gives 0.51 ± 0.06, +0.06 over random, within about one sd of a single random pick. The acquisition policy fed predictions, the campaign's priority rule with predicted inputs, gives 0.44 ± 0.06. The oracle ranking, which uses the measured SAXS, reaches 1.00: the rule separates the kinds when it sees the measurement, and the missing piece is a predictor of them. The map therefore supports controlled replication of known failures; it does not support a new-protein campaign driven by predicted error.
The priority score is not a basis for recommending measurements. The design proposed ranking systems by log₁₀ raw χ² × a reweighting weight × a tractability weight, with weights set by judgement. Fed the quantities known before a measurement, it picks high-discrepancy profiles at 0.44 against 0.44 for random choice, a difference inside the spread of a single random pick. I report the score and its test, and the campaign below does not rest on it. Arm 1 re-measures deposited systems, so it can use their measured discrepancy directly: it lists the systems that meet stated eligibility rules and have the largest observed raw χ², which are the same twelve the score ranks first; what is dropped is the judgement-set weighting. Arm 2 splits test proteins from controls by a quantity computed from the model and the sequence alone, within the archive's disordered entries whose deposited profiles pass the quality checks.
| model | R² (log₁₀ χ²) | MAE | AUC (high-discrepancy kind) | Brier |
|---|---|---|---|---|
| constant / base rate | -0.01 ± 0.01 | 0.494 | 0.50 ± 0.00 | 0.249 |
| class mean / class rate | 0.04 ± 0.01 | 0.476 | 0.42 ± 0.04 | 0.252 |
| ridge / logistic regression | 0.05 ± 0.01 | 0.473 | 0.53 ± 0.02 | 0.253 |
| gradient-boosted trees (depth 3) | 0.00 ± 0.01 | 0.471 | 0.59 ± 0.01 | 0.253 |
Mean ± sd over five repeats; n = 399 entries in 346 sequence groups. The feature the tree model uses most is the spread of the conformers' Rg (interquartile range over median) (importance 0.065 ± 0.047).
| policy | top 10: yield | strongly reweightable | target fit not reached | top 20: yield | strongly reweightable | target fit not reached |
|---|---|---|---|---|---|---|
| random | 0.44 ± 0.00 | 0.27 | 0.18 | 0.44 ± 0.00 | 0.27 | 0.18 |
| predicted error | 0.51 ± 0.06 | 0.34 | 0.17 | 0.51 ± 0.02 | 0.34 | 0.17 |
| acquisition policy fed predictions | 0.44 ± 0.06 | 0.23 | 0.21 | 0.47 ± 0.03 | 0.28 | 0.19 |
| oracle ranking (uses measured SAXS) | 1.00 ± 0.01 | 0.96 | 0.04 | 0.93 ± 0.01 | 0.82 | 0.11 |
Mean ± sd over five repeats of the grouped 5-fold split; random is averaged over 200 draws per fold, and one random pick of ten spreads by ±0.15. Pool shares: 44% high-discrepancy, 27% strongly reweightable, 18% target fit not reached.
What it shows. Left: calibration of the tree classifier, quintiles of predicted probability against the observed share. Right: drop in held-out R² when each feature is permuted, eight most used features.
What BioEmu-1 was trained on, what the SAXS archive covers and where the two leave gaps are the evidence for which protein data to collect. This section measures all three, relates BioEmu-1's error to closeness to its training data, and then lists the categories of proteins the evidence supports, before the two arms of the campaign.
BioEmu-1 learned in three stages [1]. Pre-training used 50,228 AlphaFold Database sequence clusters (179,422 predicted structures; the cluster definitions are published with the model), from which clusters whose structures were all more than half coil had been removed. Fine-tuning used molecular dynamics: 1,093 CATH domains of up to 199 residues, 271 MEGAscale wild types with 21,458 point mutants, 1,100 octapeptides, and published simulations of a few named proteins (DDR1 kinases, SETD8, coronavirus proteins, MHC class II, barnase–barstar). A last stage fitted measured folding stabilities of 361 MEGAscale wild-type domains and their mutants [11]. The DESRES fast-folder simulations trained a separate model only. I read the sequences of each part (the simulated topologies, the cluster members from UniProt) and described them as I described the SAXS set: CATH superfamilies, chain length, disorder predicted from sequence by one predictor for every set [12], and organism.
What it shows. A: CATH class of protein–superfamily units: SAXS constructs (CATH-Gene3D), AFDB pre-training clusters (CATH-Gene3D of the input chain, 41,488 units), simulated CATH domains (1,077) and CATH S35 clusters. B: chain length. C: predicted disorder (metapredict), share of chains mostly disordered and share with a disordered run of at least 30 residues. D: source organism; the reference is the PDB 30% clusters. MEGAscale domains are in the tables below.
How to read it. A bar of the SAXS set far from the AFDB and simulated bars marks a region the archive measures and the training data barely contain.
| set | chains | superfamilies | mainly α | mainly β | α/β | Rossmann | TIM barrel | median length | > 500 | mostly disordered | disordered run ≥ 30 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SAXS constructs | 399 | 215 | 26% | 33% | 40% | 7.4% | 0.9% | 252 | 0.0% | 15.5% | 19.1% |
| AFDB clusters | 50,228 | 6,120 | 38% | 11% | 51% | 16.0% | 1.6% | 261 | 19.5% | 1.7% | 6.7% |
| simulated CATH domains | 1,093 | 471 | 32% | 17% | 51% | 14.3% | 0.0% | 100 | 0.0% | 0.3% | 0.5% |
| MEGAscale domains | 477 | 71 | 28% | 39% | 31% | 0.0% | 0.0% | 56 | 0.0% | 6.3% | 0.0% |
| CATH S35 | 37,350 | 6,630 | 25% | 23% | 52% | 12.6% | 2.9% | 133 | 0.5% | 2.1% | 1.9% |
Class and fold shares are over protein–superfamily units (CATH S35: clusters). MEGAscale domains: the 478 curated wild types of the MEGAscale study and the MegaSim wild types, one row per distinct sequence, superfamilies from the best sequence hit in CATH; 34% of them are de novo designs. These domains were selected for folding-stability measurements, yet the predictor calls 6.3% of them mostly disordered: 22 of 317 natural (6.9%) and 8 of 160 designed domains (5.0%; Fisher p 0.55). The domains so called are shorter than the rest (median 43 residues, range 32–72, against 56; Mann–Whitney p 0.0066). A sequence-based call does not show whether a domain is disordered in solution, and these numbers do not tell whether the call reflects disorder or the short length. Octapeptides (1,100 systems, 8 residues) are counted and not compared.
| organism | SAXS | AFDB | simulated CATH | MEGAscale | PDB 30% |
|---|---|---|---|---|---|
| Human | 39.9% | 0.2% | 15.0% | 15.1% | 16.1% |
| Mouse | 3.8% | 0.1% | 2.9% | 1.7% | 2.8% |
| E. coli | 6.8% | 1.9% | 9.8% | 3.1% | 4.8% |
| Yeast | 3.5% | 0.1% | 5.9% | 1.3% | 5.8% |
| Other bacteria | 22.6% | 81.2% | 42.2% | 21.6% | 34.2% |
| Viruses | 4.8% | 0.1% | 4.5% | 2.7% | 6.5% |
| Plants | 4.3% | 2.7% | 2.3% | 1.3% | 5.7% |
| Other | 14.5% | 13.7% | 17.5% | 19.7% | 24.1% |
| Designed | 0.0% | 0.0% | 0.0% | 33.5% | 0.0% |
| SAXS-set superfamilies found in | superfamilies |
|---|---|
| AFDB | 146 of 215 |
| CATH MD | 63 of 215 |
| MEGAscale | 20 of 215 |
| MD (CATH MD and MegaSim) | 67 of 215 |
| any training component | 157 of 215 |
The other way round, the SAXS set contains 146 of the 6,120 superfamilies of the AFDB clusters (2.4%) and 63 of the 471 of the simulated domains (13%).
As the reference for experimentally known structure space I use CATH 4.4.0 at the S35 level (domains clustered at 35% sequence identity) for folds, and non-redundant PDB clusters (30% identity) and Swiss-Prot for organism, length and disorder. CATH and the PDB describe what has been solved; the comparison with the training data is above.
What it shows. A: CATH 4.4.0 superfamilies ranked by their number of S35 clusters, in bins; the bar is the share present in the SAXS set (a CATH-Gene3D domain inside a construct), with Wilson 95% intervals and counts. B: CATH class composition of the 345 unique protein–superfamily units in the set against the CATH S35 clusters, with bootstrap intervals over proteins; the few-secondary-structure class (about 1% of both) is not drawn.
How to read it. In A a set that sampled fold space evenly by cluster would show bars of similar height. In B a ratio above 1 marks a class more frequent in the SAXS set than among solved domains.
What it shows. A: source organism of the SAXS profiles (with Wilson 95% intervals), of PDB 30% clusters of chains of at least 30 residues, and of Swiss-Prot entries. B: CATH-Gene3D domains per protein in the SAXS construct, in the full-length UniProt protein it was cut from, and in Swiss-Prot proteins.
Interpretation. Set against the measured composition of the training data, three of the archive's skews fall where the training data are thinnest: disorder (1.7% of AFDB input chains and 0.3% of simulated domains predicted mostly disordered, against 15.5% of the constructs), human proteins (0.2% of AFDB clusters, 15% of simulated domains, 40% of the profiles) and mainly-β domains (11% of AFDB units, 17% of simulated ones, 33% of the set). The depletion of α/β enzyme folds is a gap of the archive only: the Rossmann fold is 16% of AFDB units and 14% of simulated ones, above its CATH share. Long chains are present in pre-training as single predicted structures (20% of AFDB inputs above 500 residues) and absent from both the simulations and the archive.
What it shows. Ratio of the SAXS share to the CATH S35 share for each architecture with a CATH share of at least 1% or at least three units in the set; units in brackets; intervals from the bootstrap over proteins.
| CATH class | SAXS units | SAXS share (95%) | CATH S35 clusters | CATH share | ratio (95%) |
|---|---|---|---|---|---|
| Mainly alpha | 89 | 26% (21%–31%) | 9,071 | 25% | 1.04 (0.85–1.23) |
| Mainly beta | 115 | 33% (28%–39%) | 8,232 | 23% | 1.48 (1.26–1.71) |
| Alpha beta | 137 | 40% (35%–45%) | 18,814 | 52% | 0.77 (0.67–0.87) |
| Few secondary structures | 4 | 1% (0%–2%) | 411 | 1% | 1.03 (0.25–2.14) |
Intervals: bootstrap over UniProt accessions (2,000 resamples). 345 units in classes 1–4 from 251 proteins.
Units: 351 unique pairs of UniProt accession and CATH-Gene3D superfamily inside the constructs; expected = units × the CATH S35 share; the Poisson tails are uncorrected and serve for ranking only.
Superfamilies, over-represented
| CATH id | name | observed | expected | CATH S35 clusters | Poisson p |
|---|---|---|---|---|---|
| 2.30.30.40 | SH3 Domains | 7 | 1.4 | 150 | < 0.001 |
| 1.20.58.60 | unnamed | 4 | 0.4 | 41 | < 0.001 |
| 1.10.8.50 | unnamed | 3 | 0.2 | 19 | < 0.001 |
| 1.10.238.10 | EF-hand | 7 | 1.8 | 194 | 0.0028 |
| 1.10.510.10 | Transferase(Phosphotransferase) domain 1 | 6 | 1.6 | 166 | 0.0054 |
| 1.10.8.10 | Ubiquitin-associated (UBA) domain | 4 | 0.7 | 76 | 0.0062 |
| 2.60.40.790 | unnamed | 3 | 0.4 | 40 | 0.0067 |
| 2.30.29.30 | Pleckstrin-homology domain (PH domain)/Phosphotyrosine-binding domain (PTB) | 6 | 1.7 | 178 | 0.0074 |
Superfamilies, under-represented
| CATH id | name | observed | expected | CATH S35 clusters | Poisson p |
|---|---|---|---|---|---|
| 3.40.50.720 | NAD(P)-binding Rossmann-like Domain | 0 | 6.4 | 685 | 0.0016 |
| 3.40.50.150 | Vaccinia Virus protein VP39 | 0 | 2.9 | 311 | 0.05 |
| 1.10.10.10 | Winged helix-like DNA-binding domain superfamily/Winged helix DNA-binding domain | 2 | 5.6 | 598 | 0.08 |
| 3.40.50.2300 | Response regulator | 1 | 4.1 | 434 | 0.09 |
| 2.60.120.10 | Jelly Rolls | 0 | 2.0 | 212 | 0.14 |
| 3.40.630.30 | Gcn5-related N-acetyltransferase (GNAT) | 0 | 1.9 | 206 | 0.14 |
| 3.90.1150.10 | Aspartate Aminotransferase, domain 1 | 0 | 1.9 | 204 | 0.15 |
| 3.40.640.10 | Type I PLP-dependent aspartate aminotransferase-like (Major domain) | 0 | 1.8 | 188 | 0.17 |
Topologies, over-represented
| CATH id | name | observed | expected | CATH S35 clusters | Poisson p |
|---|---|---|---|---|---|
| 2.70.50 | Coagulation Factor XIII; Chain A, domain 1 | 3 | 0.3 | 28 | 0.0025 |
| 1.10.510 | Transferase(Phosphotransferase); domain 1 | 6 | 1.6 | 167 | 0.0055 |
| 2.60.40 | Immunoglobulin-like | 35 | 22.1 | 2,348 | 0.0066 |
| 1.10.238 | Recoverin; domain 1 | 7 | 2.3 | 242 | 0.0088 |
| 2.30.30 | SH3 type barrels. | 11 | 4.9 | 522 | 0.01 |
| 1.10.8 | Helicase, Ruva Protein; domain 3 | 8 | 3.1 | 331 | 0.01 |
| 2.30.29 | PH-domain like | 6 | 1.9 | 207 | 0.01 |
| 3.30.505 | SHC Adaptor Protein | 3 | 0.7 | 73 | 0.03 |
Topologies, under-represented
| CATH id | name | observed | expected | CATH S35 clusters | Poisson p |
|---|---|---|---|---|---|
| 3.40.50 | Rossmann fold | 26 | 44.1 | 4,698 | 0.0022 |
| 3.20.20 | TIM Barrel | 3 | 10.0 | 1,064 | 0.01 |
| 3.30.450 | Beta-Lactamase | 0 | 2.8 | 302 | 0.06 |
| 3.90.1150 | Aspartate Aminotransferase, domain 1 | 0 | 2.2 | 235 | 0.11 |
| 6.10.250 | Single alpha-helices involved in coiled-coils or other helix-helix interfaces | 1 | 3.4 | 359 | 0.15 |
| 3.40.640 | Aspartate Aminotransferase; domain 2 | 0 | 1.8 | 188 | 0.17 |
| 1.20.5 | Single alpha-helices involved in coiled-coils or other helix-helix interfaces | 3 | 5.5 | 581 | 0.21 |
| 1.10.357 | Tetracycline Repressor; domain 2 | 0 | 1.6 | 166 | 0.21 |
| chain length (residues) | 10th | 25th | median | 75th | 90th |
|---|---|---|---|---|---|
| SAXS constructs | 98 | 154 | 252 | 357 | 427 |
| their full-length proteins | 160 | 290 | 448 | 854 | 1688 |
| PDB 30% clusters, ≥ 30 residues | 74 | 121 | 212 | 357 | 540 |
| Swiss-Prot | 100 | 169 | 296 | 451 | 665 |
| UniProt "Disordered" annotation | SAXS constructs | full-length proteins | Swiss-Prot | Swiss-Prot with a PDB entry |
|---|---|---|---|---|
| ≥ 50% of residues disordered | 6% | 4% | 1.6% | 2% |
| a disordered region of ≥ 30 residues, under 50% | 13% | 41% | 12.3% | 22% |
UniProt REGION features with the note "Disordered", mostly MobiDB-lite predictions.
| organism | SAXS entries | SAXS share | PDB 30% clusters | Swiss-Prot |
|---|---|---|---|---|
| Human | 159 | 40% | 16% | 3.5% |
| Mouse | 15 | 4% | 3% | 3.0% |
| E. coli | 27 | 7% | 5% | 4.0% |
| Yeast | 14 | 4% | 6% | 2.3% |
| Other bacteria | 90 | 23% | 34% | 54.5% |
| Viruses | 19 | 5% | 7% | 3.0% |
| Plants | 17 | 4% | 6% | 7.3% |
| Other | 58 | 15% | 24% | 22.2% |
37,352 PDB clusters of chains of at least 30 residues with an experimental entity; 575,748 Swiss-Prot entries.
390 of 399 profiles map to a UniProtKB entry: 353 through the accession in the SASBDB record, 23 through a PDB sequence search (identity ≥ 90%) and the SIFTS reference of the hit, and 14 through a UniProt peptide search; a local alignment to the canonical sequence then defines the construct region (median identity 1.00). For 17 profiles the SASBDB accession was set aside because it pointed to a UniProtKB entry deleted since deposition or aligned to a small part of the construct; all but 1 of them were mapped again by sequence search. The 9 unmapped profiles are SASDAT3 (immunoglobulin- like filamin two-domain …), SASDCV7 (TET12(2.3)SN-f5b), SASDDL7 (anti-DISC1 single-domain camelid antibody VHH …), SASDKX6 (PSK, an antimicrobial peptide from Chrysomya …), SASDM97 (Recombinant monoclonal anti-proNGF antibody …), SASDMT7 (Lanthionine synthetase C-like protein), SASDNW2 (Fab fragment in complex with small molecule …), SASDS37 (Harpin Z2), SASDSV6 (Anti-Idiotypic-Palivizumab-Nanobody1). Domains are CATH-Gene3D assignments whose residues lie at least half inside the construct region; PDB coverage counts residues of atomic-model chains listed by SIFTS.
| source | version |
|---|---|
| SASBDB | REST API |
| UniProtKB | 2026_03 |
| UniProtKB/Swiss-Prot flat file | UniProtKB/Swiss-Prot Release 2026_03 of 02-Sep-2026 |
| RCSB clusters-by-entity-30 | Sun, 04 Oct 2026 02:09:51 GMT |
| InterPro:interpro | 110.0 |
| InterPro:cathgene3d | 4.3.0 |
| InterPro:pfam | 38.2 |
| PDBe SIFTS best_structures | live API (weekly SIFTS update) |
| TED | TED v1 (AFDB v4 models, CATH labels) |
| CATH | v4.4.0 |
| BioEmu AFDB cluster definitions | file added 2025-04-25 |
| Zenodo 15629740 | published 2025-06-10 |
| UniProtKB (AFDB cluster members) | 2026_03 |
| Zenodo 15641199 | published 2025-06-11 |
| Zenodo 15641184 | published 2025-06-11 |
| Zenodo 7992926 | v2_230420 |
| MEGAscale (Tsuboyama et al. 2023) | REST API |
| Other public MD sets (named proteins) | REST API |
| UniParc (AFDB cluster members deleted from UniProtKB) | REST API |
| DESRES fast-folder sequences | REST API |
| metapredict | 3.1.1 |
| MMseqs2 | release 18-8cc5c |
| UniProtKB (AFDB cluster members): counts | REST API |
CATH v4.4.0: 601,328 domains, 37,350 S35 clusters, 6,630 superfamilies, 1,472 topologies, 43 architectures. CATH-Gene3D in InterPro is release 4.3.0; every superfamily it assigns here is present in v4.4.0.
For each construct I searched every part of the training data for the most similar sequence over a local alignment that covers at least half of the construct [13], and checked whether the construct's CATH superfamilies occur there. BioEmu-1's training data were filtered against the paper's own test proteins only, so overlap with these proteins was possible.
What it shows. A: share of the 399 profiles whose nearest training sequence (local alignment covering at least half of the construct) reaches each identity, for all training data, the AFDB cluster members and the simulated systems; constructs without such an alignment count as below every threshold. B: the profiles by identity bin and disorder class.
How to read it. A curve that stays high to the right would mean many measured proteins were nearly in the training data; at 50% identity the share is below one in five.
Pooled, the raw fit improves with closeness: χ² ≤ 2 is met by 32% of profiles below 30% identity and 52% at 90% or more (Spearman ρ +0.11, p 0.02), and the Rg ratio falls slightly (ρ -0.12, p 0.01); raw χ² itself does not reach significance (p 0.08). The bins differ in disorder class: 32 of the 234 profiles below 30% are disordered and 2 of the 23 at 90% or more. Within classes no measure varies with closeness: across the 12 within-class Spearman tests (three classes, raw χ², raw fit, NRMSD and Rg ratio) the smallest nominal p is 0.03 (disordered, NRMSD) and every Holm-adjusted p is at least 0.35; a rank test stratified by class finds no difference in raw χ² between constructs with a training neighbour of at least 50% identity and the rest (van Elteren p 0.17). Superfamily presence does not separate the errors either (pooled Mann–Whitney p 0.97; restricted to simulated superfamilies p 0.13, stratified by class p 0.12).
In this archive, closeness to the training data does not explain BioEmu-1's raw error beyond the disorder class. The over-extended disordered ensembles are also the proteins furthest from the training data, and the archive cannot separate the two: only 9 disordered constructs have a training neighbour at 30% identity or more.
What it shows. One point per profile: raw reduced χ² (top, log scale) and the ratio of ensemble Rg to measured Rg (bottom), by the identity of the nearest training sequence, within each disorder class. Bars: group medians (groups of three or more); dashed lines χ² = 2 and ratio 1.
How to read it. If closeness to the training data mattered, the points would step down (top) or towards 1 (bottom) from left to right within a class.
| nearest training identity | n | χ² ≤ 2 (95%) | median raw χ² | median Rg ratio |
|---|---|---|---|---|
| all classes | ||||
| below 30% | 234 | 32% (27%–39%) | 3.8 | 0.99 |
| 30–50% | 99 | 37% (28%–47%) | 3.2 | 0.98 |
| 50–90% | 43 | 49% (35%–63%) | 2.5 | 0.97 |
| 90% or more | 23 | 52% (33%–71%) | 1.8 | 0.99 |
| folded · Spearman with identity: χ² p 0.59, Rg ratio p 0.99 (Holm 1.00, 1.00) | ||||
| below 30% | 127 | 45% (37%–54%) | 2.3 | 0.98 |
| 30–50% | 70 | 44% (33%–56%) | 2.4 | 0.99 |
| 50–90% | 32 | 53% (36%–69%) | 1.9 | 0.96 |
| 90% or more | 17 | 53% (31%–74%) | 1.8 | 0.99 |
| partly disordered · Spearman with identity: χ² p 0.53, Rg ratio p 0.34 (Holm 1.00, 1.00) | ||||
| below 30% | 75 | 20% (13%–30%) | 5.5 | 0.99 |
| 30–50% | 23 | 26% (13%–46%) | 3.6 | 0.96 |
| 50–90% | 10 | 40% (17%–69%) | 2.5 | 0.99 |
| 90% or more | 4 | 50% (15%–85%) | 3.3 | 1.10 |
| disordered · Spearman with identity: χ² p 0.42, Rg ratio p 0.05 (Holm 1.00, 0.59) | ||||
| below 30% | 32 | 12% (5%–28%) | 12.1 | 1.23 |
| 30–50% | 6 | 0% (0%–39%) | 5.7 | 1.04 |
| 50–90% | 1 | 0% (0%–79%) | 9.4 | 1.12 |
| 90% or more | 2 | 50% (9%–91%) | 3.5 | 1.07 |
| superfamily in | group | n | χ² ≤ 2 | median raw χ² | median Rg ratio |
|---|---|---|---|---|---|
| any part of the training data | superfamily present | 254 | 40% | 2.9 | 0.98 |
| any part of the training data | superfamily absent | 35 | 34% | 3.4 | 0.99 |
| any part of the training data | no CATH superfamily | 110 | 29% | 4.6 | 1.01 |
| the simulated domains | superfamily present | 143 | 45% | 2.4 | 0.98 |
| the simulated domains | superfamily absent | 146 | 34% | 3.5 | 0.98 |
| the simulated domains | no CATH superfamily | 110 | 29% | 4.6 | 1.01 |
Present against absent, two-sided Mann–Whitney on raw χ², NRMSD and Rg ratio, pooled and within each class (Holm over the within-class tests with both groups present).
| SASBDB | residues | class | nearest training set | sequence | identity | construct covered |
|---|---|---|---|---|---|---|
| SASDAQ2 | 76 | folded | AFDB cluster member | A0A4W5NE66 | 100% | 100% |
| SASDBN2 | 152 | folded | AFDB cluster member | A0A4W5NE66 | 100% | 50% |
| SASDEK7 | 171 | partly disordered | AFDB cluster member | A0A735B4C1 | 100% | 93% |
| SASDJG4 | 261 | folded | named MD protein | SARS-CoV-2 | 100% | 95% |
| SASDVL5 | 62 | disordered | AFDB cluster member | A0A803KJN6 | 100% | 100% |
| SASDSZ9 | 80 | disordered | named MD protein | SARS-CoV-2 | 100% | 88% |
| SASDN65 | 197 | folded | AFDB cluster member | A0A2J8Q1C0 | 100% | 75% |
| SASDPY2 | 283 | folded | named MD protein | SARS-CoV-2 | 100% | 70% |
| SASDR33 | 138 | folded | named MD protein | SARS-CoV-2 | 100% | 99% |
| SASDJY2 | 182 | folded | AFDB cluster member | P24200 | 100% | 96% |
| SASDRH9 | 214 | partly disordered | AFDB cluster member | A0A2K6BBN2 | 100% | 73% |
| SASDCK7 | 347 | folded | AFDB cluster member | O08967 | 99% | 97% |
| SASDC88 | 155 | folded | AFDB cluster member | A0A735B4C1 | 99% | 94% |
| SASDBP2 | 228 | folded | AFDB cluster member | Q9XXZ6 | 99% | 100% |
| SASDMS4 | 320 | folded | AFDB cluster member | A0A825WEZ1 | 99% | 100% |
| SASDSV9 | 255 | partly disordered | named MD protein | SARS-CoV-2 | 99% | 96% |
| SASDCK3 | 451 | folded | AFDB cluster member | F5RY53 | 98% | 99% |
| SASDCD3 | 275 | folded | AFDB cluster member | A0A6P6CPY3 | 97% | 54% |
| SASDP28 | 240 | folded | AFDB cluster member | A0A091H9H1 | 96% | 81% |
| SASDVT4 | 365 | partly disordered | AFDB cluster member | A0A091U471 | 96% | 63% |
| SASDRK7 | 292 | folded | AFDB cluster member | Q9XXZ6 | 95% | 100% |
| SASDVR6 | 297 | folded | AFDB cluster member | A0A5N4EKN6 | 94% | 93% |
| SASDKK9 | 289 | folded | AFDB cluster member | A0A1J3I1H7 | 94% | 100% |
Profiles with a training sequence of at least 90% identity over at least half of the construct. AFDB sequences are named by UniProt accession; the named MD proteins are full UniProt chains, so their match can extend beyond what was simulated.
Of the 390 mapped profiles, 290 have an experimental structure covering at least half of the construct, 25 a partial one and 75 none. Pooled, BioEmu-1's raw ensemble meets χ² ≤ 2 for 41% (35%–46%) of the first group, 20% (9%–39%) of the second and 28% (19%–39%) of the third, with median raw χ² 2.8, 8.9 and 4.1 (at least half covered against the rest, two-sided Mann–Whitney p 0.01). The groups differ in disorder: a structure covers at least half of the construct for 83% of folded, 69% of partly disordered and 39% of disordered profiles. Within classes the difference vanishes for folded (p 0.35) and partly disordered proteins (p 0.31; here the median χ² is higher with a structure, 5.3 against 3.0), and a rank test stratified by class finds no difference in raw χ² (van Elteren p 0.39) or in the Rg ratio (p 0.38). NRMSD does not differ even pooled (p 0.54).
Among disordered proteins, raw χ² is lower when a structure exists (median 5.0 against 12.4, p 0.04, n = 16 against 25), but the difference does not survive a Holm correction over the six within-class tests (p 0.22) and is absent under NRMSD (p 0.86). At the median these ensembles remain larger than measured: Rg ratio 1.15 with a structure (95% interval 0.97–1.25, which reaches 1) and 1.22 (1.12–1.39) without; the two do not differ (p 0.18). Whether the construct has an experimental structure does not explain BioEmu-1's raw error beyond what the disorder class explains.
| construct structure | n | χ² ≤ 2 (95%) | median raw χ² (95%) | median NRMSD | median Rg ratio (95%) | target fit not reached |
|---|---|---|---|---|---|---|
| all classes | ||||||
| PDB ≥ 50% | 290 | 41% (35%–46%) | 2.8 (2.3–3.6) | 0.030 | 0.98 (0.98–0.99) | 15% |
| partial | 25 | 20% (9%–39%) | 8.9 (4.8–26.9) | 0.037 | 1.06 (1.01–1.30) | 36% |
| no PDB | 75 | 28% (19%–39%) | 4.1 (2.5–5.9) | 0.032 | 0.99 (0.97–1.04) | 17% |
| folded · Mann–Whitney, PDB ≥ 50% against the rest: χ² p 0.35, Rg ratio p 0.40 | ||||||
| PDB ≥ 50% | 200 | 48% (41%–55%) | 2.2 (1.8–2.8) | 0.030 | 0.99 (0.98–0.99) | 16% |
| partial | 8 | 38% (14%–69%) | 7.5 (1.1–9.6) | 0.040 | 1.01 (0.92–1.06) | 50% |
| no PDB | 33 | 42% (27%–59%) | 3.1 (1.4–5.3) | 0.030 | 0.96 (0.94–0.99) | 24% |
| partly disordered · Mann–Whitney, PDB ≥ 50% against the rest: χ² p 0.31, Rg ratio p 0.11 | ||||||
| PDB ≥ 50% | 74 | 26% (17%–37%) | 5.3 (3.0–9.4) | 0.032 | 0.97 (0.95–0.99) | 15% |
| partial | 6 | 33% (10%–70%) | 9.2 (1.2–26.8) | 0.055 | 1.17 (0.91–1.49) | 17% |
| no PDB | 28 | 18% (8%–36%) | 3.0 (2.4–5.1) | 0.034 | 1.05 (0.97–1.09) | 14% |
| disordered · Mann–Whitney, PDB ≥ 50% against the rest: χ² p 0.04, Rg ratio p 0.18 | ||||||
| PDB ≥ 50% | 16 | 19% (7%–43%) | 5.0 (3.3–8.5) | 0.027 | 1.15 (0.97–1.25) | 0% |
| partial | 11 | 0% (0%–26%) | 26.9 (5.4–37.1) | 0.031 | 1.28 (1.09–1.44) | 36% |
| no PDB | 14 | 14% (4%–40%) | 12.1 (5.6–13.5) | 0.033 | 1.21 (1.02–1.38) | 7% |
PDB ≥ 50%: experimental chains (X-ray, electron microscopy or NMR) cover at least half of the construct; partial: some overlap, under half; no PDB: no overlap. Shares: Wilson 95% intervals; medians: percentile bootstrap over entries (2,000 resamples). 9 profiles without a UniProt mapping are left out.
What it shows. One point per profile, raw reduced χ² of the BioEmu-1 ensemble (log scale), by disorder class and by PDB coverage of the construct: at least half (filled), partial (light) and none (open). Bars: group median and its 95% bootstrap interval (groups of five or more). The p value above each class compares at least half covered against the rest.
How to read it. Within a class, a solved structure would help if the filled group sat lower than the other two.
Each row joins three measurements: how much of the category the training data contain, how much the archive measures, and BioEmu-1's raw error on it here. A category is kept when the training data are thin in it and either the error on it exceeds the error on the rest of the archive or the archive holds no measurement of it. The order is my judgement from that evidence: the first row combines the thinnest training data with the largest error and a direction SAXS reads directly; the third is a gap in both the simulations and the archive, so its error is unknown. I give no numeric priority: the heuristic priority score tested above picked high-discrepancy profiles no better than random choice.
| protein category | in BioEmu-1's training data | in the SAXS archive | BioEmu-1's raw SAXS error here | what SAXS shows | feasibility |
|---|---|---|---|---|---|
| Disordered proteins (at least half of the residues predicted disordered) | thin: 1.7% of AFDB inputs, 0.3% of simulated domains | 62 profiles (16%) | 13% meet χ² ≤ 2 (rest 41%); Rg ratio 1.17 (rest 0.98); n = 62 | the size error: ensembles too extended, read from the Guinier region | usually soluble and monomeric; low mass gives weak scattering, so a concentration series matters |
| Folded proteins with a long disordered region (a run of at least 30 residues, under half the chain) | thin: 6.7% of AFDB inputs, 0.5% of simulated domains | 76 profiles (19%) | 29% meet χ² ≤ 2 (rest 38%); Rg ratio 0.99 (rest 0.99); n = 76 | a shape error (raw χ² higher, Mann–Whitney p 0.0071) with the size close to measured | tails and linkers are prone to proteolysis; intact mass by SDS-PAGE or mass spectrometry before measuring |
| Full-length multidomain proteins above 500 residues | as predicted structures only: 20% of AFDB inputs exceed 500 residues, 10% carry two or more superfamilies; no simulated chain exceeds 199 residues | none: the longest construct has 496 residues | not measured: no profile in the archive | domain arrangement and interdomain flexibility, at low resolution | expression and stability of long constructs; size-exclusion-coupled SAXS to separate aggregates |
Disorder of the training chains and of the constructs from the same sequence predictor (metapredict); "rest" is every other clean profile. Feasibility notes are judgement.
Four candidates from the coverage analysis are not supported: α/β enzyme folds (Rossmann, TIM barrel), bacteria other than E. coli, superfamilies beyond the 200 most populated, proteins with no experimental structure. Either the training data are not thin in them (the Rossmann fold is more frequent in pre-training and simulation than in CATH; 81% of the pre-training clusters are bacterial) or BioEmu-1's raw error on them matches or is below the error on the rest of the archive. Human proteins and mainly-β domains are thin in pre-training, but the error on them matches the rest; so does the error on constructs with two or more domains up to 500 residues.
| category | SAXS profiles | χ² ≤ 2, in / rest | median Rg ratio | raw χ², Mann–Whitney p |
|---|---|---|---|---|
| α/β enzyme folds (Rossmann, TIM barrel) | 31 | 39% / 36% | 0.99 | 0.99 |
| bacteria other than E. coli | 90 | 42% / 35% | 0.98 | 0.01 |
| superfamilies beyond the 200 most populated | 130 | 32% / 39% | 0.98 | 0.74 |
| proteins with no experimental structure | 75 | 28% / 39% | 0.99 | 0.34 |
| human proteins | 159 | 38% / 35% | 0.99 | 0.78 |
| mainly-β domains | 114 | 35% / 37% | 0.97 | 0.73 |
| constructs with two or more domains (up to 500 residues) | 81 | 37% / 36% | 0.98 | 0.95 |
Training side of the dropped categories: Rossmann folds are 16% of AFDB units and 14% of simulated ones against 13% of CATH; 81% of AFDB clusters are from bacteria other than E. coli; superfamilies beyond the 200 most populated are 27% of AFDB units and 48% of simulated ones (47% of CATH); the AFDB clusters need no experimental structure; human proteins are 0.2% of AFDB clusters and 15% of the simulated domains; mainly-β domains are 11% of AFDB units and 17% of simulated ones (23% of CATH); 10% of AFDB inputs and no simulated domain carry two or more superfamilies.
Feasibility · disordered and full-length proteins
How such data would be used. BioEmu-1's last training stage fitted measured folding stabilities by property-prediction fine-tuning: a foldedness observable computed from sampled structures is matched to the measurement by back-propagating through the sampler [1]. A SAXS observable could enter the same way, as the radius of gyration of the sampled conformers or, through a differentiable forward model, as the scattering curve. Reweighting the ensemble toward a measured profile, as in the reweighting section, gives a second route: the reweighted conformers serve as conformer-level targets. Neither route was tried in this study; the rows above are where either would add information the training data lack.
I simulated 40 of the 41 disordered proteins whose profiles pass the quality checks with CALVADOS 3 [18, 16], a model with one bead per residue whose parameters were fitted to SAXS radii and NMR data of disordered proteins, each at the temperature and pH recorded with its profile and an ionic strength read from its buffer. The longest chain (475 residues) was left out for computing time, and the next 3 longest (404–439 residues) were run at half length. The ratio of the simulated radius of gyration to the measured one has a median of 1.04 (95% interval 1.01–1.10), against 1.21 (1.13–1.30) for BioEmu-1 on the same proteins; the CALVADOS 3 ratio is the closer to 1 for 33 of 40 proteins (sign test p < 0.001).
What it shows. Each line joins one protein's BioEmu-1 ratio (left) to its CALVADOS 3 ratio (right), 40 disordered proteins, log scale. Open circles: proteins at least 90% identical to a protein CALVADOS was trained or validated on. Bars: medians with 95% bootstrap intervals; the band marks 0.90–1.10.
How to read it. A protein on the dashed line has the measured size. Lines that slope towards the band mark proteins where the simulation is the closer.
| expectation, fixed before the runs | reasoning | outcome |
|---|---|---|
| E10. CALVADOS 3's median Rg ratio lies within 0.90–1.10. | The model's parameters were fitted to SAXS radii of disordered proteins. | met median 1.04, 1.01–1.10 |
| E11. CALVADOS 3's ratio is closer to 1 than BioEmu-1's for most proteins (sign test p < 0.05). | BioEmu-1's training data hold almost no disordered chains, and its disordered ensembles are too extended. | met 33 of 40, p < 0.001 |
Circularity. 6 of the 40 simulated proteins are at least 90% identical to a protein CALVADOS was trained on, and 5 more to one it was validated on. Without the 6 the median ratio is 1.05 (1.00–1.11, n = 34; E10 met, E11 met), and without all 11 it is 1.03 (0.99–1.11, n = 29; E10 met, E11 met).
For collecting data. Simulation with a model fitted to existing SAXS data puts 22 of these 40 disordered proteins within 10% of the measured size, against 6 for BioEmu-1, at a median of 1.3 CPU hours per protein, so measured profiles of disordered proteins are most needed for the others, which it misses by up to a factor of 2.2, and for chains longer than the 439 residues simulated here.
| proteins | n | CALVADOS 3 median ratio (95%) | BioEmu-1 median ratio (95%) | CALVADOS 3 closer to 1 | sign test p | E10 | E11 |
|---|---|---|---|---|---|---|---|
| all simulated proteins | 40 | 1.04 (1.01–1.10) | 1.21 (1.13–1.30) | 33 of 40 | < 0.001 | met | met |
| without proteins in CALVADOS training sets | 34 | 1.05 (1.00–1.11) | 1.21 (1.12–1.34) | 28 of 34 | < 0.001 | met | met |
| without proteins in CALVADOS training or validation sets | 29 | 1.03 (0.99–1.11) | 1.19 (1.02–1.25) | 23 of 29 | 0.0023 | met | met |
| without profiles measured in urea | 36 | 1.06 (1.03–1.10) | 1.23 (1.18–1.36) | 30 of 36 | < 0.001 | met | met |
| runs that pass the convergence check | 34 | 1.04 (0.99–1.09) | 1.21 (1.12–1.29) | 29 of 34 | < 0.001 | met | met |
| the same comparison with like observables | |||||||
| bead radius against BioEmu-1's Cα radius | 40 | 1.04 (1.01–1.10) | 1.36 (1.23–1.47) | 33 of 40 | < 0.001 | met | met |
| both radii from Guinier fits of computed curves | 40 | 0.99 (0.98–1.04) | 1.21 (1.13–1.30) | 32 of 40 | < 0.001 | met | met |
CALVADOS 3 ratio: root-mean-square radius of gyration of the residue beads (mass-weighted) over the production frames, divided by the Guinier radius of the measured profile. BioEmu-1 ratio: the Guinier radius of its ensemble curve over the same measured radius, as in the size section. Intervals: percentile bootstrap over proteins (2,000 resamples).
Observables. The measured radius comes from a Guinier fit up to q·Rg = 1.3; applied to the scattering curve of an ideal random chain, the same fit returns 0.92 times its true radius. A bead radius carries no such offset, so the primary comparison can place a correct chain model above 1 by up to that factor. When both radii come from Guinier fits of computed curves, the CALVADOS 3 median is 0.99 (0.98–1.04).
Scattering curves. Each ensemble was reduced to 100 evenly spaced frames, rebuilt to all atoms with PULCHRA [19] and passed to Pepsi-SAXS [6] with the settings used for the archive's curves, then scored as BioEmu-1's raw ensembles are. Median reduced χ² 2.7 for CALVADOS 3 against 9.0 for BioEmu-1; χ² ≤ 2 for 42% against 12%; CALVADOS 3 has the lower χ² for 30 of 40 (sign test p 0.0022). Back-mapped coarse-grained frames carry reconstruction error that all-atom conformers do not, so these χ² values are indicative.
Runs and conditions. 34 of 40 runs pass the convergence check (block standard error under 2% of the radius and halves within 5%). Temperatures 4–25 °C, ionic strengths 0.02–0.54 M; 4 profiles were measured with urea, which the model does not represent. The runs took 118 single-thread CPU hours in total, the pilot of the chain left out included.
The error map identifies where BioEmu-1 disagrees with deposited data; it cannot yet say where it will disagree on a new protein. The campaign has two arms that follow from that. Arm 2 tests the size bias in the category this analysis ranks first, disordered proteins; Arm 1 checks that deposited profiles reproduce under controlled conditions, which any profile needs before it can serve as a training label. The full proposal gives constructs, assay, scripted quality criteria, success criteria, contingencies and work packages for an external provider.
High-error, reweightable monomers from the archive are re-measured by SEC-SAXS in one standard buffer and temperature, with tags removed and a concentration series. Standardised measurement reduces condition heterogeneity and tests whether the discrepancy is reproducible. One common buffer and temperature can itself shift some ensembles, so each new profile is compared with the deposited one before it is compared with the model. The five eligible systems with the largest raw χ²:
| SASBDB | length | class | raw χ² | kind | model Rg | deposited study |
|---|---|---|---|---|---|---|
| SASDAG8 | 189 | folded | 156.2 | strongly reweightable | too compact | Flexible Structure of Peptide-Bound Filamin A … |
| SASDJ84 | 215 | folded | 112.2 | strongly reweightable | too compact | The extracellular juncture domains in the … |
| SASDCP8 | 211 | folded | 96.2 | strongly reweightable | too compact | The SAXS Analysis of the human GTPase Rab1a |
| SASDPA8 | 200 | folded | 77.1 | strongly reweightable | too compact | Structural plasticity of NFU1 upon interaction … |
| SASDFK8 | 114 | partly disordered | 62.1 | strongly reweightable | too extended | Defects in t6A tRNA modification due … |
Eligible: reweighting reaches χ² ≤ 2, measured mass matches a monomer, at most 350 residues, folded or partly disordered (101 entries). Listed by raw χ², largest first: the order sorts by the size of the known discrepancy, which is what a replication tests, and is not an estimate of what each measurement is worth; the final choice needs a feasibility review. The first 12 are in the campaign proposal.
The hypothesis came from this archive, so the arm is a prospective replication test with pre-specified model-based selection and new SAXS measurements. Test proteins are disordered proteins whose BioEmu-1 ensemble Rg exceeds the scaling-law Rg for disordered chains (1.927 N0.598 Å [9]) by more than 10%, a quantity computed from the model and the sequence alone, drawn from the archive's disordered entries whose deposited profiles pass the quality checks, spanning chain length and net charge (19 of 41 disordered entries qualify). The scaling-law-matched controls are disordered proteins of the same length bins whose ensemble Rg is within 10% of the scaling law (13 qualify). In the archive their ratio of ensemble Rg to measured Rg is 1.19 at the median, against 1.36 for the test group, so the controls are not known to agree with experiment. Pre-specified outcome: the median Rgexp/Rgmodel across the test arm from Guinier fits of the new profiles. The bias is refuted if that median is at least 0.95; it is confirmed if it is at most 0.90 in the test arm and at least 0.95 in the controls.
| SASBDB | length | net charge / residue | charged fraction | ensemble Rg (Å) | scaling-law Rg (Å) | model / law | model / measured (archive) |
|---|---|---|---|---|---|---|---|
| SASDJG6 | 40 | -0.15 | 0.20 | 22.9 | 17.5 | 1.31 | 2.43 |
| SASDRF8 | 89 | -0.02 | 0.29 | 32.5 | 28.2 | 1.15 | 1.44 |
| SASDUL9 | 76 | +0.09 | 0.35 | 28.5 | 25.7 | 1.11 | 1.12 |
| SASDLF9 | 133 | -0.13 | 0.32 | 46.3 | 35.9 | 1.29 | 1.40 |
| SASDJJ6 | 191 | +0.00 | 0.20 | 49.3 | 44.6 | 1.11 | 1.45 |
| SASDV46 | 130 | +0.06 | 0.23 | 42.8 | 35.4 | 1.21 | 1.26 |
| SASDJK6 | 209 | -0.01 | 0.17 | 52.7 | 47.0 | 1.12 | 1.38 |
| SASDF34 | 271 | +0.00 | 0.21 | 61.1 | 54.9 | 1.11 | 1.32 |
| SASDLS4 | 255 | +0.07 | 0.24 | 65.6 | 53.0 | 1.24 | 1.40 |
Controls
Feasibility judgement
Both arms use size-exclusion-coupled SAXS at three concentrations with a protein standard in every session, and eight scripted quality criteria (Guinier linearity, no low-angle upturn and Rg agreeing across concentrations, no radiation damage across frames, molecular weight from I(0) within 20% of the sequence mass, no negative intensities, a repeated deposited entry reproduced at χ² < 2, I(0)/c constant across concentrations, and a constant Rg across the SEC peak). Entries whose target fit is not reached get hydrogen–deuterium exchange mass spectrometry on the same batch of protein. Every batch of twelve constructs re-runs this pipeline, so the error map is updated before the next batch is chosen.
The archive carries computed curves from other models for the same entries, scored here identically and without reweighting: the ensemble generators Boltz-2, IDPFold, idpSAM, PepTron, and the single-structure predictors AlphaFold2 and ESMFold as reference baselines. On the shared entries BioEmu-1 has the lowest median raw χ² of the seven, and the ordering is kept under NRMSD. PepTron comes closest: it beats BioEmu-1 on 40% of the 399 shared entries (95% interval 36%–45%). BioEmu-1 is the only tested model with a disordered median Rg ratio above 1.1; PepTron sits at 1.01.
What it shows. Left: median raw reduced χ² of each model's unweighted ensemble by disorder class, on the profiles that pass quality checks; BioEmu-1 and PepTron emphasised, single-structure predictors grouped below. Right: median Rg ratio of each model on the disordered entries; solid line 1, dashed line 1.1.
| model | shared entries | median raw χ² | BioEmu-1, same entries | median log₁₀ χ² ratio (95%) | entries where it beats BioEmu-1 (95%) | same, by NRMSD | disordered Rg ratio |
|---|---|---|---|---|---|---|---|
| PepTron | 399 | 4.2 | 3.3 | +0.02 (+0.01, +0.04) | 40% (36%–45%) | 43% | 1.01 |
| Boltz-2 | 399 | 6.8 | 3.3 | +0.14 (+0.10, +0.20) | 21% (17%–25%) | 25% | 0.52 |
| IDPFold | 236 | 18.3 | 3.2 | +0.54 (+0.42, +0.75) | 24% (19%–30%) | 32% | 1.00 |
| idpSAM | 374 | 310.5 | 3.3 | +1.85 (+1.74, +1.93) | 1% (0%–2%) | 1% | 0.40 |
| AlphaFold2 (single structure) | 399 | 4.6 | 3.3 | +0.05 (+0.03, +0.07) | 34% (30%–39%) | 33% | 1.01 |
| ESMFold (single structure) | 399 | 5.0 | 3.3 | +0.09 (+0.07, +0.12) | 29% (25%–33%) | 30% | 0.88 |
Intervals: percentile bootstrap over the shared entries. The log₁₀ χ² ratio is comparator over BioEmu-1, positive when BioEmu-1 fits better. BioEmu-1 disordered Rg ratio: 1.21.
Five further folders in the archive are not used: Boltz-1x and ESMFlow are superseded by models already included, idpGAN and IDP-o are disorder-only generators represented here by idpSAM, and PepTron-base is the un-finetuned PepTron. AF-CALVADOS [10] was evaluated on this benchmark but its predictions were not in the archive.
The ensembles are those in the PeptoneBench archive, which records the BioEmu code version but not the checkpoint. The bioemu-v1.2 re-sample covers 51 proteins with one run of 100 samples each, so its seed-to-seed variation is not measured, and the distance between the two checkpoints, and any difference in sampler settings, is unknown. SASBDB profiles differ in buffer, temperature and construct, none of which the model sees, so part of the raw error is condition mismatch. The forward model is treated as exact; its hydration-shell parameters bias Rg by a few per cent, which is why modestly reweightable cases are read with care. Disorder classes come from a sequence-based predictor. The model comparison is restricted to the six comparators on the shared entries. CATH and the PDB serve as the reference for experimentally known structure space. BioEmu-1's training data were measured from their public releases: the AFDB cluster definitions and the simulated systems are those published with the model, the 361 MEGAscale wild types used for fine-tuning are a subset of the curated set the paper does not list, and the other published simulations are represented by the full UniProt chains of the named proteins, an upper bound on their overlap. Disorder of the training chains is predicted from sequence, without examining the training structures, and AFDB superfamilies are the CATH-Gene3D assignments of the full chain. PDB coverage of a construct comes from SIFTS residue ranges, which count residues listed in the structure record and can include residues not modelled in the coordinates. The prediction and selection tests are retrospective and use 399 entries; a predictor that fails at this size could succeed with more data or with richer features of the model's own ensemble. Profiles with a low-angle upturn above 10%, an intensity more than three standard errors below zero, a reported error of zero or less, no valid Guinier region, or a measured mass above 1.6 times the sequence mass are kept out of every statistic (1, 12, 1, 7 and 18 profiles; 37 in all, as some carry more than one flag).
The pipeline runs in named stages (download, score, coordinate radii, analysis, selection test, prediction, envelopes, report) from a locked environment, with unit tests on synthetic data. Derived results are tracked in the repository, and the page build reads every number from them and stops if a sentence that states a direction or a comparison no longer matches. Key code: coordinate radii, acquisition policies, error prediction. The ensembles and computed curves are those published by the PeptoneBench authors [3] (Zenodo 17306061, CC-BY 4.0).