How models are evaluated

Every model you train on deepmirror comes with a performance estimate. This page explains, at a high level, where that estimate comes from: which molecules the model is scored on, how they are chosen, which numbers we report, and what those numbers can and cannot tell you.

The short version. We score a model on molecules it was not trained on. Where your dataset has a numeric molecule ID and enough measurements, those molecules are your newest ones. Otherwise they come from cross-validation. In both cases the model you actually use for predictions is then refit on all of your data.

Which evaluation your model gets

The method is chosen per property, not per dataset. For each property you train on, we count the molecules that have a measured value for it and check whether the dataset has a numeric molecule ID column.

Situation How the model is scored Labelled in the app as
Numeric molecule ID and 100 or more measurements for the property Newest molecules held back (temporal validation) Temporal split
Numeric molecule ID but fewer than 100 measurements 5-fold scaffold-grouped cross-validation Scaffold split
No numeric molecule ID 5-fold scaffold-grouped cross-validation Scaffold split
Classification model (30 or more molecules) 5-fold random cross-validation, stratified by class Random split
Very small dataset (fewer than 20 molecules for regression, fewer than 30 for classification) Leave-one-out cross-validation Random split

One dataset can therefore carry a temporal score on a well-measured property and a cross-validated score on one you have measured less often.

Temporal validation: holding back your newest molecules

When a numeric molecule ID is available we sort the molecules by that ID, treat the highest IDs as the most recent, and hold them back. The model is trained on everything older and scored only on the held-back set. How many we hold back depends on how many measurements the property has:

Molecules measured for that property Held back to score the model
1,000 or more 200 newest
500 – 999 100 newest
100 – 499 50 newest
Fewer than 100 none — cross-validated

This is the most honest test we can run without you supplying a separate test set. Cross-validation mixes old and new molecules together, so the model is always tested on compounds that have close analogues in training. Holding back your newest chemistry asks the question that matters in a live project: how well does the model do on compounds it genuinely has not seen? Expect temporal scores to be lower than cross-validated ones. A lower number here reflects a more realistic estimate, not a worse model.

What counts as a numeric molecule ID

Any ID column works as long as every non-empty value contains digits. We strip everything that is not a digit and sort on the number that remains, so CMP-00123, DM_123 and 123 all sort as 123. Rows with an empty ID are treated as the oldest. If a column contains values with no digits at all, it cannot be used and the model falls back to cross-validation.

Row order and re-imports make no difference. The split depends only on the values in your molecule ID column, not on the order of rows in your file or when they were uploaded. Importing the same data in a different order, or into a new dataset, holds back the same molecules.

If your IDs don't reflect chronology. Purchased libraries and merged datasets often have IDs that say nothing about the order compounds were made or acquired. You can assign your own sequential IDs to define a chronology and use that column as the molecule ID. Keep a lookup table mapping back to the originals.

If you want a cross-validated score instead. You can't choose the method directly, but you can choose whether a molecule ID is used. If your IDs say nothing about chronology and you don't want to renumber them, change the molecule ID column's type to Text in Edit columns (or when importing) and retrain. The model will then be scored with scaffold-grouped cross-validation. Random splits are used only for classification models and very small datasets, and can't be selected.

Cross-validation: how the folds are built

When no molecules can be held back, we cross-validate. The training molecules are split into five groups (folds). Five models are trained in turn, each leaving one fold out, and each is scored on the fold it did not see. Every molecule is therefore predicted exactly once by a model that never trained on it. The reported score is the average across the five folds.

Folds are grouped by scaffold. Before splitting, every molecule is assigned its Bemis–Murcko scaffold (the ring systems and the linkers joining them, with substituents removed). Molecules sharing a scaffold are always placed in the same fold. This stops the model from being scored on a close analogue of something it trained on, which would flatter the result. Scaffold-grouped cross-validation needs at least five distinct scaffolds in the training set.

The split is deterministic. Scaffold groups are shuffled with a fixed random seed before being dealt into folds, so training the same data twice gives the same folds and a comparable score.

Very small datasets use leave-one-out. Regression datasets with fewer than 20 measurements, and classification datasets with fewer than 30, use leave-one-out cross-validation instead: each molecule is predicted in turn by a model trained on all the others. This is the only option that leaves enough data to train on, but scores on datasets this small are unreliable and should be read as a rough indication only.

Classification models are cross-validated with random rather than scaffold-grouped folds, stratified so each fold has a similar class balance.

The numbers we report

For regression models the app shows two headline numbers, both computed on the held-back or out-of-fold predictions.

Spearman's ρ measures how well the model ranks molecules against each other. It runs from −1 to 1, where 1 means the predicted order matches the measured order exactly and 0 means no relationship. A model whose predictions are all shifted by a constant still ranks perfectly, so ρ is insensitive to systematic offsets. Ranking is usually what you rely on when choosing between designs.

MAE (mean absolute error) is the average size of the gap between predicted and measured values, ignoring direction. Ranking is not the whole story: if every compound is correctly predicted to be poor, you need new designs rather than the best of a bad lot, and MAE tells you how far off the absolute predictions are. MAE is reported in the units your property is measured in, so there is no conversion to do.

One exception on units. If a property's values span more than two orders of magnitude (a raw IC50 in nM that runs from 5 to 50,000, for example), we model it on a natural-log scale because the model performs better that way. For those properties MAE is reported on the same log scale. If you want errors in measurement units for such a property, store it as a log quantity such as pIC50 instead.

For classification models the headline numbers are Matthews correlation coefficient (MCC) and ROC-AUC, alongside a confusion matrix.

How much to trust the score

The score is a measurement made on a finite set of molecules, and like any measurement it has noise.

  • Smaller evaluation sets give noisier scores. A ρ measured on 50 held-back molecules will move around more between retrains than one measured on 200. Treat small differences on small sets with caution. This is also why properties with fewer than 100 measurements stay cross-validated rather than getting a tiny held-out set.
  • A narrow measured range depresses every metric. If all your molecules sit within half a log unit of each other, there is little ranking to get right and ρ will be low even for a sound model.
  • Temporal scores depend on how different your newest chemistry is. A series that has moved to a new scaffold will score lower than one that has stayed close to home. That is the point of the test, but it means temporal scores are not directly comparable between projects.
  • Cross-validated scores are optimistic on new chemistry. Even with scaffold grouping, the folds are drawn from the same chemistry the model trained on. Use them to compare models on the same data, not as a forecast of performance on a new series.

Your served model is trained on everything

Evaluation is a measurement, not a restriction on training. Once the score has been computed, the model you use for predictions is refit on all of your data, including the held-back newest molecules. We have checked directly that this refit model performs the same as one trained without a hold-out step; see Updates to how model performance is reported for that comparison.

Data requirements

  • At least 5 molecules with distinct measured values (and, for classification, at least 5 molecules per class).
  • At most 10,000 molecules per model.
  • No molecule heavier than 4,000 Da. Training is rejected if one is present, so remove or flag it first.