Here is a situation everyone who has trained a model has been in.
You have a number. It's 0.78. You are fairly sure it came from the model in the folder called run_final_v2_actually_final. You are less sure which images it was scored on, because you added forty more last Tuesday and re-split at some point after that. You cannot reproduce it, and you cannot say whether the 0.81 you got this morning is an improvement or a different measurement entirely.
The number is not evidence. It is a rumor about a model.
The fix is boring
Freeze the dataset before the run consumes it.
In Orinth, a dataset version is a materialized, preprocessed snapshot with a name. Preprocessing and augmentation are applied and written, not recomputed at load time. The split assignment is fixed. The version is immutable.
A training run records the version it consumed — not the dataset, the version. So does a testing run.
That's it. That's the whole mechanism.
What it buys you
Comparability. Two runs against the same version are comparable. Two runs against "the dataset" at different times are not, and no amount of experiment tracking fixes that, because the thing that changed wasn't tracked.
Reproducibility from the record alone. The run stores its full configuration — every hyperparameter, including the family defaults you never touched — plus the version ID. You can rebuild the run from its record without remembering anything.
A real held-out split. "Scored on data the model never saw" is only checkable if the split assignment was frozen before training started. Re-splitting between train and test quietly leaks, and the leak looks exactly like an improvement.
What it costs
Disk. A frozen version is a real copy of preprocessed data, so ten versions of a large dataset take real space.
We think that's the right trade at this scale. Recomputing preprocessing at load time saves the disk and gives it all back in ambiguity — you no longer know what the model actually consumed, because it depends on the code at the time it ran.
Original uploads are never modified, so versions are additive on top of a source of truth you can always return to.
The habit underneath
Most of Orinth's design comes out of one belief: a measurement you can't trace is worse than no measurement, because you'll act on it anyway.
That's also why every surface makes its context visible — which project, which dataset, which split, which model, which run. And why per-item inspection sits behind every aggregate number. An aggregate is a summary of rows, and the rows are where the interesting failures live.
None of this is clever. It's the ML equivalent of committing before you refactor.