In the scenario Tal Sofer opens with on the lakeFS blog, a credit model has been in production since March and the validator asks for one thing: the exact data it was trained on. The training config answers with s3://ml-data/features/. Three backfills have rewritten that prefix since March, a labeling fix touched a third of the rows, and the engineer who ran the job has moved teams. What is left is a notebook called features_final_v3.ipynb. The review stalls.
Overwriting a partition during a backfill is normal in the BI stack Sofer describes. The platform did its job and, in doing it, erased the evidence the review needed. This is a structural problem, not a bug.
A path is a promise about location
A path says where to look. It does not record what was there when the model read it. The missing piece is a record of data state, not a better tool.
Sofer proposes recording the dataset as an immutable commit and saving the commit identifier in the run metadata next to the code version. A commit identifier is also a promise. It holds only while the bytes it points at are kept and never rewritten.
An isolated copy kept unchanged and an immutable versioned store with retention both preserve that state. The two are the same guarantee by different mechanisms. Sofer acknowledges that keeping history means keeping old objects and requires a retention policy. I would require a content hash recorded at training time and a retrieval test at review to turn either reference into evidence. Recovering the inputs does not establish that the model is valid.
Four things, written separately
The policy: a model must have reproducible training inputs for as long as it can be reviewed.
The standard: an immutable identifier, a content hash of the inputs, and retention tied to the life of the model, including the period in which it can be reviewed.
The procedure: start with Sofer's advice to pick one training pipeline that touches production data, put it on versioned, branched data, record the commit identifier with every run, and validate changes before merging. I would also record the content hash of the training inputs with each run.
The verification: at review, I would start from the model register entry, resolve the identifier, retrieve the inputs, recompute the hash and compare it with the value recorded at training time. A populated field is not evidence. A retrieval that matches is.