Reproducibility: Seeds, Environments and Experiment Records
Reproducibility means someone else can rebuild your model from scratch and get the same result. This is essential for regulatory scrutiny. An auditor might ask: “Show me exactly what you did.
I want to verify it myself.” If you cannot reproduce your own model, you cannot answer that request and you lose credibility. In a real-world scenario, a data scientist at a pharmaceutical company was asked to justify a machine learning model used to predict drug efficacy.
Because the model was not reproducible, the regulatory body questioned its validity and delayed approval. This led to a costly delay in product development and a loss of trust in the team’s processes. Reproducibility is not just about technical accuracy; it is about transparency and accountability in decision-making.
Reproducibility requires three things: fixed random seeds, recorded environment specifications and complete experiment records.
Random seeds control the randomness in model training. Machine learning involves randomness: random initialisation of weights, random shuffling of data, random selection of hyperparameters in some algorithms. If you set a seed before training, you will get the same model every time.
If you do not set a seed, the model will vary slightly between runs due to random variation. Document which seed you used. A reviewer might ask whether the model depends on a lucky random seed.
If you used seed 42 but results are poor with seed 123, the model is unstable and might not generalise. For example, a team at a fintech company was developing a fraud detection model. They initially used no seed and observed inconsistent performance.
After setting a fixed seed, they were able to replicate results and ensure model stability. This allowed them to confidently deploy the model in production. Always log the seed used in your experiment logs and consider running multiple trials with different seeds to assess model resilience.
Environment specifications include the programming language version, library versions, operating system and hardware. A model trained with TensorFlow 2.10 might behave slightly differently than one trained with TensorFlow 2.11. If you cannot specify which versions you used, someone else cannot reproduce your results.
Use a requirements file (Python), environment file (Conda) or a container (Docker) to lock all dependencies precisely. A data scientist at a logistics firm once had to retrain a model that had been working perfectly in development. The model failed in production due to a version mismatch in NumPy.
Had they locked their environment, this issue would have been avoided. Using tools like Docker containers ensures that the execution environment is identical across all systems. This is especially important when deploying models in cloud environments or across multiple teams.
Always create and maintain a version-controlled environment specification file that can be used to recreate the exact setup.
Experiment records document every model you trained during development: hyperparameters, performance metrics, date and time, who ran it, which code version was used. This allows you to trace results back to specific code and to understand why you chose a particular model for deployment. If you trained 50 models during exploration, document all 50 and explain why you chose the final one.
Do not hide the exploration process. A machine learning engineer at a retail company was tasked with improving demand forecasting. They ran over 100 experiments, each with different hyperparameters and data preprocessing steps.
By maintaining detailed experiment logs, they were able to explain to stakeholders why a specific model was chosen. This transparency led to better buy-in and improved model performance. Experiment logs should include not only the model performance but also the reasoning behind each decision.
This helps future team members understand the process and avoid repeating past mistakes.
Do not delete experiment records because results were poor or disappointing. A poor result is still a result and tells an important story. If you only keep successful experiments, you hide information from auditors and create false impressions of model quality. Auditors expect to see your full development process and history. A case in point is a team at a financial services company that was investigating a model that underperformed. They had discarded all earlier experiments, which led to confusion when auditors asked for the rationale behind the final model. Had they retained all logs, they could have shown how they iterated and improved. This also helps in identifying patterns in model failure and in building better models in the future. Always treat experiment logs as a historical record of your work. They are not just for reporting; they are for learning and continuous improvement. Keep all logs in a centralised repository, even those that led to dead ends. This ensures that future audits or reviews are not hindered by incomplete data. Documenting even failed experiments provides valuable insights into model limitations and helps prevent similar issues in future projects. This is particularly important in regulated industries where transparency and traceability are critical. A well-maintained experiment log can be the difference between a model that is accepted and one that is rejected. It also provides a benchmark for future experiments and helps in identifying which approaches are most effective. Ultimately, reproducibility is not just a technical requirement; it is a cultural practice that enhances the quality and reliability of data science projects.