Splitting Data Honestly and Avoiding Leakage

How you split your data into training, validation and test sets determines whether you can trust your model’s performance estimates. The wrong split introduces severe bias into your evaluation and makes you think the model is better than it actually is. The worst mistake is data leakage, where information from the test set leaks into the training process, so you train on data you later claim to have left held out for objective evaluation. This is particularly problematic in financial services, healthcare and e-commerce domains where model performance must reflect real-world conditions.

Temporal Split: The Gold Standard

The gold standard is temporal split. If you are building a model to predict events in the future, you train exclusively on data from the past and test exclusively on data from the future. If you train a model to predict loan default, you train on loans originated before 31 December 2019 and test on loans originated after 1 January 2020. This ensures the test set is truly held out and the evaluation reflects what the model will actually do on future data it has never seen. Temporal split is the most honest way to evaluate a model because it mirrors how the model will be used in production.

For example, a retail company building a demand forecasting model for seasonal products must ensure that training data only includes historical sales before the start of a new season. If the model is trained on data that includes sales from the latest season, it will appear to perform well on test data but will fail when applied to future demand. This is especially critical in supply chain management, where poor forecasting can lead to overstocking or stockouts. A temporal split ensures that the model is evaluated under realistic conditions that reflect how it will be used in production environments.

Random Split: A Practical Alternative

If you cannot do temporal split, random split is the alternative. You randomly assign records to train, validation and test sets, usually approximately 70 percent training, 15 percent validation, 15 percent test. The critical rule is that the test set is left completely untouched until final evaluation is complete. You do not use the test set to tune hyperparameters. You do not use the test set to choose between model architectures. You do not even look at test set performance during development.

For instance, in a customer churn prediction model for a telecom company, a random split ensures that the model is evaluated on a truly independent sample. If the company were to use a non-random split, such as grouping customers by region or tenure, it could lead to overfitting to specific segments. A practical step is to use stratified sampling to maintain the same proportion of churners and non-churners in each set. This helps ensure that the model is dependable and generalises well across different customer types. In a real-world scenario, a telecom firm might split customer data into training, validation and test sets using a random seed to ensure reproducibility and prevent accidental leakage.

Understanding Data Leakage

Data leakage occurs when information from the test set influences the training process, either directly or indirectly. The most obvious form is accidentally using a variable that is only known after the decision has been made. In a fraud detection model, if you use “account has been reported as fraudulent” as a predictor, you are using information that only becomes available after someone has suffered the fraud. This is leakage. The model appears to perform perfectly on test data but fails completely in production where you do not have access to that variable.

Consider a healthcare model that predicts patient readmission. If the model uses “readmission within 30 days” as a feature, it will appear highly accurate on test data but will be useless in practice. A real-world example is a hospital using patient data to predict which patients are at risk of readmission. If the model includes a variable that only becomes known after the patient is discharged, it will not reflect the true predictive capability. This is a common pitfall in medical data science projects where features are often derived from outcomes rather than predictors.

Subtle Leakage in Preprocessing

Subtle leakage happens when you preprocess data before splitting. If you standardise all variables using mean and standard deviation calculated from the full dataset (train and test combined), you have leaked information from the test set into the training process. The correct order is: split data first, calculate preprocessing parameters from training set only, then apply those parameters to validation and test sets.

For example, in a credit scoring model, a data scientist might standardise income variables using the mean and standard deviation of the entire dataset. This means that information about future customers (those in the test set) is used to transform the training data. A practical step is to implement a preprocessing pipeline that separates the training and test data before any standardisation. This can be done using scikit-learn’s Pipeline class or by manually applying transformations. In a financial institution, this step is critical when building models for lending decisions, as even small amounts of leakage can lead to overoptimistic performance metrics and poor real-world performance. A common mistake is to standardise features using the entire dataset before splitting, which can artificially inflate model performance and lead to incorrect business decisions.