My Model Was Cheating on Its Own Test

a model gets to see information it should not have access to yet, and that hidden peek makes its score look better than it really is. Kaufman, Rosset, Perlich, and Stitelman describe this clearly in their 2012 paper in ACM Transactions on Knowledge Discovery from Data, “Leakage in Data Mining: Formulation, Detection, and Avoidance.” One example they give is the INFORMS 2010 Data Mining Challenge. Competitors were supposed to predict stock price movements using only a training set, then be scored on a separate test set. Several competitors figured out which real stocks were hidden in the test set by matching patterns against public finance data. That let them pull in information the test set was supposed to keep hidden, and their scores looked better than their models actually deserved. My own leak was much smaller and much easier to miss. It did not involve figuring out hidden stock identities. It came from running two ordinary lines of preprocessing code in the wrong order. I trained a small neural network, using scikit-learn’s MLPRegressor, to predict car prices from a used car dataset. The first version of this project reported a strong test score: an R squared of 0.887. After I fixed the order of two lines, the honest score was 0.767. The model did not get worse. My first measurement had been quietly reading part of the answer key. The number that looked solid R squared is a common score for how well a model predicts a number. It runs from 0 to 1. Higher is better. An R squared of 0.887 means the model explains about 89 percent of the differences in car price on data it had never trained on. The original version of this project reported a test R squared of 0.887 and a test error, measured as mean squared error, of about 6.9 million. I reran the exact same code, with the exact same settings, to check whether that number was real, and it matched. For a small dataset of under 200 cars, that is a genuinely strong result, the kind that ends a homework assignment without further questions. The experiment This project started from a leak I found in a car price model I had built for a class assignment. That version used a dataset with no license listed anywhere on the page it came from, so instead of reproducing it here, I rebuilt the same pipeline on UCI’s Automobile dataset, donated by Jeffrey Schlimmer in 1987 and sourced from the 1985 Ward’s Automotive Yearbook. It is released under a Creative Commons Attribution 4.0 license, which permits reuse like this with credit. The two datasets describe the same kind of thing: one row per car, with its specifications and its price. Each row lists a car’s make, body style, engine specs, fuel type, an assigned insurance risk rating, and price. After dropping a column with a large number of missing values and removing the rows with any remaining missing values, 193 cars remain. Of the 24 columns left to predict price from, 16 are numeric measurements, such as engine size, horsepower, and curb weight, and 8 are categorical, such as fuel type, drive wheel, and engine location. The goal is regression: predict price, which is a number, rather than sorting cars into categories. I used a multilayer perceptron, or MLP. An MLP is a type of neural network: a model made of layers of small connected units, where each unit combines its inputs and passes the result forward. This one has two hidden layers of 64 units each: MLPRegressor( hidden_layer_sizes=(64, 64), max_iter=1000, random_state=42, ) I split the 193 rows into three groups, matching the same proportions as the original assignment: 60 percent for training, 20 percent for validation, and 20 percent for test, which comes to 115 training rows, 39 validation rows, and 39 test rows. The training set is what the model actually learns from. The validation set is meant to check the model while it is still being developed. The test set is a final, one time check, meant to be looked at only once the model is finished. Two lines, run in the wrong order Before any model can use this data, the raw columns need to be prepared. Numeric columns like horsepower need to be put on a common scale, since some numbers use small ranges and others use large ones. Categorical columns like fuel type need to be converted into numbers a model can use, commonly by turning each category into its own 0 or 1 column, a method called one hot encoding. This preparation step is often called preprocessing, and in scikit-learn it is usually done with a small chain of steps called a pipeline. Here is the preprocessing code, in the order it ran to produce the strong looking result above: # Outlier handling via IQR capping for col in numerical_cols + ["price"]: Q1 = df[col].quantile(0.25) Q3 = df[col].quantile(0.75) IQR = Q3 - Q1 lower_bound = Q1 - 1.5 * IQR upper_bound = Q3 + 1.5 * IQR df[col] = df[col].clip(lower=lower_bound, upper=upper_bound) # Preprocessing pipeline preprocessor = ColumnTransformer(transformers=[ ("num", StandardScaler(), numerical_cols), ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_cols), ]) # Transform the data X_processed = preprocessor.fit_transform(X) # Train-validation-test split X_train_val, X_test, y_train_val, y_test = train_test_split( X_processed, y, test_size=0.2, random_state=42 ) Nothing about this code signals a problem, which is part of the point. It is the same shape of mistake I originally found in the class assignment this project is based on, just written against a differently sourced dataset here so the result can be shared freely. Read the code in order, and watch what happens before the split. The first step caps extreme outlier values in each column, using a common statistical rule: anything more than 1.5 times the interquartile range (the middle 50 percent of the data) above or below the typical range gets pulled in to that boundary. That boundary is calculated from every row in the dataset, including rows that will become the test set two steps later. StandardScaler, which rescales numeric columns, calculates its scaling numbers from every row too. OneHotEncoder, which builds the 0 and 1 columns for categories, learns its list of categories from every row as well. Only after all three of these steps finish does train_test_split divide the data into separate pieces. None of these three steps looks like a mistake by itself. Capping outliers is a normal thing to do. Scaling numbers is a normal thing to do. Calling fit_transform, which both learns the transformation and applies it in one step, is the normal way to use a scikit-learn transformer. The mistake is entirely about order: every one of these steps was allowed to look at the test rows before the test set was supposed to exist. This is a quieter kind of problem than the more obvious version of a leak, where a single line compares validation labels with themselves and produces an impossible perfect score. A leak like this one does not announce itself. The model still makes real predictions. The test rows were not copied into training, they only quietly nudged the scaling numbers, the outlier boundaries, and the category list. The resulting score looks like an ordinary strong result, not a broken one. That is exactly what makes it worth checking for, even when nothing looks wrong. The corrected pipeline The fix is simple to state: split the data first, then fit every preprocessing step on the training rows only. The validation and test rows should only ever be transformed using numbers already learned from training, never used to help calculate those numbers. X_train_val, X_test, y_train_val, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) X_train, X_val, y_train, y_val = train_test_split( X_train_val, y_train_val, test_size=0.25, random_state=42 ) # IQR bounds computed from the training rows only bounds = {} for col in numerical_cols: Q1 = X_train[col].quantile(0.25) Q3 = X_train[col].quantile(0.75) IQR = Q3 - Q1 bounds[col] = (Q1 - 1.5 * IQR, Q3 + 1.5 * IQR) def apply_bounds(frame): frame = frame.copy() for col, (lower, upper) in bounds.items(): frame[col] = frame[col].clip(lower=lower, upper=upper) return frame X_train = apply_bounds(X_train) X_val = apply_bounds(X_val) X_test = apply_bounds(X_test) # Cap outlier prices in the training target only, using bounds from # training prices. Validation and test prices are left as observed, # since scoring against a clipped target would hide real errors. price_q1, price_q3 = y_train.quantile(0.25), y_train.quantile(0.75) price_iqr = price_q3 - price_q1 y_train = y_train.clip( lower=price_q1 - 1.5 * price_iqr, upper=price_q3 + 1.5 * price_iqr ) preprocessor = ColumnTransformer(transformers=[ ("num", StandardScaler(), numerical_cols), ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_cols), ]) X_train_p = preprocessor.fit_transform(X_train) X_val_p = preprocessor.transform(X_val) X_test_p = preprocessor.transform(X_test) mlp = MLPRegressor(hidden_layer_sizes=(64, 64), max_iter=1000, random_state=42) mlp.fit(X_train_p, y_train) Notice the pattern: fit_transform only ever runs on X_train. Validation and test data only ever go through transform, which applies numbers already learned, without learning anything new from them. What the leak was worth The corrected test R squared is 0.767, down from 0.887. The corrected test error is about 26.2 million, nearly four times the original 6.9 million. In terms that are easier to picture, the typical prediction error, measured as root mean squared error, grew from about $2,630 to about $5,120 per car. The leaky pipeline reports a test R squared of 0.887. Once preprocessing is fit on training rows only, the same model and the same random seed produce a test R squared of 0.767. Higher is better, and the honest number is the lower one. Chart by author, generated from a rerun of both pipelines on the same data and split. Twelve points of R squared is not a rounding error. It is the difference between a result worth highlighting in a report and a result you would call solid but ordinary. The whole gap comes from which rows were allowed to influence the scaler, the outlier boundaries, and the category list before the model was ever tested. One honest limit on this number: 193 rows is a small dataset, and the test set is only 39 cars. A gap of exactly twelve points of R squared is specific to this dataset, this split, and this random seed. On a larger dataset, the same mistake would likely produce a smaller gap, since a bigger training set moves the scaler’s mean and the outlier boundaries less when a few test rows are removed from the calculation. The direction of the mistake, an inflated score, holds regardless of dataset size. The exact size of the inflation does not. The validation set nobody asked for A validation set only earns its place if something actually gets scored on it. It is a common enough shortcut to build one, in the same line that builds the test set, and then never call predict on it, leaving the model’s fit quality unchecked between training and the final test. Scoring the corrected pipeline on all three splits shows why that matters: Train R squared is 0.872, validation is 0.787, and test is 0.767 for the corrected pipeline. Higher is better. The drop from train to validation is the generalization check a validation set exists to provide. Chart by author, generated from the corrected pipeline’s rerun. A model usually fits its training data a little better than it predicts on new data. That is normal, up to a point, and the size of the gap is what tells you whether that point has been passed. Here, training R squared and test R squared differ by about 0.10, a moderate gap for a model with two 64 unit hidden layers trained on only 115 rows. That gap is not evidence of a broken model. It is evidence that the validation set, once it is actually used, does the job it was built for. Reading the corrected result Each point is one test car: actual price on the x axis, predicted price on the y axis. Points close to the diagonal line are accurate predictions. Most points sit near the line, with wider spread at higher prices, where there are fewer training examples to learn from. Chart by author, generated from the corrected pipeline’s test set. Each point shows the actual price minus the predicted price, plotted against the predicted price. Points scattered evenly around the zero line mean there is no strong pattern in the errors. Wider spread at higher predicted prices shows the model is less precise for expensive cars, which matches there being fewer expensive cars in the training data. Chart by author, generated from the corrected pipeline’s test set. Neither chart is dramatic, and that is a good sign. A correctly measured result should look like a decent model with a normal amount of error, not a broken one. Fixing the leak was never about making the model look worse. It was about making the score describe what the model actually does. A short checklist for pipeline order Here is a short checklist for preprocessing order: Split the data before fitting anything that learns from it: scalers, encoders, missing value fillers, outlier boundaries. If a ColumnTransformer or Pipeline calls fit_transform, check exactly what data it was called on. Never call fit or fit_transform on validation or test data. Only ever call transform on it. If a validation set is built, use it. An unused validation set is not a safeguard, it is a step someone meant to take and did not. Compare train, validation, and test scores together. A single test score, on its own, cannot show you the gap that matters. What I learned An evaluation bug that compares labels with themselves is impossible to defend once you look at it: literally 100 percent accuracy from comparing an array with itself. A leak in preprocessing order is quieter, because fit_transform on the full dataset is valid, working Python, produces a believable number, and passes a casual read of the code. The only way to catch it is to ask a specific question about every step: which rows were used to calculate this mean, this boundary, this list of categories, and does that match which rows were later used to test the model. The lesson is simple to state and easy to skip in practice: a reported score is the output of a full pipeline, not just a model, and every step in that pipeline, including the evaluation code, is something to check before trusting the number it produces. That check extends to the data itself. The class assignment that first showed me this leak used a dataset with no listed license, which is its own kind of thing to catch before publishing a result built on it. Reproduce the experiment The companion folder contains the dataset, both pipeline versions, saved charts, and metrics: car-price-regression-leakage-bug/ code/ car_price_leakage_comparison.py data/ car_price_dataset.csv media/ leaky_vs_corrected_r2.png corrected_train_val_test_r2.png corrected_actual_vs_predicted.png corrected_residuals_vs_predicted.png outputs/ leaky_vs_corrected_metrics.csv run_summary.json From the article folder, install the packages and run both pipelines: python3 -m venv .venv source .venv/bin/activate pip install pandas numpy scikit-learn matplotlib seaborn ucimlrepo python code/car_price_leakage_comparison.py The script downloads the UCI Automobile dataset directly if a local copy is not already saved, prints train, validation, and test scores for both the leaky and corrected pipelines, and saves the comparison table and all four charts. Selected sources Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4), Article 15. Schlimmer, J. (1985). Automobile [Dataset]. UCI Machine Learning Repository. Licensed under CC BY 4.0. scikit-learn: ColumnTransformer scikit-learn: Pipeline and preventing data leakage scikit-learn: MLPRegressor