Feature scaling before fitting
When one column runs to thousands and another to fractions, the larger column dominates the distance the model is minimising.
Why the units matter
Many models measure similarity or error as a distance in the space the columns define. If one column is measured in metres and another in millimetres, the second column contributes a thousand times more to that distance for the same physical difference. The model is not wrong; it is answering the question the units asked.
Distance-based methods, anything regularised, and anything trained by gradient descent are all sensitive to this. Tree-based methods split one column at a time and are largely indifferent.
Two common transforms
Standardising subtracts each column's mean and divides by its standard deviation, leaving every column centred at zero with comparable spread. It suits data that is roughly bell-shaped and it tolerates outliers reasonably.
Normalising to a fixed range instead maps the smallest observed value to zero and the largest to one. It preserves the shape of the distribution but a single extreme value squashes everything else into a narrow band.
The ordering mistake
The statistics used for scaling have to be computed on the training split alone and then applied unchanged to the validation and test splits. Computing them over the whole dataset first lets information about the held-out rows reach the model, and the reported score comes out better than the model deserves.
A safe ordering
- Split the rows into training and held-out sets first, before touching any column.
- Compute the scaling statistics on the training rows only.
- Apply those statistics to the training rows.
- Apply the same stored statistics, unchanged, to the held-out rows.
- Store the statistics alongside the model so anything served later is scaled identically.
scaler = StandardScaler()
train_x = scaler.fit_transform(train_x) # learns mean and spread here
test_x = scaler.transform(test_x) # reuses them, learns nothingStatistics fitted once, applied twice