Cross-validation framework — Scikit-learn course (2024)

Training error vs testing error#

To solve this regression task, we will use a decision tree regressor.

from sklearn.tree import DecisionTreeRegressorregressor = DecisionTreeRegressor(random_state=0)regressor.fit(data, target)

DecisionTreeRegressor(random_state=0)

After training the regressor, we would like to know its potentialgeneralization performance once deployed in production. For this purpose, weuse the mean absolute error, which gives us an error in the native unit, i.e.k$.

from sklearn.metrics import mean_absolute_errortarget_predicted = regressor.predict(data)score = mean_absolute_error(target, target_predicted)print(f"On average, our regressor makes an error of {score:.2f} k$")

On average, our regressor makes an error of 0.00 k$

We get perfect prediction with no error. It is too optimistic and almostalways revealing a methodological problem when doing machine learning.

Indeed, we trained and predicted on the same dataset. Since our decision treewas fully grown, every sample in the dataset is stored in a leaf node.Therefore, our decision tree fully memorized the dataset given during fitand therefore made no error when predicting.

This error computed above is called the empirical error or trainingerror.

We trained a predictive model to minimize the training error but our aim is tominimize the error on data that has not been seen during training.

This error is also called the generalization error or the “true” testingerror.

Thus, the most basic evaluation involves:

• splitting our dataset into two subsets: a training set and a testing set;

• fitting the model on the training set;

• estimating the training error on the training set;

• estimating the testing error on the testing set.

So let’s split our dataset.

from sklearn.model_selection import train_test_splitdata_train, data_test, target_train, target_test = train_test_split( data, target, random_state=0)

Then, let’s train our model.

regressor.fit(data_train, target_train)

DecisionTreeRegressor(random_state=0)

Finally, we estimate the different types of errors. Let’s start by computingthe training error.

target_predicted = regressor.predict(data_train)score = mean_absolute_error(target_train, target_predicted)print(f"The training error of our model is {score:.2f} k$")

The training error of our model is 0.00 k$

We observe the same phenomena as in the previous experiment: our modelmemorized the training set. However, we now compute the testing error.

target_predicted = regressor.predict(data_test)score = mean_absolute_error(target_test, target_predicted)print(f"The testing error of our model is {score:.2f} k$")

The testing error of our model is 47.28 k$

This testing error is actually about what we would expect from our model if itwas used in a production environment.

Stability of the cross-validation estimates#

When doing a single train-test split we don’t give any indication regardingthe robustness of the evaluation of our predictive model: in particular, ifthe test set is small, this estimate of the testing error will be unstable andwouldn’t reflect the “true error rate” we would have observed with the samemodel on an unlimited amount of test data.

For instance, we could have been lucky when we did our random split of ourlimited dataset and isolated some of the easiest cases to predict in thetesting set just by chance: the estimation of the testing error would beoverly optimistic, in this case.

Cross-validation allows estimating the robustness of a predictive model byrepeating the splitting procedure. It will give several training and testingerrors and thus some estimate of the variability of the model generalizationperformance.

There are different cross-validationstrategies,for now we are going to focus on one called “shuffle-split”. At each iterationof this strategy we:

• randomly shuffle the order of the samples of a copy of the full dataset;

• split the shuffled dataset into a train and a test set;

• train a new model on the train set;

• evaluate the testing error on the test set.

We repeat this procedure n_splits times. Keep in mind that the computationalcost increases with n_splits.

In this case we will set n_splits=40, meaning that we will train 40 modelsin total and all of them will be discarded: we just record theirgeneralization performance on each variant of the test set.

To evaluate the generalization performance of our regressor, we can usesklearn.model_selection.cross_validatewith asklearn.model_selection.ShuffleSplitobject:

from sklearn.model_selection import cross_validatefrom sklearn.model_selection import ShuffleSplitcv = ShuffleSplit(n_splits=40, test_size=0.3, random_state=0)cv_results = cross_validate( regressor, data, target, cv=cv, scoring="neg_mean_absolute_error")

The results cv_results are stored into a Python dictionary. We will convertit into a pandas dataframe to ease visualization and manipulation.

import pandas as pdcv_results = pd.DataFrame(cv_results)cv_results

Let us revert the negation to get the actual error:

cv_results["test_error"] = -cv_results["test_score"]

Let’s check the results reported by the cross-validation.

cv_results.head(10)

We get timing information to fit and predict at each cross-validationiteration. Also, we get the test score, which corresponds to the testing erroron each of the splits.

len(cv_results)

40

We get 40 entries in our resulting dataframe because we performed 40 splits.Therefore, we can show the testing error distribution and thus, have anestimate of its variability.

import matplotlib.pyplot as pltcv_results["test_error"].plot.hist(bins=10, edgecolor="black")plt.xlabel("Mean absolute error (k$)")_ = plt.title("Test error distribution")

We observe that the testing error is clustered around 47 k$ and ranges from43 k$ to 50 k$.

print( "The mean cross-validated testing error is: " f"{cv_results['test_error'].mean():.2f} k$")

The mean cross-validated testing error is: 46.36 k$

print( "The standard deviation of the testing error is: " f"{cv_results['test_error'].std():.2f} k$")

The standard deviation of the testing error is: 1.17 k$

Note that the standard deviation is much smaller than the mean: we couldsummarize that our cross-validation estimate of the testing error is 46.36 ±1.17 k$.

If we were to train a single model on the full dataset (withoutcross-validation) and then later had access to an unlimited amount of testdata, we would expect its true testing error to fall close to that region.

While this information is interesting in itself, it should be contrasted tothe scale of the natural variability of the vector target in our dataset.

Let us plot the distribution of the target variable:

target.plot.hist(bins=20, edgecolor="black")plt.xlabel("Median House Value (k$)")_ = plt.title("Target distribution")

print(f"The standard deviation of the target is: {target.std():.2f} k$")

The standard deviation of the target is: 115.40 k$

The target variable ranges from close to 0 k$ up to 500 k$ and, with astandard deviation around 115 k$.

We notice that the mean estimate of the testing error obtained bycross-validation is a bit smaller than the natural scale of variation of thetarget variable. Furthermore, the standard deviation of the cross validationestimate of the testing error is even smaller.

This is a good start, but not necessarily enough to decide whether thegeneralization performance is good enough to make our prediction useful inpractice.

We recall that our model makes, on average, an error around 47 k$. With thisinformation and looking at the target distribution, such an error might beacceptable when predicting houses with a 500 k$. However, it would be anissue with a house with a value of 50 k$. Thus, this indicates that ourmetric (Mean Absolute Error) is not ideal.

We might instead choose a metric relative to the target value to predict: themean absolute percentage error would have been a much better choice.

But in all cases, an error of 47 k$ might be too large to automatically useour model to tag house values without expert supervision.

More detail regarding cross_validate#

During cross-validation, many models are trained and evaluated. Indeed, thenumber of elements in each array of the output of cross_validate is a resultfrom one of these fit/score procedures. To make it explicit, it ispossible to retrieve these fitted models for each of the splits/folds bypassing the option return_estimator=True in cross_validate.

cv_results = cross_validate(regressor, data, target, return_estimator=True)cv_results

{'fit_time': array([0.1656003 , 0.16090298, 0.16149712, 0.16149282, 0.15634871]), 'score_time': array([0.00256753, 0.00256181, 0.00258613, 0.00303054, 0.00268364]), 'estimator': [DecisionTreeRegressor(random_state=0), DecisionTreeRegressor(random_state=0), DecisionTreeRegressor(random_state=0), DecisionTreeRegressor(random_state=0), DecisionTreeRegressor(random_state=0)], 'test_score': array([0.26291527, 0.41947109, 0.44492564, 0.23357874, 0.40788361])}

cv_results["estimator"]

[DecisionTreeRegressor(random_state=0), DecisionTreeRegressor(random_state=0), DecisionTreeRegressor(random_state=0), DecisionTreeRegressor(random_state=0), DecisionTreeRegressor(random_state=0)]

The five decision tree regressors corresponds to the five fitted decisiontrees on the different folds. Having access to these regressors is handybecause it allows to inspect the internal fitted parameters of theseregressors.

In the case where you only are interested in the test score, scikit-learnprovide a cross_val_score function. It is identical to calling thecross_validate function and to select the test_score only (as weextensively did in the previous notebooks).

from sklearn.model_selection import cross_val_scorescores = cross_val_score(regressor, data, target)scores

array([0.26291527, 0.41947109, 0.44492564, 0.23357874, 0.40788361])

Summary#

In this notebook, we saw:

• the necessity of splitting the data into a train and test set;

• the meaning of the training and testing errors;

• the overall cross-validation framework with the possibility to studygeneralization performance variations.