Machine Learning: Classification, Regression and Clustering

14.1 Introduction to Machine Learning

Machine learning is an important part of artificial intelligence. The basic idea is simple: learn patterns from data rather than manually programming every rule.

What is machine learning?

Suppose you want a program to recognize handwritten digits. Writing rules for every possible way a person can write the digit 7 would be extremely difficult. A machine-learning model can instead study many labeled examples and learn patterns that help it recognize new examples.

Prediction

Machine learning can be used for many prediction problems: weather forecasting, medical diagnosis, fraud detection, customer churn, recommendation systems, handwriting recognition, voice recognition, spam filtering and many others.

14.1.1 Scikit-Learn

scikit-learn (often imported as sklearn) packages many machine-learning algorithms as easy-to-use estimators. The mathematical details are largely hidden inside these objects. You generally create an estimator, train it, evaluate it and use it for predictions.

14.1.2 Types of Machine Learning

Supervised learning

Each sample has a target (label). For example, an email might have the target spam or not spam.

Unsupervised learning

There are no target labels supplied to the algorithm. The model tries to discover structure in the data. Clustering groups similar samples.

14.1.3 Datasets Bundled with Scikit-Learn

Scikit-learn includes small "toy" datasets and several larger real-world datasets for experimentation. Examples include Iris plants, diabetes, handwritten digits, wine recognition, breast cancer data, Olivetti faces, 20 newsgroups and California Housing.

14.1.4 Steps in a Typical Data Science Study

1. Load the dataset.
2. Explore the data with pandas and visualizations.
3. Clean or transform the data when necessary.
4. Split data for training and testing.
5. Create a model.
6. Train the model.
7. Test and evaluate the model.
8. Tune the model when appropriate.
9. Make predictions on new data.

14.2 Classification with k-Nearest Neighbors and the Digits Dataset, Part 1

The case study uses the scikit-learn Digits dataset. It contains 8×8 handwritten-digit images represented by 64 numeric features. There are 10 possible classes: digits 0 through 9.

Classification problem

Classification predicts which category a sample belongs to. Two categories give binary classification; more than two categories give multi-classification.

14.2.1 k-Nearest Neighbors Algorithm

k-nearest neighbors (k-NN) predicts a sample by looking at the k training samples closest to it. The class receiving the most votes wins.

New sample
   ↓
Find the k closest training samples
   ↓
Look at their classes
   ↓
Majority vote
   ↓
Predicted class

For example, if three nearest neighbors are B, C and C, the prediction is C.

Hyperparameters

A model has parameters it learns from data and hyperparameters that are specified before training. In k-NN, k (represented by n_neighbors) is a hyperparameter.

14.2.2 Loading the Dataset

from sklearn.datasets import load_digits

digits = load_digits()

print(digits.data.shape)
print(digits.target.shape)

digits.data contains the samples and their 64 features. digits.target contains the correct digit for each sample.

14.2.3 Visualizing the Data

Before training a model, explore the data. For digit images, Matplotlib can display the 8×8 pixel arrays so you can visually understand what the model is seeing.

import matplotlib.pyplot as plt

figure, axes = plt.subplots(nrows=4, ncols=6, figsize=(6, 4))

for ax, image, target in zip(axes.ravel(),
                             digits.images,
                             digits.target):
    ax.imshow(image, cmap=plt.cm.gray_r)
    ax.set_xticks([])
    ax.set_yticks([])
    ax.set_title(target)

plt.tight_layout()

14.2.4 Splitting the Data for Training and Testing

The model should not be evaluated only on the data it has already seen. We therefore keep part of the dataset aside for testing.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    digits.data,
    digits.target,
    random_state=11
)

print(X_train.shape)
print(X_test.shape)

By default, this split uses about 75% for training and 25% for testing. The random_state value makes the random split reproducible.

14.2.5 Creating the Model

from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier()

Scikit-learn calls these model objects estimators.

14.2.6 Training the Model

knn.fit(X=X_train, y=y_train)

For k-NN, the fit operation mainly stores the training data because k-NN is a lazy learning algorithm: much of its work happens when predictions are requested.

14.2.7 Predicting Digit Classes

predicted = knn.predict(X=X_test)
expected = y_test

wrong = [(p, e) for p, e in zip(predicted, expected) if p != e]

print(wrong)

In the chapter's run, 10 of 450 test samples were incorrect, giving about 97.78% prediction accuracy.

14.3 Classification with k-Nearest Neighbors and the Digits Dataset, Part 2

14.3.1 Metrics for Model Accuracy

After training and testing, you need to measure how well the model performed.

The score() method

print(f'{knn.score(X_test, y_test):.2%}')

Confusion matrix

A confusion matrix shows the model's hits and misses for each class. Rows represent the actual classes and columns represent predicted classes. Correct predictions appear on the main diagonal.

from sklearn.metrics import confusion_matrix

confusion = confusion_matrix(
    y_true=expected,
    y_pred=predicted
)

Classification report

from sklearn.metrics import classification_report

names = [str(digit) for digit in digits.target_names]

print(classification_report(
    expected,
    predicted,
    target_names=names
))

Visualizing the confusion matrix

import pandas as pd
import seaborn as sns

confusion_df = pd.DataFrame(
    confusion,
    index=range(10),
    columns=range(10)
)

sns.heatmap(confusion_df, annot=True)

14.3.2 K-Fold Cross-Validation

K-fold cross-validation repeatedly trains and tests the model using different parts of the dataset. The data is divided into k folds. Each fold gets a turn as the test fold while the remaining folds are used for training.

from sklearn.model_selection import KFold, cross_val_score

kfold = KFold(
    n_splits=10,
    random_state=11,
    shuffle=True
)

scores = cross_val_score(
    estimator=knn,
    X=digits.data,
    y=digits.target,
    cv=kfold
)

print(f'Mean accuracy: {scores.mean():.2%}')
print(f'Accuracy standard deviation: {scores.std():.2%}')

14.3.3 Running Multiple Models to Find the Best One

It is difficult to know beforehand which estimator will perform best. Scikit-learn makes it easy to compare several models using the same cross-validation procedure.

from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB

estimators = {
    'KNeighborsClassifier': knn,
    'SVC': SVC(gamma='scale'),
    'GaussianNB': GaussianNB()
}

for name, estimator in estimators.items():
    kfold = KFold(n_splits=10, random_state=11, shuffle=True)
    scores = cross_val_score(
        estimator=estimator,
        X=digits.data,
        y=digits.target,
        cv=kfold
    )
    print(f'{name}: mean={scores.mean():.2%}, '
          f'std={scores.std():.2%}')

The lesson is not "SVC is always best." The lesson is: test multiple appropriate models and compare them on your data.

14.3.4 Hyperparameter Tuning

Instead of accepting the default k=5, try several values and compare their performance.

for k in range(1, 20, 2):
    kfold = KFold(
        n_splits=10,
        random_state=11,
        shuffle=True
    )
    knn = KNeighborsClassifier(n_neighbors=k)
    scores = cross_val_score(
        estimator=knn,
        X=digits.data,
        y=digits.target,
        cv=kfold
    )
    print(
        f'k={k}: mean={scores.mean():.2%}, '
        f'std={scores.std():.2%}'
    )

14.4 Case Study: Time Series and Simple Linear Regression

Simple linear regression describes the relationship between one independent variable and one dependent variable using a straight line:

y = mx + b

The chapter revisits average New York City January high-temperature data from 1895 through 2018. The year is the independent variable and temperature is the dependent variable.

Loading the data

import pandas as pd

nyc = pd.read_csv('ave_hi_nyc_jan_1895-2018.csv')
nyc.columns = ['Date', 'Temperature', 'Anomaly']
nyc.Date = nyc.Date.floordiv(100)

nyc.head(3)

Preparing one feature for scikit-learn

A DataFrame column normally produces a one-dimensional Series, while scikit-learn estimators expect the features in a two-dimensional structure. reshape(-1, 1) converts one column of values into rows with one feature.

X_train, X_test, y_train, y_test = train_test_split(
    nyc.Date.values.reshape(-1, 1),
    nyc.Temperature.values,
    random_state=11
)

Training

from sklearn.linear_model import LinearRegression

linear_regression = LinearRegression()
linear_regression.fit(X=X_train, y=y_train)

print(linear_regression.coef_)
print(linear_regression.intercept_)

The estimator finds a best-fitting line by minimizing the sum of squared distances of the data points from the line.

Making predictions

predicted = linear_regression.predict(X_test)

predict = lambda x: (
    linear_regression.coef_ * x +
    linear_regression.intercept_
)

print(predict(2019))
print(predict(1890))

The model can also be visualized with a scatter plot and the regression line using Seaborn and Matplotlib.

Overfitting and Underfitting

14.5 Multiple Linear Regression with the California Housing Dataset

Instead of predicting a value from one feature, multiple linear regression uses several numerical features together.

The California Housing dataset

The chapter uses 20,640 samples with eight numerical features. The target is the median house value.

14.5.1 Loading the Dataset

from sklearn.datasets import fetch_california_housing

california = fetch_california_housing()

print(california.data.shape)
print(california.target.shape)

14.5.2 Exploring the Data with Pandas

import pandas as pd

california_df = pd.DataFrame(
    california.data,
    columns=california.feature_names
)

california_df['MedHouseValue'] = pd.Series(
    california.target
)

california_df.head()
california_df.describe()

Exploration lets you inspect ranges, averages, spread and unusual values before training.

14.5.3 Visualizing the Features

The chapter samples 10% of the rows for visualization and creates scatter plots of each feature against median house value. Visualization can reveal relationships and unusual patterns that are hard to notice from raw numbers.

sample_df = california_df.sample(
    frac=0.1,
    random_state=17
)

import seaborn as sns
import matplotlib.pyplot as plt

for feature in california.feature_names:
    plt.figure(figsize=(16, 9))
    sns.scatterplot(
        data=sample_df,
        x=feature,
        y='MedHouseValue'
    )

14.5.4 Splitting the Data for Training and Testing

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    california.data,
    california.target,
    random_state=11
)

print(X_train.shape)
print(X_test.shape)

14.5.5 Training the Model

from sklearn.linear_model import LinearRegression

linear_regression = LinearRegression()
linear_regression.fit(
    X=X_train,
    y=y_train
)

The model produces one coefficient for each feature and one intercept. Conceptually:

y = m1x1 + m2x2 + ... + mnxn + b

A positive coefficient means the predicted target tends to increase as that feature increases, while a negative coefficient means it tends to decrease, holding the model's other terms in the equation constant.

14.5.6 Testing the Model

predicted = linear_regression.predict(X_test)
expected = y_test

print(predicted[:5])

14.5.7 Visualizing Expected vs. Predicted Prices

Create a DataFrame containing the expected and predicted values and plot them. A perfect model would place every point on the diagonal line where expected equals predicted.

df = pd.DataFrame()
df['Expected'] = pd.Series(expected)
df['Predicted'] = pd.Series(predicted)

sns.scatterplot(
    data=df,
    x='Expected',
    y='Predicted'
)

start = min(expected.min(), predicted.min())
end = max(expected.max(), predicted.max())

plt.plot([start, end], [start, end])

14.5.8 Regression Model Metrics

R² score

The R² score, or coefficient of determination, indicates how well the model accounts for the dependent variable based on the independent variables. In the chapter's treatment, 1.0 is best and 0.0 indicates no predictive accuracy based on those variables.

from sklearn import metrics

metrics.r2_score(expected, predicted)

Mean squared error

Mean squared error (MSE) finds each prediction error, squares it, and averages the squared errors. When comparing models with MSE, values closer to zero are better.

metrics.mean_squared_error(expected, predicted)

14.5.9 Choosing the Best Model

The chapter compares several regression estimators: LinearRegression, ElasticNet, Lasso and Ridge. They are evaluated using 10-fold cross-validation and R² scoring.

14.6 Unsupervised Machine Learning, Part 1 — Dimensionality Reduction

Some datasets have many dimensions. The Digits dataset has 64 features; real-world datasets can have hundreds, thousands or even millions.

Dimensionality reduction compresses a high-dimensional dataset into fewer dimensions, often two or three, so that humans can visualize it and sometimes so models can work with a simpler representation.

t-SNE with the Digits dataset

from sklearn.datasets import load_digits
from sklearn.manifold import TSNE

digits = load_digits()

tsne = TSNE(
    n_components=2,
    random_state=11
)

reduced_data = tsne.fit_transform(digits.data)

print(reduced_data.shape)

The original 64 features have been reduced to two new dimensions. A scatter plot can then reveal clusters of similar handwritten digits.

import matplotlib.pyplot as plt

plt.scatter(
    reduced_data[:, 0],
    reduced_data[:, 1],
    c=digits.target
)

plt.colorbar()

14.7 Unsupervised Machine Learning, Part 2 — k-Means Clustering

k-means is an unsupervised algorithm that attempts to divide unlabeled samples into a specified number of clusters.

How k-means works

1. Choose the desired number of clusters, k.
2. Choose initial centroids.
3. Assign samples to their closest centroid.
4. Recalculate the centroids.
5. Repeat the assignment and recalculation process until the clusters stabilize.

Data
 ↓
Choose k
 ↓
Initial centroids
 ↓
Assign points to nearest centroid
 ↓
Recalculate centroids
 ↓
Repeat
 ↓
Final clusters

A centroid is the center point of a cluster.

Iris Dataset

The Iris dataset has 150 samples, four numerical features and three species: Iris setosa, Iris versicolor and Iris virginica.

14.7.1 Loading the Iris Dataset

from sklearn.datasets import load_iris

iris = load_iris()

print(iris.data.shape)
print(iris.target.shape)
print(iris.target_names)

For the clustering experiment, we intentionally ignore the known labels while building the clusters. The labels can later help us judge the clustering result.

14.7.2 Exploring the Iris Dataset: Descriptive Statistics with Pandas

import pandas as pd

iris_df = pd.DataFrame(
    iris.data,
    columns=iris.feature_names
)

iris_df['species'] = [
    iris.target_names[i]
    for i in iris.target
]

iris_df.head()
iris_df.describe()

14.7.3 Visualizing the Dataset with a Seaborn pairplot

A pairplot shows relationships between pairs of features. It is particularly useful when the dataset has only a few features and samples.

import seaborn as sns

sns.pairplot(
    data=iris_df,
    vars=iris_df.columns[0:4],
    hue='species'
)

The visualizations show that setosa is relatively easy to separate, while versicolor and virginica have some overlap in certain feature combinations.

14.7.4 Using a KMeans Estimator

from sklearn.cluster import KMeans

kmeans = KMeans(
    n_clusters=3,
    random_state=11
)

kmeans.fit(iris.data)

After fitting, the estimator contains:

• labels_ — the cluster assigned to each sample.
• cluster_centers_ — the centroid of each cluster.

14.7.5 Dimensionality Reduction with Principal Component Analysis

The chapter uses PCA (Principal Component Analysis) to reduce the Iris dataset's four features to two dimensions for visualization.

from sklearn.decomposition import PCA

pca = PCA(
    n_components=2,
    random_state=11
)

pca.fit(iris.data)

iris_pca = pca.transform(iris.data)

print(iris_pca.shape)

The same trained PCA estimator can also transform the cluster centroids so that the centroids can be displayed on the same two-dimensional graph.

iris_pca_df = pd.DataFrame(
    iris_pca,
    columns=['Component1', 'Component2']
)

iris_pca_df['species'] = iris_df.species

iris_centers = pca.transform(
    kmeans.cluster_centers_
)

14.7.6 Choosing the Best Clustering Estimator

The chapter compares several clustering estimators on the Iris data:

• KMeans
• DBSCAN
• MeanShift
• SpectralClustering
• AgglomerativeClustering

from sklearn.cluster import (
    DBSCAN,
    MeanShift,
    SpectralClustering,
    AgglomerativeClustering
)

estimators = {
    'KMeans': kmeans,
    'DBSCAN': DBSCAN(),
    'MeanShift': MeanShift(),
    'SpectralClustering':
        SpectralClustering(n_clusters=3),
    'AgglomerativeClustering':
        AgglomerativeClustering(n_clusters=3)
}

for name, estimator in estimators.items():
    estimator.fit(iris.data)
    print(name)

The chapter demonstrates that different clustering algorithms can produce different groupings. For example, DBSCAN found three clusters but grouped many versicolor and virginica samples together; MeanShift found two clusters in this experiment. The best algorithm therefore depends on the dataset and the goal of the study.

14.8 Wrap-Up

This chapter introduces practical machine learning with scikit-learn through classification, regression and clustering.

Quick Memory Map

Machine Learning
│
├── Supervised
│   ├── Classification
│   │   └── k-NN → Digits
│   └── Regression
│       ├── Simple Linear Regression → Temperature
│       └── Multiple Linear Regression → California Housing
│
└── Unsupervised
    ├── Dimensionality Reduction
    │   ├── t-SNE → Digits
    │   └── PCA → Iris
    └── Clustering
        └── k-Means → Iris

Common Beginner Mistakes

• Training and testing on the same data: the model may look better than it really is.
• Skipping data exploration: important patterns or unusual values can be missed.
• Confusing parameters and hyperparameters: hyperparameters are selected before training.
• Confusing k-NN's k with k-fold's k: they are different concepts.
• Thinking high accuracy always means a good model: use suitable metrics and unseen data.
• Assuming cluster labels have meaning: cluster number 0 or 1 is only an identifier.
• Ignoring reproducibility: use random_state when appropriate.
• Using categorical values directly: scikit-learn models generally require numerical features; categorical data must be transformed when necessary.

Revision Questions

1. What is machine learning?
2. What is the difference between supervised and unsupervised learning?
3. What is classification? Give two examples.
4. What is regression?
5. What are features and targets?
6. What does train_test_split() do?
7. Why should test data be unseen during training?
8. How does k-nearest neighbors make a prediction?
9. What is a hyperparameter?
10. What is k-fold cross-validation?
11. What does a confusion matrix show?
12. What are precision, recall and F1-score?
13. What is overfitting?
14. What is underfitting?
15. What is the purpose of R² and mean squared error?
16. Why is dimensionality reduction useful?
17. What is a centroid in k-means?
18. Why can different clustering algorithms produce different results?

Practice Programs

Teaching edition created from the chapter structure and concepts in the supplied Python for Programmers source. Examples are rewritten for beginner-friendly explanation rather than reproducing the book verbatim.