[AI from Scratch] Episode 224: Practical Hyperparameter Tuning — Using Grid Search to Find Optimal Parameters

Recap and Today’s Theme

Hello! In the previous episode, we discussed various methods for evaluating machine learning models, including accuracy, precision, recall, F1 score, ROC curves, and AUC scores. These metrics allow us to evaluate model performance comprehensively.

Today, we will explore hyperparameter tuning, a process for improving model performance. Hyperparameters are parameters that need to be set before training a model, and their values can significantly affect model accuracy. We will learn how to use Grid Search to find the optimal hyperparameters and enhance model performance. Let’s dive into the details!

What Are Hyperparameters?

Hyperparameters are parameters set before training a model. Examples include:

• Decision Trees: Maximum depth (max_depth), minimum samples required for splitting (min_samples_split).
• Logistic Regression: Regularization parameter (C), solver method (solver).
• Support Vector Machine (SVM): Kernel type (kernel), regularization parameter (C).

Hyperparameters vary for each model, and setting them correctly can maximize model performance.

What Is Grid Search?

Grid Search is a method that exhaustively searches for the best combination of hyperparameters from specified ranges. Scikit-learn’s GridSearchCV makes it easy to implement this method.

Grid Search Workflow

1. Set the range or candidates for the hyperparameters you want to tune.
2. Train the model for each combination and evaluate performance using cross-validation.
3. Choose the hyperparameter combination with the best performance.

Implementing Grid Search

Let’s implement Grid Search using a logistic regression model as an example.

1. Preparing the Dataset

We will use Scikit-learn’s breast_cancer dataset to train a logistic regression model.

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Loading and preparing the data
cancer = load_breast_cancer()
df = pd.DataFrame(cancer.data, columns=cancer.feature_names)
df['target'] = cancer.target

# Separating features and target
X = df.drop('target', axis=1)
y = df['target']

# Standardizing the data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Splitting the data
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)

2. Setting Up Grid Search

Next, we configure Grid Search by setting the hyperparameters for logistic regression: C (regularization parameter) and solver (optimization algorithm).

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV

# Defining the model
model = LogisticRegression()

# Setting hyperparameter grid
param_grid = {
    'C': [0.01, 0.1, 1, 10, 100],
    'solver': ['newton-cg', 'lbfgs', 'liblinear']
}

# Defining Grid Search (using 5-fold cross-validation)
grid_search = GridSearchCV(model, param_grid, cv=5, scoring='accuracy')

# Running Grid Search
grid_search.fit(X_train, y_train)

• param_grid: Specifies hyperparameter candidates in a dictionary format.
• GridSearchCV(): Runs Grid Search with cross-validation. The cv parameter sets the number of splits (5-fold in this case), and scoring specifies the evaluation metric (accuracy).

3. Checking the Results

We check the best hyperparameter combination and evaluate the model’s performance.

# Displaying the best parameters
print(f"Best Parameters: {grid_search.best_params_}")

# Evaluating accuracy on test data
best_model = grid_search.best_estimator_
accuracy = best_model.score(X_test, y_test)
print(f"Test Accuracy: {accuracy:.2f}")

• best_params_: Displays the best hyperparameters found by Grid Search.
• best_estimator_: Retrieves the model retrained with the best hyperparameters.

Grid Search Examples for Other Models

Grid Search can be applied to other models, such as decision trees and support vector machines (SVM).

Grid Search for Decision Trees

from sklearn.tree import DecisionTreeClassifier

# Defining the model
tree_model = DecisionTreeClassifier()

# Setting hyperparameter grid
tree_param_grid = {
    'max_depth': [3, 5, 7, 10],
    'min_samples_split': [2, 5, 10]
}

# Defining Grid Search
tree_grid_search = GridSearchCV(tree_model, tree_param_grid, cv=5, scoring='accuracy')

# Running Grid Search
tree_grid_search.fit(X_train, y_train)

# Displaying the best parameters
print(f"Best Parameters for Decision Tree: {tree_grid_search.best_params_}")

Grid Search for SVM

from sklearn.svm import SVC

# Defining the model
svm_model = SVC()

# Setting hyperparameter grid
svm_param_grid = {
    'C': [0.1, 1, 10],
    'kernel': ['linear', 'rbf'],
    'gamma': ['scale', 'auto']
}

# Defining Grid Search
svm_grid_search = GridSearchCV(svm_model, svm_param_grid, cv=5, scoring='accuracy')

# Running Grid Search
svm_grid_search.fit(X_train, y_train)

# Displaying the best parameters
print(f"Best Parameters for SVM: {svm_grid_search.best_params_}")

Considerations for Grid Search

• Computational Cost: Grid Search tests all combinations, so if there are many parameter candidates, the computation time increases. It’s essential to set a reasonable range and number of candidates.
• Evaluation Metrics: Choosing the right evaluation metric (e.g., accuracy, recall, F1 score) based on the model’s objective helps find the best model for the task.

Summary

In this episode, we explained how to use Grid Search for hyperparameter tuning. Finding the optimal hyperparameters can significantly improve model performance. By using Scikit-learn’s GridSearchCV, you can efficiently search for the best parameters across various models. Experiment with different models and parameters to explore Grid Search’s full potential!

Next Episode Preview

Next time, we will introduce the implementation of a Neural Network, explaining how to build a basic neural network model. Learn the fundamentals of neural networks and start building your own!

---

Annotations

• Hyperparameter: A parameter set before training a model, affecting the training process.
• Grid Search: A method that exhaustively searches for the best combination of hyperparameters.