[AI from Scratch] Episode 284: Fine-Tuning — Retraining Parts of Transfer Learning Models

Recap and Today’s Theme

Hello! In the previous episode, we covered transfer learning, where we leveraged pre-trained models (like VGG16) and adapted them for new image classification tasks. Transfer learning allowed us to build high-accuracy models efficiently, even with limited data, by using the feature extraction capabilities of models trained on large datasets.

In this episode, we will dive into an advanced technique within transfer learning known as Fine-Tuning. Fine-tuning is the process of retraining certain layers of a pre-trained model to further optimize it for a specific task. This article will explain the concept of fine-tuning and walk through its implementation.

What is Fine-Tuning?

Fine-tuning is the process of selectively re-training the deeper layers of a pre-trained model to adapt it more precisely to the target task. In basic transfer learning, only the final output layer is trained, while the rest of the model’s weights remain unchanged. Fine-tuning, however, involves “unfreezing” some of the deeper layers and adjusting their weights to further improve the model’s accuracy for the new task.

Benefits of Fine-Tuning

• Improved Accuracy: Fine-tuning allows for more precise adjustments to the model, improving its performance on the target task.
• Flexibility: You can selectively retrain only certain layers, which reduces the computational cost while still optimizing for specific datasets.

How Fine-Tuning Works

The typical steps for fine-tuning are:

1. Load Pre-Trained Model: Start by loading a pre-trained model (e.g., VGG, ResNet).
2. Add and Train Output Layer: Add a new output layer and train it (this is standard transfer learning).
3. Unfreeze Specific Layers: Choose which layers to unfreeze for retraining (often the deeper layers).
4. Perform Fine-Tuning: Retrain the unfrozen layers along with the output layer to optimize the model for the specific task.

Fine-Tuning with Python and Keras

We will now demonstrate how to implement fine-tuning using Python and Keras. In this example, we will use a pre-trained VGG16 model and fine-tune it for a binary classification task.

1. Installing Required Libraries

pip install tensorflow

2. Fine-Tuning Example

The following code shows how to perform fine-tuning using VGG16 on a new dataset:

import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow.keras.applications import VGG16
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.optimizers import Adam

# Load pre-trained VGG16 model without the output layer
base_model = VGG16(weights='imagenet', include_top=False, input_shape=(150, 150, 3))

# Freeze the layers of the base model
base_model.trainable = False

# Build the transfer learning model
model = models.Sequential([
    base_model,
    layers.Flatten(),
    layers.Dense(256, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(1, activation='sigmoid')  # Binary classification
])

# Compile the model
model.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=['accuracy'])

# Data generators (with data augmentation)
train_datagen = ImageDataGenerator(rescale=1./255, rotation_range=20, width_shift_range=0.2, height_shift_range=0.2, horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1./255)

# Load datasets
train_generator = train_datagen.flow_from_directory('data/train', target_size=(150, 150), batch_size=32, class_mode='binary')
validation_generator = test_datagen.flow_from_directory('data/validation', target_size=(150, 150), batch_size=32, class_mode='binary')

# Train the output layer
history = model.fit(train_generator, epochs=5, validation_data=validation_generator)

# Prepare for fine-tuning (unfreeze some layers)
base_model.trainable = True
for layer in base_model.layers[:15]:  # Keep early layers frozen
    layer.trainable = False

# Recompile the model with a lower learning rate
model.compile(optimizer=Adam(learning_rate=1e-5), loss='binary_crossentropy', metrics=['accuracy'])

# Fine-tuning
history_finetune = model.fit(train_generator, epochs=5, validation_data=validation_generator)

# Plot training results
import matplotlib.pyplot as plt
plt.plot(history_finetune.history['accuracy'], label='accuracy')
plt.plot(history_finetune.history['val_accuracy'], label='val_accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()
plt.show()

• Freezing Layers: Initially, we freeze all layers of the pre-trained model and only train the output layer.
• Unfreezing Layers: During fine-tuning, we unfreeze deeper layers and retrain them along with the output layer.
• Learning Rate: It is recommended to use a lower learning rate during fine-tuning (e.g., 1e-5) to avoid overwriting the pre-trained weights too drastically.

Key Points to Consider for Fine-Tuning

1. Selecting Layers to Retrain

It is common to only retrain the deeper layers of the model, as these capture more complex features specific to the task at hand. The earlier layers, which capture more general features like edges and textures, are typically left frozen.

2. Learning Rate Adjustments

Fine-tuning requires careful learning rate adjustments. A low learning rate helps prevent the pre-trained weights from being significantly altered, ensuring that the model retains the useful knowledge it gained during its initial training.

3. Data Augmentation and Regularization

To avoid overfitting, especially when fine-tuning on small datasets, it’s important to use techniques like data augmentation and regularization (e.g., dropout). These methods help improve the model’s generalization ability by increasing the variability of the training data.

Applications of Fine-Tuning

1. Medical Diagnosis

Fine-tuning is used in medical imaging to improve diagnostic accuracy with small datasets. Pre-trained models are fine-tuned to detect specific diseases or abnormalities in medical images such as X-rays or MRIs.

2. Autonomous Driving

In self-driving cars, fine-tuning is applied to pre-trained object detection models to adapt them to specific environments, such as recognizing traffic signs, pedestrians, and vehicles in different cities or regions.

3. Surveillance Systems

Fine-tuning can improve accuracy in video surveillance systems by allowing the model to adapt to various environmental conditions, lighting, and camera angles.

Summary

In this episode, we introduced Fine-Tuning, an advanced technique that builds on transfer learning by selectively retraining parts of a pre-trained model to optimize it for specific tasks. Fine-tuning can significantly improve model accuracy, especially when working with smaller datasets. In the next episode, we will cover the basics of Object Detection, a key technique used to identify objects within images.

Next Episode Preview

Next time, we will dive into Object Detection, a foundational technology for identifying and classifying objects within an image. We will explore its basic algorithms and practical implementation strategies.

---

Notes

• Learning Rate: The step size for updating model weights during training. Fine-tuning often requires a small learning rate to prevent over-adjusting pre-trained weights.
• Regularization: Techniques such as dropout are used to reduce overfitting by adding constraints on the model’s complexity.