[AI from Scratch] Episode 232: Practical Transfer Learning — Utilizing Pre-trained Models

Recap and Today’s Theme

Hello! In the previous episode, we explored data augmentation, demonstrating how to expand image datasets using Keras. We confirmed that data augmentation not only helps models learn from more diverse data but also prevents overfitting.

Today, we will discuss Transfer Learning, a technique that leverages pre-trained models for new tasks. Transfer learning is particularly effective when building high-accuracy models with limited data. Many pre-trained models are available for image recognition, allowing us to efficiently build high-performance models. Let’s explore how to implement transfer learning using Keras.

What Is Transfer Learning?

Transfer Learning is a method that utilizes the knowledge learned by a model from one dataset or task to apply it to a new task. Pre-trained models are typically trained on large datasets (e.g., ImageNet) and have already learned significant features of images. By using these models, we can achieve high accuracy even with a small amount of data for new tasks.

Advantages of Transfer Learning

1. Faster Training: Since many parameters are already optimized, models can quickly adapt to new datasets.
2. High Accuracy with Small Data: Even with limited data, accuracy can be improved by using pre-trained models.
3. Resource Efficiency: Saves time and computational resources needed for training.

Implementing Transfer Learning

Let’s implement transfer learning using Keras. We will use a pre-trained MobileNetV2 model and apply it to the CIFAR-10 dataset for image classification.

1. Importing the Necessary Libraries

First, import TensorFlow and Keras libraries.

import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow.keras.applications import MobileNetV2
import matplotlib.pyplot as plt

2. Preparing the Dataset

We load the CIFAR-10 dataset, which is built into Keras, and normalize the image data.

# Loading the CIFAR-10 dataset
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()

# Scaling image data to 0-1
x_train = x_train.astype('float32') / 255.0
x_test = x_test.astype('float32') / 255.0

3. Loading and Configuring the Pre-trained Model

Next, we load the MobileNetV2 model provided by Keras. We use the pre-trained weights (imagenet) and remove the final layer to retain only the feature extraction part.

# Loading the pre-trained MobileNetV2 model
base_model = MobileNetV2(weights='imagenet', include_top=False, input_shape=(32, 32, 3))

# Freezing the base model (no training)
base_model.trainable = False

• weights=’imagenet’: Uses weights trained on the ImageNet dataset.
• include_top=False: Excludes the final dense layer, keeping only the feature extraction part.
• trainable=False: Freezes the base model weights to prevent re-training.

4. Adding a New Output Layer

We add a new classification layer on top of the base model to create a model suitable for the 10 classes in the CIFAR-10 dataset.

# Defining the model
model = models.Sequential([
    base_model,
    layers.GlobalAveragePooling2D(),
    layers.Dense(128, activation='relu'),
    layers.Dense(10, activation='softmax')
])

• GlobalAveragePooling2D: A pooling layer used before the dense layers to average the spatial dimensions of the feature map.
• Dense: Adds fully connected layers for 10-class classification.

5. Compiling the Model

Next, we compile the model.

# Compiling the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

• optimizer=’adam’: Uses the Adam optimization algorithm.
• loss=’sparse_categorical_crossentropy’: Suitable for integer labels in CIFAR-10.
• metrics=[‘accuracy’]: Evaluates model performance using accuracy.

6. Training the Model

We train the model using the dataset.

# Training the model
history = model.fit(x_train, y_train, epochs=10, batch_size=64, validation_split=0.2)

• epochs=10: Trains the model for 10 epochs.
• batch_size=64: Uses a batch size of 64 for training efficiency.
• validation_split=0.2: Uses 20% of the training data as validation data.

7. Evaluating the Model

We evaluate the model’s performance using the test data.

# Evaluating the model
test_loss, test_accuracy = model.evaluate(x_test, y_test)
print(f"Test accuracy: {test_accuracy:.2f}")

8. Fine-Tuning

If further accuracy improvement is needed, we can fine-tune some layers of the base model. Fine-tuning involves re-training specific layers to adapt the model further to the new dataset.

# Unfreezing certain layers for fine-tuning
base_model.trainable = True

# Re-compiling the model for fine-tuning
model.compile(optimizer=tf.keras.optimizers.Adam(1e-5),  # Lower learning rate for fine-tuning
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Training for fine-tuning
history_fine = model.fit(x_train, y_train, epochs=5, batch_size=64, validation_split=0.2)

• trainable=True: Unfreezes part or all of the base model layers for re-training.
• Lower learning rate: A smaller learning rate prevents large changes in weights during fine-tuning.

Applications of Transfer Learning

Transfer learning is widely applied in various fields:

1. Image Classification: High-accuracy image classification using pre-trained models like ImageNet, even with small datasets.
2. Object Detection: Efficient model construction for detecting objects and their positions using transfer learning.
3. Natural Language Processing (NLP): Leveraging pre-trained models like BERT and GPT for text classification, text generation, and translation tasks.

Summary

In this episode, we explored Transfer Learning using Keras. We demonstrated how to quickly build a high-accuracy model with limited data by using pre-trained models. Transfer learning is a powerful tool in deep learning, especially in image recognition and NLP. Utilize this technique to streamline model development!

Next Episode Preview

Next time, we will discuss model deployment, covering how to use trained models in real-world applications. We’ll learn the steps from deploying a model to making predictions in a live environment!

---

Notes

• Pre-trained Model: A model trained on a large dataset, used as a base for building new models with limited data.
• Fine-Tuning: A technique where certain layers of an existing model are re-trained to adapt to new tasks.