[AI from Scratch] Episode 255: Fine-Tuning BERT

Recap and Today’s Theme

Hello! In the previous episode, we discussed the implementation of the Transformer model, an innovative architecture that uses Self-Attention to enhance NLP performance.

Today, we dive into fine-tuning BERT (Bidirectional Encoder Representations from Transformers). BERT, developed by Google, is a pre-trained model widely used in NLP tasks. By leveraging the knowledge from its pre-training phase, we can adapt it to specific NLP tasks. This episode covers the basic concepts of BERT, steps for fine-tuning, and a detailed implementation example.

What is BERT?

1. Basic Concept of BERT

BERT is a language model based on the encoder component of the Transformer architecture. Unlike traditional models that process sequences in a single direction (left-to-right or right-to-left), BERT uses a bidirectional approach, considering both past and future contexts simultaneously to understand the meaning of words. This bidirectionality allows BERT to achieve a deeper understanding of text, making it highly effective in tasks such as sentiment analysis, question answering, and document classification.

BERT is pre-trained on a large corpus using two main tasks:

• Masked Language Model (MLM): Certain words in the input text are masked, and the model predicts these words.
• Next Sentence Prediction (NSP): The model predicts whether one sentence logically follows another.

2. What is Fine-Tuning?

Fine-tuning is the process of further training a pre-trained BERT model on a specific task. Since BERT already has general language knowledge, fine-tuning tailors it to specific tasks (e.g., sentiment analysis, question answering, document classification) by training it on task-specific data.

Steps for Fine-Tuning BERT

Fine-tuning BERT involves the following steps:

1. Load the Pre-trained BERT Model: Use a library like Hugging Face’s Transformers to load a pre-trained BERT model.
2. Preprocess Input Data: Convert text data into a format compatible with BERT, including tokenization and padding.
3. Build the Model: Attach additional layers for classification or other tasks.
4. Fine-Tune the Model: Train the model with the task-specific data.
5. Evaluate: Assess the model’s performance using test data.

Below is an implementation example of fine-tuning BERT for sentiment analysis on IMDb movie reviews.

Implementation Example of Fine-Tuning BERT

We use the Hugging Face Transformers library to implement BERT fine-tuning for sentiment analysis.

1. Preparation

First, install the required libraries:

pip install transformers
pip install torch
pip install tensorflow

2. Load the Pre-trained BERT Model

Load the BERT model and tokenizer:

from transformers import BertTokenizer, TFBertForSequenceClassification

# Load the BERT tokenizer and model
model_name = "bert-base-uncased"
tokenizer = BertTokenizer.from_pretrained(model_name)
model = TFBertForSequenceClassification.from_pretrained(model_name, num_labels=2)

This code loads the bert-base-uncased model configured for binary classification.

3. Data Preprocessing

Prepare the text data to fit the input format BERT expects:

from tensorflow.keras.preprocessing.sequence import pad_sequences
import tensorflow as tf

def preprocess_texts(texts, tokenizer, max_len=128):
    # Tokenize texts
    input_ids = [tokenizer.encode(text, add_special_tokens=True, max_length=max_len, truncation=True) for text in texts]
    # Pad sequences
    input_ids = pad_sequences(input_ids, maxlen=max_len, dtype="long", truncating="post", padding="post")
    # Create attention masks
    attention_masks = [[float(i > 0) for i in seq] for seq in input_ids]
    return input_ids, attention_masks

# Example texts
texts = ["I love this movie!", "This movie was terrible..."]

# Preprocess texts
input_ids, attention_masks = preprocess_texts(texts, tokenizer)

This code tokenizes the text and generates input_ids and attention_masks for BERT.

4. Fine-Tuning the Model

Prepare the dataset and fine-tune the model:

from sklearn.model_selection import train_test_split

# Split the data
X_train, X_val, y_train, y_val = train_test_split(input_ids, labels, test_size=0.2, random_state=42)
train_masks, val_masks = train_test_split(attention_masks, test_size=0.2, random_state=42)

# Create TensorFlow datasets
train_dataset = tf.data.Dataset.from_tensor_slices(({"input_ids": X_train, "attention_mask": train_masks}, y_train)).batch(32)
val_dataset = tf.data.Dataset.from_tensor_slices(({"input_ids": X_val, "attention_mask": val_masks}, y_val)).batch(32)

# Compile the model
optimizer = tf.keras.optimizers.Adam(learning_rate=2e-5)
model.compile(optimizer=optimizer, loss=model.compute_loss, metrics=["accuracy"])

# Train the model
history = model.fit(train_dataset, epochs=3, validation_data=val_dataset)

The model is compiled and trained using a small learning rate (2e-5) and a batch size of 32. We split the data into training and validation sets for monitoring performance.

5. Model Evaluation

Finally, evaluate the model with test data:

# Preprocess test data
test_texts = ["The movie was fantastic!", "I didn't like the film."]
test_ids, test_masks = preprocess_texts(test_texts, tokenizer)

# Make predictions
predictions = model.predict({"input_ids": test_ids, "attention_mask": test_masks})
predicted_labels = tf.argmax(predictions.logits, axis=1).numpy()

# Display results
for text, label in zip(test_texts, predicted_labels):
    print(f"Text: {text} => Sentiment: {'Positive' if label == 1 else 'Negative'}")

This code makes predictions on new text inputs and outputs the sentiment classification.

Tips for Fine-Tuning BERT

1. Learning Rate

The learning rate for fine-tuning is typically set very low (e.g., 2e-5 to 5e-5) to prevent overfitting and degradation of the pre-trained model.

2. Number of Epochs

The number of epochs is usually kept low (e.g., 3 to 5) to avoid overfitting. Adjust this based on the dataset size and task complexity.

3. Data Augmentation and Normalization

For smaller datasets, data augmentation can improve performance. Additionally, text normalization (removing unnecessary characters and standardizing text) is important for consistency.

Summary

This episode explained the process of fine-tuning BERT, detailing the steps and implementation. By adapting a pre-trained BERT model to specific tasks, it is possible to build high-accuracy models even with limited data. Next time, we will explore Topic Modeling (LDA), a technique to extract latent topics from documents.

Next Episode Preview

Next time, we will discuss Topic Modeling (LDA), learning how to extract latent topics within documents using LDA.

---

Notes

1. Fine-Tuning: Additional training applied to a pre-trained model for specific tasks.
2. Masked Language Model (MLM): A task where some words in the text are masked, and the model predicts them.
3. Next Sentence Prediction (NSP): A task to determine if two sentences logically follow each other.