[AI from Scratch] Episode 252: Text Classification Using LSTM

Recap and Today’s Theme

Hello! In the previous episode, we covered the basics of sentiment analysis, explaining how to determine emotions and opinions from text data using dictionary-based and machine learning-based methods. We also highlighted how deep learning models, particularly context-aware models, can achieve high accuracy in sentiment analysis.

Today, we will delve into text classification using LSTM (Long Short-Term Memory), a type of deep learning model designed to handle sequence data. LSTM is particularly effective in retaining context, making it suitable for tasks involving time series or natural language processing. In this episode, we will explain the basics of LSTM and how to implement text classification using Python and Keras.

What is LSTM?

1. Basic Concept of LSTM

LSTM is a type of Recurrent Neural Network (RNN) designed to process sequence data. Traditional RNNs struggle to learn long-term dependencies due to issues like vanishing gradients and exploding gradients. LSTM addresses these issues by introducing cell states and gating mechanisms that allow the network to retain important information over longer periods while discarding irrelevant details.

LSTM uses three gates:

• Input Gate: Determines what new information should be added to the cell state.
• Forget Gate: Decides how much of the past information should be forgotten.
• Output Gate: Controls the extent of the cell state that should be output at each step.

2. Handling Sequence Data

LSTM is ideal for processing sequence data where the order of elements is important, such as text, audio, and time-series data. It excels in tasks like text classification, machine translation, and speech recognition by maintaining and leveraging context.

Workflow of Text Classification Using LSTM

The steps for implementing text classification using LSTM are as follows:

1. Data Preparation: Collect and label the text data.
2. Data Preprocessing: Clean the text, tokenize it, and apply padding to ensure uniform sequence lengths.
3. Model Construction: Build a neural network with LSTM layers.
4. Model Training: Train the model using labeled data.
5. Model Evaluation: Evaluate the model’s performance on test data.

Below, we implement a movie review sentiment classifier using Python and Keras.

Implementing Text Classification with LSTM

We use the IMDb dataset, which contains movie reviews labeled as either positive (1) or negative (0).

1. Data Preparation

First, we load the IMDb dataset:

from keras.datasets import imdb

# Load the IMDb dataset
max_features = 10000  # Limit to the top 10,000 most frequent words
(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=max_features)

print(f"Number of training samples: {len(X_train)}, Number of test samples: {len(X_test)}")

This code loads the dataset, restricting the vocabulary to the 10,000 most common words to manage model complexity.

2. Data Preprocessing

Next, we preprocess the data by padding sequences to ensure uniform length.

from keras.preprocessing.sequence import pad_sequences

# Set the sequence length
maxlen = 200

# Pad sequences to make all reviews the same length
X_train = pad_sequences(X_train, maxlen=maxlen)
X_test = pad_sequences(X_test, maxlen=maxlen)

print(f"Training data shape: {X_train.shape}, Test data shape: {X_test.shape}")

In this code, each review is either truncated or padded to 200 words to ensure consistency in input size.

3. Model Construction

We then build an LSTM-based neural network model:

from keras.models import Sequential
from keras.layers import Embedding, LSTM, Dense, Dropout

# Build the model
model = Sequential()
model.add(Embedding(input_dim=max_features, output_dim=128, input_length=maxlen))
model.add(LSTM(64, return_sequences=False))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

# Compile the model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

print(model.summary())

The model includes the following layers:

• Embedding Layer: Converts words into vectors.
• LSTM Layer: Processes the sequence data with 64 LSTM units.
• Dropout Layer: Reduces overfitting by randomly setting 50% of neurons to zero.
• Dense Layer: A fully connected layer with a sigmoid activation function for binary classification.

4. Model Training

Next, we train the model using the training data.

# Train the model
batch_size = 32
epochs = 5

history = model.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=0.2)

We set the batch size to 32 and train for 5 epochs. Additionally, 20% of the training data is used as validation data to monitor and reduce overfitting.

5. Model Evaluation

Finally, we evaluate the trained model using the test data.

# Evaluate the model on test data
loss, accuracy = model.evaluate(X_test, y_test)
print(f"Test loss: {loss:.4f}, Test accuracy: {accuracy:.4f}")

This code measures the model’s performance by calculating the loss and accuracy on the test dataset.

Improving the LSTM Model

1. Hyperparameter Tuning

Adjusting the following hyperparameters can further enhance the model’s performance:

• Number of LSTM Units: Changing the number of units in the LSTM layer adjusts the model’s capacity.
• Batch Size and Epochs: Optimizing these can stabilize and improve training.
• Learning Rate: Modifying the learning rate of the optimizer improves training efficiency and stability.

2. Using Bidirectional LSTM

A Bidirectional LSTM processes sequences from both forward and backward directions, allowing the model to capture context from both ends, which can significantly enhance classification accuracy.

from keras.layers import Bidirectional

# Add a Bidirectional LSTM layer
model = Sequential()
model.add(Embedding(input_dim=max_features, output_dim=128, input_length=maxlen))
model.add(Bidirectional(LSTM(64, return_sequences=False)))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

3. Combining LSTM with CNN

Combining LSTM with Convolutional Neural Networks (CNNs) allows the model to capture both local features (via CNNs) and sequential dependencies (via LSTM), enhancing its ability to classify complex texts.

Summary

This episode introduced the basics of text classification using LSTM, explaining how to build and implement a model in Python. LSTM is a powerful tool for handling sequence data and learning long-term dependencies. Next time, we will explore the attention mechanism, focusing on how models can attend to important words within sequences for more sophisticated text processing.

Next Episode Preview

Next time, we will discuss the implementation of attention mechanisms, showing how models can focus on crucial words to improve text processing capabilities. Stay tuned!

---

Notes

1. Vanishing Gradient Problem: An issue where gradients become too small during training, hindering learning.
2. Padding: A method to ensure all input data has the same length by adding padding values.
3. Bidirectional LSTM: An LSTM variant that processes sequences in both directions to better capture context.