[AI from Scratch] Episode 249: Overview of FastText

Recap and Today’s Theme

Hello! In the previous episode, we covered the GloVe model, which captures the semantic relationships between words using a co-occurrence matrix. However, GloVe has limitations in dealing with out-of-vocabulary words.

This time, we will discuss FastText, a word embedding method developed to overcome this issue. FastText utilizes subword (partial word) information, making it effective even for unknown words and variations in spelling. In this article, we will explore the basic concept of FastText, its mechanism, and how it differs from other methods.

What is FastText?

1. Basic Concept of FastText

FastText is a word embedding method developed by Facebook AI Research, positioned as an extension of Word2Vec. Like Word2Vec, it represents words as low-dimensional vectors, but the significant difference lies in its use of subword (partial word) information.

FastText divides each word into subwords (n-grams) and combines them to learn the word vectors. This approach allows FastText to effectively generate embedding vectors for new or morphologically varied words.

2. What are Subwords?

Subwords are small segments (substrings) of a word. For example, if the word “running” is split into 3-grams (three-character substrings), the subwords would be “run,” “unn,” “nni,” “nin,” and “ing.” By using this subword information, FastText can capture the meaning of words in more detail.

How FastText Works

1. Leveraging Subword Information

FastText learns by breaking down each word into subwords and learning vectors for each subword. The learning process follows these steps:

1. Decomposing Words: Words are divided into n-grams of a specified length. For example, splitting “cat” into 3-grams produces subwords like “” (where “<” and “>” indicate the start and end of the word).
2. Learning Subword Vectors: Vectors are learned for each subword. The overall word vector is then represented as the sum or average of the vectors of its constituent subwords.
3. Running Prediction Tasks: Using the combined subword vectors, the model predicts other words in the context (Skip-gram) or predicts the center word from the surrounding words (CBOW).

2. Handling Out-of-Vocabulary Words

One of FastText’s major strengths is its ability to handle words not present in the training data. Unlike traditional Word2Vec, which cannot generate vectors for unknown words, FastText uses subword information to estimate vectors for new words. This flexibility allows it to accommodate misspellings and morphological variations.

3. Selecting Subwords and Hyperparameters

Subword selection involves several hyperparameters:

• Length of n-grams: Typically, n-grams of length between 3 and 6 are chosen. This range provides subwords of appropriate size, capturing the meaning of words effectively.
• Minimum and Maximum n-gram Length: For instance, setting a minimum of 3 and a maximum of 6 means words are divided into subwords ranging from 3 to 6 characters.

Properly setting these hyperparameters helps optimize the model’s performance.

Advantages of FastText

1. Strong Adaptability to Unknown Words and Morphological Variations

By using subword information, FastText is effective at handling unknown words and morphological changes. This makes it particularly useful in languages with many inflectional variations (e.g., English past and plural forms, French gender and number changes).

2. Computational Efficiency

FastText employs Skip-gram and CBOW approaches like Word2Vec, allowing it to efficiently train on large datasets. Moreover, by leveraging subword information, it can generate high-precision word embeddings even with limited data.

3. Improved Similarity Calculations

With the consideration of subword information, FastText’s word vectors more accurately reflect semantic meaning. This improvement leads to higher precision in tasks such as finding similar words and analogy reasoning.

Comparison with Other Word Embedding Methods

1. Differences from Word2Vec

• Word2Vec: Represents each word with a single vector, struggling to handle unknown words.
• FastText: Uses subword information, allowing it to generate vectors for both in-vocabulary and out-of-vocabulary words, making it a more flexible model.

2. Differences from GloVe

• GloVe: A model based on global statistical information from co-occurrence matrices, learning word vectors that reflect overall co-occurrence patterns.
• FastText: Learns using local context information and subwords, enabling it to handle unknown and newly encountered words, unlike GloVe.

Applications of FastText

1. Document Classification

By generating word vectors using subword information and aggregating these vectors for entire documents, FastText is applied in text classification tasks. It shows high performance in tasks such as categorizing news articles and filtering spam emails.

2. Machine Translation

FastText’s ability to handle unknown words and spelling variations makes it particularly useful in machine translation, improving translation accuracy.

3. Named Entity Recognition and Spell Correction

FastText also finds applications in named entity recognition and spell correction. Its use of subword information is effective in recognizing complex morphological variations and correcting spelling errors.

Challenges of FastText

1. Model Size

Since FastText incorporates subword information, its model size can become larger than that of Word2Vec. This is because it must store all subword vectors. Solutions include compressing the model or limiting the number of subwords.

2. Context Dependency Issue

FastText is a context-independent word embedding method, meaning it represents words with the same vector regardless of context. It struggles to handle polysemous words (words with multiple meanings). Combining FastText with context-dependent methods like BERT can address this limitation.

Summary

This episode covered the basics of FastText, its mechanism, and how it differs from other word embedding methods. FastText utilizes subword information to flexibly handle unknown words and morphological variations, making it highly effective in tasks such as document classification and machine translation. However, challenges such as model size and context dependency remain.

Next Episode Preview

Next time, we will discuss implementing document classification, explaining how to build a news article categorization model with specific code examples. Stay tuned!

---

Notes

1. Subword: Small substrings that make up a word; used in FastText for learning word vectors.
2. n-gram: A unit consisting of n consecutive characters or words. FastText uses n-grams to create subwords.
3. Context-Independent: Words are represented by the same vector regardless of context, not reflecting meaning changes based on context.