[AI from Scratch] Episode 259: Thesaurus and WordNet

Recap and Today’s Theme

Hello! In the previous episode, we discussed cosine similarity for text comparison, a technique used to measure the similarity between documents by converting them into vectors. Cosine similarity is widely applied in text search and duplicate detection.

Today, we will cover Thesaurus and WordNet, resources that capture semantic relationships between words, playing an important role in natural language processing (NLP). This episode introduces the basic concepts, structure, and NLP applications of Thesaurus and WordNet.

What is a Thesaurus?

1. Basic Concept of a Thesaurus

A Thesaurus is a dictionary that lists words along with their synonyms and antonyms. It helps find words with similar meanings or opposites. For example, a thesaurus entry for “happy” might include synonyms like “joyful,” “cheerful,” and “content,” as well as antonyms like “sad” and “unhappy.”

2. Structure of a Thesaurus

A thesaurus typically provides a list of synonyms and antonyms for each word, grouping them based on their meanings. It is useful in various fields, including writing support, composition, and translation, as it enables the selection of alternative expressions and enhances diversity in text.

3. Digital Thesaurus

Unlike traditional print dictionaries, digital thesauruses offer search and filtering capabilities. They can also be algorithmically utilized to calculate semantic distances between words, providing more dynamic and computational use cases.

What is WordNet?

1. Basic Concept of WordNet

WordNet is a large lexical database for English that models the semantic relationships between words. Developed at Princeton University, WordNet groups words into synsets (synonym sets) based on their meanings and defines relationships between these synsets.

WordNet models various semantic relationships, such as synonyms, antonyms, hypernyms (superordinate terms), hyponyms (subordinate terms), and meronymy (part-whole relationships), making it an essential resource for understanding word meanings.

2. Structure of WordNet

WordNet organizes words using relationships such as:

• Synset (Synonym Set): A group of words with similar meanings. For example, “car,” “automobile,” and “motorcar” belong to the same synset.
• Hypernym: A word representing a general concept (e.g., “vehicle” is the hypernym of “car”).
• Hyponym: A word representing a more specific concept (e.g., “sedan” or “SUV” are hyponyms of “car”).
• Antonym: Words with opposite meanings (e.g., “hot” and “cold”).
• Meronym: Words that denote a part of a whole (e.g., “wheel” is a meronym of “car”).

3. Using WordNet

WordNet is widely used to understand the semantic relationships between words, supporting various NLP tasks such as semantic similarity calculation, word sense disambiguation, and question answering systems.

Applications of Thesaurus and WordNet in NLP

1. Semantic Similarity Calculation

Thesaurus and WordNet can be used to calculate the semantic distance between words, which helps in building algorithms that measure the similarity between documents. WordNet is particularly effective in evaluating the semantic closeness of words using information on synsets, hypernyms, and hyponyms.

2. Word Sense Disambiguation

Thesaurus and WordNet help resolve word ambiguities when a word has multiple meanings. For example, the word “bank” could mean “the side of a river” or “a financial institution.” WordNet can select the appropriate synset based on the context to clarify the intended meaning.

3. Question Answering Systems

WordNet enables the evaluation of semantic relationships between a user’s query and potential answers. By utilizing synonyms, hypernyms, and related words found in the question, WordNet allows for broader and more accurate answer matching.

4. Text Classification and Clustering

By considering the semantic relationships between words, Thesaurus and WordNet can improve the accuracy of document classification and clustering. Assigning related words to the same category allows for a more precise evaluation of document similarity.

Implementation Example Using WordNet

Here is an example of how to use Python’s nltk library to access basic WordNet functions.

1. Installing and Setting Up WordNet

First, install the nltk library and download the WordNet data:

pip install nltk

In the Python code, import WordNet and download the necessary data:

import nltk
nltk.download('wordnet')
nltk.download('omw-1.4')
from nltk.corpus import wordnet as wn

2. Retrieving Synsets

The following code shows how to retrieve synsets (groups of synonyms) for a word:

# Retrieve synsets for "car"
synsets = wn.synsets('car')

# Display synset details
for synset in synsets:
    print(f"Name: {synset.name()}, Definition: {synset.definition()}")

This code retrieves synsets associated with “car” and displays their names and definitions.

3. Retrieving Hypernyms and Hyponyms

You can use WordNet to obtain hypernyms (general concepts) and hyponyms (specific concepts) for a word:

# Get a synset for "car"
car_synset = wn.synset('car.n.01')

# Retrieve hypernyms
hypernyms = car_synset.hypernyms()
print("Hypernyms:", [hypernym.name() for hypernym in hypernyms])

# Retrieve hyponyms
hyponyms = car_synset.hyponyms()
print("Hyponyms:", [hyponym.name() for hyponym in hyponyms])

This code retrieves and displays the hypernyms and hyponyms of “car”.

4. Calculating Semantic Similarity

WordNet also provides a method for calculating semantic similarity between synsets based on their common hypernyms:

# Synsets for "car" and "bicycle"
car_synset = wn.synset('car.n.01')
bicycle_synset = wn.synset('bicycle.n.01')

# Calculate semantic similarity
similarity = car_synset.wup_similarity(bicycle_synset)
print(f"Semantic similarity between car and bicycle: {similarity:.2f}")

This code calculates the semantic similarity score between “car” and “bicycle”.

Limitations and Challenges of Thesaurus and WordNet

1. Language Dependency

Thesaurus and WordNet are primarily developed for English, limiting their applicability to other languages. Multilingual resources and language-specific datasets are necessary for broader usage.

2. Ambiguity of Meanings

Choosing the correct synset can be difficult when words have ambiguous meanings. Additional contextual information is required to resolve these ambiguities accurately.

Summary

This episode introduced Thesaurus and WordNet, their structures, and applications in NLP. These resources help understand semantic relationships between words and are used in various tasks, such as semantic similarity calculation and word sense disambiguation.

Next Episode Preview

Next time, we will cover the basics of dialogue systems, exploring how chatbots work and learning simple implementation methods.

---

Notes

1. Synset (Synonym Set): A group of synonyms in WordNet, categorized by different meanings of a word.
2. Hypernym: A word that represents a more general category of a concept.
3. Hyponym: A word that represents a more specific type of a concept.