[AI from Scratch] Episode 191: Evaluation Metrics for Image Generation — FID Score and Other Methods

Recap: Pix2Pix

In the previous episode, we covered Pix2Pix, a model for image-to-image translation. Pix2Pix can be applied to various transformation tasks, such as colorizing black-and-white images or generating realistic images from sketches. While it uses the collaboration of a generator and a discriminator to create high-quality images, it’s crucial to understand how to evaluate the quality of these generated images.

This time, we will discuss evaluation metrics for image generation, focusing particularly on the FID score (Fréchet Inception Distance) and other methods used to quantitatively evaluate image quality.

What Are Image Generation Evaluation Metrics?

To evaluate the quality of images produced by image generation models, quantitative evaluation metrics are essential. Human judgment alone is not sufficient, so methods that mechanically measure image quality are necessary.

Some of the primary evaluation metrics include:

• FID Score (Fréchet Inception Distance)
• IS (Inception Score)
• PSNR (Peak Signal-to-Noise Ratio)
• SSIM (Structural Similarity Index)

1. FID Score (Fréchet Inception Distance)

The FID Score is one of the most widely used metrics for evaluating the quality of generated images. It measures the distance between the generated images and real data (training data). Specifically, it evaluates the similarity between the distributions of the features extracted from images, providing a quantitative measure of how closely the generated images resemble the real ones.

How to Calculate the FID Score

1. The Inception Network is used to extract feature vectors from both the generated images and the real images.
2. The feature vectors are modeled as multidimensional normal distributions.
3. The Fréchet distance (the difference between the distributions) between the generated and real images is calculated and used as the score.

Interpreting the FID Score

A lower FID score is better. The lower the score, the closer the generated images are to the real ones. For instance, when generating realistic landscapes or human faces, the goal is to achieve a low FID score, indicating that the images are similar to real-world data.

2. IS (Inception Score)

The Inception Score (IS) is another metric developed to evaluate GAN performance. This score measures “how diverse and classifiable” the generated images are. Specifically, it checks if the generated images belong to different classes and how confidently they fit into those classes.

How to Calculate the IS

1. The Inception Network is used to classify the generated images.
2. The diversity of classifications (whether the images belong to different classes) and the confidence (likelihood of belonging to a specific class) are measured and combined into a score.

Interpreting the IS

A higher IS is better. It indicates that the generated images are diverse and can be classified into distinct categories. However, IS has some limitations as it may not fully evaluate the realism of the images from a human perspective.

3. PSNR (Peak Signal-to-Noise Ratio)

PSNR is mainly used for image reconstruction tasks to measure how close the reconstructed image is to the original. It quantifies the level of noise in the image, evaluating the accuracy of reconstruction compared to the original image.

How to Calculate the PSNR

PSNR calculates the pixel-by-pixel difference between the reconstructed and original images and outputs the inverse of this difference as the score. A higher score indicates less noise and greater similarity to the original image.

Interpreting the PSNR

A higher PSNR is better. It suggests that the image has low noise and closely matches the original. However, because PSNR is based on pixel-level errors, it may not accurately reflect visual realism.

4. SSIM (Structural Similarity Index)

SSIM measures the structural similarity between images. Rather than focusing on pixel-by-pixel differences, SSIM assesses the overall structure of the image, providing a more accurate representation of visual realism. It is often used to evaluate the quality of images after compression.

How to Calculate SSIM

SSIM compares the brightness, contrast, and structural similarity between the reconstructed and original images to derive a score, indicating how visually similar the images are.

Interpreting the SSIM

An SSIM score close to 1 is better. A score near 1 indicates that the reconstructed image is highly similar to the original in terms of structure and visual quality.

Challenges in Evaluating Image Generation

Although evaluation metrics are crucial, accurately measuring the quality of generated images remains a challenge. Quantitative metrics like FID and IS may not fully capture the realism and naturalness of images. Therefore, it is recommended to combine these metrics with visual inspection or subjective evaluation to ensure a comprehensive assessment.

Summary

In this episode, we explored evaluation metrics for image generation. The FID score is one of the most commonly used methods, providing a quantitative evaluation of how closely the generated images resemble real data. However, other metrics such as Inception Score (IS), PSNR, and SSIM also offer various perspectives on image quality. Next time, we will dive into text generation models, exploring the mechanisms and applications of language models for automatic text generation.

---

Preview of the Next Episode

Next time, we will discuss the details of text generation models. We will learn about the mechanisms of text generation using language models and explore their applications, discovering how text can be automatically generated. Stay tuned!

---

Annotations

1. FID Score (Fréchet Inception Distance): An evaluation metric that measures the distance between the distributions of generated images and real data. Lower scores are better.
2. Inception Score (IS): A metric that evaluates the diversity and classifiability of generated images. Higher scores are better.
3. PSNR (Peak Signal-to-Noise Ratio): Used in image reconstruction tasks to measure pixel-level differences between reconstructed and original images. Higher scores indicate better quality.
4. SSIM (Structural Similarity Index): Evaluates the structural similarity between images. Scores closer to 1 indicate higher similarity.