[AI from Scratch] Episode 294: Evaluation Metrics in Computer Vision — Accuracy, IoU, mAP, and More

Recap and Today’s Theme

Hello! In the previous episode, we covered pose estimation using OpenPose, a technique for detecting human joint positions and analyzing poses. OpenPose has applications in fields like sports analysis, rehabilitation support, and entertainment.

In this episode, we will focus on evaluation metrics for computer vision models. When working on tasks such as image classification, object detection, and segmentation, it is essential to measure the performance of models using metrics like Accuracy, IoU (Intersection over Union), and mAP (mean Average Precision). Understanding these metrics provides the foundation for evaluating and improving model performance.

What are Computer Vision Evaluation Metrics?

In computer vision tasks, various evaluation metrics are used to measure model performance. These metrics quantify how accurately a model can classify, detect, or segment objects in an image.

Main Tasks and Evaluation Metrics

1. Image Classification: Predicting which category an object in an image belongs to. Metrics like Accuracy and F1 score are typically used.
2. Object Detection: Predicting both the position (bounding box) and category of objects within an image. The primary metrics are IoU and mAP.
3. Segmentation: Predicting the category of objects at the pixel level within an image. Common metrics include IoU and Dice coefficient.

Evaluation Metrics for Image Classification

1. Accuracy

Accuracy is the most basic metric for classification tasks, representing the proportion of correctly classified samples.

• Formula:
  [
  \text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}}
  ]

While easy to understand, accuracy can be misleading when dealing with imbalanced classes. In such cases, F1 score or other metrics are often used together with accuracy.

2. F1 Score

The F1 score is the harmonic mean of precision and recall, making it useful when class imbalance exists.

• Precision: The proportion of true positives among all positive predictions.
• Recall: The proportion of true positives among all actual positives.
• F1 score formula:
  [
  \text{F1 Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}
  ]

The F1 score balances precision and recall, making it useful when one of them is significantly higher than the other.

Evaluation Metrics for Object Detection

In object detection, models must predict both the position (bounding box) and the category of objects. The two main metrics used for object detection are IoU and mAP.

1. IoU (Intersection over Union)

IoU measures how well the predicted bounding box overlaps with the ground truth box.

• Formula:
  [
  \text{IoU} = \frac{\text{Area of Overlap between Prediction and Ground Truth}}{\text{Area of Union between Prediction and Ground Truth}}
  ]

IoU ranges from 0 to 1, with 1 indicating perfect overlap. In object detection tasks, an IoU of 0.5 or higher is typically considered a correct prediction.

2. mAP (mean Average Precision)

mAP is a comprehensive metric for evaluating object detection models, representing the average precision over all classes.

• AP (Average Precision): The area under the Precision-Recall curve for a given class, calculated based on IoU thresholds.
• mAP is the mean AP across all classes.

mAP provides a holistic view of the detection model’s performance, accounting for both precision and recall at various IoU thresholds.

Evaluation Metrics for Segmentation

In image segmentation, the goal is to accurately predict the category of each pixel. Two commonly used metrics are IoU and Dice coefficient.

1. IoU (Segmentation)

For segmentation tasks, IoU is calculated similarly to object detection but at the pixel level.

• Segmentation IoU formula:
  [
  \text{IoU} = \frac{\text{Number of Overlapping Pixels}}{\text{Number of Pixels in the Union of Prediction and Ground Truth}}
  ]

2. Dice Coefficient

The Dice coefficient is another metric used in segmentation tasks, giving more weight to overlapping pixels compared to IoU. It is commonly used in medical imaging.

• Dice coefficient formula:
  [
  \text{Dice} = \frac{2 \times \text{Number of Overlapping Pixels}}{\text{Total Number of Predicted Pixels} + \text{Total Number of Ground Truth Pixels}}
  ]

The Dice coefficient ranges from 0 to 1, with higher values indicating better overlap.

Practical Example of Evaluation Metrics in Computer Vision

Now that we understand the evaluation metrics, let’s implement a simple example in Python to calculate IoU and mAP for an object detection task.

Required Libraries

pip install numpy

IoU Calculation Example

import numpy as np

def calculate_iou(box1, box2):
    # Get the coordinates of the boxes
    x1, y1, x2, y2 = box1
    x1_prime, y1_prime, x2_prime, y2_prime = box2

    # Calculate the coordinates of the intersection box
    x_left = max(x1, x1_prime)
    y_top = max(y1, y1_prime)
    x_right = min(x2, x2_prime)
    y_bottom = min(y2, y2_prime)

    if x_right < x_left or y_bottom < y_top:
        return 0.0

    # Calculate the area of the intersection and the union
    intersection_area = (x_right - x_left) * (y_bottom - y_top)
    box1_area = (x2 - x1) * (y2 - y1)
    box2_area = (x2_prime - x1_prime) * (y2_prime - y1_prime)
    union_area = box1_area + box2_area - intersection_area

    return intersection_area / union_area

# Test bounding boxes
boxA = [50, 50, 150, 150]
boxB = [60, 60, 140, 140]

iou = calculate_iou(boxA, boxB)
print(f"IoU: {iou:.2f}")

Code Explanation

• calculate_iou(): A function that calculates IoU between two bounding boxes.
• boxA and boxB are bounding boxes representing objects in an image.
• The IoU result ranges from 0 to 1, with higher values indicating better overlap between the boxes.

Summary

In this episode, we explored key computer vision evaluation metrics such as accuracy, IoU, mAP, and the Dice coefficient. These metrics are crucial for accurately assessing model performance in image classification, object detection, and segmentation tasks. Understanding and implementing these metrics helps build a solid foundation for evaluating and improving computer vision models.

Next Episode Preview

In the next episode, we will delve into the basics of 3D vision, focusing on how to handle depth information. Learn about techniques that enable 3D data analysis and their applications!

---

Notes

• IoU (Intersection over Union): A metric used in object detection and segmentation to evaluate the overlap between predicted and actual regions.
• mAP (mean Average Precision): A comprehensive metric used in object detection to measure model performance across all classes.