Computer vision metrics

Object Detection Metrics

Object detection models predict bounding boxes around objects and classify them.

1. Intersection over Union (IoU): Measures the overlap between predicted and ground truth bounding boxes.

$$\text{IoU}=\frac{\text{Area of Overlap}}{\text{Area of Union}}$$

2. Average Precision (AP): The area under the Precision-Recall curve for a specific class, calculated at a specific IoU threshold.

$$\text{AP}={\int }_0^{1}p(r)dr$$

Where $p(r)$ is the precision at recall level $r$.

3. Mean Average Precision (mAP): The mean of AP values across all object classes, often calculated at multiple IoU thresholds.

$$\text{mAP}=\frac{1}{n}\sum _{i=1}^n{\text{AP}}_i$$

Semantic Segmentation Metrics

In semantic segmentation, each pixel belongs to one class.

1. Pixel Accuracy: The proportion of correctly classified pixels among all pixels. Can be dominated by large classes (background).

$$\text{Pixel Accuracy}=\frac{\text{Number of correctly classified pixels}}{\text{Total number of pixels}}$$

2. Mean Intersection over Union (mIoU): The average IoU across all classes.

$${\text{IoU}}_c=\frac{{\text{TP}}_c}{{\text{TP}}_c+{\text{FP}}_c+{\text{FN}}_c}$$ $$\text{mIoU}=\frac{1}{n_c}\sum _{c=1}^{n_c}{\text{IoU}}_c$$

Where:

• ${\text{TP}}_c$ is the number of true positive pixels for class $c$
• ${\text{FP}}_c$ is the number of false positive pixels for class $c$
• ${\text{FN}}_c$ is the number of false negative pixels for class $c$
• $n_c$ is the number of classes

3. Frequency Weighted IoU (FWIoU): A weighted version of mIoU that accounts for class imbalance.

$$\text{FWIoU}=\frac{1}{{\sum }_{k=1}^{n_c}{t}_k}\sum _{c=1}^{n_c}{t}_c\cdot {\text{IoU}}_c$$

Where $t_c$ is the total number of pixels that truly belong to class $c$.

4. Dice Coefficient: (F1 Score equivalent for segmentation).

$$Dic{e}_{class}=\frac{2\ast T{P}_{class}}{(2\ast T{P}_{class}+F{P}_{class}+F{N}_{class})}=\frac{2\ast IoU}{(IoU+1)}$$

Instance Segmentation Metrics

Instance segmentation involves both semantic segmentation and instance differentiation (separating individual objects).

1. Mask AP: Average Precision calculated based on IoU between predicted and ground truth masks instead of bounding boxes.

2. Panoptic Quality (PQ): Combines recognition and segmentation quality for panoptic segmentation tasks.

$$\text{PQ}=\underset{\text{segmentation quality (SQ)}}{\underbrace{\frac{{\sum }_{(p,g)\in TP}\text{IoU}(p,g)}{|TP|}}}\times \underset{\text{recognition quality (RQ)}}{\underbrace{\frac{|TP|}{|TP|+\frac{1}{2}|FP|+\frac{1}{2}|FN|}}}$$

Where:

• $p$ is a predicted segment
• $g$ is a ground truth segment
• $TP$, $FP$, $FN$ are true positives, false positives, and false negatives

Image Generation and Synthesis Metrics

These metrics evaluate the quality, diversity, and realism of generated images.

1. Fréchet Inception Distance (FID): Measures the distance between the distribution of features from generated images and real images, extracted using a pre-trained Inception network. Compares mean and covariance of these feature distributions. Lower values indicate that generated images are more similar to real images in terms of deep features.

$$\text{FID}=||{\mu }_r-{\mu }_g|{|}^2+\text{Tr}({\Sigma }_r+{\Sigma }_g-2\sqrt{{\Sigma }_r{\Sigma }_g})$$

Where:

• ${\mu }_r$ and ${\mu }_g$ are the mean feature representations of real and generated images
• ${\Sigma }_r$ and ${\Sigma }_g$ are the covariance matrices of the feature representations

2. Inception Score (IS): Measures the quality (sharpness, recognizability by a pre-trained Inception network) and diversity of generated images.

$$\text{IS}=\exp \left({\mathbb{E}}_x[\text{KL}(p(y|x)||p(y))]\right)$$

Where:

• $p(y|x)$ is the conditional class distribution for image $x$
• $p(y)$ is the marginal class distribution

3. Structural Similarity Index (SSIM): Measures the perceptual difference between two images based on luminance, contrast, and structure. Ranges from -1 to 1 (or 0 to 1). 1 indicates perfect similarity. Aims to be more consistent with human perception than PSNR/MSE.

$$\text{SSIM}(x,y)=\frac{(2{\mu }_x{\mu }_y+c_1)(2{\sigma }_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\sigma }_x^2+{\sigma }_y^2+c_2)}$$

Where:

• ${\mu }_x$ and ${\mu }_y$ are the average pixel values
• ${\sigma }_x^2$ and ${\sigma }_y^2$ are the variances
• ${\sigma }_{xy}$ is the covariance
• $c_1$ and $c_2$ are constants to avoid division by zero

4. Peak Signal-to-Noise Ratio (PSNR): Measures the quality of reconstructed images in tasks like denoising or super-resolution. Ratio between the maximum possible power of a signal and the power of corrupting noise that affects its fidelity. Based on MSE.

$$\text{PSNR}=10\cdot {\log }_{10}\left(\frac{{\text{MAX}}_I^2}{\text{MSE}}\right)$$

Where:

• ${\text{MAX}}_I$ is the maximum possible pixel value
• $\text{MSE}$ is the mean squared error between images

5. Learned Perceptual Image Patch Similarity (LPIPS): Measures perceptual similarity using deep features from pre-trained networks (VGG, AlexNet). Aims to align better with human perception of similarity than pixel-wise metrics like MSE.

3D Vision Metrics

Metrics for evaluating 3D reconstruction, depth estimation, and point cloud processing.

1. Depth Estimation Metrics:

   • Mean Absolute Error (MAE): Average absolute difference between predicted and ground truth depths.
   • Root Mean Squared Error (RMSE): Square root of the average squared differences.
   • Threshold Accuracy: Percentage of pixels with ratio of predicted to ground truth depth within threshold $t$ (commonly $t\in 1.25,1.25^2,1.25^3$).

2. Point Cloud Metrics:

   • Chamfer Distance: Measures the average distance from each point in one point cloud to its nearest neighbor in another point cloud.

   $\text{CD}(S_1,S_2)=\frac{1}{|S_1|}{\sum }_{x\in S_1}{\min }_{y\in S_2}||x-y|{|}_2^2+\frac{1}{|S_2|}{\sum }_{y\in S_2}{\min }_{x\in S_1}||y-x|{|}_2^2$

   • Earth Mover’s Distance (EMD): The minimum “cost” to transform one point cloud into another.
   • F-Score: The harmonic mean of precision and recall at a specific distance threshold.

3. 3D Reconstruction Metrics:

   • Volumetric IoU: The intersection over union of 3D volumes.
   • Surface-to-Surface Distance: The average distance between reconstructed and ground truth surfaces.

Human Pose Estimation Metrics

1. Percentage of Correct Keypoints (PCK): The percentage of predicted keypoints that fall within a distance threshold of the ground truth keypoints.

2. Mean Per Joint Position Error (MPJPE): The average Euclidean distance between predicted and ground truth joint positions.

3. Object Keypoint Similarity (OKS): Similar to IoU but for keypoints, accounting for keypoint visibility and scale.

$$\text{OKS}=\frac{{\sum }_i\exp (-d_i^2/(2s^2{k}_i^2))\delta (v_i>0)}{{\sum }_i\delta (v_i>0)}$$

Where:

• $d_i$ is the Euclidean distance between predicted and ground truth keypoint $i$
• $s$ is the object scale
• $k_i$ is the per-keypoint constant
• $v_i$ is the visibility flag for keypoint $i$

Links

• COCO Evaluation Metrics
• Scikit-image Comparison Metrics
• PyTorch Vision Metrics