Finding Important Metrics
Visualise the relationship between your model performance and metrics
With this workflow, you will be able to identify the most important metrics for your model performance and prioritise further data exploration and actions.
Prerequisites: Dataset, Labels, Predictions
Navigate to the Model Quality > Metrics tab.
Select label classes to include in the top left drop down menu.
Determine IoU threshold using the slider in the top bar. By default, IoU threshold is set to 0.50.
Next, Encord Active automatically computes mAP, mAR, Metric Importance, and Metric Correlation.
Metric importance: Measures the strength of the dependency between a metric and model performance. A high value means that the model performance would be strongly affected by a change in the metric. For example, a high importance in 'Brightness' implies that a change in that quantity would strongly affect model performance. Values range from 0 (no dependency) to 1 (perfect dependency, one can completely predict model performance simply by looking at this metric).
Metric correlation: Measures the linearity and direction of the dependency between a metric and model performance. Crucially, this metric tells us whether a positive change in a metric will lead to a positive change (positive correlation) or a negative change (negative correlation) in model performance. Values range from -1 to 1.
Metrics denoted with (P) are Prediction-level metrics and metrics with (F) are Frame-level metrics.
Once an important metric is identified, navigate to Performance By Metric in the Model Quality tab.
Select the important metric you want to understand using the drop-down menu on the top bar.
By default, the performance chart is shown in aggregate for all classes, optionally you can choose to decompose performance by class or select individual classes to be shown in the top left drop down menu.
The plot shows the True Positive Rate (TPR) and the False Negative Rate (FNR) by metric to help you identify which metric characteristics you model have a hard time predicting.
Metric importance plots indicate that Object Area - Relative (P) is an important metric that has an important relationship with the model performance.
In this case, go to Performance By Metric page and choose "Object Area - Relative (P)" in the Select metric for your predictions drop down menu. Here, you can understand why Object Area - Relative (P) has a relationship with the model performance, and you can act based upon insights you got from here. Let's examine Object Area - Relative (P) metric:
As indicated in the details, this metric refers to the object area as a percentage of total image area. Blue dashed horizontal line (around 0.17 TPR and 0.77 FNR) is the average true positive rate and false negative rate of the selected classes, respectively. So, what we get from the above graph is that objects, whose area is less than the 0.24%, have a very low performance. In other words, the model predictions that are small are very often incorrect. Similarly, labelled objects, for which the area is small, has a high false negative rate.
Based on this insight, you may improve your model with several actions, such as:
- Filtering model predictions to not include the smallest objects
- Increasing your model's input resolution
- Increasing the confidence threshold for small objects