Touring the Quickstart dataset

In this tutorial, we will dive into the quickstart dataset and show you some cool features of Encord Active. You will go through the following steps:

1. Opening the quickstart dataset.
2. Finding and tagging outliers.
3. Exporting samples for relabeling or other actions.
4. Figuring out what metrics influence model performance.

ℹ️

Note

This tutorial assumes that you have installed encord-active.

1. Opening the quickstart dataset

To open the quickstart dataset run:

encord-active quickstart

Encord Active downloads the dataset and opens the UI in your browser.

ℹ️

Note

If the terminal just seems to get stuck and nothing happens, try visiting http://localhost:8000 in your browser.

About the dataset

The Quickstart dataset contains images and labels for 200 random samples from the COCO 2017 validation set with a pre-trained MASK R-CNN RESNET50 FPN V2 model.

2. Finding and tagging outliers

First, we will find and tag image outliers.

Identifying metrics with outliers

When you open Encord Active, you will start on the landing page.

Click the quickstart project. The "Summary" page for the project appears:

On the Summary > Data page, you can see all the data outliers that Encord Active automatically found based on all the Quality Metrics that were computed for the images.

👍

Tip

You check the metric of distribution for outliers using the Metrics Distribution graph and selecting the outlier from the drop down. Good places to start could be the "Brightness" and "Sharpness" entries.

On the Summary > Annotations page, you can see annotation outliers that Encord Active automatically found.

Tagging outliers

To tag an image identified as an outlier, go to the Explorer page and select one or more images. The TAG button is enabled. Click the TAG button and specify a new tag.

Once the tag is created, you can add the tag to the images by selecting images and clicking the TAG button and selecting a tag from the list of tags.

👍

Tip

Use the Explorer scatter plot graph, the filter feature (click the FILTERS button repeatedly to add multiple filters including tags or labels) and queries (queries are only available from the web-app) to specify the images you want to tag.

Once you are satisfied with your tagged subset of images, you can move on to exporting.

ℹ️

Note

Mulitple subsets can be created in the web-app.

3. Figuring out what metrics influence model performance

Encord Active also allows you to figure out which metrics influence your model performance the most.
In this section, we'll go through a subset of those:

• The high level view of model performance.
• Inspecting model performance for a specific metric.

The high level view of model performance

mAP and mAR scores

First, navigate to the Predictions > Summary page where you find multiple insights into your model performance.

The first section displays the mean Average Precision (mAP), mean Average Recall (mAR), true positive (TP), false positive (FP), and false negative (FN) of your model based on the IOU threshold set in the top of the page.

Dragging the IOU slider changes the scores.
You can also choose to see the aggregate score for certain classes by selecting them in the drop-down to the left.

Metric importance and correlation

Scrolling down the Summary page, the importances and correlations of your model performance display as functions of metrics.

From this overview, you can see that, for example "Confidence" has a high importance for the model performance.

Next, we can jump to the Metric Performance page and take a closer look at exactly how the model performance is affected by this metric. However, we want to show you the rest of this page prior to doing this.

You can skip straight ahead to the Inspecting Model Performance for a Specific Metric if you are too curious to wait.

Before jumping into specific metrics, we want to show you the decomposition of the model performance based on individual classes. Scrolling down the Summary page, the Per Class average precision, average recall, and precision recall curve scores for each individual class appears.

Inspecting model performance for a specific metric

Using the Metric Performance and Explorer pages you can see how specific metrics affect the model performance:

1. Go to Predictions > Metric Performance.
2. Select the "Confidence" metric from the Metric drop down list.

The plot shows the precision and the false negative rate as a function of the selected metric; "Confidence" in this case.

3. Go to Predictions > Explorer.
4. Filter the data based on a data or prediction metric and the prediction outcome.

ℹ️

Note

Queries are only available in the web-app version of Active.

Summary

This concludes the tour of the quickstart dataset. In this tutorial we covered opening the quickstart dataset, finding image outliers, and analysing the performance of an off-the-shelf object detection model on the dataset. By now, you should have a good idea about how Encord Active can be used to understand your data, labels, and model.

Next steps

• We have only covered a few of the page in the app briefly in this tutorial.
• To learn more about concrete actionable steps you can take to improve your model performance, we suggest that you have a look at the Workflow section.
• If you want to learn more about the existing metrics or want to build your own metric function, the Metrics section is where you should continue reading.
• Finally, we have also included some in-depth descriptions the Command Line Interface.