Benchmark QA

Benchmark QA Projects rely on Manual QA Projects that share the same Ontology as a benchmark that provides ground-truth labels. To create a Benchmark QA Project, you must first complete a Manual QA Project with the same Ontology.

The terms ‘Automated QA (Auto QA)’ and ‘Benchmark QA (BMQA)’ are used interchangeably throughout the documentation.

​

Creating benchmark QA Projects

Before creating a Benchmark QA Project, you must complete a Manual QA Project with the same Ontology to serve as a benchmark. This benchmark project provides ground-truth labels, enabling the evaluation of labels in the Benchmark QA Project you are creating.

In the Annotate section of the navigation bar, select Annotation projects. Click the + New annotation project button to create a new Project.

​

1. Enter details

Ensure you select the Automated QA Project type. Then give your Project a meaningful title and description.

If you are part of an Organization, you see an optional Project tags drop-down. Project tags are useful for categorizing your Projects. Select as many tags as are relevant to your Project.

​

2. Attach Ontology and Datasets

A Project requires an Ontology to define the labels being applied, and at least one Dataset to determine what is being labeled.

​

Attach Ontology

Click the Attach ontology button.

Select an existing Ontology from the list using the Attach button, or create a new Ontology by clicking the New ontology button.

Click OK to attach the Ontology to the Project.

​

Attach Datasets

Click the Attach datasets button.

Select an existing Dataset from the list using the Attach button, or create a new Dataset by clicking the New Dataset button.

Click OK to attach the Dataset(s) to the Project.

​

3. Select the benchmark Project

Benchmark QA Projects rely on Manual QA Projects that share the same Ontology as a benchmark that provides ground-truth labels. To create a Benchmark QA Project, you must first complete a Manual QA Project with the same Ontology.

Select the Manual QA Project you want to use as a benchmark source.

​

4. Set up your scoring function

Click the Scoring function tab to customize the scoring function. The scoring function determines how your automated labels are evaluated.

• Single frame-level classification only evaluates classification instances.

• Dynamic benchmark is used for combinations of objects and classifications and can be calibrated by setting the relative weights of the following evaluation metrics:

  • Intersection over Union (IoU) is an evaluation metric that assesses the accuracy of labels compared to the benchmark. If labels fully overlap with those in the ground truth full points are awarded. Conversely, if there is no overlap between a label and the ground truth labels then no points are awarded.
  For Keypoints the IoU represents a measure of distance between the annotator’s point, and the benchmark point. The larger the distance, the lower the IoU score.
  • Category is an evaluation metric based on correctly identifying the ontology category. In the example above correctly identifying an ‘Apple’ awards 100 points, while an incorrect or a missing category awards no points.

👍 Tip IoU and Category values can be adjusted in the Settings tab after your project has been created.

​

5. Create Project

To create your Benchmark QA Project, click Create project in the top right corner of the screen.

---

​

Roles and permissions

---

​

Supported annotation types

Object annotation type	Benchmark QA Available?
Bounding box	✅
Rotatable bounding box	✅
Polygon	✅
Polyline	❌
Keypoint	✅
Bitmask	❌
Object primitive	❌

---

​

Performance evaluation

Results of the Benchmark QA process for each annotator can be viewed on the Performance dashboard (if the projects has benchmark quality assurance). You can also export a CSV containing a breakdown of annotator performance for each instance.

---

​

Project dashboard

Selecting a project from the list of annotation projects takes you to its ‘Project dashboard’.

This is where you monitor and manage your project. For example, you can view your project’s summary statistics, manage labeling tasks, view your team’s productivity, train models and invite collaborators.

The dashboard is split into 7 tabs:

• Summary: a high-level view of labeling and productivity statistics.
• Explore: a quick way to explore the distribution of instances and labels across data assets in the project.
• Labels: for managing all the project’s labeling activity and tasks.
• Performance: a more detailed view of your team’s labeling and productivity.
• Models: for administering models in this project.
• Export: for exporting your labeling data.
• Settings: editing project options, ontology, team collaborators, and other general project settings.

Access to each tab is associated with the various project roles as follows:

---

​

Summary

Clicking an annotation project takes you to its Summary dashboard. This dashboard has 2 components and gives you a rich visual display of your project’s progress at a high level.

‘Project task status overview’ displays the number of annotation tasks that are in either the Annotate, or Completed state.

• Annotate: The task is ready to be annotated.
• Completed: The task has been annotated, and automatically reviewed. There is no further action to be taken.

---

​

Explore

The Explore page provides interfaces to help you understand how project’s annotations are distributed amongst the data assets at both an instance and label level. It allows a deeper exploration through attributes on objects, as well as frame-level classifications.

• Instance statistics: Class distribution across data assets in the given project.
• Label statistics: Label distributions within data assets, objects and classifications.

---

​

Instance statistics

This section provides the total count of all instances across the datasets in your project.

• Project total: Shows total instances (both objects and classifications) across the project by default. To get instance statistics for individual data files, click the drop-down to select a data file.
• Select class: Shows the total instances for a particular class. This is a summary of how a given class is
  distributed across your project’s data assets. The pie chart segments show a breakdown of how that class is split across the data assets.
• Display timestamps: Flip the toggle to switch between frame numbers and timestamps for the labels.

---

​

Label statistics

This is a summary of how your labels are distributed across the project. The pie chart shows a breakdown of how many labels there are for a given class.

• Project total: Shows the total number of labels across different datasets in the project. To get label stats for individual data files, click the drop-down to select a data file.
• Objects: Click on the pie chart segment of a class to see the total number of labels and its attributes (sometimes called nested attributes) if available for that class.
• Classifications: Shows the global classification at project or individual video level. For example, location, time of day, etc.

​

Quick definitions of classes, instances and labels

• Class: Fundamental unit of the project’s ontology. For example the ontology of a project annotating traffic videos could have classes such as Car, Truck, Bicycle, and so on. For more information on objects and classifications, see Ontologies Overview.
• Instance: Specific occurrence of a class. Car(0) is an instance of the Car class, for example, it could be a specific black sedan. The single Car(0) instance can appear in a single frame or a range of frames. Therefore, instances may contain multiple labels across frames.
• Label: An frame-specific annotation of an instance. For example the annotation of Car(0) on frame 201 is a label.

---

​

Labels

The Labels page is your gateway to creating labels in your project. Access to each pane will depend on the user’s project role. We briefly summarize the purpose of each tab, and the roles which can access each below.

Role	Activity	Queue	Data	Instances
Annotator	✅	✅	❌	❌
Team Manager	✅	✅	✅	✅
Admin	✅	✅	✅	✅

The labels dashboard features the following tabs:

• Activity: View all Completed tasks, and confirm the status of labels within tasks by clicking View.
• Queue: The Queue tab is where all labeling actions are initiated, regardless of a user’s role within the project.
• Data: Use the Data tab to get an overview of every data asset in the project, regardless of status.
• Instances: The Instances tab lets you use the unique instance identifier to search the project for a specific instance, and jump directly into the editor to confirm the status of an annotation visually.

---

​

Activity

The Activity tab allows you to quickly monitor annotation activity in your project by showing tasks and providing a summary interface.

• File, Search, & Reopen: The name of the specific data unit or data asset. This is the same as the name in the dataset to which this data asset is a part of. Use the search box to filter the list by file name, and send tasks back to annotation using the ‘Reopen’ feature.
• Dataset : The dataset the data asset belongs to.
• Type : The type of the data, such as an image or video. For more on our supported data types, see our supported data documentation.
• Status: The status of this task.
• Frames : The number of frames in the data asset. For a DICOM series, this will be the number of slices.
• Submitted : Indicates when the last submit action, whether for an annotation was made against any of the labels in this data asset.
• Submitted by : Who last submitted the annotations.
• Actions : Click the View link to open the label editor. Note: this feature is only available to Team Managers and Administrators as an extra method of submissions. We advise extra caution if you decide to edit the labels from this interface. If significant work needs to be done, we strongly recommend to ‘Reopen’ the task to prevent possible errors from simultaneous edits.
• Filter: Use the filter drop-down to only show tasks with the selected status. Only filtered results will be shown in the Label Editor.

​

File, Search, and Reopen

The file column shows the name of the data asset. For files uploaded via the GUI, they keep the name they were uploaded with. For files added from your cloud storage, this will normally be the path under the bucket they are stored on.

Use the search interface to quickly filter and display only those tasks with file names matching your desired text. Even partial matches will be shown. For example: searching “fly” will return file names containing “flyover” and “flyaround.”

The Reopen button allows Administrators and Team Managers to send tasks which are currently Completed back to the annotation stage. Select your target tasks using the checkboxes in the File column to select individual assets, or select the checkbox in the column header to select all tasks, and press the Reopen button to move all selected tasks back to the annotation stage. Tasks reopened in this way will have the status Returned in the Queue tab. No labels are lost by reopening a task. The ‘Reopen’ action is only applied to tasks which are both visible (i.e. not filtered out by the file search) and selected.

​

Status

This column shows the status of this task. The Activity pane only shows assets which have had some action done on them, and therefore only reflects Completed tasks.

For a comprehensive summary of the possible task states, see the status section of the Data tab, below.

---

​

Queue

The Queue tab is where annotators look to find their next task. The Start labeling button visible throughout the project open the label editor with the next task in the queue according to the relevant task type.

The Queue tab can be used to assess the number of tasks assigned to you as an annotator and therefore estimate your likely workload. Administrators and Team Managers can also use it to quickly verify the current assignments per team member, and change assignments as necessary.

• A. File, Search, & Assign: The name of the specific data unit or data asset. This is the same as the name in the dataset to which this data asset is a part of. Use the search box to filter the list by file name, and send tasks back to annotation using the ‘Reopen’ feature.
• B. Dataset : The dataset the data asset belongs to.
• C. Type : The type of the data, such as an image or video. For more on our supported data types, see our documentation here.
• D. Status and Task: The status and category of this task.
• E. Last Edited : When the task was last edited
• F. Reserved by : Who the task has been assigned to or reserved by
• G. Actions: Depending on your collaborator role, you can initiate or reassign the task.
• H. Filter: Use the filter drop-down to only show tasks of the selected status. Only filtered results will be shown in the Label Editor.

​

File, Search, and Assign

The file column shows the name of the data asset. For files uploaded via the GUI, they keep the name they were uploaded with. For files added from your cloud storage, this will normally be the path under the bucket they are stored on.

Use the search interface to quickly filter and display only those tasks with file names matching your desired text. Even partial matches will be shown. For example: searching “fly” will return file names containing “flyover” and “flyaround.”

The ‘Assign’ button allows Administrators and Team Managers to allocate unassigned tasks to specific collaborators for annotation. Select your target tasks using the checkboxes in the File column to select individual assets, or select the checkbox in the column header to select all tasks, and press the ‘Assign’ button open the task assignment popup.

Confirm the selected tasks are as intended, then select the target collaborator from the drop-down and press assign. Tasks which have already been assigned to another collaborator, as indicated by the email in the ‘Reserved by’ column, can not be reassigned until they have first been released.

​

Status and Task

The Queue tab only shows tasks which have remaining annotation work to be done. Therefore, the stage of the task within the TMS is understood by reading the Status and Task columns together.

Benchmark QA projects contain only annotation tasks. Therefore, only tasks with the status Queued will be represented. Queued tasks are ready for annotation. For an annotation tasks to be Queued it must not be assigned to a user, and must have no submitted labels. It may have been previously assigned to a user, but subsequently released before any annotations were submitted.

​

Actions

There are two relevant actions that can be done on each task from the Queue tab. Press ‘Initiate’ to open the label editor and proceed with annotation, depending on the task type.

Additionally, Administrators and Team Managers can click the three vertical dots to open the expanded menu, to access the ‘Release task’ function. Tasks must be explicitly released before they can be reassigned.

​

Data

The Data tab gives a complete overview of all the data asset tasks in the project, regardless of their progress. Therefore, this is the first place Administrators and Team Managers should check if they want to confirm the status of a given task.

• A. File & Search: The name of the specific data unit or data asset. This is the same as the name in the dataset to which this data asset is a part of. Use the search box to filter the list by file name.
• B. Dataset : The dataset the data asset belongs to.
• C. Type : The type of the data, such as an image or video. For more on our supported data types, see our documentation here for more details.
• D. Status: The status of this task.
• E. Frames : The total frames in this data asset. This will apply to videos, image sequences and DICOM. Images always only have 1 frame.
• F. FPS : the frames per second of the data asset. This only applies for data of type video. Others will show a dash (-).
• G. Created : When the task was created. Tasks are created when the dataset containing the data asset is attached to the project.
• H. Last edited by : the last collaborator to edit the task in any capacity, and when.
• I. Actions: The Data page allows users to view the task in the label editor, as well as get a code snippet for using the SDK with this task, and confirming the edit actions via the Activity Log.
• J. Filter by : Use the filter drop-down to view only tasks with the selected Status.

​

File and Search

The file column shows the name of the data asset. For files uploaded via the GUI, they keep the name they were uploaded with. For files added from your cloud storage, this will normally be the path under the bucket they are stored on.

Use the search interface to quickly filter and display only those tasks with file names matching your desired text. Even partial matches will be shown. For example: searching “fly” will return file names containing “flyover” and “flyaround.”

​

Status

The data tab provides the most comprehensive overview of all the tasks associated with each data asset in a given project. As such, this is the first place to check to see the status of various tasks.

• Queued : The task is ready for annotation. For a task to be ‘Queued’ it must not be assigned to a user, and have no submitted labels. A queued task may have been previously assigned to a user, but subsequently released before any annotations were submitted. Queued tasks are shown in light orange.
• Assigned : An annotation task has been assigned to a specific user. Assigned tasks are shown in aqua green.
• Completed : The annotation task has been submitted and completed. Completed task status is shown in green.

​

Actions

Clicking View drops you into the label editor to do a live audit of the annotations in this data asset. The Data tab is only visible to Administrators and Team Managers and so grants great power to view any data asset. Encord advises edit actions are NOT taken from the Data tab unless you have received confirmation no one else is concurrently editing the asset.

Other possible actions include ‘API Details’ which show a popup with sample code you can use to get started with our SDK to access this particular data asset, often known as a label row in the SDK. Click ‘Activity log’ to see a popup with a graphical summary of add / edit / delete actions on this data asset indexed by annotator or ontology class. Click ‘Display logs’ in the lower right to show all actions in reverse chronological order.

---

​

Instances

The Instances tab allows Administrators and Team Managers to search within the data to directly find specific instances. Recall that an annotation instance correlates to a unique instantiation of a specific ontology class in a data asset.

For example, if you have the ‘Person’ class in your ontology, the first instance of a ‘Person’ in a given data asset will be indicated in the interface as ‘Person (0)’, the second as ‘Person (1)’ and so on. Instances, therefore, can exist in multiple frames of a data asset, and indicate the same object. Use the Instances tab to search for specific instances of objects or classifications using their Identifier.

Instance identifiers are unique at the project scope, and can be found in any of the following ways:

• From inside the label editor, by clicking on a particular instance, and then selecting ‘Copy identifier’ from the instance action menu.
• From inside exported labels, where they are known as the objectHash or classificationHash as appropriate.
• When uploading labels using the SDK, you may specify your own objectHash or classificationHash.

Once you have an identifier of interest, use the ‘Search instance’ interface to filter the instances by identifier to quickly find the instance you’re interested in. This can be particularly handy when you want to visually confirm an annotation you may not have seen before, but for which you have the identifier.

After locating your instance of interest, click View from the ‘Actions’ column to jump deeply into the dataset straight to where the instance is first annotated.

---

​

Performance

Switch between a summary, or detailed view of the Performance page to different metrics relating to your team’s performance.

​

Performance - Summary

The Summary tab of the performance dashboard provides an overview of your team’s manual labeling and productivity.

​

Task actions over time

View the number of tasks in a project that have been approved, rejected, and submitted for review over a given period of time.

• The height of a bar represents the total number of tasks.
• The height of each color within a bar represents the number of approved, rejected, and submitted tasks.

• A: Set the time period you would like to see displayed by selecting a range of dates.
• B: The Hide days without any actions toggle removes all days at which no actions were taken from the view.
• C: Download a CSV file of the data.
• D: Display the data as a bar chart, or a table. While the chart provides a clear visual representation, the table provides exact figures for a more detailed picture of your team’s performance.

​

Instance Label actions over time

View the number of instance label actions in a project that have been approved, rejected, and submitted for review over a given period of time.

• A: Set the time period you would like to see displayed by selecting a range of dates.
• B: Download a CSV file of the data.
• C: Display the data as a bar chart, or a table. While the chart provides a clear visual representation, the table provides exact figures for a more detailed picture of your team’s performance.

Within your specified time period, you can choose which dates to display by using the slider located beneath the graph.

​

Team collaborators

The ‘Team collaborators’ section shows the duration of time each project collaborator spend working on a given file.

A. ‘Data file’ displays session time collaborators spent working on individual files. ‘Project’ displays session time collaborators have spent working on the project.

B. Table entries can be filtered according to dates by clicking the range of dates, and selecting the start and end date of the period you would like to see table entries displayed for.

C. Table entries can be downloaded in CSV format by clicking the Download CSV button.

D. When lots of entries are present they will be split across a number of different pages. The number of table entries per table can be adjusted.

---

​

Performance - Details

The Details tab of the performance dashboard gives a more detailed view of your team’s labeling and productivity. The below details will be displayed for Benchmark QA projects.

​

Training performance

The ‘Training performance’ section provides you with insights into labeling activities of annotators in the project.

• Click the Re-calculate scores button to re-calculate all benchmark scores displayed on this page.

• Click the Download CSV button to download a CSV file of these records.

​

Overview tab

The Overview tab can be seen in the screenshot above. It lists the following information:

• User: The user’s email address.
• Total benchmark tasks: The number of tasks in the benchmark Project.
• Submitted benchmark tasks: The number of tasks submitted belonging to the benchmark Dataset.
• Benchmark results: The user’s percentage score against benchmark labels.
• Progress: The percentage of benchmark tasks completed relative to the total.

​

Annotator submissions

The Annotator submissions tab lists the following information:

• Annotator: The user’s email address.
• Data Unit: The task’s file name.
• Instance: The ontology instance being referred to.
• Benchmark answer: The benchmark classification and label. To view the label, click .
• Annotator answer: The annotator’s classification and label. To view the label, click .
• Score: A score measuring the annotator’s label when compared to the benchmark label.

​

Annotator session time

The Annotator session time table displays the duration of time each annotator spent labeling the tasks in their queue.

• Data file: The task’s file name.
• User: The user’s email address.
• User role: The user’s role.
• Session time: Time spent labeling tasks.

​

Submissions

The Submissions graph displays there number of submitted labels within a given range of time.

---

​

Models

The Models page allowed you to manage models linked to your project. For information on models, please see our documentation here.

---

​

Export

Use the Export page to export your data. Please see our exporting data page to learn how to do this.

---

​

Settings

The Settings tab allows you to make modifications to your project using the following tabs:

• Options - Copy a project, modify datasets, modify ontology, upload annotation instructions, modify project tags, QA settings.
• Team - Manage collaborators on a project.
• Danger zone - Delete your project.

---

​

Options

​

Copy a Project

To copy a Project, click the Copy project button in the Options section of the project’s Settings. This opens the copy Project window. From the Copy Project window, you can pick the various parts of your project you want to copy over into your new project.

​

1. Select copy options

Choose the parts of your project you want to copy.

You can copy any combination of the following assets:

• Labels: this will copy the labels within videos and image sequences of your choice.
• Models: this will copy all the models in your project along with their training logs.
• Collaborators: copy all project users with their respective roles. Project admins are copied regardless of whether this is selected.
• All datasets: all datasets will be copied, and new annotation tasks will be created for all videos and image sequences if their labels were not copied over (see next line).

The new annotation Project uses the same ontology as the original. This can be changed in the Project settings if required.

• If you do not want to copy labels, press Copy project. This creates a copy of your Project, which you can then access in the Projects tab.

• If you choose to copy over labels, you will be asked to select the data assets for which you would like labels copied over. To begin the process, press Next: configure labels. Continue to step 2. below.

​

2. Select labels to be copied

Select the data units with the labels that you want to copy into your new project.

Click Next to continue.

​

3. Configure labels

Select the statuses of the files you want copied over into your new project.

This means that all tasks will be Annotate tasks, and their status will be Queued.
All tasks will have to be re-assigned after being copied.

Click the Copy project button to complete the process.

​

Upload annotation instructions

• Click the Add instructions button to upload instructions for your annotators in PDF format.

• To ensure the best possible results, provide as much detail as possible about what you would like annotated and how precise bounding boxes should be drawn. For example, instead of saying ‘person’, consider defining what should constitute a person for your annotators - only a full person? A torso? Or should any part of a person in a frame be labeled as a ‘person’?

• Once uploaded, annotation instructions will be accessible within the Label Editor.

​

Project tags

You can add tags to a Project if you are part of an Organization.

Project tags allow you to:

• Flexibly categorize and group your Projects.

• Filter your Projects.

​

Adding and removing tags

You can add tags to your projects in:

• When creating a Project.

• In the Settings page of a project.

To add tags to your Projects in the Settings page, navigate to the Options tab and click the Project tags drop-down. Here you will see the available tags in your organization. Click on a tag to add it to a Project. You can remove a tag from your Project by clicking the same tag again, or clicking the x button next to its name.

​

Filtering Projects by tags

You can filter your Projects based on the tags they contain. Click on the Projects tab in the navigation bar, click the Filter by tags drop-down and select one or more Project tags. This will result in only Projects with the tags being selected being displayed.

​

Edit project Ontology

You can view or switch the Ontology attached to your Project.

• Click the Switch ontology button to switch the ontology linked to your Project.
  The resulting pop-up allows you to choose an existing ontology from a list, or create a new ontology for this Project.

• Click the View ontology button to view the details of the Ontology that is attached to the current Project.

​

Edit Datasets attached to a Project

The Datasets section allows you to attach or detach any number of Datasets to your Project. You must create a new Dataset in the Datasets section for it to become available in a Project’s settings.

​

Quality Assurance

The Automatic QA section in the Options tab allows you to configure the way that benchmark quality assurance is implemented.

• The Benchmark project source allows you to see which project is being used as the gold standard benchmark function against which QA is done.

• Click Edit next to the Benchmark scoring function to adjust the relative weight of the IoU and Category evaluation metrics. Click Save when you are done.

---

​

Team

To manage project collaborators, select the Team tab in your project Settings.

Here you can invite collaborators to the Project, and configure their roles.

​

1. Add a collaborator

To invite collaborators to your project, click the + Invite collaborators button. This opens a new window where you can enter email addresses of the people you want to invite.

Once you have entered everybody you want to invite, press the Add button.

​

2. Add collaborators as a group

Collaborators can be added to a Project as a group - which can save time as well as ensure that no individual is forgotten.

In the ‘Groups’ section of the page, click on Manage to make the ‘Manage Groups’ pop-up appear.

Click the ‘Select group’ drop-down and pick a group you would like to add as collaborators. After selecting a group, click the ‘Select Role’ drop-down to assign a role to the group of collaborators. Click Add to add the group.

The group you just added appears under the ‘Added groups’ heading. Repeat the process if you want to add more groups with different roles to the Project.

To delete a group from the project, simply click the button next to the group name.

​

2. Change collaborator role

A project admin can modify the different roles of collaborators, using the drop-down on the right.

You can assign the following roles to collaborators:

• Annotator: annotators are responsible for labeling. This is the default role for all collaborators.
• Team manager: a team manager can assign tasks to other users, and add collaborators to the project.
• Admin: gives this collaborator full administrative control over this project. Caution: this is an irreversible action.

Please confirm or cancel your selection when making a collaborator a project Admin.

---

​

Danger zone

You can delete your project by going to the Danger zone tab at the bottom of the menu, and clicking the red Delete project button, shown below.

Deleting your project does not delete the datasets in the project, but will delete the project’s labels and ontology.