Skip to main content

Labels

The labels page is your gateway to annotating, reviewing, and auditing the labels made against all the datasets in your project.

The labels page presents several different tabs, each with a different purpose. Access to each pane will depend on the user's project role. We quickly summarize the purpose of each tab, and the roles which can access each below.

The labels dashboard features the following tabs:

  • Activity: Quickly view all the tasks that are either In review or Completed, as well as confirm the status of the labels within those tasks.
  • Queue: The Queue tab is where all labeling and review actions should be initiated. Regardless of a user's role within the project, all annotations and reviews should be initiated from within the Queue pane, as it pulls tasks from the Task Management System. This ensures collaborators' efforts will not overwrite each other.
  • Data: Use the Data tab to get an overview of every data asset in the project, regardless of status within the Task Management System.
  • Instances: The Instances tab lets you use the unique instance identifier to search across the project to find a specific instance, and then jump directly into the editor to confirm the status of the annotation visually.

Access to each tab is associated with the various project roles as follows:

RoleActivityQueueDataInstances
Annotatorโœ…โœ…โŒโŒ
Reviewerโœ…โœ…โŒโŒ
Annotator & Reviewerโœ…โœ…โŒโŒ
Team Managerโœ…โœ…โœ…โœ…
Adminโœ…โœ…โœ…โœ…

Read on for the details about how to use each pane.

Activity#

The actvity screen allows you to quickly monitor annotation and review activity in your project by showing tasks and providing a summary interface to see the status of reviewed labels inside each task. See below for a detailed explanation of the various columns and interfaces. Tasks are displayed in most recently edited order from top to bottom. The most recently edited is at the top.

  • A. File, Search, & Reopen: The name of the sepcific data unit or data asset. This is the same as the name in the dataset to which this data asset is a part of. Use the search box to filter the list by file name, and send tasks back to annotation using the 'Reopen' feature.
  • B. Dataset: The dataset the data asset belongs to.
  • C. Type: The type of the data, such as an image or video. For more on our supported data types, see our label editor documentation.
  • D. Status: The status of this task within the Task Management System.
  • E. Frames: The number of frames in the data asset. For a DICOM series, this will be the number of slices.
  • F. Reviews: How many annotation instances are in this data asset.
  • G. Submitted: Indicates when the last submit action, whether for an annotation or review, was made against any of the labels in this data asset.
  • H. Submitted by: Who last submitted the annotations.
  • I. Reviewed by: Who submitted the most recent review.
  • J. Actions: Click the 'View' link to open the label editor. Note: this feature is only available to Team Managers and Administrators as an extra method of reviewing submissions outside of the TMS. We advise extra caution if you decide to edit the labels from this interface. If significant work needs to be done, we strongly recommend to 'Reopen' the task to prevent possible errors from simultaneous edits.
  • K. Filter: Use the filter dropdown to only show tasks with the selected status.

File, Search, and Reopen#

The file column shows the name of the data asset. For files uploaded via the GUI, they keep the name they were uploaded with. For files added from your cloud storage, this will normally be the path under the bucket they are stored on.

Use the search interface to quickly filter and display only those tasks with file names matching your desired text. Even partial matches will be shown. For example: searching "fly" will return file names containing "flyover" and "flyaround."

The 'Reopen' button allows Administrators and Team Managers to send tasks which are currently Completed or In review back to annotation. Select your target tasks using the checkboxes in the File column to select individual assets, or select the checkbox in the column header to select all tasks, and press the 'Reopen' button to move all selected tasks back to the annotation stage. Tasks reopened in this way will have the status 'Returned,' in the 'Queue' tab. No labels are lost by reopening a task. The 'Reopen' action is only applied to tasks which are both visible (i.e. not filtered out by the file search) and selected.

Status#

This column shows the status of this task within the Task Management System. The Activity pane only shows assets which have had some action done on them, and therefore only reflects tasks with the following statuses:

  • In review: The annotation task has been submitted but outstanding review tasks remain. In review task status is shown in blue.
  • Completed: The annotation task has been submitted and all reviews have been completed. Completed task status is shown in green.

For a comprehensive summary of the possible task states, see the status section of the Data tab, below.

Reviews#

Show a count of how many instances have been reviewed in a given data asset. Click the number to open a panel which shows the last review action taken on each instance, and who originally created the annotation and when. Note that unless the review was done by an 'Expert Reviewer', all reviewed annotations must be either 'Approved' or 'Deleted' before a task can be 'Completed.' Read more about the Expert Review feauture here.

Queue#

The 'Queue' is where annotators and reviewers look to find their next task. The 'Start labeling' and 'Start reviewing' buttons visible throughout the project open the label editor with the next task in the queue according to the relevant task type. This tab can be used to assess the number of tasks assigned to you as an annotator or reviewer and therefore estimate your likely workload. Administrators and Team Managers can also use it to quickly verify the current assignments per team member, and change assignments as necessary.

  • A. File, Search, & Assign: The name of the specific data unit or data asset. This is the same as the name in the dataset to which this data asset is a part of. Use the search box to filter the list by file name, and send tasks back to annotation using the 'Reopen' feature.
  • B. Dataset: The dataset the data asset belongs to.
  • C. Type: The type of the data, such as an image or video. For more on our supported data types, see our label editor documentation.
  • D. Status and Task: The status and category of this task within the Task Management System.
  • E. Last Edited: When the task was lated edited
  • F. Reserved by: Who the task has been assigned to or reserved by
  • G. Actions: Depending on your collaborator role, you can initiate or reassign the task.
  • H. Filter: Use the filter drop-down to only show tasks of the selected status.

File, Search, and Assign#

The file column shows the name of the data asset. For files uploaded via the GUI, they keep the name they were uploaded with. For files added from your cloud storage, this will normally be the path under the bucket they are stored on.

Use the search interface to quickly filter and display only those tasks with file names matching your desired text. Even partial matches will be shown. For example: searching "fly" will return file names containing "flyover" and "flyaround."

The 'Assign' button allows Administrators and Team Managers to allocate unassigned tasks to specific collaborators for annotation or review. Select your target tasks using the checkboxes in the File column to select individual assets, or select the checkbox in the column header to select all tasks, and press the 'Assign' button open the task assignment popup. Confirm the selected tasks are as intended, then select the target collaborator from the dropdown and press assign. Tasks which have already been assigned to another collaborator, as indicated by the email in the 'Reserved by' column, can not be reassigned until they have first been released.

Status and Task#

The 'Queue' tab only shows tasks which have remaining annotation or review work to be done within the Task Management System. Therefore, the stage of the task within the TMS is understood by reading the Status and Task columns together.

The two types of tasks are 'Annotate' and 'Review' which can be in any of the following states:

  • Queued: The task is ready for annotation or review. For an annotation tasks to be 'Queued' it must not be assigned to a user, and must have no submitted labels. It may have beenpreviously assigned to a user, but subsequently released before any annotations were submitted.
  • Assigned: The annotation or review task is assigned to a specific user.
  • Returned: The annotation task was previously submitted, and either several of the annotations were rejected by the reviewer or it was 'reopened' after completion by a Team Manager or Administrator.

Actions#

There are two relevant actions that can be done on each task from the 'Queue' pane. Press 'Initiate' to open the label editor and proceed with annotation or review, depending on the task type.

Additionally, Administrators and Team Managers can click the three vertical dots to open the expanded menu, to access the 'Release task' function. Tasks must be explicitly released before they can be reassigned.

Data#

The Data tab gives a complete overview of all the data asset tasks in the project, regardless of their progress through the Task Management System. Therefore, this is first place Administrators and Team Managers should check if they want confirm the status of a given task.

  • A. File & Search: The name of the specific data unit or data asset. This is the same as the name in the dataset to which this data asset is a part of. Use the search box to filter the list by file name.
  • B. Dataset: The dataset the data asset belongs to.
  • C. Type: The type of the data, such as an image or video. For more on our supported data types, see our [label
  • D. Status: The status of this task within the Task Management System.
  • E. Frames: The total frames in this data asset. This will apply to videos, image sequences and DICOM. Images always only have 1 frame.
  • F. FPS: the frames per second of the data asset. This only applies for data of type video. Others will show a dash (-).
  • G. Created: When the task was created. Tasks are created when the dataset containing the data asset is attached to the project.
  • H. Last edited by: the lat collobator to edit the task in any capacity (such as annotate or review), and when.
  • I. Actions: The 'Data' tab allows users to view the task in the label editor, as well as get a code snippet for using the SDK with this task, and confirming the edit actions via the Activity Log.
  • J. Filter by: Use the filter dropdown to view only tasks with the selected Status.

File and Search#

The file column shows the name of the data asset. For files uploaded via the GUI, they keep the name they were uploaded with. For files added from your cloud storage, this will normally be the path under the bucket they are stored on.

Use the search interface to quickly filter and display only those tasks with file names matching your desired text. Even partial matches will be shown. For example: searching "fly" will return file names containing "flyover" and "flyaround."

Status#

The data tab provides the most comprehensive overview of all the tasks associated with each data asset in a given project. As such, this is the first place to check to see the status of various tasks.

  • Queued: The task is ready for annotation. For a task to be 'Queued' it must not be assigned to a user, and have no submitted labels. A queued task may have been previously assigned to a user, but subsequently released before any annotations were submitted. Queued tasks are shown in light orange.
  • Assigned: An annotation task has been assigned to a specific user. Assigned tasks are shown in aqua green.
  • In review: The annotation task has been submitted but outstanding review tasks remain. In review task status is shown in blue.
  • Returned: The task was previously submitted, and either several of the annotations were rejected by the reviewer or it was 'reopened' after completion by a team manager or administrator.
  • Completed: The annotation task has been submitted and all reviews have been completed. Completed task status is shown in green.

Actions#

Clicking View will drop you into the label editor to do a live audit of the annotations in this data asset. The 'Data' tab is only visible to Administrators and Team Managers and so grants great power to view any data asset, however appropriate care must be taken to ensure annotations are not simultaneously edited from the 'Queue' pane by an annotator or reviewer. Encord advises edit actions are NOT taken from the Data tab unless you have received confirmation no one else is concurrently editing the asset.

caution

In order to prevent any possible issues of annotator work being overwritten, it's critical that all annotations are done via the Task Management System's Queue tab, and only the person assigned to the task makes annotations at any given time.

Other possible actions include 'API Details' which show a popup with sample code you can use to get started with our SDK to access this particular data asset, often known as a label row in the SDK. Click 'Acitvity log' to see a popup with a graphical summary of add / edit / delete actions on this data asset indexed by annotator or ontology class. Click 'Display logs' in the lower right to show all actions in reverse chronological order.

Instances#

The 'Instances' tab allows Administrators and Team Managers to search within the data to directly find specific instances. Recall that an annotation instance correlates to a unique instantiation of a specific ontology class in a data asset. For example, if you have the 'Person' class in your ontology, the first instance of a 'Person' in a given data asset will be indicated in the interface as 'Person (0)', the second as 'Person (1)' and so on. Instances, therefore, can exist in multiple frames of a data asset, and indicate the same object. Use the 'Instances' tab to search for specific instances of objects or classifications using their Identifier.

Instance identifiers are unique at the project scope, and can be found in any of the following ways:

  • From inside the label editor, by clicking on a particular instance, and then selecting 'Copy identifier' from the instance action menu.
  • From inside exported labels, where they are known as the objectHash or classificationHash as appropriate.
  • When uploading labels using the SDK, you may specify your own objectHash or classificationHash.

Once you have an identifier of intereset, use the 'Search instance' interface to filter the instances by identifier to quickly find the instance you're interested in. This can be particuarly handy when you want to visually confirm an annotation you may not have seen before, but for which you have the identifier.

After locating your instance of interest, click 'View' from the Actions column to jump deeply into the dataset straight to where the instance is first annotated.