Label Editor - Review

To ensure seamless data loading in Encord, whitelist https://app.encord.com/ or https://app.us.encord.com/ in any 3rd party VPNs, firewalls, or URL isolators. This helps to prevent potential issues with opening your data in Encord.

This documentation provides a comprehensive guide to using the Label Editor for review, covering its components, features, and the display of various elements.

Introductory Videos

The following video goes through the basics of reviewing images in the Label Editor.

The following video goes through the basics of reviewing videos in the Label Editor.

As a reviewer, your label editor workspace will look as below. Learn how to use the features in the components illustrated below.

Review Mode Components

A. Editor menu
B. Single label review toggle
C. Quick toolbar
D. Pending reviews pane
E. Completed reviews pane
F. Reject and Approve buttons
G. Editor timeline (for videos and image sequences only)
H. Editor header
I. Edit labels during review
J. Approve and Reject all in frame buttons

Single Label Review

The normal review mode displays all labels simultaneously, allowing you to quickly approve or reject many labels in bulk actions. Alternately, enter the Single label review mode by toggling the switch at the top left. The single label review mode automatically highlights the selected instance, and hides all other objects. This allows you to carefully review one instance at a time including their attributes, and quickly iterate through the different instances using the Up and Down keys on your keyboard or by clicking the appropriate buttons in the UI.

The Single label review feature can also be considered “turbo review mode,” analogous to the “turbo attribute mode” in annotation mode, because it allows the reviewer to quickly step through and review instance attributes. “Turbo attribute mode” in the editor settings can be disabled while in review mode.

Single Label Review Settings

The default zoom level for single label review can be adjusted to suit your needs.

Open the editor settings by clicking in the editor header.
In the General settings section adjust the Set the default zoom level for single label view value highlighted in the screenshot below. The settings will be applied for the next selected label.

Initially, the default is set to 60%
20% corresponds to no zoom. It’s the magnification at which the image or frame is displayed in the Label Editor.
To zoom in, increase the percentage value.
To zoom out, decrease the percentage value. The minimum value is 1%.

Report Missing Labels

This feature is only available for Manual QA projects.

If a reviewer determines that a label is missing entirely (as opposed to annotated incorrectly), they can use the report missing labels feature to indicate labels are missing in a given frame or image. Navigate to the frame with the missing labels and use the Report missing labels button to create a report at that frame. You can only create one missing label report per frame per review task — subsequent reports at the same frame in the same task will overwrite previous reports. Missing label reports will be sent back to the annotator via the same issues list as rejected labels.

Edit Labels During Review

It is sometimes faster, more convenient, and less error-prone to have reviewers make edits to annotations, instead of outright rejecting them. Rejection requires the return of the entire set of labels to the annotator.

Edit review allows reviewers to make changes to labels, remove labels, and add new labels. Approved or rejected labels cannot be edited. Only Pending labels can be edited by reviewers. New labels made by editors are automatically approved, and cannot be changed once they have been saved. Once all labels have been approved the task moves to the Complete stage.

We recommend enabling Strict review on Review blocks when configuring Workflows. Strict review allows reviewers to revert approved labels back to Pending, giving reviewers the option to make changes to new labels drawn while editing labels.

To edit labels in review mode:

Click the Edit labels button.

Add or remove labels, or edit existing labels as necessary using the same interface annotators use to create labels.
Click the Save and return to review button to save your edits and return to review mode.

The Save and return to review button replaces the Save and Cancel buttons used in earlier versions of this feature.

Edit review is primarily optimized for use in Workflow Projects. It is available for Manual QA Projects, but lacks some functionality.

There is no way to edit Classifications.
There is no way to edit approved or rejected labels, as Strict review is not available in Manual QA Projects
Labels created during Edit review are not added to the Review stage and can not be seen in the Label Editor.

Pending Reviews Pane

All labels for review for a particular data asset assigned to the reviewer are automatically loaded into then Pending reviews pane. The labels are grouped by Ontology class.

Completed Reviews Pane

Completed reviews are displayed in the Completed reviews pane. You can click on specific objects to highlight them. Labels can be selected and then approved or rejected for a given instance or in bulk using the Reject and Approve buttons or the matching hotkeys, b for reject and n for approve. Approved labels appear in a green background and rejected labels appear in red.

Approve and Reject All In Frame Buttons

The Approve all in frame and Reject all in frame buttons take the named action without having to select a particular instance or set of instances. This speeds up workflows, particularly in images, where you can judge whether all instances are correct or not by looking at an individual frame.

The Approve all in frame and Reject all in frame buttons apply to all instances present in a frame, rather than labels. This means that all instances with labels in the current frame will have all of their labels approved or rejected.

The Approve all in frame and Reject all in frame buttons can be hidden in the general settings.

Reject and Approve Buttons

To reject/approve multiple object instances across frames, select the instances, and then click the Reject or Approve button, as appropriate. Alternately, you may press b for reject and n for approve.

Change Review Judgement

This feature is not available for Manual QA projects.

Reviewed instances can be reverted to pending. Reverting a review allows reviewers to make adjustments to one or more instances after making their initial review.

Revert labels to pending :

Revert an approved task to pending by clicking the green tick icon next to the instance.
Revert a rejected task to pending by clicking the red cross icon next to the instance.

Once the reviewer completes their review, they need to click the Approve task or Reject task to finalize the review.

Controls Common To Both Modes

The current workflow status is displayed in the top left corner of the label editor.

Editor Layout

Contact support to gain access to custom editor layouts.

You can customize the default Label Editor layout by uploading a JSON file that defines your preferred arrangement. These JSON files can be added in the Project settings and are applied to all tasks in a Project.

Custom editor layouts depend on a file’s client_metadata or their DICOM tags. This means that custom editor layouts only work for files that contain client_metadata or DICOM tags. Watch the video tutorials below to learn how it fits together.

To upload an editor layout:

Navigate to the Project settings.
Click on Layouts.
Click Upload JSON.
Select the JSON file containing the layout you want the Label Editor to have.

JSON file requirements:

The JSON file must follow the JSON schema defined here. The following fields are required:

In Workflow Projects, tasks with matching values in the field specified by the topLevelGridFilter progress through the Workflow as a unit. When encord-LayoutGroup is set as the topLevelGridFilter, for example, tasks sharing the same encord-LayoutGroup value are opened, submitted, reviewed, and reopened together, enabling seamless coordination across the Workflow. This behavior does not apply in Manual QA Projects.

The operator value must be =. This means the value must be equal to.

"topLevelGridFilter": {
  "client_metadata": {
    "key": "encord-LayoutGroup",
    "operator": "="
  }
}

Another example is displaying images from the same patient together in the Label Editor using the topLevelGridFilter to group tasks based on a patient-id.

"topLevelGridFilter": {
  "client_metadata": {
    "key": "patient-id",
    "operator": "="
  }
}

The grid key defines the basic layout of the Label Editor. The grid is a list of lists, where each value represents a window.

In the following example, the values 0, 1, 2, and 3 represent individual windows. The layout is defined as two separate lists [0, 1] and [2, 3], resulting in a 2x2 grid configuration:

  "grid": [[0, 1], [2, 3]]

This configuration translates to the following visual layout:

The gridContent key specifies what content to display in each window defined in the grid. It includes the following elements:

Grid Values: Assigns content to each window based on its grid value.
Content Query: Defines the criteria for displaying content, which can be either dicom_tags_query or client_metadata_query.
Conditions: Specifies the conditions for displaying the content, which can be either and_conditions or or_conditions.
Operator value can only be =. This means the value must be equal to.

In the following example, window 0 displays content based on a dicom_tags_query with specific conditions.

The dicom_tags_query specifies that the content is determined by DICOM tags.
The and_conditions key specifies that all conditions listed must be met for the content to be displayed.
There are two conditions:
1. The DICOM tag 00185101 must equal CC.
2. The DICOM tag 00200062 must equal R.

This setup ensures that the content for window 0 is only displayed if both conditions are satisfied.

  "gridContent": {
    "0": {
      "dicom_tags_query": {
        "and_conditions": [
          {
            "tag": "00185101",
            "operator": "=",
            "value": "CC"
          },
          {
            "tag": "00200062",
            "operator": "=",
            "value": "R"
          }
        ]
      }
    }
    }

In the following example, window 1 displays content based on a client_metadata_query with specific conditions.

The client_metadata_query specifies that the content is determined by client metadata.
The or_conditions key specifies that any of the conditions listed must be met for the content to be displayed.
There are two conditions:
1. The client metadata group-id must equal A.
2. The client metadata group-id must equal B.

This setup ensures that the content for window 1 is only displayed if the group-id is either A or B.

  "gridContent": {
    "1": {
      "client_metadata_query": {
        "or_conditions": [
          {
            "key": "group-id",
            "operator": "=",
            "value": "A"
          },
          {
            "key": "group-id",
            "operator": "=",
            "value": "B"
          }
        ]
      }
    }
  }

Complete example

Add client metadata to files

Editor layouts are based on DICOM tags for DICOM files or on client_metadata for all other use cases. We provide templates for common mammography layouts based on DICOM tags here.

This example uses client_metadata. The following SDK script can be used to add client metadata to specific data units in a Dataset.

from encord import EncordUserClient
from encord.client import DatasetAccessSettings

# Instantiate Encord client by substituting the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
                ssh_private_key_path="<private_key_path>",
            )


# Specify a Dataset to read or write metadata to
dataset = user_client.get_dataset("<dataset_hash>")

# Fetch the Dataset's metadata
dataset.set_access_settings(DatasetAccessSettings(fetch_client_metadata=True))

# Add your file names and client metadata here
layouts = {
  'Image1.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': '1'},
  'Image2.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '1'},
  'Image3.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': "2"},
  'Image4.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '2'},
  'Image5.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': '1'},
  'Image6.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '1'},
  'Image7.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': "2"},
  'Image8.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '2'},
  'Image9.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': "1"},
  'Image10.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '1'},
  'Image11.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': "2"},
  'Image12.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '2'},
}

for dr in dataset.data_rows:
  dr.client_metadata = layouts[dr['data_title']]
  print (dr.title)
  print (dr.client_metadata)
  dr.save()

Create a JSON file

Create a JSON for specifying the editor layout that suits your needs.

In the JSON file below:

The grid arrangement is configured to display two files side by side in the Label Editor.
The gridContent section specifies that tasks in either position (0 or 1) can have client metadata values for encord-EditorGridPosition set to either A or B.

The topLevelGridFilter is defined as encord-LayoutGroup, meaning tasks with matching encord-LayoutGroup metadata values are displayed together in the Label Editor, ensuring they appear side by side when they share the same encord-LayoutGroup values.

{
  "grid": [
    [0],
    [1]
  ],
  "gridContent": {
    "0": {
      "client_metadata_query": {
        "or_conditions": [
          {
            "key": "encord-EditorGridPosition",
            "operator": "=",
            "value": "A"
          },
          {
            "key": "encord-EditorGridPosition",
            "operator": "=",
            "value": "B"
          }
        ]
      }
    },
    "1": {
      "client_metadata_query": {
        "or_conditions": [
          {
            "key": "encord-EditorGridPosition",
            "operator": "=",
            "value": "A"
          },
          {
            "key": "encord-EditorGridPosition",
            "operator": "=",
            "value": "B"
          }
        ]
      }
    }
  },
  "topLevelGridFilter": {
    "client_metadata": {
      "key": "encord-LayoutGroup",
      "operator": "="
    }
  },
  "$schema": "https://raw.githubusercontent.com/encord-team/Annotate/main/editor-layouts/editor_layout_schema.json"
}

Create a JSON file

Upload the JSON file to your Project in Encord.

Navigate to the Project settings.
Click on Layouts.
Click Upload JSON.
Select the JSON file containing the layout you want the Label Editor to have.

Create a JSON file

Open any task in the task queue of your Project.

The JSON file in this example results in the following Label Editor layout.

We provide an end-to-end walkthroughs for how to set up ustom editor layouts using :

clientMetadata here.
DICOM Tags here.

Open the editor menu by clicking the menu icon in the top left corner of the Label Editor.

The editor menu contains the following controls:

Return to Project - Returns the user to the Project navigation.
Save changes - Saves the current state of all labels.
View - Allows users to change what information is displayed next to all labels.
Bulk label operations (annotation mode only) - Allows the user to perform actions on many labels at once.
Automated labeling (annotation mode only) - Lets the user choose between several automated labeling techniques.
Dark mode - Toggle dark mode on and off.
Keyboard shortcuts - Displays a full list of keyboard shortcuts.
Editor settings - Opens the Editor settings.

Editor Header

The Editor header at the top-right of the navigation bar contains buttons and menu items to perform labeling activities as well as manage general app settings and notifications.

Icon	Name	Description
	Help	Links to the: quick start guide, documentation, learning hub, and Encord support team.
	Notifications	Shows the current status and history of actions taken in the application. This includes uploading datasets, training models, and exporting labels.
	Share	Copies the URL of the asset currently being labeled or reviewed.
	Save labels	Manually saves labels on the Encord server.
	Label editor settings	Settings controlling your labeling and reviewing experience.
	Comments	Add and view comments pertaining to a frame, or the entire data unit.
	Issues	Shows any issues with submitting the current task, such as any objects or classifications marked as Required.

A toolbar with handy tools to:

Rotate the label editor.
Adjust filters; contrast, brightness, gamma, and pixelated scaling.
Remove polygon vertices.
Add comments to a frame or image.
Quickly center an image.
Zoom in and out.

Remove polygon vertices (the scissors tool) is currently not available in review or review-edit mode.

Click the zoom icon again to disable zoom.

Rotate the Label Editor

Click the Rotate toolbar icon on the quick toolbar to bring up a pop-up that enables you to rotate the label editor by using the slider, as shown below.

Click the Rotate 90 degrees icon to rotate the editor in increments of 90 degrees.

Pixelated Scaling

The Pixelated scaling feature enhances the visibility of individual pixels in an image or frame by disabling the browser’s automatic edge smoothing. This increased pixel clarity aids annotators in creating precise, pixel-perfect annotations, thereby improving their annotation accuracy, especially when zooming into an image.

Pixelated Scaling turned on	Pixelated Scaling turned off

Enable Pixelated scaling by selecting the Adjust filters icon ()in the Quick toolbar and enabling the Pixelated scaling checkbox, as shown below.

Pixelated scaling is a browser-specific feature. The table below shows which types of files Pixelated scaling works for Firefox and Chrome.

File type	Chrome	Firefox
Image	✅	✅
Image group	✅	✅
Video	❌	✅
Image sequence	❌	✅

The video frame slider lets you quickly scroll through frames in a video asset. Below the frame slider are a range of controls to let you play, skip and navigate through media. Notice that the navigation controls are not displayed in single images files, as well as any media with only 1 frame.

Icon descriptions:

The Time and Frame rate information for each frame is displayed in the far lower right corner.

Comments

Comments can be added during the annotation or review stages and are visible to any user accessing the data unit. All comments are visible in the comments pane, where any user can add a reply.

You can add comments to:

Task comments

Task comments relate to the entire task / data unit. For example, a comment relating to the resolution of a video is a task comment.

To add a comment to the entire task:

Open the comments pane.
Click + New comment.
In the Apply to section, ensure that Entire file is selected.
Type your comment.
Press Enter on your keyboard to add the comment.

To reply to a task comment:

Open the comments pane.
Click Reply on the comment you want to reply to.
Type the reply you want to add.
Press the Enter key to add the reply.

Frame comments

Frame comments relate to the entire frame. For example, comments relating to all annotations on a particular frame are frame comments.

Open the comments pane.
Click + New comment.
In the Apply to section, ensure that Current frame is selected.
Type your comment.
Press Enter on your keyboard to add the comment.

To reply to a frame comment:

Open the comments pane.
Click Reply on the comment you want to reply to.
Type the reply you want to add.
Press the Enter key to add the reply.

In-frame comments

In-frame comments can be added anywhere on a frame. For example, a comment regarding a particular object label is an in-frame comment. Comments cannot be added directly on an object label.

To add a comment to a specific location:

Right-click the location you want to add the comment and select Add comment.
Type your comment.
Press the Enter key to add the comment. The comment icon now indicates in-frame comment.

To reply to an in-frame comment:

Click the comment icon to open the in-frame comment. =
Click Reply.
Type the reply you want to add.
Press the Enter key to add the reply.

Comments Pane

The comments pane allows you to see all comments on a data unit and to filter comments by type. Access the comments pane by clicking the icon, located in the top right corner of the Label Editor.

The symbols to the left of each comment imply whether the comment pertains to a particular frame, or the entire task.

Icon	Meaning
	This comment applies to the entire task
	This comment applies to a particular frame
	This comment applies to a particular location within a frame

The numbers on frame-level and location specific comments show which frame the comment applies to. Click on the number to open up the frame in the Label Editor.

To filter comments by type, click the settings icon in the comments pane and select the type of comment you want to display.

Issues Drawer

The Issues drawer will show any outstanding issues the user needs to resolve before moving on to the next task. It is accessed by clicking the icon on the editor header. The red number next to the icon indicates the number of issues that need to be resolved.

Issues include objects and classifications that have been marked as Required in the ontology.

Resolving Issues

To resolve all outstanding issues follow the steps below:

Click the Issues icon to open the issues drawer.
Click Create.
Create the Required object or classification that’s missing and click Confirm.
Re-open the Issues drawer and resolve the next issue.

See our documentation on objects and classifications for more information on these topics.

Editor Lock

The “Editor lock” is a safety feature we implemented to prevent several people making changes to the same file - annotators labeling, or reviewers reviewing the same data unit.

A warning is triggered in two cases:

When you have the same task open in two different tabs or browser windows:

When a different user is editing the same task:

Get started

General

Index

Annotate

Active

Other

Label Editor - Review

Introductory Videos

Review Mode Components

Single Label Review

Report Missing Labels

Edit Labels During Review

Pending Reviews Pane

Completed Reviews Pane

Approve and Reject All In Frame Buttons

Reject and Approve Buttons

Change Review Judgement

Controls Common To Both Modes

Editor Layout

Editor Menu

Editor Header

Rotate the Label Editor

Pixelated Scaling

Media Navigation Controls

Comments

Comments Pane

Issues Drawer

Resolving Issues

Editor Lock

Get started

General

Index

Annotate

Active

Other

​Introductory Videos

​Review Mode Components

​Single Label Review

​Report Missing Labels

​Edit Labels During Review

​Pending Reviews Pane

​Completed Reviews Pane

​Approve and Reject All In Frame Buttons

​Reject and Approve Buttons

​Change Review Judgement

​Controls Common To Both Modes

​Editor Layout

​Editor Menu

​Editor Header

​Quick Toolbar

​Rotate the Label Editor

​Pixelated Scaling

​Media Navigation Controls

​Comments

​Comments Pane

​Issues Drawer

​Resolving Issues

​Editor Lock

Introductory Videos

Review Mode Components

Single Label Review

Report Missing Labels

Edit Labels During Review

Pending Reviews Pane

Completed Reviews Pane

Approve and Reject All In Frame Buttons

Reject and Approve Buttons

Change Review Judgement

Controls Common To Both Modes

Editor Layout

Editor Menu

Editor Header

Quick Toolbar

Rotate the Label Editor

Pixelated Scaling

Media Navigation Controls

Comments

Comments Pane

Issues Drawer

Resolving Issues

Editor Lock