Label Editor - Review

To ensure seamless data loading in Encord, whitelist https://app.encord.com/ or https://app.us.encord.com/ in any 3rd party VPNs, firewalls, or URL isolators. This helps to prevent potential issues with opening your data in Encord.

This documentation provides a comprehensive guide to using the Label Editor for review, covering its components, features, and the display of various elements.

Review Mode

Introductory Videos

The following video goes through the basics of reviewing images in the Label Editor.

The following video goes through the basics of reviewing videos in the Label Editor.

As a reviewer, your label editor workspace will look as below. Learn how to use the features in the components illustrated below.

Use the show-hide labels buttons, workspace settings, and the drag and select in the workspace to speed up your approved label selection and review process.

Review Mode Components

A. Editor menu
B. Single label review toggle
C. Quick toolbar
D. Pending reviews pane
E. Completed reviews pane
F. Reject and Approve buttons
G. Editor timeline (for videos and image sequences only)
H. Editor header
I. Edit labels during review
J. Approve and Reject all in frame buttons

Single Label Review

The normal review mode displays all labels simultaneously, allowing you to quickly approve or reject many labels in bulk actions. Alternately, enter the Single label review mode by toggling the switch at the top left. The single label review mode automatically highlights the selected instance, and hides all other objects. This allows you to carefully review one instance at a time including their attributes, and quickly iterate through the different instances using the Up and Down keys on your keyboard or by clicking the appropriate buttons in the UI.

The Single label review feature can also be considered “turbo review mode,” analogous to the “turbo attribute mode” in annotation mode, because it allows the reviewer to quickly step through and review instance attributes. “Turbo attribute mode” in the editor settings can be disabled while in review mode.

Single Label Review Settings

The default zoom level for single label review can be adjusted to suit your needs.

Open the editor settings by clicking in the editor header.
In the General settings section adjust the Set the default zoom level for single label view value highlighted in the screenshot below. The settings will be applied for the next selected label.

Initially, the default is set to 60%
20% corresponds to no zoom. It’s the magnification at which the image or frame is displayed in the Label Editor.
To zoom in, increase the percentage value.
To zoom out, decrease the percentage value. The minimum value is 1%.

Report Missing Labels

This feature is only available for Manual QA projects.

If a reviewer determines that a label is missing entirely (as opposed to annotated incorrectly), they can use the report missing labels feature to indicate labels are missing in a given frame or image. Navigate to the frame with the missing labels and use the Report missing labels button to create a report at that frame. You can only create one missing label report per frame per review task — subsequent reports at the same frame in the same task will overwrite previous reports. Missing label reports will be sent back to the annotator via the same issues list as rejected labels.

Edit Labels During Review

It is sometimes faster, more convenient, and less error-prone to have reviewers make edits to annotations, instead of outright rejecting them. Rejection requires the return of the entire set of labels to the annotator.

Edit review allows reviewers to make changes to labels, remove labels, and add new labels. Approved or rejected labels cannot be edited. Only Pending labels can be edited by reviewers. New labels made by editors are automatically approved, and cannot be changed once they have been saved. Once all labels have been approved the task moves to the Complete stage.

We recommend enabling Strict review on Review blocks when configuring Workflows. Strict review allows reviewers to revert approved labels back to Pending, giving reviewers the option to make changes to new labels drawn while editing labels.

To edit labels in review mode:

Click the Edit labels button.

Add or remove labels, or edit existing labels as necessary using the same interface annotators use to create labels.
Click the Save and return to review button to save your edits and return to review mode.

The Save and return to review button replaces the Save and Cancel buttons used in earlier versions of this feature.

Edit review is primarily optimized for use in Workflow Projects. It is available for Manual QA Projects, but lacks some functionality.

There is no way to edit Classifications.
There is no way to edit approved or rejected labels, as Strict review is not available in Manual QA Projects
Labels created during Edit review are not added to the Review stage and can not be seen in the Label Editor.

Pending Reviews Pane

All labels for review for a particular data asset assigned to the reviewer are automatically loaded into then Pending reviews pane. The labels are grouped by Ontology class.

Completed Reviews Pane

Completed reviews are displayed in the Completed reviews pane. You can click on specific objects to highlight them. Labels can be selected and then approved or rejected for a given instance or in bulk using the Reject and Approve buttons or the matching hotkeys, b for reject and n for approve. Approved labels appear in a green background and rejected labels appear in red.

Approve and Reject All In Frame Buttons

The Approve all in frame and Reject all in frame buttons take the named action without having to select a particular instance or set of instances. This speeds up workflows, particularly in images, where you can judge whether all instances are correct or not by looking at an individual frame.

The Approve all in frame and Reject all in frame buttons apply to all instances present in a frame, rather than labels. This means that all instances with labels in the current frame will have all of their labels approved or rejected.

The Approve all in frame and Reject all in frame buttons can be hidden in the general settings.

Reject and Approve Buttons

To reject/approve multiple object instances across frames, select the instances, and then click the Reject or Approve button, as appropriate. Alternately, you may press b for reject and n for approve.

Change Review Judgement

Video Tutorial - Strict review & change review judgement

This feature is not available for Manual QA projects.

Reviewed instances can be reverted to pending. Reverting a review allows reviewers to make adjustments to one or more instances after making their initial review.

Revert labels to pending :

Revert an approved task to pending by clicking the green tick icon next to the instance.
Revert a rejected task to pending by clicking the red cross icon next to the instance.

Once the reviewer completes their review, they need to click the Approve task or Reject task to finalize the review.

Controls Common To Both Modes

The current workflow status is displayed in the top left corner of the label editor.

Editor Layout

You can customize the default Label Editor layout by uploading a JSON file that defines your preferred arrangement. These JSON files can be added in the Project settings and are applied to all tasks in a Project.

Custom editor layouts depend on a file’s client_metadata or their DICOM tags. This means that custom editor layouts only work for files that contain client_metadata or DICOM tags. Watch the video tutorials below to learn how it fits together.

Introduction to custom editor layouts for DICOM - Video Tutorial

Introduction to custom editor layouts (Client metadata) - Video Tutorial

To upload an editor layout:

Navigate to the Project settings.
Click on Layouts.
Click Upload JSON.
Select the JSON file containing the layout you want the Label Editor to have.

JSON file requirements:

The JSON file must follow the JSON schema defined here. The following fields are required:

topLevelGridFilter

In Workflow Projects, tasks with matching values in the field specified by the topLevelGridFilter progress through the Workflow as a unit. When encord-LayoutGroup is set as the topLevelGridFilter, for example, tasks sharing the same encord-LayoutGroup value are opened, submitted, reviewed, and reopened together, enabling seamless coordination across the Workflow. This behavior does not apply in Manual QA Projects.

The operator value must be =. This means the value must be equal to.

"topLevelGridFilter": {
  "client_metadata": {
    "key": "encord-LayoutGroup",
    "operator": "="
  }
}

Another example is displaying images from the same patient together in the Label Editor using the topLevelGridFilter to group tasks based on a patient-id.

"topLevelGridFilter": {
  "client_metadata": {
    "key": "patient-id",
    "operator": "="
  }
}

grid

The grid key defines the basic layout of the Label Editor. The grid is a list of lists, where each value represents a window.

In the following example, the values 0, 1, 2, and 3 represent individual windows. The layout is defined as two separate lists [0, 1] and [2, 3], resulting in a 2x2 grid configuration:

  "grid": [[0, 1], [2, 3]]

This configuration translates to the following visual layout:

gridContent

The gridContent key specifies what content to display in each window defined in the grid. It includes the following elements:

Grid Values: Assigns content to each window based on its grid value.
Content Query: Defines the criteria for displaying content, which can be either dicom_tags_query or client_metadata_query.
Conditions: Specifies the conditions for displaying the content, which can be either and_conditions or or_conditions.
Operator value can only be =. This means the value must be equal to.

In the following example, window 0 displays content based on a dicom_tags_query with specific conditions.

The dicom_tags_query specifies that the content is determined by DICOM tags.
The and_conditions key specifies that all conditions listed must be met for the content to be displayed.
There are two conditions:
1. The DICOM tag 00185101 must equal CC.
2. The DICOM tag 00200062 must equal R.

This setup ensures that the content for window 0 is only displayed if both conditions are satisfied.

  "gridContent": {
    "0": {
      "dicom_tags_query": {
        "and_conditions": [
          {
            "tag": "00185101",
            "operator": "=",
            "value": "CC"
          },
          {
            "tag": "00200062",
            "operator": "=",
            "value": "R"
          }
        ]
      }
    }
    }

In the following example, window 1 displays content based on a client_metadata_query with specific conditions.

The client_metadata_query specifies that the content is determined by client metadata.
The or_conditions key specifies that any of the conditions listed must be met for the content to be displayed.
There are two conditions:
1. The client metadata group-id must equal A.
2. The client metadata group-id must equal B.

This setup ensures that the content for window 1 is only displayed if the group-id is either A or B.

  "gridContent": {
    "1": {
      "client_metadata_query": {
        "or_conditions": [
          {
            "key": "group-id",
            "operator": "=",
            "value": "A"
          },
          {
            "key": "group-id",
            "operator": "=",
            "value": "B"
          }
        ]
      }
    }
  }

Complete example

Add client metadata to files

Editor layouts are based on DICOM tags for DICOM files or on client_metadata for all other use cases. We provide templates for common mammography layouts based on DICOM tags here.

This example uses client_metadata. The following SDK script can be used to add client metadata to specific data units in a Dataset.

from encord import EncordUserClient
from encord.client import DatasetAccessSettings

# Instantiate Encord client by substituting the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
                ssh_private_key_path="<private_key_path>",
            )


# Specify a Dataset to read or write metadata to
dataset = user_client.get_dataset("<dataset_hash>")

# Fetch the Dataset's metadata
dataset.set_access_settings(DatasetAccessSettings(fetch_client_metadata=True))

# Add your file names and client metadata here
layouts = {
  'Image1.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': '1'},
  'Image2.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '1'},
  'Image3.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': "2"},
  'Image4.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '2'},
  'Image5.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': '1'},
  'Image6.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '1'},
  'Image7.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': "2"},
  'Image8.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '2'},
  'Image9.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': "1"},
  'Image10.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '1'},
  'Image11.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': "2"},
  'Image12.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '2'},
}

for dr in dataset.data_rows:
  dr.client_metadata = layouts[dr['data_title']]
  print (dr.title)
  print (dr.client_metadata)
  dr.save()

Create a JSON file

Create a JSON for specifying the editor layout that suits your needs.

In the JSON file below:

The grid arrangement is configured to display two files side by side in the Label Editor.
The gridContent section specifies that tasks in either position (0 or 1) can have client metadata values for encord-EditorGridPosition set to either A or B.

The topLevelGridFilter is defined as encord-LayoutGroup, meaning tasks with matching encord-LayoutGroup metadata values are displayed together in the Label Editor, ensuring they appear side by side when they share the same encord-LayoutGroup values.

{
  "grid": [
    [0],
    [1]
  ],
  "gridContent": {
    "0": {
      "client_metadata_query": {
        "or_conditions": [
          {
            "key": "encord-EditorGridPosition",
            "operator": "=",
            "value": "A"
          },
          {
            "key": "encord-EditorGridPosition",
            "operator": "=",
            "value": "B"
          }
        ]
      }
    },
    "1": {
      "client_metadata_query": {
        "or_conditions": [
          {
            "key": "encord-EditorGridPosition",
            "operator": "=",
            "value": "A"
          },
          {
            "key": "encord-EditorGridPosition",
            "operator": "=",
            "value": "B"
          }
        ]
      }
    }
  },
  "topLevelGridFilter": {
    "client_metadata": {
      "key": "encord-LayoutGroup",
      "operator": "="
    }
  },
  "$schema": "https://raw.githubusercontent.com/encord-team/Annotate/main/editor-layouts/editor_layout_schema.json"
}

Create a JSON file

Upload the JSON file to your Project in Encord.

Navigate to the Project settings.
Click on Layouts.
Click Upload JSON.
Select the JSON file containing the layout you want the Label Editor to have.

Create a JSON file

Open any task in the task queue of your Project.

The JSON file in this example results in the following Label Editor layout.

We provide end-to-end walkthroughes for how to set up custom editor layouts using:

clientMetadata here.
DICOM Tags here.

Open the editor menu by clicking the menu icon in the top left corner of the Label Editor.

The editor menu contains the following controls:

Return to Project - Returns the user to the Project navigation.
Save changes - Saves the current state of all labels.
View - Allows users to change what information is displayed next to all labels.
Bulk label operations (annotation mode only) - Allows the user to perform actions on many labels at once.
Automated labeling (annotation mode only) - Lets the user choose between several automated labeling techniques.
Dark mode - Toggle dark mode on and off.
Keyboard shortcuts - Displays a full list of keyboard shortcuts.
Editor settings - Opens the Editor settings.

Editor Header

The Editor header at the top-right of the navigation bar contains buttons and menu items to perform labeling activities as well as manage general app settings and notifications.

Icon	Name	Description
	Help	Links to the: quick start guide, documentation, learning hub, and Encord support team.
	Notifications	Shows the current status and history of actions taken in the application. This includes uploading datasets, training models, and exporting labels.
	Share	Copies the URL of the asset currently being labeled or reviewed.
	Save labels	Manually saves labels on the Encord server.
	Label Editor settings	Settings controlling your labeling and reviewing experience.
	Issues	Shows any issues and comments for the current task, such as any objects or classifications marked as Required, or comments added by other users.

A toolbar with handy tools to:

Rotate the label editor.
Adjust filters; contrast, brightness, gamma, pixelated scaling, and grayscaling.
Remove polygon vertices.
Add comments or issues to a frame or image.
Quickly center an image.
Zoom in and out.

Remove polygon vertices (the scissors tool) is currently not available in review or review-edit mode.

Click the zoom icon again to disable zoom.

Rotate the Label Editor

Click the Rotate toolbar icon on the quick toolbar to bring up a pop-up that enables you to rotate the label editor by using the slider, as shown below.

Click the Rotate 90 degrees icon to rotate the editor in increments of 90 degrees.

Pixelated Scaling

The Pixelated scaling feature enhances the visibility of individual pixels in an image or frame by disabling the browser’s automatic edge smoothing. This increased pixel clarity aids annotators in creating precise, pixel-perfect annotations, thereby improving their annotation accuracy, especially when zooming into an image.

Pixelated Scaling turned on	Pixelated Scaling turned off

Enable Pixelated scaling by selecting the Adjust filters icon ()in the Quick toolbar and enabling the Pixelated scaling checkbox, as shown below.

Pixelated scaling is a browser-specific feature. The table below shows which types of files Pixelated scaling works for Firefox and Chrome.

File type	Chrome	Firefox
Image	✅	✅
Image group	✅	✅
Video	❌	✅
Image sequence	❌	✅

Grayscale Filter

The Grayscale filter feature enhances the visibility across an image or frame by applying an RGB grayscale to the image/frame. The filter applies across the pixels of an image/frame using the value for Red, Green, or Blue as a base then converting to grayscale.

You can use the Grayscale filter on images, videos, and DICOM (DICOM and NIfTi) series.

Grayscale Image/Frame

Grayscale DICOM/NIfTi

The video frame slider lets you quickly scroll through frames in a video asset. Below the frame slider are a range of controls to let you play, skip and navigate through media. Notice that the navigation controls are not displayed in single images files, as well as any media with only 1 frame.

Icon descriptions:

The Time and Frame rate information for each frame is displayed in the far lower right corner.

Issues and Comments

Add Issue Tags to a Project to flexibly categorize and group your issues.
Use the Issues tab of your Project Analytics to learn about issues in your Project.

A complete walkthrough of all things Issues and Comments related can be found here.

Issues can be added to annotation and review tasks to communicate with other users. All issues contain comment threads relating to the entire task or specific sections of the task. A small notification badge on the Issues icon displays the number of unresolved issues in a task.

Types of Issues:

Label rejection: Created when a reviewer rejects labels. The reviewer is prompted to provide a reason for the rejection in the form of a comment. If all instances in a frame or image are rejected, a separate rejection issue is created for each label. Labels with rejection issues display a red Issues icon next to the instance name, which turns green once the issue is resolved.
File: Apply to the entire file and can be created at any stage within the Label Editor.
Pinned: Apply to a specific part of the file or frame and can be created at any stage within the Label Editor. Red Pin icons indicate unresolved issues and remain red until they are marked as resolved.
Frame: Apply to a specific frame and can be created at any stage within the Label Editor. Frame issues can only be created when multiple frames are present; such as in a video, image sequence, or DICOM volume.
Frame range: Apply to a range of frames and can be created at any stage within the Label Editor. Frame range issues can only be created when multiple frames are present; such as in a video, image sequence, or DICOM volume. Frame range issues can only be created in Encord Active.
Task requirement: Are created for each object or classification marked Required in the Ontology. The issue is resolved when the required object or classification is applied. Task requirement issues are the only issues visible in a Consensus annotate stage.

Selecting an issue card highlights the associated label. If a frame number is present, clicking it navigates directly to the corresponding frame.
Issue threads can be filtered by Resolved and Unresolved issues.

To Create a new Issue:

Issues cannot be created in a Consensus annotate stage.

Open the Issues pane by clicking the Issues icon.
Click the New issue icon (or keyboard shortcut Ctrl+N).
Click the Apply to dropdown to select the type of issue you want to create.
Select any Issue Tags you want to apply to the issue.
Add a comment to the issue.
If you selected ‘Pin on canvass’ in Step 3, click the desired location on the canvas.
Click Create issue.

To Resolve an Issue:

Open the Issues pane by clicking the Issues icon.
Click the tick to mark a thread as resolved.

Resolved issues remain visible but can be filtered out. All users can resolve issues.

To Delete an Issue Thread:

Deleted threads are NOT counted in the Issues analytics.

Open the Issues pane by clicking the Issues icon.
Click the Ellipsis icon to show more options.
Click Delete thread.

Users can only delete issues that they created. Label rejections cannot be deleted.

To Reopen an Issue Thread:

Open the Issues pane by clicking the Issues icon.
Click Reopen on an issue that was previously resolved.

Use Cases:

Ask questions: Annotators and reviewers can ask questions about the task. Admins can use this feedback to improve guidance.
Explain task rejection: Reviewers can provide reasons for rejection, helping annotators avoid repeating mistakes.
Track unresolved issues: Keep a record of open discussions that need follow-up before finalizing annotations.

Editor Lock

The ‘Editor lock’ is a safety feature we implemented to prevent several people making changes to the same file - annotators labeling, or reviewers reviewing the same data unit.

A warning is triggered in two cases:

When you have the same task open in two different tabs or browser windows:

When a different user is editing the same task:

Get Started

General

Index

Annotate

Active

Other

Label Editor - Review

Review Mode

Introductory Videos

Review Mode Components

Single Label Review

Report Missing Labels

Edit Labels During Review

Pending Reviews Pane

Completed Reviews Pane

Approve and Reject All In Frame Buttons

Reject and Approve Buttons

Change Review Judgement

Controls Common To Both Modes

Editor Layout

Editor Menu

Editor Header

Rotate the Label Editor

Pixelated Scaling

Grayscale Filter

Media Navigation Controls

Issues and Comments

Editor Lock

Get Started

General

Index

Annotate

Active

Other

​Review Mode

​Introductory Videos

​Review Mode Components

​Single Label Review

​Report Missing Labels

​Edit Labels During Review

​Pending Reviews Pane

​Completed Reviews Pane

​Approve and Reject All In Frame Buttons

​Reject and Approve Buttons

​Change Review Judgement

​Controls Common To Both Modes

​Editor Layout

​Editor Menu

​Editor Header

​Quick Toolbar

​Rotate the Label Editor

​Pixelated Scaling

​Grayscale Filter

​Media Navigation Controls

​Issues and Comments

​Editor Lock

Review Mode

Introductory Videos

Review Mode Components

Single Label Review

Report Missing Labels

Edit Labels During Review

Pending Reviews Pane

Completed Reviews Pane

Approve and Reject All In Frame Buttons

Reject and Approve Buttons

Change Review Judgement

Controls Common To Both Modes

Editor Layout

Editor Menu

Editor Header

Quick Toolbar

Rotate the Label Editor

Pixelated Scaling

Grayscale Filter

Media Navigation Controls

Issues and Comments

Editor Lock