To ensure seamless data loading in Encord, whitelist https://app.encord.com/ or https://app.us.encord.com/ in any 3rd party VPNs, firewalls, or URL isolators. This helps to prevent potential issues with opening your data in Encord.

This documentation provides a comprehensive guide to using the Label Editor for review, covering its components, features, and the display of various elements.

Introductory Videos

The following video goes through the basics of reviewing images in the Label Editor.

The following video goes through the basics of reviewing videos in the Label Editor.

As a reviewer, your label editor workspace will look as below. Learn how to use the features in the components illustrated below.


Review Mode Components


Single Label Review

The normal review mode displays all labels simultaneously, allowing you to quickly approve or reject many labels in bulk actions. Alternately, enter the Single label review mode by toggling the switch at the top left. The single label review mode automatically highlights the selected instance, and hides all other objects. This allows you to carefully review one instance at a time including their attributes, and quickly iterate through the different instances using the Up and Down keys on your keyboard or by clicking the appropriate buttons in the UI.

The Single label review feature can also be considered “turbo review mode,” analogous to the “turbo attribute mode” in annotation mode, because it allows the reviewer to quickly step through and review instance attributes. “Turbo attribute mode” in the editor settings can be disabled while in review mode.

Single Label Review Settings

The default zoom level for single label review can be adjusted to suit your needs.

  1. Open the editor settings by clicking in the editor header.

  2. In the General settings section adjust the Set the default zoom level for single label view value highlighted in the screenshot below. The settings will be applied for the next selected label.

  • Initially, the default is set to 60%
  • 20% corresponds to no zoom. It’s the magnification at which the image or frame is displayed in the Label Editor.
  • To zoom in, increase the percentage value.
  • To zoom out, decrease the percentage value. The minimum value is 1%.

Report Missing Labels

This feature is only available for Manual QA projects.

If a reviewer determines that a label is missing entirely (as opposed to annotated incorrectly), they can use the report missing labels feature to indicate labels are missing in a given frame or image. Navigate to the frame with the missing labels and use the Report missing labels button to create a report at that frame. You can only create one missing label report per frame per review task — subsequent reports at the same frame in the same task will overwrite previous reports. Missing label reports will be sent back to the annotator via the same issues list as rejected labels.


Edit Labels During Review

It is sometimes faster, more convenient, and less error-prone to have reviewers make edits to annotations, instead of outright rejecting them. Rejection requires the return of the entire set of labels to the annotator.

Edit review allows reviewers to make changes to labels, remove labels, and add new labels. Approved or rejected labels cannot be edited. Only Pending labels can be edited by reviewers. New labels made by editors are automatically approved, and cannot be changed once they have been saved. Once all labels have been approved the task moves to the Complete stage.

We recommend enabling Strict review on Review blocks when configuring Workflows. Strict review allows reviewers to revert approved labels back to Pending, giving reviewers the option to make changes to new labels drawn while editing labels.

To edit labels in review mode:

  1. Click the Edit labels button.
  1. Add or remove labels, or edit existing labels as necessary using the same interface annotators use to create labels.

  2. Click the Save and return to review button to save your edits and return to review mode.

The Save and return to review button replaces the Save and Cancel buttons used in earlier versions of this feature.

Edit review is primarily optimized for use in Workflow Projects. It is available for Manual QA Projects, but lacks some functionality.

  • There is no way to edit Classifications.
  • There is no way to edit approved or rejected labels, as Strict review is not available in Manual QA Projects
  • Labels created during Edit review are not added to the Review stage and can not be seen in the Label Editor.

Pending Reviews Pane

All labels for review for a particular data asset assigned to the reviewer are automatically loaded into then Pending reviews pane. The labels are grouped by Ontology class.


Completed Reviews Pane

Completed reviews are displayed in the Completed reviews pane. You can click on specific objects to highlight them. Labels can be selected and then approved or rejected for a given instance or in bulk using the Reject and Approve buttons or the matching hotkeys, b for reject and n for approve. Approved labels appear in a green background and rejected labels appear in red.


Approve and Reject All In Frame Buttons

The Approve all in frame and Reject all in frame buttons take the named action without having to select a particular instance or set of instances. This speeds up workflows, particularly in images, where you can judge whether all instances are correct or not by looking at an individual frame.

The Approve all in frame and Reject all in frame buttons apply to all instances present in a frame, rather than labels. This means that all instances with labels in the current frame will have all of their labels approved or rejected.
The Approve all in frame and Reject all in frame buttons can be hidden in the general settings.

Reject and Approve Buttons

To reject/approve multiple object instances across frames, select the instances, and then click the Reject or Approve button, as appropriate. Alternately, you may press b for reject and n for approve.


Change Review Judgement

This feature is not available for Manual QA projects.

Reviewed instances can be reverted to pending. Reverting a review allows reviewers to make adjustments to one or more instances after making their initial review.

  1. Revert labels to pending :
  • Revert an approved task to pending by clicking the green tick icon next to the instance.
  • Revert a rejected task to pending by clicking the red cross icon next to the instance.
  1. Once the reviewer completes their review, they need to click the Approve task or Reject task to finalize the review.

Controls Common To Both Modes

The current workflow status is displayed in the top left corner of the label editor.

Editor Layout

Contact support to gain access to custom editor layouts.

You can customize the default Label Editor layout by uploading a JSON file that defines your preferred arrangement. These JSON files can be added in the Project settings and are applied to all tasks in a Project.

Custom editor layouts depend on a file’s client_metadata or their DICOM tags. This means that custom editor layouts only work for files that contain client_metadata or DICOM tags. Watch the video tutorials below to learn how it fits together.

To upload an editor layout:

  1. Navigate to the Project settings.
  2. Click on Layouts.
  3. Click Upload JSON.
  4. Select the JSON file containing the layout you want the Label Editor to have.

JSON file requirements:

The JSON file must follow the JSON schema defined here. The following fields are required:

Complete example

1

Add client metadata to files

Editor layouts are based on DICOM tags for DICOM files or on client_metadata for all other use cases. We provide templates for common mammography layouts based on DICOM tags here.

This example uses client_metadata. The following SDK script can be used to add client metadata to specific data units in a Dataset.

from encord import EncordUserClient
from encord.client import DatasetAccessSettings

# Instantiate Encord client by substituting the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
                ssh_private_key_path="<private_key_path>",
            )


# Specify a Dataset to read or write metadata to
dataset = user_client.get_dataset("<dataset_hash>")

# Fetch the Dataset's metadata
dataset.set_access_settings(DatasetAccessSettings(fetch_client_metadata=True))

# Add your file names and client metadata here
layouts = {
  'Image1.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': '1'},
  'Image2.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '1'},
  'Image3.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': "2"},
  'Image4.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '2'},
  'Image5.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': '1'},
  'Image6.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '1'},
  'Image7.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': "2"},
  'Image8.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '2'},
  'Image9.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': "1"},
  'Image10.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '1'},
  'Image11.jpg': {"encord-EditorGridPosition": "A", 'encord-LayoutGroup': "2"},
  'Image12.jpg': {"encord-EditorGridPosition": "B", 'encord-LayoutGroup': '2'},
}

for dr in dataset.data_rows:
  dr.client_metadata = layouts[dr['data_title']]
  print (dr.title)
  print (dr.client_metadata)
  dr.save()
2

Create a JSON file

Create a JSON for specifying the editor layout that suits your needs.

In the JSON file below:

  • The grid arrangement is configured to display two files side by side in the Label Editor.

  • The gridContent section specifies that tasks in either position (0 or 1) can have client metadata values for encord-EditorGridPosition set to either A or B.

  • The topLevelGridFilter is defined as encord-LayoutGroup, meaning tasks with matching encord-LayoutGroup metadata values are displayed together in the Label Editor, ensuring they appear side by side when they share the same encord-LayoutGroup values.

    {
      "grid": [
        [0],
        [1]
      ],
      "gridContent": {
        "0": {
          "client_metadata_query": {
            "or_conditions": [
              {
                "key": "encord-EditorGridPosition",
                "operator": "=",
                "value": "A"
              },
              {
                "key": "encord-EditorGridPosition",
                "operator": "=",
                "value": "B"
              }
            ]
          }
        },
        "1": {
          "client_metadata_query": {
            "or_conditions": [
              {
                "key": "encord-EditorGridPosition",
                "operator": "=",
                "value": "A"
              },
              {
                "key": "encord-EditorGridPosition",
                "operator": "=",
                "value": "B"
              }
            ]
          }
        }
      },
      "topLevelGridFilter": {
        "client_metadata": {
          "key": "encord-LayoutGroup",
          "operator": "="
        }
      },
      "$schema": "https://raw.githubusercontent.com/encord-team/Annotate/main/editor-layouts/editor_layout_schema.json"
    }
    
3

Create a JSON file

Upload the JSON file to your Project in Encord.

  1. Navigate to the Project settings.
  2. Click on Layouts.
  3. Click Upload JSON.
  4. Select the JSON file containing the layout you want the Label Editor to have.
4

Create a JSON file

Open any task in the task queue of your Project.

The JSON file in this example results in the following Label Editor layout.

We provide end-to-end walkthroughes for how to set up custom editor layouts using:

Editor Menu

Open the editor menu by clicking the menu icon in the top left corner of the Label Editor.

The editor menu contains the following controls:

  • Return to Project - Returns the user to the Project navigation.
  • Save changes - Saves the current state of all labels.
  • View - Allows users to change what information is displayed next to all labels.
  • Bulk label operations (annotation mode only) - Allows the user to perform actions on many labels at once.
  • Automated labeling (annotation mode only) - Lets the user choose between several automated labeling techniques.
  • Dark mode - Toggle dark mode on and off.
  • Keyboard shortcuts - Displays a full list of keyboard shortcuts.
  • Editor settings - Opens the Editor settings.

Editor Header

The Editor header at the top-right of the navigation bar contains buttons and menu items to perform labeling activities as well as manage general app settings and notifications.

IconNameDescription
HelpLinks to the: quick start guide, documentation, learning hub, and Encord support team.
NotificationsShows the current status and history of actions taken in the application. This includes uploading datasets, training models, and exporting labels.
ShareCopies the URL of the asset currently being labeled or reviewed.
Save labelsManually saves labels on the Encord server.
Label editor settingsSettings controlling your labeling and reviewing experience.
CommentsAdd and view comments pertaining to a frame, or the entire data unit.
IssuesShows any issues with submitting the current task, such as any objects or classifications marked as Required.

Quick Toolbar

A toolbar with handy tools to:

Remove polygon vertices (the scissors tool) is currently not available in review or review-edit mode.
Click the zoom icon again to disable zoom.

Rotate the Label Editor

Click the Rotate toolbar icon on the quick toolbar to bring up a pop-up that enables you to rotate the label editor by using the slider, as shown below.

Click the Rotate 90 degrees icon to rotate the editor in increments of 90 degrees.

Pixelated Scaling

The Pixelated scaling feature enhances the visibility of individual pixels in an image or frame by disabling the browser’s automatic edge smoothing. This increased pixel clarity aids annotators in creating precise, pixel-perfect annotations, thereby improving their annotation accuracy, especially when zooming into an image.

Pixelated Scaling turned onPixelated Scaling turned off

Enable Pixelated scaling by selecting the Adjust filters icon ()in the Quick toolbar and enabling the Pixelated scaling checkbox, as shown below.

Pixelated scaling is a browser-specific feature. The table below shows which types of files Pixelated scaling works for Firefox and Chrome.

File typeChromeFirefox
Image
Image group
Video
Image sequence

Media Navigation Controls

The video frame slider lets you quickly scroll through frames in a video asset. Below the frame slider are a range of controls to let you play, skip and navigate through media. Notice that the navigation controls are not displayed in single images files, as well as any media with only 1 frame.

Icon descriptions:

The Time and Frame rate information for each frame is displayed in the far lower right corner.


Comments and Issues

Issues can be added to annotation and review tasks to communicate with other users. All issues contain comment threads relating to the entire task or specific sections of the task. A small notification badge on the Issues icon displays the number of unresolved issues in a task.

Selecting an issue card highlights the label the issue pertains to.

Issues can be added to:

  • The entire task.
  • The current frame (for videos, image groups, and image sequences).
  • A specific location on the frame or task.

Issue threads can be filtered by Resolved and Unresolved issues.

To add a new issue:

  1. Open the Issues pane by clicking the Issues icon.
  2. Click + New.
  3. Choose the part of the task where you want to create the issue.
  4. Add a comment on the issue.
  5. If you selected ‘Pin on canvass’ in step 3, click the desired location on the canvas.
  6. Click Create issue.

To resolve an issue:

  1. Open the Issues pane by clicking the Issues icon.
  2. Click the tick to mark a thread as resolved.

Resolved issues remain visible but can be filtered out. All users can resolve issues.

To delete an issue thread:

  1. Open the Issues pane by clicking the Issues icon.
  2. Click the Ellipsis icon to show more options.
  3. Click Delete thread.

Users can only delete issues that they created. Label rejections cannot be deleted.

Types of Issues:

There are two types of issues: general issues and rejection issues.

  • General issues serve as comments to communicate with other users and can be created at any stage in the Label Editor.
  • Rejection issues are added when a reviewer rejects a task. The reviewer is prompted to provide a reason for the rejection. When rejecting all instances in a frame or image, a rejection issue is created for each rejected label. If a comment is provided, it is applied to all rejected labels within the frame or image.

Use Cases:

  • Ask questions: Annotators and reviewers can ask questions about the task. Admins can use this feedback to improve guidance.
  • Explain task rejection: Reviewers can provide reasons for rejection, helping annotators avoid repeating mistakes.
  • Track unresolved issues: Keep a record of open discussions that need follow-up before finalizing annotations.

Editor Lock

The “Editor lock” is a safety feature we implemented to prevent several people making changes to the same file - annotators labeling, or reviewers reviewing the same data unit.

A warning is triggered in two cases:

  1. When you have the same task open in two different tabs or browser windows:
  1. When a different user is editing the same task: