Annotation and review tasks are completed in a purpose-built user interface in the label editor. The label editor interface has two modes which depend on if you're performing an annotate or a review task.
Annotation mode components
- A. Editor header
- B. Skip and Submit
- C. Quick toolbar
- D. Classes
- E. Instances and Frame labels
- F. Annotation canvas: displays the media asset and provides interface for drawing and editing labels.
- G. Automated labeling
- H. Media navigation controls
Annotation tasks are completed in the label editor. Hit the Submit button once labeling work is completed and you're confident that the quality and coverage of created labels meet the project requiremens. The submitted labels will be selected for review, depending on the project's Quality configuration.
Skipping this task will put it in the back of the task queue, and the next annotation task will automatically open in the label editor if available. When submitting, be sure to resolve all issues first, then press Submit.
In annotation mode, the Classes section of the label editor shows you the available ontology classes, both objects and classifications. You can create a new instance by either clicking the desired class, or by using the indicated instantiation hotkeys. Learn more about ontologies in general here or learn more about the details of ontology structure on Encord.
The canvas is your labeling area to annotate the 'instances' of the ontology classes. When you define object instances, sometimes known simply as objects, you are creating an instance of that class. For example, if Apple is the class, 'Apple (0)' is the object.
Clicking on a label instance on the canvas opens the instance details sidebar showing the class, the instance ID and any nested attributes, if available. You can also reach this screen by clicking the 'Edit classifications' button in the desired instance's row.
Click the magnifying glass icon to open the Search object interface. Choose the ontology class, and then the sequential ID of the desired instance, and it will appear in the search result area, directly under the class and ID selection fields.
The Instances tab shows all the instantiated objects and classifications in a given data asset.
Objects are grouped first by class, and sorted by creation time within each class, with earlier objects on top. They are named in an "ABC (0)" format, where ABC is the ontology class and 0 is the first instance annotation. Frame-level classifications are all stored under the 'Classifications' section, and sorted by creation time with the oldest first.
To work with the details of any given instance, expand its class in the Instances tab and the following icons in the desired instance row to perform any of the following actions.
Instantiate icon : Use the Instantiate icon to create new labels of the given object. You can also use the assigned instance draw hotkey to create new labels of a given instance.
Edit classifications icon : Edit classifications for an existing object. You can reach the same interface by clicking an annotation in the canvas as well.
Hide icon : Click the Hide icon at the class level to toggle display or all instances of that class.You can hide or show individual objects by clicking in the corresponding object row as well.
Click the three dots icon () to access additional actions.
Copy identifier: Each object instance is automatically assigned a unique alphanumeric ID, which can be used to identify it within the project. This ID is sometimes known alternately as a feature hash, instance hash, or instance identifier. For objects and classifications, it may also be known as the object hash or classification hash, respectively. Use this button to copy the instance identifer to your clipboard -- which may help in workflows using specific instances in source code or for other purposes.
Track from this frame: Initiates single object tracking, a form of automated labeling that will create labels for the specified object from the current frame, according to the Single object tracking settings section in the editor settings.
Go to object: Navigate to the specified object in the data asset. If it is present in more than one frame, go to the first frame.
Split object track: In some cases, particularly when many objects overlap, labels may be associated with the correct instance in several frames, but later become associated with a different instance. In those cases, use the Split object track feature, which will split the labels from one instance into two at the frame specified.
Change class: Use the dropdown in the modal to change the current class of an instance. Note that you can only switch within classes of the same annotation type, i.e. you cannot switch from a bounding box to a polygon.
Dynamic classifications: Use dynamic classifications to attach temporally sensitive attributes to objects in a video or image sequence. For example, annotate invisible and heavily occluded objects to help models perceive object occlusions in videos. Use this interface to launch the dynamic classifications editor which allows to create and edit ranges of dynamic classification on this object.
Delete object: Delete an object from a single frame or a range of frames. To delete labels between frame ranges, move the frame slider to the desired frames, or input the From and To (exclusive) values in the relevant form. Delete actions are recorded in the User activity log. You can view the deleted objects in the Activity reports.
The Frame labels tab will show all instances with a label in the current frame, and will be empty when there are no labels in a given frame.
If there are many frames or instances, using automated labeling could be a good choice. Click the Automated labeling button to open the Automated labeling drawer, where you can initiate inference with micro-models as well as tracking and interpolation for multiple objects at once. The video below briefly illustrates the flow for using our object tracking feature -- interpolation is used analogously.
Fixing incorrect labels are an important step in producing high-quality datasets. With bulk label operations, annotators can merge labels which should really be one instance, or efficiently delete multiple labels in bulk.
Merge one or more label objects into a single object. To do this, select all the invovled objects from the Select objects to be merged drop-down menu. Then, select the target object in the form field which says Select objects to be merged into. Press Merge to execute the merge operation.
Encord also lets you perform bulk delete operations as follows:
Remove labels of a specific class From the Select class menu, select a class to delete, and specify the frame range in the respective fields, and click the Delete button.
Delete all labels within a frame range Use Delete all labels from range as needed to delete all annotations across all classes within a frame range.
Delete labels below confidence Labels generated by automatic labeling agents, such as micro-models, have an associated confidence which indicates the degree of certainty the agent had when making the prediction. A confidence of 1 is essentially "ground truth", so numbers less than that are percentages of confidence. Explore your predicted labels to find confidence levels which are of acceptable / unacceptable quality, and delete all those below a given confidence threshold at once with this convenient feature.
As a reviewer, your label editor workspace will look as below. Learn how to use the features in the components illustrated below.
- A. Single label review toggle
- B. Editor header
- C. Report missing labels / Edit labels
- D. Pending reviews pane
- E. Quick toolbar
- F. Completed reviews pane
- G. Approve and Reject all in frame buttons
- H. Reject and Approve buttons
- I. Media navigation controls
The normal review mode displays all labels simultaneously, allowing you to quickly approve or reject many labels in bulk actions. Alternately, enter the Single label review mode by toggling the switch at the top left. The single label review mode automatically highlights the selected instance, and hides all other objects. This allows you to carefully review one instance at a time including their nested classifications, and quickly iterate through the different instances using the Up and Down keys on your keyboard or by clicking the appropriate buttons in the UI.
- This Single label review feature can also be considered "turbo review mode," analogous to the "turbo attribute mode" in annotation mode, because it allows the reviewer to quickly step through and review instance attributes.
- "Turbo attribute mode" in the editor settings is disabled while in review mode.
If a reviewer determines that a label is missing entirely (as opposed to annotated incorrectly), they can use the report missing labels feature to indicate labels are missing in a given frame or image. Navigate to the frame with the missing labels and use the Report missing labels button to create a report at that frame. You can only create one missing label report per frame per review task -- subsequent reports at the same frame in the same task will overwrite previous reports. Missing label reports will be sent back to the annotator via the same issues list as rejected labels.
It's sometimes faster, more convenient, and less error-prone to have reviewers make some edits to annotations, rather than outright rejecting them, which requires the return of the entire set of labels to the annotator. Press the Edit labels button to enter review-edit mode and make any necessary changes, and press it again to return to review mode. Currently, only a subset of label operations are supported in review-edit mode:
- Objects: moving the object or individual vertices
- Classifications: changing the classification value
- Objects and classifications: changing nested classifications
We are continuously expanding the capabilities of review-edit mode while maintaining a simple UX. Contact email@example.com if there are particular capabilities which would assist your workflow.
All labels for review for a particular data asset assigned to the reviewer are automatically loaded into the Pending reviews pane. The labels are grouped by ontology class.
Completed reviews are displayed in the Completed reviews pane. You can click on specific objects to highlight them. Labels can be selected and then approved or rejected for a given instance or in bulk using the Reject and Approve buttons or the matching hotkeys, b for reject and n for approve. Approved labels appear in a green background and rejected labels appear in red.
The Approve all in frame and Reject all in frame buttons take the named action without having to select a particular instance or set of instances. Please note that any instance with labels in current frame will have all of its labels fall under the target of the approve or reject action.
This often speeds up workflows, particularly in images, where you can judge whether all instances are correct or not by looking at an individual frame.
To reject/approve multiple object instances across frames, select the instances, and then click the Reject or Approve button, as appropriate. Alternately, you may press b for reject and n for approve.
The Editor header at the top-right of the navigation bar contains buttons and menu items to perform labeling activities as well as manage general app settings and notifications.
Notification bell icon
shows the history and status of recent actions you've taken in the application, such as uploading datasets, training models, or exporting labels.
Copy URL to clipboard
Copy the URL of the asset you are currently reviewing/annotating.
Bulk label operations (only available in Annotation mode)
Open the 'Bulk label operations' drawer, which allows one to merge label instances and and perform various bulk delete operations.
Manually initiate saving labels to the server.
Adjust brightness and contrast
Adjust the brightness and contrast of the dataset you have opened in the canvas.
View a list of keyboard shortcuts you can use in the Encord app.
Label editor settings
Settings to control your annotation and review experience and workflows.
Toggle dark mode on and off.
A floating toolbar with handy tools to quickly center an image, zoom in and out, and trim polygon vertices. Click and drag while holding the left most section of the toolbar to drag it around the canvas.
Trimming polygon vertices (the scissors tool) is currently not available in review or review-edit mode.
The video frame slider lets you quickly scroll through frames in a video asset. Below the frame slider are a range of controls to let you play, skip and navigate through media. Notice that the navigation controls are not displayed in single images files, as well as any media with only 1 frame.
Go to first / last frame
Navigate directly to the beginning and end of videos, image sequences, and DICOM files.
Jump forward / backward to annotation
Jump forward or backward from the current frame to a frame that contains annotations. Use the settings menu to add them into the player controls panel.
Frame skip backward / forward
Skip back or forth by the current frame skip interval. You can also add a custom value directly from the frame skip interval selector, or in the settings.
Play / Pause
Play or pause the media. Alternately, use the spacebar while no other page elements are selected.
Adjust the playback speed from 1/8x to 8x.
Navigate to frame
To go to a specific location in the media, click on this icon and the target frame number or timestamp in the appropriate field, then press 'Go'.
The Time and Frame rate information for each frame is displayed in the far lower right corner.
The Encord label editor is a powerful and configurable tool to help you annotate data and review annotations as quickly and efficiently as possible. The settings drawer is where you can confirm details about your data assets, change settings and access tools to customize your labeling experience and workflow.
The settings drawer is divided into four sections: Data details, General settings, Drawing settings, and Single object tracking settings, continue reading for details on how to best utilize each section.
Remember, the label editor settings are set by you and not the project. Therefore, each user can:
- control their own settings
- have settings shared across all projects
- login from any device
- persist settings across logout / login sessions
The Data details section displays information about the asset currently open for annotation, or review. This information is drawn from the data itself, and isn't available for direct manipulation. We use this section to provide you with valuable information about the data you're working on, including if we may have detected any possible issues with the data encoding.
- A. Data title: Title or name of the asset.
- B. Data type: Data format of the asset such as image, video or DICOM.
- C. Frames per second: frequently abbreviated FPS, this is the number of frames, or images, per second in a video. Higher FPS means smoother playerback. This value is only valid for videos.
- D. Browser compatibility: Some FPS and video encodings may have issues synchronizing drawn labels with frames in modern browsers. Encord is developing compatibility tests to help detect when issues may arise. Read below for more.
Modern browsers don't allow deterministic control over video rendering -- which can lead to frame synchronization issues with some FPS values or video encodings. In order to help you ensure your labels are associated with the correct frames, Encord has developed browser compatibility tests to detect problematic videos. Click Run test to start the compatibility test process.
You can read more about supported videos here and learn more about frame synchronization issues here. We've identified common issues that can cause problems with frame synchronization, we can't guarantee all issues will be idenfitied. Please reach out to firstname.lastname@example.org if you have any questions or concerns about your data.
- A. Auto-save interval
- B. Display timestamps
- C. Turbo mode
- D. Frame skip interval
- E. Enable skip to next annotation buttons
To prevent accidentally losing work, the label editor will automatically persist labels to the server at regular intervals. Use this setting to control how often the labels are automatically saved. The minimum time interval for autosaving is 15 seconds and maximum is 600 seconds. By default, the annotations are saved every 120 seconds.
Toggle on if you want to display timestamps instead of frame numbers for object and classification instances.
Turbo mode switches from the standard annotation workflow in which labelers can both create instances and modify them in detail to a separate "turbo attribute mode", which allows labelers to quickly iterate through created instances to assign or confirm nested classifications.
Toggle turbo mode on, and then select an instance to work with from the drop down menu in the top left. This will hide all other instances except the single instance being viewed and corresponding nested classifications, if applicable.
Turbo mode has an additional autozoom feature, which will automatically zoom the camera on the focused instance as you scroll through the list of instances. You can use the normal zoom control tools to to zoom out, center () or even zoom in after autozoom has centered the selected instance. This can help contextualize the object in it's surroundings while ensuring you know exactly which instance is under consideration. It will auto-zoom again when you navigate to the next/previous instance.
The frame skip interval controls how many frames are skipped at once when navigating using the frame-skip forward and backward buttons. We define a set of default frame skip intervals to start, but you can also define customized frame skip intervals here, or from the drop-down in the video playback controls section. Select the desired interval from the video playback controls section to navigate appropriately throughout your data. See all player controls
For long videos with sparsely labeled frames, sometimes it helps to quickly jump between frames with labels. Toggle on the skip to next annotation buttons to add an extra set of playback controls which enable navigation straight to the next frame, either forward or backward, with labels. This will apply for objects and classifications. The screenshot below shows the video playback controls with the toggle off and on, respectively.
- A. Polygon vertex size
- B. Object opacity
- C. Selected object opacity
- D. Object names & font size
- E. Object classifications
- F. Skeleton keypoint sensitivity
- G. Drawing line aides
- H. Freehand drawing & Polygon coarseness
- I. Permanent drawing (new instances)
- J. Permanent drawing (existing instances
- K. Skeleton points
Adjust polygon vertex size. Larger vertices are easier to spot but may disrupt visibility of the surrounding areas. The possible range is from 1 pixel to 10 pixels, but they cannot be disabled entirely.
Adjust the opacity level of object labels. Raising the opacity will make it easier to spot the labels themselves, but will make it harder to see what is being labeled. Or, in the case of overlaps, may make it hard to see overlapping labels as well. Transparent labels may be hard to spot in a quick scan. Raise and lower the opacity levels to suit your workflow at different times of the labeling process.
Note that the setting can range from 1% to 100%. 1% is essentially transparent, and 100% is completely opaque.
Adjust the opacity for the selected instance, calculated as a multiplier of the value defined in Object opacity. The opacity multiplier can be set from 1x (same as all other instances) to 10x.
Toggle the display of the ontology class of an object and the object identifier, as well as control the font size when displayed. The text will appear directly above their respective labels. Possible font sizes range from 4pt to 18pt. Choose a size which is legible, but doesn't hinder your annotation work.
If there are nested classifications on object instances (otherwise known as object attributes), enabling this option will display them over their respective labels. The classifications display can be toggled independently of the names display. When they're both enabled, classifications are displayed between the name and object itself.
Adjust this setting to control the necessary precision when adjusting skeleton keypoint positions. Low values require high precision -- you must click exactly over the keypoint in question in order to adjust it. High values give the opposite behavior -- you can select and move skeleton keypoints even while the cursor is a good distance from the point in question.
Possible values range from a minimum of 1.00 and maximum of 10.00.
Show crosshairs on the canvas to indicate where the label will be drawn. Useful to know both when you've entered draw mode, and to see from exactly where the label will be created.
Use freehand drawing to draw a polygon or polyline as you move the mouse, without explicitly having to click to place each vertex. The coarseness controls the spacing between two vertices. A low Polygon coarseness allows for high resolution polygons, but creates high vertex counts.
To avoid possible performance issues with large or complicated polygons, set the coarseness only as fine as necessary to accurately define the desired segmentation.
Continue drawing new instances without having to press the drawing hotkey for a given ontology category. After drawing an instance of a given category, the editor will stay on the same frame and immediately prompt you to create another instance of the same category. It’s normally used to create many instances of a certain class in a given frame.
Useful in semi-automated workflows where the annotator labels many instances in certain keyframes and then uses tracking or interpolation features to fill in the remainder. The video below shows the difference between creating many objects with Permanent drawing (new instances) toggled off and then on.
Continue drawing labels of the same instance without pressing the drawing hotkey every time for the given object instance. After creating a label of an already existing instance, the label editor will automatically advance forward the currently specified frame skip interval and prompt for the creation of a new label of that instance on the new frame. Accelerate workflows which create many labels for specific instances throughout a data asset and also makes it easy to apply dynamic classifications through the Preserve chosen state functionality. The below video showcases the difference in creating multiple labels of the same instance with the Permanent drawing (existing instances) on and off.
Analogous to how we toggle the display of object names and attributes, use this to toggle the display of the individual skeleton keypoint names, as defined in the skeleton template editor. Since skeleton templates may have many points in complex shapes, displaying the names can help ensure each point is placed appropriately and that the entire annotation is oriented correctly.
Single object tracking always starts from the frame it's initiated on, and will predict labels on the number of frames specified in this settings. For example, if you start on frame 50, and set Number of frames to predict to 50, the tracker will attempt to predict labels on frames up to and including frame 100. If the tracker isn't able to find an appropriate section of a frame to label however, it may not label the entire designated range. Use smaller ranges with less radical changes across frames to enhance tracking performance.
Choose between the Standard and Advanced tracking models. The Standard model runs faster, but may not be as accurate. The Advanced model has enhanced accuracy, but may take more time because it requires more computations.
Use the following hotkeys to speed up common or useful operations.
Common editor operations
|Play / Pause||Space|
|Track selected object from frame||Shift+T|
|Delete selected object from range||Shift+D|
|Mark label row as not labeled||Shift+Y|
|Mark label row as in progress||Shift+U|
|Mark label row as labeled||Shift+I|
|Mark label row as review required||Shift+O|
|Mark label row as reviewed||Shift+P|
|Toggle turbo attribute mode||Ctrl+Shift+T / Cmd+Shift+T|
|Toggle turbo attribute mode auto-zoom||Ctrl+Shift+Y / Cmd+Shift+Y|
|Toggle permanent drawing||Ctrl+Shift+P / Cmd+Shift+P|
|Render next review task (Turbo review mode only)||Up|
|Render previous review task (Turbo review mode only)||Down|
|Select additional drawing||Ctrl+ Click object / Cmd+ Click object|
|Select all drawings||Ctrl+A / Cmd+ A|
|Copy to clipboard||Ctrl+C / Cmd+ C|
|Paste from clipboard||Ctrl+V / Cmd+ V|
|Change class type||C|
|Save labels||Ctrl+S / Cmd+ S|
|Activate/deactivate add vertex||A|
|Activate/deactivate delete vertex||S|
|Enable free hand drawing||D|
|Enable edit mode for polygons and polylines||F|
|Delete drawing||Del / Backspace|
|Undo||Ctrl+Z / Cmd+ Z|
|Redo||Ctrl+Shift+Z / Cmd+Shift+Z|
|Drag image||Ctrl+ Click and Drag / Cmd+ Click and Drag|
DICOM specific operations