​
Workflows

Workflows are a powerful tool to design and build your Projects - letting you control how tasks move through different stages of the Project, and determining how different stages interact with one another. On this page you can find:

​
Creating workflows

Workflows are created when creating new annotation Projects, and when creating Workflow templates.

​
Workflow composer components

Workflow workspace
  • A. Manage Collaborators allows you to seamlessly add users to your project as collaborators, which can be added to different stages of the workflow.
  • B. The Workflow library contains all the components that make up a project. Drag the cards from the workflow library onto the workflow canvas (C) to build the stages a workflow should pass through. Some cards can be used more than once to create complex workflows. The configuration details for each component can be found in the Workflow library components section below.
  • C. The Workflow canvas is the canvas on which you design your workflow, by connecting cards from the workflow library. Set up the stages you’d like your workflow to pass through from left to right, and connect them to suit your needs. Click and drag the canvas to move it.
  • D. Zoom in and out of the Workflow canvas for convenience. Large workflow designs will require you to zoom out to see the whole picture. The zoom icon focuses in on your workflow.
  • E. Create the Project, or return to the previous step of the project creation process. Click Create Project to complete the creation process.
Currently, connections between nodes on the canvas can’t be edited within a project after its creation. However, workflow templates can be edited and configured even when attached to ongoing projects. Click here.

​
Workflow library components

This section contains details for all Workflow library components.
To add library components to the canvas, click the Add stage button and then drag-and-drop the components onto the canvas.

​
Start

Start represents the beginning an annotation task through your Workflow, and can therefore only be used once in a Project. It must have a connector linking it to the next stage. It displays the Datasets connected to the Project, and the selected Ontology.

​
Annotate

Annotate represents an annotation stage, during which a group of annotators adds labels to tasks assigned to them. Click an Annotate card on the canvas to edit. Click x next to the name to exit edit mode. Click the Edit icon to edit the name of the annotation stage.
Annotation instructions can be added in the Project’s Settings after the Project is created.
The stage will contain all available annotators selected by default. Specific annotators can be selected by clicking the Annotate component, and choosing the Selective tab seen above. Type the email address of a user you’d like to add and select them from the list. Only collaborators that have been added via Manage collaborators on the Project level will be available. Click Add to confirm the selection.
Only collaborators with the Annotator or the Annotator & Reviewer roles are available for selection. All Admins and Team Managers will be part of all annotation stages and therefore can’t be added, or removed.
A Webhook can optionally be added to an annotation stage to send out notifications whenever a task is submitted. Click the Edit icon next to Webhook and add the URL of the server you would like to receive notifications on.

​
Routers

A Router splits the path that annotate and review tasks take through the workflow. Two different types of router are available:

​
Percentage router

Pathways that data units take through the Workflow can be split according to a certain percentage of annotation tasks. For example, 50% of tasks get sent to review stage A, while the other 50% of tasks get sent to review stage B, splitting the workload between two different groups of reviewers.
Once the router selects a route for a given task, it will be routed the same way until the routing percentages are updated. This means that if a task passes through a percentage router and is routed to review stage A on its first pass, it will be routed to review stage A on each consecutive pass.
Routing percentages can be changed after Project creation. Changing routing percentages resets the percentage router, and may alter the pathway taken by a given task.

​
Collaborator router

Determine the paths that data units take through your workflow based on users. For example, if user A completes a task it gets sent to be reviewed by review stage A, while a task completed by user B gets reviewed in review stage B.
The Or else option accounts for all collaborators not explicitly added to the router. In the image above they will be directed to review stage C.
  1. Click Add pathway to select a subset of collaborators that will make up a pathway - only collaborators present in the previous stage can be selected. There is no limit on the amount of pathways you can create.
  2. Click Add to finish a pathway.
Collaborators can only be added to the Router after an input is connected.

​
Review

Review represents a review stage where tasks labeled in an annotation stage are reviewed.
To attach the Rejected node of the review stage to another review stage, contact support@encord.com
Click a Review card that has been added to the canvas to edit. Click x next to the name to exit edit mode. Click the Edit icon to edit the name of the review stage. This is particularly useful when your Project has multiple review stages.
The stage contains all available reviewers selected by default. Specific reviewers can be selected by clicking the Review component, and choosing the Selective tab seen above. Type the email address of a user you’d like to add and select them from the list. Only collaborators that have been added to the Workflow or the Project. Click Add to confirm the selection.
Only collaborators with the Reviewer and Annotator & Reviewer roles are available for selection. All Admins and Team Managers are part of all review stages and cannot be removed.

​
Strict review

Strict review is enabled by default, but can be disabled under Advanced settings.
Strict review ensures that crucial details are not overlooked during a review stage. Strict review aids in assuring higher accuracy when reviewing annotations. Strict review stages do not auto-advance to the next review task after bulk approving or rejecting labels. Reviewers can make adjustments to their review, and add comments when rejecting tasks with missing annotations.

​
User assignment restriction

The User assignment restriction feature prevents team members working on tasks in the current stage that they have done in previous stages. This can prevent individuals with the Annotator & Reviewer role from being assigned their own annotations to review. This feature does not apply to users with the Admin or Team Manager roles, since they are added to each stage by default and cannot be removed.
Enable the toggle to implement User assignment restriction.

​
Agent

The Agent component represents a Task Agent. It triggers a custom operation each time a task passes through the Agent stage. The exact operation being triggered is configured using the Encord SDK. To fully connect an Agent to other Workflow components you must add pathways that allow you to create connections from the Agent stage to other components. To add a pathway, click + next to Add pathway and give the pathway a meaningful name. Repeat this step for each additional pathway you want to add.
Learn more about Task Agents in our Task Agent section

​
Complete

Complete represents the stage of a Project at which a task is marked as complete, and can therefore only be used once per project. Having passed though your Workflow, it was deemed to be acceptable data for the training of machine learning algorithms. Complete tasks can still be accessed by admins. Optionally, add a Webhook to this stage to receive notifications whenever a task arrives at the Complete stage.Click the Edit icon next to ‘Webhook’ and add the URL of the server you would like to receive notifications on.
In Consensus Projects, tasks in the COMPLETE and ARCHIVE blocks CANNOT be reopened.

​
Managing collaborators

​
User permissions

PermissionAdminTeam ManagerReviewerAnnotatorAnnotator & Reviewer
Attach / Detach datasets✅❌❌❌❌
Attach / Switch ontology✅❌❌❌❌
Delete✅❌❌❌❌
Invite team members✅✅❌❌❌
Manage team permissions✅✅❌❌❌
Manage admins✅❌❌❌❌
Annotate & review tasks✅✅Review onlyAnnotate only✅
Confirm annotations✅✅❌❌❌
Control assignments and status✅✅❌❌❌

​
Project level collaborators

Click the Manage Collaborators button, as shown in the Managing collaborators section.
  • Add collaborators and select their roles in the Add tab.
  • Remove and manage collaborators in the Manage tab.

​
Workflow stage level collaborators

Collaborators can be added to a workflow stage by clicking + on a card that has been added to the workflow canvas.
Learn how to add collaborators to different stages of a Project by visiting the relevant sections in this document: If no collaborators are specified at a given stage, all users with permissions are added by default. For example, if no collaborators are specified at an Annotate stage then all collaborators with the following roles will be added by default:
  • Annotator
  • Annotator & Reviewer
  • Team Manager
  • Admin

​
Templates

Tired of creating the same workflow over and over again? Try creating templates of your most commonly used workflows - and use these to create annotation Projects!

​
Creating templates

There are two ways of creating a workflow template:
  • In the Workflows section.
  • While creating a new annotation project.

​
Creating templates in the Workflows section

Select Workflow templates from the Annotate section of the sidebar in the Encord platform, and click the + New workflow template button. Previously created templates are visible here, and can be edited when selected.
Create and configure your Workflow template, then click the Create workflow template button when you are done. To learn how to create and configure Workflows, see our documentation here. The Create workflow template button is grayed out until:
  • The nodes in your template have been connected.
  • The workflow contains a Complete stage.
  • The template has been given a title.

​
Creating templates during Project creation

New templates can be created during the creation of a new annotation Project. Once a Workflow has been created, click the Save as template button to save the Workflow as a template.

​
Edit templates

You can edit Workflow templates after they have been created.
  1. Navigate to Workflow templates under Annotation.
  2. Click the name of the template you want to edit.
Click the Edit button, as highlighted in the image below.
You can not edit the start nodes.
Once you have made all desired edits, click the Save template changes button. The Save template changes button will be grayed out until:
  • The nodes in your template have been connected.
  • The workflow contains a Complete stage.
  • The template has been given a title.

​
Creating annotation Projects from templates

Workflow templates can be used during Project creation. To use a template during Project creation, choose the template you would like to use and click the Select button during the workflow setup.

​
Customizing templates

You can customize/edit a Workflow template to suit a project’s specific needs. Click the Save as a new template button after customizing/editing a template to create a new template.

​
Template settings

The template settings allow you to:

​
Team

The Teams section allows you to control which users have access to the template. There are two types of user roles for templates:
  • Admins can view as well as edit the template.
  • Viewers can only view the template.
Once a new Admin has been added, they can not be removed from the template.

​
Adding users to a template

  1. Click the + Invite collaborators button.
  1. Select a user role for the collaborator by choosing an option from the list.
  2. Type the email address of the user you’d like to add and select the user from the list.
  3. Click the Add button to add the user with the specified role.

​
Deleting templates

To delete a template:
  1. Navigate to Workflows in the Annotate section of the sidebar.
  2. Select the template you would like to delete.
  3. Click the Settings tab and navigate to ‘Danger zone’.
  4. Click the Delete template button to delete the template.