Workflows and templates

Workflows

Workflows are a powerful tool to design and build your projects - letting you control how tasks move through different stages of the project, and determining how different stages interact with one another.

On this page you can find:

A step-by-step tutorial on setting up and configuring a workflow.
An explanation of all workflow composer components.
An explanation of all workflow library components.
A guide to Managing Collaborators
A guide to workflow Templates

Creating workflows

This section will outline a step-by-step tutorial to creating workflows.

ℹ️
Note
Workflows are created when creating new annotation projects or workflow templates.

1. Add users to the project

Add users from within your organization to the project by clicking the Add collaborators button, highlighted in the image below.

Collaborators are added based on their role within the project - select the role you would like the collaborator(s) to have.
Start typing the email of a user you would like to add into the area highlighted on the image below, and select the user from the list that appears. Repeat this for every user that will have the same role.
When you are done selecting users for this role, click Add.

ℹ️
Note
Learn more about managing collaborators in the Managing collaborators section.

2. Configure your workflow

Click the + New workflow button to create a new workflow. Any workflow created can be saved as a template by clicking the Save as a new template button.

Alternatively select a template that was previously created

Configure your workflow by pulling components from the Workflow library onto the canvas - the video below shows how to do this. The canvas will be populated with a Start stage, connected to an Annotate stage by default.

Add the stages and other components you require for your workflow onto the canvas:
- All workflows must begin with the Start stage.
- All workflows must contain an Annotate stage.
- Add routers to your project to determine different pathways through your workflow a task can take. Click on the Router card to determine the type of router.
- Add as many Review stages as necessary.
- All workflows must end at a Complete stage.
Link all components on the canvas by clicking and dragging from one connection point to another, as seen below.

👍
Tip
The cursor changes from to when hovering over a connection point.

3. Configure the stages of your workflow

Once you’ve arranged the stages in the composer, it’s time to configure the details of each stage.

Click an Annotate card on the canvas to start editing the annotation stage.
- Give the stage a descriptive name.
- Add annotators. If you'd like to specify annotators for this stage, add them as collaborators. For full details on how collaborators can work on tasks at each stage, see our section on managing collaborators.
- Optionally, add a Webhook to receive notifications when labels are submitted at this stage.

ℹ️
Note
More details about configuring annotation stages can be found in the Annotate section below

Click a Review card on the canvas to start editing the review stage.
- Give the stage a descriptive name.
- Add reviewers. If you'd like to specify reviewers for this stage, add them as collaborators. For full details on how collaborators can work on tasks at each stage, see our section on managing collaborators.

ℹ️
Note
More details about configuring review stages can be found in the Review section below.

Click a Router card on your canvas. Please see the router section for more details on how different types of routers can be configured.
Optionally, add a Webhook to the Complete stage to receive a notification when a task has been completed.
Optionally, add User assignment restrictions if users in this node should be prevented from being assigned to tasks they completed in the nodes listed.

ℹ️
Note
More details on routers can be found in the router section below.

4. Create your project

Once all previous stages in this tutorial have been completed and your workflows has been validated, you are ready to create your project. Simply click the Create Project button highlighted in the image below.

Workflow composer components

A. Manage Collaborators allows you to seamlessly add users to your project as collaborators, which can be added to different stages of the workflow.
B. The Workflow library contains all the components that make up a project. Drag the cards from the workflow library onto the workflow canvas (C) to build the stages a workflow should pass through. Some cards can be used more than once to create complex workflows. The configuration details for each component can be found in the Workflow library components section below.
C. The Workflow canvas is the canvas on which you design your workflow, by connecting cards from the workflow library. Set up the stages you'd like your workflow to pass through from left to right, and connect them to suit your needs. Click and drag the canvas to move it.
D. Zoom in and out of the Workflow canvas for convenience. Large workflow designs will require you to zoom out to see the whole picture. The icon focuses in on your workflow.
E. Create the project, or return to the previous step of the project creation process. Click Create Project to complete the creation process.

🚧
Caution
Currently, connections between nodes on the canvas can't be edited within a project after its creation. However, workflow templates can be edited and configured even when attached to ongoing projects. Click here.

Workflow library components

This section contains an overview of, and details on all Workflow library components.

Start

Start represents the beginning an annotation task through your workflow, and can therefore only be used once in a project. It must have a connector linking it to the next stage. It displays the datasets connected to the project, as well as the selected ontology.

Annotate

Annotate represents an annotation stage, during which a group of annotators adds labels to tasks assigned to them.

Click an Annotate card on the canvas to edit. Click x next to the name to exit edit mode.

Click the icon to edit the name of the annotation stage.

👍
Tip
Annotation instructions can be added in the project's Settings after the project has been created.

The stage will contain all available annotators selected by default. Specific annotators can be selected by clicking the Annotate component, and choosing the Selective tab seen above. Type the email address of a user you'd like to add and select them from the list. Only collaborators that have been added via Manage collaborators on the project level will be available. Click Add to confirm the selection.

👍
Tip
Annotation instructions can be added in the project's Settings after the project has been created.

ℹ️
Note
Only collaborators with the Annotator or the Annotator & Reviewer roles are available for selection. All Admins and Team Managers will be part of all annotation stages and therefore can't be added, or removed.

A Webhook can optionally be added to an annotation stage to send out notifications whenever a task is submitted. Press the + icon next to 'Webhook' and add the URL of the server you would like to receive notifications on.

Routers

A Router splits the path that annotate and review tasks take through the workflow. Two different types of router are available:

Percentage router.
Collaborator router.

Video Tutorial - Routers

Percentage router

Pathways that data units take through the workflow can be split according to a certain percentage of annotation tasks. For example, 50% of tasks get sent to review stage A, while the other 50% of tasks get sent to review stage B, splitting the workload between two different groups of reviewers.

Once the router selects a route for a given task, it will always be routed the same way. This means that if a task passes through a percentage router and is routed to review stage A on its first pass, it will be routed to review stage A on each consecutive pass.

Collaborator router

Determine the paths that data units take through your workflow based on users. For example, if user A completes a task it gets sent to be reviewed by review stage A, while a task completed by user B gets reviewed in review stage B.

The 'Or else' option accounts for all collaborators not explicitly added to the router. In the image above they will be directed to review stage C.

Click Add pathway to select a subset of collaborators that will make up a pathway - only collaborators present in the previous stage can be selected. There is no limit on the amount of pathways you can create.
Click Add to finish a pathway.

ℹ️
Note
Collaborators can only be added to the Router once an input has been connected.

Review

Review represents a review stage where tasks labeled in an annotation stage are reviewed. A project can have multiple review stages, all of which have different reviewers, and are routed according to different conditions.

ℹ️
Note
Once a task has been rejected a review stage, it can't be passed onto another review stage directly.

Click a Review card that has been added to the canvas to edit. Click x next to the name to exit edit mode.

Click the icon to edit the name of the review stage. This is particularly useful when your project has multiple review stages.

The stage will contain all available reviewers selected by default. Specific reviewers can be selected by clicking the Review component, and choosing the Selective tab seen above. Type the email address of a user you'd like to add and select them from the list. Only collaborators that have been added via Manage collaborators on the project level will be available. Click Add to confirm the selection.

ℹ️
Note
Only collaborators with the Reviewer and Annotator & Reviewer roles are available for selection. All Admins and Team Managers will be part of all review stages and therefore can't be added, or removed.

Strict review

Video Tutorial - Strict review & change review judgement

Strict review ensures that crucial details are not overlooked during a review stage. Strict review aids in assuring higher accuracy when reviewing annotations.

Strict review stages don't auto-advance to the next review task after bulk approving or rejecting labels. Reviewers can make adjustments to their review, and add comments when rejecting tasks with missing annotations.

To enable Strict review mode, enable the Strict review toggle on the Review component while creating a workflow template.

User assignment restriction

ℹ️
Note
This feature is only available to select customers. Please contact [email protected] for more information.

The User assignment restriction feature prevents users from selected workflow stages being assigned to tasks in the current workflow stage. This can prevent individuals with the 'Annotator & Reviewer' role from being assigned their own annotations to review.

This feature doesn't apply to users with the Admin or Team Manager roles, since they are added to each stage by default and can't be removed.

Enable the toggle to implement User assignment restriction. Include the stages from which you intend to restrict user assignments in the highlighted list shown in the image above. For instance, in the provided example, individuals from the Annotate 1 stage will be prevented from task assignments within the Review 1 stage.

Complete

Complete represents the stage of a project at which a task is marked as complete, and can therefore only be used once per project.

Having passed though your workflow, it was deemed to be acceptable data for the training of machine learning algorithms. Complete tasks can still be accessed by admins.

Optionally, add a Webhook to this stage to receive notifications whenever a task arrives at the Complete stage.Press the + icon next to 'Webhook' and add the URL of the server you would like to receive notifications on.

Managing collaborators

Video tutorial - Managing collaborators

User permissions

Permission	Admin	Team Manager	Reviewer	Annotator	Annotator & Reviewer
Attach / Detach datasets	✅	❌	❌	❌	❌
Attach / Switch ontology	✅	❌	❌	❌	❌
Delete	✅	❌	❌	❌	❌
Invite team members	✅	✅	❌	❌	❌
Manage team permissions	✅	✅	❌	❌	❌
Manage admins	✅	❌	❌	❌	❌
Annotate & review tasks	✅	✅	Review only	Annotate only	✅
Confirm annotations	✅	✅	❌	❌	❌
Control assignments and status	✅	✅	❌	❌	❌

Project level collaborators

Click the Manage Collaborators button, as shown in the Managing collaborators section.

Add collaborators and select their roles in the 'Add tab', shown below.
Remove and manage collaborators in the 'Manage' tab.

Workflow stage level collaborators

Collaborators can be added to a workflow stage by clicking + on a card that has been added to the workflow canvas.

Learn how to add collaborators to different stages of a project by visiting the relevant sections in this document:

Annotate collaborators.
Review collaborators.
Collaborator type Router.

If no collaborators are specified at a given stage, all users with permissions will be added by default.

For example, if no collaborators are specified at an Annotate stage then all collaborators with the following roles wil be added by default:

Annotator
Annotator & Reviewer
Team Manager
Admin

Templates

Tired of creating the same workflow over and over again? Try creating templates of your most commonly used workflows - and use these to create annotation projects!

Video tutorial - Creating workflow templates

Creating templates

There are two ways of creating a workflow template:

In the Workflows section.
While creating a new annotation project.

Creating templates in the Workflows section

Select Workflow templates from the Annotate section of the sidebar in the Encord web app, and click the + New workflow button. Previously created templates will be visible here, and can be edited when selected.

Create and configure your workflow template, then click the Create workflow template button when you are done. To learn how to create and configure workflows, see our documentation here.

The Create workflow template button will be greyed out until:

The nodes in your template have been connected.
The workflow contains a Complete stage.
The template has been given a title.

Creating templates during project creation

New templates can be created during the creation of an new annotation project.

Click the + New workflow button in stage 4 of the project creation process to create a new workflow. Any workflow created can be saved as a template by clicking the Save as a new template button.

Editing existing templates

Workflow templates can be edited after they have been created. Navigate to Workflow templates in the Annotation section of the sidebar in the Encord web app, and click the name of the template you wish to edit.

Click the Edit button, as highlighted in the image below.

ℹ️
Note
You are not able to edit the start and annotation nodes.

Once you have made all desired edits, click the Save template changes button.

The Save template changes button will be greyed out until:

The nodes in your template have been connected.
The workflow contains a Complete stage.
The template has been given a title.

Creating annotation projects from templates

Video Tutorial - Creating projects from templates

Workflow templates can be used during project creation. To use a template during project creation, choose the template you would like to use and click the Select button during the workflow setup.

Customizing workflows

You can customize a workflow template to suit a project's specific needs. After selecting a template during project creation, as outlined in the section above, click the Customize template button to edit the workflow.

👍
Tip
Customized templates can be saved as a new template. Simply click the Save as a new template in the Workflow library to save the new template.

Template settings

The template settings allow you to:

Manage the team that has access to the template.
Delete the template.

Team

The Teams section allows you to control which users have access to the template. There are two types of user roles for templates:

Admins can view as well as edit the template.
Viewers can only view the template.

ℹ️
Note
Once a new Admin has been added, they can not be removed from the template.

Adding users to a template

Click the + Invite collaborators button.

Select a user role for the collaborator by choosing an option from the list.
Type the email address of the user you'd like to add and select the user from the list.
Click the Add button to add the user with the specified role.

Deleting templates

To delete a template:

Navigate to Workflows in the Annotate section of the sidebar.
Select the template you would like to delete.
Click the Settings tab and navigate to 'Danger zone'.
Click the Delete template button to delete the template.