Workflows and templates
Workflows
Workflows are a powerful tool to design and build your projects - letting you control how tasks move through different stages of the project, and determining how different stages interact with one another.
On this page you can find:
- A step-by-step tutorial on setting up and configuring a workflow.
- An explanation of all workflow composer components.
- An explanation of all workflow library components.
- A guide to Managing Collaborators
- A guide to workflow Templates
Creating workflows
This section will outline a step-by-step tutorial to creating workflows.
Note
Workflows are created when creating new annotation projects, and when creating workflow templates.
1. Add users to the Workflow
Add users from within your Organization to the Workflow by clicking the 
icon.
- Collaborators are added based on their role within the project - select the role you would like the collaborator(s) to have.
- Start typing the email of a user you would like to add into the area highlighted on the image below, and select the user from the list that appears. Repeat this for every user that will have the same role.
- When you are done selecting users for this role, click Add.
Note
Learn more about managing collaborators in the Managing collaborators section.
2. Configure your Workflow
The canvas is populated with a simple Workflow by default.
Click the Add stage button to show all Workflow components.
Customize your Workflow by pulling components onto the canvas.
Add the stages and other components you require for your Workflow onto the canvas:
-
All workflows must begin with the Start stage.
-
All workflows must contain an Annotate stage.
-
Add routers to your project to determine different pathways through your workflow a task can take. Click on the Router card to determine the type of router.
-
Add as many Review stages as necessary.
-
All workflows must end at a Complete stage.
-
Link all components on the canvas by clicking and dragging from one connection point to another, as seen below.
Tip
The cursor changes from
to
when hovering over a connection point.
All Workflows can be saved as a template by clicking the Save as a new template button.
3. Configure the stages of your Workflow
Once you’ve arranged the stages in the composer, it’s time to configure the details of each stage.
- Click an Annotate card on the canvas to start editing the annotation stage.
- Give the stage a descriptive name.
- Add annotators. If you'd like to specify annotators for this stage, add them as collaborators. For full details on how collaborators can work on tasks at each stage, see our section on managing collaborators.
- Optionally, add a Webhook to receive notifications when labels are submitted at this stage.
Note
More details about configuring annotation stages can be found in the Annotate section below
- Click a Review card on the canvas to start editing the review stage.
- Give the stage a descriptive name.
- Add reviewers. If you'd like to specify reviewers for this stage, add them as collaborators. For full details on how collaborators can work on tasks at each stage, see our section on managing collaborators.
Note
More details about configuring review stages can be found in the Review section below.
-
Click a Router card on your canvas. Please see the router section for more details on how different types of routers can be configured.
-
Optionally, add a Webhook to the Complete stage to receive a notification when a task has been completed.
-
Optionally, add User assignment restrictions if users in this node should be prevented from being assigned to tasks they completed in the nodes listed.
Note
More details on routers can be found in the router section below.
Workflow composer components
-
A. Manage Collaborators allows you to seamlessly add users to your project as collaborators, which can be added to different stages of the workflow.
-
B. The Workflow library contains all the components that make up a project. Drag the cards from the workflow library onto the workflow canvas (C) to build the stages a workflow should pass through. Some cards can be used more than once to create complex workflows. The configuration details for each component can be found in the Workflow library components section below.
-
C. The Workflow canvas is the canvas on which you design your workflow, by connecting cards from the workflow library. Set up the stages you'd like your workflow to pass through from left to right, and connect them to suit your needs. Click and drag the canvas to move it.
-
D. Zoom in and out of the Workflow canvas for convenience. Large workflow designs will require you to zoom out to see the whole picture. The
icon focuses in on your workflow. -
E. Create the project, or return to the previous step of the project creation process. Click Create Project to complete the creation process.
Caution
Currently, connections between nodes on the canvas can't be edited within a project after its creation. However, workflow templates can be edited and configured even when attached to ongoing projects. Click here.
Workflow library components
This section contains an overview of, and details on all Workflow library components.
Start
Start represents the beginning an annotation task through your workflow, and can therefore only be used once in a project. It must have a connector linking it to the next stage. It displays the datasets connected to the project, and the selected ontology.
Annotate
Annotate represents an annotation stage, during which a group of annotators adds labels to tasks assigned to them.
Click an Annotate card on the canvas to edit. Click x next to the name to exit edit mode.
Click the 
icon to edit the name of the annotation stage.
Tip
Annotation instructions can be added in the project's Settings after the project has been created.
The stage will contain all available annotators selected by default. Specific annotators can be selected by clicking the Annotate component, and choosing the Selective tab seen above. Type the email address of a user you'd like to add and select them from the list. Only collaborators that have been added via Manage collaborators on the project level will be available. Click Add to confirm the selection.
Tip
Annotation instructions can be added in the project's Settings after the project has been created.
Note
Only collaborators with the Annotator or the Annotator & Reviewer roles are available for selection. All Admins and Team Managers will be part of all annotation stages and therefore can't be added, or removed.
A Webhook can optionally be added to an annotation stage to send out notifications whenever a task is submitted. Press the + icon next to 'Webhook' and add the URL of the server you would like to receive notifications on.
Routers
A Router splits the path that annotate and review tasks take through the workflow. Two different types of router are available:
Percentage router
Pathways that data units take through the workflow can be split according to a certain percentage of annotation tasks. For example, 50% of tasks get sent to review stage A, while the other 50% of tasks get sent to review stage B, splitting the workload between two different groups of reviewers.
Once the router selects a route for a given task, it will be routed the same way until the routing percentages are updated. This means that if a task passes through a percentage router and is routed to review stage A on its first pass, it will be routed to review stage A on each consecutive pass.
Note
Routing percentages can be changed after Project creation. Changing routing percentages resets the percentage router, and may alter the pathway taken by a given task.
Collaborator router
Determine the paths that data units take through your workflow based on users. For example, if user A completes a task it gets sent to be reviewed by review stage A, while a task completed by user B gets reviewed in review stage B.
The 'Or else' option accounts for all collaborators not explicitly added to the router. In the image above they will be directed to review stage C.
-
Click Add pathway to select a subset of collaborators that will make up a pathway - only collaborators present in the previous stage can be selected. There is no limit on the amount of pathways you can create.
-
Click Add to finish a pathway.
Note
Collaborators can only be added to the Router after an input is connected.
Review
Note
Strict review is on by default. Contact [email protected] to disable strict review from your Workflow.
Review represents a review stage where tasks labeled in an annotation stage are reviewed.
Note
To attach the Rejected node of the review stage to another review stage, contact [email protected]
Click a Review card that has been added to the canvas to edit. Click x next to the name to exit edit mode.
Click the icon to edit the name of the review stage. This is particularly useful when your Project has multiple review stages.
The stage contains all available reviewers selected by default. Specific reviewers can be selected by clicking the Review component, and choosing the Selective tab seen above. Type the email address of a user you'd like to add and select them from the list. Only collaborators that have been added to the Workflow or the Project. Click Add to confirm the selection.
Note
Only collaborators with the Reviewer and Annotator & Reviewer roles are available for selection. All Admins and Team Managers are part of all review stages and cannot be removed.
Strict review
Note
Strict review is on by default. Contact [email protected] to disable strict review from your Workflow.
Strict review ensures that crucial details are not overlooked during a review stage. Strict review aids in assuring higher accuracy when reviewing annotations.
Strict review stages do not auto-advance to the next review task after bulk approving or rejecting labels. Reviewers can make adjustments to their review, and add comments when rejecting tasks with missing annotations.
Video Tutorial - Strict review & change review judgement
User assignment restriction
Note
This feature is only available upon request. Contact [email protected] for more information.
The User assignment restriction feature prevents users from selected workflow stages being assigned to tasks in the current Workflow stage. This can prevent individuals with the 'Annotator & Reviewer' role from being assigned their own annotations to review.
This feature doesn't apply to users with the Admin or Team Manager roles, since they are added to each stage by default and can't be removed.
Enable the toggle to implement User assignment restriction. Include the stages from which you intend to restrict user assignments in the highlighted list shown in the image above. For instance, in the provided example, individuals from the Annotate 1 stage will be prevented from task assignments within the Review 1 stage.
Complete
Complete represents the stage of a project at which a task is marked as complete, and can therefore only be used once per project.
Having passed though your workflow, it was deemed to be acceptable data for the training of machine learning algorithms. Complete tasks can still be accessed by admins.
Optionally, add a Webhook to this stage to receive notifications whenever a task arrives at the Complete stage.Press the + icon next to 'Webhook' and add the URL of the server you would like to receive notifications on.
Managing collaborators
User permissions
| Permission | Admin | Team Manager | Reviewer | Annotator | Annotator & Reviewer |
|---|---|---|---|---|---|
| Attach / Detach datasets | ✅ | ❌ | ❌ | ❌ | ❌ |
| Attach / Switch ontology | ✅ | ❌ | ❌ | ❌ | ❌ |
| Delete | ✅ | ❌ | ❌ | ❌ | ❌ |
| Invite team members | ✅ | ✅ | ❌ | ❌ | ❌ |
| Manage team permissions | ✅ | ✅ | ❌ | ❌ | ❌ |
| Manage admins | ✅ | ❌ | ❌ | ❌ | ❌ |
| Annotate & review tasks | ✅ | ✅ | Review only | Annotate only | ✅ |
| Confirm annotations | ✅ | ✅ | ❌ | ❌ | ❌ |
| Control assignments and status | ✅ | ✅ | ❌ | ❌ | ❌ |
Project level collaborators
Click the Manage Collaborators button, as shown in the Managing collaborators section.
- Add collaborators and select their roles in the Add tab.
- Remove and manage collaborators in the Manage tab.
Workflow stage level collaborators
Collaborators can be added to a workflow stage by clicking + on a card that has been added to the workflow canvas.
Learn how to add collaborators to different stages of a Project by visiting the relevant sections in this document:
If no collaborators are specified at a given stage, all users with permissions will be added by default.
For example, if no collaborators are specified at an Annotate stage then all collaborators with the following roles wil be added by default:
- Annotator
- Annotator & Reviewer
- Team Manager
- Admin
Templates
Tired of creating the same workflow over and over again? Try creating templates of your most commonly used workflows - and use these to create annotation projects!
Video tutorial - Creating workflow templates
Creating templates
There are two ways of creating a workflow template:
- In the Workflows section.
- While creating a new annotation project.
Creating templates in the Workflows section
Select Workflow templates from the Annotate section of the sidebar in the Encord web app, and click the + New workflow button. Previously created templates will be visible here, and can be edited when selected.
Create and configure your workflow template, then click the Create workflow template button when you are done. To learn how to create and configure workflows, see our documentation here.
The Create workflow template button will be greyed out until:
- The nodes in your template have been connected.
- The workflow contains a Complete stage.
- The template has been given a title.
Creating templates during project creation
New templates can be created during the creation of an new annotation project.
Once a Workflow has been created, click the Save as template button to save the Workflow as a template.
Edit templates
You can edit Workflow templates after they have been created.
- Navigate to Workflow templates under Annotation.
- Click the name of the template you want to edit.
Click the Edit button, as highlighted in the image below.
Note
You can not edit the start nodes.
Once you have made all desired edits, click the Save template changes button.
The Save template changes button will be greyed out until:
- The nodes in your template have been connected.
- The workflow contains a Complete stage.
- The template has been given a title.
Creating annotation projects from templates
Workflow templates can be used during project creation. To use a template during project creation, choose the template you would like to use and click the Select button during the workflow setup.
Customizing templates
You can customize/edit a Workflow template to suit a project's specific needs.
Click the Save as a new template button after customizing/editing a template to create a new template.
Template settings
The template settings allow you to:
Team
The Teams section allows you to control which users have access to the template. There are two types of user roles for templates:
- Admins can view as well as edit the template.
- Viewers can only view the template.
Note
Once a new Admin has been added, they can not be removed from the template.
Adding users to a template
- Click the + Invite collaborators button.
-
Select a user role for the collaborator by choosing an option from the list.
-
Type the email address of the user you'd like to add and select the user from the list.
-
Click the Add button to add the user with the specified role.
Deleting templates
To delete a template:
- Navigate to Workflows in the Annotate section of the sidebar.
- Select the template you would like to delete.
- Click the Settings tab and navigate to 'Danger zone'.
- Click the Delete template button to delete the template.
Updated 10 days ago