Workflows and Templates
Workflows
Workflows are a powerful tool to design and build your Projects - letting you control how tasks move through different stages of the Project, and determining how different stages interact with one another.
On this page you can find:
- A step-by-step tutorial on setting up and configuring a Workflow.
- An explanation of all Workflow composer components.
- An explanation of all Workflow library components.
- A guide to Managing Collaborators
- A guide to Workflow Templates
Creating workflows
Workflow composer components
-
A. Manage Collaborators allows you to seamlessly add users to your project as collaborators, which can be added to different stages of the workflow.
-
B. The Workflow library contains all the components that make up a project. Drag the cards from the workflow library onto the workflow canvas (C) to build the stages a workflow should pass through. Some cards can be used more than once to create complex workflows. The configuration details for each component can be found in the Workflow library components section below.
-
C. The Workflow canvas is the canvas on which you design your workflow, by connecting cards from the workflow library. Set up the stages you’d like your workflow to pass through from left to right, and connect them to suit your needs. Click and drag the canvas to move it.
-
D. Zoom in and out of the Workflow canvas for convenience. Large workflow designs will require you to zoom out to see the whole picture. The zoom icon focuses in on your workflow.
-
E. Create the Project, or return to the previous step of the project creation process. Click Create Project to complete the creation process.
Workflow library components
This section contains details for all Workflow library components.
Start
Start represents the beginning an annotation task through your Workflow, and can therefore only be used once in a Project. It must have a connector linking it to the next stage. It displays the Datasets connected to the Project, and the selected Ontology.
Annotate
Annotate represents an annotation stage, during which a group of annotators adds labels to tasks assigned to them.
Click an Annotate card on the canvas to edit. Click x next to the name to exit edit mode.
Click the Edit icon to edit the name of the annotation stage.
Annotation instructions can be added in the Project’s Settings after the Project is created.
The stage will contain all available annotators selected by default. Specific annotators can be selected by clicking the Annotate component, and choosing the Selective tab seen above. Type the email address of a user you’d like to add and select them from the list. Only collaborators that have been added via Manage collaborators on the Project level will be available. Click Add to confirm the selection.
A Webhook can optionally be added to an annotation stage to send out notifications whenever a task is submitted. Click the Edit icon next to Webhook and add the URL of the server you would like to receive notifications on.
Routers
A Router splits the path that annotate and review tasks take through the workflow. Two different types of router are available:
Percentage router
Pathways that data units take through the Workflow can be split according to a certain percentage of annotation tasks. For example, 50% of tasks get sent to review stage A, while the other 50% of tasks get sent to review stage B, splitting the workload between two different groups of reviewers.
Once the router selects a route for a given task, it will be routed the same way until the routing percentages are updated. This means that if a task passes through a percentage router and is routed to review stage A on its first pass, it will be routed to review stage A on each consecutive pass.
Collaborator router
Determine the paths that data units take through your workflow based on users. For example, if user A completes a task it gets sent to be reviewed by review stage A, while a task completed by user B gets reviewed in review stage B.
The Or else option accounts for all collaborators not explicitly added to the router. In the image above they will be directed to review stage C.
-
Click Add pathway to select a subset of collaborators that will make up a pathway - only collaborators present in the previous stage can be selected. There is no limit on the amount of pathways you can create.
-
Click Add to finish a pathway.
Review
Review represents a review stage where tasks labeled in an annotation stage are reviewed.
Click a Review card that has been added to the canvas to edit. Click x next to the name to exit edit mode.
Click the Edit icon to edit the name of the review stage. This is particularly useful when your Project has multiple review stages.
The stage contains all available reviewers selected by default. Specific reviewers can be selected by clicking the Review component, and choosing the Selective tab seen above. Type the email address of a user you’d like to add and select them from the list. Only collaborators that have been added to the Workflow or the Project. Click Add to confirm the selection.
Strict review
Strict review ensures that crucial details are not overlooked during a review stage. Strict review aids in assuring higher accuracy when reviewing annotations.
Strict review stages do not auto-advance to the next review task after bulk approving or rejecting labels. Reviewers can make adjustments to their review, and add comments when rejecting tasks with missing annotations.
Video Tutorial - Strict review & change review judgement
Video Tutorial - Strict review & change review judgement
User assignment restriction
The User assignment restriction feature prevents team members working on tasks in the current stage that they have done in previous stages. This can prevent individuals with the Annotator & Reviewer role from being assigned their own annotations to review.
This feature does not apply to users with the Admin or Team Manager roles, since they are added to each stage by default and cannot be removed.
Enable the toggle to implement User assignment restriction.
Agent
The Agent component represents a Task Agent. It triggers a custom operation each time a task passes through the Agent stage. The exact operation being triggered is configured using the Encord SDK.
To fully connect an Agent to other Workflow components you must add pathways that allow you to create connections from the Agent stage to other components. To add a pathway, click + next to Add pathway and give the pathway a meaningful name. Repeat this step for each additional pathway you want to add.
Complete
Complete represents the stage of a Project at which a task is marked as complete, and can therefore only be used once per project.
Having passed though your Workflow, it was deemed to be acceptable data for the training of machine learning algorithms. Complete tasks can still be accessed by admins.
Optionally, add a Webhook to this stage to receive notifications whenever a task arrives at the Complete stage.Click the Edit icon next to ‘Webhook’ and add the URL of the server you would like to receive notifications on.
Managing collaborators
User permissions
| Permission | Admin | Team Manager | Reviewer | Annotator | Annotator & Reviewer |
|---|---|---|---|---|---|
| Attach / Detach datasets | ✅ | ❌ | ❌ | ❌ | ❌ |
| Attach / Switch ontology | ✅ | ❌ | ❌ | ❌ | ❌ |
| Delete | ✅ | ❌ | ❌ | ❌ | ❌ |
| Invite team members | ✅ | ✅ | ❌ | ❌ | ❌ |
| Manage team permissions | ✅ | ✅ | ❌ | ❌ | ❌ |
| Manage admins | ✅ | ❌ | ❌ | ❌ | ❌ |
| Annotate & review tasks | ✅ | ✅ | Review only | Annotate only | ✅ |
| Confirm annotations | ✅ | ✅ | ❌ | ❌ | ❌ |
| Control assignments and status | ✅ | ✅ | ❌ | ❌ | ❌ |
Project level collaborators
Click the Manage Collaborators button, as shown in the Managing collaborators section.
- Add collaborators and select their roles in the Add tab.
- Remove and manage collaborators in the Manage tab.
Workflow stage level collaborators
Collaborators can be added to a workflow stage by clicking + on a card that has been added to the workflow canvas.
Learn how to add collaborators to different stages of a Project by visiting the relevant sections in this document:
If no collaborators are specified at a given stage, all users with permissions are added by default.
For example, if no collaborators are specified at an Annotate stage then all collaborators with the following roles will be added by default:
- Annotator
- Annotator & Reviewer
- Team Manager
- Admin
Templates
Tired of creating the same workflow over and over again? Try creating templates of your most commonly used workflows - and use these to create annotation Projects!
Video tutorial - Creating workflow templates
Video tutorial - Creating workflow templates
Creating templates
There are two ways of creating a workflow template:
- In the Workflows section.
- While creating a new annotation project.
Creating templates in the Workflows section
Select Workflow templates from the Annotate section of the sidebar in the Encord platform, and click the + New workflow template button. Previously created templates are visible here, and can be edited when selected.
Create and configure your Workflow template, then click the Create workflow template button when you are done. To learn how to create and configure Workflows, see our documentation here.
The Create workflow template button is greyed out until:
- The nodes in your template have been connected.
- The workflow contains a Complete stage.
- The template has been given a title.
Creating templates during Project creation
New templates can be created during the creation of a new annotation Project.
Once a Workflow has been created, click the Save as template button to save the Workflow as a template.
Edit templates
You can edit Workflow templates after they have been created.
- Navigate to Workflow templates under Annotation.
- Click the name of the template you want to edit.
Click the Edit button, as highlighted in the image below.
Once you have made all desired edits, click the Save template changes button.
The Save template changes button will be greyed out until:
- The nodes in your template have been connected.
- The workflow contains a Complete stage.
- The template has been given a title.
Creating annotation Projects from templates
Workflow templates can be used during Project creation. To use a template during Project creation, choose the template you would like to use and click the Select button during the workflow setup.
Customizing templates
You can customize/edit a Workflow template to suit a project’s specific needs.
Click the Save as a new template button after customizing/editing a template to create a new template.
Template settings
The template settings allow you to:
Team
The Teams section allows you to control which users have access to the template. There are two types of user roles for templates:
- Admins can view as well as edit the template.
- Viewers can only view the template.
Adding users to a template
- Click the + Invite collaborators button.
-
Select a user role for the collaborator by choosing an option from the list.
-
Type the email address of the user you’d like to add and select the user from the list.
-
Click the Add button to add the user with the specified role.
Deleting templates
To delete a template:
- Navigate to Workflows in the Annotate section of the sidebar.
- Select the template you would like to delete.
- Click the Settings tab and navigate to ‘Danger zone’.
- Click the Delete template button to delete the template.
