Projects
Workflows and Templates
Workflows are a powerful tool for designing and building Projects, allowing you to control how tasks move through different stages and how those stages interact.
Annotate represents an annotation stage, during which a group of annotators adds labels to tasks assigned to them.Click an Annotate card on the canvas to edit. Click x next to the name to exit edit mode.Click the Edit icon to edit the name of the annotation stage.
The stage contains all available annotators selected by default. Specific annotators can be selected in the Selective tab. Type the email address of users you want to add and select them from the list. Only collaborators that have been added to the Workflow are available for selection. Click Add to confirm the selection.A Webhook can optionally be added to an annotation stage to send out notifications whenever a task is submitted. Click the Edit icon next to Webhook and add the URL of the server you want to receive notifications on.
Pathways that data units take through the Workflow can be split according to a certain percentage of annotation tasks. For example, 50% of tasks get sent to review stage A, while the other 50% of tasks get sent to review stage B, splitting the workload between two different groups of reviewers.
Once the router selects a route for a given task, it is routed the same way until the routing percentages are updated. This means that if a task passes through a percentage router and is routed to review stage A on its first pass, it is routed to review stage A on each consecutive pass.
Determine the paths that data units take through your workflow based on users. For example, if user A completes a task it gets sent to be reviewed by review stage A, while a task completed by user B gets reviewed in review stage B.
The Or else option accounts for all collaborators not explicitly added to the router. In the image above they will be directed to review stage C.
Review represents a review stage where tasks labeled in an annotation stage are reviewed.Click a Review card that has been added to the canvas to edit. Click x next to the name to exit edit mode.Click the Edit icon to edit the name of the review stage. This is particularly useful when your Project has multiple review stages.
The stage contains all available reviewers selected by default. Specific reviewers can be selected by clicking the Review component, and choosing the Selective tab seen above. Type the email address of a user you’d like to add and select them from the list. Only collaborators that have been added to the Workflow or the Project. Click Add to confirm the selection.
Strict review ensures that crucial details are not overlooked during a review stage. Strict review aids in assuring higher accuracy when reviewing annotations.Strict review stages do not auto-advance to the next review task after bulk approving or rejecting labels. Reviewers can make adjustments to their review, and add comments when rejecting tasks with missing annotations.
Enable the toggle to implement User assignment restriction.
The Agent component represents a Task Agent. It triggers a custom operation each time a task passes through the Agent stage. The exact operation being triggered is configured using the Encord SDK.To fully connect an Agent to other Workflow components you must add pathways that allow you to create connections from the Agent stage to other components. To add a pathway, click + next to Add pathway and give the pathway a meaningful name. Repeat this step for each additional pathway you want to add.
Complete represents the stage of a Project at which a task is marked as complete, and can therefore only be used once per project.Having passed though your Workflow, it was deemed to be acceptable data for the training of machine learning algorithms. Complete tasks can still be accessed by admins.Optionally, add a Webhook to this stage to receive notifications whenever a task arrives at the Complete stage.Click the Edit icon next to ‘Webhook’ and add the URL of the server you would like to receive notifications on.
Learn how to add collaborators to different stages of a Project by visiting the relevant sections in this document:
If no collaborators are specified at a given stage, all users with permissions are added by default.
For example, if no collaborators are specified at an Annotate stage then all collaborators with the following roles will be added by default:
Creating Workflows
Workflows are created when creating new annotation Projects, and when creating Workflow templates.
- Manage Collaborators allows you to seamlessly add users to your Workflow as collaborators, which can be added to different stages of the workflow.
- Add stage allows you to add more stages to your Workflow.
Workflow Stages
This section contains details for all Workflow library components.To add library components to the canvas, click the Add stage button and then drag-and-drop the components onto the canvas.
Start
Start
Annotate
Annotate
Annotate
Annotation instructions can be added in the Project’s Settings after the Project is created.
Only collaborators with the Annotator or the Annotator & Reviewer roles are available for selection. All Admins and Team Managers will be part of all annotation stages and therefore can’t be added, or removed.
Routers
Routers
Routers
A Router splits the path that annotate and review tasks take through the workflow. Two different types of router are available:Percentage router
Routing percentages can be changed after Project creation. Changing routing percentages resets the percentage router, and may alter the pathway taken by a given task.
Collaborator router
- Click Add pathway to select a subset of collaborators that will make up a pathway - only collaborators present in the previous stage can be selected. There is no limit on the amount of pathways you can create.
- Click Add to finish a pathway.
Collaborators can only be added to the Router after an input is connected.
Review
Review
Review
To attach the Rejected node of the review stage to another review stage, contact support@encord.com
Only collaborators with the Reviewer and Annotator & Reviewer roles are available for selection. All Admins and Team Managers are part of all review stages and cannot be removed.
Strict review
Strict review is enabled by default, but can be disabled under Advanced settings.
Video Tutorial - Strict review & change review judgment
Video Tutorial - Strict review & change review judgment
User assignment restriction
The User assignment restriction feature prevents team members working on tasks in the current stage that they have done in previous stages. This can prevent individuals with the Annotator & Reviewer role from being assigned their own annotations to review.This feature does not apply to users with the Admin or Team Manager roles, since they are added to each stage by default and cannot be removed.Agent
Agent
Agent
Learn more about Task Agents in our Task Agent section
Complete
Complete
Complete
In Consensus Projects, tasks in the COMPLETE and ARCHIVE blocks CANNOT be reopened.
Managing collaborators
User permissions
| Permission | Admin | Team Manager | Reviewer | Annotator | Annotator & Reviewer |
|---|---|---|---|---|---|
| Attach / Detach datasets | ✅ | ❌ | ❌ | ❌ | ❌ |
| Attach / Switch ontology | ✅ | ❌ | ❌ | ❌ | ❌ |
| Delete | ✅ | ❌ | ❌ | ❌ | ❌ |
| Invite team members | ✅ | ✅ | ❌ | ❌ | ❌ |
| Manage team permissions | ✅ | ✅ | ❌ | ❌ | ❌ |
| Manage admins | ✅ | ❌ | ❌ | ❌ | ❌ |
| Annotate & review tasks | ✅ | ✅ | Review only | Annotate only | ✅ |
| Confirm annotations | ✅ | ✅ | ❌ | ❌ | ❌ |
| Control assignments and status | ✅ | ✅ | ❌ | ❌ | ❌ |
Workflow stage level collaborators
Collaborators can be added to a workflow stage by clicking + on a card that has been added to the workflow canvas.- Annotator
- Annotator & Reviewer
- Team Manager
- Admin
Templates
Workflow templates allow you to save and reuse your most common workflows. You can then use these templates to create new annotation Projects, ensuring consistency and efficiency.Create Templates
New Workflow templates can be created each time a new Workflow is created.- Navigate to the Projects section of the Encord platform.
- Navigate to the Templates tab.
- Click Create template.
- Create your Workflow. Ensure your Workflow has a title.
- Click the Create workflow template button. The template is saved in the custom templates section.
Edit templates
You can edit Workflow templates after they have been created.- Navigate to the Projects section of the Encord platform.
- Navigate to the Templates tab.
- Select the template you want to Edit.
- Click the Edit button in the top-right.
- Edit the template.
- Click Save changes.
You can not edit the start node.
Template Collaborators
- Click the + Invite collaborators button.
- Select a user role for the collaborator by choosing an option from the list.
- Type the email address of the user you’d like to add and select the user from the list.
- Click the Add button to add the user with the specified role.
Delete Templates
To delete a template, click the red Delete icon next to the template.