Workflows are a powerful tool for designing and building Projects, allowing you to control how tasks move through different stages and how those stages interact.
Start represents the beginning an annotation task through your Workflow, and can therefore only be used once in a Project. It must have a connector linking it to the next stage. It displays the Datasets connected to the Project, and the selected Ontology.
Annotate represents an annotation stage, during which a group of annotators adds labels to tasks assigned to them.Click an Annotate card on the canvas to edit. Click x next to the name to exit edit mode.Click the Edit icon to edit the name of the annotation stage.
Annotation instructions can be added in the Project’s Settings after the Project is created.
The stage contains all available annotators selected by default. Specific annotators can be selected in the Selective tab. Only collaborators that are added to the Workflow are available for selection. Click Add to confirm the selection.
Only collaborators with the Annotator or the Annotator & Reviewer roles are available for selection. All Admins and Team Managers are part of all annotation stages and therefore cannot be added, or removed.
A Webhook can optionally be added to an annotation stage to send out notifications whenever a task is submitted. Click the Edit icon next to Webhook and add the URL of the server you want to receive notifications on. Once a valid URL is added and the Workflow saves, a Signing Secret becomes available that can be used to verify that incoming notifications are genuinely from Encord.Advanced SettingsAllow task reassessment: When enabled, annotators can see and review the same task again later in the workflow. Default: Disabled.Hide unassigned tasks: When enabled, users with the Annotator, Reviewer, or Annotator & Reviewer roles can only see tasks assigned to them. Admins and Team Managers can view all tasks, including unassigned ones.
Pathways that data units take through the Workflow can be split according to a certain percentage of annotation tasks. For example, 50% of tasks get sent to review stage A, while the other 50% of tasks get sent to review stage B, splitting the workload between two different groups of reviewers.
Once the router selects a route for a given task, it is routed the same way until the routing percentages are updated. This means that if a task passes through a percentage router and is routed to review stage A on its first pass, it is routed to review stage A on each consecutive pass.
Routing percentages can be changed after Project creation. Changing routing percentages resets the percentage router, and may alter the pathway taken by a given task.
Determine the paths that data units take through your workflow based on users. For example, if user A completes a task it gets sent to be reviewed by review stage A, while a task completed by user B gets reviewed in review stage B.
The Or else option accounts for all collaborators not explicitly added to the router. In this example they are directed to review stage C.
Click Add pathway to choose a subset of collaborators that form the pathway. - only collaborators present in the previous stage can be selected. There is no limit on the amount of pathways you can create.
Click Add to finish a pathway.
Collaborators can only be added to the Router after an input is connected.
Review represents a review stage where tasks labeled in an annotation stage are reviewed.
To attach the Rejected node of the review stage to another review stage, contact support@encord.com
Click a Review card that has been added to the canvas to edit. Click x next to the name to exit edit mode.Click the Edit icon to edit the name of the review stage. This is particularly useful when your Project has multiple review stages.
The stage contains all available reviewers selected by default. Specific reviewers can be selected by clicking the Review component, and choosing the Selective tab seen above. Only collaborators that have been added to the Workflow or the Project. Click Add to confirm the selection.
Only collaborators with the Reviewer and Annotator & Reviewer roles are available for selection. All Admins and Team Managers are part of all review stages and cannot be removed.
Advanced SettingsAllow task reassessment: When enabled, reviewers can review the same task multiple times as it moves through the Workflow. This also allows a user to review tasks they previously annotated. Default: Enabled.
Strict review is enabled by default, but can be disabled under Advanced settings.
Strict review ensures that crucial details are not overlooked during a review stage. Strict review aids in assuring higher accuracy when reviewing annotations.Strict review stages do not auto-advance to the next review task after bulk approving or rejecting labels. Reviewers can make adjustments to their review, and add comments when rejecting tasks with missing annotations.
Video Tutorial - Strict review & change review judgment
The User assignment restriction feature prevents team members working on tasks in the current stage that they have done in previous stages. This can prevent individuals with the Annotator & Reviewer role from being assigned their own annotations to review.This feature does not apply to users with the Admin or Team Manager roles, since they are added to each stage by default and cannot be removed.
Enable the toggle to implement User assignment restriction.
The Agent component represents an Agent. It triggers a custom operation each time a task passes through the Agent stage. The exact operation being triggered is configured using the Encord SDK.To fully connect an Agent to other Workflow components you must add pathways that allow you to create connections from the Agent stage to other components. To add a pathway, click + next to Add pathway and give the pathway a meaningful name. Repeat this step for each additional pathway you want to add.
Export represents a stage in a Project’s Workflow that automatically syncs labels to your cloud storage as tasks pass through it.Every task that passes through the Export stage is synced to your chosen storage destination. Once the sync completes, tasks automatically continue to the next stage in your Workflow. This allows you to build a fully automated labeling pipeline, from task creation through to label delivery, without any manual exports. For more information, see How to Export Labels.
Complete represents the stage of a Project at which a task is marked as complete, and can therefore only be used once per project.Having passed though your Workflow, it was deemed to be acceptable data for the training of machine learning algorithms. Complete tasks can still be accessed by admins.Optionally, add a Webhook to this stage to receive notifications whenever a task arrives at the Complete stage. Adding a valid Webhook URL and saving the Workflow generates a Signing Secret that can be used to verify that incoming notifications are genuinely from Encord.
In Consensus Projects, tasks in the COMPLETE and ARCHIVE blocks CANNOT be reopened.
If no collaborators are specified at a given stage, all users with permissions are added by default.For example, if no collaborators are specified at an Annotate stage then all collaborators with the following roles are added by default:
Workflow templates allow you to save and reuse your most common workflows. You can then use these templates to create new annotation Projects, ensuring consistency and efficiency.
Remove empty stages: Delete stages with zero tasks
System validation: Automatic checking prevents problematic deletions
Any stage containing tasks cannot be deleted. Move tasks to a different stage before deleting a stage.
Modify workflow connections
Change edge directions: Alter task flow paths
Expand router pathways: Add new routing options
Adjust percentage routing: Modify router distribution percentages
Workflow management
Save changes: Persist workflow modifications
Re-name: Change the name of stages
Pause project: Stop task flow during Workflow modifications
Track last edited: See modification history and who made changes.
Encord tracks all edits so if we need to investigate / roll back we are capable of doing so. Not all logs are exposed to customers.
Edit Workflows from the Annotate > Projects > [select a Project] > Workflow page.
You CANNOT delete stages that contain tasks. If you want to delete a stage that has tasks, first move the tasks to a different stage and then delete the stage.
