Skip to main content
From version 0.2.0 of the SDK, project.workflow raises WrongProjectTypeError for Projects whose Workflow is available only through project.workflow_v3. Check project.has_workflow before you use project.workflow. For Projects that support project.workflow_v3, see Workflows (workflow_v3).

Workflow Basics

Encord SDK supports programmatic use of Workflows with the following: WorkflowStage, WorkflowAction, and Task. Typical use would be querying a WorkflowStage to get Tasks matching some criteria, and then executing a WorkflowAction against Tasks (for example: assigning, submitting annotation tasks, or approving review tasks). The type of actions available depends on the stage queried. Here is a short description of major parts of the Workflows SDK:
  • WorkflowStage: Move your tasks through a Workflow Project using the WorkflowStage class.
    • AnnotationStage
    • ReviewStage
    • ConsensusAnnotationStage
    • ConsensusReviewStage
    • AgentStage
    • FinalStage (Complete and Archive)
  • WorkflowAction: Applies an action to a task in a workflow stage The following actions are supported:
    • assign: Assigns a user to one or more tasks.
    • submit: Moves a task to the next stage.
    • release: Releases a task from the current user.
    • approve: Approves a review task.
    • reject: Rejects a review task.
    • move: Moves a task directly to any stage. See Move Tasks Freely.
    • proceed: Moves an Agent task along one of the Agent stage’s pathways. See Move Agent tasks along a pathway.
    Annotation, Review, Consensus Review, and Agent tasks also have an issues property for working with the task’s issues (for example, task.issues.add_file_issue(...)). See Issues & Comments in Projects.
  • Task: Exposes a simple version of the tasks available in a Project.
  • TaskStatus: The following statuses are available: The following statuses are available for ANNOTATE tasks:
    • NEW: Tasks that are unassigned.
    • ASSIGNED: Tasks that are assigned to a user.
    • RELEASED: Tasks that were released from an assigned user.
    • REOPENED: Tasks that were reopened.
    • SKIPPED: Tasks that were skipped by annotators.
    • COMPLETED: Tasks that are in the Complete stage. Used for Annotation sub-tasks in Consensus Annotation tasks.
Consensus Annotation tasks consist of 1 or more Annotation sub-tasks.
The following statuses are available for REVIEW and CONSENSUS REVIEW tasks:
  • NEW: Tasks that are unassigned.
  • ASSIGNED: Tasks that are assigned to a user.
  • RELEASED: Tasks that were released from an assigned user.
  • REOPENED: Tasks that were reopened.
  • SKIPPED: Tasks that were skipped by reviewers.
AGENT tasks use the same statuses as ANNOTATE tasks.All task status enums (AnnotationTaskStatus, ReviewTaskStatus, ConsensusReviewTaskStatus, and AgentTaskStatus), as well as LabelReviewStatus, also include:
  • UNKNOWN: Returned when the Encord platform reports a status that your version of the SDK does not recognize. Upgrade the encord SDK to see the actual status.
SKIPPED for Review and Consensus Review tasks, and the UNKNOWN fallback, require encord SDK version 0.1.207 or later.
In Consensus Projects, tasks in the COMPLETE and ARCHIVE blocks CANNOT be reopened.

View all stages in a Workflow

The following code provides you with the method to view all stages in a Workflow in a Project. The code is the same for Consensus and Non-Consensus Projects.
The SDK only returns stages that hold tasks: Annotate, Review, Consensus Annotate, Consensus Review, Agent, Complete, and Archive. Routers, Export, Metadata router, Parallelization, and Checkpoint stages are not returned, so you cannot read their settings (for example, an Export stage’s destination) with the SDK.
Non-Consensus Project This is an example of a non-Consensus Workflow graph.
Consensus Workflow Project This is an example of a Consensus Workflow graph.

Non-Consensus Projects

In the SDK, a class is defined for every stage type. This allows you to see what properties and actions each stage offers. The following stages are supported for non-Consensus Projects:
  • AnnotationStage
  • ReviewStage
  • AgentStage
  • FinalStage (Complete and Archive)
Every stage has a uuid and a title, and a get_tasks() method that returns the stage’s tasks. You can filter tasks with these optional arguments:
  • assignee: A user email, or a list of user emails.
  • data_hash: A data unit ID, or a list of data unit IDs.
  • dataset_hash: A dataset ID, or a list of dataset IDs.
  • data_title: Text that the data unit’s title contains.
  • status: A task status, or a list of task statuses (for example, AnnotationTaskStatus.ASSIGNED). Not available for the Complete and Archive stages, which also don’t take assignee.
Tasks provide programmatic access to the tasks available in the UI. The following actions are supported for non-Consensus Projects:
  • assign: Assigns a user to one or more tasks.
  • submit: Moves a task to the next stage.
  • release: Releases a task from the current user.
  • approve and reject: Approve or reject a review task.
  • move: Moves a task directly to any stage.
  • proceed: Moves an Agent task along one of the Agent stage’s pathways.
An annotation task has the following fields:

Display Tasks in a Stage

Use the following scripts as templates for annotation and review tasks.

Assign a User to Tasks in a Stage

Use the following scripts as templates for annotation and review tasks.

Submit Tasks as Another User in a Stage

  • Annotation Tasks can only be submitted on behalf of users with the following roles: Annotator, Annotator + Reviewer, Team Manager, or Admin.
  • Review Tasks are approved rather than submitted: approving a Review Task moves it to the next stage. Review Tasks can only be approved on behalf of users with the following roles: Reviewer, Annotator + Reviewer, Team Manager, or Admin.

Retain Current User and Submit Task

  • Annotation Tasks can only be submitted by users with the following roles: Annotator, Annotator + Reviewer, Team Manager, or Admin.
  • Review Tasks are approved rather than submitted. Review Tasks can only be approved by users with the following roles: Reviewer, Annotator + Reviewer, Team Manager, or Admin.
  • retain_assignee=True raises an error for a task with no assignee, so the following code only acts on tasks with the ASSIGNED status.

Release Tasks in a Stage

Releasing a task removes its current assignee and returns the task to the stage’s queue, so any eligible user can pick it up. Use release() on annotation tasks, review tasks, and Consensus review tasks.

Approve and Reject Review Tasks

Approving a review task moves it along the stage’s approval pathway, and also approves its label reviews that haven’t been approved or rejected yet. Rejecting a review task moves it along the stage’s rejection pathway. Pass assignee to act as a specific user, or retain_assignee=True to keep the task’s current assignee.

Reopen Label Reviews in a Review Task

Each label in a review task has its own label review, which can be approved, rejected, or reopened. Use get_label_reviews() to list the label reviews for a task, and reopen() to set an approved or rejected label review back to an open state so that it can be reviewed again.

Approve and Reject Label Reviews

Each label in a review task has its own label review. Use approve() and reject() on a label review to approve or reject a single label. A rejection can include a comment and issue tags. Approving or rejecting label reviews doesn’t approve or reject the task itself: to move the task along, approve or reject the task, as shown in Approve and Reject Review Tasks.

Get all Tasks

The following code lists the tasks in every stage of the Workflow that holds tasks.

Filter tasks by data hash

You can filter tasks in any stage using data hashes.

Filter tasks by status

Use the status parameter of get_tasks() to return only tasks with one or more specific statuses. Each stage type has its own status enum: AnnotationTaskStatus, ReviewTaskStatus, ConsensusReviewTaskStatus, or AgentTaskStatus.If the Encord platform returns a status that your SDK version does not recognize, the task’s status is UNKNOWN rather than raising an error.

Filter tasks by dataset, title, and assignee

You can combine the filters of get_tasks(): only tasks that match all of them are returned. dataset_hash and assignee take a single value or a list, and data_title returns tasks whose data unit title contains the text.

Move all tasks in a stage to the next stage

You can submit tasks as the currently assigned user or as a different user. If you do not specify the user to submit the task, you are assigned as the user submitting the task. When you keep the currently assigned user, the code only submits tasks with the ASSIGNED status, because a task without an assignee has no assignee to keep.

Move Agent tasks along a pathway

An Agent stage has one or more pathways, each leading to another stage. Use proceed() to send a task along a pathway, by its name or its UUID. To run agents that do this automatically, see Task Agents.

Use model predictions as an annotator

The following code assigns tasks to a user, adds your model’s predictions to the tasks’ label rows, saves the labels, and then submits the tasks as the same user. match_some_criteria() and do_preannotate() are placeholders: replace their bodies with your own logic.

Consensus Projects

Consensus annotation tasks can be listed, but cannot be assigned or moved into the Consensus review stage. In Consensus Projects, tasks are broken into subtasks. One subtask for each Annotator. Subtasks can be assigned or submitted. Consensus Projects support the following stages:
  • AnnotationStage
  • ReviewStage
  • ConsensusAnnotationStage
  • ConsensusReviewStage
  • FinalStage (Complete and Archive)
The following actions are supported for Consensus review and Review and Refine:
  • assign: Assigns a user to one or more tasks.
  • approve: Moves a task to the next stage.
  • release: Releases a task from the current user.
  • reject: Rejects a task.
  • move: Moves a task directly to any stage.

Get all Subtasks

Select your labels from a Consensus Project

The following code includes produce_result(), a placeholder for your own logic. It must compare the annotators’ label rows, add the labels you choose to the target label row, and return True to approve the task or False to reject it. Annotators who have not created labels yet have no label hash, so the code leaves them out, and skips tasks where no annotator has labels.

Work with Labels on Tasks

A task tracks where a data unit is in the Workflow. The data unit’s labels are in its label row. Use the task’s data_hash to get the label row.
  • Save labels before you submit a task. submit(), approve(), and the other task actions only move the task. They don’t upload labels. Call label_row.save() first, or labels you added in your script are not included.
  • You can edit labels on tasks in the Complete stage. Saving labels does not move the task: it stays in the Complete stage, so you don’t need to reopen it first.
  • Saving labels on a task in a Review stage updates the task’s label reviews. New labels within the stage’s Ontology scope get an approved review. Restoring a rejected label with an unresolved rejection comment gives it a pending review linked to that comment. Removing a label removes its review, while editing an existing label keeps its review status. After a label removal, a Review stage configured to submit automatically leaves through the applicable configured auto-submit pathway when no label review remains pending.
The following code gets the label rows for all tasks in a stage in one call. Use data_hash to identify the data unit in the Project, and backing_item_uuid to identify the storage item, for example to add it to a Collection.

Move Tasks

To Next Stage

Submitting a task moves it to the next stage in the Workflow.

To Complete

workflow_complete() moves Workflow tasks to the Complete stage. It needs the label row to have a label_hash, so the following code first initializes the label rows that don’t have one yet. To move tasks without working with label rows, see Move tasks to Complete with move().
In Consensus Projects, tasks in the COMPLETE and ARCHIVE blocks CANNOT be reopened.

Move tasks to Complete with move()

You can also move tasks to the Complete stage with each task’s move() method. This works whether or not the tasks’ label rows have a label_hash. Moving a task to a Complete stage approves all of its labels, and webhooks don’t fire. See Move Tasks Freely.

Reopen

Use the workflow_reopen() method to reopen tasks. Reopening a task moves it back to the first annotation stage in your Workflow. No labels are lost.
  • workflow_reopen() raises an error in Projects whose first annotation stage is a Consensus stage, for every task in the Project.
  • workflow_reopen() and workflow_complete() act on the main label branch.
  • Neither method updates the label row object in your script. To see a task’s new stage, list the label row again with project.list_label_rows_v2().

Move Tasks Freely

  • Moving tasks skips all intermediate Workflow stages, placing them immediately into the selected stage.
  • All labels (object and classification) are automatically approved when tasks move to a COMPLETE stage.
  • Webhooks do not fire when tasks teleport through a workflow.
This script shows how to move a data unit directly between workflow stages, bypassing the standard workflow progression. Replace the stage names with the stage to move the task from and the stage to move it to. You can move tasks out of and into Annotate, Review, Consensus Annotate, Consensus Review, Agent, Complete, and Archive stages.

Task Priority

You can read and assign task priorities using the SDK. Priorities in the SDK are represented by numbers ranging from 0 to 1, where 1 corresponds to 100 in the Encord platform, the highest priority value.
For more information, see our platform documentation for the Queue tab, and priorities.
Setting priorities requires the Admin or Team Manager role on the Project. A bundle sends priorities in requests of up to 1,000 tasks, so you can set priorities for any number of tasks in one bundle. priority is None for tasks in the Complete stage.

End-to-End Example moving tasks through Projects

This example uses a basic non-Consensus Project Workflow moving a number of tasks from ANNOTATE to COMPLETE. End-to-end Example Workflow End-to-end Example tasks
  • Project ID: 01bd7084-b04a-4cd1-87f3-73e8e78925c4
  • Data units: apples_01.jpg, apples_02.jpg, apples_03.jpg, apples_04.jpg, apples_05.jpg
A few tasks have been annotated, but they have not been submitted yet. The following code lists all tasks in the ANNOTATE 1 stage.
From the returned results we can see that four of the data units have been assigned (apples_01 to apples_04) while one task has not (apples_05). We do not know the extent that each of the assigned data units has been annotated, but we want the tasks to move through the Workflow.
The following code submits ALL tasks to the REVIEW 1 stage. Each task is submitted as a user in the Project, by passing the user’s email as assignee. The user could be any user capable of annotating in the Project (because this is the ANNOTATE stage). For this example, the Admin of the Project (the user with SDK access) is the assignee.
Each time we perform an action, we should verify the action is successful. After submitting the tasks, we can verify that the tasks are now in REVIEW 1.
All of the tasks are now in stage REVIEW 1. None of the tasks are assigned to any users.
We know that apples_05.jpg does not have any labels, because it was not assigned to any users before we submitted the task. If we REJECT the task, according to the Workflow in the Project, the task returns to the ANNOTATE 1 stage.The following code filters the task in the REVIEW 1 stage based on the task’s data hash, and then rejects the task.
From what was returned, we can see that only data unit apples_05.jpg was acted upon.To verify that the task is now in ANNOTATE 1, run the following:
There are still four tasks in REVIEW 1. The following code approves the tasks. This moves the tasks to the COMPLETE stage. You could approve all the tasks in the REVIEW 1 stage at once, but instead we will filter by data hash and then approve.
Use the following code to verify that the tasks apples_01 to apples_04 are in the COMPLETE stage.
And from what the code returns, the tasks are now in the COMPLETE stage.