Workflow Basics
Encord SDK supports programmatic use of Workflows with the following: WorkflowStage
, WorkflowAction
, and Task
. Typical use would be querying a WorkflowStage
to get Tasks
matching some criteria, and then executing a WorkflowAction
against Tasks (for example: assigning, submitting annotation tasks, or accepting review tasks). The type of actions available depends on the stage queried.
Here is a short description of major parts of the Workflows SDK:
-
WorkflowStage
: Move your tasks through a Workflow Project using the WorkflowStage
class. However, tasks CANNOT “teleport” through your Workflow. Tasks must move through the Workflow in logical order.
These are the stages currently supported:
AnnotationStage
ReviewStage
ConsensusAnnotationStage
ConsensusReviewStage
FinalStage
(Complete and Archive)
-
WorkflowAction
: Applies an action to a task in a workflow stage
The following actions are supported:
assign
: Assigns a user to one or more tasks.
submit
: Moves a task to the next stage.
release
: Releases a task from the current user.
accept
: Accepts a task.
reject
: Rejects a task.
-
Task
: Exposes a simple version of the tasks available in a Project.
-
TaskStatus
: The following statuses are available:
The following statuses are available for ANNOTATE tasks:
-
NEW
: Tasks that are unassigned.
-
ASSIGNED
: Tasks that are assigned to a user.
-
RELEASED
: Tasks that were released from an assigned user.
-
REOPENED
: Tasks that were reopened.
-
SKIPPED
: Tasks that were skipped by annotators.
-
COMPLETED
: Tasks that are in the Complete stage. Used for Annotation sub-tasks in Consensus Annotation tasks.
Consensus Annotation tasks consist of 1 or more Annotation sub-tasks.
The following statuses are available for REVIEW and CONSENSUS REVIEW tasks:
NEW
: Tasks that are unassigned.
ASSIGNED
: Tasks that are assigned to a user.
RELEASED
: Tasks that were released from an assigned user.
REOPENED
: Tasks that were reopened.
In Consensus Projects, tasks in the COMPLETE and ARCHIVE blocks CANNOT be reopened.
View all stages in a Workflow
The following code provides you with the method to view all stages in a Workflow in a Project. The code is the same for Consensus and Non-Consensus Projects.
Routers are not displayed in the list of stages when using the SDK.
Non-Consensus Workflow Project
This is an example of a non-Consensus Workflow graph.
Consensus Workflow Project
This is an example of a Consensus Workflow graph.
Non-Consensus Workflow Projects
In the SDK, a class is defined for every stage type. This allows you to see what properties and actions each stage offers. The following stages are supported for non-Consensus Projects:
AnnotationStage
ReviewStage
FinalStage
(Complete and Archive)
class AnnotationStage(WorkflowStage):
uuid: str
title: str
def get_tasks()
Tasks
provides programmatic access to your tasks available in the UI.
The following tasks are supported for non-Consensus Projects::
assign
: Assigns a user to one or more tasks.
submit
: Moves a task to the next stage.
release
: Releases a task from the current user.
class AnnotationTask:
uuid: UUID
data_hash: UUID
data_title: str
label_branch: str
Display Tasks in a Stage
from encord import EncordUserClient, Project
from encord.workflow import(
AnnotationStage,
ReviewStage,
ConsensusAnnotationStage,
ConsensusReviewStage,
FinalStage
)
SSH_PATH = "/Users/chris-encord/sdk-ssh-private-key.txt"
PROJECT_HASH = "<project-unique-id>"
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
ssh_private_key_path=SSH_PATH
)
project: Project = user_client.get_project(PROJECT_HASH)
stage = project.workflow.get_stage(name="Annotate 1", type_=AnnotationStage)
assert isinstance(stage, AnnotationStage)
for task in stage.get_tasks():
print(f"Task: {task}")
Assign a User to Tasks in a Stage
from encord import EncordUserClient, Project
from encord.workflow import(
AnnotationStage,
ReviewStage,
ConsensusAnnotationStage,
ConsensusReviewStage,
FinalStage
)
SSH_PATH = "/Users/chris-encord/sdk-ssh-private-key.txt"
PROJECT_HASH = "<project-unique-id>"
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
ssh_private_key_path=SSH_PATH
)
project: Project = user_client.get_project(PROJECT_HASH)
stage = project.workflow.get_stage(name="Annotate 1", type_=AnnotationStage)
assert isinstance(stage, AnnotationStage)
for task in stage.get_tasks():
task.assign("<your-admin@example.com>")
print(f"Task: {task}")
Get all Tasks
from encord import EncordUserClient, Project
from encord.workflow import(
AnnotationStage,
ReviewStage,
ConsensusAnnotationStage,
ConsensusReviewStage,
FinalStage
)
SSH_PATH = "/Users/chris-encord/sdk-ssh-private-key.txt"
PROJECT_HASH = "<project-unique-id>"
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
ssh_private_key_path=SSH_PATH
)
project: Project = user_client.get_project(PROJECT_HASH)
stage = project.workflow.get_stage(name="Consensus 1", type_=ConsensusAnnotationStage)
assert isinstance(stage, ConsensusAnnotationStage)
for task in stage.get_tasks():
print(f"Task: {task}")
Filter tasks by data hash
You can filter tasks in any stage using data hashes.
Filter tasks by data hash
from encord import EncordUserClient, Project
from encord.workflow import AnnotationStage, ReviewStage, ConsensusAnnotationStage, ConsensusReviewStage, FinalStage
SSH_PATH = "<ssh-private-key>"
PROJECT_HASH = "<project-unique-id>"
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
ssh_private_key_path=SSH_PATH
)
project: Project = user_client.get_project(PROJECT_HASH)
stage = project.workflow.get_stage(name="Review 1", type_=ReviewStage)
assert isinstance(stage, ReviewStage)
data_hashes = ["<data-unique-id-01>", "<data-unique-id-02>", "<data-unique-id-03>"]
for task in stage.get_tasks(data_hash=data_hashes):
print(f"Task: {task}")
Move all tasks in a stage to the next stage
Before tasks can be moved along in the Workflow, they must be assigned to one user (for example, your admin).
from encord import EncordUserClient, Project
from encord.workflow import(
AnnotationStage,
ReviewStage,
ConsensusAnnotationStage,
ConsensusReviewStage,
FinalStage
)
SSH_PATH = "/Users/chris-encord/sdk-ssh-private-key.txt"
PROJECT_HASH = "<project-unique-id>"
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
ssh_private_key_path=SSH_PATH
)
project: Project = user_client.get_project(PROJECT_HASH)
stage = project.workflow.get_stage(name="Annotate 1", type_=AnnotationStage)
assert isinstance(stage, AnnotationStage)
for task in stage.get_tasks():
task.assign("admin-dana@encord.com")
task.submit()
print(f"Task: {task}")
Use model predictions as an annotator
from encord import EncordUserClient, Project
from encord.workflow import(
AnnotationStage,
ReviewStage,
ConsensusAnnotationStage,
ConsensusReviewStage,
FinalStage
)
SSH_PATH = "/Users/chris-encord/sdk-ssh-private-key.txt"
PROJECT_HASH = "<project-unique-id>"
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
ssh_private_key_path=SSH_PATH
)
project: Project = user_client.get_project(PROJECT_HASH)
stage = project.workflow.get_stage(name="Automated Annotation", type_=AnnotationStage)
assert isinstance(stage, AnnotationStage)
for task in stage.get_tasks():
task.assign("admin-alex@encord.com")
print(f"Task: {task}")
if match_some_criteria(task)
for label_row in project.list_label_rows_v2(data_hashes=[task.data_hash]):
label_row.initialise_labels()
do_preannotate(label_row)
label_row.save()
task.submit()
Consensus Workflow Projects
Consensus annotation tasks can be listed, but cannot be assigned or moved into the Consensus review stage.
Consensus Workflow Projects support the following stages:
AnnotationStage
ReviewStage
ConsensusAnnotationStage
ConsensusReviewStage
FinalStage
(Complete and Archive)
The following actions are supported for Consensus review and Review and Refine:
assign
: Assigns a user to one or more tasks.
approve
: Moves a task to the next stage.
release
: Releases a task from the current user.
reject
: Rejects a task.
Get all Subtasks
from encord import EncordUserClient, Project
from encord.workflow import(
AnnotationStage,
ReviewStage,
ConsensusAnnotationStage,
ConsensusReviewStage,
FinalStage
)
SSH_PATH = "/Users/chris-encord/sdk-ssh-private-key.txt"
PROJECT_HASH = "<project-unique-id>"
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
ssh_private_key_path=SSH_PATH
)
project: Project = user_client.get_project(PROJECT_HASH)
stage = project.workflow.get_stage(name="Consensus 1", type_=ConsensusAnnotationStage)
assert isinstance(stage, ConsensusAnnotationStage)
for task in stage.get_tasks():
print(f"Task: {task}")
for subtask in task.subtasks:
print(f" subtask: {subtask}")
Select your labels from a Consensus Project
The following code includes produce_result
, that is undefined. You need to create and define the custom logic that selects the labels from the label options you have available.
from encord import EncordUserClient, Project
from encord.workflow import(
AnnotationStage,
ReviewStage,
ConsensusAnnotationStage,
ConsensusReviewStage,
FinalStage
)
SSH_PATH = "/Users/chris-encord/sdk-ssh-private-key.txt"
PROJECT_HASH = "<project-unique-id>"
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
ssh_private_key_path=SSH_PATH
)
project: Project = user_client.get_project(PROJECT_HASH)
stage = project.workflow.get_stage(name="Consensus 1 Review", type_=ConsensusReviewStage)
for task in stage.get_tasks():
label_hashes = [option.label_hash for option in task.options]
label_rows_for_consensus = project.list_label_rows_v2(label_hashes=label_hashes)
for label_row in label_rows_for_consensus:
label_row.initialise_labels()
target_label_row = project.list_label_rows_v2(data_hashes=[task.data_hash])[0]
target_label_row.initialise_labels()
if is_consensus := produce_result(label_rows_for_consensus, target_label_row):
target_label_row.save()
task.assign("<your-admin@example.com>")
task.approve()
else:
task.reject()
End-to-End Example moving tasks through Projects
This example uses a basic non-Consensus Project Workflow moving a number of tasks from ANNOTATE to COMPLETE.
!End-to-end Example Workflow
!End-to-end Example tasks
- Project ID: 01bd7084-b04a-4cd1-87f3-73e8e78925c4
- Data units: apples_01.jpg, apples_02.jpg, apples_03.jpg, apples_04.jpg, apples_05.jpg
Step 1: View all tasks in the Project
A few tasks have been annotated, but they have not been submitted yet. The following code lists all tasks in the ANNOTATE 1 stage.
From the returned results we can see that four of the data units have been assigned (apples_01 to apples_04) while one task has not (apples_05). We do not know the extent that each of the assigned data units has been annotated, but we want the tasks to move through the Workflow.
Step 2: Submit all tasks for review
The following code submits ALL tasks to the REVIEW 1 stage. Before the tasks can be submitted they are first assigned to a user in the Project. The user could be any user capable of annotating in the Project (because this is the ANNOTATE stage). For this example, the Admin of the Project (the user with SDK access) is assigned.
Submit all tasks for review
from encord import EncordUserClient, Project
from encord.workflow import(
AnnotationStage,
ReviewStage,
ConsensusAnnotationStage,
ConsensusReviewStage,
FinalStage
)
SSH_PATH = "/Users/chris-encord/sdk-ssh-private-key.txt"
PROJECT_HASH = "01bd7084-b04a-4cd1-87f3-73e8e78925c4"
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
ssh_private_key_path=SSH_PATH
)
project: Project = user_client.get_project(PROJECT_HASH)
stage = project.workflow.get_stage(name="Annotate 1", type_=AnnotationStage)
assert isinstance(stage, AnnotationStage)
for task in stage.get_tasks():
task.assign("chris@acme.com")
task.submit()
print(f"Task: {task}")
Step 3: Verify that the tasks are in REVIEW 1
Each time we perform an action, we should verify the action is successful. After submitting the tasks, we can verify that the tasks are now in REVIEW 1.
All of the tasks are now in stage REVIEW 1. None of the tasks are assigned to any users.
Step 4: Reject a task
We know that apples-05.jpg
does not have any labels, because it was not assigned to any users before we submitted the task. If we REJECT the task, according to the Workflow in the Project, the task returns to the ANNOTATE 1 stage.
The following code filters the task in the REVIEW 1 stage based on the task’s data hash, and then rejects the task.
From what was returned, we can see that only data unit apples_05.jpg
was acted upon.
To verify that the task is now in ANNOTATE 1, run the following:
Step 5: Approve tasks
There are still four tasks in REVIEW 1. The following code approves the tasks. This moves the tasks to the COMPLETE stage. You could approve all the tasks at once the REVIEW 1 stage, but instead we will filter by data hash and then approve.
Step 6: Verify tasks are COMPLETE
Use the following code to verify that the tasks apples01
to apples_04
are in the COMPLETE stage.
And from what the code returns, the tasks are now in the COMPLETE stage.