Task Agents

TLDR;

Do you already know what you are doing and only want to look over a Jupyter Notebook example? We provide one here.

Task Agents are Workflow components in which a custom operation on all tasks in the Agent stage can be triggered. This allows you to setup pre-labeling, such as using foundation models such as GPT-4o, automated quality assurance, or any other custom action you need for your workflow.

Tasks in the Agent stage are processed in the order of their task priority.

Adding Task Agents to your Workflow

To add Task Agents to your Workflow you must add an Agent stage component to your Workflow.

Connect the Agent stage in the required place in your Workflow.
Click + next to Add pathway and give the pathway a meaningful name. Repeat this step for each additional pathway you want to add. Pathways allow you to connect the Agent stage to other workflow components.

Coming Soon! We’re currently evaluating webhooks for the Agent stage. Please let us know if this would help your use-case by reaching out to support@encord.com.

Configuring Task Agents

Use the Encord SDK to configure your Task Agent. The Task Agent executes the configured SDK script for all tasks that are routed through the Task Agent stage in your Workflow.

The General Example script shows how to configure a Task Agent with the name Agent 1 and with a pathway called continue to Review.

Agent nodes can be used in conjunction with bundles to efficiently handle bulk actions

Ensure that you:

Replace <private_key_path> with the path to your private key.
Replace <project_hash> with the hash of your Project.
Insert your custom logic where the comment instructs you to do so.

General Example

# Import dependencies
from encord.user_client import EncordUserClient
from encord.workflow import AgentStage

# Authenticate using the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
ssh_private_key_path="<private_key_path>"
)

# Specify the Project that contains the Task agent. Replace <project_hash> with the hash of your Project
project = user_client.get_project(<project_hash>)

# Specify the Task Agent
agent_stage = project.workflow.get_stage(name="Agent 1", type_=AgentStage)
for task in agent_stage.get_tasks():

    # Now you have the agent task containing the data hash
    # Insert you custom logic here

# When the custom logic is completed, the task can be moved forward to the selected pathway
task.proceed(pathway_name="continue to Review")

To set up a pre-labeling stage using models:

Train a model on the objects you want pre-labeled.
Set up a Workflow that includes a prelabeling Agent stage.

Create a new Project with the Workflow from Step 2, and attach a Dataset containing the images you want to pre-label.
Add the following script to the pre-labeling Agent stage. The script makes the following assumptions:
- You want to apply polygon labels.
- You only have one model and only one training iteration for that model.

Agent nodes can be used in conjunction with bundles to efficiently handle bulk actions

In the following script:

Replace <private_key_path> with the hash of your private key.
Replace <project_hash> with the hash of your Project.
Replace Agent 1 with the name of your Agent stage.

Pre-Labeling using models

# Import dependencies
from encord.user_client import EncordUserClient
from encord.workflow import AgentStage

# Authenticate using the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Specify the Project that contains the Task agent. Replace <project_hash> with the hash of your Project
project = user_client.get_project(<project_hash>)
model = project.list_models[0]

# Specify the Task Agent
prelabeling_stage = project.workflow.get_stage(name="Agent 1", type_=AgentStage)
for task in prelabeling_stage.get_tasks():
    print(f"{task.data_hash} -> {task.data_title}")

    # Getting a label row for the data unit
    lr = project.list_label_rows_v2(data_hashes=[task.data_hash])[0]
    lr.initialise_labels()

    # Infer the first training iteration of your model
    inference_results = project.model_inference(
        uid=str(
            model.model_iteration_uids[0]   # Take only the first result
        ), 
        data_hashes=[str(task.data_hash)],  # Data hashes
        detection_frame_range=[0, 0],
    )

    ontology_structure = project.ontology_structure
    for inference_result in inference_results:
        for frame, predictions in inference_result.get("predictions").items():
            for pred in predictions.get("objects"):
                ontology_object = ontology_structure.get_child_by_title(pred.get("name"))
                object_instance = ontology_object.create_instance()
                object_instance.set_for_frames(
                    coordinates=PolygonCoordinates.from_dict(pred),
                    frames=int(frame),
                    manual_annotation=False,
                    confidence=pred.get("confidence"),
                    # Add other attributes as needed
                )
                lr.add_object_instance(object_instance)
    lr.save()
    task.proceed(pathway_name="Pre-Labeled")

Triggering the Task Agent

Webhooks are coming soon, allowing you to trigger Task Agents manually.

Task Agents aggregate all tasks that reach the Agent stage in the workflow. Your custom script must be triggered at this stage before the tasks proceed further in the workflow.

End-to-End Agent Example

We provide an end-to-end example using a Jupyter Notebook here.

Get started

General

Index

Annotate

Active

Other

TLDR;

Adding Task Agents to your Workflow

Configuring Task Agents

Triggering the Task Agent

End-to-End Agent Example

Get started

General

Index

Annotate

Active

Other

​TLDR;

​Adding Task Agents to your Workflow

​Configuring Task Agents

​Triggering the Task Agent

​End-to-End Agent Example

TLDR;

Adding Task Agents to your Workflow

Configuring Task Agents

Triggering the Task Agent

End-to-End Agent Example