# Dependencies & Specs Source: https://docs.encord.com/agents-documentation/Basics/Dependencies ## Dependencies When writing agents, you often rely on common resources, whether data for running your agent or for recording the output. Encord Agents use **dependency injection** to declaratively acquire these resources, so you can focus on developing your agent instead of writing boilerplate to set them up manually. ### What is Dependency Injection? Dependency injection means your code (in this case, your path operation functions) can declare what it needs to function — its "dependencies." The system (e.g., FastAPI or encord-agents) then takes care of providing your code with those dependencies ("injecting" them). We follow the same pattern as [FastAPI Dependencies](https://fastapi.tiangolo.com/tutorial/dependencies/), where you can find more examples. ### Injecting Dependencies When defining your agents, you can inject essential dependencies, such as the path to the underlying asset or frame iterators. You can also add custom dependencies if needed. To inject dependencies, type-annotate your agent function variables using the `Depends` class. In the example below, replace `{module}` with the type of agent you're building: ```python theme={"dark"} from typing_extensions import Annotated from encord.core.dependencies import Depends # or from fastapi import Depends # if you are building a fastapi app from encord.{module}.dependencies import dep_single_frame def my_agent(frame: Annotated[np.ndarray, Depends(dep_single_frame)]): # the frame becomes available here. ``` See the [references section](/agents-documentation/Reference/Custom-Agents/Agents-Reference-Custom-Agents#dependencies-2) for available dependencies. ### Custom Dependencies To add a custom dependency: ```python theme={"dark"} def my_custom_dependency(label_row: LabelRowV2) -> dict: # 1. Define a function to load the dependency # e.g., look up additional data in own db return db.query("whatever") @runner.stage(stage="") def by_custom_data( # 2. Use that function as a dependency custom_data: Annotated[dict, Depends(my_custom_dependency)] ) -> str: # `custom_data` is automatically injected here. # ... do your thing # then, return name of task pathway. ``` Custom dependency functions can themselves rely on other dependencies, enabling more complex resource acquisition — see `dep_video_iterator` for an example of this. *** ## Custom Agents Specification This section defines the interface for Custom Agents. Use it to define agents via the library or when writing your own implementation. ### Schema ```typescript theme={"dark"} type AgentPayload = { projectHash: string; dataHash: string; frame: number; objectHashes?: string[]; }; ``` This schema aligns with the **FrameData** structure. The `objectHashes` field is optional — when present, it's a list of strings. ### Test Payload When you register your Custom Agent in the platform's **Custom Agents** section, you can test it with a test payload. * If you modify the test payload, the platform verifies that your agent has access to the associated project and data. * If you leave it unmodified, the platform instead sends a distinguished `X-Encord-Agent` header, which automatically triggers an appropriate response. This lets you confirm that your agent is deployed correctly, that your session can see the agent (all requests to your agent originate from your browser session, not the Encord backend), and that it works on specific Projects. ### Webhook Security Encord provides webhook signing secrets to verify the authenticity of requests sent to your custom agent endpoints. Each custom agent endpoint receives a unique signing secret you can use to validate incoming webhook payloads. **To access your signing secret**: 1. Navigate to your Workflow configuration in Encord. 2. Select your custom agent node. 3. Locate the **Signing secret** section. 4. Use the eye icon to reveal the secret, or the copy icon to copy it to your clipboard. The signing secret is automatically generated when you configure your custom agent endpoint URL. Save your workflow after changing the URL to update the secret. *To verify webhook signatures*: Use the signing secret to verify that webhook requests to your custom agent endpoint originate from Encord. This helps protect your endpoint from unauthorized requests and ensures data integrity. For implementation guidance, see the [webhook security documentation](/platform-documentation/Annotate/annotate-webhooks-notifications#verifying-webhook-signatures). ### Response The platform displays your agent's output using the **AgentResponse** type, making your agents more interactive and informative. For example, if the label state doesn't meet the Custom Agent's expectations, or if the agent's function is to check the validity of current labels, the response type lets you communicate that information to the annotator. ### Error Handling Raise an `EncordAgentException` to handle errors appropriately. For example, if your agent expects polygons but receives a skeleton, use it to return an informative error to the Encord platform. If an **authorization issue** occurs with the Encord platform (for example, a request tries to access a project the agent doesn't have access to), the response body includes the authorization message in an `AgentErrorResponse` type: ```typescript theme={"dark"} type EditorAgentErrorResponse = { message?: string; } ``` The platform displays this message so your agent can be used intuitively. # AWS Lambda Functions Source: https://docs.encord.com/agents-documentation/Custom-Agents/AWS-Lambda Before you start, ensure that you can [authenticate](/agents-documentation/Basics/Authentication) with Encord and your [AWS CLI is authenticated](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-authentication.html). The following example shows the general structure of how to build an AWS Lambda function. For concrete implementations of agents with specific abilities, see the [examples section](/agents-documentation/Custom-Agents/Examples). There are two different ways in which you can create lambda functions. 1. [Using a zip upload](#building-a-lambda-function-using-zip-upload) 2. [Using a Docker container](#building-a-lambda-function-with-docker) They have different applications and properties. Before you dive into one of the examples, please consider the properties in the table below. | Type | File size limits | Ease of use | | :--------------------------------------------------------- | :-------------------: | :---------: | | [Zip upload](#building-a-lambda-function-using-zip-upload) | 50MB (250MB unzipped) | `easier` | | [Docker](#building-a-lambda-function-with-docker) | NA | `harder` | AWS Lambda zip deployments have a 50MB (250MB uncompressed) size limit. This often forces the use of Docker for agents with larger dependencies, such as computer vision libraries (`pip install encord-agents[vision]`). Additionally, dependencies relying on C/C++ code require specific CPU architecture installations, often making testing only possible after deployment. More detailed AWS Documentation on python lambda functions can be found [here](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html). ## Building a lambda function using zip upload The full AWS lambda documentation for zip files is available [here](https://docs.aws.amazon.com/lambda/latest/dg/python-package.html). ### STEP 1: Create a local Project To begin, set up your local project structure. This includes creating two directories, a virtual environment (for example `venv`), and a package directory for CPU-architecture-specific dependencies tailored to your cloud infrastructure. 1. Create your virtual environment and install `encord-agents`. Refer to the [installation docs](/agents-documentation/Basics/Installation) for detailed instructions. ```shell theme={"dark"} mkdir my_project cd my_project python -m venv venv source venv/bin/activate pip install encord-agents deactivate ``` 2. Create the "mirror" package directory for the upload. We recommend explicitly installing `boto3`, the AWS SDK, even though it's typically present in Lambda environments. This ensures consistent dependency versions and avoids unexpected changes when AWS updates its infrastructure. ```shell theme={"dark"} mkdir package pip install \ --platform manylinux2014_x86_64 \ --target=package \ --implementation cp \ --python-version 3.12 \ --only-binary=:all: --upgrade \ encord-agents boto3 ``` ### STEP 2: Define the agent 1. Create a `lambda_function.py` file using the following template: ```python title="lambda_function.py" theme={"dark"} from encord.objects.ontology_labels_impl import LabelRowV2 from encord_agents.core.data_model import FrameData from encord_agents.aws import editor_agent @editor_agent() def lambda_handler(frame_data: FrameData, label_row: LabelRowV2) -> None: print(frame_data.model_dump_json()) # label_row.save() ``` Make sure to name your file `lambda_function.py` and your function `lambda_handler` if you are following this example, as these names are referenced in the `--handler` argument in [Step 6](#step-6-upload-the-zip). 2. Complete the `lambda_handler` function with the logic you want to execute when the agent is triggered. For more Custom Agent examples, see the [examples section](/agents-documentation/Custom-Agents/Examples). You can inject multiple different [dependencies](/agents-documentation/Reference/Custom-Agents/Agents-Reference-Custom-Agents#Dependencies) into the function if necessary. ### STEP 3: (Optional) Test the Agent Locally We are testing against the `venv` and not the `package` that is uploaded to AWS, so we might see slight differences in the outcome. 1. In order to test your agent locally, you can add an `if __name__ == "__main__"` declaration in the end of the file as follows: ```python title="lambda_function.py" theme={"dark"} # ... imports from before @editor_agent() def lambda_handler(frame_data: FrameData): print(frame_data.model_dump_json()) if __name__ == '__main__': event = { "body": { "projectHash": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", "dataHash": "00000000-1111-2222-3333-444444444444", "frame": 0, "objectHashes": [] } } lambda_handler(event, None) ``` 2. Make sure to update the Project and data hash to correspond to one of your data units. If you open the label editor in your browser and look at the url, it has this pattern: ``` https://app.encord.com/label_editor/{projectHash}/{dataHash} ``` 3. Run the file: ```shell theme={"dark"} source venv/bin/activate export ENCORD_SSH_KEY_FILE='/path/to/your/private-key-file' python lambda_function.py ``` ### STEP 4: Create the ZIP file 1. Create the zip file by first zipping what is in the `package` directory: ```shell theme={"dark"} cd package zip -r ../package.zip . cd .. ``` 2. Add your lambda function to the zip file: ```shell theme={"dark"} zip package.zip lambda_function.py ``` ### STEP 5: Set up an execution role Your Lambda function requires an execution role to define the permissions it needs to run. 1. Create the configuration file `trust-policy.json` to be uploaded to AWS. ```json title="trust-policy.json" theme={"dark"} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` 2. Create a role with the trust policy: ```shell theme={"dark"} aws iam create-role \ --role-name lambda-execute-encord-agents \ --assume-role-policy-document file://trust-policy.json ``` It should output a JSON similar to the following. Make sure to take a note of the `Arn`. ```json theme={"dark"} { "Role": { "Path": "/", "RoleName": "lambda-execute-encord-agents", "RoleId": "AROAQ7BEARV2DSFT2E3PI", "Arn": "arn:aws:iam::061234567890:role/lambda-execute-encord-agents", "CreateDate": "2025-05-09T10:39:35+00:00", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } } } ``` 4. Attach the basic lambda execution role to the role. ```shell theme={"dark"} aws iam attach-role-policy \ --role-name lambda-execute-encord-agents \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole ``` ### STEP 6: Upload the zip 1. To upload the zip file to AWS, use the following command. Make sure to insert the `` you want to use and the proper role `Arn` from STEP 5. ```shell theme={"dark"} aws lambda create-function --function-name \ --runtime python3.12 \ --handler lambda_function.lambda_handler \ --role arn:aws:iam::061234567890:role/lambda-execute-encord-agents \ --zip-file fileb://package.zip ``` If you want to modify your lambda, follow the steps above again but call the update method rather than the `create-function` from the AWS CLI. ```shell theme={"dark"} aws lambda update-function-code \ --function-name \ --zip-file fileb://package.zip ``` 2. Proceed to [Step A](#step-a-configure-public-endpoint) below to complete the setup. ## Building a Lambda Function with Docker The full AWS documentation for building docker images for lambda functions with Python is available [here](https://docs.aws.amazon.com/lambda/latest/dg/python-image.html). ### STEP 1: Create a Local Project Start by creating a local project directory. ```shell theme={"dark"} mkdir my_project cd my_project ``` Then add a `requirements.txt` file. ```text title="requirements.txt" theme={"dark"} boto3 encord-agents ``` #### (Optional) local environment If you want, you can create a local environment for testing before building the docker image. ``` python -m venv venv source venv/bin/activate pip install -r requirements.txt ``` ### STEP 2: Define the Agent 1. Create a `lambda_function.py` file using the following template: ```python title="lambda_function.py" theme={"dark"} from encord.objects.ontology_labels_impl import LabelRowV2 from encord_agents.core.data_model import FrameData from encord_agents.aws import editor_agent @editor_agent() def lambda_handler(frame_data: FrameData, label_row: LabelRowV2) -> None: print(frame_data.model_dump_json()) # label_row.save() ``` Make sure to name your file `lambda_function.py` and your function `lambda_handler` if you're following this example, as these names are referenced in the Docker image in [the next step](#step-3-build-the-docker-image). Complete the `lambda_handler` function with the logic you want to execute when the agent is triggered. For more Custom Agent examples, see the [examples section](/agents-documentation/Custom-Agents/Examples). You can inject multiple different [dependencies](/agents-documentation/Reference/Custom-Agents/Agents-Reference-Custom-Agents#Dependencies) into the function if necessary. You can find multiple examples of what can be done with Custom Agents [here](/agents-documentation/Custom-Agents/Examples). ### STEP 3: Build the Docker Image 1. Create a `Dockerfile` with the following content. ```Dockerfile title="Dockerfile" theme={"dark"} FROM public.ecr.aws/lambda/python:3.12 # Install the specified packages COPY requirements.txt ${LAMBDA_TASK_ROOT} RUN pip install -r requirements.txt # Copy function code COPY lambda_function.py ${LAMBDA_TASK_ROOT} # Set the CMD to your handler # (could also be done as a parameter override outside of the Dockerfile) CMD [ "lambda_function.lambda_handler" ] ``` 2. Build the image locally. You need to have `docker` and `buildx` installed. ```shell theme={"dark"} docker buildx build \ --platform linux/amd64 \ --provenance=false \ -t encord-agents-my-agent-name:latest \ . ``` ### STEP 4: (Optional) Test the agent locally To test the agent locally, you can spin up the container with the following command. ```shell theme={"dark"} docker run \ -e ENCORD_SSH_KEY="$(cat /path/to/your/private-key-file)" \ -p 9000:8080 \ --platform linux/amd64 \ -t encord-agents-my-agent-name:latest ``` The respective lines ensures that we * Add the `ENCORD_SSH_KEY` env variable * Map the port 8080 to your own local port (9000 in this example) * AWS architecture needs to be `amd64` (or `arm64` [see aws docs](https://docs.aws.amazon.com/lambda/latest/dg/python-image.html#python-image-clients)) * Run the latest built image (`encord-agents-example:latest` in this example) You can "hit" the function endpoint with a `curl` command. The `functions/function/invocation` path is required by Amazon. ```shell theme={"dark"} curl "http://localhost:9000/2015-03-31/functions/function/invocations" \ -X "POST" \ -d '{ "body": { "projectHash": "aaaaaaaa-ed79-4a1b-a0c1-b1b38ffb523c", "dataHash":"659c3a38-737f-4f56-a709-26ce01f5fd0b", "frame": 0, "objectHashes": [] } }' ``` There is a long-standing issue with Lambda Docker containers where the API differs between local and cloud execution. In this case, two relevant distinctions exist. * `Content-Type`: Locally, the container expects a post request with `Content-Type: application/x-www-form-urlencoded` while the public url hosted by AWS expects `Content-Type: application/application-json`. * `POST` data: Locally, the post data needs to be `{"body": {... content dict}}` while the public url expects `{... content dict}`. As a consequence, the equivalent `curl` request for a publicly hosted lambda function would be: ``` curl "https://.lambda-url.eu-west-1.on.aws/" \ -X "POST" \ -H "Content-Type: application/json" \ -d '{ "projectHash": "aaaaaaaa-ed79-4a1b-a0c1-b1b38ffb523c", "dataHash":"659c3a38-737f-4f56-a709-26ce01f5fd0b", "frame": 0, "objectHashes": [] }' ``` See [the Function URL section](#step-a-configure-public-endpoint) for more information on how to get the public endpoint. ### Step 5: Prepare ECR Container Repository To associate a Lambda function with your container, the container must first be uploaded to the AWS Elastic Container Registry (ECR). For this, a container repository is required. 1. Run the `aws get-login-password` command to authenticate the Docker CLI to your Amazon ECR registry. * Set the `--region` value to the AWS Region where you want to create the Amazon ECR repository (we use `eu-west-1` in this example). * Replace 111122223333 with your AWS account ID. ```shell theme={"dark"} aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 111122223333.dkr.ecr.eu-west-1.amazonaws.com ``` 2. Create a repository in Amazon ECR using the create-repository command. ```shell theme={"dark"} aws ecr create-repository \ --repository-name \ --region us-east-1 \ --image-scanning-configuration scanOnPush=true \ --image-tag-mutability MUTABLE ``` If successful, you see a response with the following format: ``` { "repository": { "repositoryArn": "arn:aws:ecr:us-east-1:111122223333:repository/", "registryId": "111122223333", "repositoryName": "", "repositoryUri": "111122223333.dkr.ecr.us-east-1.amazonaws.com/", "createdAt": "2025-05-12T10:39:01+00:00", "imageTagMutability": "MUTABLE", "imageScanningConfiguration": { "scanOnPush": true }, "encryptionConfiguration": { "encryptionType": "AES256" } } } ``` Copy the `repositoryUri` from the output. ### Step 6: Upload the Local Docker Image 1. To tag your local Docker image for Amazon ECR as the latest version, use the `docker tag` command: * `encord-agents-my-agent-name:latest` is the name and tag of your local Docker image. This is the image name and tag that you specified in the `docker build` command [above](#step-3-build-the-docker-image). * Replace `` with the `repositoryUri` that you copied [above](#step-5-prepare-ecr-container-repository). Make sure to include `:latest` at the end of the URI. ```shell theme={"dark"} docker tag encord-agents-my-agent-name:latest :latest ``` 2. Run the `docker push` command to deploy your local image to the Amazon ECR repository. Ensure you include `:latest` at the end of the repository URI. ```shell theme={"dark"} docker push 111122223333.dkr.ecr.us-east-1.amazonaws.com/hello-world:latest ``` ### STEP 7: Set up an Execution Role Your Lambda function requires an execution role to define the permissions it needs to run. 1. Create the configuration file `trust-policy.json` to be uploaded to AWS. ```json title="trust-policy.json" theme={"dark"} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` 2. Create a role with the trust policy. ```shell theme={"dark"} aws iam create-role \ --role-name lambda-execute-encord-agents \ --assume-role-policy-document file://trust-policy.json ``` It should output a JSON similar to the following. Make sure to take a note of the `Arn`. ```json theme={"dark"} { "Role": { "Path": "/", "RoleName": "lambda-execute-encord-agents", "RoleId": "AROAQ7BEARV2DSFT2E3PI", "Arn": "arn:aws:iam::061234567890:role/lambda-execute-encord-agents", "CreateDate": "2025-05-09T10:39:35+00:00", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } } } ``` 3. Attach the basic lambda execution role to the role. ```shell theme={"dark"} aws iam attach-role-policy \ --role-name lambda-execute-encord-agents \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole ``` ### STEP 8: Creating a Lambda Function For `ImageUri`, specify the repository URI from [Step 5](#step-5-prepare-ecr-container-repository). Ensure you include `:latest` at the end of the URI. ```shell theme={"dark"} aws lambda create-function \ --function-name \ --package-type Image \ --code ImageUri=111122223333.dkr.ecr.us-east-1.amazonaws.com/:latest \ --role arn:aws:iam::061234567890:role/lambda-execute-encord-agents ``` To update the container image, rebuild it (as in [Step 3](#step-3-build-the-docker-image)), then tag it with the repository URI and upload it again (as in [Step 6](#step-6-upload-the-local-docker-image)). Instead of the `create-function` call in the end, you do the following. ```shell theme={"dark"} aws lambda update-function-code \ --function-name pdf-qa \ --image-uri 111122223333.dkr.ecr.us-east-1.amazonaws.com/:latest \ --publish ``` The URL stays the same, so there is no need to reconfigure CORS or anything else. Now, proceed to [Step A](#step-a-configure-public-endpoint) below to complete the setup. ## Communication with Encord ### STEP A: Configure Public Endpoint Now the function is live but not publicly available to the internet. To make it accessible to the Encord platform, follow these steps: To make your live Lambda function publicly accessible to the Encord platform, follow these steps: 1. Go to [https://console.aws.amazon.com/lambda](https://console.aws.amazon.com/lambda) and navigate to your newly created function. 2. Go to the **Configuration** tab and choose the **Function URL** section. 3. Click the **Create function URL** button. 4. Choose `NONE` for **Auth type**. 5. Expand the **Additional settings** panel and check the **Configure cross-origin resource sharing (CORS)** box. 6. In the **Allow origin** section, add `https://app.encord.com`, `https://app.us.encord.com`, or your custom (VPC) domain. 7. Under the **Allow headers** section, add both `content-type` and `x-encord-editor-agent` values. 8. In the **Allow methods** section, choose `POST`. 9. Click **Save**. The configuration should look similar to the following image (potentially with fewer allowed origins). You can click the image to expand it. ![AWS Function url configuration](https://storage.googleapis.com/docs-media.encord.com/assets/aws-function-url-configuration.png) ### STEP B: Adding Secrets Your agent requires an access key secret. In the AWS console where you configured the function URL, click the **Environment variables** tab on the left. Add the `ENCORD_SSH_KEY` variable, along with any other necessary credentials (for example, HuggingFace, OpenAI, or Gemini). We recommend creating an Encord service account and using an associated access key. This ensures you provide only the necessary access permissions. ### STEP C: Associating the URL with Encord Now that your AWS setup is complete, copy the Function URL displayed at the top of your AWS Lambda function's web page. Then, navigate to the Encord app. 1. Click the **Agents** section on the left. The **Agents catalog** tab opens. 2. Click **Create custom agent** on the **Custom agent** tile.
3. Give the agent a meaningful name and description. 4. Paste the function URL into the **Agent endpoint** field. 5. Click **Create custom agent**. # Custom Domain Source: https://docs.encord.com/agents-documentation/Custom-Agents/Custom-Domain *** ## Custom Agent Configuration for US Domain or VPC This section guides users operating Encord with the US domain (`https://api.us.encord.com`) or within their private Virtual Private Cloud (VPC) on how to configure their Custom Agents. *** ### STEP 1: Set the `ENCORD_DOMAIN` Environment Variable Define the **Encord API domain**, not the front-end application domain. * **Running Locally:** ```shell theme={"dark"} export ENCORD_DOMAIN=https://api.us.encord.com ``` * **Deploying with a GCP Cloud Function:** Create a GCP secret and pass in the domain: ```shell theme={"dark"} --set-secrets="ENCORD_SSH_KEY=YOUR_GCP_STORED_KEY:latest,ENCORD_DOMAIN=YOUR_GCP_STORED_DOMAIN" ``` *** ### STEP 2: Set the `ENCORD_SSH_KEY` or `ENCORD_SSH_KEY_FILE` Environment Variables Provide your access key for authentication. * **Running Locally:** ```shell theme={"dark"} export ENCORD_SSH_KEY_FILE="path/to/your/key" ``` or ```shell theme={"dark"} export ENCORD_SSH_KEY="" ``` * **Deploying with a GCP Cloud Function:** Create a GCP secret and pass in the key: ```shell theme={"dark"} --set-secrets="ENCORD_SSH_KEY=YOUR_GCP_STORED_KEY:latest,ENCORD_DOMAIN=YOUR_GCP_STORED_DOMAIN" ``` *** ### STEP 3: Pass the Front-end Domain to the Custom Agent Declaration Pass the **Encord front-end domain** into the `@editor_agent` declaration to ensure the function respects Cross-Origin Resource Sharing (CORS): ```python theme={"dark"} @editor_agent(custom_cors_regex="https://app.us.encord.com") # Or the domain of your custom FE ``` # FastAPI Examples Source: https://docs.encord.com/agents-documentation/Custom-Agents/Examples/FastAPI-Examples ## Basic Geometric Example A simple example showing how to use objectHashes. ```python agent.py theme={"dark"} from typing import Annotated from encord.objects.ontology_labels_impl import LabelRowV2 from encord.objects.ontology_object_instance import ObjectInstance from fastapi import Depends, FastAPI from encord_agents.fastapi.cors import get_encord_app() from encord_agents.fastapi.dependencies import ( FrameData, dep_label_row, dep_objects, ) # Initialize FastAPI app app = get_encord_app() @app.post("/handle-object-hashes") def handle_object_hashes( frame_data: FrameData, lr: Annotated[LabelRowV2, Depends(dep_label_row)], object_instances: Annotated[list[ObjectInstance], Depends(dep_objects)], ) -> None: for object_inst in object_instances: print(object_inst) ``` **Use Case: Selective OCR on Selected Objects** This functionality allows you to apply your own OCR model to specific objects selected directly within the Encord platform. When you trigger your agent from the Encord app after selecting objects, the platform automatically sends a list of `objectHashes` to your agent. Your agent can then use the `dep_objects` method to gain immediate access to these specific object instances, which greatly simplifies integrating your OCR model for targeted processing. **Test the Agent** 1. Save the above code as `agent.py`. 2. Run the following command to run the agent in debug mode in your terminal. ```shell theme={"dark"} uvicorn main:app --reload --port 8080 ``` 3. Open your Project in the Encord platform and navigate to a frame with an object that you want to act on. Choose an object from the bottom left sider and click `Copy URL` as shown:
Copy URL from left sider
The url should have roughly this format: `"https://app.encord.com/label_editor/{project_hash}/{data_hash}/{frame}/0?other_query_params&objectHash={objectHash}"`. 4. In another shell operating from the same working directory, source your virtual environment and test the agent. ```shell theme={"dark"} source venv/bin/activate encord-agents test local agent '' ``` 5. To see if the test is successful, refresh your browser to see the action taken by the Agent. If the test has run successfully, the agent can be deployed. Visit [the deployment documentation](/agents-documentation/Custom-Agents/GCP-Cloud-Functions#step-4-deployment) to learn more. ## Nested Classification using Claude 3.5 Sonnet The goals of this example is to: 1. Create a Custom Agent that can automatically fill in frame-level classifications in the Label Editor. 2. Demonstrate how to use the [`OntologyDataModel`](/agents-documentation/Reference/Core/Agents-Reference-Core#ontologydatamodel-objects) for classifications. 3. Demonstrate how to build an agent using FastAPI that can be self-hosted. **Prerequisites** Before you begin, ensure you have: * Created a virtual Python environment. * Installed all necessary dependencies. * Have an [Anthropic API key](https://www.anthropic.com/api). * Are able to [authenticate with Encord](/agents-documentation/Basics/Authentication). Run the following commands to set up your environment: ```shell theme={"dark"} python -m venv venv # Create a virtual Python environment source venv/bin/activate # Activate the virtual environment python -m pip install "fastapi[standard]" encord-agents anthropic # Install required dependencies export ANTHROPIC_API_KEY="" # Set your Anthropic API key export ENCORD_SSH_KEY_FILE="/path/to/your/private/key" # Define your Encord access key ``` **Project Setup** Create a Project with visual content (images, image groups, image sequences, or videos) in Encord. This example uses the following Ontology, but any Ontology containing classifications can be used.
```json Ontology JSON theme={"dark"} { "objects": [], "classifications": [ { "id": "1", "featureNodeHash": "TTkHMtuD", "attributes": [ { "id": "1.1", "featureNodeHash": "+1g9I9Sg", "type": "text", "name": "scene summary", "required": false, "dynamic": false } ] }, { "id": "2", "featureNodeHash": "xGV/wCD0", "attributes": [ { "id": "2.1", "featureNodeHash": "k3EVexk7", "type": "radio", "name": "is there a person in the frame?", "required": false, "options": [ { "id": "2.1.1", "featureNodeHash": "EkGwhcO4", "label": "yes", "value": "yes", "options": [ { "id": "2.1.1.1", "featureNodeHash": "mj9QCDY4", "type": "text", "name": "What is the person doing?", "required": false } ] }, { "id": "2.1.2", "featureNodeHash": "37rMLC/v", "label": "no", "value": "no", "options": [] } ], "dynamic": false } ] } ] } ``` To construct the same Ontology as used in this example, run the following script. ```python Create Ontology theme={"dark"} import json from encord.objects.ontology_structure import OntologyStructure from encord_agents.core.utils import get_user_client encord_client = get_user_client() structure = OntologyStructure.from_dict(json.loads("{the_json_above}")) ontology = encord_client.create_ontology( title="Your ontology title", structure=structure ) print(ontology.ontology_hash) ``` The aim is to trigger an agent that transforms a labeling task from Figure A to Figure B. Figure A: No classification labels. Figure B: Multiple nested classification labels generated by an LLM. **Create the Agent** This section provides the complete code for creating your Custom Agent, along with an explanation of its internal workings. **Agent Setup Steps** 1. Import dependencies, authenticate with Encord, and set up the Project. Ensure you insert your Project's unique identifier. 2. Create a data model and a system prompt based on the Project Ontology to tell Claude how to structure its response. 3. Set up an Anthropic API client to establish communication with the Claude model. 4. Define the Custom Agent. This includes: * Receiving frame data using FastAPI's Form dependency. * Retrieving the associated label row and frame content using Encord Agents' dependencies. * Constructing a Frame object from the content. * Sending the frame image to Claude for analysis. * Parsing Claude's response into classification instances. * Adding these classifications to the label row and saving the updated data. ````python theme={"dark"} # 1. Import dependencies and set up the Project. The CORS middleware is crucial as it allows the Encord platform to make requests to your API. import os import numpy as np from anthropic import Anthropic from encord.objects.ontology_labels_impl import LabelRowV2 from fastapi import Depends from numpy.typing import NDArray from typing_extensions import Annotated from encord_agents.core.data_model import Frame from encord_agents.core.ontology import OntologyDataModel from encord_agents.core.utils import get_user_client from encord_agents.fastapi.cors import get_encord_app from encord_agents.fastapi.dependencies import ( FrameData, dep_label_row, dep_single_frame, ) # Initialize FastAPI app app = get_encord_app() # 2. Set up the Project and create a data model based on the Ontology. client = get_user_client() project = client.get_project("") data_model = OntologyDataModel(project.ontology_structure.classifications) # 3. Set up Claude and create the system prompt that tells Claude how to structure its response. system_prompt = f""" You're a helpful assistant that's supposed to help fill in json objects according to this schema: ```json {data_model.model_json_schema_str} ``` Please only respond with valid json. """ ANTHROPIC_API_KEY = os.getenv("ANTHROPIC_API_KEY") anthropic_client = Anthropic(api_key=ANTHROPIC_API_KEY) # 4. Define the Custom Agent @app.post("/frame_classification") async def classify_frame( frame_data: FrameData, lr: Annotated[LabelRowV2, Depends(dep_label_row)], content: Annotated[NDArray[np.uint8], Depends(dep_single_frame)], ): # Receives frame data using FastAPI's Form dependency. # Note: FastAPI handles parsing the incoming request body (which implicitly includes frame_data, # and the dependencies (dep_label_row, dep_single_frame) resolve the lr and content). """Classify a frame using Claude.""" # Constructs a `Frame` object with the content. frame = Frame(frame=frame_data.frame, content=content) # Sends the frame image to Claude for analysis. message = anthropic_client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, system=system_prompt, messages=[ { "role": "user", "content": [frame.b64_encoding(output_format="anthropic")], } ], ) try: # Parses Claude's response into classification instances. classifications = data_model(message.content[0].text) for clf in classifications: clf.set_for_frames(frame_data.frame, confidence=0.5, manual_annotation=False) # Adds classifications to the label row lr.add_classification_instance(clf) except Exception: import traceback traceback.print_exc() print(f"Response from model: {message.content[0].text}") # Saves the updated data. lr.save() ```` **Test the Agent** 1. In your current terminal run the following command to runFastAPI server in development mode with auto-reload enabled. ```shell theme={"dark"} uvicorn main:app --reload --port 8080 ``` 2. Open your Project in the Encord platform and navigate to a frame you want to add a classification to. Copy the URL from your browser. The url should have the following format: `"https://app.encord.com/label_editor/{project_hash}/{data_hash}/{frame}"` 3. In another shell operating from the same working directory, source your virtual environment and test the agent. ```shell theme={"dark"} source venv/bin/activate encord-agents test local frame_classification '' ``` 4. To see if the test is successful, refresh your browser to view the classifications generated by Claude. Once the test runs successfully, you are ready to deploy your agent. Visit the deployment documentation to learn more. ## Nested Attributes using Claude 3.5 Sonnet The goals of this example are: 1. Create a Custom Agent that can convert generic object annotations (class-less coordinates) into class specific annotations with nested attributes like descriptions, radio buttons, and checklists. 2. Demonstrate how to use both the [`OntologyDataModel`](/agents-documentation/Reference/Core/Agents-Reference-Core#ontologydatamodel-objects) and the [`dep_object_crops`](/agents-documentation/Reference/Custom-Agents/Agents-Reference-Custom-Agents#dep-object-crops-2) dependency. **Prerequisites** Before you begin, ensure you have: * Created a virtual Python environment. * Installed all necessary dependencies. * Have an [Anthropic API key](https://www.anthropic.com/api). * Are able to [authenticate with Encord](/agents-documentation/Basics/Authentication). Run the following commands to set up your environment: ```shell theme={"dark"} python -m venv venv # Create a virtual Python environment source venv/bin/activate # Activate the virtual environment python -m pip install encord-agents anthropic # Install required dependencies export ANTHROPIC_API_KEY="" # Set your Anthropic API key export ENCORD_SSH_KEY_FILE="/path/to/your/private/key" # Define your Encord access key ``` **Project Setup** Create a Project with visual content (images, image groups, image sequences, or videos) in Encord. This example uses the following Ontology, but any Ontology containing classifications can be used provided the object types are the same and there is one entry called "generic".
```json ontology.json theme={"dark"} { "objects": [ { "id": "1", "name": "person", "color": "#D33115", "shape": "bounding_box", "featureNodeHash": "2xlDPPAG", "required": false, "attributes": [ { "id": "1.1", "featureNodeHash": "aFCN9MMm", "type": "text", "name": "activity", "required": false, "dynamic": false } ] }, { "id": "2", "name": "animal", "color": "#E27300", "shape": "bounding_box", "featureNodeHash": "3y6JxTUX", "required": false, "attributes": [ { "id": "2.1", "featureNodeHash": "2P7LTUZA", "type": "radio", "name": "type", "required": false, "options": [ { "id": "2.1.1", "featureNodeHash": "gJvcEeLl", "label": "dolphin", "value": "dolphin", "options": [] }, { "id": "2.1.2", "featureNodeHash": "CxrftGS4", "label": "monkey", "value": "monkey", "options": [] }, { "id": "2.1.3", "featureNodeHash": "OQyWm7Sm", "label": "dog", "value": "dog", "options": [] }, { "id": "2.1.4", "featureNodeHash": "CDKmYJK/", "label": "cat", "value": "cat", "options": [] } ], "dynamic": false }, { "id": "2.2", "featureNodeHash": "5fFgrM+E", "type": "text", "name": "description", "required": false, "dynamic": false } ] }, { "id": "3", "name": "vehicle", "color": "#16406C", "shape": "bounding_box", "featureNodeHash": "llw7qdWW", "required": false, "attributes": [ { "id": "3.1", "featureNodeHash": "79mo1G7Q", "type": "text", "name": "type - short and concise", "required": false, "dynamic": false }, { "id": "3.2", "featureNodeHash": "OFrk07Ds", "type": "checklist", "name": "visible", "required": false, "options": [ { "id": "3.2.1", "featureNodeHash": "KmX/HjRT", "label": "wheels", "value": "wheels" }, { "id": "3.2.2", "featureNodeHash": "H6qbEcdj", "label": "frame", "value": "frame" }, { "id": "3.2.3", "featureNodeHash": "gZ9OucoQ", "label": "chain", "value": "chain" }, { "id": "3.2.4", "featureNodeHash": "cit3aZSz", "label": "head lights", "value": "head_lights" }, { "id": "3.2.5", "featureNodeHash": "qQ3PieJ/", "label": "tail lights", "value": "tail_lights" } ], "dynamic": false } ] }, { "id": "4", "name": "generic", "color": "#FE9200", "shape": "bounding_box", "featureNodeHash": "jootTFfQ", "required": false, "attributes": [] } ], "classifications": [] } ``` To construct the Ontology used in this example, run the following script: ```python theme={"dark"} import json from encord.objects.ontology_structure import OntologyStructure from encord_agents.core.utils import get_user_client encord_client = get_user_client() structure = OntologyStructure.from_dict(json.loads("{the_json_above}")) ontology = encord_client.create_ontology( title="Your ontology title", structure=structure ) print(ontology.ontology_hash) ``` The goal is to trigger an agent that takes a labeling task from Figure A to Figure B, below: Figure A: No classification labels. Figure B: Multiple nested classification labels generated by an LLM. **Create the Agent** This section provides the complete code for creating your Custom Agent, along with an explanation of its internal workings. **Agent Setup Steps** 1. Import Dependencies and Configure Project: Import necessary dependencies and set up your project. Remember to insert your project's unique identifier. 2. Create a data model and a system prompt based on the Project Ontology to tell Claude how to structure its response. 3. Initialize Anthropic API Client: Set up an API client to establish communication with the Claude model. 4. Define the Custom Agent: * Arguments are automatically injected when the agent is called (see dependency injection details \[suspicious link removed]). * The dep\_object\_crops dependency filters to include only "generic" object crops that still need classification. * Call Claude with Image Crops: Use the crop.b64\_encoding method to send each image crop to Claude in a format it understands. 6. Parse Claude's Response and Update Labels: The data\_model parses Claude's JSON response, creating a new Encord object instance. If successful, the original generic object is replaced with the newly classified instance on the label row. 7. Save Labels. ```python theme={"dark"} # 1. Import dependencies, authenticate with Encord, and set up the Project. import os from anthropic import Anthropic from encord.objects.ontology_labels_impl import LabelRowV2 from fastapi import Depends from typing_extensions import Annotated from encord_agents.core.data_model import InstanceCrop from encord_agents.core.ontology import OntologyDataModel from encord_agents.core.utils import get_user_client from encord_agents.fastapi.cors import get_encord_app from encord_agents.fastapi.dependencies import ( FrameData, dep_label_row, dep_object_crops, ) # Initialize FastAPI app app = get_encord_app() # User client and ontology setup client = get_user_client() # Ensure you insert your Project's unique identifier. project = client.get_project("") generic_ont_obj, *other_objects = sorted( project.ontology_structure.objects, key=lambda o: o.title.lower() == "generic", reverse=True, ) # 2. Create a data model and a system prompt based on the Project Ontology to tell Claude how to structure its response. data_model = OntologyDataModel(other_objects) system_prompt = f""" You're a helpful assistant that's supposed to help fill in json objects according to this schema: `{data_model.model_json_schema_str}` Please only respond with valid json. """ # 3. Set up an Anthropic API client to establish communication with the Claude model. ANTHROPIC_API_KEY = os.getenv("ANTHROPIC_API_KEY") anthropic_client = Anthropic(api_key=ANTHROPIC_API_KEY) # 4. Define the Custom Agent. @app.post("/object_classification") async def classify_objects( frame_data: FrameData, lr: Annotated[LabelRowV2, Depends(dep_label_row)], crops: Annotated[ list[InstanceCrop], Depends(dep_object_crops(filter_ontology_objects=[generic_ont_obj])), ], ): """Classify generic objects using Claude.""" changes = False # Iterating through each object crop. for crop in crops: # 5. Call Claude with Image Crops. # Sending each crop image to Claude for analysis. message = anthropic_client.messages.create( model="claude-3-haiku-20240307", max_tokens=1024, system=system_prompt, messages=[ { "role": "user", "content": [crop.b64_encoding(output_format="anthropic")], } ], ) # 6. Parse Claude's Response and Update Labels. try: # Parsing Claude's response into an updated object instance. instance = data_model(message.content[0].text) coordinates = crop.instance.get_annotation(frame=frame_data.frame).coordinates instance.set_for_frames( coordinates=coordinates, frames=frame_data.frame, confidence=0.5, manual_annotation=False, ) # Updating the label row by removing the original object and adding the newly classified instance. lr.remove_object(crop.instance) lr.add_object_instance(instance) changes = True except Exception: import traceback traceback.print_exc() print(f"Response from model: {message.content[0].text}") # 7. Save Labels. if changes: lr.save() ``` **Testing the Agent** 1. In your current terminal run the following command to runFastAPI server in development mode with auto-reload enabled. ```shell theme={"dark"} fastapi dev agent.py --port 8080 ``` 2. Open your Project in the Encord platform and navigate to a frame you want to add a classification to. Copy the URL from your browser. The url should have roughly this format: `"https://app.encord.com/label_editor/{project_hash}/{data_hash}/{frame}"`. 3. In another shell operating from the same working directory, source your virtual environment and test the agent: ```shell theme={"dark"} source venv/bin/activate encord-agents test local object_classification '' ``` 4. To see if the test is successful, refresh your browser to view the classifications generated by Claude. Once the test runs successfully, you are ready to deploy your agent. Visit the deployment documentation to learn more. ## Video Recaptioning using GPT-4o-mini The goals of this example are: 1. Create a Custom Agent that automatically generates multiple variations of video captions. 2. Demonstrate how to use OpenAI's GPT-4o-mini model to enhance human-created video captions with a FastAPI-based agent. **Prerequisites** Before you begin, ensure you have: * Created a virtual Python environment. * Installed all necessary dependencies. * Have an OpenAI API key. * Are able to [authenticate with Encord](/agents-documentation/Basics/Authentication). Run the following commands to set up your environment: ```shell theme={"dark"} python -m venv venv # Create a virtual Python environment source venv/bin/activate # Activate the virtual environment python -m pip install encord-agents langchain-openai "fastapi[standard]" openai # Install required dependencies export OPENAI_API_KEY="" # Set your OpenAI API key export ENCORD_SSH_KEY_FILE="/path/to/your/private/key" # Define your Encord access key ``` **Project Setup** Create a Project containing videos in Encord. This example requires an Ontology with four text classifications: * One text classification for human-created summaries of what is happening in the video. * Three text classifications to be automatically filled by the LLM. Ontology ```json theme={"dark"} { "objects": [], "classifications": [ { "id": "1", "featureNodeHash": "GCH8VHIK", "attributes": [ { "id": "1.1", "name": "Caption", "type": "text", "required": false, "featureNodeHash": "Yg7xXEfC" } ] }, { "id": "2", "featureNodeHash": "PwQAwYid", "attributes": [ { "id": "2.1", "name": "Caption Rephrased 1", "type": "text", "required": false, "featureNodeHash": "aQdXJwbG" } ] }, { "id": "3", "featureNodeHash": "3a/aSnHO", "attributes": [ { "id": "3.1", "name": "Caption Rephrased 2", "type": "text", "required": false, "featureNodeHash": "8zY6H62x" } ] }, { "id": "4", "featureNodeHash": "FNjXp5TU", "attributes": [ { "id": "4.1", "name": "Caption Rephrased 3", "type": "text", "required": false, "featureNodeHash": "sKg1Kq/m" } ] } ] } ``` To construct the Ontology used in this example, run the following script: ```python theme={"dark"} import json from encord.objects.ontology_structure import OntologyStructure from encord.objects.attributes import TextAttribute structure = OntologyStructure() caption = structure.add_classification() caption.add_attribute(TextAttribute, "Caption") re1 = structure.add_classification() re1.add_attribute(TextAttribute, "Recaption 1") re2 = structure.add_classification() re2.add_attribute(TextAttribute, "Recaption 2") re3 = structure.add_classification() re3.add_attribute(TextAttribute, "Recaption 3") print(json.dumps(structure.to_dict(), indent=2)) create_ontology = False if create_ontology: from encord.user_client import EncordUserClient client = EncordUserClient.create_with_ssh_private_key() # Look in auth section for authentication client.create_ontology("title", "description", structure) ``` The workflow for this agent is: 1. A human watches the video and enters a caption in the first text field. 2. The agent is then triggered and generates three additional caption variations for review. * Each video is first annotated by a human (ANNOTATE stage). * Next, a data agent automatically generates alternative captions (AGENT stage). * Finally, a human reviews all four captions (REVIEW stage) before the task is marked complete. If no human caption is present when the agent is triggered, the task is sent back for annotation. If the review stage results in rejection, the task is also returned for re-annotation.
Workflow
**Create the Agent** This section provides the complete code for creating your Custom Agent, along with an explanation of its internal workings. **Agent Setup Steps** 1. Set up imports and create a Pydantic model for our LLM's structured output 2. Create a detailed system prompt for the LLM that explains exactly what kind of rephrasing we want 3. We configure the LLM to use structured outputs based on our model 4. Create a helper function to prompt the model with both text and image: 5. Initialize the FastAPI app with the required CORS middleware: 6. Define the agent to handle the recaptioning. This includes: * Retrieving the existing human-created caption, prioritizing captions from the current frame or falling back to frame zero. * Sending the first frame of the video along with the human caption to the LLM. * Processing the response from the LLM, which provides three alternative phrasings of the original caption. * Updating the label row with the new captions, replacing any existing ones. ```python theme={"dark"} # 1. Set up imports and create a Pydantic model for our LLM's structured output. import os from typing import Annotated import numpy as np from encord.exceptions import LabelRowError from encord.objects.classification_instance import ClassificationInstance from encord.objects.ontology_labels_impl import LabelRowV2 from fastapi import Depends from langchain_openai import ChatOpenAI from numpy.typing import NDArray from pydantic import BaseModel from encord_agents import FrameData from encord_agents.fastapi.cors import get_encord_app from encord_agents.fastapi.dependencies import Frame, dep_label_row, dep_single_frame # The response model for the agent to follow. class AgentCaptionResponse(BaseModel): rephrase_1: str rephrase_2: str rephrase_3: str # 2. Create a detailed system prompt for the LLM that explains exactly what kind of rephrasing we want. SYSTEM_PROMPT = """ You are a helpful assistant that rephrases captions. I will provide you with a video caption and an image of the scene of the video. The captions follow this format: "The droid picks up and puts it on the ." The captions that you make should replace the tags, e.g., , with the actual object names. The replacements should be consistent with the scene. Here are three rephrases: 1. The droid picks up the blue mug and puts it on the left side of the table. 2. The droid picks up the cup and puts it to the left of the plate. 3. The droid is picking up the mug on the right side of the table and putting it down next to the plate. You will rephrase the caption in three different ways, as above, the rephrases should be 1. Diverse in terms of adjectives, object relations, and object positions. 2. Sound in relation to the scene. You cannot talk about objects you cannot see. 3. Short and concise. Keep it within one sentence. """ # 3. Configure the LLM to use structured outputs based on our model. llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.4, api_key=os.environ["OPENAI_API_KEY"]) llm_structured = llm.with_structured_output(AgentCaptionResponse) # 4. Create a helper function to prompt the model with both text and image. def prompt_gpt(caption: str, image: Frame) -> AgentCaptionResponse: prompt = [ {"role": "system", "content": SYSTEM_PROMPT}, { "role": "user", "content": [ {"type": "text", "text": f"Video caption: `{caption}`"}, image.b64_encoding(output_format="openai"), ], }, ] return llm_structured.invoke(prompt) # 5. Initialize the FastAPI app with the required CORS middleware. app = get_encord_app() # 6. Define the agent to handle the recaptioning. @app.post("/my_agent") def my_agent( frame_data: FrameData, label_row: Annotated[LabelRowV2, Depends(dep_label_row)], frame_content: Annotated[NDArray[np.uint8], Depends(dep_single_frame)], ) -> None: # Get the relevant Ontology information # Recall that we expect # [human annotation, llm recaption 1, llm recaption 2, llm recaption 3] # in the Ontology cap, *rs = label_row.ontology_structure.classifications # Retrieve the existing human-created caption, prioritizing captions from the current frame or falling back to frame zero. instances = label_row.get_classification_instances( filter_ontology_classification=cap, filter_frames=[0, frame_data.frame] ) if not instances: # nothing to do if there are no human labels return elif len(instances) > 1: def order_by_current_frame_else_frame_0( instance: ClassificationInstance, ) -> bool: try: instance.get_annotation(frame_data.frame) return 2 # The best option except LabelRowError: pass try: instance.get_annotation(0) return 1 except LabelRowError: return 0 instance = sorted(instances, key=order_by_current_frame_else_frame_0)[-1] else: instance = instances[0] # Read the actual string caption caption = instance.get_answer() # Send the first frame of the video along with the human caption to the LLM. frame = Frame(frame=0, content=frame_content) response = prompt_gpt(caption, frame) # Process the LLM's response, which contains three different versions of the original caption. # Update the label row with the new captions, replacing any existing ones. for r, t in zip(rs, [response.rephrase_1, response.rephrase_2, response.rephrase_3]): # Overwrite any existing re-captions existing_instances = label_row.get_classification_instances(filter_ontology_classification=r) for existing_instance in existing_instances: label_row.remove_classification(existing_instance) # Create new instances ins = r.create_instance() ins.set_answer(t, attribute=r.attributes[0]) ins.set_for_frames(0) label_row.add_classification_instance(ins) label_row.save() ``` **Test the Agent** 1. In your current terminal, run the following command to run the FastAPI server: ```shell theme={"dark"} ENCORD_SSH_KEY_FILE=/path/to/your_private_key \ OPENAI_API_KEY= \ fastapi dev main.py ``` 2. Open your Project in the Encord platform, navigate to a video frame, and add your initial caption. Copy the URL from your browser. 3. In another shell operating from the same working directory, source your virtual environment and test the agent: ```shell theme={"dark"} source venv/bin/activate encord-agents test local my_agent '' ``` 4. Refresh your browser to view the three AI-generated caption variations. Once the test runs successfully, you are ready to deploy your agent. Visit [the deployment documentation](/agents-documentation/Custom-Agents/GCP-Cloud-Functions#step-4-deployment) to learn more. # GCP Examples Source: https://docs.encord.com/agents-documentation/Custom-Agents/Examples/GCP-Examples ## Basic Geometric Example A simple example showing how to use objectHashes. ```python agent.py theme={"dark"} from typing import Annotated from encord.objects.ontology_labels_impl import LabelRowV2 from encord.objects.ontology_object_instance import ObjectInstance from encord_agents.core.data_model import FrameData from encord_agents.core.dependencies import Depends from encord_agents.gcp.dependencies import dep_objects from encord_agents.gcp.wrappers import editor_agent @editor_agent() def handle_object_hashes( frame_data: FrameData, lr: LabelRowV2, object_instances: Annotated[list[ObjectInstance], Depends(dep_objects)], ) -> None: for object_inst in object_instances: print(object_inst) ``` **Use Case: Selective OCR on Selected Objects** This functionality allows you to apply your own OCR model to specific objects selected directly within the Encord platform. When you trigger your agent from the Encord app after selecting objects, the platform automatically sends a list of `objectHashes` to your agent. Your agent can then use the `dep_objects` method to gain immediate access to these specific object instances, which greatly simplifies integrating your OCR model for targeted processing. **Test the Agent** 1. Save the above code as `agent.py`. 2. Run the following command to run the agent in debug mode in your terminal. ```shell theme={"dark"} functions-framework --target=handle_object_hashes --debug --source agent.py ``` 3. Open your Project in the Encord platform and navigate to a frame with an object that you want to act on. Choose an object from the bottom left sider and click `Copy URL` as shown:
Copy URL from left sider
The url should have roughly this format: `"https://app.encord.com/label_editor/{project_hash}/{data_hash}/{frame}/0?other_query_params&objectHash={objectHash}"`. 4. In another shell operating from the same working directory, source your virtual environment and test the agent. ```shell theme={"dark"} source venv/bin/activate encord-agents test local agent '' ``` 5. To see if the test is successful, refresh your browser to see the action taken by the Agent. If the test has run successfully, the agent can be deployed. Visit [the deployment documentation](/agents-documentation/Custom-Agents/GCP-Cloud-Functions#step-4-deployment) to learn more. ## Nested Classification using Claude 3.5 Sonnet The goals of this example are: 1. Create a Custom Agent that automatically adds frame-level classifications. 2. Demonstrate how to use the [`OntologyDataModel`](/agents-documentation/Reference/Core/Agents-Reference-Core#ontologydatamodel-objects) for classifications. **Prerequisites** Before you begin, ensure you have: * Created a virtual Python environment. * Installed all necessary dependencies. * Have an [Anthropic API key](https://www.anthropic.com/api). * Are able to [authenticate with Encord](/agents-documentation/Basics/Authentication). Run the following commands to set up your environment: ```shell theme={"dark"} python -m venv venv # Create a virtual Python environment source venv/bin/activate # Activate the virtual environment python -m pip install encord-agents anthropic # Install required dependencies export ANTHROPIC_API_KEY="" # Set your Anthropic API key export ENCORD_SSH_KEY_FILE="/path/to/your/private/key" # Define your Encord access key ``` **Project Setup** Create a Project with visual content (images, image groups, image sequences, or videos) in Encord. This example uses the following Ontology, but any Ontology containing classifications can be used.
```json Ontology JSON theme={"dark"} { "objects": [], "classifications": [ { "id": "1", "featureNodeHash": "TTkHMtuD", "attributes": [ { "id": "1.1", "featureNodeHash": "+1g9I9Sg", "type": "text", "name": "scene summary", "required": false, "dynamic": false } ] }, { "id": "2", "featureNodeHash": "xGV/wCD0", "attributes": [ { "id": "2.1", "featureNodeHash": "k3EVexk7", "type": "radio", "name": "is there a person in the frame?", "required": false, "options": [ { "id": "2.1.1", "featureNodeHash": "EkGwhcO4", "label": "yes", "value": "yes", "options": [ { "id": "2.1.1.1", "featureNodeHash": "mj9QCDY4", "type": "text", "name": "What is the person doing?", "required": false } ] }, { "id": "2.1.2", "featureNodeHash": "37rMLC/v", "label": "no", "value": "no", "options": [] } ], "dynamic": false } ] } ] } ``` To construct the same Ontology as used in this example, run the following script. ```python Create Ontology theme={"dark"} import json from encord.objects.ontology_structure import OntologyStructure from encord_agents.core.utils import get_user_client encord_client = get_user_client() structure = OntologyStructure.from_dict(json.loads("{the_json_above}")) ontology = encord_client.create_ontology( title="Your ontology title", structure=structure ) print(ontology.ontology_hash) ``` The aim is to trigger an agent that transforms a labeling task from Figure A to Figure B. **Figure A: No classification labels.** **Figure B: Multiple nested classification labels generated by an LLM.** **Create the Agent** This section provides the complete code for creating your Custom Agent, along with an explanation of its internal workings. **Agent Setup Steps** 1. Import dependencies, authenticate with Encord, and set up the Project. Ensure you insert your Project's unique identifier. 2. Create a data model and a system prompt based on the Project Ontology to tell Claude how to structure its response. 3. Set up an Anthropic API client to establish communication with the Claude model. 4. Define the Custom Agent. This includes * Retrieving Frame Content: It automatically fetches the current frame's image data using the `dep_single_frame` dependency. * Analyzing with Claude: The frame image is then sent to the Claude AI model for analysis. * Parsing Classifications: Claude's response is parsed and transformed into structured classification instances using the predefined data model. * Saving Results: The new classifications are added to the active label row, and the updated results are saved within the Project. ````python theme={"dark"} # 1. Import dependencies, authenticate with Encord, and set up the Project. Ensure you insert your Project's unique identifier. import os from anthropic import Anthropic from encord.objects.ontology_labels_impl import LabelRowV2 from numpy.typing import NDArray from typing_extensions import Annotated from encord_agents.core.ontology import OntologyDataModel from encord_agents.core.utils import get_user_client from encord_agents.core.video import Frame from encord_agents.gcp import Depends, editor_agent from encord_agents.gcp.dependencies import FrameData, dep_single_frame client = get_user_client() project = client.get_project("") # 2. Create a data model and a system prompt based on the Project Ontology to tell Claude how to structure its response data_model = OntologyDataModel(project.ontology_structure.classifications) system_prompt = f""" You're a helpful assistant that's supposed to help fill in json objects according to this schema: ```json {data_model.model_json_schema_str} ``` Please only respond with valid json. """ # 3. Set up an Anthropic API client to establish communication with Claude ANTHROPIC_API_KEY = os.getenv("ANTHROPIC_API_KEY") anthropic_client = Anthropic(api_key=ANTHROPIC_API_KEY) # 4. Define the Custom Agent @editor_agent() def agent( frame_data: FrameData, lr: LabelRowV2, content: Annotated[NDArray, Depends(dep_single_frame)], ): # # Retrieving Frame Content: It automatically fetches the current frame's image data using the `dep_single_frame` dependency frame = Frame(frame_data.frame, content=content) # Analyzing with Claude: The frame image is then sent to the Claude AI model for analysis message = anthropic_client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, system=system_prompt, messages=[ { "role": "user", "content": [frame.b64_encoding(output_format="anthropic")], } ], ) try: # Parsing Classifications: Claude's response is parsed and transformed into structured classification instances using the predefined data model classifications = data_model(message.content[0].text) for clf in classifications: clf.set_for_frames(frame_data.frame, confidence=0.5, manual_annotation=False) lr.add_classification_instance(clf) except Exception: import traceback traceback.print_exc() print(f"Response from model: {message.content[0].text}") # Saving Results: The new classifications are added to the active label row, and the updated results are saved within the Project. lr.save() ```` ```json theme={"dark"} { "$defs": { "IsThereAPersonInTheFrameRadioModel": { "properties": { "feature_node_hash": { "const": "k3EVexk7", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "k3EVexk7" ], "title": "Feature Node Hash", "type": "string" }, "choice": { "description": "Choose exactly one answer from the given options.", "discriminator": { "mapping": { "37rMLC/v": "#/$defs/NoNestedRadioModel", "EkGwhcO4": "#/$defs/YesNestedRadioModel" }, "propertyName": "feature_node_hash" }, "oneOf": [ { "$ref": "#/$defs/YesNestedRadioModel" }, { "$ref": "#/$defs/NoNestedRadioModel" } ], "title": "Choice" } }, "required": [ "feature_node_hash", "choice" ], "title": "IsThereAPersonInTheFrameRadioModel", "type": "object" }, "NoNestedRadioModel": { "properties": { "feature_node_hash": { "const": "37rMLC/v", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "37rMLC/v" ], "title": "Feature Node Hash", "type": "string" }, "title": { "const": "no", "default": "Constant value - should be included as-is.", "enum": [ "no" ], "title": "Title", "type": "string" } }, "required": [ "feature_node_hash" ], "title": "NoNestedRadioModel", "type": "object" }, "SceneSummaryTextModel": { "properties": { "feature_node_hash": { "const": "+1g9I9Sg", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "+1g9I9Sg" ], "title": "Feature Node Hash", "type": "string" }, "value": { "description": "Please describe the image as accurate as possible focusing on 'scene summary'", "maxLength": 1000, "minLength": 0, "title": "Value", "type": "string" } }, "required": [ "feature_node_hash", "value" ], "title": "SceneSummaryTextModel", "type": "object" }, "WhatIsThePersonDoingTextModel": { "properties": { "feature_node_hash": { "const": "mj9QCDY4", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "mj9QCDY4" ], "title": "Feature Node Hash", "type": "string" }, "value": { "description": "Please describe the image as accurate as possible focusing on 'What is the person doing?'", "maxLength": 1000, "minLength": 0, "title": "Value", "type": "string" } }, "required": [ "feature_node_hash", "value" ], "title": "WhatIsThePersonDoingTextModel", "type": "object" }, "YesNestedRadioModel": { "properties": { "feature_node_hash": { "const": "EkGwhcO4", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "EkGwhcO4" ], "title": "Feature Node Hash", "type": "string" }, "what_is_the_person_doing": { "$ref": "#/$defs/WhatIsThePersonDoingTextModel", "description": "A text attribute with carefully crafted text to describe the property." } }, "required": [ "feature_node_hash", "what_is_the_person_doing" ], "title": "YesNestedRadioModel", "type": "object" } }, "properties": { "scene_summary": { "$ref": "#/$defs/SceneSummaryTextModel", "description": "A text attribute with carefully crafted text to describe the property." }, "is_there_a_person_in_the_frame": { "$ref": "#/$defs/IsThereAPersonInTheFrameRadioModel", "description": "A mutually exclusive radio attribute to choose exactly one option that best matches to the give visual input." } }, "required": [ "scene_summary", "is_there_a_person_in_the_frame" ], "title": "ClassificationModel", "type": "object" } ``` **Test the Agent** 1. In your current terminal, run the following command to run the agent in debug mode. ```shell theme={"dark"} functions-framework --target=agent --debug --source agent.py ``` 2. Open your Project in [the Encord platform](https://app.encord.com/projects) and navigate to a frame you want to add a classification to. Copy the URL from your browser. The url should have the following format: `"https://app.encord.com/label_editor/{project_hash}/{data_hash}/{frame}"`. 3. In another shell operating from the same working directory, source your virtual environment and test the agent. ```shell theme={"dark"} source venv/bin/activate encord-agents test local agent '' ``` 4. To see if the test is successful, refresh your browser to view the classifications generated by Claude. Once the test runs successfully, you are ready to deploy your agent. Visit [the deployment documentation](/agents-documentation/Custom-Agents/GCP-Cloud-Functions#step-4-deployment) to learn more. ## Nested Attributes using Claude 3.5 Sonnet The goals of this example are: 1. Create a Custom Agent that can convert generic object annotations (class-less coordinates) into class specific annotations with nested attributes like descriptions, radio buttons, and checklists. 2. Demonstrate how to use both the [`OntologyDataModel`](/agents-documentation/Reference/Core/Agents-Reference-Core#ontologydatamodel-objects) and the [`dep_object_crops`](/agents-documentation/Reference/Custom-Agents/Agents-Reference-Custom-Agents) dependency. **Prerequisites** Before you begin, ensure you have: * Created a virtual Python environment. * Installed all necessary dependencies. * Have an [Anthropic API key](https://www.anthropic.com/api). * Are able to [authenticate with Encord](/agents-documentation/Basics/Authentication). Run the following commands to set up your environment: ```shell theme={"dark"} python -m venv venv # Create a virtual Python environment source venv/bin/activate # Activate the virtual environment python -m pip install encord-agents anthropic # Install required dependencies export ANTHROPIC_API_KEY="" # Set your Anthropic API key export ENCORD_SSH_KEY_FILE="/path/to/your/private/key" # Define your Encord access key ``` **Project Setup** Create a Project with visual content (images, image groups, image sequences, or videos) in Encord. This example uses the following Ontology, but any Ontology containing classifications can be used provided the object types are the same and there is one entry called `"generic"`.
```json ontology.json theme={"dark"} { "objects": [ { "id": "1", "name": "person", "color": "#D33115", "shape": "bounding_box", "featureNodeHash": "2xlDPPAG", "required": false, "attributes": [ { "id": "1.1", "featureNodeHash": "aFCN9MMm", "type": "text", "name": "activity", "required": false, "dynamic": false } ] }, { "id": "2", "name": "animal", "color": "#E27300", "shape": "bounding_box", "featureNodeHash": "3y6JxTUX", "required": false, "attributes": [ { "id": "2.1", "featureNodeHash": "2P7LTUZA", "type": "radio", "name": "type", "required": false, "options": [ { "id": "2.1.1", "featureNodeHash": "gJvcEeLl", "label": "dolphin", "value": "dolphin", "options": [] }, { "id": "2.1.2", "featureNodeHash": "CxrftGS4", "label": "monkey", "value": "monkey", "options": [] }, { "id": "2.1.3", "featureNodeHash": "OQyWm7Sm", "label": "dog", "value": "dog", "options": [] }, { "id": "2.1.4", "featureNodeHash": "CDKmYJK/", "label": "cat", "value": "cat", "options": [] } ], "dynamic": false }, { "id": "2.2", "featureNodeHash": "5fFgrM+E", "type": "text", "name": "description", "required": false, "dynamic": false } ] }, { "id": "3", "name": "vehicle", "color": "#16406C", "shape": "bounding_box", "featureNodeHash": "llw7qdWW", "required": false, "attributes": [ { "id": "3.1", "featureNodeHash": "79mo1G7Q", "type": "text", "name": "type - short and concise", "required": false, "dynamic": false }, { "id": "3.2", "featureNodeHash": "OFrk07Ds", "type": "checklist", "name": "visible", "required": false, "options": [ { "id": "3.2.1", "featureNodeHash": "KmX/HjRT", "label": "wheels", "value": "wheels" }, { "id": "3.2.2", "featureNodeHash": "H6qbEcdj", "label": "frame", "value": "frame" }, { "id": "3.2.3", "featureNodeHash": "gZ9OucoQ", "label": "chain", "value": "chain" }, { "id": "3.2.4", "featureNodeHash": "cit3aZSz", "label": "head lights", "value": "head_lights" }, { "id": "3.2.5", "featureNodeHash": "qQ3PieJ/", "label": "tail lights", "value": "tail_lights" } ], "dynamic": false } ] }, { "id": "4", "name": "generic", "color": "#FE9200", "shape": "bounding_box", "featureNodeHash": "jootTFfQ", "required": false, "attributes": [] } ], "classifications": [] } ``` To construct the Ontology used in this example, run the following script: ```python theme={"dark"} import json from encord.objects.ontology_structure import OntologyStructure from encord_agents.core.utils import get_user_client encord_client = get_user_client() structure = OntologyStructure.from_dict(json.loads("{the_json_above}")) ontology = encord_client.create_ontology( title="Your ontology title", structure=structure ) print(ontology.ontology_hash) ``` The goal is create an agent that takes a labeling task from Figure A to Figure B **Figure A: No classification labels.** **Figure B: Multiple nested classification labels generated by an LLM.** **Create the Agent** This section provides the complete code for creating your Custom Agent, along with an explanation of its internal workings. **Agent Setup Steps** 1. Import dependencies, authenticate with Encord, and set up the Project. Ensure you insert your Project's unique identifier. 2. Extract the generic Ontology object and the specific objects of interest. This example sorts Ontology objects based on whether their title is `"generic"`. The generic object is used to query image crops within the agent. Before that, `other_objects` is used to pass in the specific context we want Claude to focus on. The [`OntologyDataModel`](/agents-documentation/Reference/Core/Agents-Reference-Core#encord_agents.core.ontology.OntologyDataModel-Objects) class helps convert Encord Ontology [Objects](/sdk-documentation/sdk-references/objects.ontology_object) into a [Pydantic](https://docs.pydantic.dev/latest/) model and parse JSON into Encord [ObjectInstance](/sdk-documentation/sdk-references/objects.ontology_object_instance)s. 3. Prepare the system prompt for each object crop using the `data_model` to generate the JSON schema. Only `other_objects` is passed to ensure the model can choose only from non-generic object types. 4. Set up an Anthropic API client to establish communication with the Claude model. You must include your Anthropic API key. 5. Define the Custom Agent. * All arguments are automatically injected when the agent is called. For details on dependency injection, see [here](/agents-documentation/Basics/Dependencies). * The [`dep_object_crops`](/agents-documentation/Reference/Custom-Agents/Agents-Reference-Custom-Agents#dep-object-crops-2) dependency allows filtering. In this case, it includes only "generic" object crops, excluding those already converted to actual labels. 6. Query Claude using the image crops. The `crop` variable has a convenient `b64_encoding` method to produce an input that Claude understands. 7. Parse Claude's message using the `data_model`. When called with a JSON string, it attempts to parse it with respect to the JSON schema we saw above to create an Encord object instance. If successful, the old generic object can be removed and the newly classified object added. 8. Save the labels with Encord. ```python theme={"dark"} # 1. Import dependencies, authenticate with Encord, and set up the Project. Ensure you insert your Project's unique identifier import os from anthropic import Anthropic from encord.objects.ontology_labels_impl import LabelRowV2 from typing_extensions import Annotated from encord_agents.core.ontology import OntologyDataModel from encord_agents.core.utils import get_user_client from encord_agents.gcp import Depends, editor_agent from encord_agents.gcp.dependencies import FrameData, InstanceCrop, dep_object_crops # User client client = get_user_client() project = client.get_project("") # 2. Extract the generic Ontology object and the specific objects of interest. This example sorts Ontology objects based on whether their title is `"generic"` generic_ont_obj, *other_objects = sorted( project.ontology_structure.objects, key=lambda o: o.title.lower() == "generic", reverse=True, ) # 3. Prepare the system prompt for each object crop using the `data_model` to generate the JSON schema data_model = OntologyDataModel(other_objects) system_prompt = f""" You're a helpful assistant that's supposed to help fill in json objects according to this schema: `{data_model.model_json_schema_str}` Please only respond with valid json. """ # 4. Set up an Anthropic API client to establish communication with the Claude model. You must include your Anthropic API key ANTHROPIC_API_KEY = os.getenv("ANTHROPIC_API_KEY") anthropic_client = Anthropic(api_key=ANTHROPIC_API_KEY) # 5. Define the Custom Agent @editor_agent() def agent( frame_data: FrameData, lr: LabelRowV2, crops: Annotated[ list[InstanceCrop], Depends(dep_object_crops(filter_ontology_objects=[generic_ont_obj])), ], ): # 6. Query Claude using the image crops. The `crop` variable has a convenient `b64_encoding` method to produce an input that Claude understands. changes = False for crop in crops: message = anthropic_client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, system=system_prompt, messages=[ { "role": "user", "content": [crop.b64_encoding(output_format="anthropic")], } ], ) # 7. Parse Claude's message using the `data_model`. try: instance = data_model(message.content[0].text) coordinates = crop.instance.get_annotation(frame=frame_data.frame).coordinates instance.set_for_frames( coordinates=coordinates, frames=frame_data.frame, confidence=0.5, manual_annotation=False, ) lr.remove_object(crop.instance) lr.add_object_instance(instance) changes = True except Exception: import traceback traceback.print_exc() print(f"Response from model: {message.content[0].text}") # 8. Save the labels with Encord. if changes: lr.save() ``` ```json theme={"dark"} { "$defs": { "ActivityTextModel": { "properties": { "feature_node_hash": { "const": "aFCN9MMm", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "aFCN9MMm" ], "title": "Feature Node Hash", "type": "string" }, "value": { "description": "Please describe the image as accurate as possible focusing on 'activity'", "maxLength": 1000, "minLength": 0, "title": "Value", "type": "string" } }, "required": [ "feature_node_hash", "value" ], "title": "ActivityTextModel", "type": "object" }, "AnimalNestedModel": { "properties": { "feature_node_hash": { "const": "3y6JxTUX", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "3y6JxTUX" ], "title": "Feature Node Hash", "type": "string" }, "type": { "$ref": "#/$defs/TypeRadioModel", "description": "A mutually exclusive radio attribute to choose exactly one option that best matches to the give visual input." }, "description": { "$ref": "#/$defs/DescriptionTextModel", "description": "A text attribute with carefully crafted text to describe the property." } }, "required": [ "feature_node_hash", "type", "description" ], "title": "AnimalNestedModel", "type": "object" }, "DescriptionTextModel": { "properties": { "feature_node_hash": { "const": "5fFgrM+E", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "5fFgrM+E" ], "title": "Feature Node Hash", "type": "string" }, "value": { "description": "Please describe the image as accurate as possible focusing on 'description'", "maxLength": 1000, "minLength": 0, "title": "Value", "type": "string" } }, "required": [ "feature_node_hash", "value" ], "title": "DescriptionTextModel", "type": "object" }, "PersonNestedModel": { "properties": { "feature_node_hash": { "const": "2xlDPPAG", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "2xlDPPAG" ], "title": "Feature Node Hash", "type": "string" }, "activity": { "$ref": "#/$defs/ActivityTextModel", "description": "A text attribute with carefully crafted text to describe the property." } }, "required": [ "feature_node_hash", "activity" ], "title": "PersonNestedModel", "type": "object" }, "TypeRadioEnum": { "enum": [ "dolphin", "monkey", "dog", "cat" ], "title": "TypeRadioEnum", "type": "string" }, "TypeRadioModel": { "properties": { "feature_node_hash": { "const": "2P7LTUZA", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "2P7LTUZA" ], "title": "Feature Node Hash", "type": "string" }, "choice": { "$ref": "#/$defs/TypeRadioEnum", "description": "Choose exactly one answer from the given options." } }, "required": [ "feature_node_hash", "choice" ], "title": "TypeRadioModel", "type": "object" }, "TypeShortAndConciseTextModel": { "properties": { "feature_node_hash": { "const": "79mo1G7Q", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "79mo1G7Q" ], "title": "Feature Node Hash", "type": "string" }, "value": { "description": "Please describe the image as accurate as possible focusing on 'type - short and concise'", "maxLength": 1000, "minLength": 0, "title": "Value", "type": "string" } }, "required": [ "feature_node_hash", "value" ], "title": "TypeShortAndConciseTextModel", "type": "object" }, "VehicleNestedModel": { "properties": { "feature_node_hash": { "const": "llw7qdWW", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "llw7qdWW" ], "title": "Feature Node Hash", "type": "string" }, "type__short_and_concise": { "$ref": "#/$defs/TypeShortAndConciseTextModel", "description": "A text attribute with carefully crafted text to describe the property." }, "visible": { "$ref": "#/$defs/VisibleChecklistModel", "description": "A collection of boolean values indicating which concepts are applicable according to the image content." } }, "required": [ "feature_node_hash", "type__short_and_concise", "visible" ], "title": "VehicleNestedModel", "type": "object" }, "VisibleChecklistModel": { "properties": { "feature_node_hash": { "const": "OFrk07Ds", "description": "UUID for discrimination. Must be included in json as is.", "enum": [ "OFrk07Ds" ], "title": "Feature Node Hash", "type": "string" }, "wheels": { "description": "Is 'wheels' applicable or not?", "title": "Wheels", "type": "boolean" }, "frame": { "description": "Is 'frame' applicable or not?", "title": "Frame", "type": "boolean" }, "chain": { "description": "Is 'chain' applicable or not?", "title": "Chain", "type": "boolean" }, "head_lights": { "description": "Is 'head lights' applicable or not?", "title": "Head Lights", "type": "boolean" }, "tail_lights": { "description": "Is 'tail lights' applicable or not?", "title": "Tail Lights", "type": "boolean" } }, "required": [ "feature_node_hash", "wheels", "frame", "chain", "head_lights", "tail_lights" ], "title": "VisibleChecklistModel", "type": "object" } }, "properties": { "choice": { "description": "Choose exactly one answer from the given options.", "discriminator": { "mapping": { "2xlDPPAG": "#/$defs/PersonNestedModel", "3y6JxTUX": "#/$defs/AnimalNestedModel", "llw7qdWW": "#/$defs/VehicleNestedModel" }, "propertyName": "feature_node_hash" }, "oneOf": [ { "$ref": "#/$defs/PersonNestedModel" }, { "$ref": "#/$defs/AnimalNestedModel" }, { "$ref": "#/$defs/VehicleNestedModel" } ], "title": "Choice" } }, "required": [ "choice" ], "title": "ObjectsRadioModel", "type": "object" } ``` **Test the Agent** 1. In your current terminal, run the following command to run the agent in debug mode. ```shell theme={"dark"} functions-framework --target=agent --debug --source agent.py ``` 2. Open your Project in the Encord platform and navigate to a frame you want to add a generic object to. Copy the URL from your browser. The url has following format: `"https://app.encord.com/label_editor/{project_hash}/{data_hash}/{frame}"`. 3. In another shell operating from the same working directory, source your virtual environment and test the agent. ```shell theme={"dark"} source venv/bin/activate encord-agents test local agent ``` 4. To see if the test is successful, refresh your browser to view the classifications generated by Claude. Once the test runs successfully, you are ready to deploy your agent. Visit the deployment documentation to learn more. ## Video Recaptioning using GPT-4o-mini The goals of this example are: 1. Create a Custom Agent that automatically generates multiple variations of video captions. 2. Demonstrate how to use OpenAI's GPT-4o-mini model to enhance human-created video captions with a FastAPI-based agent. **Prerequisites** Before you begin, ensure you have: * Created a virtual Python environment. * Installed all necessary dependencies. * Have an OpenAI API key. * Are able to [authenticate with Encord](/agents-documentation/Basics/Authentication). Run the following commands to set up your environment: ```shell theme={"dark"} python -m venv venv # Create a virtual Python environment source venv/bin/activate # Activate the virtual environment python -m pip install encord-agents langchain-openai "fastapi[standard]" openai # Install required dependencies export OPENAI_API_KEY="" # Set your OpenAI API key export ENCORD_SSH_KEY_FILE="/path/to/your/private/key" # Define your Encord access key ``` **Project Setup** Create a Project containing videos in Encord. This example requires an Ontology with four text classifications: * One text classification for human-created summaries of what is happening in the video. * Three text classifications to be automatically filled by the LLM. **Ontology** ```json theme={"dark"} { "objects": [], "classifications": [ { "id": "1", "featureNodeHash": "GCH8VHIK", "attributes": [ { "id": "1.1", "name": "Caption", "type": "text", "required": false, "featureNodeHash": "Yg7xXEfC" } ] }, { "id": "2", "featureNodeHash": "PwQAwYid", "attributes": [ { "id": "2.1", "name": "Caption Rephrased 1", "type": "text", "required": false, "featureNodeHash": "aQdXJwbG" } ] }, { "id": "3", "featureNodeHash": "3a/aSnHO", "attributes": [ { "id": "3.1", "name": "Caption Rephrased 2", "type": "text", "required": false, "featureNodeHash": "8zY6H62x" } ] }, { "id": "4", "featureNodeHash": "FNjXp5TU", "attributes": [ { "id": "4.1", "name": "Caption Rephrased 3", "type": "text", "required": false, "featureNodeHash": "sKg1Kq/m" } ] } ] } ``` To construct the Ontology used in this example, run the following script: ```python theme={"dark"} import json from encord.objects.ontology_structure import OntologyStructure from encord.objects.attributes import TextAttribute structure = OntologyStructure() caption = structure.add_classification() caption.add_attribute(TextAttribute, "Caption") re1 = structure.add_classification() re1.add_attribute(TextAttribute, "Recaption 1") re2 = structure.add_classification() re2.add_attribute(TextAttribute, "Recaption 2") re3 = structure.add_classification() re3.add_attribute(TextAttribute, "Recaption 3") print(json.dumps(structure.to_dict(), indent=2)) create_ontology = False if create_ontology: from encord.user_client import EncordUserClient client = EncordUserClient.create_with_ssh_private_key() # Look in auth section for authentication client.create_ontology("title", "description", structure) ``` The workflow for this agent is: 1. A human watches the video and enters a caption in the first text field. 2. The agent is then triggered and generates three additional caption variations for review. * Each video is first annotated by a human (ANNOTATE stage). * Next, a data agent automatically generates alternative captions (AGENT stage). * Finally, a human reviews all four captions (REVIEW stage) before the task is marked complete. If no human caption is present when the agent is triggered, the task is sent back for annotation. If the review stage results in rejection, the task is also returned for re-annotation. **Workflow** **Create the Agent** This section provides the complete code for creating your Custom Agent, along with an explanation of its internal workings. **Agent Setup Steps** 1. Set up imports and create a Pydantic model for our LLM's structured output. 2. Create a detailed system prompt for the LLM that explains exactly what kind of rephrasing we want. 3. Configure the LLM to use structured outputs based on our model. 4. Create a helper function to prompt the model with both text and image. 5. Define the agent to handle the recaptioning. This includes: * Retrieving the existing human-created caption, prioritizing captions from the current frame or falling back to frame zero. * Sending the first frame of the video along with the human caption to the LLM. * Processing the response from the LLM, which provides three alternative phrasings of the original caption. * Updating the label row with the new captions, replacing any existing ones. ```python theme={"dark"} # 1. Set up imports and create a Pydantic model for our LLM's structured output. import os from typing import Annotated import numpy as np from encord.exceptions import LabelRowError from encord.objects.classification_instance import ClassificationInstance from encord.objects.ontology_labels_impl import LabelRowV2 from langchain_openai import ChatOpenAI from numpy.typing import NDArray from pydantic import BaseModel from encord_agents import FrameData from encord_agents.gcp import Depends, editor_agent from encord_agents.gcp.dependencies import Frame, dep_single_frame # The response model for the agent to follow. class AgentCaptionResponse(BaseModel): rephrase_1: str rephrase_2: str rephrase_3: str # 2. Create a detailed system prompt for the LLM that explains exactly what kind of rephrasing we want. SYSTEM_PROMPT = """ You are a helpful assistant that rephrases captions. I will provide you with a video caption and an image of the scene of the video. The captions follow this format: "The droid picks up and puts it on the ." The captions that you make should replace the tags, e.g., , with the actual object names. The replacements should be consistent with the scene. Here are three rephrases: 1. The droid picks up the blue mug and puts it on the left side of the table. 2. The droid picks up the cup and puts it to the left of the plate. 3. The droid is picking up the mug on the right side of the table and putting it down next to the plate. You will rephrase the caption in three different ways, as above, the rephrases should be 1. Diverse in terms of adjectives, object relations, and object positions. 2. Sound in relation to the scene. You cannot talk about objects you cannot see. 3. Short and concise. Keep it within one sentence. """ # 3. Configure the LLM to use structured outputs based on our model. llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.4, api_key=os.environ["OPENAI_API_KEY"]) llm_structured = llm.with_structured_output(AgentCaptionResponse) # 4. Create a helper function to prompt the model with both text and image. def prompt_gpt(caption: str, image: Frame) -> AgentCaptionResponse: prompt = [ {"role": "system", "content": SYSTEM_PROMPT}, { "role": "user", "content": [ {"type": "text", "text": f"Video caption: `{caption}`"}, image.b64_encoding(output_format="openai"), ], }, ] return llm_structured.invoke(prompt) # 5. Define the agent to handle the recaptioning. This includes: @editor_agent() def my_agent( frame_data: FrameData, label_row: LabelRowV2, # FrameData is automatically received by the agent frame_content: Annotated[NDArray[np.uint8], Depends(dep_single_frame)], ) -> None: # Retrieve the existing human-created caption, prioritizing captions from the current frame or falling back to frame zero. cap, *rs = label_row.ontology_structure.classifications # Read the existing human caption instances = label_row.get_classification_instances( filter_ontology_classification=cap, filter_frames=[0, frame_data.frame] ) if not instances: # nothing to do if there are no human labels return elif len(instances) > 1: def order_by_current_frame_else_frame_0( instance: ClassificationInstance, ) -> bool: try: instance.get_annotation(frame_data.frame) return 2 # The best option except LabelRowError: pass try: instance.get_annotation(0) return 1 except LabelRowError: return 0 instance = sorted(instances, key=order_by_current_frame_else_frame_0)[-1] else: instance = instances[0] # Read the actual string caption caption = instance.get_answer() # Send the first frame of the video along with the human caption to the LLM. frame = Frame(frame=0, content=frame_content) response = prompt_gpt(caption, frame) # Process the response from the LLM, which provides three alternative phrasings of the original caption. # Update the label row with the new captions, replacing any existing ones. # Upsert the new captions for r, t in zip(rs, [response.rephrase_1, response.rephrase_2, response.rephrase_3]): # Overwrite any existing re-captions existing_instances = label_row.get_classification_instances(filter_ontology_classification=r) for existing_instance in existing_instances: label_row.remove_classification(existing_instance) # Create new instances ins = r.create_instance() ins.set_answer(t, attribute=r.attributes[0]) ins.set_for_frames(0) label_row.add_classification_instance(ins) label_row.save() ``` [Click here](https://colab.research.google.com/github/encord-team/encord-agents/blob/main/examples/notebooks/recaption_video.ipynb) for a concrete Vision Language Action model use-case. **This example requires the following dependencies**: ```text theme={"dark"} encord-agents langchain-openai fastapi[standard] openai ``` **To set up and test the agent locally**: 1. Save the dependencies above into a `requirements.txt` file. 2. Set up your Python environment and run the agent: ```shell theme={"dark"} python -m venv venv source venv/bin/activate python -m pip install -r requirements.txt ENCORD_SSH_KEY_FILE=/path/to/your_private_key \ OPENAI_API_KEY= \ fastapi dev main.py ``` *(Replace `/path/to/your_private_key` and `` with your actual credentials.)* 3. In a separate terminal, test the agent: ```shell theme={"dark"} source venv/bin/activate encord-agents test local my_agent ``` *(Replace `` with the URL from your Encord Label Editor session.)* ## PDF OCR Encord Agent The goal is to create a Custom Agent that extracts text from target bounding boxes in a PDF using the Document AI API. This Agent performs the following: 1. Searches for bounding boxes in your PDF that have a Text or OCR text attributes. 2. Rasterizes PDF pages. 3. Crops each bounding box. 4. Sends the crop to [Google Document AI OCR](https://cloud.google.com/document-ai?hl=en). 5. Writes the extracted text back into the attribute on the object. 6. Saves the label row after each batch. **Prerequisites** * Create a virtual Python environment * Install all necessary dependencies * Are able to authenticate with Encord Run the following commands to set up your environment: ```bash theme={"dark"} python -m venv venv # Create a virtual Python environment source venv/bin/activate # Activate the virtual environment ``` **Project Setup** For the Agent to work, the Ontology for your Project must contain a Bounding Box object with a **Text** attribute named `Text` or `OCR`. For example, create an Ontology with the following: * PDF Document Name (bounding box) * Text (text attribute) * Error (bounding box) * OCR (text attribute) * PDF Signature Field (bounding box) * Signatory Name (text attribute) * Status (radio button) * Signed (radio button option) * Unsigned (radio button option)