To create a Benchmark QA Workflow you must create two Projects. The first Project establishes ‘ground-truth’ labels, while the second Project produces production labels.

​
STEP 1: Import Benchmark files into encord

1

Create a Cloud Integration

2

Create a Folder to Store your Files

  1. Navigate to Files under the Index heading in the Encord platform.
  2. Click the + New folder button to create a new folder. A dialog to create a new folder appears.

  1. Give the folder a meaningful name and description.

  2. Click Create to create the folder. The folder is listed in Files.

3

Create JSON or CSV for Import

To import files from cloud storage into Encord, you must create a JSON or CSV file specifying the files you want to upload.

Find helpful scripts for creating JSON and CSV files for the data upload process here.

All types of data (videos, images, image groups, image sequences, and DICOM) from a private cloud are added to a Dataset in the same way, by using a JSON or CSV file. The file includes links to all images, image groups, videos and DICOM files in your cloud storage.

For a list of supported file formats for each data type, go here
Encord supports file names up to 300 characters in length for any file or video for upload.
4

Import your Files

​
STEP 2: Create Benchmark Project

The Benchmark Project establishes ground truth labels.

1

Create a Dataset

  1. Click the New dataset button in the Datasets section in Annotate.
  1. Give your Dataset a meaningful title and description. A clear title and description keeps your data organized.
Toggle Looking to create a mirrored dataset? to create a Mirrored Dataset.
  1. Click Create dataset to create the Dataset.

Attach files

You can add data to Datasets from the Files or the Datasets screens.
We recommend uploading smaller batches of data: limit uploads to 100 videos and up to 1000 images at a time. You have the option to create multiple Datasets, all of which can be linked to a single Project. Familiarize yourself with our limits and best practices for data import before uploading data to Encord.
  1. Navigate to the Datasets section under the Annotate heading.
  2. Click the Dataset you want to attach data to.
  3. Click +Attach existing files.
If the files you want have not been uploaded into Encord yet, click +Upload files to upload new files.

  1. Select the folders containing the files you want to attach to the Dataset. To select individual files, double-click a folder to see its contents, and select the files you want to add to the Dataset.

  2. Click Attach data to attach the selected files to the Dataset.

2

Create an Ontology

We recommend learning about Ontology structure in Encord before creating Ontologies.
  1. Click the New ontology button in the Ontologies section to create a new Ontology.

  1. Give your Ontology a meaningful title and description. A clear title and description keeps your Ontologies organized.

  2. Click Next to continue.

  1. Define your Ontology structure. See our documentation on Ontology structure for more information on the various types of objects, classifications, and attributes.

To add objects:

Object labels are not supported for audio files. Ensure that Ontologies used for audio annotations do not include object labels.
  1. Click Add object to create a new object.
  2. Give the object a name. For example “Apple”.
  3. Select a shape for the object. For example polygon.
  4. Optionally, enable the Required toggle to mark the object as Required.
  5. Optionally, add attributes to the object.
  6. Repeat these steps for as many objects as necessary.

To add attributes to an object:

You can add attributes to objects that define the object’s characteristics. For example the object “Apple” can have an attribute “Color”.

  1. Click the arrow icon next to an object to add attributes to the object.

  2. Give the attribute a name. For example “Color”.

  3. Click the attribute type to change the attribute type. The default attribute type is a text field.

  4. Click Add option to add an option, if you have chosen a radio button or checklist attribute.

  5. Enter a name for the attribute option. For example, the attribute “Color” can have the options “Red”, “Green”, and “Yellow”.

  6. Click the Back to parent button to return the Ontology creation view.

To add a classification:

  1. Click Add classification to create a new classification.
  2. Give the classification a name. For example, “Time of day”.
  3. Optionally, configure the classification. The default classification type is a text field.
  4. Optionally, enable the Required toggle to mark the object as Required.
  5. Repeat these steps for as many classifications as necessary.

Configure classifications:

You can configure classifications to change the classification type, and to add classification options to radio buttons and check lists.

  1. Click the arrow icon next to an object to configure the classification.
  1. Click the classification type to change the classification type. The default classification type is a text field.
  2. Click Add option to add an option if you have selected a radio button or check list classification.
  3. Enter a name for the classification option. For example, the classification “Time of day” can have the options “Night” and “Day”.
  4. Click the Back to parent button to return the Ontology creation view.
3

Create a Workflow Template

Create a Workflow template to establish ground truth labels. and give it a meaningful name like “Establishing Benchmarks”. The following template is an example.

For information on how to create Workflow templates see our documentation here.
4

Create the Benchmark Project

Ensure that you:

  • Attach ONLY the Benchmark Dataset to the Project.
  • Attach the Benchmark Workflow Template to the Project.
  1. In the Encord platform, select Projects under Annotate.
  2. Click the + New annotation project button to create a new Project.
  1. Give the Project a meaningful title and description.

If you are part of an Organization, an optional Project tags drop-down is visible. Project tags are useful for categorizing and finding your Projects. Select as many tags as are relevant for your Project.

  1. Click the Attach ontology button.

  2. Select the Ontology you created previously from the list using the Select button.

  1. Click OK to attach the Ontology to the Project.

  2. Click the Attach datasets button.

  3. Select the Dataset you created previously from the list using the Attach button.

  1. Click OK to attach the Dataset(s) to the Project.
  1. Click the Load from template button to use a Workflow template.

  1. Select the template you want to use and click Load template.

  2. Click Create project to finish creating the Project.

​
STEP 3: Create Benchmark labels

Complete the Benchmark Project created in STEP 2 to establish a set of ground truth labels for specific data units.

To learn how to create annotations, see our documentation here.

​
STEP 4: Create Production Project

Your annotation workforce creates labels in the Production Project.

1

Create a Dataset

Create a Dataset using your Production data. Give the Dataset a meaningful name and description to distinguish it from the Benchmark Dataset created in STEP 2.

2

Create a Workflow

Create a Workflow Template with Benchmark QA.

For information on how to create Workflow templates see our documentation here.
3

Create The Production Project

Ensure that you:

  • Attach both the Benchmark Dataset AND the Production Dataset when creating the Production Project.
  • Attach the SAME Ontology you created for the Benchmark Project.
  • Attach the Production Workflow Template to the Project.
  1. In the Encord platform, select Projects under Annotate.
  2. Click the + New annotation project button to create a new Project.
  1. Give the Project a meaningful title and description.

If you are part of an Organization, an optional Project tags drop-down is visible. Project tags are useful for categorizing and finding your Projects. Select as many tags as are relevant for your Project.

  1. Click the Attach ontology button.

  2. Select the Ontology you created previously from the list using the Select button.

  1. Click OK to attach the Ontology to the Project.

  2. Click the Attach datasets button.

  3. Select the Dataset you created previously from the list using the Attach button.

  1. Click OK to attach the Dataset(s) to the Project.
  1. Click the Load from template button to use a Workflow template.

  1. Select the template you want to use and click Load template.

  2. Click Create project to finish creating the Project.

4

Create and run the SDK script for the Agent node

  1. Create the following script to check whether a data unit is part of the Benchmark Dataset, or the Production Dataset and route it accordingly.
# Import dependencies
from encord.user_client import EncordUserClient
from encord.workflow import AgentStage

#Replace <project_hash> with the hash of your Project
PROJECT_HASH = "<project_hash>"

BENCHMARK_DATASET_HASH = "<benchmark_dataset_hash>"

#Replace <private_key_path> with the full path to your private key
SSH_PATH = "<private_key_path>"

# Authenticate using the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
ssh_private_key_path=SSH_PATH
)

# Specify the Project that contains the Task agent.
project = user_client.get_project(PROJECT_HASH)

# Specify the Task Agent
agent_stage = project.workflow.get_stage(name="Benchmark Task?", type_=AgentStage)
benchmark_dataset = user_client.get_dataset(BENCHMARK_DATASET_HASH)
benchmark_data_hashes = {data_row.uid for data_row in benchmark_dataset}

for task in agent_stage.get_tasks():
    if task.data_hash in benchmark_data_hashes:
        task.proceed(pathway_name="YES")
    else: 
        task.proceed(pathway_name="NO")
5

  • Run the script.
    • 6

    Create a script for the Review & Refine stage

    Crete the following script to compare the annotator’s labels in the Production Project with the ground truth labels established in the Benchmark Project.

    # Import dependencies
from encord import EncordUserClient, Project

from encord.workflow import(
AnnotationStage, 
ReviewStage,
ConsensusAnnotationStage, 
ConsensusReviewStage,
FinalStage
)

#Replace <project_hash> with the hash of your Project
PROJECT_HASH = "<project_hash>"

#Replace <private_key_path> with the full path to your private key
SSH_PATH = "<private_key_path>"

user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Get project
project = user_client.get_project(PROJECT_HASH)

stage = project.workflow.get_stage(name="Consensus 1", type_=ConsensusReviewStage)

# Code for comparing labels goes here

    ​
    STEP 5: Create labels

    With your Production Project set up your production data can be labeled.

    Was this page helpful?

    Pre Classification of Images using GPT 4o