To set up a Benchmark QA Workflow, you need to create two distinct Projects:
  • Benchmark Project: This Project establishes the “ground-truth” labels, which serve as the benchmark for evaluating annotator performance.
  • Production Project: In this Project, annotators generate the production labels. Annotator performance is scored against the ground-truth labels from the first Project.

​
STEP 1: Register Files with Encord

You must first register your files with Encord. This includes files that are used to establish ‘ground-truth’ labels, and your production data.
1

Create a Cloud Integration

Select your cloud provider:
2

Create a Folder to Store your Files

  1. Navigate to Files under the Index heading in the Encord platform.
  2. Click the + New folder button to create a new folder. Select the type of folder you want to create.
  1. Give the folder a meaningful name and description.
  2. Click Create to create the folder. The folder is listed in Files.
3

Create JSON file for Registration

To register files from cloud storage into Encord, you must create a JSON file specifying the files you want to upload.
While you can use a CSV file, we strongly recommend using JSON files for uploading cloud data to Encord for better compatibility and performance.
Find helpful scripts for creating JSON files for the data registration process here.
All types of data (videos, images, image groups, image sequences, and DICOM) from a private cloud are added to a Dataset in the same way, by using a JSON or CSV file. The file includes links to all images, image groups, videos and DICOM files in your cloud storage.
For a list of supported file formats for each data type, go here
Encord supports file names up to 300 characters in length for any file or video for upload.
Encord enforces the following upload limits for each JSON file used for file registration:
  • Up to 1 million URLs
  • A maximum of 500,000 items (e.g. images, image groups, videos, DICOMs)
  • URLs can be up to 16 KB in size
Optimal upload chunking can vary depending on your data type and the amount of associated metadata. For tailored recommendations, contact Encord support. We recommend starting with smaller uploads and gradually increasing the size based on how quickly jobs are processed. Generally, smaller chunks result in faster data reflection within the platform.
BEST PRACTICE: If you want to use Index or Active with your video data, we STRONGLY RECOMMEND using custom metadata (clientMetadata) to specify key frames, custom metadata, and custom embeddings. For more information go here or here for information on using the SDK.
For detailed information about the JSON file format used for import go here.The information provided about each of the following data types is designed to get you up and running as quickly as possible without going too deeply into the why or how. Look at the template for each data type, then the examples, and adjust the examples to suit your needs.
If skip_duplicate_urls is set to true, all object URLs that exactly match existing images/videos in the dataset are skipped.

​
Videos

Video MetadataWhen the videoMetadata flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers.
To guarantee accurate labels, it is crucial that the videoMetadata provided is accurate.
{
  "videos": [
    {
      "objectUrl": "cloud-path-to-your-video-1"
    },
    {
      "objectUrl": "cloud-path-to-your-video-2",
        "videoMetadata": {
          "fps": frames-per-second,
          "duration": duration-in-seconds,
          "width": frame-width,
          "height": frame-height,
          "file_size": file-size-in-bytes,
          "mime_type": "MIME-file-type-extension"
        }
      }
  ],
  "skip_duplicate_urls": true
}

​
Audio Files

The following is an example JSON file for uploading two audio files to Encord.
  • Template: Imports audio files with an Encord title.
  • Audio Metadata: Imports one audio file with the audiometadata flag. When the audiometadata flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. To guarantee accurate labels, it is crucial that the metadata you provide is accurate.
{
  "audio": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-custom-audio-file-title.mp3"
    }
  ],
  "skip_duplicate_urls": true
}

​
PDFs

The following are example JSON files for uploading PDFs to Encord.
{
  "pdfs": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-document-02.pdf"
    },
    {
      "objectUrl": "<object url_3>",
      "title": "my-document-03.pdf",
      "pdfMetadata": {
        "fileSize": 300,
        "numPages": 5
        }
    }
  ],
  "skip_duplicate_urls": true
}

​
Text Files

The following are example JSON files for uploading text files to Encord.
{
  "text": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-file.html"
    },
    {
      "objectUrl": "<object url_3>",
      "title": "my-plain-text-file.txt",
      "textMetadata": {
        "fileSize": 200,
        "mime_type": "text/plain"
      }
    },
    {
      "objectUrl": "<object url_4>",
      "title": "my-csv-file.csv",
      "textMetadata": {
        "fileSize": 250,
        "mime_type": "text/csv"
      }
    },
    {
      "objectUrl": "<object url_5>",
      "title": "my-xml-file.xml",
      "textMetadata": {
        "fileSize": 300,
        "mime_type": "application/xml"
      }
    },
    {
      "objectUrl": "<object url_6>",
      "title": "my-html-file.html",
      "textMetadata": {
        "fileSize": 400,
        "mime_type": "text/html"
      }
    }
  ],
  "skip_duplicate_urls": true
}

​
Single Images

For detailed information about the JSON file format used for import go here.The JSON structure for single images parallels that of videos.Template: Provides the proper JSON format to import images into Encord.Examples:
  • Data Imports the images only.
{
  "images": [
    {
      "objectUrl": "file/path/to/images/file-name-01.file-extension"
    },
    {
      "objectUrl": "file/path/to/images/file-name-02.file-extension"
    },
    {
      "objectUrl": "file/path/to/images/file-name-03.file-extension",
      "title": "image-title.file-extension"
    }
  ],
  "skip_duplicate_urls": true
}

​
Image groups

For detailed information about the JSON file format used for import go here.
  • Image groups are collections of images that are processed as one annotation task.
  • Images within image groups remain unaltered, meaning that images of different sizes and resolutions can form an image group without the loss of data.
  • Image groups do NOT require ‘write’ permissions to your cloud storage.
  • If skip_duplicate_urls is set to true, all URLs exactly matching existing image groups in the dataset are skipped.
The position of each image within the sequence needs to be specified in the key (objectUrl_{position_number}).
Template: Provides the proper JSON format to import image groups into Encord.Examples:
  • Data: Imports the image groups only.
{
  "image_groups": [
    {
      "title": "<title 1>",
      "createVideo": false,
      "objectUrl_0": "file/path/to/images/file-name-01.file-extension",
      "objectUrl_1": "file/path/to/images/file-name-02.file-extension",
      "objectUrl_2": "file/path/to/images/file-name-03.file-extension",
    },
    {
      "title": "<title 2>",
      "createVideo": false,
      "objectUrl_0": "file/path/to/images/file-name-01.file-extension",
      "objectUrl_1": "file/path/to/images/file-name-02.file-extension",
      "objectUrl_2": "file/path/to/images/file-name-03.file-extension"
    }
  ],
  "skip_duplicate_urls": true
}

​
Image sequences

For detailed information about the JSON file format used for import go here.
  • Image sequences are collections of images that are processed as one annotation task and represented as a video.
  • Images within image sequences may be altered as images of varying sizes and resolutions are made to match that of the first image in the sequence.
  • Creating Image sequences from cloud storage requires ‘write’ permissions, as new files have to be created in order to be read as a video.
  • Each object in the image_groups array with the createVideo flag set to true represents a single image sequence.
  • If skip_duplicate_urls is set to true, all URLs exactly matching existing image sequences in the dataset are skipped.
The only difference between adding image groups and image sequences using a JSON file is that image sequences require the createVideo flag to be set to true. Both use the key image_groups.
The position of each image within the sequence needs to be specified in the key (objectUrl_{position_number}).
Encord supports up to 32,767 entries (21:50 minutes) for a single image sequence. We recommend up to 10,000 to 15,000 entries for a single image sequence for best performance. If you need a longer sequence, we recommend using video instead of an image sequence.
Template: Provides the proper JSON format to import image groups into Encord.Examples:
  • Data: Imports the images groups only.
{
  "image_groups": [
    {
      "title": "<title 1>",
      "createVideo": true,
      "objectUrl_0": "<object url>"
    },
    {
      "title": "<title 2>",
      "createVideo": true,
      "objectUrl_0": "<object url>",
      "objectUrl_1": "<object url>",
      "objectUrl_2": "<object url>"
    }
  ],
  "skip_duplicate_urls": true
}

​
DICOM

For detailed information about the JSON file format used for import go here.
  • Each dicom_series element can contain one or more DICOM series.
  • Each series requires a title and at least one object URL, as shown in the example below.
  • If skip_duplicate_urls is set to true, all object URLs exactly matching existing DICOM files in the dataset will be skipped.
Custom metadata is distinct from patient metadata, which is included in the .dcm file and does not have to be specific during the upload to Encord.
The following is an example JSON for uploading three DICOM series belonging to a study. Each title and object URL correspond to individual DICOM series.
  • The first series contains only a single object URL, as it is composed of a single file.
  • The second series contains 3 object URLs, as it is composed of three separate files.
  • The third series contains 2 object URLs, as it is composed of two separate files.
For each DICOM upload, an additional DicomSeries file is created. This file represents the series file-set. Only DicomSeries are displayed in the Encord application.
Template
{
  "dicom_series": [
    {
      "title": "Series-1",
      "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series1-file.dcm"
    },
    {
      "title": "Series-2",
      "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series2-file1.dcm",
      "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series2-file2.dcm",
      "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series2-file3.dcm",
    },
      {
      "title": "Series-3",
      "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series3-file1.dcm",
      "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series3-file2.dcm",
    }
  ],
  "skip_duplicate_urls": true
}

​
NIfTI

The following is an example JSON file for uploading two NIfTI files to Encord.
Template
{
    "nifti": [
      {
        "title": "<file-1>",
        "objectUrl": "https://my-bucket/.../nifti-file1.nii"
      },
      {
        "title": "<file-2>",
        "objectUrl": "https://my-bucket/.../nifti-file2.nii.gz"
      }
    ],
    "skip_duplicate_urls": true
  }
You can upload multiple file types using a single JSON file. The example below shows 1 image, 2 videos, 2 image sequences, and 1 image group.
Multiple file types

{
  "images": [
    {
      "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/Image1.png"
    }
  ],
  "videos": [
    {
      "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/Cooking.mp4"
    },
    {
      "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/Oranges.mp4"
    }
  ],
  "image_groups": [
    {
      "title": "apple-samsung-light",
      "createVideo": true,
      "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(32).jpg",
      "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(33).jpg",
      "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(34).jpg",
      "objectUrl_3": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(35).jpg"
    },
    {
      "title": "apple-samsung-dark",
      "createVideo": true,
      "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/2+(32).jpg",
      "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/2+(33).jpg",
      "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/2+(34).jpg",
      "objectUrl_3": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/2+(35).jpg"
    }
  ],
  "image_groups": [
    {
      "title": "apple-ios-light",
      "createVideo": false,
      "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/3+(32).jpg",
      "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/3+(33).jpg"
    }
  ],
  "skip_duplicate_urls": true
}

​
Videos

Video MetadataWhen the videoMetadata flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers.
To guarantee accurate labels, it is crucial that the videoMetadata provided is accurate.
{
  "videos": [
    {
      "objectUrl": "cloud-path-to-your-video-1"
    },
    {
      "objectUrl": "cloud-path-to-your-video-2",
        "videoMetadata": {
          "fps": frames-per-second,
          "duration": duration-in-seconds,
          "width": frame-width,
          "height": frame-height,
          "file_size": file-size-in-bytes,
          "mime_type": "MIME-file-type-extension"
        }
      }
  ],
  "skip_duplicate_urls": true
}

​
Audio Files

The following is an example JSON file for uploading two audio files to Encord.
  • Example 1 imports audio files with an Encord title.
  • Example 2 imports one audio file with the audiometadata flag. When the audiometadata flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. To guarantee accurate labels, it is crucial that the metadata you provide is accurate.
{
  "audio": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-custom-audio-file-title.mp3"
    }
  ],
  "skip_duplicate_urls": true
}

​
PDFs

The following are example JSON files for uploading PDFs to Encord.
{
  "pdfs": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-document-02.pdf"
    },
    {
      "objectUrl": "<object url_3>",
      "title": "my-document-03.pdf",
      "pdfMetadata": {
        "fileSize": 300,
        "numPages": 5
        }
    }
  ],
  "skip_duplicate_urls": true
}

​
Text Files

The following are example JSON files for uploading text files to Encord.
{
  "text": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-file.html"
    },
    {
      "objectUrl": "<object url_3>",
      "title": "my-plain-text-file.txt",
      "textMetadata": {
        "fileSize": 200,
        "mime_type": "text/plain"
      }
    },
    {
      "objectUrl": "<object url_4>",
      "title": "my-csv-file.csv",
      "textMetadata": {
        "fileSize": 250,
        "mime_type": "text/csv"
      }
    },
    {
      "objectUrl": "<object url_5>",
      "title": "my-xml-file.xml",
      "textMetadata": {
        "fileSize": 300,
        "mime_type": "application/xml"
      }
    },
    {
      "objectUrl": "<object url_6>",
      "title": "my-html-file.html",
      "textMetadata": {
        "fileSize": 400,
        "mime_type": "text/html"
      }
    }
  ],
  "skip_duplicate_urls": true
}

​
Single Images

For detailed information about the JSON file format used for import go here.The JSON structure for single images parallels that of videos.Template: Provides the proper JSON format to import images into Encord.Examples:
  • Data Imports the images only.
  • Image Metadata: Imports images with image metadata. This improves the import speed for your images. 
{
  "images": [
    {
      "objectUrl": "file/path/to/images/file-name-01.file-extension"
    },
    {
      "objectUrl": "file/path/to/images/file-name-02.file-extension"
    },
    {
      "objectUrl": "file/path/to/images/file-name-03.file-extension",
      "title": "image-title.file-extension"
    }
  ],
  "skip_duplicate_urls": true
}

​
Image groups

For detailed information about the JSON file format used for import go here.
  • Image groups are collections of images that are processed as one annotation task.
  • Images within image groups remain unaltered, meaning that images of different sizes and resolutions can form an image group without the loss of data.
  • Image groups do NOT require ‘write’ permissions to your cloud storage.
  • If skip_duplicate_urls is set to true, all URLs exactly matching existing image groups in the dataset are skipped.
The position of each image within the sequence needs to be specified in the key (objectUrl_{position_number}).
Template: Provides the proper JSON format to import image groups into Encord.Examples:
  • Data: Imports the image groups only.
{
  "image_groups": [
    {
      "title": "<title 1>",
      "createVideo": false,
      "objectUrl_0": "file/path/to/images/file-name-01.file-extension",
      "objectUrl_1": "file/path/to/images/file-name-02.file-extension",
      "objectUrl_2": "file/path/to/images/file-name-03.file-extension",
    },
    {
      "title": "<title 2>",
      "createVideo": false,
      "objectUrl_0": "file/path/to/images/file-name-01.file-extension",
      "objectUrl_1": "file/path/to/images/file-name-02.file-extension",
      "objectUrl_2": "file/path/to/images/file-name-03.file-extension"
    }
  ],
  "skip_duplicate_urls": true
}

​
Image sequences

For detailed information about the JSON file format used for import go here.
  • Image sequences are collections of images that are processed as one annotation task and represented as a video.
  • Images within image sequences may be altered as images of varying sizes and resolutions are made to match that of the first image in the sequence.
  • Creating Image sequences from cloud storage requires ‘write’ permissions, as new files have to be created in order to be read as a video.
  • Each object in the image_groups array with the createVideo flag set to true represents a single image sequence.
  • If skip_duplicate_urls is set to true, all URLs exactly matching existing image sequences in the dataset are skipped.
The only difference between adding image groups and image sequences using a JSON file is that image sequences require the createVideo flag to be set to true. Both use the key image_groups.
The position of each image within the sequence needs to be specified in the key (objectUrl_{position_number}).
Encord supports up to 32,767 entries (21:50 minutes) for a single image sequence. We recommend up to 10,000 to 15,000 entries for a single image sequence for best performance. If you need a longer sequence, we recommend using video instead of an image sequence.
Template: Provides the proper JSON format to import image groups into Encord.Examples:
  • Data: Imports the images groups only.
{
  "image_groups": [
    {
      "title": "<title 1>",
      "createVideo": true,
      "objectUrl_0": "<object url>"
    },
    {
      "title": "<title 2>",
      "createVideo": true,
      "objectUrl_0": "<object url>",
      "objectUrl_1": "<object url>",
      "objectUrl_2": "<object url>"
    }
  ],
  "skip_duplicate_urls": true
}

​
DICOM

For detailed information about the JSON file format used for import go here.
  • Each dicom_series element can contain one or more DICOM series.
  • Each series requires a title and at least one object URL, as shown in the example below.
  • If skip_duplicate_urls is set to true, all object URLs exactly matching existing DICOM files in the dataset will be skipped.
Custom metadata is distinct from patient metadata, which is included in the .dcm file and does not have to be specific during the upload to Encord.
The following is an example JSON for uploading three DICOM series belonging to a study. Each title and object URL correspond to individual DICOM series.
  • The first series contains only a single object URL, as it is composed of a single file.
  • The second series contains 3 object URLs, as it is composed of three separate files.
  • The third series contains 2 object URLs, as it is composed of two separate files.
For each DICOM upload, an additional DicomSeries file is created. This file represents the series file-set. Only DicomSeries are displayed in the Encord application.
JSON for DICOM
{
  "dicom_series": [
    {
      "title": "Series-1",
      "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/study1-series1-file.dcm"
    },
    {
      "title": "Series-2",
      "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/study1-series2-file1.dcm",
      "objectUrl_1": "https://storage.cloud.google.com/encord-image-bucket/images/study1-series2-file2.dcm",
      "objectUrl_2": "https://storage.cloud.google.com/encord-image-bucket/images/study1-series2-file3.dcm",
    },
      {
      "title": "Series-3",
      "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/study1-series3-file1.dcm",
      "objectUrl_1": "https://storage.cloud.google.com/encord-image-bucket/images/study1-series3-file2.dcm",
    }
  ],
  "skip_duplicate_urls": true
}

​
NIfTI

The following is an example JSON file for uploading two NIfTI files to Encord.
{
    "nifti": [
      {
        "title": "<file-1>",
        "objectUrl": "https://my-bucket/.../nifti-file1.nii"
      },
      {
        "title": "<file-2>",
        "objectUrl": "https://my-bucket/.../nifti-file2.nii.gz"
      }
    ],
    "skip_duplicate_urls": true
  }
You can upload multiple file types using a single JSON file. The example below shows 1 image, 2 videos, 2 image sequences, and 1 image group.
Multiple file types

{
  "images": [
    {
      "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/images/Image1.png"
    }
  ],
  "videos": [
    {
      "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/videos/Cooking.mp4"
    },
    {
      "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/videos/Oranges.mp4"
    }
  ],
  "image_groups": [
    {
      "title": "apple-samsung-light",
      "createVideo": true,
      "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/1+(32).jpg",
      "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(33).jpg",
      "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(34).jpg",
      "objectUrl_3": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(35).jpg"
    },
    {
      "title": "apple-samsung-dark",
      "createVideo": true,
      "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/2+(32).jpg",
      "objectUrl_1": "https://storage.cloud.google.com/encord-image-bucket/images/2+(33).jpg",
      "objectUrl_2": "https://storage.cloud.google.com/encord-image-bucket/images/2+(34).jpg",
      "objectUrl_3": "https://storage.cloud.google.com/encord-image-bucket/images/2+(35).jpg"
    }
  ],
  "image_groups": [
    {
      "title": "apple-ios-light",
      "createVideo": false,
      "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/3+(32).jpg",
      "objectUrl_1": "https://storage.cloud.google.com/encord-image-bucket/images/3+(33).jpg"
    }
  ],
  "skip_duplicate_urls": true
}

​
Videos

Video MetadataWhen the videoMetadata flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers.
To guarantee accurate labels, it is crucial that the videoMetadata provided is accurate.
{
  "videos": [
    {
      "objectUrl": "cloud-path-to-your-video-1"
    },
    {
      "objectUrl": "cloud-path-to-your-video-2",
        "videoMetadata": {
          "fps": frames-per-second,
          "duration": duration-in-seconds,
          "width": frame-width,
          "height": frame-height,
          "file_size": file-size-in-bytes,
          "mime_type": "MIME-file-type-extension"
        }
      }
    {
      "objectUrl": "cloud-path-to-your-video-3",
      "title": "title-for-your-video-3"

    }
  ],
  "skip_duplicate_urls": true
}

​
Audio Files

The following is an example JSON file for uploading two audio files to Encord.
  • Template: Imports audio files with an Encord title.
  • Audio Metadata: Imports one audio file with the audiometadata flag. When the audiometadata flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. To guarantee accurate labels, it is crucial that the metadata you provide is accurate.
{
  "audio": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-custom-audio-file-title.mp3"
    }
  ],
  "skip_duplicate_urls": true
}

​
PDFs

The following are example JSON files for uploading PDFs to Encord.
{
  "pdfs": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-document-02.pdf"
    },
    {
      "objectUrl": "<object url_3>",
      "title": "my-document-03.pdf",
      "pdfMetadata": {
        "fileSize": 300,
        "numPages": 5
        }
    }
  ],
  "skip_duplicate_urls": true
}

​
Text Files

The following are example JSON files for uploading text files to Encord.
{
  "text": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-file.html"
    },
    {
      "objectUrl": "<object url_3>",
      "title": "my-plain-text-file.txt",
      "textMetadata": {
        "fileSize": 200,
        "mime_type": "text/plain"
      }
    },
    {
      "objectUrl": "<object url_4>",
      "title": "my-csv-file.csv",
      "textMetadata": {
        "fileSize": 250,
        "mime_type": "text/csv"
      }
    },
    {
      "objectUrl": "<object url_5>",
      "title": "my-xml-file.xml",
      "textMetadata": {
        "fileSize": 300,
        "mime_type": "application/xml"
      }
    },
    {
      "objectUrl": "<object url_6>",
      "title": "my-html-file.html",
      "textMetadata": {
        "fileSize": 400,
        "mime_type": "text/html"
      }
    }
  ],
  "skip_duplicate_urls": true
}

​
Single Images

For detailed information about the JSON file format used for import go here.The JSON structure for single images parallels that of videos.Template: Provides the proper JSON format to import images into Encord.Examples:
  • Data Imports the images only.
  • Image Metadata: Imports images with image metadata. This improves the import speed for your images. 
{
  "images": [
    {
      "objectUrl": "file/path/to/images/file-name-01.file-extension"
    },
    {
      "objectUrl": "file/path/to/images/file-name-02.file-extension"
    },
    {
      "objectUrl": "file/path/to/images/file-name-03.file-extension",
      "title": "image-title.file-extension"
    }
  ],
  "skip_duplicate_urls": true
}

​
Image groups

For detailed information about the JSON file format used for import go here.
  • Image groups are collections of images that are processed as one annotation task.
  • Images within image groups remain unaltered, meaning that images of different sizes and resolutions can form an image group without the loss of data.
  • Image groups do NOT require ‘write’ permissions to your cloud storage.
  • If skip_duplicate_urls is set to true, all URLs exactly matching existing image groups in the dataset are skipped.
The position of each image within the sequence needs to be specified in the key (objectUrl_{position_number}).
Template: Provides the proper JSON format to import image groups into Encord.Examples:
  • Data: Imports the image groups only.
{
  "image_groups": [
    {
      "title": "<title 1>",
      "createVideo": false,
      "objectUrl_0": "file/path/to/images/file-name-01.file-extension",
      "objectUrl_1": "file/path/to/images/file-name-02.file-extension",
      "objectUrl_2": "file/path/to/images/file-name-03.file-extension"
    },
    {
      "title": "<title 2>",
      "createVideo": false,
      "objectUrl_0": "file/path/to/images/file-name-01.file-extension",
      "objectUrl_1": "file/path/to/images/file-name-02.file-extension",
      "objectUrl_2": "file/path/to/images/file-name-03.file-extension"
    }
  ],
  "skip_duplicate_urls": true
}

​
Image sequences

For detailed information about the JSON file format used for import go here.
  • Image sequences are collections of images that are processed as one annotation task and represented as a video.
  • Images within image sequences may be altered as images of varying sizes and resolutions are made to match that of the first image in the sequence.
  • Creating Image sequences from cloud storage requires ‘write’ permissions, as new files have to be created in order to be read as a video.
  • Each object in the image_groups array with the createVideo flag set to true represents a single image sequence.
  • If skip_duplicate_urls is set to true, all URLs exactly matching existing image sequences in the dataset are skipped.
The only difference between adding image groups and image sequences using a JSON file is that image sequences require the createVideo flag to be set to true. Both use the key image_groups.
The position of each image within the sequence needs to be specified in the key (objectUrl_{position_number}).
Encord supports up to 32,767 entries (21:50 minutes) for a single image sequence. We recommend up to 10,000 to 15,000 entries for a single image sequence for best performance. If you need a longer sequence, we recommend using video instead of an image sequence.
Template: Provides the proper JSON format to import image groups into Encord.Examples:
  • Data: Imports the images groups only.
{
  "image_groups": [
    {
      "title": "<title 1>",
      "createVideo": true,
      "objectUrl_0": "<object url>"
    },
    {
      "title": "<title 2>",
      "createVideo": true,
      "objectUrl_0": "<object url>",
      "objectUrl_1": "<object url>",
      "objectUrl_2": "<object url>"
    }
  ],
  "skip_duplicate_urls": true
}

​
DICOM

For detailed information about the JSON file format used for import go here.
  • Each dicom_series element can contain one or more DICOM series.
  • Each series requires a title and at least one object URL, as shown in the example below.
  • If skip_duplicate_urls is set to true, all object URLs exactly matching existing DICOM files in the dataset will be skipped.
Custom metadata is distinct from patient metadata, which is included in the .dcm file and does not have to be specific during the upload to Encord.
The following is an example JSON for uploading three DICOM series belonging to a study. Each title and object URL correspond to individual DICOM series.
  • The first series contains only a single object URL, as it is composed of a single file.
  • The second series contains 3 object URLs, as it is composed of three separate files.
  • The third series contains 2 object URLs, as it is composed of two separate files.
For each DICOM upload, an additional DicomSeries file is created. This file represents the series file-set. Only DicomSeries are displayed in the Encord application.
Template
{
  "dicom_series": [
    {
      "title": "Series-1",
      "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/study1-series1-file.dcm"
    },
    {
      "title": "Series-2",
      "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/study1-series2-file1.dcm",
      "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/study1-series2-file2.dcm",
      "objectUrl_2": "https://myaccount.blob.core.windows.net/encordcontainer/study1-series2-file3.dcm",
    },
      {
      "title": "Series-3",
      "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/study1-series3-file1.dcm",
      "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/study1-series3-file2.dcm",
    }
  ],
  "skip_duplicate_urls": true
}

​
NIfTI

The following is an example JSON file for uploading two NIfTI files to Encord.
Template
{
    "nifti": [
      {
        "title": "<file-1>",
        "objectUrl": "https://my-bucket/.../nifti-file1.nii"
      },
      {
        "title": "<file-2>",
        "objectUrl": "https://my-bucket/.../nifti-file2.nii.gz"
      }
    ],
    "skip_duplicate_urls": true
  }
You can upload multiple file types using a single JSON file. The example below shows 1 image, 2 videos, 2 image sequences, and 1 image group.

{
  "images": [
    {
      "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/Image1.png"
    }
  ],
  "videos": [
    {
      "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/Cooking.mp4"
    },
    {
      "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/Oranges.mp4"
    }
  ],
  "image_groups": [
    {
      "title": "apple-samsung-light",
      "createVideo": true,
      "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/1-Samsung-S4-Light+Environment/1+(32).jpg",
      "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/1-Samsung-S4-Light+Environment/1+(33).jpg",
      "objectUrl_2": "https://myaccount.blob.core.windows.net/encordcontainer/1-Samsung-S4-Light+Environment/1+(34).jpg",
      "objectUrl_3": "https://myaccount.blob.core.windows.net/encordcontainer/1-Samsung-S4-Light+Environment/1+(35).jpg"
    },
    {
      "title": "apple-samsung-dark",
      "createVideo": true,
      "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/2-samsung-S4-Dark+Environment/2+(32).jpg",
      "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/2-samsung-S4-Dark+Environment/2+(33).jpg",
      "objectUrl_2": "https://myaccount.blob.core.windows.net/encordcontainer/2-samsung-S4-Dark+Environment/2+(34).jpg",
      "objectUrl_3": "https://myaccount.blob.core.windows.net/encordcontainer/2-samsung-S4-Dark+Environment/2+(35).jpg"
    }
  ],
  "image_groups": [
    {
      "title": "apple-ios-light",
      "createVideo": false,
      "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/3-IOS-4-Light+Environment/3+(32).jpg",
      "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/3-IOS-4-Light+Environment/3+(33).jpg"
    }
  ],
  "skip_duplicate_urls": true
}

​
Videos

Video MetadataWhen the videoMetadata flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers.
To guarantee accurate labels, it is crucial that the videoMetadata provided is accurate.
{
  "videos": [
    {
      "objectUrl": "cloud-path-to-your-video-1"
    },
    {
      "objectUrl": "cloud-path-to-your-video-2",
        "videoMetadata": {
          "fps": frames-per-second,
          "duration": duration-in-seconds,
          "width": frame-width,
          "height": frame-height,
          "file_size": file-size-in-bytes,
          "mime_type": "MIME-file-type-extension"
        }
      }
  ],
  "skip_duplicate_urls": true
}

​
Audio Files

The following is an example JSON file for uploading two audio files to Encord.
  • Template: Imports audio files with an Encord title.
  • Audio Metadata: Imports one audio file with the audiometadata flag. When the audiometadata flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. To guarantee accurate labels, it is crucial that the metadata you provide is accurate.
{
  "audio": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-custom-audio-file-title.mp3"
    }
  ],
  "skip_duplicate_urls": true
}

​
PDFs

The following are example JSON files for uploading PDFs to Encord.
{
  "pdfs": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-document-02.pdf"
    },
    {
      "objectUrl": "<object url_3>",
      "title": "my-document-03.pdf",
      "pdfMetadata": {
        "fileSize": 300,
        "numPages": 5
        }
    }
  ],
  "skip_duplicate_urls": true
}

​
Text Files

The following are example JSON files for uploading text files to Encord.
{
  "text": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-file.html"
    },
    {
      "objectUrl": "<object url_3>",
      "title": "my-plain-text-file.txt",
      "textMetadata": {
        "fileSize": 200,
        "mime_type": "text/plain"
      }
    },
    {
      "objectUrl": "<object url_4>",
      "title": "my-csv-file.csv",
      "textMetadata": {
        "fileSize": 250,
        "mime_type": "text/csv"
      }
    },
    {
      "objectUrl": "<object url_5>",
      "title": "my-xml-file.xml",
      "textMetadata": {
        "fileSize": 300,
        "mime_type": "application/xml"
      }
    },
    {
      "objectUrl": "<object url_6>",
      "title": "my-html-file.html",
      "textMetadata": {
        "fileSize": 400,
        "mime_type": "text/html"
      }
    }
  ],
  "skip_duplicate_urls": true
}

​
Single Images

For detailed information about the JSON file format used for import go here.The JSON structure for single images parallels that of videos.Template: Provides the proper JSON format to import images into Encord.Examples:
  • Data Imports the images only.
  • Image Metadata: Imports images with image metadata. This improves the import speed for your images. 
{
  "images": [
    {
      "objectUrl": "file/path/to/images/file-name-01.file-extension"
    },
    {
      "objectUrl": "file/path/to/images/file-name-02.file-extension"
    },
    {
      "objectUrl": "file/path/to/images/file-name-03.file-extension",
      "title": "image-title.file-extension"
    }
  ],
  "skip_duplicate_urls": true
}

​
Image groups

For detailed information about the JSON file format used for import go here.
  • Image groups are collections of images that are processed as one annotation task.
  • Images within image groups remain unaltered, meaning that images of different sizes and resolutions can form an image group without the loss of data.
  • Image groups do NOT require ‘write’ permissions to your cloud storage.
  • If skip_duplicate_urls is set to true, all URLs exactly matching existing image groups in the dataset are skipped.
The position of each image within the sequence needs to be specified in the key (objectUrl_{position_number}).
Template: Provides the proper JSON format to import image groups into Encord.Examples:
  • Data: Imports the image groups only.
{
  "image_groups": [
    {
      "title": "<title 1>",
      "createVideo": false,
      "objectUrl_0": "file/path/to/images/file-name-01.file-extension",
      "objectUrl_1": "file/path/to/images/file-name-02.file-extension",
      "objectUrl_2": "file/path/to/images/file-name-03.file-extension",
    },
    {
      "title": "<title 2>",
      "createVideo": false,
      "objectUrl_0": "file/path/to/images/file-name-01.file-extension",
      "objectUrl_1": "file/path/to/images/file-name-02.file-extension",
      "objectUrl_2": "file/path/to/images/file-name-03.file-extension",
      "clientMetadata": {"optional": "metadata"}
    }
  ],
  "skip_duplicate_urls": true
}

​
Image sequences

For detailed information about the JSON file format used for import go here.
  • Image sequences are collections of images that are processed as one annotation task and represented as a video.
  • Images within image sequences may be altered as images of varying sizes and resolutions are made to match that of the first image in the sequence.
  • Creating Image sequences from cloud storage requires ‘write’ permissions, as new files have to be created in order to be read as a video.
  • Each object in the image_groups array with the createVideo flag set to true represents a single image sequence.
  • If skip_duplicate_urls is set to true, all URLs exactly matching existing image sequences in the dataset are skipped.
The only difference between adding image groups and image sequences using a JSON file is that image sequences require the createVideo flag to be set to true. Both use the key image_groups.
The position of each image within the sequence needs to be specified in the key (objectUrl_{position_number}).
Encord supports up to 32,767 entries (21:50 minutes) for a single image sequence. We recommend up to 10,000 to 15,000 entries for a single image sequence for best performance. If you need a longer sequence, we recommend using video instead of an image sequence.
Template: Provides the proper JSON format to import image groups into Encord.Examples:
  • Data: Imports the images groups only.
{
  "image_groups": [
    {
      "title": "<title 1>",
      "createVideo": true,
      "objectUrl_0": "<object url>"
    },
    {
      "title": "<title 2>",
      "createVideo": true,
      "objectUrl_0": "<object url>",
      "objectUrl_1": "<object url>",
      "objectUrl_2": "<object url>"
    }
  ],
  "skip_duplicate_urls": true
}

​
DICOM

For detailed information about the JSON file format used for import go here.
  • Each dicom_series element can contain one or more DICOM series.
  • Each series requires a title and at least one object URL, as shown in the example below.
  • If skip_duplicate_urls is set to true, all object URLs exactly matching existing DICOM files in the dataset will be skipped.
Custom metadata is distinct from patient metadata, which is included in the .dcm file and does not have to be specific during the upload to Encord.
The following is an example JSON for uploading three DICOM series belonging to a study. Each title and object URL correspond to individual DICOM series.
  • The first series contains only a single object URL, as it is composed of a single file.
  • The second series contains 3 object URLs, as it is composed of three separate files.
  • The third series contains 2 object URLs, as it is composed of two separate files.
For each DICOM upload, an additional DicomSeries file is created. This file represents the series file-set. Only DicomSeries are displayed in the Encord application.
JSON for DICOM
{
  "dicom_series": [
    {
      "title": "Series-1",
      "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/study1-series1-file.dcm"
    },
    {
      "title": "Series-2",
      "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/study1-series2-file1.dcm",
      "objectUrl_1": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/study1-series2-file2.dcm",
      "objectUrl_2": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/study1-series2-file3.dcm",
    },
      {
      "title": "Series-3",
      "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/study1-series3-file1.dcm",
      "objectUrl_1": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/study1-series3-file2.dcm",
    }
  ],
  "skip_duplicate_urls": true
}

​
NIfTI

The following is an example JSON file for uploading two NIfTI files to Encord.
{
    "nifti": [
      {
        "title": "<file-1>",
        "objectUrl": "https://my-bucket/.../nifti-file1.nii"
      },
      {
        "title": "<file-2>",
        "objectUrl": "https://my-bucket/.../nifti-file2.nii.gz"
      }
    ],
    "skip_duplicate_urls": true
  }
You can upload multiple file types using a single JSON file. The example below shows 1 image, 2 videos, 2 image sequences, and 1 image group.
Multiple file types

{
  "images": [
    {
      "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/Image1.png"
    }
  ],
  "videos": [
    {
      "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/videos/Cooking.mp4"
    },
    {
      "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/videos/Oranges.mp4"
    }
  ],
  "image_groups": [
    {
      "title": "apple-samsung-light",
      "createVideo": true,
      "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/1+(32).jpg",
      "objectUrl_1": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/1+(33).jpg",
      "objectUrl_2": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/1+(34).jpg",
      "objectUrl_3": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/1+(35).jpg"
    },
    {
      "title": "apple-samsung-dark",
      "createVideo": true,
      "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/2+(32).jpg",
      "objectUrl_1": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/2+(33).jpg",
      "objectUrl_2": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/2+(34).jpg",
      "objectUrl_3": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/2+(35).jpg"
    }
  ],
  "image_groups": [
    {
      "title": "apple-ios-light",
      "createVideo": false,
      "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/3+(32).jpg",
      "objectUrl_1": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/3+(33).jpg"
    }
  ],
  "skip_duplicate_urls": true
}
When using a Multi-Region Access Point for your AWS S3 buckets the JSON file has to be slightly different from the examples provided. Instead of an object’s URL, objects are specified using the ARN of the Multi-Region Access Point followed by the object name. The example below shows how video files from a Multi-Region Access Point would be specified.
{
  "videos": [
    {
      "objectUrl": "Multi-Region-Access-Point-ARN + <object name_1>"
    },
    {
      "objectUrl": "Multi-Region-Access-Point-ARN + <object name_2>",
      "title": "my-custom-video-title.mp4"
    }
  ],
  "skip_duplicate_urls": true
}
MRAP Example
{
  "videos": [
    {
      "objectUrl": "https://arn:aws:s3::123123123:accesspoint/frf28frarf9.mrap.s3-accesspoint.amazonaws.com/Videos/2022/video_1.mp4"
    },
    {
      "objectUrl": "https://arn:aws:s3::123123123:accesspoint/frf28frarf9.mrap.s3-accesspoint.amazonaws.com/Videos/2022/video_2.mp4",
      "title": "many-cute-cats.mp4"
    }
  ],
  "skip_duplicate_urls": true
}
4

Import your Files

To ensure smoother uploads and faster completion times, and avoid hitting absolute file limits, we recommend adding smaller batches of data. Limit uploads to 100 videos or up to 1,000 images at a time. You can also create multiple Datasets, all of which can be linked to a single Project. Familiarize yourself with our limits and best practices for data import/registration before adding data to Encord.
  1. Navigate to Files section of Index in the Encord platform.
  2. Click into a Folder.
  3. Click + Upload files. A dialog appears.
  1. Click Import from cloud data.
We recommend turning on the Ignore individual file errors feature. This ensures that individual file errors do not lead to the whole upload process being aborted.
  1. Click Add JSON or CSV files to add a JSON or CSV file specifying cloud data that is to be added.
To ensure smoother uploads and faster completion times, and avoid hitting absolute file limits, we recommend adding smaller batches of data. Limit uploads to 100 videos or up to 1,000 images at a time. You can also create multiple Datasets, all of which can be linked to a single Project. Familiarize yourself with our limits and best practices for data import/registration before adding data to Encord.
  1. Navigate to Files section of Index in the Encord platform.
  2. Click into a Folder.
  3. Click + Upload files. A dialog appears.
  1. Click one of the following:
    • Upload: Upload images, videos, and audio files.
      Upload dialog
    • Batch images as: Upload image batches as image groups or image sequences.
      Batch images as
    • DICOM/NifTi: Upload DICOM or NifTi series.
      DICOM/NifTi
  2. Click Upload after selecting your images or series. Your files upload into the Folder in Encord.

​
STEP 2: Create Benchmark Project

The Benchmark Project establishes ground truth labels.
1

Create a Benchmark Dataset

Create a Dataset containing tasks designed to establish ground truth labels. These files will be used to generate ‘gold-standard’ labels against which annotator performance will be evaluated. Be sure to give the Dataset a clear and descriptive name.
Learn how to create Datasets here
2

Create an Ontology

Create an Ontology to label your data. The same Ontology is used in the Benchmark Project AND the Production Project.
Learn how to create Ontologies here
3

Create a Workflow Template

Create a Workflow template to establish ground truth labels and give it a meaningful name like “Establishing Benchmarks”. The following example template is just one approach; however, the process for creating benchmark labels is flexible, allowing you to choose any Workflow that suits your requirements.
For information on how to create Workflow templates see our documentation here.
4

Create the Benchmark Project

Ensure that you:
  • Attach ONLY the Benchmark Dataset to the Project.
  • Attach the Benchmark Workflow Template to the Project.
  1. In the Encord platform, select Projects under Annotate.
  2. Click the + New annotation project button to create a new Project.
  1. Give the Project a meaningful title and description, for example “Benchmark Labels”.
  2. Click the Attach ontology button and attach the Ontology you created.
  3. Click the Attach dataset button and attach the Dataset you created.
  4. Click the Load from template button to attach the template you created in STEP 2.3.
  1. Click Create project to finish creating the Project. You have now created the Project to establish ground-truth labels.

​
STEP 3: Create Benchmark Labels

Complete the Benchmark Project created in STEP 2 to establish a set of ground truth labels for all data units in the Benchmark Dataset.
To learn how to create annotations, see our documentation here.

​
STEP 4: Create Production Project

Create a Project where your annotation workforce labels data and is evaluated against benchmark labels.
1

Create a Production Dataset

Create a Dataset using your Production data. Give the Dataset a meaningful name and description to distinguish it from the Benchmark Dataset created in STEP 2.
2

Create a Production Workflow Template

Create a Workflow template for labeling production data using Benchmark QA and give it a meaningful name like “Benchmark QA Production Labels”The following Workflow template is an example showing how to set up a Workflow for Benchmark QA.
  • A Task Agent is used to route tasks depending on whether they originates in the Benchmark Dataset or the Production Dataset.
  • A script is will be added to the Consensus block of the Production Workflow to evaluate annotator performance.
3

Create The Production Project

Ensure that you:
  • Attach both the Benchmark Dataset AND the Production Dataset when creating the Production Project.
  • Attach the SAME Ontology you created for the Benchmark Project.
  • Attach the Production Workflow Template to the Project.
  1. In the Encord platform, select Projects under Annotate.
  2. Click the + New annotation project button to create a new Project.
  3. Give the Project a meaningful title and description, for example “Benchmark QA Production Labels”.
  4. Click the Attach ontology button and attach the Ontology you created. Attach the SAME Ontology you created for the Benchmark Project.
  5. Click the Attach dataset button and attach the Benchmark AND the Production Datasets.
  6. Click the Load from template button to attach the “Benchmark QA Production Labels” template you created in STEP 4.2.
  7. Click Create Project to create the Project. You have now created the Project to label production data and evaluate annotators against the benchmark labels.
4

Create and run the SDK script for the Agent node

Create and run the following benchmark_routing.py script to check whether a data unit is part of the Benchmark Dataset, or the Production Dataset.
  • If a task is part of the Benchmark Dataset, the task is routed along the “Yes” pathway and proceeds to the Consensus 1 stage of the Production Project, where annotator performance is evaluated.
  • If the task is not part of the Benchmark Dataset it is routed along the “No” pathway and proceeds to the Annotate 1 stage of the Production Project, where production data is labeled.
Run this script each time new production data is added to the Production Dataset
benchmark_routing.py
# Import dependencies
from encord.user_client import EncordUserClient
from encord.workflow import AgentStage

#Replace <project_hash> with the hash of your Project
PROJECT_HASH = "<project_hash>"

BENCHMARK_DATASET_HASH = "<benchmark_dataset_hash>"

#Replace <private_key_path> with the full path to your private key
SSH_PATH = "<private_key_path>"

# Authenticate using the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
ssh_private_key_path=SSH_PATH
)

# Specify the Project that contains the Task agent.
project = user_client.get_project(PROJECT_HASH)

# Specify the Task Agent
agent_stage = project.workflow.get_stage(name="Benchmark Task?", type_=AgentStage)
benchmark_dataset = user_client.get_dataset(BENCHMARK_DATASET_HASH)
benchmark_data_hashes = {data_row.uid for data_row in benchmark_dataset}

for task in agent_stage.get_tasks():
    if task.data_hash in benchmark_data_hashes:
        task.proceed(pathway_name="YES")
    else: 
        task.proceed(pathway_name="NO")
5

Create a script for the Review & Refine stage

Crete the following sample_evaluation.py script for the Consensus 1 stage in the Production Project. The script compares the annotator’s labels in the Production Project with the ground truth labels established in the Benchmark Project.
All tasks in this stage are rejected and routed to the Archive stage, as they do not constitute production data. The point of the Consensus block is to evaluate annotator performance.
sample_evaluation.py
from encord import EncordUserClient, Project

from encord.workflow import (
    ConsensusReviewStage,
)


user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

training_project_id = "<production_project_hash>"
production_project = user_client.get_project(training_project_id)

groundtruth_project_id = "<groundtruth_project_hash>"
groundtruth_project = user_client.get_project(groundtruth_project_id)


review_stage = production_project.workflow.get_stage(name="Consensus 1 Review", type_=ConsensusReviewStage)
task_data_hashes = [t.data_hash for t in review_stage.get_tasks()]

# download all submitted labels and ground-truth labels
label_rows_to_evaluate = production_project.list_label_rows_v2(
    data_hashes=task_data_hashes, 
    include_all_label_branches=True,
)
with production_project.create_bundle() as bundle:
    for lr in label_rows_to_evaluate:
        lr.initialise_labels(bundle=bundle)

groundtruth_label_rows = groundtruth_project.list_label_rows_v2(data_hashes=task_data_hashes)
with groundtruth_project.create_bundle() as bundle:
    for lr in groundtruth_label_rows:
        lr.initialise_labels(bundle=bundle)


# for each ground-truth label row from the ground-truth project
# 1. grab the corresponding task to be benchmarked
# 2. evaluate the submitted labels against the ground truth, specifying the annotator being evaluated

for gtlr in groundtruth_label_rows:
    # get all label row submissions against the ground truth task
    benchmark_label_rows = [bmlr for bmlr in label_rows_to_evaluate if bmlr.data_hash == gtlr.data_hash]
    target_task = next(review_stage.get_tasks(data_hash=gtlr.data_hash), None)

    if target_task is None:
        print("Could not find corresponding benchmark task in production project")
        continue

    # for each label row to be benchmarked, run the evaluation passing in annotator
    for bmlr in benchmark_label_rows:
        # skip the 'main' branch, it does not yet have any annotator submissions.
        if bmlr.branch_name == "main":
            continue

        annotator = next(task_option.annotator for task_option in target_task.options if task_option.branch_name == bmlr.branch_name)
        print(f"Evaluate task. Data UUID: {gtlr.data_hash}, Label Branch: {bmlr.branch_name}, Annotator: {annotator}")
        # my_eval_function(gtlr, bmlr, gtlr) <- fill in your logic here!

​
STEP 5: Create labels

Once your Production Project is set up, annotators can begin labeling the production data. Tasks from both the Benchmark Dataset and the Production Dataset are assigned to annotators. Their performance is then assessed based on how accurately they label the Benchmark tasks.
To learn how to create annotations, see our documentation here.

​
STEP 6: Evaluate Annotator Performance

Run the sample_evaluation.py script created in STEP 4.5 to evaluate annotator performance.