Import Data from AWS (SDK)

Using cloud storage data in Encord is a multi-step process:

Set up your cloud storage so Encord can access your data
Create a cloud storage integration on Encord to link to your cloud storage
Create a Dataset
Create a JSON or CSV file to import your data
Perform the import using the JSON or CSV file

See our AWS integration documentation for a detailed guide to setting up an integration.

Step 1: Set up AWS

Before you can do anything with the Encord platform and cloud storage, you need to configure your cloud storage to work with Encord. Once the integration between Encord and your cloud storage is complete, you can then use your data in Encord.

In order to integrate with AWS S3, you need to:

Create a permission policy for your resources that allows appropriate access to Encord.
Create a role for Encord and attach the policy so that Encord can access those resources.
Activate Cross-origin resource sharing which allows Encord to access those resources from a web browser.
Test the integration to make sure it works.

See our AWS integration documentation for a detailed explanation of setting up AWS to work with Encord.

You have the following options to integrate AWS and Encord:

Use AWS UI

Use Terraform

Step 2: Create AWS-Encord Integration

Create an S3 bucket to store your files if you haven’t already. Your S3 bucket permissions should be set to be blocking all public access.

In the Integrations section of the Encord platform, click +New integration to create a new integration.

Select AWS S3 at the top of the chooser.

It is essential you do not close this tab or window until you have finished the whole integration process. If you use the AWS UI for integration, we advise opening the AWS console in a separate tab.

See our AWS integration documentation for a detailed explanation of how to set up the AWS integration.

Step 3: Create Metadata Schema

If you are not using Index or Active, you do not need to create a Custom Metadata Schema, because you will not be using custom metadata.

Before importing your custom metadata to Encord, we recommend that you import a metadata schema. Encord uses metadata schemas to validate custom metadata uploaded to Encord and to instruct Index and Active how to display your metadata.

To handle your custom metadata schema across multiple teams within the same organization, we recommend using namespacing for metadata keys in the schema. This ensures that different teams can define and manage their own metadata schema without conflicts. For example, team A could use video.description, while team B could use audio.description. Another example could be TeamName.MetadataKey. This approach maintains clarity and avoids key collisions across departments.

Benefits of Using a Metadata Schema

Using a metadata schema provides several benefits:

Validation: Ensures that all custom metadata conforms to predefined data types, reducing errors during data import and processing.
Consistency: Maintains uniformity in data types across different datasets and projects, which simplifies data management and analysis.
Filtering and Sorting: Enhances the ability to filter and sort data efficiently in the Encord platform, enabling more accurate and quick data retrieval.

Metadata Schema Table

Metadata Schema keys support letters (a-z, A-Z), numbers (0-9), and blank spaces ( ), hyphens (-), underscores (_), and periods (.). Metadata schema keys are case sensitive.

If you are unsure about the type to assign to a metadata key, we recommend using varchar as a versatile default.

Use .add_scalar() to add a scalar key to your metadata schema.

Scalar Key	Description	Display Benefits
boolean	Binary data type with values “true” or “false”.	Filtering by binary values
datetime	ISO 8601 formatted date and time.	Filtering by time and date
number	Numeric data type supporting float values.	Filtering by numeric values
uuid	Customer specified unique identifier for a data unit.	Filtering by customer specified unique identifier
varchar	Textual data type. Formally `string`. `string` can be used as an alias for `varchar`, but we STRONGLY RECOMMEND that you use `varchar`.	Filtering by string.
text	Text data with unlimited length (example: transcripts for audio). Formally `long_string`. `long_string` can be used as an alias for `text`, but we STRONGLY RECOMMEND that you use `text`.	Storing and filtering large amounts of text.

Use add_enum and add_enum_options to add an enum and enum options to your metadata schema.

Key	Description	Display Benefits
enum	Enumerated type with predefined set of values.	Facilitates categorical filtering and data validation

Use add_embedding to add an embedding to your metadata schema.

Key	Description	Display Benefits
embedding	1 to 4096 for Index. 1 to 2000 for Active.	Filtering by embeddings, similarity search, 2D scatter plot visualization (Coming Soon)

Incorrectly specifying a data type in the schema can cause errors when filtering your data in Index or Active. If you encounter errors while filtering, verify your schema is correct. If your schema has errors, correct the errors, re-import the schema, and then re-sync your Active Project.

While still supported, we STRONGLY RECOMMEND that you use the latest metadata schema instructions.

ENUM Name	String Representation	Description	Display Benefits
NUMBER	number	Numeric data type supporting float values	Filtering by numeric values
STRING	string	Textual data type	Filtering by string
BOOLEAN	boolean	Binary data type with values “true” or “false”	Filtering by binary values
DATETIME	datetime	ISO 8601 formatted date and time	Filtering by time and date
ENUM	enum	Enumerated type with predefined set of values	Facilitates categorical filtering and data validation
EMBEDDING	embedding	1 to 4096 for Index. 1 to 2000 for Active.	Filtering by embeddings, similarity search, 2D scatter plot visualization (Coming Soon)
LONG_STRING	long_string	Text data with unlimited length (example: transcripts for audio)	Storing and filtering large amounts of text

Import Your Metadata Schema to Encord

Verify Your Schema

After importing your schema to Encord we recommend that you verify that the import is successful. Run the following code to verify your metadata schema imported and that the schema is correct.

Edit Schema Keys

You can change the data type of schema keys using the .set_scalar() method. The example below shows how to update the data type for multiple metadata fields.

Restore Schema Key
# Import dependencies
from encord import EncordUserClient
from encord.metadata_schema import MetadataSchema

SSH_PATH = "/Users/chris-encord/ssh-private-key.txt"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Get your metadata schema
metadata_schema = user_client.metadata_schema()

# Edit various metadata fields
metadata_schema.set_scalar("metadata_1", data_type="number")
metadata_schema.set_scalar("metadata_2", data_type="boolean")
metadata_schema.set_scalar("metadata_3", data_type="boolean")

# Print the schema for verification
print(metadata_schema)

Delete Schema Keys

Schema key options cannot be deleted. Instead, we recommend creating new schema key options to meet your needs and phasing out any that are no longer required.

You can delete schema keys using the .delete() method.

There are two types of deletion: hard delete and soft delete. A hard delete permanently removes the key, making it impossible to restore. A soft delete allows you to restore the key later using the .restore_key() method.

The following examples show hard delete and soft deletion of a schema key called Fruit.

Restore Schema Keys

Keys that have been soft deleted can be restored using the .restore_key() method. The following example restores a schema key called Fruit.

Restore Schema Key
# Import dependencies
from encord import EncordUserClient
from encord.metadata_schema import MetadataSchema

SSH_PATH = "/Users/chris-encord/ssh-private-key.txt"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Get your metadata schema
metadata_schema = user_client.metadata_schema()

metadata_schema.restore_key(k: "Fruit")

# Print the schema for verification
print(metadata_schema)

Step 4: Create JSON or CSV for import

For a list of supported file formats for each data type, go here.

All types of data (videos, images, image groups, image sequences, and DICOM) from a private cloud are added to a Dataset in the same way, by using a JSON or CSV file. The file includes links to all images, image groups, videos and DICOM files in your cloud storage.

For a list of supported file formats for each data type, go here

Encord supports file names up to 300 characters in length for any file or video for upload.

Create JSON file for import

For detailed information about the JSON file format used for import go here.

The information provided about each of the following data types is designed to get you up and running as quickly as possible without going too deeply into the why or how. Look at the template for each data type, then the examples, and adjust the examples to suit your needs.

If skip_duplicate_urls is set to true, all object URLs that exactly match existing images/videos in the dataset are skipped.

Videos

BEST PRACTICE: If you want to use Index or Active with your video data, we STRONGLY RECOMMEND using key frames, custom metadata, and custom embeddings. When specifying key frames set the sampling_rate to 0. This imports only the first frame and any key frames you specify in the video. This can significantly speed up the import of your data into Active and Index and help you to focus on only data you identify as critical.

The following table provides some guidance for the examples provided after the table.

Title	Description
Template	Provides the proper JSON format to import videos into Encord. This template provides examples from the most basic to the most complex.
Data	Imports videos into Encord. Why would I do this? You ONLY want to add labels and classifications to your data. You DO NOT want to use Index or Active.
Key Frames	Imports videos with an Encord title and specifies key frames (frames of interest) for Active and Index. Why would I do this? You ONLY want to see frames that you deem critical in Active and Index. You want to significantly improve the time to import videos into Active and Index. `config` is optional when specifying key frames for Active and Index: Specifying a `sampling_rate` of `0` only imports the first frame and all key frames of your video into Active and Index. `"config": { "sampling_rate": "<samples-per-second>", "keyframe_mode": "frame" or "seconds", },` If `config` is not specified, the `sampling_rate` is 1 frame per second, and the `keyframe_mode` is `frame`.
Custom Metadata	Imports videos with an Encord title, specifies key frames (frames of interest), and custom metadata for Active and Index. Why would I do this? Importing custom metadata allows you to filter your data in Active and Index to make it easier to find the data you want to focus on. This speeds up creating Collections and by extension Datasets. Specifying key frames means you ONLY want to see frames that you deem critical in Active and Index AND you want to significantly improve the time to import videos into Active and Index. `config` is optional when specifying key frames for Active and Index: Specifying a `sampling_rate` of `0` only imports the first frame and all key frames of your video into Active and Index. `"config": { "sampling_rate": "<samples-per-second>", "keyframe_mode": "frame" or "seconds", },` If `config` is not specified, the `sampling_rate` is 1 frame per second, and the `keyframe_mode` is `frame`.
Embeddings	Imports videos with an Encord title, specifies key frames (frames of interest), custom metadata, and custom embeddings for Active and Index. This example includes the following custom metadata types: boolean, varchar, datetime, uuid, number. Why would I do this? Importing custom embeddings allows you to use scatter plots to examine your data AND allows you to use similarity search and natural language searches. Index supports embedding dimensions 1 to 4096, while Active supports embedding dimensions 1 to 2000. Importing custom metadata allows you to filter your data in Active and Index to make it easier to find the data you want to focus on. This speeds up creating Collections and by extension Datasets. Specifying key frames means you ONLY want to see frames that you deem critical in Active and Index AND you want to significantly improve the time to import videos into Active and Index. `config` is optional when specifying custom embeddings for Active and Index: Specifying a `sampling_rate` of `0` only imports the first frame and all key frames of your video into Active and Index. `"config": { "sampling_rate": "<samples-per-second>", "keyframe_mode": "frame" or "seconds", },` If `config` is not specified, the `sampling_rate` is 1 frame per second, and the `keyframe_mode` is `frame`. Refer to our documentation for more information about Index with Custom Metadata, Index with Custom Embeddings, Active with Custom Metadata and Active with Custom Embeddings.
Video Metadata	Imports videos with the videoMetadata flag. When the videoMetadata flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. To guarantee accurate labels, it is crucial that the metadata you provide is accurate.

DICOM

For detailed information about the JSON file format used for import go here.

Each dicom_series element can contain one or more DICOM series.
Each series requires a title and at least one object URL, as shown in the example below.
If skip_duplicate_urls is set to true, all object URLs exactly matching existing DICOM files in the dataset will be skipped.

Custom metadata is distinct from patient metadata, which is included in the .dcm file and does not have to be specific during the upload to Encord.

The following is an example JSON for uploading three DICOM series belonging to a study. Each title and object URL correspond to individual DICOM series.

The first series contains only a single object URL, as it is composed of a single file.
The second series contains 3 object URLs, as it is composed of three separate files.
The third series contains 2 object URLs, as it is composed of two separate files.

For each DICOM upload, an additional DicomSeries file is created. This file represents the series file-set. Only DicomSeries are displayed in the Encord application.

Template
{
  "dicom_series": [
    {
      "title": "Series-1",
      "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series1-file.dcm"
    },
    {
      "title": "Series-2",
      "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series2-file1.dcm",
      "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series2-file2.dcm",
      "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series2-file3.dcm",
    },
      {
      "title": "Series-3",
      "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series3-file1.dcm",
      "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series3-file2.dcm",
    }
  ],
  "skip_duplicate_urls": true
}

NIfTI

The following is an example JSON file for uploading two NIfTI files to Encord.

Template
{
    "nifti": [
      {
        "title": "<file-1>",
        "objectUrl": "https://my-bucket/.../nifti-file1.nii"
      },
      {
        "title": "<file-2>",
        "objectUrl": "https://my-bucket/.../nifti-file2.nii.gz"
      }
    ],
    "skip_duplicate_urls": true
  }

You can upload multiple file types using a single JSON file. The example below shows 1 image, 2 videos, 2 image sequences, and 1 image group.

Multiple file types

{
  "images": [
    {
      "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/Image1.png"
    }
  ],
  "videos": [
    {
      "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/Cooking.mp4"
    },
    {
      "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/Oranges.mp4"
    }
  ],
  "image_groups": [
    {
      "title": "apple-samsung-light",
      "createVideo": true,
      "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(32).jpg",
      "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(33).jpg",
      "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(34).jpg",
      "objectUrl_3": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(35).jpg"
    },
    {
      "title": "apple-samsung-dark",
      "createVideo": true,
      "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/2+(32).jpg",
      "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/2+(33).jpg",
      "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/2+(34).jpg",
      "objectUrl_3": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/2+(35).jpg"
    }
  ],
  "image_groups": [
    {
      "title": "apple-ios-light",
      "createVideo": false,
      "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/3+(32).jpg",
      "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/3+(33).jpg"
    }
  ],
  "skip_duplicate_urls": true
}

Use a Multi-Region Access Point

When using a Multi-Region Access Point for your AWS S3 buckets the JSON file has to be slightly different from the examples provided. Instead of an object’s URL, objects are specified using the ARN of the Multi-Region Access Point followed by the object name. The example below shows how video files from a Multi-Region Access Point would be specified.

{
  "videos": [
    {
      "objectUrl": "Multi-Region-Access-Point-ARN + <object name_1>"
    },
    {
      "objectUrl": "Multi-Region-Access-Point-ARN + <object name_2>",
      "title": "my-custom-video-title.mp4",
      "clientMetadata": {"optional": "metadata"}
    }
  ],
  "skip_duplicate_urls": true
}

MRAP Example
{
  "videos": [
    {
      "objectUrl": "https://arn:aws:s3::123123123:accesspoint/frf28frarf9.mrap.s3-accesspoint.amazonaws.com/Videos/2022/video_1.mp4"
    },
    {
      "objectUrl": "https://arn:aws:s3::123123123:accesspoint/frf28frarf9.mrap.s3-accesspoint.amazonaws.com/Videos/2022/video_2.mp4",
      "title": "many-cute-cats.mp4",
      "clientMetadata": {"optional": "metadata"}
    }
  ],
  "skip_duplicate_urls": true
}

Create CSV file for import

In the CSV file format, the column headers specify which type of data is being uploaded. You can add and single file format at a time, or combine multiple data types in a single CSV file.

Details for each data format are given in the sections below.

Encord supports up to 10,000 entries for upload in the CSV file.

Object URLs can’t contain whitespace.
For backwards compatibility reasons, a single column CSV is supported. A file with the single ObjectUrl column is interpreted as a request for video upload. If your objects are of a different type (for example, images), this error displays: “Expected a video, got a file of type XXX”.

Videos

A CSV file containing videos should contain two columns with the following mandatory column headings:
‘ObjectURL’ and ‘Video title’. All headings are case-insensitive.

The ‘ObjectURL’ column containing the objectUrl. This field is mandatory for each file, as it specifies the full URL of the video resource.
The ‘Video title’ column containing the video_title. If left blank, the original file name is used.

In the example below files 1, 2 and 4 will be assigned the names in the title column, while file 3 will keep its original file name.

ObjectUrl	Video title
path/to/storage-location/frame1.mp4	Video 1
path/to/storage-location/frame2.mp4	Video 2
path/to/storage-location/frame3.mp4
path/to/storage-location/frame4.mp4	Video 3

A CSV file containing single images should contain two columns with the following mandatory headings:
‘ObjectURL’ and ‘Image title’. All headings are case-insensitive.

The ‘ObjectURL’ column containing the objectUrl. This field is mandatory for each file, as it specifies the full URL of the image resource.
The ‘Image title’ column containing the image_title. If left blank, the original file name is used.

In the example below files 1, 2 and 4 will be assigned the names in the title column, while file 3 will keep its original file name.

ObjectUrl	Image title
path/to/storage-location/frame1.jpg	Image 1
path/to/storage-location/frame2.jpg	Image 2
path/to/storage-location/frame3.jpg
path/to/storage-location/frame4.jpg	Image 3

Image groups

A CSV file containing image groups should contain three columns with the following mandatory headings:
‘ObjectURL’, ‘Image group title’, and ‘Create video’. All three headings are case-insensitive.

The ‘ObjectURL’ column containing the objectUrl. This field is mandatory for each file, as it specifies the full URL of the resource.
The ‘Image group title’ column containing the image_group_title. This field is mandatory, as it determines which image group a file will be assigned to.

In the example below the first two URLs are grouped together into ‘Group 1’, while the following two files are grouped together into ‘Group 2’.

ObjectUrl	Image group title	Create video
path/to/storage-location/frame1.jpg	Group 1	false
path/to/storage-location/frame2.jpg	Group 1	false
path/to/storage-location/frame3.jpg	Group 2	false
path/to/storage-location/frame4.jpg	Group 2	false

Image groups do not require ‘write’ permissions.

Image sequences

A CSV file containing image sequences should contain three columns with the following mandatory headings: ‘ObjectURL’, ‘Image group title’, and ‘Create video’. All three headings are case-insensitive.

The ‘ObjectURL’ column containing the objectUrl. This field is mandatory for each file, as it specifies the full URL of the resource.
The ‘Image group title’ column containing the image_group_title. This field is mandatory, as it determines which image sequence a file will be assigned to. The dimensions of the image sequence are determined by the first file in the sequence.
The ‘Create video’ column. This can be left blank, as the default value is ‘true’.

In the example below the first two URLs are grouped together into ‘Sequence 1’, while the second two files are grouped together into ‘Sequence 2’.

ObjectUrl	Image group title	Create video
path/to/storage-location/frame1.jpg	Sequence 1	true
path/to/storage-location/frame2.jpg	Sequence 1	true
path/to/storage-location/frame3.jpg	Sequence 2	true
path/to/storage-location/frame4.jpg	Sequence 2	true

Image groups and image sequences are only distinguished by the presence of the ‘Create video’ column.

Image sequences require ‘write’ permissions against your storage bucket to save the compressed video.

A CSV file containing DICOM files should contain two columns with the following mandatory headings: ‘ObjectURL’ and ‘Dicom title’. Both headings are case-insensitive.

The ‘ObjectURL’ column containing the objectUrl. This field is mandatory for each file, as it specifies the full URL of the resource.
The ‘Series title’ column containing the dicom_title. When two files are given the same title they are grouped into the same DICOM series. If left blank, the original file name is used.

In the example below the first two files are grouped into ‘dicom series 1’, the next two files are grouped into ‘dicom series 2’, while the final file will remain separated as ‘dicom series 3’.

ObjectUrl	Series title
path/to/storage-location/frame1.dcm	dicom series 1
path/to/storage-location/frame2.dcm	dicom series 1
path/to/storage-location/frame3.dcm	dicom series 2
path/to/storage-location/frame4.dcm	dicom series 2
path/to/storage-location/frame5.dcm	dicom series 3

Multiple file types

You can upload multiple file types with a single CSV file by using a new header each time there is a change of file type. Three headings will be required if image sequences are included.

Since the ‘Create video’ column defaults to true all files that are not image sequences must contain the value false

The example below shows a CSV file for the following:

Two image sequences composed of 2 files each.
One image group composed of 2 files.
One single image.
One video.

ObjectUrl	Image group title	Create video
path/to/storage-location/frame1.jpg	Sequence 1	true
path/to/storage-location/frame2.jpg	Sequence 1	true
path/to/storage-location/frame3.jpg	Sequence 2	true
path/to/storage-location/frame4.jpg	Sequence 2	true
path/to/storage-location/frame5.jpg	Group 1	false
path/to/storage-location/frame6.jpg	Group 1	false
ObjectUrl	Image title	Create video
path/to/storage-location/frame1.jpg	Image 1	false
ObjectUrl	Image title	Create video
full/storage/path/video.mp4	Video 1	false

Step 5: Upload data to Encord

To use your data in Encord, it must be uploaded to the Encord Files storage. Once uploaded, your data can be reused across multiple Projects and contain no labels or annotations themselves. Files stores your data, while Projects store your labels. The following script creates a folder in Files and uses your AWS integration to upload data to that folder.

The following script creates a new folder in Files and initiates uploads from AWS. It works for all file types.

If Upload is still in progress, try again later! is returned, use the script to check the upload status to see whether the upload has finished.

Ensure that you:

Replace <private_key_path> with the path to your private key.
Replace <integration_title> with the title of the integration you want to use.
Replace <folder_name> with the folder name. The scripts assume that the specified folder name is unique.
Replace path/to/json/file.json with the path to a JSON file specifying which cloud storage files should be uploaded.
Replace A folder to store my files with a meaningful description for your folder.
Replace "my": "folder_metadata" with any metadata you want to add to the folder.

The script has several possible outputs:

“Upload is still in progress, try again later!”: The upload has not finished. Run this script again later to check if the upload has finished.
“Upload completed”: The upload completed. If any files failed to upload, the URLs are listed.
“Upload failed”: The entire upload failed, and not just individual files. Ensure your JSON file is formatted correctly.

Step 6: Check data upload

If Step 5 returns "Upload is still in progress, try again later!", run the following code to query the Encord server again. Ensure that you replace <upload_job_id> with the output by the previous code. In the example above upload_job_id=c4026edb-4fw2-40a0-8f05-a1af7f465727.

The script has several possible outputs:

“Upload is still in progress, try again later!”: The upload has not finished. Run this script again later to check if the upload has finished.
“Upload completed”: The upload completed. If any files failed to upload, the URLs are listed.
“Upload failed”: The entire upload failed, and not just individual files. Ensure your JSON file is formatted correctly.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import LongPollingStatus

upload_job_id = <upload_job_id>

# Authenticate with Encord using the path to your private key. 
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
    )

# Check upload status
res = dataset.add_private_data_to_dataset_get_result(upload_job_id, timeout_seconds=5)
print(f"Execution result: {res}")

if res.status == LongPollingStatus.PENDING:
    print("Upload is still in progress, try again later!")
elif res.status == LongPollingStatus.DONE:
    print("Upload completed")
    if res.unit_errors:
        print("The following URLs failed to upload:")
        for e in res.unit_errors:
            print(e.object_urls)
else:
    print(f"Upload failed: {res.errors}")

Omitting the timeout_seconds argument from the add_private_data_to_dataset_get_result() method performs status checks until the status upload has finished.

Step 7: Create a Dataset

Creating a Dataset and adding files to a Dataset are two distinct steps. Click here to learn how to add data to an existing Dataset.

Datasets cannot be deleted using the SDK or the API. Use the Encord platform to delete Datasets.

The following example creates a Dataset called “Houses” that expects data hosted on AWS S3.

Substitute <private_key_path> with the file path for your private key.
Replace “Houses” with the name you want your Dataset to have.

Storage location	StorageLocation method argument	Represented by
AWS S3	AWS	1
GCP	GCP	2
Azure blob	AZURE	3
Open Telekom Cloud	OTC	4
Encord storage	CORD_STORAGE	0

Step 8: Add your data to a Dataset

Now that you uploaded your data and created a Dataset, it is time to add your files to the Dataset. The following scripts add all files in a specified folder to a Dataset.

Replace <private_key_path> with the path to your private key.
Replace <folder_name> with the name you want to give your Storage folder.
Replace <dataset_hash> with the hash of the Dataset you want to add the data units to.

Files added to the folder at a later time will not be automatically added to the Dataset.

All files
from encord import EncordUserClient

# Authentication
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Find the storage folder by name
folder_name = "<folder_name>"  # Replace with your folder's name
folders = list(user_client.find_storage_folders(search=folder_name, page_size=1))

dataset = user_client.get_dataset("<dataset_hash>")

# Ensure the folder was found
if folders:
    storage_folder = folders[0]

    # List all data units
    items = list(storage_folder.list_items())

    # Collect all item UUIDs
    item_uuids = [item.uuid for item in items]

    # Output the retrieved data units
    for item in items:
        print(f"UUID: {item.uuid}, Name: {item.name}, Type: {item.item_type}")

    # Link all items at once if there are any
    if item_uuids:
        dataset.link_items(item_uuids)
else:
    print("Folder not found.")

Step 9: Verify your files are in the Dataset

After adding your files to the Dataset, verify that all the files you expect to be there made it into the Dataset.

The following script prints the URLs of all the files in a Dataset. Ensure that you:

Replace <private_key_path> with the path to your private key.
Replace <dataset_hash> with the hash of your Dataset.

Sample Code
#Import dependencies
from encord import EncordUserClient, Project,Dataset
from encord.objects.project import ProjectDataset
from encord.orm.dataset import DatasetAccessSettings

#Initiate client
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

#Files the file links to all files in the Dataset
dataset_level_file_links = []
dataset: Dataset = user_client.get_dataset("<dataset_hash>")
for data in dataset.list_data_rows():
    dataset_level_file_links.append(data.file_link)
print(dataset_level_file_links)

Step 10: Prepare your data for label/annotation import

Prepare Data for Labels

Step 11: Import labels/annotations

Import Labels/Annotations

Was this page helpful?

1. Prerequisites and Installation Import Data from GCP (SDK)

On this page

Step 1: Set up AWS
Step 2: Create AWS-Encord Integration
Step 3: Create Metadata Schema
Benefits of Using a Metadata Schema
Metadata Schema Table
Import Your Metadata Schema to Encord
Verify Your Schema
Edit Schema Keys
Delete Schema Keys
Restore Schema Keys
Step 4: Create JSON or CSV for import
Create JSON file for import
Use a Multi-Region Access Point
Create CSV file for import
Step 5: Upload data to Encord
Step 6: Check data upload
Step 7: Create a Dataset
Step 8: Add your data to a Dataset
Step 9: Verify your files are in the Dataset
Step 10: Prepare your data for label/annotation import
Step 11: Import labels/annotations

Get Started

General

SDK Ref

Index

Ontologies

Projects

Labels

Datasets

DICOM

Active API & SDK

​Step 1: Set up AWS

​Step 2: Create AWS-Encord Integration

​Step 3: Create Metadata Schema

​Benefits of Using a Metadata Schema

​Metadata Schema Table

​Import Your Metadata Schema to Encord

​Verify Your Schema

​Edit Schema Keys

​Delete Schema Keys

​Restore Schema Keys

​Step 4: Create JSON or CSV for import

​Create JSON file for import

​Use a Multi-Region Access Point

​Create CSV file for import

Videos

Image groups

Image sequences

Multiple file types

​Step 5: Upload data to Encord

​Step 6: Check data upload

​Step 7: Create a Dataset

​Step 8: Add your data to a Dataset

​Step 9: Verify your files are in the Dataset

​Step 10: Prepare your data for label/annotation import

​Step 11: Import labels/annotations

Step 1: Set up AWS

Step 2: Create AWS-Encord Integration

Step 3: Create Metadata Schema

Benefits of Using a Metadata Schema

Metadata Schema Table

Import Your Metadata Schema to Encord

Verify Your Schema

Edit Schema Keys

Delete Schema Keys

Restore Schema Keys

Step 4: Create JSON or CSV for import

Create JSON file for import

Use a Multi-Region Access Point

Create CSV file for import

Step 5: Upload data to Encord

Step 6: Check data upload

Step 7: Create a Dataset

Step 8: Add your data to a Dataset

Step 9: Verify your files are in the Dataset

Step 10: Prepare your data for label/annotation import

Step 11: Import labels/annotations