Your data is registered in the Files section of Index where it is organized into folders and sub-folders. Importing your data into Encord is a multi-step process:

  1. Set up an OTC integration.
  2. Create an OTC integration in Encord.
  3. Create a JSON or CSV.
  4. Create a folder to store your data in Encord.
  5. Upload your data to the folder.
See our OTC integration documentation for a detailed guide to setting up an integration.

Step 1: Setup OTC Integration

Before you can do anything with the Encord platform and cloud storage, you need to configure your cloud storage to work with Encord. Once the integration between Encord and your cloud storage is complete, you can then use your data in Encord.

In order to integrate with Open Telecom Cloud, you need to:

  1. Create the account which accesses data in the Object Storage Service.
  2. Give the account read access to the desired buckets by:
    1. Creating a Custom Bucket Policy.
    2. (Optional) If you have Cross-origin resource sharing (CORS) configured on your buckets, make sure that *.encord.com is given read access.
  3. Create the integration by giving Encord access to that account’s credentials.
See our OTC integration documentation for a detailed guide to setting up an OTC integration.

Step 2: Create Encord Integration

On the Encord platform enter the Access key ID and Secret access key, which should be located in the access key file, generated with the creation of the user. (if the access key has been misplaced, a new one can be created from the IAM User menu).

Optionally check the box to enable Strict client-only access, server-side media features will not be available if you would like Encord to sign URLs, but refrain from downloading any media files onto Encord servers. Read more about this feature here.

Finally, click the Create button at the bottom of the pop-up. The integration will now appear in the list of integrations in the ‘Integrations’ tab.

Step 3: Create JSON or CSV for Registration

All types of data (videos, images, image groups, image sequences, and DICOM) from a private cloud are added to a Dataset in the same way, by using a JSON or CSV file. The file includes links to all images, image groups, videos and DICOM files in your cloud storage.

For a list of supported file formats for each data type, go here
Encord supports file names up to 300 characters in length for any file or video for upload.

Encord enforces the following upload limits for each JSON file used for file registration:

  • Up to 1 million URLs
  • A maximum of 500,000 items (e.g. images, image groups, videos, DICOMs)
  • URLs can be up to 16 KB in size

Optimal upload chunking can vary depending on your data type and the amount of associated metadata. For tailored recommendations, contact Encord support. We recommend starting with smaller uploads and gradually increasing the size based on how quickly jobs are processed. Generally, smaller chunks result in faster data reflection within the platform.

BEST PRACTICE: If you want to use Index or Active with your video data, we STRONGLY RECOMMEND using custom metadata (clientMetadata) to specify key frames, custom metadata, and custom embeddings. For more information go here or here for information on using the SDK.

For detailed information about the JSON file format used for import go here.

The information provided about each of the following data types is designed to get you up and running as quickly as possible without going too deeply into the why or how. Look at the template for each data type, then the examples, and adjust the examples to suit your needs.

If skip_duplicate_urls is set to true, all object URLs that exactly match existing images/videos in the dataset are skipped.


Create CSV file for Registration

In the CSV file format, the column headers specify which type of data is being uploaded. You can add and single file format at a time, or combine multiple data types in a single CSV file.

Details for each data format are given in the sections below.

Encord supports up to 10,000 entries for upload in the CSV file.
  • Object URLs can’t contain whitespace.
  • For backwards compatibility reasons, a single column CSV is supported. A file with the single ObjectUrl column is interpreted as a request for video upload. If your objects are of a different type (for example, images), this error displays: “Expected a video, got a file of type XXX”.

Step 4: Create a Folder

  1. Navigate to Files section of Index in the Encord platform.
  2. Click + New folder. A dialog to create a new folder appears.
  1. Give the folder a meaningful name and description.
  2. Click Create to create the folder. The folder is listed in Files.

Step 5: Register your Data in the Folder

To ensure smoother uploads and faster completion times, and avoid hitting absolute file limits, we recommend adding smaller batches of data. Limit uploads to 100 videos or up to 1,000 images at a time. You can also create multiple Datasets, all of which can be linked to a single Project. Familiarize yourself with our limits and best practices for data import/registration before adding data to Encord.
  1. Navigate to Files section of Index in the Encord platform.
  2. Click + Upload files. A dialog appears.
  1. Select the folder you created in step 4.
  2. Click the Import from private cloud option.
  3. Select the integration you created in step 2 to add your cloud data.
We recommend turning on the Ignore individual file errors feature. This ensures that individual file errors do not lead to the whole upload process being aborted.
  1. Click Add JSON or CSV files to add a JSON or CSV file specifying cloud data that is to be added.