Your data is uploaded to, and securely stored in the Files section of Index where it is organized into folders and sub-folders. Importing your data into Encord is a multi-step process:

  1. Set up an AWS integration.
  2. Create an AWS integration in Encord.
  3. Create a JSON or CSV.
  4. Create a folder to store your data in Encord.
  5. Upload your data to the folder.
See our AWS integration documentation for a detailed guide to setting up an integration.

Step 1: Set up AWS

Before you can do anything with the Encord platform and cloud storage, you need to configure your cloud storage to work with Encord. Once the integration between Encord and your cloud storage is complete, you can then use your data in Encord.

In order to integrate with AWS S3, you need to:

  1. Create a permission policy for your resources that allows appropriate access to Encord.
  2. Create a role for Encord and attach the policy so that Encord can access those resources.
  3. Activate Cross-origin resource sharing which allows Encord to access those resources from a web browser.
  4. Test the integration to make sure it works.
See our AWS integration documentation for a detailed explanation of setting up AWS to work with Encord.

You have the following options to integrate AWS and Encord:

Step 2: Create AWS-Encord Integration

Create an S3 bucket to store your files if you haven’t already. Your S3 bucket permissions should be set to be blocking all public access.

In the Integrations section of the Encord platform, click +New integration to create a new integration.

Select AWS S3 at the top of the chooser.

It is essential you do not close this tab or window until you have finished the whole integration process. If you use the AWS UI for integration, we advise opening the AWS console in a separate tab.
See our AWS integration documentation for a detailed explanation of how to set up the AWS integration.

Step 3: Create Metadata Schema

Metadata schema

If you are not using Index or Active, you do not need to create a Custom Metadata Schema, because you will not be using custom metadata.

Before importing your custom metadata to Encord, we recommend that you import a metadata schema. Encord uses metadata schemas to validate custom metadata uploaded to Encord and to instruct Index and Active how to display your metadata.

To handle your custom metadata schema across multiple teams within the same organization, we recommend using namespacing for metadata keys in the schema. This ensures that different teams can define and manage their own metadata schema without conflicts. For example, team A could use video.description, while team B could use audio.description. Another example could be TeamName.MetadataKey. This approach maintains clarity and avoids key collisions across departments.

Benefits of Using a Metadata Schema

Using a metadata schema provides several benefits:

  • Validation: Ensures that all custom metadata conforms to predefined data types, reducing errors during data import and processing.
  • Consistency: Maintains uniformity in data types across different datasets and projects, which simplifies data management and analysis.
  • Filtering and Sorting: Enhances the ability to filter and sort data efficiently in the Encord platform, enabling more accurate and quick data retrieval.

Metadata Schema Table

Metadata Schema keys support letters (a-z, A-Z), numbers (0-9), and blank spaces ( ), hyphens (-), underscores (_), and periods (.). Metadata schema keys are case sensitive.

Use add_scalar to add a scalar key to your metadata schema.

Scalar KeyDescriptionDisplay Benefits
booleanBinary data type with values “true” or “false”.Filtering by binary values
datetimeISO 8601 formatted date and time.Filtering by time and date
numberNumeric data type supporting float values.Filtering by numeric values
uuidCustomer specified unique identifier for a data unit.Filtering by customer specified unique identifier
varcharTextual data type. Formally string. string can be used as an alias for varchar, but we STRONGLY RECOMMEND that you use varchar.Filtering by string.
textText data with unlimited length (example: transcripts for audio). Formally long_string. long_string can be used as an alias for text, but we STRONGLY RECOMMEND that you use text.Storing and filtering large amounts of text.

Use add_enum and add_enum_options to add an enum and enum options to your metadata schema.

KeyDescriptionDisplay Benefits
enumEnumerated type with predefined set of values.Facilitates categorical filtering and data validation

Use add_embedding to add an embedding to your metadata schema.

KeyDescriptionDisplay Benefits
embedding1 to 4096 for Index. 1 to 2000 for Active.Filtering by embeddings, similarity search, 2D scatter plot visualization (Coming Soon)

Incorrectly specifying a data type in the schema can cause errors when filtering your data in Index or Active. If you encounter errors while filtering, verify your schema is correct. If your schema has errors, correct the errors, re-import the schema, and then re-sync your Active Project.

Step 4: Create JSON or CSV for import

All types of data (videos, images, image groups, image sequences, and DICOM) from a private cloud are added to a Dataset in the same way, by using a JSON or CSV file. The file includes links to all images, image groups, videos and DICOM files in your cloud storage.

For a list of supported file formats for each data type, go here
Encord supports file names up to 300 characters in length for any file or video for upload.

Create JSON file for import

For detailed information about the JSON file format used for import go here.

The information provided about each of the following data types is designed to get you up and running as quickly as possible without going too deeply into the why or how. Look at the template for each data type, then the examples, and adjust the examples to suit your needs.

If skip_duplicate_urls is set to true, all object URLs that exactly match existing images/videos in the dataset are skipped.


Use a Multi-Region Access Point

When using a Multi-Region Access Point for your AWS S3 buckets the JSON file has to be slightly different from the examples provided. Instead of an object’s URL, objects are specified using the ARN of the Multi-Region Access Point followed by the object name. The example below shows how video files from a Multi-Region Access Point would be specified.

{
  "videos": [
    {
      "objectUrl": "Multi-Region-Access-Point-ARN + <object name_1>"
    },
    {
      "objectUrl": "Multi-Region-Access-Point-ARN + <object name_2>",
      "title": "my-custom-video-title.mp4",
      "clientMetadata": {"optional": "metadata"}
    }
  ],
  "skip_duplicate_urls": true
}

Create CSV file for import

In the CSV file format, the column headers specify which type of data is being uploaded. You can add and single file format at a time, or combine multiple data types in a single CSV file.

Details for each data format are given in the sections below.

Encord supports up to 10,000 entries for upload in the CSV file.
  • Object URLs can’t contain whitespace.
  • For backwards compatibility reasons, a single column CSV is supported. A file with the single ObjectUrl column is interpreted as a request for video upload. If your objects are of a different type (for example, images), this error displays: “Expected a video, got a file of type XXX”.

Step 5: Create a folder

  1. Navigate to Files section of Index in the Encord platform.
  2. Click + New folder. A dialog to create a new folder appears.
  1. Give the folder a meaningful name and description.
  2. Click Create to create the folder. The folder is listed in Files.

Step 6: Upload your data to the folder

We recommend uploading smaller batches of data: limit uploads to 100 videos and up to 1000 images at a time. Familiarize yourself with our limits and best practices for data import before uploading data to Encord.
  1. Navigate to Files section of Index in the Encord platform.
  2. Click + Upload files. A dialog appears.
  1. Select the folder you created in step 4.
  2. Click the Import from private cloud option.
  3. Select the integration you created in step 2 to add your cloud data.
We recommend turning on the Ignore individual file errors feature. This ensures that individual file errors do not lead to the whole upload process being aborted.
  1. Click Add JSON or CSV files to add a JSON or CSV file specifying cloud data that is to be added.