Your data is registered to the Files section of Index where it is organized into folders and sub-folders.

Registering your data into Encord is a multi-step process:

1. Set up an AWS integration.
2. Create an AWS integration in Encord.
3. Create a JSON or CSV.
4. Create a folder to store your data in Encord.
5. Register your data to the folder.

​

Step 1: Set up AWS

Before you can do anything with the Encord platform and cloud storage, you need to configure your cloud storage to work with Encord. Once the integration between Encord and your cloud storage is complete, you can then use your data in Encord.

In order to integrate with AWS S3, you need to:

1. Create a permission policy for your resources that allows appropriate access to Encord.
2. Create a role for Encord and attach the policy so that Encord can access those resources.
3. Activate Cross-origin resource sharing which allows Encord to access those resources from a web browser.
4. Test the integration to make sure it works.

You have the following options to integrate AWS and Encord:

​

Step 2: Create AWS-Encord Integration

In the Integrations section of the Encord platform, click +New integration to create a new integration.

Select AWS S3 at the top of the chooser.

​

Step 3: Create JSON or CSV for Registration

All types of data (videos, images, image groups, image sequences, and DICOM) from a private cloud are added to a Dataset in the same way, by using a JSON or CSV file. The file includes links to all images, image groups, videos and DICOM files in your cloud storage.

​

For detailed information about the JSON file format used for import go here.

The information provided about each of the following data types is designed to get you up and running as quickly as possible without going too deeply into the why or how. Look at the template for each data type, then the examples, and adjust the examples to suit your needs.

​

Use a Multi-Region Access Point

When using a Multi-Region Access Point for your AWS S3 buckets the JSON file has to be slightly different from the examples provided. Instead of an object’s URL, objects are specified using the ARN of the Multi-Region Access Point followed by the object name. The example below shows how video files from a Multi-Region Access Point would be specified.

{
  "videos": [
    {
      "objectUrl": "Multi-Region-Access-Point-ARN + <object name_1>"
    },
    {
      "objectUrl": "Multi-Region-Access-Point-ARN + <object name_2>",
      "title": "my-custom-video-title.mp4"
    }
  ],
  "skip_duplicate_urls": true
}

{
  "videos": [
    {
      "objectUrl": "https://arn:aws:s3::123123123:accesspoint/frf28frarf9.mrap.s3-accesspoint.amazonaws.com/Videos/2022/video_1.mp4"
    },
    {
      "objectUrl": "https://arn:aws:s3::123123123:accesspoint/frf28frarf9.mrap.s3-accesspoint.amazonaws.com/Videos/2022/video_2.mp4",
      "title": "many-cute-cats.mp4"
    }
  ],
  "skip_duplicate_urls": true
}

​

Create CSV File for Registration

In the CSV file format, the column headers specify which type of data is being uploaded. You can add and single file format at a time, or combine multiple data types in a single CSV file.

Details for each data format are given in the sections below.

​

Step 4: Create a Folder

1. Navigate to Files section of Index in the Encord platform.
2. Click + New folder. A dialog to create a new folder appears.

3. Give the folder a meaningful name and description.
4. Click Create to create the folder. The folder is listed in Files.

​

Step 5: Register Your Data to the Folder

1. Navigate to Files section of Index in the Encord platform.
2. Click + Upload files. A dialog appears.

3. Select the folder you created in step 4.
4. Click the Import from private cloud option.
5. Select the integration you created in step 2 to add your cloud data.

6. Click Add JSON or CSV files to add a JSON or CSV file specifying cloud data that is to be added.