Register AWS Data (SDK)

Using cloud storage data in Encord is a multi-step process:

1. Set up your cloud storage so Encord can access your data

2. Create a cloud storage integration on Encord to link to your cloud storage

3. Create a Dataset

4. Create a JSON or CSV file to register your data

5. Perform the registration using the JSON or CSV file

​

Step 1: Set up AWS

Before you can do anything with the Encord platform and cloud storage, you need to configure your cloud storage to work with Encord. Once the integration between Encord and your cloud storage is complete, you can then use your data in Encord.

In order to integrate with AWS S3, you need to:

1. Create a permission policy for your resources that allows appropriate access to Encord.
2. Create a role for Encord and attach the policy so that Encord can access those resources.
3. Activate Cross-origin resource sharing which allows Encord to access those resources from a web browser.
4. Test the integration to make sure it works.

You have the following options to integrate AWS and Encord:

​

Step 2: Create AWS-Encord Integration

In the Integrations section of the Encord platform, click +New integration to create a new integration.

Select AWS S3 at the top of the chooser.

​

Step 3: Create JSON or CSV for Registration

All types of data (videos, images, image groups, image sequences, and DICOM) from a private cloud are added to a Dataset in the same way, by using a JSON or CSV file. The file includes links to all images, image groups, videos and DICOM files in your cloud storage.

​

Create JSON file for Registration

For detailed information about the JSON file format used for import go here.

The information provided about each of the following data types is designed to get you up and running as quickly as possible without going too deeply into the why or how. Look at the template for each data type, then the examples, and adjust the examples to suit your needs.

​

Use a Multi-Region Access Point

When using a Multi-Region Access Point for your AWS S3 buckets the JSON file has to be slightly different from the examples provided. Instead of an object’s URL, objects are specified using the ARN of the Multi-Region Access Point followed by the object name. The example below shows how video files from a Multi-Region Access Point would be specified.

{
  "videos": [
    {
      "objectUrl": "Multi-Region-Access-Point-ARN + <object name_1>"
    },
    {
      "objectUrl": "Multi-Region-Access-Point-ARN + <object name_2>",
      "title": "my-custom-video-title.mp4"
    }
  ],
  "skip_duplicate_urls": true
}

{
  "videos": [
    {
      "objectUrl": "https://arn:aws:s3::123123123:accesspoint/frf28frarf9.mrap.s3-accesspoint.amazonaws.com/Videos/2022/video_1.mp4"
    },
    {
      "objectUrl": "https://arn:aws:s3::123123123:accesspoint/frf28frarf9.mrap.s3-accesspoint.amazonaws.com/Videos/2022/video_2.mp4",
      "title": "many-cute-cats.mp4"
    }
  ],
  "skip_duplicate_urls": true
}

​

Create CSV File for Registration

In the CSV file format, the column headers specify which type of data is being uploaded. You can add and single file format at a time, or combine multiple data types in a single CSV file.

Details for each data format are given in the sections below.

​

Step 4: Register Data with Encord

To use your data in Encord, it must be uploaded to the Encord Files storage. Once uploaded, your data can be reused across multiple Projects and contain no labels or annotations themselves. Files stores your data, while Projects store your labels. The following script creates a folder in Files and uses your AWS integration to register data in that folder.

The following script creates a new folder in Files and initiates uploads from AWS. It works for all file types.

Ensure that you:

• Replace <private_key_path> with the path to your private key.
• Replace <integration_title> with the title of the integration you want to use.
• Replace <folder_name> with the folder name. The scripts assume that the specified folder name is unique.
• Replace path/to/json/file.json with the path to a JSON file specifying which cloud storage files should be uploaded.
• Replace A folder to store my files with a meaningful description for your folder.
• Replace "my": "folder_metadata" with any metadata you want to add to the folder.

The script has several possible outputs:

• “Upload is still in progress, try again later!”: The registration has not finished. Run this script again later to check if the data registration has finished.
• “Upload completed”: The registration completed. If any files failed to upload, the URLs are listed.
• “Upload failed”: The entire registration failed, and not just individual files. Ensure your JSON file is formatted correctly.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import LongPollingStatus  # Ensure correct import path

# Instantiate user client. Replace <private_key_path> with the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Specify the integration you want to use
integrations = user_client.get_cloud_integrations()
integration_idx = [i.title for i in integrations].index("<integration_title>")
integration = integrations[integration_idx].id

# Create a storage folder
folder_name = "<folder_name>"
folder_description = "A folder to store my files"
folder_metadata = {"my": "folder_metadata"}
storage_folder = user_client.create_storage_folder(
    folder_name, folder_description, client_metadata=folder_metadata
)

# Initiate cloud data registration
upload_job_id = storage_folder.add_private_data_to_folder_start(
    integration_id=integration, private_files="path/to/json/file.json", ignore_errors=True
)

# Check upload status
res = storage_folder.add_private_data_to_folder_get_result(upload_job_id, timeout_seconds=5)
print(f"Execution result: {res}")

if res.status == LongPollingStatus.PENDING:
    print("Upload is still in progress, try again later!")
elif res.status == LongPollingStatus.DONE:
    print("Upload completed")
    if res.unit_errors:
        print("The following URLs failed to upload:")
        for e in res.unit_errors:
            print(e.object_urls)
else:
    print(f"Upload failed: {res.errors}")

​

Step 5: Check Registration Status

If Step 5 returns "Upload is still in progress, try again later!", run the following code to query the Encord server again. Ensure that you replace <upload_job_id> with the output by the previous code. In the example above upload_job_id=c4026edb-4fw2-40a0-8f05-a1af7f465727.

The script has several possible outputs:

• “Upload is still in progress, try again later!”: The registration has not finished. Run this script again later to check if the data registration has finished.

• “Upload completed”: The registration completed. If any files failed to upload, the URLs are listed.

• “Upload failed”: The entire registration failed, and not just individual files. Ensure your JSON file is formatted correctly.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import LongPollingStatus

upload_job_id = <upload_job_id>

# Authenticate with Encord using the path to your private key. 
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
    )

# Check upload status
res = dataset.add_private_data_to_dataset_get_result(upload_job_id, timeout_seconds=5)
print(f"Execution result: {res}")

if res.status == LongPollingStatus.PENDING:
    print("Upload is still in progress, try again later!")
elif res.status == LongPollingStatus.DONE:
    print("Upload completed")
    if res.unit_errors:
        print("The following URLs failed to upload:")
        for e in res.unit_errors:
            print(e.object_urls)
else:
    print(f"Upload failed: {res.errors}")

​

Step 6: Create a Dataset

The following example creates a Dataset called “Houses” that expects data hosted on AWS S3.

• Substitute <private_key_path> with the file path for your private key.
• Replace “Houses” with the name you want your Dataset to have.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import StorageLocation

# Authenticate with Encord using the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Create a new dataset
dataset_response = user_client.create_dataset(
    dataset_title="Houses",
    dataset_type=StorageLocation.AWS,
    create_backing_folder=False,
)

# Prints a CreateDatasetResponse object. Verify the Dataset creation
print(dataset_response)

# Print the storage location
print(f"Using storage location: AWS")

​

Step 7: Add Your Data to a Dataset

Now that you uploaded your data and created a Dataset, it is time to add your files to the Dataset. The following scripts add all files in a specified folder to a Dataset.

• Replace <private_key_path> with the path to your private key.
• Replace <folder_name> with the name you want to give your Storage folder.
• Replace <dataset_hash> with the hash of the Dataset you want to add the data units to.

from encord import EncordUserClient

# Authentication
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Find the storage folder by name
folder_name = "<folder_name>"  # Replace with your folder's name
folders = list(user_client.find_storage_folders(search=folder_name, page_size=1))

dataset = user_client.get_dataset("<dataset_hash>")

# Ensure the folder was found
if folders:
    storage_folder = folders[0]

    # List all data units
    items = list(storage_folder.list_items())

    # Collect all item UUIDs
    item_uuids = [item.uuid for item in items]

    # Output the retrieved data units
    for item in items:
        print(f"UUID: {item.uuid}, Name: {item.name}, Type: {item.item_type}")

    # Link all items at once if there are any
    if item_uuids:
        dataset.link_items(item_uuids)
else:
    print("Folder not found.")

​

Step 8: Verify Your Files Are in the Dataset

After adding your files to the Dataset, verify that all the files you expect to be there made it into the Dataset.

The following script prints the URLs of all the files in a Dataset. Ensure that you:

• Replace <private_key_path> with the path to your private key.
• Replace <dataset_hash> with the hash of your Dataset.

#Import dependencies
from encord import EncordUserClient, Project,Dataset
from encord.objects.project import ProjectDataset
from encord.orm.dataset import DatasetAccessSettings

#Initiate client
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

#Files the file links to all files in the Dataset
dataset_level_file_links = []
dataset: Dataset = user_client.get_dataset("<dataset_hash>")
for data in dataset.list_data_rows():
    dataset_level_file_links.append(data.file_link)
print(dataset_level_file_links)

​

Step 9: Prepare Your Data for Label/Annotation Import

​

Step 10: Import Labels/Annotations