Register OTC Data (SDK)

1. Set up your cloud storage so Encord can access your data
2. Create a cloud storage integration on Encord to link to your cloud storage
3. Create a JSON or CSV file to import your data
4. Create a Dataset
5. Perform the registration using the JSON or CSV file

​

Step 1: Setup OTC Integration

1. Create the account which accesses data in the Object Storage Service.
2. Give the account read access to the desired buckets by:
   1. Creating a Custom Bucket Policy.
   2. (Optional) If you have Cross-origin resource sharing (CORS) configured on your buckets, make sure that *.encord.com is given read access.
3. Create the integration by giving Encord access to that account’s credentials.

​

Step 2: Create Encord Integration

​

Step 3: Create JSON or CSV for Registration

​

Create JSON file for Registration

​

Create CSV File for Registration

​

Step 4: Register Data with Encord

• Replace <private_key_path> with the path to your private key.
• Replace <integration_title> with the title of the integration you want to use.
• Replace <folder_name> with the folder name. The scripts assume that the specified folder name is unique.
• Replace path/to/json/file.json with the path to a JSON file specifying which cloud storage files should be uploaded.
• Replace A folder to store my files with a meaningful description for your folder.
• Replace "my": "folder_metadata" with any metadata you want to add to the folder.

• “Upload is still in progress, try again later!”: The registration has not finished. Run this script again later to check if the data registration has finished.
• “Upload completed”: The registration completed. If any files failed to upload, the URLs are listed.
• “Upload failed”: The entire registration failed, and not just individual files. Ensure your JSON file is formatted correctly.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import LongPollingStatus  # Ensure correct import path

# Instantiate user client. Replace <private_key_path> with the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Specify the integration you want to use
integrations = user_client.get_cloud_integrations()
integration_idx = [i.title for i in integrations].index("<integration_title>")
integration = integrations[integration_idx].id

# Create a storage folder
folder_name = "<folder_name>"
folder_description = "A folder to store my files"
folder_metadata = {"my": "folder_metadata"}
storage_folder = user_client.create_storage_folder(
    folder_name, folder_description, client_metadata=folder_metadata
)

# Initiate cloud data registration
upload_job_id = storage_folder.add_private_data_to_folder_start(
    integration_id=integration, private_files="path/to/json/file.json", ignore_errors=True
)

# Check upload status
res = storage_folder.add_private_data_to_folder_get_result(upload_job_id, timeout_seconds=5)
print(f"Execution result: {res}")

if res.status == LongPollingStatus.PENDING:
    print("Upload is still in progress, try again later!")
elif res.status == LongPollingStatus.DONE:
    print("Upload completed")
    if res.unit_errors:
        print("The following URLs failed to upload:")
        for e in res.unit_errors:
            print(e.object_urls)
else:
    print(f"Upload failed: {res.errors}")

​

Step 5: Check Registration Status

• “Upload is still in progress, try again later!”: The registration has not finished. Run this script again later to check if the data registration has finished.
• “Upload completed”: The registration completed. If any files failed to upload, the URLs are listed.
• “Upload failed”: The entire registration failed, and not just individual files. Ensure your JSON file is formatted correctly.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import LongPollingStatus

upload_job_id = <upload_job_id>

# Authenticate with Encord using the path to your private key. 
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
    )

# Check upload status
res = dataset.add_private_data_to_dataset_get_result(upload_job_id, timeout_seconds=5)
print(f"Execution result: {res}")

if res.status == LongPollingStatus.PENDING:
    print("Upload is still in progress, try again later!")
elif res.status == LongPollingStatus.DONE:
    print("Upload completed")
    if res.unit_errors:
        print("The following URLs failed to upload:")
        for e in res.unit_errors:
            print(e.object_urls)
else:
    print(f"Upload failed: {res.errors}")

​

Step 6: Create a Dataset

• Substitute <private_key_path> with the file path for your private key.
• Replace “Houses” with the name you want your Dataset to have.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import StorageLocation

# Authenticate with Encord using the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Create a new dataset
dataset = user_client.create_dataset(
    dataset_title="Houses",
    dataset_type=StorageLocation.OTC,
    create_backing_folder=False,
)

# Prints a CreateDatasetResponse object. Verify the Dataset creation
print(dataset_response)

# Print the storage location
print(f"Using storage location: OTC")

​

Step 7: Add Your Data to a Dataset

• Replace <private_key_path> with the path to your private key.
• Replace <folder_name> with the name you want to give your Storage folder.
• Replace <dataset_hash> with the hash of the Dataset you want to add the data units to.

from encord import EncordUserClient

# Authentication
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Find the storage folder by name
folder_name = "<folder_name>"  # Replace with your folder's name
folders = list(user_client.find_storage_folders(search=folder_name, page_size=1))

dataset = user_client.get_dataset("<dataset_hash>")

# Ensure the folder was found
if folders:
    storage_folder = folders[0]

    # List all data units
    items = list(storage_folder.list_items())

    # Collect all item UUIDs
    item_uuids = [item.uuid for item in items]

    # Output the retrieved data units
    for item in items:
        print(f"UUID: {item.uuid}, Name: {item.name}, Type: {item.item_type}")

    # Link all items at once if there are any
    if item_uuids:
        dataset.link_items(item_uuids)
else:
    print("Folder not found.")

​

Step 8: Verify your files are in the Dataset

• Replace <private_key_path> with the path to your private key.
• Replace <dataset_hash> with the hash of your Dataset.

#Import dependencies
from encord import EncordUserClient, Project,Dataset
from encord.objects.project import ProjectDataset
from encord.orm.dataset import DatasetAccessSettings

#Initiate client
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

#Files the file links to all files in the Dataset
dataset_level_file_links = []
dataset: Dataset = user_client.get_dataset("<dataset_hash>")
for data in dataset.list_data_rows():
    dataset_level_file_links.append(data.file_link)
print(dataset_level_file_links)

​

Step 9: Prepare Your Data for Label/Annotation Import

​

Step 10: Import Labels/Annotations