Files Upgrade and Migration Guide

Comprehensive documentation for all features in Files can be found here

There are no breaking changes to the data upload or registration process. The current method of uploading, and registering of data continues to be supported.

This guide provides targeted instructions for customers transitioning to the new Files interface with their existing Datasets. We recommend upgrading to Files promptly to benefit from enhanced centralized data handling, flexible management, and improved metadata and permissions control. However, you can continue using your current Datasets without disruption and upgrade at your own pace. Select your preferred data management option below and follow the corresponding steps.

What should I do?

First, what happened to my existing data?

Datasets are now in the Annotate section of the application.

After Files is enabled:

All data on Encord is now managed through Files. Your existing Datasets have been converted into Mirrored Datasets, which automatically track their corresponding folders without displaying them in the Files interface. These Mirrored Datasets are visible in the Datasets section of the Encord platform and can be used just as you have always used Datasets. While this provides a simpler experience, it comes with some reduced flexibility. Mirrored Datasets are identifiable by the M on the Dataset icon.

Before Files is enabled:

After Files is enabled:

Mirrored Datasets can be upgraded to standard Datasets and folders at any time. We highly recommend upgrading your Mirrored Datasets to standard Datasets and folders to fully unlock the benefits of Files.

Choose your data management style

Choose the guide that best fits your needs to get started with Files:

Continue using Encord as you always have

Add ONLY new data to Files

Fully transition to Files

Option 1: I want to keep using Encord as I always have

Your Datasets are now referred to as Mirrored Datasets. The process for adding data to Mirrored Datasets remains unchanged in both the platform and the SDK. There are only minor adjustments to how Mirrored Datasets are created within the Encord platform. SDK users can continue creating Datasets and adding data as they have been accustomed to.

1. Create Mirrored Datasets using the Encord app:

Click the New dataset button in the Datasets section in Annotate.

Give your Dataset a meaningful title and description. A clear title and description keeps your data organized.
Hover over Looking to create a mirrored dataset? and click Yes, proceed to create a Mirrored Dataset.
Click Create dataset to create the Dataset.

Option 2: I want to add new data to Files but leave my existing data unchanged

Your existing Datasets are now Mirrored Datasets, and you can continue adding new files directly to Mirrored Datasets. Mirrored Datasets can be upgraded to standard Datasets and folders at any time.

Upgrading Mirrored Datasets is a one-way action that cannot be undone.

New files are added to, and stored in the Files section of the Encord platform.

Files must be uploaded to folders.

Managing files and folders in the Encord app

Managing files and folders using the Encord SDK

The steps below are the recommended method for uploading files to Encord and attaching them to Datasets. However, you can still upload files directly to Datasets as long as you specify a folder.

1. Creating folders

The following script creates a new folder in the root directory of Files. Ensure that you:

Replace <private_key_path> with the path to your private key.
Replace <folder_name> with the name you want to give your folder. We recommend using unique folder names.
Replace A folder to store my files with a meaningful description for your folder.

from encord import EncordUserClient

# Instantiate Encord client by substituting the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
                ssh_private_key_path="<private_key_path>"
            )

# Create a storage folder
folder_name = "<folder_name>"
folder_description = "A folder to store my files"
storage_folder = user_client.create_storage_folder(folder_name, folder_description)

2. Adding files to folders

The following scripts initiate uploads from your cloud storage to a specified folder in Encord. It works for all file types.

If Upload is still in progress, try again later! is returned, use the script to check the upload status to see whether the upload has finished.

Ensure that you:

Replace <private_key_path> with the path to your private key.
Replace <integration_title> with the title of the integration you want to use.
Replace <folder_name> with the folder name. The scripts assume that the specified folder name is unique.
Replace path/to/json/file.json with the path to a JSON file specifying which cloud storage files should be uploaded.
If creating a new folder, replace A folder to store my files with a meaningful description for your folder.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import LongPollingStatus
from encord.storage import FoldersSortBy

# Instantiate user client. Replace <private_key_path> with the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
    )

# Specify the integration you want to use by replacing <integration_title> with the integration title
integrations = user_client.get_cloud_integrations()
integration_idx = [i.title for i in integrations].index("<integration_title>")
integration = integrations[integration_idx].id

# Find the storage folder by name
folder_name = "<folder_name>"  # Replace with your folder's name
folders = list(user_client.find_storage_folders(search=folder_name, dataset_synced=None, order=FoldersSortBy.NAME, desc=False, page_size=1000))

# Ensure the folder was found
if folders:
    storage_folder = folders[0]

    # Initiate cloud data registration to the storage folder. Replace path/to/json/file.json with the path to your JSON file
    upload_job_id = storage_folder.add_private_data_to_folder_start(
        integration_id=integration, private_files="path/to/json/file.json", ignore_errors=True
    )

    # timeout_seconds determines how long the code will wait after initiating upload until continuing and checking upload status
    res = storage_folder.add_private_data_to_folder_get_result(upload_job_id, timeout_seconds=5)
    print(f"Execution result: {res}")

    if res.status == LongPollingStatus.PENDING:
        print("Upload is still in progress, try again later!")
    elif res.status == LongPollingStatus.DONE:
        print("Upload completed")
        if res.unit_errors:
            print("The following URLs failed to upload:")
            for e in res.unit_errors:
                print(e.object_urls)
    else:
        print(f"Upload failed: {res.errors}")
else:
    print("Folder not found")

3. Create a Dataset

Creating a Dataset and adding files to a Dataset are two distinct steps. Click here to learn how to add data to an existing Dataset.

Datasets cannot be deleted using the SDK or the API. Use the Encord platform to delete Datasets.

The following example creates a Dataset called “Houses” that expects data hosted on AWS S3.

Substitute <private_key_path> with the file path for your private key.
Replace “Houses” with the name you want your Dataset to have.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import StorageLocation

# Authenticate with Encord using the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Create a new dataset
dataset_response = user_client.create_dataset(
    dataset_title="Houses",
    dataset_type=StorageLocation.AWS,
    create_backing_folder=False,
)

# Prints a CreateDatasetResponse object. Verify the Dataset creation
print(dataset_response)

# Print the storage location
print(f"Using storage location: AWS")

4. Attach files to a Dataset

Now that you uploaded your data and created a Dataset, its time to add your files to the Dataset. The following scripts add all files in a specified folder to a Dataset.

Replace <private_key_path> with the path to your private key.
Replace <folder_name> with the name you want to give your Storage folder.
Replace <dataset_hash> with the hash of the Dataset you want to add the data units to.

Files added to the folder at a later time will not be automatically added to the Dataset.

All files

from encord import EncordUserClient

# Authentication
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Find the storage folder by name
folder_name = "<folder_name>"  # Replace with your folder's name
folders = list(user_client.find_storage_folders(search=folder_name, page_size=1))

dataset = user_client.get_dataset("<dataset_hash>")

# Ensure the folder was found
if folders:
    storage_folder = folders[0]

    # List all data units
    items = list(storage_folder.list_items())

    # Collect all item UUIDs
    item_uuids = [item.uuid for item in items]

    # Output the retrieved data units
    for item in items:
        print(f"UUID: {item.uuid}, Name: {item.name}, Type: {item.item_type}")

    # Link all items at once if there are any
    if item_uuids:
        dataset.link_items(item_uuids)
else:
    print("Folder not found.")

Option 3: I want to fully transition to Files

First, upgrade your Mirrored Datasets:

Upgrading Mirrored Datasets is a one-way action that changes how files are added to Datasets. We recommend uploading files through the Files interface and then attaching them to Datasets, as outlined below. However, you can still upload files directly to Datasets by specifying the folder in the upload call, as shown in the SDK reference.

Navigate to Datasets in the Index section of the Encord platform.
Select the Mirrored Dataset you want to upgrade.
Click the Upgrade to standard dataset and folder button.

New files are added to and stored in the Files section of the Encord platform.

Files must be uploaded to folders.

Managing files and folders in the Encord app

Managing files and folders using the Encord SDK

The steps below are the recommended method for uploading files to Encord and attaching them to Datasets. However, you can still upload files directly to Datasets as long as you specify a folder.

1. Creating folders

The following script creates a new folder in the root directory of Files. Ensure that you:

Replace <private_key_path> with the path to your private key.
Replace <folder_name> with the name you want to give your folder. We recommend using unique folder names.
Replace A folder to store my files with a meaningful description for your folder.

from encord import EncordUserClient

# Instantiate Encord client by substituting the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
                ssh_private_key_path="<private_key_path>"
            )



# Create a storage folder
folder_name = "<folder_name>"
folder_description = "A folder to store my files"
storage_folder = user_client.create_storage_folder(folder_name, folder_description)

2. Adding files to folders

The following scripts initiate uploads from your cloud storage to a specified folder in Encord. It works for all file types.

If Upload is still in progress, try again later! is returned, use the script to check the upload status to see whether the upload has finished.

Ensure that you:

Replace <private_key_path> with the path to your private key.
Replace <integration_title> with the title of the integration you want to use.
Replace <folder_name> with the folder name. The scripts assume that the specified folder name is unique.
Replace path/to/json/file.json with the path to a JSON file specifying which cloud storage files should be uploaded.
If creating a new folder, replace A folder to store my files with a meaningful description for your folder.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import LongPollingStatus
from encord.storage import FoldersSortBy

# Instantiate user client. Replace <private_key_path> with the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
    )

# Specify the integration you want to use by replacing <integration_title> with the integration title
integrations = user_client.get_cloud_integrations()
integration_idx = [i.title for i in integrations].index("<integration_title>")
integration = integrations[integration_idx].id

# Find the storage folder by name
folder_name = "<folder_name>"  # Replace with your folder's name
folders = list(user_client.find_storage_folders(search=folder_name, dataset_synced=None, order=FoldersSortBy.NAME, desc=False, page_size=1000))

# Ensure the folder was found
if folders:
    storage_folder = folders[0]

    # Initiate cloud data registration to the storage folder. Replace path/to/json/file.json with the path to your JSON file
    upload_job_id = storage_folder.add_private_data_to_folder_start(
        integration_id=integration, private_files="path/to/json/file.json", ignore_errors=True
    )

    # timeout_seconds determines how long the code will wait after initiating upload until continuing and checking upload status
    res = storage_folder.add_private_data_to_folder_get_result(upload_job_id, timeout_seconds=5)
    print(f"Execution result: {res}")

    if res.status == LongPollingStatus.PENDING:
        print("Upload is still in progress, try again later!")
    elif res.status == LongPollingStatus.DONE:
        print("Upload completed")
        if res.unit_errors:
            print("The following URLs failed to upload:")
            for e in res.unit_errors:
                print(e.object_urls)
    else:
        print(f"Upload failed: {res.errors}")
else:
    print("Folder not found")

3. Create a Dataset

Creating a Dataset and adding files to a Dataset are two distinct steps. Click here to learn how to add data to an existing Dataset.

Datasets cannot be deleted using the SDK or the API. Use the Encord platform to delete Datasets.

The following example creates a Dataset called “Houses” that expects data hosted on AWS S3.

Substitute <private_key_path> with the file path for your private key.
Replace “Houses” with the name you want your Dataset to have.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import StorageLocation

# Authenticate with Encord using the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Create a new dataset
dataset_response = user_client.create_dataset(
    dataset_title="Houses",
    dataset_type=StorageLocation.AWS,
    create_backing_folder=False,
)

# Prints a CreateDatasetResponse object. Verify the Dataset creation
print(dataset_response)

# Print the storage location
print(f"Using storage location: AWS")

4. Attach files to a Dataset

Now that you uploaded your data and created a Dataset, its time to add your files to the Dataset. The following scripts add all files in a specified folder to a Dataset.

Replace <private_key_path> with the path to your private key.
Replace <folder_name> with the name you want to give your Storage folder.
Replace <dataset_hash> with the hash of the Dataset you want to add the data units to.

Files added to the folder at a later time will not be automatically added to the Dataset.

All files

from encord import EncordUserClient

# Authentication
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Find the storage folder by name
folder_name = "<folder_name>"  # Replace with your folder's name
folders = list(user_client.find_storage_folders(search=folder_name, page_size=1))

dataset = user_client.get_dataset("<dataset_hash>")

# Ensure the folder was found
if folders:
    storage_folder = folders[0]

    # List all data units
    items = list(storage_folder.list_items())

    # Collect all item UUIDs
    item_uuids = [item.uuid for item in items]

    # Output the retrieved data units
    for item in items:
        print(f"UUID: {item.uuid}, Name: {item.name}, Type: {item.item_type}")

    # Link all items at once if there are any
    if item_uuids:
        dataset.link_items(item_uuids)
else:
    print("Folder not found.")

Get Started

General

Index

Annotate

Active

Other

Files Upgrade and Migration Guide

What should I do?

First, what happened to my existing data?

Choose your data management style

Option 1: I want to keep using Encord as I always have

Option 2: I want to add new data to Files but leave my existing data unchanged

1. Creating folders

2. Adding files to folders

3. Create a Dataset

4. Attach files to a Dataset

1. Creating folders

2. Adding files to folders

3. Create a Dataset

4. Attach files to a Dataset

Option 3: I want to fully transition to Files

1. Creating folders

2. Adding files to folders

3. Create a Dataset

4. Attach files to a Dataset

1. Creating folders

2. Adding files to folders

3. Create a Dataset

4. Attach files to a Dataset

Get Started

General

Index

Annotate

Active

Other

​What should I do?

​First, what happened to my existing data?

​Choose your data management style

​Option 1: I want to keep using Encord as I always have

​Option 2: I want to add new data to Files but leave my existing data unchanged

​Option 3: I want to fully transition to Files

What should I do?

First, what happened to my existing data?

Choose your data management style

Option 1: I want to keep using Encord as I always have

Option 2: I want to add new data to Files but leave my existing data unchanged

Option 3: I want to fully transition to Files