Files Upgrade and Migration Guide

You can upload files to Encord in the Datasets screen and in the Files screen.

1. Creating folders

1. Navigate to Files under the Index heading in the Encord platform.
2. Click the + New folder button to create a new folder. A dialog to create a new folder appears.

3. Give the folder a meaningful name and description.

4. Click Create to create the folder. The folder is listed in Files.

2. Adding files to folders

1. Navigate to Files under the Index heading in the Encord platform.
2. Click the +Upload files button.

3. Select a folder to upload your files to. You can click +New folder to create a new folder.
4. Select the type of data you want to add.

5. Click Import to upload the files to Encord.

3. Create a Dataset

To take advantage of all of the Files functionality you must create a standard Dataset (as opposed to a Mirrored Dataset).

1. Click the New dataset button in the Datasets section in Annotate.

2. Give your Dataset a meaningful title and description. A clear title and description keeps your data organized.

3. Click Create dataset to create the Dataset.

4. Attach files to a Dataset

1. Navigate to the Datasets section under the Annotate heading.
2. Click the Dataset you want to add data to.
3. Click +Attach existing files.

4. Select the folders containing the files you want to attach to the Dataset. To select individual files, double-click a folder to see its contents, and select the files you want to add to the Dataset.

5. Click Attach data to attach the selected files to the Dataset.

1. Creating folders

The following script creates a new folder in the root directory of Files. Ensure that you:

• Replace <private_key_path> with the path to your private key.
• Replace <folder_name> with the name you want to give your folder. We recommend using unique folder names.
• Replace A folder to store my files with a meaningful description for your folder.

2. Adding files to folders

The following scripts initiate uploads from your cloud storage to a specified folder in Encord. It works for all file types.

Ensure that you:

• Replace <private_key_path> with the path to your private key.
• Replace <integration_title> with the title of the integration you want to use.
• Replace <folder_name> with the folder name. The scripts assume that the specified folder name is unique.
• Replace path/to/json/file.json with the path to a JSON file specifying which cloud storage files should be uploaded.
• If creating a new folder, replace A folder to store my files with a meaningful description for your folder.

3. Create a Dataset

The following example creates a Dataset called “Houses” that expects data hosted on AWS S3.

• Substitute <private_key_path> with the file path for your private key.
• Replace “Houses” with the name you want your Dataset to have.

4. Attach files to a Dataset

Now that you uploaded your data and created a Dataset, its time to add your files to the Dataset. The following scripts add all files in a specified folder to a Dataset.

• Replace <private_key_path> with the path to your private key.
• Replace <folder_name> with the name you want to give your Storage folder.
• Replace <dataset_hash> with the hash of the Dataset you want to add the data units to.

from encord import EncordUserClient

# Authentication
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Find the storage folder by name
folder_name = "<folder_name>"  # Replace with your folder's name
folders = list(user_client.find_storage_folders(search=folder_name, page_size=1))

dataset = user_client.get_dataset("<dataset_hash>")

# Ensure the folder was found
if folders:
    storage_folder = folders[0]

    # List all data units
    items = list(storage_folder.list_items())

    # Collect all item UUIDs
    item_uuids = [item.uuid for item in items]

    # Output the retrieved data units
    for item in items:
        print(f"UUID: {item.uuid}, Name: {item.name}, Type: {item.item_type}")

    # Link all items at once if there are any
    if item_uuids:
        dataset.link_items(item_uuids)
else:
    print("Folder not found.")

You can upload files to Encord in the Datasets screen and in the Files screen.

1. Creating folders

1. Navigate to Files under the Index heading in the Encord platform.
2. Click the + New folder button to create a new folder. A dialog to create a new folder appears.

3. Give the folder a meaningful name and description.

4. Click Create to create the folder. The folder is listed in Files.

2. Adding files to folders

1. Navigate to Files under the Index heading in the Encord platform.
2. Click the +Upload files button.

3. Select a folder to upload your files to. You can click +New folder to create a new folder.
4. Select the type of data you want to add.

5. Click Import to upload the files to Encord.

3. Create a Dataset

To take advantage of all of the Files functionality you must create a standard Dataset (as opposed to a Mirrored Dataset).

1. Click the New dataset button in the Datasets section in Annotate.

2. Give your Dataset a meaningful title and description. A clear title and description keeps your data organized.

3. Click Create dataset to create the Dataset.

4. Attach files to a Dataset

1. Navigate to the Datasets section under the Annotate heading.
2. Click the Dataset you want to add data to.
3. Click +Attach existing files.

4. Select the folders containing the files you want to attach to the Dataset. To select individual files, double-click a folder to see its contents, and select the files you want to add to the Dataset.

5. Click Attach data to attach the selected files to the Dataset.

1. Creating folders

The following script creates a new folder in the root directory of Files. Ensure that you:

• Replace <private_key_path> with the path to your private key.
• Replace <folder_name> with the name you want to give your folder. We recommend using unique folder names.
• Replace A folder to store my files with a meaningful description for your folder.

2. Adding files to folders

The following scripts initiate uploads from your cloud storage to a specified folder in Encord. It works for all file types.

Ensure that you:

• Replace <private_key_path> with the path to your private key.
• Replace <integration_title> with the title of the integration you want to use.
• Replace <folder_name> with the folder name. The scripts assume that the specified folder name is unique.
• Replace path/to/json/file.json with the path to a JSON file specifying which cloud storage files should be uploaded.
• If creating a new folder, replace A folder to store my files with a meaningful description for your folder.

3. Create a Dataset

The following example creates a Dataset called “Houses” that expects data hosted on AWS S3.

• Substitute <private_key_path> with the file path for your private key.
• Replace “Houses” with the name you want your Dataset to have.

4. Attach files to a Dataset

Now that you uploaded your data and created a Dataset, its time to add your files to the Dataset. The following scripts add all files in a specified folder to a Dataset.

• Replace <private_key_path> with the path to your private key.
• Replace <folder_name> with the name you want to give your Storage folder.
• Replace <dataset_hash> with the hash of the Dataset you want to add the data units to.

from encord import EncordUserClient

# Authentication
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Find the storage folder by name
folder_name = "<folder_name>"  # Replace with your folder's name
folders = list(user_client.find_storage_folders(search=folder_name, page_size=1))

dataset = user_client.get_dataset("<dataset_hash>")

# Ensure the folder was found
if folders:
    storage_folder = folders[0]

    # List all data units
    items = list(storage_folder.list_items())

    # Collect all item UUIDs
    item_uuids = [item.uuid for item in items]

    # Output the retrieved data units
    for item in items:
        print(f"UUID: {item.uuid}, Name: {item.name}, Type: {item.item_type}")

    # Link all items at once if there are any
    if item_uuids:
        dataset.link_items(item_uuids)
else:
    print("Folder not found.")