Register Cloud Data
Using a JSON File (Recommended)
The following scripts initiate uploads from your cloud storage to a specified folder in Encord using a properly formatted JSON upload file. It works for all file types.
If Upload is still in progress, try again later! is returned, use the script to check the upload status to see whether the upload has finished.
Ensure that you:
- Replace <private_key_path> with the path to your private key.
- Replace <integration_title> with the title of the integration you want to use.
- Replace <folder_name> with the folder name. The scripts assume that the specified folder name is unique.
- Replace
path/to/json/file.json with the path to a JSON file specifying which cloud storage files should be uploaded.
- If creating a new folder, replace
A folder to store my files with a meaningful description for your folder.
- If creating a new folder, replace
"my": "folder_metadata" with any metadata you want to add to the folder.
To update client metadata, include the "upsert_metadata": true flag in the upload JSON file. It can only be used in conjunction with "skip_duplicate_urls": true.
Use the following script to check the status of your private cloud registration.
If Upload is still in progress, try again later! is returned, run the script again to check the status at a later time.
Ensure that you:
- Replace
<upload_job_id> with the upload_job_id shown in the output of the script used to start the upload. In the example output of the script provided, the upload_job_id = c4026edb-4fw2-40a0-8f05-a1af7f465727.
- Replace
<private_key_path> with the path to your private key.
- Replace
<folder_name> with the folder name. The scripts assume that the specified folder name is unique.
Register Cloud Data Without a JSON File
Instead of using a JSON file to specify cloud files and URLs, you can use DataUploadItems. This method is recommended for cloud data registrations in the specific use cases listed below.
Increased Image Group File Registration
You can increase the speed at which Image Groups are registered to Encord by including imageMetadata for each image belonging to the image group. For this you must use the following method of importing images and creating Image Groups.
When the audioMetadata, imageMetadata, or videoMetadata flags are present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. It is crucial that the metadata you provide is accurate.
Ensure that you:
- Replace
<path_to_private_key> with your private key path
- Enter the URLs of images you want to import and batch as an image group.
- Provide the
imageMetadata for your files by replacing the example values.
- Enter a title and an optional description for your image group.
- Specify the name of the target folder to store the image group.
- Replace
<integration_id> with the id of the integration you want to use for the data registration.
Create an Image Group from Existing Images
You can create Image Groups from individual images previously added to Encord. If the individual images include clientMetadata and are then grouped into an Image Group, this metadata is retained at the frame level within the Image Group.
The following script combines three previously uploaded images in Encord into a single Image Group.
In the following script:
- Replace
<private_key_path> with your private key path.
- Replace the titles in the
file_names list with the names of images to be grouped.
- Replace
<folder_name> with the title for the folder containing the files.
- Provide a title for the image group.
- Replace
<integration_id> with the ID of the integration being used.
- Add optional
clientMetadata for the image group.
Local Data
Upload Local Data to an Existing Folder
The following scripts add data to a folder that already exists in Data > Files & Folders.
- Replace
<folder_name> with the name of the folder you want to add data to.
- For images and videos replace
User/path/to/my/file with the path to your image or video file.
- For DICOM series replace
User/path/to/dicom/file1 and User/path/to/dicom/file2 with the paths to the DICOM files you want to create a series from. Add as many file paths as necessary.
- For image groups and image sequences, replace
User/path/to/image/file1 and User/path/to/image/file2 with the paths to the image files you want to create an image group or image sequence from. Add as many file paths as necessary.
Upload Local Data to a New Folder
The following script creates a new folder in Data > Files & Folders and uploads image, video, DICOM, image group, or image sequence files to the newly created folder. A new folder is created each time the script is run.
In the following scripts, ensure that you:
- Replace <private_key_path> with the path to your private key.
- Replace <folder_name> with the name you want to give your folder. The scripts assume that the specified folder name is unique.
- Replace
A folder to store my files with a meaningful description for your folder.
- Replace
my: folder_metadata with any metadata you want to add to the folder. Remove the line if you do not want to add any metadata to the folder.
- For images and videos replace
User/path/to/my/file with the path to your image or video file.
- For DICOM series replace
User/path/to/dicom/file1 and User/path/to/dicom/file2 with the paths to the DICOM files you want to create a series from. Add as many file paths as necessary.
- For image groups and image sequences, replace
User/path/to/image/file1 and User/path/to/image/file2 with the paths to the image files you want to create an image group or image sequence from. Add as many file paths as necessary.
List Data Units in a Folder
Listing the files in a folder can be used to verify that all data unit were successfully uploaded.
- Replace <private_key_path> with the path to your private key.
- Replace <folder_name> with the name you want to give your folder. The scripts assume that the specified folder name is unique.