Skip to main content

​
CACHE_DURATION_IN_SECONDS

1 day

​
CloudUploadSettings Objects

@dataclass
class CloudUploadSettings()
Settings for uploading data into GCP cloud storage. These apply for each individual upload and will overwrite the :meth:encord.http.constants.RequestsSettings defined during :class:encord.EncordUserClient creation.

​
max_retries

Maximum number of allowed retries when uploading a file.

​
backoff_factor

Factor used to calculate exponential backoff between retries. The delay before each retry is computed as: backoff_factor * (2 ** (retry_number - 1)).

​
allow_failures

Whether to allow partial failures during upload. If set to True, the upload will proceed even if some items fail after all retries. Failed uploads will be logged, and the process will continue. This is useful for large batch uploads where a few failures are acceptable.

​
get_batches

def get_batches(iterable: List, n: int) -> List[List]
Split an iterable into fixed-size batches. Arguments:
  • iterable List - The input list to be split.
  • n int - The maximum size of each batch.
Returns:
  • List[List] - A list of lists where each sublist represents a batch.
Notes: This can be replaced with :func:itertools.batched in Python 3.12+.

​
UploadToSignedUrlFailure Objects

@dataclass
class UploadToSignedUrlFailure()
Details of a failed upload attempt to a signed URL.

​
exception

The exception instance raised during the failed upload.

​
file_path

Path to the file that failed to upload.

​
title

Title or identifier associated with the file.

​
signed_url

The signed URL that the file upload was attempted against.

​
upload_to_signed_url_list_for_single_file

def upload_to_signed_url_list_for_single_file(
        failures: List[UploadToSignedUrlFailure], file_path: Union[str, Path],
        title: str, signed_url: str, upload_item_type: StorageItemType,
        max_retries: int, backoff_factor: float) -> None
Attempt to upload a single file to a signed URL, appending failures if any occur. Arguments:
  • failures List[UploadToSignedUrlFailure] - A list to append failures to.
  • file_path Union[str, Path] - Path of the file to upload.
  • title str - Title or identifier for the file.
  • signed_url str - The signed URL to upload the file to.
  • upload_item_type StorageItemType - The type of the file being uploaded.
  • max_retries int - Maximum number of retries in case of failure.
  • backoff_factor float - Backoff factor for retry delays.

​
UploadPresignedUrlsGetParams Objects

class UploadPresignedUrlsGetParams(BaseDTO)
Parameters for requesting presigned URLs for file uploads.

​
count

Number of presigned URLs to request, typically matching the batch size of files.

​
upload_item_type

The type of item being uploaded (e.g., IMAGE, VIDEO, AUDIO, DICOM).

​
upload_to_signed_url_list

def upload_to_signed_url_list(
        file_paths: Iterable[Union[str, Path]], config: BaseConfig,
        api_client: ApiClient, upload_item_type: StorageItemType,
        cloud_upload_settings: CloudUploadSettings) -> List[Dict]
Upload multiple files to signed URLs and return upload results. Arguments:
  • file_paths Iterable[Union[str, Path]] - Paths of files to upload.
  • config BaseConfig - Configuration object with request settings.
  • api_client ApiClient - API client used to fetch presigned URLs.
  • upload_item_type StorageItemType - Type of items being uploaded.
  • cloud_upload_settings CloudUploadSettings - Upload configuration.
Returns:
  • List[Dict] - A list of dictionaries containing upload metadata:
    • data_hash (str): Unique identifier for the file.
    • file_link (str): Link to the uploaded file in storage.
    • title (str): File name or title.
Raises:
  • EncordException - If any file path does not exist.
  • CloudUploadError - If uploads fail and allow_failures is False.