Register Cloud Data

Any files you upload to Encord must be stored in folders. Click here to learn how to create a folder to store your files.

​

Register Cloud Data to Files

​

STEP 1: Create a JSON or CSV File for Registration

Before registering your cloud data to Encord you must first create a JSON or CSV file specifying the files you want to register.

​

JSON Format

The JSON file format is a JSON object with top-level keys specifying the type of data and object URLs of the files you want to upload to Encord. You can add one data type at a time, or combine multiple data types in one JSON.

The supported top-level keys are: videos, audio, image_groups, images, and dicom_series. The details for each data format are given in the sections below.

​

CSV Format

In the CSV file format, the column headers specify which type of data is being uploaded. You can add and single file format at a time, or combine multiple data types in a single CSV file.

Details for each data format are given in the sections below.

​

STEP 2: Register Your Cloud Data

1. Navigate to Files section of Index in the Encord platform.
2. Click into a Folder.
3. Click + Upload files. A dialog appears.

4. Click Import from cloud data.

6. Click Add JSON or CSV files to add a JSON or CSV file specifying cloud data that is to be added.

​

Custom Metadata

Custom metadata, also known as client metadata, is supplementary information you can add to all data imported into Encord. It is provided in the form of a Python dictionary, as shown in examples. Client metadata serves several key functions:

• Filtering and sorting in Index and Active.
• Creating custom Label Editor layouts based on metadata.

You can optionally add some custom metadata per data item in the clientMetadata field (examples show how this is done) of your JSON file.

​

Metadata Schema

Based on your Data Discoverability Strategy, you need to create a metadata schema. The schema provides a method of organization for your custom metadata. Encord supports:

• Scalars: Methods for filtering.
• Enums: Methods with options for filtering.
• Embeddings: Method for embedding plot visualization, similarity search, and natural language search.

​

Custom metadata

Custom metadata refers to any additional information you attach to files, allowing for better data curation and management based on your specific needs. It can include any details relevant to your workflow, helping you organize, filter, and retrieve data more efficiently. For example, for a video of a construction site, custom metadata could include fields like "site_location": "Algiers", "project_phase": "foundation", or "weather_conditions": "sunny". This enables more precise tracking and management of your data.

Before importing any files with custom metadata to Encord, we recommend that you import a metadata schema. Encord uses metadata schemas to validate custom metadata uploaded to Encord and to instruct Index and Active how to display your metadata.

​

Metadata schema table

Use add_scalar to add a scalar key to your metadata schema.

Use add_enum and add_enum_options to add an enum and enum options to your metadata schema.

Use add_embedding to add an embedding to your metadata schema.

​

Import your metadata schema to Encord

# Import dependencies
from encord import EncordUserClient
from encord.metadata_schema import MetadataSchema

SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Create the schema
metadata_schema = user_client.metadata_schema()

# Add various metadata fields
metadata_schema.add_scalar("metadata_1", data_type="boolean")
metadata_schema.add_scalar("metadata_2", data_type="datetime")
metadata_schema.add_scalar("metadata_3", data_type="number")
metadata_schema.add_scalar("metadata_4", data_type="uuid")
metadata_schema.add_scalar("metadata_5", data_type="varchar")
metadata_schema.add_scalar("metadata_6", data_type="text")

# Add an enum field
metadata_schema.add_enum("my-enum", values=["enum-value-01", "enum-value-02", "enum-value-03"])

# Add embedding fields
metadata_schema.add_embedding('my-test-active-embedding', size=512)
metadata_schema.add_embedding('my-test-index-embedding', size=<values-from-1-to-4096>)

# Save the schema
metadata_schema.save()

# Print the schema for verification
print(metadata_schema)

​

Verify your schema

After importing your schema to Encord we recommend that you verify that the import is successful. Run the following code to verify your metadata schema imported and that the schema is correct.

# Import dependencies
from encord import EncordUserClient
from encord.metadata_schema import MetadataSchema

SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Create the schema
metadata_schema = user_client.metadata_schema()

# Print the schema for verification
print(metadata_schema)

​

Update Custom Metadata (JSON)

When updating custom metadata using a JSON file, you MUST specify "skip_duplicate_urls": true and "upsert_metadata": true.

Specifying the "skip_duplicate_urls": true and "upsert_metadata": true flags in the JSON file does the following:

• New files registered with Encord and custom metadata for those files is added.

• Existing files have their existing custom metadata overwritten with the custom metadata specified in the JSON file.

To update custom metadata with a JSON file:

1. Create a registration JSON file with the updated custom metadata. Include the "skip_duplicate_urls": true and "upsert_metadata": true flags.

{
  "videos": [
    {
      "objectUrl": "<object url_1>"
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-custom-video-title.mp4",
      "clientMetadata": {"optional": "metadata"}
    }
  ],
  "skip_duplicate_urls": true,
  "upsert_metadata": true
}

2. Register your files with Encord using the new JSON file.

​

Custom Embeddings

Encord enables the use of custom embeddings for images, image sequences, image groups, and individual video frames.

​

Step 1: Create a New Embedding Type

A key is required in your custom metadata schema for your embeddings. You can use any string as the key for your embeddings. We strongly recommend that you use a string that is meaningful.

If you do not include a key in your metadata schema, your imported embeddings are treated as strings.

Use add_embedding to add an embedding to your metadata schema.

# Import dependencies
from encord import EncordUserClient
from encord.metadata_schema import MetadataSchema

SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Create the schema
metadata_schema = user_client.metadata_schema()


# Add embedding fields
metadata_schema.add_embedding('my-test-active-embedding', size=512)
metadata_schema.add_embedding('my-test-index-embedding', size=<values-from-1-to-4096>)

# Save the schema
metadata_schema.save()

# Print the schema for verification
print(metadata_schema)

​

Step 2: Upload Embeddings

With the key in the custom metadata schema ready, we can now import our embeddings.

Custom embedding sizes are flexible and can be set anywhere between 1 and 4096.

You can import embeddings after you have added your data or during your data registration.

If config is not specified, the sampling_rate is 1 frame per second, and the keyframe_mode is frame.

​

How To Increase File Registration Speed

To speed up file registration with Encord, you can include metadata for each file in the upload JSON. This metadata is used directly without additional validation and is not stored on our servers. Ensuring accuracy in the metadata you provide is essential to maintain precise labels.

• imageMetadata for images:

  • mimeType: MIME type of the image (e.g., image/jpeg).
  • fileSize: Size of the file in bytes.
  • width: Width of the image in pixels.
  • height: Height of the image in pixels.

• audioMetadata for audio files:

  • duration_seconds (float): Audio duration in seconds.
  • file_size (int): Size of the audio file in bytes.
  • mime_type (str): MIME type (e.g., audio/mpeg, audio/wav).
  • sample_rate (int): Sample rate in Hz.
  • bit_depth (int): Size of each sample in bits.
  • codec (str): Codec used (e.g., mp3, pcm).
  • num_channels (int): Number of audio channels.

• videoMetadata for videos:

  • fps: Frames per second.
  • duration: Duration in seconds.
  • width / height: Dimensions in pixels.
  • file_size: File size in bytes.
  • mime_type: File type (MIME standard).

{
  "images": [
    {
      "objectUrl": "s3://my_image.jpg",
      "imageMetadata": {
        "mimeType": "image/jpg",
        "fileSize": 124,
        "width": 640,
        "height": 480
      }
    }
  ]
}

​

Check Data Registration Status

You can check the progress of the processing job by clicking the bell icon in the top right corner of the Encord app.

• A spinning progress indicator shows that the processing job is still in progress.
• If successful, the processing completes with a green tick icon.
• If unsuccessful, there is a red cross icon, as seen below.

If the upload is unsuccessful, ensure that:

• Your provider permissions are set correctly
• The object data format is supported
• The upload JSON or CSV file is correctly formatted.

​

Helpful Scripts and Examples

Use the following examples and helpful scripts to quickly learn how to create JSON and CSV files formatted for uploading cloud data to Encord, by constructing the URLs from the specified path in your private storage.