Custom Embeddings with the SDK

Encord enables the use of custom embeddings for images, image sequences, image groups, and individual video frames. Custom embeddings allow you to incorporate your own feature representations into Encord’s platform, enhancing capabilities like similarity search, visualizations, and data filtering. This flexibility supports more advanced workflows and deeper insights tailored to your specific use cases.

​

Custom Embeddings Support

We currently support embeddings of dimensions ranging from 1 to 4096 for Index, and 1 to 2000 for Active, following on from our in-house clip Embeddings.

​

Import Custom Embeddings

To bring your custom embeddings into Encord, you first need to create a key in your metadata schema. After the key is in your schema, you can import your custom embeddings.

​

Step 1: Create a New Embedding Type

A key is required in your custom metadata schema for your embeddings. You can use any string as the key for your embeddings. We strongly recommend that you use a string that is meaningful.

If you do not include a key in your metadata schema, your imported embeddings are treated as strings.

Use add_embedding to add an embedding to your metadata schema.

​

Step 2: Upload Embeddings

With the key in the custom metadata schema ready, we can now import our embeddings.

Custom embedding sizes are flexible and can be set anywhere between 1 and 4096.

You can import embeddings after you have imported your data or during your data import.

If config is not specified, the sampling_rate is 1 frame per second, and the keyframe_mode is frame.

# Import dependencies
from encord import EncordUserClient
from encord.http.bundle import Bundle

# Authentication
SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH,
)

# Define a dictionary with item UUIDs and their respective metadata updates
updates = {
    "<data-ID-1>": {"<my-embedding>": [1.0, 2.0, 3.0]},
    "<data-ID-2>": {"<my-embedding>": [1.0, 2.0, 3.0]}
}

# Use the Bundle context manager
with Bundle() as bundle:
    # Update the storage items based on the dictionary
    for item_uuid, metadata_update in updates.items():
        item = user_client.get_storage_item(item_uuid=item_uuid)

        # Make a copy of the current metadata and update it with the new metadata
        curr_metadata = item.client_metadata.copy()
        curr_metadata.update(metadata_update)

        # Update the item with the new metadata and bundle
        item.update(client_metadata=curr_metadata, bundle=bundle)

"config": {
    "sampling_rate": "<samples-per-second>",
    "keyframe_mode": "frame" or "seconds",
},

​

Final Setup for Custom Embeddings

Once you import custom embeddings to your data (during data import or to existing data in Encord), there is a bit more setup required in Index and Active. These steps can only be performed from the UI.

​

Use Custom Embeddings

Custom embeddings in Index and Active allow you to:

• Perform similarity searches on your data with custom embeddings
• Adjust the similarity distance on similarity searches
• View your data in a 2D embedding plot graph
• Filter your data based on whether it has custom embeddings or not (Index only)