Active Custom Embeddings

​

TLDR;

Do you already know what you are doing and only want to look over a Jupyter Notebook example? We provide one here.

​

Active’s Default Embeddings

Currently in Active, we calculate and display embeddings using a generic CLIP model. This model is excellent for a wide range of tasks and across the board performance. However, a generic CLIP model may struggle with highly specialized tasks. The embeddings generated from the generic model are used for: Natural Language Search (not supported for custom embeddings), image similarity search and the embeddings view (where you view reduced embeddings).

​

Custom Embeddings Support

We currently support embeddings of dimensions ranging from 1 to 4096 for Index, and 1 to 2000 for Active, following on from our in-house clip Embeddings.

We support embeddings for images, image sequences and image groups. Support for Videos is coming soon.

​

Use Custom Embeddings in Active

To use custom embeddings in Active:

1. Add an Embedding type to your custom metadata schema.
2. Upload your embeddings as custom metadata.
3. Create an Annotate Project.
4. Select your custom embeddings as you import the Project to Active.

​

Step 1: Create an Embedding Field

A key is required in your custom metadata schema for your embeddings. You can use any string as the key for your embeddings. We strongly recommend that you use a string that is meaningful.

If you do not include a key in your metadata schema, your imported embeddings are treated as strings.

Use add_embedding to add an embedding to your metadata schema.

​

Step 2: Upload Embeddings

With the key in the custom metadata schema ready, we can now import our embeddings.

Custom embedding sizes are flexible and can be set anywhere between 1 and 4096.

You can import embeddings after you have imported your data or during your data import.

If config is not specified, the sampling_rate is 1 frame per second, and the keyframe_mode is frame.

# Import dependencies
from encord import EncordUserClient
from encord.http.bundle import Bundle

# Authentication
SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH,
)

# Define a dictionary with item UUIDs and their respective metadata updates
updates = {
    "<data-ID-1>": {"<my-embedding>": [1.0, 2.0, 3.0]},
    "<data-ID-2>": {"<my-embedding>": [1.0, 2.0, 3.0]}
}

# Use the Bundle context manager
with Bundle() as bundle:
    # Update the storage items based on the dictionary
    for item_uuid, metadata_update in updates.items():
        item = user_client.get_storage_item(item_uuid=item_uuid)

        # Make a copy of the current metadata and update it with the new metadata
        curr_metadata = item.client_metadata.copy()
        curr_metadata.update(metadata_update)

        # Update the item with the new metadata and bundle
        item.update(client_metadata=curr_metadata, bundle=bundle)

"config": {
    "sampling_rate": "<samples-per-second>",
    "keyframe_mode": "frame" or "seconds",
},

​

Step 3 : Create a Project in Annotate

The following task outlines the basics of creating an Annotate Project. For detailed instructions, refer to the documentation here.

To create an Annotate Project:

1. Go to Annotate > Projects. The Annotate Projects list appears.

2. Click New annotation project. The Create New Project page appears.

3. Provide a meaningful title and description.

4. Select an Ontology.

5. Select one or more Datasets.

6. Load a Workflow template.

7. Add collaborators to the Project.

8. Click Create Project.

​

Step 4: Import Project with Custom Embeddings

Before you can use your custom embeddings in Encord Active Projects, you need to import the custom embeddings. This is performed while you import your Annotate Project into Active.

For existing Active Projects, you can import custom embeddings for your Project import if your Project imported to Data or Labels. Importing to Metrics & Embeddings requires deleting the Project in Active and re-importing the Project with your custom embeddings.

To import a Project with custom embeddings:

1. Log in to the Encord platform. The landing page for the Encord platform appears.

2. Create a Project (Annotation Project or Training Project) in Encord Annotate.

3. Click Active from the main menu. The landing page for Active appears.

4. Click the Import Annotate Project button. The Import an Annotate Project to Encord Active dialog appears.

5. Click the Import button for the Annotate Project you want to import. The Confirm Project Import dialog appears.

   .

6. Select the custom embeddings you want to import to the Project, under Metrics & Embeddings.

7. Select Import under Metrics & Embeddings.

8. Click Proceed. The Annotate Project imports with your custom embeddings.

​

End-to-End Custom Embeddings Example

We provide an end-to-end example using a Jupyter Notebook here.