Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.encord.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Default Embeddings

Embeddings are calculated using our purpose-built models, which perform well across a broad range of tasks. For highly specialized domains, custom embeddings may be more appropriate. Our built-in embeddings power the following features:
  • Natural language search (not supported for custom embeddings)
  • Image similarity search
  • Embeddings view

​
Custom Embeddings Support

We currently support embeddings of:
  • 1 to 4,096 dimensions for data curation.
  • 1 to 2,000 dimensions for label validation.
Encord enables the use of custom embeddings for images, image sequences, image groups, and individual video frames. Custom embeddings allow you to incorporate your own feature representations into Encord’s platform, enhancing capabilities like similarity search, visualizations, and data filtering. This flexibility supports more advanced workflows and deeper insights tailored to your specific use cases.

​
Use Custom Embeddings

To bring your custom embeddings into Encord, you first need to create a key in your metadata schema. After the key is in your schema, you can import your custom embeddings. Custom embeddings can be used for data curation and label validation. To use custom embeddings
  1. Create a new embedding type in your Schema.
  2. Upload your embeddings.
  3. Select your custom embeddings from the Embeddings view.
Before you can use embedding plots with your custom embeddings, you need to configure your root Folder in Files.

​
1. Create New Embedding Type

A key is required in your custom metadata schema for your embeddings. You can use any string as the key for your embeddings. We strongly recommend that you use a string that is meaningful. If you do not include a key in your metadata schema, your imported embeddings are treated as strings.
Embedding key names can contain alphanumeric (a-z, A-Z, 0-1) characters, hyphens, and underscores.
Use add_embedding to add an embedding to your metadata schema.
We currently support embeddings of:
  • 1 to 4,096 dimensions for data curation.
  • 1 to 2,000 dimensions for label validation.

# Import dependencies
from encord import EncordUserClient
from encord.metadata_schema import MetadataSchema

SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Create the schema
metadata_schema = user_client.metadata_schema()


# Add embedding fields
metadata_schema.add_embedding('my-test-active-embedding', size=512)
metadata_schema.add_embedding('my-test-index-embedding', size=<values-from-1-to-4096>)

# Save the schema
metadata_schema.save()

# Print the schema for verification
print(metadata_schema)

​
2. Upload Embeddings

With the key in the custom metadata schema ready, we can now import our embeddings. Custom embedding sizes are flexible and can be set anywhere between 1 and 4096. You can import embeddings after you have added your data or during your data registration.
Your key frames (frames specified with or without embeddings) always appear in Encord, regardless of what sampling rate you specify.
Embedding key names can contain alphanumeric (a-z, A-Z, 0-1) characters, hyphens, and underscores.
If config is not specified, the sampling_rate is 1 frame per second, and the keyframe_mode is frame.
Specifying a sampling_rate of 0 only imports the first frame and all keyframes of your video.

​
Import while importing videos

This JSON file imports embeddings while registering your data with Encord from a cloud integration.config is optional when importing your custom embeddings:
"config": {
    "sampling_rate": "<samples-per-second>",
    "keyframe_mode": "frame" or "seconds",
},
If config is not specified, the sampling_rate is 1 frame per second, and the keyframe_mode is frame.
Specifying a sampling_rate of 0 only imports the first frame and all keyframes of your video.

{
    "videos": [
        {
            "objectUrl": "<cloud-file-path-to-your-video-1>",
            "title": "<title-for-your-video-1>",
            "clientMetadata": {
                "$encord": {
                    "config": {
                        "sampling_rate": "<samples-per-second>",
                        "keyframe_mode": "frame" or "seconds",
                    },
                    "frames": {
                        "<frame-number-or-seconds>": {
                            "<my-embedding>": [1.0, 2.0, 3.0]
                        },
                        "<frame-number-or-seconds>": {
                            "<my-embedding>": [1.0, 2.0, 3.0]
                        }
                    }
                }
            }
        },
        {
            "objectUrl": "<cloud-file-path-to-your-video-2>",
            "title": "<title-for-your-vide-2>",
            "clientMetadata": {
                "$encord": {
                    "config": {
                        "sampling_rate": "<frames-per-second>",
                        "keyframe_mode": "frame" or "seconds",
                    },
                    "frames": {
                        "<frame-number-or-seconds>": {
                            "<my-embedding>": [1.0, 2.0, 3.0]
                        },
                        "<frame-number-or-seconds>": {
                            "<my-embedding>": [1.0, 2.0, 3.0]
                        }
                    }
                }
            }
        },
        {
            "objectUrl": "<cloud-path-to-your-video-3>",
            "title": "<title-for-your-video-3>",
            "clientMetadata": {
                "$encord": {
                    "config": {
                        "sampling_rate": "<frames-per-second>",
                        "keyframe_mode": "frame" or "seconds",
                    },
                    "frames": {
                        "<frame-number-or-seconds>": {
                            "<my-embedding>": [1.0, 2.0, 3.0]
                        },
                        "<frame-number-or-seconds>": {
                            "<my-embedding>": [1.0, 2.0, 3.0]
                        }
                    }
                }
            }
        }
    ],
    "skip_duplicate_urls": true
}

​
Update specific videos

# Import dependencies
from encord import EncordUserClient
from encord.http.bundle import Bundle
from encord.orm.storage import StorageFolder, StorageItem, StorageItemType, FoldersSortBy

# Authentication
SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH,
)

updates = {
    "<data-hash-1>": {
        "$encord": {
            "frames": {
                "<frame-number-1>": {
                    "<my-embedding>": [1.0, 2.0, 3.0],  # custom embedding ("embedding") with float values
                },
                "<frame-number-2>": {
                    "<my-embedding>": [1.0, 2.0, 3.0],  # custom embedding ("embedding") with float values
                }
            }
        }
    },
    "<data-hash-2>": {
        "$encord": {
            "config": {
                "sampling_rate": <samples-per-second>,  # VIDEO ONLY (optional default = 1 sample/second)
                "keyframe_mode": "frame" or "seconds",  # VIDEO ONLY (optional default = "frame")
            },
            "frames": {
                "<frame-number-1>": {
                    "<my-embedding>": [1.0, 2.0, 3.0],  # custom embedding ("embedding") with float values
                },
                "<frame-number-2>": {
                    "<my-embedding>": [1.0, 2.0, 3.0],  # custom embedding ("embedding") with float values
                }
            }
        }
    },
}

# Use the Bundle context manager
with Bundle() as bundle:
    # Update the storage items based on the dictionary
    for item_uuid, metadata_update in updates.items():
        item = user_client.get_storage_item(item_uuid=item_uuid)

        # Make a copy of the current metadata and update it with the new metadata
        curr_metadata = item.client_metadata.copy()
        curr_metadata.update(metadata_update)

        # Update the item with the new metadata and bundle
        item.update(client_metadata=curr_metadata, bundle=bundle)

​
Import while importing data units

This JSON file imports embeddings while registering your data with Encord from a cloud integration.
{
  "images": [
    {
      "objectUrl": "file/path/to/data-unit/file-name-01.file-extension",
      "title": "data-unit-title.file-extension",
      "clientMetadata": {"metadata-1": "value", "metadata-2": "value"}
    },
  ],
  "skip_duplicate_urls": true
}

​
Import specific data units

The custom embeddings format for images, text files, PDFs, and audio files follows the same format as importing custom metadata.
# Import dependencies
from encord import EncordUserClient
from encord.http.bundle import Bundle

# Authentication
SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH,
)

# Define a dictionary with item UUIDs and their respective metadata updates
updates = {
    "<data-ID-1>": {"<my-embedding>": [1.0, 2.0, 3.0]},
    "<data-ID-2>": {"<my-embedding>": [1.0, 2.0, 3.0]}
}

# Use the Bundle context manager
with Bundle() as bundle:
    # Update the storage items based on the dictionary
    for item_uuid, metadata_update in updates.items():
        item = user_client.get_storage_item(item_uuid=item_uuid)

        # Make a copy of the current metadata and update it with the new metadata
        curr_metadata = item.client_metadata.copy()
        curr_metadata.update(metadata_update)

        # Update the item with the new metadata and bundle
        item.update(client_metadata=curr_metadata, bundle=bundle)

​
3. Select your Custom Embeddings

After you import your custom embeddings they are available for:
  • Filtering using custom embeddings
  • Similarity searches using your custom embeddings
  • Embedding view and 2D plots with selection based workflows
Before you can use embedding plots with your custom embeddings, you need to configure your root Folder in Files.

​
Compute Metrics and Embeddings

Before you can perform filtering, use similarity searches, or use embedding plots with your custom embeddings, you need to compute Metrics and Embeddings.
  1. Navigate to Projects > Explore.
  2. Click Metrics & Embeddings.
  3. Click Compute for either option.
  4. Specify the following:
    • Similarity & Natural language search and quality metrics: Enable to compute embeddings and quality metrics. Access quality metrics for filtering and sorting.
    • Select embeddings: Default embeddings are computed by Encord. Alternatively, import and select your own custom embeddings.
    • Embeddings plot, Diversity and Uniqueness metrics: Enable to compute UMAP reduction to generate 2D embeddings plots to visualize your data. Also access diversity and uniqueness metrics for curation.
  5. Click Start computation.

​
Filtering with Custom Embeddings

You must calculate metrics and embeddings before you can filter with your custom embeddings.
  1. Click the Filter dropdown or press F. Filter by custom embeddings
  2. Click Custom Embeddings from the menu.
  3. Select your custom embedding to filter your data.
  4. Select True to display images, frames, or videos with the custom embeddings.

​
Similarity Searches with Custom Embeddings

You must calculate metrics and embeddings before using the similarity search with your custom embeddings.
  1. Click the Embeddings icon in the Explorer. The Embeddings screen appears.
  2. Select the embedding you want to use from the Select custom embeddings menu.
  3. Click the Grid icon.
  4. Hover over an image or frame with the custom embedding.
  5. Click the Similarity Search icon. Images and Frames sort according to similarity.

​
Adjust Similarity Search Distance

  1. Click the Embeddings icon in the Explorer. The Embeddings screen appears.
  2. Select the embedding you want to use from the Select custom embeddings menu.
  3. Click the Grid icon.
  4. Hover over an image or frame with the custom embedding.
  5. Click the Similarity Search icon. Images and Frames sort according to similarity AND a Distance filter appears. Distance filter
  6. Adjust the Distance filter slider to change the similarity search results.

​
Embedding View

You must calculate metrics and embeddings before you can view embedding plots.
Encord incorporates embedding plots — a two-dimensional visualization technique employed to represent intricate, high-dimensional data in a more comprehensible and visually coherent manner. This technique reduces data dimensionality while preserving the inherent structure and patterns within the original data. The embedding plot aids in identifying interesting/noteworthy clusters, inspecting outliers, and excluding unwanted samples. Use Custom Embedding Plots Vibrant 2D data embedding plot highlighting data patterns and clusters Notice how images are clustered around certain regions. By defining a rectangular area on the plot, users can quickly isolate and analyze data points within that defined region. This approach facilitates the exploration of commonalities among these samples. Hover over clusters or individual data points on the plot to visually check frames. Upon selecting a region, the content within the Explorer page adjusts accordingly. Various actions can be executed with the chosen group:
  • Use Collections to tag and group images.
  • Establish subsets similar to these and then conduct comparisons.