Encord enables the use of custom embeddings for images, image sequences, image groups, and individual video frames. Custom embeddings allow you to incorporate your own feature representations into Encord’s platform, enhancing capabilities like similarity search, visualizations, and data filtering. This flexibility supports more advanced workflows and deeper insights tailored to your specific use cases.
To bring your custom embeddings into Encord, you first need to create a key in your metadata schema. After the key is in your schema, you can import your custom embeddings.
To use custom embeddings in Index:
embedding
type in your Schema.A key is required in your custom metadata schema for your embeddings. You can use any string as the key for your embeddings. We strongly recommend that you use a string that is meaningful.
If you do not include a key in your metadata schema, your imported embeddings are treated as strings.
Use add_embedding
to add an embedding to your metadata schema.
Key | Description | Display Benefits |
---|---|---|
embedding | 1 to 4096 for Index. 1 to 2000 for Active | Filtering by embeddings, similarity search, 2D scatter plot visualization (Coming Soon) |
With the key in the custom metadata schema ready, we can now import our embeddings.
Custom embedding sizes are flexible and can be set anywhere between 1 and 4096.
You can import embeddings after you have added your data or during your data registration.
If config
is not specified, the sampling_rate
is 1 frame per second, and the keyframe_mode
is frame
.
sampling_rate
of 0
only imports the first frame and all keyframes of your video into Index.Import while registering images
This JSON file imports embeddings while registering your data with Index from a cloud integration.
Update specific images
The custom embeddings format for images follows the same format as importing custom metadata.
Import while registering videos
This JSON file imports embeddings while registering your data with Index from a cloud integration.
config
is optional when importing your custom embeddings:
If config
is not specified, the sampling_rate
is 1 frame per second, and the keyframe_mode
is frame
.
sampling_rate
of 0
only imports the first frame and all keyframes of your video into Index.Update specific videos
After you import your custom embeddings they are available for:
Filtering using custom embeddings
Similarity searches using your custom embeddings
Embedding view and 2D plots with selection based workflows
Before you can perform filtering, use similarity searches, or use embedding plots with your custom embeddings, you need to configure your top-level Folder in Files.
To configure Folders for Embedding Plots:
Go to Index > Files.
A list of Folders available to you appears on the My Files page.
Do one of the following:
The Upgrade Folder button appears.
Click Upgrade Folder.
The Folder upgrades dialog appears.
Expand the Custom Embeddings drop down.
Select a custom embedding from the list.
Click Add.
The custom embedding appears under Custom Embeddings.
You can add multiple embeddings. Only one embedding can be active in Index at a time.
Expand your selected custom embedding.
Select any of the following:
Click Save and process changes.
A dialog appears informing you that the folder upgrade was successful. You are now ready to use your custom embeddings.
Upgrade your top-level folder before trying to filter.
Click the Filters dropdown or press F. The Filters menu appears.
Click Custom Embeddings from the menu.
Select your custom embedding to filter your data.
Select True to display images, frames, or videos with the custom embeddings.
Upgrade your top-level folder before trying to perform a similarity search.
Click the Embeddings icon in the Explorer.
The Embeddings screen appears.
Select the embedding you want to use from the Select custom embeddings menu.
Click the Grid icon.
Hover over an image or frame with the custom embedding.
Click the Similarity Search icon. Images and Frames sort according to similarity.
Click the Embeddings icon in the Explorer. The Embeddings screen appears.
Select the embedding you want to use from the Select custom embeddings menu.
Click the Grid icon.
Hover over an image or frame with the custom embedding.
Click the Similarity Search icon. Images and Frames sort according to similarity AND a Distance filter appears.
Adjust the Distance filter slider to change the similarity search results.
Upgrade your top-level folder before trying to view embedding plots.
Encord Index incorporates embedding plots — a two-dimensional visualization technique employed to represent intricate, high-dimensional data in a more comprehensible and visually coherent manner. This technique reduces data dimensionality while preserving the inherent structure and patterns within the original data.
The embedding plot aids in identifying interesting/noteworthy clusters, inspecting outliers, and excluding unwanted samples.
Use Custom Embedding Plots
Notice how images are clustered around certain regions. By defining a rectangular area on the plot, users can quickly isolate and analyze data points within that defined region. This approach facilitates the exploration of commonalities among these samples.
Hover over clusters or individual data points on the plot to visually check frames.
Upon selecting a region, the content within the Explorer page adjusts accordingly. Various actions can be executed with the chosen group:
Encord enables the use of custom embeddings for images, image sequences, image groups, and individual video frames. Custom embeddings allow you to incorporate your own feature representations into Encord’s platform, enhancing capabilities like similarity search, visualizations, and data filtering. This flexibility supports more advanced workflows and deeper insights tailored to your specific use cases.
To bring your custom embeddings into Encord, you first need to create a key in your metadata schema. After the key is in your schema, you can import your custom embeddings.
To use custom embeddings in Index:
embedding
type in your Schema.A key is required in your custom metadata schema for your embeddings. You can use any string as the key for your embeddings. We strongly recommend that you use a string that is meaningful.
If you do not include a key in your metadata schema, your imported embeddings are treated as strings.
Use add_embedding
to add an embedding to your metadata schema.
Key | Description | Display Benefits |
---|---|---|
embedding | 1 to 4096 for Index. 1 to 2000 for Active | Filtering by embeddings, similarity search, 2D scatter plot visualization (Coming Soon) |
With the key in the custom metadata schema ready, we can now import our embeddings.
Custom embedding sizes are flexible and can be set anywhere between 1 and 4096.
You can import embeddings after you have added your data or during your data registration.
If config
is not specified, the sampling_rate
is 1 frame per second, and the keyframe_mode
is frame
.
sampling_rate
of 0
only imports the first frame and all keyframes of your video into Index.Import while registering images
This JSON file imports embeddings while registering your data with Index from a cloud integration.
Update specific images
The custom embeddings format for images follows the same format as importing custom metadata.
Import while registering videos
This JSON file imports embeddings while registering your data with Index from a cloud integration.
config
is optional when importing your custom embeddings:
If config
is not specified, the sampling_rate
is 1 frame per second, and the keyframe_mode
is frame
.
sampling_rate
of 0
only imports the first frame and all keyframes of your video into Index.Update specific videos
After you import your custom embeddings they are available for:
Filtering using custom embeddings
Similarity searches using your custom embeddings
Embedding view and 2D plots with selection based workflows
Before you can perform filtering, use similarity searches, or use embedding plots with your custom embeddings, you need to configure your top-level Folder in Files.
To configure Folders for Embedding Plots:
Go to Index > Files.
A list of Folders available to you appears on the My Files page.
Do one of the following:
The Upgrade Folder button appears.
Click Upgrade Folder.
The Folder upgrades dialog appears.
Expand the Custom Embeddings drop down.
Select a custom embedding from the list.
Click Add.
The custom embedding appears under Custom Embeddings.
You can add multiple embeddings. Only one embedding can be active in Index at a time.
Expand your selected custom embedding.
Select any of the following:
Click Save and process changes.
A dialog appears informing you that the folder upgrade was successful. You are now ready to use your custom embeddings.
Upgrade your top-level folder before trying to filter.
Click the Filters dropdown or press F. The Filters menu appears.
Click Custom Embeddings from the menu.
Select your custom embedding to filter your data.
Select True to display images, frames, or videos with the custom embeddings.
Upgrade your top-level folder before trying to perform a similarity search.
Click the Embeddings icon in the Explorer.
The Embeddings screen appears.
Select the embedding you want to use from the Select custom embeddings menu.
Click the Grid icon.
Hover over an image or frame with the custom embedding.
Click the Similarity Search icon. Images and Frames sort according to similarity.
Click the Embeddings icon in the Explorer. The Embeddings screen appears.
Select the embedding you want to use from the Select custom embeddings menu.
Click the Grid icon.
Hover over an image or frame with the custom embedding.
Click the Similarity Search icon. Images and Frames sort according to similarity AND a Distance filter appears.
Adjust the Distance filter slider to change the similarity search results.
Upgrade your top-level folder before trying to view embedding plots.
Encord Index incorporates embedding plots — a two-dimensional visualization technique employed to represent intricate, high-dimensional data in a more comprehensible and visually coherent manner. This technique reduces data dimensionality while preserving the inherent structure and patterns within the original data.
The embedding plot aids in identifying interesting/noteworthy clusters, inspecting outliers, and excluding unwanted samples.
Use Custom Embedding Plots
Notice how images are clustered around certain regions. By defining a rectangular area on the plot, users can quickly isolate and analyze data points within that defined region. This approach facilitates the exploration of commonalities among these samples.
Hover over clusters or individual data points on the plot to visually check frames.
Upon selecting a region, the content within the Explorer page adjusts accordingly. Various actions can be executed with the chosen group: