Custom Embedding Model

Custom embeddings are required for similarity search on items already in Encord, embedding plots, and filtering. Natural language search and search by uploaded image are different. When someone types a query or drops in an image, that query has to be turned into a vector in the same embedding space as your imported embeddings, at the moment they search. Encord’s built-in model cannot produce vectors in your model’s space, so you host a small endpoint that runs your model, and Encord calls it at search time.

Feature	Import embeddings	Host an embedding endpoint
Similarity search (“find items like this one”)	Required	Not needed
Embedding plots	Required	Not needed
Filter by custom embedding	Required	Not needed
Natural language (text) search	Required	Required
Search by uploaded image	Required	Required

You can use any model, including open-weight models, as long as the vectors your endpoint returns come from the same model (and the same preprocessing) you used to create the embeddings you imported.

How It Works

The call to your endpoint is made from your browser, not from Encord’s servers. Encord’s backend signs a short-lived token for each user, the browser sends the query and token to your endpoint, and then sends the returned vector to Encord to run the search. This has three consequences for how you deploy your endpoint:

Network access: your endpoint must be reachable from the browsers of everyone who searches in Encord. There is no fixed Encord IP address to allowlist. An internet-facing endpoint works for everyone; an endpoint on your internal network or VPN works only for users connected to it.
CORS: because the request comes from a browser, your endpoint must answer CORS preflight requests from the Encord app.
Authentication: you authenticate requests by verifying the signed token, not by IP address or a shared secret.

Before You Begin

Your custom embeddings are imported and enabled on your top-level folder.
You have the embedding key you added to your metadata schema (for example, my-model-embedding) and its size.
You are an Admin of your Encord Organization (needed to create a service integration).
You can host an HTTPS endpoint that runs your model.

Step 1: Build Your Endpoint

Your endpoint accepts a POST request with a JSON body and returns the embedding as a JSON array of numbers.

Request

POST /v1/embed HTTP/1.1
Host: embed.example.com
Content-Type: application/json
Authorization: Bearer <signed-jwt>

{
  "embedding_id": "my-model-embedding",
  "content_type": "text",
  "content": "a forklift carrying pallets"
}

Field	Type	Description
`embedding_id`	string	The embedding key from your metadata schema. Useful if one endpoint serves several models.
`content_type`	`"text"` or `"b64_image"`	What the user searched with.
`content`	string	The text query, or the uploaded image as a raw base64 string (no `data:image/...;base64,` prefix and no MIME type, so detect the format from the bytes).

Response

Return HTTP 200 with a JSON array of numbers:

[0.0123, -0.0456, 0.0789, ...]

The array length must match the size of the embedding in your metadata schema.
Normalize the vector the same way you normalized your imported embeddings.
Any status other than 200, or a body that is not an array of numbers, is shown to the user as a failed search.

CORS

Allow the Encord app origin to call your endpoint. Your endpoint must respond to the OPTIONS preflight with at least:

Access-Control-Allow-Origin: https://app.encord.com
Access-Control-Allow-Methods: POST
Access-Control-Allow-Headers: Authorization, Content-Type

Use https://app.us.encord.com if your Organization is on the US platform.

Step 2: Verify Every Request

Every request carries Authorization: Bearer <token>, a JWT signed by Encord. Reject any request whose token does not verify.

Check	Expected value
Signature	Valid against a key from the JWKS URL below. Match on the `kid` header, which is your service integration’s UUID.
`alg`	`EdDSA` or `ES256`, depending on the key type you choose in Step 3.
`iss`	`https://api.encord.com/v2/public/organisation/<organisation-id>/service-integrations`
`aud`	Exactly the Target domain you register in Step 3 (for example, `https://embed.example.com`).
`exp`	In the future. Tokens are valid for 15 minutes and are refreshed automatically by the Encord app.

The token also includes a sub claim containing the email address of the Encord user who ran the search, which you can use for logging or per-user access control. Public keys (JWKS):

https://api.encord.com/v2/public/organisation/<organisation-id>/service-integrations/jwks

An OpenID discovery document is also available, for libraries that support it:

https://api.encord.com/v2/public/organisation/<organisation-id>/service-integrations/.well-known/openid-configuration

US platform: replace https://api.encord.com with https://api.us.encord.com in all the URLs above.
Your Encord representative can confirm your <organisation-id> and the exact issuer URL during setup.
Cache the JWKS response rather than fetching it on every request.

Step 3: Create a Service Integration

A service integration holds the key pair Encord uses to sign tokens for your endpoint.

Go to your Workspace Organization settings and select Service integrations.
Click New service integration.
Enter:
- Title: A meaningful name for the integration. For example Embedding search.
- Target domain: The origin of your endpoint including https://, for example https://embed.example.com. Do not include the path; you send that separately in Step 4. This value is also the token’s aud claim.
- Signing algorithm: EdDSA (Ed25519) is recommended. Choose ES256 only if your JWT library does not support EdDSA.
Click Create service integration.
Copy the UUID shown next to the integration’s title. You can also click Download public key if you prefer to pin the key instead of using the JWKS URL.

Step 4: Send Your Details to Encord

Send your Encord contact:

The service integration UUID from Step 3.
Your embedding key (for example, my-model-embedding).
The endpoint path (for example, /v1/embed). Encord combines it with your Target domain to form the full URL, so https://embed.example.com + /v1/embed becomes https://embed.example.com/v1/embed.
Whether your endpoint supports text, images, or both.

Encord connects your embedding key to your endpoint. After that, your custom embedding offers text and image search.

Step 5: Test Search

Go to Data > Explore and select your top-level folder.
Select your custom embedding from the Embedding selector.
Type a search query, or upload an image.

If results don’t appear, open your browser’s developer tools and check the Network tab for the request to your endpoint.

Example Endpoint (Python)

This example uses FastAPI and PyJWT. Replace my_model with your own model code.

import base64
from typing import Literal

import jwt  # pip install "pyjwt[crypto]"
from fastapi import FastAPI, Header, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel

ENCORD_API = "https://api.encord.com"  # US platform: "https://api.us.encord.com"
ORGANISATION_ID = "<organisation-id>"
TARGET_DOMAIN = "https://embed.example.com"  # Exactly what you entered as Target domain

ISSUER = f"{ENCORD_API}/v2/public/organisation/{ORGANISATION_ID}/service-integrations"
jwks_client = jwt.PyJWKClient(f"{ISSUER}/jwks", cache_keys=True)

app = FastAPI()
app.add_middleware(
    CORSMiddleware,
    allow_origins=["https://app.encord.com"],  # US platform: "https://app.us.encord.com"
    allow_methods=["POST"],
    allow_headers=["Authorization", "Content-Type"],
)


class EmbedRequest(BaseModel):
    embedding_id: str
    content_type: Literal["text", "b64_image"]
    content: str


def verify_token(authorization: str | None) -> dict:
    if not authorization or not authorization.startswith("Bearer "):
        raise HTTPException(status_code=401, detail="Missing bearer token")
    token = authorization.removeprefix("Bearer ")
    try:
        signing_key = jwks_client.get_signing_key_from_jwt(token)
        return jwt.decode(
            token,
            signing_key.key,
            algorithms=["EdDSA", "ES256"],
            audience=TARGET_DOMAIN,
            issuer=ISSUER,
            options={"require": ["exp", "iat", "iss", "aud"]},
        )
    except jwt.PyJWTError as exc:
        raise HTTPException(status_code=401, detail="Invalid token") from exc


@app.post("/v1/embed")
def embed(req: EmbedRequest, authorization: str | None = Header(default=None)) -> list[float]:
    claims = verify_token(authorization)  # claims["sub"] is the Encord user's email

    if req.content_type == "text":
        vector = my_model.embed_text(req.content)
    elif req.content_type == "b64_image":
        image_bytes = base64.b64decode(req.content)
        vector = my_model.embed_image(image_bytes)
    else:
        raise HTTPException(status_code=400, detail="Unsupported content_type")

    return [float(x) for x in vector]

Troubleshooting

Symptom	Likely cause
Browser console shows a CORS error	Your endpoint isn’t answering the `OPTIONS` preflight, or the `Access-Control-Allow-Origin` doesn’t match the Encord app origin.
Request fails with a network error for some users only	Those users can’t reach your endpoint (for example, they’re off VPN).
Your endpoint returns `401`	`aud` doesn’t match: check that your Target domain includes `https://` and matches your code exactly. Also check `iss` uses the right region (`api.encord.com` or `api.us.encord.com`).
Search runs but results look random	The query vector doesn’t come from the same model or normalization as your imported embeddings.
Search fails after your endpoint returns `200`	The array length doesn’t match your embedding size, or the response isn’t a plain JSON array of numbers.

Get Started

Data

Fundamentals

Annotate & Review

Label and Model Evaluation

Users and Workspaces

Custom Embedding Model

How It Works

Before You Begin

Step 1: Build Your Endpoint

Request

Response

CORS

Step 2: Verify Every Request

Step 3: Create a Service Integration

Step 4: Send Your Details to Encord

Step 5: Test Search

Example Endpoint (Python)

Troubleshooting

​How It Works

​Before You Begin

​Step 1: Build Your Endpoint

​Request

​Response

​CORS

​Step 2: Verify Every Request

​Step 3: Create a Service Integration

​Step 4: Send Your Details to Encord

​Step 5: Test Search

​Example Endpoint (Python)

​Troubleshooting

How It Works

Before You Begin

Step 1: Build Your Endpoint

Request

Response

CORS

Step 2: Verify Every Request

Step 3: Create a Service Integration

Step 4: Send Your Details to Encord

Step 5: Test Search

Example Endpoint (Python)

Troubleshooting