> ## Documentation Index
> Fetch the complete documentation index at: https://docs.encord.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom Embedding Model

[Custom embeddings](https://docs.encord.com/platform-documentation/Curate/custom-embeddings.md) are required for similarity search on items already in Encord, embedding plots, and filtering.

Natural language search and search by uploaded image are different. When someone types a query or drops in an image, that query has to be turned into a vector **in the same embedding space as your imported embeddings**, at the moment they search. Encord's built-in model cannot produce vectors in your model's space, so you host a small endpoint that runs your model, and Encord calls it at search time.

| Feature | Import embeddings | Host an embedding endpoint |
| - | - | - |
| Similarity search ("find items like this one") | Required | Not needed |
| Embedding plots | Required | Not needed |
| Filter by custom embedding | Required | Not needed |
| Natural language (text) search | Required | **Required** |
| Search by uploaded image | Required | **Required** |

> [!NOTE]
>
> You can use any model, including open-weight models, as long as the vectors your endpoint returns come from the same model (and the same preprocessing) you used to create the embeddings you imported.

## How It Works

The call to your endpoint is made **from your browser**, not from Encord's servers. Encord's backend signs a short-lived token for each user, the browser sends the query and token to your endpoint, and then sends the returned vector to Encord to run the search.

```mermaid theme={"dark"}
sequenceDiagram
    participant Y as Your browser (Encord app)
    participant E as Encord API
    participant C as Your embedding endpoint
    Y->>E: Request a signed token for your service integration
    E-->>Y: JWT (valid for 15 minutes)
    Y->>C: POST query + Authorization: Bearer <JWT>
    C->>E: Fetch public keys (JWKS), cached
    C-->>Y: 200 [0.12, -0.03, ...]
    Y->>E: Search using the returned vector
    E-->>Y: Ranked results
```

This has three consequences for how you deploy your endpoint:

* **Network access:** your endpoint must be reachable from the browsers of everyone who searches in Encord. There is no fixed Encord IP address to allowlist. An internet-facing endpoint works for everyone; an endpoint on your internal network or VPN works only for users connected to it.
* **CORS:** because the request comes from a browser, your endpoint must answer CORS preflight requests from the Encord app.
* **Authentication:** you authenticate requests by verifying the signed token, not by IP address or a shared secret.

## Before You Begin

* Your custom embeddings are [imported and enabled](https://docs.encord.com/platform-documentation/Curate/custom-embeddings.md) on your top-level folder.
* You have the **embedding key** you added to your metadata schema (for example, `my-model-embedding`) and its size.
* You are an Admin of your Encord Organization (needed to create a service integration).
* You can host an HTTPS endpoint that runs your model.

## Step 1: Build Your Endpoint

Your endpoint accepts a `POST` request with a JSON body and returns the embedding as a JSON array of numbers.

### Request

```http theme={"dark"}
POST /v1/embed HTTP/1.1
Host: embed.example.com
Content-Type: application/json
Authorization: Bearer <signed-jwt>

{
  "embedding_id": "my-model-embedding",
  "content_type": "text",
  "content": "a forklift carrying pallets"
}
```

| Field | Type | Description |
| - | - | - |
| `embedding_id` | string | The embedding key from your metadata schema. Useful if one endpoint serves several models. |
| `content_type` | `"text"` or `"b64_image"` | What the user searched with. |
| `content` | string | The text query, or the uploaded image as a **raw base64 string** (no `data:image/...;base64,` prefix and no MIME type, so detect the format from the bytes). |

### Response

Return HTTP `200` with a JSON array of numbers:

```json theme={"dark"}
[0.0123, -0.0456, 0.0789, ...]
```

* The array length must match the size of the embedding in your metadata schema.
* Normalize the vector the same way you normalized your imported embeddings.
* Any status other than `200`, or a body that is not an array of numbers, is shown to the user as a failed search.

### CORS

Allow the Encord app origin to call your endpoint. Your endpoint must respond to the `OPTIONS` preflight with at least:

```http theme={"dark"}
Access-Control-Allow-Origin: https://app.encord.com
Access-Control-Allow-Methods: POST
Access-Control-Allow-Headers: Authorization, Content-Type
```

> [!NOTE]
>
> Use `https://app.us.encord.com` if your Organization is on the US platform.

## Step 2: Verify Every Request

Every request carries `Authorization: Bearer <token>`, a JWT signed by Encord. Reject any request whose token does not verify.

| Check | Expected value |
| - | - |
| Signature | Valid against a key from the JWKS URL below. Match on the `kid` header, which is your service integration's UUID. |
| `alg` | `EdDSA` or `ES256`, depending on the key type you choose in Step 3. |
| `iss` | `https://api.encord.com/v2/public/organisation/<organisation-id>/service-integrations` |
| `aud` | Exactly the **Target domain** you register in Step 3 (for example, `https://embed.example.com`). |
| `exp` | In the future. Tokens are valid for 15 minutes and are refreshed automatically by the Encord app. |

The token also includes a `sub` claim containing the email address of the Encord user who ran the search, which you can use for logging or per-user access control.

**Public keys (JWKS):**

```
https://api.encord.com/v2/public/organisation/<organisation-id>/service-integrations/jwks
```

An OpenID discovery document is also available, for libraries that support it:

```
https://api.encord.com/v2/public/organisation/<organisation-id>/service-integrations/.well-known/openid-configuration
```

> [!NOTE]
>
> * US platform: replace `https://api.encord.com` with `https://api.us.encord.com` in all the URLs above.
> * Your Encord representative can confirm your `<organisation-id>` and the exact issuer URL during setup.
> * Cache the JWKS response rather than fetching it on every request.

## Step 3: Create a Service Integration

A service integration holds the key pair Encord uses to sign tokens for your endpoint.

1. Go to your Workspace Organization settings and select **Service integrations**.
2. Click **New service integration**.
3. Enter:
   * **Title**: A meaningful name for the integration. For example `Embedding search`.
   * **Target domain**: The origin of your endpoint **including `https://`**, for example `https://embed.example.com`. Do not include the path; you send that separately in Step 4. This value is also the token's `aud` claim.
   * **Signing algorithm**: **EdDSA (Ed25519)** is recommended. Choose **ES256** only if your JWT library does not support EdDSA.
4. Click **Create service integration**.
5. Copy the **UUID** shown next to the integration's title. You can also click **Download public key** if you prefer to pin the key instead of using the JWKS URL.

## Step 4: Send Your Details to Encord

Send your Encord contact:

* The **service integration UUID** from Step 3.
* Your **embedding key** (for example, `my-model-embedding`).
* The **endpoint path** (for example, `/v1/embed`). Encord combines it with your Target domain to form the full URL, so `https://embed.example.com` + `/v1/embed` becomes `https://embed.example.com/v1/embed`.
* Whether your endpoint supports **text**, **images**, or both.

Encord connects your embedding key to your endpoint. After that, your custom embedding offers text and image search.

## Step 5: Test Search

1. Go to **Data** > **Explore** and select your top-level folder.
2. Select your custom embedding from the **Embedding** selector.
3. Type a search query, or upload an image.

If results don't appear, open your browser's developer tools and check the **Network** tab for the request to your endpoint.

## Example Endpoint (Python)

This example uses [FastAPI](https://fastapi.tiangolo.com/) and [PyJWT](https://pyjwt.readthedocs.io/). Replace `my_model` with your own model code.

```python theme={"dark"}
import base64
from typing import Literal

import jwt  # pip install "pyjwt[crypto]"
from fastapi import FastAPI, Header, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel

ENCORD_API = "https://api.encord.com"  # US platform: "https://api.us.encord.com"
ORGANISATION_ID = "<organisation-id>"
TARGET_DOMAIN = "https://embed.example.com"  # Exactly what you entered as Target domain

ISSUER = f"{ENCORD_API}/v2/public/organisation/{ORGANISATION_ID}/service-integrations"
jwks_client = jwt.PyJWKClient(f"{ISSUER}/jwks", cache_keys=True)

app = FastAPI()
app.add_middleware(
    CORSMiddleware,
    allow_origins=["https://app.encord.com"],  # US platform: "https://app.us.encord.com"
    allow_methods=["POST"],
    allow_headers=["Authorization", "Content-Type"],
)


class EmbedRequest(BaseModel):
    embedding_id: str
    content_type: Literal["text", "b64_image"]
    content: str


def verify_token(authorization: str | None) -> dict:
    if not authorization or not authorization.startswith("Bearer "):
        raise HTTPException(status_code=401, detail="Missing bearer token")
    token = authorization.removeprefix("Bearer ")
    try:
        signing_key = jwks_client.get_signing_key_from_jwt(token)
        return jwt.decode(
            token,
            signing_key.key,
            algorithms=["EdDSA", "ES256"],
            audience=TARGET_DOMAIN,
            issuer=ISSUER,
            options={"require": ["exp", "iat", "iss", "aud"]},
        )
    except jwt.PyJWTError as exc:
        raise HTTPException(status_code=401, detail="Invalid token") from exc


@app.post("/v1/embed")
def embed(req: EmbedRequest, authorization: str | None = Header(default=None)) -> list[float]:
    claims = verify_token(authorization)  # claims["sub"] is the Encord user's email

    if req.content_type == "text":
        vector = my_model.embed_text(req.content)
    elif req.content_type == "b64_image":
        image_bytes = base64.b64decode(req.content)
        vector = my_model.embed_image(image_bytes)
    else:
        raise HTTPException(status_code=400, detail="Unsupported content_type")

    return [float(x) for x in vector]
```

## Troubleshooting

| Symptom | Likely cause |
| - | - |
| Browser console shows a CORS error | Your endpoint isn't answering the `OPTIONS` preflight, or the `Access-Control-Allow-Origin` doesn't match the Encord app origin. |
| Request fails with a network error for some users only | Those users can't reach your endpoint (for example, they're off VPN). |
| Your endpoint returns `401` | `aud` doesn't match: check that your Target domain includes `https://` and matches your code exactly. Also check `iss` uses the right region (`api.encord.com` or `api.us.encord.com`). |
| Search runs but results look random | The query vector doesn't come from the same model or normalization as your imported embeddings. |
| Search fails after your endpoint returns `200` | The array length doesn't match your embedding size, or the response isn't a plain JSON array of numbers. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.