Data integrations

❗️

CRITICAL INFORMATION

Only users that are part of an Organization can create data integrations.

Create and manage data integrations from the Integrations section. Click Add integration to create a new integration.

Encord can integrate with the following cloud storage providers:

• AWS S3.
• Azure Blob Storage.
• Google Cloud Storage storage.
• Open Telekom Cloud OSS.

User roles and permissions

Integrations have the following user based access controls.

Add and remove users

ℹ️

Note

The user who creates the cloud integration becomes its administrator and gains immediate access to use it. Only integration admins can add users to an integration.

Users must be added to a cloud integration before they can use it to upload data to Encord.

Add users

Add individual users to an integration:

1. Click the Add or remove users button on the integration.
   A dialog appears.

2. Ensure the Invite tab is selected.

3. Ensure the Individual tab is selected

4. Type the email addresses of the users you want to share the integration with and press Enter on your keyboard to confirm.

5. Select the integration user role you want to assign to the users.

6. Click Add.

Add user groups to an integration:

1. Click the Add or remove users button on the integration.
   A dialog appears.

2. Ensure the Invite tab is selected.

3. Click the Groups tab.

4. Select a user group you want to add to the integration

5. Select the integration user role you want to assign to all users in the group.

6. Click Add.

Remove users

ℹ️

Note

Individuals with the Admin user role cannot be removed from an integration. To remove an admin, contact [email protected]

To remove individual users from an integration:

1. Click the Add or remove users button on the integration.
   A dialog appears.

2. Select the Manage tab.

3. Ensure the Individuals tab is selected.

4. Click the delete icon next to the user you want to remove from the integration.

To remove a user group from an integration:

1. Click the Add or remove users button on the integration.
   A dialog appears.

2. Select the Manage tab.

3. Select the Groups tab.

4. Click the delete icon next to the user group you want to remove from the integration.

Strict client-only access

Strict client-only access allows Encord to sign URLs without temporarily processing information on our servers. Strict client-only access is available for all types of private cloud integrations, including Direct access.

👍

Tip

For enhanced security, create a Direct Access integration with Strict client-only access. This combination ensures the highest level of data protection: Strict client-only access prevents Encord from accessing your data, while Direct Access keeps Encord from signing the URLs, maintaining your data's confidentiality.

There are two main consequences of making an integration client-only access:

1. All features that rely on any kind of data conversion will not work. This includes features such as:

• Image groups & image sequences
• Re-encoding videos.
• All models.
• Object tracking.
• Segment-anything model (SAM).

2. Metadata must be specified for all data in the JSON file when adding data to a dataset. The video metadata section shows how to specify the metadata.

Enabling Strict client-only access

Strict client-only access is enabled by checking the Strict client-only access, server-side media features will not be available checkbox in the Advanced settings section when creating an integration.

A green tick and the words Client-only access are visible on all client-only access integrations.

Video metadata

The JSON format allows you to specify videoMetadata for video files. videoMetadata is essential information used by the Label Editor and is crucial for aligning annotations to the correct frame.

❗️

CRITICAL INFORMATION

When the videometadata flag is included in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. To guarantee accurate labels, it is crucial that the metadata you provide is accurate.

ℹ️

Note

videoMetadata must be specified when a Strict client-only access integration is used. In all other cases videoMetadata is optional.
👍 Tip

Encord assumes the videoMetadata is correct and our servers do not download or pre-process your data. This is particularly useful for customers with strict data compliance concerns.

ℹ️

Note

videoMetadata is distinct from clientMetadata. videoMetadata is required for videos when using Strict client-only access, while clientMetadata is optional and can be included for all types of data.

One way to find the required metadata is shown below. Run the following commands in your terminal.

• ffmpeg -i 'video_title.mp4' to retrieve fps, duration, width, and height:

• ls -l 'video_title.mp4' to retrieve the file size:

{
    "videos": [
      {
        "objectUrl": "video_file.mp4",
        "videoMetadata": {
            "fps": 23.98,
            "duration": 29.09,
            "width": 1280,
            "height": 720,
            "file_size": 5468354,
            "mime_type": "video/mp4"
        }
      }
    ]
  }

Custom signed URL expiration time

A custom signed URL expiration time allows you to set the duration that an integration's pre-signed URL remains valid.

ℹ️

Note

Custom signed URL expiration times are only available for AWS, GCP, and OTC integrations.

Shorter signed URL expiration times are more secure, but can negatively impact the annotation and review experiences in the Label Editor. Ensure that the URL expiration time exceeds the amount of time that files from the integration take to annotate and review. The default value for signed URL expiration times when creating an integration is 1 week.

Setting custom signed url expiration times

To set up a custom signed URL expiration time for a new integration:

1. Click the Custom signed URL integration time checkbox in the Advanced settings when setting up your GCP, AWS, or OTC integration.

2. Set the duration for which pre-signed URLs should remain valid. We recommend configuring the URL expiration time to exceed the time required to annotate and review files from the integration in the Label Editor.

Edit custom signed url expiration times

ℹ️

Note

Custom signed URL expiration times are only available for AWS, GCP, and OTC integrations. For Azure integrations,expired SAS tokens must be updated regularly.

To change the custom signed URL expiration time for an existing integration:

1. Click the icon on your integration.

2. Select Update Signed Url Expiration.

3. Set the duration for which pre-signed URLs should remain valid. We recommend configuring the URL expiration time to exceed the time required to annotate and review files from the integration in the Label Editor.

View and join integrations in your Org

Organization Admins can search for and join any data integration that exist within the Organization. Integrations are managed in the Integrations tab of your Organization's dashboard.

When an Organization Admin joins an integration, they are automatically assigned the Admin user role for that integration.

👍

Tip

Integrations can be filtered by the owner of the integration.

👍

Tip

See all integrations you belong to by clicking the Filter by search bar, and selecting My integrations only.