Skip to main content
There are multiple parts to setting up a GCP integration in Encord:
  1. Start setting up the integration in Encord.
  2. Configure Workload Identity Federation in your GCP project.
  3. Set up a CORS configuration.
Ensure all objects in the bucket use the Standard Storage class.

1. Start Setting Up The Integration

  1. Navigate to Workspace Settings > Cloud Buckets and click +New integration.
  1. Select Google Cloud Storage.
  2. Give your integration a meaningful title.
  3. Select Workload Identity Federation (WIF).
  1. Under Choose authentication type, select either Encord service account to follow the steps below, or Workload Identity Federation (WIF) to use short-lived tokens without sharing credentials.

Use Workload Identity Federation

To use WIF, a Workspace admin must first go to Workspace Organization settings > Service integrations, click New service integration, enter a title, set Used for to GCP, and click Create service integration. After selecting Workload Identity Federation (WIF) in the GCP integration form:
  1. Select the GCP service integration that signs the tokens.
  2. Enable the IAM, Security Token Service, and IAM Service Account Credentials APIs. The form links to each API.
  3. Create a workload identity pool and add an OpenID Connect (OIDC) provider. Copy the issuer, attribute mapping, and attribute condition displayed by Encord into the provider configuration.
  4. Paste the provider’s full resource name into Encord. Use the project number, not the project ID, and omit the https://iam.googleapis.com/ prefix.
  5. Create a Google Cloud service account and grant it Storage Object Viewer on the bucket. Grant Storage Object Creator as well if Encord needs to write to the bucket.
  6. Paste the service account email into Encord, then grant Service Account Token Creator to the principal displayed in the form.
Continue with the CORS configuration in Step 3 below, then create and test the integration.

2. Configure Workload Identity Federation

The integration form provides the values you need for the following steps:
  1. Select an Encord service integration that has Used for set to GCP. If none is available, create one from the Service integrations page, then return to the integration form.
  2. Enable the IAM API, Security Token Service API, and IAM Service Account Credentials API in your GCP project.
  3. In Google Cloud, go to IAM & Admin > Workload Identity Federation and create a workload identity pool.
  4. Add a provider to the pool using the settings shown in the Encord integration form:
    • Provider type: OpenID Connect (OIDC)
    • Provider name / ID: encord
    • Issuer (URL): Copy the displayed issuer URL.
    • JWK file: Leave empty so Google fetches the keys from the issuer.
    • Audiences: Default audience
    • Attribute mapping: google.subject = assertion.sub
    • Attribute conditions: Copy the displayed assertion.sub == 'service-integration:<SERVICE_INTEGRATION_UUID>' condition.
  5. Copy the provider’s resource name from Google Cloud into Encord. Use the project number, not the project ID, and omit the https://iam.googleapis.com/ prefix. The resource name has this format:
  1. Create a Google Cloud service account and grant it access to your bucket. Assign Storage Object Viewer for read access and Storage Object Creator if you use features that write to the bucket, including re-encoding data and creating image sequences. Paste the service account email into Encord.
  2. On the service account’s Permissions tab, grant the Service Account Token Creator role to the principal shown in the Encord integration form.
Workload Identity Federation uses short-lived credentials. You do not need to create or share a service account key.

3. Create a CORS Configuration

If you use Infrastructure as Code tools like Terraform, they can overwrite your manual CORS settings. To prevent this, always include the correct CORS policy in your Terraform configuration file.
Applying Encord required CORS settings may overwrite existing configurations on your cloud storage bucket. If your bucket already has CORS settings, make sure to append Encord’s required settings instead of replacing them.
A CORS configuration must be applied to the bucket you want to integrate with Encord. A CORS configuration enables Encord to request resources from the specified bucket.
The only way to create a CORS configuration is using the command line or terminal tool.
Install Google’s Cloud CLI, and use the following gcloud command in your command line or terminal to create a CORS configuration.
  • Replace <YOUR_BUCKET_NAME> with the name of the bucket you want to integrate.
  • Replace /path/to/you/cors.json with the full path to your JSON CORS file. A sample JSON file is provided below.
Click Create in Encord to finish creating the integration, after creating a CORS configuration.

JSON configuration file

The following JSON file can be used to set up the CORS configuration in Step 4 of your GCP integration.
  • Preflight requests are valid for 1 hour. Use the maxAgeSeconds variable to adjust the number of seconds the browser is allowed to make requests before it must repeat the preflight request.
  • Successful browser requests return the Content-Type of the resource in the response, as set by the responseHeader.
  • Byte-range requests (GET/HEAD) require the Range request header and the exposed response headers shown above so large scene files (such as MCAP) can stream in chunks in the browser.
Consult the GCP documentation, to customize the file further.

4. Test Integration

Click the Run a test button on the integration to test the integration.
Paste the URL of any object in the bucket and click Check Encord can access this URL. If the test is successful a green tick appears next to Encord infrastructure and This machine.

Register GCP Data

See the Register cloud data page for guidance on how to register data stored in GCP.
We recommend setting the expiration time for pre-signed URLs (found in your cloud storage settings) to be greater than the time it takes to complete an annotation task. More information can be found in the GCP documentation.

GCP with Unsigned URLs (Securely)

To use unsigned URLs, you must set up a Direct Access integration in Encord.