GCP storage

Suggest Edits

There are two parts to setting up a GCP integration in Encord:

  1. Setting up a GCP bucket on the Google Cloud Platform so that it can be integrated with Encord. This will include setting up a CORS configuration for your bucket.

  2. Integrating your GCP bucket with Encord's platform. Strict client-only access can also be added in this step if necessary.

1. Setting up a GCP bucket

To get started, please log into your Google Cloud account and navigate to the GCP bucket you wish to integrate with Encord.

Creating a service account for Encord

In the bucket's PERMISSION tab, click Grant Access to add a service account for encord.

Add 'cord-c[email protected]' as a New Principal and assign the following roles:

  • Storage Object Viewer

  • Storage Object Creator

Creating a CORS configuration

A CORS configuration needs to be applied to the bucket you wish to integrate with Encord. This will enable your user's browsers to request resources from the specified bucket.

Example gsutil command

Below is an example gsutil command to be used in your command line or terminal to create a CORS configuration.

ℹ️

Note

The only way to create a CORS configuration is using the command line, or terminal tool.

gsutil cors set example_cors_file.json gs://example_bucket
  • Replace 'example_bucket' with the name of the bucket you wish to integrate.
  • Replace 'example_cors_file' with the name of your CORS file. A sample JSON file is provided.

JSON configuration file

Below is a simple JSON file that should work with your GCP bucket. 

[
    {
      "origin": ["https://app.encord.com", "https://api.encord.com", "https://dicom.encord.com"],
      "method": ["GET"],
      "responseHeader": ["Content-Type"],
      "maxAgeSeconds": 3600
    }
]

ℹ️

Note

  • Preflight requests are valid for 1 hour. Use the maxAgeSeconds variable to adjust the number of seconds the browser is allowed to make requests before it must repeat the preflight request.
  • Successful browser requests return the Content-Type of the resource in the response, as set by the responseHeader.

To customize the file further please consult the GCP documentation.

2. Creating the integration on Encord

Now that your GCP storage has been set up and configured, you are ready to create an integration in Encord.

Navigate to the Encord app and select Settings from the drop-down on the top right. Open the Integrations tab.

Click + Add integration.

Create the integration by selecting GCP at the top of the chooser. Enter a name for the integration, and enter the name of the bucket you wish to make available in the second dropdown of the GCP integration window.

Optionally check the box to enable Strict client-only access, server-side media features will not be available if you would like Encord to sign URLs, but refrain from downloading any media files onto Encord servers. Read more about this feature here.

Click Create to create the GCP integration.

3. Testing your integration

To test that Encord can sync with your GCP bucket, click on the icon shown below.

Enter the URL of an object you would like to test the connection with and click Check Encord can access this URI. If the test is successful a green tick will appear next to Encord infrastructure as well as This machine, as seen in the screenshot for a GCP test below.

4. Creating a dataset with GCP data

Please navigate to the private cloud integration page for guidance on how to create a dataset with data stored in GCP.

Updated 27 days ago