Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.encord.com/llms.txt

Use this file to discover all available pages before exploring further.

Your data is stored under Data > Sources, where it is organized into folders and sub-folders.
Registering your cloud data with Encord only references the files. Your data is not stored on Encord servers.
Registering your data into Encord is a multi-step process:
  1. Set up an AWS integration.
  2. Create an AWS integration in Encord.
  3. Create a Cloud-synced folder to store your data in Encord.
  4. Sync data to the folder.
See our AWS integration documentation for a detailed guide to setting up an integration.

​
Step 1: Set up AWS

Before you can do anything with the Encord platform and cloud storage, you need to configure your cloud storage to work with Encord. Once the integration between Encord and your cloud storage is complete, you can then use your data in Encord. In order to integrate with AWS S3, you need to:
  1. Create a permission policy for your resources that allows appropriate access to Encord.
  2. Create a role for Encord and attach the policy so that Encord can access those resources.
  3. Activate Cross-origin resource sharing which allows Encord to access those resources from a web browser.
  4. Test the integration to make sure it works.
See our AWS integration documentation for a detailed explanation of setting up AWS to work with Encord.
You have the following options to integrate AWS and Encord:

​
Step 2: Create AWS-Encord Integration

Create an S3 bucket to store your files if you haven’t already. Your S3 bucket permissions should be set to be blocking all public access.
In the Integrations section of the Encord platform, click +New integration to create a new integration.
Select AWS S3 at the top of the chooser.
It is essential you do not close this tab or window until you have finished the whole integration process. If you use the AWS UI for integration, we advise opening the AWS console in a separate tab.
See our AWS integration documentation for a detailed explanation of how to set up the AWS integration.

​
Step 3: Create a Cloud-synced Folder

You cannot change URI after folder creation.
  1. Go to Data > Sources.
  2. Click New folder > Cloud-synced folder. The New Cloud-synced folder dialog appears.
  1. Provide the following:
    • Title: Provide a meaningful name for the Cloud-synced folder.
    • Description: OPTIONAL - Provide a meaningful description for the Cloud-synced folder.
    • Select your integration: Select the integration to use from the drop down.
    • Storage path: Specify the storage/file path to your cloud storage. For example: gs://encord-gcp-bucket/CloudSync/ or s3://encord-aws-bucket/CloudSync.
    • Automatically sync data: Automatically syncs data from your cloud storage to Encord once every 24 hours.
    • Metadata ingestion: Enable this toggle to import custom metadata files from your cloud storage. Set your sidecar suffix (default is .metadata.json). Any file matching that suffix is treated as metadata for its paired data file and does not appear as a separate item in your folder.
  2. Click Create. The page for the new Cloud-synced folder appears.

​
Find Storage Path

Finding the Storage path for your folder or object varies across Cloud Storage platforms. AWS Find AWS storage path GCP Find GCP storage path

​
STEP 4: Sync Data Between Encord and Cloud Storage

  1. Go to Data > Sources. The Cloud-synced folder page appears. Sync your Data
  2. Click into your cloud-synced folder. Sync your Data
  3. Click Initiate sync. The sync between the folder and your cloud storage begins.