Skip to main content

AWS S3

To create an AWS S3 integration, click the + Add integration button in the Integrations tab. This will pop up the 'Add bucket integration' chooser. Please select AWS S3 at the top of the chooser.

AWS S3 integration flow​

In order to integrate with AWS S3, you will need to

  1. Create a permission policy for your resources that will allow appropriate access to Encord
  2. Create a role for Encord and attach the policy so that Encord can access those resources
  3. Activate Cross-origin resource sharing which allows Encord to access those resources from a web browser
  4. Create and test the integration

Please note that your S3 bucket permissions should be set to be blocking all public access.

1. Create a permission policy​

Log in to your AWS account. Navigate to your Identity and Access Management (IAM) dashboard. Go to 'Policies' on the left-hand side.

Click on 'Create policy' and then click on 'JSON'.

Navigate to the Integrations tab inside the Encord app, and click the + Add integration button. The Add bucket integration dialog will appear. It's essential you don't close this window until clicking the Create button to creation the integration. Inside the Create policy section of the window, click the copy icon to copy the JSON and paste into the AWS policy JSON editor opened in the previous step. Replace the arn:aws:s3:::YourBucketARN value for "Resource" with your bucket's Amazon Resource Name (ARN). You can find your ARN in the 'Properties' tab of your S3 bucket, as shown below. For example, for the bucket below, we want the value at "Resource" to be arn:aws:s3:::com.encord.docs-demo/*.

If you do not expect to be creating image groups, you can take out the s3:PutObject action.

You can also use this tool by AWS to create a policy. It is helpful if you are unsure about how to correctly define the policy file.

Click the Next:tags button to add any tags according to your organization's resource tagging policy. Encord does not require any tags to function. Click the Next:Review button to proceed to the final step. Name your policy something descriptive (we will use it in the next step) and click the Create policy button. You now have a policy to apply to Encord once it has a role defined.

2. Create a role for Encord​

Now go to 'Roles' on the left-hand side.

Click the Create role button. Select 'AWS Account' as the Trusted entity type and under the the 'An AWS Account section', select 'Another AWS account'.From the Integrations window in the Encord app, copy the Encord AWS account ID and External ID and paste them into the relevant areas of the AWS trusted entity creation form. You will have to check Require external ID under Options in the form to reveal the External ID entry form.

Click the Next button. Attach the policy we created in step 1 and click the Next button. Name your role something descriptive and click the Create role button. This is the role Encord will use to access this S3 bucket.

We now need to let the Encord platform know the details of this role. In the AWS Console, click on the role you just created to open the role details page. Copy the Role ARN.

Paste the ARN into the second entry form under Role ARN in the integration window. Copy only the text after the final / to select the role name, and paste that into the first entry area above the ARN.

Now that the role is setup, the next step is to enable Cross-origin resource sharing (CORS) on your S3 bucket to ensure data can successfully be loaded in your browser while using the Encord app. Correctly setting up the CORS permissions is a critical step in completing your S3 integration, read below for detailed instruction.

3. Allow Cross-origin resource sharing (CORS)​

Expand the Display CORS Policy heading in the integrations window. It will look something like this:

Copy the CORS JSON policy. Navigate to your S3 bucket and go to the 'Permissions' tab. Paste this JSON into the CORS editor at the bottom and save.

4. Create and test the integration​

All your policies and roles are now set. Click the Create button at the bottom of the pop-up. The integration will now appear in the list of integrations in the 'Integrations' tab.

To test that Encord can sync with your S3 bucket, click on the sync icon next to the bin icon.

note

This test checks that Encord can assume the role that that has defined for it. It does not check that we can necessarily access your buckets. If this test passes but data on-boarding still fails, please check Encord has bucket permissions and that the object URLs are correct.

If the sync is successful, you will see a message saying that the integration is functional. You may now proceed to the data creation flow to create a dataset from your S3 data. Here are a few examples and helpful scripts to get you started.