Skip to main content

Encord Best Practices - Data Management

General​

Although our platform is able to handle large volumes of data, we strongly recommend working within the following limits to ensure smooth operations.

This is to ensure that you have the optimal customer experience and avoid longer loading times.

note
The following limits apply for both uploading via the SDK and the GUI.

When using the platform we recommend using the latest version of the Chrome browser.

Upload Limitations​

Although we support uploads of any size, we limit the number of uploaded files per upload to ensure optimal performance of our platform.

File sizes​

Videos / images: Less than 2 GB, or 200,000 frames (2hrs @ 30FPS)

Please note that most browsers intelligently buffer large videos. However, if your team experiences slow loading times on large videos, reduce video size until loading operations are smooth.

JSON: 100 MB

The JSON file is used to specify URLs of files to load from a private cloud integration. The JSON upload specification file itself may not exceed 100 MB.

Annotations / label blobs: 50 MB

Commonly referred to as label rows in our documentation, label blob uploads per label row should be kept under 50 MB.

Number of files​

Videos / Images: 1,000

Separate from the size of the JSON upload specification file, the number of media files (i.e. videos, images, DICOM series) per upload batch should be limited to 1,000 files or fewer. Please observe this limit in concert with the limit of 100 MB for the JSON specification file. If you need to upload a large amount of data, we recommend to split the upload into batches and upload these separately.

Resolution of data / media​

Our platform is browser based, hence the resolution of any videos and images should be a maximum of 4k pixels or 3840 x 2160 pixels to avoid issues with displaying them. If you have other requirements, please contact us first and we will help evaluate suitability for specific use-cases.

Data Formats​

Video Formats​

We support a range of formats and codecs. For optimized performance we recommend using the following video formats:

  • .mp4 file with h.264
  • .webm with vp9

Please ensure that videos have a constant frame rate and no audio tracks. The label editor will inform you when we detect audio present in the video files. You can remove audio tracks in videos already in the Encord platform by using the media re-encoding feature in our platform, detailed here.

Other formats and codecs may be supported, but we cannot guarantee compatibility, for more information please see our blog.

In case of any questions or doubts please contact us at support@encord.com

Image Formats​

The platform supports a range of image formats. For most use cases we recommend to use:

  • .png
  • .jpeg

While other formats may be supported, we cannot guarantee compatibility. We recommend converting other image formats (e.g. .tiff) into the formats above.

In case of any questions or doubts please contact us at support@encord.com.

Projects Size Limitations​

Attaching a dataset to a project will turn each data item in the dataset into a labeling task in that project. Labels created in one project don't affect labels in another project, even connected to multiple datasets. You can attach as many datasets and labeling tasks to a project as you wish, and the platform allows you to constantly add new data to any on-going project.

However, in our experience very large projects (those with many tasks and / or annotations) will impact the loading time and hence the user experience and efficiency when using the platform. Keep in mind the working environment of your annotation team in addition to that of your operations or engineering team.

For this reason we recommend some best practices on project sizes on the platform to ensure smooth operation for you and your team.

High Level Guidance (for optimal performance)​

Projects do not expire, however under heavy usage or projects that continually grow their datasets and labeling tasks, the important limits are indicated in terms of:

  • How much data? Less than 100 GB per project

  • How many labels? Less than 1 million per project

  • How many tasks? Less than 5,000 per project

💡 Even if you stay within these limits, we recommend to create a new project every 6 months and archive the existing project