Supported data

Supported data types

Encord empowers you to work with a diverse range of data types. Whether you're looking to annotate sequences of images, videos, or even DICOM files, we've got you covered!

❗️

CRITICAL INFORMATION

To ensure seamless data loading in Encord, please whitelist https://app.encord.com in any 3rd party VPNs, firewalls, or URL isolators. This helps prevent potential issues with opening your data in Encord.

Comparing file formats

Confused about images, image groups and image sequences? Watch the video, or read below to learn all you need to know!

Comparison of image file types

* Data is lost during the creation of an image sequence when images of varying dimensions are combined into a single image sequence. For more information about this, please see the section on image sequences below.

Supported file formats

🚧

Caution

* Due to Chromium-based browser limitations, TIFF files can only be viewed in the Label Editor using the Safari browser. More details can be found here.

Single images

Single images are individual images that are uploaded to Encord as separate files. Each single image constitutes its own data package, hence images will be listed individually.

We support the following file types for single images:

.jpeg,  .png,   .webp,  .avif,  .bmp,  .tiff*,   .tif*

* Due to Chromium-based browser limitations, TIFF files can only be viewed in the label editor using the Safari browser.

👍

Tip

To learn how to upload images from your cloud storage, follow the links below:

• JSON
• CSV

Image groups

Image groups are collections of images that are grouped together into a single data unit. They can contain images of varying orientations and no data is lost in the process of creating an image group.

We support the following file types for image groups:

.jpeg,  .png,   .webp,  .avif,  .bmp,  .tiff*,   .tif*

🚧

Caution

* Due to Chromium-based browser limitations, TIFF files can only be viewed in the label editor using the Safari browser.

👍

Tip

To learn how to upload image groups from your cloud storage, follow the links below:

• JSON
• CSV

Image sequences

Image sequences are collections of images that are grouped together as a single data unit, and annotated in the same way as videos. As a result, image sequences render faster in the browser (provided images are sequential), and are able to make use of powerful machine learning features such as 'auto-segmentation' and 'automated labeling'.

We support the following file types for image sequences:

.jpeg,  .png,   .webp,  .avif,  .bmp,  .tiff*,   .tif*

ℹ️

Note

* During the creation of image sequences, TIFF files are converted into a different format. Image sequences should always be viewed in Chromium based web browsers such as Chrome, and not in the Safari browser.

👍

Tip

To learn how to upload image sequences from your cloud storage, follow the links below:

• JSON
• CSV

Creating image sequences using the Encord platform

If you use the Encord platform to upload your images and create image sequences, Encord will automatically group images of different heights and widths into separate image sequences. This is done to preserve the quality of each frame within the resulting image sequences.

Creating image sequences from cloud storage using JSON or CSV

If you upload your data from cloud storage using JSON or CSV file formats, you retain full control over which image sequence every image will end up in. However, if images you choose to form into a sequence have varying dimensions they will be homogenized. This means that every image within your defined sequence will be made to have the same dimensions as the first image in that sequence. As a result, data may be lost when forming image sequences.

ℹ️

Note

The same applies when uploading image sequences from the SDK.

Creating image sequences using the SDK

For information on how to create any type of dataset using the SDK, please see our dedicated SDK documentation.

Videos

Video files can be uploaded onto the Encord platform and annotated frame by frame, enabling you to track elements within the video. Even though frames are treated as individual images when being annotated, the video itself constitutes a single data package.

The following video file types are supported by Encord:

.mp4  .mov .webm  .mkv

For optimized performance we recommend using the following video formats:

• .mp4 file with h.264.
• .mp4 with vp9.

File uploads must be less than 1 GB or 100,000 frames (1hr at 30FPS). Please see our guidelines for optimal performance for more details.

👍

Tip

To learn how to upload videos from your cloud storage, follow the links below:

• JSON
• CSV

Resizing videos

Videos can be re-sized to fit the required specifications by lowering their resolution and / or frame rate using ffmpeg.

Follow the steps below to reduce the video file size by adjusting the resolution, as well as the number of frames per second (fps).

1. If you haven't already, download and install ffmpeg.
2. Open Command Line (on Windows) or Terminal (on Mac & Linux).
3. Run the command shown below, substituting the values for your desired resolution as well as fps, followed by the path to the video you'd like to downsize.

ffmpeg -i normal-video.mp4 -c:v libx264 -an -movflags faststart -tune zerolatency -f mp4 -vf "scale=-1:1080,fps=40" </Users/Desktop/normal-video-downsampled.mp4>

The example above resizes normal-video-downsampled.mp4 to 40 fps and a resolution of -1:1080px. This means that the video height is set to 1080px, while the -1 adjusts the width so that the aspect ratio remains the same as in the original.

👍

Tip

Click here to learn more about scaling with ffmpeg.

🚧

Caution

Decreasing the resolution and / or frame rate involves decreasing the video quality. Therefore, it may only be suitable for clients who don't rely on high-quality videos.

Pixel aspect ratio

To ensure accurate labels, only videos with a pixel aspect ratio of 1:1 should be used in Encord.

To check the pixel aspect ratio of your file, run the following ffprobe command, where input-file.mp4 is the full file path to the file you want to output the pixel aspect ratio for. ffprobe refers to pixel aspect ratio as 'sample aspect ratio (SAR)'.

ffprobe -v error -select_streams v:0 -show_entries stream=sample_aspect_ratio -of default=noprint_wrappers=1:nokey=1 input-file.mp4

ℹ️

Note

The ffprobe command will output 'N/A' is the file's SAR is not defined in the video's metadata.

Converting videos to images

In machine learning, most infrastructures and codebases are tailored for image data rather than video. Therefore, when working with videos, a common prerequisite is to convert videos into a series of images. This process is essential for training ML models that do not directly support video inputs. We outline two approaches for extracting images from videos, each catering to different needs.

Approach 1: Extracting Every Frame as an Image

ℹ️

Note

This approach requires ffmpeg. Download and install ffmpeg if you haven't already.

This method involves decompressing each frame of a video into separate image files. It's straightforward and useful when you need to analyze or process every frame individually. Navigate to the directory your video is in and run the following command in your Terminal, replacing your_video.mp4 with the name of your video file, and /path/to/output/dir/frame with the path to your output directory.

ffmpeg -i your_video.mp4 -start_number 0 /path/to/output/dir/frame_%d.png -hide_banner

Approach 2: In-Memory Frame Iteration with Python

For scenarios where saving individual image files is unnecessary, you can iterate over video frames directly in memory using Python. This method is efficient for immediate processing or conversion to formats like TFRecords.

ℹ️

Note

This approach requires that the pyav Python library is installed.

import av

# Open the video file
with av.open('your_video.mp4') as container:
    # Loop through streams in the container
    for frame in container.decode(video=0):
        # Process the frame. For example, convert the frame to a PIL Image and show it
        img = frame.to_image()
        img.show()

DICOM

Encord provides native support for Digital Imaging and Communications in Medicine (DICOM) browser rendering and data annotations. Our ontologies allow you to create any type of labeling protocol - for example RECIST, which requires measuring the longest diameter of a lesion.

With the DICOM editor, you can:

• Annotate modalities such as CT, X-ray, and MRI.
• Label using any annotation type in 2D (with 3D in the works), and seamlessly toggle between axial, coronal, and sagittal views.
• Render 20,000 pixel intensities and set custom window widths & levels.
• Natively display DICOM metadata.
• Track and interpolate objects between slices.
• Reduce manual annotations with automation features.

ℹ️

Note

Due to DICOM being a large and open standard Encord accepts most data types. Examples include: CT, MR, US, MG, TOMO, but many more are accepted.

👍

Tip

To learn how to upload DICOM files from your cloud storage, follow the link below:

• JSON

Multi-frame DICOM Support

We support multi-frame DICOMs, making it even easier and faster to load data onto our platform! Multi-frame DICOM support can save time and streamline workflows due to a reduced amount of header data repetition.

Simply upload your multi-frame DICOM files like you would any other files - either via the user interface, or using our python SDK.

Caching DICOM data

Caching allows frequently accessed DICOM files to be stored securely in your local browser memory, allowing them to be retrieved quickly and efficiently. As a result you can expect faster load times when accessing medical imaging data.

This is particularly beneficial for busy healthcare professionals who require quick access to patient records and diagnostic imaging.

Data Optimization

This section will run through all topics related optimizing your data to work with Encord’s features and modern web browsers.

Re-encode your data

ℹ️

Note

Re-encoding is only available to Enterprise customers. Please contact [email protected] for more information.

If we detect that an uploaded video cannot be added to a dataset, the video must be re-encoded. This simply means bringing the video in line with Encord's requirements to avoid frame synchronization issues and ensure accurate labels.

Your video needs re-encoding if you see after successfully uploading your video. Hover over the icon to read what issues we have found with the video.

You can always check the progress of any process by clicking the icon. The number displayed in red signifies how many new notifications you have.

ℹ️

Note

The way in which individual frames within different browsers are displayed may vary. To avoid browser-related frame synchronization issues we strongly recommend the Google Chrome browser when using our platform.

Re-encode (Auto)

Auto re-encoding serves common frame synchronization issues and ensures there is no quality loss during re-encoding. If for example the only issue is that audio is present in the video, auto re-encoding only removes the audio while keeping the video as-is.

To automatically re-encode your video, simply select the file in question and click the Re-encode (auto) button to prevent any frame synchronization issues from arising. A notification telling you your video is re-encoded should appear at the top of your screen.

Re-encode (Full)

The Force full re-encoding button converts the video to a specific frame rate and video codec regardless of why frame synchronization issues are occurring. A full re-encoding takes longer and can lead to issues with existing labels.

Re-encode locally

Some cases, such as corrupted metadata, might require you to re-encode your data locally before uploading them to the Encord platform.

To do this, try the following ffmpeg command, replacing "video.mp4" with the name of the file you would like to re-encode, and "re-encoded-video.mp4" with the name you'd like the re-encoded file to have:

ffmpeg -err_detect aggressive -fflags discardcorrupt -i video.mp4 -c:v libx264 -movflags faststart -an -tune zerolatency re-encoded-video.mp4

Frame synchronization

Our servers use the FFmpeg project to extract frames from a video one by one. That way every label for a video is mapped to a specific frame index within that video, providing a reliable way to map labels to the specific frames in the video.

This process is not as straight forward in the browser. When a video gets embedded in the browser, we have no control over how exactly the individual frames are displayed when the video is played. Even if you upload a video which has a constant frame rate of 30 fps, this unfortunately does not guarantee that the browser will play exactly 30 frames in a second.

In order to seek individual frames we have to seek specific timestamps, and thus we are at the mercy of the browser to decide which frames start and end at which timestamps. We have worked hard to create the necessary safety mechanisms to detect possible issues and provide a robust solution to ensure labels are saved to the correct frame index.

Frame synchronization issue causes and solutions

We report frame synchronization issues in the dataset view after a video has been uploaded. Hover over the icon in the video row to see which issues have been identified.

We also report frame synchronization issues when the video is loaded in the label editor with the following warning:

Our recommended solution is to re-encode problematic videos in your dataset. You can use Encord to re-encode your videos using the following steps:

1. Navigate to your dataset in the dataset settings.
2. Select all the videos that need re-encoding.
3. Hit the Re-encode button.
4. Wait until you see the re-encoded videos added to the dataset. The re-encoded videos will have the same title as the selected videos, with "_normalized" appended in the title.
5. Optionally remove the previously selected videos to avoid having problematic videos being labeled.

When you click the re-encode button, our platform will apply the least invasive transformation to the video to fix the issues. This is so that the re-encoded video and the original video are as similar as possible which will reduce possible frictions in your integration pipeline with the Encord platform. You can read more about what happens during re-encoding in the examples below.

To detect some of these problems you can inspect your video with ffprobe. You will need to download FFmpeg to do so.

ffprobe -i $YOUR_FILE -show_streams -select_streams a -loglevel error

ffprobe -show_packets -select_streams v -read_intervals %+3 -of json $YOUR_FILE

    [...]
    "packets": [
        {
            "codec_type": "video",
            "stream_index": 0,
            "pts": -16384,
            "pts_time": "-1.066667",
            "dts": -16896,
            "dts_time": "-1.100000",
            "duration": 512,
            "duration_time": "0.033333",
            "size": "71396",
            "pos": "37476",
            "flags": "KD"
        },
        [...]

Existing labels before re-encoding

In certain scenarios, you may have pre-existing labels that you want to upload to our platform. If we advise you to re-encode the video, there is a possibility that your new labels might become misplaced. This situation can arise if:

• The labels were created incorrectly before, because of frame synchronization issues on other platforms
• You have labels for a variable frame rate video and now the frame rate has changed

If you upload labels and need to re-encode your video, we strongly encourage you to double-check the accuracy of your pre-existing labels after re-encoding the video.

If you believe that your labels are misaligned, please do reach out to the Encord team so we can help you with the uploading process.

Fixing cloud storage buffering issues

A number of factors determine how fast remote content is rendered and displayed in a web browser:

• Geographical distance from host bucket.
• Local internet speed.
• Use of a Virtual Private Network (VPN).
• The bitrate of the video file being requested.

While these factors are outside of Encord's influence, we have put together a set of common issues when loading data from private cloud, along with possible solutions to improve your user experience.

ℹ️

Note

You can see the speed at which a video is being downloaded at the bottom of the 'editor pane' in the 'label editor' on Encord's platform. It also displays the download speed required to display a video without buffering.

Geographical distance from host

The physical distance between you and the server hosting your data will affect how quickly content can be accessed. If, for example, an Amazon S3 bucket is hosted in the 'EU (London) eu-west-2' region, then users from India, or Brazil might experience latency issues due to data having to travel a long way.

The solution is to make sure that users always connect to their closest server, reducing the time it takes for data to travel from one place to another - however, setting this up differs between different cloud storage providers.

Local internet speed

Test your internet connection to make sure it is fast enough to download your video from cloud storage without buffering. Consistently low internet speeds might mean your provider isn't providing you with high enough internet speeds to load large quantities of data from your cloud storage.

You can see the speed at which a video is being downloaded at the bottom of the 'editor pane' in the 'label editor' on Encord's platform. It also displays the download speed required to display a video without buffering.

VPNs

VPNs can slow down your connection by requiring the data travel through the VPN's server before arriving to you.

We recommend you turn off your VPN when using Encord's platform if you are experiencing buffering issues.

High bitrates in videos

Video content itself varies widely in terms of frames rate, compression, definition and size. The most important factor when determining how quickly a remote video will load in the browser is the bitrate. The higher the bitrate, the more data will need to be loaded per second of video.

👍

Tip

To make sure videos load and display quickly in the Label Editor:

1. An annotator’s download speed from private cloud storage has to be at least 3-4 times the video bitrate.
2. Have enough keyframes to ensure smooth frame-by-frame navigation (roughly every 50 frames for high-resolution videos).

Lowering a video's bitrate will reduce the time it takes to load from your cloud storage. This can be achieved using tools such as FFmpeg. Please see our section on resizing videos for information on how this can be achieved.

ℹ️

Note

Lowering the bitrate involves decreasing the video quality. Therefore, it may only be suitable for clients who don't rely on high-quality videos.

Compatibility

Data integrity is a top priority at Encord.

Modern browsers give no control over deterministically displaying frames and rendering videos - therefore, we've introduced security mechanisms to ensure that annotations are correctly displayed in the browser. Please follow the guidelines below to ensure those security mechanisms will be working for you.

Supported data formats

We support all the major video and image formats, however not all video formats are supported by all browsers for playback. If you have trouble with playback in the Label editor, it may be a browser issue. Refer to your browser documentation for more information if issues persist after implementing the guidelines listed below.

Please see the sections on images and videos for more information.

Supported browsers

We recommend the Google Chrome or Brave web-browsers when using Encord.

ℹ️

Note

We have not tested frame synchronization and security mechanisms on other browsers, so we strongly encourage to use Google Chrome for annotating data and using our platform in general.

Additionally, we recommend disabling hardware acceleration in Chrome. Hardware acceleration can introduce another layer of uncertainty to video rendering in the browser, potentially leading to unexpected behavior. By disabling it, you can ensure a more stable and consistent experience when working with videos on our platform.