Skip to main content

Supported videos

Data integrity is a top priority at Encord. Modern browsers give no control over deterministically displaying frames and rendering videos. Therefore, we have introduced security mechanisms to ensure that annotations are correctly displayed in the browser. Please follow the guidelines below to ensure those security mechanisms will be actively working for you.

Supported file formats and browsers

Supported files#

Currently we support the following file types:

  • MPEG-4 (.mp4)
  • WebM (.webm)

Supported browsers#

We strongly encourage you use Google Chrome. Please note that we have not tested frame synchronisation and our security mechanisms on other browsers, so we strongly encourage to use Google Chrome for annotating data and using our platform in general.

We also recommend turning off hardware acceleration in Chrome. Hardware acceleration adds another layer of indeterminism to video rendering in the browser. To eliminate unexpected behaviour it is best to disable it.

Frame synchronisation

Our servers use the FFmpeg project to extract frames from a video one by one. Every label for a video is mapped to a specific frame index in that video. By using FFmpeg in our backend, we have a reliable way to map the labels to the exact frames in the video.

This process is not as straight forward in the browser. When a video gets embedded in the browser, we have no control over how exactly the individual frames are displayed when the video is played. Even if you upload a video which has a constant frame rate of 30 fps, this unfortunately does not guarantee that the browser will play exactly 30 frames in a second. In order to seek individual frames we have to seek specific timestamps, and thus we are at the mercy of the browser to decide which frames start and end at which timestamps.

We have worked hard to create the necessary safety mechanisms to detect possible issues and provide a robust solution to ensure labels are saved to the correct frame index.

Frame synchronisation issue causes and solutions#

We report frame synchronisation issues in the dataset view after a video has been uploaded. Hover over the yellow triangle in the video row to read what issues we have found with the video.

We additionally report frame synchronisation issues when the video is loaded in the label editor.

Our recommended solution is to re-encode problematic videos in your dataset. You can use Encord to re-encode your videos using the following steps

  1. Navigate to your dataset in the dataset settings
  2. Select all the videos that need re-encoding
  3. Hit the "Re-encode" button
  4. Wait until you see the re-encoded videos added to the dataset. The re-encoded videos will have the same title as the selected videos, but they additionally have "_normalised" appended in the title
  5. Optionally remove the previously selected videos to avoid having problematic videos being labelled

When you click the re-encode button, our platform will apply the least invasive transformation to the video to fix the issues. This is so that the re-encoded video and the original video are as similar as possible which will reduce possible frictions in your integration pipeline with the Encord platform. You can read more about what happens during re-encoding in the examples below.

Re-encoding a video to fix frame synchronisation issues.

To detect some of these problems you can inspect your video with ffprobe. You will need to download FFmpeg to do so.

Audio frames

A video has "audio frames" whenever the video has any audio with it. We have observed unexpected behaviour where the browser video player would sometimes increase the display length of specific frames when audio frames were present. As we can only seek frames by timestamps in the browser, whenever we detect audio frames we recommend to remove those.

Detecting the problem:

  • Check the warnings on the Encord platform
  • Listen for sound in the video. Unfortunately this is not a guarantee as sometimes there can be audio frames even when there is no audible sound present
  • By running
ffprobe -i $YOUR_FILE -show_streams -select_streams a -loglevel error

If there is no output, there are no audio frames. If you see some output, there are some audio frames.

Fixing the problem: Re-encode the video via the Encord platform. This will copy all the video frames but will drop all the audio frames.

Ghost frames
An issue that we call "ghost frames" comes from having video frames with negative timestamps. This could possibly come from videos which are cut at the beginning, but where a keyframe with a negative time stamp is kept.

Different video players will display between 0 and all of the frames with negative timestamps. With no control over the browser's video player, we cannot deterministically tell how many negative keyframes the browser will display.

We want to make sure that we are not labeling any negative timestamp frames. Therefore, we require you to re-encode the video which will remove any negative timestamps.

Detecting the problem:

  • Check the warnings on the Encord platform
  • By using different media players and observing whether some of them start at different frames in the video.
  • By running
ffprobe -show_packets -select_streams v -read_intervals %+3 -of json $YOUR_FILE

This will show the video packets of the first 3 seconds in a JSON format. You will see an output such as

    [...]    "packets": [        {            "codec_type": "video",            "stream_index": 0,            "pts": -16384,            "pts_time": "-1.066667",            "dts": -16896,            "dts_time": "-1.100000",            "duration": 512,            "duration_time": "0.033333",            "size": "71396",            "pos": "37476",            "flags": "KD"        },        [...]

If in any of the packets you have a negative value for pts, you know that there might be some ghost frames.

Fixing the problem: Re-encode the video via the Encord platform. These are the transformations we apply when we detect ghost frames:

  • The audio frames are dropped
  • The video frames are re-encoded (i.e. the images in the videos are decompressed and compressed again)
  • All unsupported file formats are converted to .mp4 files
  • Corrupted frames are dropped
Variable frame rates
Some videos have variable frame rates. This could come for example from a dashcam which tries to write frames as fast as it can.

Detecting the problem:

  • Check the warnings on the Encord platform
  • Run the same ffprobe command as for ghost frames (above) and check if the distance between the pts is variable

Fixing the problem: Re-encode the video via the Encord platform. These are the transformations we apply when we detect a variable frame rate:

  • A constant frame rate is applied to the video. This requires some dropping and duplicating of frames and is handled by the -vsync cfr FFmpeg flag
  • All the transformations that are applied to videos with ghost frames

Existing labels before re-encoding

In some cases you might already have some existing labels that you upload on our platform. If this is the case and we suggest you to re-encode the video, then your new labels might be misplaced. This could be the case if

  • The labels were created incorrectly before, because of frame synchronisation issues on other platforms
  • You have labels for a variable frame rate video and now the frame rate has changed

If you upload labels and need to re-encode your video we strongly encourage you to check for the correctness of your pre-existing labels after re-encoding the video.

If you believe that your labels are misaligned, please do reach out to the Encord team so we can help you with the uploading process.