> ## Documentation Index > Fetch the complete documentation index at: https://docs.encord.com/llms.txt > Use this file to discover all available pages before exploring further. # Annotator Training Follow these steps to set up annotator training in Encord. The annotator training workflow enables you to assess the accuracy and performance of your annotation workforce. ## Overview 1. [Add your training files to Encord](/end-to-end/BenchmarkQA/annotator-training#step-1%3A-add-files-to-encord). 2. [Create a Benchmark Project](/end-to-end/BenchmarkQA/annotator-training#step-2%3A-create-a-benchmark-project) Establish ground truth labels by having a trusted expert annotate the data. This project must be completed before annotator training begins. 3. [Set up Annotator Training Projects](/end-to-end/BenchmarkQA/annotator-training#step-3%3A-create-annotator-training-projects) You must create one training Project per annotator. Use the same Dataset as the Benchmark Project in each annotator training Project. Annotators label the data, and their work is compared against the gold standard created in the benchmark. 4. [Annotators Label Training Tasks](/end-to-end/BenchmarkQA/annotator-training#step-4%3A-annotator-training) Annotators must complete all the training tasks assigned to them. 5. [Evaluate Annotator Performance](/end-to-end/BenchmarkQA/annotator-training#step-5%3A-evaluate-annotators) Use the provided SDK script to compare annotator labels with the benchmark. Analyze the results to assess accuracy and provide targeted feedback. ## STEP 1: Add Files to Encord You must first add your data Encord. These files are used to train your annotators. Select your cloud provider:
Clickable Div
AWS Integration GCP Integration Azure Integration OTC Integration Wasabi Integration Oracle Integration Direct Access Integration
1. Navigate to **Data** > **Files & Folders** 2. Click the **+ New folder** button to create a new folder. Select the type of folder you want to create.
3. Give the folder a meaningful name and description. 4. Click **Create** to create the folder. The folder is listed in **Files & Folders**. To register files from cloud storage into Encord, you must create a JSON file specifying the files you want to upload. While you can [use a CSV file](/platform-documentation/Curate/add-files/index-register-cloud-data#csv-format), we strongly recommend using JSON files for uploading cloud data to Encord for better compatibility and performance. Find helpful scripts for creating JSON files for the data registration process [here](/platform-documentation/Curate/add-files/index-register-cloud-data#helpful-scripts-and-examples). All types of data (videos, images, image groups, image sequences, and DICOM) from a private cloud are added to a Dataset in the same way, by using a JSON or CSV file. The file includes links to all images, image groups, videos and DICOM files in your cloud storage. For a list of supported file formats for each data type, go [here](/platform-documentation/General/general-supported-data) Encord supports file names up to 300 characters in length for any file or video for upload. Encord enforces the following upload limits for each JSON file used for file registration: * Up to 1 million URLs * A maximum of 500,000 items (e.g. images, image groups, videos, DICOMs) * URLs can be up to 16 KB in size Optimal upload chunking can vary depending on your data type and the amount of associated metadata. For tailored recommendations, contact Encord support. We recommend starting with smaller uploads and gradually increasing the size based on how quickly jobs are processed. Generally, smaller chunks result in faster data reflection within the platform. BEST PRACTICE: If you want to use Index or Active with your video data, we STRONGLY RECOMMEND using custom metadata (`clientMetadata`) to specify key frames, custom metadata, and custom embeddings. For more information go [here](/platform-documentation/Curate/custom-metadata/import-custom-metadata) or [here for information on using the SDK](/sdk-documentation/index-sdk/custom-metadata/sdk-import-custom-metadata). For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). The information provided about each of the following data types is designed to get you up and running as quickly as possible without going too deeply into the why or how. Look at the template for each data type, then the examples, and adjust the examples to suit your needs. If `skip_duplicate_urls` is set to `true`, all object URLs that exactly match existing images/videos in the dataset are skipped. #### [Videos](#videos) **Video Metadata** When the `videoMetadata` flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. To guarantee accurate labels, it is crucial that the `videoMetadata` provided is accurate. ```json Template theme={"dark"} { "videos": [ { "objectUrl": "cloud-path-to-your-video-1" }, { "objectUrl": "cloud-path-to-your-video-2", "videoMetadata": { "fps": frames-per-second, "duration": duration-in-seconds, "width": frame-width, "height": frame-height, "file_size": file-size-in-bytes, "mime_type": "MIME-file-type-extension" } } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "videos": [ { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/BerghouseLeopardJog.mp4" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/blue_bus_video.mp4" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/bluemlisalphutte_flyover.mp4" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/boat_lake_normalized.mp4" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/boat_lake.MP4" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/boat_lake.MP4" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/snow_sled.MOV" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/cyclists.MP4" } ], "skip_duplicate_urls": true } ``` ```json Video Metadata theme={"dark"} { "videos": [ { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/video_file.mp4", "videoMetadata": { "fps": 23.98, "duration": 29.09, "width": 1280, "height": 720, "file_size": 5468354, "mime_type": "video/mp4" } } ], "skip_duplicate_urls": true } ``` #### [Audio Files](#audio-files) The following is an example JSON file for uploading two audio files to Encord. Waveform generation for long audio files can cause lag. To avoid this, generate the waveform offline and upload it with the audio file. For more information go [here](/platform-documentation/Curate/index-audio#long-audio-files). * **Template:** Imports audio files with an Encord title. * **Audio Metadata:** Imports one audio file with the `audiometadata` flag. When the `audiometadata` flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. To guarantee accurate labels, it is crucial that the metadata you provide is accurate. ```json Template theme={"dark"} { "audio": [ { "objectUrl": "" }, { "objectUrl": "", "title": "my-custom-audio-file-title.mp3" } ], "skip_duplicate_urls": true } ``` ```json Audio Metadata theme={"dark"} { "audio": [ { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/audio_file.mp3", "audioMetadata": { "duration": 23.98, "file_size": 2900000, "mime_type": "audio/mp3", "sample_rate": 44100, "bit_depth": 24, "codec": "mp3", "num_channels": 2 } } ], "skip_duplicate_urls": true } ``` #### [PDFs](#pdfs) The following are example JSON files for uploading PDFs to Encord. ```json Template theme={"dark"} { "pdfs": [ { "objectUrl": "" }, { "objectUrl": "", "title": "my-document-02.pdf" }, { "objectUrl": "", "title": "my-document-03.pdf", "pdfMetadata": { "fileSize": 300, "numPages": 5 } } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "pdfs": [ { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/pdfs/my-document-01.pdf", }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/pdfs/my-document-02.pdf", "title": "my-document-02.pdf" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/pdfs/my-document-03.pdf", "title": "my-document-03.pdf", "pdfMetadata": { "fileSize": 300, "numPages": 5 } } ], "skip_duplicate_urls": true } ``` #### [Text Files](#text-files) The following are example JSON files for uploading text files to Encord. ```json Template theme={"dark"} { "text": [ { "objectUrl": "" }, { "objectUrl": "", "title": "my-file.html" }, { "objectUrl": "", "title": "my-plain-text-file.txt", "textMetadata": { "fileSize": 200, "mime_type": "text/plain" } }, { "objectUrl": "", "title": "my-csv-file.csv", "textMetadata": { "fileSize": 250, "mime_type": "text/csv" } }, { "objectUrl": "", "title": "my-xml-file.xml", "textMetadata": { "fileSize": 300, "mime_type": "application/xml" } }, { "objectUrl": "", "title": "my-html-file.html", "textMetadata": { "fileSize": 400, "mime_type": "text/html" } } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "text": [ { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/text/my-file-01.html" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/text/my-plain-text-file-01.txt", "title": "my-plain-text-file-01.txt" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/text/my-plain-text-file-02.txt", "title": "my-plain-text-file-02.txt", "textMetadata": { "fileSize": 200, "mime_type": "text/plain" } }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/text/my-csv-file-01.csv", "title": "my-csv-file-01.csv", "textMetadata": { "fileSize": 250, "mime_type": "text/csv" } }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/text/my-xml-file-01.xml", "title": "my-xml-file-01.xml", "textMetadata": { "fileSize": 300, "mime_type": "application/xml" } }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/text/my-html-file-01.html", "title": "my-html-file-01.html", "textMetadata": { "fileSize": 400, "mime_type": "text/html" } } ], "skip_duplicate_urls": true } ``` #### [Single Images](#single-images) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). The JSON structure for single images parallels that of videos. **Template:** Provides the proper JSON format to import images into Encord. **Examples:** * **Data** Imports the images only. ```json Template theme={"dark"} { "images": [ { "objectUrl": "file/path/to/images/file-name-01.file-extension" }, { "objectUrl": "file/path/to/images/file-name-02.file-extension" }, { "objectUrl": "file/path/to/images/file-name-03.file-extension", "title": "image-title.file-extension" } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "images": [ { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/0001.jpg" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/0002.jpg" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/0003.jpg" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/DALL%C2%B7E+2022-09-08+18.53.25+-+firefighter+extinguishing+flames+around+computer+in+software+office+overgrown+by+foliage.png" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/DALL%C2%B7E+2022-12-08+22.16.52+-+steampunk+combustion+engine+that+is+fueled+by+data+and+produces+computer+vision+algorithms.png" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/large_images/pexels-ivo-rainha-00057.png" } ] } ``` ```json Image Metadata theme={"dark"} { "images": [ { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/my_image.jpg", "imageMetadata": { "mimeType": "image/jpg", "fileSize": 124, "width": 640, "height": 480 } } ], "skip_duplicate_urls": true } ``` #### [Image groups](#image-groups) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). * Image groups are collections of images that are processed as one annotation task. * Images within image groups remain unaltered, meaning that images of different sizes and resolutions can form an image group without the loss of data. * Image groups do NOT require 'write' permissions to your cloud storage. * If `skip_duplicate_urls` is set to `true`, all URLs exactly matching existing image groups in the dataset are skipped. The position of each image within the sequence needs to be specified in the key (`objectUrl_{position_number}`). **Template:** Provides the proper JSON format to import image groups into Encord. **Examples:** * **Data:** Imports the image groups only. ```json Template theme={"dark"} { "image_groups": [ { "title": "", "createVideo": false, "objectUrl_0": "file/path/to/images/file-name-01.file-extension", "objectUrl_1": "file/path/to/images/file-name-02.file-extension", "objectUrl_2": "file/path/to/images/file-name-03.file-extension", }, { "title": "<title 2>", "createVideo": false, "objectUrl_0": "file/path/to/images/file-name-01.file-extension", "objectUrl_1": "file/path/to/images/file-name-02.file-extension", "objectUrl_2": "file/path/to/images/file-name-03.file-extension" } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "image_groups": [ { "title": "Image group 01", "createVideo": false, "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/0001.jpg", "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/0002.jpg", "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/DALL%C2%B7E+2022-09-08+18.53.25+-+firefighter+extinguishing+flames+around+computer+in+software+office+overgrown+by+foliage.png" }, { "title": "Image group 02", "createVideo": false, "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/thing-0001.jpg", "objectUrl_1": "<https://encord-integration.s3.eu-west-2.amazonaws.com/images/thing-0002.jpg", "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/thing-0003.jpg" } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Image sequences"> #### [Image sequences](#image-sequences) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). * Image sequences are collections of images that are processed as one annotation task and represented as a video. * Images within image sequences may be altered as images of varying sizes and resolutions are made to match that of the first image in the sequence. * Creating Image sequences from cloud storage requires 'write' permissions, as new files have to be created in order to be read as a video. * Each object in the `image_groups` array with the `createVideo` flag set to `true` represents a single image sequence. * If `skip_duplicate_urls` is set to `true`, all URLs exactly matching existing image sequences in the dataset are skipped. <Tip> The only difference between adding image groups and image sequences using a JSON file is that image sequences require the `createVideo` flag to be set to `true`. Both use the key `image_groups`.</Tip> <Note>The position of each image within the sequence needs to be specified in the key (`objectUrl_{position_number}`).</Note> <Info>Encord supports up to 32,767 entries (21:50 minutes) for a single image sequence. We recommend up to 10,000 to 15,000 entries for a single image sequence for best performance. If you need a longer sequence, we recommend using video instead of an image sequence.</Info> **Template:** Provides the proper JSON format to import image groups into Encord. **Examples:** * **Data:** Imports the images groups only. <CodeGroup> ```json Template theme={"dark"} { "image_groups": [ { "title": "<title 1>", "createVideo": true, "objectUrl_0": "<object url>" }, { "title": "<title 2>", "createVideo": true, "objectUrl_0": "<object url>", "objectUrl_1": "<object url>", "objectUrl_2": "<object url>" } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "image_groups": [ { "title": "Image sequence 001", "createVideo": true, "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/01.jpg", "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/02.jpg", "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/DALL%C2%B7E+2022-09-08+18.53.25+-+firefighter+extinguishing+flames+around+computer+in+software+office+overgrown+by+foliage.png" }, { "title": "Image sequence 002", "createVideo": true, "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/thing-01.jpg", "objectUrl_1": "<https://encord-integration.s3.eu-west-2.amazonaws.com/images/thing-02.jpg", "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/thing-03.jpg" } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="DICOM"> #### [DICOM](#dicom) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). * Each `dicom_series` element can contain one or more DICOM series. * Each series requires a title and at least one object URL, as shown in the example below. * If `skip_duplicate_urls` is set to `true`, all object URLs exactly matching existing DICOM files in the dataset will be skipped. <Note>Custom metadata is distinct from patient metadata, which is included in the `.dcm` file and does not have to be specific during the upload to Encord. </Note> The following is an example JSON for uploading three DICOM series belonging to a study. Each title and object URL correspond to individual DICOM series. * The first series contains only a single object URL, as it is composed of a single file. * The second series contains 3 object URLs, as it is composed of three separate files. * The third series contains 2 object URLs, as it is composed of two separate files. <Info> For each DICOM upload, an additional `DicomSeries` file is created. This file represents the series file-set. Only `DicomSeries` are displayed in the Encord application. </Info> ```json Template theme={"dark"} { "dicom_series": [ { "title": "Series-1", "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series1-file.dcm" }, { "title": "Series-2", "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series2-file1.dcm", "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series2-file2.dcm", "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series2-file3.dcm", }, { "title": "Series-3", "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series3-file1.dcm", "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/study1-series3-file2.dcm", } ], "skip_duplicate_urls": true } ``` </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="NIfTI"> #### [NIfTI](#nifti) The following is an example JSON file for uploading two NIfTI files to Encord. ```json Template theme={"dark"} { "nifti": [ { "title": "<file-1>", "objectUrl": "https://my-bucket/.../nifti-file1.nii" }, { "title": "<file-2>", "objectUrl": "https://my-bucket/.../nifti-file2.nii.gz" } ], "skip_duplicate_urls": true } ``` </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title=" Multiple file types "> You can upload multiple file types using a single JSON file. The example below shows 1 image, 2 videos, 2 image sequences, and 1 image group. ```json Multiple file types theme={"dark"} { "images": [ { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/Image1.png" } ], "videos": [ { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/Cooking.mp4" }, { "objectUrl": "https://encord-integration.s3.eu-west-2.amazonaws.com/videos/Oranges.mp4" } ], "image_groups": [ { "title": "apple-samsung-light", "createVideo": true, "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(32).jpg", "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(33).jpg", "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(34).jpg", "objectUrl_3": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(35).jpg" }, { "title": "apple-samsung-dark", "createVideo": true, "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/2+(32).jpg", "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/2+(33).jpg", "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/2+(34).jpg", "objectUrl_3": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/2+(35).jpg" } ], "image_groups": [ { "title": "apple-ios-light", "createVideo": false, "objectUrl_0": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/3+(32).jpg", "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/3+(33).jpg" } ], "skip_duplicate_urls": true } ``` </Accordion> </AccordionGroup> </Accordion> <Accordion title="JSON for GCP"> <AccordionGroup> <Accordion title="Videos"> #### [Videos](#videos) **Video Metadata** When the `videoMetadata` flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. <Warning>To guarantee accurate labels, it is crucial that the `videoMetadata` provided is accurate.</Warning> <CodeGroup> ```json Template theme={"dark"} { "videos": [ { "objectUrl": "cloud-path-to-your-video-1" }, { "objectUrl": "cloud-path-to-your-video-2", "videoMetadata": { "fps": frames-per-second, "duration": duration-in-seconds, "width": frame-width, "height": frame-height, "file_size": file-size-in-bytes, "mime_type": "MIME-file-type-extension" } } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "videos": [ { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/eLeopardJog.mp4" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/_video.mp4" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/bluemlisalphutte_flyover.mp4" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/boat_lake_normalized.mp4" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/boat_lake.MP4" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/boat_lake.MP4" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/snow_sled.MOV" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/cyclists.MP4" } ], "skip_duplicate_urls": true } ``` ```json Video Metadata theme={"dark"} { "videos": [ { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/es/video_file.mp4", "videoMetadata": { "fps": 23.98, "duration": 29.09, "width": 1280, "height": 720, "file_size": 5468354, "mime_type": "video/mp4" } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Audio files"> #### [Audio Files](#audio-files) The following is an example JSON file for uploading two audio files to Encord. <Tip> Waveform generation for long audio files can cause lag. To avoid this, generate the waveform offline and upload it with the audio file. For more information go [here](/platform-documentation/Curate/index-audio#long-audio-files). </Tip> * Example 1 imports audio files with an Encord title. * Example 2 imports one audio file with the `audiometadata` flag. When the `audiometadata` flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. To guarantee accurate labels, it is crucial that the metadata you provide is accurate. <CodeGroup> ```json Template theme={"dark"} { "audio": [ { "objectUrl": "<object url_1>" }, { "objectUrl": "<object url_2>", "title": "my-custom-audio-file-title.mp3" } ], "skip_duplicate_urls": true } ``` ```json Audio Metadata theme={"dark"} { "audio": [ { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/images/audio_file.mp3", "audioMetadata": { "duration": 23.98, "file_size": 2900000, "mime_type": "audio/mp3", "sample_rate": 44100, "bit_depth": 24, "codec": "mp3", "num_channels": 2 } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="PDFs"> #### [PDFs](#pdfs) The following are example JSON files for uploading PDFs to Encord. <CodeGroup> ```json Template theme={"dark"} { "pdfs": [ { "objectUrl": "<object url_1>" }, { "objectUrl": "<object url_2>", "title": "my-document-02.pdf" }, { "objectUrl": "<object url_3>", "title": "my-document-03.pdf", "pdfMetadata": { "fileSize": 300, "numPages": 5 } } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "pdfs": [ { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/my-document-01.pdf", }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/my-document-02.pdf", "title": "my-document-02.pdf" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/my-document-03.pdf", "title": "my-document-03.pdf", "pdfMetadata": { "fileSize": 300, "numPages": 5 } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Text Files"> #### [Text Files](#text-files) The following are example JSON files for uploading text files to Encord. <CodeGroup> ```json Template theme={"dark"} { "text": [ { "objectUrl": "<object url_1>" }, { "objectUrl": "<object url_2>", "title": "my-file.html" }, { "objectUrl": "<object url_3>", "title": "my-plain-text-file.txt", "textMetadata": { "fileSize": 200, "mime_type": "text/plain" } }, { "objectUrl": "<object url_4>", "title": "my-csv-file.csv", "textMetadata": { "fileSize": 250, "mime_type": "text/csv" } }, { "objectUrl": "<object url_5>", "title": "my-xml-file.xml", "textMetadata": { "fileSize": 300, "mime_type": "application/xml" } }, { "objectUrl": "<object url_6>", "title": "my-html-file.html", "textMetadata": { "fileSize": 400, "mime_type": "text/html" } } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "text": [ { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/my-file-01.html" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/my-plain-text-file-01.txt", "title": "my-plain-text-file-01.txt" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/my-plain-text-file-02.txt", "title": "my-plain-text-file-02.txt", "textMetadata": { "fileSize": 200, "mime_type": "text/plain" } }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/my-csv-file-01.csv", "title": "my-csv-file-01.csv", "textMetadata": { "fileSize": 250, "mime_type": "text/csv" } }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/my-xml-file-01.xml", "title": "my-xml-file-01.xml", "textMetadata": { "fileSize": 300, "mime_type": "application/xml" } }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/my-html-file-01.html", "title": "my-html-file-01.html", "textMetadata": { "fileSize": 400, "mime_type": "text/html" } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Single images"> #### [Single Images](#single-images) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). The JSON structure for single images parallels that of videos. **Template:** Provides the proper JSON format to import images into Encord. **Examples:** * **Data** Imports the images only. * **Image Metadata:** Imports images with image metadata. This improves the import speed for your images. <CodeGroup> ```json Template theme={"dark"} { "images": [ { "objectUrl": "file/path/to/images/file-name-01.file-extension" }, { "objectUrl": "file/path/to/images/file-name-02.file-extension" }, { "objectUrl": "file/path/to/images/file-name-03.file-extension", "title": "image-title.file-extension" } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "images": [ { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/es/0001.jpg" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/images/0002.jpg" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/images/0003.jpg" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/images/DALL%C2%B7E+2022-09-08+18.53.25+-+firefighter+extinguishing+flames+around+computer+in+software+office+overgrown+by+foliage.png" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/images/DALL%C2%B7E+2022-12-08+22.16.52+-+steampunk+combustion+engine+that+is+fueled+by+data+and+produces+computer+vision+algorithms.png" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/images/large_images/pexels-ivo-rainha-00057.png" } ], "skip_duplicate_urls": true } ``` ```json Image Metadata theme={"dark"} { "images": [ { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/images/my_image.jpg", "imageMetadata": { "mimeType": "image/jpg", "fileSize": 124, "width": 640, "height": 480 } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Image groups"> #### [Image groups](#image-groups) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). * Image groups are collections of images that are processed as one annotation task. * Images within image groups remain unaltered, meaning that images of different sizes and resolutions can form an image group without the loss of data. * Image groups do NOT require 'write' permissions to your cloud storage. * If `skip_duplicate_urls` is set to `true`, all URLs exactly matching existing image groups in the dataset are skipped. <Note>The position of each image within the sequence needs to be specified in the key (`objectUrl_{position_number}`).</Note> **Template:** Provides the proper JSON format to import image groups into Encord. **Examples:** * **Data:** Imports the image groups only. <CodeGroup> ```json Template theme={"dark"} { "image_groups": [ { "title": "<title 1>", "createVideo": false, "objectUrl_0": "file/path/to/images/file-name-01.file-extension", "objectUrl_1": "file/path/to/images/file-name-02.file-extension", "objectUrl_2": "file/path/to/images/file-name-03.file-extension", }, { "title": "<title 2>", "createVideo": false, "objectUrl_0": "file/path/to/images/file-name-01.file-extension", "objectUrl_1": "file/path/to/images/file-name-02.file-extension", "objectUrl_2": "file/path/to/images/file-name-03.file-extension" } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "image_groups": [ { "title": "Image group 01", "createVideo": false, "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/0001.jpg", "objectUrl_1": "https://storage.cloud.google.com/encord-image-bucket/images/0002.jpg", "objectUrl_2": "https://storage.cloud.google.com/encord-image-bucket/images/DALL%C2%B7E+2022-09-08+18.53.25+-+firefighter+extinguishing+flames+around+computer+in+software+office+overgrown+by+foliage.png" }, { "title": "Image group 02", "createVideo": false, "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/thing-0001.jpg", "objectUrl_1": "<https://storage.cloud.google.com/encord-image-bucket/images/thing-0002.jpg", "objectUrl_2": "https://storage.cloud.google.com/encord-image-bucket/images/thing-0003.jpg" } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Image sequences"> #### [Image sequences](#image-sequences) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). * Image sequences are collections of images that are processed as one annotation task and represented as a video. * Images within image sequences may be altered as images of varying sizes and resolutions are made to match that of the first image in the sequence. * Creating Image sequences from cloud storage requires 'write' permissions, as new files have to be created in order to be read as a video. * Each object in the `image_groups` array with the `createVideo` flag set to `true` represents a single image sequence. * If `skip_duplicate_urls` is set to `true`, all URLs exactly matching existing image sequences in the dataset are skipped. <Tip> The only difference between adding image groups and image sequences using a JSON file is that image sequences require the `createVideo` flag to be set to `true`. Both use the key `image_groups`.</Tip> <Note>The position of each image within the sequence needs to be specified in the key (`objectUrl_{position_number}`).</Note> <Info>Encord supports up to 32,767 entries (21:50 minutes) for a single image sequence. We recommend up to 10,000 to 15,000 entries for a single image sequence for best performance. If you need a longer sequence, we recommend using video instead of an image sequence.</Info> **Template:** Provides the proper JSON format to import image groups into Encord. **Examples:** * **Data:** Imports the images groups only. <CodeGroup> ```json Template theme={"dark"} { "image_groups": [ { "title": "<title 1>", "createVideo": true, "objectUrl_0": "<object url>" }, { "title": "<title 2>", "createVideo": true, "objectUrl_0": "<object url>", "objectUrl_1": "<object url>", "objectUrl_2": "<object url>" } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "image_groups": [ { "title": "Image sequence 001", "createVideo": true, "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/01.jpg", "objectUrl_1": "https://storage.cloud.google.com/encord-image-bucket/images/02.jpg", "objectUrl_2": "https://storage.cloud.google.com/encord-image-bucket/images/DALL%C2%B7E+2022-09-08+18.53.25+-+firefighter+extinguishing+flames+around+computer+in+software+office+overgrown+by+foliage.png" }, { "title": "Image sequence 002", "createVideo": true, "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/thing-01.jpg", "objectUrl_1": "<https://storage.cloud.google.com/encord-image-bucket/images/thing-02.jpg", "objectUrl_2": "https://storage.cloud.google.com/encord-image-bucket/images/thing-03.jpg" } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="DICOM"> #### [DICOM](#dicom) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). * Each `dicom_series` element can contain one or more DICOM series. * Each series requires a title and at least one object URL, as shown in the example below. * If `skip_duplicate_urls` is set to `true`, all object URLs exactly matching existing DICOM files in the dataset will be skipped. <Note>Custom metadata is distinct from patient metadata, which is included in the `.dcm` file and does not have to be specific during the upload to Encord. </Note> The following is an example JSON for uploading three DICOM series belonging to a study. Each title and object URL correspond to individual DICOM series. * The first series contains only a single object URL, as it is composed of a single file. * The second series contains 3 object URLs, as it is composed of three separate files. * The third series contains 2 object URLs, as it is composed of two separate files. <Info> For each DICOM upload, an additional `DicomSeries` file is created. This file represents the series file-set. Only `DicomSeries` are displayed in the Encord application. </Info> ```json JSON for DICOM theme={"dark"} { "dicom_series": [ { "title": "Series-1", "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/study1-series1-file.dcm" }, { "title": "Series-2", "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/study1-series2-file1.dcm", "objectUrl_1": "https://storage.cloud.google.com/encord-image-bucket/images/study1-series2-file2.dcm", "objectUrl_2": "https://storage.cloud.google.com/encord-image-bucket/images/study1-series2-file3.dcm", }, { "title": "Series-3", "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/study1-series3-file1.dcm", "objectUrl_1": "https://storage.cloud.google.com/encord-image-bucket/images/study1-series3-file2.dcm", } ], "skip_duplicate_urls": true } ``` </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="NIfTI"> #### [NIfTI](#nifti) The following is an example JSON file for uploading two NIfTI files to Encord. ```json theme={"dark"} { "nifti": [ { "title": "<file-1>", "objectUrl": "https://my-bucket/.../nifti-file1.nii" }, { "title": "<file-2>", "objectUrl": "https://my-bucket/.../nifti-file2.nii.gz" } ], "skip_duplicate_urls": true } ``` </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title=" Multiple file types "> You can upload multiple file types using a single JSON file. The example below shows 1 image, 2 videos, 2 image sequences, and 1 image group. ```json Multiple file types theme={"dark"} { "images": [ { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/images/Image1.png" } ], "videos": [ { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/videos/Cooking.mp4" }, { "objectUrl": "https://storage.cloud.google.com/encord-image-bucket/videos/Oranges.mp4" } ], "image_groups": [ { "title": "apple-samsung-light", "createVideo": true, "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/1+(32).jpg", "objectUrl_1": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(33).jpg", "objectUrl_2": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(34).jpg", "objectUrl_3": "https://encord-integration.s3.eu-west-2.amazonaws.com/images/1+(35).jpg" }, { "title": "apple-samsung-dark", "createVideo": true, "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/2+(32).jpg", "objectUrl_1": "https://storage.cloud.google.com/encord-image-bucket/images/2+(33).jpg", "objectUrl_2": "https://storage.cloud.google.com/encord-image-bucket/images/2+(34).jpg", "objectUrl_3": "https://storage.cloud.google.com/encord-image-bucket/images/2+(35).jpg" } ], "image_groups": [ { "title": "apple-ios-light", "createVideo": false, "objectUrl_0": "https://storage.cloud.google.com/encord-image-bucket/images/3+(32).jpg", "objectUrl_1": "https://storage.cloud.google.com/encord-image-bucket/images/3+(33).jpg" } ], "skip_duplicate_urls": true } ``` </Accordion> </AccordionGroup> </Accordion> <Accordion title="JSON for Azure"> <AccordionGroup> <Accordion title="Videos"> #### [Videos](#videos) **Video Metadata** When the `videoMetadata` flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. <Warning>To guarantee accurate labels, it is crucial that the `videoMetadata` provided is accurate.</Warning> <CodeGroup> ```json Template theme={"dark"} { "videos": [ { "objectUrl": "cloud-path-to-your-video-1" }, { "objectUrl": "cloud-path-to-your-video-2", "videoMetadata": { "fps": frames-per-second, "duration": duration-in-seconds, "width": frame-width, "height": frame-height, "file_size": file-size-in-bytes, "mime_type": "MIME-file-type-extension" } } { "objectUrl": "cloud-path-to-your-video-3", "title": "title-for-your-video-3" } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "videos": [ { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/BerghouseLeopardJog.mp4" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/blue_bus_video.mp4" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/bluemlisalphutte_flyover.mp4" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/boat_lake_normalized.mp4" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/boat_lake.MP4" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/boat_lake.MP4" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/snow_sled.MOV" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/cyclists.MP4" } ], "skip_duplicate_urls": true } ``` ```json Video Metadata theme={"dark"} { "videos": [ { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/video_file.mp4", "videoMetadata": { "fps": 23.98, "duration": 29.09, "width": 1280, "height": 720, "file_size": 5468354, "mime_type": "video/mp4" } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Audio files"> #### [Audio Files](#audio-files) The following is an example JSON file for uploading two audio files to Encord. <Tip> Waveform generation for long audio files can cause lag. To avoid this, generate the waveform offline and upload it with the audio file. For more information go [here](/platform-documentation/Curate/index-audio#long-audio-files). </Tip> * **Template:** Imports audio files with an Encord title. * **Audio Metadata:** Imports one audio file with the `audiometadata` flag. When the `audiometadata` flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. To guarantee accurate labels, it is crucial that the metadata you provide is accurate. <CodeGroup> ```json Template theme={"dark"} { "audio": [ { "objectUrl": "<object url_1>" }, { "objectUrl": "<object url_2>", "title": "my-custom-audio-file-title.mp3" } ], "skip_duplicate_urls": true } ``` ```json Audio Metadata theme={"dark"} { "audio": [ { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/audio_file.mp3", "audioMetadata": { "duration": 23.98, "file_size": 2900000, "mime_type": "audio/mp3", "sample_rate": 44100, "bit_depth": 24, "codec": "mp3", "num_channels": 2 } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="PDFs"> #### [PDFs](#pdfs) The following are example JSON files for uploading PDFs to Encord. <CodeGroup> ```json Template theme={"dark"} { "pdfs": [ { "objectUrl": "<object url_1>" }, { "objectUrl": "<object url_2>", "title": "my-document-02.pdf" }, { "objectUrl": "<object url_3>", "title": "my-document-03.pdf", "pdfMetadata": { "fileSize": 300, "numPages": 5 } } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "pdfs": [ { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/my-document-01.pdf", }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/my-document-02.pdf", "title": "my-document-02.pdf" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/my-document-03.pdf", "title": "my-document-03.pdf", "pdfMetadata": { "fileSize": 300, "numPages": 5 } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Text Files"> #### [Text Files](#text-files) The following are example JSON files for uploading text files to Encord. <CodeGroup> ```json Template theme={"dark"} { "text": [ { "objectUrl": "<object url_1>" }, { "objectUrl": "<object url_2>", "title": "my-file.html" }, { "objectUrl": "<object url_3>", "title": "my-plain-text-file.txt", "textMetadata": { "fileSize": 200, "mime_type": "text/plain" } }, { "objectUrl": "<object url_4>", "title": "my-csv-file.csv", "textMetadata": { "fileSize": 250, "mime_type": "text/csv" } }, { "objectUrl": "<object url_5>", "title": "my-xml-file.xml", "textMetadata": { "fileSize": 300, "mime_type": "application/xml" } }, { "objectUrl": "<object url_6>", "title": "my-html-file.html", "textMetadata": { "fileSize": 400, "mime_type": "text/html" } } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "text": [ { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/my-file-01.html" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/my-plain-text-file-01.txt", "title": "my-plain-text-file-01.txt" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/my-plain-text-file-02.txt", "title": "my-plain-text-file-02.txt", "textMetadata": { "fileSize": 200, "mime_type": "text/plain" } }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/my-csv-file-01.csv", "title": "my-csv-file-01.csv", "textMetadata": { "fileSize": 250, "mime_type": "text/csv" } }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/my-xml-file-01.xml", "title": "my-xml-file-01.xml", "textMetadata": { "fileSize": 300, "mime_type": "application/xml" } }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/my-html-file-01.html", "title": "my-html-file-01.html", "textMetadata": { "fileSize": 400, "mime_type": "text/html" } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Single images"> #### [Single Images](#single-images) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). The JSON structure for single images parallels that of videos. **Template:** Provides the proper JSON format to import images into Encord. **Examples:** * **Data** Imports the images only. * **Image Metadata:** Imports images with image metadata. This improves the import speed for your images. <CodeGroup> ```json Template theme={"dark"} { "images": [ { "objectUrl": "file/path/to/images/file-name-01.file-extension" }, { "objectUrl": "file/path/to/images/file-name-02.file-extension" }, { "objectUrl": "file/path/to/images/file-name-03.file-extension", "title": "image-title.file-extension" } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "images": [ { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/0001.jpg" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/0002.jpg" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/0003.jpg" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/DALL%C2%B7E+2022-09-08+18.53.25+-+firefighter+extinguishing+flames+around+computer+in+software+office+overgrown+by+foliage.png" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/DALL%C2%B7E+2022-12-08+22.16.52+-+steampunk+combustion+engine+that+is+fueled+by+data+and+produces+computer+vision+algorithms.png" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/large_images/pexels-ivo-rainha-00057.png" } ], "skip_duplicate_urls": true } ``` ```json Image Metadata theme={"dark"} { "images": [ { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/my_image.jpg", "imageMetadata": { "mimeType": "image/jpg", "fileSize": 124, "width": 640, "height": 480 } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Image groups"> #### [Image groups](#image-groups) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). * Image groups are collections of images that are processed as one annotation task. * Images within image groups remain unaltered, meaning that images of different sizes and resolutions can form an image group without the loss of data. * Image groups do NOT require 'write' permissions to your cloud storage. * If `skip_duplicate_urls` is set to `true`, all URLs exactly matching existing image groups in the dataset are skipped. <Note>The position of each image within the sequence needs to be specified in the key (`objectUrl_{position_number}`).</Note> **Template:** Provides the proper JSON format to import image groups into Encord. **Examples:** * **Data:** Imports the image groups only. <CodeGroup> ```json Template theme={"dark"} { "image_groups": [ { "title": "<title 1>", "createVideo": false, "objectUrl_0": "file/path/to/images/file-name-01.file-extension", "objectUrl_1": "file/path/to/images/file-name-02.file-extension", "objectUrl_2": "file/path/to/images/file-name-03.file-extension" }, { "title": "<title 2>", "createVideo": false, "objectUrl_0": "file/path/to/images/file-name-01.file-extension", "objectUrl_1": "file/path/to/images/file-name-02.file-extension", "objectUrl_2": "file/path/to/images/file-name-03.file-extension" } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "image_groups": [ { "title": "Image group 01", "createVideo": false, "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/0001.jpg", "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/0002.jpg", "objectUrl_2": "https://myaccount.blob.core.windows.net/encordcontainer/DALL%C2%B7E+2022-09-08+18.53.25+-+firefighter+extinguishing+flames+around+computer+in+software+office+overgrown+by+foliage.png" }, { "title": "Image group 02", "createVideo": false, "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/images/thing-0001.jpg", "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/images/thing-0002.jpg", "objectUrl_2": "https://myaccount.blob.core.windows.net/encordcontainer/thing-0003.jpg" } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Image sequences"> #### [Image sequences](#image-sequences) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). * Image sequences are collections of images that are processed as one annotation task and represented as a video. * Images within image sequences may be altered as images of varying sizes and resolutions are made to match that of the first image in the sequence. * Creating Image sequences from cloud storage requires 'write' permissions, as new files have to be created in order to be read as a video. * Each object in the `image_groups` array with the `createVideo` flag set to `true` represents a single image sequence. * If `skip_duplicate_urls` is set to `true`, all URLs exactly matching existing image sequences in the dataset are skipped. <Tip> The only difference between adding image groups and image sequences using a JSON file is that image sequences require the `createVideo` flag to be set to `true`. Both use the key `image_groups`.</Tip> <Note>The position of each image within the sequence needs to be specified in the key (`objectUrl_{position_number}`).</Note> <Info>Encord supports up to 32,767 entries (21:50 minutes) for a single image sequence. We recommend up to 10,000 to 15,000 entries for a single image sequence for best performance. If you need a longer sequence, we recommend using video instead of an image sequence.</Info> **Template:** Provides the proper JSON format to import image groups into Encord. **Examples:** * **Data:** Imports the images groups only. <CodeGroup> ```json Template theme={"dark"} { "image_groups": [ { "title": "<title 1>", "createVideo": true, "objectUrl_0": "<object url>" }, { "title": "<title 2>", "createVideo": true, "objectUrl_0": "<object url>", "objectUrl_1": "<object url>", "objectUrl_2": "<object url>" } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "image_groups": [ { "title": "Image sequence 001", "createVideo": true, "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/01.jpg", "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/02.jpg", "objectUrl_2": "https://myaccount.blob.core.windows.net/encordcontainer/DALL%C2%B7E+2022-09-08+18.53.25+-+firefighter+extinguishing+flames+around+computer+in+software+office+overgrown+by+foliage.png" }, { "title": "Image sequence 002", "createVideo": true, "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/thing-01.jpg", "objectUrl_1": "<https://myaccount.blob.core.windows.net/encordcontainer/thing-02.jpg", "objectUrl_2": "https://myaccount.blob.core.windows.net/encordcontainer/thing-03.jpg" } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="DICOM"> #### [DICOM](#dicom) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). * Each `dicom_series` element can contain one or more DICOM series. * Each series requires a title and at least one object URL, as shown in the example below. * If `skip_duplicate_urls` is set to `true`, all object URLs exactly matching existing DICOM files in the dataset will be skipped. <Note>Custom metadata is distinct from patient metadata, which is included in the `.dcm` file and does not have to be specific during the upload to Encord. </Note> The following is an example JSON for uploading three DICOM series belonging to a study. Each title and object URL correspond to individual DICOM series. * The first series contains only a single object URL, as it is composed of a single file. * The second series contains 3 object URLs, as it is composed of three separate files. * The third series contains 2 object URLs, as it is composed of two separate files. <Info> For each DICOM upload, an additional `DicomSeries` file is created. This file represents the series file-set. Only `DicomSeries` are displayed in the Encord application. </Info> ```json Template theme={"dark"} { "dicom_series": [ { "title": "Series-1", "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/study1-series1-file.dcm" }, { "title": "Series-2", "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/study1-series2-file1.dcm", "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/study1-series2-file2.dcm", "objectUrl_2": "https://myaccount.blob.core.windows.net/encordcontainer/study1-series2-file3.dcm", }, { "title": "Series-3", "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/study1-series3-file1.dcm", "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/study1-series3-file2.dcm", } ], "skip_duplicate_urls": true } ``` </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="NIfTI"> #### [NIfTI](#nifti) The following is an example JSON file for uploading two NIfTI files to Encord. ```json Template theme={"dark"} { "nifti": [ { "title": "<file-1>", "objectUrl": "https://my-bucket/.../nifti-file1.nii" }, { "title": "<file-2>", "objectUrl": "https://my-bucket/.../nifti-file2.nii.gz" } ], "skip_duplicate_urls": true } ``` </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title=" Multiple file types "> You can upload multiple file types using a single JSON file. The example below shows 1 image, 2 videos, 2 image sequences, and 1 image group. ```json theme={"dark"} { "images": [ { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/Image1.png" } ], "videos": [ { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/Cooking.mp4" }, { "objectUrl": "https://myaccount.blob.core.windows.net/encordcontainer/Oranges.mp4" } ], "image_groups": [ { "title": "apple-samsung-light", "createVideo": true, "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/1-Samsung-S4-Light+Environment/1+(32).jpg", "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/1-Samsung-S4-Light+Environment/1+(33).jpg", "objectUrl_2": "https://myaccount.blob.core.windows.net/encordcontainer/1-Samsung-S4-Light+Environment/1+(34).jpg", "objectUrl_3": "https://myaccount.blob.core.windows.net/encordcontainer/1-Samsung-S4-Light+Environment/1+(35).jpg" }, { "title": "apple-samsung-dark", "createVideo": true, "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/2-samsung-S4-Dark+Environment/2+(32).jpg", "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/2-samsung-S4-Dark+Environment/2+(33).jpg", "objectUrl_2": "https://myaccount.blob.core.windows.net/encordcontainer/2-samsung-S4-Dark+Environment/2+(34).jpg", "objectUrl_3": "https://myaccount.blob.core.windows.net/encordcontainer/2-samsung-S4-Dark+Environment/2+(35).jpg" } ], "image_groups": [ { "title": "apple-ios-light", "createVideo": false, "objectUrl_0": "https://myaccount.blob.core.windows.net/encordcontainer/3-IOS-4-Light+Environment/3+(32).jpg", "objectUrl_1": "https://myaccount.blob.core.windows.net/encordcontainer/3-IOS-4-Light+Environment/3+(33).jpg" } ], "skip_duplicate_urls": true } ``` </Accordion> </AccordionGroup> </Accordion> <Accordion title="JSON for OTC"> <AccordionGroup> <Accordion title="Videos"> #### [Videos](#videos) **Video Metadata** When the `videoMetadata` flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. <Warning>To guarantee accurate labels, it is crucial that the `videoMetadata` provided is accurate.</Warning> <CodeGroup> ```json Template theme={"dark"} { "videos": [ { "objectUrl": "cloud-path-to-your-video-1" }, { "objectUrl": "cloud-path-to-your-video-2", "videoMetadata": { "fps": frames-per-second, "duration": duration-in-seconds, "width": frame-width, "height": frame-height, "file_size": file-size-in-bytes, "mime_type": "MIME-file-type-extension" } } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "videos": [ { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/og.mp4" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/videos/blue_bus_video.mp4" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/videos/bluemlisalphutte_flyover.mp4" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/videos/boat_lake_normalized.mp4" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/videos/boat_lake.MP4" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/videos/boat_lake.MP4" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/videos/snow_sled.MOV" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/videos/cyclists.MP4" } ], "skip_duplicate_urls": true } ``` ```json Video Metadata theme={"dark"} { "videos": [ { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/videos/video_file.mp4", "videoMetadata": { "fps": 23.98, "duration": 29.09, "width": 1280, "height": 720, "file_size": 5468354, "mime_type": "video/mp4" } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Audio files"> #### [Audio Files](#audio-files) The following is an example JSON file for uploading two audio files to Encord. <Tip> Waveform generation for long audio files can cause lag. To avoid this, generate the waveform offline and upload it with the audio file. For more information go [here](/platform-documentation/Curate/index-audio#long-audio-files). </Tip> * **Template:** Imports audio files with an Encord title. * **Audio Metadata:** Imports one audio file with the `audiometadata` flag. When the `audiometadata` flag is present in the JSON file, we directly use the supplied metadata without performing any additional validation, and do not store the file on our servers. To guarantee accurate labels, it is crucial that the metadata you provide is accurate. <CodeGroup> ```json Template theme={"dark"} { "audio": [ { "objectUrl": "<object url_1>" }, { "objectUrl": "<object url_2>", "title": "my-custom-audio-file-title.mp3" } ], "skip_duplicate_urls": true } ``` ```json Audio Metadata theme={"dark"} { "audio": [ { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/audio/audio_file.mp3", "audioMetadata": { "duration": 23.98, "file_size": 2900000, "mime_type": "audio/mp3", "sample_rate": 44100, "bit_depth": 24, "codec": "mp3", "num_channels": 2 } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="PDFs"> #### [PDFs](#pdfs) The following are example JSON files for uploading PDFs to Encord. <CodeGroup> ```json Template theme={"dark"} { "pdfs": [ { "objectUrl": "<object url_1>" }, { "objectUrl": "<object url_2>", "title": "my-document-02.pdf" }, { "objectUrl": "<object url_3>", "title": "my-document-03.pdf", "pdfMetadata": { "fileSize": 300, "numPages": 5 } } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "pdfs": [ { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/my-document-01.pdf", }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/my-document-02.pdf", "title": "my-document-02.pdf" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/my-document-3.pdf", "title": "my-document-03.pdf", "pdfMetadata": { "fileSize": 300, "numPages": 5 } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Text Files"> #### [Text Files](#text-files) The following are example JSON files for uploading text files to Encord. <CodeGroup> ```json Template theme={"dark"} { "text": [ { "objectUrl": "<object url_1>" }, { "objectUrl": "<object url_2>", "title": "my-file.html" }, { "objectUrl": "<object url_3>", "title": "my-plain-text-file.txt", "textMetadata": { "fileSize": 200, "mime_type": "text/plain" } }, { "objectUrl": "<object url_4>", "title": "my-csv-file.csv", "textMetadata": { "fileSize": 250, "mime_type": "text/csv" } }, { "objectUrl": "<object url_5>", "title": "my-xml-file.xml", "textMetadata": { "fileSize": 300, "mime_type": "application/xml" } }, { "objectUrl": "<object url_6>", "title": "my-html-file.html", "textMetadata": { "fileSize": 400, "mime_type": "text/html" } } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "text": [ { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/my-file-01.html" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/my-plain-text-file-01.txt", "title": "my-plain-text-file-01.txt" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/my-plain-text-file-02.txt", "title": "my-plain-text-file-02.txt", "textMetadata": { "fileSize": 200, "mime_type": "text/plain" } }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/my-csv-file-01.csv", "title": "my-csv-file-01.csv", "textMetadata": { "fileSize": 250, "mime_type": "text/csv" } }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/my-xml-file-01.xml", "title": "my-xml-file-01.xml", "textMetadata": { "fileSize": 300, "mime_type": "application/xml" } }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/my-html-file-01.html", "title": "my-html-file-01.html", "textMetadata": { "fileSize": 400, "mime_type": "text/html" } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Single images"> #### [Single Images](#single-images) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). The JSON structure for single images parallels that of videos. **Template:** Provides the proper JSON format to import images into Encord. **Examples:** * **Data** Imports the images only. * **Image Metadata:** Imports images with image metadata. This improves the import speed for your images. <CodeGroup> ```json Template theme={"dark"} { "images": [ { "objectUrl": "file/path/to/images/file-name-01.file-extension" }, { "objectUrl": "file/path/to/images/file-name-02.file-extension" }, { "objectUrl": "file/path/to/images/file-name-03.file-extension", "title": "image-title.file-extension" } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "images": [ { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/0001.jpg" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/0002.jpg" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/0003.jpg" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/DALL%C2%B7E+2022-09-08+18.53.25+-+firefighter+extinguishing+flames+around+computer+in+software+office+overgrown+by+foliage.png" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/DALL%C2%B7E+2022-12-08+22.16.52+-+steampunk+combustion+engine+that+is+fueled+by+data+and+produces+computer+vision+algorithms.png" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/large_images/pexels-ivo-rainha-00057.png" } ], "skip_duplicate_urls": true } ``` ```json Image Metadata theme={"dark"} { "images": [ { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/my_image.jpg", "imageMetadata": { "mimeType": "image/jpg", "fileSize": 124, "width": 640, "height": 480 } } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Image groups"> #### [Image groups](#image-groups) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). * Image groups are collections of images that are processed as one annotation task. * Images within image groups remain unaltered, meaning that images of different sizes and resolutions can form an image group without the loss of data. * Image groups do NOT require 'write' permissions to your cloud storage. * If `skip_duplicate_urls` is set to `true`, all URLs exactly matching existing image groups in the dataset are skipped. <Note>The position of each image within the sequence needs to be specified in the key (`objectUrl_{position_number}`).</Note> **Template:** Provides the proper JSON format to import image groups into Encord. **Examples:** * **Data:** Imports the image groups only. <CodeGroup> ```json Template theme={"dark"} { "image_groups": [ { "title": "<title 1>", "createVideo": false, "objectUrl_0": "file/path/to/images/file-name-01.file-extension", "objectUrl_1": "file/path/to/images/file-name-02.file-extension", "objectUrl_2": "file/path/to/images/file-name-03.file-extension", }, { "title": "<title 2>", "createVideo": false, "objectUrl_0": "file/path/to/images/file-name-01.file-extension", "objectUrl_1": "file/path/to/images/file-name-02.file-extension", "objectUrl_2": "file/path/to/images/file-name-03.file-extension", "clientMetadata": {"optional": "metadata"} } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "image_groups": [ { "title": "Image group 01", "createVideo": false, "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/0001.jpg", "objectUrl_1": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/0002.jpg", "objectUrl_2": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/DALL%C2%B7E+2022-09-08+18.53.25+-+firefighter+extinguishing+flames+around+computer+in+software+office+overgrown+by+foliage.png" }, { "title": "Image group 02", "createVideo": false, "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/thing-0001.jpg", "objectUrl_1": "<https://encord-bucket.obs.eu-de.otc.t-systems.com/images/thing-0002.jpg", "objectUrl_2": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/thing-0003.jpg" } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Image sequences"> #### [Image sequences](#image-sequences) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). * Image sequences are collections of images that are processed as one annotation task and represented as a video. * Images within image sequences may be altered as images of varying sizes and resolutions are made to match that of the first image in the sequence. * Creating Image sequences from cloud storage requires 'write' permissions, as new files have to be created in order to be read as a video. * Each object in the `image_groups` array with the `createVideo` flag set to `true` represents a single image sequence. * If `skip_duplicate_urls` is set to `true`, all URLs exactly matching existing image sequences in the dataset are skipped. <Tip> The only difference between adding image groups and image sequences using a JSON file is that image sequences require the `createVideo` flag to be set to `true`. Both use the key `image_groups`.</Tip> <Note>The position of each image within the sequence needs to be specified in the key (`objectUrl_{position_number}`).</Note> <Info>Encord supports up to 32,767 entries (21:50 minutes) for a single image sequence. We recommend up to 10,000 to 15,000 entries for a single image sequence for best performance. If you need a longer sequence, we recommend using video instead of an image sequence.</Info> **Template:** Provides the proper JSON format to import image groups into Encord. **Examples:** * **Data:** Imports the images groups only. <CodeGroup> ```json Template theme={"dark"} { "image_groups": [ { "title": "<title 1>", "createVideo": true, "objectUrl_0": "<object url>" }, { "title": "<title 2>", "createVideo": true, "objectUrl_0": "<object url>", "objectUrl_1": "<object url>", "objectUrl_2": "<object url>" } ], "skip_duplicate_urls": true } ``` ```json Data theme={"dark"} { "image_groups": [ { "title": "Image sequence 001", "createVideo": true, "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/01.jpg", "objectUrl_1": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/02.jpg", "objectUrl_2": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/DALL%C2%B7E+2022-09-08+18.53.25+-+firefighter+extinguishing+flames+around+computer+in+software+office+overgrown+by+foliage.png" }, { "title": "Image sequence 002", "createVideo": true, "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/thing-01.jpg", "objectUrl_1": "<https://encord-bucket.obs.eu-de.otc.t-systems.com/images/thing-02.jpg", "objectUrl_2": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/thing-03.jpg" } ], "skip_duplicate_urls": true } ``` </CodeGroup> </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="DICOM"> #### [DICOM](#dicom) For detailed information about the JSON file format used for import go [here](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format). * Each `dicom_series` element can contain one or more DICOM series. * Each series requires a title and at least one object URL, as shown in the example below. * If `skip_duplicate_urls` is set to `true`, all object URLs exactly matching existing DICOM files in the dataset will be skipped. <Note>Custom metadata is distinct from patient metadata, which is included in the `.dcm` file and does not have to be specific during the upload to Encord. </Note> The following is an example JSON for uploading three DICOM series belonging to a study. Each title and object URL correspond to individual DICOM series. * The first series contains only a single object URL, as it is composed of a single file. * The second series contains 3 object URLs, as it is composed of three separate files. * The third series contains 2 object URLs, as it is composed of two separate files. <Info> For each DICOM upload, an additional `DicomSeries` file is created. This file represents the series file-set. Only `DicomSeries` are displayed in the Encord application. </Info> ```json JSON for DICOM theme={"dark"} { "dicom_series": [ { "title": "Series-1", "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/study1-series1-file.dcm" }, { "title": "Series-2", "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/study1-series2-file1.dcm", "objectUrl_1": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/study1-series2-file2.dcm", "objectUrl_2": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/study1-series2-file3.dcm", }, { "title": "Series-3", "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/study1-series3-file1.dcm", "objectUrl_1": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/study1-series3-file2.dcm", } ], "skip_duplicate_urls": true } ``` </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="NIfTI"> #### [NIfTI](#nifti) The following is an example JSON file for uploading two NIfTI files to Encord. ```json theme={"dark"} { "nifti": [ { "title": "<file-1>", "objectUrl": "https://my-bucket/.../nifti-file1.nii" }, { "title": "<file-2>", "objectUrl": "https://my-bucket/.../nifti-file2.nii.gz" } ], "skip_duplicate_urls": true } ``` </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title=" Multiple file types "> You can upload multiple file types using a single JSON file. The example below shows 1 image, 2 videos, 2 image sequences, and 1 image group. ```json Multiple file types theme={"dark"} { "images": [ { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/Image1.png" } ], "videos": [ { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/videos/Cooking.mp4" }, { "objectUrl": "https://encord-bucket.obs.eu-de.otc.t-systems.com/videos/Oranges.mp4" } ], "image_groups": [ { "title": "apple-samsung-light", "createVideo": true, "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/1+(32).jpg", "objectUrl_1": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/1+(33).jpg", "objectUrl_2": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/1+(34).jpg", "objectUrl_3": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/1+(35).jpg" }, { "title": "apple-samsung-dark", "createVideo": true, "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/2+(32).jpg", "objectUrl_1": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/2+(33).jpg", "objectUrl_2": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/2+(34).jpg", "objectUrl_3": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/2+(35).jpg" } ], "image_groups": [ { "title": "apple-ios-light", "createVideo": false, "objectUrl_0": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/3+(32).jpg", "objectUrl_1": "https://encord-bucket.obs.eu-de.otc.t-systems.com/images/3+(33).jpg" } ], "skip_duplicate_urls": true } ``` </Accordion> </AccordionGroup> </Accordion> <Accordion title="JSON for AWS Multi-Region Access Point"> When using a Multi-Region Access Point for your AWS S3 buckets the JSON file has to be slightly different from the examples provided. Instead of an object's URL, objects are specified using the ARN of the Multi-Region Access Point followed by the object name. The example below shows how video files from a Multi-Region Access Point would be specified. ```json theme={"dark"} { "videos": [ { "objectUrl": "Multi-Region-Access-Point-ARN + <object name_1>" }, { "objectUrl": "Multi-Region-Access-Point-ARN + <object name_2>", "title": "my-custom-video-title.mp4" } ], "skip_duplicate_urls": true } ``` ```json MRAP Example theme={"dark"} { "videos": [ { "objectUrl": "https://arn:aws:s3::123123123:accesspoint/frf28frarf9.mrap.s3-accesspoint.amazonaws.com/Videos/2022/video_1.mp4" }, { "objectUrl": "https://arn:aws:s3::123123123:accesspoint/frf28frarf9.mrap.s3-accesspoint.amazonaws.com/Videos/2022/video_2.mp4", "title": "many-cute-cats.mp4" } ], "skip_duplicate_urls": true } ``` </Accordion> </Accordion> </AccordionGroup> </Step> <Step title="Import your Files"> <AccordionGroup> <Accordion title="Import cloud data"> <Tip>To ensure smoother uploads and faster completion times, and avoid hitting absolute file limits, we recommend adding smaller batches of data. Limit uploads to 100 videos or up to 1,000 images at a time. You can also create multiple Datasets, all of which can be linked to a single Project. Familiarize yourself with our [limits and best practices for data import/registration](/platform-documentation/General/best-practices#import-into-encord) before adding data to Encord. </Tip> 1. Navigate to **Data** > **Files & Folders** in the Encord platform. 2. Click into a Folder. 3. Click **+ Upload files**. A dialog appears. <div class="flex justify-center"> <img src="https://storage.googleapis.com/docs-media.encord.com/static/img/upload-files-new.png" width="800" /> </div> 4. Click **Import from cloud data**. <div class="flex justify-center"> <img src="https://storage.googleapis.com/docs-media.encord.com/static/img/Index/upload-data-dialog-04.png" width="550" /> </div> <Tip>We recommend turning on the **Ignore individual file errors** feature. This ensures that individual file errors do not lead to the whole upload process being aborted. </Tip> 6. Click **Add JSON or CSV files** to add a [JSON](/platform-documentation/Curate/add-files/index-register-cloud-data#json-format) or [CSV](/platform-documentation/Curate/add-files/index-register-cloud-data#csv-format) file specifying cloud data that is to be added. </Accordion> </AccordionGroup> <AccordionGroup> <Accordion title="Import local data"> <Tip>To ensure smoother uploads and faster completion times, and avoid hitting absolute file limits, we recommend adding smaller batches of data. Limit uploads to 100 videos or up to 1,000 images at a time. You can also create multiple Datasets, all of which can be linked to a single Project. Familiarize yourself with our [limits and best practices for data import/registration](/platform-documentation/General/best-practices#import-into-encord) before adding data to Encord. </Tip> 1. Navigate to **Data** > **Files & Folders** in the Encord platform. 2. Click into a Folder. 3. Click **+ Upload files**. A dialog appears. <img src="https://storage.googleapis.com/docs-media.encord.com/static/img/upload-files-new.png" width="600" /> 4. Click one of the following: * **Upload:** Upload images, videos, and audio files. ![Upload dialog](https://storage.googleapis.com/docs-media.encord.com/static/img/Index/upload-data-dialog-001.png) * **Batch images as:** Upload image batches as image groups or image sequences. ![Batch images as](https://storage.googleapis.com/docs-media.encord.com/static/img/Index/upload-data-dialog-02.png) * **DICOM/NifTi:** Upload DICOM or NifTi series. ![DICOM/NifTi](https://storage.googleapis.com/docs-media.encord.com/static/img/Index/upload-data-dialog-03.png) 5. Click **Upload** after selecting your images or series. Your files upload into the Folder in Encord. </Accordion> </AccordionGroup> </Step> </Steps> ## STEP 2: Create a Benchmark Project <Info> The benchmark Project contains reference labels used to evaluate your annotators' labels. These gold standard labels should be created by a trusted expert to ensure accurate assessment. </Info> <Steps> <Step title="Create a Training Dataset"> Create a Dataset containing tasks designed to establish ground truth labels. These files are used to generate 'gold-standard' labels against which annotator performance can be evaluated. Give the Dataset a meaningful name. <Tip> Learn how to create Datasets [here](/platform-documentation/Annotate/annotate-datasets/annotate-datasets).</Tip> </Step> <Step title="Create an Ontology"> Create an Ontology to label your data. The same Ontology must be used in the Benchmark Project AND the Annotator Training Project. <Tip> Learn how to create Ontologies [here](/platform-documentation/Annotate/annotate-ontologies/annotate-create-ontologies).</Tip> </Step> <Step title="Create the Benchmark Project"> <Warning> Ensure that you attach ONLY the Training Dataset to the Project. </Warning> 1. Go to **Annotate > Projects**. 2. Click the **+ New annotation project** button to create a new Project. <div class="flex justify-center"> <img src="https://storage.googleapis.com/docs-media.encord.com/static/img/create-project-new.png" width="600" /> </div> 3. Give the Project a meaningful title and description. For example "Benchmark Labels". 4. Click the **Attach ontology** button and attach the Ontology you created. 5. Click the **Attach dataset** button and attach the Benchmark Dataset you created. <div class="flex justify-center"> <img src="https://storage.googleapis.com/docs-media.encord.com/static/img/simple-workflow.png" /> </div> 7. Click **Create project** to finish creating the Project. You have now created the Project to Establish ground-truth labels. </Step> </Steps> ## STEP 3: Create Annotator Training Projects Create a Project where your annotation workforce labels data and is evaluated against benchmark labels. <Steps> <Step title="Create an Annotator Training Workflow Template"> Create a Workflow template and give it a meaningful name like "Annotator Training". <Tip>Creating templates makes creating one Project per annotator quicker and easier.</Tip> Create the following Workflow template for your Annotator Training Projects. Documentation on how to create new Workflow templates can be found [here](/platform-documentation/Annotate/annotate-projects/annotate-workflows-and-templates#templates). <div class="flex justify-center"> <img src="https://storage.googleapis.com/docs-media.encord.com/static/img/annotator-training.png" /> </div> </Step> <Step title="Create Annotator Training Projects"> You must create one Annotator Training Project per annotator. This step must be repeated for each annotator. <Warning> Ensure that you: * Attach the Training Dataset you created in **Step 2.1** for the Benchmark Project. * Attach the SAME Ontology you created in **Step 2.2** for the Benchmark Project. * Attach the Annotator Training Workflow Template to the Project. </Warning> 1. Go to **Annotate > Projects**. 2. Click the **+ New annotation project** button to create a new Project. 3. Give the Project a meaningful title and description. For example "Annotator Training - Alex" for an annotator named Alex. 4. Click the **Attach ontology** button and attach the Ontology you created. Attach the SAME Ontology you created in **Step 2.2** for the Benchmark Project. 5. Click the **Attach dataset** button and attach the training Dataset you created in **Step 2.1**. 6. Click the **Load from template** button to attach the "Annotator Training" template you created in **Step 3.1**. 7. Click **Invite collaborators**. [Add the annotator](/platform-documentation/Annotate/annotate-projects/annotate-workflows-and-templates#annotate) you want to train in this Project to the annotation stage. 8. Click **Create Project** to create the Project. You have now created the Project to train the selected annotator. <div class="flex justify-center"> <img src="https://storage.googleapis.com/docs-media.encord.com/static/img/create-training-project.png" /> </div> </Step> </Steps> ## STEP 4: Annotator Training Your annotators must now complete all tasks in the Annotator Training Project they are assigned to. Only tasks in the Complete stage are evaluated. <Tip> Information on how to label can be found [here](/platform-documentation/GettingStarted/gettingstarted-labeling). </Tip> ## STEP 5: Evaluate Annotators <Info> This example only evaluates Bounding Boxes. </Info> Save and run the following script to evaluate annotator performance. The script must be run once for each Annotator Training Project. It outputs a CSV file called `iou_results.csv` containing the results. The evaluation metrics used are Intersection over Union (IoU) and Class score. <Info> * **IoU (Intersection over Union)**: Quantifies the overlap between predicted labels and the ground truth. It ranges from 0 to 1: 1.0: Indicates a perfect overlap between the predicted label and the ground truth. 0.0: Indicates no overlap between the predicted label and the ground truth. Values between 0 and 1: Represent the percentage of overlap. For example, an IoU of 0.6 signifies that 60% of the predicted label area overlaps with the ground truth label area. * **Class Score (0 or 1)**: 1: The label was created using the correct class. 0: The label was created using the wrong class. </Info> Ensure that you: * Replace `<private_key_path>` with the full path to your private access key. * Replace `<benchmark-project-id>` with the id of your Benchmark Project. * Replace `<training-project-id>` with the id of the Training Project you want to evaluate. <CodeGroup> ```python Annotator Training Evaluation theme={"dark"} from encord import EncordUserClient from encord.objects.common import Shape from encord.objects.coordinates import BoundingBoxCoordinates import pandas as pd from encord.user_client import EncordUserClient import os # Instantiate Encord client by substituting the path to your private key user_client = EncordUserClient.create_with_ssh_private_key( ssh_private_key_path="<private_key_path>" ) training_project_id = "<training-project-id>" benchmark_project_id = "<benchmark-project-id>" training_project = user_client.get_project(training_project_id) benchmark_project = user_client.get_project(benchmark_project_id) training_label_rows = training_project.list_label_rows_v2(workflow_graph_node_title_eq='Complete') benchmark_label_rows = benchmark_project.list_label_rows_v2(workflow_graph_node_title_eq='Complete') # Match by data_hash benchmark_dict = {lr.data_hash: lr for lr in benchmark_label_rows} paired_label_rows = [ (benchmark_dict[lr.data_hash], lr) for lr in training_label_rows if lr.data_hash in benchmark_dict ] # Initialize labels with training_project.create_bundle() as bundle: for _, prod_lr in paired_label_rows: prod_lr.initialise_labels(bundle=bundle, overwrite=True) with benchmark_project.create_bundle() as bundle: for bm_lr, _ in paired_label_rows: bm_lr.initialise_labels(bundle=bundle, overwrite=True) # IoU calculation def calculate_iou(bbox1: BoundingBoxCoordinates, bbox2: BoundingBoxCoordinates) -> float: x_left = max(bbox1.top_left_x, bbox2.top_left_x) y_top = max(bbox1.top_left_y, bbox2.top_left_y) x_right = min(bbox1.top_left_x + bbox1.width, bbox2.top_left_x + bbox2.width) y_bottom = min(bbox1.top_left_y + bbox1.height, bbox2.top_left_y + bbox2.height) intersection = max(0, x_right - x_left) * max(0, y_bottom - y_top) union = bbox1.width * bbox1.height + bbox2.width * bbox2.height - intersection return intersection / union if union > 0 else 0.0 # Compare labels and extract information results = [] for bm_lr, prod_lr in paired_label_rows: prod_instances = [oi for oi in prod_lr.get_object_instances() if oi.ontology_item.shape == Shape.BOUNDING_BOX and oi.get_annotation(0)] bm_instances = [oi for oi in bm_lr.get_object_instances() if oi.ontology_item.shape == Shape.BOUNDING_BOX and oi.get_annotation(0)] training_data_unit_name = prod_lr.data_title training_label_id = prod_lr.label_hash for prod_obj in prod_instances: best_iou = 0.0 best_match_hash = None prod_bbox = prod_obj.get_annotation(0).coordinates training_email = prod_obj.get_annotation(0).created_by for bm_obj in bm_instances: bm_bbox = bm_obj.get_annotation(0).coordinates iou = calculate_iou(prod_bbox, bm_bbox) if iou > best_iou: best_iou = iou best_match_hash = bm_obj.feature_hash class_score = 1.0 if best_match_hash == prod_obj.feature_hash and best_match_hash is not None else 0.0 results.append({ 'training_email': training_email, 'data_unit_name': training_data_unit_name, 'label_id': training_label_id, 'iou_score': best_iou, 'class_score': class_score }) # Output the results to a CSV file if results: df_results = pd.DataFrame(results) script_dir = os.path.dirname(os.path.abspath(__file__)) csv_file_path = os.path.join(script_dir, "iou_results.csv") df_results.to_csv(csv_file_path, index=False) print(f"Results saved to: {csv_file_path}") else: print("No matching label rows found for comparison.") ``` ```csv Example Output theme={"dark"} training_email,data_unit_name,label_id,iou_score,class_score david.babuschkin@encord.com,Image3.jpg,a4a3d707-e077-47f1-9bad-d05a1fcc28aa,0.0,0.0 david.babuschkin@encord.com,Image6.jpg,be53b547-7a5e-4476-9018-903ce7ad4d99,0.5506647634569868,1.0 ``` </CodeGroup>