Audio for Direct Comparison (RLHF)

Encord’s multimodal platform and configurable label editor make it possible to support data curation and annotation workloads on various types of media, and in various annotation configurations. Follow the end to end walkthrough below to learn how to setup an Audio annotation workload where each task allows you to directly compare two audio files. This could be applicable for example when rating two audio files against each other, or in RLHF flows.

​

STEP 1: Create Metadata Schema

The schema organizes your custom metadata. Encord supports:

• Scalars: Methods for filtering and specifying data for custom Label Editor layouts.
• Enums: Methods with options for filtering.
• Embeddings: Method for embedding plot visualization, similarity search, and natural language search.

​

Custom metadata

Custom metadata can store any information which is not the data itself in a key-value format. Custom metadata is often used to describe information useful in curation and management at scale. Encord also uses metadata to create annotation specific layouts in the editor. You set up both using a metadata schema.

​

Metadata schema table

Use add_scalar to add a scalar key to your metadata schema.

Use add_enum and add_enum_options to add an enum and enum options to your metadata schema.

Use add_embedding to add an embedding to your metadata schema.

​

Import your metadata schema to Encord

# Import dependencies
from encord import EncordUserClient
from encord.metadata_schema import MetadataSchema

SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Create the schema
metadata_schema = user_client.metadata_schema()

# Add display parameters for data appearing in custom editor layouts
metadata_schema.add_scalar("encord-layout-group", data_type="varchar")
metadata_schema.add_scalar("encord-editor-grid-position", data_type="varchar")

# Add various metadata fields
metadata_schema.add_scalar("metadata_1", data_type="boolean")
metadata_schema.add_scalar("metadata_2", data_type="datetime")
metadata_schema.add_scalar("metadata_3", data_type="number")
metadata_schema.add_scalar("metadata_4", data_type="uuid")
metadata_schema.add_scalar("metadata_5", data_type="varchar")
metadata_schema.add_scalar("metadata_6", data_type="text")

# Add an enum field
metadata_schema.add_enum("my-enum", values=["enum-value-01", "enum-value-02", "enum-value-03"])

# Add embedding fields
metadata_schema.add_embedding('my-test-active-embedding', size=512)
metadata_schema.add_embedding('my-test-index-embedding', size=<values-from-1-to-4096>)

# Save the schema
metadata_schema.save()

# Print the schema for verification
print(metadata_schema)

​

STEP 2: Create JSON for Import

The following is an example JSON file for uploading two audio files to Encord. Both files include clientMetadata to ensure they display correctly in custom Label Editor layouts.

• The "encord-layout-group" key determines which files are shown together — files with the same "encord-layout-group" value are displayed simultaneously. Both files in the example appears in the Label Editor simultaneously since they have the same "encord-layout-group" value.

• The "encord-editor-grid-position" key, set to either A or B, specifies the exact position of each file within the label editor.

One audio file includes audioMetadata and one does not.

{
  "audio": [
    {
      "objectUrl": "<object url_1>",
      "clientMetadata": {"encord-editor-grid-position": "A", "encord-layout-group": "1"}
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-custom-audio-file-title.mp3",
      "audioMetadata": {
        "duration":  file-duration-in-seconds,
        "file_size": file-size-in-bytes,
        "mime_type": "MIME-type-of-the-audio-file-example-audio/mpeg",
        "sample_rate": sample-rate-in-Hz,
        "bit_depth": size-of-each-sample-in-bits,
        "codec": "file-codec-example-mp3",
        "num_channels": number-of-channels
        },
       "clientMetadata": {"encord-editor-grid-position": "B", "encord-layout-group": "1"}
    }
  ],
  "skip_duplicate_urls": true
}

​

STEP 3: Import your Audio files

You can import your audio data, using the JSON file you created, using the UI or the SDK.

​

Import with the UI

1. Navigate to Files section of Index in the Encord platform.
2. Click into a Folder.
3. Click + Upload files. A dialog appears.

4. Click Import from cloud data.

6. Click Add JSON or CSV files to add a JSON or CSV file specifying cloud data that is to be added.

​

Import with the SDK

To use your data in Encord, it must be uploaded to the Encord Files storage. Once uploaded, your data can be reused across multiple Projects and contain no labels or annotations themselves. Files stores your data, while Projects store your labels. The following script creates a folder in Files and uses your AWS integration to register data in that folder.

The following script creates a new folder in Files and initiates uploads from AWS. It works for all file types.

Ensure that you:

• Replace <private_key_path> with the path to your private key.
• Replace <integration_title> with the title of the integration you want to use.
• Replace <folder_name> with the folder name. The scripts assume that the specified folder name is unique.
• Replace path/to/json/file.json with the path to a JSON file specifying which cloud storage files should be uploaded.
• Replace A folder to store my files with a meaningful description for your folder.
• Replace "my": "folder_metadata" with any metadata you want to add to the folder.

The script has several possible outputs:

• “Upload is still in progress, try again later!”: The registration has not finished. Run this script again later to check if the data registration has finished.
• “Upload completed”: The registration completed. If any files failed to upload, the URLs are listed.
• “Upload failed”: The entire registration failed, and not just individual files. Ensure your JSON file is formatted correctly.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import LongPollingStatus  # Ensure correct import path

# Instantiate user client. Replace <private_key_path> with the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Specify the integration you want to use
integrations = user_client.get_cloud_integrations()
integration_idx = [i.title for i in integrations].index("<integration_title>")
integration = integrations[integration_idx].id

# Create a storage folder
folder_name = "<folder_name>"
folder_description = "A folder to store my files"
folder_metadata = {"my": "folder_metadata"}
storage_folder = user_client.create_storage_folder(
    folder_name, folder_description, client_metadata=folder_metadata
)

# Initiate cloud data registration
upload_job_id = storage_folder.add_private_data_to_folder_start(
    integration_id=integration, private_files="path/to/json/file.json", ignore_errors=True
)

# Check upload status
res = storage_folder.add_private_data_to_folder_get_result(upload_job_id, timeout_seconds=5)
print(f"Execution result: {res}")

if res.status == LongPollingStatus.PENDING:
    print("Upload is still in progress, try again later!")
elif res.status == LongPollingStatus.DONE:
    print("Upload completed")
    if res.unit_errors:
        print("The following URLs failed to upload:")
        for e in res.unit_errors:
            print(e.object_urls)
else:
    print(f"Upload failed: {res.errors}")

​

STEP 4: Create a Dataset

1. Click the New dataset button in the Datasets section in Annotate.

2. Give your Dataset a meaningful title and description. A clear title and description keeps your data organized.

3. Click Create dataset to create the Dataset.

​

Attach files

1. Navigate to the Datasets section under the Annotate heading.
2. Click the Dataset you want to attach data to.
3. Click +Attach existing files.

4. Select the folders containing the files you want to attach to the Dataset. To select individual files, double-click a folder to see its contents, and select the files you want to add to the Dataset.

5. Click Attach data to attach the selected files to the Dataset.

​

STEP 5: Create an Ontology

1. Click the New ontology button in the Ontologies section to create a new Ontology.

2. Give your Ontology a meaningful title and description. A clear title and description keeps your Ontologies organized. Click Next to continue.

3. Define your Ontology structure.

• Click Add classification
• Type ‘Winner?’
• Click on 0 options to configure options
• Click Add option
• Enter the word Yes as an option
• Click Add option
• Enter the word No as an option
• Click Create ontology

​

STEP 6: Create a Project

1. In the Encord platform, select Projects under Annotate.
2. Click the + New annotation project button to create a new Project.

3. Give the Project a meaningful title and description.

4. Click the Attach ontology button.

5. Select the Ontology you created in STEP 5 and click the Attach button.

6. Click OK to attach the Ontology to the Project.

7. Click the Attach datasets button.

8. Select the Dataset you created in STEP 4 and click the Attach button.

9. Click OK to attach the Dataset(s) to the Project.

10. Ensure the default Workflow shown suits your needs.

11. Click Create project to finish creating the Project.

​

STEP 7: Import Custom Layout

After creating your Project, import a custom layout for the Label Editor. Start by creating a JSON file that determines the layout.

​

Create a JSON for the Layout

{
  "grid": [
    [0], [1]
  ],
  "gridContent": {
    "0": {
      "client_metadata_query": {
        "or_conditions": [
          {
            "key": "encord-editor-grid-position",
            "operator": "=",
            "value": "A"
          },
          {
            "key": "encord-editor-grid-position",
            "operator": "=",
            "value": "B"
          }
        ]
      }
    },
    "1": {
      "client_metadata_query": {
        "or_conditions": [
          {
            "key": "encord-editor-grid-position",
            "operator": "=",
            "value": "A"
          },
          {
            "key": "encord-editor-grid-position",
            "operator": "=",
            "value": "B"
          }
        ]
      }
    }
  },
  "topLevelGridFilter": {
    "client_metadata": {
      "key": "encord-layout-group",
      "operator": "="
    }
  },
  "$schema": "https://raw.githubusercontent.com/encord-team/Annotate/main/editor-layouts/editor_layout_schema.json"
}

​

Import the JSON

1. Navigate to the Project settings.
2. Click on Layouts.
3. Click Upload JSON.
4. Select the JSON file containing the layout you want the Label Editor to have.

​

STEP 8: Label and Submit Audio Data

More than a single file appears in the Label Editor. Switch back and forth between the files you are labeling or reviewing using the Annotate from this tile icon.

​

Label Audio Files

Label audio files using the Classification specified from your Ontology.

To label audio files:

1. Go to Project > [Your Project Title] > Queue > Annotate.

   The Project Annotate Queue appears with a list of audio files for labelling.

2. Click Initiate next to a file from the Project Queue.

   Two audio files appear in the Label Editor.

3. Press 2 to select the Winner Classification.

   The options for the Classification appear.

4. Press Q to select Yes.

5. Click and then drag the sections of the audio file that appears at the top of the Label Editor.

6. Press N to save the Classifications on the audio file.

7. Click the Annotate from this tile icon to switch to annotating the bottom audio file.

8. Press Q to select Yes.

9. Click and then drag the sections of the audio file that appears at the top of the Label Editor.

10. Press N to save the Classifications on the audio file.

10. Click Submit.

    The next set of audio files for annotation appears.

​

STEP 9: Review Audio Data

Approving tasks moves the tasks to the next stage in a workflow. In this example, that is Complete.

Rejecting tasks sends the tasks back to the first stage in a workflow. In this example, that is Annotate.

​

Approve

To approve labels on audio files:

1. Go to Project > [Your Project Title] > Queue > Review.

   The Project Review Queue appears with a list of audio files for labelling.

2. Click Initiate next to a file from the Project Queue.

   Two audio files appear in the Label Editor.

3. Expand the Winner Classification.

   The options for the Classification appear.

4. Select the Classifications applied to the audio file.

5. Press N to approve the Classifications on the audio file.

6. Press the Review from this file icon.

   .

7. Expand the Winner Classification.

   The options for the Classification appear.

   .

8. Select the Classifications applied to the audio file.

9. Press N to approve the Classifications on the audio file.

10. Click Approve task to move the task to the next stage.

    .

    In this example, that is the Complete stage.

    .

​

Reject

To reject labels on audio files:

1. Go to Project > [Your Project Title] > Queue > Review.

   The Project Review Queue appears with a list of audio files for labelling.

2. Click Initiate next to a file from the Project Queue.

   Two audio files appear in the Label Editor.

3. Expand the Winner Classification.

   The options for the Classification appear.

4. Select the Classifications applied to the audio file.

5. Press B to reject the Classifications on the audio file.

   A dialog appears.

6. Provide a reason for rejecting the Classification.

7. Press the Review from this file icon.

   .

8. Expand the Winner Classification.

   The options for the Classification appear.

   .

9. Select the Classifications applied to the audio file.

10. Press B to reject the Classifications on the audio file.

11. Click Reject task to reject the tasks.

.

Both tasks then appear in the Annotate stage.

​

Complete

Files in the Complete stage can be reopened.

.