Audio for Direct Comparison (RLHF)

​

STEP 1: Create Metadata Schema

• Scalars: Methods for filtering and specifying data for custom Label Editor layouts.
• Enums: Methods with options for filtering.
• Embeddings: Method for embedding plot visualization, similarity search, and natural language search.

​

Custom metadata

​

Metadata schema table

​

Import your metadata schema to Encord

# Import dependencies
from encord import EncordUserClient
from encord.metadata_schema import MetadataSchema

SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

# Create the schema
metadata_schema = user_client.metadata_schema()

# Add display parameters for data appearing in custom editor layouts
metadata_schema.add_scalar("encord-layout-group", data_type="varchar")
metadata_schema.add_scalar("encord-editor-grid-position", data_type="varchar")

# Add various metadata fields
metadata_schema.add_scalar("metadata_1", data_type="boolean")
metadata_schema.add_scalar("metadata_2", data_type="datetime")
metadata_schema.add_scalar("metadata_3", data_type="number")
metadata_schema.add_scalar("metadata_4", data_type="uuid")
metadata_schema.add_scalar("metadata_5", data_type="varchar")
metadata_schema.add_scalar("metadata_6", data_type="text")

# Add an enum field
metadata_schema.add_enum("my-enum", values=["enum-value-01", "enum-value-02", "enum-value-03"])

# Add embedding fields
metadata_schema.add_embedding('my-test-active-embedding', size=512)
metadata_schema.add_embedding('my-test-index-embedding', size=<values-from-1-to-4096>)

# Save the schema
metadata_schema.save()

# Print the schema for verification
print(metadata_schema)

​

STEP 2: Create JSON for Import

• The "encord-layout-group" key determines which files are shown together — files with the same "encord-layout-group" value are displayed simultaneously. Both files in the example appears in the Label Editor simultaneously since they have the same "encord-layout-group" value.
• The "encord-editor-grid-position" key, set to either A or B, specifies the exact position of each file within the label editor.

{
  "audio": [
    {
      "objectUrl": "<object url_1>",
      "clientMetadata": {"encord-editor-grid-position": "A", "encord-layout-group": "1"}
    },
    {
      "objectUrl": "<object url_2>",
      "title": "my-custom-audio-file-title.mp3",
      "audioMetadata": {
        "duration":  file-duration-in-seconds,
        "file_size": file-size-in-bytes,
        "mime_type": "MIME-type-of-the-audio-file-example-audio/mpeg",
        "sample_rate": sample-rate-in-Hz,
        "bit_depth": size-of-each-sample-in-bits,
        "codec": "file-codec-example-mp3",
        "num_channels": number-of-channels
        },
       "clientMetadata": {"encord-editor-grid-position": "B", "encord-layout-group": "1"}
    }
  ],
  "skip_duplicate_urls": true
}

​

STEP 3: Import your Audio files

​

Import with the UI

1. Navigate to Files section of Index in the Encord platform.
2. Click into a Folder.
3. Click + Upload files. A dialog appears.

4. Click Import from cloud data.

6. Click Add JSON or CSV files to add a JSON or CSV file specifying cloud data that is to be added.

​

Import with the SDK

• Replace <private_key_path> with the path to your private key.
• Replace <integration_title> with the title of the integration you want to use.
• Replace <folder_name> with the folder name. The scripts assume that the specified folder name is unique.
• Replace path/to/json/file.json with the path to a JSON file specifying which cloud storage files should be uploaded.
• Replace A folder to store my files with a meaningful description for your folder.
• Replace "my": "folder_metadata" with any metadata you want to add to the folder.

• “Upload is still in progress, try again later!”: The registration has not finished. Run this script again later to check if the data registration has finished.
• “Upload completed”: The registration completed. If any files failed to upload, the URLs are listed.
• “Upload failed”: The entire registration failed, and not just individual files. Ensure your JSON file is formatted correctly.

# Import dependencies
from encord import EncordUserClient
from encord.orm.dataset import LongPollingStatus  # Ensure correct import path

# Instantiate user client. Replace <private_key_path> with the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Specify the integration you want to use
integrations = user_client.get_cloud_integrations()
integration_idx = [i.title for i in integrations].index("<integration_title>")
integration = integrations[integration_idx].id

# Create a storage folder
folder_name = "<folder_name>"
folder_description = "A folder to store my files"
folder_metadata = {"my": "folder_metadata"}
storage_folder = user_client.create_storage_folder(
    folder_name, folder_description, client_metadata=folder_metadata
)

# Initiate cloud data registration
upload_job_id = storage_folder.add_private_data_to_folder_start(
    integration_id=integration, private_files="path/to/json/file.json", ignore_errors=True
)

# Check upload status
res = storage_folder.add_private_data_to_folder_get_result(upload_job_id, timeout_seconds=5)
print(f"Execution result: {res}")

if res.status == LongPollingStatus.PENDING:
    print("Upload is still in progress, try again later!")
elif res.status == LongPollingStatus.DONE:
    print("Upload completed")
    if res.unit_errors:
        print("The following URLs failed to upload:")
        for e in res.unit_errors:
            print(e.object_urls)
else:
    print(f"Upload failed: {res.errors}")

​

STEP 4: Create a Dataset

1. Click the New dataset button in the Datasets section in Annotate.

2. Give your Dataset a meaningful title and description. A clear title and description keeps your data organized.

3. Click Create dataset to create the Dataset.

​

Attach files

1. Navigate to the Datasets section under the Annotate heading.
2. Click the Dataset you want to attach data to.
3. Click +Attach existing files.

4. Select the folders containing the files you want to attach to the Dataset. To select individual files, double-click a folder to see its contents, and select the files you want to add to the Dataset.
5. Click Attach data to attach the selected files to the Dataset.

​

STEP 5: Create an Ontology

1. Click the New ontology button in the Ontologies section to create a new Ontology.

2. Give your Ontology a meaningful title and description. A clear title and description keeps your Ontologies organized. Click Next to continue.

3. Define your Ontology structure.

• Click Add classification
• Type ‘Winner?’
• Click on 0 options to configure options
• Click Add option
• Enter the word Yes as an option
• Click Add option
• Enter the word No as an option
• Click Create ontology

​

STEP 6: Create a Project

1. In the Encord platform, select Projects under Annotate.
2. Click the + New annotation project button to create a new Project.

3. Give the Project a meaningful title and description.

4. Click the Attach ontology button.
5. Select the Ontology you created in STEP 5 and click the Attach button.

6. Click OK to attach the Ontology to the Project.
7. Click the Attach datasets button.
8. Select the Dataset you created in STEP 4 and click the Attach button.

9. Click OK to attach the Dataset(s) to the Project.

10. Ensure the default Workflow shown suits your needs.
11. Click Create project to finish creating the Project.

​

STEP 7: Import Custom Layout

​

Create a JSON for the Layout

{
  "grid": [
    [0], [1]
  ],
  "gridContent": {
    "0": {
      "client_metadata_query": {
        "or_conditions": [
          {
            "key": "encord-editor-grid-position",
            "operator": "=",
            "value": "A"
          },
          {
            "key": "encord-editor-grid-position",
            "operator": "=",
            "value": "B"
          }
        ]
      }
    },
    "1": {
      "client_metadata_query": {
        "or_conditions": [
          {
            "key": "encord-editor-grid-position",
            "operator": "=",
            "value": "A"
          },
          {
            "key": "encord-editor-grid-position",
            "operator": "=",
            "value": "B"
          }
        ]
      }
    }
  },
  "topLevelGridFilter": {
    "client_metadata": {
      "key": "encord-layout-group",
      "operator": "="
    }
  },
  "$schema": "https://raw.githubusercontent.com/encord-team/Annotate/main/editor-layouts/editor_layout_schema.json"
}

​

Import the JSON

1. Navigate to the Project settings.
2. Click on Editor Layouts.
3. Click Upload JSON.
4. Select the JSON file containing the layout you want the Label Editor to have.

​

STEP 8: Label and Submit Audio Data

​

Label Audio Files

1. Go to Project > [Your Project Title] > Queue > Annotate. The Project Annotate Queue appears with a list of audio files for labeling.
2. Click Initiate next to a file from the Project Queue. Two audio files appear in the Label Editor.
3. Press 2 to select the Winner Classification. The options for the Classification appear.
4. Press Q to select Yes.
5. Click and then drag the sections of the audio file that appears at the top of the Label Editor.
6. Press N to save the Classifications on the audio file.
7. Click the Annotate from this tile icon to switch to annotating the bottom audio file.
8. Press Q to select Yes.
9. Click and then drag the sections of the audio file that appears at the top of the Label Editor.
10. Press N to save the Classifications on the audio file.

10. Click Submit. The next set of audio files for annotation appears.

​

STEP 9: Review Audio Data

​

Approve

1. Go to Project > [Your Project Title] > Queue > Review. The Project Review Queue appears with a list of audio files for labeling.
2. Click Initiate next to a file from the Project Queue. Two audio files appear in the Label Editor.
3. Expand the Winner Classification. The options for the Classification appear.
4. Select the Classifications applied to the audio file.
5. Press N to approve the Classifications on the audio file.
6. Press the Review from this file icon. .
7. Expand the Winner Classification. The options for the Classification appear. .
8. Select the Classifications applied to the audio file.
9. Press N to approve the Classifications on the audio file.
10. Click Approve task to move the task to the next stage. . In this example, that is the Complete stage. .

​

Reject

1. Go to Project > [Your Project Title] > Queue > Review. The Project Review Queue appears with a list of audio files for labeling.
2. Click Initiate next to a file from the Project Queue. Two audio files appear in the Label Editor.
3. Expand the Winner Classification. The options for the Classification appear.
4. Select the Classifications applied to the audio file.
5. Press B to reject the Classifications on the audio file. A dialog appears.
6. Provide a reason for rejecting the Classification.
7. Press the Review from this file icon. .
8. Expand the Winner Classification. The options for the Classification appear. .
9. Select the Classifications applied to the audio file.
10. Press B to reject the Classifications on the audio file.
11. Click Reject task to reject the tasks.

.

​

Complete

.