Ontologies

Before we can create machine learning and computer vision applications that understand the data they look at, we need to establish just exactly what is to be understood and recognized. An ontology, sometimes known as a taxonomy or labeling protocol, does exactly that. Ontologies establish the set of concepts, their relationships and their representations in our data, and are the layer of communication which allow us to semantically program our machine learning applications.

ℹ️

Note

Our DICOM customers might be more familiar with the term 'labeling protocol', which is equivalent to an ontology.

Watch the video below to learn how to create ontologies.

Create Ontologies

Navigate to the Ontologies tab in the Navigation bar to show a list of Ontologies visible to your user.

1. Click the the +New ontology button to create a new Ontology.

2. Give your Ontology a meaningful title and description. Click Next to continue.

3. Create your Ontology structure.

Use the + buttons under the Objects and Classifications section to add them to the structure. You can use attributes to create expressive, hierarchically descriptive structures and attach classification modifiers to aid in the annotation process or dynamically annotate data, as necessary. Reference the structure section of the documentation below for full details on the significance and configuration details of each component.

As you are editing the structure, the 'Sandbox' view offers a preview of how the Ontology appears to your annotators in the label editor. You can expand and contract individual elements by clicking them to confirm the tree structure of the Ontology. You can also view the Ontology's JSON representation by toggling the Display JSON toggle on the upper right. This may be useful if you are using Encord's SDK or API.

{
  "objects": [
    {
      "id": "1",
      "name": "Apple",
      "color": "#D33115",
      "shape": "bounding_box",
      "featureNodeHash": "dfa43291",
      "attributes": [
        {
          "id": "1.1",
          "name": "Color",
          "type": "radio",
          "required": false,
          "featureNodeHash": "3a2d7e3e",
          "options": [
            {
              "id": "1.1.1",
              "label": "Red",
              "value": "red",
              "featureNodeHash": "d262efe3"
            },
            {
              "id": "1.1.2",
              "label": "Green",
              "value": "green",
              "featureNodeHash": "05e37fad"
            }
          ]
        }
      ]
    }
  ],
  "classifications": [
    {
      "id": "1",
      "featureNodeHash": "621710a2",
      "attributes": [
        {
          "id": "1.1",
          "name": "Orientation",
          "type": "radio",
          "required": false,
          "featureNodeHash": "2313aee5",
          "options": [
            {
              "id": "1.1.1",
              "label": "Top view",
              "value": "top_view",
              "featureNodeHash": "cc336600"
            },
            {
              "id": "1.1.2",
              "label": "Side view",
              "value": "side_view",
              "featureNodeHash": "02d60e6f"
            }
          ]
        }
      ]
    }
  ]
}

Supported annotation types

Intro to Ontology structure

Ontologies are hierarchical structures which capture not only the top-level concepts and categories present in your data, but also allow nested attributes for fine-grained differentiation or detailed annotations. At the top most level, ontologies are composed of classes (sometimes known as categories), the first level of concepts you wish to represent. Ontology classes can be either objects or classifications. Hierarchical attributes are added via attributes.

Objects are used in object detection and segmentation, when you want to annotate not only the category but also the location. Classifications (sometimes known as frame level classifications on Encord) annotate whether something is present in a given frame, but don't require localization data. Attributes can be nested under objects, classifications, and even other attributes to create deeply nested structures capable of modeling complex and even conditional attribute relationships.

Ontology Structure

Objects

Objects are configured with a title, an object annotation type and optionally any attributes you wish to assign. You may also change the color if you wish. Object annotation types are as follows:

• Box: Frequently known as a bounding box, they are quick to draw and work with many advanced automated labeling techniques.
• Rotatable box: Also known as a Rotatable bounding box, allows you to draw bounding boxes that can be rotated to create more accurate labels.
• Polygon: Useful to capture complex shapes for which a bounding box is not sufficiently precise. They are sometimes known as segmentations. Polygons cannot be self-intersecting.
• Polyline: An unclosed polygon, useful for representing long, thin annotations.
• Keypoint: A simple geometric point which can be useful for tracking small objects or particular points of interest on larger objects.
• Bitmask: Create labels using a brush tool to select parts of an image, which may be topologically separate from each other. Optionally you can select a range of intensity values within which labels will be created.
• Object primitive: Formerly called Skeleton template. A collection of geometric points connected in the manner of your choosing, primitives excel at representing particularly complex shapes such as those found in pose estimation problems.

Classifications

Since classifications apply to the frame as a whole, there is no need for a corresponding color. Classification annotation types may be any of the following:

• Checklist: For multi-valued classifications. An example could be talking about weather features -- for example, it could be both cloudy and rainy.
• Radio: Single-valued classifications. A canonical example is scene classification: is that an office? home? etc.
• Text: The annotator fills in the value freely in each situation.

ℹ️

Note

Radio buttons and text fields can be nested up to 7 layers deep. Check boxes do not allow further nesting.

Attributes

Attributes (sometimes called nested classifications) can either be nested under objects or under any classification with an annotation type of Radio. After setting the type as Radio, click the Configure button (stacked data items which display Nested options on mouseover) next to the value under which you wish to create a further attribute. This allows for deep hierarchies and conditional annotation structures, which can help describe even the most complex annotation scenarios.

Required attributes

Any object, classification and attribute can be set as Required - which means that annotators will have to include at least one instance of the required feature in each task before it can be submitted.

Annotators trying to submit a task without an object or classification marked as Required will see the following warning message.

Click View issue to open the Issues drawer. From there annotators can seamlessly resolve all Required issues before moving onto the next task.

👍

Tip

All Required objects and classifications will appear as items in the Issues drawer of the label editor.

Enable the Required feature when creating the ontology object or classification to mark it as Required. The screenshot below shows the feature being enabled for a top-level classification.

👍

Tip

Ontology objects and classifications marked as Required are always listed alongside the symbol.

Dynamic attributes

First-level attributes on objects can be marked as dynamic attributes. Normally, attributes on objects are static -- that is, they apply across the lifetime of the instance. By marking an attribute as dynamic, it's possible to apply that particular attribute to a selected subset of frames. This is very useful for indicating temporary attributes on an object, such as mode of transport, direction, state of occlusion, and so on.

👍

Tip

Refer to our documentation on Keyframe-based attribute propagation to learn how to use dynamic attributes in the Label Editor, and seamlessly apply dynamic attributes to a range of frames in a video.

Preserve current state

A 'Preserve current state' checkbox is available for the first instance of a label with a dynamic classification. Selecting this option propagates the classification for all future instance labels. This means that all labels created for this instance have the same classification as the initial instance label.

The 'Preserve current state' functionality also holds for instance labels created automatically with interpolation.

Relation attributes

The Relation attribute allows you link objects, and specify the relationship between them using text regardless of the annotation type used.

ℹ️

Note

Only one of the linked objects need a Relation attribute for the objects to be considered linked. For example, consider 2 Ontology objects: a chicken, and a chicken wing. To allow these objects to be linked, a Relation attribute must created for the chicken or the chicken wing, while setting up the Ontology.

All Relation attributes must be text fields. They cannot be radio buttons or checklists. Relation attributes can be applied to object labels of any kind, but not classifications.

To create a Relation attribute, enable the Relation feature when creating an attribute during Ontology creation. The default name for all Relation attributes is #relation.

ℹ️

Note

Objects are linked in the Label Editor during annotation, not during Ontology creation.

Using Relation attributes in the Label Editor

Once an Ontology with Relation attributes has been set up, instances can be linked in the Label Editor during annotation.

1. Create both instance labels. In this example a chicken and its wing have been labeled using bounding boxes.

2. Click the Edit classifications button for the object with the Relation attribute - in this example the chicken wing, as seen below.

3. Click the Set relation... bar and select the instance you want to link the selected instance to. In this example the chicken and the wing appear on the same frame, and therefore appear under the This frame heading. Instances in different frames appear under the heading Rest.

4. Click Done. The instances are now linked. This is shown in the Instance labels section with the name of the linked instance being displayed.

Considerations when designing ontologies

Creating an ontology is an important prerequisite when creating effective machine learning applications, and it can be helpful to keep some of the following in mind when designing your ontologies:

• The Problem Domain: it's important that ontologies are exhaustive, be sure there is a class or representation for the important concepts you want to address. Also keep in mind at what level to separate concepts. E.g. an application focused on recognizing various animals might have top-level classes of "cat" and "dog," but a problem focused on differentiating dog breeds might more appropriately feature "German Shepherd" and "Border Collie" as top-level classes.
• The Team: be sure to frame the classes and attributes in terms that are communicable across your entire team of annotators, reviewers, project managers, algorithmic developers or other involved stakeholders.
• The Workflow: annotation can be a difficult and time-consuming process. It's important to represent classes and their attributes appropriately, but designing ontologies such that objects and scenes can be labeled both correctly and quickly can lead to a more efficient labelling process.

Entity relationships

The diagram below illustrates the relationship different entities in Encord have to each other.

• Projects bring together ontologies and datasets.
• A project can have multiple datasets attached to it, but only one ontology.
• One ontology can be attached to multiple projects.