You’ve got your data into the Encord, and now you want to evaluate inter-annotator agreement. This is where Consensus Projects come into play.

Consensus Projects consist of Ontologies, Consensus Workflows, Datasets, and Collaborators.

We’ve already got the E2E Fruit Dataset, so we’re going to make the Ontology. We’ll use the default Consensus workflow and add some Collaborators.

In Consensus Projects, each data unit needs to be annotated and submitted by multiple Annotators before the data unit moves to the REVIEW stage. The default Consensus workflow requires 3 Annotators to annotate and submit a data unit before the data unit moved to the REVIEW stage. The number of Annotators required for a data unit can be adjusted, but we will use the default value (3 Annotators).

1

Create Ontology

Our Ontology E2E Fruit Ontology is going to have a number of object label types and classifications.

This Ontology is used for all Project creation types, from Basic to SOTA.

Class nameLabel TypeAttributes
BlueberriesPolygon

Radio button:

Type?

  • Bluegold
  • Duke
  • Blueray
CherriesBounding box

Radio button:

Type?

  • Bing
  • King
  • Rainier
ApplesBitmask

Radio button:

Types?

  • Sugar Bee
  • Granny Smith
  • Honey Crisp
BananasPolygon

Radio button:

Type?

  • Cavendish
  • Red
  • Blue Java
KiwiBounding Box

Radio button:

Type?

  • Golden
  • Green
  • Kiwi Berry
PersimmonsBounding Box

Radio button:

Type?

  • Hachiya
  • Fuyu
  • Rojo Brilliante
StrawberriesBounding Box

Radio button:

Type?

  • Albion
  • Redcoat
  • Sweet Kiss
Ripe?Classification

Radio button:

  • Yes
  • No
Type of fruit?Classification

Radio button:

  • Blueberries
  • Cherries
  • Apples
  • Bananas
  • Kiwi
  • Persimmons
  • Strawberries
  • Other > What type? - Text box

Start Ontology Creation

Add Objects

Add Classifications

Save Ontology

2

Create Project

This step brings the Dataset and Ontology you created together with Collaborators and the Workflow.

We’ll use the default Consensus workflow for this Consensus Project. This is what the Consensus workflow looks like:

To create the Consensus Project:

  1. Click Projects > New annotation project.

  2. Add the following:

    • Project name: E2E - Fruit - Project I
    • Ontology: E2E Fruit Ontology
    • Dataset: E2E Fruit Dataset
    • Collaborators: Add your Annotators (label data), Reviewers (review labels), Team Managers (manage Annotators and Reviewers), and Project Admins (manage Project and export labels).
    • Workflow: We’ll use the default Consensus workflow for the Project. This means 3 Annotators are required to annotate and submit a data unit before the data unit moves to the REVIEW stage.

3

Label Data