You’ve got your data into the Encord, and now you want to evaluate inter-annotator agreement. This is where Consensus Projects come into play. Consensus Projects consist of Ontologies, Consensus Workflows, Datasets, and Collaborators. We’ve already got the E2E Fruit Dataset, so we’re going to make the Ontology. We’ll use the default Consensus workflow and add some Collaborators. In Consensus Projects, each data unit needs to be annotated and submitted by multiple Annotators before the data unit moves to the REVIEW stage. The default Consensus workflow requires 3 Annotators to annotate and submit a data unit before the data unit moved to the REVIEW stage. The number of Annotators required for a data unit can be adjusted, but we will use the default value (3 Annotators).
1

Create Ontology

Our Ontology E2E Fruit Ontology is going to have a number of object label types and classifications.E2E Fruit OntologyThis Ontology is used for all Project creation types, from Basic to SOTA.
Class nameLabel TypeAttributes
BlueberriesPolygonRadio button:Type?
  • Bluegold
  • Duke
  • Blueray
CherriesBounding boxRadio button:Type?
  • Bing
  • King
  • Rainier
ApplesBitmaskRadio button:Types?
  • Sugar Bee
  • Granny Smith
  • Honey Crisp
BananasPolygonRadio button:Type?
  • Cavendish
  • Red
  • Blue Java
KiwiBounding BoxRadio button:Type?
  • Golden
  • Green
  • Kiwi Berry
PersimmonsBounding BoxRadio button:Type?
  • Hachiya
  • Fuyu
  • Rojo Brilliante
StrawberriesBounding BoxRadio button:Type?
  • Albion
  • Redcoat
  • Sweet Kiss
Ripe?ClassificationRadio button:
  • Yes
  • No
Type of fruit?ClassificationRadio button:
  • Blueberries
  • Cherries
  • Apples
  • Bananas
  • Kiwi
  • Persimmons
  • Strawberries
  • Other > What type? - Text box
Start Ontology CreationCreate OntologyAdd ObjectsCreate ObjectsAdd ClassificationsCreate ClassificationsSave OntologySave Ontology
2

Create Project

This step brings the Dataset and Ontology you created together with Collaborators and the Workflow.We’ll use the default Consensus workflow for this Consensus Project. This is what the Consensus workflow looks like:Consensus workflowTo create the Consensus Project:
  1. Click Projects > New annotation project.
  2. Add the following:
    • Project name: E2E - Fruit - Project I
    • Ontology: E2E Fruit Ontology
    • Dataset: E2E Fruit Dataset
    • Collaborators: Add your Annotators (label data), Reviewers (review labels), Team Managers (manage Annotators and Reviewers), and Project Admins (manage Project and export labels).
    • Workflow: We’ll use the default Consensus workflow for the Project. This means 3 Annotators are required to annotate and submit a data unit before the data unit moves to the REVIEW stage.
Create Consensus Project
3

Label Data