Review Consensus Projects (BETA)

Aligned Image with Page Break

Reveiwers on Consensus Projects have a different goal from Reviewers on other types of Encord Projects. Reviewers on Consensus Projects review ALL the labels on a data unit, made by a specified number of annotators. The reviewers then make a decision on whether there is agreement/consensus on ALL the labels on a task (data unit).

If the reviewer decides that ALL the labels on a data unit are correct/in agreement, the reviewer selects the annotators that are in agreement and clicks Consensus reached. If the reviewer decides that there is no agreement on the labels or that the labels are incorrect, the reviewer clicks No Consensus.

Currently, Classifications are supported for consensus review on all data unit types. Data units are images, image groups, image sequences, videos, and DICOM series.

To review a Consensus Project:

  1. Log in to Encord.
    Your review task queue appears for the latest Project you are assigned for review.

  2. Go to Annotate > Projects > Annotate projects.
    The Annotate projects page appears displaying a list of Projects you are assigned to review.

  3. Click a Consensus Project.
    The Project opens with the Review populated with review tasks.

  4. Click Initiate on a task.
    The Reviewing page appears.

  5. Review the labels from each annotator.

❗️

CRITICAL INFORMATION

Consensus in Encord is based on Consensus for ALL labels on a data unit. A data unit being an image, an image group, an image sequence, a video, or a DICOM series. Encord Consensus is not granular to the label itself (yet). This means reviewers must ensure ALL labels on a data unit are in consensus before deciding consensus is reached.

The Reviewing page is different for single images and all other data units. When reviewing single images, expand each user to display their labels and decide on consensus. When reviewing videos, image groups, image sequences, and DICOM series a timeline appears under the data unit. Use the timeline to speed up reviewing all the labels on the data unit

Single Image, Image Group

Video, Image Sequence, DICOM Series

  1. If the labels are correct:

a. Select the annotators that have done the best job of labeling the data unit.
The Consensus reached button becomes clickable once the required number of annotators required are selected.

b. Click the Consensus reached button.

c. Select the annotator that did the best job. That annotator's work on the data unit is used if an Admin or Team Manager export the labels for the data unit.

d. Click Submit.

Single Image, Image Group

Video, Image Sequence, DICOM Series

  1. Click No consensus if the labels are not correct OR if not enough of the annotators have labeled the data unit correctly.