The Consensus Stage

Suggest Edits

Consensus stages enable the quality assurance of annotations by measuring the extent of agreement between different annotators (and/or models) and allowing the selection of the best annotation when there is a disagreement. This page describes how to effectively use Consensus Stages in your workflows.

Overview

A Consensus Stage has three parts: the stage setup, annotation and review. This can best be understood via an example, as described below.

A medical team has 2 radiologists and a cancer segmentation model which it wants to use to annotate cancerous/non-cancerous DICOM images. At a high level:

  1. The workforce manager sets up a Consensus Stage and assigns the 2 radiologists and the model to parallel stages within the Consensus stage.
  2. The annotators request work and independently annotate the items in the Consensus Stage, whilst the model generates its predictions.
  3. If there is disagreement between the annotators and/or model, the items enter a Review Stage, where the workforce manager reviews and resolves any disagreements.

How to Setup a Consensus Stage

  1. Add a Consensus Stage to your workflow in the workflows editor.
  2. Add and specify which models and/or annotators will participate in the Consensus Stage. You can add multiple parallel stages and annotators per parallel stage.
1812
  1. Configure the Consensus threshold per class. This specifies the minimum IoU score required for an item to automatically pass through the Consensus Stage.
276
  1. Specify which model or annotator will act as your Champion Stage. If an item automatically passes through a Consensus Stage due to an agreement, the Champion Stage's annotation is deemed as the ‘correct’ annotation and is used moving forward.
322
  1. Save your workflow and let your annotators request work.

How to Review Disagreements

  1. After all annotators and/or models have added their annotations, the disagreements can be reviewed in a review stage.
344
  1. In the annotation panel, go to the "issues" tab and view the disagreements
462
  1. To resolve a disagreement, either accept a single annotation, or delete other annotations in the disagreement until you have the “winning” annotation left.
1470

Parallel Blind Reads with the Consensus Stage

A common use case for the Consensus Stage is for having multiple labelers annotate the same image blindly in parallel.

In medical use cases where assessment can be subjective, and one reader is not necessarily more correct than the other - the Consensus Stage allows you to collect multiple readers opinions for analysis of inter-reader variability on your dataset.

To use the Consensus Stage for Parallel Blind Reads, set the disagreement threshold to 100%. This will mean all annotations pass through to the next stage.

230

Supported Classes

V7's consensus stage is compatible with images, videos, DICOMs and PDFs, for any number of annotators and/or models.

V7 supports automated disagreement checking for:

  • Bounding Boxes (IoU)
  • Polygons (IoU)
  • Tags (voting)

V7 allows you to use the consensus stage for blind parallel reads (without automated disagreement checking) for the following classes:

  • Polylines
  • Ellipses
  • Keypoints
  • Keypoint Skeletons

This means you can still do a simple automated review in the V7 Platform. If you have a consensus use case for one of these annotation types, please ask us to turn this setting on for you by contacting support or via your dedicated Customer Success Manager.

Updated 18 days ago