Creating and configuring Workflows

Workflows are a powerful tool to design and build your projects - letting you control how an annotation task moves through different stages of the project, and determining how different stages are structured and interact with one another.

Select the Workflows (Beta) tab when setting up quality assurance during project creation to create a workflow project.

On this page you can find:

• A step-by-step tutorial on setting up and configuring your workflow project.
• A detailed explanation of all workflow composer components.

ℹ️

Note

Workflows are replacing the previous Task Manager which will be gradually phased out.

ℹ️

Note

We are rolling this feature out in Beta mode. Please understand there might be some blips in stability as we continuously add new features and improve performance.

---

Workflows - tutorial

This section will outline a step-by-step tutorial to create a project using Workflows.

To get started, create a new annotation project - select the datasets you'd like to add, as well an ontology for the project. In the final Set up quality assurance stage, select the Workflows (Beta) tab to create a workflow project.

---

1. Add users to the project

Add users from within your organization to the project by clicking the Add collaborators button, highlighted in the image below.

• Collaborators are added based on their role within the project - select the role you would like the collaborator(s) to have.
• Start typing the email of a user you would like to add into the area highlighted on the image below, and select the user from the list that appears. Repeat this for every user that will have the same role.
• When you are done selecting users for this role, click Add.

ℹ️

Note

Learn more about managing collaborators in the Managing collaborators section.

---

2. Configure your workflow

Click the + New workflow button to create a workflow. Any workflow created can be saved as a template by clicking the Save as a new template button.

Alternatively, you can use a template that was previously created. Click here to learn more.

Configure your workflow by pulling components from the Workflow library onto the canvas - the video below shows how to do this. The canvas will be populated with a Start stage, connected to an Annotate stage by default.

• Add the stages and other components you require for your workflow onto the canvas:

  • All workflows must begin with the Start stage.
  • All workflows must contain an Annotate stage.
  • Add routers to your project to determine different pathways through your workflow a task can take. Click on the Router card to determine the type of router.
  • Add as many Review stages as necessary.
  • All workflows must end at a Complete stage.

• Link all components on the canvas by clicking and dragging from one connection point to another, as seen below.

👍

Tip

The cursor changes from     to   when hovering over a connection point.

---

3. Configure the stages of your workflow

Once you’ve arranged the stages in the composer, it’s time to configure the details of each stage.

• Click an Annotate card on the canvas to start editing the annotation stage.
  • Give the stage a descriptive name.
  • Add annotators. If you'd like to specify annotators for this stage, add them as collaborators. For full details on how collaborators can work on tasks at each stage, see our section on managing collaborators.
  • Optionally, add a Webhook to receive notifications when labels are submitted at this stage.

ℹ️

Note

More details about configuring annotation stages can be found in the Annotate section below

• Click a Review card on the canvas to start editing the review stage.
  • Give the stage a descriptive name.
  • Add reviewers. If you'd like to specify reviewers for this stage, add them as collaborators. For full details on how collaborators can work on tasks at each stage, see our section on managing collaborators.

ℹ️

Note

More details about configuring review stages can be found in the Review section below.

• Click a Router card on your canvas. Please see the router section for more details on how different types of routers can be configured.

• Optionally, add a Webhook to the Complete stage to receive a notification when a task has been completed.

• Optionally, add User assignment restrictions if users in this node should be prevented from being assigned to tasks they completed in the nodes listed.

ℹ️

Note

More details on routers can be found in the router section below.

5. Create your project

Once all previous stages in this tutorial have been completed, and your graph has been validated you are ready to create your project. Simply click the Create Project button highlighted in the image below.

---

Workflow composer components

• A. Manage Collaborators allows you to seamlessly add users to your project as collaborators, which can be added to different stages of the workflow.

• B. The Workflow library contains all the components that make up a project. Drag the cards from the workflow library onto the workflow canvas (C) to build the stages a workflow should pass through. Some cards can be used more than once to create complex workflows. The configuration details for each component can be found in the Workflow library components section below.

• C. The Workflow canvas is the canvas on which you design your workflow, by connecting cards from the workflow library. Set up the stages you'd like your workflow to pass through from left to right, and connect them to suit your needs. Click and drag the canvas to move it.

• D. Zoom in and out of the Workflow canvas for convenience. Large workflow designs will require you to zoom out to see the whole picture. The icon focuses in on your workflow.

• E. Create the project, or return to the previous step of the project creation process. Click Create Project to complete the creation process.

🚧

Caution

Currently connections between nodes on the canvas can't be edited within a project after its creation. However, workflow templates can be edited and configured even when attached to ongoing projects. Click here.

---

Workflow library components

This section contains an overview of, and details on all Workflow library components.

---

Start

Start represents the beginning an annotation task through your workflow, and can therefore only be used once in a project. It must have a connector linking it to the next stage. It displays the datasets connected to the project, as well as the selected ontology.

---

Annotate

Annotate represents an annotation stage, during which a group of annotators adds labels to tasks assigned to them.

Click an Annotate card on the canvas to edit. Click x next to the name to exit edit mode.

Click the icon to edit the name of the annotation stage.

👍

Tip

Annotation instructions can be added in the project's Settings after the project has been created.

The stage will contain all available annotators selected by default. Specific annotators can be selected by clicking the Annotate component, and choosing the Selective tab seen above. Type the email address of a user you'd like to add and select them from the list. Only collaborators that have been added via Manage collaborators on the project level will be available. Click Add to confirm the selection.

👍

Tip

Annotation instructions can be added in the project's Settings after the project has been created.

ℹ️

Note

Only collaborators with the Annotator or the Annotator & Reviewer roles are available for selection. All Admins and Team Managers will be part of all annotation stages and therefore can't be added, or removed.

A Webhook can optionally be added to an annotation stage to send out notifications whenever a task is submitted. Press the + icon next to 'Webhook' and add the URL of the server you would like to receive notifications on.

---

Router

A Router splits the path that annotate and review tasks take through the workflow. See the different types of routers available below.

To set the type of router:

1. A the Router component to the canvas.
2. Click the Router component.
3. Click Select router type and select the type of router you would like this component to be.

ℹ️

Note

The type of router can't be changed once it's been determined. The only option is to delete the router, and create a new of a different type

Percentage router

Pathways that data units take through the workflow can be split according to a certain percentage of annotation tasks. For example, 50% of tasks get sent to review stage A, while the other 50% of tasks get sent to review stage B, splitting the workload between two different groups of reviewers.

Once the router selects a route for a given task, it will always be routed the same way. This means that if a task passes through a percentage router and is routed to review stage A on its first pass, it will be routed to review stage A on each consecutive pass.

Collaborator router

Determine the paths that data units take through your workflow based on users. For example, if user A completes a task it gets sent to be reviewed by review stage A, while a task completed by user B gets reviewed in review stage B.

The 'Or else' option accounts for all collaborators not explicitly added to the router. In the image above they will will be directed to review stage C.

1. Click Add pathway to select a subset of collaborators that will make up a pathway - only collaborators present in the previous stage can be selected. There is no limit on the amount of pathways you can create.

2. Click Add to finish a pathway.

ℹ️

Note

Collaborators can only be added to the Router once an input has been connected.

---

Review

Review represents a review stage where tasks created in an annotation stage are reviewed. A project can have multiple review stages, all of which have different reviewers, and are routed according to different conditions.

ℹ️

Note

Once a task has been rejected a review stage, it can't be passed onto another review stage directly.

Click a Review card that has been added to the canvas to edit. Click x next to the name to exit edit mode.

Click the icon to edit the name of the review stage. This is particularly useful when your project has multiple review stages.

The stage will contain all available reviewers selected by default. Specific reviewers can be selected by clicking the Review component, and choosing the Selective tab seen above. Type the email address of a user you'd like to add and select them from the list. Only collaborators that have been added via Manage collaborators on the project level will be available. Click Add to confirm the selection.

ℹ️

Note

Only collaborators with the Reviewer and Annotator & Reviewer roles are available for selection. All Admins and Team Managers will be part of all review stages and therefore can't be added, or removed.

User assignment restriction

The User assignment restriction feature prevents users from selected workflow stages being assigned to tasks in the current workflow stage. This can prevent individuals with the 'Annotator & Reviewer' role from being assigned their own annotations to review.

ℹ️

Note

This feature doesn't apply to users with the Admin or Team Manager roles, since they are added to each stage by default and can't be removed.

Enable the toggle to implement User assignment restriction. Include the stages from which you intend to restrict user assignments in the highlighted list shown in the image above. For instance, in the provided example, individuals from the Annotate 1 stage will be prevented from task assignments within the Review 1 stage.

---

Complete

Complete represents the stage of a project at which a task is marked as complete, and can therefore only be used once per project.

Having passed though your workflow, it was deemed to be acceptable data for the training of machine learning algorithms. Complete tasks can still be accessed by admins.

Optionally, add a Webhook to this stage to receive notifications whenever a task arrives at the Complete stage.Press the + icon next to 'Webhook' and add the URL of the server you would like to receive notifications on.

---

Managing collaborators

Video tutorial - Managing workflow collaborators

User permissions

Project level collaborators

Click the Manage Collaborators button, as shown in the Creating and configuring Workflows section.

• Add collaborators and select their roles in the 'Add tab', shown below.
• Remove and manage collaborators in the 'Manage' tab.

Workflow stage level collaborators

Collaborators can be added to a workflow stage by clicking + on a card that has been added to the workflow canvas.

Learn how to add collaborators to different stages of a project by visiting the relevant sections in this document:

• Annotate collaborators.
• Review collaborators.
• Collaborator type Router.

If no collaborators are specified at a given stage, all users with permissions will be added by default.

For example, if no collaborators are specified at an Annotate stage then all collaborators with the following roles wil be added by default:

• Annotator
• Annotator & Reviewer
• Team Manager
• Admin

---

Working with Workflows

Workflow annotation projects are projects which have been created using Workflows (Beta). Click here to learn about creating and configuring Workflow projects.

---

Project dashboard

Clicking on a project in the 'Projects overview' screen takes you to its 'Project dashboard'.

This is where you monitor and manage your project - view your project's summary statistics, manage labeling tasks, view your team's productivity, train models and invite collaborators.

The dashboard is split into 7 tabs:

• Summary: a high-level view of labeling and productivity statistics.
• Explore: a quick way to explore the distribution of instances and labels across data assets in the
  project.
• Labels: for managing all the project's labeling activity and tasks.
• Performance: a more detailed view of your team's manual labeling and productivity.
• Models: for creating, training and using pre-trained computer vision models with your labeled training data.
• Export: for exporting your labeling data.
• Settings: editing project options, ontology, team collaborators, and other general project settings.

---

Summary

Clicking a workflow project takes you to its summary dashboard. This dashboard has 2 components and gives you a rich visual display of your project's progress at a high level.

Project task status overview

Displays the number of tasks that are in each state of your workflow project. The number of states and their names reflect the choices made during workflow project creation.

👍

Tip

Click on the stages in the chart key to remove, or include them from the donut view

---

Explore

ℹ️

Note

The Explore tab is only visible to project Admins and Team Managers.

The Explore tab helps you understand how project annotations are distributed among data assets, at both an instance and label level. It allows a deeper exploration through attributes on objects, as well as frame-level classifications.

• Instance statistics: Class distribution across data assets in the given project.
• Label statistics: Label distributions within data assets, objects and classifications.

---

Instance statistics

This section provides the total count of all instances across the datasets in your project.

• Project total: Shows total instances (both objects and classifications) across the project by default. To get instance statistics for individual data files, click the drop-down to select a data file.
• Select class: Shows the total instances for a particular class. This is a summary of how a given class is distributed across your project's data assets. The pie chart segments show a breakdown of how that class is split across the data assets.
• Display timestamps: Flip the toggle to switch between frame numbers and timestamps for the labels.

---

Label statistics

This is a summary of how your labels are distributed across the project. The pie chart shows a breakdown of how many labels there are for a given class.

• Project total: Shows the total number of labels across different datasets in the project. To get label stats for individual data files, click the drop-down to select a data file.
• Objects: Click on the pie chart segment of a class to see the total number of labels and its attributes (sometimes called nested attributes) if available for that class.
• Classifications: Shows the global classification at project or individual video level. For example, location, time of day, etc.

Quick definitions of classes, instances and labels

• Class: Fundamental unit of the project's ontology. For example the ontology of a project annotating traffic
  videos could have classes such as Car, Truck, Bicycle, and so on. For more information on objects and classifications, see Ontologies Overview.
• Instance: Specific occurrence of a class. Car(0) is an instance of the Car class, for example, it could be a specific black sedan. The single Car(0) instance can appear in a single frame or a range of frames. Therefore, instances may contain multiple labels across frames.
• Label: An frame-specific annotation of an instance. For example the annotation of Car(0) on frame 201 is a label.

---

Queue

The Queue tab is where tasks are assigned and labeling is initiated for all users attached to a project. The queue tab will look different depending on the user's permissions.

• Annotator's Queue.
• Reviewer's Queue.
• Annotator & Reviewer, Admin, and Team Manager Queue.

Queue (Annotators)

Annotators will be presented with the following Queue tab, from which they can manage their annotations.

• A - Use the search bar to filter the list of data units being displayed, or to look for a particular data unit.
• B - The list of annotation stages shows how many data units each stage currently contains. If more than one stage is listed, clicking a stage lets you view the tasks it contains.
• C - Filter the list of data units being displayed by task status by Dataset, User, or Data type.
• D - The list of tasks / data units in your queue. Unassigned tasks are also visible as these can be initiated by all Annotators.
• E - This column shows which user a task is assigned to. Unassigned tasks are also visible.
• F - Click the Initiate button next to a task to start annotating. Clicking the Start annotating button will start the annotation process from the first task in the queue.

ℹ️

Note

Initiating an unassigned task automatically assigns the task to the Annotator.

Queue (Reviewers)

• A - Use the search bar to filter the list of data units being displayed, or to look for a particular data unit.
• B - The list of review stages shows how many data units each stage currently contains. If more than one stage is listed, clicking a stage lets you view the tasks it contains.
• C - Filter the list of data units being displayed by task status by Dataset, User, or Data type.
• D - The list of tasks / data units in your queue. Unassigned tasks are also visible as these can be initiated by all Reviewers.
• E - This column shows which user a task is assigned to. Unassigned tasks are also visible.
• F - Click the Initiate button next to a task to start reviewing. Clicking the Start reviewing button will start the review process from the first task in the queue.

ℹ️

Note

Initiating an unassigned task automatically assigns the task to the Reviewer.

Queue (Annotator & Reviewer, Admin, Team Manager)

• A - Use the search bar to filter the list of data units being displayed, or to look for a particular data unit.
• B - The list of all workflow stages shows how many data units each stage currently contains. Each stage can be selected; the screenshot above shows a list of data units in the Annotate 1 stage.
• C - Clicking the Initiate button will initiate a task. If an annotation stage is selected, it will be an annotation task. If a review stage is selected a review task will be initiated.
• D - Use the Start labeling and Start reviewing buttons to begin labeling or reviewing a task.
• E - Filter the list of data units being displayed by task status by Dataset, User, or Data type.

Assigning and releasing tasks

ℹ️

Note

Only Admins and Task Managers can assign and release tasks.

ℹ️

Note

Tasks can only be assigned to collaborators that have been added to a given workflow stage. Find out more in our section on managing collaborators.

Tasks can be assigned to specific users by selecting them from the list and clicking the Assign button, as shown below. Once a task is assigned, only the assigned user is able to open the task. Alternatively, click the button in the Assigned to column to assign an individual data unit.

Releasing a task is the opposite of assigning a task, and removes any user the task was assigned to. To release any number of tasks, select them from the list and click the Release button located next to the Assign button shown above.

👍

Tip

Unassigned tasks can be initiated by all users.

---

Labels

ℹ️

Note

The Labels tab is only visible to project Admins and Team Managers.

The Labels tab is your gateway to auditing the labels made against all the datasets in your project.

Access to each pane will depend on the user's project role. We quickly summarize the purpose of each tab, and the roles which can access each below.

The labels dashboard features the following tabs:

• Activity: An overview of all data units in the project as well as their task status.
• Data: A complete overview of all tasks in the project, regardless of their progress through the project's workflow
• Instances: The Instances tab lets you use the unique instance identifier to search the project for a specific instance, and jump directly into the editor to confirm the status of an annotation visually.

---

Activity

The Activity tab shows the activity log for all completed data units. This includes information on which users submitted and reviewed each task, as well as how many reviews each task has been through before being completed.

• A : Search, Filter, and Reopen.
• B : Activity columns.

Search, Filter and Reopen

• Use the Search to quickly filter and display only those tasks with file names matching your desired text. Even partial matches will be shown. For example: searching "fly" will return file names containing "flyover" and "flyaround."

• You can Filter the list of data units being displayed by task status.

• The Reopen button allows Administrators and Team Managers to send tasks which are currently Completed or In review back to annotation. Select your target tasks using the checkboxes in the File column to select individual assets, or select the checkbox in the column header to select all tasks, and press the 'Reopen' button to move all selected tasks back to the annotation stage. Tasks reopened in this way will have the status 'Returned,' in the 'Queue' tab. No labels are lost by reopening a task. The 'Reopen' action is only applied to tasks which are both visible (i.e. not filtered out by the file search) and selected.

Activity columns

Reviews

The Reviews column shows a count of how many instances have been reviewed in a given data asset.

Click the number to open a panel which shows the last review action taken on each instance, as well as who originally created the annotation and when. Note that unless the review was done by an 'Expert Reviewer', all reviewed annotations must be either 'Approved' or 'Deleted' before a task can be 'Completed.' Read more about the Expert Review feature here.

---

Data

The Data tab gives a comprehensive overview of all tasks in the project, regardless of their status and progress through the project's workflow.

• A. Search and Filter: The name of the specific data unit or data asset. This is the same as the name in the dataset to which this data asset is a part of. Use the search box to filter the list by file name.
• B. Data columns: The dataset the data asset belongs to.

Search and Filter

• Use the Search to quickly filter and display only those tasks with file names matching your desired text. Even partial matches will be shown. For example: searching "fly" will return file names containing "flyover" and "flyaround."

• You can Filter the list of data units being displayed by task status.

Data columns

👍

Tip

Confused about the difference between image groups and image sequences? See our documentation here to learn about different data types in Encord.

Actions

Clicking View will drop you into the label editor to do a live audit of the annotations in this data asset. The 'Data' tab is only visible to Administrators and Team Managers and so grants great power to view any data asset, however appropriate care must be taken to ensure annotations are not simultaneously edited from the 'Queue' pane by an annotator or reviewer. Encord advises edit actions are NOT taken from the Data tab unless you have received confirmation no one else is concurrently editing the asset.

🚧

Caution

In order to prevent any possible issues of annotator work being overwritten, it's critical that all annotations are done via the workflow's Queue tab, and only the person assigned to the task makes annotations at any given time.

Other possible actions include 'API Details' which show a popup with sample code you can use to get started with our SDK to access this particular data asset, often known as a label row in the SDK. Click Activity log to see a popup with a graphical summary of add / edit / delete actions on this data asset indexed by annotator or ontology class. Click Display logs in the lower right to show all actions in reverse chronological order.

---

Instances

The 'Instances' tab allows Administrators and Team Managers to search within the data to directly find specific instances Recall that an annotation instance correlates to a unique instantiation of a specific ontology class in a data asset. For example, if you have the 'Person' class in your ontology, the first instance of a 'Person' in a given data asset will be indicated in the interface as 'Person (0)', the second as 'Person (1)' and so on. Instances, therefore, can exist in multiple frames of a data asset, and indicate the same object. Use the 'Instances' tab to search for specific instances of objects or classifications using their Identifier.

Instance identifiers are unique at the project scope, and can be found in any of the following ways:

• From inside the label editor, by clicking on a particular instance, and then selecting 'Copy identifier' from the instance action menu.
• From inside exported labels, where they are known as the objectHash or classificationHash as appropriate.
• When uploading labels using the SDK, you may specify your own objectHash or classificationHash.

Once you have an identifier of interest, use the 'Search instance' interface to filter the instances by identifier to quickly find the instance you're interested in. This can be particularly handy when you want to visually confirm an annotation you may not have seen before, but for which you have the identifier.

After locating your instance of interest, click View from the 'Actions' column to jump deeply into the dataset straight to where the instance is first annotated.

---

Performance - Summary

ℹ️

Note

The Performance dashboard is only visible to project Admins and Team Managers.

The 'Summary' tab of the performance dashboard provides an overview of your team's labeling and productivity. You're able to switch between a summary, or detailed view of performance metrics using the toggle found at the top of the 'Performance' tab.

Task actions over time

View the number of tasks in a project that have been approved, rejected, and submitted for review over a given period of time.

• The height of a bar represents the total number of tasks.
• The height of each color within a bar represents the number of approved, rejected, and submitted tasks.

• A: Set the time period you would like to see displayed by selecting a range of dates.
• B: The Hide days without any actions toggle removes all days at which no actions were taken from the view.
• C: Download a CSV file of the data.
• D: Display the data as a bar chart, or a table. While the chart provides a clear visual representation, the table provides exact figures for a more detailed picture of your team's performance.

Instance Label actions over time

View the number of instance label actions in a project that have been approved, rejected, and submitted for review over a given period of time.

• A: Set the time period you would like to see displayed by selecting a range of dates.
• B: Download a CSV file of the data.
• C: Display the data as a bar chart, or a table. While the chart provides a clear visual representation, the table provides exact figures for a more detailed picture of your team's performance.

Within your specified time period, you can choose which dates to display by using the slider located beneath the graph.

Team collaborators

The 'Team collaborators' section the duration of time each project collaborator spend working on a given file.

A. Table entries can be filtered according to dates by clicking the range of dates, and selecting the start and end date of the period you would like to see table entries displayed for.

B. Table entries can be downloaded in CSV format by clicking the Download CSV button.

C. When lots of entries are present they will be split across a number of different pages. The number of table entries per table can be adjusted.

Performance - Details

The 'Details' tab of the performance dashboard gives a more detailed view of your team's labeling and productivity. This section will cover manual QA projects.

👍

Tip

You can specify a range of dates, as well as whether statics should be displayed for labels, or instances. More information on instances and labels can be found here.

---

Submissions chart

The submissions chart displays the number of submitted labels or instances over the specified time period. The chart can be filtered to show submissions for specific annotators or classes.

If you filter on both Annotators and Classes then the resulting chart will show the submission statistics for the selected annotators and for selected labels.

---

Reviews chart

The reviews chart displays the cumulative number of accepted and rejected labels or instances over the specified time period.

---

Annotators' table

The annotators' table displays all the relevant statistics for all annotators in a project. It can be filtered on classes to show annotator statistics only for the selected classes.

• User: The annotator's email.
• Rejection rate: Percentage of their labels or instances that have been rejected in the review process.
• Submitted labels / instances: Number of labels or instances that the annotator has submitted for review
  • Repeated submissions are not counted.
• Accepted labels / instances: Number of labels or instances that the annotator created that passed the review process.
• Rejected labels / instances: Number of labels or instances that the annotator created that we're rejected during the review process. Note that this can be higher than the number of submitted labels / instances since a label or instance can be rejected multiple times during the review process but the submission will only be logged once.
• Total session time: Time spent labeling.

---

Reviewers table

• User: The reviewers email.
• Rejection rate: Percentage of labels or instances that they rejected in the review process.
• Accepted labels / instances: Number of labels or instances that the reviewer accepted.
• Rejected labels / instances: Number of labels or instances that the reviewer rejected.
• Total session time: Time spent reviewing.

---

Objects and classifications table

Each row in the objects and classifications table can be expanded to show statistics attributes.

• Class: The class name.
• Rejection rate: Percentage of labels or instances rejected in the review process.
• Reviewed labels / instances: Number of labels or instances of the class that have gone through the review process.
• Accepted labels / instances: Number of labels or instances of the class that have passed the review process.
• Rejected labels / instances: Number of labels or instances of the class that failed the review process.
• Avg. time to annotate: Average time spent annotating this class.

---

---

Models

ℹ️

Note

The Models tab is only visible to project Admins and Team Managers.

The 'Models' tab is where you attach and manage models for automated labeling to your project. Encord's Apollo allows you to build models for automated data labeling.

Please see our documentation on models here.

---

Export

ℹ️

Note

The Export tab is only visible to project Admins and Team Managers.

Use the 'Export' tab to export your data. Please see our exporting data page to learn how to do this.

---

Workflow

ℹ️

Note

The Workflow tab is only visible to project Admins and Team Managers.

The 'Workflow' tab lets you view the project's workflow, as well as adjust the following:

• Manage project collaborators as well as collaborators at for each stage within the workflow.

• Add, remove, or edit Webhooks at the Annotate and Complete stages of the workflow.

Video Tutorial - Creating webhooks in Workflow projects

Settings

ℹ️

Note

The Settings tab is only visible to project Admins and Team Managers.

The Settings tab allows you to make modifications to your project using the following tabs:

• Options - Copy a project, modify datasets, modify ontology, upload annotation instructions, modify project tags, QA settings.
• Danger zone - Delete your project.

Options

Modify datasets, modify project tags, modify ontology, and upload annotation instructions.

Upload annotation instructions

• Click the Add instructions button to upload instructions for your annotators in PDF format.

• To ensure the best possible results, provide as much detail as possible about what you would like annotated and how precise bounding boxes should be drawn. For example, instead of saying 'person', consider defining what should constitute a person for your annotators - only a full person? A torso? Or should any part of a person in a frame be labeled as a 'person'?

ℹ️

Note

The more specific your annotator instructions, the higher the chances that your annotators will perform well.

• Once uploaded, annotation instructions will be accessible within the Label Editor, or the Workflow tab of your project.

---

Project tags

ℹ️

Note

Tags are created and managed on the Organization level. Once created they can be added to individual projects.

You can add tags to a project if you are part of an organization.

Project tags allow you to:

• Flexibly categorize and group your projects.
• Filter your projects.

Adding and removing tags

You can add tags to your projects in:

• When creating a project.

• In the Settings page of a project. This process is described below.

To add tags to your projects in the Settings page, navigate to the 'Options' tab and click the 'Project tags' drop-down. Here you will see the available tags in your organization. Click on a tag to add it to a project. You can remove a tag from your project by clicking the same tag again, or clicking the x button next to its name.

Filtering projects by tags

You can filter your projects based on the tags they contain. To do so, click on the 'Projects tab' in the Navigation bar, click the 'Filter by tags' drop-down and select one or more project tags. This will result in only projects with the tags being selected being displayed.

---

Edit project ontology

You can view or switch the ontology attached to your project.

ℹ️

Note

Changing the ontology can render existing labels invalid and lead to data inconsistency.

• Click the Switch ontology button to switch the ontology linked to your project. The resulting pop-up allows you to choose an existing ontology from a list, or create a new ontology for this project.

• Click the View ontology button to view the details of the ontology that is attached to the current project.

---

Edit datasets attached to a project

The 'Datasets' section allows you to attach or detach any number of datasets to your project by clicking Manage. You will have to create a new dataset in the 'Datasets' section for it to become available in a project's settings.

Label Editor

Projects created using the 'Workflows' include some minor changes to the label editor.

1. In Annotation as well as Review stages the current workflow status is displayed in the top left corner of the label editor.

2. The Edit labels and Report missing labels buttons have been removed, and this functionality is not supported for projects created using 'Workflows'.

---

Team

To manage project collaborators, select the 'Team' pane in your project Settings.

Here you can invite collaborators to the project, and configure their roles.

1. Add a collaborator

To invite collaborators to your project, click the + Invite collaborators button. This will open a new window where you can enter email addresses of the people you would like to invite.

Once you have entered everybody you want to invite, press the Add button.

2. Add collaborators as a group

ℹ️

Note

To add collaborators as a group, your organization needs to have user groups. Navigate to our documentation on creating user groups for more information.

Collaborators can be added to a project as a group - which can save time as well as ensure that no individual is forgotten.

In the 'Groups' section of the page, click on Manage to make the 'Manage Groups' pop-up appear.

Click the 'Select group' drop-down and pick a group you would like to add as collaborators. After selecting a group, click the 'Select Role' drop-down to assign a role to the group of collaborators. Click Add to add the group.

The group you just added will appear under the 'Added groups' heading. Repeat the process if you'd like to add more groups with different roles to the project.

To delete a group from the project, simply click the button next to the group name.

2. Change collaborator role

A project admin can modify the different roles of collaborators, using the drop-down on the right.

You can assign the following roles to collaborators:

• Annotator: annotators are responsible for labeling. This is the default role for all collaborators.
• Reviewer: for reviewing labeled tasks.
• Annotator & reviewer: a combination of annotator and reviewer.
• Team manager: a team manager can assign tasks to other users, and add collaborators to the project.
• Admin: gives this collaborator full administrative control over this project. Caution: this is an irreversible action.

* Please note that Reviewer and Annotator + Reviewer roles are only available for projects with manual QA.

Please confirm or cancel your selection when making a collaborator a project admin.

---

Danger zone

You can delete your project by going to the 'Danger zone' tab at the bottom of the menu, and clicking the red Delete project button, shown below.

Deleting your project does not delete the datasets in the project, but will delete the project's labels and ontology.

---

Workflow templates

Tired of creating the same workflow over and over again? Try creating templates of your most commonly used workflows - and attach these to any new workflow project during project creation!

Video tutorial - Creating workflow templates

Creating templates

There are two ways of creating a workflow template:

• In the Workflow templates section.
• While creating a workflow project.

Creating templates in the Workflows section

Select Workflows from the Annotate section of the sidebar in the Encord web app, and click the + New workflow button. Previously created templates will be visible here, and can be edited when selected.

Create and configure your workflow template, then click the Create workflow template button when you are done. To learn how to create and configure workflows, see our documentation here.

The Create workflow template button will be greyed out until:

• The nodes in your template have been connected.
• The workflow contains a Complete stage.
• The template has been given a title.

Creating templates during project creation

Templates can be created during the creation of a workflow project.

Click the + New workflow button to create a new workflow. Any workflow created can be saved as a template by clicking the Save as a new template button.

Editing existing templates

Workflow templates can be edited after they have been created. To do so, navigate to Workflow templates in the Annotation section of the sidebar in the Encord web app, and click the name of the template you wish to edit.

Click the Edit template button, as highlighted in the image below.

ℹ️

Note

You are not able to edit the start and annotation nodes.

Once you have made all desired edits, click the Save template changes button.

The Save template changes button will be greyed out until:

• The nodes in your template have been connected.
• The workflow contains a Complete stage.
• The template has been given a title.

Using workflow templates in projects

Workflow templates can be used during project creation. To do so choose the template you would like to use and click the Select button.

For more information, see our documentation on creating and configuring workflow projects.

Customizing workflows

You can customize a workflow template to suit a project's specific needs. After selecting a template during project creation, as outlined in the section above, click the Customize template button to edit the workflow.

Deleting templates

To delete a template:

1. Navigate to Workflows in the Annotate section of the sidebar.
2. Select the template you would like to delete.
3. Click the Settings tab and navigate to 'Danger zone'.
4. Click the Delete template button to delete the template.