Highlighter Documentation

Human Assessment Steps

Learn how to set up human assessment steps where contributors manually evaluate and process data in your workflows.

Overview

Human assessment steps involve manual review and processing by human experts. These steps are essential for tasks requiring human judgment, quality assurance, edge case handling, and training data creation.

Creating Human Assessment Steps

Prerequisites

  • Defined review criteria and guidelines
  • Trained reviewers with appropriate expertise
  • Clear workflow requirements and expectations
  • Access to workflow configuration tools

Basic Setup

When creating a human assessment step, you configure the following:

  1. Queue Name: Descriptive name for the assessment queue
  2. Step Type: Select "Human Assessment"
  3. Contributor Assignment: Assign team members who can work on tasks in this step
  4. Previous Step (optional): Link this step to a previous step in your workflow
  5. Queue Settings:
    • Queue Capacity (optional): Maximum number of tasks in the queue
    • Batch Size (optional): Number of items to display at once
    • Consumable: Whether tasks are consumed when completed
    • Lockable: Whether tasks can be locked for exclusive access

Note: Contributors work across all workflow step types using a unified interface. See Assigning Contributors to Workflow Steps for detailed assignment instructions.

Shadow Entities

By default, an assessor sees every entity carried over from earlier assessment stages. For a step that only cares about a subset of those entities, you can shadow the irrelevant ones so they are hidden on the canvas (but kept on the case for later use). For example, a "vehicle damage" step can hide the trees and pedestrians detected by an upstream "object detection" step while still showing the vehicles.

In the human assessment step's edit form, the Shadow Entities section controls this:

  1. Tick Enable Shadow Entities to turn the feature on for the step.
  2. Under Key Attributes, add one or more rules that describe which entities stay visible. Each rule is:
    • An entity attribute (required) — choose the attribute that identifies relevant entities (for example, "Species").
    • An optional value. Leave it as Any value to match any entity that has the attribute, or pick a specific value (for example, "Cat") to match only entities with that value.
  3. Use + Add Key Attribute to add more rules, and the × button to remove one.
  4. Save the step.

Rules combine with OR logic: an entity stays visible if it matches at least one rule. Any entity that matches no rule is shadowed. If you enable the feature but add no rules, every carried-over entity is shadowed.

The step details page shows a read-only summary of the configured key attribute rules.

Assessors can reveal shadowed entities by hovering, promote them back into the active set by clicking, and manually shadow entities themselves — see Work With Shadow Entities.

Assigning Contributors

How to Assign Contributors

Contributors are assigned when creating or editing a human assessment step:

  1. Navigate to your workflow and create or edit a human assessment step
  2. In the Assignees field, select one or more users from your account
  3. Assigned contributors will be able to work on tasks for this step
  4. Save the step to apply the assignments

Monitoring Assigned Contributors

On the step details page, you can view information about assigned contributors:

  • Name and Email: Contributor contact information
  • Checked Out: Number of tasks currently being worked on
  • Submissions: Total number of completed submissions
  • Assigned Orders: Number of workflow orders assigned to the contributor
  • Available Tasks: Number of tasks available to work on

For step-by-step instructions on assigning contributors, see Assigning Contributors to Workflow Steps.

Using Human Assessment Steps

Contributor Workflow

Contributors assigned to human assessment steps follow this workflow:

  1. Task Access
  • Log into the Highlighter platform
  • Navigate to assigned tasks queue
  • Select tasks based on priority and deadline
  • Begin review process following guidelines
  1. Review Process
  • Data Examination: Thorough review of input data
  • Annotation/Classification: Apply labels or classifications
  • Quality Assessment: Evaluate data quality and completeness
  • Decision Making: Make required judgments or decisions
  1. Task Completion
  • Result Recording: Document findings and decisions
  • Quality Verification: Double-check work for accuracy
  • Submission: Submit completed reviews
  • Notes Documentation: Add relevant comments or observations

Step Management Features

Available Actions

On the step details page, you can:

  • View Task Statistics: See total, pending, and succeeded task counts
  • Unlock Remaining Tasks: Release locked tasks back to the queue
  • View Filters: See any filters applied to the queue
  • Monitor Contributors: Track who is working on tasks and their progress

Workflow Integration

Human assessment steps can be:

  • Connected in Sequence: Link steps together to create multi-step workflows
  • Combined with Machine Steps: Use machine assessment before or after human review
  • Filtered: Apply filters to determine which cases flow to this step

Best Practices

Effective Step Configuration

  • Use descriptive queue names that clearly indicate the purpose of the step
  • Set appropriate batch sizes for your team's review capacity
  • Configure queue capacity limits to prevent overwhelming contributors
  • Link steps in a logical sequence that matches your workflow process

Contributor Management

  • Assign contributors with relevant expertise to each step
  • Monitor the contributor metrics on the step details page
  • Balance workload across team members
  • Provide clear instructions and training for the assessment tasks

Workflow Design

  • Start with simple single-step workflows before adding complexity
  • Use filters to route appropriate cases to each step
  • Combine machine and human assessment steps where beneficial
  • Monitor task statistics to identify bottlenecks

Troubleshooting

Common Issues

Contributors Cannot Access Tasks

  • Verify the contributor is assigned to the step in the Assignees field
  • Check that there are tasks available in the queue (check the Pending count)
  • Ensure the workflow step is properly configured and linked to previous steps

Tasks Not Flowing to the Step

  • Check that the previous step is correctly linked
  • Verify any filters are configured correctly
  • Ensure tasks exist in the workflow and are being created

High Task Backlog

  • Check the Pending task count on the step details page
  • Consider adding more contributors to the step
  • Review if batch size or queue capacity settings are appropriate
  • Use the "Unlock Remaining" button if tasks are stuck in a locked state

Step Configuration Issues

  • Ensure the queue name is set
  • Verify contributors are properly assigned
  • Check that the step has an associated image queue

Getting Help

For additional assistance with human assessment steps:

  • Review the step details page for current status and metrics
  • Check the task statistics to understand workflow health
  • Consult your workflow administrator
  • Contact Highlighter support team
← Troubleshooting Annotation Tool
How To Create and Edit a Machine Agent →