Highlighter Documentation

Human Assessment Stages

Learn how to set up human assessment stages where expert reviewers manually evaluate and process data in your workflows.

Overview

Human assessment stages involve manual review and processing by human experts. These stages are essential for tasks requiring human judgment, quality assurance, edge case handling, and training data creation.

Creating Human Assessment Stages

Prerequisites

  • Defined review criteria and guidelines
  • Trained reviewers with appropriate expertise
  • Clear workflow requirements and expectations
  • Access to workflow configuration tools

Basic Setup

  1. Stage Configuration

    • Stage Name: Descriptive name for the assessment stage
    • Stage Type: Select "Human Assessment"
    • Review Type: Classification, annotation, validation, or quality control
    • Input Requirements: Define data types and formats expected

  2. Reviewer Assignment

    • Reviewer Groups: Assign specific groups or individuals
    • Expertise Requirements: Match reviewers to task complexity
    • Workload Distribution: Configure how tasks are distributed
    • Backup Reviewers: Assign secondary reviewers for coverage

  3. Review Parameters

    • Review Guidelines: Detailed instructions for reviewers
    • Quality Standards: Define acceptable quality levels
    • Time Limits: Set reasonable completion timeframes
    • Escalation Rules: Handle complex or problematic cases

Creating Human File Assessment Stages

Stage Configuration

  1. File Processing Setup

    • File Types: Specify supported file formats
    • Batch Size: Number of files per review session
    • Priority Rules: Determine file processing order
    • Routing Logic: Direct files to appropriate reviewers

  2. Review Interface

    • Annotation Tools: Configure available labelling tools
    • Review Templates: Set up standardized review forms
    • Decision Options: Define possible review outcomes
    • Comment System: Enable reviewer feedback and notes

Assigning to Inspectors

  1. Inspector Selection

    • Skill Matching: Match inspectors to file types and complexity
    • Workload Balancing: Distribute work evenly across team
    • Availability Management: Consider inspector schedules and capacity
    • Performance History: Use past performance to guide assignments

  2. Assignment Methods

    • Manual Assignment: Direct assignment by supervisors
    • Automatic Distribution: Algorithm-based assignment
    • Self-Service: Inspectors select from available work
    • Hybrid Approach: Combination of manual and automatic methods

Using Human Assessment Stages

Reviewer Workflow

  1. Task Access

    • Log into the Highlighter platform
    • Navigate to assigned tasks queue
    • Select tasks based on priority and deadline
    • Begin review process following guidelines

  2. Review Process

    • Data Examination: Thorough review of input data
    • Annotation/Classification: Apply labels or classifications
    • Quality Assessment: Evaluate data quality and completeness
    • Decision Making: Make required judgments or decisions

  3. Task Completion

    • Result Recording: Document findings and decisions
    • Quality Verification: Double-check work for accuracy
    • Submission: Submit completed reviews
    • Notes Documentation: Add relevant comments or observations

Quality Control

  • Peer Review: Secondary review by another expert
  • Supervisor Approval: Management review of critical decisions
  • Consensus Building: Multiple reviewers for complex cases
  • Audit Trails: Complete documentation of review decisions

Advanced Features

Multi-Stage Reviews

  • Sequential Review: Multiple reviewers in sequence
  • Parallel Review: Independent reviews compared for consensus
  • Hierarchical Review: Junior reviewers supervised by senior experts
  • Specialized Stages: Different experts for different aspects

Workflow Integration

  • Conditional Routing: Route based on review results
  • Machine-Human Hybrid: Combine AI and human assessment
  • Feedback Loops: Use human reviews to improve machine learning
  • Exception Handling: Special processing for unusual cases

Performance Management

  • Reviewer Metrics: Track accuracy, speed, and consistency
  • Training Programs: Ongoing education and skill development
  • Calibration Sessions: Ensure consistency across reviewers
  • Performance Reviews: Regular evaluation of reviewer performance

Best Practices

Review Quality

  • Provide clear, detailed guidelines
  • Regular training and calibration sessions
  • Consistent application of standards
  • Regular quality audits and feedback

Workflow Efficiency

  • Optimize batch sizes for reviewer productivity
  • Minimize context switching between different types of tasks
  • Provide efficient tools and interfaces
  • Regular workflow optimization based on feedback

Reviewer Management

  • Match reviewer expertise to task complexity
  • Provide adequate training and support
  • Maintain reasonable workloads and deadlines
  • Recognize and reward good performance

Integration with Machine Learning

  • Use human reviews to create training data
  • Feedback loops to improve AI performance
  • Human oversight of machine decisions
  • Continuous learning from human expertise

Troubleshooting

Common Issues

Inconsistent Reviews

  • Review and update guidelines
  • Conduct calibration sessions
  • Provide additional training
  • Implement peer review processes

Review Bottlenecks

  • Analyze workload distribution
  • Add additional reviewers if needed
  • Optimize review processes
  • Identify and address inefficiencies

Quality Issues

  • Review reviewer performance metrics
  • Provide targeted training
  • Adjust quality standards if needed
  • Implement additional quality controls

Performance Optimization

  • Monitor review times and accuracy
  • Optimize review interfaces and tools
  • Regular feedback from reviewers
  • Continuous improvement of processes

Support and Training

Reviewer Onboarding

  • Comprehensive training programs
  • Hands-on practice sessions
  • Mentorship with experienced reviewers
  • Regular competency assessments

Ongoing Support

  • Help desk for technical issues
  • Guidelines and documentation updates
  • Regular team meetings and feedback sessions
  • Performance coaching when needed

For additional assistance:

  • Contact reviewer support team
  • Consult workflow documentation
  • Participate in training sessions
← Troubleshooting Annotation Tool
How To Create and Edit a Machine Agent →