Highlighter Documentation

Machine Assessment Steps

Learn how to set up and configure machine assessment steps that use AI agents to automatically process and evaluate data.

Overview

Machine assessment steps are automated workflow components that use trained AI agents to process data without human intervention. These steps can perform tasks like classification, detection, segmentation, and quality assessment at scale.

Creating Machine Assessment Steps

Prerequisites

  • A trained AI agent or capability
  • Access to workflow configuration
  • Appropriate data sources and formats
  • Defined success criteria and quality thresholds

Basic Setup

  1. Step Configuration

    • Step Name: Descriptive identifier for the assessment step
    • Step Type: Select "Machine Assessment"
    • Input Requirements: Define expected data formats and requirements
    • Output Specifications: Configure output format and destinations

  2. Agent Assignment

    • Select Agent: Choose the trained AI agent for this step
    • Agent Version: Specify which version to use
    • Fallback Options: Configure backup agents if primary fails
    • Performance Monitoring: Set up performance tracking

  3. Processing Parameters

    • Batch Size: Number of items processed simultaneously
    • Processing Timeout: Maximum time allowed per item
    • Resource Allocation: CPU, GPU, and memory requirements
    • Scaling Rules: Auto-scaling based on workload

Configuration Options

Quality Thresholds

  • Confidence Thresholds: Minimum confidence levels for automated decisions
  • Quality Gates: Criteria that must be met to proceed
  • Error Handling: Actions to take when quality thresholds aren't met
  • Escalation Rules: When to involve human reviewers

Workflow Integration

  • Input Filters: Pre-processing filters and data validation
  • Output Routing: Direct results to appropriate next steps
  • Conditional Logic: Branch workflow based on results
  • Error Recovery: Handle processing failures gracefully

Performance Optimization

  • Parallel Processing: Configure concurrent processing streams
  • Resource Management: Optimize resource utilization
  • Caching: Cache frequently used models and data
  • Load Balancing: Distribute work across available resources

Adding Tasks to Machine Assessment Steps

Task Types

  • Classification Tasks: Categorize input data
  • Detection Tasks: Identify and locate objects
  • Segmentation Tasks: Precise boundary detection
  • Quality Assessment: Evaluate data quality and completeness

Task Configuration

  1. Task Definition

    • Task Name: Clear identifier for the task
    • Input Specification: Define what data the task receives
    • Processing Logic: Configure the AI processing pipeline
    • Output Format: Specify result structure and format

  2. Quality Controls

    • Validation Rules: Automated checks on task results
    • Confidence Requirements: Minimum confidence for acceptance
    • Review Triggers: Conditions that require human review
    • Approval Workflows: Multi-step approval processes

  3. Performance Monitoring

    • Metrics Collection: Track accuracy, speed, and throughput
    • Alert Conditions: Set up notifications for issues
    • Reporting: Generate performance and quality reports
    • Continuous Improvement: Use metrics to refine processes

Advanced Features

Multi-Agent Workflows

  • Agent Chains: Sequential processing with multiple agents
  • Consensus Systems: Multiple agents voting on results
  • Specialized Agents: Different agents for different data types
  • Agent Orchestration: Complex routing and decision logic

Dynamic Configuration

  • Adaptive Thresholds: Adjust quality gates based on performance
  • Learning Integration: Incorporate new training data automatically
  • A/B Testing: Compare different agent configurations
  • Rollback Mechanisms: Revert to previous configurations if needed

Integration Capabilities

  • API Endpoints: Programmatic access to step configuration
  • Webhook Notifications: Real-time status updates
  • External Systems: Integration with business applications
  • Data Pipeline: Seamless data flow between systems

Monitoring and Optimization

Performance Metrics

  • Processing Speed: Items processed per unit time
  • Accuracy Rates: Percentage of correct automated decisions
  • Resource Utilization: CPU, memory, and GPU usage
  • Queue Lengths: Backlog of items waiting for processing

Quality Assurance

  • Automated Testing: Regular validation of agent performance
  • Sample Review: Human review of automated decisions
  • Feedback Loops: Incorporate review results into training
  • Version Control: Track agent versions and performance changes

Troubleshooting

  • Error Tracking: Monitor and categorize processing errors
  • Performance Degradation: Identify and address performance issues
  • Resource Bottlenecks: Optimize resource allocation
  • Data Quality Issues: Handle problematic input data

Correcting a Task's Status

An account admin can change the status of a task from the step's page, for example to send a failed task back to be picked up again. This works on the Tasks panel of every step type, not only Machine Assessment Steps.

  1. Open the step's page and find the task in the Tasks panel. The panel opens on the All tab; use the Pending, Running, Success or Failed tab to narrow the list.
  2. Click the task's status label in the first column. Labels you can change show Change task status when you hover over them.
  3. In the Change task status dialog, choose Pending, Running, Failed or Success from Status.
  4. Click Save status. The page confirms with Task#<id> status updated to <status>. and returns you to the same tab and page of the list.

What each change does:

  • Pending clears the task's previous lease, so an agent or assessor can pick it up again.
  • Success advances the workflow, as it does when a task completes normally.
  • Any change may send notifications and start or stop agents.

Two kinds of task cannot be changed, and their status label is not clickable:

  • A task that is already success. This keeps completed workflow state intact.
  • A running task that an agent or assessor currently holds a lease on. Wait for the lease to expire or for the task to finish.

If the task becomes ineligible between opening the dialog and saving, nothing changes and the page reports Successful tasks and tasks with an active running lease cannot be changed.

Users who are not admins see the status labels but cannot click them.

Best Practices

Step Design

  • Start with simple, well-defined tasks
  • Build in appropriate quality controls
  • Plan for edge cases and error conditions
  • Design for scalability and maintenance

Agent Management

  • Use version control for agent deployments
  • Maintain performance baselines and benchmarks
  • Regular retraining with new data
  • Monitor for concept drift and model degradation

Workflow Integration

  • Ensure smooth data flow between steps
  • Implement comprehensive logging and monitoring
  • Plan for failure scenarios and recovery
  • Document configuration decisions and rationale

Performance Optimization

  • Optimize batch sizes for your specific use case
  • Monitor resource usage and adjust allocation
  • Implement caching where appropriate
  • Regular performance tuning and optimization

Getting Help

For technical support:

  • Review agent performance logs
  • Check system resource utilization
  • Consult the troubleshooting guide
  • Contact technical support team
← Troubleshooting Annotation Tool
How To Create and Edit a Machine Agent →