Human in the Loop - Documentation

Getting Started

Human in the Loop is a simple API service that allows your AI agents to pause for human judgment when needed. When your agent encounters uncertainty or needs approval for a critical action, it can create a review request and wait for human input before continuing.

Basic Flow

1. Your AI agent calls our API with the decision context
2. We return a review URL that you can share with a human reviewer
3. The reviewer makes a decision through our interface
4. We notify your system via webhook so the agent can continue

API Reference

Create Review Request

Endpoint: POST https://human-in-the-loop.derrick-i.workers.dev/create

{
  "message": "AI agent wants to delete 500 records from production database. Confidence: 67%",
  "webhook": "https://your-ai-system.com/webhooks/review-complete",
  "status": "pending",
  "actions": ["approve", "reject", "investigate"],
  "autoAction": "approve",
  "metadata": {
    "agent_id": "data-cleanup-agent",
    "confidence_score": "0.67",
    "affected_records": "500"
  },
  "redirectUrl": "https://your-dashboard.com/reviews",
  "redirectText": "Back to Dashboard",
  "messageType": "markdown"
}

Request Fields

message
The content to be reviewed. Supports Markdown and HTML formatting. Required
webhook
URL that will receive the decision data after human review. Required
status
Initial status of the review request (typically "pending"). Required
actions
Custom action buttons for the reviewer. Defaults to ["accept", "reject"] if not provided. Optional
autoAction
If present and matches one of the actions, this action will be executed immediately (webhook called, status updated), but the review UI will still be available for audit/history. Optional
metadata
Additional context data displayed to the reviewer (agent ID, confidence scores, etc.). Optional
redirectUrl
URL to redirect the user to after they take action. Optional
redirectText
Button text for the redirect button (defaults to "Continue"). Optional
messageType
Specify "html" or "markdown" for content formatting (defaults to "markdown"). Optional

Response

Returns a JSON object with the unique review URL:

{
  "url": "https://human-in-the-loop.derrick-i.workers.dev/7ba8e7d8c3262780"
}

Webhook Notification

After the human reviewer makes a decision, your webhook endpoint will receive a POST request:

{
  "message": "AI agent wants to delete 500 records...",
  "webhook": "https://your-ai-system.com/webhooks/review-complete",
  "status": "approve",
  "actions": ["approve", "reject", "investigate"],
  "metadata": {
    "agent_id": "data-cleanup-agent",
    "confidence_score": "0.67",
    "affected_records": "500"
  }
}

The status field will contain the action selected by the reviewer.

Integration Examples

Data Cleanup Agent

Agent pauses before deleting records when confidence is low.

if confidence < 0.7: review_url = create_review({ "message": f"Delete {count} records?", "actions": ["approve", "reject", "review_manually"] }) await_human_decision(review_url)

Customer Service Bot

Escalate complex complaints to human agents.

if sentiment_score < -0.8: review_url = create_review({ "message": f"Escalate complaint: {summary}", "actions": ["escalate", "handle_automatically"] }) await_human_decision(review_url)

Content Publishing AI

Get approval before publishing generated content.

if content_risk_score > 0.3: review_url = create_review({ "message": generated_content, "messageType": "html", "actions": ["publish", "edit", "reject"] }) await_human_decision(review_url)

Trading Algorithm

Require approval for large transactions.

if trade_amount > 10000: review_url = create_review({ "message": f"Execute ${trade_amount} trade?", "actions": ["execute", "cancel", "reduce_amount"] }) await_human_decision(review_url)

Best Practices

When to Use Human-in-the-Loop

Consider adding human oversight when:

AI confidence scores are below your threshold
Actions have significant business impact (money, data, reputation)
Dealing with sensitive or controversial content
Regulatory compliance requires human approval
Edge cases that weren't covered in training

Message Formatting

For better review experiences:

Use clear, concise language explaining the situation
Include relevant context (confidence scores, impact, risks)
Use Markdown formatting for better readability
Add metadata for additional context without cluttering the message

Action Design

When defining custom actions:

Use descriptive action names ("approve_with_changes" vs "maybe")
Limit to 2-4 actions to avoid decision paralysis
Consider the reviewer's workflow and next steps
Use consistent action naming across your system

Webhook Handling

For reliable integration:

Implement proper error handling and retry logic
Use HTTPS URLs for security
Consider adding authentication tokens to webhook URLs
Log webhook responses for debugging
Handle timeouts gracefully