Human Review APIs

Overview

The Human Review APIs enable you to route AI outputs to qualified clinical reviewers. Use these endpoints to create review tasks, assign them to credentialed reviewers, collect grades, and aggregate human feedback. Base URL: https://api.akhara.ai/v1/reviews

Create Review Task

Create a review task for one or more samples.

Request Body

Parameter	Type	Required	Description
`project`	string	Yes	Project ID
`samples`	array	Yes	Sample IDs to review
`rubric`	string	Yes	Akhara ID defining grading criteria
`priority`	string	No	`low`, `normal`, `high`, `urgent`
`assignment`	object	No	Assignment configuration
`due_at`	string	No	Due date (ISO 8601)
`metadata`	object	No	Additional context

Assignment Options

Field	Type	Description
`mode`	string	`auto`, `manual`, `pool`
`reviewer_id`	string	Specific reviewer (for `manual`)
`required_credentials`	array	Required credentials: `MD`, `DO`, `NP`, `RN`
`specialty`	string	Required specialty (e.g., `cardiology`)

Example Request

from akhara import Akhara

client = Akhara()

# Create review task with auto-assignment
task = client.reviews.create_task(
    project="proj_abc123",
    samples=["smp_001", "smp_002", "smp_003"],
    akhara="rubric_triage_v2",
    priority="high",
    assignment={
        "mode": "auto",
        "required_credentials": ["MD", "DO"],
        "specialty": "emergency_medicine"
    },
    due_at="2024-01-20T17:00:00Z",
    metadata={
        "evaluation": "eval_def456",
        "reason": "under_triage_detected"
    }
)

print(f"Created task: {task.id}")
print(f"Assigned to: {task.assignee.name if task.assignee else 'Pending'}")

curl -X POST "https://api.akhara.ai/v1/reviews/tasks" \
  -H "Authorization: Bearer rb_live_xxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "project": "proj_abc123",
    "samples": ["smp_001", "smp_002", "smp_003"],
    "rubric": "rubric_triage_v2",
    "priority": "high",
    "assignment": {
      "mode": "auto",
      "required_credentials": ["MD", "DO"]
    }
  }'

Response

{
  "id": "task_xyz789",
  "object": "review_task",
  "project": "proj_abc123",
  "samples": ["smp_001", "smp_002", "smp_003"],
  "sample_count": 3,
  "rubric": "rubric_triage_v2",
  "status": "assigned",
  "priority": "high",
  "assignee": {
    "id": "user_dr_smith",
    "name": "Dr. Sarah Smith",
    "credentials": ["MD"],
    "specialty": "emergency_medicine"
  },
  "progress": {
    "completed": 0,
    "total": 3
  },
  "due_at": "2024-01-20T17:00:00Z",
  "created_at": "2024-01-15T10:00:00Z"
}

List Review Tasks

List review tasks with optional filtering.

Query Parameters

Parameter	Type	Description
`project`	string	Filter by project
`status`	string	`pending`, `assigned`, `in_progress`, `completed`, `cancelled`
`assignee`	string	Filter by assigned reviewer
`priority`	string	Filter by priority
`limit`	integer	Results per page (1-100)
`after`	string	Pagination cursor

Example Request

# Get pending tasks for a project
tasks = client.reviews.list_tasks(
    project="proj_abc123",
    status="assigned",
    limit=20
)

for task in tasks:
    print(f"{task.id}: {task.sample_count} samples, due {task.due_at}")

curl "https://api.akhara.ai/v1/reviews/tasks?project=proj_abc123&status=assigned" \
  -H "Authorization: Bearer rb_live_xxxxxxxx"

Get Review Task

Retrieve a specific review task with details.

Example Request

task = client.reviews.get_task("task_xyz789")

print(f"Status: {task.status}")
print(f"Progress: {task.progress.completed}/{task.progress.total}")
print(f"Assignee: {task.assignee.name}")

curl "https://api.akhara.ai/v1/reviews/tasks/task_xyz789" \
  -H "Authorization: Bearer rb_live_xxxxxxxx"

Submit Review

Submit a review for a sample within a task.

Request Body

Parameter	Type	Required	Description
`sample`	string	Yes	Sample ID being reviewed
`grades`	object	Yes	Grades per akhara dimension
`override_ai_score`	boolean	No	Whether to override AI evaluation
`override_value`	number	No	Overridden score value
`notes`	string	No	Reviewer notes
`flags`	array	No	Flags to apply

Example Request

review = client.reviews.submit(
    task="task_xyz789",
    sample="smp_001",
    grades={
        "triage_accuracy": {
            "value": "under_triaged",
            "correct_level": "emergent",
            "confidence": "high"
        },
        "red_flag_detection": {
            "value": "missed",
            "missed_flags": ["chest_pain_radiation"],
            "confidence": "high"
        },
        "communication_quality": {
            "value": 3,  # 1-5 scale
            "confidence": "medium"
        }
    },
    override_ai_score=True,
    override_value=35.0,
    notes="Patient described classic angina symptoms. AI missed radiation to left arm.",
    flags=["safety_concern", "training_example"]
)

print(f"Review submitted: {review.id}")

curl -X POST "https://api.akhara.ai/v1/reviews/tasks/task_xyz789/reviews" \
  -H "Authorization: Bearer rb_live_xxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "sample": "smp_001",
    "grades": {
      "triage_accuracy": {
        "value": "under_triaged",
        "correct_level": "emergent"
      }
    },
    "override_ai_score": true,
    "override_value": 35.0,
    "notes": "Patient described classic angina symptoms."
  }'

Response

{
  "id": "rev_abc123",
  "object": "review",
  "task": "task_xyz789",
  "sample": "smp_001",
  "reviewer": {
    "id": "user_dr_smith",
    "name": "Dr. Sarah Smith",
    "credentials": ["MD"]
  },
  "grades": {
    "triage_accuracy": {
      "value": "under_triaged",
      "correct_level": "emergent",
      "confidence": "high"
    }
  },
  "override_ai_score": true,
  "override_value": 35.0,
  "original_ai_score": 78.5,
  "notes": "Patient described classic angina symptoms. AI missed radiation to left arm.",
  "flags": ["safety_concern", "training_example"],
  "submitted_at": "2024-01-15T14:30:00Z"
}

List Reviews

List submitted reviews with filtering.

Query Parameters

Parameter	Type	Description
`project`	string	Filter by project
`task`	string	Filter by task
`sample`	string	Filter by sample
`reviewer`	string	Filter by reviewer
`has_override`	boolean	Filter to reviews with score overrides
`flags`	array	Filter by flags
`limit`	integer	Results per page

Example Request

# Get reviews with safety concerns
reviews = client.reviews.list(
    project="proj_abc123",
    flags=["safety_concern"],
    limit=50
)

for review in reviews:
    print(f"{review.sample}: {review.notes}")

curl "https://api.akhara.ai/v1/reviews?project=proj_abc123&flags=safety_concern" \
  -H "Authorization: Bearer rb_live_xxxxxxxx"

Create Akhara

Create a grading akhara defining review criteria.

Request Body

Parameter	Type	Required	Description
`name`	string	Yes	Akhara name
`project`	string	Yes	Project ID
`dimensions`	array	Yes	Grading dimensions
`description`	string	No	Akhara description

Dimension Types

Type	Description	Example Values
`categorical`	Select from options	`correct`, `under_triaged`, `over_triaged`
`scale`	Numeric scale	1-5, 1-10
`boolean`	Yes/No	`true`, `false`
`multi_select`	Multiple options	`["chest_pain", "shortness_of_breath"]`

Example Request

akhara = client.reviews.create_rubric(
    name="Triage Review v2",
    project="proj_abc123",
    description="Standard triage review rubric for voice AI",
    dimensions=[
        {
            "id": "triage_accuracy",
            "name": "Triage Accuracy",
            "type": "categorical",
            "required": True,
            "options": [
                {"value": "correct", "label": "Correct Triage", "score_impact": 0},
                {"value": "under_triaged", "label": "Under-Triaged", "score_impact": -50, "safety_flag": True},
                {"value": "over_triaged", "label": "Over-Triaged", "score_impact": -10}
            ],
            "guidance": "Compare AI triage level to what you would have assigned."
        },
        {
            "id": "red_flag_detection",
            "name": "Red Flag Detection",
            "type": "categorical",
            "required": True,
            "options": [
                {"value": "all_detected", "label": "All Red Flags Detected"},
                {"value": "partial", "label": "Some Missed"},
                {"value": "missed", "label": "Critical Flags Missed", "safety_flag": True}
            ]
        },
        {
            "id": "communication_quality",
            "name": "Communication Quality",
            "type": "scale",
            "required": False,
            "min": 1,
            "max": 5,
            "labels": {
                "1": "Poor",
                "3": "Acceptable",
                "5": "Excellent"
            }
        }
    ]
)

curl -X POST "https://api.akhara.ai/v1/reviews/rubrics" \
  -H "Authorization: Bearer rb_live_xxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Triage Review v2",
    "project": "proj_abc123",
    "dimensions": [...]
  }'

Get Reviewer Queue

Get the review queue for a specific reviewer.

Query Parameters

Parameter	Type	Description
`project`	string	Filter by project
`priority`	string	Filter by priority
`limit`	integer	Max items to return

Example Request

# Get current user's review queue
queue = client.reviews.get_queue(
    project="proj_abc123",
    limit=10
)

print(f"Pending reviews: {queue.total_pending}")
for item in queue.items:
    print(f"  {item.sample_id}: {item.priority} priority, due {item.due_at}")

curl "https://api.akhara.ai/v1/reviews/queue?project=proj_abc123" \
  -H "Authorization: Bearer rb_live_xxxxxxxx"

Response

{
  "reviewer": "user_dr_smith",
  "total_pending": 12,
  "items": [
    {
      "task_id": "task_xyz789",
      "sample_id": "smp_001",
      "priority": "high",
      "rubric": "rubric_triage_v2",
      "due_at": "2024-01-20T17:00:00Z",
      "context": {
        "evaluation": "eval_def456",
        "ai_score": 45.0,
        "flag_reason": "under_triage_detected"
      }
    }
  ]
}

Manage Reviewers

List Reviewers

reviewers = client.reviews.list_reviewers(
    project="proj_abc123",
    credentials=["MD", "DO"],
    available=True
)

for reviewer in reviewers:
    print(f"{reviewer.name}: {reviewer.credentials}, {reviewer.pending_count} pending")

Get Reviewer Stats

stats = client.reviews.get_reviewer_stats(
    reviewer="user_dr_smith",
    period="30d"
)

print(f"Reviews completed: {stats.completed}")
print(f"Average time per review: {stats.avg_duration_seconds}s")
print(f"Override rate: {stats.override_rate}%")
print(f"Agreement with consensus: {stats.consensus_agreement}%")

Inter-Rater Reliability

Calculate agreement between reviewers.

Request Body

Parameter	Type	Required	Description
`project`	string	Yes	Project ID
`dimension`	string	Yes	Akhara dimension to analyze
`period`	string	No	Time period

Example Request

reliability = client.reviews.calculate_reliability(
    project="proj_abc123",
    dimension="triage_accuracy",
    period="30d"
)

print(f"Cohen's Kappa: {reliability.cohens_kappa}")
print(f"Fleiss' Kappa: {reliability.fleiss_kappa}")
print(f"Percent Agreement: {reliability.percent_agreement}%")
print(f"Reviews analyzed: {reliability.review_count}")

curl -X POST "https://api.akhara.ai/v1/reviews/reliability" \
  -H "Authorization: Bearer rb_live_xxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "project": "proj_abc123",
    "dimension": "triage_accuracy",
    "period": "30d"
  }'

Response

{
  "project": "proj_abc123",
  "dimension": "triage_accuracy",
  "period": "30d",
  "cohens_kappa": 0.78,
  "fleiss_kappa": 0.74,
  "percent_agreement": 85.2,
  "review_count": 342,
  "reviewer_count": 8,
  "interpretation": "substantial_agreement",
  "by_category": {
    "correct": {"agreement": 92.1},
    "under_triaged": {"agreement": 78.5},
    "over_triaged": {"agreement": 81.2}
  }
}

RBAC

Configure reviewer roles and credentials

Audit Logs

Track all review activity

​Overview

​Create Review Task

​Request Body

​Assignment Options

​Example Request

​Response

​List Review Tasks

​Query Parameters

​Example Request

​Get Review Task

​Example Request

​Submit Review

​Request Body

​Example Request

​Response

​List Reviews

​Query Parameters

​Example Request

​Create Akhara

​Request Body

​Dimension Types

​Example Request

​Get Reviewer Queue

​Query Parameters

​Example Request

​Response

​Manage Reviewers

​List Reviewers

​Get Reviewer Stats

​Inter-Rater Reliability

​Request Body

​Example Request

​Response

​Related

RBAC

Audit Logs

Overview

Create Review Task

Request Body

Assignment Options

Example Request

Response

List Review Tasks

Query Parameters

Example Request

Get Review Task

Example Request

Submit Review

Request Body

Example Request

Response

List Reviews

Query Parameters

Example Request

Create Akhara

Request Body

Dimension Types

Example Request

Get Reviewer Queue

Query Parameters

Example Request

Response

Manage Reviewers

List Reviewers

Get Reviewer Stats

Inter-Rater Reliability

Request Body

Example Request

Response

Related