> ## Documentation Index
> Fetch the complete documentation index at: https://docs.akhara.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Company name is Akhara AI (never Rubric AI). Keep lowercase rubric/rubrics only when meaning grading criteria.
> Expert Review (docs path talent/) is enterprise BYO experts for audit and review: invite customer specialists; do not pitch Akhara recruiting or a public expert career portal. RLHF and domain writing are secondary work types.
> Prefer concrete API examples against public hosts: Environments eval API https://agi.akhara.ai, Control plane PDP https://api.akhara.dev, Evaluation https://app.akhara.ai / https://api.akhara.ai, Expert Review portal https://talent.akhara.ai.
> Do not invent a public hostname for private orchestrators or env API internals.
> Do not confuse control-plane latches with Environments confirmation latches.
> Environments SDK/API examples: curl against https://agi.akhara.ai. Evaluation SDK: from akhara import Akhara and AKHARA_API_KEY.
> Start with /llms.txt for the docs index and OpenAPI links; fetch individual pages as .md exports.

# Human Review APIs

> Route AI outputs to clinical experts for review, grading, and feedback.

## Overview

The Human Review APIs enable you to route AI outputs to qualified clinical reviewers. Use these endpoints to create review tasks, assign them to credentialed reviewers, collect grades, and aggregate human feedback.

**Base URL:** `https://api.akhara.ai/v1/reviews`

***

## Create Review Task

Create a review task for one or more samples.

<EndpointBlock method="POST" path="/v1/reviews/tasks" />

### Request Body

| Parameter    | Type   | Required | Description                         |
| ------------ | ------ | -------- | ----------------------------------- |
| `project`    | string | Yes      | Project ID                          |
| `samples`    | array  | Yes      | Sample IDs to review                |
| `rubric`     | string | Yes      | Akhara ID defining grading criteria |
| `priority`   | string | No       | `low`, `normal`, `high`, `urgent`   |
| `assignment` | object | No       | Assignment configuration            |
| `due_at`     | string | No       | Due date (ISO 8601)                 |
| `metadata`   | object | No       | Additional context                  |

### Assignment Options

| Field                  | Type   | Description                                  |
| ---------------------- | ------ | -------------------------------------------- |
| `mode`                 | string | `auto`, `manual`, `pool`                     |
| `reviewer_id`          | string | Specific reviewer (for `manual`)             |
| `required_credentials` | array  | Required credentials: `MD`, `DO`, `NP`, `RN` |
| `specialty`            | string | Required specialty (e.g., `cardiology`)      |

### Example Request

<CodeGroup>
  ```python Python theme={null}
  from akhara import Akhara

  client = Akhara()

  # Create review task with auto-assignment
  task = client.reviews.create_task(
      project="proj_abc123",
      samples=["smp_001", "smp_002", "smp_003"],
      akhara="rubric_triage_v2",
      priority="high",
      assignment={
          "mode": "auto",
          "required_credentials": ["MD", "DO"],
          "specialty": "emergency_medicine"
      },
      due_at="2024-01-20T17:00:00Z",
      metadata={
          "evaluation": "eval_def456",
          "reason": "under_triage_detected"
      }
  )

  print(f"Created task: {task.id}")
  print(f"Assigned to: {task.assignee.name if task.assignee else 'Pending'}")
  ```

  ```bash cURL theme={null}
  curl -X POST "https://api.akhara.ai/v1/reviews/tasks" \
    -H "Authorization: Bearer rb_live_xxxxxxxx" \
    -H "Content-Type: application/json" \
    -d '{
      "project": "proj_abc123",
      "samples": ["smp_001", "smp_002", "smp_003"],
      "rubric": "rubric_triage_v2",
      "priority": "high",
      "assignment": {
        "mode": "auto",
        "required_credentials": ["MD", "DO"]
      }
    }'
  ```
</CodeGroup>

### Response

```json theme={null}
{
  "id": "task_xyz789",
  "object": "review_task",
  "project": "proj_abc123",
  "samples": ["smp_001", "smp_002", "smp_003"],
  "sample_count": 3,
  "rubric": "rubric_triage_v2",
  "status": "assigned",
  "priority": "high",
  "assignee": {
    "id": "user_dr_smith",
    "name": "Dr. Sarah Smith",
    "credentials": ["MD"],
    "specialty": "emergency_medicine"
  },
  "progress": {
    "completed": 0,
    "total": 3
  },
  "due_at": "2024-01-20T17:00:00Z",
  "created_at": "2024-01-15T10:00:00Z"
}
```

***

## List Review Tasks

List review tasks with optional filtering.

<EndpointBlock method="GET" path="/v1/reviews/tasks" />

### Query Parameters

| Parameter  | Type    | Description                                                    |
| ---------- | ------- | -------------------------------------------------------------- |
| `project`  | string  | Filter by project                                              |
| `status`   | string  | `pending`, `assigned`, `in_progress`, `completed`, `cancelled` |
| `assignee` | string  | Filter by assigned reviewer                                    |
| `priority` | string  | Filter by priority                                             |
| `limit`    | integer | Results per page (1-100)                                       |
| `after`    | string  | Pagination cursor                                              |

### Example Request

<CodeGroup>
  ```python Python theme={null}
  # Get pending tasks for a project
  tasks = client.reviews.list_tasks(
      project="proj_abc123",
      status="assigned",
      limit=20
  )

  for task in tasks:
      print(f"{task.id}: {task.sample_count} samples, due {task.due_at}")
  ```

  ```bash cURL theme={null}
  curl "https://api.akhara.ai/v1/reviews/tasks?project=proj_abc123&status=assigned" \
    -H "Authorization: Bearer rb_live_xxxxxxxx"
  ```
</CodeGroup>

***

## Get Review Task

Retrieve a specific review task with details.

<EndpointBlock method="GET" path="/v1/reviews/tasks/{task_id}" />

### Example Request

<CodeGroup>
  ```python Python theme={null}
  task = client.reviews.get_task("task_xyz789")

  print(f"Status: {task.status}")
  print(f"Progress: {task.progress.completed}/{task.progress.total}")
  print(f"Assignee: {task.assignee.name}")
  ```

  ```bash cURL theme={null}
  curl "https://api.akhara.ai/v1/reviews/tasks/task_xyz789" \
    -H "Authorization: Bearer rb_live_xxxxxxxx"
  ```
</CodeGroup>

***

## Submit Review

Submit a review for a sample within a task.

<EndpointBlock method="POST" path="/v1/reviews/tasks/{task_id}/reviews" />

### Request Body

| Parameter           | Type    | Required | Description                       |
| ------------------- | ------- | -------- | --------------------------------- |
| `sample`            | string  | Yes      | Sample ID being reviewed          |
| `grades`            | object  | Yes      | Grades per akhara dimension       |
| `override_ai_score` | boolean | No       | Whether to override AI evaluation |
| `override_value`    | number  | No       | Overridden score value            |
| `notes`             | string  | No       | Reviewer notes                    |
| `flags`             | array   | No       | Flags to apply                    |

### Example Request

<CodeGroup>
  ```python Python theme={null}
  review = client.reviews.submit(
      task="task_xyz789",
      sample="smp_001",
      grades={
          "triage_accuracy": {
              "value": "under_triaged",
              "correct_level": "emergent",
              "confidence": "high"
          },
          "red_flag_detection": {
              "value": "missed",
              "missed_flags": ["chest_pain_radiation"],
              "confidence": "high"
          },
          "communication_quality": {
              "value": 3,  # 1-5 scale
              "confidence": "medium"
          }
      },
      override_ai_score=True,
      override_value=35.0,
      notes="Patient described classic angina symptoms. AI missed radiation to left arm.",
      flags=["safety_concern", "training_example"]
  )

  print(f"Review submitted: {review.id}")
  ```

  ```bash cURL theme={null}
  curl -X POST "https://api.akhara.ai/v1/reviews/tasks/task_xyz789/reviews" \
    -H "Authorization: Bearer rb_live_xxxxxxxx" \
    -H "Content-Type: application/json" \
    -d '{
      "sample": "smp_001",
      "grades": {
        "triage_accuracy": {
          "value": "under_triaged",
          "correct_level": "emergent"
        }
      },
      "override_ai_score": true,
      "override_value": 35.0,
      "notes": "Patient described classic angina symptoms."
    }'
  ```
</CodeGroup>

### Response

```json theme={null}
{
  "id": "rev_abc123",
  "object": "review",
  "task": "task_xyz789",
  "sample": "smp_001",
  "reviewer": {
    "id": "user_dr_smith",
    "name": "Dr. Sarah Smith",
    "credentials": ["MD"]
  },
  "grades": {
    "triage_accuracy": {
      "value": "under_triaged",
      "correct_level": "emergent",
      "confidence": "high"
    }
  },
  "override_ai_score": true,
  "override_value": 35.0,
  "original_ai_score": 78.5,
  "notes": "Patient described classic angina symptoms. AI missed radiation to left arm.",
  "flags": ["safety_concern", "training_example"],
  "submitted_at": "2024-01-15T14:30:00Z"
}
```

***

## List Reviews

List submitted reviews with filtering.

<EndpointBlock method="GET" path="/v1/reviews" />

### Query Parameters

| Parameter      | Type    | Description                            |
| -------------- | ------- | -------------------------------------- |
| `project`      | string  | Filter by project                      |
| `task`         | string  | Filter by task                         |
| `sample`       | string  | Filter by sample                       |
| `reviewer`     | string  | Filter by reviewer                     |
| `has_override` | boolean | Filter to reviews with score overrides |
| `flags`        | array   | Filter by flags                        |
| `limit`        | integer | Results per page                       |

### Example Request

<CodeGroup>
  ```python Python theme={null}
  # Get reviews with safety concerns
  reviews = client.reviews.list(
      project="proj_abc123",
      flags=["safety_concern"],
      limit=50
  )

  for review in reviews:
      print(f"{review.sample}: {review.notes}")
  ```

  ```bash cURL theme={null}
  curl "https://api.akhara.ai/v1/reviews?project=proj_abc123&flags=safety_concern" \
    -H "Authorization: Bearer rb_live_xxxxxxxx"
  ```
</CodeGroup>

***

## Create Akhara

Create a grading akhara defining review criteria.

<EndpointBlock method="POST" path="/v1/reviews/rubrics" />

### Request Body

| Parameter     | Type   | Required | Description        |
| ------------- | ------ | -------- | ------------------ |
| `name`        | string | Yes      | Akhara name        |
| `project`     | string | Yes      | Project ID         |
| `dimensions`  | array  | Yes      | Grading dimensions |
| `description` | string | No       | Akhara description |

### Dimension Types

| Type           | Description         | Example Values                             |
| -------------- | ------------------- | ------------------------------------------ |
| `categorical`  | Select from options | `correct`, `under_triaged`, `over_triaged` |
| `scale`        | Numeric scale       | 1-5, 1-10                                  |
| `boolean`      | Yes/No              | `true`, `false`                            |
| `multi_select` | Multiple options    | `["chest_pain", "shortness_of_breath"]`    |

### Example Request

<CodeGroup>
  ```python Python theme={null}
  akhara = client.reviews.create_rubric(
      name="Triage Review v2",
      project="proj_abc123",
      description="Standard triage review rubric for voice AI",
      dimensions=[
          {
              "id": "triage_accuracy",
              "name": "Triage Accuracy",
              "type": "categorical",
              "required": True,
              "options": [
                  {"value": "correct", "label": "Correct Triage", "score_impact": 0},
                  {"value": "under_triaged", "label": "Under-Triaged", "score_impact": -50, "safety_flag": True},
                  {"value": "over_triaged", "label": "Over-Triaged", "score_impact": -10}
              ],
              "guidance": "Compare AI triage level to what you would have assigned."
          },
          {
              "id": "red_flag_detection",
              "name": "Red Flag Detection",
              "type": "categorical",
              "required": True,
              "options": [
                  {"value": "all_detected", "label": "All Red Flags Detected"},
                  {"value": "partial", "label": "Some Missed"},
                  {"value": "missed", "label": "Critical Flags Missed", "safety_flag": True}
              ]
          },
          {
              "id": "communication_quality",
              "name": "Communication Quality",
              "type": "scale",
              "required": False,
              "min": 1,
              "max": 5,
              "labels": {
                  "1": "Poor",
                  "3": "Acceptable",
                  "5": "Excellent"
              }
          }
      ]
  )
  ```

  ```bash cURL theme={null}
  curl -X POST "https://api.akhara.ai/v1/reviews/rubrics" \
    -H "Authorization: Bearer rb_live_xxxxxxxx" \
    -H "Content-Type: application/json" \
    -d '{
      "name": "Triage Review v2",
      "project": "proj_abc123",
      "dimensions": [...]
    }'
  ```
</CodeGroup>

***

## Get Reviewer Queue

Get the review queue for a specific reviewer.

<EndpointBlock method="GET" path="/v1/reviews/queue" />

### Query Parameters

| Parameter  | Type    | Description         |
| ---------- | ------- | ------------------- |
| `project`  | string  | Filter by project   |
| `priority` | string  | Filter by priority  |
| `limit`    | integer | Max items to return |

### Example Request

<CodeGroup>
  ```python Python theme={null}
  # Get current user's review queue
  queue = client.reviews.get_queue(
      project="proj_abc123",
      limit=10
  )

  print(f"Pending reviews: {queue.total_pending}")
  for item in queue.items:
      print(f"  {item.sample_id}: {item.priority} priority, due {item.due_at}")
  ```

  ```bash cURL theme={null}
  curl "https://api.akhara.ai/v1/reviews/queue?project=proj_abc123" \
    -H "Authorization: Bearer rb_live_xxxxxxxx"
  ```
</CodeGroup>

### Response

```json theme={null}
{
  "reviewer": "user_dr_smith",
  "total_pending": 12,
  "items": [
    {
      "task_id": "task_xyz789",
      "sample_id": "smp_001",
      "priority": "high",
      "rubric": "rubric_triage_v2",
      "due_at": "2024-01-20T17:00:00Z",
      "context": {
        "evaluation": "eval_def456",
        "ai_score": 45.0,
        "flag_reason": "under_triage_detected"
      }
    }
  ]
}
```

***

## Manage Reviewers

### List Reviewers

<EndpointBlock method="GET" path="/v1/reviews/reviewers" />

```python theme={null}
reviewers = client.reviews.list_reviewers(
    project="proj_abc123",
    credentials=["MD", "DO"],
    available=True
)

for reviewer in reviewers:
    print(f"{reviewer.name}: {reviewer.credentials}, {reviewer.pending_count} pending")
```

### Get Reviewer Stats

<EndpointBlock method="GET" path="/v1/reviews/reviewers/{reviewer_id}/stats" />

```python theme={null}
stats = client.reviews.get_reviewer_stats(
    reviewer="user_dr_smith",
    period="30d"
)

print(f"Reviews completed: {stats.completed}")
print(f"Average time per review: {stats.avg_duration_seconds}s")
print(f"Override rate: {stats.override_rate}%")
print(f"Agreement with consensus: {stats.consensus_agreement}%")
```

***

## Inter-Rater Reliability

Calculate agreement between reviewers.

<EndpointBlock method="POST" path="/v1/reviews/reliability" />

### Request Body

| Parameter   | Type   | Required | Description                 |
| ----------- | ------ | -------- | --------------------------- |
| `project`   | string | Yes      | Project ID                  |
| `dimension` | string | Yes      | Akhara dimension to analyze |
| `period`    | string | No       | Time period                 |

### Example Request

<CodeGroup>
  ```python Python theme={null}
  reliability = client.reviews.calculate_reliability(
      project="proj_abc123",
      dimension="triage_accuracy",
      period="30d"
  )

  print(f"Cohen's Kappa: {reliability.cohens_kappa}")
  print(f"Fleiss' Kappa: {reliability.fleiss_kappa}")
  print(f"Percent Agreement: {reliability.percent_agreement}%")
  print(f"Reviews analyzed: {reliability.review_count}")
  ```

  ```bash cURL theme={null}
  curl -X POST "https://api.akhara.ai/v1/reviews/reliability" \
    -H "Authorization: Bearer rb_live_xxxxxxxx" \
    -H "Content-Type: application/json" \
    -d '{
      "project": "proj_abc123",
      "dimension": "triage_accuracy",
      "period": "30d"
    }'
  ```
</CodeGroup>

### Response

```json theme={null}
{
  "project": "proj_abc123",
  "dimension": "triage_accuracy",
  "period": "30d",
  "cohens_kappa": 0.78,
  "fleiss_kappa": 0.74,
  "percent_agreement": 85.2,
  "review_count": 342,
  "reviewer_count": 8,
  "interpretation": "substantial_agreement",
  "by_category": {
    "correct": {"agreement": 92.1},
    "under_triaged": {"agreement": 78.5},
    "over_triaged": {"agreement": 81.2}
  }
}
```

***

## Related

<CardGroup cols={2}>
  <Card title="RBAC" icon="lock" href="/evaluation/api-reference/authentication/rbac">
    Configure reviewer roles and credentials
  </Card>

  <Card title="Audit Logs" icon="scroll" href="/evaluation/api-reference/authentication/audit-logs">
    Track all review activity
  </Card>
</CardGroup>