> ## Documentation Index
> Fetch the complete documentation index at: https://docs.akhara.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Company name is Akhara AI (never Rubric AI). Keep lowercase rubric/rubrics only when meaning grading criteria.
> Expert Review (docs path talent/) is enterprise BYO experts for audit and review: invite customer specialists; do not pitch Akhara recruiting or a public expert career portal. RLHF and domain writing are secondary work types.
> Prefer concrete API examples against public hosts: Environments eval API https://agi.akhara.ai, Control plane PDP https://api.akhara.dev, Evaluation https://app.akhara.ai / https://api.akhara.ai, Expert Review portal https://talent.akhara.ai.
> Do not invent a public hostname for private orchestrators or env API internals.
> Do not confuse control-plane latches with Environments confirmation latches.
> Environments SDK/API examples: curl against https://agi.akhara.ai. Evaluation SDK: from akhara import Akhara and AKHARA_API_KEY.
> Start with /llms.txt for the docs index and OpenAPI links; fetch individual pages as .md exports.

# Evaluation APIs

> Create and manage evaluations to assess your healthcare AI outputs.

## Overview

Evaluations run one or more evaluators against samples in a dataset. Each evaluation produces scores and detailed results that you can use to measure AI performance.

**Base URL:** `https://api.akhara.ai/v1/evaluations`

***

## Create Evaluation

Create and start a new evaluation run.

<EndpointBlock method="POST" path="/v1/evaluations" />

### Request Body

| Parameter       | Type    | Required | Description                            |
| --------------- | ------- | -------- | -------------------------------------- |
| `name`          | string  | Yes      | Human-readable name for the evaluation |
| `project`       | string  | Yes      | Project ID                             |
| `dataset`       | string  | Yes      | Dataset ID to evaluate                 |
| `evaluators`    | array   | Yes      | List of evaluator configurations       |
| `sample_filter` | object  | No       | Filter which samples to evaluate       |
| `run_async`     | boolean | No       | Run asynchronously (default: true)     |
| `metadata`      | object  | No       | Arbitrary key-value pairs              |

### Evaluator Configuration

| Field    | Type   | Required | Description                                 |
| -------- | ------ | -------- | ------------------------------------------- |
| `type`   | string | Yes      | Evaluator type (e.g., `triage_accuracy`)    |
| `config` | object | No       | Evaluator-specific settings                 |
| `weight` | number | No       | Weight for aggregate scoring (default: 1.0) |

### Example Request

<CodeGroup>
  ```python Python theme={null}
  from akhara import Akhara

  client = Akhara()

  evaluation = client.evaluations.create(
      name="Triage Model v2.4 - Weekly",
      project="proj_abc123",
      dataset="ds_xyz789",
      evaluators=[
          {
              "type": "triage_accuracy",
              "config": {
                  "severity_weights": {
                      "under_triage": 5.0,
                      "over_triage": 1.0
                  }
              },
              "weight": 2.0
          },
          {
              "type": "red_flag_detection",
              "config": {
                  "protocols": ["chest_pain", "stroke", "sepsis"]
              },
              "weight": 3.0
          },
          {
              "type": "guideline_compliance",
              "config": {
                  "guideline_set": "schmitt_thompson_adult"
              }
          }
      ],
      metadata={
          "triggered_by": "ci_pipeline",
          "model_version": "v2.4.1",
          "commit_sha": "abc123"
      }
  )

  print(f"Created evaluation: {evaluation.id}")
  print(f"Status: {evaluation.status}")
  ```

  ```typescript TypeScript theme={null}
  import Akhara from '@akhara/sdk';

  const client = new Akhara();

  const evaluation = await client.evaluations.create({
    name: "Triage Model v2.4 - Weekly",
    project: "proj_abc123",
    dataset: "ds_xyz789",
    evaluators: [
      {
        type: "triage_accuracy",
        config: {
          severity_weights: {
            under_triage: 5.0,
            over_triage: 1.0
          }
        },
        weight: 2.0
      },
      {
        type: "red_flag_detection",
        config: {
          protocols: ["chest_pain", "stroke", "sepsis"]
        },
        weight: 3.0
      }
    ],
    metadata: {
      triggered_by: "ci_pipeline",
      model_version: "v2.4.1"
    }
  });
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.akhara.ai/v1/evaluations \
    -H "Authorization: Bearer rb_live_xxxxxxxx" \
    -H "Content-Type: application/json" \
    -d '{
      "name": "Triage Model v2.4 - Weekly",
      "project": "proj_abc123",
      "dataset": "ds_xyz789",
      "evaluators": [
        {
          "type": "triage_accuracy",
          "config": {
            "severity_weights": {
              "under_triage": 5.0,
              "over_triage": 1.0
            }
          }
        },
        {
          "type": "red_flag_detection",
          "config": {
            "protocols": ["chest_pain", "stroke", "sepsis"]
          }
        }
      ]
    }'
  ```
</CodeGroup>

### Response

```json theme={null}
{
  "id": "eval_def456",
  "object": "evaluation",
  "name": "Triage Model v2.4 - Weekly",
  "project": "proj_abc123",
  "dataset": "ds_xyz789",
  "status": "pending",
  "evaluators": [
    {
      "type": "triage_accuracy",
      "config": { "severity_weights": { ... } },
      "weight": 2.0
    },
    {
      "type": "red_flag_detection",
      "config": { "protocols": [...] },
      "weight": 3.0
    }
  ],
  "progress": {
    "total": 1247,
    "completed": 0,
    "failed": 0
  },
  "created_at": "2024-01-15T10:30:00Z",
  "started_at": null,
  "completed_at": null,
  "metadata": {
    "triggered_by": "ci_pipeline",
    "model_version": "v2.4.1"
  }
}
```

***

## List Evaluations

List evaluations with optional filtering.

<EndpointBlock method="GET" path="/v1/evaluations" />

### Query Parameters

| Parameter | Type    | Description                                                                |
| --------- | ------- | -------------------------------------------------------------------------- |
| `project` | string  | Filter by project ID                                                       |
| `dataset` | string  | Filter by dataset ID                                                       |
| `status`  | string  | Filter by status: `pending`, `running`, `completed`, `failed`, `cancelled` |
| `limit`   | integer | Results per page (1-100)                                                   |
| `after`   | string  | Pagination cursor                                                          |

### Example Request

<CodeGroup>
  ```python Python theme={null}
  # List recent evaluations
  evaluations = client.evaluations.list(
      project="proj_abc123",
      status="completed",
      limit=10
  )

  for eval in evaluations:
      print(f"{eval.name}: {eval.results.aggregate_score}%")
  ```

  ```bash cURL theme={null}
  curl "https://api.akhara.ai/v1/evaluations?project=proj_abc123&status=completed" \
    -H "Authorization: Bearer rb_live_xxxxxxxx"
  ```
</CodeGroup>

***

## Get Evaluation

Retrieve a specific evaluation with full results.

<EndpointBlock method="GET" path="/v1/evaluations/{evaluation_id}" />

### Example Request

<CodeGroup>
  ```python Python theme={null}
  evaluation = client.evaluations.get("eval_def456")

  print(f"Status: {evaluation.status}")
  print(f"Progress: {evaluation.progress.completed}/{evaluation.progress.total}")

  if evaluation.status == "completed":
      print(f"Aggregate Score: {evaluation.results.aggregate_score}%")
      for evaluator, score in evaluation.results.by_evaluator.items():
          print(f"  {evaluator}: {score}%")
  ```

  ```bash cURL theme={null}
  curl "https://api.akhara.ai/v1/evaluations/eval_def456" \
    -H "Authorization: Bearer rb_live_xxxxxxxx"
  ```
</CodeGroup>

### Response (Completed)

```json theme={null}
{
  "id": "eval_def456",
  "object": "evaluation",
  "name": "Triage Model v2.4 - Weekly",
  "status": "completed",
  "progress": {
    "total": 1247,
    "completed": 1247,
    "failed": 3
  },
  "results": {
    "aggregate_score": 84.2,
    "by_evaluator": {
      "triage_accuracy": 78.5,
      "red_flag_detection": 92.1,
      "guideline_compliance": 81.8
    },
    "distributions": {
      "triage_accuracy": {
        "correct": 892,
        "under_triage": 156,
        "over_triage": 196
      }
    },
    "flagged_samples": 42
  },
  "created_at": "2024-01-15T10:30:00Z",
  "started_at": "2024-01-15T10:30:05Z",
  "completed_at": "2024-01-15T10:45:32Z",
  "duration_seconds": 927
}
```

***

## Get Evaluation Results

Retrieve detailed per-sample results.

<EndpointBlock method="GET" path="/v1/evaluations/{evaluation_id}/results" />

### Query Parameters

| Parameter     | Type    | Description                            |
| ------------- | ------- | -------------------------------------- |
| `evaluator`   | string  | Filter by specific evaluator           |
| `score_below` | number  | Filter samples scoring below threshold |
| `flagged`     | boolean | Filter to flagged samples only         |
| `limit`       | integer | Results per page (1-100)               |
| `after`       | string  | Pagination cursor                      |

### Example Request

<CodeGroup>
  ```python Python theme={null}
  # Get low-scoring samples for review
  results = client.evaluations.get_results(
      evaluation="eval_def456",
      evaluator="triage_accuracy",
      score_below=70,
      limit=50
  )

  for result in results:
      print(f"Sample {result.sample_id}: {result.score}%")
      print(f"  AI output: {result.output.triage_level}")
      print(f"  Expected: {result.expected.triage_level}")
      print(f"  Details: {result.details}")
  ```

  ```bash cURL theme={null}
  curl "https://api.akhara.ai/v1/evaluations/eval_def456/results?evaluator=triage_accuracy&score_below=70" \
    -H "Authorization: Bearer rb_live_xxxxxxxx"
  ```
</CodeGroup>

### Response

```json theme={null}
{
  "object": "list",
  "data": [
    {
      "sample_id": "smp_abc123",
      "evaluator": "triage_accuracy",
      "score": 45.0,
      "passed": false,
      "output": {
        "triage_level": "routine"
      },
      "expected": {
        "triage_level": "urgent"
      },
      "details": {
        "classification": "under_triage",
        "severity_penalty": 5.0,
        "reason": "Patient reported chest pain with exertion, warranting urgent evaluation"
      },
      "flagged": true,
      "flag_reason": "Critical under-triage: chest pain symptoms"
    }
  ],
  "has_more": true
}
```

***

## Cancel Evaluation

Cancel a running evaluation.

<EndpointBlock method="POST" path="/v1/evaluations/{evaluation_id}/cancel" />

### Example Request

<CodeGroup>
  ```python Python theme={null}
  client.evaluations.cancel("eval_def456")
  ```

  ```bash cURL theme={null}
  curl -X POST "https://api.akhara.ai/v1/evaluations/eval_def456/cancel" \
    -H "Authorization: Bearer rb_live_xxxxxxxx"
  ```
</CodeGroup>

### Response

```json theme={null}
{
  "id": "eval_def456",
  "object": "evaluation",
  "status": "cancelled",
  "progress": {
    "total": 1247,
    "completed": 523,
    "failed": 2
  },
  "cancelled_at": "2024-01-15T10:35:00Z"
}
```

***

## Compare Evaluations

Compare results between two or more evaluations.

<EndpointBlock method="POST" path="/v1/evaluations/compare" />

### Request Body

| Parameter     | Type   | Required | Description                       |
| ------------- | ------ | -------- | --------------------------------- |
| `evaluations` | array  | Yes      | List of evaluation IDs to compare |
| `baseline`    | string | No       | Evaluation ID to use as baseline  |
| `metrics`     | array  | No       | Specific metrics to compare       |

### Example Request

<CodeGroup>
  ```python Python theme={null}
  comparison = client.evaluations.compare(
      evaluations=["eval_v2_3", "eval_v2_4"],
      baseline="eval_v2_3"
  )

  print(f"Baseline: {comparison.baseline.name}")
  print(f"Candidate: {comparison.candidate.name}")
  print(f"Improvement: {comparison.improvement_pct}%")
  print(f"Statistically significant: {comparison.is_significant}")

  for metric, diff in comparison.by_metric.items():
      print(f"  {metric}: {diff.baseline} → {diff.candidate} ({diff.change:+.1f}%)")
  ```

  ```bash cURL theme={null}
  curl -X POST "https://api.akhara.ai/v1/evaluations/compare" \
    -H "Authorization: Bearer rb_live_xxxxxxxx" \
    -H "Content-Type: application/json" \
    -d '{
      "evaluations": ["eval_v2_3", "eval_v2_4"],
      "baseline": "eval_v2_3"
    }'
  ```
</CodeGroup>

### Response

```json theme={null}
{
  "object": "comparison",
  "baseline": {
    "id": "eval_v2_3",
    "name": "Triage v2.3",
    "aggregate_score": 81.2
  },
  "candidate": {
    "id": "eval_v2_4",
    "name": "Triage v2.4",
    "aggregate_score": 84.2
  },
  "improvement_pct": 3.7,
  "is_significant": true,
  "p_value": 0.023,
  "by_metric": {
    "triage_accuracy": {
      "baseline": 75.1,
      "candidate": 78.5,
      "change": 4.5,
      "significant": true
    },
    "red_flag_detection": {
      "baseline": 91.8,
      "candidate": 92.1,
      "change": 0.3,
      "significant": false
    }
  },
  "regressions": [],
  "improvements": ["triage_accuracy"]
}
```

***

## Built-in Evaluators

### Triage Accuracy

Evaluates correctness of urgency level classification.

```python theme={null}
{
    "type": "triage_accuracy",
    "config": {
        "levels": ["emergent", "urgent", "semi_urgent", "routine"],
        "severity_weights": {
            "under_triage": 5.0,  # Penalize under-triage heavily
            "over_triage": 1.0
        },
        "require_expected": True
    }
}
```

### Red Flag Detection

Checks if critical symptoms were identified.

```python theme={null}
{
    "type": "red_flag_detection",
    "config": {
        "protocols": ["chest_pain", "stroke", "sepsis", "pediatric_fever"],
        "require_escalation": True,
        "min_confidence": 0.8
    }
}
```

### Guideline Compliance

Measures adherence to clinical protocols.

```python theme={null}
{
    "type": "guideline_compliance",
    "config": {
        "guideline_set": "schmitt_thompson_adult",
        "required_questions": ["onset", "severity", "associated_symptoms"],
        "scoring_mode": "weighted"
    }
}
```

### Symptom Extraction

Evaluates completeness of symptom identification.

```python theme={null}
{
    "type": "symptom_extraction",
    "config": {
        "taxonomy": "snomed_ct",
        "require_negation_handling": True,
        "min_recall": 0.9
    }
}
```

### Custom Evaluator

Use your own evaluation logic via webhook.

```python theme={null}
{
    "type": "custom",
    "config": {
        "webhook_url": "https://your-server.com/evaluate",
        "timeout_seconds": 30,
        "retry_count": 3
    }
}
```

***

## Related

<CardGroup cols={2}>
  <Card title="Scoring & Metrics APIs" icon="chart-bar" href="/evaluation/api-reference/scores/overview">
    Detailed scoring and metrics endpoints
  </Card>

  <Card title="Human Review APIs" icon="user-md" href="/evaluation/api-reference/reviews/overview">
    Route flagged samples for clinician review
  </Card>
</CardGroup>