Overview
Evaluations run one or more evaluators against samples in a dataset. Each evaluation produces scores and detailed results that you can use to measure AI performance. Base URL:https://api.akhara.ai/v1/evaluations
Create Evaluation
Create and start a new evaluation run.Request Body
| Parameter | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Human-readable name for the evaluation |
project | string | Yes | Project ID |
dataset | string | Yes | Dataset ID to evaluate |
evaluators | array | Yes | List of evaluator configurations |
sample_filter | object | No | Filter which samples to evaluate |
run_async | boolean | No | Run asynchronously (default: true) |
metadata | object | No | Arbitrary key-value pairs |
Evaluator Configuration
| Field | Type | Required | Description |
|---|---|---|---|
type | string | Yes | Evaluator type (e.g., triage_accuracy) |
config | object | No | Evaluator-specific settings |
weight | number | No | Weight for aggregate scoring (default: 1.0) |
Example Request
from akhara import Akhara
client = Akhara()
evaluation = client.evaluations.create(
name="Triage Model v2.4 - Weekly",
project="proj_abc123",
dataset="ds_xyz789",
evaluators=[
{
"type": "triage_accuracy",
"config": {
"severity_weights": {
"under_triage": 5.0,
"over_triage": 1.0
}
},
"weight": 2.0
},
{
"type": "red_flag_detection",
"config": {
"protocols": ["chest_pain", "stroke", "sepsis"]
},
"weight": 3.0
},
{
"type": "guideline_compliance",
"config": {
"guideline_set": "schmitt_thompson_adult"
}
}
],
metadata={
"triggered_by": "ci_pipeline",
"model_version": "v2.4.1",
"commit_sha": "abc123"
}
)
print(f"Created evaluation: {evaluation.id}")
print(f"Status: {evaluation.status}")
import Akhara from '@akhara/sdk';
const client = new Akhara();
const evaluation = await client.evaluations.create({
name: "Triage Model v2.4 - Weekly",
project: "proj_abc123",
dataset: "ds_xyz789",
evaluators: [
{
type: "triage_accuracy",
config: {
severity_weights: {
under_triage: 5.0,
over_triage: 1.0
}
},
weight: 2.0
},
{
type: "red_flag_detection",
config: {
protocols: ["chest_pain", "stroke", "sepsis"]
},
weight: 3.0
}
],
metadata: {
triggered_by: "ci_pipeline",
model_version: "v2.4.1"
}
});
curl -X POST https://api.akhara.ai/v1/evaluations \
-H "Authorization: Bearer rb_live_xxxxxxxx" \
-H "Content-Type: application/json" \
-d '{
"name": "Triage Model v2.4 - Weekly",
"project": "proj_abc123",
"dataset": "ds_xyz789",
"evaluators": [
{
"type": "triage_accuracy",
"config": {
"severity_weights": {
"under_triage": 5.0,
"over_triage": 1.0
}
}
},
{
"type": "red_flag_detection",
"config": {
"protocols": ["chest_pain", "stroke", "sepsis"]
}
}
]
}'
Response
{
"id": "eval_def456",
"object": "evaluation",
"name": "Triage Model v2.4 - Weekly",
"project": "proj_abc123",
"dataset": "ds_xyz789",
"status": "pending",
"evaluators": [
{
"type": "triage_accuracy",
"config": { "severity_weights": { ... } },
"weight": 2.0
},
{
"type": "red_flag_detection",
"config": { "protocols": [...] },
"weight": 3.0
}
],
"progress": {
"total": 1247,
"completed": 0,
"failed": 0
},
"created_at": "2024-01-15T10:30:00Z",
"started_at": null,
"completed_at": null,
"metadata": {
"triggered_by": "ci_pipeline",
"model_version": "v2.4.1"
}
}
List Evaluations
List evaluations with optional filtering.Query Parameters
| Parameter | Type | Description |
|---|---|---|
project | string | Filter by project ID |
dataset | string | Filter by dataset ID |
status | string | Filter by status: pending, running, completed, failed, cancelled |
limit | integer | Results per page (1-100) |
after | string | Pagination cursor |
Example Request
# List recent evaluations
evaluations = client.evaluations.list(
project="proj_abc123",
status="completed",
limit=10
)
for eval in evaluations:
print(f"{eval.name}: {eval.results.aggregate_score}%")
curl "https://api.akhara.ai/v1/evaluations?project=proj_abc123&status=completed" \
-H "Authorization: Bearer rb_live_xxxxxxxx"
Get Evaluation
Retrieve a specific evaluation with full results.Example Request
evaluation = client.evaluations.get("eval_def456")
print(f"Status: {evaluation.status}")
print(f"Progress: {evaluation.progress.completed}/{evaluation.progress.total}")
if evaluation.status == "completed":
print(f"Aggregate Score: {evaluation.results.aggregate_score}%")
for evaluator, score in evaluation.results.by_evaluator.items():
print(f" {evaluator}: {score}%")
curl "https://api.akhara.ai/v1/evaluations/eval_def456" \
-H "Authorization: Bearer rb_live_xxxxxxxx"
Response (Completed)
{
"id": "eval_def456",
"object": "evaluation",
"name": "Triage Model v2.4 - Weekly",
"status": "completed",
"progress": {
"total": 1247,
"completed": 1247,
"failed": 3
},
"results": {
"aggregate_score": 84.2,
"by_evaluator": {
"triage_accuracy": 78.5,
"red_flag_detection": 92.1,
"guideline_compliance": 81.8
},
"distributions": {
"triage_accuracy": {
"correct": 892,
"under_triage": 156,
"over_triage": 196
}
},
"flagged_samples": 42
},
"created_at": "2024-01-15T10:30:00Z",
"started_at": "2024-01-15T10:30:05Z",
"completed_at": "2024-01-15T10:45:32Z",
"duration_seconds": 927
}
Get Evaluation Results
Retrieve detailed per-sample results.Query Parameters
| Parameter | Type | Description |
|---|---|---|
evaluator | string | Filter by specific evaluator |
score_below | number | Filter samples scoring below threshold |
flagged | boolean | Filter to flagged samples only |
limit | integer | Results per page (1-100) |
after | string | Pagination cursor |
Example Request
# Get low-scoring samples for review
results = client.evaluations.get_results(
evaluation="eval_def456",
evaluator="triage_accuracy",
score_below=70,
limit=50
)
for result in results:
print(f"Sample {result.sample_id}: {result.score}%")
print(f" AI output: {result.output.triage_level}")
print(f" Expected: {result.expected.triage_level}")
print(f" Details: {result.details}")
curl "https://api.akhara.ai/v1/evaluations/eval_def456/results?evaluator=triage_accuracy&score_below=70" \
-H "Authorization: Bearer rb_live_xxxxxxxx"
Response
{
"object": "list",
"data": [
{
"sample_id": "smp_abc123",
"evaluator": "triage_accuracy",
"score": 45.0,
"passed": false,
"output": {
"triage_level": "routine"
},
"expected": {
"triage_level": "urgent"
},
"details": {
"classification": "under_triage",
"severity_penalty": 5.0,
"reason": "Patient reported chest pain with exertion, warranting urgent evaluation"
},
"flagged": true,
"flag_reason": "Critical under-triage: chest pain symptoms"
}
],
"has_more": true
}
Cancel Evaluation
Cancel a running evaluation.Example Request
client.evaluations.cancel("eval_def456")
curl -X POST "https://api.akhara.ai/v1/evaluations/eval_def456/cancel" \
-H "Authorization: Bearer rb_live_xxxxxxxx"
Response
{
"id": "eval_def456",
"object": "evaluation",
"status": "cancelled",
"progress": {
"total": 1247,
"completed": 523,
"failed": 2
},
"cancelled_at": "2024-01-15T10:35:00Z"
}
Compare Evaluations
Compare results between two or more evaluations.Request Body
| Parameter | Type | Required | Description |
|---|---|---|---|
evaluations | array | Yes | List of evaluation IDs to compare |
baseline | string | No | Evaluation ID to use as baseline |
metrics | array | No | Specific metrics to compare |
Example Request
comparison = client.evaluations.compare(
evaluations=["eval_v2_3", "eval_v2_4"],
baseline="eval_v2_3"
)
print(f"Baseline: {comparison.baseline.name}")
print(f"Candidate: {comparison.candidate.name}")
print(f"Improvement: {comparison.improvement_pct}%")
print(f"Statistically significant: {comparison.is_significant}")
for metric, diff in comparison.by_metric.items():
print(f" {metric}: {diff.baseline} → {diff.candidate} ({diff.change:+.1f}%)")
curl -X POST "https://api.akhara.ai/v1/evaluations/compare" \
-H "Authorization: Bearer rb_live_xxxxxxxx" \
-H "Content-Type: application/json" \
-d '{
"evaluations": ["eval_v2_3", "eval_v2_4"],
"baseline": "eval_v2_3"
}'
Response
{
"object": "comparison",
"baseline": {
"id": "eval_v2_3",
"name": "Triage v2.3",
"aggregate_score": 81.2
},
"candidate": {
"id": "eval_v2_4",
"name": "Triage v2.4",
"aggregate_score": 84.2
},
"improvement_pct": 3.7,
"is_significant": true,
"p_value": 0.023,
"by_metric": {
"triage_accuracy": {
"baseline": 75.1,
"candidate": 78.5,
"change": 4.5,
"significant": true
},
"red_flag_detection": {
"baseline": 91.8,
"candidate": 92.1,
"change": 0.3,
"significant": false
}
},
"regressions": [],
"improvements": ["triage_accuracy"]
}
Built-in Evaluators
Triage Accuracy
Evaluates correctness of urgency level classification.{
"type": "triage_accuracy",
"config": {
"levels": ["emergent", "urgent", "semi_urgent", "routine"],
"severity_weights": {
"under_triage": 5.0, # Penalize under-triage heavily
"over_triage": 1.0
},
"require_expected": True
}
}
Red Flag Detection
Checks if critical symptoms were identified.{
"type": "red_flag_detection",
"config": {
"protocols": ["chest_pain", "stroke", "sepsis", "pediatric_fever"],
"require_escalation": True,
"min_confidence": 0.8
}
}
Guideline Compliance
Measures adherence to clinical protocols.{
"type": "guideline_compliance",
"config": {
"guideline_set": "schmitt_thompson_adult",
"required_questions": ["onset", "severity", "associated_symptoms"],
"scoring_mode": "weighted"
}
}
Symptom Extraction
Evaluates completeness of symptom identification.{
"type": "symptom_extraction",
"config": {
"taxonomy": "snomed_ct",
"require_negation_handling": True,
"min_recall": 0.9
}
}
Custom Evaluator
Use your own evaluation logic via webhook.{
"type": "custom",
"config": {
"webhook_url": "https://your-server.com/evaluate",
"timeout_seconds": 30,
"retry_count": 3
}
}
Related
Scoring & Metrics APIs
Detailed scoring and metrics endpoints
Human Review APIs
Route flagged samples for clinician review