Scoring & Metrics APIs

Overview

The Scoring & Metrics APIs provide access to evaluation scores at multiple granularities: per-sample, per-evaluator, aggregate, and over time. Use these endpoints to build dashboards, track regressions, and analyze AI performance. Base URL: https://api.akhara.ai/v1/scores

Get Sample Scores

Retrieve all scores for a specific sample.

Query Parameters

Parameter	Type	Description
`evaluation`	string	Filter to specific evaluation
`evaluator`	string	Filter to specific evaluator

Example Request

from akhara import Akhara

client = Akhara()

# Get all scores for a sample
scores = client.scores.get_sample("smp_abc123")

for score in scores:
    print(f"{score.evaluator}: {score.value}%")
    print(f"  Details: {score.details}")

curl "https://api.akhara.ai/v1/scores/samples/smp_abc123" \
  -H "Authorization: Bearer rb_live_xxxxxxxx"

Response

{
  "sample_id": "smp_abc123",
  "scores": [
    {
      "evaluation": "eval_def456",
      "evaluator": "triage_accuracy",
      "value": 78.5,
      "passed": true,
      "details": {
        "predicted": "urgent",
        "expected": "urgent",
        "classification": "correct"
      },
      "created_at": "2024-01-15T10:32:00Z"
    },
    {
      "evaluation": "eval_def456",
      "evaluator": "red_flag_detection",
      "value": 100.0,
      "passed": true,
      "details": {
        "flags_detected": ["chest_pain"],
        "flags_expected": ["chest_pain"],
        "missed": [],
        "false_positives": []
      },
      "created_at": "2024-01-15T10:32:00Z"
    }
  ]
}

Get Evaluation Scores

Retrieve aggregate scores for an evaluation.

Query Parameters

Parameter	Type	Description
`include_distribution`	boolean	Include score distributions
`include_breakdown`	boolean	Include per-evaluator breakdown

Example Request

scores = client.scores.get_evaluation(
    evaluation="eval_def456",
    include_distribution=True,
    include_breakdown=True
)

print(f"Aggregate: {scores.aggregate}%")
print(f"Samples evaluated: {scores.sample_count}")

for evaluator, data in scores.breakdown.items():
    print(f"{evaluator}:")
    print(f"  Mean: {data.mean}%")
    print(f"  Median: {data.median}%")
    print(f"  Std Dev: {data.std_dev}")
    print(f"  Pass Rate: {data.pass_rate}%")

curl "https://api.akhara.ai/v1/scores/evaluations/eval_def456?include_distribution=true&include_breakdown=true" \
  -H "Authorization: Bearer rb_live_xxxxxxxx"

Response

{
  "evaluation_id": "eval_def456",
  "aggregate": 84.2,
  "weighted_aggregate": 82.1,
  "sample_count": 1247,
  "pass_rate": 91.3,
  "breakdown": {
    "triage_accuracy": {
      "mean": 78.5,
      "median": 82.0,
      "std_dev": 15.2,
      "min": 0.0,
      "max": 100.0,
      "pass_rate": 87.4,
      "weight": 2.0
    },
    "red_flag_detection": {
      "mean": 92.1,
      "median": 100.0,
      "std_dev": 18.5,
      "min": 0.0,
      "max": 100.0,
      "pass_rate": 94.2,
      "weight": 3.0
    }
  },
  "distribution": {
    "buckets": [
      {"range": "0-10", "count": 12},
      {"range": "10-20", "count": 8},
      {"range": "20-30", "count": 15},
      {"range": "30-40", "count": 23},
      {"range": "40-50", "count": 45},
      {"range": "50-60", "count": 89},
      {"range": "60-70", "count": 156},
      {"range": "70-80", "count": 287},
      {"range": "80-90", "count": 356},
      {"range": "90-100", "count": 256}
    ]
  }
}

Get Project Metrics

Retrieve aggregate metrics across all evaluations in a project.

Query Parameters

Parameter	Type	Description
`start_date`	string	Start of date range (ISO 8601)
`end_date`	string	End of date range (ISO 8601)
`evaluator`	string	Filter to specific evaluator
`granularity`	string	`daily`, `weekly`, `monthly`

Example Request

from datetime import datetime, timedelta

metrics = client.scores.get_project_metrics(
    project="proj_abc123",
    start_date=datetime.now() - timedelta(days=30),
    end_date=datetime.now(),
    granularity="daily"
)

print(f"Current score: {metrics.current}%")
print(f"30-day trend: {metrics.trend:+.1f}%")
print(f"Evaluations run: {metrics.evaluation_count}")

curl "https://api.akhara.ai/v1/scores/projects/proj_abc123/metrics?start_date=2024-01-01&end_date=2024-01-31&granularity=daily" \
  -H "Authorization: Bearer rb_live_xxxxxxxx"

Response

{
  "project_id": "proj_abc123",
  "period": {
    "start": "2024-01-01T00:00:00Z",
    "end": "2024-01-31T23:59:59Z"
  },
  "current": 84.2,
  "previous": 81.5,
  "trend": 2.7,
  "evaluation_count": 45,
  "sample_count": 12847,
  "by_evaluator": {
    "triage_accuracy": {
      "current": 78.5,
      "trend": 3.2
    },
    "red_flag_detection": {
      "current": 92.1,
      "trend": -0.8
    }
  },
  "time_series": [
    {"date": "2024-01-01", "score": 80.2, "evaluations": 2},
    {"date": "2024-01-02", "score": 81.5, "evaluations": 1},
    {"date": "2024-01-03", "score": 79.8, "evaluations": 3}
  ]
}

Get Score Trends

Analyze score trends over time with statistical analysis.

Query Parameters

Parameter	Type	Required	Description
`project`	string	Yes	Project ID
`evaluator`	string	No	Filter to specific evaluator
`period`	string	No	`7d`, `30d`, `90d`, `1y` (default: 30d)
`include_regression_analysis`	boolean	No	Include regression detection

Example Request

trends = client.scores.get_trends(
    project="proj_abc123",
    evaluator="triage_accuracy",
    period="90d",
    include_regression_analysis=True
)

print(f"Trend direction: {trends.direction}")
print(f"Slope: {trends.slope:+.2f}% per week")
print(f"R-squared: {trends.r_squared}")

if trends.regressions:
    print("Detected regressions:")
    for reg in trends.regressions:
        print(f"  {reg.date}: {reg.before}% → {reg.after}% ({reg.change:+.1f}%)")

curl "https://api.akhara.ai/v1/scores/trends?project=proj_abc123&evaluator=triage_accuracy&period=90d&include_regression_analysis=true" \
  -H "Authorization: Bearer rb_live_xxxxxxxx"

Response

{
  "project_id": "proj_abc123",
  "evaluator": "triage_accuracy",
  "period": "90d",
  "direction": "improving",
  "slope": 0.45,
  "slope_unit": "percent_per_week",
  "r_squared": 0.72,
  "data_points": 45,
  "current": 78.5,
  "period_start": 72.1,
  "period_end": 78.5,
  "min": 68.2,
  "max": 82.1,
  "regressions": [
    {
      "date": "2024-01-15",
      "evaluation": "eval_xyz",
      "before": 76.8,
      "after": 71.2,
      "change": -5.6,
      "severity": "moderate",
      "recovered": true,
      "recovery_date": "2024-01-18"
    }
  ],
  "time_series": [
    {"date": "2024-01-01", "score": 72.1, "sample_count": 150},
    {"date": "2024-01-08", "score": 74.5, "sample_count": 180}
  ]
}

Get Threshold Analysis

Analyze how scores compare against configurable thresholds.

Request Body

Parameter	Type	Required	Description
`project`	string	Yes	Project ID
`evaluation`	string	No	Specific evaluation (or latest)
`thresholds`	object	Yes	Threshold definitions

Example Request

analysis = client.scores.analyze_thresholds(
    project="proj_abc123",
    thresholds={
        "triage_accuracy": {
            "critical": 70,
            "warning": 80,
            "target": 90
        },
        "red_flag_detection": {
            "critical": 85,
            "warning": 95,
            "target": 99
        }
    }
)

for evaluator, result in analysis.results.items():
    print(f"{evaluator}:")
    print(f"  Current: {result.current}%")
    print(f"  Status: {result.status}")  # critical, warning, passing, exceeding
    print(f"  Gap to target: {result.gap_to_target:+.1f}%")

curl -X POST "https://api.akhara.ai/v1/scores/threshold-analysis" \
  -H "Authorization: Bearer rb_live_xxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "project": "proj_abc123",
    "thresholds": {
      "triage_accuracy": {"critical": 70, "warning": 80, "target": 90}
    }
  }'

Response

{
  "project_id": "proj_abc123",
  "evaluation_id": "eval_def456",
  "results": {
    "triage_accuracy": {
      "current": 78.5,
      "status": "warning",
      "thresholds": {
        "critical": 70,
        "warning": 80,
        "target": 90
      },
      "gap_to_target": -11.5,
      "gap_to_warning": -1.5,
      "percentile_rank": 65
    },
    "red_flag_detection": {
      "current": 92.1,
      "status": "passing",
      "thresholds": {
        "critical": 85,
        "warning": 95,
        "target": 99
      },
      "gap_to_target": -6.9,
      "percentile_rank": 78
    }
  },
  "overall_status": "warning",
  "recommendations": [
    "triage_accuracy is below warning threshold. Review recent regressions.",
    "red_flag_detection is passing but 6.9% below target."
  ]
}

Export Scores

Export scores to various formats for external analysis.

Request Body

Parameter	Type	Required	Description
`project`	string	Yes	Project ID
`evaluations`	array	No	Specific evaluations (or all)
`format`	string	Yes	`csv`, `json`, `parquet`
`include_samples`	boolean	No	Include sample-level scores

Example Request

export = client.scores.export(
    project="proj_abc123",
    format="csv",
    include_samples=True
)

export.wait()
export.download("scores_export.csv")

curl -X POST "https://api.akhara.ai/v1/scores/export" \
  -H "Authorization: Bearer rb_live_xxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "project": "proj_abc123",
    "format": "csv",
    "include_samples": true
  }'

Webhooks for Score Alerts

Configure webhooks to receive alerts when scores cross thresholds.

client.webhooks.create(
    event_types=["score.threshold_crossed"],
    url="https://your-server.com/webhook",
    config={
        "project": "proj_abc123",
        "thresholds": {
            "triage_accuracy": {"below": 75},
            "red_flag_detection": {"below": 90}
        }
    }
)

Webhook Payload

{
  "event": "score.threshold_crossed",
  "timestamp": "2024-01-15T10:45:00Z",
  "data": {
    "project": "proj_abc123",
    "evaluation": "eval_def456",
    "evaluator": "triage_accuracy",
    "score": 72.3,
    "threshold": 75,
    "direction": "below",
    "previous_score": 78.5,
    "change": -6.2
  }
}

Evaluation APIs

Create and manage evaluations

Human Review APIs

Route low-scoring samples for review

​Overview

​Get Sample Scores

​Query Parameters

​Example Request

​Response

​Get Evaluation Scores

​Query Parameters

​Example Request

​Response

​Get Project Metrics

​Query Parameters

​Example Request

​Response

​Get Score Trends

​Query Parameters

​Example Request

​Response

​Get Threshold Analysis

​Request Body

​Example Request

​Response

​Export Scores

​Request Body

​Example Request

​Webhooks for Score Alerts

​Webhook Payload

​Related

Evaluation APIs

Human Review APIs

Overview

Get Sample Scores

Query Parameters

Example Request

Response

Get Evaluation Scores

Query Parameters

Example Request

Response

Get Project Metrics

Query Parameters

Example Request

Response

Get Score Trends

Query Parameters

Example Request

Response

Get Threshold Analysis

Request Body

Example Request

Response

Export Scores

Request Body

Example Request

Webhooks for Score Alerts

Webhook Payload

Related