> ## Documentation Index
> Fetch the complete documentation index at: https://docs.akhara.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Company name is Akhara AI (never Rubric AI). Keep lowercase rubric/rubrics only when meaning grading criteria.
> Expert Review (docs path talent/) is enterprise BYO experts for audit and review: invite customer specialists; do not pitch Akhara recruiting or a public expert career portal. RLHF and domain writing are secondary work types.
> Prefer concrete API examples against public hosts: Environments eval API https://agi.akhara.ai, Control plane PDP https://api.akhara.dev, Evaluation https://app.akhara.ai / https://api.akhara.ai, Expert Review portal https://talent.akhara.ai.
> Do not invent a public hostname for private orchestrators or env API internals.
> Do not confuse control-plane latches with Environments confirmation latches.
> Environments SDK/API examples: curl against https://agi.akhara.ai. Evaluation SDK: from akhara import Akhara and AKHARA_API_KEY.
> Start with /llms.txt for the docs index and OpenAPI links; fetch individual pages as .md exports.

# Metrics Reference

> Quantitative measures for AI evaluation. Each metric is designed for interpretability and audit-ready documentation.

## Accuracy metrics

### Classification accuracy

Measures the percentage of cases where the AI assigned the correct label or level, for example the priority a support assistant assigns to tickets.

```
Classification Accuracy = (Correct Classifications) / (Total Cases) × 100%

Weighted Accuracy = Σ(weight_i × correct_i) / Σ(weight_i)

Where weights reflect the severity of each level.
```

| Metric            | Formula             | Interpretation                         |
| ----------------- | ------------------- | -------------------------------------- |
| Raw Accuracy      | Correct / Total     | Simple correctness rate                |
| Weighted Accuracy | Σ(w × correct) / Σw | Accounts for severity differences      |
| Under-rating Rate | Under-rated / Total | Cases assigned less urgent than needed |
| Over-rating Rate  | Over-rated / Total  | Cases assigned more urgent than needed |

<Warning>
  **Under-rating vs over-rating**: In most risk-bearing domains, under-rating (missing critical cases) is far more dangerous than over-rating (unnecessary escalations). Weight your metrics accordingly.
</Warning>

## Sensitivity and specificity

Critical metrics for evaluating detection of specific conditions or red flags.

```
Sensitivity (Recall) = True Positives / (True Positives + False Negatives)
"Of all actual positives, how many did we catch?"

Specificity = True Negatives / (True Negatives + False Positives)
"Of all actual negatives, how many did we correctly rule out?"

Positive Predictive Value (PPV) = TP / (TP + FP)
"When we say positive, how often are we right?"

Negative Predictive Value (NPV) = TN / (TN + FN)
"When we say negative, how often are we right?"
```

### Choosing a priority metric

| Detection target                         | Priority metric | Target | Rationale                     |
| ---------------------------------------- | --------------- | ------ | ----------------------------- |
| Account compromise                       | Sensitivity     | ≥ 99%  | Cannot miss takeovers         |
| Crisis disclosure                        | Sensitivity     | ≥ 99%  | Safety-critical detection     |
| Prohibited content                       | Sensitivity     | ≥ 98%  | Policy and legal exposure     |
| Routine follow-up                        | Specificity     | ≥ 85%  | Avoid unnecessary escalations |
| Clinical red flags (healthcare vertical) | Sensitivity     | ≥ 99%  | Time-critical intervention    |

```python title="sensitivity_calculation.py" theme={null}
from akhara import Akhara

client = Akhara()

# Get evaluation results with sensitivity/specificity breakdown
results = client.evaluations.get("eval_abc123")

for target in results.detection_targets:
    print(f"""
Target: {target.name}
  Sensitivity: {target.sensitivity:.1%}
  Specificity: {target.specificity:.1%}
  PPV: {target.ppv:.1%}
  NPV: {target.npv:.1%}
  F1 Score: {target.f1:.3f}

  Confusion Matrix:
    TP: {target.true_positives}  FP: {target.false_positives}
    FN: {target.false_negatives}  TN: {target.true_negatives}
""")
```

## Rubric-based scoring

For complex outputs like long-form responses or generated documents, rubric-based scoring provides structured evaluation across multiple dimensions.

```python title="rubric_definition.py" theme={null}
rubric = {
    "name": "Support Response Quality",
    "version": "2.1",
    "dimensions": [
        {
            "name": "Factual Accuracy",
            "weight": 0.30,
            "criteria": [
                {"score": 5, "description": "All facts accurate, appropriate terminology"},
                {"score": 4, "description": "Minor terminology issues, facts correct"},
                {"score": 3, "description": "Some inaccuracies, no user harm"},
                {"score": 2, "description": "Multiple inaccuracies, potential confusion"},
                {"score": 1, "description": "Significant errors, harmful guidance"}
            ]
        },
        {
            "name": "Completeness",
            "weight": 0.25,
            "criteria": [
                {"score": 5, "description": "All required elements present, thorough"},
                {"score": 4, "description": "Minor omissions, key info captured"},
                {"score": 3, "description": "Some gaps but usable"},
                {"score": 2, "description": "Missing important elements"},
                {"score": 1, "description": "Critically incomplete"}
            ]
        },
        {
            "name": "Policy Adherence",
            "weight": 0.25,
            "criteria": [
                {"score": 5, "description": "All policy constraints respected, clear disclosures"},
                {"score": 4, "description": "Policy respected, minor disclosure gaps"},
                {"score": 3, "description": "Basic policy compliance"},
                {"score": 2, "description": "Policy gaps present"},
                {"score": 1, "description": "Critical policy violations"}
            ]
        },
        {
            "name": "Actionability",
            "weight": 0.20,
            "criteria": [
                {"score": 5, "description": "Clear next steps, specific instructions"},
                {"score": 4, "description": "Good guidance, minor ambiguity"},
                {"score": 3, "description": "Adequate but could be clearer"},
                {"score": 2, "description": "Vague or confusing recommendations"},
                {"score": 1, "description": "No clear action items"}
            ]
        }
    ],

    "aggregate_method": "weighted_average",
    "passing_threshold": 3.5
}

# Use in evaluation
evaluation = client.evaluations.create(
    name="Support Response Evaluation",
    dataset="ds_support_responses",
    evaluators=[{
        "type": "rubric",
        "config": {"rubric": rubric}
    }]
)
```

## Custom metrics

Define metrics specific to your domain and use cases.

```python title="custom_metrics.py" theme={null}
from akhara import Metric, MetricResult

class TopicCoverageMetric(Metric):
    """
    Measures what percentage of the user's raised topics
    were addressed in the AI's response.
    """

    name = "topic_coverage"
    display_name = "Topic Coverage"
    description = "Percentage of raised topics addressed in the response"

    def calculate(self, sample) -> MetricResult:
        raised_topics = set(sample.input.get("topics", []))
        addressed_topics = set(sample.ai_output.get("addressed_topics", []))

        if not raised_topics:
            return MetricResult(
                value=1.0,
                details={"note": "No topics raised"}
            )

        coverage = len(addressed_topics & raised_topics) / len(raised_topics)
        missed = raised_topics - addressed_topics

        return MetricResult(
            value=coverage,
            details={
                "total_topics": len(raised_topics),
                "addressed": len(addressed_topics & raised_topics),
                "missed_topics": list(missed)
            },
            flags=["incomplete_response"] if coverage < 0.8 else []
        )


class TimeToResolutionMetric(Metric):
    """Measures efficiency of the agent's decision-making."""

    name = "time_to_resolution"
    display_name = "Time to Resolution Decision"
    description = "Conversation turns before a resolution decision"

    def calculate(self, sample) -> MetricResult:
        turns = sample.ai_output.get("conversation_turns", 0)
        resolution_turn = sample.ai_output.get("resolution_decision_turn", turns)

        # Benchmark: resolve within 5 turns for most cases
        efficiency_score = max(0, 1 - (resolution_turn - 3) / 10)

        return MetricResult(
            value=efficiency_score,
            details={
                "total_turns": turns,
                "resolution_turn": resolution_turn,
                "benchmark": 5
            }
        )

# Register custom metrics
client.metrics.register(TopicCoverageMetric)
client.metrics.register(TimeToResolutionMetric)
```

## Confidence intervals

All metrics include confidence intervals to quantify uncertainty, especially important for small sample sizes.

```python theme={null}
results = client.evaluations.get("eval_abc123")

print(f"""
Classification Accuracy: {results.classification_accuracy.value:.1%}
  95% CI: [{results.classification_accuracy.ci_lower:.1%}, {results.classification_accuracy.ci_upper:.1%}]
  Sample Size: {results.classification_accuracy.n}

Sensitivity (Account Compromise): {results.sensitivity_account_compromise.value:.1%}
  95% CI: [{results.sensitivity_account_compromise.ci_lower:.1%}, {results.sensitivity_account_compromise.ci_upper:.1%}]
""")
```

<Info>
  **Sample size matters**: For rare cases, you may need larger datasets to achieve narrow confidence intervals. Akhara warns when sample sizes are too small for reliable conclusions.
</Info>

## Metric aggregation

Combine multiple metrics into composite scores for overall model assessment.

| Method           | Description                     | Use Case                   |
| ---------------- | ------------------------------- | -------------------------- |
| Weighted Average | Sum of (weight × metric)        | General performance score  |
| Minimum          | Lowest individual metric        | Ensure no weak spots       |
| Geometric Mean   | ∏(metric)^(1/n)                 | Balance across metrics     |
| Threshold Gate   | Pass only if all thresholds met | Safety-critical deployment |

```python theme={null}
evaluation = client.evaluations.create(
    name="Production Readiness Check",
    dataset="ds_validation",
    evaluators=[...],

    aggregation={
        "method": "threshold_gate",
        "thresholds": {
            "classification_accuracy": {"min": 0.85},
            "sensitivity_critical": {"min": 0.95},
            "hallucination_rate": {"max": 0.01},
            "red_flag_detection": {"min": 0.98}
        },
        "require_all": True  # Must pass ALL thresholds
    }
)
```

## Exporting metrics

Export metrics in formats suitable for compliance documentation, dashboards, or CI/CD pipelines.

```python theme={null}
# Export for compliance documentation
client.evaluations.export(
    "eval_abc123",
    format="compliance_report",
    output_path="./reports/quality_review.pdf",
    include=[
        "methodology",
        "dataset_description",
        "metric_definitions",
        "results_summary",
        "confidence_intervals",
        "failure_analysis"
    ]
)

# Export for CI/CD
results = client.evaluations.export("eval_abc123", format="json")

# Use in deployment decision
if results["composite_score"] >= 0.9 and results["safety_gate"] == "pass":
    deploy_model()
else:
    block_deployment(reason=results["failure_reasons"])
```