> ## Documentation Index
> Fetch the complete documentation index at: https://docs.akhara.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Company name is Akhara AI (never Rubric AI). Keep lowercase rubric/rubrics only when meaning grading criteria.
> Expert Review (docs path talent/) is enterprise BYO experts for audit and review: invite customer specialists; do not pitch Akhara recruiting or a public expert career portal. RLHF and domain writing are secondary work types.
> Prefer concrete API examples against public hosts: Environments eval API https://agi.akhara.ai, Control plane PDP https://api.akhara.dev, Evaluation https://app.akhara.ai / https://api.akhara.ai, Expert Review portal https://talent.akhara.ai.
> Do not invent a public hostname for private orchestrators or env API internals.
> Do not confuse control-plane latches with Environments confirmation latches.
> Environments SDK/API examples: curl against https://agi.akhara.ai. Evaluation SDK: from akhara import Akhara and AKHARA_API_KEY.
> Start with /llms.txt for the docs index and OpenAPI links; fetch individual pages as .md exports.

# Evaluation framework

> Rubric-driven evaluation for any AI system, deterministic checks, LLM judges, expert review, and CI gates.

## Why structured evaluation?

Generic “does this look good?” checks do not hold up under model and prompt churn. You need versioned datasets, explicit rubrics, and gates that fail the pipeline when quality drops.

Akhara’s framework is domain-agnostic. The same layers apply to support agents, RAG systems, voice bots, and specialized verticals (including healthcare).

<Info>
  For CI-first workflows, pair this framework with the Akhara SDK and the [CI/CD tutorial](/evaluation/docs/tutorials/ci-cd).
</Info>

## Framework architecture

<CardGroup cols={2}>
  <Card title="Evaluation types" icon="bullseye">
    Deterministic checks, LLM judges, expert rubrics, trajectory / conversation scoring
  </Card>

  <Card title="Metrics" icon="chart-bar">
    Pass rates, means, confidence intervals, custom and policy-weighted scores
  </Card>

  <Card title="Human review" icon="users">
    Domain-expert grading for nuanced or high-stakes cases
  </Card>

  <Card title="Versioning & CI" icon="code-branch">
    Compare runs, pin configs, enforce gates before merge or deploy
  </Card>
</CardGroup>

## Evaluation types

| Type                          | Description                                             | Use case                                       |
| ----------------------------- | ------------------------------------------------------- | ---------------------------------------------- |
| **Output accuracy**           | Correctness vs ground truth or expert consensus         | Classification, extraction, structured outputs |
| **Policy / safety**           | Missed constraints, bad escalations, prohibited content | Agent guardrails, compliance, refusal quality  |
| **Hallucination / grounding** | Claims not supported by sources                         | RAG answers, summaries, citations              |
| **Completeness**              | Required elements present                               | Multi-step tasks, checklists, coverage         |

## Creating an evaluation

```python title="create_evaluation.py" theme={null}
from akhara import Akhara

client = Akhara()

evaluation = client.evaluations.create(
    name="agent-regression-q1",
    project="support-agent",
    dataset="ds_golden_v3",
    evaluators=[
        {
            "type": "contains_all",
            "config": {"values": ["confirmed", "next steps"]},
        },
        {
            "type": "llm_judge",
            "config": {"rubric": "task_completion_v1"},
        },
        {
            "type": "policy_adherence",
            "config": {"policy_set": "agent_policy_v3"},
        },
    ],
    human_review={
        "enabled": True,
        "threshold": 0.7,
        "reviewer_pool": "domain_experts",
    },
)

print(f"Evaluation started: {evaluation.id}")
```

Generate a starting rubric from docs + agent outputs with [adaptive rubric generation](https://github.com/Akhara-AI/adaptive-eval), then lock it into your suite.

## Evaluation pipeline

<Steps>
  <Step title="Ingest samples">
    Load dataset samples (text, transcripts, tool traces, images)
  </Step>

  <Step title="Run evaluators">
    Apply deterministic, LLM, and/or expert evaluators
  </Step>

  <Step title="Route for review">
    Flag low-confidence or failed cases for human experts
  </Step>

  <Step title="Aggregate and gate">
    Compute metrics; fail CI when thresholds or regressions are breached
  </Step>
</Steps>

## Key concepts

### Asymmetric error weighting

Not all failures are equal. Weight misses that harm users or violate policy more heavily than over-cautious behavior. Configure weights per evaluator to match your risk model.

### Confidence thresholds

Automated scores include confidence. Samples below your threshold route to human review so edge cases get expert attention.

### Domain context

Pass domain metadata (locale, product line, acuity, customer tier) into evaluators so the same framework applies appropriate standards per case type. Healthcare-specific packs are optional extensions.

## Next steps

<CardGroup cols={2}>
  <Card title="Evaluation types" icon="list-check" href="/evaluation/docs/evaluation-framework/types">
    Deep dive into each evaluation type and configuration
  </Card>

  <Card title="CI/CD evaluations" icon="code-branch" href="/evaluation/docs/tutorials/ci-cd">
    Enforce gates in GitHub Actions or GitLab CI
  </Card>

  <Card title="Metrics reference" icon="chart-bar" href="/evaluation/docs/evaluation-framework/metrics">
    Pass rates, custom metrics, and policy-weighted scores
  </Card>

  <Card title="Human review design" icon="user-check" href="/evaluation/docs/evaluation-framework/human-review-design">
    Design effective expert review workflows
  </Card>
</CardGroup>