Evaluation framework

Why structured evaluation?

Generic “does this look good?” checks do not hold up under model and prompt churn. You need versioned datasets, explicit rubrics, and gates that fail the pipeline when quality drops. Akhara’s framework is domain-agnostic. The same layers apply to support agents, RAG systems, voice bots, and specialized verticals (including healthcare).

For CI-first workflows, pair this framework with the Akhara SDK and the CI/CD tutorial.

Framework architecture

Evaluation types

Deterministic checks, LLM judges, expert rubrics, trajectory / conversation scoring

Metrics

Pass rates, means, confidence intervals, custom and policy-weighted scores

Human review

Domain-expert grading for nuanced or high-stakes cases

Versioning & CI

Compare runs, pin configs, enforce gates before merge or deploy

Evaluation types

Type	Description	Use case
Output accuracy	Correctness vs ground truth or expert consensus	Classification, extraction, structured outputs
Policy / safety	Missed constraints, bad escalations, prohibited content	Agent guardrails, compliance, refusal quality
Hallucination / grounding	Claims not supported by sources	RAG answers, summaries, citations
Completeness	Required elements present	Multi-step tasks, checklists, coverage

Creating an evaluation

create_evaluation.py

from akhara import Akhara

client = Akhara()

evaluation = client.evaluations.create(
    name="agent-regression-q1",
    project="support-agent",
    dataset="ds_golden_v3",
    evaluators=[
        {
            "type": "contains_all",
            "config": {"values": ["confirmed", "next steps"]},
        },
        {
            "type": "llm_judge",
            "config": {"rubric": "task_completion_v1"},
        },
        {
            "type": "policy_adherence",
            "config": {"policy_set": "agent_policy_v3"},
        },
    ],
    human_review={
        "enabled": True,
        "threshold": 0.7,
        "reviewer_pool": "domain_experts",
    },
)

print(f"Evaluation started: {evaluation.id}")

Generate a starting rubric from docs + agent outputs with adaptive rubric generation, then lock it into your suite.

Evaluation pipeline

Ingest samples

Load dataset samples (text, transcripts, tool traces, images)

Run evaluators

Apply deterministic, LLM, and/or expert evaluators

Route for review

Flag low-confidence or failed cases for human experts

Aggregate and gate

Compute metrics; fail CI when thresholds or regressions are breached

Key concepts

Asymmetric error weighting

Not all failures are equal. Weight misses that harm users or violate policy more heavily than over-cautious behavior. Configure weights per evaluator to match your risk model.

Confidence thresholds

Automated scores include confidence. Samples below your threshold route to human review so edge cases get expert attention.

Domain context

Pass domain metadata (locale, product line, acuity, customer tier) into evaluators so the same framework applies appropriate standards per case type. Healthcare-specific packs are optional extensions.

Next steps

Evaluation types

Deep dive into each evaluation type and configuration

CI/CD evaluations

Enforce gates in GitHub Actions or GitLab CI

Open-source tooling

Local suites, adaptive rubrics, dashboard source

Human review design

Design effective expert review workflows

Overview

Getting Started

Core Concepts

Onboarding Guides