Open-source tooling

Akhara Evaluation is a stack, hosted product surfaces plus open-source tools you can run locally or inspect as source.

Components

Component	Role	Surface / source
Eval dashboard	Projects, cases, rubrics, reviewer performance, run detail	app.akhara.ai · source
Eval framework	Datasets, deterministic checks, LLM judges, expert rubric workflows for local and CI suites	source
Adaptive rubric generation	Infer domain themes from a knowledge folder and agent outputs, emit a tailored rubric, then score	source

Knowledge / golden data ──▶ adaptive rubric gen (optional)
                                    │
                                    ▼
                         Akhara eval suite + gates
                                    │
                    ┌───────────────┼───────────────┐
                    ▼               ▼               ▼
              Local scripts    CI quality gate    Hosted API / dashboard
              (SDK / configs)  (PR / deploy)     (app.akhara.ai)

Hosted path (recommended)

To get started, use the Akhara SDK and dashboard:

pip install akhara
export AKHARA_API_KEY="your_key_here"

from akhara import Akhara

client = Akhara()
evaluation = client.evaluations.create(
    name="agent-regression",
    dataset="golden-set",
    evaluators=[
        {"type": "contains_all", "config": {"values": ["confirmed", "next steps"]}},
        {"type": "llm_judge", "config": {"rubric": "task_completion_v1"}},
    ],
    ci_mode=True,
)
results = client.evaluations.wait(evaluation.id, stage="automated")
print(results.metrics)

Full pipeline wiring: CI/CD evaluations.

Local and CI suites

Use configuration-driven suites when you want versioned YAML in git, metric thresholds, and a non-zero exit when gates fail. In CI, install akhara and run a small script that creates an evaluation, waits on automated scoring, and exits non-zero if gates fail:

evaluations/regression_gate.yaml

name: Agent Regression Gate
version: 1.0.0

evaluators:
  - type: contains_all
    config:
      values: ["confirmed", "next steps"]
  - type: llm_judge
    config:
      rubric: task_completion_v1

gates:
  - name: contains_required
    metric: contains_all
    operator: gte
    threshold: 0.9
  - name: task_completion
    metric: task_completion
    operator: gte
    threshold: 0.85

pip install akhara
python scripts/run_evaluation.py \
  --config evaluations/regression_gate.yaml \
  --dataset golden-set \
  --output results.json

See the full script and GitHub Actions / GitLab examples in the CI/CD tutorial. Open-source framework source lives at Akhara-AI/rubric-eval.

Adaptive rubric generation

Use when you do not yet have a hand-authored rubric. Point the tool at a knowledge folder and agent answers, it detects domain themes, generates a structured rubric, then scores outputs. Source: Akhara-AI/adaptive-eval. Clone the repo and follow its README for install and run commands. Typical inputs: a knowledge directory, agent answer file, and model choice. Outputs include dimension scores, per-criterion justifications, letter grades (penalty-based), and aggregate weakest/strongest areas. Rubrics follow a fixed JSON schema so results stay comparable across providers (litellm).

Eval dashboard

The dashboard is the review and operations surface:

Project / case organization
Upload → transcribe → evaluate pipelines (where applicable)
Rubric bank and audit-task flows
Reviewer metrics and case-level drill-down

Production app: https://app.akhara.ai
Source: https://github.com/Akhara-AI/rubric-app

When to use which

Goal	Start with
Block a bad PR with metric gates	Akhara SDK + `scripts/run_evaluation.py` in CI
Bootstrap a domain rubric from docs + agent outputs	Adaptive rubric generation
Review cases, manage rubrics, track reviewer quality	Eval dashboard
Persist datasets / evaluations / human review at org scale	Hosted API + dashboard

Overview

Getting Started

Core Concepts

Onboarding Guides

Evaluation Framework

Tutorials

Integrations

Open-source tooling

Components

Hosted path (recommended)

Local and CI suites

Adaptive rubric generation

Eval dashboard

When to use which

Next steps

CI/CD tutorial

Quickstart

​Components

​Hosted path (recommended)

​Local and CI suites

​Adaptive rubric generation

​Eval dashboard

​When to use which

​Next steps

CI/CD tutorial

Quickstart

Components

Hosted path (recommended)

Local and CI suites

Adaptive rubric generation

Eval dashboard

When to use which

Next steps