Platform capabilities

Evaluation stack

Hosted dashboard

Projects, cases, rubrics, and reviewer workflows at app.akhara.ai

Akhara SDK & API

Datasets, evaluations, scores, and CI gates via pip install akhara

Open-source tooling

Local suites, adaptive rubric generation, and dashboard source

CI/CD evaluations

First-class quality gates for pull requests and deploys, not an afterthought.

Capability	Description
Config-as-code suites	Versioned YAML/JSON eval definitions in your repo
SDK + gate script	Non-zero exit codes when thresholds fail (`python scripts/run_evaluation.py`)
Hosted CI mode	API evaluations tagged for pipelines; wait on automated scoring only
Regression checks	Compare candidate vs production baseline; cap allowed drop per metric
PR reporting	Post scorecards to GitHub / GitLab merge requests
Artifact archives	Persist results JSON for audit and dashboards

See the CI/CD tutorial for GitHub Actions and GitLab examples.

Data modalities

The core model is modality-agnostic (input / output / expected / metadata). Common surfaces:

Text & structured

Chat turns, tool calls, JSON outputs, RAG answers

Voice & audio

Recordings, speaker-labeled transcripts, live streams

Images & docs

Documents, screenshots, and domain imaging where needed

Healthcare-specific schemas (clinical notes, DICOM) remain available as vertical extensions. See Notes and Imaging sections.

Evaluation framework

Evaluation types

Deterministic checks

Exact match, JSON schema, regex, numeric tolerance, required phrases, code execution.

LLM judges

Structured-output scoring with portable providers (via litellm-style routing).

Expert rubrics

Export/import human scoring tasks with rubric versioning and preference pairs.

Adaptive rubrics

Generate scoring dimensions from your knowledge base, then reuse the rubric. See open-source tooling.

Trajectory / conversation

Multi-turn and tool-path checks for agents.

Metrics

Metric style	Description
Pass rate / mean	Per-evaluator aggregates with optional bootstrap CIs
Custom metrics	Define domain-specific scores and weights
Safety / policy	Asymmetric penalties (e.g. under-refusal vs over-refusal)
Cost & latency	Judge cost tracking alongside quality

Human review

Review templates and multi-criteria grading rubrics
Annotation (highlight, comment, label)
Side-by-side comparison of model output vs source
Multi-reviewer consensus, adjudication, inter-rater reliability

Versioning and experiments

Version history and rollback for evaluation configs
A/B comparison of model / prompt / rubric variants
Deterministic sampling, input hashing, environment pinning

experiments.py

experiment = client.experiments.create(
    name="agent-v2-vs-v3",
    project="support-agent",
    dataset="ds_golden_test",
    variants=[
        {"name": "v2-baseline", "model": "agent-v2"},
        {"name": "v3-candidate", "model": "agent-v3"},
    ],
    evaluators=[
        {"type": "task_completion"},
        {"type": "policy_adherence"},
        {"type": "latency"},
    ],
    significance_level=0.05,
    min_sample_size=500,
)

results = client.experiments.get_results(experiment.id)
print(results.winner, results.improvement_pct, results.is_significant)

Observability

Structured logging

Log inputs, outputs, and metadata with stable schemas

Dashboard

Monitor metrics, error rates, and trends at app.akhara.ai

Alerting

Notify when metrics degrade or gates would fail in production traffic

Tracing

Follow multi-step agent pipelines end to end

Expert network

Route outputs to credentialed domain experts when automated scores are not enough, specialty matching, workload balancing, conflict-of-interest controls, and calibration against gold sets. Healthcare specialties are available; the same workflows apply to other regulated or high-stakes domains.

Security & compliance

Enterprise controls

RBAC, SSO, MFA, audit logs; BAA / HIPAA where required

SOC 2 Type II

Annual audits and security controls

Encryption

AES-256 at rest, TLS 1.3 in transit

Data residency

US, EU, or custom regions; configurable retention

Integrations

LLM providers

OpenAI, Anthropic, Azure, AWS Bedrock, and more

CI/CD

GitHub Actions, GitLab CI, Jenkins

Monitoring

Datadog, Grafana, PagerDuty

Voice platforms

Twilio, Vonage, Amazon Connect

Webhooks

Push evaluation events into your systems

Warehouses

Export scores and provenance for analytics

Next steps

Open-source tooling

Local suites, adaptive rubrics, dashboard source

CI/CD tutorial

Wire gates into your pipeline

​Evaluation stack

Hosted dashboard

Akhara SDK & API

Open-source tooling

​CI/CD evaluations

​Data modalities

Text & structured

Voice & audio

Images & docs

​Evaluation framework

​Evaluation types

​Metrics

​Human review

​Versioning and experiments

​Observability

Structured logging

Dashboard

Alerting

Tracing

​Expert network

​Security & compliance

Enterprise controls

SOC 2 Type II

Encryption

Data residency

​Integrations

LLM providers

CI/CD

Monitoring

Voice platforms

Webhooks

Warehouses

​Next steps

Open-source tooling

CI/CD tutorial

Evaluation stack

CI/CD evaluations

Data modalities

Evaluation framework

Evaluation types

Metrics

Human review

Versioning and experiments

Observability

Expert network

Security & compliance

Integrations

Next steps