> ## Documentation Index
> Fetch the complete documentation index at: https://docs.akhara.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Company name is Akhara AI (never Rubric AI). Keep lowercase rubric/rubrics only when meaning grading criteria.
> Expert Review (docs path talent/) is enterprise BYO experts for audit and review: invite customer specialists; do not pitch Akhara recruiting or a public expert career portal. RLHF and domain writing are secondary work types.
> Prefer concrete API examples against public hosts: Environments eval API https://agi.akhara.ai, Control plane PDP https://api.akhara.dev, Evaluation https://app.akhara.ai / https://api.akhara.ai, Expert Review portal https://talent.akhara.ai.
> Do not invent a public hostname for private orchestrators or env API internals.
> Do not confuse control-plane latches with Environments confirmation latches.
> Environments SDK/API examples: curl against https://agi.akhara.ai. Evaluation SDK: from akhara import Akhara and AKHARA_API_KEY.
> Start with /llms.txt for the docs index and OpenAPI links; fetch individual pages as .md exports.

# Platform capabilities

> Datasets, rubrics, CI/CD gates, expert review, and a hosted dashboard, domain-agnostic evaluation for any AI system.

## Evaluation stack

<CardGroup cols={3}>
  <Card title="Hosted dashboard" icon="chart-line" href="https://app.akhara.ai">
    Projects, cases, rubrics, and reviewer workflows at [app.akhara.ai](https://app.akhara.ai)
  </Card>

  <Card title="Akhara SDK & API" icon="terminal" href="/evaluation/docs/getting-started/quickstart">
    Datasets, evaluations, scores, and CI gates via `pip install akhara`
  </Card>

  <Card title="CI quality gates" icon="code-branch" href="/evaluation/docs/tutorials/ci-cd">
    Gate PRs and deploys with versioned suites and gate scripts
  </Card>
</CardGroup>

## CI/CD evaluations

First-class quality gates for pull requests and deploys, not an afterthought.

| Capability            | Description                                                                   |
| --------------------- | ----------------------------------------------------------------------------- |
| Config-as-code suites | Versioned YAML/JSON eval definitions in your repo                             |
| SDK + gate script     | Non-zero exit codes when thresholds fail (`python scripts/run_evaluation.py`) |
| Hosted CI mode        | API evaluations tagged for pipelines; wait on automated scoring only          |
| Regression checks     | Compare candidate vs production baseline; cap allowed drop per metric         |
| PR reporting          | Post scorecards to GitHub / GitLab merge requests                             |
| Artifact archives     | Persist results JSON for audit and dashboards                                 |

See the [CI/CD tutorial](/evaluation/docs/tutorials/ci-cd) for GitHub Actions and GitLab examples.

## Data modalities

The core model is modality-agnostic (input / output / expected / metadata). Common surfaces:

<CardGroup cols={3}>
  <Card title="Text & structured" icon="file-lines">
    Chat turns, tool calls, JSON outputs, RAG answers
  </Card>

  <Card title="Voice & audio" icon="microphone">
    Recordings, speaker-labeled transcripts, live streams
  </Card>

  <Card title="Images & docs" icon="image">
    Documents, screenshots, and domain imaging where needed
  </Card>
</CardGroup>

Healthcare-specific schemas (clinical notes, DICOM) remain available as vertical extensions.

## Evaluation framework

### Evaluation types

<AccordionGroup>
  <Accordion title="Deterministic checks" icon="bullseye">
    Exact match, JSON schema, regex, numeric tolerance, required phrases, code execution.
  </Accordion>

  <Accordion title="LLM judges" icon="microchip">
    Structured-output scoring with portable providers (via litellm-style routing).
  </Accordion>

  <Accordion title="Expert rubrics" icon="clipboard-check">
    Export/import human scoring tasks with rubric versioning and preference pairs.
  </Accordion>

  <Accordion title="Adaptive rubrics" icon="wand-magic-sparkles">
    Generate scoring dimensions from your knowledge base, then reuse the rubric across runs.
  </Accordion>

  <Accordion title="Trajectory / conversation" icon="route">
    Multi-turn and tool-path checks for agents.
  </Accordion>
</AccordionGroup>

### Metrics

| Metric style     | Description                                               |
| ---------------- | --------------------------------------------------------- |
| Pass rate / mean | Per-evaluator aggregates with optional bootstrap CIs      |
| Custom metrics   | Define domain-specific scores and weights                 |
| Safety / policy  | Asymmetric penalties (e.g. under-refusal vs over-refusal) |
| Cost & latency   | Judge cost tracking alongside quality                     |

### Human review

* Review templates and multi-criteria grading rubrics
* Annotation (highlight, comment, label)
* Side-by-side comparison of model output vs source
* Multi-reviewer consensus, adjudication, inter-rater reliability

### Versioning and experiments

* Version history and rollback for evaluation configs
* A/B comparison of model / prompt / rubric variants
* Deterministic sampling, input hashing, environment pinning

```python experiments.py theme={null}
experiment = client.experiments.create(
    name="agent-v2-vs-v3",
    project="support-agent",
    dataset="ds_golden_test",
    variants=[
        {"name": "v2-baseline", "model": "agent-v2"},
        {"name": "v3-candidate", "model": "agent-v3"},
    ],
    evaluators=[
        {"type": "task_completion"},
        {"type": "policy_adherence"},
        {"type": "latency"},
    ],
    significance_level=0.05,
    min_sample_size=500,
)

results = client.experiments.get_results(experiment.id)
print(results.winner, results.improvement_pct, results.is_significant)
```

## Observability

<CardGroup cols={2}>
  <Card title="Structured logging" icon="database">
    Log inputs, outputs, and metadata with stable schemas
  </Card>

  <Card title="Dashboard" icon="chart-line">
    Monitor metrics, error rates, and trends at [app.akhara.ai](https://app.akhara.ai)
  </Card>

  <Card title="Alerting" icon="bell">
    Notify when metrics degrade or gates would fail in production traffic
  </Card>

  <Card title="Tracing" icon="route">
    Follow multi-step agent pipelines end to end
  </Card>
</CardGroup>

## Expert network

Route outputs to credentialed domain experts when automated scores are not enough, specialty matching, workload balancing, conflict-of-interest controls, and calibration against gold sets. Healthcare specialties are available; the same workflows apply to other regulated or high-stakes domains.

## Security & compliance

<CardGroup cols={2}>
  <Card title="Enterprise controls" icon="shield-check">
    RBAC, SSO, MFA, audit logs; BAA / HIPAA program in progress
  </Card>

  <Card title="SOC 2 Type II" icon="certificate">
    Audit program in progress; security controls in place
  </Card>

  <Card title="Encryption" icon="lock">
    AES-256 at rest, TLS 1.3 in transit
  </Card>

  <Card title="Data residency" icon="globe">
    US, EU, or custom regions; configurable retention
  </Card>
</CardGroup>

## Integrations

<CardGroup cols={3}>
  <Card title="LLM providers" icon="microchip">
    OpenAI, Anthropic, Azure, AWS Bedrock, and more
  </Card>

  <Card title="CI/CD" icon="code-branch">
    GitHub Actions, GitLab CI, Jenkins
  </Card>

  <Card title="Monitoring" icon="chart-mixed">
    Datadog, Grafana, PagerDuty
  </Card>

  <Card title="Voice platforms" icon="phone">
    Twilio, Vonage, Amazon Connect
  </Card>

  <Card title="Webhooks" icon="webhook">
    Push evaluation events into your systems
  </Card>

  <Card title="Warehouses" icon="database">
    Export scores and provenance for analytics
  </Card>
</CardGroup>

## Next steps

<CardGroup cols={2}>
  <Card title="Quickstart" icon="play" href="/evaluation/docs/getting-started/quickstart">
    Install the SDK and run your first evaluation
  </Card>

  <Card title="CI/CD tutorial" icon="code-branch" href="/evaluation/docs/tutorials/ci-cd">
    Wire gates into your pipeline
  </Card>
</CardGroup>