Quickstart

Overview

This guide gets you from zero to a scored evaluation in under 5 minutes. You’ll:

Install the Akhara SDK
Get your API key
Create a sample
Run evaluators and view results in the dashboard

Prefer CI quality gates? See the CI/CD tutorial. For local and open-source tooling, see Open-source tooling.

Prerequisites

Python 3.8+ or Node.js 16+

Step 1: Install the SDK

pip install akhara

npm install @akhara/sdk

yarn add @akhara/sdk

Step 2: Get your API key

Sign up at app.akhara.ai
Navigate to Settings → API Keys
Click Create API Key
Copy your key

Keep your API key secure. Never commit it to version control or expose it in client-side code.

export AKHARA_API_KEY="your_key_here"

Step 3: Create a sample

from akhara import Akhara

client = Akhara()  # Uses AKHARA_API_KEY env var

client.samples.create(
    dataset="my-first-dataset",
    input={"query": "Cancel my order and issue a refund."},
    output={
        "response": "Done. I've cancelled the order and started your refund.",
        "actions": ["cancel_order", "issue_refund"],
    },
    expected={
        "must_include": ["cancel", "refund"],
        "must_not": ["upsell"],
    },
)

print("Sample created")

import Akhara from '@akhara/sdk';

const client = new Akhara();  // Uses AKHARA_API_KEY env var

await client.samples.create({
    dataset: "my-first-dataset",
    input: { query: "Cancel my order and issue a refund." },
    output: {
        response: "Done. I've cancelled the order and started your refund.",
        actions: ["cancel_order", "issue_refund"],
    },
    expected: {
        mustInclude: ["cancel", "refund"],
        mustNot: ["upsell"],
    },
});

console.log("Sample created");

Step 4: View in the dashboard

Open app.akhara.ai
Open your project / dataset
Inspect the sample and any automated scores

Step 5: Run an evaluation

evaluation = client.evaluations.create(
    name="My First Evaluation",
    dataset="my-first-dataset",
    evaluators=[
        {
            "type": "contains_all",
            "config": {"fields": ["must_include"]},
        },
        {
            "type": "llm_judge",
            "config": {"rubric": "task_completion_v1"},
        },
    ],
)

print(f"Evaluation started: {evaluation.id}")
results = client.evaluations.wait(evaluation.id)
print(results.metrics)

Gate PRs and deploys

Wire the same evaluation into CI with the Akhara SDK. See CI/CD evaluations.

What’s next?

Account setup

Configure your workspace, team, and settings

First evaluation

Deep dive into creating evaluations

Open-source tooling

Local suites, adaptive rubrics, dashboard source

CI/CD tutorial

Gate PRs and deploys on eval metrics

Common issues

API key not found

Set AKHARA_API_KEY, or pass it directly:

client = Akhara(api_key="your_key")

Project or dataset not found

Datasets are created on first write in many setups. Check the name for typos, or create the dataset explicitly in the dashboard.

Rate limit exceeded

Free tiers throttle burst traffic. Add retries or upgrade for higher limits.

Overview

Getting Started

Core Concepts

Onboarding Guides

Evaluation Framework

Tutorials

Integrations

Overview

Prerequisites

Step 1: Install the SDK

Step 2: Get your API key

Step 3: Create a sample

Step 4: View in the dashboard

Step 5: Run an evaluation

Gate PRs and deploys

What’s next?

Account setup

First evaluation

Open-source tooling

CI/CD tutorial

Common issues

​Overview

​Prerequisites

​Step 1: Install the SDK

​Step 2: Get your API key

​Step 3: Create a sample

​Step 4: View in the dashboard

​Step 5: Run an evaluation

​Gate PRs and deploys

​What’s next?

Account setup

First evaluation

Open-source tooling

CI/CD tutorial

​Common issues

Overview

Prerequisites

Step 1: Install the SDK

Step 2: Get your API key

Step 3: Create a sample

Step 4: View in the dashboard

Step 5: Run an evaluation

Gate PRs and deploys

What’s next?

Common issues