> ## Documentation Index
> Fetch the complete documentation index at: https://docs.akhara.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Company name is Akhara AI (never Rubric AI). Keep lowercase rubric/rubrics only when meaning grading criteria.
> Expert Review (docs path talent/) is enterprise BYO experts for audit and review: invite customer specialists; do not pitch Akhara recruiting or a public expert career portal. RLHF and domain writing are secondary work types.
> Prefer concrete API examples against public hosts: Environments eval API https://agi.akhara.ai, Control plane PDP https://api.akhara.dev, Evaluation https://app.akhara.ai / https://api.akhara.ai, Expert Review portal https://talent.akhara.ai.
> Do not invent a public hostname for private orchestrators or env API internals.
> Do not confuse control-plane latches with Environments confirmation latches.
> Environments SDK/API examples: curl against https://agi.akhara.ai. Evaluation SDK: from akhara import Akhara and AKHARA_API_KEY.
> Start with /llms.txt for the docs index and OpenAPI links; fetch individual pages as .md exports.

# Core Objects

> Understanding Akhara's fundamental data model: Datasets, Tasks, Models, Evaluations, Reviewers, and Scores.

## Overview

Akhara's data model is built around six core objects that work together to enable AI evaluation:

<CardGroup cols={3}>
  <Card title="Datasets" icon="database">
    Collections of samples for evaluation
  </Card>

  <Card title="Tasks" icon="clipboard-list">
    Individual items needing human review
  </Card>

  <Card title="Models" icon="microchip">
    AI models being evaluated
  </Card>

  <Card title="Evaluations" icon="chart-bar">
    Automated scoring runs
  </Card>

  <Card title="Reviewers" icon="user-check">
    Experts who review AI outputs
  </Card>

  <Card title="Scores" icon="star">
    Evaluation results and metrics
  </Card>
</CardGroup>

## Datasets

Datasets are collections of samples that share a common purpose, typically test sets for evaluation.

### Schema

```json theme={null}
{
  "id": "ds_abc123",
  "object": "dataset",
  "name": "Support Golden Set Q1 2025",
  "description": "Quarterly evaluation test set",
  "project": "proj_xyz789",
  "sample_count": 247,
  "created_at": "2025-01-15T10:30:00Z",
  "updated_at": "2025-01-15T14:22:00Z",
  "tags": ["golden-set", "quarterly"],
  "metadata": {
    "annotator": "qa_team",
    "version": "3"
  }
}
```

### Usage

```python theme={null}
# Create a dataset
dataset = client.datasets.create(
    name="Support Golden Set Q1 2025",
    description="Quarterly evaluation test set",
    project="support-agent",
    tags=["golden-set", "quarterly"]
)

# List datasets
datasets = client.datasets.list(project="support-agent")

# Get dataset details
dataset = client.datasets.get("ds_abc123")

# Update dataset
client.datasets.update(
    "ds_abc123",
    description="Updated description"
)

# Delete dataset
client.datasets.delete("ds_abc123")
```

### Samples

Samples are the individual data points within a dataset:

```json theme={null}
{
  "id": "smp_def456",
  "object": "sample",
  "dataset": "ds_abc123",
  "input": {
    "transcript": [...],
    "audio_url": "s3://bucket/call.wav"
  },
  "output": {
    "priority": "high",
    "topics": ["billing_dispute"]
  },
  "expected": {
    "priority": "critical"
  },
  "metadata": {
    "conversation_id": "conv_123"
  },
  "created_at": "2025-01-15T10:30:00Z"
}
```

## Tasks

Tasks represent individual items requiring human review. They're created automatically when AI outputs need expert oversight.

### Schema

```json theme={null}
{
  "id": "task_ghi789",
  "object": "task",
  "type": "output_review",
  "status": "pending",
  "priority": "high",
  "sample": "smp_def456",
  "project": "proj_xyz789",
  "assigned_to": null,
  "created_at": "2025-01-15T10:30:00Z",
  "due_at": "2025-01-15T18:00:00Z",
  "flag_reason": "low_confidence",
  "metadata": {
    "confidence_score": 0.72,
    "red_flags_detected": true
  }
}
```

### Task Status Flow

```
pending → assigned → in_progress → completed
                  ↘            ↗
                   → skipped →
```

| Status        | Description                   |
| ------------- | ----------------------------- |
| `pending`     | Awaiting assignment           |
| `assigned`    | Assigned to a reviewer        |
| `in_progress` | Reviewer is actively working  |
| `completed`   | Review submitted              |
| `skipped`     | Reviewer skipped (reassigned) |

### Usage

```python theme={null}
# List pending tasks
tasks = client.tasks.list(
    project="support-agent",
    status="pending"
)

# Get task details
task = client.tasks.get("task_ghi789")

# Assign task to reviewer
client.tasks.assign(
    "task_ghi789",
    reviewer="rev_jkl012"
)

# Complete a task with review
client.tasks.complete(
    "task_ghi789",
    review={
        "priority_correct": False,
        "correct_priority": "critical",
        "notes": "Missed outage indicators in the transcript"
    }
)
```

## Models

Models represent the AI systems being evaluated. Track different versions and configurations.

### Schema

```json theme={null}
{
  "id": "mod_mno345",
  "object": "model",
  "name": "Support Voice Agent",
  "description": "Production support model",
  "project": "proj_xyz789",
  "versions": [
    {
      "version": "v2.3.1",
      "created_at": "2025-01-10T10:00:00Z",
      "config": {
        "temperature": 0.3,
        "max_tokens": 1024
      }
    },
    {
      "version": "v2.3.0",
      "created_at": "2025-01-01T10:00:00Z"
    }
  ],
  "current_version": "v2.3.1",
  "created_at": "2024-06-15T10:00:00Z"
}
```

### Usage

```python theme={null}
# Register a model
model = client.models.register(
    name="Support Voice Agent",
    project="support-agent",
    version="v2.3.1",
    config={
        "temperature": 0.3,
        "max_tokens": 1024,
        "base_model": "claude-sonnet-4-20250514"
    }
)

# List models
models = client.models.list(project="support-agent")

# Get model details
model = client.models.get("mod_mno345")

# Add new version
client.models.add_version(
    "mod_mno345",
    version="v2.4.0",
    config={...}
)
```

## Evaluations

Evaluations are automated scoring runs that assess AI performance against a dataset.

### Schema

```json theme={null}
{
  "id": "eval_pqr678",
  "object": "evaluation",
  "name": "Weekly Support Agent Evaluation",
  "status": "completed",
  "project": "proj_xyz789",
  "dataset": "ds_abc123",
  "model": "mod_mno345",
  "evaluators": [
    {"type": "classification_accuracy", "config": {...}},
    {"type": "red_flag_detection", "config": {...}}
  ],
  "progress": {
    "total": 247,
    "completed": 247,
    "failed": 3
  },
  "metrics": {
    "classification_accuracy": 0.89,
    "red_flag_recall": 0.97,
    "under_rating_rate": 0.02
  },
  "created_at": "2025-01-15T10:30:00Z",
  "started_at": "2025-01-15T10:30:05Z",
  "completed_at": "2025-01-15T10:35:22Z"
}
```

### Evaluation Status Flow

```
pending → running → completed
              ↘         ↗
               → failed
              ↘
               → cancelled
```

### Usage

```python theme={null}
# Create evaluation
evaluation = client.evaluations.create(
    name="Weekly Support Agent Evaluation",
    project="support-agent",
    dataset="ds_abc123",
    evaluators=[
        {"type": "classification_accuracy"},
        {"type": "red_flag_detection"}
    ]
)

# Check status
status = client.evaluations.get_status(evaluation.id)

# Wait for completion
result = client.evaluations.wait(evaluation.id)

# Get detailed results
results = client.evaluations.get(evaluation.id)

# Get per-sample results
samples = client.evaluations.get_samples(evaluation.id)

# Compare evaluations
comparison = client.evaluations.compare([eval1.id, eval2.id])
```

## Reviewers

Reviewers are domain experts who provide human oversight on AI outputs.

### Schema

```json theme={null}
{
  "id": "rev_stu901",
  "object": "reviewer",
  "user": "user_abc123",
  "name": "Sarah Chen",
  "email": "sarah.chen@example.com",
  "credentials": [
    {
      "type": "senior_engineer",
      "issuer": "employer_verification",
      "verified": true,
      "expires_at": "2026-12-31"
    }
  ],
  "specialties": ["api_integrations", "billing_systems"],
  "capacity": {
    "max_daily_tasks": 50,
    "current_assigned": 12
  },
  "stats": {
    "total_reviews": 1247,
    "avg_review_time_seconds": 180,
    "agreement_rate": 0.94
  },
  "created_at": "2024-06-15T10:00:00Z"
}
```

### Credential Types

Credentials are configurable per domain. Examples:

| Domain                         | Example credential              | Can Review                            |
| ------------------------------ | ------------------------------- | ------------------------------------- |
| Software                       | Senior engineer, staff engineer | Code review, tool-use trajectories    |
| Support                        | Team lead, QA specialist        | Response quality, policy adherence    |
| Finance                        | CPA, licensed adviser           | Advice accuracy, compliance           |
| Legal                          | Attorney, paralegal             | Contract analysis, regulatory content |
| Healthcare (optional vertical) | MD, NP, RN                      | Clinical decisions, documentation     |

### Usage

```python theme={null}
# Add a reviewer
reviewer = client.reviewers.create(
    user="user_abc123",
    credentials=[{
        "type": "senior_engineer",
        "issuer": "employer_verification"
    }],
    specialties=["api_integrations"]
)

# List reviewers
reviewers = client.reviewers.list(
    credential_type="senior_engineer",
    available=True
)

# Get reviewer stats
stats = client.reviewers.get_stats("rev_stu901")

# Update capacity
client.reviewers.update(
    "rev_stu901",
    capacity={"max_daily_tasks": 75}
)
```

## Scores

Scores are the evaluation results, both from automated evaluators and human reviewers.

### Schema

```json theme={null}
{
  "id": "scr_vwx234",
  "object": "score",
  "sample": "smp_def456",
  "evaluation": "eval_pqr678",
  "source": "evaluator",
  "evaluator_type": "classification_accuracy",
  "value": 0.0,
  "passed": false,
  "details": {
    "predicted": "high",
    "expected": "critical",
    "error_type": "under_rating"
  },
  "created_at": "2025-01-15T10:32:00Z"
}
```

### Score Sources

| Source      | Description                        |
| ----------- | ---------------------------------- |
| `evaluator` | Automated evaluation score         |
| `reviewer`  | Human reviewer score               |
| `consensus` | Aggregated from multiple reviewers |

### Usage

```python theme={null}
# Get scores for an evaluation
scores = client.scores.list(evaluation="eval_pqr678")

# Get scores for a sample
scores = client.scores.list(sample="smp_def456")

# Get score details
score = client.scores.get("scr_vwx234")

# Aggregate scores
aggregates = client.scores.aggregate(
    evaluation="eval_pqr678",
    group_by="evaluator_type"
)
```

## Object Relationships

```mermaid theme={null}
flowchart TB
    Project --> Datasets
    Datasets --> Samples
    
    Project --> Models
    Models --> Versions
    
    Project --> Evaluations
    Evaluations --> Scores
    Evaluations -.->|reference| Samples
    
    Project --> Tasks
    Tasks -.->|reference| Samples
    Tasks -.->|assignment| Reviewers
    
    Reviewers --> Reviews
    Reviews --> Scores
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Evaluation Lifecycle" icon="arrows-spin" href="/evaluation/docs/core-concepts/evaluation-lifecycle">
    Understand evaluation states and triggers
  </Card>

  <Card title="Human vs Automated" icon="users" href="/evaluation/docs/core-concepts/human-vs-automated">
    Learn about review workflows
  </Card>
</CardGroup>