Scores and stats

Live evaluation scores

After POST /evaluations/run, poll:

GET /evaluations/run/{run_id}/status

Per-rollout fields:

Field	Meaning
`reward`	Terminal hybrid score: `+1` pass, `0` incomplete, `-1` fail
`task_passed`	Boolean derived from verifiers / stop
`task_outcome`	Human-readable outcome (`pass`, `fail`, …)
`verifiers`	Final VU / Room objects
`is_truncated`	Hit `max_steps` without clean terminal
`stop_condition`	e.g. `terminal_action`
`metrics`	Extra counters (do not confuse shaping with terminal reward)

List historical evals:

GET /evaluations?environment_id=amazon_gym_android&limit=20

Trajectory data

Need	Endpoint
Full step list	`GET /dashboard/runs/{run_id}/rollouts/{rollout_id}`
Step screenshot	`GET /dashboard/runs/{run_id}/rollouts/{rollout_id}/steps/{i}/screenshot`
Run + all rollouts	`GET /dashboard/runs/{run_id}`
Episode zip (post-sync)	`GET /v1/export/run/{episode_id}` on private orchestrator

rollout_id = {run_id}_{example_idx}_{rollout_idx}. On-disk capture (operator side) also lands under outputs/runs/{run_id}/ when the wrapper is configured to write run capture.

Aggregate stats

curl -s https://agi.akhara.ai/dashboard/stats

{
  "total_runs": 42,
  "active_runs": 1,
  "total_rollouts": 500,
  "completed_rollouts": 480,
  "failed_rollouts": 12,
  "success_rate": 0.3125,
  "avg_reward": 0.05
}

Pass@N

GET /dashboard/runs/{run_id}/pass_at_n
GET /dashboard/tasks/pass_at_n_calibration

Use these when rollouts_per_example > 1.

Offline sample pack metrics

The shopping-environment sample pack ships precomputed metrics:

File	Contents
`metrics/summary.json`	`pass_rate`, `outcome_counts`, `family_counts`
`metrics/task_results.csv`	Per-episode table
`episode_index.jsonl`	One row per episode
`trajectories/.../metadata.json`	`terminal_reward`, `final_verifier_verdicts`

Example summary.json:

{
  "episodes": 12,
  "tasks": 12,
  "pass_rate": 0.25,
  "failure_rate": 0.75,
  "outcome_counts": { "FAIL": 9, "PASS": 3 },
  "family_counts": {
    "checkout": 5,
    "reasoning_checkout": 2,
    "search_add_checkout": 5
  }
}

Interpreting a PASS

A checkout PASS requires more than reaching ORDER_CONFIRMATION:

Primary VU (e.g. VU-CHECKOUT) status PASS
ROOM_DEVICE_DB PASS when hybrid Room scoring is on
Confirmation / preselection gates satisfied

See Verifiers and the worked example in Sample trajectories.

Getting started

Mechanics

API reference

Scores and trajectories

Scores and stats

Live evaluation scores

Trajectory data

Aggregate stats

Pass@N

Offline sample pack metrics

Interpreting a PASS

​Live evaluation scores

​Trajectory data

​Aggregate stats

​Pass@N

​Offline sample pack metrics

​Interpreting a PASS

Live evaluation scores

Trajectory data

Aggregate stats

Pass@N

Offline sample pack metrics

Interpreting a PASS