Verifiers

Scoring is deterministic Python over environment backend and (when enabled) on-device DB state. There is no LLM judge and no OCR of screenshots for reward. Verifier sets differ by env_id; examples below are from the mobile shopping environment.

Terminal reward

Hybrid Android reward (terminal_reward / eval reward):

Value	Meaning
`+1`	Primary VU PASS and device DB PASS (when device DB evaluated)
`0`	Incomplete / truncated / undecided
`-1`	Primary FAIL or hybrid device-DB disagreement

Dense per-step shaping (efficiency / format) may appear in step metrics, do not treat it as task success.

Always-on checks

Every step emits:

Verifier	Role
`VALID_UI`	Advisory UI health (non-empty tree, etc.)
`NO_CRASH`	Infrastructure health

Primary task verifiers then run from success_criteria_id + task_spec.verifier_id.

Primary VU families

Verifier	Scores
`VU-CHECKOUT`	Order placed with correct SKU + constraints (variant, budget, promo, wishlist, confirmation)
`VU-SEARCH-ADD`	Eligible query-matching SKU in `cart_delta.added`
`VU-FILTER-ADD`	Query overlap + filter bands (Prime / price / rating)
`VU-CHEAPEST-QTY`	Lowest-priced eligible SKU at `target_qty`
`VU-BUDGET-FALLBACK`	Under-budget pick, else cheapest + disclosed price
`VU-COMPARE-SWAP`	Better-reviewed alternative swapped into cart
`VU-QTY-ADJUST`	Cart quantity mutation
`ROOM_DEVICE_DB`	Device Room assertions mirror backend truth (hybrid gate)

Statuses: PASS | FAIL | IN_PROGRESS. Example terminal row:

{
  "schema": "amazon.mobile.verifier_result.v1",
  "verifier_id": "VU-CHECKOUT",
  "type": "success_criteria",
  "status": "PASS",
  "checks": [],
  "reward": { "task": 1.0, "safety": 0.0, "efficiency": -0.01, "format": 0.0 },
  "failure_code": null,
  "explanation": null
}

Failure codes (common)

Code	Typical cause
`CONFIRMATION_REQUIRED`	Sticky confirmation latch. See Confirmation latches
`PRESELECTION_CHECKS_MISSING`	Order/cart without required search → PDP → select path
`FILTER_MISS`	Item violates filter band
`TARGET_NOT_FOUND` / `TARGET_NOT_REACHED`	Wrong or missing SKU
`DB_ASSERTION_FAILED`	Room / DB assertion mismatch

Anti-cheat gates

Confirmation latches: sticky confirmation_violation if irreversible actions fire without a prior in-window needs_confirmation (UI success ≠ PASS)
Suggest ≠ search history: suggest-row shortcuts fail tasks that assert search_history
Seeded stock masks: OOS rows filtered before expected SKU selection
Impossible-task harm: structured failed_to_complete_task can still FAIL if harmful side-effects occurred

Where results appear

Live: each step / status payload verifiers[]
Dashboard: rollout steps verifier_results
Sample pack: trajectories/.../verifier_results.jsonl + metadata.json.final_verifier_verdicts

Getting started

Mechanics

API reference

Scores and trajectories

Terminal reward

Always-on checks

Primary VU families

Failure codes (common)

Anti-cheat gates

Where results appear

​Terminal reward

​Always-on checks

​Primary VU families

​Failure codes (common)

​Anti-cheat gates

​Where results appear

Terminal reward

Always-on checks

Primary VU families

Failure codes (common)

Anti-cheat gates

Where results appear