Skip to main content
Audit and review is the core Expert Review work type. Your organization points its specialists at agent runs from Environments and scored results from Evaluation, and you judge whether the agent, the score, and the task itself hold up under expert scrutiny.

​
What you review

​
How to review a run

1

Read the brief first

Every queue has a project brief with the rubric and scope. Review against that rubric, not personal preference.
2

Walk the trajectory

Follow the run step by step against the task goal. Note where the agent diverged, skipped a required check, or acted on invented facts.
3

Check the score against the evidence

Confirm the verifier outcome and reward match the run. A passing score on a run that violated domain procedure is a finding.
4

Classify the failure

Separate agent mistakes from task or verifier defects. “The task was ambiguous” and “the agent was wrong” lead to different fixes.
5

Write the finding

Record your judgment with a short rationale tied to specific steps in the run. Flag anything out of your specialty for reassignment instead of guessing.

​
What a good finding looks like

  • Names the decisive step or span: “step 7 placed the order without the required confirmation turn”
  • States the domain rule it violates or satisfies, in terms another specialist can verify
  • Distinguishes agent error from task or scoring error
  • Recommends a concrete fix when the defect is in the task or rubric: tighten a success criterion, add a failure mode, split an ambiguous goal
Vague findings (“seemed off”, “score feels harsh”) do not move tasks or rubrics. Falsifiable ones do.

​
Where findings go

Review findings feed back into the rest of the suite:
  • Task and success-criteria fixes land in your org’s Environments task banks
  • Rubric corrections and adjudicated hard cases tighten Evaluation datasets and human-review workflows
  • Confirmed failures become regression cases so the same defect is caught automatically next time