Actions

Agents return one action per step. The wrapper/orchestrator translate into executor device operations.

Real-app (Android) surface

Primary production actions:

Action	Params	Notes
`click`	`point_2d: [x, y]`	Tap in screenshot coordinates
`type`	`text`	Type into focused field
`drag`	start/end points (impl-specific)	Scroll / swipe gestures
`press_back`	-	Android back
`start_app`	-	Foreground app only (not a full reset)
`needs_confirmation`	message / content	Required before irreversible actions when gated. See Confirmation latches
`finished`	optional `text`	Agent claims success → terminal
`failed_to_complete_task`	optional reason	Structured give-up

Example:

{
  "action": "click",
  "params": { "point_2d": [310, 125] }
}

Canonical schema types

Capabilities also advertise schema types used in mock / element-id mode: tap, type_text, scroll, select, copy, paste, back, finish with optional target.element_id (e.g. home.search.input).

{
  "episode_id": "ep_demo_001",
  "step_id": 1,
  "type": "type_text",
  "target": { "element_id": "home.search.input" },
  "params": { "text": "hoodie" }
}

Reset vs `start_app`

Call	Semantics
`POST /v1/env/reset` / worker run start / executor reset	Full episode reset: force-stop, relaunch, reseed session
`start_app` step action	Foreground only: no DB wipe, no fixture reseed

Termination

Episodes end when:

Agent emits finished / failed_to_complete_task (or schema finish)
max_steps hit → truncated (reward often 0 or fail depending on verifiers)
Infra error → treat as infrastructure failure, not model skill

Terminal UI success for checkout is usually screen_id=ORDER_CONFIRMATION plus verifier PASS. UI alone is not enough.

Getting started

Mechanics

API reference

Scores and trajectories

Real-app (Android) surface

Canonical schema types

Reset vs `start_app`

Termination

​Real-app (Android) surface

​Canonical schema types

​Reset vs start_app

​Termination

Real-app (Android) surface

Canonical schema types

Reset vs `start_app`

Termination