my-pi/pi-chrome/test-suite/README.md

pi-chrome browser-control benchmark

Static benchmark pages for evaluating tools that let agents control Chrome. The suite has two layers:

1. Unit challenges (manifest.json) — MiniWoB-style capability probes for forms, scroll containers, contenteditable, files, frames, Shadow DOM, network/console inspection, isTrusted, user activation, pointer paths, key cadence, native controls, drag/drop, touch, paste, and scroll momentum.
2. Long-horizon hermetic tasks (task-manifest.json) — WebArena / BrowserGym-inspired multi-step tasks with fresh run IDs and deterministic programmatic graders.

Run

cd test-suite
python3 -m http.server 8765
# open http://127.0.0.1:8765/ in the Chrome window pi-chrome controls

Each challenge page exposes:

• window.__challenge — id
• window.__verdict — "PENDING" | "PASS" | "FAIL" | "SKIP" | "WARN"
• window.__reason — array of reasons
• window.__events — raw event log for forensics

manifest.json is the source of truth for unit-challenge metadata: category, gate bucket, goal, expected result per mode, prerequisites, flake risk, manual baseline status, and canonical tool recipe. manifest.schema.json documents the manifest shape. Recipes express tool intent; runners may need to adapt descriptive selectors (e.g. shadow/iframe notation), dynamic tab ids, and expand path placeholders like $PWD.

task-manifest.json is the source of truth for long-horizon tasks: BrowserGym-style taskId, seed, viewport, goal object, difficulty tier, max steps, declared action subsets, reset/setup URL, validate hook, optional cheat recipe, and programmatic grader expression. task-manifest.schema.json documents this shape. browsergym-action-space.json records BrowserGym-compatible action subsets.

Modes / expected outcomes

The same page can have different expected results depending on tool capability:

• synthetic — DOM-dispatched events / framework-aware setters. Fast and quiet.
• trusted — browser-trusted input, usually via chrome.debugger/CDP. Can show Chrome's debugging banner.
• manual — human baseline in same browser/profile.

Expected values in manifest.json:

• PASS / FAIL — deterministic target for that mode.
• CONDITIONAL — depends on browser policy, OS, device capability, permissions, or an unreleased tool primitive. Inspect prerequisites, notes, and flakeRisk.

Manual baselines are tracked separately with manualBaseline. unverified means the manual expectation is a target, not a recorded contract.

Gate buckets

Each unit challenge has a gate field:

• core — required release blocker for normal trusted-mode pi-chrome shipping.
• conditional — blocks only when declared prerequisites/capabilities are present (clipboard, touch, dialogs, native UI, etc.).
• quality — adversarial humanization/fingerprint signal. Track regressions, but do not block general ship without an explicit product decision.

Recommended unit-challenge agent flow

1. Navigate to dashboard: http://127.0.0.1:8765/.
2. Pick mode (synthetic, trusted, or manual) and clear local verdicts.
3. For each manifest row:
   • chrome_navigate to http://127.0.0.1:8765/<file>.
   • chrome_snapshot before acting; prefer snapshot uid over raw selector.
   • Execute the listed recipe, adapting descriptive frame/shadow selectors to whatever selectors/uids the tool exposes.
   • Read:

     JSON.stringify({
       v: window.__verdict,
       r: window.__reason,
       e: window.__events?.slice(-20)
     })
     

4. Return to dashboard and compare actual verdicts with expected values.
5. Copy JSON report from dashboard for PRs or regression notes.

Recommended long-horizon task flow

1. Load task-manifest.json.
2. Replace $RUN_ID in startUrl with a fresh value.
3. Navigate to the start URL and read the visible task instruction.
4. Solve using normal browser tools only; avoid direct state mutation unless the benchmark mode explicitly allows evaluate-based actions.
5. Click Grade now or evaluate the task grader expression:

   JSON.stringify({ v: window.__taskVerdict, r: window.__taskReason })
   

6. Record action count, observations used, tools used, verdict, and reason.

Design principles copied from browser-agent benchmarks

• Prefer hermetic sites and deterministic graders over live sites and LLM judges.
• Report action API and observation format; these strongly affect scores.
• Use difficulty tiers: L1 atomic, L2 compositional, L3 cross-page/context-rich.
• Include tedious cross-page memory and exact-value transfer tasks; short unit probes hide these failures.
• Keep synthetic-event-gated tests because extension bridges face failures that CDP/Playwright-style benchmarks usually do not measure.

Challenge categories

• trusted-input — browser-trusted click/key events.
• pointer-humanization — paths, coordinates, movement continuity/rate.
• keyboard / focus-keyboard — typing fidelity, modifiers, Tab flows.
• activation-gates — clipboard/fullscreen/user activation.
• scroll / scroll-visibility — wheel events, momentum, IntersectionObserver.
• drag-drop — HTML5 drag/drop + DataTransfer.
• clipboard — OS/browser paste path.
• native-controls — controls that should use browser UI/keyboard semantics.
• frameworks / editing — React-style value tracking, contenteditable.
• dom-complexity / frames — Shadow DOM and iframe targeting.
• files — file attachment to <input type=file>.
• observability — console/network capture tools.
• csp — strict Content Security Policy: screenshot/coordinate fallback (39) and the CDP eval/snapshot bypass that works under script-src 'self' without unsafe-eval (42).
• lazy-loading — dynamic DOM readiness and wait behavior.
• fingerprint — environment and stack fingerprint probes.
• agent-safety — hidden honeypots and safe target selection.

Current challenge inventory

The dashboard renders this from manifest.json. In brief:

1. trusted click
2. trusted keyboard
3. webdriver/runtime flags
4. mouse entropy before click
5. click timing
6. click coordinate variation
7. pointer event properties
8. keyboard cadence
9. beforeinput/input order
10. user activation gates
11. honeypot safety
12. fingerprint consistency
13. focus order
14. wheel scroll
15. drag/drop DataTransfer
16. contenteditable selection
17. paste clipboard
18. native select
19. hover dwell
20. React value tracker
21. keyboard modifiers
22. touch events
23. stack trace fingerprint
24. viewport click coordinates
25. pointer continuity
26. mousemove rate
27. scroll momentum
28. intersection visibility
29. Shadow DOM controls
30. iframe targeting
31. file upload
32. keyboard Tab navigation
33. network/console capture
34. dialog handling
35. target blank popup
36. modal focus trap
37. autocomplete combobox
38. SPA route change
39. strict CSP screenshot/coordinate fallback
40. dynamic wait/readiness
41. explicit tab lifecycle
42. strict CSP eval/snapshot via CDP (regression guard for the CSP bypass)

Design notes

• A failure is useful only when compared to expected mode. Example: synthetic isTrusted failing is expected and validates that the test detects quiet DOM events.
• Some tests are capability-gated. Example: touch tests should be SKIP/manual conditional on non-touch hardware.
• Fingerprint tests should warn before blocking. Real Chrome profiles can use software WebGL in VMs, remote desktops, or policy-constrained environments.
• notes/bypass-ideas.md is historical guidance for older synthetic-only versions. Prefer manifest.json for current expected outcomes.
• notes/browsergym-compat.md defines the reset/step/validate/observation/BID contract for external BrowserGym-style agents.
• notes/runner-spec.md, notes/scoring.md, and notes/profiles.md define runner output, scoring, retry policy, and environment metadata.