feat(chrome): hand snapshots to context mode

This commit is contained in:
云服务部-叶林立
2026-08-28 14:45:37 +08:00
parent 221e978622
commit f3a2abe1e4
96 changed files with 12800 additions and 26 deletions
+194
View File
@@ -0,0 +1,194 @@
# pi-chrome browser-control benchmark
Static benchmark pages for evaluating tools that let agents control Chrome. The suite has two layers:
1. **Unit challenges** (`manifest.json`) — MiniWoB-style capability probes for
forms, scroll containers, contenteditable, files, frames, Shadow DOM,
network/console inspection, `isTrusted`, user activation, pointer paths, key
cadence, native controls, drag/drop, touch, paste, and scroll momentum.
2. **Long-horizon hermetic tasks** (`task-manifest.json`) — WebArena /
BrowserGym-inspired multi-step tasks with fresh run IDs and deterministic
programmatic graders.
## Run
```bash
cd test-suite
python3 -m http.server 8765
# open http://127.0.0.1:8765/ in the Chrome window pi-chrome controls
```
Each challenge page exposes:
- `window.__challenge` — id
- `window.__verdict``"PENDING" | "PASS" | "FAIL" | "SKIP" | "WARN"`
- `window.__reason` — array of reasons
- `window.__events` — raw event log for forensics
`manifest.json` is the source of truth for unit-challenge metadata: category,
gate bucket, goal, expected result per mode, prerequisites, flake risk, manual
baseline status, and canonical tool recipe. `manifest.schema.json` documents the
manifest shape. Recipes express tool intent; runners may need to adapt
descriptive selectors (e.g. shadow/iframe notation), dynamic tab ids, and expand
path placeholders like `$PWD`.
`task-manifest.json` is the source of truth for long-horizon tasks: BrowserGym-style
`taskId`, seed, viewport, goal object, difficulty tier, max steps, declared
action subsets, reset/setup URL, validate hook, optional cheat recipe, and
programmatic grader expression. `task-manifest.schema.json` documents this shape.
`browsergym-action-space.json` records BrowserGym-compatible action subsets.
## Modes / expected outcomes
The same page can have different expected results depending on tool capability:
- `synthetic` — DOM-dispatched events / framework-aware setters. Fast and quiet.
- `trusted` — browser-trusted input, usually via `chrome.debugger`/CDP. Can show
Chrome's debugging banner.
- `manual` — human baseline in same browser/profile.
Expected values in `manifest.json`:
- `PASS` / `FAIL` — deterministic target for that mode.
- `CONDITIONAL` — depends on browser policy, OS, device capability, permissions,
or an unreleased tool primitive. Inspect `prerequisites`, `notes`, and
`flakeRisk`.
Manual baselines are tracked separately with `manualBaseline`. `unverified`
means the manual expectation is a target, not a recorded contract.
## Gate buckets
Each unit challenge has a `gate` field:
- `core` — required release blocker for normal trusted-mode pi-chrome shipping.
- `conditional` — blocks only when declared prerequisites/capabilities are present
(clipboard, touch, dialogs, native UI, etc.).
- `quality` — adversarial humanization/fingerprint signal. Track regressions, but
do not block general ship without an explicit product decision.
## Recommended unit-challenge agent flow
1. Navigate to dashboard:
`http://127.0.0.1:8765/`.
2. Pick mode (`synthetic`, `trusted`, or `manual`) and clear local verdicts.
3. For each manifest row:
- `chrome_navigate` to `http://127.0.0.1:8765/<file>`.
- `chrome_snapshot` before acting; prefer snapshot `uid` over raw selector.
- Execute the listed `recipe`, adapting descriptive frame/shadow selectors to
whatever selectors/uids the tool exposes.
- Read:
```js
JSON.stringify({
v: window.__verdict,
r: window.__reason,
e: window.__events?.slice(-20)
})
```
4. Return to dashboard and compare actual verdicts with expected values.
5. Copy JSON report from dashboard for PRs or regression notes.
## Recommended long-horizon task flow
1. Load `task-manifest.json`.
2. Replace `$RUN_ID` in `startUrl` with a fresh value.
3. Navigate to the start URL and read the visible task instruction.
4. Solve using normal browser tools only; avoid direct state mutation unless the
benchmark mode explicitly allows evaluate-based actions.
5. Click **Grade now** or evaluate the task grader expression:
```js
JSON.stringify({ v: window.__taskVerdict, r: window.__taskReason })
```
6. Record action count, observations used, tools used, verdict, and reason.
## Design principles copied from browser-agent benchmarks
- Prefer hermetic sites and deterministic graders over live sites and LLM judges.
- Report action API and observation format; these strongly affect scores.
- Use difficulty tiers: L1 atomic, L2 compositional, L3 cross-page/context-rich.
- Include tedious cross-page memory and exact-value transfer tasks; short unit
probes hide these failures.
- Keep synthetic-event-gated tests because extension bridges face failures that
CDP/Playwright-style benchmarks usually do not measure.
## Challenge categories
- `trusted-input` — browser-trusted click/key events.
- `pointer-humanization` — paths, coordinates, movement continuity/rate.
- `keyboard` / `focus-keyboard` — typing fidelity, modifiers, Tab flows.
- `activation-gates` — clipboard/fullscreen/user activation.
- `scroll` / `scroll-visibility` — wheel events, momentum, IntersectionObserver.
- `drag-drop` — HTML5 drag/drop + `DataTransfer`.
- `clipboard` — OS/browser paste path.
- `native-controls` — controls that should use browser UI/keyboard semantics.
- `frameworks` / `editing` — React-style value tracking, contenteditable.
- `dom-complexity` / `frames` — Shadow DOM and iframe targeting.
- `files` — file attachment to `<input type=file>`.
- `observability` — console/network capture tools.
- `csp` — strict Content Security Policy: screenshot/coordinate fallback (39) and the CDP eval/snapshot bypass that works under `script-src 'self'` without `unsafe-eval` (42).
- `lazy-loading` — dynamic DOM readiness and wait behavior.
- `fingerprint` — environment and stack fingerprint probes.
- `agent-safety` — hidden honeypots and safe target selection.
## Current challenge inventory
The dashboard renders this from `manifest.json`. In brief:
1. trusted click
2. trusted keyboard
3. webdriver/runtime flags
4. mouse entropy before click
5. click timing
6. click coordinate variation
7. pointer event properties
8. keyboard cadence
9. beforeinput/input order
10. user activation gates
11. honeypot safety
12. fingerprint consistency
13. focus order
14. wheel scroll
15. drag/drop `DataTransfer`
16. contenteditable selection
17. paste clipboard
18. native select
19. hover dwell
20. React value tracker
21. keyboard modifiers
22. touch events
23. stack trace fingerprint
24. viewport click coordinates
25. pointer continuity
26. mousemove rate
27. scroll momentum
28. intersection visibility
29. Shadow DOM controls
30. iframe targeting
31. file upload
32. keyboard Tab navigation
33. network/console capture
34. dialog handling
35. target blank popup
36. modal focus trap
37. autocomplete combobox
38. SPA route change
39. strict CSP screenshot/coordinate fallback
40. dynamic wait/readiness
41. explicit tab lifecycle
42. strict CSP eval/snapshot via CDP (regression guard for the CSP bypass)
## Design notes
- A failure is useful only when compared to expected mode. Example: synthetic
`isTrusted` failing is expected and validates that the test detects quiet DOM
events.
- Some tests are capability-gated. Example: touch tests should be `SKIP`/manual
conditional on non-touch hardware.
- Fingerprint tests should warn before blocking. Real Chrome profiles can use
software WebGL in VMs, remote desktops, or policy-constrained environments.
- `notes/bypass-ideas.md` is historical guidance for older synthetic-only
versions. Prefer `manifest.json` for current expected outcomes.
- `notes/browsergym-compat.md` defines the reset/step/validate/observation/BID
contract for external BrowserGym-style agents.
- `notes/runner-spec.md`, `notes/scoring.md`, and `notes/profiles.md` define
runner output, scoring, retry policy, and environment metadata.
+130
View File
@@ -0,0 +1,130 @@
// Tiny shared harness for challenge pages.
// Each page calls Challenge.init({id, instructions}) then Challenge.pass()/fail()
// based on its own listeners.
(function () {
const events = [];
const state = {
id: null,
verdict: "PENDING",
reason: [],
events,
details: [],
thresholds: {},
};
function render() {
const el = document.getElementById("__verdict");
if (!el) return;
el.textContent = state.verdict;
el.dataset.verdict = state.verdict;
el.style.background =
state.verdict === "PASS" ? "#1f7a1f" :
state.verdict === "FAIL" ? "#a11" :
state.verdict === "SKIP" ? "#76520b" :
state.verdict === "WARN" ? "#6b5d00" : "#444";
el.style.color = "#fff";
const r = document.getElementById("__reason");
if (r) r.textContent = state.reason.join("\n");
}
function log(name, detail) {
events.push({ t: performance.now(), name, ...detail });
if (events.length > 500) events.shift();
}
const Challenge = {
init({ id, instructions, thresholds = {} }) {
state.id = id;
state.thresholds = parseThresholds(thresholds);
document.title = `[${state.verdict}] ${id}`;
const root = document.body;
const bar = document.createElement("div");
bar.style.cssText =
"position:sticky;top:0;background:#111;color:#eee;padding:8px 12px;font:13px monospace;border-bottom:1px solid #333;z-index:9999";
bar.innerHTML = `
<b>${id}</b>
<span id="__verdict" style="margin-left:8px;padding:2px 8px;border-radius:4px;background:#444">PENDING</span>
<span style="margin-left:12px;opacity:.7">${instructions}</span>
<pre id="__reason" style="white-space:pre-wrap;margin:6px 0 0;color:#bbb;font:12px monospace"></pre>
`;
root.insertBefore(bar, root.firstChild);
window.__challenge = id;
window.__verdict = state.verdict;
window.__reason = state.reason;
window.__events = state.events;
render();
},
pass(...reasons) {
if (state.verdict === "FAIL") return; // sticky
state.verdict = "PASS";
state.reason.push(...reasons.map((r) => "✓ " + r));
window.__verdict = state.verdict;
document.title = `[PASS] ${state.id}`;
persist(); render();
},
fail(...reasons) {
state.verdict = "FAIL";
state.reason.push(...reasons.map((r) => "✗ " + r));
window.__verdict = state.verdict;
document.title = `[FAIL] ${state.id}`;
persist(); render();
},
skip(...reasons) {
if (state.verdict === "FAIL" || state.verdict === "PASS") return;
state.verdict = "SKIP";
state.reason.push(...reasons.map((r) => "↷ " + r));
window.__verdict = state.verdict;
document.title = `[SKIP] ${state.id}`;
persist(); render();
},
warn(...reasons) {
if (state.verdict === "FAIL" || state.verdict === "PASS") return;
state.verdict = "WARN";
state.reason.push(...reasons.map((r) => "! " + r));
window.__verdict = state.verdict;
document.title = `[WARN] ${state.id}`;
persist(); render();
},
partial({ name, pass, reason, data }) {
state.details.push({ name, pass: !!pass, reason: reason || "", data });
log("partial", { name, pass: !!pass, reason, data });
persist(); render();
return !!pass;
},
finishPartials() {
const failed = state.details.filter(d => !d.pass);
if (failed.length) Challenge.fail(...failed.map(d => `${d.name}: ${d.reason || "failed"}`));
else Challenge.pass(...state.details.map(d => `${d.name}: ok`));
},
getThreshold(name, fallback) {
return Object.prototype.hasOwnProperty.call(state.thresholds, name) ? state.thresholds[name] : fallback;
},
log,
state,
};
function persist() {
try {
localStorage.setItem(
"pi-chrome-suite:" + state.id,
JSON.stringify({ id: state.id, verdict: state.verdict, reason: state.reason, details: state.details, thresholds: state.thresholds, events: state.events.slice(-50), ts: Date.now() })
);
} catch {}
}
function parseThresholds(defaults) {
const out = { ...defaults };
try {
const qs = new URLSearchParams(location.search);
for (const [k, v] of qs) {
if (!k.startsWith("threshold.")) continue;
const key = k.slice("threshold.".length);
const num = Number(v);
out[key] = Number.isFinite(num) ? num : v;
}
} catch {}
return out;
}
window.Challenge = Challenge;
})();
+31
View File
@@ -0,0 +1,31 @@
body { font: 14px system-ui, sans-serif; margin: 0; background: #1a1a1a; color: #eee; }
main { padding: 24px; max-width: 720px; }
code { background:#222;padding:1px 4px;border-radius:3px }
a { color: #6cf; }
button, input, select { font: 14px system-ui, sans-serif; }
button { padding: 7px 11px; border-radius: 6px; border: 1px solid #555; background: #262626; color: #eee; cursor:pointer; }
button:hover { background:#333; }
input, select { padding: 6px 8px; border-radius: 6px; border: 1px solid #555; background:#111; color:#eee; }
table { border-collapse: collapse; width: 100%; font: 13px ui-monospace, SFMono-Regular, Menlo, monospace; }
th, td { border-bottom: 1px solid #333; padding: 8px; vertical-align: top; text-align:left; }
th { position: sticky; top: 0; background:#151515; z-index:1; }
tr.ok { background: rgba(31,122,31,.08); }
tr.mismatch { background: rgba(170,17,17,.12); }
tr.conditional { background: rgba(118,82,11,.12); }
.panel { background:#202020; border:1px solid #333; border-radius:10px; padding:12px; margin:16px 0; }
.controls { display:flex; flex-wrap:wrap; gap:10px; align-items:center; }
.hint, .notes, .reason { color:#aaa; font-size:12px; margin-top:4px; }
.summary { display:flex; flex-wrap:wrap; gap:8px; margin: 14px 0; }
.pill { display:inline-block; border-radius:999px; padding:2px 8px; font-size:12px; font-weight:700; background:#444; color:#fff; }
.pass { background:#1f7a1f; }
.fail { background:#a11; }
.skip, .conditional { background:#76520b; }
.warn { background:#6b5d00; }
.pending { background:#444; }
.expected { background:#2b4b6b; }
.risk { color:#ffd479; }
.code, pre { background:#111; color:#eee; padding:12px; border-radius:6px; overflow:auto; }
.copy-fallback { display:none; width:100%; min-height:180px; margin-top:10px; font:12px ui-monospace, SFMono-Regular, Menlo, monospace; background:#111; color:#eee; border:1px solid #555; border-radius:6px; padding:10px; }
.copy-fallback[data-copied="true"] { border-color:#1f7a1f; }
.copy-fallback[data-copied="false"] { border-color:#a11; }
details summary { cursor:pointer; color:#9cf; }
Binary file not shown.

After

Width:  |  Height:  |  Size: 429 KiB

@@ -0,0 +1,44 @@
{
"subsets": {
"chat": ["send_msg_to_user"],
"infeas": ["report_infeasible"],
"bid": ["click", "dblclick", "hover", "fill", "clear", "press", "focus", "select_option", "scroll", "drag_and_drop", "upload_file"],
"coord": ["mouse_move", "mouse_up", "mouse_down", "mouse_click", "mouse_dblclick", "mouse_drag_and_drop", "mouse_upload_file", "scroll_at", "keyboard_down", "keyboard_up", "keyboard_press", "keyboard_type", "keyboard_insert_text"],
"nav": ["go_back", "go_forward", "goto"],
"tab": ["tab_close", "tab_focus", "new_tab"]
},
"signatures": {
"fill": "fill(bid, value, enable_autocomplete_menu=false)",
"click": "click(bid, button='left', modifiers=[])",
"dblclick": "dblclick(bid, button='left', modifiers=[])",
"hover": "hover(bid)",
"press": "press(bid, key_comb)",
"focus": "focus(bid)",
"clear": "clear(bid)",
"select_option": "select_option(bid, options)",
"scroll": "scroll(delta_x, delta_y)",
"drag_and_drop": "drag_and_drop(from_bid, to_bid)",
"upload_file": "upload_file(bid, file)",
"mouse_move": "mouse_move(x, y)",
"mouse_click": "mouse_click(x, y, button='left')",
"mouse_dblclick": "mouse_dblclick(x, y, button='left')",
"mouse_up": "mouse_up(x, y, button='left')",
"mouse_down": "mouse_down(x, y, button='left')",
"mouse_drag_and_drop": "mouse_drag_and_drop(from_x, from_y, to_x, to_y)",
"scroll_at": "scroll_at(x, y, dx, dy)",
"keyboard_press": "keyboard_press(key)",
"keyboard_down": "keyboard_down(key)",
"keyboard_up": "keyboard_up(key)",
"keyboard_type": "keyboard_type(text)",
"keyboard_insert_text": "keyboard_insert_text(text)",
"goto": "goto(url)",
"go_back": "go_back()",
"go_forward": "go_forward()",
"tab_close": "tab_close()",
"tab_focus": "tab_focus(index)",
"new_tab": "new_tab()",
"send_msg_to_user": "send_msg_to_user(text)",
"report_infeasible": "report_infeasible(reason)",
"noop": "noop(wait_ms=1000)"
}
}
@@ -0,0 +1,28 @@
<!doctype html>
<meta charset="utf-8">
<title>01 isTrusted click</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: click the green button. Page only accepts <code>event.isTrusted === true</code>.</p>
<button id="go" style="padding:20px 32px;font-size:18px;background:#1f7a1f;color:#fff;border:0;border-radius:6px">Click me</button>
</main>
<script>
Challenge.init({
id: "is-trusted-click",
instructions: "click the green button; isTrusted must be true",
});
const btn = document.getElementById("go");
btn.addEventListener("click", (e) => {
Challenge.log("click", { isTrusted: e.isTrusted, x: e.clientX, y: e.clientY });
if (e.isTrusted) Challenge.pass("click.isTrusted === true");
else Challenge.fail("click.isTrusted === false (synthetic dispatchEvent)");
}, { capture: true });
// Also watch pointerdown to detect bot earlier.
btn.addEventListener("pointerdown", (e) => {
Challenge.log("pointerdown", { isTrusted: e.isTrusted, pressure: e.pressure, pointerType: e.pointerType });
if (!e.isTrusted) Challenge.fail("pointerdown.isTrusted === false");
});
</script>
</body>
@@ -0,0 +1,50 @@
<!doctype html>
<meta charset="utf-8">
<title>02 isTrusted keyboard</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: type <code>hello</code> into the box. Each character must arrive via a trusted keydown/keypress/input.</p>
<input id="t" placeholder="type hello" style="font-size:18px;padding:8px 12px;width:240px">
</main>
<script>
Challenge.init({
id: "is-trusted-keyboard",
instructions: "type 'hello'; every keydown.isTrusted must be true",
});
const t = document.getElementById("t");
let trustedKeys = 0, untrustedKeys = 0, inputs = 0, untrustedInputs = 0;
t.addEventListener("keydown", (e) => {
Challenge.log("keydown", { key: e.key, isTrusted: e.isTrusted });
if (e.isTrusted) trustedKeys++; else untrustedKeys++;
});
t.addEventListener("input", (e) => {
Challenge.log("input", { isTrusted: e.isTrusted, inputType: e.inputType, data: e.data });
inputs++; if (!e.isTrusted) untrustedInputs++;
});
t.addEventListener("keyup", () => {
if (t.value === "hello") {
if (untrustedKeys || untrustedInputs) {
Challenge.fail(`untrusted keydowns=${untrustedKeys}, untrusted inputs=${untrustedInputs}`);
} else if (trustedKeys < 5) {
Challenge.fail(`only ${trustedKeys} trusted keydowns (need >=5)`);
} else {
Challenge.pass(`${trustedKeys} trusted keydowns, ${inputs} trusted inputs`);
}
}
});
// Programmatic value-set detection.
const desc = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, "value");
Object.defineProperty(t, "value", {
get() { return desc.get.call(this); },
set(v) {
Challenge.log("value-set", { v });
if (untrustedKeys === 0 && trustedKeys === 0) {
Challenge.fail("value set programmatically without any keydown");
}
return desc.set.call(this, v);
},
});
</script>
</body>
@@ -0,0 +1,51 @@
<!doctype html>
<meta charset="utf-8">
<title>03 webdriver / runtime fingerprint</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Auto-checks runtime properties commonly inspected by bot-detection scripts.</p>
<pre id="rep" style="font:12px monospace;background:#111;color:#eee;padding:12px;border-radius:6px"></pre>
</main>
<script>
Challenge.init({
id: "webdriver-flag",
instructions: "no interaction; auto-verdict on load",
});
const checks = [];
function chk(name, bad, note) {
checks.push({ name, bad, note });
if (bad) Challenge.log("flag", { name, note });
}
chk("navigator.webdriver", navigator.webdriver === true, String(navigator.webdriver));
chk("languages-empty", !navigator.languages || navigator.languages.length === 0,
JSON.stringify(navigator.languages));
chk("plugins-empty", !navigator.plugins || navigator.plugins.length === 0,
`length=${navigator.plugins && navigator.plugins.length}`);
chk("permissions-notifications-quirk", false, "checked async below");
(async () => {
try {
const p = await navigator.permissions.query({ name: "notifications" });
// Headless Chrome historically returned "denied" while Notification.permission === "default".
const mismatch = p.state === "denied" && Notification.permission === "default";
checks.push({ name: "permissions-notifications-quirk", bad: mismatch,
note: `perm=${p.state} api=${Notification.permission}` });
finalize();
} catch (e) {
checks.push({ name: "permissions-notifications-quirk", bad: false, note: "n/a " + e.message });
finalize();
}
})();
function finalize() {
const rep = document.getElementById("rep");
rep.textContent = checks.map(c => `${c.bad ? "✗" : "✓"} ${c.name} ${c.note}`).join("\n");
const failed = checks.filter(c => c.bad);
if (failed.length) Challenge.fail(...failed.map(c => `${c.name}: ${c.note}`));
else Challenge.pass("all runtime checks clean");
}
</script>
</body>
@@ -0,0 +1,34 @@
<!doctype html>
<meta charset="utf-8">
<title>04 mouse entropy</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: click the green button after moving the mouse in an organic path. Page requires
≥15 <code>mousemove</code> events with non-zero <code>movementX/Y</code> variance before the click.</p>
<button id="go" style="padding:20px 32px;font-size:18px;background:#1f7a1f;color:#fff;border:0;border-radius:6px">Click me</button>
</main>
<script>
Challenge.init({
id: "mouse-entropy",
instructions: "click the button after natural mouse movement",
});
const moves = [];
window.addEventListener("mousemove", (e) => {
moves.push({ x: e.clientX, y: e.clientY, mx: e.movementX, my: e.movementY, t: e.timeStamp });
if (moves.length > 200) moves.shift();
});
document.getElementById("go").addEventListener("click", (e) => {
if (moves.length < 15) return Challenge.fail(`only ${moves.length} mousemove events before click`);
const dx = moves.map(m => Math.abs(m.mx)).reduce((a,b)=>a+b,0);
const dy = moves.map(m => Math.abs(m.my)).reduce((a,b)=>a+b,0);
if (dx + dy < 50) return Challenge.fail(`movement deltas tiny: dx=${dx}, dy=${dy}`);
// Detect "teleport then click" — all moves clustered at the button.
const r = document.getElementById("go").getBoundingClientRect();
const inside = moves.filter(m => m.x>=r.left && m.x<=r.right && m.y>=r.top && m.y<=r.bottom).length;
if (inside === moves.length) return Challenge.fail("all mousemoves inside target rect (teleport)");
Challenge.pass(`${moves.length} moves, |dx|+|dy|=${dx+dy}, ${inside} inside target`);
});
</script>
</body>
@@ -0,0 +1,34 @@
<!doctype html>
<meta charset="utf-8">
<title>05 event timing</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: click the button 3 times. Page rejects when <code>pointerdown→pointerup</code> is &lt;30ms
or always identical, and when between-click gaps are too uniform.</p>
<button id="go" style="padding:20px 32px;font-size:18px;background:#1f7a1f;color:#fff;border:0;border-radius:6px">Click me (0/3)</button>
</main>
<script>
Challenge.init({ id: "event-timing", instructions: "click button 3 times with human-like timing" });
const btn = document.getElementById("go");
const ups = [], downs = [], clickTs = [];
btn.addEventListener("pointerdown", (e) => downs.push(e.timeStamp));
btn.addEventListener("pointerup", (e) => ups.push(e.timeStamp));
btn.addEventListener("click", (e) => {
clickTs.push(e.timeStamp);
btn.textContent = `Click me (${clickTs.length}/3)`;
if (clickTs.length < 3) return;
const holds = downs.map((d,i)=> (ups[i]??d) - d);
const allSame = holds.every(h => Math.abs(h - holds[0]) < 0.5);
const tooFast = holds.some(h => h < 30);
const gaps = clickTs.slice(1).map((t,i)=> t - clickTs[i]);
const gapsUniform = gaps.length > 1 && Math.abs(gaps[0]-gaps[1]) < 5;
Challenge.log("timing", { holds, gaps });
if (tooFast) return Challenge.fail(`pointer hold too short: ${holds.join(",")}ms`);
if (allSame) return Challenge.fail(`pointer holds identical: ${holds.join(",")}ms`);
if (gapsUniform) return Challenge.fail(`between-click gaps suspiciously uniform: ${gaps.join(",")}ms`);
Challenge.pass(`holds=${holds.map(h=>h.toFixed(1)).join(",")}ms gaps=${gaps.map(g=>g.toFixed(0)).join(",")}ms`);
});
</script>
</body>
@@ -0,0 +1,29 @@
<!doctype html>
<meta charset="utf-8">
<title>06 click coordinates</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: click the green button 5 times. Page rejects when clicks always land on the exact element center.</p>
<button id="go" style="padding:24px 40px;font-size:18px;background:#1f7a1f;color:#fff;border:0;border-radius:6px">Click me (0/5)</button>
</main>
<script>
Challenge.init({ id: "click-coordinates", instructions: "click button 5 times; coordinates must vary" });
const btn = document.getElementById("go");
const pts = [];
btn.addEventListener("click", (e) => {
const r = btn.getBoundingClientRect();
const cx = r.left + r.width/2, cy = r.top + r.height/2;
pts.push({ x: e.clientX, y: e.clientY, dx: e.clientX-cx, dy: e.clientY-cy });
btn.textContent = `Click me (${pts.length}/5)`;
if (pts.length < 5) return;
const onCenter = pts.filter(p => Math.abs(p.dx) < 0.51 && Math.abs(p.dy) < 0.51).length;
const unique = new Set(pts.map(p => `${p.x},${p.y}`)).size;
Challenge.log("coords", { pts });
if (onCenter >= 4) return Challenge.fail(`${onCenter}/5 clicks on exact center`);
if (unique <= 1) return Challenge.fail(`only ${unique} unique click coords`);
Challenge.pass(`${unique} unique coords, ${onCenter} on center`);
});
</script>
</body>
@@ -0,0 +1,29 @@
<!doctype html>
<meta charset="utf-8">
<title>07 pointer properties</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: click the button. Page inspects <code>pointerType</code>, <code>pressure</code>,
<code>movementX/Y</code> on the preceding pointermove, and that pointerId is non-zero.</p>
<button id="go" style="padding:20px 32px;font-size:18px;background:#1f7a1f;color:#fff;border:0;border-radius:6px">Click me</button>
</main>
<script>
Challenge.init({ id: "pointer-properties", instructions: "click button; pointer event details must look real" });
const btn = document.getElementById("go");
let lastMove = null;
window.addEventListener("pointermove", (e) => { lastMove = { mx: e.movementX, my: e.movementY, pid: e.pointerId, type: e.pointerType }; });
btn.addEventListener("pointerdown", (e) => {
Challenge.log("pointerdown", { type: e.pointerType, pressure: e.pressure, pid: e.pointerId, mx: e.movementX, my: e.movementY });
const bad = [];
if (e.pointerType !== "mouse" && e.pointerType !== "touch" && e.pointerType !== "pen") bad.push(`pointerType=${e.pointerType}`);
if (e.pointerType === "mouse" && e.pressure !== 0.5) bad.push(`mouse pressure=${e.pressure} (real=0.5)`);
if (e.pointerId === 0) bad.push("pointerId=0");
if (!lastMove) bad.push("no preceding pointermove");
else if (lastMove.mx === 0 && lastMove.my === 0) bad.push("preceding pointermove had movementX=movementY=0");
if (bad.length) Challenge.fail(...bad);
else Challenge.pass(`type=${e.pointerType} pressure=${e.pressure} pid=${e.pointerId}`);
});
</script>
</body>
@@ -0,0 +1,37 @@
<!doctype html>
<meta charset="utf-8">
<title>08 keyboard cadence</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: type <code>secret</code>. Page demands per-key keydown+keypress+keyup with non-uniform gaps and non-zero hold time.</p>
<input id="t" placeholder="type 'secret'" style="font-size:18px;padding:8px 12px;width:240px">
</main>
<script>
Challenge.init({ id: "keyboard-cadence", instructions: "type 'secret' with realistic per-key cadence" });
const t = document.getElementById("t");
const evs = [];
["keydown","keypress","keyup","input"].forEach(name => {
t.addEventListener(name, (e) => evs.push({ name, key: e.key ?? e.data, t: e.timeStamp, trusted: e.isTrusted }));
});
t.addEventListener("keyup", () => {
if (t.value !== "secret") return;
const downs = evs.filter(e => e.name === "keydown");
const presses = evs.filter(e => e.name === "keypress");
const ups = evs.filter(e => e.name === "keyup");
const bad = [];
if (downs.length < 6) bad.push(`only ${downs.length} keydowns (need >=6)`);
if (presses.length < 6) bad.push(`only ${presses.length} keypress events`);
if (ups.length < 6) bad.push(`only ${ups.length} keyups`);
const holds = downs.slice(0, ups.length).map((d,i)=> ups[i].t - d.t);
if (holds.some(h => h <= 0)) bad.push(`some keyup at-or-before keydown: ${holds.join(",")}`);
if (holds.length && holds.every(h => Math.abs(h-holds[0]) < 0.5)) bad.push(`hold times identical: ${holds.join(",")}`);
const gaps = downs.slice(1).map((d,i)=> d.t - downs[i].t);
if (gaps.length && gaps.every(g => Math.abs(g-gaps[0]) < 0.5)) bad.push(`keydown gaps identical: ${gaps.join(",")}`);
Challenge.log("cadence", { holds, gaps });
if (bad.length) Challenge.fail(...bad);
else Challenge.pass(`holds=${holds.map(h=>h.toFixed(0)).join(",")} gaps=${gaps.map(g=>g.toFixed(0)).join(",")}`);
});
</script>
</body>
@@ -0,0 +1,45 @@
<!doctype html>
<meta charset="utf-8">
<title>09 framework input invariants</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: type <code>abc</code>. Page asserts React-style invariants: <code>beforeinput</code>
fires <em>before</em> <code>input</code>, both per-character, <code>inputType==="insertText"</code>,
value mutates between events, and no <code>compositionstart</code> for plain typing.</p>
<input id="t" placeholder="type abc" style="font-size:18px;padding:8px 12px;width:240px">
</main>
<script>
Challenge.init({ id: "composition-input", instructions: "type 'abc' producing framework-correct event order" });
const t = document.getElementById("t");
const evs = [];
let sawCompositionStart = false;
["beforeinput","input","compositionstart","compositionend"].forEach(name => {
t.addEventListener(name, (e) => {
evs.push({ name, t: e.timeStamp, inputType: e.inputType, data: e.data, value: t.value });
if (name === "compositionstart") sawCompositionStart = true;
});
});
t.addEventListener("input", () => {
if (t.value !== "abc") return;
const before = evs.filter(e => e.name === "beforeinput");
const inp = evs.filter(e => e.name === "input");
const bad = [];
if (sawCompositionStart) bad.push("compositionstart fired for plain ASCII typing");
if (before.length !== 3) bad.push(`beforeinput count=${before.length} (need 3)`);
if (inp.length !== 3) bad.push(`input count=${inp.length} (need 3)`);
if (before.some(b => b.inputType !== "insertText")) bad.push(`beforeinput.inputType=${before.map(b=>b.inputType).join(",")}`);
// beforeinput must precede matching input.
for (let i = 0; i < Math.min(before.length, inp.length); i++) {
if (before[i].t > inp[i].t) bad.push(`beforeinput[${i}] after input[${i}]`);
}
// Value must reflect each char incrementally.
const seq = inp.map(e => e.value).join("|");
if (seq !== "a|ab|abc") bad.push(`value seq at input events: ${seq} (need a|ab|abc)`);
Challenge.log("seq", { evs });
if (bad.length) Challenge.fail(...bad);
else Challenge.pass(`event order and invariants OK; seq=${seq}`);
});
</script>
</body>
@@ -0,0 +1,40 @@
<!doctype html>
<meta charset="utf-8">
<title>10 user activation gates</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: click the button. Handler then tries gated APIs (clipboard.writeText, fullscreen).
Success requires <code>navigator.userActivation.isActive</code> and at least one gated API to succeed.</p>
<button id="go" style="padding:20px 32px;font-size:18px;background:#1f7a1f;color:#fff;border:0;border-radius:6px">Click me</button>
</main>
<script>
Challenge.init({ id: "user-activation", instructions: "click button; user-activation must unlock gated APIs" });
document.getElementById("go").addEventListener("click", async () => {
// Capture activation state synchronously before any await (activation can be consumed by gated APIs).
const ua = navigator.userActivation;
const wasActive = !!ua?.isActive;
const hadBeenActive = !!ua?.hasBeenActive;
Challenge.log("activation", { isActive: wasActive, hasBeenActive: hadBeenActive });
let clip = "skip", fs = "skip";
try { await navigator.clipboard.writeText("pi-chrome-test"); clip = "ok"; }
catch (e) { clip = "err:" + e.name; }
try { await document.documentElement.requestFullscreen(); fs = "ok"; document.exitFullscreen?.(); }
catch (e) { fs = "err:" + e.name; }
Challenge.log("gates", { clip, fs });
if (!wasActive && !hadBeenActive) {
return Challenge.fail("userActivation.isActive/hasBeenActive both false (synthetic click)");
}
const okCount = [clip, fs].filter(x => x === "ok").length;
if (okCount === 0) {
return Challenge.skip(`activation present, but gated APIs blocked by environment/policy: clipboard=${clip}; fullscreen=${fs}`);
}
const warnings = [];
if (clip.startsWith("err")) warnings.push("clipboard.writeText " + clip);
if (fs.startsWith("err")) warnings.push("requestFullscreen " + fs);
if (warnings.length) Challenge.pass(`activation active; ${okCount}/2 gates succeeded; ${warnings.join("; ")}`);
else Challenge.pass(`activation active; clipboard=${clip}; fullscreen=${fs}`);
});
</script>
</body>
@@ -0,0 +1,36 @@
<!doctype html>
<meta charset="utf-8">
<title>11 honeypot fields</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: fill <em>only</em> the visible "Name" field with <code>alex</code> and submit.
Three honeypots (a, b, c) are present that a human cannot see/tab to. Filling any honeypot fails.</p>
<form id="f">
<label>Name <input name="name" id="name"></label>
<!-- off-screen -->
<input name="a" aria-hidden="true" tabindex="-1" autocomplete="off"
style="position:absolute;left:-9999px;top:-9999px">
<!-- display:none -->
<input name="b" aria-hidden="true" tabindex="-1" autocomplete="off" style="display:none">
<!-- zero-size + visually hidden -->
<input name="c" aria-hidden="true" tabindex="-1" autocomplete="off"
style="opacity:0;width:0;height:0;border:0;padding:0">
<button type="submit">Submit</button>
</form>
</main>
<script>
Challenge.init({ id: "honeypot", instructions: "fill only the Name field with 'alex', then submit" });
const form = document.getElementById("f");
form.addEventListener("submit", (e) => {
e.preventDefault();
const fd = new FormData(form);
const filled = ["a","b","c"].filter(k => (fd.get(k) ?? "").toString().length > 0);
Challenge.log("submit", { name: fd.get("name"), a: fd.get("a"), b: fd.get("b"), c: fd.get("c") });
if (filled.length) return Challenge.fail("honeypot(s) filled: " + filled.join(","));
if ((fd.get("name") ?? "") !== "alex") return Challenge.fail(`name="${fd.get("name")}" (need 'alex')`);
Challenge.pass("only visible field filled");
});
</script>
</body>
@@ -0,0 +1,59 @@
<!doctype html>
<meta charset="utf-8">
<title>12 fingerprint consistency</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Auto-check. Looks for cross-API consistency mismatches that betray instrumented Chrome.</p>
<pre id="rep" style="font:12px monospace;background:#111;color:#eee;padding:12px;border-radius:6px"></pre>
</main>
<script>
Challenge.init({ id: "fingerprint", instructions: "no interaction" });
(async () => {
const checks = [];
const ua = navigator.userAgent;
const isChrome = /Chrome\/\d+/.test(ua);
checks.push({ name: "ua-is-chrome", bad: !isChrome, note: ua });
// UA-CH: Chrome should expose userAgentData with brands incl Chromium.
const uad = navigator.userAgentData;
const hasUAD = !!uad;
checks.push({ name: "userAgentData-present", bad: isChrome && !hasUAD, note: hasUAD ? JSON.stringify(uad.brands) : "missing" });
// Chrome runtime object exists in normal Chrome.
checks.push({ name: "window.chrome", bad: isChrome && typeof window.chrome === "undefined",
note: typeof window.chrome });
// languages must match accept-language semantics.
checks.push({ name: "languages-includes-language", bad: !navigator.languages?.includes(navigator.language),
note: `${navigator.language} vs ${JSON.stringify(navigator.languages)}` });
// WebGL vendor/renderer should not be Brian Paul / SwiftShader on user profile.
try {
const gl = document.createElement("canvas").getContext("webgl");
const dbg = gl.getExtension("WEBGL_debug_renderer_info");
const vendor = dbg ? gl.getParameter(dbg.UNMASKED_VENDOR_WEBGL) : gl.getParameter(gl.VENDOR);
const renderer = dbg ? gl.getParameter(dbg.UNMASKED_RENDERER_WEBGL) : gl.getParameter(gl.RENDERER);
const swiftshader = /SwiftShader|llvmpipe|Software/i.test(renderer);
checks.push({ name: "webgl-not-software", bad: false, warn: swiftshader, note: `${vendor} / ${renderer}${swiftshader ? " (software renderer; warning in VM/remote contexts)" : ""}` });
} catch (e) {
checks.push({ name: "webgl-not-software", bad: false, note: "n/a " + e.message });
}
// hardwareConcurrency / deviceMemory plausibility.
checks.push({ name: "hardwareConcurrency", bad: !(navigator.hardwareConcurrency > 0), note: String(navigator.hardwareConcurrency) });
// navigator.webdriver again, separate from challenge 03 so it appears here too.
checks.push({ name: "webdriver-false", bad: navigator.webdriver === true, note: String(navigator.webdriver) });
const rep = document.getElementById("rep");
rep.textContent = checks.map(c => `${c.bad ? "✗" : c.warn ? "!" : "✓"} ${c.name} ${c.note}`).join("\n");
const failed = checks.filter(c => c.bad);
const warned = checks.filter(c => c.warn);
if (failed.length) Challenge.fail(...failed.map(c => `${c.name}: ${c.note}`));
else if (warned.length) Challenge.warn(...warned.map(c => `${c.name}: ${c.note}`));
else Challenge.pass("fingerprint consistent with real Chrome");
})();
</script>
</body>
@@ -0,0 +1,31 @@
<!doctype html>
<meta charset="utf-8">
<title>13 focus order</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: focus the input by <em>clicking</em> it (not <code>.focus()</code>), then type <code>x</code>.
Pointerdown must precede focus, and <code>:focus-visible</code> must be false for pointer focus.</p>
<input id="t" placeholder="click then type x" style="font-size:18px;padding:8px 12px;width:240px">
</main>
<script>
Challenge.init({ id: "focus-order", instructions: "click input then type 'x'" });
const t = document.getElementById("t");
let lastPointer = -Infinity, lastFocus = -Infinity, sawPointer = false;
t.addEventListener("pointerdown", (e) => { lastPointer = e.timeStamp; sawPointer = true; });
t.addEventListener("focus", (e) => {
lastFocus = e.timeStamp;
if (!sawPointer) Challenge.fail("focus arrived with no preceding pointerdown");
else if (lastFocus < lastPointer) Challenge.fail("focus before pointerdown");
});
t.addEventListener("input", () => {
if (t.value !== "x") return;
// For pointer-driven focus, :focus-visible should be false.
const fv = t.matches(":focus-visible");
Challenge.log("focus-visible", { fv });
if (fv) Challenge.fail(":focus-visible true after pointer click (looks like keyboard focus)");
else if (window.__verdict !== "FAIL") Challenge.pass("pointerdown→focus→input order ok, :focus-visible=false");
});
</script>
</body>
@@ -0,0 +1,28 @@
<!doctype html>
<meta charset="utf-8">
<title>14 wheel scroll</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: scroll the box to the bottom. Page rejects raw <code>scrollTop</code> assignment without
<code>wheel</code> events.</p>
<div id="box" style="height:200px;overflow:auto;border:1px solid #555;background:#fff;color:#111;padding:8px">
<div style="height:1200px">scroll me ⬇<br><br>...lots of content...</div>
</div>
</main>
<script>
Challenge.init({ id: "wheel-scroll", instructions: "scroll the box to its bottom" });
const box = document.getElementById("box");
let wheelCount = 0, lastWheelTs = -Infinity;
box.addEventListener("wheel", (e) => { wheelCount++; lastWheelTs = e.timeStamp; Challenge.log("wheel", { dy: e.deltaY }); }, { passive: true });
box.addEventListener("scroll", () => {
Challenge.log("scroll", { top: box.scrollTop });
if (box.scrollTop + box.clientHeight >= box.scrollHeight - 2) {
if (wheelCount === 0) Challenge.fail("scrolled to bottom with zero wheel events");
else if (performance.now() - lastWheelTs > 1500) Challenge.fail("wheel events too far before final scroll");
else Challenge.pass(`${wheelCount} wheel events accompanied scroll`);
}
});
</script>
</body>
@@ -0,0 +1,73 @@
<!doctype html>
<meta charset="utf-8">
<title>15 drag-drop DataTransfer</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: drag the <b>red box</b> into the <b>drop zone</b>. Page asserts a full HTML5
drag sequence with a populated <code>DataTransfer</code> — synthetic pointer drags
dispatched without <code>dragstart</code>/<code>drop</code> + DataTransfer payload fail.</p>
<div style="display:flex;gap:24px;align-items:center;margin-top:16px">
<div id="src" draggable="true"
style="width:80px;height:80px;background:#c33;color:#fff;display:flex;align-items:center;justify-content:center;border-radius:6px;cursor:grab">
DRAG
</div>
<div id="dst"
style="width:200px;height:120px;border:2px dashed #888;display:flex;align-items:center;justify-content:center;color:#888">
DROP HERE
</div>
</div>
</main>
<script>
Challenge.init({ id: "drag-drop-datatransfer", instructions: "drag red box into drop zone via real HTML5 drag" });
const src = document.getElementById("src");
const dst = document.getElementById("dst");
const seen = { dragstart: 0, drag: 0, dragenter: 0, dragover: 0, drop: 0, dragend: 0 };
let dtHadTypes = false, dtPayload = null, isTrustedAll = true;
src.addEventListener("dragstart", (e) => {
seen.dragstart++;
if (!e.isTrusted) isTrustedAll = false;
try { e.dataTransfer.setData("text/plain", "payload-" + Math.random().toString(36).slice(2,8)); } catch {}
Challenge.log("dragstart", { hasDt: !!e.dataTransfer });
});
src.addEventListener("drag", (e) => { seen.drag++; if (!e.isTrusted) isTrustedAll = false; });
src.addEventListener("dragend", (e) => { seen.dragend++; if (!e.isTrusted) isTrustedAll = false; });
dst.addEventListener("dragenter", (e) => { seen.dragenter++; e.preventDefault(); if (!e.isTrusted) isTrustedAll = false; });
dst.addEventListener("dragover", (e) => { seen.dragover++; e.preventDefault(); if (!e.isTrusted) isTrustedAll = false; });
dst.addEventListener("drop", (e) => {
e.preventDefault();
seen.drop++;
if (!e.isTrusted) isTrustedAll = false;
const dt = e.dataTransfer;
if (dt) {
dtHadTypes = (dt.types && dt.types.length > 0);
try { dtPayload = dt.getData("text/plain"); } catch {}
}
Challenge.log("drop", { types: dt && [...dt.types], payload: dtPayload });
// dragend fires AFTER drop per spec; give it a tick.
setTimeout(evaluate, 50);
});
function evaluate() {
const bad = [];
if (!isTrustedAll) bad.push("at least one drag event isTrusted=false");
for (const k of ["dragstart","dragover","drop","dragend"]) {
if (!seen[k]) bad.push(`missing ${k}`);
}
if (!dtHadTypes) bad.push("DataTransfer.types empty on drop");
if (!dtPayload || !dtPayload.startsWith("payload-")) bad.push("DataTransfer payload missing on drop");
if (bad.length) Challenge.fail(...bad);
else Challenge.pass(`full drag cycle, payload=${dtPayload}`);
}
// Fallback: pointer-drag without dragstart triggers fail after a delay.
let pointerDownInSrc = false;
src.addEventListener("pointerdown", () => { pointerDownInSrc = true;
setTimeout(() => { if (!seen.dragstart && pointerDownInSrc) Challenge.fail("pointerdown on src but no dragstart fired"); }, 1500);
});
</script>
</body>
@@ -0,0 +1,54 @@
<!doctype html>
<meta charset="utf-8">
<title>16 contenteditable selection</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: focus the editable region and type <code>hello</code>. Page asserts that
<code>window.getSelection()</code> reflects per-keystroke caret movement
(<code>rangeCount===1</code>, <code>collapsed</code> caret, anchor inside the editor,
offset advances 1 per char) and that <code>selectionchange</code> fires.</p>
<div id="ed" contenteditable="true"
style="min-height:60px;padding:10px;border:1px solid #555;background:#fff;color:#111;font:16px monospace"></div>
</main>
<script>
Challenge.init({ id: "contenteditable-selection", instructions: "focus editor and type 'hello'" });
const ed = document.getElementById("ed");
const offsets = [];
let selChanges = 0;
document.addEventListener("selectionchange", () => {
selChanges++;
const s = window.getSelection();
if (!s || s.rangeCount === 0) return;
if (!ed.contains(s.anchorNode)) return;
offsets.push({ off: s.anchorOffset, collapsed: s.isCollapsed, t: performance.now(), len: ed.textContent.length });
});
ed.addEventListener("input", async () => {
if (ed.textContent !== "hello") return;
// selectionchange is async — wait a tick so the final caret update is observed.
await new Promise((r) => setTimeout(r, 30));
const bad = [];
const sel = window.getSelection();
if (!sel) bad.push("getSelection()==null");
else {
if (sel.rangeCount !== 1) bad.push(`rangeCount=${sel.rangeCount} (need 1)`);
if (!sel.isCollapsed) bad.push("selection not collapsed after typing");
if (!ed.contains(sel.anchorNode)) bad.push("selection anchor outside editor");
if (sel.anchorOffset !== 5) bad.push(`anchorOffset=${sel.anchorOffset} (need 5)`);
}
if (selChanges < 5) bad.push(`selectionchange count=${selChanges} (need ≥5)`);
// offsets should advance monotonically 1,2,3,4,5 (allowing extras from focus)
const monotonic = offsets.map(o => o.off);
const peak = Math.max(0, ...monotonic);
if (peak !== 5) bad.push(`peak caret offset=${peak} during typing (need 5)`);
// every caret reading must be collapsed (no spurious ranges)
if (offsets.some(o => !o.collapsed)) bad.push("non-collapsed range observed during typing");
Challenge.log("sel", { offsets, selChanges });
if (bad.length) Challenge.fail(...bad);
else Challenge.pass(`selectionchanges=${selChanges}, caret advanced 0→5`);
});
</script>
</body>
@@ -0,0 +1,48 @@
<!doctype html>
<meta charset="utf-8">
<title>17 paste clipboard</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: paste the string <code>pi-chrome</code> into the input via a real OS paste
(Cmd/Ctrl+V firing a trusted <code>paste</code> event with populated
<code>clipboardData</code>). Programmatic <code>value=</code> or <code>setRangeText</code>
fail. The <code>input</code> event's <code>inputType</code> must be
<code>insertFromPaste</code>.</p>
<input id="t" placeholder="paste here" style="font-size:18px;padding:8px 12px;width:280px">
</main>
<script>
Challenge.init({ id: "paste-clipboard", instructions: "paste 'pi-chrome' via OS clipboard" });
const t = document.getElementById("t");
let sawPaste = false, pasteIsTrusted = false, pastePayload = null;
t.addEventListener("paste", (e) => {
sawPaste = true;
pasteIsTrusted = e.isTrusted;
try { pastePayload = e.clipboardData && e.clipboardData.getData("text/plain"); } catch {}
Challenge.log("paste", { isTrusted: e.isTrusted, payload: pastePayload });
});
let lastInputType = null;
t.addEventListener("input", (e) => {
lastInputType = e.inputType;
if (t.value !== "pi-chrome") return;
const bad = [];
if (!sawPaste) bad.push("no paste event fired before value matched");
if (!pasteIsTrusted) bad.push("paste event isTrusted=false");
if (pastePayload !== "pi-chrome") bad.push(`clipboardData payload="${pastePayload}" (need 'pi-chrome')`);
if (lastInputType !== "insertFromPaste") bad.push(`input.inputType="${lastInputType}" (need 'insertFromPaste')`);
if (bad.length) Challenge.fail(...bad);
else Challenge.pass("trusted paste with clipboardData text/plain match");
});
// Also fail loudly if value is set without any input event (raw .value=)
let inputFired = false;
t.addEventListener("input", () => { inputFired = true; });
const obs = new MutationObserver(() => {});
obs.observe(t, { attributes: true, attributeFilter: ["value"] });
setInterval(() => {
if (t.value === "pi-chrome" && !inputFired) Challenge.fail("value matched without input event firing");
}, 200);
</script>
</body>
@@ -0,0 +1,56 @@
<!doctype html>
<meta charset="utf-8">
<title>18 native select option</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: select the option <code>banana</code> from the native <code>&lt;select&gt;</code>.
Page asserts a real picker interaction: <code>mousedown</code>/<code>pointerdown</code>
on the &lt;select&gt; (opening the OS picker), then a <code>change</code> event with
<code>isTrusted=true</code>. Programmatically setting <code>.value</code> or clicking the
option element directly fails (clicking &lt;option&gt; does not change selection in
real browsers).</p>
<select id="s" style="font-size:18px;padding:6px 10px">
<option value="">— pick a fruit —</option>
<option value="apple">apple</option>
<option value="banana">banana</option>
<option value="cherry">cherry</option>
</select>
</main>
<script>
Challenge.init({ id: "native-select", instructions: "pick 'banana' from native select" });
const s = document.getElementById("s");
let sawPointerDownOnSelect = false, pointerDownTrusted = false;
s.addEventListener("pointerdown", (e) => { sawPointerDownOnSelect = true; pointerDownTrusted = e.isTrusted; });
s.addEventListener("mousedown", (e) => { sawPointerDownOnSelect = true; pointerDownTrusted ||= e.isTrusted; });
// Trap: if a click bubbles up from an <option>, that's bot-like (real OS pickers don't dispatch this).
s.addEventListener("click", (e) => {
if (e.target && e.target.tagName === "OPTION") {
Challenge.fail("click event with target=<option> bubbled to <select> (synthetic option click)");
}
});
let changeTrusted = false;
s.addEventListener("change", (e) => {
changeTrusted = e.isTrusted;
Challenge.log("change", { v: s.value, isTrusted: e.isTrusted });
if (s.value !== "banana") return Challenge.fail(`selected="${s.value}" (need 'banana')`);
const bad = [];
if (!changeTrusted) bad.push("change event isTrusted=false");
if (!sawPointerDownOnSelect) bad.push("no pointerdown/mousedown on <select> before change");
if (!pointerDownTrusted) bad.push("pointerdown on select isTrusted=false");
if (bad.length) Challenge.fail(...bad);
else Challenge.pass("native picker opened and trusted change fired");
});
// Also trip if value mutates without change firing.
let changed = false;
s.addEventListener("change", () => { changed = true; });
setInterval(() => {
if (s.value === "banana" && !changed) Challenge.fail("value='banana' without any change event");
}, 200);
</script>
</body>
@@ -0,0 +1,50 @@
<!doctype html>
<meta charset="utf-8">
<title>19 hover dwell</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: hover the trigger and wait until the <b>Confirm</b> button reveals (≥600ms dwell
with continuous <code>pointermove</code> activity inside the trigger), then click Confirm.
Instant hover-then-click without dwell, or hover with no intermediate pointermove,
looks robotic and fails.</p>
<div id="trig" style="display:inline-block;padding:16px 24px;background:#2a4;color:#fff;border-radius:6px;cursor:pointer">
hover me
</div>
<button id="go" style="display:none;margin-left:12px;padding:10px 18px;background:#1f7a1f;color:#fff;border:0;border-radius:6px">Confirm</button>
</main>
<script>
Challenge.init({ id: "hover-dwell", instructions: "hover ≥600ms with movement, then click Confirm" });
const trig = document.getElementById("trig");
const btn = document.getElementById("go");
let enterTs = 0, lastMoveTs = 0, moveCount = 0;
trig.addEventListener("pointerenter", (e) => { enterTs = e.timeStamp; lastMoveTs = e.timeStamp; moveCount = 0; });
trig.addEventListener("pointermove", (e) => { lastMoveTs = e.timeStamp; moveCount++; });
trig.addEventListener("pointerleave", () => {
if (!btn.dataset.revealed) { enterTs = 0; lastMoveTs = 0; moveCount = 0; }
});
// Reveal after sustained dwell + activity.
setInterval(() => {
if (btn.dataset.revealed) return;
if (!enterTs) return;
const dwell = performance.now() - enterTs;
if (dwell >= 600 && moveCount >= 3) {
btn.style.display = "inline-block";
btn.dataset.revealed = "1";
btn.dataset.revealTs = performance.now();
Challenge.log("revealed", { dwell, moveCount });
}
}, 50);
btn.addEventListener("click", (e) => {
if (!btn.dataset.revealed) return Challenge.fail("Confirm clicked before reveal (display:none)");
const sinceReveal = e.timeStamp - Number(btn.dataset.revealTs);
if (sinceReveal < 80) return Challenge.fail(`clicked ${sinceReveal.toFixed(0)}ms after reveal (too instant)`);
if (!e.isTrusted) return Challenge.fail("Confirm click isTrusted=false");
Challenge.pass(`dwell+activity+human-latency reveal (Δ=${sinceReveal.toFixed(0)}ms)`);
});
</script>
</body>
@@ -0,0 +1,78 @@
<!doctype html>
<meta charset="utf-8">
<title>20 React _valueTracker</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: set the input to <code>react-ok</code> in a way that React would accept.
The page mimics React's <code>_valueTracker</code>: it stamps the input with a
hidden tracker holding the last-seen value. An <code>input</code> event is only
considered "framework-real" if the tracker's recorded value differs from the
current <code>input.value</code> at the moment the event fires
(i.e. somebody used the native <code>HTMLInputElement.value</code> setter and
dispatched <code>input</code> — what React internals require).</p>
<p>Setting <code>el.value = ...</code> directly and dispatching a synthetic <code>input</code>
fails because the native setter wasn't used. Calling <code>setRangeText</code> without
dispatching <code>input</code> also fails.</p>
<input id="t" placeholder="type or fill 'react-ok'" style="font-size:18px;padding:8px 12px;width:240px">
</main>
<script>
Challenge.init({ id: "react-value-tracker", instructions: "make input value 'react-ok' as React expects" });
const t = document.getElementById("t");
// Install a fake React-style _valueTracker.
(function attachTracker(node) {
const nativeProto = Object.getPrototypeOf(node);
const desc = Object.getOwnPropertyDescriptor(nativeProto, "value");
const nativeGet = desc.get, nativeSet = desc.set;
let tracked = nativeGet.call(node); // last value the "framework" has seen
node._valueTracker = {
getValue() { return tracked; },
setValue(v) { tracked = v; },
};
// Wrap the value property on the *instance* to intercept assignments.
Object.defineProperty(node, "value", {
configurable: true,
get() { return nativeGet.call(this); },
set(v) {
// If somebody assigns via instance property (the bad path), sync the tracker
// so the input event below will look "stale" and we can detect it.
tracked = String(v);
nativeSet.call(this, v);
},
});
})(t);
let passed = false;
t.addEventListener("input", (e) => {
if (passed) return;
const proto = Object.getPrototypeOf(t);
const nativeGet = Object.getOwnPropertyDescriptor(proto, "value").get;
const real = nativeGet.call(t);
const tracked = t._valueTracker.getValue();
Challenge.log("input", { real, tracked, isTrusted: e.isTrusted });
if (real !== "react-ok") return;
if (!e.isTrusted && tracked === real) {
return Challenge.fail("synthetic input event but _valueTracker already up-to-date (instance value= setter used, not native)");
}
// The "good" path: native setter ran (tracker still stale), then input dispatched.
// After the framework consumes the event it would syncs the tracker.
t._valueTracker.setValue(real);
passed = true;
Challenge.pass(`value 'react-ok' delivered with stale tracker (native setter path); isTrusted=${e.isTrusted}`);
});
// Trap: value mutates without input ever firing.
setInterval(() => {
if (passed) return;
const proto = Object.getPrototypeOf(t);
const nativeGet = Object.getOwnPropertyDescriptor(proto, "value").get;
if (nativeGet.call(t) === "react-ok" && !passed) {
// Only fail if we've been sitting on the value with no event for a while.
setTimeout(() => { if (!passed && nativeGet.call(t) === "react-ok") Challenge.fail("value='react-ok' but no input event consumed it"); }, 600);
}
}, 300);
</script>
</body>
@@ -0,0 +1,65 @@
<!doctype html>
<meta charset="utf-8">
<title>21 keyboard modifiers</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: focus the input and type a capital <code>A</code> via <b>Shift+a</b>
(not Caps Lock, not pasted, not assigned). Page asserts a real Shift chord:
<code>keydown</code> for <code>Shift</code> arrives <em>before</em> <code>keydown</code> for
<code>A</code>; the <code>A</code> event has <code>shiftKey=true</code>,
<code>key==="A"</code>, <code>code==="KeyA"</code>, <code>getModifierState("Shift")===true</code>;
and <code>keyup</code> for Shift arrives <em>after</em> <code>keyup</code> for <code>A</code>.</p>
<input id="t" placeholder="type Shift+a" style="font-size:18px;padding:8px 12px;width:240px">
</main>
<script>
Challenge.init({ id: "keyboard-modifiers", instructions: "type capital A using Shift+a chord" });
const t = document.getElementById("t");
const log = [];
for (const name of ["keydown","keypress","keyup","input"]) {
t.addEventListener(name, (e) => {
log.push({
name,
key: e.key, code: e.code,
shiftKey: e.shiftKey, isTrusted: e.isTrusted,
modShift: e.getModifierState ? e.getModifierState("Shift") : null,
t: e.timeStamp,
});
});
}
t.addEventListener("input", () => {
if (t.value !== "A") return;
const bad = [];
const downShift = log.find(e => e.name === "keydown" && e.key === "Shift");
const downA = log.find(e => e.name === "keydown" && e.code === "KeyA");
const upShift = log.find(e => e.name === "keyup" && e.key === "Shift");
const upA = log.find(e => e.name === "keyup" && e.code === "KeyA");
if (!downShift) bad.push("no keydown for Shift");
if (!downA) bad.push("no keydown for KeyA");
if (!upShift) bad.push("no keyup for Shift");
if (!upA) bad.push("no keyup for KeyA");
if (downShift && downA && downShift.t > downA.t) bad.push("Shift keydown after A keydown");
if (upShift && upA && upShift.t < upA.t) bad.push("Shift keyup before A keyup");
if (downA) {
if (downA.key !== "A") bad.push(`A keydown.key="${downA.key}" (need 'A')`);
if (downA.code !== "KeyA") bad.push(`A keydown.code="${downA.code}"`);
if (!downA.shiftKey) bad.push("A keydown.shiftKey=false");
if (downA.modShift !== true) bad.push("getModifierState('Shift')!==true on A keydown");
if (!downA.isTrusted) bad.push("A keydown isTrusted=false");
}
if (downShift) {
if (!downShift.isTrusted) bad.push("Shift keydown isTrusted=false");
}
Challenge.log("chord", { log });
if (bad.length) Challenge.fail(...bad);
else Challenge.pass("Shift+a chord, modifiers + ordering correct");
});
</script>
</body>
@@ -0,0 +1,66 @@
<!doctype html>
<meta charset="utf-8">
<title>22 touch events</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: tap the green pad on a touch-capable device (or via DevTools touch
emulation). Page asserts a real <code>touchstart</code>/<code>touchend</code> sequence:
<code>touches</code> / <code>changedTouches</code> are
<code>TouchList</code> objects containing <code>Touch</code> instances with
<code>identifier</code>, <code>clientX/Y</code>, <code>radiusX/Y</code>, and
<code>force</code> in [0,1]. Synthetic <code>PointerEvent</code> with
<code>pointerType="touch"</code> alone does not satisfy this — touch events must
fire too.</p>
<p><b>Note for caller:</b> requires Chrome touch emulation on (DevTools → Sensors →
"Touch: Force enabled") or a real touchscreen. This is intentional: pi-chrome's
synthetic <code>pointerType:'touch'</code> does NOT also dispatch TouchEvents.</p>
<div id="pad" style="width:200px;height:200px;background:#1f7a1f;border-radius:12px;color:#fff;display:flex;align-items:center;justify-content:center;font:18px monospace;touch-action:none;user-select:none">
TAP
</div>
</main>
<script>
Challenge.init({ id: "touch-events", instructions: "tap the pad on a touch-capable device" });
const pad = document.getElementById("pad");
let touchStart = null, touchEnd = null;
pad.addEventListener("touchstart", (e) => {
touchStart = e;
Challenge.log("touchstart", { tl: e.touches.length, ct: e.changedTouches.length, isTrusted: e.isTrusted });
}, { passive: true });
pad.addEventListener("touchend", (e) => {
touchEnd = e;
const bad = [];
if (!touchStart) return Challenge.fail("touchend without touchstart");
if (!e.isTrusted || !touchStart.isTrusted) bad.push("touch event isTrusted=false");
const ts = touchStart.changedTouches;
if (!ts || ts.length === 0) bad.push("touchstart.changedTouches empty");
else {
const T = ts[0];
// Window.Touch should exist as the constructor.
if (typeof window.Touch !== "function") bad.push("window.Touch constructor missing (no native TouchEvent support)");
if (!(T instanceof Touch)) bad.push("changedTouches[0] not instanceof Touch");
if (typeof T.identifier !== "number") bad.push("Touch.identifier not number");
if (!Number.isFinite(T.clientX) || !Number.isFinite(T.clientY)) bad.push("Touch.clientX/Y invalid");
if (!("force" in T) || typeof T.force !== "number" || T.force < 0 || T.force > 1) bad.push(`Touch.force=${T.force}`);
if (!("radiusX" in T) || typeof T.radiusX !== "number") bad.push("Touch.radiusX missing");
}
if (!(touchStart instanceof TouchEvent)) bad.push("touchstart not instanceof TouchEvent");
if (bad.length) Challenge.fail(...bad);
else Challenge.pass("real TouchEvent + Touch object surface present");
}, { passive: true });
// Capability check: if TouchEvent constructor missing entirely, hint immediately on click.
pad.addEventListener("click", () => {
if (!touchStart && typeof window.TouchEvent !== "function") {
Challenge.fail("TouchEvent constructor not available on this UA (enable touch emulation)");
} else if (!touchStart) {
Challenge.fail("click fired but no touchstart — synthetic pointer without TouchEvent");
}
});
</script>
</body>
@@ -0,0 +1,76 @@
<!doctype html>
<meta charset="utf-8">
<title>23 stack-trace fingerprint</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: click the button. The page samples call stacks from inside several
instrumented globals (<code>Function.prototype.toString</code>,
<code>document.querySelector</code>, <code>Element.prototype.click</code>) and inspects
the stack of <em>this script's own</em> handler. It fails if it sees telltales of
evaluator-injected frames (e.g. <code>at &lt;anonymous&gt;</code> as the only frame, the
bridge's <code>new Function</code> wrapper, <code>callFunctionOn</code>,
<code>executeScript</code>, or extension URLs).</p>
<button id="go" style="padding:14px 22px;font-size:16px;background:#1f7a1f;color:#fff;border:0;border-radius:6px">Click me</button>
</main>
<script>
Challenge.init({ id: "stack-trace-fingerprint", instructions: "click the button" });
// Suspicious stack-frame patterns commonly observed when code is invoked via
// CDP Runtime.evaluate / chrome.scripting.executeScript / new Function bodies
// dispatched from an MV3 service worker.
const SUSPICIOUS = [
/chrome-extension:\/\//i,
/\bnew Function\b/,
/Runtime\.evaluate/i,
/Runtime\.callFunctionOn/i,
/executeScript/i,
/content[_-]?script/i,
/^\s*at\s+eval\b/m,
];
function inspectStack(stack) {
if (!stack) return ["empty stack"];
const hits = SUSPICIOUS.filter(r => r.test(stack)).map(r => r.source);
const lines = stack.split("\n").filter(l => l.trim().startsWith("at "));
// Do not fail on generic <anonymous> frames alone: inline scripts, extensions,
// and browser versions vary here. This test should catch concrete automation
// tells, not punish legitimate stack formatting differences.
const reasons = [];
if (hits.length) reasons.push("suspicious frames: " + hits.join(","));
return reasons;
}
// Hook some commonly-touched APIs so any pre-click bridge instrumentation also
// leaves a trail. Their stacks get inspected the moment the click handler fires.
const probeStacks = [];
const oTo = Function.prototype.toString;
Function.prototype.toString = function () {
probeStacks.push({ where: "Function.toString", stack: new Error().stack });
return oTo.apply(this, arguments);
};
const oQS = Document.prototype.querySelector;
Document.prototype.querySelector = function (sel) {
probeStacks.push({ where: "document.querySelector", sel, stack: new Error().stack });
return oQS.apply(this, arguments);
};
document.getElementById("go").addEventListener("click", (e) => {
const ownStack = new Error().stack || "";
const reasons = inspectStack(ownStack);
// Also check any probe stacks gathered before the click — the bridge often
// queries the DOM right before dispatching.
const probeBad = [];
for (const p of probeStacks) {
const r = inspectStack(p.stack);
if (r.length) probeBad.push(`${p.where}: ${r.join("; ")}`);
}
Challenge.log("stacks", { ownStack, probeStacks });
if (!e.isTrusted) return Challenge.fail("click isTrusted=false");
if (reasons.length) return Challenge.fail(...reasons);
if (probeBad.length) return Challenge.fail(...probeBad.slice(0, 3));
Challenge.pass("call stack matches an in-page event handler");
});
</script>
</body>
@@ -0,0 +1,51 @@
<!doctype html>
<meta charset="utf-8">
<title>24 viewport edge clicks</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: click the button. Page rejects any pointer event whose
<code>clientX</code>/<code>clientY</code> falls outside the viewport
(<code>x &lt; 0</code>, <code>x &gt; innerWidth</code>, etc.) or lands on a coordinate
that's not actually inside the button's <code>getBoundingClientRect()</code>.
Real pointing devices can't dispatch clicks at negative coordinates;
bridges that pass uncoerced selector-center coords (or coordinates from a
stale layout) trip this.</p>
<button id="go" style="margin:40px;padding:14px 22px;font-size:16px;background:#1f7a1f;color:#fff;border:0;border-radius:6px">Click me</button>
</main>
<script>
Challenge.init({ id: "viewport-edge-clicks", instructions: "click the button at valid viewport coords" });
const btn = document.getElementById("go");
// Track ALL pointer/click events on document — even ones that miss the button.
let suspicious = null;
for (const name of ["pointerdown","pointerup","click","mousedown","mouseup"]) {
document.addEventListener(name, (e) => {
const x = e.clientX, y = e.clientY;
const w = window.innerWidth, h = window.innerHeight;
if (x < 0 || y < 0 || x > w || y > h) {
suspicious = `${name} at (${x},${y}) outside viewport ${w}x${h}`;
Challenge.fail(suspicious);
}
if ((x === 0 && y === 0) && e.isTrusted === false) {
suspicious = `${name} at (0,0) with isTrusted=false (default synthetic coords)`;
Challenge.fail(suspicious);
}
}, true);
}
btn.addEventListener("click", (e) => {
if (suspicious) return;
if (!e.isTrusted) return Challenge.fail("click isTrusted=false");
const r = btn.getBoundingClientRect();
const x = e.clientX, y = e.clientY;
const inside = x >= r.left && x <= r.right && y >= r.top && y <= r.bottom;
if (!inside) return Challenge.fail(`click at (${x},${y}) outside button rect ${JSON.stringify({l:r.left,t:r.top,r:r.right,b:r.bottom})}`);
// Coord must not be exactly integer-center (covered by #06 but reinforced here).
Challenge.log("click", { x, y, rect: r });
Challenge.pass(`click at (${x.toFixed(1)},${y.toFixed(1)}) inside button rect`);
});
</script>
</body>
@@ -0,0 +1,62 @@
<!doctype html>
<meta charset="utf-8">
<title>25 pointer continuity</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: click <b>A</b>, then click <b>B</b>. Page requires pointer continuity:
between the two clicks there must be intermediate <code>pointermove</code> events
whose path roughly connects the two click points (no teleport). Two clicks
&gt;100px apart with zero moves between is a synthetic-bridge tell.</p>
<div style="display:flex;justify-content:space-between;margin-top:20px">
<button id="a" style="padding:14px 22px;background:#36c;color:#fff;border:0;border-radius:6px">A</button>
<div style="flex:1"></div>
<button id="b" style="padding:14px 22px;background:#c36;color:#fff;border:0;border-radius:6px">B</button>
</div>
</main>
<script>
Challenge.init({ id: "pointer-continuity", instructions: "click A, move mouse to B, click B" });
const a = document.getElementById("a");
const b = document.getElementById("b");
const moves = [];
window.addEventListener("pointermove", (e) => {
moves.push({ x: e.clientX, y: e.clientY, t: e.timeStamp });
if (moves.length > 500) moves.shift();
});
let aClickTs = 0, aPoint = null;
a.addEventListener("click", (e) => {
if (!e.isTrusted) return Challenge.fail("A click isTrusted=false");
aClickTs = e.timeStamp;
aPoint = { x: e.clientX, y: e.clientY };
Challenge.log("A click", aPoint);
});
b.addEventListener("click", (e) => {
if (!aClickTs) return Challenge.fail("B clicked before A");
if (!e.isTrusted) return Challenge.fail("B click isTrusted=false");
const bPoint = { x: e.clientX, y: e.clientY };
const between = moves.filter(m => m.t > aClickTs && m.t < e.timeStamp);
Challenge.log("B click", { bPoint, betweenCount: between.length });
const dist = Math.hypot(bPoint.x - aPoint.x, bPoint.y - aPoint.y);
if (dist < 100) return Challenge.fail(`A-B distance only ${dist.toFixed(0)}px — buttons should be further apart in viewport`);
if (between.length < 5) return Challenge.fail(`only ${between.length} pointermove(s) between A and B (need ≥5)`);
// Path coverage: at least some intermediate move must be ≥30% of the way across.
const inSpan = between.filter(m => {
const px = (m.x - aPoint.x) / (bPoint.x - aPoint.x || 1);
return px > 0.25 && px < 0.75;
});
if (inSpan.length === 0) return Challenge.fail("no pointermove samples in mid-span between A and B (teleport)");
// Move timestamps should be spread, not bunched in <20ms.
const span = between[between.length-1].t - between[0].t;
if (span < 30) return Challenge.fail(`pointermoves bunched in ${span.toFixed(0)}ms (need spread ≥30ms)`);
Challenge.pass(`${between.length} moves bridging A→B over ${span.toFixed(0)}ms, dist=${dist.toFixed(0)}px`);
});
</script>
</body>
@@ -0,0 +1,57 @@
<!doctype html>
<meta charset="utf-8">
<title>26 mousemove rate</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: sweep the mouse across the box for ~1s, then click the green button.
Page measures the inter-event timestamp distribution of <code>mousemove</code>
events. Real pointing devices fire at ~60250 Hz (median Δt 418ms). Bridges
that flood synthetic moves on a tight <code>for</code>-loop or
<code>setTimeout(...,1)</code> produce a degenerate distribution
(most Δt &lt; 2ms, or all Δt identical, or median &gt; 50ms with no jitter).</p>
<div id="pad" style="height:160px;border:1px solid #555;background:#fff;color:#111;display:flex;align-items:center;justify-content:center;font:13px monospace;user-select:none">
sweep me
</div>
<button id="go" style="margin-top:14px;padding:12px 20px;background:#1f7a1f;color:#fff;border:0;border-radius:6px">Done</button>
</main>
<script>
Challenge.init({ id: "mousemove-rate", instructions: "sweep mouse across pad ~1s, then click Done" });
const pad = document.getElementById("pad");
const stamps = [];
pad.addEventListener("mousemove", (e) => { stamps.push(e.timeStamp); });
document.getElementById("go").addEventListener("click", (e) => {
if (!e.isTrusted) return Challenge.fail("Done click isTrusted=false");
if (stamps.length < 20) return Challenge.fail(`only ${stamps.length} mousemove events (need ≥20)`);
const deltas = [];
for (let i = 1; i < stamps.length; i++) deltas.push(stamps[i] - stamps[i-1]);
deltas.sort((a,b) => a-b);
const median = deltas[Math.floor(deltas.length/2)];
const p10 = deltas[Math.floor(deltas.length*0.1)];
const p90 = deltas[Math.floor(deltas.length*0.9)];
const mean = deltas.reduce((s,x)=>s+x,0) / deltas.length;
const variance = deltas.reduce((s,x)=>s+(x-mean)**2,0) / deltas.length;
const std = Math.sqrt(variance);
const allSame = deltas.every(d => Math.abs(d - deltas[0]) < 0.05);
const allTiny = deltas.every(d => d < 1.5);
Challenge.log("dist", { n: deltas.length, median, p10, p90, mean, std });
const bad = [];
if (allSame) bad.push(`all Δt identical (~${deltas[0].toFixed(2)}ms) — scripted loop`);
if (allTiny) bad.push(`all Δt < 1.5ms — synthetic flood`);
if (median < 2) bad.push(`median Δt ${median.toFixed(2)}ms too fast for a real device`);
if (median > 60) bad.push(`median Δt ${median.toFixed(0)}ms too slow (real moves ~418ms)`);
if (std < 0.5) bad.push(`Δt std=${std.toFixed(2)}ms — no jitter (scripted)`);
// Ratio p90/p10 should be ≥1.8 for organic motion.
if (p10 > 0 && (p90 / p10) < 1.5) bad.push(`p90/p10 ratio ${(p90/p10).toFixed(2)} — distribution too narrow`);
if (bad.length) Challenge.fail(...bad);
else Challenge.pass(`n=${deltas.length}, median=${median.toFixed(1)}ms, std=${std.toFixed(1)}ms`);
});
</script>
</body>
@@ -0,0 +1,66 @@
<!doctype html>
<meta charset="utf-8">
<title>27 scroll momentum tail</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: scroll the box to its bottom using a real wheel/trackpad gesture.
Page records the <code>deltaY</code> sequence and requires a momentum-like tail:
a peak <code>|deltaY|</code>, then a decreasing decay over several events
(≥4 wheel events whose <code>|deltaY|</code> is strictly less than the previous
by &gt;5%). Constant or single-shot wheel events (one big delta then nothing)
fail.</p>
<div id="box" style="height:200px;overflow:auto;border:1px solid #555;background:#fff;color:#111;padding:8px">
<div style="height:2400px">scroll me ⬇ … continues …</div>
</div>
</main>
<script>
Challenge.init({ id: "scroll-momentum", instructions: "scroll the box to its bottom with a real gesture" });
const box = document.getElementById("box");
const deltas = []; // signed
const tsLog = [];
box.addEventListener("wheel", (e) => {
deltas.push(e.deltaY);
tsLog.push(e.timeStamp);
if (deltas.length > 200) { deltas.shift(); tsLog.shift(); }
Challenge.log("wheel", { dy: e.deltaY, isTrusted: e.isTrusted });
if (!e.isTrusted) Challenge.fail("wheel event isTrusted=false");
}, { passive: true });
box.addEventListener("scroll", () => {
if (box.scrollTop + box.clientHeight < box.scrollHeight - 2) return;
// We hit bottom. Analyze the deltaY trace.
if (deltas.length === 0) return Challenge.fail("reached bottom with zero wheel events");
if (deltas.length < 6) return Challenge.fail(`only ${deltas.length} wheel events (need ≥6 for momentum)`);
const abs = deltas.map(Math.abs);
// All same sign? Trackpad inertia keeps signs consistent.
const allSameSign = deltas.every(d => Math.sign(d) === Math.sign(deltas[0]));
if (!allSameSign) return Challenge.fail("wheel deltaY signs flip — not a continuous scroll gesture");
// All identical magnitudes? Robotic.
const allEqual = abs.every(a => Math.abs(a - abs[0]) < 0.01);
if (allEqual) return Challenge.fail(`all |deltaY|=${abs[0]} — no momentum`);
// Find peak and require decay tail after it.
const peakIdx = abs.indexOf(Math.max(...abs));
const tail = abs.slice(peakIdx);
let decays = 0;
for (let i = 1; i < tail.length; i++) {
if (tail[i] < tail[i-1] * 0.95) decays++;
}
Challenge.log("trace", { abs, peakIdx, decays });
if (decays < 3) return Challenge.fail(`only ${decays} decay steps after peak — no momentum tail`);
// Timestamps within the tail should span ≥150ms (real inertia lasts hundreds of ms).
const tailTs = tsLog.slice(peakIdx);
const span = tailTs[tailTs.length-1] - tailTs[0];
if (span < 100) return Challenge.fail(`momentum tail only spans ${span.toFixed(0)}ms`);
Challenge.pass(`${deltas.length} wheels, peak |dY|=${abs[peakIdx].toFixed(1)}, ${decays} decay steps over ${span.toFixed(0)}ms`);
});
</script>
</body>
@@ -0,0 +1,72 @@
<!doctype html>
<meta charset="utf-8">
<title>28 IntersectionObserver visibility</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: scroll until the hidden <b>TARGET</b> button comes into view, then click it.
Page uses an <code>IntersectionObserver</code> to track when TARGET crosses the
viewport. Real scrolling produces a sequence of intersection updates with
gradually increasing <code>intersectionRatio</code> (0 → 1) across animation
frames. Programmatic teleports (<code>scrollIntoView({behavior:"instant"})</code>,
direct <code>scrollTop=</code>) produce exactly one observer callback with
<code>ratio≈1</code> and no rAF samples in between — fail.</p>
<div id="scroller" style="height:240px;overflow:auto;border:1px solid #555;background:#fff;color:#111">
<div style="height:1600px;padding:8px">keep scrolling …</div>
<button id="target" style="margin:0 auto 1600px;display:block;padding:14px 22px;background:#1f7a1f;color:#fff;border:0;border-radius:6px">TARGET</button>
</div>
</main>
<script>
Challenge.init({ id: "intersection-visibility", instructions: "scroll until TARGET visible, then click" });
const scroller = document.getElementById("scroller");
const target = document.getElementById("target");
const ratioSamples = [];
const rafSamples = [];
const io = new IntersectionObserver((entries) => {
for (const e of entries) ratioSamples.push({ r: e.intersectionRatio, t: performance.now() });
}, { root: scroller, threshold: Array.from({length: 21}, (_,i) => i/20) });
io.observe(target);
// Track rAF ticks during which scroll was happening.
let lastScrollTop = scroller.scrollTop;
let scrollingFrames = 0;
function tick() {
if (scroller.scrollTop !== lastScrollTop) {
scrollingFrames++;
rafSamples.push({ st: scroller.scrollTop, t: performance.now() });
lastScrollTop = scroller.scrollTop;
}
requestAnimationFrame(tick);
}
requestAnimationFrame(tick);
target.addEventListener("click", (e) => {
if (!e.isTrusted) return Challenge.fail("TARGET click isTrusted=false");
// Need the target to actually be in the viewport at click time.
const rect = target.getBoundingClientRect();
const sRect = scroller.getBoundingClientRect();
const inViewport = rect.top < sRect.bottom && rect.bottom > sRect.top;
if (!inViewport) return Challenge.fail("TARGET clicked while still off-screen");
// Require at least a gradient of intersection ratios from low to high.
const lows = ratioSamples.filter(s => s.r > 0 && s.r < 0.3).length;
const mids = ratioSamples.filter(s => s.r >= 0.3 && s.r < 0.7).length;
const highs = ratioSamples.filter(s => s.r >= 0.7).length;
Challenge.log("io", { samples: ratioSamples.length, lows, mids, highs, scrollingFrames });
const bad = [];
if (ratioSamples.length < 4) bad.push(`only ${ratioSamples.length} IntersectionObserver samples (teleport)`);
if (lows === 0) bad.push("no low-ratio IO samples — target appeared instantly");
if (mids === 0) bad.push("no mid-ratio IO samples — no smooth approach");
if (scrollingFrames < 5) bad.push(`only ${scrollingFrames} rAF frames saw scroll motion`);
if (bad.length) Challenge.fail(...bad);
else Challenge.pass(`IO gradient seen: lows=${lows} mids=${mids} highs=${highs}, rafFrames=${scrollingFrames}`);
});
</script>
</body>
@@ -0,0 +1,44 @@
<!doctype html>
<meta charset="utf-8">
<title>29 shadow DOM controls</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: discover controls inside an <code>open</code> Shadow DOM, click the button, then type <code>shadow-ok</code> into the shadow input.</p>
<div id="host"></div>
</main>
<script>
Challenge.init({ id: "shadow-dom-controls", instructions: "click shadow button, type 'shadow-ok' in shadow input" });
const host = document.getElementById("host");
const root = host.attachShadow({ mode: "open" });
root.innerHTML = `
<style>
.card { border:1px solid #555; border-radius:8px; padding:16px; background:#222; display:inline-block; }
button,input { font:16px system-ui; padding:8px 10px; margin:6px; }
</style>
<div class="card" role="group" aria-label="Shadow test controls">
<button id="shadowBtn">Arm shadow form</button>
<input id="shadowInput" aria-label="Shadow value" placeholder="shadow-ok" disabled>
<span id="status">waiting</span>
</div>
`;
const btn = root.getElementById("shadowBtn");
const input = root.getElementById("shadowInput");
const status = root.getElementById("status");
let armed = false;
btn.addEventListener("click", (e) => {
armed = true;
input.disabled = false;
input.focus();
status.textContent = `armed; click trusted=${e.isTrusted}`;
Challenge.log("shadow-click", { isTrusted: e.isTrusted });
});
input.addEventListener("input", (e) => {
Challenge.log("shadow-input", { value: input.value, isTrusted: e.isTrusted });
if (input.value !== "shadow-ok") return;
if (!armed) return Challenge.fail("input reached target value before shadow button click armed the form");
Challenge.pass("shadow button clicked and shadow input filled");
});
</script>
</body>
@@ -0,0 +1,44 @@
<!doctype html>
<meta charset="utf-8">
<title>30 iframe targeting</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: interact with controls inside a same-origin iframe. Type <code>frame-ok</code> then click Submit inside the frame.</p>
<iframe id="ifr" title="same-origin challenge frame" style="width:520px;height:220px;border:1px solid #555;border-radius:8px;background:#fff"></iframe>
</main>
<script>
Challenge.init({ id: "iframe-targeting", instructions: "type 'frame-ok' and click Submit inside iframe" });
const frame = document.getElementById("ifr");
frame.srcdoc = `<!doctype html><meta charset="utf-8">
<style>body{font:16px system-ui;margin:24px;background:#202020;color:#eee}input,button{font:16px system-ui;padding:8px 10px;margin:6px}</style>
<p>Inside frame: fill and submit.</p>
<input id="insideText" aria-label="Frame value" placeholder="frame-ok">
<button id="insideButton">Submit frame</button>
<pre id="log"></pre>
<script>
const input = document.getElementById('insideText');
const btn = document.getElementById('insideButton');
const log = document.getElementById('log');
let focused = false;
input.addEventListener('focus', () => { focused = true; parent.postMessage({ type:'frame-focus' }, '*'); });
input.addEventListener('input', e => parent.postMessage({ type:'frame-input', value: input.value, trusted: e.isTrusted }, '*'));
btn.addEventListener('click', e => {
const payload = { type:'frame-submit', value: input.value, focused, trusted: e.isTrusted };
log.textContent = JSON.stringify(payload, null, 2);
parent.postMessage(payload, '*');
});
<\/script>`;
window.addEventListener("message", (e) => {
if (!e.data || typeof e.data !== "object") return;
Challenge.log("frame-message", e.data);
if (e.data.type !== "frame-submit") return;
const bad = [];
if (e.data.value !== "frame-ok") bad.push(`iframe input value="${e.data.value}" (need frame-ok)`);
if (!e.data.focused) bad.push("iframe input was not focused before submit");
if (bad.length) Challenge.fail(...bad);
else Challenge.pass(`iframe controls reached; submit trusted=${e.data.trusted}`);
});
</script>
</body>
@@ -0,0 +1,30 @@
<!doctype html>
<meta charset="utf-8">
<title>31 file upload</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: attach <code>test-suite/fixtures/pi-chrome-upload.txt</code>. Page reads name and contents from the File object.</p>
<input id="file" type="file" aria-label="Upload fixture file" style="font-size:16px;padding:10px;border:1px solid #555;border-radius:6px">
<pre id="rep" style="font:12px monospace;background:#111;color:#eee;padding:12px;border-radius:6px"></pre>
</main>
<script>
Challenge.init({ id: "file-upload", instructions: "upload fixtures/pi-chrome-upload.txt" });
const input = document.getElementById("file");
const rep = document.getElementById("rep");
input.addEventListener("change", async (e) => {
const f = input.files && input.files[0];
if (!f) return Challenge.fail("change fired but input.files[0] missing");
const text = await f.text();
const info = { name: f.name, size: f.size, type: f.type, trusted: e.isTrusted, text };
rep.textContent = JSON.stringify(info, null, 2);
Challenge.log("file-change", info);
const bad = [];
if (f.name !== "pi-chrome-upload.txt") bad.push(`file.name=${f.name}`);
if (text !== "pi-chrome upload fixture\n") bad.push(`file content=${JSON.stringify(text)}`);
if (bad.length) Challenge.fail(...bad);
else Challenge.pass(`uploaded ${f.name}; change.isTrusted=${e.isTrusted}`);
});
</script>
</body>
@@ -0,0 +1,61 @@
<!doctype html>
<meta charset="utf-8">
<title>32 keyboard tab navigation</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: use keyboard only. Click Start, then Tab through fields, type <code>one</code>, <code>two</code>, <code>three</code>, Tab to Submit, press Enter.</p>
<button id="start">Start here</button>
<form id="f" style="margin-top:16px;display:grid;gap:10px;max-width:360px">
<label>First <input id="a" name="a" autocomplete="off"></label>
<label>Second <input id="b" name="b" autocomplete="off"></label>
<label>Third <input id="c" name="c" autocomplete="off"></label>
<button id="submit" type="submit">Submit</button>
</form>
<pre id="rep" style="font:12px monospace;background:#111;color:#eee;padding:12px;border-radius:6px"></pre>
</main>
<script>
Challenge.init({ id: "keyboard-tab-navigation", instructions: "keyboard-only tab flow; submit with Enter" });
const ids = ["start", "a", "b", "c", "submit"];
const focusLog = [];
const keyLog = [];
const rep = document.getElementById("rep");
for (const id of ids) {
const el = document.getElementById(id);
el.addEventListener("focus", (e) => {
focusLog.push({ id, t: e.timeStamp });
rep.textContent = JSON.stringify({ focusLog, keyLog }, null, 2);
});
el.addEventListener("keydown", (e) => {
keyLog.push({ id, key: e.key, trusted: e.isTrusted, t: e.timeStamp });
});
}
let sawPointerAfterStart = false;
document.addEventListener("pointerdown", (e) => {
if (e.target && e.target.id !== "start") sawPointerAfterStart = true;
});
document.getElementById("start").addEventListener("click", () => setTimeout(() => document.getElementById("start").focus(), 0));
document.getElementById("f").addEventListener("submit", (e) => {
e.preventDefault();
const vals = {
a: document.getElementById("a").value,
b: document.getElementById("b").value,
c: document.getElementById("c").value
};
const bad = [];
if (vals.a !== "one" || vals.b !== "two" || vals.c !== "three") bad.push(`values=${JSON.stringify(vals)}`);
const seq = focusLog.map(x => x.id).join(">");
for (const id of ids) if (!focusLog.some(x => x.id === id)) bad.push(`never focused ${id}`);
const enter = keyLog.find(x => x.id === "submit" && x.key === "Enter");
if (!enter) bad.push("no Enter keydown on submit button");
else if (!enter.trusted) bad.push("Enter on submit isTrusted=false");
const tabCount = keyLog.filter(x => x.key === "Tab").length;
if (tabCount < 4) bad.push(`only ${tabCount} Tab keydowns (need ≥4)`);
if (sawPointerAfterStart) bad.push("pointerdown after Start; flow was not keyboard-only");
Challenge.log("keyboard-submit", { vals, seq, keyLog, sawPointerAfterStart });
if (bad.length) Challenge.fail(...bad);
else Challenge.pass(`keyboard-only focus sequence ${seq}`);
});
</script>
</body>
@@ -0,0 +1,33 @@
<!doctype html>
<meta charset="utf-8">
<title>33 network and console capture</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: click button. Page emits console messages and performs a fetch. Benchmark harness should verify <code>chrome_list_console_messages</code> and <code>chrome_list_network_requests</code> captured them.</p>
<button id="go" style="padding:16px 24px;font-size:18px">Emit console + fetch</button>
<pre id="rep" style="font:12px monospace;background:#111;color:#eee;padding:12px;border-radius:6px"></pre>
</main>
<script>
Challenge.init({ id: "network-console-capture", instructions: "clear capture, click, then inspect console/network tools" });
document.getElementById("go").addEventListener("click", async (e) => {
console.log("pi-chrome-benchmark-console", { clicked: true, trusted: e.isTrusted });
console.warn("pi-chrome-benchmark-warning");
let data;
try {
const res = await fetch("data:application/json,%7B%22ok%22%3Atrue%2C%22source%22%3A%22pi-chrome-benchmark%22%7D");
data = await res.json();
} catch (err) {
return Challenge.fail("fetch failed: " + err.message);
}
document.getElementById("rep").textContent = JSON.stringify(data, null, 2);
Challenge.log("fetch-result", data);
if (data && data.ok && data.source === "pi-chrome-benchmark") {
Challenge.pass("page fetch completed; verify tool-level console/network capture separately");
} else {
Challenge.fail("unexpected fetch payload: " + JSON.stringify(data));
}
});
</script>
</body>
@@ -0,0 +1,28 @@
<!doctype html>
<meta charset="utf-8">
<title>34 dialog handling</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: handle browser dialogs. Click Prompt, enter <code>pi-chrome</code>, accept it; then click Confirm and accept it. Browser automation must expose/dismiss native dialogs.</p>
<button id="promptBtn">Prompt</button>
<button id="confirmBtn">Confirm</button>
<button id="alertBtn">Alert smoke</button>
<pre id="rep" class="code"></pre>
</main>
<script>
Challenge.init({ id: "dialog-handling", instructions: "accept prompt('pi-chrome') and confirm()" });
const state = { prompt: null, confirm: null, alert: false };
function paint(){ rep.textContent = JSON.stringify(state, null, 2); }
function check(){
paint();
if (state.prompt === "pi-chrome" && state.confirm === true) Challenge.pass("prompt and confirm handled with expected values");
}
promptBtn.addEventListener("click", () => { state.prompt = prompt("Enter token", ""); Challenge.log("prompt-result", { value: state.prompt }); check(); });
confirmBtn.addEventListener("click", () => { state.confirm = confirm("Accept benchmark confirm?"); Challenge.log("confirm-result", { value: state.confirm }); check(); });
alertBtn.addEventListener("click", () => { alert("pi-chrome alert smoke"); state.alert = true; Challenge.log("alert-dismissed", {}); paint(); });
window.addEventListener("beforeunload", e => { e.preventDefault(); e.returnValue = "benchmark beforeunload"; });
paint();
</script>
</body>
@@ -0,0 +1,35 @@
<!doctype html>
<meta charset="utf-8">
<title>35 target blank popup</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: click <code>target=_blank</code> link, detect new tab, switch to it, and verify child page reports PASS.</p>
<a id="open" target="_blank" rel="opener" href="35-target-blank-popup.html?child=1" style="font-size:18px">Open child tab</a>
<p id="msg"></p>
</main>
<script>
Challenge.init({ id: "target-blank-popup", instructions: "open child tab and inspect it" });
const qs = new URLSearchParams(location.search);
if (qs.get("child") === "1") {
document.getElementById("msg").textContent = "Child tab opened.";
Challenge.pass("child tab loaded on same origin");
try { localStorage.setItem("pi-chrome-suite:target-blank-popup", JSON.stringify({ id:"target-blank-popup", verdict:"PASS", reason:["✓ child tab loaded"], ts: Date.now() })); } catch {}
} else {
document.getElementById("msg").textContent = "Parent page. Link should open new tab.";
function maybePassFromStorage(raw) {
if (!raw) return;
try {
const v = JSON.parse(raw);
if (v.verdict === "PASS") Challenge.pass("child reported PASS via storage event/localStorage");
} catch {}
}
window.addEventListener("storage", e => {
if (e.key === "pi-chrome-suite:target-blank-popup") maybePassFromStorage(e.newValue);
});
maybePassFromStorage(localStorage.getItem("pi-chrome-suite:target-blank-popup"));
document.getElementById("open").addEventListener("click", e => Challenge.log("popup-click", { trusted: e.isTrusted }));
}
</script>
</body>
@@ -0,0 +1,49 @@
<!doctype html>
<meta charset="utf-8">
<title>36 modal focus trap</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: open modal, use Tab to cycle focus inside modal only, then Escape closes it and focus returns to opener.</p>
<button id="open">Open modal</button>
<div id="modal" role="dialog" aria-modal="true" aria-label="Focus trap modal" hidden style="position:fixed;inset:20%;background:#222;border:2px solid #6cf;border-radius:10px;padding:20px;z-index:5">
<button id="first">First</button>
<input id="middle" aria-label="Modal input">
<button id="last">Last</button>
</div>
</main>
<script>
Challenge.init({ id: "modal-focus-trap", instructions: "open modal, Tab cycles inside, Escape closes" });
const order = [];
const modal = document.getElementById("modal");
const nodes = [first, middle, last];
let escaped = false, modalOpened = false, outsideFocusWhileOpen = false;
for (const el of [open, ...nodes]) el.addEventListener("focus", () => { order.push(el.id); Challenge.log("focus", { id: el.id }); });
document.addEventListener("focusin", e => {
if (modalOpened && !modal.hidden && !modal.contains(e.target)) outsideFocusWhileOpen = true;
});
open.addEventListener("click", () => { modal.hidden = false; modalOpened = true; first.focus(); });
document.addEventListener("keydown", e => {
if (modal.hidden) return;
if (e.key === "Tab") {
const i = nodes.indexOf(document.activeElement);
if (i >= 0) { e.preventDefault(); nodes[(i + (e.shiftKey ? -1 : 1) + nodes.length) % nodes.length].focus(); }
}
if (e.key === "Escape") { modal.hidden = true; escaped = true; open.focus(); queueMicrotask(check); }
});
function hasAdjacent(seq, a, b) {
return seq.some((x, i) => x === a && seq[i + 1] === b);
}
function check(){
const afterOpen = order.slice(order.indexOf("first"));
const bad = [];
if (!escaped) bad.push("Escape did not close modal");
if (document.activeElement !== open) bad.push("focus did not return to opener");
if (!afterOpen.includes("middle") || !afterOpen.includes("last")) bad.push(`focus cycle incomplete: ${afterOpen.join(">")}`);
if (!hasAdjacent(afterOpen, "last", "first") && !hasAdjacent(afterOpen, "first", "last")) bad.push(`no wrap-around focus trap observed: ${afterOpen.join(">")}`);
if (outsideFocusWhileOpen) bad.push("focus escaped outside modal while open");
if (bad.length) Challenge.fail(...bad); else Challenge.pass(`modal focus trap ok: ${afterOpen.join(">")}`);
}
</script>
</body>
@@ -0,0 +1,42 @@
<!doctype html>
<meta charset="utf-8">
<title>37 autocomplete combobox</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: type <code>as</code>, use ArrowDown then Enter to choose <code>Aster Lamp</code> from ARIA combobox.</p>
<label>Product <input id="combo" role="combobox" aria-autocomplete="list" aria-controls="list" aria-expanded="false" autocomplete="off"></label>
<ul id="list" role="listbox" style="border:1px solid #555;max-width:260px;padding:6px;display:none"></ul>
</main>
<script>
Challenge.init({ id: "autocomplete-combobox", instructions: "type 'as', ArrowDown, Enter to choose Aster Lamp" });
const items = ["Aster Lamp", "Aspen Desk", "Cable Kit", "Desk Mat", "Red Mug"];
let active = -1, opened = false, keyPath = [];
combo.addEventListener("input", renderList);
combo.addEventListener("keydown", e => {
keyPath.push(e.key);
const opts = [...list.querySelectorAll('[role="option"]')];
if (e.key === "ArrowDown") { e.preventDefault(); active = Math.min(active + 1, opts.length - 1); paintActive(opts); }
if (e.key === "Enter" && active >= 0 && opts[active]) { e.preventDefault(); combo.value = opts[active].textContent; close(); check(e); }
});
function renderList(){
const q = combo.value.toLowerCase();
const matches = items.filter(x => x.toLowerCase().includes(q));
list.innerHTML = matches.map((x,i)=>`<li role="option" id="opt-${i}" style="padding:4px;cursor:pointer">${x}</li>`).join("");
list.style.display = matches.length ? "block" : "none";
combo.setAttribute("aria-expanded", matches.length ? "true" : "false");
active = -1; opened ||= matches.length > 0;
}
function paintActive(opts){ opts.forEach((o,i)=>o.style.background = i===active ? "#2b4b6b" : ""); if(opts[active]) combo.setAttribute("aria-activedescendant", opts[active].id); }
function close(){ list.style.display="none"; combo.setAttribute("aria-expanded","false"); }
function check(e){
const bad=[];
if (!opened) bad.push("listbox never opened");
if (combo.value !== "Aster Lamp") bad.push(`selected ${combo.value}`);
if (!keyPath.includes("ArrowDown") || !keyPath.includes("Enter")) bad.push(`missing keyboard selection path: ${keyPath.join(",")}`);
if (!e.isTrusted) bad.push("Enter key isTrusted=false");
if (bad.length) Challenge.fail(...bad); else Challenge.pass("ARIA combobox selected via keyboard");
}
</script>
</body>
@@ -0,0 +1,40 @@
<!doctype html>
<meta charset="utf-8">
<title>38 SPA route change</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: navigate within SPA using links/buttons. URL must change via <code>history.pushState</code> without full reload. Then go Back to Settings, forward to Billing, and Save.</p>
<nav><a href="/settings" id="settings">Settings</a> <a href="/settings/billing" id="billing">Billing</a></nav>
<section id="view"></section>
</main>
<script>
Challenge.init({ id: "spa-route-change", instructions: "Settings → Billing → Back → Billing → Save, no full reload" });
let navs = 0, popstates = 0;
window.__spaBootId = window.__spaBootId || Math.random().toString(36).slice(2);
const bootId = window.__spaBootId;
const navigationEntriesAtBoot = performance.getEntriesByType("navigation").length;
function route(path){ history.pushState({}, "", path); navs++; render(); Challenge.log("route", { path, navs }); }
settings.addEventListener("click", e => { e.preventDefault(); route("/settings"); });
billing.addEventListener("click", e => { e.preventDefault(); route("/settings/billing"); });
window.addEventListener("popstate", () => { popstates++; render(); Challenge.log("popstate", { path: location.pathname, popstates }); });
function render(){
if (location.pathname.endsWith("/settings/billing")) view.innerHTML = `<h2>Billing</h2><button id="save">Save billing settings</button>`;
else if (location.pathname.endsWith("/settings")) view.innerHTML = `<h2>Settings</h2><p>Choose Billing.</p>`;
else view.innerHTML = `<h2>Home</h2>`;
const save = document.getElementById("save");
if (save) save.addEventListener("click", e => {
const bad=[];
if (!location.pathname.endsWith("/settings/billing")) bad.push(`wrong route ${location.pathname}`);
if (navs < 2) bad.push(`only ${navs} SPA route changes`);
if (popstates < 1) bad.push("browser Back/popstate was not observed");
if (window.__spaBootId !== bootId) bad.push("window sentinel changed; reload detected");
if (performance.getEntriesByType("navigation").length !== navigationEntriesAtBoot) bad.push("navigation entry count changed; reload suspected");
if (!e.isTrusted) bad.push("save click isTrusted=false");
if (bad.length) Challenge.fail(...bad); else Challenge.pass("SPA pushState flow completed without reload");
});
}
render();
</script>
</body>
@@ -0,0 +1,14 @@
<!doctype html>
<meta charset="utf-8">
<meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'none'; base-uri 'none'">
<title>39 strict CSP fallback</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: page blocks <code>unsafe-eval</code>. Agent should fall back to screenshot/coordinate observation and click the target without relying on snapshot/evaluate.</p>
<p id="hint">The green target is fixed at known viewport coordinates. Use a screenshot, then click inside it.</p>
<button id="cspTarget" aria-label="strict CSP target">CSP TARGET</button>
</main>
<script src="39-strict-csp-fallback.js"></script>
</body>
@@ -0,0 +1,27 @@
Challenge.init({ id: "strict-csp-fallback", instructions: "use screenshot/coordinates; click CSP TARGET" });
const btn = document.getElementById("cspTarget");
btn.style.cssText = [
"position:fixed",
"left:220px",
"top:220px",
"width:180px",
"height:72px",
"font:700 16px system-ui",
"background:#1f7a1f",
"color:white",
"border:0",
"border-radius:10px",
"box-shadow:0 0 0 4px rgba(31,122,31,.25)"
].join(";");
document.getElementById("cspTarget").addEventListener("click", (e) => {
const r = btn.getBoundingClientRect();
const bad = [];
if (!e.isTrusted) bad.push("click isTrusted=false");
if (e.clientX < r.left || e.clientX > r.right || e.clientY < r.top || e.clientY > r.bottom) {
bad.push(`click coordinates ${e.clientX},${e.clientY} outside target rect ${Math.round(r.left)},${Math.round(r.top)},${Math.round(r.right)},${Math.round(r.bottom)}`);
}
if (bad.length) Challenge.fail(...bad);
else Challenge.pass("strict CSP page completed via trusted viewport click");
});
@@ -0,0 +1,41 @@
<!doctype html>
<meta charset="utf-8">
<title>40 dynamic wait readiness</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: wait until an async control exists and is enabled, then click it. Direct early clicks or fixed sleeps should not be required.</p>
<div id="status" role="status">Loading async action…</div>
<div id="mount" style="margin-top:20px"></div>
</main>
<script>
Challenge.init({ id: "dynamic-wait-readiness", instructions: "wait for async button, then click" });
const start = performance.now();
setTimeout(() => {
const btn = document.createElement("button");
btn.id = "readyAction";
btn.textContent = "Run ready action";
btn.disabled = true;
btn.style.cssText = "padding:14px 22px;font-size:16px;background:#1f7a1f;color:#fff;border:0;border-radius:6px";
document.getElementById("mount").appendChild(btn);
document.getElementById("status").textContent = "Button mounted; enabling soon…";
Challenge.log("mounted", { at: performance.now() - start });
setTimeout(() => {
btn.disabled = false;
btn.dataset.ready = "true";
document.getElementById("status").textContent = "Ready.";
Challenge.log("enabled", { at: performance.now() - start });
}, 450);
btn.addEventListener("click", (e) => {
const elapsed = performance.now() - start;
const bad = [];
if (!e.isTrusted) bad.push("click isTrusted=false");
if (btn.disabled || btn.dataset.ready !== "true") bad.push("clicked before enabled/ready");
if (elapsed < 650) bad.push(`clicked too early at ${elapsed.toFixed(0)}ms`);
if (bad.length) Challenge.fail(...bad);
else Challenge.pass(`waited for async ready state (${elapsed.toFixed(0)}ms)`);
});
}, 350);
</script>
</body>
@@ -0,0 +1,44 @@
<!doctype html>
<meta charset="utf-8">
<title>41 tab lifecycle</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: create a new tab, inspect it, close it, return to parent, then verify lifecycle state.</p>
<p id="msg"></p>
<button id="verify">Verify tab lifecycle</button>
</main>
<script>
Challenge.init({ id: "tab-lifecycle", instructions: "new tab → inspect child → close child → return parent → Verify" });
const qs = new URLSearchParams(location.search);
const key = "pi-chrome-suite:tab-lifecycle:state";
function readState(){ try { return JSON.parse(localStorage.getItem(key) || "{}"); } catch { return {}; } }
function replaceState(next){ localStorage.setItem(key, JSON.stringify({ ...next, ts: Date.now() })); }
function writeState(patch){ localStorage.setItem(key, JSON.stringify({ ...readState(), ...patch, ts: Date.now() })); }
if (qs.get("child") === "1") {
document.title = "[CHILD] 41 tab lifecycle";
document.getElementById("msg").textContent = "Child tab opened. Close this tab, then return to parent.";
writeState({ opened: true, childUrl: location.href, childTitle: document.title });
window.addEventListener("pagehide", () => writeState({ closed: true, closedAt: Date.now() }));
window.addEventListener("beforeunload", () => writeState({ closed: true, closedAt: Date.now() }));
// Do not call Challenge.pass here: parent verification must be the only suite PASS.
Challenge.log("child-loaded", { url: location.href });
} else {
document.getElementById("msg").textContent = "Parent tab. Runner should open this same file with ?child=1 in a new tab, inspect it, close it, then return here.";
replaceState({ parentLoaded: true, closed: false });
document.getElementById("verify").addEventListener("click", (e) => {
const s = readState();
const bad = [];
if (!e.isTrusted) bad.push("verify click isTrusted=false");
if (!s.opened) bad.push("child tab did not record opened=true");
if (!s.childUrl || !s.childUrl.includes("child=1")) bad.push("child URL not recorded");
if (!s.closed) bad.push("child tab close/pagehide was not recorded");
Challenge.log("tab-state", s);
if (bad.length) Challenge.fail(...bad);
else Challenge.pass("new/inspect/close/return tab lifecycle completed");
});
}
</script>
</body>
@@ -0,0 +1,16 @@
<!doctype html>
<meta charset="utf-8">
<meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'none'; base-uri 'none'">
<title>42 strict CSP evaluate/snapshot</title>
<link rel="stylesheet" href="../_style.css">
<script src="../_lib.js"></script>
<body>
<main>
<p>Goal: this page ships a strict CSP (<code>script-src 'self'</code>, no <code>unsafe-eval</code>), which blocks <code>eval</code>/<code>new Function</code>. <code>chrome_evaluate</code> and <code>chrome_snapshot</code> must still work because they run through CDP, which is not subject to page CSP.</p>
<p id="hint">A secret token is exposed only at <code>window.__cspToken</code> — it is never written into the DOM. Use <code>chrome_evaluate</code> to read it, type it into the field (snapshot/uid to find the field), then click Verify.</p>
<label for="tokenInput">Token:</label>
<input id="tokenInput" type="text" autocomplete="off" aria-label="csp token">
<button id="verify" aria-label="verify token">Verify</button>
</main>
<script src="42-strict-csp-evaluate.js"></script>
</body>
@@ -0,0 +1,21 @@
Challenge.init({
id: "strict-csp-evaluate",
instructions: "under strict CSP: read window.__cspToken via chrome_evaluate, type it into the field, click Verify",
});
// Secret available only via JS evaluation. It is intentionally NOT rendered into the DOM and
// is defined non-enumerable, so the only way to obtain it is to evaluate window.__cspToken in
// the page (which proves chrome_evaluate works despite script-src 'self' blocking eval).
const token = "csp-" + Math.random().toString(36).slice(2, 10);
Object.defineProperty(window, "__cspToken", { value: token, enumerable: false, configurable: false, writable: false });
document.getElementById("verify").addEventListener("click", (e) => {
const bad = [];
if (!e.isTrusted) bad.push("verify click isTrusted=false (use trusted/CDP input)");
const val = (document.getElementById("tokenInput").value || "").trim();
if (val !== token) {
bad.push(`token mismatch: got "${val}" expected "${token}" — chrome_evaluate must read window.__cspToken under strict CSP`);
}
if (bad.length) Challenge.fail(...bad);
else Challenge.pass("strict CSP: chrome_evaluate read the hidden token via CDP and trusted input submitted it");
});
@@ -0,0 +1 @@
pi-chrome upload fixture
@@ -0,0 +1,12 @@
window.MiniShopCheats = {
"mini-shop-red-second-cheapest": [
{ tool: "chrome_click", params: { selector: "button[aria-label=\"add Ruby Notebook\"]" } },
{ tool: "chrome_click", params: { selector: "#checkout" } }
],
run: {
"mini-shop-red-second-cheapest": async () => {
document.querySelector('button[aria-label="add Ruby Notebook"]').click();
document.querySelector('#checkout').click();
}
}
};
@@ -0,0 +1,29 @@
window.MiniShopGrader = {
validate(task, state, products, run, seed = 0) {
let bad = [];
const rubric = {};
if (task === 'mini-shop-red-second-cheapest') {
const reds = products.filter(p => p.color === 'red').sort((a,b)=>a.price-b.price);
const target = reds[1];
rubric.addedTarget = state.cart.includes(target.id);
rubric.noWrongItems = state.cart.every(id => id === target.id);
rubric.checkedOut = /^MS-\d{4}$/.test(state.orderId || '');
if (!rubric.addedTarget) bad.push(`cart missing second-cheapest red item ${target.name}`);
const wrong = state.cart.filter(id => id !== target.id);
if (wrong.length) bad.push(`cart contains wrong product(s): ${wrong.join(',')}`);
if (!rubric.checkedOut) bad.push('missing checkout order id');
} else bad.push(`unknown task ${task}`);
const passed = bad.length === 0;
return { reward: passed ? 1 : 0, done: passed, message: passed ? 'all deterministic mini-shop checks passed' : bad.join('; '), info: { task, run, seed, rubric, bad, state } };
},
grade(task, state, products, run, seed = 0) {
const validation = this.validate(task, state, products, run, seed);
const verdict = validation.done ? 'PASS' : 'PENDING';
window.__taskId = task;
window.__taskVerdict = verdict;
window.__taskReason = validation.done ? [validation.message] : validation.info.bad;
window.__taskValidation = validation;
try { localStorage.setItem(`pi-chrome-task:${task}:${run}`, JSON.stringify({ id: task, run, seed, verdict, reason: window.__taskReason, validation, state, ts: Date.now() })); } catch {}
return validation;
}
};
@@ -0,0 +1,55 @@
<!doctype html>
<meta charset="utf-8">
<title>MiniShop hermetic site</title>
<link rel="stylesheet" href="../../../_style.css">
<style>
main{max-width:1100px}.products{display:grid;grid-template-columns:repeat(auto-fit,minmax(210px,1fr));gap:12px}.card{border:1px solid #333;border-radius:10px;background:#202020;padding:12px}.cart{position:sticky;top:8px;background:#111;border:1px solid #555;border-radius:10px;padding:12px;margin-bottom:12px}.red{color:#ff8b8b}.blue{color:#8bbcff}.green{color:#9f9}.muted{color:#aaa;font-size:12px}</style>
<body>
<main>
<h1>MiniShop</h1>
<div class="cart"><b>Cart</b> <span id="cartCount">0</span> items <button id="checkout" data-task-role="checkout">Checkout</button> <span id="order"></span> <span id="reward" class="muted"></span><div id="cartItems"></div></div>
<p id="taskText"></p>
<section class="products" id="productsEl"></section>
</main>
<script src="grader.js?v=2"></script>
<script src="cheats.js?v=2"></script>
<script>
(function(){
const params=new URLSearchParams(location.search);
if(!params.get('run')){params.set('run', crypto.randomUUID ? crypto.randomUUID() : String(Date.now())); history.replaceState(null,'',`?${params}`);}
const task=params.get('task')||'mini-shop-red-second-cheapest'; const run=params.get('run'); const seed=Number(params.get('seed')||0);
const key=`mini-shop:${task}:${run}:${seed}`;
const products=[
{id:'p1',name:'Crimson Mug',color:'red',price:12.00},
{id:'p2',name:'Ruby Notebook',color:'red',price:7.50},
{id:'p3',name:'Scarlet Pen',color:'red',price:3.25},
{id:'p4',name:'Blue Cable',color:'blue',price:5.10},
{id:'p5',name:'Green Mat',color:'green',price:9.00},
{id:'p6',name:'Red Stapler',color:'red',price:18.00},
{id:'p7',name:'Blue Lamp',color:'blue',price:22.00},
{id:'p8',name:'Green Clip',color:'green',price:2.40}
];
let state=JSON.parse(localStorage.getItem(key)||'{"cart":[],"orderId":"","actionCount":0,"startedAt":0}');
if(!state.startedAt) state.startedAt=Date.now();
taskText.textContent='Task: add the second-cheapest red product to cart, checkout, and leave order id visible.';
function save(){localStorage.setItem(key,JSON.stringify(state)); window.__miniShopState=state; const v=MiniShopGrader.grade(task,state,products,run,seed); window.__taskReward=v.reward; window.__taskTerminated=v.done; window.__taskInfo=v.info; reward.textContent=`reward=${v.reward} actions=${state.actionCount||0}`; renderCart();}
function assignBids(){let n=0; document.querySelectorAll('button,a,input,select,textarea').forEach(el=>{const bid='b'+(++n).toString(36); el.setAttribute('bid',bid); el.setAttribute('data-bid',bid); const r=el.getBoundingClientRect(); el.setAttribute('browsergym_visibility_ratio',(r.width&&r.height?1:0).toString());});}
function render(){productsEl.innerHTML=''; products.forEach(p=>{const d=document.createElement('div');d.className='card';d.innerHTML=`<h3 class="${p.color}">${p.name}</h3><p>Color: ${p.color}</p><p>Price: $${p.price.toFixed(2)}</p><button aria-label="add ${p.name}" data-id="${p.id}" data-task-role="add-to-cart">Add to cart</button>`;productsEl.appendChild(d)});document.querySelectorAll('[data-id]').forEach(b=>b.onclick=()=>{state.actionCount=(state.actionCount||0)+1;state.cart.push(b.dataset.id);save()});renderCart();assignBids();}
function renderCart(){
cartCount.textContent=state.cart.length; order.textContent=state.orderId?`Order ${state.orderId}`:'';
cartItems.innerHTML=state.cart.map((id,i)=>{const p=products.find(x=>x.id===id);return `<div>${p?.name||id} <button data-remove-index="${i}" aria-label="remove ${p?.name||id}" data-task-role="remove-from-cart">Remove</button></div>`}).join('');
cartItems.querySelectorAll('[data-remove-index]').forEach(b=>b.onclick=()=>{state.actionCount=(state.actionCount||0)+1;state.cart.splice(Number(b.dataset.removeIndex),1);save();render();});
assignBids();
}
function orderId(){ return 'MS-' + String(1000 + ((seed * 2654435761 + state.cart.join('').length * 97) % 9000)).padStart(4, '0'); }
checkout.onclick=()=>{state.actionCount=(state.actionCount||0)+1;state.orderId=orderId();save();};
function observation(){return {goal:taskText.textContent,url:location.href,title:document.title,reward:window.__taskReward,actionCount:state.actionCount||0,focusedElementBid:document.activeElement?.getAttribute('bid')||null};}
async function runCheat(){const fn=window.MiniShopCheats?.run?.[task]; if(fn) await fn(); else for(const step of (window.MiniShopCheats?.[task]||[])) console.log('cheat step',step); save();}
window.__miniShopState=state; window.__miniShopProducts=products; window.__miniShopTask=task;
window.__task={id:task,run,seed,goal:taskText.textContent,difficulty:'L2',step:()=>{const v=MiniShopGrader.grade(task,state,products,run,seed);return {observation:observation(),reward:v.reward,terminated:v.done,truncated:false,info:v.info}},reset:()=>{localStorage.removeItem(key);state={cart:[],orderId:'',actionCount:0,startedAt:Date.now()};render();save();return {observation:observation(),info:{task,run,seed}}},rubric:()=>MiniShopGrader.validate(task,state,products,run,seed).info.rubric,cheat:runCheat,validate:()=>MiniShopGrader.validate(task,state,products,run,seed),teardown:()=>undefined};
window.__miniShop=window.__task;
window.__miniShop.cheatHelpers={click:(sel)=>document.querySelector(sel).click()};
render(); save();
})();
</script>
</body>
@@ -0,0 +1,9 @@
[
{
"id": "mini-shop-red-second-cheapest",
"difficulty": "L2",
"intent": "Add the second-cheapest red product to cart, checkout, and leave the generated order id visible.",
"gradeSource": "harness",
"grader": "MiniShopGrader.grade(task, state, products, run)"
}
]
+193
View File
@@ -0,0 +1,193 @@
<!doctype html>
<meta charset="utf-8">
<title>pi-chrome browser-control benchmark</title>
<link rel="stylesheet" href="_style.css">
<body>
<main style="max-width:1280px">
<h1>pi-chrome browser-control benchmark</h1>
<p>This benchmark measures how well Chrome-control tools let agents do real browser work: DOM discovery, trusted input, keyboard/focus, scroll, drag/drop, files, frames, clipboard, and observability.</p>
<section class="panel">
<div class="controls">
<button id="refresh">Refresh verdicts</button>
<button id="clear">Clear local verdicts</button>
<button id="copy">Copy JSON report</button>
<label>Filter <input id="filter" placeholder="category, id, expected..." /></label>
<label>Mode
<select id="mode">
<option value="trusted">trusted</option>
<option value="synthetic">synthetic</option>
<option value="manual">manual</option>
</select>
</label>
</div>
<p class="hint">Open a row, drive it with <code>chrome_*</code> tools, then return here. Verdicts are read from <code>localStorage</code>. Expected outcomes come from <code>manifest.json</code>.</p>
<textarea id="copyFallback" class="copy-fallback" readonly aria-label="JSON report fallback"></textarea>
</section>
<div id="summary" class="summary"></div>
<table id="tbl">
<thead>
<tr>
<th>id</th><th>gate</th><th>category</th><th>verdict</th><th>expected</th><th>baseline</th><th>risk</th><th>goal / notes</th><th>open</th>
</tr>
</thead>
<tbody></tbody>
</table>
<h2>Long-horizon hermetic tasks</h2>
<p class="hint">These WebArena/BrowserGym-inspired tasks use deterministic in-page graders and fresh <code>$RUN_ID</code> state. They test multi-step browser work beyond event fidelity.</p>
<table id="taskTbl">
<thead><tr><th>id</th><th>difficulty</th><th>category</th><th>intent</th><th>requires</th><th>open</th></tr></thead>
<tbody></tbody>
</table>
<h2>Agent run loop</h2>
<pre class="code">python3 -m http.server 8765
# For each manifest entry:
# 1. chrome_navigate http://127.0.0.1:8765/&lt;file&gt;
# 2. chrome_snapshot before acting; prefer uid selectors.
# 3. Execute recipe with selected mode, adapting descriptive frame/shadow selectors to tool uids.
# 4. chrome_evaluate JSON.stringify({v:window.__verdict,r:window.__reason,e:window.__events?.slice(-20)})
# 5. Compare verdict to manifest.expected[mode]; CONDITIONAL means inspect prerequisites/notes.</pre>
</main>
<script>
let manifest = [];
let taskManifest = [];
const tbody = document.querySelector("#tbl tbody");
const taskTbody = document.querySelector("#taskTbl tbody");
const modeEl = document.getElementById("mode");
const filterEl = document.getElementById("filter");
function verdictFor(id) {
const raw = localStorage.getItem("pi-chrome-suite:" + id);
if (!raw) return null;
try { return JSON.parse(raw); } catch { return null; }
}
function classFor(v) {
if (!v) return "pending";
return String(v).toLowerCase();
}
function expectedClass(actual, expected) {
if (!actual) return "pending";
if (expected === "CONDITIONAL") return actual === "PASS" ? "ok" : "conditional";
return actual === expected ? "ok" : "mismatch";
}
function report() {
const mode = modeEl.value;
return manifest.map(m => {
const v = verdictFor(m.id);
return {
id: m.id,
category: m.category,
gate: m.gate ?? "core",
file: m.file,
verdict: v?.verdict ?? "PENDING",
reason: v?.reason ?? [],
expected: m.expected?.[mode] ?? "—",
timestamp: v?.ts ?? null,
goal: m.goal,
prerequisites: m.prerequisites ?? [],
notes: m.notes ?? []
};
});
}
function paint() {
const q = filterEl.value.trim().toLowerCase();
const mode = modeEl.value;
tbody.innerHTML = "";
const rows = report();
const counts = rows.reduce((acc, r) => (acc[r.verdict] = (acc[r.verdict] || 0) + 1, acc), {});
const mismatches = rows.filter(r => r.verdict !== "PENDING" && r.expected !== "CONDITIONAL" && r.expected !== "—" && r.verdict !== r.expected).length;
const coreMismatches = rows.filter(r => (r.gate ?? "core") === "core" && r.verdict !== "PENDING" && r.expected !== "CONDITIONAL" && r.expected !== "—" && r.verdict !== r.expected).length;
const gates = rows.reduce((acc, r) => (acc[r.gate ?? "core"] = (acc[r.gate ?? "core"] || 0) + 1, acc), {});
document.getElementById("summary").innerHTML = `
<span class="pill pass">PASS ${counts.PASS || 0}</span>
<span class="pill fail">FAIL ${counts.FAIL || 0}</span>
<span class="pill skip">SKIP ${counts.SKIP || 0}</span>
<span class="pill warn">WARN ${counts.WARN || 0}</span>
<span class="pill pending">PENDING ${counts.PENDING || 0}</span>
<span class="pill ${coreMismatches ? "fail" : "pass"}">core unexpected ${coreMismatches}</span>
<span class="pill ${mismatches ? "fail" : "pass"}">all unexpected ${mismatches}</span>
<span class="pill expected">core ${gates.core || 0}</span>
<span class="pill expected">conditional ${gates.conditional || 0}</span>
<span class="pill expected">quality ${gates.quality || 0}</span>
`;
for (const m of manifest) {
const v = verdictFor(m.id);
const verdict = v?.verdict ?? "PENDING";
const expected = m.expected?.[mode] ?? "—";
const hay = JSON.stringify({ id:m.id, gate:m.gate, category:m.category, verdict, expected, goal:m.goal, notes:m.notes }).toLowerCase();
if (q && !hay.includes(q)) continue;
const tr = document.createElement("tr");
tr.className = expectedClass(verdict, expected);
const reason = (v?.reason || []).join(" / ");
const notes = [...(m.prerequisites || []).map(x => "pre: " + x), ...(m.notes || [])].join("; ");
tr.innerHTML = `
<td><code>${m.id}</code></td>
<td><span class="pill expected">${m.gate || "core"}</span></td>
<td>${m.category || "—"}</td>
<td><span class="pill ${classFor(verdict)}">${verdict}</span><div class="reason">${escapeHtml(reason)}</div></td>
<td><span class="pill expected">${expected}</span></td>
<td><span class="pill ${m.manualBaseline === "verified" ? "pass" : "pending"}">${m.manualBaseline || "unverified"}</span></td>
<td>${m.flakeRisk ? `<span class="risk">${m.flakeRisk}</span>` : ""}</td>
<td><div>${escapeHtml(m.goal || "")}</div><div class="notes">${escapeHtml(notes)}</div><details><summary>recipe</summary><pre>${escapeHtml(JSON.stringify(m.recipe || [], null, 2))}</pre></details></td>
<td><a target="_blank" href="${m.file}">open</a></td>
`;
tbody.appendChild(tr);
}
}
function escapeHtml(s) {
return String(s).replace(/[&<>"']/g, c => ({"&":"&amp;","<":"&lt;",">":"&gt;","\"":"&quot;","'":"&#39;"}[c]));
}
function paintTasks() {
taskTbody.innerHTML = "";
const run = "manual-" + new Date().toISOString().slice(0,10);
for (const t of taskManifest) {
const url = t.startUrl.replaceAll("$RUN_ID", run).replaceAll("$SEED", String(t.seed ?? 0));
const tr = document.createElement("tr");
tr.innerHTML = `
<td><code>${t.taskId || t.id}</code><div class="notes">seed=${t.seed ?? 0} maxSteps=${t.maxSteps ?? "—"}</div></td>
<td><span class="pill expected">${t.difficulty}</span></td>
<td>${t.category}</td>
<td>${escapeHtml(t.goal || t.intent)}</td>
<td><div class="notes">${escapeHtml((t.requires || []).join(", "))}</div><div class="notes">actions: ${escapeHtml((t.actionSubsets || []).join(", "))}</div></td>
<td><a target="_blank" href="${url}">open</a></td>
`;
taskTbody.appendChild(tr);
}
}
async function init() {
manifest = await fetch("manifest.json", { cache: "no-store" }).then(r => r.json());
taskManifest = await fetch("task-manifest.json", { cache: "no-store" }).then(r => r.json()).catch(() => []);
paint();
paintTasks();
}
document.getElementById("refresh").addEventListener("click", paint);
document.getElementById("clear").addEventListener("click", () => {
if (!confirm("Clear pi-chrome-suite verdicts for this origin?")) return;
for (const m of manifest) localStorage.removeItem("pi-chrome-suite:" + m.id);
paint();
});
document.getElementById("copy").addEventListener("click", async () => {
const text = JSON.stringify({ mode: modeEl.value, generatedAt: new Date().toISOString(), rows: report() }, null, 2);
const box = document.getElementById("copyFallback");
box.value = text;
box.style.display = "block";
box.focus();
box.select();
let copied = false;
try { await navigator.clipboard.writeText(text); copied = true; } catch {}
if (!copied) {
try { copied = document.execCommand("copy"); } catch {}
}
box.dataset.copied = copied ? "true" : "false";
console.log(text);
});
modeEl.addEventListener("change", paint);
filterEl.addEventListener("input", paint);
window.addEventListener("storage", paint);
setInterval(paint, 1000);
init();
</script>
</body>
File diff suppressed because it is too large Load Diff
+73
View File
@@ -0,0 +1,73 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "pi-chrome benchmark manifest",
"description": "Manifest for browser-control benchmark pages. Recipes are canonical intent, not guaranteed raw tool JSON: runners may need to adapt descriptive selectors (frame/shadow) and expand path placeholders such as ${REPO_ROOT} before invoking tools.",
"type": "array",
"items": {
"type": "object",
"required": ["id", "file", "category", "goal", "expected", "recipe"],
"additionalProperties": true,
"properties": {
"id": { "type": "string" },
"file": { "type": "string" },
"category": { "$ref": "#/$defs/category" },
"difficulty": { "enum": ["L1", "L2", "L3"] },
"gate": { "description": "Ship-gate bucket. core blocks releases, conditional requires declared capability/environment, quality is adversarial/flaky signal and should not block general ship.", "enum": ["core", "conditional", "quality"] },
"tags": { "type": "array", "items": { "type": "string" } },
"goal": { "type": "string" },
"gradeSource": { "enum": ["page", "harness", "both"], "default": "page" },
"expected": {
"type": "object",
"description": "Expected verdict by execution mode. CONDITIONAL means prerequisites, browser policy, hardware, or missing tool primitives decide outcome.",
"required": ["synthetic", "trusted", "manual"],
"properties": {
"synthetic": { "$ref": "#/$defs/outcome" },
"trusted": { "$ref": "#/$defs/outcome" },
"manual": { "$ref": "#/$defs/outcome" }
}
},
"manualBaseline": { "description": "Whether human baseline has been explicitly verified in this repo/environment.", "enum": ["unverified", "verified"] },
"manualBaselineVerifiedAt": { "type": "string", "format": "date-time" },
"recipe": {
"type": "array",
"description": "Ordered tool intent. Params may include descriptive selectors (e.g. frame/shadow notation) or path placeholders for runners to adapt.",
"items": {
"type": "object",
"required": ["tool", "params"],
"properties": {
"tool": { "type": "string" },
"params": { "type": "object", "description": "Tool params. For upload recipes, paths are strings after runner expansion; ${REPO_ROOT} means repository root." }
}
}
},
"requires": {
"type": "object",
"additionalProperties": true,
"properties": {
"cdp": { "type": "boolean" },
"clipboard": { "type": "boolean" },
"touch": { "type": "boolean" },
"secureOrigin": { "type": "boolean" },
"fileSystem": { "type": "boolean" },
"dialogs": { "type": "boolean" },
"downloads": { "type": "boolean" },
"popups": { "type": "boolean" }
}
},
"prerequisites": { "type": "array", "items": { "type": "string" } },
"notes": { "type": "array", "items": { "type": "string" } },
"thresholds": { "type": "object", "additionalProperties": true },
"timeoutMs": { "type": "number" },
"verdictDelayMs": { "type": "number" },
"flakeRisk": { "enum": ["low", "medium", "high"] }
}
},
"$defs": {
"outcome": { "enum": ["PASS", "FAIL", "SKIP", "WARN", "CONDITIONAL"] },
"category": {
"enum": [
"activation-gates", "agent-safety", "aria-snapshot", "canvas-svg", "clipboard", "contextmenu", "csp", "dialogs", "dom-complexity", "downloads", "drag-drop", "editing", "files", "fingerprint", "focus-keyboard", "forms", "frames", "frameworks", "hover", "keyboard", "lazy-loading", "native-controls", "observability", "pointer-humanization", "popups", "scroll", "scroll-visibility", "spa-routing", "task-completion", "timing", "touch", "trusted-input"
]
}
}
}
@@ -0,0 +1,70 @@
# BrowserGym compatibility contract
Pi-chrome benchmark tasks mirror BrowserGym without importing Gymnasium.
## Task fields
Each `task-manifest.json` entry maps to `AbstractBrowserTask` concepts:
- `taskId` / `id` — stable namespaced id
- `seed` — deterministic state seed
- `viewport`, `slowMoMs`, `timeoutMs`, `locale`, `timezoneId`
- `goal` and `goalObject` — text and OpenAI-style message objects
- `setupUrl` — page URL that performs setup from `task`, `run`, `seed`
- `validateScript` — JS validate hook returning reward contract
- `cheatScript` — gold recipe used to sanity-check grader
- `maxSteps`, `humanBaseline`, `difficulty`, `tags`
- `actionSubsets`, `allowedActions` — BrowserGym HighLevelActionSet metadata
## Reset / step return
Runner should expose:
```text
reset() -> (obs, info)
step(action) -> (obs, reward, terminated, truncated, info)
```
`terminated` means task done. `truncated` means max step or timeout.
## Validate return
In-page validators return:
```js
{ reward, done, message, info }
```
`reward` is incremental. Current hermetic tasks use binary terminal reward. `info.rubric` carries per-check details.
## Observation schema target
Runner observations should include BrowserGym-like keys:
- `chat_messages`
- `goal`, `goal_object`
- `open_pages_urls`, `open_pages_titles`, `active_page_index`
- `url`, `screenshot`, `dom_object`, `axtree_object`
- `extra_element_properties`
- `focused_element_bid`
- `last_action`, `last_action_error`
- `elapsed_time`
## BID plan
BrowserGym uses `bid` attributes as stable element handles. Pi-chrome currently returns snapshot `uid`. Compatibility path:
1. During snapshot, assign every interactive element a deterministic `bid`.
2. Return both `uid` and `bid`.
3. Add `visibility`, `bbox`, `clickable`, `set_of_marks` metadata.
4. Maintain uid↔bid mapping so agents can use either addressing mode.
Target attributes:
- `bid`
- `browsergym_visibility_ratio`
- `browsergym_set_of_marks`
## Action subsets
See `../browsergym-action-space.json` for BrowserGym-compatible subsets and function signatures.
@@ -0,0 +1,79 @@
# Historical bypass notes
This file predates the current `trusted:true` / `chrome.debugger` path and is
kept as design history. For current benchmark expectations, recipes, and
capability notes, use `../manifest.json`.
Older context: `pi-chrome` originally ran mostly through a content-script bridge.
That limited the solution space — most "real-event" tricks needed the
`chrome.debugger` API or `chrome.input.synthesizeMouseEvent`.
## 0102 isTrusted click / keyboard
**Hard.** `Event.isTrusted` is true only for events the browser dispatched
itself. Two viable paths:
- **`chrome.debugger` + `Input.dispatchMouseEvent` / `Input.dispatchKeyEvent`.**
Adds `"debugger"` permission. Chrome shows a "X is debugging this browser"
banner, but events arrive as `isTrusted=true`. Best fix for the highest-value
cases; gate it behind a toggle the user opts into.
- **`chrome.input.ime.*`** — not applicable for general typing.
Today's `dispatchEvent` path will always fail any `isTrusted` check.
## 03 / 12 webdriver + fingerprint
Cheap wins:
- Stub `navigator.webdriver` via `chrome.scripting.executeScript({ world: "MAIN", injectImmediately: true, … })` that defines `navigator.webdriver` getter to `undefined`. Already half-supported through `initScript` on `chrome_navigate` — extend to a per-tab content script registered with `chrome.scripting.registerContentScripts({ runAt: "document_start", world: "MAIN" })`.
- Spoof languages/plugins/permissions only if the user explicitly enables it — by default the real Chrome profile already looks legitimate. The bigger risk is `chrome_evaluate` *itself* leaving traces (Function-constructor stack frames in errors).
## 04 mouse entropy
Generate a humanised pointer path before any click:
- Bezier interpolation from current cursor pos (or a random off-target start) to the target point.
- 2060 steps, easing curve, ±jitter on each step, varying `movementX/Y`.
- Implement in `clickPage` before the existing pointerdown sequence; reuse `pointerEventSequence` for each interpolated point.
## 05 event timing
- Insert `await sleep(rand(40,140))` between `pointerdown` and `pointerup`.
- Insert `await sleep(rand(80,220))` between successive synthesised clicks when
the caller issues them in a loop.
- Vary by ±20%.
## 06 click coordinates
- After resolving target rect in `clickPage`, pick `(cx ± rand(-rw*0.3,rw*0.3), cy ± rand(-rh*0.3,rh*0.3))` instead of dead center. Already returns the chosen point — keep that.
## 07 pointer properties
- Set `pressure: 0.5` for `pointerdown` (currently inits don't set pressure; browser default for mouse pointerdown is 0.5).
- Set `pointerId: 1` for mouse (already done) and 2+ for touch.
- Carry `movementX/Y` from the previous interpolated step (needs path state).
## 08 / 09 keyboard cadence + framework invariants
- Per character: dispatch `keydown` (cancelable, fills `key` + `code` + `keyCode`), then `keypress` for printables, then `beforeinput` `{inputType:"insertText", data:ch}`, then mutate the input's value to `cur+ch` via the native setter, then `input` `{inputType:"insertText", data:ch}`, then `keyup`. Sleep 40120 ms between chars.
- Today the bridge fills the entire value in one go and emits one `beforeinput`/`input`. Replace with the per-char loop above.
- Make sure not to dispatch `compositionstart`/`compositionend` for plain ASCII.
## 10 user activation
**Impossible with synthetic events.** `navigator.userActivation.isActive` only
flips on browser-trusted events. Same fix as 01/02 — `chrome.debugger` path.
Document the limitation; offer a `useDebuggerForActivationGates` opt-in.
## 11 honeypot
Pure agent-side concern. Before filling/clicking by selector, the bridge could:
- Reject targets with `display:none`, `visibility:hidden`, `aria-hidden=true`,
or off-screen position (`getBoundingClientRect()` outside viewport bounds and
not scrolled-into-view), unless the caller passes `force: true`.
- That logic largely exists (`elementVisible` / `occludedBy` are returned) —
promote it from informational to a guard.
## 13 focus order
- When clicking a focusable element, dispatch `pointerdown` first; only after
`mousedown` defaults run let focus settle naturally (browser will move focus
on the synthetic `mousedown` only if `isTrusted` is true — alas, no). Manual
workaround: dispatch the pointer/mouse sequence, then explicitly call
`target.focus({ preventScroll: true })`, then dispatch `focus` with
`{ relatedTarget: previouslyFocused }`. Set `:focus-visible` heuristic via
`target.blur(); target.focus({ focusVisible: false })` (Chrome 124+).
## 14 wheel scroll
- For scroll operations, dispatch sequential `wheel` events with `deltaY`
chunks (e.g., 60120 per tick), and let the browser scroll, instead of
setting `scrollTop` directly. Provide `chrome_scroll({ uid, dy, dx })` as a
first-class tool.
+22
View File
@@ -0,0 +1,22 @@
# Calibration profiles
Benchmark results depend on Chrome profile, OS, input device, and extension mode.
Record these with every run:
- OS and version
- Chrome version
- pi-chrome package version
- companion extension version
- trusted mode: off / auto / on
- input hardware: trackpad vs detent mouse, touch support
- permissions: clipboard, fullscreen, downloads
- viewport size and deviceScaleFactor
- loaded privacy/security extensions
Known environment-sensitive areas:
- WebGL fingerprint can warn/fail in VMs, remote desktops, GPU-disabled Chrome.
- Scroll momentum expects trackpad-like decay; detent mouse wheels may fail manually.
- Touch tests require touch events enabled/supported.
- Dialog/download/file-picker tests require native browser UI handling or manual intervention.
- Stack traces vary across Chrome versions; tests should only fail concrete automation URLs/Runtime frames.
+29
View File
@@ -0,0 +1,29 @@
# Recipe runner spec
Future Node/Pi runner should:
1. Read `manifest.json`.
2. Probe environment capabilities (`trusted`, `clipboard`, `touch`, `dialogs`, `downloads`, `fileSystem`, `strictCspFallback`).
3. For each entry:
- read `gate` (`core`, `conditional`, `quality`) and score in that bucket
- skip if `requires` is unsatisfied and expected is `CONDITIONAL`
- navigate to challenge URL
- execute `recipe` in order
- wait `verdictDelayMs || 500`
- evaluate `JSON.stringify({v:window.__verdict,r:window.__reason,d:window.Challenge?.state?.details,e:window.__events?.slice(-20)})`, except strict-CSP pages where eval is expected to fail; read persisted verdict from dashboard/localStorage after leaving the CSP page
- compare to `expected[mode]`
4. Output:
- Markdown summary
- JSON details
- JUnit XML for CI
Runner must adapt recipe intent:
- expand `${REPO_ROOT}` path placeholders
- adapt unsupported shadow/iframe selector notation to snapshot uids or evaluate fallback
- preserve hook install ordering for console/network capture tests
- substitute dynamic tab ids from `chrome_tab list` recipes
- support screenshot/coordinate fallback for strict CSP pages where snapshot/evaluate are blocked
- record whether trusted/CDP path was used
Long-horizon task runner should read `task-manifest.json`, replace `$RUN_ID`, solve task, then evaluate task grader expression.
+44
View File
@@ -0,0 +1,44 @@
# Benchmark scoring
## Unit challenges (`manifest.json`)
Score actual verdict against `expected[mode]`, not raw PASS count.
Gate buckets:
- `core`: headline release gate. Exact expected verdict required.
- `conditional`: gate only when declared capability/prerequisite is present; otherwise report skipped/conditional.
- `quality`: adversarial humanization/fingerprint signal. Report trend and regressions, but do not block general release unless explicitly promoted.
Expected values:
- `PASS` / `FAIL` expected: exact match required inside active gate bucket.
- `CONDITIONAL`: exclude from core headline score unless environment prerequisite is declared present.
- `SKIP` / `WARN`: report separately.
Recommended flake policy:
- Run each non-destructive challenge up to 2 retries.
- Take best verdict only for known-flaky timing/scroll tests.
- Always keep all reasons/events in detailed report.
Difficulty weighting:
- L1 = 1
- L2 = 2
- L3 = 3
Report both unweighted and weighted score. Also report per-category score.
## Long-horizon tasks (`task-manifest.json`)
Task score is deterministic grader PASS / not PASS. Record:
- action count
- wall time
- tools used
- observation mode (`snapshot`, screenshot, accessibility, evaluate)
- whether direct state mutation/evaluate was allowed
- final grader reason
Do not use LLM judge for hermetic tasks unless grader cannot express the target.
@@ -0,0 +1,49 @@
window.ChoreDeskCheats = {
"choredesk-l1-ticket-priority": [
{ tool: "chrome_fill", params: { selector: "select[aria-label=\"T-104 priority\"]", text: "High" } },
{ tool: "chrome_fill", params: { selector: "select[aria-label=\"T-104 status\"]", text: "In Progress" } },
{ tool: "chrome_click", params: { selector: "button[aria-label=\"save tickets T-104\"]" } }
],
"choredesk-l2-refund-message": [
{ tool: "chrome_click", params: { selector: "a[data-view=\"customers\"]" } },
{ tool: "chrome_click", params: { selector: "a[data-view=\"orders\"]" } },
{ tool: "chrome_click", params: { selector: "a[data-view=\"messages\"]" } },
{ tool: "chrome_fill", params: { selector: "#msgSubject", text: "Refund review ORD-778" } },
{ tool: "chrome_fill", params: { selector: "#msgBody", text: "Noah Kim noah.kim@example.com order ORD-778 total $84.20" } },
{ tool: "chrome_click", params: { selector: "#sendMsg" } }
],
"choredesk-l3-restock-ticket": [
{ tool: "chrome_click", params: { selector: "a[data-view=\"catalog\"]" } },
{ tool: "chrome_click", params: { selector: "a[data-view=\"inventory\"]" } },
{ tool: "chrome_fill", params: { selector: "input[aria-label=\"AST-LAMP reorder quantity\"]", text: "12" } },
{ tool: "chrome_click", params: { selector: "button[aria-label=\"save inventory AST-LAMP\"]" } },
{ tool: "chrome_click", params: { selector: "a[data-view=\"tickets\"]" } },
{ tool: "chrome_fill", params: { selector: "input[aria-label=\"T-205 internal note\"]", text: "reordered 12 AST-LAMP" } },
{ tool: "chrome_click", params: { selector: "button[aria-label=\"save tickets T-205\"]" } }
],
run: {
"choredesk-l1-ticket-priority": async () => {
await window.__choredesk.cheatHelpers.route("tickets");
window.__choredesk.cheatHelpers.set("select[aria-label=\"T-104 priority\"]", "High");
window.__choredesk.cheatHelpers.set("select[aria-label=\"T-104 status\"]", "In Progress");
document.querySelector("button[aria-label=\"save tickets T-104\"]").click();
},
"choredesk-l2-refund-message": async () => {
await window.__choredesk.cheatHelpers.route("customers");
await window.__choredesk.cheatHelpers.route("orders");
await window.__choredesk.cheatHelpers.route("messages");
window.__choredesk.cheatHelpers.set("#msgSubject", "Refund review ORD-778");
window.__choredesk.cheatHelpers.set("#msgBody", "Noah Kim noah.kim@example.com order ORD-778 total $84.20");
document.querySelector("#sendMsg").click();
},
"choredesk-l3-restock-ticket": async () => {
await window.__choredesk.cheatHelpers.route("catalog");
await window.__choredesk.cheatHelpers.route("inventory");
window.__choredesk.cheatHelpers.set("input[aria-label=\"AST-LAMP reorder quantity\"]", "12");
document.querySelector("button[aria-label=\"save inventory AST-LAMP\"]").click();
await window.__choredesk.cheatHelpers.route("tickets");
window.__choredesk.cheatHelpers.set("input[aria-label=\"T-205 internal note\"]", "reordered 12 AST-LAMP");
document.querySelector("button[aria-label=\"save tickets T-205\"]").click();
}
}
};
@@ -0,0 +1,239 @@
<!doctype html>
<meta charset="utf-8">
<title>ChoreDesk hermetic task site</title>
<link rel="stylesheet" href="../../_style.css">
<style>
main { max-width: 1180px; }
.top { display:flex; gap:12px; align-items:center; flex-wrap:wrap; }
.tabs { display:flex; gap:8px; margin:16px 0; flex-wrap:wrap; }
.tab { border:1px solid #444; border-radius:999px; padding:6px 10px; color:#eee; text-decoration:none; background:#222; }
.tab[aria-current="page"] { background:#2b4b6b; }
.card { border:1px solid #333; border-radius:10px; background:#202020; padding:14px; }
table { font-size:13px; }
td input, td select { width: 100%; box-sizing:border-box; }
textarea { width:100%; min-height:110px; background:#111; color:#eee; border:1px solid #555; border-radius:6px; padding:8px; }
.muted { color:#aaa; font-size:12px; }
.taskbar { border:1px solid #4b3b16; background:#221b0d; border-radius:10px; padding:12px; margin:12px 0; }
.verdict { display:inline-block; border-radius:999px; padding:2px 8px; font-weight:700; background:#444; }
.verdict.PASS { background:#1f7a1f; }
.verdict.FAIL { background:#a11; }
.verdict.PENDING { background:#444; }
</style>
<body>
<main>
<div class="top">
<h1 style="margin-right:auto">ChoreDesk</h1>
<button id="reset" data-task-role="reset-button">Reset run</button>
<button id="grade" data-task-role="grade-button">Grade now</button>
<button id="cheat" data-task-role="cheat-button">Run cheat</button>
</div>
<div class="taskbar">
<div><b id="taskId"></b> <span id="difficulty"></span> <span id="verdict" class="verdict PENDING">PENDING</span> <span id="reward" class="muted"></span></div>
<p id="instruction"></p>
<pre id="reason" class="muted"></pre>
</div>
<nav class="tabs" aria-label="ChoreDesk sections"></nav>
<section id="view"></section>
</main>
<script src="cheats.js"></script>
<script>
(function () {
const params = new URLSearchParams(location.search);
if (!params.get("run")) {
params.set("run", crypto.randomUUID ? crypto.randomUUID() : String(Date.now()));
history.replaceState(null, "", `?${params}`);
}
const task = params.get("task") || "choredesk-l1-ticket-priority";
const run = params.get("run");
const seed = Number(params.get("seed") || 0);
const nonce = params.get("nonce") || (crypto.randomUUID ? crypto.randomUUID() : Math.random().toString(36).slice(2));
if (!params.get("nonce")) { params.set("nonce", nonce); history.replaceState(null, "", `?${params}`); }
const storeKey = `pi-chrome-task-state:${task}:${run}:${seed}:${nonce}`;
const verdictKey = `pi-chrome-task:${task}:${run}`;
const views = ["tickets", "customers", "orders", "catalog", "inventory", "messages", "audit"];
let view = params.get("view") || "tickets";
let bidCounter = 0;
const TASKS = {
"choredesk-l1-ticket-priority": {
difficulty: "L1 atomic",
instruction: "Set ticket T-104 priority to High and status to In Progress. Save the ticket.",
rubric(s) {
const t = s.tickets.find(x => x.id === "T-104");
return [
{ id:"priorityHigh", weight:.33, pass:t?.priority === "High", description:"T-104 priority is High", reason:`T-104 priority=${t?.priority}` },
{ id:"statusInProgress", weight:.33, pass:t?.status === "In Progress", description:"T-104 status is In Progress", reason:`T-104 status=${t?.status}` },
{ id:"saved", weight:.34, pass:s.audit.some(a => a.kind === "ticket-save" && a.id === "T-104" && a.via === "ui-event"), description:"T-104 saved through UI", reason:"no UI save audit for T-104" }
];
}
},
"choredesk-l2-refund-message": {
difficulty: "L2 compositional",
instruction: "A customer named Noah Kim reports a duplicate charge. Find Noah's email and order ORD-778 total, then send Billing a message with subject 'Refund review ORD-778'. Body must include Noah's email and the exact order total.",
rubric(s) {
const msg = s.messages.find(m => m.to === "Billing" && m.subject.trim() === "Refund review ORD-778");
return [
{ id:"visitedCustomers", weight:.15, pass:!!s.viewsVisited.customers, description:"Opened Customers view", reason:"never opened Customers view" },
{ id:"visitedOrders", weight:.15, pass:!!s.viewsVisited.orders, description:"Opened Orders view", reason:"never opened Orders view" },
{ id:"subject", weight:.20, pass:!!msg, description:"Billing message with exact subject exists", reason:"missing Billing message with exact subject" },
{ id:"email", weight:.25, pass:!!msg && /\bnoah\.kim@example\.com\b/i.test(msg.body), description:"Body includes Noah email", reason:"message body missing Noah email" },
{ id:"total", weight:.25, pass:!!msg && /(?<![\d.])\$?84\.20\b/.test(msg.body), description:"Body includes exact ORD-778 total", reason:"message body missing exact ORD-778 total 84.20" }
];
}
},
"choredesk-l3-restock-ticket": {
difficulty: "L3 cross-page tedious",
instruction: "Ticket T-205 says Aster Lamp is delayed. Find the Aster Lamp SKU in Catalog, set its Inventory reorder quantity to 12, then add an internal note to T-205 saying 'reordered 12 AST-LAMP'.",
rubric(s) {
const inv = s.inventory.find(x => x.sku === "AST-LAMP");
const t = s.tickets.find(x => x.id === "T-205");
return [
{ id:"visitedCatalog", weight:.15, pass:!!s.viewsVisited.catalog, description:"Opened Catalog view", reason:"never opened Catalog view" },
{ id:"visitedInventory", weight:.15, pass:!!s.viewsVisited.inventory, description:"Opened Inventory view", reason:"never opened Inventory view" },
{ id:"inventory", weight:.25, pass:Number(inv?.reorderQty) === 12 && s.audit.some(a => a.kind === "inventory-save" && a.id === "AST-LAMP" && a.via === "ui-event"), description:"AST-LAMP reorderQty=12 saved through UI", reason:`AST-LAMP reorderQty=${inv?.reorderQty}; missing UI save audit` },
{ id:"ticketNote", weight:.25, pass:!!t && t.note.includes("reordered 12 AST-LAMP"), description:"T-205 note contains exact phrase", reason:"T-205 internal note missing exact phrase" },
{ id:"ticketSaved", weight:.20, pass:s.audit.some(a => a.kind === "ticket-save" && a.id === "T-205" && a.via === "ui-event"), description:"T-205 saved through UI", reason:"no UI save audit for T-205" }
];
}
}
};
const SEED = {
tickets: [
{ id:"T-101", customer:"Rosa Patel", subject:"Where is my desk mat?", priority:"Normal", status:"Open", note:"" },
{ id:"T-104", customer:"Iris Zhao", subject:"Invoice address typo", priority:"Low", status:"Open", note:"" },
{ id:"T-205", customer:"Mateo Silva", subject:"Aster Lamp delayed", priority:"Normal", status:"Open", note:"" },
{ id:"T-309", customer:"Noah Kim", subject:"Duplicate charge", priority:"Normal", status:"Open", note:"" }
],
customers: [
{ name:"Noah Kim", email:"noah.kim@example.com", tier:"Gold" },
{ name:"Rosa Patel", email:"rosa.patel@example.com", tier:"Silver" },
{ name:"Iris Zhao", email:"iris.zhao@example.com", tier:"Bronze" },
{ name:"Mateo Silva", email:"mateo.silva@example.com", tier:"Gold" }
],
orders: [
{ id:"ORD-778", customer:"Noah Kim", item:"Cable Kit", total:"$84.20", status:"Paid" },
{ id:"ORD-812", customer:"Rosa Patel", item:"Desk Mat", total:"$28.00", status:"Shipped" },
{ id:"ORD-919", customer:"Mateo Silva", item:"Aster Lamp", total:"$134.50", status:"Delayed" }
],
catalog: [
{ item:"Aster Lamp", sku:"AST-LAMP", vendor:"Northwind Lighting" },
{ item:"Desk Mat", sku:"DSK-MAT", vendor:"Woven Works" },
{ item:"Cable Kit", sku:"CBL-KIT", vendor:"Copper Lane" }
],
inventory: [
{ sku:"AST-LAMP", onHand:0, reorderQty:0 },
{ sku:"DSK-MAT", onHand:42, reorderQty:0 },
{ sku:"CBL-KIT", onHand:6, reorderQty:0 }
],
messages: [],
audit: [],
viewsVisited: {},
actionCount: 0,
startedAt: Date.now()
};
let state = loadState();
function clone(x){ return JSON.parse(JSON.stringify(x)); }
function loadState(){ const raw = localStorage.getItem(storeKey); return raw ? JSON.parse(raw) : clone(SEED); }
function saveState(){ localStorage.setItem(storeKey, JSON.stringify(state)); }
function countAction(kind){ state.actionCount = (state.actionCount || 0) + 1; state.lastAction = kind; }
function audit(entry){ state.audit.push({ ts:new Date().toISOString(), via:"ui-event", trusted:!!entry.trusted, ...entry }); saveState(); grade(false); }
function rubric(){ return (TASKS[task] || TASKS["choredesk-l1-ticket-priority"]).rubric(state); }
function validate(){
const items = rubric();
const total = items.reduce((a,x)=>a+x.weight,0) || 1;
const got = items.filter(x=>x.pass).reduce((a,x)=>a+x.weight,0);
const reward = Math.round((got / total) * 1000) / 1000;
const done = reward === 1;
const bad = items.filter(x=>!x.pass).map(x=>x.reason || `${x.id} failed`);
return { reward, done, message: done ? "all deterministic checks passed" : bad.join("; "), info: { task, seed, run, actionCount:state.actionCount||0, elapsedMs:Date.now()-(state.startedAt||Date.now()), viewsVisited:state.viewsVisited, rubric:items, state } };
}
function observation(){
return { goal:TASKS[task]?.instruction, url:location.href, activeView:view, title:document.title, reward:window.__taskReward, actionCount:state.actionCount||0, viewsVisited:state.viewsVisited, focusedElementBid:document.activeElement?.getAttribute("bid") || null };
}
function setVerdict(verdict, reason, validation = validate()){
window.__taskVerdict = verdict;
window.__taskReason = reason;
window.__taskValidation = validation;
window.__taskReward = validation.reward;
window.__taskTerminated = validation.done;
window.__taskTruncated = (state.actionCount || 0) >= Number(params.get("maxSteps") || 9999);
window.__taskInfo = validation.info;
localStorage.setItem(verdictKey, JSON.stringify({ id: task, run, seed, verdict, reason, validation, state, ts: Date.now() }));
document.getElementById("verdict").textContent = verdict;
document.getElementById("verdict").className = "verdict " + verdict;
document.getElementById("reward").textContent = `reward=${validation.reward} actions=${state.actionCount || 0}`;
document.getElementById("reason").textContent = reason.join("\n");
}
function grade(showPass=true){ const v = validate(); if (!v.done) setVerdict(showPass ? "FAIL" : "PENDING", v.info.rubric.filter(x=>!x.pass).map(x=>`${x.id}: ${x.reason}`), v); else setVerdict("PASS", [v.message], v); return v; }
function route(v){ view = v; state.viewsVisited[v] = (state.viewsVisited[v] || 0) + 1; countAction("route:" + v); saveState(); history.replaceState(null, "", `?task=${encodeURIComponent(task)}&run=${encodeURIComponent(run)}&seed=${seed}&nonce=${nonce}&view=${encodeURIComponent(view)}`); render(); }
function renderShell(){
const spec = TASKS[task] || TASKS["choredesk-l1-ticket-priority"];
document.getElementById("taskId").textContent = task;
document.getElementById("difficulty").textContent = spec.difficulty;
document.getElementById("instruction").textContent = spec.instruction;
const nav = document.querySelector(".tabs"); nav.innerHTML = "";
for (const v of views) { const a = document.createElement("a"); a.href="#"; a.className="tab"; a.textContent=v[0].toUpperCase()+v.slice(1); a.dataset.view=v; a.setAttribute("data-task-role", "nav-tab"); if(v===view)a.setAttribute("aria-current","page"); a.addEventListener("click", e=>{e.preventDefault(); route(v);}); nav.appendChild(a); }
}
function render(){
renderShell(); bidCounter = 0;
const root = document.getElementById("view"); root.innerHTML = "";
if (view === "tickets") root.appendChild(tableEditor("tickets", state.tickets, ["id","customer","subject","priority","status","note"], saveTicket));
if (view === "customers") root.appendChild(readTable(state.customers, ["name","email","tier"]));
if (view === "orders") root.appendChild(readTable(state.orders, ["id","customer","item","total","status"]));
if (view === "catalog") root.appendChild(readTable(state.catalog, ["item","sku","vendor"]));
if (view === "inventory") root.appendChild(tableEditor("inventory", state.inventory, ["sku","onHand","reorderQty"], saveInventory));
if (view === "messages") root.appendChild(messageComposer());
if (view === "audit") root.appendChild(readTable(state.audit.map((entry,i)=>({ n:i+1, entry: JSON.stringify(entry) })), ["n","entry"]));
assignBids(); grade(false);
}
function readTable(rows, cols){
const wrap = document.createElement("div"); wrap.className="card";
wrap.innerHTML = `<table><thead><tr>${cols.map(c=>`<th>${c}</th>`).join("")}</tr></thead><tbody></tbody></table>`;
const tb = wrap.querySelector("tbody");
for (const r of rows) { const tr = document.createElement("tr"); tr.innerHTML = cols.map(c => `<td>${escapeHtml(r[c] ?? "")}</td>`).join(""); tb.appendChild(tr); }
return wrap;
}
function tableEditor(kind, rows, cols, onSave){
const wrap = document.createElement("div"); wrap.className="card";
wrap.innerHTML = `<table><thead><tr>${cols.map(c=>`<th>${c}</th>`).join("")}<th>save</th></tr></thead><tbody></tbody></table>`;
const tb = wrap.querySelector("tbody");
rows.forEach((r) => {
const key = r.id || r.sku; const tr = document.createElement("tr"); tr.dataset.itemId = key; if (r.id) tr.dataset.ticketId = r.id; if (r.sku) tr.dataset.sku = r.sku;
for (const c of cols) {
const td = document.createElement("td");
const stableAttrs = `${r.id ? `data-ticket-id="${r.id}"` : ""} ${r.sku ? `data-sku="${r.sku}"` : ""}`;
if (c === "priority") td.innerHTML = `<select data-item-id="${key}" ${stableAttrs} data-col="${c}" aria-label="${key} priority"><option>Low</option><option>Normal</option><option>High</option></select>`;
else if (c === "status") td.innerHTML = `<select data-item-id="${key}" ${stableAttrs} data-col="${c}" aria-label="${key} status"><option>Open</option><option>In Progress</option><option>Waiting</option><option>Closed</option></select>`;
else if (c === "note") td.innerHTML = `<input data-item-id="${key}" ${stableAttrs} data-col="${c}" aria-label="${key} internal note">`;
else if (c === "reorderQty") td.innerHTML = `<input type="number" data-item-id="${key}" ${stableAttrs} data-col="${c}" aria-label="${key} reorder quantity">`;
else td.textContent = r[c];
tr.appendChild(td);
}
const save = document.createElement("td"); save.innerHTML = `<button data-save-id="${key}" data-task-role="save-button" aria-label="save ${kind} ${key}">Save</button>`; tr.appendChild(save); tb.appendChild(tr);
});
wrap.querySelectorAll("[data-item-id]").forEach(el => { const row = rows.find(r => (r.id || r.sku) === el.dataset.itemId); el.value = row[el.dataset.col]; el.addEventListener("input", () => { row[el.dataset.col] = el.value; }); });
wrap.querySelectorAll("[data-save-id]").forEach(btn => btn.addEventListener("click", e => onSave(rows.find(r => (r.id || r.sku) === btn.dataset.saveId), e)));
return wrap;
}
function saveTicket(t, e){ countAction("save-ticket"); saveState(); audit({ kind:"ticket-save", id:t.id, trusted:e?.isTrusted }); render(); }
function saveInventory(i, e){ countAction("save-inventory"); i.reorderQty = Number(i.reorderQty); saveState(); audit({ kind:"inventory-save", id:i.sku, trusted:e?.isTrusted }); render(); }
function messageComposer(){
const wrap = document.createElement("div"); wrap.className="card";
wrap.innerHTML = `<label>From <input id="msgFrom" aria-label="message from" value="Agent"></label><br><br><label>To <select id="msgTo"><option>Billing</option><option>Support</option><option>Warehouse</option></select></label><br><br><label>Subject <input id="msgSubject" aria-label="message subject"></label><br><br><label>Body <textarea id="msgBody" aria-label="message body"></textarea></label><br><button id="sendMsg" data-task-role="send-message">Send message</button><h3>Sent messages</h3><div id="sent"></div>`;
wrap.querySelector("#sendMsg").addEventListener("click", e => { countAction("send-message"); state.messages.push({ from:msgFrom.value, to:msgTo.value, subject:msgSubject.value, body:msgBody.value, trusted:e.isTrusted }); audit({ kind:"message-send", id:msgSubject.value, trusted:e.isTrusted }); render(); });
wrap.querySelector("#sent").appendChild(readTable(state.messages, ["from","to","subject","body"]));
return wrap;
}
function assignBids(){ document.querySelectorAll("button,input,select,textarea,a[href]").forEach(el => { const id = `b${(++bidCounter).toString(36)}`; el.setAttribute("bid", id); el.setAttribute("data-bid", id); const r = el.getBoundingClientRect(); el.setAttribute("browsergym_visibility_ratio", (r.width > 0 && r.height > 0 ? 1 : 0).toString()); }); }
function escapeHtml(s){ return String(s).replace(/[&<>"']/g, c => ({"&":"&amp;","<":"&lt;",">":"&gt;","\"":"&quot;","'":"&#39;"}[c])); }
async function runCheat(){ const fn = window.ChoreDeskCheats?.run?.[task]; if (fn) await fn(); else for (const step of (window.ChoreDeskCheats?.[task] || [])) console.log("cheat step", step); grade(true); }
document.getElementById("reset").addEventListener("click", () => { localStorage.removeItem(storeKey); localStorage.removeItem(verdictKey); state = loadState(); render(); });
document.getElementById("grade").addEventListener("click", () => grade(true));
document.getElementById("cheat").addEventListener("click", () => runCheat());
window.__task = { id:task, run, seed, goal:TASKS[task]?.instruction, difficulty:TASKS[task]?.difficulty, step:() => { const v=grade(false); return { observation:observation(), reward:v.reward, terminated:v.done, truncated:window.__taskTruncated, info:v.info }; }, reset:() => { localStorage.removeItem(storeKey); state=loadState(); render(); return { observation:observation(), info:{task,run,seed} }; }, rubric:() => rubric(), cheat:runCheat, validate, teardown:()=>undefined };
window.__choredesk = window.__task;
window.__choredesk.cheatHelpers = { route: async (v) => { route(v); await new Promise(r => setTimeout(r, 0)); }, set: (sel, value) => { const el = document.querySelector(sel); if (!el) throw new Error('cheat selector missing: ' + sel); el.value = value; el.dispatchEvent(new Event('input', { bubbles:true })); el.dispatchEvent(new Event('change', { bubbles:true })); } };
state.viewsVisited[view] = (state.viewsVisited[view] || 0) + 1; saveState(); render();
})();
</script>
</body>
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Serve the test suite on a fixed local port so pi-chrome can drive it.
cd "$(dirname "$0")"
PORT="${PORT:-8765}"
echo "serving http://127.0.0.1:${PORT}/"
exec python3 -m http.server "${PORT}" --bind 127.0.0.1
+416
View File
@@ -0,0 +1,416 @@
[
{
"id": "choredesk-l1-ticket-priority",
"site": "choredesk",
"startUrl": "scenarios/choredesk/index.html?task=choredesk-l1-ticket-priority&run=$RUN_ID&seed=$SEED",
"difficulty": "L1",
"category": "enterprise-crud",
"intent": "Set ticket T-104 priority to High and status to In Progress, then save it.",
"expectedActions": {
"min": 3,
"max": 6
},
"requires": [
"snapshot",
"click",
"select/change",
"form editing"
],
"grader": {
"type": "inPageProgrammatic",
"expression": "window.__taskVerdict || (window.__taskValidation?.done ? \"PASS\" : \"FAIL\")",
"passValue": "PASS"
},
"browserGymLike": {
"observation": [
"dom",
"accessibility_tree",
"screenshot_optional"
],
"actionSpace": [
"click",
"type",
"select",
"evaluate_optional"
],
"reset": "navigate startUrl with fresh $RUN_ID or click Reset run",
"stepReturn": "obs,reward,terminated,truncated,info"
},
"viewport": {
"width": 1280,
"height": 720
},
"slowMoMs": 1000,
"timeoutMs": 5000,
"locale": null,
"timezoneId": null,
"humanBaseline": null,
"actionSubsets": [
"bid",
"nav",
"tab",
"infeas",
"chat"
],
"rewardShape": "binary",
"taskId": "choredesk.L1.ticket-priority",
"seed": 101,
"maxSteps": 8,
"allowedActions": [
"click",
"fill",
"select_option",
"goto",
"report_infeasible",
"send_msg_to_user"
],
"setupUrl": "scenarios/choredesk/index.html?task=choredesk-l1-ticket-priority&run=$RUN_ID&seed=$SEED",
"validateScript": "scenarios/choredesk/index.html#window.__choredesk.validate()",
"cheatScript": "scenarios/choredesk/cheats.js#choredesk-l1-ticket-priority",
"tags": [
"form-fill",
"ticket",
"select"
],
"rubric": [
{
"id": "priorityHigh",
"weight": 0.33,
"description": "T-104 priority is High"
},
{
"id": "statusInProgress",
"weight": 0.33,
"description": "T-104 status is In Progress"
},
{
"id": "saved",
"weight": 0.34,
"description": "T-104 was saved"
}
],
"goal": "Set ticket T-104 priority to High and status to In Progress, then save it.",
"goalObject": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Set ticket T-104 priority to High and status to In Progress, then save it."
}
]
}
],
"gradeSource": "page"
},
{
"id": "choredesk-l2-refund-message",
"site": "choredesk",
"startUrl": "scenarios/choredesk/index.html?task=choredesk-l2-refund-message&run=$RUN_ID&seed=$SEED",
"difficulty": "L2",
"category": "cross-page-memory",
"intent": "Resolve a customer billing task by looking up customer email and order total on separate pages, then composing a message with exact required fields.",
"expectedActions": {
"min": 8,
"max": 14
},
"requires": [
"multi-page navigation",
"table lookup",
"working memory",
"form composition"
],
"grader": {
"type": "inPageProgrammatic",
"expression": "window.__taskVerdict || (window.__taskValidation?.done ? \"PASS\" : \"FAIL\")",
"passValue": "PASS"
},
"browserGymLike": {
"observation": [
"dom",
"accessibility_tree",
"screenshot_optional"
],
"actionSpace": [
"click",
"type",
"select",
"evaluate_optional"
],
"reset": "navigate startUrl with fresh $RUN_ID or click Reset run",
"stepReturn": "obs,reward,terminated,truncated,info"
},
"viewport": {
"width": 1280,
"height": 720
},
"slowMoMs": 1000,
"timeoutMs": 5000,
"locale": null,
"timezoneId": null,
"humanBaseline": null,
"actionSubsets": [
"bid",
"nav",
"tab",
"infeas",
"chat"
],
"rewardShape": "binary",
"taskId": "choredesk.L2.refund-message",
"seed": 202,
"maxSteps": 16,
"allowedActions": [
"click",
"fill",
"select_option",
"goto",
"report_infeasible",
"send_msg_to_user"
],
"setupUrl": "scenarios/choredesk/index.html?task=choredesk-l2-refund-message&run=$RUN_ID&seed=$SEED",
"validateScript": "scenarios/choredesk/index.html#window.__choredesk.validate()",
"cheatScript": "scenarios/choredesk/cheats.js#choredesk-l2-refund-message",
"tags": [
"lookup",
"cross-page-memory",
"message-compose"
],
"rubric": [
{
"id": "messageSubject",
"weight": 0.34,
"description": "Billing message subject matches"
},
{
"id": "email",
"weight": 0.33,
"description": "Body includes Noah email"
},
{
"id": "total",
"weight": 0.33,
"description": "Body includes exact order total"
}
],
"goal": "Resolve a customer billing task by looking up customer email and order total on separate pages, then composing a message with exact required fields.",
"goalObject": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Resolve a customer billing task by looking up customer email and order total on separate pages, then composing a message with exact required fields."
}
]
}
],
"gradeSource": "page"
},
{
"id": "choredesk-l3-restock-ticket",
"site": "choredesk",
"startUrl": "scenarios/choredesk/index.html?task=choredesk-l3-restock-ticket&run=$RUN_ID&seed=$SEED",
"difficulty": "L3",
"category": "cross-page-state-update",
"intent": "Use ticket text to identify an item, map item to SKU in Catalog, update Inventory reorder quantity, and add exact internal note back on the ticket.",
"expectedActions": {
"min": 12,
"max": 20
},
"requires": [
"multi-page navigation",
"entity resolution",
"state mutation",
"exact text note",
"audit-save behavior"
],
"grader": {
"type": "inPageProgrammatic",
"expression": "window.__taskVerdict || (window.__taskValidation?.done ? \"PASS\" : \"FAIL\")",
"passValue": "PASS"
},
"browserGymLike": {
"observation": [
"dom",
"accessibility_tree",
"screenshot_optional"
],
"actionSpace": [
"click",
"type",
"select",
"evaluate_optional"
],
"reset": "navigate startUrl with fresh $RUN_ID or click Reset run",
"stepReturn": "obs,reward,terminated,truncated,info"
},
"viewport": {
"width": 1280,
"height": 720
},
"slowMoMs": 1000,
"timeoutMs": 5000,
"locale": null,
"timezoneId": null,
"humanBaseline": null,
"actionSubsets": [
"bid",
"nav",
"tab",
"infeas",
"chat"
],
"rewardShape": "binary",
"taskId": "choredesk.L3.restock-ticket",
"seed": 303,
"maxSteps": 24,
"allowedActions": [
"click",
"fill",
"select_option",
"goto",
"report_infeasible",
"send_msg_to_user"
],
"setupUrl": "scenarios/choredesk/index.html?task=choredesk-l3-restock-ticket&run=$RUN_ID&seed=$SEED",
"validateScript": "scenarios/choredesk/index.html#window.__choredesk.validate()",
"cheatScript": "scenarios/choredesk/cheats.js#choredesk-l3-restock-ticket",
"tags": [
"entity-resolution",
"cross-page-state-update",
"exact-note"
],
"rubric": [
{
"id": "inventory",
"weight": 0.4,
"description": "AST-LAMP reorder quantity is 12 and saved"
},
{
"id": "ticketNote",
"weight": 0.4,
"description": "T-205 note contains exact phrase"
},
{
"id": "audit",
"weight": 0.2,
"description": "Both save audit entries exist"
}
],
"goal": "Use ticket text to identify an item, map item to SKU in Catalog, update Inventory reorder quantity, and add exact internal note back on the ticket.",
"goalObject": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Use ticket text to identify an item, map item to SKU in Catalog, update Inventory reorder quantity, and add exact internal note back on the ticket."
}
]
}
],
"gradeSource": "page"
},
{
"id": "mini-shop-red-second-cheapest",
"site": "mini-shop",
"startUrl": "fixtures/sites/mini-shop/index.html?task=mini-shop-red-second-cheapest&run=$RUN_ID&seed=$SEED",
"difficulty": "L2",
"category": "task-completion",
"intent": "Add the second-cheapest red product to cart, checkout, and leave the generated order id visible.",
"expectedActions": {
"min": 5,
"max": 9
},
"requires": [
"filter/sort reasoning",
"cart state",
"checkout flow"
],
"grader": {
"type": "inPageProgrammatic",
"expression": "window.__taskVerdict || (window.__taskValidation?.done ? \"PASS\" : \"FAIL\")",
"passValue": "PASS"
},
"browserGymLike": {
"observation": [
"dom",
"accessibility_tree",
"screenshot_optional"
],
"actionSpace": [
"click",
"read_table",
"evaluate_optional"
],
"reset": "navigate startUrl with fresh $RUN_ID",
"stepReturn": "obs,reward,terminated,truncated,info"
},
"viewport": {
"width": 1280,
"height": 720
},
"slowMoMs": 1000,
"timeoutMs": 5000,
"locale": null,
"timezoneId": null,
"humanBaseline": null,
"actionSubsets": [
"bid",
"nav",
"tab",
"infeas",
"chat"
],
"rewardShape": "binary",
"taskId": "mini-shop.L2.red-second-cheapest",
"seed": 404,
"maxSteps": 12,
"allowedActions": [
"click",
"goto",
"report_infeasible",
"send_msg_to_user"
],
"setupUrl": "fixtures/sites/mini-shop/index.html?task=mini-shop-red-second-cheapest&run=$RUN_ID&seed=$SEED",
"validateScript": "fixtures/sites/mini-shop/grader.js#MiniShopGrader.validate",
"cheatScript": "fixtures/sites/mini-shop/cheats.js#mini-shop-red-second-cheapest",
"tags": [
"shopping",
"sort",
"cart",
"checkout"
],
"rubric": [
{
"id": "addedTarget",
"weight": 0.4,
"description": "Cart includes second-cheapest red product"
},
{
"id": "noWrongItems",
"weight": 0.3,
"description": "Cart has no wrong products"
},
{
"id": "checkedOut",
"weight": 0.3,
"description": "Checkout produced order id"
}
],
"goal": "Add the second-cheapest red product to cart, checkout, and leave the generated order id visible.",
"goalObject": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Add the second-cheapest red product to cart, checkout, and leave the generated order id visible."
}
]
}
],
"gradeSource": "both"
}
]
@@ -0,0 +1,59 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "pi-chrome long-horizon task manifest",
"description": "BrowserGym-inspired task manifest: setup URL, deterministic seed, goal object, action subsets, reward contract, optional cheat recipe, and in-page validate hook.",
"type": "array",
"items": {
"type": "object",
"required": ["id", "taskId", "site", "startUrl", "setupUrl", "difficulty", "category", "goal", "goalObject", "grader", "actionSubsets", "maxSteps"],
"additionalProperties": true,
"properties": {
"id": { "type": "string" },
"taskId": { "type": "string" },
"site": { "type": "string" },
"seed": { "type": "integer" },
"startUrl": { "type": "string", "description": "Relative URL. $RUN_ID and $SEED should be replaced by runner for isolated deterministic state." },
"setupUrl": { "type": "string" },
"validateScript": { "type": "string" },
"cheatScript": { "type": "string" },
"viewport": { "type": "object", "properties": { "width": { "type": "integer" }, "height": { "type": "integer" } } },
"slowMoMs": { "type": "integer" },
"timeoutMs": { "type": "integer" },
"locale": { "type": ["string", "null"] },
"timezoneId": { "type": ["string", "null"] },
"difficulty": { "enum": ["L1", "L2", "L3"] },
"category": { "enum": ["enterprise-crud", "cross-page-memory", "cross-page-state-update", "task-completion"] },
"gradeSource": { "enum": ["page", "harness", "both"] },
"tags": { "type": "array", "items": { "type": "string" } },
"intent": { "type": "string" },
"goal": { "type": "string" },
"goalObject": { "type": "array", "items": { "type": "object" } },
"expectedActions": { "type": "object", "required": ["min", "max"], "properties": { "min": { "type": "integer" }, "max": { "type": "integer" } } },
"maxSteps": { "type": "integer" },
"humanBaseline": { "type": ["number", "null"], "minimum": 0, "maximum": 1 },
"actionSubsets": { "type": "array", "items": { "enum": ["chat", "infeas", "bid", "coord", "nav", "tab"] } },
"allowedActions": { "type": "array", "items": { "type": "string" } },
"requires": { "type": "array", "items": { "type": "string" } },
"rewardShape": { "enum": ["binary", "graded", "shaped"] },
"rubric": { "type": "array", "items": { "type": "object", "required": ["id", "weight", "description"], "properties": { "id": { "type": "string" }, "weight": { "type": "number" }, "description": { "type": "string" } } } },
"grader": {
"type": "object",
"required": ["type", "expression", "passValue"],
"properties": {
"type": { "enum": ["inPageProgrammatic"] },
"expression": { "type": "string", "description": "Should evaluate to BrowserGym-like validate object or legacy {v,r}." },
"passValue": { "type": "string" }
}
},
"browserGymLike": {
"type": "object",
"properties": {
"observation": { "type": "array", "items": { "type": "string" } },
"actionSpace": { "type": "array", "items": { "type": "string" } },
"reset": { "type": "string" },
"stepReturn": { "const": "obs,reward,terminated,truncated,info" }
}
}
}
}
}
@@ -0,0 +1,395 @@
// Unit harness for pi-chrome's dedicated automation tab/window isolation in service_worker.js.
//
// Feature under test: pi-chrome must never navigate or replace the user's active tab. Page and
// navigation actions without an explicit target are routed to a dedicated automation target that
// the *calling Pi session* created and owns. Ownership is session-scoped (one extension brokers
// every session) and mirrored to chrome.storage.session so a service-worker restart re-hydrates
// it instead of orphaning the window. Cleanup closes only the calling session's owned target.
//
// Like csp-eval.test.mjs we load the *real* worker into a vm sandbox with a stateful chrome.*
// mock, then exercise the real helpers and the real dispatch() paths. Chrome state (tabs/windows/
// storage.session) can be shared across two sandbox loads to simulate a service-worker restart.
import vm from "node:vm";
import fs from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const workerPath = path.resolve(__dirname, "../../extensions/chrome-profile-bridge/browser-extension/service_worker.js");
const src = fs.readFileSync(workerPath, "utf8");
let failures = 0;
let passes = 0;
function ok(cond, msg) {
if (cond) { passes++; }
else { failures++; console.error(`${msg}`); }
}
async function throwsWith(fn, re, msg) {
try { await fn(); ok(false, `${msg} (expected throw)`); }
catch (e) { ok(re.test(String(e.message || e)), `${msg} (got: ${e.message})`); }
}
// ---- stateful Chrome mock. `state` (tabs/windows/storage) can be shared to simulate a
// service-worker restart: the browser keeps its tabs/windows/session-storage, the worker memory
// is wiped (a fresh sandbox).
function makeChromeState() {
const tabs = new Map(); // id -> { id, windowId, url, active, groupId }
const windows = new Map(); // id -> { id }
const groups = new Map(); // groupId -> { id, title, color, collapsed, windowId }
const storage = {}; // chrome.storage.session backing
let nextTabId = 1;
let nextWindowId = 1;
let nextGroupId = 1;
const alloc = { tab: () => nextTabId++, window: () => nextWindowId++, group: () => nextGroupId++ };
// Seed a user window with two real user tabs (Gmail + a research article, the active one).
const userWindowId = alloc.window();
windows.set(userWindowId, { id: userWindowId });
const userGmail = { id: alloc.tab(), windowId: userWindowId, url: "https://mail.google.com/", active: false, groupId: -1 };
const userArticle = { id: alloc.tab(), windowId: userWindowId, url: "https://example.com/research-article", active: true, groupId: -1 };
tabs.set(userGmail.id, userGmail);
tabs.set(userArticle.id, userArticle);
return { tabs, windows, groups, storage, alloc, userWindowId, userGmail, userArticle };
}
function makeChrome(state, { withWindows = true, withStorage = true, withTabGroups = false } = {}) {
const { tabs, windows, groups, storage, alloc, userWindowId } = state;
const noop = () => {};
const listener = { addListener: noop, removeListener: noop };
const chrome = {
runtime: { id: "unittestextension", getManifest: () => ({ version: "0.0.0" }), onInstalled: listener, onStartup: listener, lastError: null },
alarms: { onAlarm: listener, create: noop, clear: noop, clearAll: noop },
action: { onClicked: listener },
debugger: { sendCommand: noop, attach: async () => {}, detach: async () => {}, getTargets: (cb) => cb([]), onDetach: listener },
scripting: { executeScript: async () => [{ result: undefined }], registerContentScripts: async () => {}, unregisterContentScripts: async () => {} },
webNavigation: { onCommitted: listener },
tabs: {
onUpdated: listener,
query: async (q = {}) => {
let list = [...tabs.values()];
if (q.active === true) list = list.filter((t) => t.active);
if (typeof q.windowId === "number") list = list.filter((t) => t.windowId === q.windowId);
return list.map((t) => ({ ...t }));
},
get: async (id) => { const t = tabs.get(id); if (!t) throw new Error(`No tab with id ${id}`); return { ...t }; },
create: async ({ url = "about:blank", active = false, windowId = userWindowId } = {}) => {
const tab = { id: alloc.tab(), windowId, url, active, groupId: -1 };
tabs.set(tab.id, tab);
return { ...tab };
},
update: async (id, props = {}) => { const t = tabs.get(id); if (!t) throw new Error(`No tab with id ${id}`); Object.assign(t, props); return { ...t }; },
remove: async (id) => { tabs.delete(id); },
group: async ({ groupId, tabIds = [] } = {}) => {
let gid = groupId;
if (typeof gid !== "number") {
gid = alloc.group();
const firstTab = tabs.get(tabIds[0]);
groups.set(gid, { id: gid, title: "", color: "grey", collapsed: false, windowId: firstTab ? firstTab.windowId : userWindowId });
}
for (const tid of tabIds) { const t = tabs.get(tid); if (t) t.groupId = gid; }
return gid;
},
ungroup: async (id) => { const ids = Array.isArray(id) ? id : [id]; for (const tid of ids) { const t = tabs.get(tid); if (t) t.groupId = -1; } },
},
storage: withStorage ? {
session: {
get: async (key) => (key in storage ? { [key]: storage[key] } : {}),
set: async (obj) => { Object.assign(storage, obj); },
},
} : undefined,
};
if (withTabGroups) {
chrome.tabGroups = {
query: async ({ windowId } = {}) => [...groups.values()].filter((g) => windowId === undefined || g.windowId === windowId).map((g) => ({ ...g })),
get: async (id) => { const g = groups.get(id); if (!g) throw new Error(`No group ${id}`); return { ...g }; },
update: async (id, props = {}) => { const g = groups.get(id); if (!g) throw new Error(`No group ${id}`); Object.assign(g, props); return { ...g }; },
};
}
if (withWindows) {
chrome.windows = {
create: async ({ url = "about:blank", focused = false } = {}) => {
const id = alloc.window();
windows.set(id, { id });
const tab = { id: alloc.tab(), windowId: id, url, active: true, groupId: -1 };
tabs.set(tab.id, tab);
return { id, focused, tabs: [{ ...tab }] };
},
get: async (id) => { const w = windows.get(id); if (!w) throw new Error(`No window with id ${id}`); return { ...w }; },
remove: async (id) => { windows.delete(id); for (const [tid, t] of [...tabs]) if (t.windowId === id) tabs.delete(tid); },
update: async () => {},
};
} else {
chrome.windows = { update: async () => {} }; // no create/get/remove -> tab fallback path
}
return chrome;
}
function loadWorker(chrome) {
const noop = () => {};
const sandbox = {
console, JSON, Date, Math, Promise, Array, Object, String, Number, Boolean,
Error, TypeError, Map, Set, BigInt, Symbol, structuredClone,
setTimeout, clearTimeout, setInterval: () => 0, clearInterval: noop,
fetch: async () => { throw new Error("no network in unit test"); },
navigator: { userAgent: "unit-test" },
WebSocket: function () {},
chrome,
};
sandbox.globalThis = sandbox;
sandbox.self = sandbox;
vm.createContext(sandbox);
vm.runInContext(src, sandbox);
return sandbox;
}
const SK = "session:alpha"; // a representative sessionKey
async function run() {
// ===== Isolation: navigation does not touch the user's active/other tabs. =====
{
const state = makeChromeState();
const w = loadWorker(makeChrome(state));
const userActiveUrl = state.userArticle.url;
const nav = await w.dispatch("page.navigate", { url: "https://pi.test/task", waitUntilLoad: false, sessionKey: SK });
ok(state.userArticle.url === userActiveUrl, "navigate: active user tab (research article) is not overwritten");
ok(state.userGmail.url === "https://mail.google.com/", "navigate: other user tab (Gmail) untouched");
ok(nav.url === "https://pi.test/task", "navigate: automation target navigated to requested URL");
ok(nav.id !== state.userArticle.id && nav.id !== state.userGmail.id, "navigate: did not reuse any user tab");
ok(nav.windowId !== state.userWindowId, "navigate: automation target lives in a dedicated window");
const status = await w.dispatch("automation.status", { sessionKey: SK });
ok(status.tabId === nav.id && status.windowId === nav.windowId, "ownership: target ids tracked for the session");
ok(w.isPiChromeOwnedTarget(nav.id, SK) === true, "ownership: isPiChromeOwnedTarget(owned, session) === true");
ok(w.isPiChromeOwnedTarget(state.userArticle.id) === false, "ownership: user tab is never owned (any session)");
// Reuse: a later navigation reuses the same owned target.
const nav2 = await w.dispatch("page.navigate", { url: "https://pi.test/step-2", waitUntilLoad: false, sessionKey: SK });
ok(nav2.id === nav.id && nav2.windowId === nav.windowId, "reuse: second navigation reuses the same automation window/tab");
ok(state.userArticle.url === userActiveUrl, "reuse: user tab still untouched after second navigation");
// Cleanup closes only the owned window; user tabs/windows survive.
const cleanup = await w.dispatch("automation.cleanup", { sessionKey: SK });
ok(cleanup.closedWindowId === nav.windowId, "cleanup: closed the owned window");
ok(state.tabs.has(state.userArticle.id) && state.tabs.has(state.userGmail.id), "cleanup: user tabs never closed");
ok(state.windows.has(state.userWindowId), "cleanup: user window never closed");
ok(!state.tabs.has(nav.id), "cleanup: the owned automation tab is gone");
const status2 = await w.dispatch("automation.status", { sessionKey: SK });
ok(status2.tabId === null && status2.windowId === null, "cleanup: ownership cleared");
}
// ===== Session-group integration: the dedicated-window tab joins this session's group. =====
{
const state = makeChromeState();
const w = loadWorker(makeChrome(state, { withTabGroups: true }));
// index.ts tags page.* actions with joinSessionGroup + sessionGroupTitle; replicate that here.
const groupTitle = "Pi Session: alpha";
const nav = await w.dispatch("page.navigate", {
url: "https://pi.test/grouped", waitUntilLoad: false,
sessionKey: SK, joinSessionGroup: true, sessionGroupTitle: groupTitle,
});
const navTab = state.tabs.get(nav.id);
ok(navTab.windowId !== state.userWindowId, "group: automation tab is in its dedicated window");
ok(typeof navTab.groupId === "number" && navTab.groupId >= 0, "group: automation tab joined a tab group");
const grp = state.groups.get(navTab.groupId);
ok(grp && grp.title === groupTitle, "group: the group is titled with this session's title");
ok(grp.windowId === navTab.windowId, "group: the session group lives inside the dedicated automation window (not the user window)");
// A second page action reuses the same tab and does not spawn a second group.
const groupsBefore = state.groups.size;
await w.dispatch("page.navigate", { url: "https://pi.test/grouped-2", waitUntilLoad: false, sessionKey: SK, joinSessionGroup: true, sessionGroupTitle: groupTitle });
ok(state.groups.size === groupsBefore, "group: reusing the automation tab does not create a second group");
}
// ===== tab.new joins the existing session group instead of creating one group per window. =====
{
const state = makeChromeState();
const w = loadWorker(makeChrome(state, { withTabGroups: true }));
const groupTitle = "Pi Session: alpha";
const nav = await w.dispatch("page.navigate", {
url: "https://pi.test/group-owner", waitUntilLoad: false,
sessionKey: SK, joinSessionGroup: true, sessionGroupTitle: groupTitle,
});
const navTab = state.tabs.get(nav.id);
const groupId = navTab.groupId;
const groupsBefore = state.groups.size;
const opened = await w.dispatch("tab.new", { url: "https://pi.test/new-tab", groupTitle, sessionKey: SK });
ok(state.groups.size === groupsBefore, "tab.new-group: did not create another same-session group");
ok(opened.tab.groupId === groupId, "tab.new-group: opened tab joined the existing session group");
ok(opened.tab.windowId === nav.windowId, "tab.new-group: opened tab was created in the existing group's window");
const forced = await w.dispatch("tab.new", { url: "https://pi.test/no-opt-out", groupTitle, group: false, sessionKey: SK });
ok(forced.tab.groupId === groupId, "tab.new-group: group:false is ignored; tab still joins the session group");
ok(state.groups.size === groupsBefore, "tab.new-group: group:false does not create another group");
const blankTitle = await w.dispatch("tab.new", { url: "https://pi.test/blank-title", groupTitle: "", group: false, sessionKey: SK });
ok(typeof blankTitle.tab.groupId === "number" && blankTitle.tab.groupId >= 0, "tab.new-group: groupTitle:'' still creates a grouped tab");
ok(blankTitle.group.title === "Pi", "tab.new-group: blank groupTitle falls back to a group instead of opting out");
const nav2 = await w.dispatch("page.navigate", {
url: "https://pi.test/new-automation-target", waitUntilLoad: false,
sessionKey: "session:beta", joinSessionGroup: true, sessionGroupTitle: groupTitle,
});
ok(state.groups.size === groupsBefore + 1, "automation-target-group: reused the existing session group, only blank-title Pi group was extra");
ok(nav2.groupId === groupId, "automation-target-group: new automation target joined the existing session group");
ok(nav2.windowId === nav.windowId, "automation-target-group: new automation target was created in the existing group's window");
}
// ===== tab.new never leaves an ungrouped tab behind when grouping fails. =====
{
const state = makeChromeState();
const chrome = makeChrome(state, { withTabGroups: true });
const w = loadWorker(chrome);
const tabsBefore = state.tabs.size;
chrome.tabs.group = async () => { throw new Error("group blew up"); };
await throwsWith(
() => w.dispatch("tab.new", { url: "https://pi.test/group-fail", groupTitle: "Pi Session: alpha", sessionKey: SK }),
/group blew up/,
"tab.new-group-fail: surfaces grouping error",
);
ok(state.tabs.size === tabsBefore, "tab.new-group-fail: closes the created tab instead of leaving it ungrouped");
}
// ===== Grouping is best-effort: a tabGroups failure must not break navigation. =====
{
const state = makeChromeState();
const chrome = makeChrome(state, { withTabGroups: true });
chrome.tabs.group = async () => { throw new Error("group blew up"); };
const w = loadWorker(chrome);
const nav = await w.dispatch("page.navigate", { url: "https://pi.test/group-fail", waitUntilLoad: false, sessionKey: SK, joinSessionGroup: true, sessionGroupTitle: "Pi Session: alpha" });
ok(nav.url === "https://pi.test/group-fail", "group-fail: navigation still succeeds when grouping throws");
ok(state.tabs.get(nav.id).windowId !== state.userWindowId, "group-fail: still used the dedicated automation window");
}
// ===== Concurrency: two sessions get separate windows; cleanup is per-session. =====
{
const state = makeChromeState();
const w = loadWorker(makeChrome(state));
const a = await w.dispatch("page.navigate", { url: "https://pi.test/a", waitUntilLoad: false, sessionKey: "session:A" });
const b = await w.dispatch("page.navigate", { url: "https://pi.test/b", waitUntilLoad: false, sessionKey: "session:B" });
ok(a.id !== b.id && a.windowId !== b.windowId, "concurrency: each session gets its own dedicated window/tab");
ok(w.isPiChromeOwnedTarget(a.id, "session:A") && !w.isPiChromeOwnedTarget(a.id, "session:B"), "concurrency: ownership is scoped to the creating session");
// Cleaning up session A must not touch session B's target.
await w.dispatch("automation.cleanup", { sessionKey: "session:A" });
ok(!state.tabs.has(a.id), "concurrency: cleanup closed session A's tab");
ok(state.tabs.has(b.id), "concurrency: cleanup left session B's tab open");
const bStatus = await w.dispatch("automation.status", { sessionKey: "session:B" });
ok(bStatus.tabId === b.id, "concurrency: session B still owns its target after A cleanup");
}
// ===== Service-worker restart / reconnect: persisted ownership re-hydrates from storage. =====
{
const state = makeChromeState();
const w1 = loadWorker(makeChrome(state));
const nav = await w1.dispatch("page.navigate", { url: "https://pi.test/persist", waitUntilLoad: false, sessionKey: SK });
ok(typeof state.storage.piChromeAutomationTargets === "object", "restart: ownership was persisted to storage.session");
// Simulate the MV3 service worker being suspended and restarted: fresh sandbox (memory wiped),
// same browser tabs/windows + same session storage.
const w2 = loadWorker(makeChrome(state));
const statusAfterRestart = await w2.dispatch("automation.status", { sessionKey: SK });
ok(statusAfterRestart.tabId === nav.id && statusAfterRestart.windowId === nav.windowId, "restart: re-hydrated the owned target from storage");
// A navigation after restart must REUSE the existing window, not orphan it with a new one.
const windowsBefore = state.windows.size;
const nav2 = await w2.dispatch("page.navigate", { url: "https://pi.test/persist-2", waitUntilLoad: false, sessionKey: SK });
ok(nav2.id === nav.id && nav2.windowId === nav.windowId, "restart: navigation after restart reuses the persisted window (no orphan)");
ok(state.windows.size === windowsBefore, "restart: no new window created after restart");
// Cleanup after restart works and clears persisted state.
await w2.dispatch("automation.cleanup", { sessionKey: SK });
const persisted = state.storage.piChromeAutomationTargets || {};
ok(!(SK in persisted), "restart: cleanup removed the session from persisted storage");
}
// ===== Restart after the user manually closed the window: no orphan, fresh target. =====
{
const state = makeChromeState();
const w1 = loadWorker(makeChrome(state));
const nav = await w1.dispatch("page.navigate", { url: "https://pi.test/closed", waitUntilLoad: false, sessionKey: SK });
await state.windows.delete(nav.windowId); // user closed pi-chrome's window
for (const [tid, t] of [...state.tabs]) if (t.windowId === nav.windowId) state.tabs.delete(tid);
const w2 = loadWorker(makeChrome(state)); // SW restart
const nav2 = await w2.dispatch("page.navigate", { url: "https://pi.test/reopened", waitUntilLoad: false, sessionKey: SK });
ok(nav2.id !== nav.id, "restart-after-close: a fresh automation target is created when the persisted one is gone");
ok(state.tabs.has(nav2.id), "restart-after-close: new target exists");
}
// ===== tab.* management never auto-creates / never falls back to the user's active tab. =====
{
const state = makeChromeState();
const w = loadWorker(makeChrome(state));
const windowsBefore = state.windows.size;
const tabsBefore = state.tabs.size;
await throwsWith(
() => w.dispatch("tab.close", { sessionKey: SK }),
/no automation tab yet|Pass targetId/,
"tab.close: with no target and no owned target, errors instead of closing the user's active tab",
);
ok(state.tabs.has(state.userArticle.id), "tab.close: user's active tab was NOT closed");
ok(state.windows.size === windowsBefore && state.tabs.size === tabsBefore, "tab.close: did not spawn a throwaway tab/window");
await throwsWith(() => w.dispatch("tab.activate", { sessionKey: SK }), /no automation tab yet|Pass targetId/, "tab.activate: errors with no target/owned target");
// Once an automation target exists, management actions operate on it (not on the user tab).
const nav = await w.dispatch("page.navigate", { url: "https://pi.test/manage", waitUntilLoad: false, sessionKey: SK });
const closed = await w.dispatch("tab.close", { sessionKey: SK });
ok(closed.closed === nav.id, "tab.close: with an owned target, closes that target");
ok(state.tabs.has(state.userArticle.id), "tab.close: user tab still safe after closing the owned target");
}
// ===== Explicit targeting still works on any existing tab (no regression). =====
{
const state = makeChromeState();
const w = loadWorker(makeChrome(state));
const nav = await w.dispatch("page.navigate", { url: "https://pi.test/explicit", targetId: String(state.userGmail.id), waitUntilLoad: false, sessionKey: SK });
ok(nav.id === state.userGmail.id, "explicit: targetId routes to the requested existing tab");
ok(state.userGmail.url === "https://pi.test/explicit", "explicit: explicitly targeted tab is navigated");
const status = await w.dispatch("automation.status", { sessionKey: SK });
ok(status.tabId === null, "explicit: explicit targeting does not create/claim an automation target");
}
// ===== Window-unavailable fallback: a dedicated TAB is used, and the user's window is safe. =====
{
const state = makeChromeState();
const w = loadWorker(makeChrome(state, { withWindows: false }));
const target = await w.getOrCreateAutomationTarget(SK);
ok(target.id !== state.userArticle.id && target.id !== state.userGmail.id, "fallback: created a dedicated tab, not a user tab");
ok(w.isPiChromeOwnedTarget(target.id, SK) === true, "fallback: dedicated tab is owned");
const cleanup = await w.cleanupAutomationTarget(SK);
ok(cleanup.closedTabId === target.id && cleanup.closedWindowId === null, "fallback: cleanup closes only the owned tab (never the shared window)");
ok(state.windows.has(state.userWindowId), "fallback: cleanup never closes the user/shared window");
ok(state.tabs.has(state.userArticle.id) && state.tabs.has(state.userGmail.id), "fallback: cleanup leaves user tabs intact");
}
// ===== Robust cleanup: no-op when nothing created, and when target already closed manually. =====
{
const state = makeChromeState();
const w = loadWorker(makeChrome(state));
const empty = await w.cleanupAutomationTarget(SK);
ok(empty.closedWindowId === null && empty.closedTabId === null, "cleanup: no-op when nothing was ever created");
const t = await w.getOrCreateAutomationTarget(SK);
// User closed pi-chrome's window manually (Chrome closes its tabs too).
state.windows.delete(t.windowId);
for (const [tid, tab] of [...state.tabs]) if (tab.windowId === t.windowId) state.tabs.delete(tid);
const stale = await w.cleanupAutomationTarget(SK);
ok(stale.closedWindowId === null && stale.closedTabId === null, "cleanup: robust when owned window was already closed");
}
console.log(`\n${passes} passed, ${failures} failed`);
if (failures) process.exit(1);
}
run().catch((e) => { console.error(e); process.exit(1); });
+192
View File
@@ -0,0 +1,192 @@
// Unit harness for the CSP-bypass layer in service_worker.js.
//
// The real CSP bypass (CDP Runtime.evaluate not being subject to page CSP) can only be
// proven in a browser — see challenge 39-strict-csp-fallback. These tests instead validate
// the JS *logic* of the refactor that the bypass depends on:
// - evaluateInTab: wrapper-string construction, expression/statement fallback, value
// marker round-trip (undefined/function/symbol/bigint/Error/DOMRect), error propagation.
// - executeInTab: 2-phase define-then-invoke, envelope unwrap, error propagation, and that
// all real HELPER_FUNCS serialize+assign without a parse error.
// - page.waitFor: service-worker-side polling via evaluateInTab (selector + expression).
//
// We load the worker into a vm sandbox with mocked chrome.* APIs, then replace `cdp` with a
// shim that evaluates the expression in a separate "page world" vm context (simulating CDP
// Runtime.evaluate returnByValue). No browser, no network, no deps.
import vm from "node:vm";
import fs from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const workerPath = path.resolve(__dirname, "../../extensions/chrome-profile-bridge/browser-extension/service_worker.js");
const src = fs.readFileSync(workerPath, "utf8");
let failures = 0;
let passes = 0;
function ok(cond, msg) {
if (cond) { passes++; }
else { failures++; console.error(`${msg}`); }
}
async function throwsWith(fn, re, msg) {
try { await fn(); ok(false, `${msg} (expected throw)`); }
catch (e) { ok(re.test(String(e.message || e)), `${msg} (got: ${e.message})`); }
}
// ---- page world: simulates the page's MAIN world for Runtime.evaluate ----
const pageGlobals = {
console, JSON, Date, Math, Promise, Object, Array, String, Number, Boolean,
Error, TypeError, SyntaxError, RangeError, BigInt, Symbol, structuredClone,
setTimeout, parseInt, parseFloat, isNaN,
document: {
title: "page title",
_present: new Set(),
querySelector(sel) { return this._present.has(sel) ? { sel } : null; },
},
};
pageGlobals.window = pageGlobals;
pageGlobals.globalThis = pageGlobals;
const pageWorld = vm.createContext(pageGlobals);
// Simulate CDP Runtime.evaluate returnByValue serialization.
function toCdpResult(v) {
if (v === undefined) return { result: { type: "undefined" } };
if (v === null) return { result: { type: "object", subtype: "null", value: null } };
const t = typeof v;
if (t === "number" || t === "string" || t === "boolean")
return { result: { type: t, value: v } };
// object/array: returnByValue deep-clones JSON-able structures
return { result: { type: "object", value: JSON.parse(JSON.stringify(v)) } };
}
// ---- worker sandbox ----
const noop = () => {};
const listener = { addListener: noop, removeListener: noop };
const sandbox = {
console, JSON, Date, Math, Promise, Array, Object, String, Number, Boolean,
Error, TypeError, Map, Set, BigInt, Symbol, structuredClone,
setTimeout, clearTimeout,
setInterval: () => 0,
clearInterval: noop,
fetch: async () => { throw new Error("no network in unit test"); },
navigator: { userAgent: "unit-test" },
WebSocket: function () {},
chrome: {
runtime: { id: "unittestextension", getManifest: () => ({ version: "0.0.0" }), onInstalled: listener, onStartup: listener, lastError: null },
alarms: { onAlarm: listener, create: noop, clear: noop, clearAll: noop },
action: { onClicked: listener },
debugger: { sendCommand: noop, attach: async () => {}, detach: async () => {}, getTargets: (cb) => cb([]) },
scripting: { executeScript: async () => [{ result: undefined }] },
tabs: { query: async () => [], get: async () => ({}), create: async () => ({}), update: async () => ({}), remove: async () => {} },
windows: { update: async () => {} },
webNavigation: { onCommitted: listener },
},
};
sandbox.globalThis = sandbox;
sandbox.self = sandbox;
vm.createContext(sandbox);
vm.runInContext(src, sandbox);
// ---- override the page-touching primitives with the page-world shim ----
sandbox.attachDebugger = async () => ({});
sandbox.bringToFront = async () => {};
sandbox.getTabByParams = async (p) => ({ id: (p && p.targetId) || 1, windowId: 1 });
sandbox.cdp = async (_tabId, method, params) => {
if (method !== "Runtime.evaluate") return {};
try {
const value = await vm.runInContext(params.expression, pageWorld);
return toCdpResult(value);
} catch (e) {
return { exceptionDetails: { exception: { className: e.name, description: String(e.stack || e.message) }, text: "Uncaught " + String(e) } };
}
};
// Phase-2 of executeInTab: run the injected wrapper func against the page world,
// where Phase-1 (via cdp shim above) already defined window.__piAction + helpers.
sandbox.chrome.scripting.executeScript = async ({ func, args }) => {
const fn = vm.runInContext("(" + func.toString() + ")", pageWorld);
const result = await fn(...(args || []));
return [{ result }];
};
const { evaluateInTab, executeInTab, dispatch } = sandbox;
async function run() {
// ===== evaluateInTab: primitives & objects =====
ok((await evaluateInTab({ expression: "2 + 2" })) === 4, "evaluate: arithmetic expression");
ok((await evaluateInTab({ expression: "document.title" })) === "page title", "evaluate: expression without return");
ok((await evaluateInTab({ expression: "'a' + 'b'" })) === "ab", "evaluate: string concat");
const obj = await evaluateInTab({ expression: "({a:1, b:[2,3]})" });
ok(obj && obj.a === 1 && obj.b[1] === 3, "evaluate: object literal round-trips");
// ===== value markers =====
ok((await evaluateInTab({ expression: "void 0" })) === undefined, "evaluate: undefined marker -> undefined");
ok((await evaluateInTab({ expression: "10n" })) === "10", "evaluate: bigint marker -> string");
ok(/^\[Function:/.test(await evaluateInTab({ expression: "(function foo(){})" })), "evaluate: function marker");
ok((await evaluateInTab({ expression: "Promise.resolve(42)" })) === 42, "evaluate: promise is awaited");
// DOMRect-like (toJSON + width/height/top) is expanded, not flattened to {}
const rect = await evaluateInTab({ expression: "({ x:1,y:2,width:3,height:4,top:2,right:4,bottom:6,left:1, toJSON(){return {}} })" });
ok(rect && rect.width === 3 && rect.bottom === 6, "evaluate: DOMRect-like expanded");
// ===== statement-form fallback (expression form is a SyntaxError) =====
// `let x=...; x` is not a valid expression, so the wrapper must retry as a statement body.
ok((await evaluateInTab({ expression: "let x = 5; x" })) === undefined, "evaluate: statement form falls back (no return -> undefined)");
ok((await evaluateInTab({ expression: "let y = 7; return y" })) === 7, "evaluate: statement form with explicit return");
// ===== error propagation =====
await throwsWith(() => evaluateInTab({ expression: "throw new Error('boom')" }), /chrome_evaluate failed[\s\S]*boom/, "evaluate: runtime error propagates");
// ===== executeInTab: 2-phase define + invoke =====
// Real HELPER_FUNCS get serialized + assigned in Phase 1; a parse error there would throw here.
const sum = await executeInTab({ targetId: 1 }, function add(a, b) { return a + b; }, [3, 4]);
ok(sum === 7, "executeInTab: action runs with args after helper injection");
const asyncResult = await executeInTab({ targetId: 1 }, async function asyncEcho(v) { return v * 2; }, [21]);
ok(asyncResult === 42, "executeInTab: async action awaited");
await throwsWith(
() => executeInTab({ targetId: 1 }, function boom() { throw new Error("action failed"); }, []),
/action failed/,
"executeInTab: thrown action error propagates via envelope",
);
// ===== page.waitFor (service-worker-side polling) =====
pageGlobals.document._present.add("#ready");
const wf = await dispatch("page.waitFor", { targetId: 1, kind: "selector", value: "#ready", timeoutMs: 1000, intervalMs: 20 });
ok(wf && typeof wf.elapsedMs === "number", "waitFor: selector present resolves");
const wfExpr = await dispatch("page.waitFor", { targetId: 1, kind: "expression", value: "1 === 1", timeoutMs: 1000, intervalMs: 20 });
ok(wfExpr && typeof wfExpr.elapsedMs === "number", "waitFor: truthy expression resolves");
await throwsWith(
() => dispatch("page.waitFor", { targetId: 1, kind: "selector", value: "#never", timeoutMs: 120, intervalMs: 30 }),
/Timed out after 120ms/,
"waitFor: missing selector times out",
);
// ===== usKeyLayoutForChar / cdpKeyInfo: US-layout key codes =====
// Regression: punctuation must NOT use charCodeAt() (".":46 collides with VK_DELETE,
// "-":45 with VK_INSERT), which made apps drop the char on keydown.
const { usKeyLayoutForChar, cdpKeyInfo } = sandbox;
const period = usKeyLayoutForChar(".");
ok(period.code === "Period" && period.keyCode === 190 && !period.needShift, "keylayout: '.' -> Period/190 (not 46)");
const dash = usKeyLayoutForChar("-");
ok(dash.code === "Minus" && dash.keyCode === 189, "keylayout: '-' -> Minus/189 (not 45)");
const slash = usKeyLayoutForChar("/");
ok(slash.code === "Slash" && slash.keyCode === 191, "keylayout: '/' -> Slash/191");
const at = usKeyLayoutForChar("@");
ok(at.code === "Digit2" && at.keyCode === 50 && at.needShift, "keylayout: '@' -> Digit2/50 + shift");
const A = usKeyLayoutForChar("A");
ok(A.code === "KeyA" && A.keyCode === 65 && A.needShift, "keylayout: 'A' -> KeyA/65 + shift");
const a = usKeyLayoutForChar("a");
ok(a.code === "KeyA" && a.keyCode === 65 && !a.needShift, "keylayout: 'a' -> KeyA/65 no shift");
const dot = cdpKeyInfo(".");
ok(dot.code === "Period" && dot.windowsVirtualKeyCode === 190 && dot.text === ".", "cdpKeyInfo: '.' -> Period/190 with text");
const ent = cdpKeyInfo("Enter");
ok(ent.code === "Enter" && ent.windowsVirtualKeyCode === 13, "cdpKeyInfo: named key 'Enter' unaffected");
console.log(`\n${passes} passed, ${failures} failed`);
if (failures) process.exit(1);
}
run().catch((e) => { console.error(e); process.exit(1); });