mirror of
https://bitbucket.org/siakitem/my-pi.git
synced 2026-08-28 08:35:57 +00:00
feat(chrome): hand snapshots to context mode
This commit is contained in:
云服务部-叶林立
@@ -0,0 +1,70 @@
|
||||
# BrowserGym compatibility contract
|
||||
|
||||
Pi-chrome benchmark tasks mirror BrowserGym without importing Gymnasium.
|
||||
|
||||
## Task fields
|
||||
|
||||
Each `task-manifest.json` entry maps to `AbstractBrowserTask` concepts:
|
||||
|
||||
- `taskId` / `id` — stable namespaced id
|
||||
- `seed` — deterministic state seed
|
||||
- `viewport`, `slowMoMs`, `timeoutMs`, `locale`, `timezoneId`
|
||||
- `goal` and `goalObject` — text and OpenAI-style message objects
|
||||
- `setupUrl` — page URL that performs setup from `task`, `run`, `seed`
|
||||
- `validateScript` — JS validate hook returning reward contract
|
||||
- `cheatScript` — gold recipe used to sanity-check grader
|
||||
- `maxSteps`, `humanBaseline`, `difficulty`, `tags`
|
||||
- `actionSubsets`, `allowedActions` — BrowserGym HighLevelActionSet metadata
|
||||
|
||||
## Reset / step return
|
||||
|
||||
Runner should expose:
|
||||
|
||||
```text
|
||||
reset() -> (obs, info)
|
||||
step(action) -> (obs, reward, terminated, truncated, info)
|
||||
```
|
||||
|
||||
`terminated` means task done. `truncated` means max step or timeout.
|
||||
|
||||
## Validate return
|
||||
|
||||
In-page validators return:
|
||||
|
||||
```js
|
||||
{ reward, done, message, info }
|
||||
```
|
||||
|
||||
`reward` is incremental. Current hermetic tasks use binary terminal reward. `info.rubric` carries per-check details.
|
||||
|
||||
## Observation schema target
|
||||
|
||||
Runner observations should include BrowserGym-like keys:
|
||||
|
||||
- `chat_messages`
|
||||
- `goal`, `goal_object`
|
||||
- `open_pages_urls`, `open_pages_titles`, `active_page_index`
|
||||
- `url`, `screenshot`, `dom_object`, `axtree_object`
|
||||
- `extra_element_properties`
|
||||
- `focused_element_bid`
|
||||
- `last_action`, `last_action_error`
|
||||
- `elapsed_time`
|
||||
|
||||
## BID plan
|
||||
|
||||
BrowserGym uses `bid` attributes as stable element handles. Pi-chrome currently returns snapshot `uid`. Compatibility path:
|
||||
|
||||
1. During snapshot, assign every interactive element a deterministic `bid`.
|
||||
2. Return both `uid` and `bid`.
|
||||
3. Add `visibility`, `bbox`, `clickable`, `set_of_marks` metadata.
|
||||
4. Maintain uid↔bid mapping so agents can use either addressing mode.
|
||||
|
||||
Target attributes:
|
||||
|
||||
- `bid`
|
||||
- `browsergym_visibility_ratio`
|
||||
- `browsergym_set_of_marks`
|
||||
|
||||
## Action subsets
|
||||
|
||||
See `../browsergym-action-space.json` for BrowserGym-compatible subsets and function signatures.
|
||||
@@ -0,0 +1,79 @@
|
||||
# Historical bypass notes
|
||||
|
||||
This file predates the current `trusted:true` / `chrome.debugger` path and is
|
||||
kept as design history. For current benchmark expectations, recipes, and
|
||||
capability notes, use `../manifest.json`.
|
||||
|
||||
Older context: `pi-chrome` originally ran mostly through a content-script bridge.
|
||||
That limited the solution space — most "real-event" tricks needed the
|
||||
`chrome.debugger` API or `chrome.input.synthesizeMouseEvent`.
|
||||
|
||||
## 01–02 isTrusted click / keyboard
|
||||
**Hard.** `Event.isTrusted` is true only for events the browser dispatched
|
||||
itself. Two viable paths:
|
||||
|
||||
- **`chrome.debugger` + `Input.dispatchMouseEvent` / `Input.dispatchKeyEvent`.**
|
||||
Adds `"debugger"` permission. Chrome shows a "X is debugging this browser"
|
||||
banner, but events arrive as `isTrusted=true`. Best fix for the highest-value
|
||||
cases; gate it behind a toggle the user opts into.
|
||||
- **`chrome.input.ime.*`** — not applicable for general typing.
|
||||
|
||||
Today's `dispatchEvent` path will always fail any `isTrusted` check.
|
||||
|
||||
## 03 / 12 webdriver + fingerprint
|
||||
Cheap wins:
|
||||
- Stub `navigator.webdriver` via `chrome.scripting.executeScript({ world: "MAIN", injectImmediately: true, … })` that defines `navigator.webdriver` getter to `undefined`. Already half-supported through `initScript` on `chrome_navigate` — extend to a per-tab content script registered with `chrome.scripting.registerContentScripts({ runAt: "document_start", world: "MAIN" })`.
|
||||
- Spoof languages/plugins/permissions only if the user explicitly enables it — by default the real Chrome profile already looks legitimate. The bigger risk is `chrome_evaluate` *itself* leaving traces (Function-constructor stack frames in errors).
|
||||
|
||||
## 04 mouse entropy
|
||||
Generate a humanised pointer path before any click:
|
||||
- Bezier interpolation from current cursor pos (or a random off-target start) to the target point.
|
||||
- 20–60 steps, easing curve, ±jitter on each step, varying `movementX/Y`.
|
||||
- Implement in `clickPage` before the existing pointerdown sequence; reuse `pointerEventSequence` for each interpolated point.
|
||||
|
||||
## 05 event timing
|
||||
- Insert `await sleep(rand(40,140))` between `pointerdown` and `pointerup`.
|
||||
- Insert `await sleep(rand(80,220))` between successive synthesised clicks when
|
||||
the caller issues them in a loop.
|
||||
- Vary by ±20%.
|
||||
|
||||
## 06 click coordinates
|
||||
- After resolving target rect in `clickPage`, pick `(cx ± rand(-rw*0.3,rw*0.3), cy ± rand(-rh*0.3,rh*0.3))` instead of dead center. Already returns the chosen point — keep that.
|
||||
|
||||
## 07 pointer properties
|
||||
- Set `pressure: 0.5` for `pointerdown` (currently inits don't set pressure; browser default for mouse pointerdown is 0.5).
|
||||
- Set `pointerId: 1` for mouse (already done) and 2+ for touch.
|
||||
- Carry `movementX/Y` from the previous interpolated step (needs path state).
|
||||
|
||||
## 08 / 09 keyboard cadence + framework invariants
|
||||
- Per character: dispatch `keydown` (cancelable, fills `key` + `code` + `keyCode`), then `keypress` for printables, then `beforeinput` `{inputType:"insertText", data:ch}`, then mutate the input's value to `cur+ch` via the native setter, then `input` `{inputType:"insertText", data:ch}`, then `keyup`. Sleep 40–120 ms between chars.
|
||||
- Today the bridge fills the entire value in one go and emits one `beforeinput`/`input`. Replace with the per-char loop above.
|
||||
- Make sure not to dispatch `compositionstart`/`compositionend` for plain ASCII.
|
||||
|
||||
## 10 user activation
|
||||
**Impossible with synthetic events.** `navigator.userActivation.isActive` only
|
||||
flips on browser-trusted events. Same fix as 01/02 — `chrome.debugger` path.
|
||||
Document the limitation; offer a `useDebuggerForActivationGates` opt-in.
|
||||
|
||||
## 11 honeypot
|
||||
Pure agent-side concern. Before filling/clicking by selector, the bridge could:
|
||||
- Reject targets with `display:none`, `visibility:hidden`, `aria-hidden=true`,
|
||||
or off-screen position (`getBoundingClientRect()` outside viewport bounds and
|
||||
not scrolled-into-view), unless the caller passes `force: true`.
|
||||
- That logic largely exists (`elementVisible` / `occludedBy` are returned) —
|
||||
promote it from informational to a guard.
|
||||
|
||||
## 13 focus order
|
||||
- When clicking a focusable element, dispatch `pointerdown` first; only after
|
||||
`mousedown` defaults run let focus settle naturally (browser will move focus
|
||||
on the synthetic `mousedown` only if `isTrusted` is true — alas, no). Manual
|
||||
workaround: dispatch the pointer/mouse sequence, then explicitly call
|
||||
`target.focus({ preventScroll: true })`, then dispatch `focus` with
|
||||
`{ relatedTarget: previouslyFocused }`. Set `:focus-visible` heuristic via
|
||||
`target.blur(); target.focus({ focusVisible: false })` (Chrome 124+).
|
||||
|
||||
## 14 wheel scroll
|
||||
- For scroll operations, dispatch sequential `wheel` events with `deltaY`
|
||||
chunks (e.g., 60–120 per tick), and let the browser scroll, instead of
|
||||
setting `scrollTop` directly. Provide `chrome_scroll({ uid, dy, dx })` as a
|
||||
first-class tool.
|
||||
@@ -0,0 +1,22 @@
|
||||
# Calibration profiles
|
||||
|
||||
Benchmark results depend on Chrome profile, OS, input device, and extension mode.
|
||||
Record these with every run:
|
||||
|
||||
- OS and version
|
||||
- Chrome version
|
||||
- pi-chrome package version
|
||||
- companion extension version
|
||||
- trusted mode: off / auto / on
|
||||
- input hardware: trackpad vs detent mouse, touch support
|
||||
- permissions: clipboard, fullscreen, downloads
|
||||
- viewport size and deviceScaleFactor
|
||||
- loaded privacy/security extensions
|
||||
|
||||
Known environment-sensitive areas:
|
||||
|
||||
- WebGL fingerprint can warn/fail in VMs, remote desktops, GPU-disabled Chrome.
|
||||
- Scroll momentum expects trackpad-like decay; detent mouse wheels may fail manually.
|
||||
- Touch tests require touch events enabled/supported.
|
||||
- Dialog/download/file-picker tests require native browser UI handling or manual intervention.
|
||||
- Stack traces vary across Chrome versions; tests should only fail concrete automation URLs/Runtime frames.
|
||||
@@ -0,0 +1,29 @@
|
||||
# Recipe runner spec
|
||||
|
||||
Future Node/Pi runner should:
|
||||
|
||||
1. Read `manifest.json`.
|
||||
2. Probe environment capabilities (`trusted`, `clipboard`, `touch`, `dialogs`, `downloads`, `fileSystem`, `strictCspFallback`).
|
||||
3. For each entry:
|
||||
- read `gate` (`core`, `conditional`, `quality`) and score in that bucket
|
||||
- skip if `requires` is unsatisfied and expected is `CONDITIONAL`
|
||||
- navigate to challenge URL
|
||||
- execute `recipe` in order
|
||||
- wait `verdictDelayMs || 500`
|
||||
- evaluate `JSON.stringify({v:window.__verdict,r:window.__reason,d:window.Challenge?.state?.details,e:window.__events?.slice(-20)})`, except strict-CSP pages where eval is expected to fail; read persisted verdict from dashboard/localStorage after leaving the CSP page
|
||||
- compare to `expected[mode]`
|
||||
4. Output:
|
||||
- Markdown summary
|
||||
- JSON details
|
||||
- JUnit XML for CI
|
||||
|
||||
Runner must adapt recipe intent:
|
||||
|
||||
- expand `${REPO_ROOT}` path placeholders
|
||||
- adapt unsupported shadow/iframe selector notation to snapshot uids or evaluate fallback
|
||||
- preserve hook install ordering for console/network capture tests
|
||||
- substitute dynamic tab ids from `chrome_tab list` recipes
|
||||
- support screenshot/coordinate fallback for strict CSP pages where snapshot/evaluate are blocked
|
||||
- record whether trusted/CDP path was used
|
||||
|
||||
Long-horizon task runner should read `task-manifest.json`, replace `$RUN_ID`, solve task, then evaluate task grader expression.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Benchmark scoring
|
||||
|
||||
## Unit challenges (`manifest.json`)
|
||||
|
||||
Score actual verdict against `expected[mode]`, not raw PASS count.
|
||||
|
||||
Gate buckets:
|
||||
|
||||
- `core`: headline release gate. Exact expected verdict required.
|
||||
- `conditional`: gate only when declared capability/prerequisite is present; otherwise report skipped/conditional.
|
||||
- `quality`: adversarial humanization/fingerprint signal. Report trend and regressions, but do not block general release unless explicitly promoted.
|
||||
|
||||
Expected values:
|
||||
|
||||
- `PASS` / `FAIL` expected: exact match required inside active gate bucket.
|
||||
- `CONDITIONAL`: exclude from core headline score unless environment prerequisite is declared present.
|
||||
- `SKIP` / `WARN`: report separately.
|
||||
|
||||
Recommended flake policy:
|
||||
|
||||
- Run each non-destructive challenge up to 2 retries.
|
||||
- Take best verdict only for known-flaky timing/scroll tests.
|
||||
- Always keep all reasons/events in detailed report.
|
||||
|
||||
Difficulty weighting:
|
||||
|
||||
- L1 = 1
|
||||
- L2 = 2
|
||||
- L3 = 3
|
||||
|
||||
Report both unweighted and weighted score. Also report per-category score.
|
||||
|
||||
## Long-horizon tasks (`task-manifest.json`)
|
||||
|
||||
Task score is deterministic grader PASS / not PASS. Record:
|
||||
|
||||
- action count
|
||||
- wall time
|
||||
- tools used
|
||||
- observation mode (`snapshot`, screenshot, accessibility, evaluate)
|
||||
- whether direct state mutation/evaluate was allowed
|
||||
- final grader reason
|
||||
|
||||
Do not use LLM judge for hermetic tasks unless grader cannot express the target.
|
||||
Reference in New Issue
Block a user