mirror of
https://bitbucket.org/siakitem/my-pi.git
synced 2026-08-28 08:35:57 +00:00
feat(chrome): hand snapshots to context mode
This commit is contained in:
云服务部-叶林立
@@ -0,0 +1,74 @@
|
||||
# pi-chrome architecture
|
||||
|
||||
`pi-chrome` connects Pi to your existing Chrome profile through a local-only bridge and an unpacked Chrome extension.
|
||||
|
||||
```text
|
||||
+----------------------+ +--------------------------+
|
||||
| Pi agent (terminal) | -- 127.0.0.1:17318 ->| Chrome extension |
|
||||
| chrome_* tools | | (your real profile) |
|
||||
+-----------+----------+ +-------------+------------+
|
||||
| same machine |
|
||||
v v
|
||||
Other Pi sessions Tabs you already have open
|
||||
share same bridge (GitHub, Linear, Stripe, etc.)
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
- **Pi extension** — exposes `chrome_*` tools and `/chrome` commands inside Pi.
|
||||
- **Loopback bridge** — listens on `127.0.0.1:17318`; no external network bind by default.
|
||||
- **Chrome companion extension** — loaded unpacked into your real Chrome profile.
|
||||
- **Chrome debugger / CDP** — drives input, screenshots, network/console observation, and evaluation.
|
||||
|
||||
## Session model
|
||||
|
||||
Multiple Pi sessions can use same Chrome companion extension. First session opens local bridge; later sessions detect it and pipe commands through.
|
||||
|
||||
Each Pi session owns its own automation target:
|
||||
|
||||
- First chrome action without explicit target opens dedicated automation window.
|
||||
- If separate window cannot be created, pi-chrome falls back to dedicated tab.
|
||||
- Target survives `/reload` and Chrome service-worker restarts.
|
||||
- Ownership is tracked by id and mirrored to `chrome.storage.session`.
|
||||
- Cleanup closes only calling session's own target, never user tabs/windows or other sessions' targets.
|
||||
|
||||
To point pi-chrome at an existing tab, pass `targetId`, `urlIncludes`, or `titleIncludes`.
|
||||
|
||||
## Tab management guards
|
||||
|
||||
`chrome_tab` management actions are guarded:
|
||||
|
||||
- `activate`, `close`, `group`, and `ungroup` without explicit target act on session automation tab if it exists.
|
||||
- If no automation tab exists, operation errors instead of touching your active tab.
|
||||
|
||||
## Background mode
|
||||
|
||||
By default, chrome calls run in background so Chrome does not steal focus.
|
||||
|
||||
```text
|
||||
/chrome background on # background mode
|
||||
/chrome background off # foreground/watch mode
|
||||
```
|
||||
|
||||
Per-call `background: false` brings Chrome forward for that action. Per-call `background: true` forces background.
|
||||
|
||||
## Authorization
|
||||
|
||||
Bridge connection alone is not enough. Chrome control stays locked until current Pi session runs:
|
||||
|
||||
```text
|
||||
/chrome authorize
|
||||
```
|
||||
|
||||
Authorization expires after configured duration, on `/chrome revoke`, or when Pi exits.
|
||||
|
||||
## Unpacked extension choice
|
||||
|
||||
`pi-chrome` ships browser extension source as an unpacked folder on purpose:
|
||||
|
||||
- easy to inspect before loading
|
||||
- no Web Store release delay
|
||||
- MIT-licensed source in repo
|
||||
- `/chrome doctor` can compare loaded extension version against installed package
|
||||
|
||||
Loaded extension has broad tab/scripting permissions inside profile where it is installed. Install only from trusted package source.
|
||||
@@ -0,0 +1,166 @@
|
||||
# pi-chrome vs. the rest of the browser-automation landscape
|
||||
|
||||
This is the honest "which tool when" page. The browser-automation space has three different layers and people often compare across them — let's not do that.
|
||||
|
||||
We benchmark in public — see [`../test-suite/`](../test-suite). Where exact scores matter (WebVoyager, WorkArena++, BrowseComp, Mind2Web 2), check the live leaderboards; they shift monthly.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
| You are… | Use… |
|
||||
| -------------------------------------------------------------- | ------------------------------- |
|
||||
| A Pi agent operator who wants the agent to use **your real Chrome** (logged-in tabs, cookies, extensions) | **pi-chrome** |
|
||||
| Building a Pi/LLM agent and want low-level browser primitives | **pi-chrome** |
|
||||
| Writing deterministic end-to-end tests in CI | Playwright / Cypress |
|
||||
| Building a hosted scraping/agent fleet on isolated profiles | Playwright / Puppeteer + Browserbase / Steel |
|
||||
| Want a turnkey "natural-language agent" with built-in loop | Browser Use / Stagehand / Skyvern |
|
||||
| Want a hosted, vendor-managed agent | OpenAI Operator / Project Mariner / Surfer |
|
||||
| Debugging your own app from your editor without leaving your real session | **pi-chrome** |
|
||||
|
||||
`pi-chrome` is **primitives**, not an opinionated agent loop. Think of it as **"Playwright for the Chrome you're already signed into"** — and pluggable under any agent framework above the line.
|
||||
|
||||
---
|
||||
|
||||
## The three axes you should compare on
|
||||
|
||||
| Axis | Examples | What it gives you |
|
||||
| --------------------------------- | --------------------------------------------------------------------- | ----------------------------- |
|
||||
| **1. Driver / transport** | Playwright, Puppeteer, Selenium, CDP raw, `puppeteer-stealth`, **pi-chrome** | low-level tools (click, type, navigate) |
|
||||
| **2. Agent framework** | Browser Use, Stagehand, Skyvern, Magnitude, Alumnium, LangGraph-with-Playwright, Operator, Mariner, Surfer | LLM loop, planning, NL API |
|
||||
| **3. Cloud browser provider** | Browserbase, Steel.dev, Hyperbrowser, Anchor, Browserless | managed Chromes, sessions, quotas |
|
||||
|
||||
`pi-chrome` is **axis 1**. You can wrap it with any axis-2 agent framework, or run it directly from a Pi agent's `chrome_*` tool calls. It explicitly does NOT compete with axis 3 — it runs locally inside *your* Chrome.
|
||||
|
||||
---
|
||||
|
||||
## Axis 1 — drivers (where pi-chrome lives)
|
||||
|
||||
| Tool | Transport | Profile | Browser input | Banner when controlling | Default detectable as bot |
|
||||
| --------------------------------- | ------------------------------------------ | ---------------------------------- | -------------------- | ----------------------------------- | ------------------------- |
|
||||
| Playwright | CDP (own driver) | throwaway by default | always | always ("controlled by test software") | yes (webdriver flag, automation flags) |
|
||||
| Puppeteer | CDP | throwaway by default | always | always | yes |
|
||||
| Selenium | WebDriver / BiDi | throwaway | partial (BiDi improves) | always | most detectable |
|
||||
| puppeteer-stealth / playwright-extra | CDP + patches | throwaway | always | always | medium (patches flags) |
|
||||
| Raw CDP | direct devtools protocol | either (needs `--remote-debugging-port`) | always | always | yes |
|
||||
| **pi-chrome** | **Chrome extension bridge → local loopback** | **your real Chrome profile, signed-in cookies, extensions, history** | **always for input tools** | **while Chrome input is attached** | **low (real profile + Chrome input)¹** |
|
||||
|
||||
¹ pi-chrome uses `chrome.debugger` for browser input and shows Chrome's banner like other CDP-based tools. The [`test-suite/`](../test-suite) grades browser-control behavior against common detection signals.
|
||||
|
||||
### What makes pi-chrome different on this axis
|
||||
|
||||
1. **Profile attach, not driver launch.** Every other driver fights cookie persistence, login walls, MFA, and extension state. pi-chrome inherits all of it because it *is* your Chrome.
|
||||
2. **Chrome input against your real profile.** Interactive tools use CDP input for reliability while still controlling the Chrome profile you already use.
|
||||
3. **Extension bridge transport.** No `--remote-debugging-port`, no throwaway Chromium. Survives Chrome auto-updates. Works alongside your normal Chrome usage.
|
||||
4. **Structured action results.** Input tools return target coordinates/tags and can include a fresh snapshot (`includeSnapshot`) so agents can verify state instead of blindly retrying.
|
||||
5. **Multi-session shared bridge.** Planner + worker + audit Pi sessions all drive the same Chrome concurrently.
|
||||
6. **Stable element uids.** `chrome_snapshot` returns deterministic uids you can pass to subsequent actions — similar to BrowserGym's `bid`, but built into the snapshot tool itself.
|
||||
|
||||
---
|
||||
|
||||
## Axis 2 — agent frameworks (built on top of axis 1)
|
||||
|
||||
These wrap a driver with an LLM loop. They are **higher-level than pi-chrome** and **complementary**, not competitors.
|
||||
|
||||
| Framework | Driver underneath | Approach | Open source |
|
||||
| ------------------------ | ------------------------------ | --------------------------------------------------------------------------------------------- | --------------- |
|
||||
| **Browser Use** | Playwright | DOM + a11y tree → LLM → action JSON. Open-source leader; widely cited on WebVoyager. | MIT (Python) |
|
||||
| **Stagehand** (Browserbase) | Playwright | Natural-language `.act()` / `.observe()` / `.extract()`; deterministic + AI mix. | MIT (TypeScript)|
|
||||
| **Skyvern** | Playwright + own DOM model | Vision-first + DOM; YAML workflows for form/workflow automation. | AGPL (Python) |
|
||||
| **Magnitude** | Playwright | NL test authoring; QA-focused. | open |
|
||||
| **Alumnium** | Selenium / Playwright | Test-author NL → agent. QA-focused. | open |
|
||||
| **LangGraph / AutoGen + Playwright** | Playwright | Generic agent graph + browser tools. | open |
|
||||
| **OpenAI Operator** | proprietary | OpenAI's own VLM + browser; ChatGPT-integrated. | closed, hosted |
|
||||
| **Project Mariner** (Google) | proprietary Chrome integration | Google's own VLM Chrome experiment. | closed |
|
||||
| **Surfer 2 / Surfer-H** (H Company) | proprietary | Hosted proprietary agent stack. | closed, hosted |
|
||||
| **Anthropic Computer Use** | OS-level screenshots + mouse/keyboard | Broader than browser; OS-level events. | closed (API) |
|
||||
|
||||
**Why pi-chrome is not on this list:** it's intentionally **not an agent**. There's no LLM loop, no `.act("click the blue button")`. Pi handles the loop; pi-chrome provides the primitives. This means:
|
||||
|
||||
- You can use pi-chrome under Browser Use, Stagehand, LangGraph, or any other agent framework (see [Interop](#interop)).
|
||||
- You don't pay for an opinion you don't want.
|
||||
- Agent improvements compound across all your workflows because the primitives stay stable.
|
||||
|
||||
---
|
||||
|
||||
## Axis 3 — cloud browser providers (orthogonal)
|
||||
|
||||
| Provider | What it sells |
|
||||
| ----------------- | -------------------------------------------------------- |
|
||||
| **Browserbase** | Managed browsers; pairs with Stagehand. |
|
||||
| **Steel.dev** | Managed browsers + public agent leaderboards. |
|
||||
| **Hyperbrowser** | Managed browsers, session APIs. |
|
||||
| **Anchor Browser**| Managed browsers. |
|
||||
| **Browserless** | Managed Chrome, scraping focus. |
|
||||
|
||||
**pi-chrome doesn't compete here.** It runs locally in *your* Chrome. The right framing if someone asks: *"no cloud cost, no session-handoff, no rate limits — and the agent runs against your real logged-in state."* If you need fleets of isolated Chromes in CI, you want one of these.
|
||||
|
||||
---
|
||||
|
||||
## Interop
|
||||
|
||||
`pi-chrome` exposes tools that any Pi agent can call. If you want to use it from outside Pi:
|
||||
|
||||
1. The local bridge speaks HTTP JSON over `127.0.0.1:17318` (default). The API is internal; use the Pi tool surface unless you are building an adapter.
|
||||
2. Tool surface mirrors Playwright closely (click/type/navigate/snapshot/screenshot/evaluate/wait_for) so adapter code is short.
|
||||
3. `includeSnapshot` on input tools lets agent harnesses verify state after actions.
|
||||
|
||||
If you want a first-class pi-chrome adapter for Browser Use / Stagehand / LangGraph, file an issue with your use case.
|
||||
|
||||
---
|
||||
|
||||
## "But Playwright has `storageState` / Puppeteer has user-data-dir"
|
||||
|
||||
Yes — you can export cookies and replay them, or point Playwright at your existing profile directory. In practice for agent workflows that breaks down fast:
|
||||
|
||||
1. **OAuth + SSO** providers (Okta, Google, GitHub) often pin sessions to TLS fingerprints, device IDs, and browser-extension state that doesn't survive replay or a parallel Chrome instance.
|
||||
2. **MFA** tokens expire mid-run.
|
||||
3. **Internal admin tools** hard-pin to your real device.
|
||||
4. **Pointing Playwright at your real `user-data-dir`** requires closing your normal Chrome (Chrome won't share the profile lock). pi-chrome doesn't fight you for the profile because it lives *inside* it.
|
||||
5. **Watching the agent work** in your real window is a different UX than a hidden parallel Chrome. Demos, pair-driving, and confidence-building all want axis-1-with-attach.
|
||||
|
||||
---
|
||||
|
||||
## "Is this safer than CDP?"
|
||||
|
||||
Different security boundary, not strictly safer.
|
||||
|
||||
- **CDP-based tools** require `chrome --remote-debugging-port=...`. That port is unauthenticated and exposes the whole browser to any local process. Easy to misconfigure.
|
||||
- **pi-chrome** runs through an extension you install yourself with broad permissions (tabs, scripting, debugger, webNavigation). The bridge listens on `127.0.0.1:17318` loopback only, rejects browser-origin command requests, and keeps chrome_* tools locked until `/chrome authorize` is run in the current Pi session. **Only install the bundled extension if you trust the source you got the npm package from.**
|
||||
|
||||
If your threat model excludes extensions with broad permissions, neither approach is a fit — you want a sandboxed CI runner.
|
||||
|
||||
---
|
||||
|
||||
## Public benchmarks worth knowing (for axis 2 / axis 3 comparison)
|
||||
|
||||
Pi-chrome itself ships a benchmark suite ([`../test-suite/`](../test-suite)) of **42 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks** covering trusted input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, iframes, file uploads, strict-CSP screenshot fallback and CDP eval/snapshot bypass, dynamic waits, tab lifecycle, network observability, fingerprint leaks, and agent-safety honeypots. Scoring tracks expected outcomes per challenge instead of raw PASS count, with `core`, `conditional`, and `quality` gate buckets. That's **driver-level** grading.
|
||||
|
||||
For **agent-level** comparison (axis 2), the public benchmarks worth citing:
|
||||
|
||||
| Benchmark | What it measures | Notes |
|
||||
| ---------------- | ------------------------------------------------- | ------------------------------------------------ |
|
||||
| **WebArena** (CMU) | Hermetic, programmatic graders | Gold standard for reproducibility. |
|
||||
| **WorkArena++** (ServiceNow) | Enterprise SaaS workflows | Hardest realistic benchmark; <5% frontier. |
|
||||
| **BrowseComp** (OpenAI) | Hard info-retrieval | Not saturated. |
|
||||
| **Mind2Web 2** (NeurIPS '25) | Long-horizon, rubric-tree judge | New, well-designed. |
|
||||
| **WebChoreArena** | Tedious cross-page workflows | Reflects real ops work. |
|
||||
| **WebVoyager** | Live web tasks | **Saturated** — 90%+ scores common; cite only as smoke test. |
|
||||
| **VisualWebArena** | Multimodal | |
|
||||
| **MiniWoB++** | Classic unit-task suite | |
|
||||
| **BrowserGym + AgentLab** | Research harness covering the above | The de-facto research API; pi-chrome's snapshot uid is comparable to BrowserGym's `bid`. |
|
||||
|
||||
Cite live leaderboards rather than hard-coded numbers; agent scores shift monthly.
|
||||
|
||||
---
|
||||
|
||||
## Reproducing pi-chrome's driver-level claims
|
||||
|
||||
Run [`../test-suite/`](../test-suite) against any browser-control tool. Each challenge exposes `window.__verdict` / `window.__reason` / `window.__events`, so any tool (Playwright, Puppeteer, Selenium, Stagehand, pi-chrome) can grade itself deterministically. Headline release scoring should use the `core` gate; `conditional` depends on declared environment capabilities, and `quality` tracks adversarial/humanization regressions.
|
||||
|
||||
```bash
|
||||
cd test-suite && python3 -m http.server 8765
|
||||
# open http://127.0.0.1:8765/ in the Chrome window the tool controls
|
||||
```
|
||||
|
||||
If you build a competing tool, please open a PR with your scores.
|
||||
@@ -0,0 +1,166 @@
|
||||
# pi-chrome examples
|
||||
|
||||
Real, useful agent prompts. Drop any of these into Pi after running `/chrome onboard`, then `/chrome authorize`. Each one uses Chrome tabs and accounts you already have.
|
||||
|
||||
## Daily workflow
|
||||
|
||||
### PR triage
|
||||
|
||||
```text
|
||||
Use chrome_tab list to find my GitHub notifications tab.
|
||||
chrome_snapshot it. Group PRs by:
|
||||
- awaiting my review
|
||||
- blocked on me (changes requested back)
|
||||
- mergeable (approved + green CI)
|
||||
Output a 5-bullet ranked triage. Do not click anything.
|
||||
```
|
||||
|
||||
### Linear standup
|
||||
|
||||
```text
|
||||
Open my Linear current cycle in the active tab.
|
||||
chrome_snapshot, then write yesterday/today/blockers
|
||||
in the exact format my standup channel uses.
|
||||
```
|
||||
|
||||
### Slack catch-up
|
||||
|
||||
```text
|
||||
For each unread channel in my Slack tab, chrome_snapshot,
|
||||
extract the top 3 messages that mention me or my team,
|
||||
and summarize what I missed in <100 words total.
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
### Reproduce a customer bug with evidence
|
||||
|
||||
```text
|
||||
1. chrome_navigate to https://staging.acme.com/orders/<id>
|
||||
2. chrome_snapshot
|
||||
3. Click "Refund" with chrome_click
|
||||
4. Use chrome_list_network_requests to capture the API call
|
||||
5. chrome_get_network_request on the failing one — give me the response body
|
||||
6. chrome_screenshot the error state to ./repro/refund-bug.png
|
||||
```
|
||||
|
||||
### Visual diff local vs staging
|
||||
|
||||
```text
|
||||
chrome_screenshot http://localhost:3000/pricing → ./diff/local.png
|
||||
chrome_screenshot https://staging.acme.com/pricing → ./diff/staging.png
|
||||
Then read both, describe layout differences in plain English.
|
||||
```
|
||||
|
||||
### Console + network forensics
|
||||
|
||||
```text
|
||||
Reproduce the checkout bug on the active tab.
|
||||
After the failure:
|
||||
- chrome_list_console_messages
|
||||
- chrome_list_network_requests
|
||||
Cross-reference the timestamps and tell me what broke first.
|
||||
```
|
||||
|
||||
## Admin / ops
|
||||
|
||||
### Multi-tab cross-check
|
||||
|
||||
```text
|
||||
I have Stripe, Postmark, and our internal admin open in 3 tabs.
|
||||
For user <id>, chrome_snapshot each tab in turn and find
|
||||
any field where state disagrees. Output a 3-column table.
|
||||
```
|
||||
|
||||
### Bulk gentle action (safe form-fill, no submit)
|
||||
|
||||
```text
|
||||
Open our vendor portal "Add Vendor" form.
|
||||
For each row in ./vendors.csv:
|
||||
- chrome_fill the form
|
||||
- chrome_screenshot it
|
||||
- STOP before submit
|
||||
- chrome_evaluate "history.back()" to return to the list
|
||||
I will review screenshots and submit manually.
|
||||
```
|
||||
|
||||
### Auth-only data pull
|
||||
|
||||
```text
|
||||
My analytics dashboard is open and the cookie auth would die in headless mode.
|
||||
chrome_evaluate to read window.__APP_STATE__.dashboardData
|
||||
and dump today's KPIs as JSON.
|
||||
```
|
||||
|
||||
## Demos / PRs
|
||||
|
||||
### Capture screenshots for a PR description
|
||||
|
||||
```text
|
||||
On localhost:3000/feature-x:
|
||||
- empty state → ./pr/01-empty.png
|
||||
- filled state → ./pr/02-filled.png
|
||||
- error state (delete the API key from devtools first) → ./pr/03-error.png
|
||||
Save each with chrome_screenshot. Output a markdown block I can paste into the PR.
|
||||
```
|
||||
|
||||
### Record a guided demo flow
|
||||
|
||||
```text
|
||||
On my staging app:
|
||||
1. Walk the new-onboarding flow start to finish
|
||||
2. After each chrome_click or chrome_navigate, chrome_screenshot
|
||||
3. Save numbered PNGs under ./demo/
|
||||
4. Write narration captions for each step
|
||||
```
|
||||
|
||||
## Forms with frameworks
|
||||
|
||||
### React controlled inputs
|
||||
|
||||
```text
|
||||
Use `chrome_fill` for React inputs when you want to replace the full value.
|
||||
Pass `includeSnapshot=true` to verify the component re-rendered with the new value.
|
||||
```
|
||||
|
||||
### File upload without the native picker
|
||||
|
||||
```text
|
||||
chrome_upload_file paths=[./fixtures/avatar.png] selector="input[type=file]"
|
||||
# Uses Chrome file-input control. No native file picker opens.
|
||||
```
|
||||
|
||||
### Drag-to-reorder lists
|
||||
|
||||
```text
|
||||
chrome_drag fromUid=row-3 toUid=row-1
|
||||
# Uses Chrome pointer drag through its input layer.
|
||||
```
|
||||
|
||||
## Multi-session patterns
|
||||
|
||||
`pi-chrome` shares one bridge across all Pi sessions on the same machine. Useful patterns:
|
||||
|
||||
### Planner + Worker
|
||||
|
||||
- **Planner session** stays high level: "find the bug, decide the fix."
|
||||
- **Worker session** runs the actual `chrome_*` tools.
|
||||
- Both see the same Chrome state because they're both pointing at your real profile.
|
||||
|
||||
### Watcher
|
||||
|
||||
A third Pi session can run `chrome_snapshot` periodically in `background: true` mode and post summaries via `pi-qq` — handy for long-running flows.
|
||||
|
||||
## Chrome input
|
||||
|
||||
Interactive tools use Chrome's real input layer by default: clicks, typing, fill, keys, hover, drag, scroll, and touch. This is reliable for:
|
||||
|
||||
- sign-in flows
|
||||
- guarded buttons
|
||||
- audio/video controls
|
||||
- fullscreen and other user-activation checks
|
||||
- pages where DOM injection/evaluate is limited, if the agent can use screenshots + coordinates
|
||||
|
||||
Strict CSP note: `chrome_snapshot`/`chrome_evaluate` work even on pages that disallow `unsafe-eval`, because they run via CDP `Runtime.evaluate` (not page-level `eval`/`new Function`), which is not subject to page CSP. `chrome_screenshot`, tab/navigation tools, and real input also work under any CSP.
|
||||
|
||||
Chrome may show its debugger banner while pi-chrome is attached.
|
||||
@@ -0,0 +1,84 @@
|
||||
# pi-chrome FAQ
|
||||
|
||||
## Does this work with Brave / Arc / Edge / Vivaldi?
|
||||
|
||||
Yes. Any Chromium-based browser that supports unpacked extensions and the `chrome.debugger` API will work. The extension is named "Pi Chrome Connector" but the source is browser-agnostic. Firefox / WebKit are out of scope (different extension models).
|
||||
|
||||
## Will it slow my browser down?
|
||||
|
||||
The companion extension is idle when no Pi command is in flight. It uses Manifest V3 service worker activation, so it wakes for a request and goes back to sleep. No content script is injected globally.
|
||||
|
||||
## Does it work in Chrome incognito?
|
||||
|
||||
By default no — extensions need explicit "Allow in incognito" permission. Toggle it on `chrome://extensions` if you want pi-chrome to see incognito tabs. We don't recommend it for sensitive work.
|
||||
|
||||
## Will sites detect that I'm automating?
|
||||
|
||||
Interactive controls use Chrome's real input layer via CDP, so normal user-activation gates are satisfied and input is closer to real browser use than DOM-dispatched events. pi-chrome also shapes pointer/keyboard/scroll behavior, but this is not a guarantee of undetectability. Some detectors check for the `chrome.debugger` API attached, and Chrome will show the "Chrome is being debugged" banner.
|
||||
|
||||
The [`test-suite/`](../test-suite) grades browser-control behavior against common detection signals. Its `quality` bucket is adversarial signal, not a blanket promise that every site will treat automation as human.
|
||||
|
||||
## Why do I see a banner saying "Pi Chrome Connector started debugging this browser"?
|
||||
|
||||
That's Chrome's built-in warning when an extension uses `chrome.debugger`. pi-chrome uses Chrome's input layer for interactive controls, so the banner appears while attached.
|
||||
|
||||
## Can a malicious page escape and access my other tabs?
|
||||
|
||||
No — pages cannot directly talk to extensions. Commands flow agent → local bridge (`127.0.0.1:17318`) → extension → tab. The bridge binds to loopback only and rejects browser-origin command requests, so ordinary web pages cannot use CORS to drive it.
|
||||
|
||||
Chrome control is also locked per Pi session until you run `/chrome authorize`; `/chrome revoke` locks it again. The remaining risk surface is **other local processes running as you** that can connect to loopback and imitate Pi. If that's in your threat model, run pi-chrome in a separate OS user account.
|
||||
|
||||
## Can multiple Pi sessions use it at once?
|
||||
|
||||
Yes. The first session opens the local bridge; later sessions detect it and pipe their commands through the same bridge. Each Pi session must be authorized with `/chrome authorize` before its chrome_* tools work. Each session also owns its **own** dedicated automation window (ownership is keyed by session id inside the one extension), so concurrent sessions never navigate into or close each other's tabs.
|
||||
|
||||
## Does pi-chrome navigate my current tab?
|
||||
|
||||
No. The first chrome_* action that has no explicit target opens a **dedicated automation window** that pi-chrome owns (falling back to a dedicated tab only if a separate window can't be created), and reuses it for the rest of the session. Your existing tabs and windows are never reused or overwritten. Pass `targetId`/`urlIncludes`/`titleIncludes` to deliberately act on a tab you already have open.
|
||||
|
||||
The window survives `/reload` and Chrome service-worker restarts because ownership is tracked by id and mirrored to `chrome.storage.session`. It is closed when you run `/chrome revoke` and on real session end (not on `/reload`); cleanup only ever closes that session's own window/tab.
|
||||
|
||||
**After a full browser restart**, `chrome.storage.session` is cleared by Chrome. If Chrome's session-restore reopens the old automation window, pi-chrome no longer recognizes it (its tracking is gone), so it is left alone as an ordinary window — pi-chrome will open a fresh dedicated window for the new run rather than reclaim or close the restored one. pi-chrome never closes a window it can't positively identify as its own, so a user window is never at risk.
|
||||
|
||||
## Why ship as an unpacked extension?
|
||||
|
||||
pi-chrome ships as an unpacked extension so the source and broad browser permissions are easy to inspect and update with the npm package. The downside: you load it manually from `chrome://extensions` and reload it after package updates.
|
||||
|
||||
## What happens when I update pi-chrome?
|
||||
|
||||
`/chrome doctor` will warn you if the loaded extension is older than the installed `pi-chrome`. Reload it from `chrome://extensions` to pick up the new version. Updates that add Chrome permissions may require re-approval once.
|
||||
|
||||
## What's the install footprint?
|
||||
|
||||
- Pi side: one extension that registers 19 tools and a few slash commands.
|
||||
- Chrome side: one unpacked extension, ~2000 LOC of plain JavaScript, no dependencies.
|
||||
|
||||
## Can I script it without Pi?
|
||||
|
||||
The Pi-facing tools are thin wrappers around an HTTP bridge at `127.0.0.1:17318`. You could call it directly from any process, but the API is internal and may change. If you need a stable scripting interface, file an issue and we'll consider stabilizing.
|
||||
|
||||
## What can humans do that pi-chrome cannot?
|
||||
|
||||
pi-chrome controls web pages through Chrome extension APIs, page inspection, screenshots, and browser input. It is not full OS-level human control. Known gaps include native Chrome/OS dialogs (print/save-as, some permission bubbles, password-manager prompts), arbitrary OS app interaction, visual CAPTCHA challenges, hardware-backed auth (passkeys/security keys/biometrics), rich multi-touch/pinch/stylus gestures, and DOM inspection inside cross-origin iframes. Some of these can still be handled with screenshot + coordinate input or user assistance, but they are not first-class deterministic workflows.
|
||||
|
||||
## Does `chrome_evaluate` work on strict-CSP pages?
|
||||
|
||||
Yes. `chrome_evaluate` and `chrome_snapshot` run in the page's MAIN world through CDP `Runtime.evaluate`, which is a DevTools protocol command and is **not** subject to the page's Content-Security-Policy. They work even on pages that block `'unsafe-eval'` (e.g. github.com and many bank/SaaS apps). `chrome_navigate`'s `initScript` injects at document_start via CDP and likewise bypasses CSP. `chrome_screenshot`, tab tools, and real Chrome input also keep working under any CSP.
|
||||
|
||||
## How do I tell whether a click or type worked?
|
||||
|
||||
Use `includeSnapshot=true` on `chrome_click`, `chrome_type`, `chrome_fill`, or `chrome_key`. The tool returns the Chrome-input result plus a fresh snapshot, so the agent can verify text, URL, visible elements, or form values before continuing.
|
||||
|
||||
If the page did not change, take a fresh snapshot or screenshot and check for overlays, disabled controls, stale element uids, or app-side validation.
|
||||
|
||||
## How do I attach a file to a React file input?
|
||||
|
||||
`chrome_upload_file` — uses Chrome DevTools file-input control and fires `input` + `change` events. It does **not** open the native file picker. Works with React/Vue/Angular controlled inputs.
|
||||
|
||||
## Can it record videos?
|
||||
|
||||
Not yet. Screenshots only. Video recording is on the roadmap.
|
||||
|
||||
## How do I file a good bug report?
|
||||
|
||||
Include `/chrome doctor` output, the exact tool call, and the result envelope. If the page is public, link to it; if private, distill it into a benchmark page under `test-suite/challenges/`. See [CONTRIBUTING.md](../CONTRIBUTING.md).
|
||||
Reference in New Issue
Block a user