my-pi/pi-chrome/docs/COMPARISON.md

pi-chrome vs. the rest of the browser-automation landscape

This is the honest "which tool when" page. The browser-automation space has three different layers and people often compare across them — let's not do that.

We benchmark in public — see ../test-suite/. Where exact scores matter (WebVoyager, WorkArena++, BrowseComp, Mind2Web 2), check the live leaderboards; they shift monthly.

---

TL;DR

You are…	Use…
A Pi agent operator who wants the agent to use your real Chrome (logged-in tabs, cookies, extensions)	pi-chrome
Building a Pi/LLM agent and want low-level browser primitives	pi-chrome
Writing deterministic end-to-end tests in CI	Playwright / Cypress
Building a hosted scraping/agent fleet on isolated profiles	Playwright / Puppeteer + Browserbase / Steel
Want a turnkey "natural-language agent" with built-in loop	Browser Use / Stagehand / Skyvern
Want a hosted, vendor-managed agent	OpenAI Operator / Project Mariner / Surfer
Debugging your own app from your editor without leaving your real session	pi-chrome

pi-chrome is primitives, not an opinionated agent loop. Think of it as "Playwright for the Chrome you're already signed into" — and pluggable under any agent framework above the line.

---

The three axes you should compare on

Axis	Examples	What it gives you
1. Driver / transport	Playwright, Puppeteer, Selenium, CDP raw, puppeteer-stealth, pi-chrome	low-level tools (click, type, navigate)
2. Agent framework	Browser Use, Stagehand, Skyvern, Magnitude, Alumnium, LangGraph-with-Playwright, Operator, Mariner, Surfer	LLM loop, planning, NL API
3. Cloud browser provider	Browserbase, Steel.dev, Hyperbrowser, Anchor, Browserless	managed Chromes, sessions, quotas

pi-chrome is axis 1. You can wrap it with any axis-2 agent framework, or run it directly from a Pi agent's chrome_* tool calls. It explicitly does NOT compete with axis 3 — it runs locally inside your Chrome.

---

Axis 1 — drivers (where pi-chrome lives)

Tool	Transport	Profile	Browser input	Banner when controlling	Default detectable as bot
Playwright	CDP (own driver)	throwaway by default	always	always ("controlled by test software")	yes (webdriver flag, automation flags)
Puppeteer	CDP	throwaway by default	always	always	yes
Selenium	WebDriver / BiDi	throwaway	partial (BiDi improves)	always	most detectable
puppeteer-stealth / playwright-extra	CDP + patches	throwaway	always	always	medium (patches flags)
Raw CDP	direct devtools protocol	either (needs --remote-debugging-port)	always	always	yes
pi-chrome	Chrome extension bridge → local loopback	your real Chrome profile, signed-in cookies, extensions, history	always for input tools	while Chrome input is attached	low (real profile + Chrome input)¹

¹ pi-chrome uses chrome.debugger for browser input and shows Chrome's banner like other CDP-based tools. The test-suite/ grades browser-control behavior against common detection signals.

What makes pi-chrome different on this axis

1. Profile attach, not driver launch. Every other driver fights cookie persistence, login walls, MFA, and extension state. pi-chrome inherits all of it because it is your Chrome.
2. Chrome input against your real profile. Interactive tools use CDP input for reliability while still controlling the Chrome profile you already use.
3. Extension bridge transport. No --remote-debugging-port, no throwaway Chromium. Survives Chrome auto-updates. Works alongside your normal Chrome usage.
4. Structured action results. Input tools return target coordinates/tags and can include a fresh snapshot (includeSnapshot) so agents can verify state instead of blindly retrying.
5. Multi-session shared bridge. Planner + worker + audit Pi sessions all drive the same Chrome concurrently.
6. Stable element uids. chrome_snapshot returns deterministic uids you can pass to subsequent actions — similar to BrowserGym's bid, but built into the snapshot tool itself.

---

Axis 2 — agent frameworks (built on top of axis 1)

These wrap a driver with an LLM loop. They are higher-level than pi-chrome and complementary, not competitors.

Framework	Driver underneath	Approach	Open source
Browser Use	Playwright	DOM + a11y tree → LLM → action JSON. Open-source leader; widely cited on WebVoyager.	MIT (Python)
Stagehand (Browserbase)	Playwright	Natural-language .act() / .observe() / .extract(); deterministic + AI mix.	MIT (TypeScript)
Skyvern	Playwright + own DOM model	Vision-first + DOM; YAML workflows for form/workflow automation.	AGPL (Python)
Magnitude	Playwright	NL test authoring; QA-focused.	open
Alumnium	Selenium / Playwright	Test-author NL → agent. QA-focused.	open
LangGraph / AutoGen + Playwright	Playwright	Generic agent graph + browser tools.	open
OpenAI Operator	proprietary	OpenAI's own VLM + browser; ChatGPT-integrated.	closed, hosted
Project Mariner (Google)	proprietary Chrome integration	Google's own VLM Chrome experiment.	closed
Surfer 2 / Surfer-H (H Company)	proprietary	Hosted proprietary agent stack.	closed, hosted
Anthropic Computer Use	OS-level screenshots + mouse/keyboard	Broader than browser; OS-level events.	closed (API)

Why pi-chrome is not on this list: it's intentionally not an agent. There's no LLM loop, no .act("click the blue button"). Pi handles the loop; pi-chrome provides the primitives. This means:

• You can use pi-chrome under Browser Use, Stagehand, LangGraph, or any other agent framework (see Interop).
• You don't pay for an opinion you don't want.
• Agent improvements compound across all your workflows because the primitives stay stable.

---

Axis 3 — cloud browser providers (orthogonal)

Provider	What it sells
Browserbase	Managed browsers; pairs with Stagehand.
Steel.dev	Managed browsers + public agent leaderboards.
Hyperbrowser	Managed browsers, session APIs.
Anchor Browser	Managed browsers.
Browserless	Managed Chrome, scraping focus.

pi-chrome doesn't compete here. It runs locally in your Chrome. The right framing if someone asks: "no cloud cost, no session-handoff, no rate limits — and the agent runs against your real logged-in state." If you need fleets of isolated Chromes in CI, you want one of these.

---

Interop

pi-chrome exposes tools that any Pi agent can call. If you want to use it from outside Pi:

1. The local bridge speaks HTTP JSON over 127.0.0.1:17318 (default). The API is internal; use the Pi tool surface unless you are building an adapter.
2. Tool surface mirrors Playwright closely (click/type/navigate/snapshot/screenshot/evaluate/wait_for) so adapter code is short.
3. includeSnapshot on input tools lets agent harnesses verify state after actions.

If you want a first-class pi-chrome adapter for Browser Use / Stagehand / LangGraph, file an issue with your use case.

---

"But Playwright has storageState / Puppeteer has user-data-dir"

Yes — you can export cookies and replay them, or point Playwright at your existing profile directory. In practice for agent workflows that breaks down fast:

1. OAuth + SSO providers (Okta, Google, GitHub) often pin sessions to TLS fingerprints, device IDs, and browser-extension state that doesn't survive replay or a parallel Chrome instance.
2. MFA tokens expire mid-run.
3. Internal admin tools hard-pin to your real device.
4. Pointing Playwright at your real user-data-dir requires closing your normal Chrome (Chrome won't share the profile lock). pi-chrome doesn't fight you for the profile because it lives inside it.
5. Watching the agent work in your real window is a different UX than a hidden parallel Chrome. Demos, pair-driving, and confidence-building all want axis-1-with-attach.

---

"Is this safer than CDP?"

Different security boundary, not strictly safer.

• CDP-based tools require chrome --remote-debugging-port=.... That port is unauthenticated and exposes the whole browser to any local process. Easy to misconfigure.
• pi-chrome runs through an extension you install yourself with broad permissions (tabs, scripting, debugger, webNavigation). The bridge listens on 127.0.0.1:17318 loopback only, rejects browser-origin command requests, and keeps chrome_* tools locked until /chrome authorize is run in the current Pi session. Only install the bundled extension if you trust the source you got the npm package from.

If your threat model excludes extensions with broad permissions, neither approach is a fit — you want a sandboxed CI runner.

---

Public benchmarks worth knowing (for axis 2 / axis 3 comparison)

Pi-chrome itself ships a benchmark suite (../test-suite/) of 42 primitive challenges plus 4 hermetic BrowserGym-style long-horizon tasks covering trusted input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, iframes, file uploads, strict-CSP screenshot fallback and CDP eval/snapshot bypass, dynamic waits, tab lifecycle, network observability, fingerprint leaks, and agent-safety honeypots. Scoring tracks expected outcomes per challenge instead of raw PASS count, with core, conditional, and quality gate buckets. That's driver-level grading.

For agent-level comparison (axis 2), the public benchmarks worth citing:

Benchmark	What it measures	Notes
WebArena (CMU)	Hermetic, programmatic graders	Gold standard for reproducibility.
WorkArena++ (ServiceNow)	Enterprise SaaS workflows	Hardest realistic benchmark; <5% frontier.
BrowseComp (OpenAI)	Hard info-retrieval	Not saturated.
Mind2Web 2 (NeurIPS '25)	Long-horizon, rubric-tree judge	New, well-designed.
WebChoreArena	Tedious cross-page workflows	Reflects real ops work.
WebVoyager	Live web tasks	Saturated — 90%+ scores common; cite only as smoke test.
VisualWebArena	Multimodal
MiniWoB++	Classic unit-task suite
BrowserGym + AgentLab	Research harness covering the above	The de-facto research API; pi-chrome's snapshot uid is comparable to BrowserGym's bid.

Cite live leaderboards rather than hard-coded numbers; agent scores shift monthly.

---

Reproducing pi-chrome's driver-level claims

Run ../test-suite/ against any browser-control tool. Each challenge exposes window.__verdict / window.__reason / window.__events, so any tool (Playwright, Puppeteer, Selenium, Stagehand, pi-chrome) can grade itself deterministically. Headline release scoring should use the core gate; conditional depends on declared environment capabilities, and quality tracks adversarial/humanization regressions.

cd test-suite && python3 -m http.server 8765
# open http://127.0.0.1:8765/ in the Chrome window the tool controls

If you build a competing tool, please open a PR with your scores.