feat(pi-overflow): 新增统一的新工具结果展示截断扩展

此前截断分散在多层且互相冲突:Pi bash 保留尾部 50KB,RTK 兜底截断又只留
开头 12000 字符(砍掉报错与完整输出路径),pi-chrome 截开头 30K,web_fetch
各 provider 不一致。新扩展统一负责"新结果给模型看多少":
- capture.ts 最先加载暂存原文,index.ts 最后加载(system-prompt 之前)
- 超过 24K 字符时原文落盘 <agentDir>/my-pi/pi-overflow/(700/600、原子写),
  优先取 bash fullOutputPath 完整文件;模型看到 头 + 中段信号行 + 尾 的有界
  视图,行号基于落盘原文、结尾永远可见
- overflow_query:按 id 正则搜索、行区间、jsonPath 只读查询,id 校验防穿越
- 收编:RTK truncate 默认关闭、删除 pi-chrome truncateText、web_fetch 默认
  每页 20000(上限 100000)、web_search max_results 限定 1–20
- pi-compaction 遮蔽带 details.overflow.id 的结果时在占位符提示 overflow_query
- 登记 tools.jsonc / routing / 权限 allow / tool-search 常驻与分组

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
云服务部-叶林立
2026-10-21 20:25:44 -07:00
co-authored by Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
parent a6d8416c76
commit 2a6bf8c929
44 changed files with 1882 additions and 34 deletions
+2 -1
View File
@@ -19,6 +19,7 @@
| `pi-minimal-footer/` | 紧凑 footer:上下文仪表、订阅用量条、扩展状态 | | `pi-minimal-footer/` | 紧凑 footer:上下文仪表、订阅用量条、扩展状态 |
| `pi-notify/` | Kitty OSC 99 完成与权限确认通知,精确聚焦来源窗口 | | `pi-notify/` | Kitty OSC 99 完成与权限确认通知,精确聚焦来源窗口 |
| `pi-compaction/` | 压力触发的观察遮蔽(Pi context_edit)、阈值整段摘要、effective token、native compaction | | `pi-compaction/` | 压力触发的观察遮蔽(Pi context_edit)、阈值整段摘要、effective token、native compaction |
| `pi-overflow/` | 新工具结果统一截断:超阈值原文落盘 + 有界视图 + `overflow_query` |
| `extensions/fff-override.ts` | FFF `override` 模式接管 `find`/`grep`/`multi_grep` 与 `@` 补全 | | `extensions/fff-override.ts` | FFF `override` 模式接管 `find`/`grep`/`multi_grep` 与 `@` 补全 |
| `extensions/codegraph.ts` | 原生 `codegraph_explore`,自有 stdio JSON-RPC 直连 `codegraph serve`(`codegraph-client.ts`) | | `extensions/codegraph.ts` | 原生 `codegraph_explore`,自有 stdio JSON-RPC 直连 `codegraph serve`(`codegraph-client.ts`) |
| `extensions/context-mode.ts` | 经 `prompt-guard.ts` 加载 context-mode adapter 与 skills(`context-mode-filter.ts` 过滤冲突锚点) | | `extensions/context-mode.ts` | 经 `prompt-guard.ts` 加载 context-mode adapter 与 skills(`context-mode-filter.ts` 过滤冲突锚点) |
@@ -35,7 +36,7 @@
## 全局硬规则 ## 全局硬规则
- 系统提示词单一所有权:`extensions/system-prompt.ts` 排在 `pi.extensions` 最后,是唯一允许在 `before_agent_start` 返回 `systemPrompt` 的扩展;其他扩展只能经 `systemPromptOptions.sections.<name>` 贡献分区,第三方扩展必须经 `extensions/` wrapper + `prompt-guard.ts` 加载,新增工具必须在 `config/prompt/tools.jsonc` 登记。宿主传入 `customPrompt`(子 Agent / SYSTEM.md)时不覆盖其开场白;子 Agent 有独立提示词画像 `config/prompt/subagent/`(preamble + 内置 Agent),判定规则与路由 scope/`subagentExclude` 细则见 `extensions/AGENTS.md`。 - 系统提示词单一所有权:`extensions/system-prompt.ts` 排在 `pi.extensions` 最后,是唯一允许在 `before_agent_start` 返回 `systemPrompt` 的扩展;其他扩展只能经 `systemPromptOptions.sections.<name>` 贡献分区,第三方扩展必须经 `extensions/` wrapper + `prompt-guard.ts` 加载,新增工具必须在 `config/prompt/tools.jsonc` 登记。宿主传入 `customPrompt`(子 Agent / SYSTEM.md)时不覆盖其开场白;子 Agent 有独立提示词画像 `config/prompt/subagent/`(preamble + 内置 Agent),判定规则与路由 scope/`subagentExclude` 细则见 `extensions/AGENTS.md`。
- 加载顺序:`permission-system` → `subagents` → `pi-ssh` → `pi-chrome` → `pi-tool-search` → `system-prompt`(最后);`pi-hermes-memory` 必须排在 `pi-compaction` 之前(`session_before_compact` 取消语义),`pi-compaction` 在 `pi-minimal-footer` 与 `context-view` 之前。 - 加载顺序:`pi-overflow/capture.ts` 必须排在最前(先于 RTK 等结果改写扩展暂存原文),`permission-system` → `subagents` → `pi-ssh` → `pi-chrome` → `pi-tool-search` → `pi-overflow/index.ts` → `system-prompt`(最后);`pi-hermes-memory` 必须排在 `pi-compaction` 之前(`session_before_compact` 取消语义),`pi-compaction` 在 `pi-minimal-footer` 与 `context-view` 之前。
- 依赖:外部依赖在根 `package.json` 用精确版本,自维护扩展一律 `file:` 本地依赖并更新锁文件;Pi 核心包只作 optional host peer,根 runtime deps 不含 Pi runtime,任何目录不得再装一套 Pi runtime;各子包用于测试的 `@earendil-works/*` devDependencies 一律写 `"latest"`,经 `scripts/update-pi-devdeps.sh` 统一刷新;原生构建依赖只按锁定版本进根 `allowScripts`(当前仅 `better-sqlite3@12.11.1`;pi-hermes-memory 的 `better-sqlite3@13` 用包内 N-API 预编译,不新增放行)。不要用仓库级 `.pi/settings.json` 代替组合包依赖。根 `files` 对子包按运行时子路径白名单列出(源码、package.json、README/LICENSE/CHANGELOG/UPSTREAM),不得整目录列入,以免打入嵌套 node_modules、dist 和测试;由 `tests/package-files.test.ts` 守护。 - 依赖:外部依赖在根 `package.json` 用精确版本,自维护扩展一律 `file:` 本地依赖并更新锁文件;Pi 核心包只作 optional host peer,根 runtime deps 不含 Pi runtime,任何目录不得再装一套 Pi runtime;各子包用于测试的 `@earendil-works/*` devDependencies 一律写 `"latest"`,经 `scripts/update-pi-devdeps.sh` 统一刷新;原生构建依赖只按锁定版本进根 `allowScripts`(当前仅 `better-sqlite3@12.11.1`;pi-hermes-memory 的 `better-sqlite3@13` 用包内 N-API 预编译,不新增放行)。不要用仓库级 `.pi/settings.json` 代替组合包依赖。根 `files` 对子包按运行时子路径白名单列出(源码、package.json、README/LICENSE/CHANGELOG/UPSTREAM),不得整目录列入,以免打入嵌套 node_modules、dist 和测试;由 `tests/package-files.test.ts` 守护。
- 配置:`config/` 下全部文件是权威源、必须提交;不得读写 `settings.json` 自定义段、agent 目录独立配置文件或以环境变量作持久默认值;`config/` 只放机器无关值,路径类默认值在代码中运行时计算。`<agentDir>/my-pi/` 只放可丢弃运行时状态(日志、缓存),连同旧 `extensions/pi-permission-system/` 残留、构建产物与依赖目录都不提交;用户数据(记忆、会话、`auth.json` 等)不属于组合包配置,也不得放入该目录。 - 配置:`config/` 下全部文件是权威源、必须提交;不得读写 `settings.json` 自定义段、agent 目录独立配置文件或以环境变量作持久默认值;`config/` 只放机器无关值,路径类默认值在代码中运行时计算。`<agentDir>/my-pi/` 只放可丢弃运行时状态(日志、缓存),连同旧 `extensions/pi-permission-system/` 残留、构建产物与依赖目录都不提交;用户数据(记忆、会话、`auth.json` 等)不属于组合包配置,也不得放入该目录。
- 组合包不加载任何 MCP adapter/SDK:Exa 与 CodeGraph 均用自有直连客户端。 - 组合包不加载任何 MCP adapter/SDK:Exa 与 CodeGraph 均用自有直连客户端。
+9
View File
@@ -13,6 +13,7 @@
- 本仓库维护的 `pi-minimal-footer`:基于 `@ogulcancelik/pi-minimal-footer@0.1.10`,用紧凑的上下文仪表和订阅用量条替换默认 footer。 - 本仓库维护的 `pi-minimal-footer`:基于 `@ogulcancelik/pi-minimal-footer@0.1.10`,用紧凑的上下文仪表和订阅用量条替换默认 footer。
- 本仓库维护的 `pi-notify`:基于 `@smoose/pi-notify@0.1.1`,使用 Kitty OSC 99 在 Pi 完全 settled 或权限系统即将请求人工确认时通知,并支持点击精确聚焦来源 Kitty 窗口。 - 本仓库维护的 `pi-notify`:基于 `@smoose/pi-notify@0.1.1`,使用 Kitty OSC 99 在 Pi 完全 settled 或权限系统即将请求人工确认时通知,并支持点击精确聚焦来源 Kitty 窗口。
- 本仓库维护的 `pi-compaction`:压力触发时以 Pi `context_edit` 批量遮蔽旧工具结果、空闲时按阈值触发整段摘要、effective token 计量、`context_tree_query`、portable checkpoint 与 Codex native compaction。 - 本仓库维护的 `pi-compaction`:压力触发时以 Pi `context_edit` 批量遮蔽旧工具结果、空闲时按阈值触发整段摘要、effective token 计量、`context_tree_query`、portable checkpoint 与 Codex native compaction。
- 本仓库自维护的 `pi-overflow`:统一的新工具结果展示截断——超阈值原文落盘、向模型展示「头 + 信号行 + 尾」的有界视图,并提供 `overflow_query` 按 id 搜索、行区间或 JSON 路径取回原文。
- 本仓库维护的 `pi-ask-user`:提供模型主动调用的 `ask_user_question`;TUI 中保留单题、多题、选择、文本、自定义回答与提交前复核,Paseo 等兼容 RPC 客户端通过标准 Pi extension UI 逐题交互。 - 本仓库维护的 `pi-ask-user`:提供模型主动调用的 `ask_user_question`;TUI 中保留单题、多题、选择、文本、自定义回答与提交前复核,Paseo 等兼容 RPC 客户端通过标准 Pi extension UI 逐题交互。
- 本仓库自维护的 `pi-worktree`:纯斜杠命令(`/wt` / `/worktree`)git worktree 管理,`/wt new` 一键拆分、`/wt merge` 一键合并回来源分支(默认 ff-only,可选 `--squash`),通过 `SessionManager.forkFrom` + `switchSession` 切换会话 cwd;登记状态在 `<agentDir>/pi-worktree/state.json`,不注册模型工具、不 push。 - 本仓库自维护的 `pi-worktree`:纯斜杠命令(`/wt` / `/worktree`)git worktree 管理,`/wt new` 一键拆分、`/wt merge` 一键合并回来源分支(默认 ff-only,可选 `--squash`),通过 `SessionManager.forkFrom` + `switchSession` 切换会话 cwd;登记状态在 `<agentDir>/pi-worktree/state.json`,不注册模型工具、不 push。
- 本仓库维护的 `pi-tool-search`:从完整工具定义生成并缓存经过校验的工作流分组,默认以最多 5 个动态组、28 个动态工具的 LRU 策略加载原始完整 schema,并支持原子批量激活。 - 本仓库维护的 `pi-tool-search`:从完整工具定义生成并缓存经过校验的工作流分组,默认以最多 5 个动态组、28 个动态工具的 LRU 策略加载原始完整 schema,并支持原子批量激活。
@@ -386,3 +387,11 @@ compaction 和排队 follow-up 完成前不会提前通知;同时监听 `pi-pe
原生远程压缩同时支持 `openai-codex` + `openai-codex-responses` 和 `sub-api` + `openai-responses`,使用触发时当前会话模型。SubAPI 保留现有模型配置,从 `baseUrl` 调用公共 `/responses` 并使用 `compaction_trigger` 流式协议;优先保留非空显式 `Authorization`,否则使用 API key,也支持仅凭非空 `cf-aig-authorization` 的代理认证。缺失或空白凭据在请求前拒绝。SubAPI 不要求客户端解析 ChatGPT account ID,也不调用独立 `/responses/compact`;Codex 的 JWT 要求保持不变。网关部署须支持 Sub2API 原生压缩 v2;请求失败沿用 Pi 回退,切换模型或 Provider 时使用 portable summary。 原生远程压缩同时支持 `openai-codex` + `openai-codex-responses` 和 `sub-api` + `openai-responses`,使用触发时当前会话模型。SubAPI 保留现有模型配置,从 `baseUrl` 调用公共 `/responses` 并使用 `compaction_trigger` 流式协议;优先保留非空显式 `Authorization`,否则使用 API key,也支持仅凭非空 `cf-aig-authorization` 的代理认证。缺失或空白凭据在请求前拒绝。SubAPI 不要求客户端解析 ChatGPT account ID,也不调用独立 `/responses/compact`;Codex 的 JWT 要求保持不变。网关部署须支持 Sub2API 原生压缩 v2;请求失败沿用 Pi 回退,切换模型或 Provider 时使用 portable summary。
本实现不读取或迁移旧 `contextPrune`、`context-prune-*` 或旧 Codex native checkpoint;切换后应创建新 Session。它与 Context Mode 职责互补:Context Mode 防止大型原始内容进入会话,Pi Compaction 管理已经进入会话的长期历史。 本实现不读取或迁移旧 `contextPrune`、`context-prune-*` 或旧 Codex native checkpoint;切换后应创建新 Session。它与 Context Mode 职责互补:Context Mode 防止大型原始内容进入会话,Pi Compaction 管理已经进入会话的长期历史。
### Pi Overflow
组合包自维护的 `pi-overflow/` 统一负责「每个新工具结果给模型看多少」。它有两个入口:`capture.ts` 排在 `pi.extensions` 最前,只把原始结果文本暂存到进程内 Map(容量 64),绝不修改结果;`index.ts` 排在 `system-prompt.ts` 之前、所有其他扩展之后,此时 RTK 等 handler 已经完成变换——它用最终可见文本判断是否超阈值,落盘的原文取 bash 的 `details.fullOutputPath` 文件、暂存原文或当前文本三者中的最完整版本,视图与其中的行号则一律基于落盘原文生成,保证与 `overflow_query` 读取的内容一致。
当最终文本仍超过 `thresholdChars`(默认 24000)时,模型看到的是确定性的「头部 40 行 + 中段匹配 error/warn 等信号行(至多 40 条)+ 尾部 120 行」视图,并写入 `details.overflow.id`;当上游已把结果压短但原文超阈值时,只追加一行 `[pi-overflow id=…]` 恢复标记。原文以 `sha1(sessionId+toolCallId)` 前 12 位 hex 为 id 存入 `<agentDir>/my-pi/pi-overflow/`(目录 `700`、文件 `600`、原子写入),session 启动时按 `retentionDays`/`maxStoreBytes` 清理。模型可用 `overflow_query` 按 id 正则搜索(带上下文合并)、按行区间或 `jsonPath` 读取;`read`/`edit`/`write`/`overflow_query`/`ask_user_question` 与嵌套调用永远豁免。配置为组合包内 `config/pi-overflow.json`,详见 [`pi-overflow/README.md`](pi-overflow/README.md)。
RTK 的兜底截断与 pi-chrome 的 30K 字符截断已收编进本扩展:`config/rtk-optimizer.json` 中 `outputCompaction.truncate.enabled` 默认关闭,chrome_* 工具返回完整文本;pi-web-search 的 `web_fetch` 默认 `max_characters` 提高到 20000(上限 100000),`web_search` 的 `max_results` 约束在 1–20。
+13
View File
@@ -0,0 +1,13 @@
{
"enabled": true,
"thresholdChars": 24000,
"headLines": 40,
"tailLines": 120,
"maxSignalLines": 40,
"signalPattern": "\\b(error|errors|fail|failed|failure|warn|warning|exception|traceback|panic|fatal)\\b",
"maxLineChars": 500,
"exemptTools": ["read", "edit", "write", "overflow_query", "ask_user_question"],
"retentionDays": 7,
"maxStoreBytes": 1073741824,
"maxEntryBytes": 67108864
}
+3
View File
@@ -86,6 +86,9 @@
// index and never touches the workspace. It is pinned in Tool Search's // index and never touches the workspace. It is pinned in Tool Search's
// alwaysEnabled set, so it must not pay an AutoReview round trip per turn. // alwaysEnabled set, so it must not pay an AutoReview round trip per turn.
"context_tree_query": "allow", "context_tree_query": "allow",
// pi-overflow's inspector reads only the extension's own stored output
// directory (id-validated); pinned in Tool Search's alwaysEnabled set.
"overflow_query": "allow",
// Model-chosen URLs can exfiltrate data — same reason curl/wget ask. // Model-chosen URLs can exfiltrate data — same reason curl/wget ask.
"web_fetch": "ask", "web_fetch": "ask",
+4
View File
@@ -65,6 +65,10 @@
"whenAll": ["chrome_snapshot", "ctx_execute_file"], "whenAll": ["chrome_snapshot", "ctx_execute_file"],
"text": "After chrome_snapshot or a Chrome action with includeSnapshot=true, immediately analyze the returned workspace-relative capture path with ctx_execute_file; never use read or cat for the raw JSON. Print only the uids, text, or state needed for the next action, and take a fresh snapshot after any interaction because the previous capture may be stale." "text": "After chrome_snapshot or a Chrome action with includeSnapshot=true, immediately analyze the returned workspace-relative capture path with ctx_execute_file; never use read or cat for the raw JSON. Print only the uids, text, or state needed for the next action, and take a fresh snapshot after any interaction because the previous capture may be stale."
}, },
{
"whenAny": ["overflow_query"],
"text": "When a tool result starts with or ends with a [pi-overflow id=…] marker, the full original output is stored; inspect it with overflow_query (search, line ranges, or jsonPath) instead of re-running the command or dumping it again."
},
{ {
"whenAny": ["read"], "whenAny": ["read"],
"text": "Use read only when exact source is needed or the file is small. Before editing, request the smallest useful offset/limit range; do not read a complete large file merely to copy, compare, hash, count, explore, or summarize it." "text": "Use read only when exact source is needed or the file is small. Before editing, request the smallest useful offset/limit range; do not read a complete large file merely to copy, compare, hash, count, explore, or summarize it."
+1
View File
@@ -62,6 +62,7 @@
"memory_remove": { "inherit": true }, "memory_remove": { "inherit": true },
"skill_manage": { "inherit": true }, "skill_manage": { "inherit": true },
"context_tree_query": { "inherit": true }, "context_tree_query": { "inherit": true },
"overflow_query": { "inherit": true },
"ask_user_question": { "inherit": true }, "ask_user_question": { "inherit": true },
+1 -1
View File
@@ -11,7 +11,7 @@
"enabled": false "enabled": false
}, },
"truncate": { "truncate": {
"enabled": true, "enabled": false,
"maxChars": 12000 "maxChars": 12000
}, },
"sourceCodeFilteringEnabled": false, "sourceCodeFilteringEnabled": false,
+2 -1
View File
@@ -11,7 +11,8 @@
"memory_replace", "memory_replace",
"memory_remove", "memory_remove",
"skill_manage", "skill_manage",
"context_tree_query" "context_tree_query",
"overflow_query"
], ],
"showToolSearchFooterStatus": false, "showToolSearchFooterStatus": false,
"maxActiveGroups": 5, "maxActiveGroups": 5,
+28
View File
@@ -21,6 +21,7 @@
"pi-hashline-edit": "0.8.3", "pi-hashline-edit": "0.8.3",
"pi-hermes-memory": "file:./pi-hermes-memory", "pi-hermes-memory": "file:./pi-hermes-memory",
"pi-lsp": "0.1.7", "pi-lsp": "0.1.7",
"pi-overflow": "file:./pi-overflow",
"pi-ssh": "file:./pi-ssh", "pi-ssh": "file:./pi-ssh",
"pi-tool-search": "file:./pi-tool-search", "pi-tool-search": "file:./pi-tool-search",
"pi-web-search": "file:./pi-web-search", "pi-web-search": "file:./pi-web-search",
@@ -5571,6 +5572,10 @@
"typebox": "*" "typebox": "*"
} }
}, },
"node_modules/pi-overflow": {
"resolved": "pi-overflow",
"link": true
},
"node_modules/pi-ssh": { "node_modules/pi-ssh": {
"resolved": "pi-ssh", "resolved": "pi-ssh",
"link": true "link": true
@@ -7093,6 +7098,29 @@
"@earendil-works/pi-coding-agent": "*" "@earendil-works/pi-coding-agent": "*"
} }
}, },
"pi-overflow": {
"version": "0.1.0",
"license": "MIT",
"devDependencies": {
"@earendil-works/pi-coding-agent": "latest",
"typebox": "latest"
},
"engines": {
"node": ">=22.19.0"
},
"peerDependencies": {
"@earendil-works/pi-coding-agent": "*",
"typebox": "*"
},
"peerDependenciesMeta": {
"@earendil-works/pi-coding-agent": {
"optional": true
},
"typebox": {
"optional": true
}
}
},
"pi-permission-system": { "pi-permission-system": {
"name": "@gotgenes/pi-permission-system", "name": "@gotgenes/pi-permission-system",
"version": "26.2.1", "version": "26.2.1",
+11
View File
@@ -59,6 +59,14 @@
"pi-subagents/SECURITY.md", "pi-subagents/SECURITY.md",
"pi-subagents/LICENSE", "pi-subagents/LICENSE",
"pi-subagents/UPSTREAM.md", "pi-subagents/UPSTREAM.md",
"pi-overflow/index.ts",
"pi-overflow/capture.ts",
"pi-overflow/settings.ts",
"pi-overflow/src",
"pi-overflow/package.json",
"pi-overflow/README.md",
"pi-overflow/LICENSE",
"pi-overflow/UPSTREAM.md",
"pi-notify/src", "pi-notify/src",
"pi-notify/package.json", "pi-notify/package.json",
"pi-notify/README.md", "pi-notify/README.md",
@@ -111,6 +119,7 @@
], ],
"pi": { "pi": {
"extensions": [ "extensions": [
"./pi-overflow/capture.ts",
"./extensions/fff-override.ts", "./extensions/fff-override.ts",
"./pi-web-search/index.ts", "./pi-web-search/index.ts",
"./extensions/codegraph.ts", "./extensions/codegraph.ts",
@@ -130,6 +139,7 @@
"./pi-worktree/index.ts", "./pi-worktree/index.ts",
"./pi-chrome/extensions/chrome-profile-bridge/index.ts", "./pi-chrome/extensions/chrome-profile-bridge/index.ts",
"./pi-tool-search/extensions/index.ts", "./pi-tool-search/extensions/index.ts",
"./pi-overflow/index.ts",
"./extensions/system-prompt.ts" "./extensions/system-prompt.ts"
], ],
"skills": [ "skills": [
@@ -149,6 +159,7 @@
"pi-compaction": "file:./pi-compaction", "pi-compaction": "file:./pi-compaction",
"pi-hashline-edit": "0.8.3", "pi-hashline-edit": "0.8.3",
"pi-hermes-memory": "file:./pi-hermes-memory", "pi-hermes-memory": "file:./pi-hermes-memory",
"pi-overflow": "file:./pi-overflow",
"pi-lsp": "0.1.7", "pi-lsp": "0.1.7",
"pi-ssh": "file:./pi-ssh", "pi-ssh": "file:./pi-ssh",
"pi-tool-search": "file:./pi-tool-search", "pi-tool-search": "file:./pi-tool-search",
+1
View File
@@ -21,5 +21,6 @@
- click/type/fill/key 上的 `includeSnapshot=true` 会保存动作后的快照并返回其捕获路径,而不是把快照嵌入工具结果。 - click/type/fill/key 上的 `includeSnapshot=true` 会保存动作后的快照并返回其捕获路径,而不是把快照嵌入工具结果。
- 授权后的 `before_agent_start` 不再返回 `systemPrompt`:Chrome 使用提示写入 `event.systemPromptOptions.sections.chrome_profile_bridge` 结构化分区,由组合包单一提示词出口 `extensions/system-prompt.ts` 统一组装,顺序由 `config/prompt/sections.jsonc` 控制;未授权时不贡献任何提示词内容。 - 授权后的 `before_agent_start` 不再返回 `systemPrompt`:Chrome 使用提示写入 `event.systemPromptOptions.sections.chrome_profile_bridge` 结构化分区,由组合包单一提示词出口 `extensions/system-prompt.ts` 统一组装,顺序由 `config/prompt/sections.jsonc` 控制;未授权时不贡献任何提示词内容。
- 移除 `PI_CHROME_BRIDGE_HOST`/`PI_CHROME_BRIDGE_PORT` 环境变量,host/port 固定为 `127.0.0.1:17318`(Companion 扩展 `service_worker.js` 同样硬编码该地址,单侧可配只会断连且非回环 host 有安全风险)。 - 移除 `PI_CHROME_BRIDGE_HOST`/`PI_CHROME_BRIDGE_PORT` 环境变量,host/port 固定为 `127.0.0.1:17318`(Companion 扩展 `service_worker.js` 同样硬编码该地址,单侧可配只会断连且非回环 host 有安全风险)。
- 移除 `MAX_TEXT_CHARS`/`truncateText` 的 30K 字符文本截断:chrome_* 工具返回完整文本,超长结果的展示截断由 pi-overflow 统一负责;`MAX_ELEMENTS` 等条数上限保留不变。
同步上游时,请以导入 commit 为基准比较并选择性移植改动。不得整体覆盖这些本地生命周期、隐私和 Context Mode 集成。 同步上游时,请以导入 commit 为基准比较并选择性移植改动。不得整体覆盖这些本地生命周期、隐私和 Context Mode 集成。
@@ -67,14 +67,8 @@ const PI_CHROME_AUTH_KEY = "__piChromeProfileBridgeAuth__";
const DEFAULT_HOST = "127.0.0.1"; const DEFAULT_HOST = "127.0.0.1";
const DEFAULT_PORT = 17318; const DEFAULT_PORT = 17318;
const DEFAULT_TIMEOUT_MS = 30_000; const DEFAULT_TIMEOUT_MS = 30_000;
const MAX_TEXT_CHARS = 30_000;
const MAX_ELEMENTS = 80; const MAX_ELEMENTS = 80;
function truncateText(text: string, maxChars = MAX_TEXT_CHARS): string {
if (text.length <= maxChars) return text;
return `${text.slice(0, maxChars)}\n\n[truncated ${text.length - maxChars} characters]`;
}
function safeJson(value: unknown): string { function safeJson(value: unknown): string {
return JSON.stringify(value, null, 2); return JSON.stringify(value, null, 2);
} }
@@ -142,7 +136,7 @@ function formatChromeInspect(inspect: any): string {
lines.push("\n## Ancestors"); lines.push("\n## Ancestors");
for (const a of inspect.ancestors.slice(0, 6)) lines.push(`- ${a.uid} ${a.role || a.tag} ${compactLine(a.label || a.selector, 120)}`); for (const a of inspect.ancestors.slice(0, 6)) lines.push(`- ${a.uid} ${a.role || a.tag} ${compactLine(a.label || a.selector, 120)}`);
} }
return truncateText(lines.join("\n")); return lines.join("\n");
} }
function extensionRoot(): string { function extensionRoot(): string {
@@ -1512,7 +1506,7 @@ Usage rules:
: typeof value === "string" : typeof value === "string"
? value ? value
: safeJson(value) ?? "undefined"; : safeJson(value) ?? "undefined";
return { content: [{ type: "text", text: truncateText(text) }], details: { value: value as Json } }; return { content: [{ type: "text", text }], details: { value: value as Json } };
}, },
}); });
@@ -1691,7 +1685,7 @@ Usage rules:
}), }),
async execute(_id, params, signal): Promise<ToolTextResult> { async execute(_id, params, signal): Promise<ToolTextResult> {
const result = await authorizedBridgeSend("page.console.list", withBackground(params), DEFAULT_TIMEOUT_MS, signal); const result = await authorizedBridgeSend("page.console.list", withBackground(params), DEFAULT_TIMEOUT_MS, signal);
return { content: [{ type: "text", text: truncateText(safeJson(result)) }], details: { result: result as Json } }; return { content: [{ type: "text", text: safeJson(result) }], details: { result: result as Json } };
}, },
}); });
@@ -1713,7 +1707,7 @@ Usage rules:
}), }),
async execute(_id, params, signal): Promise<ToolTextResult> { async execute(_id, params, signal): Promise<ToolTextResult> {
const result = await authorizedBridgeSend("page.network.list", withBackground(params), DEFAULT_TIMEOUT_MS, signal); const result = await authorizedBridgeSend("page.network.list", withBackground(params), DEFAULT_TIMEOUT_MS, signal);
return { content: [{ type: "text", text: truncateText(safeJson(result)) }], details: { result: result as Json } }; return { content: [{ type: "text", text: safeJson(result) }], details: { result: result as Json } };
}, },
}); });
@@ -1733,7 +1727,7 @@ Usage rules:
}), }),
async execute(_id, params, signal): Promise<ToolTextResult> { async execute(_id, params, signal): Promise<ToolTextResult> {
const result = await authorizedBridgeSend("page.network.get", withBackground(params), DEFAULT_TIMEOUT_MS, signal); const result = await authorizedBridgeSend("page.network.get", withBackground(params), DEFAULT_TIMEOUT_MS, signal);
return { content: [{ type: "text", text: truncateText(safeJson(result)) }], details: { result: result as Json } }; return { content: [{ type: "text", text: safeJson(result) }], details: { result: result as Json } };
}, },
}); });
+1 -1
View File
@@ -7,7 +7,7 @@
- `pi-compaction` 负责已经进入 Session 的历史投影,与 Context Mode 的输入隔离职责互补。首次 Session hydration 允许一次完整扫描;后续普通 `context` hot path 必须复用不可变 snapshot 或只处理 delta,不得调用 summarizer、写 Session 或重新构建完整 branch。 - `pi-compaction` 负责已经进入 Session 的历史投影,与 Context Mode 的输入隔离职责互补。首次 Session hydration 允许一次完整扫描;后续普通 `context` hot path 必须复用不可变 snapshot 或只处理 delta,不得调用 summarizer、写 Session 或重新构建完整 branch。
- L2 遮蔽只用 Pi 原生 `context_edit`(`{ type: "context_edit", targetId, replacement }`)处理保护窗口外的旧 toolResult:成批提交、受 trigger/target 回差与 `minClearTokens` 约束,不逐条即时替换;不为遮蔽写 `segment-summary` 等 custom entry,原始条目、UI 与导出保持不变。 - L2 遮蔽只用 Pi 原生 `context_edit`(`{ type: "context_edit", targetId, replacement }`)处理保护窗口外的旧 toolResult:成批提交、受 trigger/target 回差与 `minClearTokens` 约束,不逐条即时替换;不为遮蔽写 `segment-summary` 等 custom entry,原始条目、UI 与导出保持不变。
- 不得改写最近 `keepRecentTurns` 轮;不得遮蔽 user/assistant 消息、`excludeTools` 中的工具、`SKILL.md` 的 `read` 结果、已遮蔽或已省略的条目、以及小于 `minEntryTokens` 的结果。 - 不得改写最近 `keepRecentTurns` 轮;不得遮蔽 user/assistant 消息、`excludeTools` 中的工具、`SKILL.md` 的 `read` 结果、已遮蔽或已省略的条目、以及小于 `minEntryTokens` 的结果。
- 遮蔽占位符必须确定性、单行、以 `MASK_PREFIX`(`"[pi-compaction] Earlier "`)开头,并在文本中指向 `context_tree_query` 恢复;`context_tree_query` 与 native checkpoint 的 portable summary 都必须有界。 - 遮蔽占位符必须确定性、单行、以 `MASK_PREFIX`(`"[pi-compaction] Earlier "`)开头,并在文本中指向 `context_tree_query` 恢复;`context_tree_query` 与 native checkpoint 的 portable summary 都必须有界。被遮蔽的 toolResult 若带 `details.overflow.id`(pi-overflow 已存原文),占位符末尾追加 `overflow_query` 提示——只读该公开字段,不得 import pi-overflow 内部实现。
- L3 整段摘要只能从空闲的 `agent_settled` 通过 `ctx.compact` 触发(`ctx.compact` 会中止当前运行,非空闲时调用会破坏运行);同一 leaf 不重复触发。阈值以绝对 token 计并受窗口比例约束(`effectiveThreshold = min(tokens, floor(ratio × contextWindow))`)。 - L3 整段摘要只能从空闲的 `agent_settled` 通过 `ctx.compact` 触发(`ctx.compact` 会中止当前运行,非空闲时调用会破坏运行);同一 leaf 不重复触发。阈值以绝对 token 计并受窗口比例约束(`effectiveThreshold = min(tokens, floor(ratio × contextWindow))`)。
- `context` 钩子不得改写消息——非 checkpoint 路径只更新 snapshot 并返回 undefined;唯一例外是 native checkpoint 路径:checkpoint 之后的条目在投影时必须应用各 target 最新的 `context_edit`(null 省略、字符串规范成 text block),因为 `sessionEntryToContextMessages` 不处理 edits。 - `context` 钩子不得改写消息——非 checkpoint 路径只更新 snapshot 并返回 undefined;唯一例外是 native checkpoint 路径:checkpoint 之后的条目在投影时必须应用各 target 最新的 `context_edit`(null 省略、字符串规范成 text block),因为 `sessionEntryToContextMessages` 不处理 edits。
- `session_before_compact` 只负责生成 portable/native 摘要,不得按自身阈值取消 Pi 的压缩(Pi 的 `contextTokens > window - reserveTokens` 是最终安全网)。 - `session_before_compact` 只负责生成 portable/native 摘要,不得按自身阈值取消 Pi 的压缩(Pi 的 `contextTokens > window - reserveTokens` 是最终安全网)。
+1
View File
@@ -79,6 +79,7 @@ Compaction entries still carry `pi-compaction` version 1 details; native checkpo
## Interfaces ## Interfaces
- `context_tree_query`: recover the original output of a tool result that pi-compaction cleared from context, by tool call ID (1–20 ids per call, bounded output). Use it when a `[pi-compaction] Earlier` placeholder lacks evidence needed for the current task. - `context_tree_query`: recover the original output of a tool result that pi-compaction cleared from context, by tool call ID (1–20 ids per call, bounded output). Use it when a `[pi-compaction] Earlier` placeholder lacks evidence needed for the current task.
- pi-overflow linkage: when the masked tool result carries `details.overflow.id` (pi-overflow stored the oversized original), the placeholder ends with `Full original stored: overflow_query {"id":"…"}`. pi-compaction only reads that public field and never imports pi-overflow internals.
- `/compaction-status`: show raw/effective tokens, reclaimed tokens, confidence, segments, masked results, and generation. - `/compaction-status`: show raw/effective tokens, reclaimed tokens, confidence, segments, masked results, and generation.
- Service `my-pi.pi-compaction.service.v1`: `getSnapshot`, `getMetrics`, and `query(sessionId, ids)` — `query` returns a placeholder record only for tool call IDs that are currently masked, so consumers such as the permission system's AutoReview see the same evidence the model does. - Service `my-pi.pi-compaction.service.v1`: `getSnapshot`, `getMetrics`, and `query(sessionId, ids)` — `query` returns a placeholder record only for tool call IDs that are currently masked, so consumers such as the permission system's AutoReview see the same evidence the model does.
+1
View File
@@ -33,3 +33,4 @@
## 本地改动记录 ## 本地改动记录
- 上下文管理重构为 L2/L3:L2 在上下文压力下经 `turn_end` 提交 Pi 原生 `context_edit`,把保护窗口外的旧 toolResult 批量遮蔽为指向 `context_tree_query` 的确定性占位符(原始条目保留、不写 custom entry);L3 只在空闲 `agent_settled` 且达到绝对 token 阈值(受窗口比例约束)时调用一次 `ctx.compact`。删除逐条确定性存根、后台语义 worker、`softRatio`/`hardRatio`/`targetRatio` 配置与 `session_before_compact` 的自行阈值取消;Pi 原生阈值压缩保留为最终安全网。 - 上下文管理重构为 L2/L3:L2 在上下文压力下经 `turn_end` 提交 Pi 原生 `context_edit`,把保护窗口外的旧 toolResult 批量遮蔽为指向 `context_tree_query` 的确定性占位符(原始条目保留、不写 custom entry);L3 只在空闲 `agent_settled` 且达到绝对 token 阈值(受窗口比例约束)时调用一次 `ctx.compact`。删除逐条确定性存根、后台语义 worker、`softRatio`/`hardRatio`/`targetRatio` 配置与 `session_before_compact` 的自行阈值取消;Pi 原生阈值压缩保留为最终安全网。
- L2 遮蔽占位符联动 pi-overflow:被遮蔽的 toolResult 若携带 `details.overflow.id`,占位符末尾追加 `overflow_query {"id":"…"}` 恢复提示。
+21 -3
View File
@@ -52,14 +52,25 @@ function json(value: unknown): string {
} }
} }
function maskPlaceholder(toolName: string, args: unknown, toolCallId: string, originalTokens: number): string { function maskPlaceholder(
toolName: string,
args: unknown,
toolCallId: string,
originalTokens: number,
overflowId?: string,
): string {
let argsText = json(args); let argsText = json(args);
if (argsText.length > MAX_PLACEHOLDER_ARGS_CHARS) { if (argsText.length > MAX_PLACEHOLDER_ARGS_CHARS) {
argsText = `${argsText.slice(0, MAX_PLACEHOLDER_ARGS_CHARS)}…`; argsText = `${argsText.slice(0, MAX_PLACEHOLDER_ARGS_CHARS)}…`;
} }
// `details.overflow.id` is pi-overflow's public contract — read the field
// only, never import its implementation.
const overflowHint = typeof overflowId === "string" && overflowId.length > 0
? ` Full original stored: overflow_query {"id":"${overflowId}"}.`
: "";
return `${MASK_PREFIX}${toolName} result cleared to save context (~${originalTokens} tokens). ` + return `${MASK_PREFIX}${toolName} result cleared to save context (~${originalTokens} tokens). ` +
`Call args: ${argsText}. Re-run the call if still needed, or recover the original with ` + `Call args: ${argsText}. Re-run the call if still needed, or recover the original with ` +
`context_tree_query {"toolCallIds":["${toolCallId}"]}.`; `context_tree_query {"toolCallIds":["${toolCallId}"]}.${overflowHint}`;
} }
function isSkillRead(toolName: string, args: unknown): boolean { function isSkillRead(toolName: string, args: unknown): boolean {
@@ -136,7 +147,14 @@ export function planMasking(params: {
if (contentText(message.content).startsWith(MASK_PREFIX)) continue; if (contentText(message.content).startsWith(MASK_PREFIX)) continue;
const originalTokens = params.estimate(message); const originalTokens = params.estimate(message);
if (originalTokens < masking.minEntryTokens) continue; if (originalTokens < masking.minEntryTokens) continue;
const placeholder = maskPlaceholder(toolName, args, message.toolCallId, originalTokens); const overflowId = (message.details as { overflow?: { id?: unknown } } | undefined)?.overflow?.id;
const placeholder = maskPlaceholder(
toolName,
args,
message.toolCallId,
originalTokens,
typeof overflowId === "string" ? overflowId : undefined,
);
const placeholderTokens = params.estimate({ ...message, content: [{ type: "text", text: placeholder }] }); const placeholderTokens = params.estimate({ ...message, content: [{ type: "text", text: placeholder }] });
clearedTokens += originalTokens - placeholderTokens; clearedTokens += originalTokens - placeholderTokens;
masked.push({ masked.push({
+21
View File
@@ -137,6 +137,27 @@ test("placeholder is deterministic, names the tool and call id, and truncates ar
assert.ok(placeholder.includes("…")); assert.ok(placeholder.includes("…"));
}); });
test("a masked result carrying details.overflow.id gets an overflow_query hint in the placeholder", () => {
const withOverflow = {
...toolResultMessage("r1", "c1", "a".repeat(4_000)),
details: { overflow: { id: "0123456789ab", totalLines: 5, totalBytes: 4000, truncated: true } },
};
const withoutOverflow = toolResultMessage("r2", "c2", "b".repeat(4_000));
const entries = [
entry("e1", assistantMessage("a1", [{ id: "c1", name: "bash" }, { id: "c2", name: "bash" }])),
entry("e2", withOverflow),
entry("e3", withoutOverflow),
...protect(),
];
const masking = { ...MASKING, minClearTokens: 1 };
const plan = planMasking({ contextEntries: entries, currentTokens: 3_500, contextWindow: 0, masking, estimate });
assert.equal(plan.masked.length, 2);
const first = plan.masked.find((m) => m.toolCallId === "c1")!.placeholder;
const second = plan.masked.find((m) => m.toolCallId === "c2")!.placeholder;
assert.ok(first.includes(` Full original stored: overflow_query {"id":"0123456789ab"}.`));
assert.ok(!second.includes("overflow_query"));
});
test("the small-window ratio caps both trigger and target", () => { test("the small-window ratio caps both trigger and target", () => {
const masking: MaskingConfig = { const masking: MaskingConfig = {
...MASKING, ...MASKING,
+18
View File
@@ -0,0 +1,18 @@
# my-pi 维护约定(pi-overflow)
- 完全自维护(无上游)的新结果展示截断扩展,负责"每个新 tool_result 给模型看多少":`capture.ts` 最先加载只暂存原文,`index.ts` 最后加载(在 `system-prompt.ts` 之前)生成有界视图并注册 `overflow_query`。
- 根 `pi.extensions` 顺序是硬约束:`./pi-overflow/capture.ts` 必须第一,`./pi-overflow/index.ts` 必须在所有其他扩展之后、`./extensions/system-prompt.ts` 之前;跨扩展暂存走 `globalThis[Symbol.for("my-pi.pi-overflow.stash.v1")]`,容量 64 条、淘汰最旧。
- `tool_result` handler 链中 capture 不得修改结果;index 改写 content 时必须原样回传 `structuredContent`、保留 image 部分、合并而非丢弃已有 `details`(如 RTK 压缩元数据、bash `fullOutputPath`)。
- `read`/`edit`/`write`/`overflow_query`/`ask_user_question` 与嵌套调用(`parentToolCallId`)永远豁免;read 靠自身 offset/limit 分页,不得破坏 hashline 锚点。
- 视图与行号一律基于刚落盘的存储原文(不是 current 文本):文件来源用流式逐行扫描收集 head/环形 tail/signals,文本来源切同构片段;视图必须确定性、不超 `thresholdChars`、结尾永远可见;落盘原文优先取 `details.fullOutputPath` 文件(超 `maxEntryBytes` 只留头尾各半),其次暂存原文,再次当前文本。
- 超预算收缩优先级为 tail > signals > head:head 先缩到 min(10, headLines),signals 逐步减半至 10 再到 0,再按约 35/65 缩 head/tail,按字符截断兜底。
- signal 扫描与 `overflow_query` 的 search 正则只匹配每行前 10,000 字符;query 输出每行按 `maxLineChars` 截断。
- 存储位于 `<agentDir>/my-pi/pi-overflow/`(目录 `700`、文件 `600`、原子写入),id 为 `sha1(sessionId + "\0" + toolCallId)` 前 12 位 hex;`overflow_query` 的 id 必须匹配 `^[0-9a-f]{12}$` 防路径穿越;`session_start` 按 `retentionDays`/`maxStoreBytes` 清理并删除超过 1 小时的 `*.tmp-*` 残留,失败不得抛出。
- 持久配置只读组合包 `config/pi-overflow.json`(`settings.ts` 逐字段校验、非法回退默认、signalPattern 编译失败回退默认且不敏感大小写);不得读写 `settings.json` 或以环境变量作持久默认。
- `overflow_query` 只读本扩展自己的存储目录;非法 id/正则/JSON/路径一律返回清楚错误文本,不得抛出。
- 与 pi-compaction 的唯一联动是被遮蔽 toolResult 的 `details.overflow.id` 公开字段——pi-compaction 只读它、不得 import 本包任何实现。
- 测试必须注入临时存储目录,不得写真实 agent 目录。
## 验证
- 在本目录内至少运行 `npm test` 与 `npm run check`;涉及加载入口、`pi.extensions` 顺序或 `files` 白名单时还要运行根 `node --test tests/extension-load.test.ts` 与 `tests/package-files.test.ts`。
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 sia
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+66
View File
@@ -0,0 +1,66 @@
# pi-overflow
Bounded display for oversized tool results. When a tool result exceeds
`thresholdChars`, pi-overflow persists the full original to disk and shows the
model a deterministic view — head lines, signal lines (error/fail/warn/…), and
tail lines — plus an `overflow_query` tool to search or read the stored
original by id.
## Two-stage load order
- `capture.ts` — **first** in `pi.extensions`. A `tool_result` handler that
never modifies anything; it stashes the untouched original text in
`globalThis[Symbol.for("my-pi.pi-overflow.stash.v1")]` (capacity 64) so the
bound stage can still reach the pre-RTK original.
- `index.ts` — **last** content extension, right before `system-prompt.ts`.
Its `tool_result` handler sees the final text after every other handler
(RTK semantic compaction included) has run, then:
- resolves the *original*: `details.fullOutputPath` file (streamed, capped
at `maxEntryBytes` head+tail) → stashed raw text → current text;
- if the current text still exceeds `thresholdChars`, replaces it with the
bounded view **built from the stored file** — line numbers and totals in
the view are exactly what `overflow_query` will read, even when RTK or
bash already cut the in-context text (file sources are scanned
line-by-line in a stream, never held in memory);
- if the current text fits but the original did not (an earlier extension
already cut it), keeps the text and appends a one-line
`[pi-overflow id=…]` recovery marker;
- preserves image parts and `structuredContent`, and merges
`details.overflow = { id, totalLines, totalBytes, truncated }` into the
existing details.
- Exempt tools (`read`, `edit`, `write`, `overflow_query`,
`ask_user_question`) and nested calls (`parentToolCallId`) are never
touched.
## Stored originals
`<agentDir>/my-pi/pi-overflow/<id>.txt` + `<id>.json` metadata, where
`id = sha1(sessionId + "\0" + toolCallId)[0:12]`. Directories are `0700`,
files `0600`, writes are atomic. `session_start` cleans entries older than
`retentionDays` and evicts oldest-first past `maxStoreBytes`.
## `overflow_query`
- `id` (required, `^[0-9a-f]{12}$` — path traversal safe)
- `search` — case-insensitive regex; hits come back with `context` lines and
adjacent hits merge; capped by `maxResults`
- `lines: { start, end }` — 1-based inclusive line range
- `jsonPath` — dot/bracket path into stored JSON (`a.b[3].c`)
- No selector → metadata + first-screen view. Output is always bounded by
`thresholdChars` and reports whether it was truncated.
## Configuration
`config/pi-overflow.json` inside the installed bundle. Every field is
validated independently and falls back to its default; an uncompilable
`signalPattern` falls back to the default pattern (case-insensitive).
## Development
```sh
npm test # node --test test/*.test.ts
npm run check # strip-types syntax check + tests
```
Tests inject a temporary store root — they never write to a real agent
directory.
+8
View File
@@ -0,0 +1,8 @@
# 上游来源
- 本项目完全自维护,无上游;未从任何外部仓库或 npm 包导入源码。
- 实现仅依赖 Pi 扩展宿主 API(`@earendil-works/pi-coding-agent`)与 `typebox`,两者均为组合包 peer 依赖。
## 本地改动记录
- 初始实现:`capture.ts`(首个加载,暂存原始 tool_result 文本)、`index.ts`(最后加载,生成有界视图 + 注册 `overflow_query`)、`src/`(视图、存储、查询、跨扩展暂存区)。
+30
View File
@@ -0,0 +1,30 @@
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
import { readOverflowSettings } from "./settings.ts";
import { getStash, stashResult } from "./src/stash.ts";
/**
* Pre-bound capture: loaded FIRST in pi.extensions, before RTK or any other
* result-rewriting extension. It only stashes the untouched original text —
* it never modifies the result. The bound stage (`index.ts`, loaded last)
* consumes the stash after all other handlers have run.
*/
export default function overflowCapture(pi: ExtensionAPI): void {
const settings = readOverflowSettings();
pi.on("tool_result", (event) => {
if (!settings.enabled) return undefined;
if (settings.exemptTools.includes(event.toolName)) return undefined;
if (event.parentToolCallId) return undefined;
const rawText = event.content
.filter((part): part is { type: "text"; text: string } => part.type === "text")
.map((part) => part.text)
.join("\n");
const fullOutputPath =
typeof event.details?.fullOutputPath === "string" ? event.details.fullOutputPath : undefined;
if (rawText.length <= settings.thresholdChars && !fullOutputPath) return undefined;
stashResult(getStash(), event.toolCallId, {
rawText,
...(fullOutputPath ? { fullOutputPath } : {}),
});
return undefined;
});
}
+167
View File
@@ -0,0 +1,167 @@
import { getAgentDir, type ExtensionAPI } from "@earendil-works/pi-coding-agent";
import { Type } from "typebox";
import { stat } from "node:fs/promises";
import { join } from "node:path";
import { readOverflowSettings, type OverflowSettings } from "./settings.ts";
import { getStash } from "./src/stash.ts";
import { buildView, collectViewFromFile, collectViewFromText } from "./src/view.ts";
import { cleanStore, entryId, storeEntry } from "./src/store.ts";
import { runOverflowQuery } from "./src/query.ts";
function formatBytes(bytes: number): string {
if (bytes < 1024) return `${bytes} B`;
if (bytes < 1_048_576) return `${(bytes / 1024).toFixed(1)} KB`;
return `${(bytes / 1_048_576).toFixed(1)} MB`;
}
export interface OverflowExtensionOptions {
settings?: OverflowSettings;
storeRoot?: string;
}
/**
* Bound stage: loaded LAST among content extensions (before system-prompt).
* By the time this handler runs, every other tool_result handler — RTK
* included — has already transformed the result, so the bounded view is
* built from the final visible text while the stored original comes from
* capture.ts's stash or bash's fullOutputPath file.
*/
export default function overflowExtension(pi: ExtensionAPI, options: OverflowExtensionOptions = {}): void {
const settings = options.settings ?? readOverflowSettings();
const storeRoot = () => options.storeRoot ?? join(getAgentDir(), "my-pi", "pi-overflow");
pi.on("session_start", () => {
// Retention/size cleanup never throws (cleanStore swallows failures)
// and the returned promise lets tests await it.
return cleanStore(storeRoot(), settings.retentionDays, settings.maxStoreBytes);
});
pi.on("tool_result", async (event, ctx) => {
const stash = getStash();
const staged = stash.get(event.toolCallId);
try {
if (
!settings.enabled ||
settings.exemptTools.includes(event.toolName) ||
event.parentToolCallId
) {
return undefined;
}
const current = event.content
.filter((part): part is { type: "text"; text: string } => part.type === "text")
.map((part) => part.text)
.join("\n");
const fullOutputPath =
typeof event.details?.fullOutputPath === "string"
? event.details.fullOutputPath
: staged?.fullOutputPath;
// Resolve the original: bash's spilled file wins (it is complete even
// when the in-context text was already cut), then the capture stash,
// then the current text itself.
let source: { kind: "text"; text: string } | { kind: "file"; path: string };
let originalSize: number;
if (fullOutputPath) {
try {
const info = await stat(fullOutputPath);
if (!info.isFile()) throw new Error("not a file");
source = { kind: "file", path: fullOutputPath };
originalSize = info.size;
} catch {
const fallback = staged?.rawText ?? current;
source = { kind: "text", text: fallback };
originalSize = fallback.length;
}
} else {
const fallback = staged?.rawText ?? current;
source = { kind: "text", text: fallback };
originalSize = fallback.length;
}
if (current.length <= settings.thresholdChars && originalSize <= settings.thresholdChars) {
return undefined;
}
const sessionId = ctx.sessionManager?.getSessionId?.() ?? "session";
const id = entryId(sessionId, event.toolCallId);
const stored = await storeEntry(
storeRoot(),
id,
source,
{ tool: event.toolName, args: event.input },
settings.maxEntryBytes,
);
const overflow = {
id,
totalLines: stored.totalLines,
totalBytes: stored.totalBytes,
truncated: current.length > settings.thresholdChars,
};
const images = event.content.filter((part) => part.type !== "text");
let text: string;
if (current.length > settings.thresholdChars) {
// The view always describes the stored file — its line numbers and
// totals must match what overflow_query reads, which is not true
// of `current` when RTK rewrote or bash pre-truncated the result.
const viewInput =
stored.storedText !== undefined
? collectViewFromText(stored.storedText, settings)
: await collectViewFromFile(join(storeRoot(), `${id}.txt`), settings);
text = buildView(viewInput, settings, { id, tool: event.toolName });
} else {
text = `${current}\n[pi-overflow id=${id}: full original output stored (${stored.totalLines} lines, ${formatBytes(stored.totalBytes)}); use overflow_query to inspect it]`;
}
return {
content: [{ type: "text", text }, ...images],
structuredContent: event.structuredContent,
details: { ...event.details, overflow },
};
} finally {
stash.delete(event.toolCallId);
}
});
pi.registerTool({
name: "overflow_query",
label: "Overflow Query",
description:
"Inspect the full original of a tool output that pi-overflow stored because it was too long to show " +
"(search with regex, read line ranges, or extract a JSON path).",
promptSnippet: "Inspect a stored oversized tool output by id: regex search, line ranges, or JSON path.",
promptGuidelines: [
"When a tool result carries a [pi-overflow id=…] marker, the model is seeing a bounded view; the complete original is stored on disk.",
"Call overflow_query with the marker's id and a regex, line range, or jsonPath instead of re-running the tool or asking for the full dump.",
],
parameters: Type.Object({
id: Type.String({ description: "12-hex storage id from a [pi-overflow id=…] marker." }),
search: Type.Optional(
Type.String({ description: "Case-insensitive regular expression matched per line." }),
),
context: Type.Optional(
Type.Number({ minimum: 0, maximum: 10, description: "Context lines around each search hit (default 2)." }),
),
lines: Type.Optional(
Type.Object({
start: Type.Number({ minimum: 1, description: "First line to return (1-based, inclusive)." }),
end: Type.Number({ minimum: 1, description: "Last line to return (inclusive)." }),
}),
),
jsonPath: Type.Optional(
Type.String({ description: "Dot/bracket path into the stored JSON, e.g. snapshot.nodes[3].name." }),
),
maxResults: Type.Optional(
Type.Number({ minimum: 1, maximum: 200, description: "Maximum search hits to report (default 50)." }),
),
}),
async execute(_toolCallId, params) {
const result = await runOverflowQuery(storeRoot(), params, settings);
return {
content: [{ type: "text", text: result.text }],
details: { error: result.error === true, truncated: result.truncated },
isError: result.error === true,
};
},
});
}
+49
View File
@@ -0,0 +1,49 @@
{
"name": "pi-overflow",
"version": "0.1.0",
"description": "Bounded views for oversized tool results: store the full original, show head/signals/tail, recover with overflow_query.",
"private": true,
"type": "module",
"license": "MIT",
"main": "./index.ts",
"exports": {
".": "./index.ts"
},
"files": [
"index.ts",
"capture.ts",
"settings.ts",
"src",
"README.md",
"LICENSE"
],
"pi": {
"extensions": [
"./capture.ts",
"./index.ts"
]
},
"scripts": {
"test": "node --test test/*.test.ts",
"check": "node --experimental-strip-types --check index.ts && node --test test/*.test.ts"
},
"peerDependencies": {
"@earendil-works/pi-coding-agent": "*",
"typebox": "*"
},
"peerDependenciesMeta": {
"@earendil-works/pi-coding-agent": {
"optional": true
},
"typebox": {
"optional": true
}
},
"devDependencies": {
"@earendil-works/pi-coding-agent": "latest",
"typebox": "latest"
},
"engines": {
"node": ">=22.19.0"
}
}
+100
View File
@@ -0,0 +1,100 @@
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
export interface OverflowSettings {
enabled: boolean;
thresholdChars: number;
headLines: number;
tailLines: number;
maxSignalLines: number;
signalPattern: string;
signalRegex: RegExp;
maxLineChars: number;
exemptTools: string[];
retentionDays: number;
maxStoreBytes: number;
maxEntryBytes: number;
}
export const DEFAULT_SIGNAL_PATTERN =
"\\b(error|errors|fail|failed|failure|warn|warning|exception|traceback|panic|fatal)\\b";
export const OVERFLOW_DEFAULTS = {
enabled: true,
thresholdChars: 24_000,
headLines: 40,
tailLines: 120,
maxSignalLines: 40,
signalPattern: DEFAULT_SIGNAL_PATTERN,
maxLineChars: 500,
exemptTools: ["read", "edit", "write", "overflow_query", "ask_user_question"],
retentionDays: 7,
maxStoreBytes: 1_073_741_824,
maxEntryBytes: 67_108_864,
} as const;
// Bundle-level `config/` is the single source of truth: the extension reads
// `config/pi-overflow.json` from inside the installed bundle (resolved
// relative to this package-root source file), never `settings.json`.
export const OVERFLOW_CONFIG_PATH = fileURLToPath(
new URL("../config/pi-overflow.json", import.meta.url),
);
function isObject(value: unknown): value is Record<string, unknown> {
return value !== null && typeof value === "object" && !Array.isArray(value);
}
function bool(value: unknown, fallback: boolean): boolean {
return typeof value === "boolean" ? value : fallback;
}
function intInRange(value: unknown, fallback: number, min: number, max: number): number {
return typeof value === "number" && Number.isInteger(value) && value >= min && value <= max
? value
: fallback;
}
function compileSignalPattern(value: unknown): { pattern: string; regex: RegExp } {
const pattern = typeof value === "string" && value.length > 0 ? value : DEFAULT_SIGNAL_PATTERN;
try {
return { pattern, regex: new RegExp(pattern, "i") };
} catch {
return { pattern: DEFAULT_SIGNAL_PATTERN, regex: new RegExp(DEFAULT_SIGNAL_PATTERN, "i") };
}
}
function stringList(value: unknown, fallback: readonly string[]): string[] {
if (!Array.isArray(value)) return [...fallback];
const list = value.filter((item): item is string => typeof item === "string" && item.length > 0);
return [...new Set(list)];
}
/**
* Read `config/pi-overflow.json`. Every field is validated independently and
* falls back to its bundle default when missing or invalid; an unreadable or
* non-object file yields all defaults. Tests inject an explicit path.
*/
export function readOverflowSettings(configPath: string = OVERFLOW_CONFIG_PATH): OverflowSettings {
let parsed: unknown = {};
try {
parsed = JSON.parse(readFileSync(configPath, "utf8"));
} catch {
parsed = {};
}
const config = isObject(parsed) ? parsed : {};
const signal = compileSignalPattern(config.signalPattern);
return {
enabled: bool(config.enabled, OVERFLOW_DEFAULTS.enabled),
thresholdChars: intInRange(config.thresholdChars, OVERFLOW_DEFAULTS.thresholdChars, 1, 10_000_000),
headLines: intInRange(config.headLines, OVERFLOW_DEFAULTS.headLines, 0, 10_000),
tailLines: intInRange(config.tailLines, OVERFLOW_DEFAULTS.tailLines, 0, 10_000),
maxSignalLines: intInRange(config.maxSignalLines, OVERFLOW_DEFAULTS.maxSignalLines, 0, 1_000),
signalPattern: signal.pattern,
signalRegex: signal.regex,
maxLineChars: intInRange(config.maxLineChars, OVERFLOW_DEFAULTS.maxLineChars, 10, 100_000),
exemptTools: stringList(config.exemptTools, OVERFLOW_DEFAULTS.exemptTools),
retentionDays: intInRange(config.retentionDays, OVERFLOW_DEFAULTS.retentionDays, 0, 3_650),
maxStoreBytes: intInRange(config.maxStoreBytes, OVERFLOW_DEFAULTS.maxStoreBytes, 1_024, 1_099_511_627_776),
maxEntryBytes: intInRange(config.maxEntryBytes, OVERFLOW_DEFAULTS.maxEntryBytes, 1_024, 1_099_511_627_776),
};
}
+188
View File
@@ -0,0 +1,188 @@
import { ID_PATTERN, readStoredEntry, type EntryMeta } from "./store.ts";
import type { OverflowSettings } from "../settings.ts";
export interface QueryParams {
id: string;
search?: string;
context?: number;
lines?: { start: number; end: number };
jsonPath?: string;
maxResults?: number;
}
export interface QueryResult {
text: string;
truncated: boolean;
error?: boolean;
}
function fail(text: string): QueryResult {
return { text, truncated: false, error: true };
}
function boundOutput(text: string, limit: number): QueryResult {
if (text.length <= limit) return { text, truncated: false };
const marker = `\n[output truncated at ${limit} chars; narrow the query (smaller line range, tighter regex, or jsonPath)]`;
const keep = Math.max(0, limit - marker.length);
return { text: text.slice(0, keep) + marker, truncated: true };
}
function metaLine(id: string, meta: EntryMeta | null): string {
if (!meta) return `[pi-overflow id=${id}]`;
return `[pi-overflow id=${id} tool=${meta.tool} ${meta.totalLines} lines / ${meta.totalBytes} bytes]`;
}
/** Regexes only inspect this prefix of each stored line — a model-supplied
* pattern must not backtrack catastrophically across a megabyte-long line. */
const MATCH_SCAN_CHARS = 10_000;
function clipLine(line: string, maxLineChars: number): string {
return line.length > maxLineChars ? `${line.slice(0, maxLineChars)}…` : line;
}
function evalJsonPath(root: unknown, path: string): { ok: true; value: unknown } | { ok: false; error: string } {
const tokens: Array<string | number> = [];
const pattern = /(?:^|\.)([A-Za-z_$][A-Za-z0-9_$]*)|\[\s*(\d+)\s*\]|\[\s*["']([^"']+)["']\s*\]/g;
let index = 0;
let match: RegExpExecArray | null;
while ((match = pattern.exec(path)) !== null) {
if (match.index !== index) return { ok: false, error: `cannot parse jsonPath near "${path.slice(index)}"` };
index = pattern.lastIndex;
if (match[1] !== undefined) tokens.push(match[1]);
else if (match[2] !== undefined) tokens.push(Number(match[2]));
else tokens.push(match[3] as string);
}
if (index !== path.length || tokens.length === 0) {
return { ok: false, error: `cannot parse jsonPath "${path}"` };
}
let value: unknown = root;
for (const token of tokens) {
if (value === null || value === undefined) {
return { ok: false, error: `jsonPath stops at null/undefined before "${String(token)}"` };
}
if (typeof token === "number") {
if (!Array.isArray(value) || token >= value.length) {
return { ok: false, error: `index [${token}] out of range` };
}
value = value[token];
} else {
if (typeof value !== "object" || Array.isArray(value)) {
return { ok: false, error: `"${token}" is not a property of this value` };
}
const record = value as Record<string, unknown>;
if (!(token in record)) return { ok: false, error: `property "${token}" not found` };
value = record[token];
}
}
return { ok: true, value };
}
function mergeRanges(hits: number[], context: number): Array<{ start: number; end: number }> {
const ranges: Array<{ start: number; end: number }> = [];
for (const hit of hits) {
const start = Math.max(1, hit - context);
const end = hit + context;
const last = ranges[ranges.length - 1];
if (last && start <= last.end + 1) {
last.end = Math.max(last.end, end);
} else {
ranges.push({ start, end });
}
}
return ranges;
}
/**
* Execute an overflow_query against the store. Never throws on bad input —
* invalid ids, regexes, JSON, and paths all come back as clear error text.
*/
export async function runOverflowQuery(
root: string,
params: QueryParams,
settings: OverflowSettings,
): Promise<QueryResult> {
if (typeof params.id !== "string" || !ID_PATTERN.test(params.id)) {
return fail(`invalid overflow id "${String(params.id)}" — expected 12 lowercase hex chars from a [pi-overflow id=…] marker`);
}
let stored: { text: string; meta: EntryMeta | null };
try {
stored = await readStoredEntry(root, params.id);
} catch {
return fail(`no stored output for id ${params.id} (it may have been cleaned up or never stored)`);
}
const lines = stored.text.split("\n");
const head = metaLine(params.id, stored.meta);
const context = Math.min(Math.max(Math.floor(params.context ?? 2), 0), 10);
const maxResults = Math.min(Math.max(Math.floor(params.maxResults ?? 50), 1), 200);
if (typeof params.search === "string") {
let regex: RegExp;
try {
regex = new RegExp(params.search, "i");
} catch (error) {
return fail(`invalid search regex: ${error instanceof Error ? error.message : String(error)}`);
}
const hits: number[] = [];
for (let index = 0; index < lines.length && hits.length < maxResults; index++) {
regex.lastIndex = 0;
if (regex.test(lines[index].slice(0, MATCH_SCAN_CHARS))) hits.push(index + 1);
}
if (hits.length === 0) {
return boundOutput(`${head}\nno lines match /${params.search}/i`, settings.thresholdChars);
}
const parts = [head, `${hits.length}${hits.length >= maxResults ? "+" : ""} matching lines:`];
for (const range of mergeRanges(hits, context)) {
const end = Math.min(range.end, lines.length);
parts.push(`--- lines ${range.start}-${end} ---`);
for (let line = range.start; line <= end; line++) {
parts.push(`L${line}: ${clipLine(lines[line - 1], settings.maxLineChars)}`);
}
}
return boundOutput(parts.join("\n"), settings.thresholdChars);
}
if (params.lines && typeof params.lines === "object") {
const start = Math.floor(params.lines.start);
const end = Math.floor(params.lines.end);
if (!Number.isFinite(start) || !Number.isFinite(end) || start < 1 || end < start) {
return fail(`invalid line range {start: ${params.lines.start}, end: ${params.lines.end}} — need 1 <= start <= end`);
}
const clippedEnd = Math.min(end, lines.length);
if (start > lines.length) {
return fail(`start ${start} is beyond the stored ${lines.length} lines`);
}
const parts = [head, `--- lines ${start}-${clippedEnd} ---`];
for (let line = start; line <= clippedEnd; line++) {
parts.push(`L${line}: ${clipLine(lines[line - 1], settings.maxLineChars)}`);
}
return boundOutput(parts.join("\n"), settings.thresholdChars);
}
if (typeof params.jsonPath === "string") {
let parsed: unknown;
try {
parsed = JSON.parse(stored.text);
} catch (error) {
return fail(`stored output is not valid JSON: ${error instanceof Error ? error.message : String(error)}`);
}
const result = evalJsonPath(parsed, params.jsonPath);
if (!result.ok) return fail(`jsonPath error: ${result.error}`);
let rendered: string;
try {
rendered = JSON.stringify(result.value, null, 2) ?? "undefined";
} catch {
rendered = String(result.value);
}
return boundOutput(`${head}\njsonPath ${params.jsonPath}:\n${rendered}`, settings.thresholdChars);
}
// No selector: metadata plus a bounded head/tail preview.
const headCount = Math.min(settings.headLines, lines.length);
const tailCount = Math.min(settings.tailLines, Math.max(0, lines.length - headCount));
const parts = [head, `--- lines 1-${headCount} ---`, ...lines.slice(0, headCount)];
if (tailCount > 0) {
parts.push(`… ${lines.length - headCount - tailCount} lines omitted …`);
parts.push(`--- lines ${lines.length - tailCount + 1}-${lines.length} ---`, ...lines.slice(lines.length - tailCount));
}
return boundOutput(parts.join("\n"), settings.thresholdChars);
}
+35
View File
@@ -0,0 +1,35 @@
/**
* Cross-extension staging for pre-bound originals. `capture.ts` (loaded
* first) stashes the raw tool_result text; `index.ts` (loaded last) consumes
* it after every other handler — RTK in particular — has run. Sharing goes
* through `globalThis[Symbol.for(...)]`, the same mechanism pi-compaction
* uses for its metrics service.
*/
export interface StashedResult {
rawText: string;
fullOutputPath?: string;
}
const STASH_KEY = Symbol.for("my-pi.pi-overflow.stash.v1");
const STASH_LIMIT = 64;
export function getStash(): Map<string, StashedResult> {
const scope = globalThis as Record<symbol, unknown>;
let stash = scope[STASH_KEY] as Map<string, StashedResult> | undefined;
if (!(stash instanceof Map)) {
stash = new Map<string, StashedResult>();
scope[STASH_KEY] = stash;
}
return stash;
}
export function stashResult(stash: Map<string, StashedResult>, toolCallId: string, entry: StashedResult): void {
stash.delete(toolCallId);
stash.set(toolCallId, entry);
while (stash.size > STASH_LIMIT) {
const oldest = stash.keys().next().value;
if (oldest === undefined) break;
stash.delete(oldest);
}
}
+267
View File
@@ -0,0 +1,267 @@
import { createHash } from "node:crypto";
import { createReadStream, createWriteStream, type WriteStream } from "node:fs";
import { chmod, mkdir, open, readFile, readdir, rename, stat, unlink, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { pipeline } from "node:stream/promises";
export interface EntryMeta {
tool: string;
argsSummary: string;
totalLines: number;
totalBytes: number;
createdAt: string;
omittedMiddle?: boolean;
}
export interface StoredEntry {
id: string;
totalLines: number;
totalBytes: number;
/** Present when the source was text: the exact content written to disk. */
storedText?: string;
}
export const ID_PATTERN = /^[0-9a-f]{12}$/;
const TMP_MAX_AGE_MS = 3_600_000;
export function entryId(sessionId: string, toolCallId: string): string {
return createHash("sha1").update(`${sessionId}\0${toolCallId}`).digest("hex").slice(0, 12);
}
async function ensureDir(dir: string): Promise<void> {
await mkdir(dir, { recursive: true, mode: 0o700 });
try {
await chmod(dir, 0o700);
} catch {
// Best-effort permission tightening only.
}
}
function tmpPath(path: string): string {
return `${path}.tmp-${process.pid}-${Math.random().toString(36).slice(2)}`;
}
async function writeFileAtomic(path: string, content: string | Buffer): Promise<void> {
const tmp = tmpPath(path);
await writeFile(tmp, content, { mode: 0o600 });
await rename(tmp, path);
}
function countNewlines(text: string): number {
let count = 0;
for (let index = 0; index < text.length; index++) {
if (text.charCodeAt(index) === 10) count++;
}
return count;
}
export function lineCount(text: string): number {
if (text.length === 0) return 0;
return countNewlines(text) + (text.endsWith("\n") ? 0 : 1);
}
function truncateSummary(value: string, maxChars = 300): string {
return value.length > maxChars ? `${value.slice(0, maxChars)}…` : value;
}
export function argsSummary(input: unknown): string {
try {
return truncateSummary(JSON.stringify(input) ?? "undefined");
} catch {
return "[unserializable]";
}
}
/**
* Copy `sourcePath` into `<root>/<id>.txt` without holding it all in memory.
* Files larger than `maxEntryBytes` keep the first and last halves with an
* omission marker between them.
*/
async function copyFileBounded(
sourcePath: string,
destPath: string,
maxEntryBytes: number,
): Promise<{ bytes: number; omittedMiddle: boolean }> {
const info = await stat(sourcePath);
if (!info.isFile()) throw new Error("full output path is not a file");
const size = info.size;
if (size <= maxEntryBytes) {
const tmp = tmpPath(destPath);
const out: WriteStream = createWriteStream(tmp, { mode: 0o600 });
await pipeline(createReadStream(sourcePath), out);
await rename(tmp, destPath);
return { bytes: size, omittedMiddle: false };
}
const half = Math.floor(maxEntryBytes / 2);
const omitted = size - half * 2;
const source = await open(sourcePath, "r");
try {
const head = Buffer.alloc(half);
const tail = Buffer.alloc(half);
await source.read(head, 0, half, 0);
await source.read(tail, 0, half, size - half);
const marker = Buffer.from(`\n… ${omitted} bytes omitted (exceeded ${maxEntryBytes}-byte store limit) …\n`);
await writeFileAtomic(destPath, Buffer.concat([head, marker, tail]));
} finally {
await source.close();
}
return { bytes: size, omittedMiddle: true };
}
async function countFileLines(path: string): Promise<number> {
let count = 0;
let lastByte = -1;
let seen = false;
for await (const chunk of createReadStream(path)) {
seen = true;
const buffer = chunk as Buffer;
for (let index = 0; index < buffer.length; index++) {
if (buffer[index] === 10) count++;
}
if (buffer.length > 0) lastByte = buffer[buffer.length - 1];
}
if (!seen) return 0;
return count + (lastByte === 10 ? 0 : 1);
}
/**
* Persist an original tool output under `<root>/<id>.txt` plus `<id>.json`
* metadata. Directories are 0700, files 0600, writes are atomic. The caller
* chooses `root` (defaults come from getAgentDir() in index.ts; tests inject
* a temp directory).
*/
export async function storeEntry(
root: string,
id: string,
source: { kind: "text"; text: string } | { kind: "file"; path: string },
meta: { tool: string; args: unknown },
maxEntryBytes: number,
): Promise<StoredEntry> {
await ensureDir(root);
const textPath = join(root, `${id}.txt`);
const metaPath = join(root, `${id}.json`);
let totalLines = 0;
let totalBytes = 0;
let omittedMiddle = false;
let storedText: string | undefined;
if (source.kind === "text") {
let text = source.text;
const bytes = Buffer.byteLength(text, "utf8");
totalBytes = bytes;
if (bytes > maxEntryBytes) {
const halfChars = Math.floor(maxEntryBytes / 2);
const omitted = bytes - halfChars * 2;
text = `${text.slice(0, halfChars)}\n… ${omitted} bytes omitted (exceeded ${maxEntryBytes}-byte store limit) …\n${text.slice(text.length - halfChars)}`;
omittedMiddle = true;
}
totalLines = lineCount(text);
storedText = text;
await writeFileAtomic(textPath, text);
} else {
const copied = await copyFileBounded(source.path, textPath, maxEntryBytes);
totalBytes = copied.bytes;
omittedMiddle = copied.omittedMiddle;
totalLines = await countFileLines(textPath).catch(() => 0);
}
const entry: EntryMeta = {
tool: meta.tool,
argsSummary: argsSummary(meta.args),
totalLines,
totalBytes,
createdAt: new Date().toISOString(),
...(omittedMiddle ? { omittedMiddle: true } : {}),
};
await writeFileAtomic(metaPath, `${JSON.stringify(entry, null, 2)}\n`);
return { id, totalLines, totalBytes, ...(storedText !== undefined ? { storedText } : {}) };
}
export async function readStoredEntry(root: string, id: string): Promise<{ text: string; meta: EntryMeta | null }> {
if (!ID_PATTERN.test(id)) throw new Error(`invalid overflow id: ${id}`);
const textPath = join(root, `${id}.txt`);
const metaPath = join(root, `${id}.json`);
const text = await readFile(textPath, "utf8");
let meta: EntryMeta | null = null;
try {
const parsed = JSON.parse(await readFile(metaPath, "utf8"));
if (parsed && typeof parsed === "object") meta = parsed as EntryMeta;
} catch {
meta = null;
}
return { text, meta };
}
/**
* Delete tmp files left by interrupted writes (older than one hour), entries
* older than `retentionDays`, then evict oldest-first until the store is
* under `maxStoreBytes`. All failures are swallowed — cleanup must never
* break a session.
*/
export async function cleanStore(root: string, retentionDays: number, maxStoreBytes: number): Promise<void> {
try {
const now = Date.now();
const cutoff = now - retentionDays * 86_400_000;
const names = await readdir(root);
// Stale tmp files from interrupted writes never become entries; remove
// them once they are clearly abandoned.
for (const name of names) {
if (!name.includes(".tmp-")) continue;
try {
const info = await stat(join(root, name));
if (info.isFile() && info.mtimeMs < now - TMP_MAX_AGE_MS) {
await unlink(join(root, name));
}
} catch {
// Already gone or unreadable — nothing to do.
}
}
const ids = new Set(
names
.filter((name) => !name.includes(".tmp-"))
.map((name) => name.replace(/\.(txt|json)$/, "")),
);
const entries: Array<{ id: string; bytes: number; mtime: number }> = [];
for (const id of ids) {
let bytes = 0;
let mtime = 0;
let seen = false;
for (const suffix of ["txt", "json"] as const) {
const file = join(root, `${id}.${suffix}`);
try {
const info = await stat(file);
bytes += info.size;
mtime = Math.max(mtime, info.mtimeMs);
seen = true;
} catch {
// Missing half of a pair still counts toward cleanup.
}
}
if (seen) entries.push({ id, bytes, mtime });
}
const remove = async (id: string) => {
for (const suffix of ["txt", "json"] as const) {
await unlink(join(root, `${id}.${suffix}`)).catch(() => {});
}
};
let total = 0;
const survivors: typeof entries = [];
for (const entry of entries) {
if (entry.mtime < cutoff) {
await remove(entry.id);
} else {
total += entry.bytes;
survivors.push(entry);
}
}
if (total > maxStoreBytes) {
survivors.sort((a, b) => a.mtime - b.mtime);
for (const entry of survivors) {
if (total <= maxStoreBytes) break;
await remove(entry.id);
total -= entry.bytes;
}
}
} catch {
// Cleanup failures never propagate.
}
}
+246
View File
@@ -0,0 +1,246 @@
import { createReadStream } from "node:fs";
import { open, stat } from "node:fs/promises";
import type { OverflowSettings } from "../settings.ts";
export interface ViewMeta {
id: string;
tool: string;
}
export interface SignalHit {
line: number;
text: string;
}
/**
* Pre-collected view segments of the STORED original (never the current —
* possibly RTK- or bash-truncated — text). Line numbers here are the stored
* file's line numbers, which is exactly what overflow_query reads.
*/
export interface ViewInput {
head: string[];
signals: SignalHit[];
tail: string[];
totalLines: number;
totalBytes: number;
/** First ~thresholdChars chars of the stored content, for char fallback. */
rawHead: string;
/** Last ~thresholdChars chars of the stored content, for char fallback. */
rawTail: string;
}
/** Regexes only inspect this prefix of each line — catastrophic backtracking
* on a giant single-line blob is not allowed to stall the handler. */
export const MATCH_SCAN_CHARS = 10_000;
function clip(line: string, maxLineChars: number): string {
return line.length > maxLineChars ? `${line.slice(0, maxLineChars)}…` : line;
}
/** Slice a text original into the same view segments a file scan produces. */
export function collectViewFromText(text: string, settings: OverflowSettings): ViewInput {
const lines = text.split("\n");
const total = lines.length;
const headCount = Math.min(settings.headLines, total);
const tailCount = Math.min(settings.tailLines, Math.max(0, total - headCount));
const head = lines.slice(0, headCount).map((line) => clip(line, settings.maxLineChars));
const tail = tailCount > 0 ? lines.slice(total - tailCount).map((line) => clip(line, settings.maxLineChars)) : [];
const signals: SignalHit[] = [];
const regex = new RegExp(settings.signalRegex.source, "i");
for (
let index = headCount;
index < total - tailCount && signals.length < settings.maxSignalLines;
index++
) {
regex.lastIndex = 0;
if (regex.test(lines[index].slice(0, MATCH_SCAN_CHARS))) {
signals.push({ line: index + 1, text: clip(lines[index], settings.maxLineChars) });
}
}
return {
head,
signals,
tail,
totalLines: total,
totalBytes: Buffer.byteLength(text, "utf8"),
rawHead: text.slice(0, settings.thresholdChars),
rawTail: text.slice(Math.max(0, text.length - settings.thresholdChars)),
};
}
/**
* Stream the stored file line by line — never materialize it. Collects the
* first headLines lines, keeps the last tailLines lines in a ring buffer,
* and records at most maxSignalLines signal hits (line number + clipped
* text); hits landing inside the final tail range are dropped afterwards.
*/
export async function collectViewFromFile(path: string, settings: OverflowSettings): Promise<ViewInput> {
const head: string[] = [];
const tail: string[] = [];
const signals: SignalHit[] = [];
let totalLines = 0;
let remainder = "";
const regex = new RegExp(settings.signalRegex.source, "i");
const handleLine = (line: string) => {
totalLines++;
if (head.length < settings.headLines) {
head.push(clip(line, settings.maxLineChars));
} else if (signals.length < settings.maxSignalLines) {
regex.lastIndex = 0;
if (regex.test(line.slice(0, MATCH_SCAN_CHARS))) {
signals.push({ line: totalLines, text: clip(line, settings.maxLineChars) });
}
}
tail.push(clip(line, settings.maxLineChars));
if (tail.length > settings.tailLines) tail.shift();
};
for await (const chunk of createReadStream(path, { encoding: "utf8" })) {
remainder += chunk;
let index = remainder.indexOf("\n");
while (index >= 0) {
handleLine(remainder.slice(0, index));
remainder = remainder.slice(index + 1);
index = remainder.indexOf("\n");
}
}
if (remainder.length > 0) handleLine(remainder);
const tailStart = totalLines - tail.length + 1;
const middleSignals = signals.filter((hit) => hit.line < tailStart);
const info = await stat(path);
const take = Math.min(settings.thresholdChars, info.size);
let rawHead = "";
let rawTail = "";
if (take > 0) {
const handle = await open(path, "r");
try {
const headBuffer = Buffer.alloc(take);
const tailBuffer = Buffer.alloc(take);
await handle.read(headBuffer, 0, take, 0);
await handle.read(tailBuffer, 0, take, info.size - take);
rawHead = headBuffer.toString("utf8");
rawTail = tailBuffer.toString("utf8");
} finally {
await handle.close();
}
}
return {
head,
signals: middleSignals,
tail,
totalLines,
totalBytes: info.size,
rawHead,
rawTail,
};
}
function formatBytes(bytes: number): string {
if (bytes < 1024) return `${bytes} B`;
if (bytes < 1_048_576) return `${(bytes / 1024).toFixed(1)} KB`;
if (bytes < 1_073_741_824) return `${(bytes / 1_048_576).toFixed(1)} MB`;
return `${(bytes / 1_073_741_824).toFixed(2)} GB`;
}
function signalDescription(pattern: string): string {
const compact = pattern.replace(/\\b/g, "").replace(/[()]/g, "");
return compact.length > 60 ? `${compact.slice(0, 60)}…` : compact;
}
/**
* Deterministic bounded view of the stored original: head lines, signal
* lines from the middle section, and tail lines — all with the stored
* file's line numbers. Output never exceeds settings.thresholdChars and the
* tail of the stored content is always visible.
*
* Shrink order under budget pressure is tail > signals > head: head first
* collapses to min(10, headLines), signals halve stepwise to 10 then 0, then
* head/tail shrink ~35/65 with tail preferred, and char truncation of the
* stored head/tail is the final fallback.
*/
export function buildView(input: ViewInput, settings: OverflowSettings, meta: ViewMeta): string {
const budget = settings.thresholdChars;
const header =
`[pi-overflow id=${meta.id} tool=${meta.tool} ${input.totalLines} lines / ${formatBytes(input.totalBytes)}` +
` — full output stored; use overflow_query to search or read ranges]`;
const render = (headLines: string[], signals: SignalHit[], tailLines: string[]): string => {
const parts: string[] = [header];
if (headLines.length > 0) {
parts.push(`--- lines 1-${headLines.length} ---`, ...headLines);
}
if (signals.length > 0) {
parts.push(
`--- signals (lines matching ${signalDescription(settings.signalPattern)}, up to ${settings.maxSignalLines}) ---`,
...signals.map((hit) => `L${hit.line}: ${hit.text}`),
);
}
if (tailLines.length > 0) {
const first = input.totalLines - tailLines.length + 1;
parts.push(`--- lines ${first}-${input.totalLines} ---`, ...tailLines);
}
return parts.join("\n");
};
const charFallback = (): string => {
const markerReserve = 64;
const bodyBudget = Math.max(64, budget - header.length - markerReserve);
const headChars = Math.floor(bodyBudget * 0.35);
const tailChars = bodyBudget - headChars;
const headText = input.rawHead.slice(0, headChars);
const tailText = input.rawTail.slice(Math.max(0, input.rawTail.length - tailChars));
const omitted = Math.max(0, input.totalBytes - headText.length - tailText.length);
return `${header}\n${headText}\n… ${omitted} chars omitted …\n${tailText}`;
};
let headList = input.head;
let signalList = input.signals;
let tailList = input.tail;
let result = render(headList, signalList, tailList);
if (result.length <= budget) return result;
// 1. Collapse head to its minimum useful window.
headList = headList.slice(0, Math.min(10, settings.headLines));
result = render(headList, signalList, tailList);
if (result.length <= budget) return result;
// 2. Halve signals stepwise until 10 remain, then drop them entirely.
while (signalList.length > 10) {
signalList = signalList.slice(0, Math.ceil(signalList.length / 2));
result = render(headList, signalList, tailList);
if (result.length <= budget) return result;
}
if (signalList.length > 0) {
signalList = [];
result = render(headList, signalList, tailList);
if (result.length <= budget) return result;
}
// When head+tail already span every stored line there is no line-level
// slack left: go straight to char truncation so the ending survives.
if (input.head.length + input.tail.length >= input.totalLines) return charFallback();
// 3. Shrink head/tail ~35/65; tail keeps the larger share and always keeps
// its last line (the stored ending).
while (headList.length + tailList.length > 0) {
result = render(headList, signalList, tailList);
if (result.length <= budget) return result;
const headChars = headList.reduce((sum, line) => sum + line.length + 1, 0);
const tailChars = tailList.reduce((sum, line) => sum + line.length + 1, 0);
if (tailList.length > 1 && (tailChars >= headChars * (65 / 35) || headList.length === 0)) {
tailList.shift();
} else if (headList.length > 0) {
headList.pop();
} else {
tailList.shift();
}
}
// 4. Last resort: keep the stored text's own head 35% / tail 65%.
return charFallback();
}
+314
View File
@@ -0,0 +1,314 @@
import test from "node:test";
import assert from "node:assert/strict";
import { mkdtemp, readFile, readdir, stat, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import overflowCapture from "../capture.ts";
import overflowExtension from "../index.ts";
import { getStash } from "../src/stash.ts";
import { readOverflowSettings, OVERFLOW_DEFAULTS, type OverflowSettings } from "../settings.ts";
function harness() {
const handlers = new Map<string, Function[]>();
const tools = new Map<string, any>();
const pi: any = {
on(name: string, handler: Function) {
const list = handlers.get(name) ?? [];
list.push(handler);
handlers.set(name, list);
},
registerTool(tool: any) { tools.set(tool.name, tool); },
};
return { pi, handlers, tools };
}
function ctx() {
return { sessionManager: { getSessionId: () => "session-test" } } as any;
}
function settings(overrides: Partial<OverflowSettings> = {}): OverflowSettings {
return { ...readOverflowSettings("/nonexistent/pi-overflow.json"), ...overrides };
}
function textEvent(toolCallId: string, text: string, extra: Record<string, unknown> = {}) {
return {
toolCallId,
toolName: "bash",
input: { command: "run" },
content: [{ type: "text", text }],
details: {},
...extra,
};
}
async function runHandlers(handlers: Map<string, Function[]>, event: any, context: any) {
let current = event;
for (const handler of handlers.get("tool_result") ?? []) {
const result = await handler(current, context);
if (result) current = { ...current, ...result };
}
return current;
}
async function makeStore() {
return mkdtemp(join(tmpdir(), "pi-overflow-test-"));
}
test("capture stashes the original before another handler shortens it; bound stage stores original but views current", async () => {
const root = await makeStore();
getStash().clear();
const capture = harness();
overflowCapture(capture.pi);
const bound = harness();
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
const big = Array.from({ length: 3000 }, (_, i) => `orig ${i}`).join("\n");
const event = textEvent("call-1", big);
const afterCapture = await runHandlers(capture.handlers, event, ctx());
assert.equal(afterCapture, event, "capture must not modify the result");
assert.ok(getStash().has("call-1"), "capture must stash the original");
// Simulate RTK: replace content with a shortened version that is still
// over threshold.
const rtkShortened = Array.from({ length: 900 }, (_, i) => `rtk ${i}`).join("\n") + "\n" + "z".repeat(30_000);
const mid = { ...afterCapture, content: [{ type: "text", text: rtkShortened }] };
const final = await runHandlers(bound.handlers, mid, ctx());
assert.ok(!getStash().has("call-1"), "stash entry must be consumed");
const text = final.content[0].text;
assert.match(text, /\[pi-overflow id=[0-9a-f]{12} tool=bash/);
assert.ok(text.length <= settings().thresholdChars);
const id = final.details.overflow.id;
const stored = await readFile(join(root, `${id}.txt`), "utf8");
assert.ok(stored.includes("orig 0"), "store must keep the ORIGINAL, not the RTK-shortened text");
assert.ok(stored.includes("orig 2999"));
assert.equal(final.details.overflow.truncated, true);
});
test("details.fullOutputPath file is used as the original", async () => {
const root = await makeStore();
getStash().clear();
const bound = harness();
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
const spill = join(root, "spill.txt");
await writeFile(spill, Array.from({ length: 5000 }, (_, i) => `spill ${i}`).join("\n"));
const event = textEvent("call-2", "x".repeat(30_000), { details: { fullOutputPath: spill } });
const final = await runHandlers(bound.handlers, event, ctx());
const stored = await readFile(join(root, `${final.details.overflow.id}.txt`), "utf8");
assert.ok(stored.includes("spill 0"));
assert.ok(stored.includes("spill 4999"));
});
test("view line numbers track the STORED file, not the current text", async () => {
const root = await makeStore();
getStash().clear();
const bound = harness();
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
// bash spill: 5000 lines on disk, but the in-context result only carries
// the last ~100 lines (still over threshold via padding).
const spill = join(root, "spill-5000.txt");
await writeFile(
spill,
Array.from({ length: 5000 }, (_, i) => `spill line ${i + 1}`).join("\n"),
);
const current = Array.from({ length: 100 }, (_, i) => `spill line ${4901 + i}`).join("\n") + "\n" + "pad".repeat(10_000);
const event = textEvent("call-lines", current, { details: { fullOutputPath: spill } });
const final = await runHandlers(bound.handlers, event, ctx());
const text = final.content[0].text;
const id = final.details.overflow.id;
assert.match(text, /--- lines 4881-5000 ---/, "tail range must come from the stored 5000-line file");
assert.match(text, /spill line 5000/);
assert.match(text, /\[pi-overflow id=[0-9a-f]{12} tool=bash 5000 lines/);
// overflow_query with the view's tail range returns the same lines.
const tool = bound.tools.get("overflow_query");
const result = await tool.execute("q1", { id, lines: { start: 4881, end: 4883 } });
assert.match(result.content[0].text, /L4881: spill line 4881/);
assert.match(result.content[0].text, /L4883: spill line 4883/);
});
test("RTK scenario: view line numbers come from the stored original text", async () => {
const root = await makeStore();
getStash().clear();
const capture = harness();
overflowCapture(capture.pi);
const bound = harness();
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
const original = Array.from({ length: 3000 }, (_, i) => `orig ${i + 1}`).join("\n");
await runHandlers(capture.handlers, textEvent("call-rtk", original), ctx());
// RTK rewrites the visible text but it stays over threshold.
const mid = {
...textEvent("call-rtk", ""),
content: [{ type: "text", text: "z".repeat(30_000) }],
};
const final = await runHandlers(bound.handlers, mid, ctx());
const text = final.content[0].text;
assert.match(text, /--- lines 2881-3000 ---/, "view tail must be the stored original's tail");
assert.ok(text.includes("orig 3000"));
});
test("file sources over maxEntryBytes keep real head and tail halves", async () => {
const root = await makeStore();
const { storeEntry } = await import("../src/store.ts");
const src = join(root, "big.bin");
const content = "HEAD-".repeat(200) + "MID-".repeat(2000) + "TAIL-".repeat(200);
await writeFile(src, content);
const maxEntryBytes = 2000;
await storeEntry(root, "0123456789ab", { kind: "file", path: src }, { tool: "bash", args: {} }, maxEntryBytes);
const stored = await readFile(join(root, "0123456789ab.txt"), "utf8");
assert.ok(stored.startsWith("HEAD-"));
assert.ok(stored.endsWith("TAIL-"));
assert.match(stored, /bytes omitted/);
// Head and tail halves must equal the source's real bytes.
const half = Math.floor(maxEntryBytes / 2);
assert.equal(stored.slice(0, half), content.slice(0, half));
assert.equal(stored.slice(-half), content.slice(-half));
assert.ok(!stored.includes("\0".repeat(100)), "must not store zeroed content");
});
test("interrupted-write tmp files older than an hour are cleaned", async () => {
const root = await makeStore();
const bound = harness();
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
const staleTmp = join(root, "deadbeef.txt.tmp-999-zzz");
const freshTmp = join(root, "livebeef.txt.tmp-111-aaa");
await writeFile(staleTmp, "partial");
await writeFile(freshTmp, "partial");
const old = new Date(Date.now() - 2 * 3_600_000);
const { utimes } = await import("node:fs/promises");
await utimes(staleTmp, old, old);
for (const handler of bound.handlers.get("session_start") ?? []) await handler({}, ctx());
const { existsSync } = await import("node:fs");
assert.ok(!existsSync(staleTmp), "stale tmp must be removed");
assert.ok(existsSync(freshTmp), "fresh tmp must survive");
});
test("current under threshold but original over → one-line marker appended", async () => {
const root = await makeStore();
getStash().clear();
const capture = harness();
overflowCapture(capture.pi);
const bound = harness();
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
const big = "a".repeat(30_000);
const event = textEvent("call-3", big);
await runHandlers(capture.handlers, event, ctx());
const mid = { ...event, content: [{ type: "text", text: "short summary" }] };
const final = await runHandlers(bound.handlers, mid, ctx());
const text = final.content[0].text;
assert.ok(text.startsWith("short summary"));
assert.match(text, /\[pi-overflow id=[0-9a-f]{12}: full original output stored \(\d+ lines, .*; use overflow_query/);
assert.equal(final.details.overflow.truncated, false);
assert.ok(final.details.overflow.id);
});
test("exempt tools and nested calls are ignored", async () => {
const root = await makeStore();
getStash().clear();
const bound = harness();
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
const big = "x".repeat(30_000);
const exempt = await runHandlers(bound.handlers, textEvent("c-ex", big, { toolName: "read" }), ctx());
assert.equal(exempt.content[0].text, big, "read results must pass through untouched");
const nested = await runHandlers(bound.handlers, textEvent("c-nest", big, { parentToolCallId: "parent" }), ctx());
assert.equal(nested.content[0].text, big, "nested calls must pass through untouched");
});
test("image parts and structuredContent survive", async () => {
const root = await makeStore();
getStash().clear();
const bound = harness();
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
const event = textEvent("call-4", "x".repeat(30_000));
event.content.push({ type: "image", data: "b64" } as any);
(event as any).structuredContent = { rows: 3 };
const final = await runHandlers(bound.handlers, event, ctx());
assert.deepEqual(final.structuredContent, { rows: 3 });
assert.equal(final.content.at(-1).type, "image");
assert.equal(final.content.at(-1).data, "b64");
});
test("existing details fields survive the merge", async () => {
const root = await makeStore();
getStash().clear();
const bound = harness();
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
const event = textEvent("call-5", "x".repeat(30_000), { details: { rtk: { saved: 100 }, fullOutputPath: undefined } });
const final = await runHandlers(bound.handlers, event, ctx());
assert.deepEqual(final.details.rtk, { saved: 100 });
assert.ok(final.details.overflow.id);
});
test("files are 0600 and directories 0700", async () => {
const root = await makeStore();
getStash().clear();
const bound = harness();
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
const final = await runHandlers(bound.handlers, textEvent("call-6", "x".repeat(30_000)), ctx());
const id = final.details.overflow.id;
const dirMode = (await stat(root)).mode & 0o777;
const fileMode = (await stat(join(root, `${id}.txt`))).mode & 0o777;
const metaMode = (await stat(join(root, `${id}.json`))).mode & 0o777;
assert.equal(dirMode, 0o700);
assert.equal(fileMode, 0o600);
assert.equal(metaMode, 0o600);
});
test("retention and total-size cleanup remove old entries", async () => {
const root = await makeStore();
const bound = harness();
overflowExtension(bound.pi, {
storeRoot: root,
settings: settings({ retentionDays: 1, maxStoreBytes: 100 }),
});
// An old entry (mtime in the past) must be deleted by age.
const oldId = "aaaaaaaaaaaa";
await writeFile(join(root, `${oldId}.txt`), "old");
await writeFile(join(root, `${oldId}.json`), "{}");
const past = new Date(Date.now() - 10 * 86_400_000);
const { utimes } = await import("node:fs/promises");
await utimes(join(root, `${oldId}.txt`), past, past);
await utimes(join(root, `${oldId}.json`), past, past);
for (const handler of bound.handlers.get("session_start") ?? []) await handler({}, ctx());
const names = await readdir(root);
assert.ok(!names.some((n) => n.startsWith(oldId)), "old entry must be deleted");
// Fresh entries exceeding maxStoreBytes get evicted oldest-first.
for (const [i, id] of ["bbbbbbbbbbbb", "cccccccccccc", "dddddddddddd"].entries()) {
await writeFile(join(root, `${id}.txt`), "y".repeat(80));
const t = new Date(Date.now() - (3 - i) * 60_000);
await utimes(join(root, `${id}.txt`), t, t);
}
for (const handler of bound.handlers.get("session_start") ?? []) await handler({}, ctx());
const after = (await readdir(root)).filter((n) => n.endsWith(".txt"));
assert.ok(after.length <= 2, `expected eviction under maxStoreBytes, got ${after.join(",")}`);
assert.ok(!after.includes("bbbbbbbbbbbb.txt"), "oldest entry must be evicted first");
});
test("disabled setting clears the stash and returns undefined", async () => {
const root = await makeStore();
getStash().clear();
const bound = harness();
overflowExtension(bound.pi, { storeRoot: root, settings: settings({ enabled: false }) });
getStash().set("c-off", { rawText: "x".repeat(30_000) });
const event = textEvent("c-off", "x".repeat(30_000));
const result = await runHandlers(bound.handlers, event, ctx());
assert.equal(result, event);
assert.ok(!getStash().has("c-off"));
});
+112
View File
@@ -0,0 +1,112 @@
import test from "node:test";
import assert from "node:assert/strict";
import { mkdtemp, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { runOverflowQuery } from "../src/query.ts";
import { storeEntry } from "../src/store.ts";
import { readOverflowSettings, type OverflowSettings } from "../settings.ts";
function settings(overrides: Partial<OverflowSettings> = {}): OverflowSettings {
return { ...readOverflowSettings("/nonexistent/pi-overflow.json"), ...overrides };
}
async function store(text: string, id = "0123456789ab") {
const root = await mkdtemp(join(tmpdir(), "pi-overflow-query-"));
await storeEntry(root, id, { kind: "text", text }, { tool: "bash", args: { command: "run" } }, 1_000_000);
return root;
}
test("search returns hits with merged context", async () => {
const lines = Array.from({ length: 100 }, (_, i) => (i === 9 || i === 11 || i === 50 ? `line ${i + 1} FAIL` : `line ${i + 1}`));
const root = await store(lines.join("\n"));
const result = await runOverflowQuery(root, { id: "0123456789ab", search: "FAIL", context: 1 }, settings());
assert.ok(!result.error);
assert.match(result.text, /L10: line 10 FAIL/);
assert.match(result.text, /L12: line 12 FAIL/);
assert.match(result.text, /L51: line 51 FAIL/);
// hits at lines 10 and 12 with context 1 merge into one range 9-13
assert.match(result.text, /--- lines 9-13 ---/);
assert.match(result.text, /--- lines 50-52 ---/);
});
test("line-range retrieval returns numbered lines", async () => {
const lines = Array.from({ length: 50 }, (_, i) => `row ${i + 1}`);
const root = await store(lines.join("\n"));
const result = await runOverflowQuery(root, { id: "0123456789ab", lines: { start: 10, end: 15 } }, settings());
assert.match(result.text, /--- lines 10-15 ---/);
assert.match(result.text, /L10: row 10/);
assert.match(result.text, /L15: row 15/);
assert.ok(!result.text.includes("row 16"));
});
test("jsonPath extracts nested values", async () => {
const doc = JSON.stringify({ snapshot: { nodes: [{ name: "a" }, { name: "b" }, { name: "c" }] } });
const root = await store(doc);
const result = await runOverflowQuery(root, { id: "0123456789ab", jsonPath: "snapshot.nodes[2].name" }, settings());
assert.match(result.text, /"c"/);
});
test("invalid id, regex, JSON and path return clear errors without throwing", async () => {
const root = await store('{"a": {"b": 1}}');
assert.ok((await runOverflowQuery(root, { id: "../etc/passwd" }, settings())).error);
assert.ok((await runOverflowQuery(root, { id: "zzzzzzzzzzzz" }, settings())).error);
assert.ok((await runOverflowQuery(root, { id: "0123456789ab", search: "([" }, settings())).error);
const goodPath = await runOverflowQuery(root, { id: "0123456789ab", jsonPath: "a.b" }, settings());
assert.ok(!goodPath.error, "valid path on valid JSON must succeed");
const badPath = await runOverflowQuery(root, { id: "0123456789ab", jsonPath: "a.b.c!!" }, settings());
assert.ok(badPath.error);
const missing = await runOverflowQuery(root, { id: "ffffffffffff" }, settings());
assert.ok(missing.error);
});
test("non-JSON content + jsonPath reports the parse failure", async () => {
const root = await store("not json at all");
const result = await runOverflowQuery(root, { id: "0123456789ab", jsonPath: "a" }, settings());
assert.ok(result.error);
assert.match(result.text, /not valid JSON/);
});
test("output is bounded by thresholdChars and marked truncated", async () => {
const lines = Array.from({ length: 5000 }, (_, i) => `line ${i + 1} pad ${"p".repeat(40)}`);
const root = await store(lines.join("\n"));
const result = await runOverflowQuery(
root,
{ id: "0123456789ab", lines: { start: 1, end: 5000 } },
settings({ thresholdChars: 4000 }),
);
assert.ok(result.text.length <= 4000);
assert.equal(result.truncated, true);
assert.match(result.text, /truncated/);
});
test("lines and search output clip each line to maxLineChars", async () => {
const long = `row ${"q".repeat(2_000)}`;
const root = await store(`first\n${long}\n${long} ERROR\nlast`);
const s = settings({ maxLineChars: 60 });
const lines = await runOverflowQuery(root, { id: "0123456789ab", lines: { start: 2, end: 3 } }, s);
assert.ok(lines.text.includes("q".repeat(56) + "…"));
assert.ok(!lines.text.includes("q".repeat(61)));
const search = await runOverflowQuery(root, { id: "0123456789ab", search: "ERROR", context: 0 }, s);
assert.ok(search.text.includes("q".repeat(56) + "…"));
assert.ok(!search.text.includes("q".repeat(61)));
});
test("search only scans the first 10000 chars of each line", async () => {
const earlyHit = `hit ${"a".repeat(100)} FAIL`;
const lateHit = `${"a".repeat(10_500)} FAIL`;
const root = await store(`${earlyHit}\n${lateHit}\nplain`);
const result = await runOverflowQuery(root, { id: "0123456789ab", search: "FAIL", context: 0 }, settings());
assert.match(result.text, /L1: hit /);
assert.ok(!result.text.includes("L2:"), "match beyond char 10000 must not be reported");
});
test("default query returns metadata and a first-screen view", async () => {
const lines = Array.from({ length: 300 }, (_, i) => `row ${i + 1}`);
const root = await store(lines.join("\n"));
const result = await runOverflowQuery(root, { id: "0123456789ab" }, settings());
assert.ok(!result.error);
assert.match(result.text, /\[pi-overflow id=0123456789ab tool=bash \d+ lines \/ \d+ bytes\]/);
assert.match(result.text, /--- lines 1-40 ---/);
assert.ok(result.text.includes("row 300"), "tail of the stored file must appear");
});
+86
View File
@@ -0,0 +1,86 @@
import test from "node:test";
import assert from "node:assert/strict";
import { buildView, collectViewFromText } from "../src/view.ts";
import { readOverflowSettings, type OverflowSettings } from "../settings.ts";
function settings(overrides: Partial<OverflowSettings> = {}): OverflowSettings {
return {
...readOverflowSettings("/nonexistent/pi-overflow.json"),
...overrides,
};
}
const meta = { id: "abc123def456", tool: "bash" };
function view(text: string, s = settings()) {
return buildView(collectViewFromText(text, s), s, meta);
}
test("below threshold remains unchanged", () => {
const out = view("line one\nline two\nline three");
assert.ok(out.includes("line one"));
assert.ok(out.includes("line three"));
});
test("above threshold includes header, line ranges, signal lines and tail", () => {
const lines: string[] = [];
for (let i = 1; i <= 1000; i++) lines.push(`row ${i} ${i === 500 ? "ERROR boom" : "ok"}`);
const out = view(lines.join("\n"));
assert.match(out, /\[pi-overflow id=abc123def456 tool=bash 1000 lines/);
assert.match(out, /--- lines 1-40 ---/);
assert.match(out, /L500: row 500 ERROR boom/);
assert.match(out, /--- lines 881-1000 ---/);
assert.ok(out.includes("row 1000"));
});
test("output never exceeds thresholdChars", () => {
const lines = Array.from({ length: 5000 }, (_, i) => `line ${i} ${"x".repeat(60)}`);
const s = settings({ thresholdChars: 2000 });
const out = view(lines.join("\n"), s);
assert.ok(out.length <= 2000, `view ${out.length} exceeds budget`);
assert.ok(out.includes("line 4999"), "tail must survive");
});
test("few long lines still expose the ending", () => {
const lines = ["start", ...Array.from({ length: 5 }, (_, i) => `middle${i} ${"y".repeat(4000)}`), "THE_END_MARKER"];
const s = settings({ thresholdChars: 3000, headLines: 40, tailLines: 120 });
const out = view(lines.join("\n"), s);
assert.ok(out.includes("THE_END_MARKER"));
assert.ok(out.length <= 3000 + 100, `view ${out.length} far exceeds budget`);
});
test("same input produces identical output", () => {
const lines = Array.from({ length: 800 }, (_, i) => `entry ${i} ${i % 77 === 0 ? "warning" : "fine"}`);
const text = lines.join("\n");
assert.equal(view(text), view(text));
});
test("a single line over maxLineChars is truncated", () => {
const long = "z".repeat(2000);
const out = view(`a\n${long}\nb`, settings({ maxLineChars: 50 }));
assert.ok(out.includes("z".repeat(50) + "…"));
assert.ok(!out.includes("z".repeat(51)));
});
test("signals outrank head and survive when head+tail exceed the budget", () => {
// ~150-char lines: head 40 + tail 120 = 160 lines * 150 = ~24K plus
// overhead, over budget. Shrink order must keep some signals while head
// collapses to 10 lines before the 35/65 phase kicks in.
const lines = Array.from({ length: 400 }, (_, i) => {
const marker = i === 200 ? "ERROR detected here" : "ok";
return `line ${i + 1} ${marker} ${"x".repeat(140)}`;
});
const out = view(lines.join("\n"));
assert.match(out, /--- signals/);
assert.match(out, /L201: line 201 ERROR detected here/);
// head collapsed below the configured 40 lines
assert.match(out, /--- lines 1-(?:[0-9]|10) ---/);
assert.ok(out.includes("line 400"));
});
test("signal hits inside the tail region are not duplicated", () => {
const lines = Array.from({ length: 100 }, (_, i) => `line ${i + 1} ${"y".repeat(40)}`);
lines[99] = "line 100 tail-side ERROR";
const out = view(lines.join("\n"));
assert.ok(!out.includes("L100:"), "tail-region hit must not appear in signals");
});
+1 -1
View File
@@ -2,7 +2,7 @@
- 扩展入口是 `pi-rtk-optimizer/index.ts`,主要实现位于 `pi-rtk-optimizer/src/`。 - 扩展入口是 `pi-rtk-optimizer/index.ts`,主要实现位于 `pi-rtk-optimizer/src/`。
- 组合包运行环境为 Node.js 22.19 或更高版本;本子目录开发验证还需要 npm、Bun 和项目声明的开发依赖。 - 组合包运行环境为 Node.js 22.19 或更高版本;本子目录开发验证还需要 npm、Bun 和项目声明的开发依赖。
- RTK 默认只压缩非搜索输出,包括本地 `bash` 与远端 `ssh_bash` 的 ANSI 清理、测试聚合、构建过滤、Git 压缩、Lint 聚合和兜底截断;`ssh_bash` 只复用输出处理,远端命令不得进入 RTK rewrite。 - RTK 默认只压缩非搜索输出,包括本地 `bash` 与远端 `ssh_bash` 的 ANSI 清理、测试聚合、构建过滤、Git 压缩、Lint 聚合;兜底截断默认关闭,新结果的展示截断由 pi-overflow 统一负责;`ssh_bash` 只复用输出处理,远端命令不得进入 RTK rewrite。
- RTK 命令改写默认关闭(`commandRewritingEnabled` 默认 `false`);默认安装不依赖系统 `rtk` CLI,只有用户主动开启命令改写时才需要 `rtk rewrite`,其命令支持策略由已安装的 `rtk rewrite` 决定,不在扩展内重复维护规则。 - RTK 命令改写默认关闭(`commandRewritingEnabled` 默认 `false`);默认安装不依赖系统 `rtk` CLI,只有用户主动开启命令改写时才需要 `rtk rewrite`,其命令支持策略由已安装的 `rtk rewrite` 决定,不在扩展内重复维护规则。
- FFF override 独占搜索工具与搜索输出;RTK 不得处理本地或远端搜索工具的调用/结果(含 `grep` 工具结果),也不得通过默认命令改写接管 `rg`、`grep`、`find`、`fd`。 - FFF override 独占搜索工具与搜索输出;RTK 不得处理本地或远端搜索工具的调用/结果(含 `grep` 工具结果),也不得通过默认命令改写接管 `rg`、`grep`、`find`、`fd`。
- `readCompaction.enabled`、`sourceCodeFilteringEnabled` 和 `smartTruncate.enabled` 当前均默认 `false`,源码读取保持原样;未经用户明确决定,不因节省上下文而改变这些默认值。若以后开启 read 压缩,优先考虑 `readCompaction + smartTruncate`,源码过滤仍独立评估;必须保留精确 `offset/limit` 读取、短文件和行锚点的完整性。 - `readCompaction.enabled`、`sourceCodeFilteringEnabled` 和 `smartTruncate.enabled` 当前均默认 `false`,源码读取保持原样;未经用户明确决定,不因节省上下文而改变这些默认值。若以后开启 read 压缩,优先考虑 `readCompaction + smartTruncate`,源码过滤仍独立评估;必须保留精确 `offset/limit` 读取、短文件和行锚点的完整性。
+2 -2
View File
@@ -154,7 +154,7 @@ When `read` output uses Pi hashline/anchor prefixes, the compactor treats each a
|--------|------|---------|-------|-------------| |--------|------|---------|-------|-------------|
| `outputCompaction.smartTruncate.enabled` | boolean | `false` | — | Enable smart line-based truncation for read output when read compaction is enabled | | `outputCompaction.smartTruncate.enabled` | boolean | `false` | — | Enable smart line-based truncation for read output when read compaction is enabled |
| `outputCompaction.smartTruncate.maxLines` | number | `220` | 40–4000 | Maximum lines after smart truncation | | `outputCompaction.smartTruncate.maxLines` | number | `220` | 40–4000 | Maximum lines after smart truncation |
| `outputCompaction.truncate.enabled` | boolean | `true` | — | Enable hard character truncation | | `outputCompaction.truncate.enabled` | boolean | `false` | — | Enable hard character truncation (off in this bundle — pi-overflow owns display truncation) |
| `outputCompaction.truncate.maxChars` | number | `12000` | 1000–200000 | Maximum characters in final output | | `outputCompaction.truncate.maxChars` | number | `12000` | 1000–200000 | Maximum characters in final output |
### Source Code Filtering Levels ### Source Code Filtering Levels
@@ -195,7 +195,7 @@ When `read` output uses Pi hashline/anchor prefixes, the compactor treats each a
"maxLines": 220 "maxLines": 220
}, },
"truncate": { "truncate": {
"enabled": true, "enabled": false,
"maxChars": 12000 "maxChars": 12000
} }
} }
+1
View File
@@ -13,3 +13,4 @@
- 默认只负责非搜索工具输出压缩;RTK 命令改写收敛为可选能力(`commandRewritingEnabled` 默认 `false`)。 - 默认只负责非搜索工具输出压缩;RTK 命令改写收敛为可选能力(`commandRewritingEnabled` 默认 `false`)。
- `readCompaction`、`sourceCodeFiltering`、`smartTruncate` 默认关闭,源码读取保持原样。 - `readCompaction`、`sourceCodeFiltering`、`smartTruncate` 默认关闭,源码读取保持原样。
- `outputCompaction.truncate` 兜底截断默认关闭(组合包 `config/rtk-optimizer.json`),新结果的展示截断由 pi-overflow 统一负责;RTK 只保留语义压缩。
+1
View File
@@ -12,6 +12,7 @@
- Added checked-in authoritative groups for every tool exposed by the standard my-pi bundle, eliminating first-use model generation and user caching unless unrecognized third-party tools are present. - Added checked-in authoritative groups for every tool exposed by the standard my-pi bundle, eliminating first-use model generation and user caching unless unrecognized third-party tools are present.
- Replaced the Hermes and third-party Hippo assignments with the five tools exposed by the official Hippo Pi extension, grouped into recall and management workflows. - Replaced the Hermes and third-party Hippo assignments with the five tools exposed by the official Hippo Pi extension, grouped into recall and management workflows.
- Replaced the retired Hippo `hippo_*` assignments with the vendored Hermes memory extension's six tools, grouped into `memory-recall` (`memory_search`, `session_search`, `context_tree_query`) and `memory-management` (`memory_add`, `memory_replace`, `memory_remove`, `skill_manage`). - Replaced the retired Hippo `hippo_*` assignments with the vendored Hermes memory extension's six tools, grouped into `memory-recall` (`memory_search`, `session_search`, `context_tree_query`) and `memory-management` (`memory_add`, `memory_replace`, `memory_remove`, `skill_manage`).
- Added `overflow_query` to `memory-recall` and pinned it in `alwaysEnabled` — inspecting a pi-overflow-stored original must not depend on a `tool_search` activation.
- Added the enabled bundle's `Agent`, `get_subagent_result`, and `steer_subagent` tools to a checked-in `subagents` orchestration group. - Added the enabled bundle's `Agent`, `get_subagent_result`, and `steer_subagent` tools to a checked-in `subagents` orchestration group.
- Added authoritative `ssh-connection`, `ssh-remote-files`, `ssh-remote-search`, and `ssh-remote-shell` groups for the maintained `pi-ssh` tools, keeping reviewed connection, bounded discovery, structured file operations, and reviewed shell execution separate; the connection workflow now requires sequential `ssh_connect` to succeed before dependent tools. - Added authoritative `ssh-connection`, `ssh-remote-files`, `ssh-remote-search`, and `ssh-remote-shell` groups for the maintained `pi-ssh` tools, keeping reviewed connection, bounded discovery, structured file operations, and reviewed shell execution separate; the connection workflow now requires sequential `ssh_connect` to succeed before dependent tools.
- Added sequential `ssh_cd` to the authoritative remote-shell workflow and direct the model to wait for a successful workspace change before dependent remote calls, avoiding both the unknown-tools fallback and cwd races. - Added sequential `ssh_cd` to the authoritative remote-shell workflow and direct the model to wait for a successful workspace change before dependent remote calls, avoiding both the unknown-tools fallback and cwd races.
+3 -2
View File
@@ -87,7 +87,8 @@ Pi reads this configuration directly from the installed bundle's `config/tool-se
"memory_replace", "memory_replace",
"memory_remove", "memory_remove",
"skill_manage", "skill_manage",
"context_tree_query" "context_tree_query",
"overflow_query"
], ],
"showToolSearchFooterStatus": false, "showToolSearchFooterStatus": false,
"maxActiveGroups": 5, "maxActiveGroups": 5,
@@ -99,7 +100,7 @@ Pi reads this configuration directly from the installed bundle's `config/tool-se
| Key | Bundle default | Description | | Key | Bundle default | Description |
|---|---|---| |---|---|---|
| `alwaysEnabled` | CodeGraph explore, LSP diagnostics, Context execution trio, all six Hermes memory tools, `context_tree_query` | Exact names pinned alongside Pi core tools | | `alwaysEnabled` | CodeGraph explore, LSP diagnostics, Context execution trio, all six Hermes memory tools, `context_tree_query`, `overflow_query` | Exact names pinned alongside Pi core tools |
| `showToolSearchFooterStatus` | `false` | Show active tool/group counts | | `showToolSearchFooterStatus` | `false` | Show active tool/group counts |
| `maxActiveGroups` | `5` | Maximum simultaneously active dynamic groups | | `maxActiveGroups` | `5` | Maximum simultaneously active dynamic groups |
| `maxToolsPerGroup` | `8` | Maximum size accepted from model-generated groups | | `maxToolsPerGroup` | `8` | Maximum size accepted from model-generated groups |
+1 -1
View File
@@ -13,7 +13,7 @@
9. Natural-language lookup gives exact tool names and workflow aliases more weight than descriptive overlap. Weak or closely tied matches return candidates without activation. 9. Natural-language lookup gives exact tool names and workflow aliases more weight than descriptive overlap. Weak or closely tied matches return candidates without activation.
10. `turn_start` notices tool or grouping-configuration changes by hash, resets stale dynamic groups, and reconstructs the checked-in/hybrid catalog. 10. `turn_start` notices tool or grouping-configuration changes by hash, resets stale dynamic groups, and reconstructs the checked-in/hybrid catalog.
The standard bundle assigns the Hermes memory extension's six tools to recall and management groups: `memory_search` and `session_search` join `context_tree_query` in `memory-recall`, while `memory_add`, `memory_replace`, `memory_remove`, and `skill_manage` form `memory-management`. All six are pinned by default, because the injected `<memory-policy>` tells the model to consult and write memory during normal work, and a schema the model cannot see would silently disable that loop. Third-party `tff-memory_*` names and the retired Hippo `hippo_*` names are not seeded. It also assigns `Agent`, `get_subagent_result`, and `steer_subagent` to the `subagents` orchestration group. The standard bundle assigns the Hermes memory extension's six tools to recall and management groups: `memory_search`, `session_search`, and `overflow_query` join `context_tree_query` in `memory-recall`, while `memory_add`, `memory_replace`, `memory_remove`, and `skill_manage` form `memory-management`. All six are pinned by default, because the injected `<memory-policy>` tells the model to consult and write memory during normal work, and a schema the model cannot see would silently disable that loop. Third-party `tff-memory_*` names and the retired Hippo `hippo_*` names are not seeded. It also assigns `Agent`, `get_subagent_result`, and `steer_subagent` to the `subagents` orchestration group.
The maintained `pi-ssh` tools are also split by workflow: sequential reviewed agent-controlled connection lives in `ssh-connection`, structured `ssh_read`/`ssh_write`/`ssh_edit` file operations live in `ssh-remote-files`, bounded `ssh_find`/`ssh_grep` discovery lives in `ssh-remote-search`, and sequential reviewed workspace changes plus reviewed command execution live in `ssh-remote-shell`. The workflows tell the model to call `ssh_connect` or `ssh_cd` separately and wait for success before issuing remote calls that depend on the new connection or cwd. The maintained `pi-ssh` tools are also split by workflow: sequential reviewed agent-controlled connection lives in `ssh-connection`, structured `ssh_read`/`ssh_write`/`ssh_edit` file operations live in `ssh-remote-files`, bounded `ssh_find`/`ssh_grep` discovery lives in `ssh-remote-search`, and sequential reviewed workspace changes plus reviewed command execution live in `ssh-remote-shell`. The workflows tell the model to call `ssh_connect` or `ssh_cd` separately and wait for success before issuing remote calls that depend on the new connection or cwd.
The maintained `ask_user_question` tool has its own `user-interaction` group so the model can load the complete interactive schema only when a missing decision or preference must come from the user. The maintained `ask_user_question` tool has its own `user-interaction` group so the model can load the complete interactive schema only when a missing decision or preference must come from the user.
The pinned `pi-chrome` bridge registers its 21 `chrome_*` tools only after explicit per-session authorization. Once registered, catalog refresh assigns them to `chrome-navigation`, `chrome-interaction`, and `chrome-debugging`; the 28-tool default allows all three groups to coexist. Revocation remains enforced inside every upstream tool execution even if a schema is later reactivated by Tool Search. The pinned `pi-chrome` bridge registers its 21 `chrome_*` tools only after explicit per-session authorization. Once registered, catalog refresh assigns them to `chrome-navigation`, `chrome-interaction`, and `chrome-debugging`; the 28-tool default allows all three groups to coexist. Revocation remains enforced inside every upstream tool execution even if a schema is later reactivated by Tool Search.
+3 -3
View File
@@ -124,11 +124,11 @@ export const BUNDLE_GROUP_DEFINITIONS: BundleGroupDefinition[] = [
{ {
id: "memory-recall", id: "memory-recall",
title: "Persistent memory recall", title: "Persistent memory recall",
summary: "Search durable user, global, project, and failure memories, search indexed past conversations, or recover pruned tool-call output.", summary: "Search durable user, global, project, and failure memories, search indexed past conversations, or recover pruned or overflow-stored tool-call output.",
useWhen: ["The task may depend on durable context from previous sessions: preferences, project conventions, prior decisions, known failures, corrections, or condensed outputs"], useWhen: ["The task may depend on durable context from previous sessions: preferences, project conventions, prior decisions, known failures, corrections, or condensed outputs"],
avoidWhen: ["The required evidence is already in the current visible context, or the question is a generic one-off that durable memory would not help"], avoidWhen: ["The required evidence is already in the current visible context, or the question is a generic one-off that durable memory would not help"],
keywords: ["memory", "recall", "search", "session", "history", "recover", "记忆", "召回", "检索", "历史会话", "恢复输出"], keywords: ["memory", "recall", "search", "session", "history", "recover", "overflow", "记忆", "召回", "检索", "历史会话", "恢复输出"],
tools: ["memory_search", "session_search", "context_tree_query"], tools: ["memory_search", "session_search", "context_tree_query", "overflow_query"],
}, },
{ {
id: "memory-management", id: "memory-management",
+1
View File
@@ -35,6 +35,7 @@ export const BUNDLE_TOOL_SEARCH_DEFAULTS = {
"memory_remove", "memory_remove",
"skill_manage", "skill_manage",
"context_tree_query", "context_tree_query",
"overflow_query",
], ],
showToolSearchFooterStatus: false, showToolSearchFooterStatus: false,
maxActiveGroups: 5, maxActiveGroups: 5,
+1
View File
@@ -43,6 +43,7 @@ test("bundle defaults keep the committed tool list", () => {
"memory_remove", "memory_remove",
"skill_manage", "skill_manage",
"context_tree_query", "context_tree_query",
"overflow_query",
]); ]);
assert.deepEqual(bundleFallbackConfig(), { assert.deepEqual(bundleFallbackConfig(), {
alwaysEnabled: [...BUNDLE_TOOL_SEARCH_DEFAULTS.alwaysEnabled], alwaysEnabled: [...BUNDLE_TOOL_SEARCH_DEFAULTS.alwaysEnabled],
+4 -1
View File
@@ -6,9 +6,12 @@ Unified web search and fetch for Pi. The main model only ever sees two tools —
- `web_search(query, max_results?, site?, published_after?, published_before?, depth?)` - `web_search(query, max_results?, site?, published_after?, published_before?, depth?)`
routes each query to Tavily, Exa, or Keenable using the session's current main routes each query to Tavily, Exa, or Keenable using the session's current main
model (`ctx.model` + `ctx.modelRegistry.complete`), guided by per-provider model (`ctx.model` + `ctx.modelRegistry.complete`), guided by per-provider
rules. Results include `details.provider` / `details.routedBy` for debugging. rules. `max_results` defaults to 8 and is clamped to 1–20. Results include
`details.provider` / `details.routedBy` for debugging.
- `web_fetch(urls, max_characters?)` always reads pages via Exa - `web_fetch(urls, max_characters?)` always reads pages via Exa
(`web_fetch_exa`), degrading silently to Keenable then Tavily. (`web_fetch_exa`), degrading silently to Keenable then Tavily.
`max_characters` defaults to 20,000 and is clamped to 1–100,000 per page;
oversized fetch output is further bounded by pi-overflow downstream.
## Providers (`clients/`) ## Providers (`clients/`)
+17 -6
View File
@@ -21,8 +21,15 @@ import {
} from "./router.ts"; } from "./router.ts";
const DEFAULT_MAX_RESULTS = 8; const DEFAULT_MAX_RESULTS = 8;
const MAX_RESULTS_LIMIT = 20;
const MAX_FETCH_URLS = 10; const MAX_FETCH_URLS = 10;
const DEFAULT_FETCH_MAX_CHARACTERS = 3000; const DEFAULT_FETCH_MAX_CHARACTERS = 20000;
const FETCH_MAX_CHARACTERS_LIMIT = 100000;
function clampInt(value: number | undefined, fallback: number, min: number, max: number): number {
if (typeof value !== "number" || !Number.isFinite(value)) return fallback;
return Math.min(Math.max(Math.floor(value), min), max);
}
export interface WebSearchParams { export interface WebSearchParams {
query: string; query: string;
@@ -48,7 +55,7 @@ function formatSearchPayload(payload: unknown): string {
export function toTavilyParams(params: WebSearchParams) { export function toTavilyParams(params: WebSearchParams) {
return { return {
query: params.query, query: params.query,
max_results: params.max_results ?? DEFAULT_MAX_RESULTS, max_results: clampInt(params.max_results, DEFAULT_MAX_RESULTS, 1, MAX_RESULTS_LIMIT),
search_depth: (params.depth ?? "basic") as "basic" | "advanced", search_depth: (params.depth ?? "basic") as "basic" | "advanced",
include_domains: params.site ? [params.site] : undefined, include_domains: params.site ? [params.site] : undefined,
start_date: params.published_after, start_date: params.published_after,
@@ -59,7 +66,7 @@ export function toTavilyParams(params: WebSearchParams) {
export function toExaParams(params: WebSearchParams) { export function toExaParams(params: WebSearchParams) {
return { return {
query: params.query, query: params.query,
numResults: params.max_results ?? DEFAULT_MAX_RESULTS, numResults: clampInt(params.max_results, DEFAULT_MAX_RESULTS, 1, MAX_RESULTS_LIMIT),
includeDomains: params.site ? [params.site] : undefined, includeDomains: params.site ? [params.site] : undefined,
startPublishedDate: params.published_after, startPublishedDate: params.published_after,
endPublishedDate: params.published_before, endPublishedDate: params.published_before,
@@ -171,7 +178,7 @@ export async function dispatchFetch(
if (urls.length > MAX_FETCH_URLS) { if (urls.length > MAX_FETCH_URLS) {
return toResult(`web_fetch accepts at most ${MAX_FETCH_URLS} URLs per call; received ${urls.length}.`, { error: true }); return toResult(`web_fetch accepts at most ${MAX_FETCH_URLS} URLs per call; received ${urls.length}.`, { error: true });
} }
const maxCharacters = params.max_characters ?? DEFAULT_FETCH_MAX_CHARACTERS; const maxCharacters = clampInt(params.max_characters, DEFAULT_FETCH_MAX_CHARACTERS, 1, FETCH_MAX_CHARACTERS_LIMIT);
const order: SearchProvider[] = ["exa", "keenable"]; const order: SearchProvider[] = ["exa", "keenable"];
if (tavilyAvailable(resolve)) order.push("tavily"); if (tavilyAvailable(resolve)) order.push("tavily");
@@ -213,7 +220,9 @@ export default function webSearchExtension(pi: ExtensionAPI): void {
}, },
max_results: { max_results: {
type: "number", type: "number",
description: `Maximum number of results to return (default ${DEFAULT_MAX_RESULTS}).`, minimum: 1,
maximum: MAX_RESULTS_LIMIT,
description: `Maximum number of results to return (default ${DEFAULT_MAX_RESULTS}, max ${MAX_RESULTS_LIMIT}).`,
}, },
site: { site: {
type: "string", type: "string",
@@ -268,7 +277,9 @@ export default function webSearchExtension(pi: ExtensionAPI): void {
}, },
max_characters: { max_characters: {
type: "number", type: "number",
description: `Maximum characters to extract per page (default ${DEFAULT_FETCH_MAX_CHARACTERS}).`, minimum: 1,
maximum: FETCH_MAX_CHARACTERS_LIMIT,
description: `Maximum characters to extract per page (default ${DEFAULT_FETCH_MAX_CHARACTERS}, max ${FETCH_MAX_CHARACTERS_LIMIT}).`,
}, },
}, },
required: ["urls"], required: ["urls"],
+15
View File
@@ -198,6 +198,21 @@ test("all package extensions load together without global registration conflicts
"./extensions/system-prompt.ts", "./extensions/system-prompt.ts",
"the single system-prompt owner must be the last loaded extension", "the single system-prompt owner must be the last loaded extension",
); );
// pi-overflow has two entry points with strict ordering: capture.ts must
// run first so it sees results before RTK rewrites them, and index.ts must
// run last (before system-prompt) so it bounds the final visible text.
assert.equal(
packageJson.pi.extensions.at(0),
"./pi-overflow/capture.ts",
"pi-overflow capture must be the first loaded extension",
);
assert.equal(packageJson.dependencies["pi-overflow"], "file:./pi-overflow");
assert.ok(filesIncludePackageDir(packageJson.files, "pi-overflow"), "the packed bundle must include pi-overflow source");
assert.equal(
packageJson.pi.extensions.at(-2),
"./pi-overflow/index.ts",
"pi-overflow bound stage must load after all other content extensions and before system-prompt",
);
assert.ok( assert.ok(
packageJson.pi.extensions.includes("./extensions/context-mode.ts"), packageJson.pi.extensions.includes("./extensions/context-mode.ts"),
"Context Mode must load through the bundle's prompt-guarded wrapper", "Context Mode must load through the bundle's prompt-guarded wrapper",