mirror of
https://bitbucket.org/siakitem/my-pi.git
synced 2026-10-26 10:45:22 +00:00
feat(pi-overflow): 新增统一的新工具结果展示截断扩展
此前截断分散在多层且互相冲突：Pi bash 保留尾部 50KB，RTK 兜底截断又只留 开头 12000 字符（砍掉报错与完整输出路径），pi-chrome 截开头 30K，web_fetch 各 provider 不一致。新扩展统一负责"新结果给模型看多少"： - capture.ts 最先加载暂存原文，index.ts 最后加载（system-prompt 之前） - 超过 24K 字符时原文落盘 <agentDir>/my-pi/pi-overflow/（700/600、原子写）， 优先取 bash fullOutputPath 完整文件；模型看到 头 + 中段信号行 + 尾 的有界 视图，行号基于落盘原文、结尾永远可见 - overflow_query：按 id 正则搜索、行区间、jsonPath 只读查询，id 校验防穿越 - 收编：RTK truncate 默认关闭、删除 pi-chrome truncateText、web_fetch 默认 每页 20000（上限 100000）、web_search max_results 限定 1–20 - pi-compaction 遮蔽带 details.overflow.id 的结果时在占位符提示 overflow_query - 登记 tools.jsonc / routing / 权限 allow / tool-search 常驻与分组 Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
云服务部-叶林立
co-authored by
Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
parent
a6d8416c76
commit
2a6bf8c929
@@ -0,0 +1,18 @@
|
||||
# my-pi 维护约定（pi-overflow）
|
||||
|
||||
- 完全自维护（无上游）的新结果展示截断扩展，负责"每个新 tool_result 给模型看多少"：`capture.ts` 最先加载只暂存原文，`index.ts` 最后加载（在 `system-prompt.ts` 之前）生成有界视图并注册 `overflow_query`。
|
||||
- 根 `pi.extensions` 顺序是硬约束：`./pi-overflow/capture.ts` 必须第一，`./pi-overflow/index.ts` 必须在所有其他扩展之后、`./extensions/system-prompt.ts` 之前；跨扩展暂存走 `globalThis[Symbol.for("my-pi.pi-overflow.stash.v1")]`，容量 64 条、淘汰最旧。
|
||||
- `tool_result` handler 链中 capture 不得修改结果；index 改写 content 时必须原样回传 `structuredContent`、保留 image 部分、合并而非丢弃已有 `details`（如 RTK 压缩元数据、bash `fullOutputPath`）。
|
||||
- `read`/`edit`/`write`/`overflow_query`/`ask_user_question` 与嵌套调用（`parentToolCallId`）永远豁免；read 靠自身 offset/limit 分页，不得破坏 hashline 锚点。
|
||||
- 视图与行号一律基于刚落盘的存储原文（不是 current 文本）：文件来源用流式逐行扫描收集 head/环形 tail/signals，文本来源切同构片段；视图必须确定性、不超 `thresholdChars`、结尾永远可见；落盘原文优先取 `details.fullOutputPath` 文件（超 `maxEntryBytes` 只留头尾各半），其次暂存原文，再次当前文本。
|
||||
- 超预算收缩优先级为 tail > signals > head：head 先缩到 min(10, headLines)，signals 逐步减半至 10 再到 0，再按约 35/65 缩 head/tail，按字符截断兜底。
|
||||
- signal 扫描与 `overflow_query` 的 search 正则只匹配每行前 10,000 字符；query 输出每行按 `maxLineChars` 截断。
|
||||
- 存储位于 `<agentDir>/my-pi/pi-overflow/`（目录 `700`、文件 `600`、原子写入），id 为 `sha1(sessionId + "\0" + toolCallId)` 前 12 位 hex；`overflow_query` 的 id 必须匹配 `^[0-9a-f]{12}$` 防路径穿越；`session_start` 按 `retentionDays`/`maxStoreBytes` 清理并删除超过 1 小时的 `*.tmp-*` 残留，失败不得抛出。
|
||||
- 持久配置只读组合包 `config/pi-overflow.json`（`settings.ts` 逐字段校验、非法回退默认、signalPattern 编译失败回退默认且不敏感大小写）；不得读写 `settings.json` 或以环境变量作持久默认。
|
||||
- `overflow_query` 只读本扩展自己的存储目录；非法 id/正则/JSON/路径一律返回清楚错误文本，不得抛出。
|
||||
- 与 pi-compaction 的唯一联动是被遮蔽 toolResult 的 `details.overflow.id` 公开字段——pi-compaction 只读它、不得 import 本包任何实现。
|
||||
- 测试必须注入临时存储目录，不得写真实 agent 目录。
|
||||
|
||||
## 验证
|
||||
|
||||
- 在本目录内至少运行 `npm test` 与 `npm run check`；涉及加载入口、`pi.extensions` 顺序或 `files` 白名单时还要运行根 `node --test tests/extension-load.test.ts` 与 `tests/package-files.test.ts`。
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 sia
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,66 @@
|
||||
# pi-overflow
|
||||
|
||||
Bounded display for oversized tool results. When a tool result exceeds
|
||||
`thresholdChars`, pi-overflow persists the full original to disk and shows the
|
||||
model a deterministic view — head lines, signal lines (error/fail/warn/…), and
|
||||
tail lines — plus an `overflow_query` tool to search or read the stored
|
||||
original by id.
|
||||
|
||||
## Two-stage load order
|
||||
|
||||
- `capture.ts` — **first** in `pi.extensions`. A `tool_result` handler that
|
||||
never modifies anything; it stashes the untouched original text in
|
||||
`globalThis[Symbol.for("my-pi.pi-overflow.stash.v1")]` (capacity 64) so the
|
||||
bound stage can still reach the pre-RTK original.
|
||||
- `index.ts` — **last** content extension, right before `system-prompt.ts`.
|
||||
Its `tool_result` handler sees the final text after every other handler
|
||||
(RTK semantic compaction included) has run, then:
|
||||
- resolves the *original*: `details.fullOutputPath` file (streamed, capped
|
||||
at `maxEntryBytes` head+tail) → stashed raw text → current text;
|
||||
- if the current text still exceeds `thresholdChars`, replaces it with the
|
||||
bounded view **built from the stored file** — line numbers and totals in
|
||||
the view are exactly what `overflow_query` will read, even when RTK or
|
||||
bash already cut the in-context text (file sources are scanned
|
||||
line-by-line in a stream, never held in memory);
|
||||
- if the current text fits but the original did not (an earlier extension
|
||||
already cut it), keeps the text and appends a one-line
|
||||
`[pi-overflow id=…]` recovery marker;
|
||||
- preserves image parts and `structuredContent`, and merges
|
||||
`details.overflow = { id, totalLines, totalBytes, truncated }` into the
|
||||
existing details.
|
||||
- Exempt tools (`read`, `edit`, `write`, `overflow_query`,
|
||||
`ask_user_question`) and nested calls (`parentToolCallId`) are never
|
||||
touched.
|
||||
|
||||
## Stored originals
|
||||
|
||||
`<agentDir>/my-pi/pi-overflow/<id>.txt` + `<id>.json` metadata, where
|
||||
`id = sha1(sessionId + "\0" + toolCallId)[0:12]`. Directories are `0700`,
|
||||
files `0600`, writes are atomic. `session_start` cleans entries older than
|
||||
`retentionDays` and evicts oldest-first past `maxStoreBytes`.
|
||||
|
||||
## `overflow_query`
|
||||
|
||||
- `id` (required, `^[0-9a-f]{12}$` — path traversal safe)
|
||||
- `search` — case-insensitive regex; hits come back with `context` lines and
|
||||
adjacent hits merge; capped by `maxResults`
|
||||
- `lines: { start, end }` — 1-based inclusive line range
|
||||
- `jsonPath` — dot/bracket path into stored JSON (`a.b[3].c`)
|
||||
- No selector → metadata + first-screen view. Output is always bounded by
|
||||
`thresholdChars` and reports whether it was truncated.
|
||||
|
||||
## Configuration
|
||||
|
||||
`config/pi-overflow.json` inside the installed bundle. Every field is
|
||||
validated independently and falls back to its default; an uncompilable
|
||||
`signalPattern` falls back to the default pattern (case-insensitive).
|
||||
|
||||
## Development
|
||||
|
||||
```sh
|
||||
npm test # node --test test/*.test.ts
|
||||
npm run check # strip-types syntax check + tests
|
||||
```
|
||||
|
||||
Tests inject a temporary store root — they never write to a real agent
|
||||
directory.
|
||||
@@ -0,0 +1,8 @@
|
||||
# 上游来源
|
||||
|
||||
- 本项目完全自维护，无上游；未从任何外部仓库或 npm 包导入源码。
|
||||
- 实现仅依赖 Pi 扩展宿主 API（`@earendil-works/pi-coding-agent`）与 `typebox`，两者均为组合包 peer 依赖。
|
||||
|
||||
## 本地改动记录
|
||||
|
||||
- 初始实现：`capture.ts`（首个加载，暂存原始 tool_result 文本）、`index.ts`（最后加载，生成有界视图 + 注册 `overflow_query`）、`src/`（视图、存储、查询、跨扩展暂存区）。
|
||||
@@ -0,0 +1,30 @@
|
||||
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
|
||||
import { readOverflowSettings } from "./settings.ts";
|
||||
import { getStash, stashResult } from "./src/stash.ts";
|
||||
|
||||
/**
|
||||
* Pre-bound capture: loaded FIRST in pi.extensions, before RTK or any other
|
||||
* result-rewriting extension. It only stashes the untouched original text —
|
||||
* it never modifies the result. The bound stage (`index.ts`, loaded last)
|
||||
* consumes the stash after all other handlers have run.
|
||||
*/
|
||||
export default function overflowCapture(pi: ExtensionAPI): void {
|
||||
const settings = readOverflowSettings();
|
||||
pi.on("tool_result", (event) => {
|
||||
if (!settings.enabled) return undefined;
|
||||
if (settings.exemptTools.includes(event.toolName)) return undefined;
|
||||
if (event.parentToolCallId) return undefined;
|
||||
const rawText = event.content
|
||||
.filter((part): part is { type: "text"; text: string } => part.type === "text")
|
||||
.map((part) => part.text)
|
||||
.join("\n");
|
||||
const fullOutputPath =
|
||||
typeof event.details?.fullOutputPath === "string" ? event.details.fullOutputPath : undefined;
|
||||
if (rawText.length <= settings.thresholdChars && !fullOutputPath) return undefined;
|
||||
stashResult(getStash(), event.toolCallId, {
|
||||
rawText,
|
||||
...(fullOutputPath ? { fullOutputPath } : {}),
|
||||
});
|
||||
return undefined;
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,167 @@
|
||||
import { getAgentDir, type ExtensionAPI } from "@earendil-works/pi-coding-agent";
|
||||
import { Type } from "typebox";
|
||||
import { stat } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { readOverflowSettings, type OverflowSettings } from "./settings.ts";
|
||||
import { getStash } from "./src/stash.ts";
|
||||
import { buildView, collectViewFromFile, collectViewFromText } from "./src/view.ts";
|
||||
import { cleanStore, entryId, storeEntry } from "./src/store.ts";
|
||||
import { runOverflowQuery } from "./src/query.ts";
|
||||
|
||||
function formatBytes(bytes: number): string {
|
||||
if (bytes < 1024) return `${bytes} B`;
|
||||
if (bytes < 1_048_576) return `${(bytes / 1024).toFixed(1)} KB`;
|
||||
return `${(bytes / 1_048_576).toFixed(1)} MB`;
|
||||
}
|
||||
|
||||
export interface OverflowExtensionOptions {
|
||||
settings?: OverflowSettings;
|
||||
storeRoot?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Bound stage: loaded LAST among content extensions (before system-prompt).
|
||||
* By the time this handler runs, every other tool_result handler — RTK
|
||||
* included — has already transformed the result, so the bounded view is
|
||||
* built from the final visible text while the stored original comes from
|
||||
* capture.ts's stash or bash's fullOutputPath file.
|
||||
*/
|
||||
export default function overflowExtension(pi: ExtensionAPI, options: OverflowExtensionOptions = {}): void {
|
||||
const settings = options.settings ?? readOverflowSettings();
|
||||
const storeRoot = () => options.storeRoot ?? join(getAgentDir(), "my-pi", "pi-overflow");
|
||||
|
||||
pi.on("session_start", () => {
|
||||
// Retention/size cleanup never throws (cleanStore swallows failures)
|
||||
// and the returned promise lets tests await it.
|
||||
return cleanStore(storeRoot(), settings.retentionDays, settings.maxStoreBytes);
|
||||
});
|
||||
|
||||
pi.on("tool_result", async (event, ctx) => {
|
||||
const stash = getStash();
|
||||
const staged = stash.get(event.toolCallId);
|
||||
try {
|
||||
if (
|
||||
!settings.enabled ||
|
||||
settings.exemptTools.includes(event.toolName) ||
|
||||
event.parentToolCallId
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
const current = event.content
|
||||
.filter((part): part is { type: "text"; text: string } => part.type === "text")
|
||||
.map((part) => part.text)
|
||||
.join("\n");
|
||||
const fullOutputPath =
|
||||
typeof event.details?.fullOutputPath === "string"
|
||||
? event.details.fullOutputPath
|
||||
: staged?.fullOutputPath;
|
||||
|
||||
// Resolve the original: bash's spilled file wins (it is complete even
|
||||
// when the in-context text was already cut), then the capture stash,
|
||||
// then the current text itself.
|
||||
let source: { kind: "text"; text: string } | { kind: "file"; path: string };
|
||||
let originalSize: number;
|
||||
if (fullOutputPath) {
|
||||
try {
|
||||
const info = await stat(fullOutputPath);
|
||||
if (!info.isFile()) throw new Error("not a file");
|
||||
source = { kind: "file", path: fullOutputPath };
|
||||
originalSize = info.size;
|
||||
} catch {
|
||||
const fallback = staged?.rawText ?? current;
|
||||
source = { kind: "text", text: fallback };
|
||||
originalSize = fallback.length;
|
||||
}
|
||||
} else {
|
||||
const fallback = staged?.rawText ?? current;
|
||||
source = { kind: "text", text: fallback };
|
||||
originalSize = fallback.length;
|
||||
}
|
||||
|
||||
if (current.length <= settings.thresholdChars && originalSize <= settings.thresholdChars) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const sessionId = ctx.sessionManager?.getSessionId?.() ?? "session";
|
||||
const id = entryId(sessionId, event.toolCallId);
|
||||
const stored = await storeEntry(
|
||||
storeRoot(),
|
||||
id,
|
||||
source,
|
||||
{ tool: event.toolName, args: event.input },
|
||||
settings.maxEntryBytes,
|
||||
);
|
||||
const overflow = {
|
||||
id,
|
||||
totalLines: stored.totalLines,
|
||||
totalBytes: stored.totalBytes,
|
||||
truncated: current.length > settings.thresholdChars,
|
||||
};
|
||||
|
||||
const images = event.content.filter((part) => part.type !== "text");
|
||||
let text: string;
|
||||
if (current.length > settings.thresholdChars) {
|
||||
// The view always describes the stored file — its line numbers and
|
||||
// totals must match what overflow_query reads, which is not true
|
||||
// of `current` when RTK rewrote or bash pre-truncated the result.
|
||||
const viewInput =
|
||||
stored.storedText !== undefined
|
||||
? collectViewFromText(stored.storedText, settings)
|
||||
: await collectViewFromFile(join(storeRoot(), `${id}.txt`), settings);
|
||||
text = buildView(viewInput, settings, { id, tool: event.toolName });
|
||||
} else {
|
||||
text = `${current}\n[pi-overflow id=${id}: full original output stored (${stored.totalLines} lines, ${formatBytes(stored.totalBytes)}); use overflow_query to inspect it]`;
|
||||
}
|
||||
|
||||
return {
|
||||
content: [{ type: "text", text }, ...images],
|
||||
structuredContent: event.structuredContent,
|
||||
details: { ...event.details, overflow },
|
||||
};
|
||||
} finally {
|
||||
stash.delete(event.toolCallId);
|
||||
}
|
||||
});
|
||||
|
||||
pi.registerTool({
|
||||
name: "overflow_query",
|
||||
label: "Overflow Query",
|
||||
description:
|
||||
"Inspect the full original of a tool output that pi-overflow stored because it was too long to show " +
|
||||
"(search with regex, read line ranges, or extract a JSON path).",
|
||||
promptSnippet: "Inspect a stored oversized tool output by id: regex search, line ranges, or JSON path.",
|
||||
promptGuidelines: [
|
||||
"When a tool result carries a [pi-overflow id=…] marker, the model is seeing a bounded view; the complete original is stored on disk.",
|
||||
"Call overflow_query with the marker's id and a regex, line range, or jsonPath instead of re-running the tool or asking for the full dump.",
|
||||
],
|
||||
parameters: Type.Object({
|
||||
id: Type.String({ description: "12-hex storage id from a [pi-overflow id=…] marker." }),
|
||||
search: Type.Optional(
|
||||
Type.String({ description: "Case-insensitive regular expression matched per line." }),
|
||||
),
|
||||
context: Type.Optional(
|
||||
Type.Number({ minimum: 0, maximum: 10, description: "Context lines around each search hit (default 2)." }),
|
||||
),
|
||||
lines: Type.Optional(
|
||||
Type.Object({
|
||||
start: Type.Number({ minimum: 1, description: "First line to return (1-based, inclusive)." }),
|
||||
end: Type.Number({ minimum: 1, description: "Last line to return (inclusive)." }),
|
||||
}),
|
||||
),
|
||||
jsonPath: Type.Optional(
|
||||
Type.String({ description: "Dot/bracket path into the stored JSON, e.g. snapshot.nodes[3].name." }),
|
||||
),
|
||||
maxResults: Type.Optional(
|
||||
Type.Number({ minimum: 1, maximum: 200, description: "Maximum search hits to report (default 50)." }),
|
||||
),
|
||||
}),
|
||||
async execute(_toolCallId, params) {
|
||||
const result = await runOverflowQuery(storeRoot(), params, settings);
|
||||
return {
|
||||
content: [{ type: "text", text: result.text }],
|
||||
details: { error: result.error === true, truncated: result.truncated },
|
||||
isError: result.error === true,
|
||||
};
|
||||
},
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"name": "pi-overflow",
|
||||
"version": "0.1.0",
|
||||
"description": "Bounded views for oversized tool results: store the full original, show head/signals/tail, recover with overflow_query.",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"license": "MIT",
|
||||
"main": "./index.ts",
|
||||
"exports": {
|
||||
".": "./index.ts"
|
||||
},
|
||||
"files": [
|
||||
"index.ts",
|
||||
"capture.ts",
|
||||
"settings.ts",
|
||||
"src",
|
||||
"README.md",
|
||||
"LICENSE"
|
||||
],
|
||||
"pi": {
|
||||
"extensions": [
|
||||
"./capture.ts",
|
||||
"./index.ts"
|
||||
]
|
||||
},
|
||||
"scripts": {
|
||||
"test": "node --test test/*.test.ts",
|
||||
"check": "node --experimental-strip-types --check index.ts && node --test test/*.test.ts"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@earendil-works/pi-coding-agent": "*",
|
||||
"typebox": "*"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@earendil-works/pi-coding-agent": {
|
||||
"optional": true
|
||||
},
|
||||
"typebox": {
|
||||
"optional": true
|
||||
}
|
||||
},
|
||||
"devDependencies": {
|
||||
"@earendil-works/pi-coding-agent": "latest",
|
||||
"typebox": "latest"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=22.19.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
import { readFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
export interface OverflowSettings {
|
||||
enabled: boolean;
|
||||
thresholdChars: number;
|
||||
headLines: number;
|
||||
tailLines: number;
|
||||
maxSignalLines: number;
|
||||
signalPattern: string;
|
||||
signalRegex: RegExp;
|
||||
maxLineChars: number;
|
||||
exemptTools: string[];
|
||||
retentionDays: number;
|
||||
maxStoreBytes: number;
|
||||
maxEntryBytes: number;
|
||||
}
|
||||
|
||||
export const DEFAULT_SIGNAL_PATTERN =
|
||||
"\\b(error|errors|fail|failed|failure|warn|warning|exception|traceback|panic|fatal)\\b";
|
||||
|
||||
export const OVERFLOW_DEFAULTS = {
|
||||
enabled: true,
|
||||
thresholdChars: 24_000,
|
||||
headLines: 40,
|
||||
tailLines: 120,
|
||||
maxSignalLines: 40,
|
||||
signalPattern: DEFAULT_SIGNAL_PATTERN,
|
||||
maxLineChars: 500,
|
||||
exemptTools: ["read", "edit", "write", "overflow_query", "ask_user_question"],
|
||||
retentionDays: 7,
|
||||
maxStoreBytes: 1_073_741_824,
|
||||
maxEntryBytes: 67_108_864,
|
||||
} as const;
|
||||
|
||||
// Bundle-level `config/` is the single source of truth: the extension reads
|
||||
// `config/pi-overflow.json` from inside the installed bundle (resolved
|
||||
// relative to this package-root source file), never `settings.json`.
|
||||
export const OVERFLOW_CONFIG_PATH = fileURLToPath(
|
||||
new URL("../config/pi-overflow.json", import.meta.url),
|
||||
);
|
||||
|
||||
function isObject(value: unknown): value is Record<string, unknown> {
|
||||
return value !== null && typeof value === "object" && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function bool(value: unknown, fallback: boolean): boolean {
|
||||
return typeof value === "boolean" ? value : fallback;
|
||||
}
|
||||
|
||||
function intInRange(value: unknown, fallback: number, min: number, max: number): number {
|
||||
return typeof value === "number" && Number.isInteger(value) && value >= min && value <= max
|
||||
? value
|
||||
: fallback;
|
||||
}
|
||||
|
||||
function compileSignalPattern(value: unknown): { pattern: string; regex: RegExp } {
|
||||
const pattern = typeof value === "string" && value.length > 0 ? value : DEFAULT_SIGNAL_PATTERN;
|
||||
try {
|
||||
return { pattern, regex: new RegExp(pattern, "i") };
|
||||
} catch {
|
||||
return { pattern: DEFAULT_SIGNAL_PATTERN, regex: new RegExp(DEFAULT_SIGNAL_PATTERN, "i") };
|
||||
}
|
||||
}
|
||||
|
||||
function stringList(value: unknown, fallback: readonly string[]): string[] {
|
||||
if (!Array.isArray(value)) return [...fallback];
|
||||
const list = value.filter((item): item is string => typeof item === "string" && item.length > 0);
|
||||
return [...new Set(list)];
|
||||
}
|
||||
|
||||
/**
|
||||
* Read `config/pi-overflow.json`. Every field is validated independently and
|
||||
* falls back to its bundle default when missing or invalid; an unreadable or
|
||||
* non-object file yields all defaults. Tests inject an explicit path.
|
||||
*/
|
||||
export function readOverflowSettings(configPath: string = OVERFLOW_CONFIG_PATH): OverflowSettings {
|
||||
let parsed: unknown = {};
|
||||
try {
|
||||
parsed = JSON.parse(readFileSync(configPath, "utf8"));
|
||||
} catch {
|
||||
parsed = {};
|
||||
}
|
||||
const config = isObject(parsed) ? parsed : {};
|
||||
const signal = compileSignalPattern(config.signalPattern);
|
||||
return {
|
||||
enabled: bool(config.enabled, OVERFLOW_DEFAULTS.enabled),
|
||||
thresholdChars: intInRange(config.thresholdChars, OVERFLOW_DEFAULTS.thresholdChars, 1, 10_000_000),
|
||||
headLines: intInRange(config.headLines, OVERFLOW_DEFAULTS.headLines, 0, 10_000),
|
||||
tailLines: intInRange(config.tailLines, OVERFLOW_DEFAULTS.tailLines, 0, 10_000),
|
||||
maxSignalLines: intInRange(config.maxSignalLines, OVERFLOW_DEFAULTS.maxSignalLines, 0, 1_000),
|
||||
signalPattern: signal.pattern,
|
||||
signalRegex: signal.regex,
|
||||
maxLineChars: intInRange(config.maxLineChars, OVERFLOW_DEFAULTS.maxLineChars, 10, 100_000),
|
||||
exemptTools: stringList(config.exemptTools, OVERFLOW_DEFAULTS.exemptTools),
|
||||
retentionDays: intInRange(config.retentionDays, OVERFLOW_DEFAULTS.retentionDays, 0, 3_650),
|
||||
maxStoreBytes: intInRange(config.maxStoreBytes, OVERFLOW_DEFAULTS.maxStoreBytes, 1_024, 1_099_511_627_776),
|
||||
maxEntryBytes: intInRange(config.maxEntryBytes, OVERFLOW_DEFAULTS.maxEntryBytes, 1_024, 1_099_511_627_776),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
import { ID_PATTERN, readStoredEntry, type EntryMeta } from "./store.ts";
|
||||
import type { OverflowSettings } from "../settings.ts";
|
||||
|
||||
export interface QueryParams {
|
||||
id: string;
|
||||
search?: string;
|
||||
context?: number;
|
||||
lines?: { start: number; end: number };
|
||||
jsonPath?: string;
|
||||
maxResults?: number;
|
||||
}
|
||||
|
||||
export interface QueryResult {
|
||||
text: string;
|
||||
truncated: boolean;
|
||||
error?: boolean;
|
||||
}
|
||||
|
||||
function fail(text: string): QueryResult {
|
||||
return { text, truncated: false, error: true };
|
||||
}
|
||||
|
||||
function boundOutput(text: string, limit: number): QueryResult {
|
||||
if (text.length <= limit) return { text, truncated: false };
|
||||
const marker = `\n[output truncated at ${limit} chars; narrow the query (smaller line range, tighter regex, or jsonPath)]`;
|
||||
const keep = Math.max(0, limit - marker.length);
|
||||
return { text: text.slice(0, keep) + marker, truncated: true };
|
||||
}
|
||||
|
||||
function metaLine(id: string, meta: EntryMeta | null): string {
|
||||
if (!meta) return `[pi-overflow id=${id}]`;
|
||||
return `[pi-overflow id=${id} tool=${meta.tool} ${meta.totalLines} lines / ${meta.totalBytes} bytes]`;
|
||||
}
|
||||
|
||||
/** Regexes only inspect this prefix of each stored line — a model-supplied
|
||||
* pattern must not backtrack catastrophically across a megabyte-long line. */
|
||||
const MATCH_SCAN_CHARS = 10_000;
|
||||
|
||||
function clipLine(line: string, maxLineChars: number): string {
|
||||
return line.length > maxLineChars ? `${line.slice(0, maxLineChars)}…` : line;
|
||||
}
|
||||
|
||||
function evalJsonPath(root: unknown, path: string): { ok: true; value: unknown } | { ok: false; error: string } {
|
||||
const tokens: Array<string | number> = [];
|
||||
const pattern = /(?:^|\.)([A-Za-z_$][A-Za-z0-9_$]*)|\[\s*(\d+)\s*\]|\[\s*["']([^"']+)["']\s*\]/g;
|
||||
let index = 0;
|
||||
let match: RegExpExecArray | null;
|
||||
while ((match = pattern.exec(path)) !== null) {
|
||||
if (match.index !== index) return { ok: false, error: `cannot parse jsonPath near "${path.slice(index)}"` };
|
||||
index = pattern.lastIndex;
|
||||
if (match[1] !== undefined) tokens.push(match[1]);
|
||||
else if (match[2] !== undefined) tokens.push(Number(match[2]));
|
||||
else tokens.push(match[3] as string);
|
||||
}
|
||||
if (index !== path.length || tokens.length === 0) {
|
||||
return { ok: false, error: `cannot parse jsonPath "${path}"` };
|
||||
}
|
||||
let value: unknown = root;
|
||||
for (const token of tokens) {
|
||||
if (value === null || value === undefined) {
|
||||
return { ok: false, error: `jsonPath stops at null/undefined before "${String(token)}"` };
|
||||
}
|
||||
if (typeof token === "number") {
|
||||
if (!Array.isArray(value) || token >= value.length) {
|
||||
return { ok: false, error: `index [${token}] out of range` };
|
||||
}
|
||||
value = value[token];
|
||||
} else {
|
||||
if (typeof value !== "object" || Array.isArray(value)) {
|
||||
return { ok: false, error: `"${token}" is not a property of this value` };
|
||||
}
|
||||
const record = value as Record<string, unknown>;
|
||||
if (!(token in record)) return { ok: false, error: `property "${token}" not found` };
|
||||
value = record[token];
|
||||
}
|
||||
}
|
||||
return { ok: true, value };
|
||||
}
|
||||
|
||||
function mergeRanges(hits: number[], context: number): Array<{ start: number; end: number }> {
|
||||
const ranges: Array<{ start: number; end: number }> = [];
|
||||
for (const hit of hits) {
|
||||
const start = Math.max(1, hit - context);
|
||||
const end = hit + context;
|
||||
const last = ranges[ranges.length - 1];
|
||||
if (last && start <= last.end + 1) {
|
||||
last.end = Math.max(last.end, end);
|
||||
} else {
|
||||
ranges.push({ start, end });
|
||||
}
|
||||
}
|
||||
return ranges;
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute an overflow_query against the store. Never throws on bad input —
|
||||
* invalid ids, regexes, JSON, and paths all come back as clear error text.
|
||||
*/
|
||||
export async function runOverflowQuery(
|
||||
root: string,
|
||||
params: QueryParams,
|
||||
settings: OverflowSettings,
|
||||
): Promise<QueryResult> {
|
||||
if (typeof params.id !== "string" || !ID_PATTERN.test(params.id)) {
|
||||
return fail(`invalid overflow id "${String(params.id)}" — expected 12 lowercase hex chars from a [pi-overflow id=…] marker`);
|
||||
}
|
||||
let stored: { text: string; meta: EntryMeta | null };
|
||||
try {
|
||||
stored = await readStoredEntry(root, params.id);
|
||||
} catch {
|
||||
return fail(`no stored output for id ${params.id} (it may have been cleaned up or never stored)`);
|
||||
}
|
||||
const lines = stored.text.split("\n");
|
||||
const head = metaLine(params.id, stored.meta);
|
||||
const context = Math.min(Math.max(Math.floor(params.context ?? 2), 0), 10);
|
||||
const maxResults = Math.min(Math.max(Math.floor(params.maxResults ?? 50), 1), 200);
|
||||
|
||||
if (typeof params.search === "string") {
|
||||
let regex: RegExp;
|
||||
try {
|
||||
regex = new RegExp(params.search, "i");
|
||||
} catch (error) {
|
||||
return fail(`invalid search regex: ${error instanceof Error ? error.message : String(error)}`);
|
||||
}
|
||||
const hits: number[] = [];
|
||||
for (let index = 0; index < lines.length && hits.length < maxResults; index++) {
|
||||
regex.lastIndex = 0;
|
||||
if (regex.test(lines[index].slice(0, MATCH_SCAN_CHARS))) hits.push(index + 1);
|
||||
}
|
||||
if (hits.length === 0) {
|
||||
return boundOutput(`${head}\nno lines match /${params.search}/i`, settings.thresholdChars);
|
||||
}
|
||||
const parts = [head, `${hits.length}${hits.length >= maxResults ? "+" : ""} matching lines:`];
|
||||
for (const range of mergeRanges(hits, context)) {
|
||||
const end = Math.min(range.end, lines.length);
|
||||
parts.push(`--- lines ${range.start}-${end} ---`);
|
||||
for (let line = range.start; line <= end; line++) {
|
||||
parts.push(`L${line}: ${clipLine(lines[line - 1], settings.maxLineChars)}`);
|
||||
}
|
||||
}
|
||||
return boundOutput(parts.join("\n"), settings.thresholdChars);
|
||||
}
|
||||
|
||||
if (params.lines && typeof params.lines === "object") {
|
||||
const start = Math.floor(params.lines.start);
|
||||
const end = Math.floor(params.lines.end);
|
||||
if (!Number.isFinite(start) || !Number.isFinite(end) || start < 1 || end < start) {
|
||||
return fail(`invalid line range {start: ${params.lines.start}, end: ${params.lines.end}} — need 1 <= start <= end`);
|
||||
}
|
||||
const clippedEnd = Math.min(end, lines.length);
|
||||
if (start > lines.length) {
|
||||
return fail(`start ${start} is beyond the stored ${lines.length} lines`);
|
||||
}
|
||||
const parts = [head, `--- lines ${start}-${clippedEnd} ---`];
|
||||
for (let line = start; line <= clippedEnd; line++) {
|
||||
parts.push(`L${line}: ${clipLine(lines[line - 1], settings.maxLineChars)}`);
|
||||
}
|
||||
return boundOutput(parts.join("\n"), settings.thresholdChars);
|
||||
}
|
||||
|
||||
if (typeof params.jsonPath === "string") {
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(stored.text);
|
||||
} catch (error) {
|
||||
return fail(`stored output is not valid JSON: ${error instanceof Error ? error.message : String(error)}`);
|
||||
}
|
||||
const result = evalJsonPath(parsed, params.jsonPath);
|
||||
if (!result.ok) return fail(`jsonPath error: ${result.error}`);
|
||||
let rendered: string;
|
||||
try {
|
||||
rendered = JSON.stringify(result.value, null, 2) ?? "undefined";
|
||||
} catch {
|
||||
rendered = String(result.value);
|
||||
}
|
||||
return boundOutput(`${head}\njsonPath ${params.jsonPath}:\n${rendered}`, settings.thresholdChars);
|
||||
}
|
||||
|
||||
// No selector: metadata plus a bounded head/tail preview.
|
||||
const headCount = Math.min(settings.headLines, lines.length);
|
||||
const tailCount = Math.min(settings.tailLines, Math.max(0, lines.length - headCount));
|
||||
const parts = [head, `--- lines 1-${headCount} ---`, ...lines.slice(0, headCount)];
|
||||
if (tailCount > 0) {
|
||||
parts.push(`… ${lines.length - headCount - tailCount} lines omitted …`);
|
||||
parts.push(`--- lines ${lines.length - tailCount + 1}-${lines.length} ---`, ...lines.slice(lines.length - tailCount));
|
||||
}
|
||||
return boundOutput(parts.join("\n"), settings.thresholdChars);
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
/**
|
||||
* Cross-extension staging for pre-bound originals. `capture.ts` (loaded
|
||||
* first) stashes the raw tool_result text; `index.ts` (loaded last) consumes
|
||||
* it after every other handler — RTK in particular — has run. Sharing goes
|
||||
* through `globalThis[Symbol.for(...)]`, the same mechanism pi-compaction
|
||||
* uses for its metrics service.
|
||||
*/
|
||||
|
||||
export interface StashedResult {
|
||||
rawText: string;
|
||||
fullOutputPath?: string;
|
||||
}
|
||||
|
||||
const STASH_KEY = Symbol.for("my-pi.pi-overflow.stash.v1");
|
||||
const STASH_LIMIT = 64;
|
||||
|
||||
export function getStash(): Map<string, StashedResult> {
|
||||
const scope = globalThis as Record<symbol, unknown>;
|
||||
let stash = scope[STASH_KEY] as Map<string, StashedResult> | undefined;
|
||||
if (!(stash instanceof Map)) {
|
||||
stash = new Map<string, StashedResult>();
|
||||
scope[STASH_KEY] = stash;
|
||||
}
|
||||
return stash;
|
||||
}
|
||||
|
||||
export function stashResult(stash: Map<string, StashedResult>, toolCallId: string, entry: StashedResult): void {
|
||||
stash.delete(toolCallId);
|
||||
stash.set(toolCallId, entry);
|
||||
while (stash.size > STASH_LIMIT) {
|
||||
const oldest = stash.keys().next().value;
|
||||
if (oldest === undefined) break;
|
||||
stash.delete(oldest);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,267 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { createReadStream, createWriteStream, type WriteStream } from "node:fs";
|
||||
import { chmod, mkdir, open, readFile, readdir, rename, stat, unlink, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
|
||||
export interface EntryMeta {
|
||||
tool: string;
|
||||
argsSummary: string;
|
||||
totalLines: number;
|
||||
totalBytes: number;
|
||||
createdAt: string;
|
||||
omittedMiddle?: boolean;
|
||||
}
|
||||
|
||||
export interface StoredEntry {
|
||||
id: string;
|
||||
totalLines: number;
|
||||
totalBytes: number;
|
||||
/** Present when the source was text: the exact content written to disk. */
|
||||
storedText?: string;
|
||||
}
|
||||
|
||||
export const ID_PATTERN = /^[0-9a-f]{12}$/;
|
||||
|
||||
const TMP_MAX_AGE_MS = 3_600_000;
|
||||
|
||||
export function entryId(sessionId: string, toolCallId: string): string {
|
||||
return createHash("sha1").update(`${sessionId}\0${toolCallId}`).digest("hex").slice(0, 12);
|
||||
}
|
||||
|
||||
async function ensureDir(dir: string): Promise<void> {
|
||||
await mkdir(dir, { recursive: true, mode: 0o700 });
|
||||
try {
|
||||
await chmod(dir, 0o700);
|
||||
} catch {
|
||||
// Best-effort permission tightening only.
|
||||
}
|
||||
}
|
||||
|
||||
function tmpPath(path: string): string {
|
||||
return `${path}.tmp-${process.pid}-${Math.random().toString(36).slice(2)}`;
|
||||
}
|
||||
|
||||
async function writeFileAtomic(path: string, content: string | Buffer): Promise<void> {
|
||||
const tmp = tmpPath(path);
|
||||
await writeFile(tmp, content, { mode: 0o600 });
|
||||
await rename(tmp, path);
|
||||
}
|
||||
|
||||
function countNewlines(text: string): number {
|
||||
let count = 0;
|
||||
for (let index = 0; index < text.length; index++) {
|
||||
if (text.charCodeAt(index) === 10) count++;
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
export function lineCount(text: string): number {
|
||||
if (text.length === 0) return 0;
|
||||
return countNewlines(text) + (text.endsWith("\n") ? 0 : 1);
|
||||
}
|
||||
|
||||
function truncateSummary(value: string, maxChars = 300): string {
|
||||
return value.length > maxChars ? `${value.slice(0, maxChars)}…` : value;
|
||||
}
|
||||
|
||||
export function argsSummary(input: unknown): string {
|
||||
try {
|
||||
return truncateSummary(JSON.stringify(input) ?? "undefined");
|
||||
} catch {
|
||||
return "[unserializable]";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Copy `sourcePath` into `<root>/<id>.txt` without holding it all in memory.
|
||||
* Files larger than `maxEntryBytes` keep the first and last halves with an
|
||||
* omission marker between them.
|
||||
*/
|
||||
async function copyFileBounded(
|
||||
sourcePath: string,
|
||||
destPath: string,
|
||||
maxEntryBytes: number,
|
||||
): Promise<{ bytes: number; omittedMiddle: boolean }> {
|
||||
const info = await stat(sourcePath);
|
||||
if (!info.isFile()) throw new Error("full output path is not a file");
|
||||
const size = info.size;
|
||||
if (size <= maxEntryBytes) {
|
||||
const tmp = tmpPath(destPath);
|
||||
const out: WriteStream = createWriteStream(tmp, { mode: 0o600 });
|
||||
await pipeline(createReadStream(sourcePath), out);
|
||||
await rename(tmp, destPath);
|
||||
return { bytes: size, omittedMiddle: false };
|
||||
}
|
||||
const half = Math.floor(maxEntryBytes / 2);
|
||||
const omitted = size - half * 2;
|
||||
const source = await open(sourcePath, "r");
|
||||
try {
|
||||
const head = Buffer.alloc(half);
|
||||
const tail = Buffer.alloc(half);
|
||||
await source.read(head, 0, half, 0);
|
||||
await source.read(tail, 0, half, size - half);
|
||||
const marker = Buffer.from(`\n… ${omitted} bytes omitted (exceeded ${maxEntryBytes}-byte store limit) …\n`);
|
||||
await writeFileAtomic(destPath, Buffer.concat([head, marker, tail]));
|
||||
} finally {
|
||||
await source.close();
|
||||
}
|
||||
return { bytes: size, omittedMiddle: true };
|
||||
}
|
||||
|
||||
async function countFileLines(path: string): Promise<number> {
|
||||
let count = 0;
|
||||
let lastByte = -1;
|
||||
let seen = false;
|
||||
for await (const chunk of createReadStream(path)) {
|
||||
seen = true;
|
||||
const buffer = chunk as Buffer;
|
||||
for (let index = 0; index < buffer.length; index++) {
|
||||
if (buffer[index] === 10) count++;
|
||||
}
|
||||
if (buffer.length > 0) lastByte = buffer[buffer.length - 1];
|
||||
}
|
||||
if (!seen) return 0;
|
||||
return count + (lastByte === 10 ? 0 : 1);
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist an original tool output under `<root>/<id>.txt` plus `<id>.json`
|
||||
* metadata. Directories are 0700, files 0600, writes are atomic. The caller
|
||||
* chooses `root` (defaults come from getAgentDir() in index.ts; tests inject
|
||||
* a temp directory).
|
||||
*/
|
||||
export async function storeEntry(
|
||||
root: string,
|
||||
id: string,
|
||||
source: { kind: "text"; text: string } | { kind: "file"; path: string },
|
||||
meta: { tool: string; args: unknown },
|
||||
maxEntryBytes: number,
|
||||
): Promise<StoredEntry> {
|
||||
await ensureDir(root);
|
||||
const textPath = join(root, `${id}.txt`);
|
||||
const metaPath = join(root, `${id}.json`);
|
||||
let totalLines = 0;
|
||||
let totalBytes = 0;
|
||||
let omittedMiddle = false;
|
||||
let storedText: string | undefined;
|
||||
if (source.kind === "text") {
|
||||
let text = source.text;
|
||||
const bytes = Buffer.byteLength(text, "utf8");
|
||||
totalBytes = bytes;
|
||||
if (bytes > maxEntryBytes) {
|
||||
const halfChars = Math.floor(maxEntryBytes / 2);
|
||||
const omitted = bytes - halfChars * 2;
|
||||
text = `${text.slice(0, halfChars)}\n… ${omitted} bytes omitted (exceeded ${maxEntryBytes}-byte store limit) …\n${text.slice(text.length - halfChars)}`;
|
||||
omittedMiddle = true;
|
||||
}
|
||||
totalLines = lineCount(text);
|
||||
storedText = text;
|
||||
await writeFileAtomic(textPath, text);
|
||||
} else {
|
||||
const copied = await copyFileBounded(source.path, textPath, maxEntryBytes);
|
||||
totalBytes = copied.bytes;
|
||||
omittedMiddle = copied.omittedMiddle;
|
||||
totalLines = await countFileLines(textPath).catch(() => 0);
|
||||
}
|
||||
const entry: EntryMeta = {
|
||||
tool: meta.tool,
|
||||
argsSummary: argsSummary(meta.args),
|
||||
totalLines,
|
||||
totalBytes,
|
||||
createdAt: new Date().toISOString(),
|
||||
...(omittedMiddle ? { omittedMiddle: true } : {}),
|
||||
};
|
||||
await writeFileAtomic(metaPath, `${JSON.stringify(entry, null, 2)}\n`);
|
||||
return { id, totalLines, totalBytes, ...(storedText !== undefined ? { storedText } : {}) };
|
||||
}
|
||||
|
||||
export async function readStoredEntry(root: string, id: string): Promise<{ text: string; meta: EntryMeta | null }> {
|
||||
if (!ID_PATTERN.test(id)) throw new Error(`invalid overflow id: ${id}`);
|
||||
const textPath = join(root, `${id}.txt`);
|
||||
const metaPath = join(root, `${id}.json`);
|
||||
const text = await readFile(textPath, "utf8");
|
||||
let meta: EntryMeta | null = null;
|
||||
try {
|
||||
const parsed = JSON.parse(await readFile(metaPath, "utf8"));
|
||||
if (parsed && typeof parsed === "object") meta = parsed as EntryMeta;
|
||||
} catch {
|
||||
meta = null;
|
||||
}
|
||||
return { text, meta };
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete tmp files left by interrupted writes (older than one hour), entries
|
||||
* older than `retentionDays`, then evict oldest-first until the store is
|
||||
* under `maxStoreBytes`. All failures are swallowed — cleanup must never
|
||||
* break a session.
|
||||
*/
|
||||
export async function cleanStore(root: string, retentionDays: number, maxStoreBytes: number): Promise<void> {
|
||||
try {
|
||||
const now = Date.now();
|
||||
const cutoff = now - retentionDays * 86_400_000;
|
||||
const names = await readdir(root);
|
||||
// Stale tmp files from interrupted writes never become entries; remove
|
||||
// them once they are clearly abandoned.
|
||||
for (const name of names) {
|
||||
if (!name.includes(".tmp-")) continue;
|
||||
try {
|
||||
const info = await stat(join(root, name));
|
||||
if (info.isFile() && info.mtimeMs < now - TMP_MAX_AGE_MS) {
|
||||
await unlink(join(root, name));
|
||||
}
|
||||
} catch {
|
||||
// Already gone or unreadable — nothing to do.
|
||||
}
|
||||
}
|
||||
const ids = new Set(
|
||||
names
|
||||
.filter((name) => !name.includes(".tmp-"))
|
||||
.map((name) => name.replace(/\.(txt|json)$/, "")),
|
||||
);
|
||||
const entries: Array<{ id: string; bytes: number; mtime: number }> = [];
|
||||
for (const id of ids) {
|
||||
let bytes = 0;
|
||||
let mtime = 0;
|
||||
let seen = false;
|
||||
for (const suffix of ["txt", "json"] as const) {
|
||||
const file = join(root, `${id}.${suffix}`);
|
||||
try {
|
||||
const info = await stat(file);
|
||||
bytes += info.size;
|
||||
mtime = Math.max(mtime, info.mtimeMs);
|
||||
seen = true;
|
||||
} catch {
|
||||
// Missing half of a pair still counts toward cleanup.
|
||||
}
|
||||
}
|
||||
if (seen) entries.push({ id, bytes, mtime });
|
||||
}
|
||||
const remove = async (id: string) => {
|
||||
for (const suffix of ["txt", "json"] as const) {
|
||||
await unlink(join(root, `${id}.${suffix}`)).catch(() => {});
|
||||
}
|
||||
};
|
||||
let total = 0;
|
||||
const survivors: typeof entries = [];
|
||||
for (const entry of entries) {
|
||||
if (entry.mtime < cutoff) {
|
||||
await remove(entry.id);
|
||||
} else {
|
||||
total += entry.bytes;
|
||||
survivors.push(entry);
|
||||
}
|
||||
}
|
||||
if (total > maxStoreBytes) {
|
||||
survivors.sort((a, b) => a.mtime - b.mtime);
|
||||
for (const entry of survivors) {
|
||||
if (total <= maxStoreBytes) break;
|
||||
await remove(entry.id);
|
||||
total -= entry.bytes;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Cleanup failures never propagate.
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,246 @@
|
||||
import { createReadStream } from "node:fs";
|
||||
import { open, stat } from "node:fs/promises";
|
||||
import type { OverflowSettings } from "../settings.ts";
|
||||
|
||||
export interface ViewMeta {
|
||||
id: string;
|
||||
tool: string;
|
||||
}
|
||||
|
||||
export interface SignalHit {
|
||||
line: number;
|
||||
text: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pre-collected view segments of the STORED original (never the current —
|
||||
* possibly RTK- or bash-truncated — text). Line numbers here are the stored
|
||||
* file's line numbers, which is exactly what overflow_query reads.
|
||||
*/
|
||||
export interface ViewInput {
|
||||
head: string[];
|
||||
signals: SignalHit[];
|
||||
tail: string[];
|
||||
totalLines: number;
|
||||
totalBytes: number;
|
||||
/** First ~thresholdChars chars of the stored content, for char fallback. */
|
||||
rawHead: string;
|
||||
/** Last ~thresholdChars chars of the stored content, for char fallback. */
|
||||
rawTail: string;
|
||||
}
|
||||
|
||||
/** Regexes only inspect this prefix of each line — catastrophic backtracking
|
||||
* on a giant single-line blob is not allowed to stall the handler. */
|
||||
export const MATCH_SCAN_CHARS = 10_000;
|
||||
|
||||
function clip(line: string, maxLineChars: number): string {
|
||||
return line.length > maxLineChars ? `${line.slice(0, maxLineChars)}…` : line;
|
||||
}
|
||||
|
||||
/** Slice a text original into the same view segments a file scan produces. */
|
||||
export function collectViewFromText(text: string, settings: OverflowSettings): ViewInput {
|
||||
const lines = text.split("\n");
|
||||
const total = lines.length;
|
||||
const headCount = Math.min(settings.headLines, total);
|
||||
const tailCount = Math.min(settings.tailLines, Math.max(0, total - headCount));
|
||||
const head = lines.slice(0, headCount).map((line) => clip(line, settings.maxLineChars));
|
||||
const tail = tailCount > 0 ? lines.slice(total - tailCount).map((line) => clip(line, settings.maxLineChars)) : [];
|
||||
const signals: SignalHit[] = [];
|
||||
const regex = new RegExp(settings.signalRegex.source, "i");
|
||||
for (
|
||||
let index = headCount;
|
||||
index < total - tailCount && signals.length < settings.maxSignalLines;
|
||||
index++
|
||||
) {
|
||||
regex.lastIndex = 0;
|
||||
if (regex.test(lines[index].slice(0, MATCH_SCAN_CHARS))) {
|
||||
signals.push({ line: index + 1, text: clip(lines[index], settings.maxLineChars) });
|
||||
}
|
||||
}
|
||||
return {
|
||||
head,
|
||||
signals,
|
||||
tail,
|
||||
totalLines: total,
|
||||
totalBytes: Buffer.byteLength(text, "utf8"),
|
||||
rawHead: text.slice(0, settings.thresholdChars),
|
||||
rawTail: text.slice(Math.max(0, text.length - settings.thresholdChars)),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream the stored file line by line — never materialize it. Collects the
|
||||
* first headLines lines, keeps the last tailLines lines in a ring buffer,
|
||||
* and records at most maxSignalLines signal hits (line number + clipped
|
||||
* text); hits landing inside the final tail range are dropped afterwards.
|
||||
*/
|
||||
export async function collectViewFromFile(path: string, settings: OverflowSettings): Promise<ViewInput> {
|
||||
const head: string[] = [];
|
||||
const tail: string[] = [];
|
||||
const signals: SignalHit[] = [];
|
||||
let totalLines = 0;
|
||||
let remainder = "";
|
||||
const regex = new RegExp(settings.signalRegex.source, "i");
|
||||
|
||||
const handleLine = (line: string) => {
|
||||
totalLines++;
|
||||
if (head.length < settings.headLines) {
|
||||
head.push(clip(line, settings.maxLineChars));
|
||||
} else if (signals.length < settings.maxSignalLines) {
|
||||
regex.lastIndex = 0;
|
||||
if (regex.test(line.slice(0, MATCH_SCAN_CHARS))) {
|
||||
signals.push({ line: totalLines, text: clip(line, settings.maxLineChars) });
|
||||
}
|
||||
}
|
||||
tail.push(clip(line, settings.maxLineChars));
|
||||
if (tail.length > settings.tailLines) tail.shift();
|
||||
};
|
||||
|
||||
for await (const chunk of createReadStream(path, { encoding: "utf8" })) {
|
||||
remainder += chunk;
|
||||
let index = remainder.indexOf("\n");
|
||||
while (index >= 0) {
|
||||
handleLine(remainder.slice(0, index));
|
||||
remainder = remainder.slice(index + 1);
|
||||
index = remainder.indexOf("\n");
|
||||
}
|
||||
}
|
||||
if (remainder.length > 0) handleLine(remainder);
|
||||
|
||||
const tailStart = totalLines - tail.length + 1;
|
||||
const middleSignals = signals.filter((hit) => hit.line < tailStart);
|
||||
|
||||
const info = await stat(path);
|
||||
const take = Math.min(settings.thresholdChars, info.size);
|
||||
let rawHead = "";
|
||||
let rawTail = "";
|
||||
if (take > 0) {
|
||||
const handle = await open(path, "r");
|
||||
try {
|
||||
const headBuffer = Buffer.alloc(take);
|
||||
const tailBuffer = Buffer.alloc(take);
|
||||
await handle.read(headBuffer, 0, take, 0);
|
||||
await handle.read(tailBuffer, 0, take, info.size - take);
|
||||
rawHead = headBuffer.toString("utf8");
|
||||
rawTail = tailBuffer.toString("utf8");
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
head,
|
||||
signals: middleSignals,
|
||||
tail,
|
||||
totalLines,
|
||||
totalBytes: info.size,
|
||||
rawHead,
|
||||
rawTail,
|
||||
};
|
||||
}
|
||||
|
||||
function formatBytes(bytes: number): string {
|
||||
if (bytes < 1024) return `${bytes} B`;
|
||||
if (bytes < 1_048_576) return `${(bytes / 1024).toFixed(1)} KB`;
|
||||
if (bytes < 1_073_741_824) return `${(bytes / 1_048_576).toFixed(1)} MB`;
|
||||
return `${(bytes / 1_073_741_824).toFixed(2)} GB`;
|
||||
}
|
||||
|
||||
function signalDescription(pattern: string): string {
|
||||
const compact = pattern.replace(/\\b/g, "").replace(/[()]/g, "");
|
||||
return compact.length > 60 ? `${compact.slice(0, 60)}…` : compact;
|
||||
}
|
||||
|
||||
/**
|
||||
* Deterministic bounded view of the stored original: head lines, signal
|
||||
* lines from the middle section, and tail lines — all with the stored
|
||||
* file's line numbers. Output never exceeds settings.thresholdChars and the
|
||||
* tail of the stored content is always visible.
|
||||
*
|
||||
* Shrink order under budget pressure is tail > signals > head: head first
|
||||
* collapses to min(10, headLines), signals halve stepwise to 10 then 0, then
|
||||
* head/tail shrink ~35/65 with tail preferred, and char truncation of the
|
||||
* stored head/tail is the final fallback.
|
||||
*/
|
||||
export function buildView(input: ViewInput, settings: OverflowSettings, meta: ViewMeta): string {
|
||||
const budget = settings.thresholdChars;
|
||||
const header =
|
||||
`[pi-overflow id=${meta.id} tool=${meta.tool} ${input.totalLines} lines / ${formatBytes(input.totalBytes)}` +
|
||||
` — full output stored; use overflow_query to search or read ranges]`;
|
||||
|
||||
const render = (headLines: string[], signals: SignalHit[], tailLines: string[]): string => {
|
||||
const parts: string[] = [header];
|
||||
if (headLines.length > 0) {
|
||||
parts.push(`--- lines 1-${headLines.length} ---`, ...headLines);
|
||||
}
|
||||
if (signals.length > 0) {
|
||||
parts.push(
|
||||
`--- signals (lines matching ${signalDescription(settings.signalPattern)}, up to ${settings.maxSignalLines}) ---`,
|
||||
...signals.map((hit) => `L${hit.line}: ${hit.text}`),
|
||||
);
|
||||
}
|
||||
if (tailLines.length > 0) {
|
||||
const first = input.totalLines - tailLines.length + 1;
|
||||
parts.push(`--- lines ${first}-${input.totalLines} ---`, ...tailLines);
|
||||
}
|
||||
return parts.join("\n");
|
||||
};
|
||||
|
||||
const charFallback = (): string => {
|
||||
const markerReserve = 64;
|
||||
const bodyBudget = Math.max(64, budget - header.length - markerReserve);
|
||||
const headChars = Math.floor(bodyBudget * 0.35);
|
||||
const tailChars = bodyBudget - headChars;
|
||||
const headText = input.rawHead.slice(0, headChars);
|
||||
const tailText = input.rawTail.slice(Math.max(0, input.rawTail.length - tailChars));
|
||||
const omitted = Math.max(0, input.totalBytes - headText.length - tailText.length);
|
||||
return `${header}\n${headText}\n… ${omitted} chars omitted …\n${tailText}`;
|
||||
};
|
||||
|
||||
let headList = input.head;
|
||||
let signalList = input.signals;
|
||||
let tailList = input.tail;
|
||||
|
||||
let result = render(headList, signalList, tailList);
|
||||
if (result.length <= budget) return result;
|
||||
|
||||
// 1. Collapse head to its minimum useful window.
|
||||
headList = headList.slice(0, Math.min(10, settings.headLines));
|
||||
result = render(headList, signalList, tailList);
|
||||
if (result.length <= budget) return result;
|
||||
|
||||
// 2. Halve signals stepwise until 10 remain, then drop them entirely.
|
||||
while (signalList.length > 10) {
|
||||
signalList = signalList.slice(0, Math.ceil(signalList.length / 2));
|
||||
result = render(headList, signalList, tailList);
|
||||
if (result.length <= budget) return result;
|
||||
}
|
||||
if (signalList.length > 0) {
|
||||
signalList = [];
|
||||
result = render(headList, signalList, tailList);
|
||||
if (result.length <= budget) return result;
|
||||
}
|
||||
|
||||
// When head+tail already span every stored line there is no line-level
|
||||
// slack left: go straight to char truncation so the ending survives.
|
||||
if (input.head.length + input.tail.length >= input.totalLines) return charFallback();
|
||||
|
||||
// 3. Shrink head/tail ~35/65; tail keeps the larger share and always keeps
|
||||
// its last line (the stored ending).
|
||||
while (headList.length + tailList.length > 0) {
|
||||
result = render(headList, signalList, tailList);
|
||||
if (result.length <= budget) return result;
|
||||
const headChars = headList.reduce((sum, line) => sum + line.length + 1, 0);
|
||||
const tailChars = tailList.reduce((sum, line) => sum + line.length + 1, 0);
|
||||
if (tailList.length > 1 && (tailChars >= headChars * (65 / 35) || headList.length === 0)) {
|
||||
tailList.shift();
|
||||
} else if (headList.length > 0) {
|
||||
headList.pop();
|
||||
} else {
|
||||
tailList.shift();
|
||||
}
|
||||
}
|
||||
|
||||
// 4. Last resort: keep the stored text's own head 35% / tail 65%.
|
||||
return charFallback();
|
||||
}
|
||||
@@ -0,0 +1,314 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, readFile, readdir, stat, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import overflowCapture from "../capture.ts";
|
||||
import overflowExtension from "../index.ts";
|
||||
import { getStash } from "../src/stash.ts";
|
||||
import { readOverflowSettings, OVERFLOW_DEFAULTS, type OverflowSettings } from "../settings.ts";
|
||||
|
||||
function harness() {
|
||||
const handlers = new Map<string, Function[]>();
|
||||
const tools = new Map<string, any>();
|
||||
const pi: any = {
|
||||
on(name: string, handler: Function) {
|
||||
const list = handlers.get(name) ?? [];
|
||||
list.push(handler);
|
||||
handlers.set(name, list);
|
||||
},
|
||||
registerTool(tool: any) { tools.set(tool.name, tool); },
|
||||
};
|
||||
return { pi, handlers, tools };
|
||||
}
|
||||
|
||||
function ctx() {
|
||||
return { sessionManager: { getSessionId: () => "session-test" } } as any;
|
||||
}
|
||||
|
||||
function settings(overrides: Partial<OverflowSettings> = {}): OverflowSettings {
|
||||
return { ...readOverflowSettings("/nonexistent/pi-overflow.json"), ...overrides };
|
||||
}
|
||||
|
||||
function textEvent(toolCallId: string, text: string, extra: Record<string, unknown> = {}) {
|
||||
return {
|
||||
toolCallId,
|
||||
toolName: "bash",
|
||||
input: { command: "run" },
|
||||
content: [{ type: "text", text }],
|
||||
details: {},
|
||||
...extra,
|
||||
};
|
||||
}
|
||||
|
||||
async function runHandlers(handlers: Map<string, Function[]>, event: any, context: any) {
|
||||
let current = event;
|
||||
for (const handler of handlers.get("tool_result") ?? []) {
|
||||
const result = await handler(current, context);
|
||||
if (result) current = { ...current, ...result };
|
||||
}
|
||||
return current;
|
||||
}
|
||||
|
||||
async function makeStore() {
|
||||
return mkdtemp(join(tmpdir(), "pi-overflow-test-"));
|
||||
}
|
||||
|
||||
test("capture stashes the original before another handler shortens it; bound stage stores original but views current", async () => {
|
||||
const root = await makeStore();
|
||||
getStash().clear();
|
||||
const capture = harness();
|
||||
overflowCapture(capture.pi);
|
||||
const bound = harness();
|
||||
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
|
||||
|
||||
const big = Array.from({ length: 3000 }, (_, i) => `orig ${i}`).join("\n");
|
||||
const event = textEvent("call-1", big);
|
||||
const afterCapture = await runHandlers(capture.handlers, event, ctx());
|
||||
assert.equal(afterCapture, event, "capture must not modify the result");
|
||||
assert.ok(getStash().has("call-1"), "capture must stash the original");
|
||||
|
||||
// Simulate RTK: replace content with a shortened version that is still
|
||||
// over threshold.
|
||||
const rtkShortened = Array.from({ length: 900 }, (_, i) => `rtk ${i}`).join("\n") + "\n" + "z".repeat(30_000);
|
||||
const mid = { ...afterCapture, content: [{ type: "text", text: rtkShortened }] };
|
||||
const final = await runHandlers(bound.handlers, mid, ctx());
|
||||
|
||||
assert.ok(!getStash().has("call-1"), "stash entry must be consumed");
|
||||
const text = final.content[0].text;
|
||||
assert.match(text, /\[pi-overflow id=[0-9a-f]{12} tool=bash/);
|
||||
assert.ok(text.length <= settings().thresholdChars);
|
||||
|
||||
const id = final.details.overflow.id;
|
||||
const stored = await readFile(join(root, `${id}.txt`), "utf8");
|
||||
assert.ok(stored.includes("orig 0"), "store must keep the ORIGINAL, not the RTK-shortened text");
|
||||
assert.ok(stored.includes("orig 2999"));
|
||||
assert.equal(final.details.overflow.truncated, true);
|
||||
});
|
||||
|
||||
test("details.fullOutputPath file is used as the original", async () => {
|
||||
const root = await makeStore();
|
||||
getStash().clear();
|
||||
const bound = harness();
|
||||
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
|
||||
|
||||
const spill = join(root, "spill.txt");
|
||||
await writeFile(spill, Array.from({ length: 5000 }, (_, i) => `spill ${i}`).join("\n"));
|
||||
const event = textEvent("call-2", "x".repeat(30_000), { details: { fullOutputPath: spill } });
|
||||
const final = await runHandlers(bound.handlers, event, ctx());
|
||||
const stored = await readFile(join(root, `${final.details.overflow.id}.txt`), "utf8");
|
||||
assert.ok(stored.includes("spill 0"));
|
||||
assert.ok(stored.includes("spill 4999"));
|
||||
});
|
||||
|
||||
test("view line numbers track the STORED file, not the current text", async () => {
|
||||
const root = await makeStore();
|
||||
getStash().clear();
|
||||
const bound = harness();
|
||||
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
|
||||
|
||||
// bash spill: 5000 lines on disk, but the in-context result only carries
|
||||
// the last ~100 lines (still over threshold via padding).
|
||||
const spill = join(root, "spill-5000.txt");
|
||||
await writeFile(
|
||||
spill,
|
||||
Array.from({ length: 5000 }, (_, i) => `spill line ${i + 1}`).join("\n"),
|
||||
);
|
||||
const current = Array.from({ length: 100 }, (_, i) => `spill line ${4901 + i}`).join("\n") + "\n" + "pad".repeat(10_000);
|
||||
const event = textEvent("call-lines", current, { details: { fullOutputPath: spill } });
|
||||
const final = await runHandlers(bound.handlers, event, ctx());
|
||||
|
||||
const text = final.content[0].text;
|
||||
const id = final.details.overflow.id;
|
||||
assert.match(text, /--- lines 4881-5000 ---/, "tail range must come from the stored 5000-line file");
|
||||
assert.match(text, /spill line 5000/);
|
||||
assert.match(text, /\[pi-overflow id=[0-9a-f]{12} tool=bash 5000 lines/);
|
||||
|
||||
// overflow_query with the view's tail range returns the same lines.
|
||||
const tool = bound.tools.get("overflow_query");
|
||||
const result = await tool.execute("q1", { id, lines: { start: 4881, end: 4883 } });
|
||||
assert.match(result.content[0].text, /L4881: spill line 4881/);
|
||||
assert.match(result.content[0].text, /L4883: spill line 4883/);
|
||||
});
|
||||
|
||||
test("RTK scenario: view line numbers come from the stored original text", async () => {
|
||||
const root = await makeStore();
|
||||
getStash().clear();
|
||||
const capture = harness();
|
||||
overflowCapture(capture.pi);
|
||||
const bound = harness();
|
||||
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
|
||||
|
||||
const original = Array.from({ length: 3000 }, (_, i) => `orig ${i + 1}`).join("\n");
|
||||
await runHandlers(capture.handlers, textEvent("call-rtk", original), ctx());
|
||||
// RTK rewrites the visible text but it stays over threshold.
|
||||
const mid = {
|
||||
...textEvent("call-rtk", ""),
|
||||
content: [{ type: "text", text: "z".repeat(30_000) }],
|
||||
};
|
||||
const final = await runHandlers(bound.handlers, mid, ctx());
|
||||
const text = final.content[0].text;
|
||||
assert.match(text, /--- lines 2881-3000 ---/, "view tail must be the stored original's tail");
|
||||
assert.ok(text.includes("orig 3000"));
|
||||
});
|
||||
|
||||
test("file sources over maxEntryBytes keep real head and tail halves", async () => {
|
||||
const root = await makeStore();
|
||||
const { storeEntry } = await import("../src/store.ts");
|
||||
const src = join(root, "big.bin");
|
||||
const content = "HEAD-".repeat(200) + "MID-".repeat(2000) + "TAIL-".repeat(200);
|
||||
await writeFile(src, content);
|
||||
const maxEntryBytes = 2000;
|
||||
await storeEntry(root, "0123456789ab", { kind: "file", path: src }, { tool: "bash", args: {} }, maxEntryBytes);
|
||||
const stored = await readFile(join(root, "0123456789ab.txt"), "utf8");
|
||||
assert.ok(stored.startsWith("HEAD-"));
|
||||
assert.ok(stored.endsWith("TAIL-"));
|
||||
assert.match(stored, /bytes omitted/);
|
||||
// Head and tail halves must equal the source's real bytes.
|
||||
const half = Math.floor(maxEntryBytes / 2);
|
||||
assert.equal(stored.slice(0, half), content.slice(0, half));
|
||||
assert.equal(stored.slice(-half), content.slice(-half));
|
||||
assert.ok(!stored.includes("\0".repeat(100)), "must not store zeroed content");
|
||||
});
|
||||
|
||||
test("interrupted-write tmp files older than an hour are cleaned", async () => {
|
||||
const root = await makeStore();
|
||||
const bound = harness();
|
||||
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
|
||||
|
||||
const staleTmp = join(root, "deadbeef.txt.tmp-999-zzz");
|
||||
const freshTmp = join(root, "livebeef.txt.tmp-111-aaa");
|
||||
await writeFile(staleTmp, "partial");
|
||||
await writeFile(freshTmp, "partial");
|
||||
const old = new Date(Date.now() - 2 * 3_600_000);
|
||||
const { utimes } = await import("node:fs/promises");
|
||||
await utimes(staleTmp, old, old);
|
||||
|
||||
for (const handler of bound.handlers.get("session_start") ?? []) await handler({}, ctx());
|
||||
const { existsSync } = await import("node:fs");
|
||||
assert.ok(!existsSync(staleTmp), "stale tmp must be removed");
|
||||
assert.ok(existsSync(freshTmp), "fresh tmp must survive");
|
||||
});
|
||||
|
||||
test("current under threshold but original over → one-line marker appended", async () => {
|
||||
const root = await makeStore();
|
||||
getStash().clear();
|
||||
const capture = harness();
|
||||
overflowCapture(capture.pi);
|
||||
const bound = harness();
|
||||
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
|
||||
|
||||
const big = "a".repeat(30_000);
|
||||
const event = textEvent("call-3", big);
|
||||
await runHandlers(capture.handlers, event, ctx());
|
||||
const mid = { ...event, content: [{ type: "text", text: "short summary" }] };
|
||||
const final = await runHandlers(bound.handlers, mid, ctx());
|
||||
|
||||
const text = final.content[0].text;
|
||||
assert.ok(text.startsWith("short summary"));
|
||||
assert.match(text, /\[pi-overflow id=[0-9a-f]{12}: full original output stored \(\d+ lines, .*; use overflow_query/);
|
||||
assert.equal(final.details.overflow.truncated, false);
|
||||
assert.ok(final.details.overflow.id);
|
||||
});
|
||||
|
||||
test("exempt tools and nested calls are ignored", async () => {
|
||||
const root = await makeStore();
|
||||
getStash().clear();
|
||||
const bound = harness();
|
||||
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
|
||||
|
||||
const big = "x".repeat(30_000);
|
||||
const exempt = await runHandlers(bound.handlers, textEvent("c-ex", big, { toolName: "read" }), ctx());
|
||||
assert.equal(exempt.content[0].text, big, "read results must pass through untouched");
|
||||
|
||||
const nested = await runHandlers(bound.handlers, textEvent("c-nest", big, { parentToolCallId: "parent" }), ctx());
|
||||
assert.equal(nested.content[0].text, big, "nested calls must pass through untouched");
|
||||
});
|
||||
|
||||
test("image parts and structuredContent survive", async () => {
|
||||
const root = await makeStore();
|
||||
getStash().clear();
|
||||
const bound = harness();
|
||||
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
|
||||
|
||||
const event = textEvent("call-4", "x".repeat(30_000));
|
||||
event.content.push({ type: "image", data: "b64" } as any);
|
||||
(event as any).structuredContent = { rows: 3 };
|
||||
const final = await runHandlers(bound.handlers, event, ctx());
|
||||
assert.deepEqual(final.structuredContent, { rows: 3 });
|
||||
assert.equal(final.content.at(-1).type, "image");
|
||||
assert.equal(final.content.at(-1).data, "b64");
|
||||
});
|
||||
|
||||
test("existing details fields survive the merge", async () => {
|
||||
const root = await makeStore();
|
||||
getStash().clear();
|
||||
const bound = harness();
|
||||
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
|
||||
|
||||
const event = textEvent("call-5", "x".repeat(30_000), { details: { rtk: { saved: 100 }, fullOutputPath: undefined } });
|
||||
const final = await runHandlers(bound.handlers, event, ctx());
|
||||
assert.deepEqual(final.details.rtk, { saved: 100 });
|
||||
assert.ok(final.details.overflow.id);
|
||||
});
|
||||
|
||||
test("files are 0600 and directories 0700", async () => {
|
||||
const root = await makeStore();
|
||||
getStash().clear();
|
||||
const bound = harness();
|
||||
overflowExtension(bound.pi, { storeRoot: root, settings: settings() });
|
||||
|
||||
const final = await runHandlers(bound.handlers, textEvent("call-6", "x".repeat(30_000)), ctx());
|
||||
const id = final.details.overflow.id;
|
||||
const dirMode = (await stat(root)).mode & 0o777;
|
||||
const fileMode = (await stat(join(root, `${id}.txt`))).mode & 0o777;
|
||||
const metaMode = (await stat(join(root, `${id}.json`))).mode & 0o777;
|
||||
assert.equal(dirMode, 0o700);
|
||||
assert.equal(fileMode, 0o600);
|
||||
assert.equal(metaMode, 0o600);
|
||||
});
|
||||
|
||||
test("retention and total-size cleanup remove old entries", async () => {
|
||||
const root = await makeStore();
|
||||
const bound = harness();
|
||||
overflowExtension(bound.pi, {
|
||||
storeRoot: root,
|
||||
settings: settings({ retentionDays: 1, maxStoreBytes: 100 }),
|
||||
});
|
||||
|
||||
// An old entry (mtime in the past) must be deleted by age.
|
||||
const oldId = "aaaaaaaaaaaa";
|
||||
await writeFile(join(root, `${oldId}.txt`), "old");
|
||||
await writeFile(join(root, `${oldId}.json`), "{}");
|
||||
const past = new Date(Date.now() - 10 * 86_400_000);
|
||||
const { utimes } = await import("node:fs/promises");
|
||||
await utimes(join(root, `${oldId}.txt`), past, past);
|
||||
await utimes(join(root, `${oldId}.json`), past, past);
|
||||
|
||||
for (const handler of bound.handlers.get("session_start") ?? []) await handler({}, ctx());
|
||||
const names = await readdir(root);
|
||||
assert.ok(!names.some((n) => n.startsWith(oldId)), "old entry must be deleted");
|
||||
|
||||
// Fresh entries exceeding maxStoreBytes get evicted oldest-first.
|
||||
for (const [i, id] of ["bbbbbbbbbbbb", "cccccccccccc", "dddddddddddd"].entries()) {
|
||||
await writeFile(join(root, `${id}.txt`), "y".repeat(80));
|
||||
const t = new Date(Date.now() - (3 - i) * 60_000);
|
||||
await utimes(join(root, `${id}.txt`), t, t);
|
||||
}
|
||||
for (const handler of bound.handlers.get("session_start") ?? []) await handler({}, ctx());
|
||||
const after = (await readdir(root)).filter((n) => n.endsWith(".txt"));
|
||||
assert.ok(after.length <= 2, `expected eviction under maxStoreBytes, got ${after.join(",")}`);
|
||||
assert.ok(!after.includes("bbbbbbbbbbbb.txt"), "oldest entry must be evicted first");
|
||||
});
|
||||
|
||||
test("disabled setting clears the stash and returns undefined", async () => {
|
||||
const root = await makeStore();
|
||||
getStash().clear();
|
||||
const bound = harness();
|
||||
overflowExtension(bound.pi, { storeRoot: root, settings: settings({ enabled: false }) });
|
||||
getStash().set("c-off", { rawText: "x".repeat(30_000) });
|
||||
const event = textEvent("c-off", "x".repeat(30_000));
|
||||
const result = await runHandlers(bound.handlers, event, ctx());
|
||||
assert.equal(result, event);
|
||||
assert.ok(!getStash().has("c-off"));
|
||||
});
|
||||
@@ -0,0 +1,112 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { runOverflowQuery } from "../src/query.ts";
|
||||
import { storeEntry } from "../src/store.ts";
|
||||
import { readOverflowSettings, type OverflowSettings } from "../settings.ts";
|
||||
|
||||
function settings(overrides: Partial<OverflowSettings> = {}): OverflowSettings {
|
||||
return { ...readOverflowSettings("/nonexistent/pi-overflow.json"), ...overrides };
|
||||
}
|
||||
|
||||
async function store(text: string, id = "0123456789ab") {
|
||||
const root = await mkdtemp(join(tmpdir(), "pi-overflow-query-"));
|
||||
await storeEntry(root, id, { kind: "text", text }, { tool: "bash", args: { command: "run" } }, 1_000_000);
|
||||
return root;
|
||||
}
|
||||
|
||||
test("search returns hits with merged context", async () => {
|
||||
const lines = Array.from({ length: 100 }, (_, i) => (i === 9 || i === 11 || i === 50 ? `line ${i + 1} FAIL` : `line ${i + 1}`));
|
||||
const root = await store(lines.join("\n"));
|
||||
const result = await runOverflowQuery(root, { id: "0123456789ab", search: "FAIL", context: 1 }, settings());
|
||||
assert.ok(!result.error);
|
||||
assert.match(result.text, /L10: line 10 FAIL/);
|
||||
assert.match(result.text, /L12: line 12 FAIL/);
|
||||
assert.match(result.text, /L51: line 51 FAIL/);
|
||||
// hits at lines 10 and 12 with context 1 merge into one range 9-13
|
||||
assert.match(result.text, /--- lines 9-13 ---/);
|
||||
assert.match(result.text, /--- lines 50-52 ---/);
|
||||
});
|
||||
|
||||
test("line-range retrieval returns numbered lines", async () => {
|
||||
const lines = Array.from({ length: 50 }, (_, i) => `row ${i + 1}`);
|
||||
const root = await store(lines.join("\n"));
|
||||
const result = await runOverflowQuery(root, { id: "0123456789ab", lines: { start: 10, end: 15 } }, settings());
|
||||
assert.match(result.text, /--- lines 10-15 ---/);
|
||||
assert.match(result.text, /L10: row 10/);
|
||||
assert.match(result.text, /L15: row 15/);
|
||||
assert.ok(!result.text.includes("row 16"));
|
||||
});
|
||||
|
||||
test("jsonPath extracts nested values", async () => {
|
||||
const doc = JSON.stringify({ snapshot: { nodes: [{ name: "a" }, { name: "b" }, { name: "c" }] } });
|
||||
const root = await store(doc);
|
||||
const result = await runOverflowQuery(root, { id: "0123456789ab", jsonPath: "snapshot.nodes[2].name" }, settings());
|
||||
assert.match(result.text, /"c"/);
|
||||
});
|
||||
|
||||
test("invalid id, regex, JSON and path return clear errors without throwing", async () => {
|
||||
const root = await store('{"a": {"b": 1}}');
|
||||
assert.ok((await runOverflowQuery(root, { id: "../etc/passwd" }, settings())).error);
|
||||
assert.ok((await runOverflowQuery(root, { id: "zzzzzzzzzzzz" }, settings())).error);
|
||||
assert.ok((await runOverflowQuery(root, { id: "0123456789ab", search: "([" }, settings())).error);
|
||||
const goodPath = await runOverflowQuery(root, { id: "0123456789ab", jsonPath: "a.b" }, settings());
|
||||
assert.ok(!goodPath.error, "valid path on valid JSON must succeed");
|
||||
const badPath = await runOverflowQuery(root, { id: "0123456789ab", jsonPath: "a.b.c!!" }, settings());
|
||||
assert.ok(badPath.error);
|
||||
const missing = await runOverflowQuery(root, { id: "ffffffffffff" }, settings());
|
||||
assert.ok(missing.error);
|
||||
});
|
||||
|
||||
test("non-JSON content + jsonPath reports the parse failure", async () => {
|
||||
const root = await store("not json at all");
|
||||
const result = await runOverflowQuery(root, { id: "0123456789ab", jsonPath: "a" }, settings());
|
||||
assert.ok(result.error);
|
||||
assert.match(result.text, /not valid JSON/);
|
||||
});
|
||||
|
||||
test("output is bounded by thresholdChars and marked truncated", async () => {
|
||||
const lines = Array.from({ length: 5000 }, (_, i) => `line ${i + 1} pad ${"p".repeat(40)}`);
|
||||
const root = await store(lines.join("\n"));
|
||||
const result = await runOverflowQuery(
|
||||
root,
|
||||
{ id: "0123456789ab", lines: { start: 1, end: 5000 } },
|
||||
settings({ thresholdChars: 4000 }),
|
||||
);
|
||||
assert.ok(result.text.length <= 4000);
|
||||
assert.equal(result.truncated, true);
|
||||
assert.match(result.text, /truncated/);
|
||||
});
|
||||
|
||||
test("lines and search output clip each line to maxLineChars", async () => {
|
||||
const long = `row ${"q".repeat(2_000)}`;
|
||||
const root = await store(`first\n${long}\n${long} ERROR\nlast`);
|
||||
const s = settings({ maxLineChars: 60 });
|
||||
const lines = await runOverflowQuery(root, { id: "0123456789ab", lines: { start: 2, end: 3 } }, s);
|
||||
assert.ok(lines.text.includes("q".repeat(56) + "…"));
|
||||
assert.ok(!lines.text.includes("q".repeat(61)));
|
||||
const search = await runOverflowQuery(root, { id: "0123456789ab", search: "ERROR", context: 0 }, s);
|
||||
assert.ok(search.text.includes("q".repeat(56) + "…"));
|
||||
assert.ok(!search.text.includes("q".repeat(61)));
|
||||
});
|
||||
|
||||
test("search only scans the first 10000 chars of each line", async () => {
|
||||
const earlyHit = `hit ${"a".repeat(100)} FAIL`;
|
||||
const lateHit = `${"a".repeat(10_500)} FAIL`;
|
||||
const root = await store(`${earlyHit}\n${lateHit}\nplain`);
|
||||
const result = await runOverflowQuery(root, { id: "0123456789ab", search: "FAIL", context: 0 }, settings());
|
||||
assert.match(result.text, /L1: hit /);
|
||||
assert.ok(!result.text.includes("L2:"), "match beyond char 10000 must not be reported");
|
||||
});
|
||||
|
||||
test("default query returns metadata and a first-screen view", async () => {
|
||||
const lines = Array.from({ length: 300 }, (_, i) => `row ${i + 1}`);
|
||||
const root = await store(lines.join("\n"));
|
||||
const result = await runOverflowQuery(root, { id: "0123456789ab" }, settings());
|
||||
assert.ok(!result.error);
|
||||
assert.match(result.text, /\[pi-overflow id=0123456789ab tool=bash \d+ lines \/ \d+ bytes\]/);
|
||||
assert.match(result.text, /--- lines 1-40 ---/);
|
||||
assert.ok(result.text.includes("row 300"), "tail of the stored file must appear");
|
||||
});
|
||||
@@ -0,0 +1,86 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { buildView, collectViewFromText } from "../src/view.ts";
|
||||
import { readOverflowSettings, type OverflowSettings } from "../settings.ts";
|
||||
|
||||
function settings(overrides: Partial<OverflowSettings> = {}): OverflowSettings {
|
||||
return {
|
||||
...readOverflowSettings("/nonexistent/pi-overflow.json"),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
const meta = { id: "abc123def456", tool: "bash" };
|
||||
|
||||
function view(text: string, s = settings()) {
|
||||
return buildView(collectViewFromText(text, s), s, meta);
|
||||
}
|
||||
|
||||
test("below threshold remains unchanged", () => {
|
||||
const out = view("line one\nline two\nline three");
|
||||
assert.ok(out.includes("line one"));
|
||||
assert.ok(out.includes("line three"));
|
||||
});
|
||||
|
||||
test("above threshold includes header, line ranges, signal lines and tail", () => {
|
||||
const lines: string[] = [];
|
||||
for (let i = 1; i <= 1000; i++) lines.push(`row ${i} ${i === 500 ? "ERROR boom" : "ok"}`);
|
||||
const out = view(lines.join("\n"));
|
||||
assert.match(out, /\[pi-overflow id=abc123def456 tool=bash 1000 lines/);
|
||||
assert.match(out, /--- lines 1-40 ---/);
|
||||
assert.match(out, /L500: row 500 ERROR boom/);
|
||||
assert.match(out, /--- lines 881-1000 ---/);
|
||||
assert.ok(out.includes("row 1000"));
|
||||
});
|
||||
|
||||
test("output never exceeds thresholdChars", () => {
|
||||
const lines = Array.from({ length: 5000 }, (_, i) => `line ${i} ${"x".repeat(60)}`);
|
||||
const s = settings({ thresholdChars: 2000 });
|
||||
const out = view(lines.join("\n"), s);
|
||||
assert.ok(out.length <= 2000, `view ${out.length} exceeds budget`);
|
||||
assert.ok(out.includes("line 4999"), "tail must survive");
|
||||
});
|
||||
|
||||
test("few long lines still expose the ending", () => {
|
||||
const lines = ["start", ...Array.from({ length: 5 }, (_, i) => `middle${i} ${"y".repeat(4000)}`), "THE_END_MARKER"];
|
||||
const s = settings({ thresholdChars: 3000, headLines: 40, tailLines: 120 });
|
||||
const out = view(lines.join("\n"), s);
|
||||
assert.ok(out.includes("THE_END_MARKER"));
|
||||
assert.ok(out.length <= 3000 + 100, `view ${out.length} far exceeds budget`);
|
||||
});
|
||||
|
||||
test("same input produces identical output", () => {
|
||||
const lines = Array.from({ length: 800 }, (_, i) => `entry ${i} ${i % 77 === 0 ? "warning" : "fine"}`);
|
||||
const text = lines.join("\n");
|
||||
assert.equal(view(text), view(text));
|
||||
});
|
||||
|
||||
test("a single line over maxLineChars is truncated", () => {
|
||||
const long = "z".repeat(2000);
|
||||
const out = view(`a\n${long}\nb`, settings({ maxLineChars: 50 }));
|
||||
assert.ok(out.includes("z".repeat(50) + "…"));
|
||||
assert.ok(!out.includes("z".repeat(51)));
|
||||
});
|
||||
|
||||
test("signals outrank head and survive when head+tail exceed the budget", () => {
|
||||
// ~150-char lines: head 40 + tail 120 = 160 lines * 150 = ~24K plus
|
||||
// overhead, over budget. Shrink order must keep some signals while head
|
||||
// collapses to 10 lines before the 35/65 phase kicks in.
|
||||
const lines = Array.from({ length: 400 }, (_, i) => {
|
||||
const marker = i === 200 ? "ERROR detected here" : "ok";
|
||||
return `line ${i + 1} ${marker} ${"x".repeat(140)}`;
|
||||
});
|
||||
const out = view(lines.join("\n"));
|
||||
assert.match(out, /--- signals/);
|
||||
assert.match(out, /L201: line 201 ERROR detected here/);
|
||||
// head collapsed below the configured 40 lines
|
||||
assert.match(out, /--- lines 1-(?:[0-9]|10) ---/);
|
||||
assert.ok(out.includes("line 400"));
|
||||
});
|
||||
|
||||
test("signal hits inside the tail region are not duplicated", () => {
|
||||
const lines = Array.from({ length: 100 }, (_, i) => `line ${i + 1} ${"y".repeat(40)}`);
|
||||
lines[99] = "line 100 tail-side ERROR";
|
||||
const out = view(lines.join("\n"));
|
||||
assert.ok(!out.includes("L100:"), "tail-region hit must not appear in signals");
|
||||
});
|
||||
Reference in New Issue
Block a user