# Oversized tool-result spill + budget-delta flush

Date: 2026-06-02
Branch/worktree: `oversized-output-spill`
Status: spec (pre-implementation)

## Problem

A single large tool result can dominate the context window before any
existing pruning mechanism reacts. Observed: a session at ~70% of a 1M-token
window where one `fetch` result was ~1.08 MB (~270K tokens, ~27% of the
window) on its own.

Why nothing "kicked in":

1. **No single-result cap.** Every captured result — regardless of size —
   waits for normal batch close → async summarization → rolling-window aging
   before its raw body leaves context. A 1 MB output sits at full size for
   several turns.
2. **Budget trigger is a single aggregate gate.** `shouldBudgetFlush`
   (`src/budget.ts`) fires only when `usage.tokens / usage.contextWindow >=
   autoBudgetThreshold`, evaluated once per `turn_end`
   (`index.ts:714-730`). The user's `autoBudgetThreshold` was `0.8`
   (`DEFAULT_CONFIG` ships `null` = off); at 70% it had not yet tripped, and a
   single turn can jump from below to over-limit between two boundaries.

This spec adds two layers:

- **A — eager spill:** synchronously offload any single oversized result to a
  file and index it immediately, so it never reaches a request at full size.
- **B — budget-delta flush:** force a flush when one turn adds a large
  fraction of the window, independent of the absolute aggregate threshold.

Both reuse existing machinery (`Indexer.addBatch` → `isSummarized` →
`pruneMessages` stub-replace; `shouldBudgetFlush` + `turn_end` flush).

## Ground truth (package namespace)

This extension currently pins and imports **`@mariozechner/*`**
(`pi-coding-agent`, `pi-ai`, `pi-tui`, `pi-agent-core`) — that is what is
installed and what every source file imports, so API verification for this
spec resolves under `node_modules/@mariozechner/pi-coding-agent/dist`.

**Caveat:** the entire `@mariozechner/*` family is **deprecated**; each
package's npm metadata says "please use `@earendil-works/<pkg>` instead going
forward" (the global pi harness already runs `@earendil-works/pi-coding-agent`).
`@earendil-works/*` is the renamed continuation — same symbols, same paths —
so the table below is expected to hold verbatim after migration. Migrating
this repo's peer deps from `@mariozechner` to `@earendil-works` is a
repo-wide dependency rename, **out of scope for this spec** and tracked
separately; this feature builds against whatever the peer deps resolve to at
implementation time.

Symbols this spec depends on, verified present (in the installed
`@mariozechner` types):

| symbol | location | note |
|---|---|---|
| `getSessionDir()`, `getSessionId()`, `getSessionFile()` | `core/session-manager.d.ts:189-191` (on `ReadonlySessionManager`, l.136) | spill dir + id |
| `getContextUsage(): ContextUsage \| undefined` | `core/extensions/types.d.ts:231` | budget triggers |
| `ContextUsage.tokens: number \| null`, `contextWindow: number`, `percent: number \| null` | `core/extensions/types.d.ts:192-198` | **`tokens` is null right after compaction** — must guard |
| `truncateHead(content, options?)`, `DEFAULT_MAX_BYTES`, `DEFAULT_MAX_LINES` | `core/tools/truncate.d.ts` (re-exported from index) | options object, not positional |

## Non-goals

- Changing the `fetch` / `bash` tools (harness-owned, separate repo). A is
  tool-agnostic and covers them all.
- Orphan-blob garbage collection / startup sweep. Out of scope for v1; blob
  cleanup rides on session deletion only (see Cleanup).
- Replacing the LLM summarizer. Eager-spilled results are NOT summarized by an
  LLM; the stub is mechanical.

## A — Eager single-result spill

### Trigger

At tool-result capture (live `turn_end` path in `src/batch-capture.ts`, and
the reconstruction path `captureUnindexedBatchesFromSession`), evaluate each
`CapturedToolCall` independently. A call is **oversized** when
`resultText.length >= spillThreshold` (chars — matches `resultText.length`,
not bytes).

Ordering relative to existing pre-flush passes (precedence is strict):

1. **Protected-tools filter first** — a `protectedTools` result is never
   spilled.
2. **Dedup check second** — compute the content hash on the full in-memory
   body. If it matches an existing original, alias as today (`registerDuplicate`)
   and **do not** write a file. A duplicate of an already-spilled original
   aliases to that original's record (and thus its `spillPath`); the byte cost
   is paid once.
3. **Spill last** — only a non-duplicate oversized result is written to a
   sidecar file and indexed.

The spill + `addBatch` happens before the existing `flushPending` call in the
`turn_end` handler. No extra `context` event is forced; the stub takes effect
on the next natural `context` build (same as today's summarized records).

### Storage (hybrid)

- Bodies `< spillThreshold` stay **inline** in the `context-prune-index`
  entry exactly as today (`IndexEntryData.toolCalls[].resultText`). Portable:
  the JSONL remains self-contained for all normal content.
- Bodies `>= spillThreshold` are written to a **sidecar file**; the index
  entry stores a reference + head preview instead of the full body.

File location:

```
<sessionDir>/<sessionId>-blobs/<toolCallId>.txt
```

- `sessionDir = ctx.sessionManager.getSessionDir()`, `sessionId =
  ctx.sessionManager.getSessionId()` (confirmed on `ReadonlySessionManager`,
  `session-manager.d.ts`). Persistent root alongside the session JSONL —
  survives reboot/resume, not subject to `os.tmpdir()` reapers.
- One file per tool call, full verbatim `resultText`, UTF-8.
- `toolCallId` is filename-safe in practice (provider ids like
  `toolu_...`); sanitize defensively (`[^A-Za-z0-9_-]` → `_`) to avoid path
  traversal / separators.

### Record / persistence changes

`ToolCallRecord` (`src/types.ts`) gains optional fields:

```ts
spillPath?: string;      // absolute path to the sidecar blob
spillBytes?: number;     // full byte length of the spilled body
resultPreview?: string;  // head preview kept inline when spilled
contentHash?: string;    // dedup hash of the FULL body, persisted
```

**`resultText` invariant.** Persisting the full body in the index entry would
defeat the spill (MB back in the JSONL). So for a spilled record the persisted
entry carries `resultPreview` (not the full body), `spillPath`, `spillBytes`,
and `contentHash`. The full `resultText` exists only transiently in memory at
capture time — long enough to compute `contentHash` and write the file — and
is not serialized.

**Dedup hash persistence.** Today `addBatch` computes the content hash from
`resultText` and `reconstructFromSession` recomputes it on reload. A spilled
record has no full `resultText` after reload, so the hash is computed once at
spill time and **persisted** as `contentHash`; reconstruct reads it back
instead of recomputing. Non-spilled records keep the existing
recompute-from-`resultText` path unchanged.

**No summarizer path.** Eager-spilled records bypass
`serializeBatchForSummarizer` entirely (they are never sent to the LLM
summarizer), so there is no double-truncation between preview and summarizer
input.

`IndexEntryData.toolCalls[]` already serializes `ToolCallRecord`, so adding the
optional fields persists them with no new custom entry type.
`Indexer.reconstructFromSession` replays them unchanged; the preview + hash
travel inline, the full body stays on disk.

### What the model sees (stub)

`pruneMessages` already stub-replaces any `isSummarized` tool result. For a
spilled record the stub text is **mechanical** (no LLM summary) and contains:

- tool name + a compact rendering of args,
- size (`spillBytes` and line count),
- the head preview (`resultText`),
- the absolute `spillPath` and an explicit hint:
  `read this file (offset/limit supported) to access the full output`,
- the existing short-ref line so `context_tree_query` still works.

This gives file-handle ergonomics: the model slices the body on demand with
the native `read` tool; no LLM round-trip happened on capture.

### Stub branching (`src/pruner.ts`)

`pruneMessages` Phase 1 already replaces a summarized tool result with a single
hardcoded `[Summarized … ref tN]` stub. It must branch on the resolved record:

- `getRecord(toolCallId)` has `spillPath` set → emit the **mechanical spill
  stub**: tool name, compact args, `spillBytes` + line count, the
  `resultPreview`, the absolute `spillPath`, a `read this file (offset/limit
  supported)` hint, and the existing short-ref line.
- otherwise → emit the **existing summary-ref stub** unchanged.

Both branches return a stub message (never delete the result) so pi-ai's
orthan-tool-result repair is not triggered.

### Indexing (synchronous, no summarizer)

Eager spill must make the result `isSummarized` immediately so `pruneMessages`
stubs it on the very next `context` build, without waiting for `flushPending`
or an LLM call. Reuse `Indexer.addBatch` with a single-call batch carrying the
spilled record(s); it issues short refs, persists the index entry via
`appendEntry`, and updates the in-memory map. No `context-prune-summary`
message is emitted for eager-spilled results (the stub is self-describing).

### Recovery

`context_tree_query` (`src/query-tool.ts`): when the resolved record has
`spillPath`, read the file and apply the **existing**
`truncateHead(content, { maxBytes: DEFAULT_MAX_BYTES, maxLines:
DEFAULT_MAX_LINES })` call already used in this file to the file contents
instead of to `resultText` — no new constants or exports. Falls back to
`resultPreview` if the file is missing. The primary recovery path the model
should use is the native `read` tool on `spillPath` (full slicing);
`context_tree_query` remains the head-truncated fallback.

### Failure handling

- **Write-then-mutate (atomic).** Write the sidecar file first; mutate the
  record fields (`spillPath`/`spillBytes`/`resultPreview`/`contentHash`) only
  after `writeFile` resolves. A throw leaves the `CapturedToolCall` untouched
  and the result falls through to the normal flush pipeline. Never drop a
  result. Log at debug.
- **Filename collision.** `toolCallId` is unique per call and the sanitized
  alphabet (`[^A-Za-z0-9_-]` → `_`) collides only on pathological ids; before
  writing, if the target path already exists for a different record, fall back
  to inline rather than overwrite.
- Missing blob at query time → return `resultPreview` + a note that the full
  body is unavailable.

## B — Budget-delta flush

Add a second, delta-based flush trigger in the `turn_end` handler, evaluated
alongside the existing absolute `shouldBudgetFlush`.

- New config `budgetTurnDelta: number | null` (fraction of window, default
  `null` = off; suggested operational value `0.15`).
- Track the previous turn's usage fraction (`tokens / contextWindow`) as a
  module-level variable in `index.ts`, set to `null` on `session_start`.
  After a restart the first turn cannot fire the delta trigger (no previous
  fraction); this gap is accepted because the absolute `autoBudgetThreshold`
  still guards the post-restart aggregate. On a turn where `usage.tokens` is
  null (e.g. immediately after compaction), leave `previousFraction`
  unchanged rather than overwriting it, so the next real reading is compared
  against the last known fraction.
- If `(currentFraction - previousFraction) >= budgetTurnDelta`, call
  `flushPending(ctx, { delivery: "session" })` — same flush the aggregate
  trigger uses.
- Either trigger (absolute OR delta) firing causes a flush; they are ORed.

Rationale: catches a turn that adds a large aggregate (many medium outputs, or
a spike that A's per-result threshold didn't individually catch) without
waiting to cross the absolute `autoBudgetThreshold`.

Helper lives in `src/budget.ts` next to `shouldBudgetFlush`:

```ts
export function shouldDeltaFlush(
  usage: ContextUsage | undefined,
  previousFraction: number | null,
  delta: number | null,
): boolean
```

Pure; mirrors `shouldBudgetFlush`'s guards exactly — returns `false` when
`!usage || usage.tokens == null || !(usage.contextWindow > 0) || delta == null
|| previousFraction == null`. Uses the same 0–1 `tokens / contextWindow`
fraction (not `ContextUsage.percent`, which is 0–100 and null when `tokens`
is). Because `tokens` is null right after a compaction, `previousFraction` is
left unchanged on a null-token turn (no spurious delta when usage
"reappears").

## Config (`DEFAULT_CONFIG`, `src/types.ts`)

| field | type | default | meaning |
|---|---|---|---|
| `spillThreshold` | `number` | `65536` | min chars (`resultText.length`) for a single result to spill |
| `spillPreviewBytes` | `number` | `2048` | head-preview size in **bytes** kept inline as `resultPreview` |
| `budgetTurnDelta` | `number \| null` | `null` | per-turn usage-fraction increase that forces a flush |

`spillThreshold` is intentionally high (64 KB): only genuinely huge outputs
spill; medium outputs (tens of KB) keep flowing through normal
summarization/dedup so the model sees them without a `read` round-trip.

## Cleanup

- The blob dir is a sibling of the session file; deleting the session dir
  removes its blobs. No separate GC in v1.
- No orphan sweep on startup (explicit non-goal). Accepted: blobs for deleted
  sessions whose dir was partially removed can linger; revisit if it matters.

## Touched files

| file | change |
|---|---|
| `src/types.ts` | `ToolCallRecord.spillPath?/spillBytes?`; `DEFAULT_CONFIG` + `ContextPruneConfig` gain `spillThreshold`, `spillPreviewBytes`, `budgetTurnDelta` |
| `src/spill.ts` (new) | pure-ish helpers: `blobDirFor`, `blobPathFor`, `headPreview`, `writeSpill`, `readSpill`; isolates fs + path sanitize |
| `src/batch-capture.ts` | protected filter → dedup → spill non-duplicate oversized `CapturedToolCall`s (write file, set `spillPath`/`spillBytes`/`resultPreview`/`contentHash`), `addBatch` synchronously |
| `src/indexer.ts` | persist + replay `contentHash` for spilled records (skip recompute when present); pass new optional fields through `reconstructFromSession` |
| `src/pruner.ts` | branch on `spillPath`: mechanical spill stub vs. existing summary-ref stub |
| `src/query-tool.ts` | `spillPath` → read file then `truncateHead`; fallback to preview |
| `src/budget.ts` | `shouldDeltaFlush` |
| `index.ts` | `turn_end`: track previous usage fraction; OR `shouldDeltaFlush` into the flush decision; reset on `session_start` |
| `README.md` | document the 3 new settings + `read`-the-blob recovery |
| `PRUNING.md` | document eager-spill layer + budget-delta trigger |

## Testing

Pure unit:
- threshold gating (`< / == / >` `spillThreshold`).
- `headPreview` (line-boundary cut, byte cap, multibyte safety).
- blob path sanitize (no traversal/separators).
- `shouldDeltaFlush` truth table (null delta, missing usage, below/at/above).

Integration (Pi extension smoke, per AGENTS.md):
- oversized result → sidecar file exists with full body; index entry has
  `spillPath` + preview, not full body.
- `pruneMessages` stubs the spilled result on the next `context` build with
  **no** `context-prune-summary` and **no** summarizer LLM call.
- `context_tree_query` on the spilled id returns file-backed (head-truncated)
  content; `read` on `spillPath` returns the full body.
- spill + dedup: a large result spills once; an identical later result aliases
  to the spilled original (no second file write) and resolves to the same
  `spillPath`.
- reconstruct-from-session: after restart, record still resolves, blob still
  read.
- budget-delta: a turn whose usage jump `>= budgetTurnDelta` triggers
  `flushPending`; below it does not; a null-`tokens` turn neither fires nor
  overwrites `previousFraction`.

Typecheck before commit per AGENTS.md (`bun x tsc --noEmit ...`).

## Open questions

None. Threshold (64 KB), B in scope, no orphan sweep — all decided.