mirror of
https://bitbucket.org/siakitem/my-pi.git
synced 2026-10-26 10:45:22 +00:00
1167 lines
98 KiB
Markdown
1167 lines
98 KiB
Markdown
# Understanding Context Pruning in Pi
|
||
|
||
> How `pi-condense` compresses tool-call history, why it matters for long-running sessions, and how it balances context size against provider-side prefix caching.
|
||
|
||
---
|
||
|
||
## Table of Contents
|
||
|
||
1. [What Does a Long Session Look Like?](#what-does-a-long-session-look-like)
|
||
2. [What Pruning Does](#what-pruning-does)
|
||
3. [Pruned Data Is Still Available](#pruned-data-is-still-available)
|
||
4. [What Actually Lives in the Pruner Index](#what-actually-lives-in-the-pruner-index)
|
||
5. [How the Model Re-reads Raw Outputs](#how-the-model-re-reads-raw-outputs)
|
||
6. [How Prefix Caching Works](#how-prefix-caching-works)
|
||
7. [Why Frequent Pruning Busts Cache](#why-frequent-pruning-busts-cache)
|
||
8. [The Sweet Spot: Batch and Prune](#the-sweet-spot-batch-and-prune)
|
||
9. [Pre-flush Pipeline & Safeguards](#pre-flush-pipeline--safeguards)
|
||
- [Stub-replace instead of delete](#stub-replace-instead-of-delete)
|
||
- [Protected tools](#protected-tools)
|
||
- [Eager single-result spill](#eager-single-result-spill)
|
||
- [Trivial-batch skip (minBatchChars)](#trivial-batch-skip-minbatchchars)
|
||
- [Content-hash dedup](#content-hash-dedup)
|
||
- [Oversized summary skip](#oversized-summary-skip)
|
||
- [Frontier persistence](#frontier-persistence)
|
||
- [Other UI / observability features](#other-ui--observability-features)
|
||
- [Token-budget auto-flush trigger](#token-budget-auto-flush-trigger)
|
||
- [Budget-delta flush](#budget-delta-flush)
|
||
10. [Chain Compression](#chain-compression)
|
||
- [Protected-output relocation](#protected-output-relocation)
|
||
11. [Occurrence Identity](#occurrence-identity)
|
||
12. [Error Purge](#error-purge)
|
||
13. [Orphan Sweep](#orphan-sweep)
|
||
14. [Diagnostics](#diagnostics)
|
||
15. [Why Summarization Works: Research Evidence](#why-summarization-works-research-evidence)
|
||
- [SUPO — Summarization augmented Policy Optimization](#supo--summarization-augmented-policy-optimization)
|
||
- [ReSum — Recursive Summarization for Long-Horizon Agents](#resum--recursive-summarization-for-long-horizon-agents)
|
||
- [ACON — Agent Context Optimization](#acon--agent-context-optimization)
|
||
16. [Summary](#summary)
|
||
|
||
---
|
||
|
||
## What Does a Long Session Look Like?
|
||
|
||
In Pi, every assistant turn that calls tools produces a sequence of messages in the context tree. In a long coding or research session, this accumulates rapidly:
|
||
|
||
### ASCII: A typical Pi context tree (before pruning)
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ SESSION CONTEXT (growing without bound) │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ [system] You are Pi, a helpful coding assistant... │
|
||
│ │
|
||
│ [user] Build a React component that fetches data from │
|
||
│ an API and displays it in a table... │
|
||
│ │
|
||
│ ── Turn 1 ───────────────────────────────────────── │
|
||
│ [assistant] <tool_call name="read_file" id="tc-001"> │
|
||
│ {"path": "src/App.tsx"} │
|
||
│ [tool] export default function App() { ... } ← 45 tokens │
|
||
│ │
|
||
│ ── Turn 2 ───────────────────────────────────────── │
|
||
│ [assistant] <tool_call name="read_file" id="tc-002"> │
|
||
│ {"path": "package.json"} │
|
||
│ [tool] { "dependencies": { "react": "^18.2.0", ... } ← 120 tok │
|
||
│ │
|
||
│ ── Turn 3 ───────────────────────────────────────── │
|
||
│ [assistant] <tool_call name="web_search" id="tc-003"> │
|
||
│ <tool_call name="read_file" id="tc-004"> │
|
||
│ [tool-003] React Table v7 docs, TanStack Table API... ← 3,400 tok │
|
||
│ [tool-004] import { useState } from 'react'; ... ← 200 tokens │
|
||
│ │
|
||
│ ── Turn 4 ───────────────────────────────────────── │
|
||
│ [assistant] <tool_call name="edit_file" id="tc-005"> │
|
||
│ [tool] ✔︎ File updated successfully ← 15 tokens │
|
||
│ │
|
||
│ ── Turn 5 ───────────────────────────────────────── │
|
||
│ [assistant] <tool_call name="bash" id="tc-006"> │
|
||
│ <tool_call name="read_file" id="tc-007"> │
|
||
│ [tool-006] BUILD OUTPUT (npm run build): ← 2,800 tok │
|
||
│ [warn] Circular dependency detected... │
|
||
│ [warn] Chunk size exceeds 500kb... │
|
||
│ [error] TypeScript compilation failed... │
|
||
│ [tool-007] Updated file contents... ← 180 tokens │
|
||
│ │
|
||
│ ── Turn 6 ── ... (more turns, more tool calls) ── │
|
||
│ │
|
||
│ ═══════════════════════════════════════════════════════ │
|
||
│ Context size: ~15,000 tokens and growing... │
|
||
│ Most tokens are raw tool outputs the model already "consumed" │
|
||
│ ═══════════════════════════════════════════════════════ │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
In a long session this can grow to **30k–100k+ tokens**. The model pays for every token on every subsequent request. More importantly, the "signal" (what the model actually needs to know) is buried in a mountain of "noise" (full build logs, search results, file contents it already processed).
|
||
|
||
---
|
||
|
||
## What Pruning Does
|
||
|
||
`pi-condense` intercepts completed tool-call batches, summarizes them, and replaces each raw `ToolResultMessage` in future context with a small breadcrumb stub that points at `context_tree_query` for recovery. The original full output is archived in the session index.
|
||
|
||
### ASCII: The same session *after* pruning Turns 1–5
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ SESSION CONTEXT (after pruning Turns 1-5) │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ [system] You are Pi, a helpful coding assistant... │
|
||
│ │
|
||
│ [user] Build a React component that fetches data from │
|
||
│ an API and displays it in a table... │
|
||
│ │
|
||
│ [summary] ╔════════════════════════════════════════════╗ │
|
||
│ ║ ⚃ [pruner] Turn 1–5 summary (7 tools) ║ │
|
||
│ ║ ║ │
|
||
│ ║ • Read existing App.tsx and package.json ║ │
|
||
│ ║ • Searched React Table docs; decided on ║ │
|
||
│ ║ @tanstack/react-table v8 ║ │
|
||
│ ║ • Created DataTable component with ║ │
|
||
│ ║ sorting, pagination, useQuery hook ║ │
|
||
│ ║ • Build failed: circular dependency in ║ │
|
||
│ ║ utils/index.ts → fix by inlining helpers ║ │
|
||
│ ║ ║ │
|
||
│ ║ Summarized tool refs: t1..t7 ║ │
|
||
│ ║ Use context_tree_query for raw outputs ║ │
|
||
│ ╚════════════════════════════════════════════╝ │
|
||
│ ← ~200 tokens (was ~6,760 tokens) │
|
||
│ │
|
||
│ ── Turn 6 ───────────────────────────────────────── │
|
||
│ [assistant] <tool_call name="read_file" id="tc-008"> │
|
||
│ <tool_call name="edit_file" id="tc-009"> │
|
||
│ [tool-008] import { helperA } from './helpers'; ... ← 90 tokens │
|
||
│ [tool-009] ✔︎ File updated successfully ← 15 tokens │
|
||
│ │
|
||
│ ═══════════════════════════════════════════════════════ │
|
||
│ Context size: ~500 tokens (plus current turn) │
|
||
│ ~96% reduction in "stale" context tokens │
|
||
│ ═══════════════════════════════════════════════════════ │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Mermaid: The pruning transformation
|
||
|
||
```mermaid
|
||
%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#e1f5fe', 'primaryTextColor': '#01579b', 'primaryBorderColor': '#0288d1', 'lineColor': '#0288d1', 'secondaryColor': '#fff3e0', 'tertiaryColor': '#e8f5e9'}}}%%
|
||
|
||
graph TB
|
||
subgraph Before["Context BEFORE Pruning"]
|
||
direction TB
|
||
U1["👤 User Request"] --> A1["🤖 Assistant (tool calls)"]
|
||
A1 --> T1["🔧 Tool Result 1<br/>~3,400 tokens"]
|
||
A1 --> T2["🔧 Tool Result 2<br/>~200 tokens"]
|
||
T1 --> A2["🤖 Assistant (tool calls)"]
|
||
A2 --> T3["🔧 Tool Result 3<br/>~2,800 tokens"]
|
||
A2 --> T4["🔧 Tool Result 4<br/>~180 tokens"]
|
||
T3 --> A3["🤖 Assistant (text response)"]
|
||
end
|
||
|
||
subgraph After["Context AFTER Pruning"]
|
||
direction TB
|
||
U2["👤 User Request"] --> S["📋 Summary Message<br/>~200 tokens"]
|
||
S --> A4["🤖 Assistant (tool calls)"]
|
||
A4 --> T5["🔧 Tool Result 5<br/>~90 tokens"]
|
||
A4 --> T6["🔧 Tool Result 6<br/>~15 tokens"]
|
||
end
|
||
|
||
Before -->|"prune summarized<br/>tool results"| After
|
||
|
||
style T1 fill:#ffebee
|
||
style T3 fill:#ffebee
|
||
style T2 fill:#ffebee
|
||
style T4 fill:#ffebee
|
||
style T5 fill:#e8f5e9
|
||
style T6 fill:#e8f5e9
|
||
style S fill:#fff3e0
|
||
```
|
||
|
||
**Key points:**
|
||
|
||
- The `AssistantMessage` tool-call blocks are **kept** (they carry the `toolCallId`s the model uses to reference originals via `context_tree_query`).
|
||
- `ToolResultMessage` entries for summarized tool calls are **replaced with a small stub** (`[Summarized in pruner summary, ref \`tN\`. Use context_tree_query to retrieve full output.]`) carrying`role: "toolResult"`, the original`toolCallId`/`toolName`/`timestamp`, and`isError: false`. The stub preserves role alternation, so pi-ai's`transformMessages.insertSyntheticToolResults` no longer injects a synthetic `{ isError: true, "No result provided" }` for the (no-longer-)orphaned tool call. See [Stub-replace instead of delete](#stub-replace-instead-of-delete).
|
||
- Every pruned tool call is also copied into the pruner's runtime/session index with its `toolCallId`, tool name, args, status, turn index, timestamp, and full `resultText`.
|
||
- A summary message is injected as a `"steer"` (`pi.sendMessage` runtime path) or appended directly via `sessionManager.appendCustomMessageEntry` (session path, used when Pi may already be shutting down). Both deliver before the next LLM call.
|
||
- The session JSONL file retains the original tool-result entries unchanged — pruning only affects what the *next* request sees in active context.
|
||
- **Recovery grace window (`recoveryGraceTurns`, default 3):** after the model recovers a tool call via `context_tree_query`, that tool call's output is rendered **verbatim** (not re-stubbed) for the next N user-turn-groups, then reverts to the normal stub. Without this, the pruner re-prunes its own recovery output on the very next flush, forcing the model into a retrieve -> re-stub -> re-query loop ("fighting the pruner") whenever it keeps referencing the same recovered data across a few turns.
|
||
- Enforced at **render time**, in Phase 1 stub-replace and in chain-compression eligibility — NOT at capture time. Capture-time exclusion would collide with frontier trim: a tool call already past the frontier is dropped forever, so excluding a recovered call from capture would either need to resurrect frontier state or degrade the lifetime bound into permanent verbatim retention for anything ever recovered.
|
||
- Trade-off: this bounds but does not eliminate regrowth. A tool call still referenced after its grace window expires is re-stubbed and may be re-queried again — the accepted cost of keeping the window's context-growth impact bounded instead of unbounded.
|
||
|
||
---
|
||
|
||
## Pruned Data Is Still Available
|
||
|
||
Pruning does **not** delete data. It moves raw tool results out of the hot path (active LLM context) and into an indexed archive the model can query later.
|
||
|
||
There are two separate things happening during pruning:
|
||
|
||
1. **Context filtering:** future requests stop including the old `toolResult` messages.
|
||
2. **Index preservation:** the extension stores each summarized tool call in the pruner index, keyed by its occurrence key (`id@resultTimestamp`, or the bare id for legacy records - see [Occurrence Identity](#occurrence-identity)).
|
||
|
||
That distinction is the core idea:
|
||
|
||
- **Pruned from context** does **not** mean **lost**
|
||
- It means **hidden from the default prompt**, but still **recoverable on demand**
|
||
|
||
### ASCII: How `context_tree_query` recovers pruned data
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ RECOVERING PRUNED DATA via context_tree_query │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ [summary] ... build failed: circular dependency ... │
|
||
│ - `t1` read config.ts -> 3 exports │
|
||
│ Summarized tool refs: `t1` │
|
||
│ Use `context_tree_query` with these refs │
|
||
│ │
|
||
│ ── LLM calls context_tree_query({ toolCallIds: ["t1"] }) ── │
|
||
│ │
|
||
│ [tool] ⌕ context_tree_query result │
|
||
│ ┌─────────────────────────────────────────────────────┐ │
|
||
│ │ Tool: bash (t1) │ │
|
||
│ │ Status: OK │ │
|
||
│ │ ───────────────────────────────────────────────── │ │
|
||
│ │ $ npm run build │ │
|
||
│ │ > react-app@0.1.0 build │ │
|
||
│ │ > tsc && vite build │ │
|
||
│ │ │ │
|
||
│ │ [warn] Circular dependency: src/utils/index.ts -> │ │
|
||
│ │ src/utils/helpers.ts -> src/utils/index.ts │ │
|
||
│ │ [warn] (!) Some chunks are larger than 500 kBs │ │
|
||
│ │ [error] TS2345: Argument of type 'X' not assignable│ │
|
||
│ │ to parameter of type 'Y'... │ │
|
||
│ │ │ │
|
||
│ │ [Output truncated: 200/512 lines shown] │ │
|
||
│ └─────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
│ The LLM now has the full build log back in context, on demand, │
|
||
│ without permanently inflating the context window. │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## What Actually Lives in the Pruner Index
|
||
|
||
When a batch is summarized, the extension writes a record for each tool call into `ToolCallIndexer` and persists that record into the session as a custom index entry.
|
||
|
||
Conceptually, each indexed record looks like this (see `ToolCallRecord` in `src/types.ts`):
|
||
|
||
```ts
|
||
{
|
||
toolCallId: "tc-006",
|
||
toolName: "bash",
|
||
args: { command: "npm run build" },
|
||
resultText: "full original raw output...",
|
||
isError: false,
|
||
turnIndex: 5,
|
||
timestamp: 1745251200000 // epoch ms
|
||
}
|
||
```
|
||
|
||
This matters because the summary is **not** the only surviving representation of the old tool call.
|
||
The model still has access to:
|
||
|
||
- the original `toolCallId`
|
||
- the tool name and arguments
|
||
- whether the tool errored
|
||
- which turn it came from
|
||
- the full original raw result text
|
||
|
||
So after pruning, the model is working with a **two-layer memory**:
|
||
|
||
1. **Hot memory:** compact summary text kept directly in context
|
||
2. **Cold memory:** full raw tool outputs stored in the pruner index and retrievable by ID
|
||
|
||
### What is removed vs what is preserved
|
||
|
||
| Part of old turn | After pruning | Why |
|
||
|---|---|---|
|
||
| Assistant tool-call block | **Kept in context** | Preserves the `toolCallId` anchors the model uses to reference originals |
|
||
| Tool result message | **Replaced by a short stub in active context** | Saves tokens (typical 50–100× reduction per call) while keeping the `toolCallId` anchor and giving the model an explicit `context_tree_query` breadcrumb. The stub keeps `role: "toolResult"` and `isError: false` so role alternation stays intact |
|
||
| Summary message | **Added to context** | Gives the model a compact description of what happened |
|
||
| Indexed tool-call record | **Stored in pruner index** (`context-prune-index` session entry) | Lets the model re-open the original raw output later via `context_tree_query` |
|
||
| Duplicate of an already-indexed record (same toolName + content) | **Aliased to the original; no new summary, no LLM call** (`context-prune-dedup-alias` session entry) | See [Content-hash dedup](#content-hash-dedup) |
|
||
|
||
## How the Model Re-reads Raw Outputs
|
||
|
||
The intended recovery flow is:
|
||
|
||
1. The model reads a summary message.
|
||
2. The summary lists the short refs (`t1`, `t2`, …) that were summarized. Each per-tool bullet also carries its own inline `` `tN` `` ref (copied from a `[[N:toolname]]` label the summarizer emits, validated against the tool at that position), so the model can jump from a bullet straight to its ref; the flat footer still lists every ref as a fallback.
|
||
3. The model decides the summary is not enough and wants exact raw output.
|
||
4. The model calls `context_tree_query({ toolCallIds: ["t1", ...] })`. The tool accepts short refs and full `toolCallId`s interchangeably (`indexer.resolveToolCallId`).
|
||
5. The tool looks up those IDs in the pruner index.
|
||
6. The tool returns the original stored output back into the current turn.
|
||
7. The model can now inspect that raw result and continue reasoning.
|
||
|
||
### ASCII: end-to-end "prune, then re-read" flow
|
||
|
||
```text
|
||
assistant turn with tools
|
||
│
|
||
▼
|
||
raw tool results exist in context
|
||
│
|
||
▼
|
||
batch gets summarized
|
||
│
|
||
├─► summary message added to context
|
||
│ └─► includes short refs (`t1`, `t2`, …)
|
||
│
|
||
├─► tool results indexed by occurrence key (id@resultTimestamp)
|
||
│ └─► full raw resultText stored in index/session
|
||
│
|
||
└─► old toolResult messages removed from future context
|
||
|
||
later...
|
||
│
|
||
▼
|
||
model sees summary and decides: "I need the exact old output"
|
||
│
|
||
▼
|
||
context_tree_query({ toolCallIds: ["t1"] })
|
||
│
|
||
▼
|
||
query tool resolves short ref / id and loads the indexed record
|
||
│
|
||
▼
|
||
original raw output is returned into the current turn
|
||
│
|
||
▼
|
||
model continues with exact old context back in view
|
||
```
|
||
|
||
### Why this is important
|
||
|
||
- summaries keep the default context small
|
||
- short refs / `toolCallId`s keep old work addressable
|
||
- `context_tree_query` makes the archive readable again
|
||
- the model can "page in" exact old context only when it actually needs it
|
||
|
||
Summaries are the default view; raw data remains addressable through `context_tree_query`.
|
||
|
||
---
|
||
|
||
## How Prefix Caching Works
|
||
|
||
Modern LLM API providers (Anthropic, OpenAI, vLLM, etc.) implement **prefix caching** (also called "prompt caching") to speed up repeated requests with similar prompts.
|
||
|
||
### How it works
|
||
|
||
LLM inference has two phases:
|
||
|
||
1. **Prefill** — compute Key-Value (KV) attention states for all input tokens
|
||
2. **Decode** — generate output tokens autoregressively, reusing the cached KV states
|
||
|
||
Prefix caching stores the KV states for an exact token sequence on the provider's GPU. When a new request shares an identical prefix, the provider **skips prefill** for that prefix and starts from the cached KV state.
|
||
|
||
### ASCII: Cache hit vs cache miss
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ WITHOUT PREFIX CACHING │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ Request 1: [System] [User Q1] ──► LLM computes KV for ALL tokens │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ Generate answer │
|
||
│ │
|
||
│ Request 2: [System] [User Q2] ──► LLM computes KV for ALL tokens │
|
||
│ ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ │ (EVERYTHING recomputed) │
|
||
│ Same prefix as Request 1 ▼ │
|
||
│ Generate answer │
|
||
│ │
|
||
│ Time: ████████████████████████████████████████ ~2.5s each │
|
||
│ Cost: Full input tokens priced at standard rate │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ WITH PREFIX CACHING (HIT) │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ Request 1: [System] [User Q1] ──► LLM computes KV for ALL tokens │
|
||
│ │ │
|
||
│ ┌───────────────────┘ │
|
||
│ ▼ │
|
||
│ [STORED IN CACHE] │
|
||
│ │
|
||
│ Request 2: [System] [User Q2] ──► SKIP! KV loaded from cache │
|
||
│ ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ │ (prefix match = instant) │
|
||
│ Cache hit! ▼ │
|
||
│ Only compute NEW tokens │
|
||
│ Generate answer │
|
||
│ │
|
||
│ Time: ████████████████████░░░░░░░░░░░░░░░░░░░ ~0.5s (80% faster) │
|
||
│ Cost: Cached prefix at 50-90% discount (provider-dependent) │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Provider specifics
|
||
|
||
| Provider | Cache Activation | Match Type | Duration |
|
||
|---|---|---|---|
|
||
| **Anthropic** | Manual — `cache_control: {type: "ephemeral"}` on message blocks | Exact prefix from breakpoints | Tied to usage pattern |
|
||
| **OpenAI** | Automatic for prompts ≥1,024 tokens | Exact prefix match; ~first 256 tokens hashed for routing | 5–10 min inactivity (up to 1 hr) |
|
||
| **vLLM / Self-hosted** | Automatic via hash-based block matching | Exact block match | Instance lifetime |
|
||
|
||
### Critical rule
|
||
|
||
> **Cache matching requires *exact* token sequences.** Any change — reordering messages, editing text, adding/removing tool results, even whitespace — alters the prefix hash and triggers a **cache miss**.
|
||
|
||
---
|
||
|
||
## Why Frequent Pruning Busts Cache
|
||
|
||
Every time you prune, you **rewrite the prefix**. The message that was previously a 3,400-token tool result is now a 200-token summary. That's a different token sequence, so the cache is invalidated.
|
||
|
||
### ASCII: The per-turn pruning trap (naive per-turn pruning)
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ NAIVE PER-TURN PRUNING — AGGRESSIVE BUT CACHE-UNFRIENDLY │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ Turn 1: Read file → 45 tokens │
|
||
│ └──► summarize → inject summary │
|
||
│ │ │
|
||
│ └──► CACHE BUST: context changed from [toolResult] to [sum] │
|
||
│ │
|
||
│ Turn 2: Read package.json → 120 tokens │
|
||
│ └──► summarize → inject summary │
|
||
│ │ │
|
||
│ └──► CACHE BUST again: prefix rewritten AGAIN │
|
||
│ │
|
||
│ Turn 3: Web search + read file → 3,600 tokens │
|
||
│ └──► summarize → inject summary │
|
||
│ │ │
|
||
│ └──► CACHE BUST again: prefix rewritten AGAIN │
|
||
│ │
|
||
│ After 5 turns: │
|
||
│ • 5 summarizer LLM calls (latency + cost) │
|
||
│ • 5 cache busts (FULL prefill every time) │
|
||
│ • No prefix ever stayed stable long enough to benefit from caching │
|
||
│ │
|
||
│ Time: ████████████████░░░░░░ ~80% spent on re-computing prefixes │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## The Sweet Spot: Batch and Prune
|
||
|
||
The insight is simple: **batch many tool turns, then prune once**. Everything before the prune point stays in the prefix cache and remains cacheable. Only the new suffix (since the last prune) needs fresh computation.
|
||
|
||
### ASCII: Batch pruning (`agent-message` mode)
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ agent-message MODE — BATCH THEN PRUNE (RECOMMENDED) │
|
||
├─────────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||
│ │ BATCH PHASE: Tool turns accumulate (not pruned yet) │ │
|
||
│ │ │ │
|
||
│ │ Turn 1: Read file → 45 tokens ═══════╗ │ │
|
||
│ │ Turn 2: Read package → 120 tokens ════╬══════╗ │ │
|
||
│ │ Turn 3: Web search + read → 3,600 tok ═════╬═╬═════╗ │ │
|
||
│ │ Turn 4: Edit file → 15 tokens ═══════╬═╬═╬═╬════╬═════╗ │ │
|
||
│ │ Turn 5: Build + read → 2,980 tokens ═╬═╬═╬═╬════╬═════╬═╗ │ │
|
||
│ │ Turn 6: Read + edit → 105 tokens ═════╬═╬═╬═╬════╬═════╬═╬ │ │
|
||
│ │ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ │ │
|
||
│ │ All these tool results stay in context UNCHANGED │ │
|
||
│ │ → Prefix cache is STABLE → cache HITS on every turn │ │
|
||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||
│ │ │
|
||
│ │ Agent sends final text reply │
|
||
│ ▼ │
|
||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||
│ │ PRUNE PHASE: Single summary replaces batch │ │
|
||
│ │ │ │
|
||
│ │ [summary] "Built React table component. Key decisions:..." │ │
|
||
│ │ │ │
|
||
│ │ ONE cache bust, then context is STABLE again │ │
|
||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||
│ │ │
|
||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||
│ │ STABLE PHASE: New requests reuse cached prefix │ │
|
||
│ │ │ │
|
||
│ │ User: "Now add sorting" │ │
|
||
│ │ → Cache HIT on [system] + [user] + [summary] prefix │ │
|
||
│ │ → Only "Now add sorting" needs prefill │ │
|
||
│ │ → Fast + cheap │ │
|
||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
│ Result: 1 cache bust per meaningful work unit, not per turn │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Mermaid: Cache-friendly pruning lifecycle
|
||
|
||
```mermaid
|
||
%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#e8f5e9', 'primaryTextColor': '#1b5e20', 'primaryBorderColor': '#4caf50', 'lineColor': '#4caf50', 'secondaryColor': '#ffebee', 'tertiaryColor': '#fff3e0'}}}%%
|
||
|
||
graph LR
|
||
subgraph Phase1["🟢 Stable Prefix<br/>(Cacheable)"]
|
||
direction TB
|
||
S1["System Prompt"] --> U1["User Request"]
|
||
U1 --> P1["Previous Summary"]
|
||
P1 --> C1["Current Turn"]
|
||
end
|
||
|
||
subgraph Phase2["🟡 Batch & Grow<br/>(Still Stable)"]
|
||
direction TB
|
||
B1["Turn N: tool calls"]
|
||
B2["Turn N+1: tool calls"]
|
||
B3["Turn N+2: tool calls"]
|
||
B1 --> B2 --> B3
|
||
end
|
||
|
||
subgraph Phase3["🔴 Prune Once<br/>(Cache Bust)"]
|
||
P["Single Summarize + Prune"]
|
||
end
|
||
|
||
subgraph Phase4["🟢 Stable Prefix Again<br/>(Cacheable)"]
|
||
direction TB
|
||
S2["System Prompt"] --> U2["User Request"]
|
||
U2 --> P2["New Summary"]
|
||
P2 --> C2["Current Turn"]
|
||
end
|
||
|
||
Phase1 -->|"agent starts<br/>tool work"| Phase2
|
||
Phase2 -->|"final text reply<br/>or loop ends"| Phase3
|
||
Phase3 -->|"context now<br/>compact & stable"| Phase4
|
||
Phase4 -->|"next task batch"| Phase2
|
||
|
||
style Phase1 fill:#e8f5e9
|
||
style Phase2 fill:#fff3e0
|
||
style Phase3 fill:#ffebee
|
||
style Phase4 fill:#e8f5e9
|
||
```
|
||
|
||
### Cache impact trade-off
|
||
|
||
| Mode | Cache Busts per 5 Turns | Context Reclaimed | Recommended for |
|
||
|---|---|---|---|
|
||
| `agent-message` | 1 | After batch | **Default** — best balance |
|
||
| `on-demand` | 0–1 | When you say so | Maximum cache preservation |
|
||
|
||
> **Everything before the previous pruning point stays in the prefix cache.** The cached prefix is the stable foundation; only the new suffix (recent turns since last prune) changes per request.
|
||
|
||
---
|
||
|
||
## Pre-flush Pipeline & Safeguards
|
||
|
||
`flushPending` runs a deterministic pipeline BEFORE any summarizer LLM call. Each step is configurable and each one can drop a batch entirely, in which case the prune frontier still advances so the same tool calls are not reconsidered next flush.
|
||
|
||
```
|
||
captured batches (from turn_end or session scan)
|
||
│
|
||
├─ 1. Protected-tools/paths filter (capture-time, see below)
|
||
│ tool calls whose toolName is in protectedTools, OR whose args.path
|
||
│ matches any protectedPaths glob, never enter the batch
|
||
│
|
||
├─ 2. Eager single-result spill (config: spillThreshold, default 65536)
|
||
│ turn_end: single result >= spillThreshold chars → write sidecar;
|
||
│ index immediately via addBatch (no LLM); pruner emits file-pointer stub;
|
||
│ dedup precedence: protected → dedup → spill (duplicate → alias, no second file)
|
||
│
|
||
├─ 3. Frontier trim (drop tool calls already past the frontier)
|
||
│
|
||
├─ 4. Content-hash dedup (config: dedupByContentHash, default ON)
|
||
│ identical (toolName, normalize(resultText)) → alias of original;
|
||
│ no LLM call; persist as context-prune-dedup-alias
|
||
│
|
||
├─ 5. Trivial-batch skip (config: minBatchChars, default 1000)
|
||
│ batches whose remaining raw chars < threshold → skip; no LLM call;
|
||
│ leave originals in context; advance frontier
|
||
│
|
||
├─ 6. Summarizer LLM call (parallel: one call per batch)
|
||
│ resolveModel + summarizeBatch / summarizeBatches
|
||
│
|
||
└─ 7. Oversized post-check (summary >= raw? → skip; advance frontier)
|
||
```
|
||
|
||
The outcome label written into `context-prune-frontier` is one of `summarized`, `skipped-deduped`, `skipped-trivial`, or `skipped-oversized` so the audit trail captures *why* a range was passed over.
|
||
|
||
### Stub-replace instead of delete
|
||
|
||
Before v0.11.0 the pruner deleted summarized `ToolResultMessage` entries outright. This worked, but it left the matching `AssistantMessage.toolCall` blocks orphaned, and pi-ai's `transformMessages.insertSyntheticToolResults` would inject `{ isError: true, content: "No result provided" }` for every orphan at LLM call time. The model then saw a parade of fake "tool errors" before the unrelated summary user-message appeared later, which mildly confused larger reasoning models.
|
||
|
||
The pruner now keeps the toolResult message but replaces its content with a small stub:
|
||
|
||
```
|
||
[Summarized in pruner summary, ref `tN`. Use context_tree_query to retrieve full output.]
|
||
```
|
||
|
||
Properties:
|
||
|
||
- `role: "toolResult"` and the original `toolCallId` / `toolName` / `timestamp` are preserved — role alternation is intact; no synthetic-result injection.
|
||
- `isError: false`, so the model does not interpret the stub as a tool failure.
|
||
- The stub references the **short ref** (`t1`, `t2`, …) the indexer assigned at summary time. Legacy entries from before short-refs landed fall back to the raw `toolCallId`.
|
||
- Deterministic per occurrence key — the stub text never changes across renders of the same `toolResult`, so the prefix cache continues to hit on the pruned range. A reused `toolCallId` is a *different* occurrence (different `resultTimestamp`) and gets its own stub and its own short ref.
|
||
|
||
Implementation: `src/pruner.ts` `pruneMessages(messages, indexer)` returns `{ messages, pruned }`. When `pruned === false`, the original array reference is returned and the `context` handler skips reconstruction entirely.
|
||
|
||
### Protected tools & paths
|
||
|
||
A tool call is protected if **either** its `toolName` is in `protectedTools` **or** its `args.path` (string) matches any glob in `protectedPaths`. Protected calls are filtered out **at capture time** - they never enter the `pendingBatches` queue, so their raw `ToolResultMessage` stays verbatim in future LLM context.
|
||
|
||
**`protectedTools: string[]`** (default `[]`) - allowlist of tool names. Covers tools whose output is a small handle that must be reused byte-for-byte (e.g. a session-id) or planning tools like `todowrite` / `todoread`.
|
||
|
||
**`protectedPaths: string[]`** (default `["**/skills/**/*.md"]`) - glob list matched against `args.path`. Designed for skill files that carry multi-step workflow gates; summarizing them is categorically lossy. Non-string or missing `path` arguments never match. Set `[]` to disable. Edit with `/pruner protected-paths`.
|
||
|
||
Glob contract: full-path match against the raw `args.path` string with `\` normalized to `/`. `*` and `?` match within a segment (no `/`); `**` crosses segments; `**/` also matches zero directories (so `**/SKILL.md` matches a bare relative `SKILL.md`). Case-sensitive. All other characters are regex-escaped literals.
|
||
|
||
**Render-time re-check:** stub replacement runs in-flight on every turn (`pruneMessages`). If a tool call's persisted `args` now satisfy `isProtected` (e.g. a pattern was added mid-session), the stub is skipped and the raw result is left verbatim - this repairs already-summarized records in existing sessions with no schema change. Declared limitation: records inside already-compressed chains (`context-prune-chain` entries) are NOT repaired - their `protectedToolCallIds` set is fixed at compression time (forward-only). Dedup-alias edge: an alias resolving to an unprotected original stays stubbed.
|
||
|
||
Names and patterns that don't match any captured tool call are silently ignored.
|
||
|
||
### Eager single-result spill
|
||
|
||
`spillThreshold: number` (default `65536`) is a capture-time safeguard for outsized single tool results (e.g. a 1 MB web fetch, a full binary diff). When a single `ToolResultMessage`'s `resultText.length` reaches the threshold, the result is spilled immediately at `turn_end` — before the pending-queue trim and before any LLM call.
|
||
|
||
**Sidecar location:** `<sessionDir>/<sessionId>-blobs/<sanitizedToolCallId>.txt`.
|
||
|
||
**Index entry:** `addBatch` is called synchronously with the spilled body (no LLM round-trip). The record is immediately `isSummarized = true`; the pruner emits a mechanical file-pointer stub:
|
||
|
||
```
|
||
[Spilled: <toolName> (<N> bytes). Head preview:
|
||
<first spillPreviewBytes bytes>
|
||
Full output: <sidecar path>. Use context_tree_query(<shortRef>) to retrieve.]
|
||
```
|
||
|
||
**Dedup precedence:** `protected → dedup → spill`. An oversized result that is also a content-hash duplicate of a prior record is aliased to the original; no second sidecar is written.
|
||
|
||
**Atomicity:** the sidecar is written first; only on success is the in-memory record mutated. A write failure leaves the result inline for the normal flush and is logged via `console.error` — no data is lost.
|
||
|
||
**Hybrid storage:** bodies below `spillThreshold` stay inline in the `context-prune-index` session entry (portable, as before); only oversized bodies are spilled. Moving the session `.jsonl` without its `-blobs/` directory loses only the giant-blob recovery path; the stub and head preview remain in the index entry.
|
||
|
||
### Trivial-batch skip (minBatchChars)
|
||
|
||
`minBatchChars: number` (default `1000`) is a pre-flush guard against "summary would be roughly the same size as the input" cases. If the total raw `resultText` across a batch is below the threshold, the batch is skipped: no summarizer LLM call, no `context-prune-index` entry, no `context-prune-summary` injection. The frontier still advances, so the same tool calls are not reconsidered next flush.
|
||
|
||
Why it exists: a short LLM summary like "Tool X did Y" is itself ~50–150 chars per call. For a 200-byte file read or an `ls` of a short directory, the summary is the same size or larger than the input — the post-call `skipped-oversized` mechanism would catch it anyway, but only after the LLM round-trip and the cost. `minBatchChars` short-circuits the obvious cases at zero LLM cost.
|
||
|
||
Set `minBatchChars: 0` to disable. The default `1000` skips obvious trivial batches (`git status`, small file reads, short directory listings) without affecting realistic tool outputs. Edit with `/pruner min-batch-chars <n>` or via the settings overlay.
|
||
|
||
### Content-hash dedup
|
||
|
||
`dedupByContentHash: boolean` (default `true`) catches re-reads of already-pruned tool outputs at zero LLM cost.
|
||
|
||
Mechanism:
|
||
|
||
1. When a batch enters `flushPending`, each tool call is hashed by `SHA-1(toolName + "\0" + normalize(resultText))`.
|
||
2. The indexer's `contentHashToOriginal` map (populated by every earlier `addBatch` / `reconstructFromSession`) is consulted.
|
||
3. A hit means an earlier prune already covered identical content. The duplicate is registered as an alias of the original via `indexer.registerDuplicate(newKey, originalKey, appendEntry)`, where both are occurrence keys (or legacy bare ids):
|
||
- `dedupAliasToOriginal[newKey] = originalKey` (so `isSummarized(newKey) === true` and `resolveToolCallId(newKey) === originalKey`).
|
||
- `toolCallIdToAlias[newKey] = toolCallIdToAlias[originalKey]` (so `getShortRefForToolCallId(newKey)` returns the **same** `tN` as the original).
|
||
- Keying by occurrence key, not bare id, means a duplicate is only ever aliased to the specific earlier *occurrence* it matches byte-for-byte - a reused id whose later occurrence has different content is not conflated with the stale one.
|
||
- A `context-prune-dedup-alias` custom entry is persisted so `reconstructFromSession` rebuilds the maps after a restart.
|
||
4. The duplicate is removed from the batch — no summarizer call, no new index entry.
|
||
5. Later, `pruneMessages` stub-replaces the duplicate's `ToolResultMessage` using the original's short ref, and `context_tree_query` returns the original's record whether the model passes the duplicate's id or the original's.
|
||
|
||
Normalization is conservative: `\r\n` → `\n`, per-line trailing whitespace stripping, final `trim()`. Internal whitespace, tabs, and capitalization are preserved so two genuinely different outputs do **not** collide.
|
||
|
||
Typical wins: re-reading an unchanged file, repeated `git status` / `ls`, retries of the same command. v1 only matches against records **already in the indexer** (cross-flush dedup); intra-flush dedup is deferred so canonicals that get skipped as oversized / trivial never produce dangling aliases.
|
||
|
||
Edit with `/pruner dedup on|off|status` or the settings overlay.
|
||
|
||
### Oversized summary skip
|
||
|
||
Last-resort safeguard: if the summarizer LLM produces a summary longer than the raw tool-result text it would replace, the batch is left untouched — the original tool results stay in context, no summary is injected, and the frontier still advances so the next prune attempt starts after this range instead of retrying it. The `quietOversizedSkips` config silences the info notification (the skip itself still happens).
|
||
|
||
This is rare in practice once `minBatchChars` is on, because the cases where summarization makes things bigger are exactly the cases the trivial-batch skip already catches earlier.
|
||
|
||
### Frontier persistence
|
||
|
||
The last attempted prune boundary is persisted as `context-prune-frontier` so `flushPending` knows where the previous attempt left off, even if that attempt was a skip rather than a real summary. Without this, a batch that's been skipped as oversized would be re-attempted (with the same LLM call, the same oversize result, the same skip) on every subsequent flush.
|
||
|
||
### Other UI / observability features
|
||
|
||
- **Tree browser (`/pruner tree`):** interactive, foldable tree of pruned tool calls grouped under their summaries. `Ctrl-O` on a summary node opens the full markdown summary in a bordered overlay.
|
||
- **Configurable summarizer thinking (`summarizerThinking`):** trade summary cost / latency for quality (`off` / `minimal` / `low` / `medium` / `high` / `xhigh`). `default` omits the option entirely so the provider chooses.
|
||
- **Cumulative stats:** `context-prune-stats` entries track input/output tokens and cost of every summarizer call; full detail surfaces in `/pruner stats`. Cost is also emitted on the `cost:external` pi.events channel for external aggregators (cumulative per session, live only).
|
||
- **Live reclaim ratio:** measured once per `pruneMessages` call via `sizeMessages(messages) = JSON.stringify(messages).length`, comparing the input array before pruning to the result after. Estimated tokens = chars / 4. The measurement covers all four phases in a single point (stub-replace, error-purge, chain-range-prune, orphan-sweep); appears on the status line as `│ prune: ON · 92.0k->14.0k (-85%)` once at least one prune has occurred (the leading `│` keeps the segment visually isolated in the shared footer, load-order independent - there is no trailing divider, since the footer's own space-join between segments already provides one).
|
||
- **Live progress for `/pruner now`:** an `aboveEditor` widget shows one row per pending batch with braille spinner, streamed summary-char count, and ✓ / ⚠ status.
|
||
|
||
### Summarizer outage fallback
|
||
|
||
Provider incidents are routinely per-model: a cheap summarizer model (e.g.
|
||
Haiku) can be down while the session's main model (Sonnet/Opus) stays healthy.
|
||
Without a fallback, `runSummarization` returns `null`, the batch is re-queued,
|
||
and the next flush retries the same dead model — pruning stalls for the whole
|
||
outage and context grows unbounded.
|
||
|
||
`src/summarizer-fallback.ts` adds a sticky, in-memory `FallbackController` that
|
||
engages only on **transient** failures of the configured summarizer model and
|
||
only when a distinct fallback model exists (`summarizerModel` != `default` and
|
||
the resolved model differs from `ctx.model`):
|
||
|
||
- **Classification is coarse** (pi-ai surfaces no status code on the throw):
|
||
`auth` (pre-flight key failure) and `unusable` (empty / length-truncated)
|
||
never trip the controller; a `transient` stream error / `stopReason: error`
|
||
does. Aborts propagate unchanged.
|
||
- **Enter:** a transient primary failure is retried once on the session model.
|
||
If that succeeds, the session flips to fallback and a one-time warning fires.
|
||
- **Sticky + probe:** while in fallback, all calls route to the session model.
|
||
After a 10-minute cooldown (`COOLDOWN_MS`, internal, not configurable) one
|
||
batch of the next flush probes the primary; success recovers (info notify),
|
||
failure stays in fallback.
|
||
- **In-memory only:** no `context-prune-*` entry; `reset()` on `session_start`.
|
||
A restart mid-outage re-detects on the next flush.
|
||
|
||
Cost note: the initial detection flush can fire up to N doomed primary calls
|
||
before the outage is known; steady state is 0 doomed calls, plus exactly 1
|
||
probe per 10 minutes. After the primary recovers, summarization keeps running
|
||
on the (often pricier) session model for up to one cooldown before the probe
|
||
switches back.
|
||
|
||
---
|
||
|
||
## Why Summarization Works: Research Evidence
|
||
|
||
Summarizing tool-call history is not just a hack — it is an active research area with strong empirical support. Three recent papers establish the benefits:
|
||
|
||
---
|
||
|
||
### SUPO — Summarization augmented Policy Optimization
|
||
|
||
> **Paper:** *SUPO (arXiv:2510.06727)* — Miao Lu et al.
|
||
> **TL;DR:** RL-trained agents with built-in summarization outperform standard agents on long-horizon tasks while using *less* context.
|
||
|
||
**Core idea:**
|
||
SUPO integrates summarization directly into the RL training pipeline for tool-using agents. Instead of treating context compression as an afterthought, the policy gradient is derived to optimize **both** tool-use behavior **and** summarization strategy end-to-end.
|
||
|
||
**Method:**
|
||
|
||
- Periodically compresses tool-using history via LLM-generated summaries
|
||
- Retains task-relevant information in compact form
|
||
- Derives a policy gradient that lets standard LLM RL infrastructure optimize both behaviors simultaneously
|
||
- Enables training beyond fixed context limits
|
||
|
||
**Key results:**
|
||
|
||
- Significantly improved success rate on interactive function calling and search tasks
|
||
- **Same or lower working context length** compared to baselines that don't summarize
|
||
- Test-time scaling: increasing the maximum summarization rounds during evaluation further improves performance
|
||
|
||
**Why this matters for Pi:**
|
||
SUPO proves that summarization is not just about saving tokens — it actively **improves task success** on long-horizon multi-turn problems by preventing context overflow and keeping relevant signals prominent.
|
||
|
||
---
|
||
|
||
### ReSum — Recursive Summarization for Long-Horizon Agents
|
||
|
||
> **Paper:** *ReSum (arXiv:2509.13313)* — Xixi Wu et al. (Alibaba)
|
||
> **TL;DR:** A plug-and-play summarization tool enables web agents to explore indefinitely without hitting context limits, achieving 4.5–12.7% gains over ReAct.
|
||
|
||
**Core idea:**
|
||
ReSum addresses the fundamental conflict between **exploration** (needing many tool calls) and **context limits** (fixed window size). Current agents append every thought/action/observation to history until they crash into the context ceiling.
|
||
|
||
**Method:**
|
||
|
||
- Periodically invokes an external **summary tool** to condense interaction history
|
||
- The agent restarts reasoning from the compressed summary
|
||
- Introduces **ReSum-GRPO**: adapts Group Relative Policy Optimization with **advantage broadcasting** — propagates final trajectory rewards across all segments so early exploration steps get proper credit
|
||
- Trained a specialized **ReSumTool-30B** to extract key evidence and propose next steps
|
||
|
||
**Key results:**
|
||
|
||
- **4.5% improvement** over ReAct in training-free settings
|
||
- **Further 8.2% gain** with ReSum-GRPO training
|
||
- A 30B ReSum-enhanced agent with only 1K training samples achieves competitive performance with leading open-source models
|
||
- Enables "unbounded exploration" — the agent never hits a hard context wall
|
||
|
||
**Why this matters for Pi:**
|
||
ReSum validates the exact architecture `pi-condense` uses: an external summarizer module, periodic compression, and recovery from compressed state. The plug-and-play nature means it works with off-the-shelf agents — no retraining required.
|
||
|
||
---
|
||
|
||
### ACON — Agent Context Optimization
|
||
|
||
> **Paper:** *ACON (arXiv:2510.00615)* — Minki Kang et al. (Microsoft/ KAIST)
|
||
> **TL;DR:** Optimized compression guidelines reduce memory by 26–54% while preserving accuracy; distilled compressors retain >95% of performance.
|
||
|
||
**Core idea:**
|
||
ACON is a unified framework that compresses **both** environment observations and interaction histories into "concise yet informative condensations." It treats compression as an optimization problem: maximize task reward while minimizing context cost.
|
||
|
||
**Method:**
|
||
|
||
- **Gradient-free** — uses natural language space optimization (no model fine-tuning)
|
||
- **Failure-driven guideline optimization:** runs the agent with and without compression, collects cases where compression caused failure, and uses an optimizer LLM to refine compression guidelines
|
||
- Two-step alternation:
|
||
1. **Utility maximization** — ensure task success is preserved
|
||
2. **Compression maximization** — make summaries shorter while keeping sufficiency
|
||
- **Distillation:** optimized compressor can be distilled into smaller models (e.g., Qwen-14B) with >95% accuracy retention
|
||
|
||
**Key results:**
|
||
|
||
- **26–54% reduction in peak tokens** across AppWorld, OfficeBench, and Multi-objective QA
|
||
- Preserves task performance with large models
|
||
- **Smaller LMs improve 20–46%** as agents when context compression removes distracting noise
|
||
- Distilled compressor retains **>95% accuracy**
|
||
|
||
**Why this matters for Pi:**
|
||
ACON demonstrates that **compression not only saves tokens but can improve agent performance** — especially for smaller models, where long noisy context actively degrades reasoning quality. The failure-driven optimization approach shows that even simple summarization, when guided by task structure, preserves critical signals.
|
||
|
||
### Token-budget auto-flush trigger
|
||
|
||
`autoBudgetThreshold` (maintained default `0.735`) is an ADDITIONAL flush trigger orthogonal to `pruneOn`. When set to a fraction in `(0, 1]`, the extension evaluates context usage at the end of every tool-using turn; when `tokens` reaches `min(MAX_BUDGET_WINDOW, threshold * contextWindow)` - i.e. the configured share of the model's window, or 300,000 tokens, whichever comes first - all pending batches are flushed immediately regardless of the configured `pruneOn` mode. `null` disables the trigger.
|
||
|
||
**Why we compute this ourselves rather than using `ContextUsage.percent`:** the provider's `percent` field is a 0-100 value, and both it and `tokens` are `null` immediately after a provider-side compaction. Comparing `tokens` against a token level we derive ourselves keeps the unit unambiguous and is independently null-safe - a `null` tokens value makes the trigger a no-op until usage is reported again. Note this trigger never divides: the fraction form (`tokens / min(contextWindow, MAX_BUDGET_WINDOW)`) belongs to `usageFraction` and the delta trigger below, and reading the threshold as a share of a capped window would be wrong - on a 1M-window model, `0.4` fires at 300,000 tokens, not at 120,000.
|
||
|
||
**Why the 300k ceiling (`MAX_BUDGET_WINDOW`, `src/budget.ts`):** the trigger was originally a pure fraction of the advertised window, which made it unreachable as windows grew. On a 1M-window model, `0.9` means 900k tokens - a session ends long before that, so users observed "pruning never happens" (issue #7), while the same setting worked on a 200k model. The ceiling is deliberately chosen so it never binds at or below a 300k advertised window (a 256k model at `0.9` still fires at 230.4k), so it changes behavior only where the fraction was already unusable. Users keep full downward control: `0.1` on a 1M model still means 100k tokens.
|
||
|
||
Lineage: simplified take on DCP's `maxContextLimit` nudging — a single threshold that forces a flush rather than separate nudge/force thresholds.
|
||
|
||
### Budget-delta flush
|
||
|
||
`budgetTurnDelta: number | null` (maintained default `0.04`) is a per-turn usage-jump trigger ORed with `autoBudgetThreshold`. When set to a fraction in `(0, 1]`, the extension compares the current turn's usage fraction (`tokens / min(contextWindow, MAX_BUDGET_WINDOW)`) to the previous turn's and forces a flush if the jump meets or exceeds the delta. Because the denominator carries the same 300k ceiling, the required growth is `delta * min(contextWindow, MAX_BUDGET_WINDOW)` tokens: `0.04` = +12k on any model at or above 300k, +8k on a 200k model. The ceiling enters through the denominator here rather than bounding a level, because a 300k ceiling on one turn's growth could never bind. Note the fraction is therefore not bounded by 1 above the ceiling (600k tokens on a 1M window = 2.0) and is deliberately not clamped - clamping would saturate this trigger and stop it re-arming.
|
||
|
||
Use case: a single enormous tool result can jump context usage by 20–30 percentage points in one turn; `autoBudgetThreshold` misses this until the next turn. `budgetTurnDelta` catches the spike immediately.
|
||
|
||
**`previousFraction` tracking:**
|
||
|
||
- Reset to `null` on `session_start` and `session_tree` (session reload).
|
||
- Left unchanged on a null-tokens turn immediately following a provider-side compaction (treating a post-compaction null as `0` would produce a spurious spike on the next real turn).
|
||
- The post-restart first turn cannot fire a delta trigger (no prior fraction to compare) and falls back to `autoBudgetThreshold` alone.
|
||
|
||
`null` = off; the maintained bundle enables this trigger at `0.04`.
|
||
|
||
---
|
||
|
||
## Chain Compression
|
||
|
||
Chain compression is a second layer on top of the per-batch tool-result stub pruner. It operates on entire closed conversation chains rather than individual tool results, reclaiming the tokens that the stub pruner cannot touch: assistant thinking blocks, encrypted thinking signatures, and tool-call argument bodies.
|
||
|
||
### What a closed chain is
|
||
|
||
A **closed chain** is a span of messages from one user message through any number of tool-using assistant turns and their results, ending in a final text-only assistant reply:
|
||
|
||
```
|
||
[user msg] ← chain start (kept raw)
|
||
[assistant: thinking + toolCalls] ← middle (dropped)
|
||
[toolResult] ← middle (dropped)
|
||
[assistant: thinking + toolCalls] ← middle (dropped)
|
||
[toolResult] ← middle (dropped)
|
||
[assistant: text-only] ← chain close (kept, thinking stripped)
|
||
```
|
||
|
||
### What gets dropped vs. kept
|
||
|
||
| Part | After chain compression |
|
||
|---|---|
|
||
| Start user message | **Kept raw** |
|
||
| Middle assistant turns (all) | **Dropped** — assistant thinking + signatures + toolCall argument blocks |
|
||
| Middle tool results (all) | **Dropped** — already stub-replaced by the per-batch pruner; now fully removed |
|
||
| Per-batch summary message(s) for this chain | **Suppressed** — replaced by the chain-level synthetic |
|
||
| Final text-only assistant | **Kept**, thinking blocks stripped (safe — no following tool cycle depends on the signature) |
|
||
| Synthetic `<compressed-chain>` user message | **Injected** immediately after the start user message |
|
||
|
||
**Why there is no separate thinking strip.** The synthetic `<compressed-chain>` block is injected as a `role: "user"` text message, which Anthropic reads as a genuine user turn. That closes the assistant cycle, and the API drops every prior thinking block from the context window server-side - verified against the live API: the same 8-turn loop bills 3152 thinking tokens with a full `toolResult` history and 0 once a user text block precedes the final assistant. So chain compression already reclaims thinking mass for free, as a side effect of the range drop that was rewriting that region anyway. Stub replacement does not have this effect: it preserves `role: "toolResult"`, so the cycle stays open. A dedicated strip phase shipped through v2.4.3; removed in v2.5.0 - see `doc/specs/2026-08-05-remove-thinking-strip.md`.
|
||
|
||
### Transform composition order
|
||
|
||
```
|
||
raw messages from session
|
||
│
|
||
├─ [1] tool-result stub-replace (per-batch; existing)
|
||
├─ [2] error-purge (phase 2)
|
||
├─ [3] chain-range-prune (runs AFTER stubs)
|
||
│ resolve each entry to a positional index range
|
||
│ drop assistant / toolResult / per-batch-summary inside the range
|
||
│ inject <compressed-chain> after start user
|
||
│ strip thinking from final assistant
|
||
└─ [4] orphan sweep (structural post-condition, see below)
|
||
```
|
||
|
||
### Identification model: positional ranges, not ids
|
||
|
||
Pi-ai's `Message` union (`UserMessage | AssistantMessage | ToolResultMessage`) has no `.id` field, and provider `toolCallId`s are not session-durable (see [Occurrence Identity](#occurrence-identity) below). Chain compression therefore decides **what to drop** by message position, not by id membership.
|
||
|
||
`resolveRange` (`src/chain-range-prune.ts`) maps a persisted `ChainCompressionEntry` to an index range in the current message array:
|
||
|
||
- exactly one `user`-role message at `entry.startUserTimestamp`
|
||
- exactly one `assistant`-role message at `entry.finalAssistantTimestamp`
|
||
- `startIndex < endIndex`
|
||
|
||
Any other outcome - zero or multiple matches on either boundary, `finalAssistantTimestamp === null`, or a non-ordered pair - resolves to `null` and the entry drops **nothing** and inserts **no synthetic** for that range. This is fail-closed on purpose: an id-set or timestamp-window fallback (accepting a match even when it's ambiguous) risks deleting a live assistant turn that reuses a compressed chain's provider id, orphaning its tool result and producing a rejected request. A resolution failure is invisible in context - it is recorded as an `unresolved-range` diagnostic (see [Diagnostics](#diagnostics) below) and the affected range simply stays raw in context instead of being silently mis-compressed.
|
||
|
||
**Drops are role-restricted**, not index-membership-restricted: inside an accepted range, only `assistant`, `toolResult`, and per-batch-summary (`custom`, `context-prune-summary`) messages are removed. `user`-role messages inside the range - including the already-inserted `<compressed-chain>` synthetic, which is itself `role: "user"` - and any third-party `custom_message` entries survive untouched. Preserving the synthetic is what makes re-application idempotent: calling `applyChainCompressions` a second time with the same entries sees its own synthetic already present (matched by `blockId`) and skips re-inserting it.
|
||
|
||
**Per-batch summary suppression is coverage-based, not index-membership-based.** Under `batchingMode: "agent-message"` a per-batch summary is appended *after* `finalAssistantTimestamp`, i.e. outside the resolved range. Suppressing it by index membership would miss it entirely, so suppression instead checks whether the summary's own `toolCallRefs` overlap the set of ids actually dropped in this pass (`perBatchSummaryOverlapsDropped`).
|
||
|
||
**Diagnostic cross-check.** Each accepted entry also carries `droppedToolCallIds` from detection time. At render time the ids actually inside the resolved range are compared against that recorded set purely as a health check - a mismatch never changes what gets dropped (the range always wins) but emits a `range-id-mismatch` diagnostic, surfacing drift between detection-time and render-time state without ever acting on it.
|
||
|
||
### Rolling window
|
||
|
||
`chainCompression.rollingWindow` (default `3`) controls how many recently-closed chains stay raw. Once the (K+1)-th chain closes, the oldest chain beyond the window is compressed.
|
||
|
||
- With K=3: the three most-recently-closed chains stay as-is; the fourth triggers compression of the oldest. Exactly K closed chains are kept raw (the open/live chain is never counted).
|
||
- `/pruner compact` bypasses the window and compresses every eligible chain immediately (retroactive).
|
||
|
||
**Closing-message threading.** In agent-message mode the flush runs from `message_end`, which pi emits to extensions *before* it persists the message to the session (`agent-session.js` runs `_emitExtensionEvent` ahead of `sessionManager.appendMessage`). So at flush time `getBranch()` is missing the just-closed final assistant, and the newest chain would read as open — over-retaining by one (effective K+1). `index.ts` threads the triggering `event.message` through `FlushOptions.closingMessage` into `withClosingMessage` so the chain closes and the window is exactly K.
|
||
|
||
### Synthetic message format
|
||
|
||
The injected user message body:
|
||
|
||
```
|
||
<compressed-chain id="b1" tools="t3,t4,t5">
|
||
[existing per-batch summary text]
|
||
</compressed-chain>
|
||
```
|
||
|
||
- `id="b1"` is a stable monotonic block ID (`b1`, `b2`, …) assigned at compression time and persisted in the `context-prune-chain` session entry.
|
||
- `tools="t3,t4,t5"` lists the short `tN` refs from the per-batch index so the model can call `context_tree_query` to recover individual tool outputs.
|
||
- The body is the cohesive LLM **range summary** when present (see below), else the existing `context-prune-summary` text reused verbatim.
|
||
|
||
### Range summary fusion
|
||
|
||
By default (`chainCompression.fuseRangeSummary`, default `true`) a compressed chain's per-batch summaries are fused into ONE cohesive summary by a single summarizer call at compression time, persisted as `rangeSummaryText` on the `context-prune-chain` entry and used as the synthetic body. This is recursive summarization (summary-of-summaries): the input is the already-pruned per-batch summary text, so it never re-sends raw tool output.
|
||
|
||
- **Gate:** only spans with >= 2 distinct per-batch summaries are fused (nothing to fuse otherwise); single-summary spans use that summary directly.
|
||
- **Non-fatal:** if the fusion call fails or returns empty, the entry stores no `rangeSummaryText` and the renderer falls back to the per-batch concatenation — the chain still compresses.
|
||
- **Cost:** one extra summarizer call per multi-batch span, charged to the same summarizer model and folded into the usage stats (`rangesSummarized` counter). Set `fuseRangeSummary: false` to keep the zero-LLM concatenation.
|
||
- **Recovery:** unchanged — the span's tool outputs stay in `context-prune-index` and are recoverable via `context_tree_query`; the raw span text stays in the session JSONL.
|
||
|
||
### Cache impact
|
||
|
||
Each new chain compression busts the prefix cache from the affected chain's start-user-message onward. Mitigation: the rolling window concentrates compression decisions at predictable points (one event per chain close), so cache is only invalidated when an old chain ages out of the window — not on every turn.
|
||
|
||
### Recovery
|
||
|
||
Chain compression does not delete data from the session JSONL. The original tool outputs remain in `context-prune-index` entries and are recoverable via `context_tree_query`. To undo compression of a specific chain, delete the matching `context-prune-chain` entry from the session file — the chain will re-appear in context on next load.
|
||
|
||
### Protected-output relocation
|
||
|
||
**Contract:** outputs protected by tool name (`protectedTools`) or path glob (`protectedPaths`) must never be pruned from LLM context. The per-batch pipeline honours this at capture time (protected tool calls never enter the batch). Chain compression, however, operates at the *message range* level and drops entire middle turns wholesale - which would silently remove any protected `ToolResultMessage` residing in those turns.
|
||
|
||
**Mechanism:** detection (`detectChains`) records which middle tool-call ids are protected on the `ChainRange`; `compressEligible` copies that id list onto the persisted `ChainCompressionEntry.protectedToolCallIds` (it stores ids only, no text). At render time, `applyChainCompressions` pulls each protected `ToolResultMessage`'s verbatim text live from the raw branch and embeds it inside the synthetic `<compressed-chain>` block:
|
||
|
||
```
|
||
<compressed-chain id="b1" tools="t3,t4,t5">
|
||
[range summary text]
|
||
<protected-output tool="todoread">...verbatim output...</protected-output>
|
||
</compressed-chain>
|
||
```
|
||
|
||
The protected output is relocated (moved), not copied — the original `ToolResultMessage` is dropped with the rest of the middle turns. The text stays in LLM context because it is embedded in the surviving synthetic block. It is NOT registered in the tool-call index and is NOT recoverable via `context_tree_query`; it does not need to be, because it is present verbatim.
|
||
|
||
The `context-prune-chain` session entry carries the matching `protectedToolCallIds` array so `session_start` reconstruction can re-embed the outputs on reload.
|
||
|
||
**Rejected alternative:** skip compression for any chain that contains a protected tool. Rejected because `todowrite`/`todoread` recur in most chains for opted-in users, so this strategy would forfeit most chain compression for the people who most need `protectedTools`.
|
||
|
||
### Deterministic fallback (uncovered chains)
|
||
|
||
A chain becomes eligible for compression (closed, older than the rolling window) independently of whether its middle tool calls were ever summarized. Per-batch coverage can be zero - a trivial batch, an oversized-skip, a fully-deduped batch, or a plain capture miss - and historically that meant `compressEligible` skipped the chain forever with `reason: "no-summary"`: the prune frontier had already advanced past the span, so no future flush would recapture it. A 639-call, ~935k-char chain stranded live this way (`doc/specs/2026-08-14-uncovered-chain-deterministic-backfill.md`).
|
||
|
||
When `hasPerBatchSummaryCoveringAny` is false for a chain's middle ids, `compressEligible` takes a second, zero-LLM branch instead of skipping:
|
||
|
||
1. **Resolve + extract.** `extractChainRecords` resolves the chain's span with the same `resolveRange` used by the drop path, then walks it positionally. Middles are excluded when protected (relocated verbatim at render, never phase-1 stubbed) or already indexed (occurrence-key membership in the indexer's record map - not `isSummarized`, which also covers dedup aliases and would wrongly re-strand a fully-deduped chain). Each surviving call becomes a `ToolCallRecord` with `turnIndex: -1` (no batch turn; `context_tree_query` renders `Turn: -1`, a pinned cosmetic).
|
||
2. **Backfill, atomically.** `indexer.backfillChainRecords` spills oversized results via the shared spill helpers (`blobPathFor`/`applySpill`), a fail-closed variant of the eager `spillOversizedBatch` path, allocates `t<N>` refs, and appends **one** `context-prune-index` entry carrying the records plus `backfilled: true` and the allocated refs - append-before-commit, so refs are durable the instant the records are. Only after the append succeeds does it commit to the in-memory index and alias maps. Backfilled records never seed `contentHashToOriginal`: a poisoned dedup canonical could point future identical outputs at a record whose durability was never actually verified end-to-end for that purpose, so backfilled entries are excluded from canonical-seeding on both the live path and `reconstructFromSession`.
|
||
3. **Compose a deterministic body.** `buildDeterministicBody` builds a zero-LLM stub - call count, a tool-name histogram, span duration, and a `First:`/`Last:` line with args JSON capped at 200 chars, plus the `t<N>` refs - and stores it as `rangeSummaryText` on the `context-prune-chain` entry with `bodySource: "deterministic"`. The renderer already prefers `entry.rangeSummaryText` (`src/pruner.ts`), so no renderer change was needed; entries without `bodySource` keep today's semantics unchanged.
|
||
4. **Fail closed.** Any throw during backfill (spill I/O, append failure, malformed span) preserves the historical `no-summary` skip - nothing partial is committed. A chain with zero freshly-extracted records is only treated as a genuine span mismatch (fail-closed skip + `backfill-empty` diagnostic) when it ALSO has zero members already present in the index; if a prior attempt's index append succeeded but the chain-entry append then failed, the retry composes and persists straight from the durable index records instead of re-extracting, so recovery never produces a second `context-prune-index` entry for the same span.
|
||
|
||
**Fully-protected exception.** If every middle id in a zero-extractable, zero-indexed chain is protected (`chain.protectedToolCallIds` covers all of `chain.middleToolCallIds`), the chain stays on the plain `no-summary` skip with no `backfill-empty` diagnostic. Compressing would relocate every output verbatim into the synthetic body anyway (zero tokens saved), and the diagnostic would misreport a healthy span - firing on every restart for protected-heavy configs.
|
||
|
||
**Known limitation.** `flushPending` returns early when there are no pending batches, before chain detection runs at all. A chain stranded in an otherwise-idle session is not healed by `/pruner now` on an empty queue - it heals on the next flush that has any work, or immediately via `/pruner compact`.
|
||
|
||
**Cache-prefix impact.** Same as any chain compression: rewriting the span busts the prefix cache from that chain's start-user-message onward, once. The deterministic body has no `now()` or LLM nondeterminism, so it renders byte-identical on every subsequent pass - no repeated invalidation from retries or reloads.
|
||
|
||
### Deferred
|
||
|
||
- **Model-driven trigger.** The compressor is autonomous (rolling window). A model-callable compress tool (DCP-style: the model compresses a sub-task as it closes) is not implemented; the earlier scaffolded `agentic-auto` mode + `context_prune` tool were removed in v1.0.0.
|
||
- **Multi-turn span merging.** Each compressed span maps 1:1 to a closed chain (one user -> text-only-assistant round). Merging several consecutive closed spans into one topic summary is future work.
|
||
|
||
### Single-chain sessions
|
||
|
||
Phase 3 (chain compression) only ever acts on **closed** chains - a chain closes when a text-only assistant reply follows the tool-call round (see [What a closed chain is](#what-a-closed-chain-is)). A session that runs one long, uninterrupted tool chain - an autonomous agent loop that never emits a text-only reply - never closes a chain, so the rolling window has nothing to compress. The entire run accumulates as one open segment that Phase 3 cannot touch. This is a known design property, not a bug: closed-chain detection is load-bearing for Phase 3's positional-range resolution (`resolveRange`), and there is no forced-close/reopen valve (see [Deferred](#deferred) above and issue #6 - a corpus replay would be needed before any such valve is considered).
|
||
|
||
Phase 1 (per-batch summarization) is unaffected by chain closure and remains the only lever for this shape of session. For long autonomous runs where no chain is expected to close:
|
||
|
||
- Lower `autoBudgetThreshold` (e.g. `0.5`-`0.6` instead of `0.8`) so the budget trigger fires well before the open segment dominates the window. On a window larger than 300k the ceiling already caps the trigger point at 300,000 tokens once `threshold * contextWindow` exceeds it - that is any setting at or above `300k / contextWindow` (`0.3` on a 1M window, `0.75` on a 400k one) - so lowering the setting only matters below that point.
|
||
- Set `budgetTurnDelta` so a single turn's sudden context jump force-flushes even between budget-threshold crossings - this catches spikes a static threshold misses until the next turn.
|
||
|
||
Neither knob makes a chain close; they just keep Phase 1 flushing on schedule so raw toolResults do not pile up unsummarized for the whole run.
|
||
|
||
**Observability metrics** (`src/context-metrics.ts`, `computeContextMetrics`) exist precisely to make this shape of session visible instead of silently reporting `calls: 1` the way the triggering incident did. All three are chars/4 token estimates (`Math.round`, same convention as the reclaim footer) and surface on `/pruner status` (a `--- context ---` block), the footer status line (a compact suffix, shown only when the frontier gap is non-zero), and a `context-prune-flush-metrics` session entry written once per flush attempt regardless of outcome:
|
||
|
||
| Metric | Definition |
|
||
|---|---|
|
||
| Open-cycle thinking tokens | Est. tokens of `thinking` content blocks in assistant messages strictly after the last text-only assistant reply (the open segment). Not windowed by the frontier, so a skip/oversized/trivial outcome (which advances the frontier without removing anything from context) never masks stranded thinking as ~0. |
|
||
| Largest-chain share | `max(largest closed chain, open segment)` chars, as a percentage of total branch chars. Interrupted chains count as closed for this purpose - they are retained context regardless of compressibility. A single-chain session's whole branch is its open segment, so this reads near 100% for the incident shape. |
|
||
| Frontier gap | Est. tokens of `ToolResultMessage`s after the persisted prune frontier that are summarization-eligible: not already summarized, not protected (`protectedTools`/`protectedPaths`). 0 when there is nothing left to capture. |
|
||
|
||
See `doc/specs/2026-08-12-single-chain-observability-trigger-repair.md` for the incident and full design rationale.
|
||
|
||
### Reload rearm
|
||
|
||
A reload (`session_start` or `session_tree`) rescans the branch for completed-but-unflushed tool-call batches past the frontier (the same rescan `flushPending` itself uses, no queue reconstruction). If that rescan finds recoverable work, a transient in-memory flag arms: the next `turn_end` evaluates the budget/delta trigger even on a turn that contributes no new tool results itself, instead of silently requiring a fresh batch to reach the gate. No flush runs at reload time - arming only changes what an *existing* trigger sees on the next turn.
|
||
|
||
The flag clears the moment any flush attempt runs (any outcome) and is re-evaluated on the next reload. A reload followed by total idleness - no further turns at all - gets visibility only: the footer, `/pruner status`, and the `agent_end` pending notice (`prune: recovered pending (reload)`) all reflect the recoverable work, but nothing flushes automatically. This is by design - no immediate flush at boot (print-mode sessions may die under it; a boot-time compact was not requested by the user).
|
||
|
||
---
|
||
|
||
## Occurrence Identity
|
||
|
||
Every lookup keyed on a tool call - the indexer's record map, dedup aliases, `isSummarized`, batch capture - needs a string that uniquely names one *occurrence* of a tool call for the lifetime of a session.
|
||
|
||
### Why the provider `toolCallId` cannot be that string
|
||
|
||
A provider's `toolCallId` (e.g. `bash_23`) is unique only within the single response that produced it. Some providers restart a `${tool}_${n}` counter, so the same bare id recurs across turns and across a session, denoting a different tool call each time. The indexer cannot treat it as session-durable identity by itself - see [Identification model](#identification-model-positional-ranges-not-ids) for how chain compression sidesteps the same problem by keying on position instead of id.
|
||
|
||
### Why `ToolCallRecord.timestamp` cannot serve as the discriminant either
|
||
|
||
`ToolCallRecord.timestamp` is the *batch's* timestamp, computed once per captured batch (`src/batch-capture.ts`) and shared by every tool call inside that batch - it cannot discriminate between two calls captured together, let alone across a reused id. It is also not computed the way this discriminant would need: the live `turn_end` path stamps it unconditionally with `Date.now()` (`index.ts`'s `turn_end` handler), not from the assistant message's own timestamp. This is exactly why a separate per-tool-call field was needed - `resultTimestamp`, sourced from the `ToolResultMessage`'s own `timestamp` instead of the batch. On reconstruction, `reconstructFromSession` replays whatever value was persisted on the record verbatim; it does not recompute it.
|
||
|
||
### The key: `id@resultTimestamp`
|
||
|
||
The one value read identically at capture time and at every later render or rescan is the `ToolResultMessage`'s own `timestamp` field, stamped once by the message itself. `src/occurrence-key.ts` combines it with the bare id:
|
||
|
||
```
|
||
occKey(toolCallId, resultTimestamp) = `${toolCallId}@${resultTimestamp}`
|
||
```
|
||
|
||
A record with no `resultTimestamp` is keyed by its bare id - the shape for any record that predates this field, not a separate code path.
|
||
|
||
**Where the key lives:** `ToolCallRecord.resultTimestamp` and `SummaryToolCallRef.resultTimestamp` (both optional), the dedup-alias pair `newResultTimestamp` / `originalResultTimestamp`, and `ChainCompressionEntry.droppedOccurrenceKeys`. Bare-id and occurrence keys coexist in the same maps: `getRecord` / `isSummarized` compare keys as opaque strings, with no casting between the two shapes.
|
||
|
||
**`isSummarized` is strict.** It does an exact lookup on the occurrence key (or dedup-alias map) and never falls back to the bare id on a miss. The sole sanctioned bare-id path is `hasLegacyBareRecord(toolCallId)`, true only when the bare id is indexed **and** has no occurrence-keyed siblings (`bareIdToKeys`). A bare id with mixed legacy and occurrence-keyed entries fails closed on both checks - a permissive fallback would stub or recover the wrong occurrence whenever an id was reused.
|
||
|
||
**Spill sidecars** are named from the occurrence key, so a reused id spills to distinct files per occurrence. Recovery reads the `spillPath` persisted on the record rather than re-deriving it from the id, so bare-id-named sidecars still resolve.
|
||
|
||
**Render-time (`src/pruner.ts` stub-replace) gates the bare-id fallback on `hasLegacyBareRecord`, not on "message has no timestamp".** The first lookup key is `occKey(msg.toolCallId, msg.timestamp)` when the message carries a `timestamp` (a live `ToolResultMessage` almost always does) or the bare id otherwise. If that first key misses `isSummarized`, the bare id is tried as a second rung - but **only** when `indexer.hasLegacyBareRecord(msg.toolCallId)` is true (bare id indexed, no occurrence-keyed siblings), regardless of whether the message itself carried a timestamp. Gating that second rung on "the message has no timestamp" instead would mean a pure-legacy session (every record captured before `resultTimestamp` existed, so every occurrence-key lookup misses) never stubs at all, since a timestamped message would never even attempt the bare-id rung. A mixed bare+occurrence id still fails closed on both lookups.
|
||
|
||
### No migration, and the limitation that leaves
|
||
|
||
A record persisted before `resultTimestamp` existed is bare-keyed and stays that way - `reconstructFromSession` does not scan the branch to retroactively assign it one. No migration runs on load; `hasLegacyBareRecord` is the only accommodation, and it is a derivation (single-key check on `bareIdToKeys`), not a rewrite.
|
||
|
||
This leaves one accepted, documented exposure: within a session that spans the upgrade, a live tool result whose provider id collides with a pre-upgrade summarized record can be stub-replaced with that record's stale content - the render-time bare-id fallback in `src/pruner.ts` cannot tell the two apart once both denote the same bare `toolCallId` with no occurrence-keyed siblings. New records (both sides occurrence-keyed) are unaffected; the exposure is confined to the pre-upgrade half of a spanning session and disappears entirely once a session contains no legacy records.
|
||
|
||
### Multi-occurrence recovery via `context_tree_query`
|
||
|
||
Short refs (`tN`) always resolve 1:1 - one ref, one record - and are the primary, recommended recovery path. A raw provider `toolCallId` passed to `context_tree_query` is looked up against every record that shares it via `getRecordsForId` (`src/indexer.ts`): if the id was reused, the tool returns **every** matching occurrence instead of silently picking one, one block each, chronologically, each labelled `id@timestamp`. This includes occurrences that were content-deduplicated to an earlier record: `getRecordsForId` also scans the dedup-alias map for aliases sharing the queried bare id and returns each aliased occurrence too, labelled with **its own** occurrence timestamp rather than the original's, so a reader can tell the two apart. A record with no `resultTimestamp` caught up in such a set is labelled with the bare id plus an explicit `Occurrence: legacy (no resultTimestamp)` line.
|
||
|
||
---
|
||
|
||
## Error Purge
|
||
|
||
Failed tool calls often embed large argument bodies in the assistant message — a `write` call with a 30 KB file body, an `edit` call with a multiline diff. The error result is small (e.g. `"Error: file not found"`), but the original `arguments` stay in the assistant turn indefinitely.
|
||
|
||
Error purge replaces those arg bodies with compact stubs after the error has cooled down:
|
||
|
||
```
|
||
{ "_purged": "<purged-errored-args size=\"N\"/>" }
|
||
```
|
||
|
||
**What triggers a purge:**
|
||
|
||
- The matching `ToolResultMessage` has `isError: true`.
|
||
- The error occurred at least `purgeErrors.cooldownTurns` assistant turns ago (default 2). The cooldown gives the model 1–2 turns to retry before context is mutated.
|
||
- The JSON-stringified argument body is at least `purgeErrors.minArgChars` characters long (default 500). Small args are not worth the substitution.
|
||
|
||
**What error purge does NOT touch:**
|
||
|
||
- The `ToolResultMessage` content — the error message stays visible so the model can see what went wrong.
|
||
- Non-errored `toolCall` argument bodies.
|
||
- Argument bodies below `minArgChars`.
|
||
- Anything when `purgeErrors.enabled` is `false`.
|
||
|
||
**Transform position:** Error purge runs in Phase 2, after stub-replace and before chain range prune.
|
||
|
||
```
|
||
[stub-replace] → [error-purge] → [chain-range-prune] → [orphan-sweep]
|
||
```
|
||
|
||
**Config keys:**
|
||
|
||
| Key | Default | Description |
|
||
|---|---|---|
|
||
| `purgeErrors.enabled` | `true` | Master toggle |
|
||
| `purgeErrors.cooldownTurns` | `2` | Turns to wait after error before purging |
|
||
| `purgeErrors.minArgChars` | `500` | Minimum argument body size to purge |
|
||
|
||
---
|
||
|
||
## Orphan Sweep
|
||
|
||
`pruneMessages` ends with an unconditional structural pass, `sweepOrphanToolResults` (`src/orphan-sweep.ts`): a `toolResult` message whose id is not open is removed, where "open" means: opened by the most recent `assistant` message and not interrupted by any barrier since. It runs after every other phase, over whatever the previous three produced, as a final post-condition rather than a targeted fix for one code path.
|
||
|
||
**Why it exists.** pi-ai's auto-repair (`insertSyntheticToolResults`) only fills in a missing `toolResult` for an orphaned `toolCall` - it has no equivalent repair for the opposite shape, an orphaned `toolResult` with no matching `toolCall`. Providers reject that shape outright (Anthropic: `unexpected tool_use_id found in tool_result blocks`; Kimi K3: a tool message needs a resolvable preceding tool_call). An orphaned `toolResult` can appear from any combination of id reuse, an unresolved chain range, or a bug in an upstream phase; the sweep is the last line of defense regardless of cause, and it is intentionally not keyed to any provider's specific error string - the fix is structural ("does this id have an open call"), not reactive to how one provider happens to phrase rejection.
|
||
|
||
**Per-turn, not cumulative, open-call tracking.** The sweep does one forward pass over the message array. Each `assistant` message **replaces** the current "open" id set with its own `toolCall` ids; each `toolResult` message either consumes (removes) a matching id from that set or, if its id is not open, is swept. Any message that is neither `assistant` nor `toolResult` is a **barrier** that clears the open set. This tracks the post-`convertToLlm` flush points of pi-ai's `insertSyntheticToolResults` - `convertToLlm` maps `custom`, `branchSummary`, `compactionSummary`, and `bashExecution` to `role: "user"`, and pi-ai flushes synthetic tool results at both `assistant` and `user` boundaries - **plus** a deliberate conservative over-sweep: `bashExecution` with `excludeFromContext: true` and unknown roles are dropped by `convertToLlm` and strictly need not be barriers, but a role allowlist would have to track an open interface for zero observed benefit. Not an exact mirror, and not an allowlist. Trade-off: a foreign message spliced mid-cycle now costs one tool output - the interleaved real result is swept and pi-ai injects its repairable `{isError: true, "No result provided"}` synthetic - converting an unrepairable provider shape (duplicate `tool_use_id`, permanent Anthropic 400) into a visible-but-recoverable tool failure. See `doc/specs/2026-08-18-gh-11-orphan-sweep-barrier.md`. A cumulative seen-set across the whole array would be wrong here: it would let an id used validly by an early turn license a *later* genuine orphan under the same reused id - exactly the collision scenario this exists to catch. Per-turn tracking means only the most recent assistant turn, uninterrupted by a barrier, can vouch for a `toolResult`'s id.
|
||
|
||
**Provider-agnostic and reference-preserving.** The sweep has no knowledge of which provider is in play; it operates purely on message shape. When nothing is swept it returns the **same array reference** it was given, so a clean render is a true no-op and the no-op / prompt-cache-prefix invariant of `doc/specs/2026-08-04-pruner-noop-serialization.md` holds - the sweep never forces a cache bust on a session where nothing is actually orphaned.
|
||
|
||
Every swept id is reported once (hashed batch, not per-id) as an `orphan-sweep` diagnostic.
|
||
|
||
---
|
||
|
||
## Diagnostics
|
||
|
||
Prune-time degradations - an unresolvable chain range, a detection/render id mismatch, a swept orphan, a chain with nothing left to backfill - are recorded on an out-of-band channel instead of surfacing in the model's context. `DiagnosticSink` (`src/diagnostics.ts`) writes a `context-prune-diagnostic` session entry (`{ kind, detail }`) for each of four kinds:
|
||
|
||
| Kind | Emitted from | Meaning |
|
||
|---|---|---|
|
||
| `unresolved-range` | `applyChainCompressions` | A persisted chain entry's boundaries didn't resolve to a unique range (or the range was rejected as nested/duplicate) - the entry compressed nothing |
|
||
| `range-id-mismatch` | `applyChainCompressions` | The ids actually inside a resolved range don't match the entry's recorded `droppedToolCallIds` - informational only, the range still wins |
|
||
| `orphan-sweep` | `pruneMessages` (Phase 4) | One or more `toolResult` messages were removed for having no open matching `toolCall` |
|
||
| `backfill-empty` | `chain-compressor.compressEligible` | A zero-coverage chain's deterministic-backfill span yielded zero extractable records AND zero members already in the index AND the chain is not fully-protected - a genuine span mismatch, not a partial-failure retry state. Fully-protected zero-coverage chains skip silently instead (see [Deterministic fallback § Fully-protected exception](#deterministic-fallback-uncovered-chains)). Deduped per chain start timestamp (`chain.startUserTimestamp`). Widget letter `b` |
|
||
|
||
**Never in LLM context.** These are session entries only - zero tokens added, zero cache-prefix change, never read back into the message array the model sees.
|
||
|
||
**Deduped per `(kind, dedupKey)`**, so a permanently degraded condition (e.g. the same chain entry failing to resolve on every render) writes one entry, not one per render. The dedup key is entry-specific: a chain `blockId` for `range-id-mismatch` and for a genuinely unresolvable `unresolved-range`; `overlap:<blockId>` for the same `unresolved-range` kind when the entry was instead skipped as nested inside or duplicating another range's start (a distinct dedup-key prefix keeps the two cases greppable under one kind); and a short hash of the sorted swept-id list for `orphan-sweep` (hashing rather than joining the raw list avoids `DiagnosticSink`'s dedup set retaining an ever-longer key per additional orphan over a session's lifetime). A write is only marked seen after the `appendEntry` call succeeds, so a failed write is retried on the next render instead of being silently dropped.
|
||
|
||
**Reset on `session_start` and `session_tree`**, matching every other in-memory, non-persisted piece of prune state.
|
||
|
||
**Surfaced on the status line.** The footer status widget (`setPruneStatusWidget`, gated by `showPruneStatusLine`) appends a self-hiding ` · diag u<N>/m<N>/o<N>/b<N>` segment (u = `unresolved-range`, m = `range-id-mismatch`, o = `orphan-sweep`, b = `backfill-empty`) built from the sink's live counters via `pruneStatusText`. Each letter is omitted when its counter is zero, and the whole segment is absent when all four are zero. `/pruner status` (the slash command) prints a separate settings/stats block and does not include this segment.
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
| Concern | How Pruning Addresses It |
|
||
|---|---|
|
||
| **Context grows without bound** | Replaces raw tool outputs (~thousands of tokens) with compact summaries (~hundreds) |
|
||
| **Signal lost in noise** | Summaries surface the key decisions and facts; raw data is demoted to on-demand query |
|
||
| **Cache performance** | Batch-then-prune (`agent-message`) minimizes cache invalidation; stub-replace keeps the pruned tail deterministic so it stays cacheable |
|
||
| **Data availability** | `context_tree_query` recovers full original outputs at any time, including aliased duplicates |
|
||
| **Wasted LLM calls** | Pre-flush content-hash dedup catches re-reads at zero cost; `minBatchChars` short-circuits batches too small to be worth summarizing |
|
||
| **Plan / state churn** | `protectedTools` keeps allowlisted tool outputs verbatim across turns |
|
||
| **Empirical benefit** | SUPO, ReSum, and ACON all show summarization improves or preserves task success while reducing context length 26–54% |
|
||
|
||
### Choosing a mode
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ MODE DECISION TREE │
|
||
├─────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ "I want maximum control" │
|
||
│ └──► on-demand + /pruner now │
|
||
│ │
|
||
│ "I want the best balance of automation, savings, and cache hits" │
|
||
│ └──► agent-message ◄── DEFAULT │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Recommended reading
|
||
|
||
- Anthropic prompt caching docs: <https://docs.claude.com/en/docs/build-with-claude/prompt-caching>
|
||
- OpenAI prompt caching docs: <https://platform.openai.com/docs/guides/prompt-caching>
|
||
- SUPO: <https://arxiv.org/abs/2510.06727>
|
||
- ReSum: <https://arxiv.org/abs/2509.13313>
|
||
- ACON: <https://arxiv.org/abs/2510.00615>
|