my-pi/pi-hermes-memory/docs/0.4/PLAN.md

v0.4 Plan: SQLite FTS5 Session Search + Hybrid Memory

Problem

The current memory architecture has two scaling bottlenecks:

1. Memory capacity: MEMORY.md is capped at 2,200 chars. Power users accumulate knowledge faster than consolidation can manage. Important facts get pruned.
2. No session history search: Past conversations are stored as JSONL files in ~/.pi/agent/sessions/<project>/, but there's no way to search them. When the agent needs context from a previous session, it's gone forever.

Solution: Hybrid Memory Architecture

Core memory (always injected, unchanged)

• MEMORY.md — 5,000 chars (up from 2,200)
• USER.md — 5,000 chars (up from 1,375)
• Still injected into every session via <memory-context> tags
• Still human-readable, still editable

Extended memory (SQLite, searchable on demand)

• ~/.pi/agent/memory/sessions.db
• memories table — unlimited entries, searchable via FTS5
• Agent uses memory_search tool to query when it needs context
• Not automatically injected — agent must explicitly search

Session history (SQLite, searchable on demand)

• Same sessions.db file
• sessions + messages tables — all past conversations indexed
• session_fts FTS5 index — full-text search across all sessions
• Agent uses session_search tool to find relevant past context

Architecture

Session starts
    ↓
┌─────────────────────────────────────────────────┐
│ System Prompt (always injected)                 │
│ ┌─────────────────────────────────────────────┐ │
│ │ <memory-context>                            │ │
│ │ MEMORY (your personal notes) [5,000 chars]  │ │
│ │ ═══ END MEMORY ═══                         │ │
│ │ </memory-context>                           │ │
│ │ <memory-context>                            │ │
│ │ USER PROFILE [5,000 chars]                  │ │
│ │ ═══ END MEMORY ═══                         │ │
│ │ </memory-context>                           │ │
│ │ <memory-context>                            │ │
│ │ PROJECT MEMORY [5,000 chars]                │ │
│ │ ═══ END MEMORY ═══                         │ │
│ │ </memory-context>                           │ │
│ └─────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────┘

Agent has access to tools:
    memory_search("prisma migration")
        → Searches memories table (global + project)
        → Returns top-10 relevant entries

    session_search("how we fixed the test hang")
        → Searches session history via FTS5
        → Returns relevant conversation snippets

Data Model

-- Session metadata
CREATE TABLE sessions (
  id TEXT PRIMARY KEY,           -- UUID from JSONL
  project TEXT NOT NULL,         -- decoded cwd path
  started_at TEXT NOT NULL,      -- ISO timestamp
  ended_at TEXT,                 -- ISO timestamp (null if still running)
  message_count INTEGER DEFAULT 0
);

-- All messages from all sessions
CREATE TABLE messages (
  id TEXT PRIMARY KEY,           -- message ID from JSONL
  session_id TEXT NOT NULL REFERENCES sessions(id),
  role TEXT NOT NULL,            -- 'user', 'assistant', 'system'
  content TEXT NOT NULL,         -- extracted text content
  timestamp TEXT NOT NULL,       -- ISO timestamp
  tool_calls TEXT                -- JSON array of tool call names (for assistant messages)
);

-- FTS5 index for full-text search across messages
CREATE VIRTUAL TABLE message_fts USING fts5(
  content,
  content='messages',
  content_rowid='rowid'
);

-- Extended memory entries (beyond MEMORY.md limit)
CREATE TABLE memories (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  project TEXT,                  -- NULL for global, project name for project-specific
  target TEXT NOT NULL,          -- 'memory' or 'user'
  content TEXT NOT NULL,
  created DATE NOT NULL,
  last_referenced DATE NOT NULL
);

-- FTS5 index for memory search
CREATE VIRTUAL TABLE memory_fts USING fts5(
  content,
  content='memories',
  content_rowid='id'
);

Key Design Decisions

1. Session indexing is lazy (on session end)

• Don't parse JSONL files on every startup
• Index a session when session_shutdown fires
• Bulk import existing sessions via /memory-index-sessions command

2. FTS5 for both memories and sessions

• Keyword search is sufficient for v0.4
• Embeddings deferred to v0.5+ if search quality isn't enough
• better-sqlite3 includes FTS5 by default

3. Single DB file

• ~/.pi/agent/memory/sessions.db stores everything
• Memories + sessions + FTS indices in one file
• Simple backup (copy one file), simple cleanup

4. Agent-driven search

• memory_search and session_search are LLM tools
• Agent decides when to search (not automatic)
• Avoids injecting irrelevant context into every session

5. Char limit increase to 5,000

• MEMORY.md: 2,200 → 5,000 chars
• USER.md: 1,375 → 5,000 chars
• Project MEMORY.md: 2,200 → 5,000 chars
• More room for core memories before consolidation kicks in

Dependencies

Package	Purpose	Size
better-sqlite3	SQLite with FTS5	~1MB native addon

Risks

Risk	Mitigation
better-sqlite3 is a native C++ addon	Standard for dev tools; CI has build tools
FTS5 search quality	Start with keyword search, add embeddings later if needed
Session JSONL format changes	Parse defensively, skip unknown message types
Large session history (1000+ sessions)	FTS5 handles this well; add pagination to results
DB corruption	Atomic writes, WAL mode, backup before migrations

Success Criteria

1. session_search("prisma migration") returns relevant conversation snippets from past sessions
2. memory_search("auth setup") returns relevant entries from extended memory store
3. MEMORY.md limit raised to 5,000 chars without breaking existing functionality
4. Existing session files indexed without data loss
5. All tests pass, zero regressions