my-pi/pi-permission-system/docs/plans/archive/0072-shell-quote-tokenizer.md

issue, issue_title

issue	issue_title
72	Replace regex-based bash tokenizer with shell-quote or tree-sitter-bash

Replace regex-based bash tokenizer with shell-quote

Problem Statement

extractExternalPathsFromBashCommand in src/external-directory.ts uses a hand-rolled regex tokenizer (stripQuotedStrings + split(/[|;&><\s]+/)) that produces false positives on edge cases. Issue #68 fixed bare-slash tokens, but the underlying tokenizer remains fragile:

1. stripQuotedStrings breaks on escaped quotes (\"), leaking post-break content into the token stream.
2. Shell comments (# ...) are not stripped — path-like tokens after # are scanned.
3. Heredoc content is tokenized as if it were command arguments.
4. Operators are treated as simple delimiters, losing structural information.

These are not hypothetical — the escaped-quote bug triggered a false-positive external-directory prompt during #68 dog-fooding.

Goals

• Replace stripQuotedStrings and the split(/[|;&><\s]+/) tokenizer with shell-quote's parse().
• Eliminate false positives from escaped quotes, shell comments, and operator conflation.
• Keep classifyTokenAsPathCandidate as a separate concern operating on properly tokenized string arguments.
• Add shell-quote and @types/shell-quote as runtime and dev dependencies respectively.
• Add regression tests for the edge cases that the regex tokenizer gets wrong.

Non-Goals

• Adopting web-tree-sitter + tree-sitter-bash — deferred to a follow-up issue. shell-quote is sufficient for path extraction and avoids the 1.5MB WASM overhead.
• Handling heredocs — shell-quote flattens heredoc content into tokens, which is the same behavior as the current tokenizer. This is a known limitation shared by both approaches.
• Changing classifyTokenAsPathCandidate logic — the classification heuristics are orthogonal to tokenization.
• Changing any permission surface, config format, or merge precedence.

Background

• Permission surface: external_directory (bash variant).
• Module: src/external-directory.ts — extractExternalPathsFromBashCommand is the entry point; stripQuotedStrings and classifyTokenAsPathCandidate are internal helpers.
• Tests: tests/bash-external-directory.test.ts (388 lines) covers extraction, formatting, and edge cases.
• Prerequisite: #68 (bare-slash fix) — already shipped in v4.0.1.

shell-quote API

shell-quote exports parse(cmd) returning ParseEntry[] where:

type ParseEntry =
  | string                          // plain argument (quotes resolved)
  | { op: string }                  // shell operator (|, &&, ;, etc.)
  | { op: "glob"; pattern: string } // glob pattern
  | { comment: string };            // shell comment

String entries have quotes already resolved — parse('git commit -m "fix /etc/hosts"') returns ["git", "commit", "-m", "fix /etc/hosts"]. Operator and comment entries are objects, trivially filtered out.

Design Overview

Tokenization change

Replace:

const unquoted = stripQuotedStrings(command);
const tokens = unquoted.split(/[|;&><\s]+/).filter(Boolean);

With:

import { parse } from "shell-quote";

const entries = parse(command);
const tokens = entries.filter((e): e is string => typeof e === "string");

This single change fixes escaped quotes, comments, and operator handling in one shot. classifyTokenAsPathCandidate continues to receive plain strings and is unchanged.

Dead code removal

stripQuotedStrings becomes dead code and is removed. The bare-slash guard in classifyTokenAsPathCandidate (/^\/+$/.test(token)) is kept — shell-quote can still produce bare-slash strings (e.g., parse("echo /") → ["echo", "/"]), so the guard remains a valid defense-in-depth layer.

Dependency addition

• shell-quote as a runtime dependency (dependencies in package.json).
• @types/shell-quote as a dev dependency (devDependencies).

This is the first runtime dependency for this package. shell-quote is 23KB, zero transitive dependencies, MIT license, 47M+ weekly downloads — low risk.

Module-Level Changes

File	Change
package.json	Add shell-quote to dependencies, @types/shell-quote to devDependencies.
src/external-directory.ts	Import parse from shell-quote. Replace stripQuotedStrings + split() in extractExternalPathsFromBashCommand with parse() + type filter. Remove stripQuotedStrings function.
tests/bash-external-directory.test.ts	Add tests for: escaped quotes in double-quoted strings, shell comments containing paths, operators as typed tokens (not leaking into path stream). Verify existing tests still pass.

TDD Order

1. test: add failing tests for regex tokenizer edge cases (#72) Add a new describe("shell-quote tokenizer edge cases") block in tests/bash-external-directory.test.ts with:

   • Escaped double quote: git commit -m "fix \"the /etc/hosts\" issue" → no external path (path is inside quotes).
   • Shell comment: echo hello # read /etc/shadow → no external path (path is in comment).
   • Comment alongside real path: cat /etc/hosts # /etc/shadow → only /etc/hosts.
   • Operator tokens don't leak: cat /etc/hosts | grep foo → only /etc/hosts, not | or grep.
   • Semicolons: echo ok; cat /etc/hosts → /etc/hosts extracted correctly. These tests will fail against the current regex tokenizer (red). Commit: test: add failing cases for regex tokenizer edge cases (#72)

2. feat: replace regex tokenizer with shell-quote (#72)

   • Add shell-quote and @types/shell-quote dependencies.
   • In extractExternalPathsFromBashCommand, replace stripQuotedStrings + split() with parse() + string filter.
   • Remove the stripQuotedStrings function.
   • All new tests pass (green). Run full suite to confirm no regressions. Commit: feat: replace regex tokenizer with shell-quote (#72)

3. test: verify defense-in-depth for bare-slash tokens (#72) Add or confirm a test that parse("echo /") still produces / as a token and classifyTokenAsPathCandidate still rejects it. This validates the bare-slash guard remains necessary even with shell-quote. Commit: test: confirm bare-slash guard with shell-quote tokenizer (#72)

4. docs: update plan retro and close issue (#72) Optional retro in docs/retro/0072-shell-quote-tokenizer.md if anything surprising surfaces. Commit: docs: retro for shell-quote tokenizer migration (#72)

Risks and Mitigations

Risk	Mitigation
Could this silently weaken a permission?	No — shell-quote produces fewer tokens than the regex tokenizer (comments and operators are filtered out as non-strings). Fewer tokens means fewer path candidates, which means fewer prompts, never fewer blocks. A path that was correctly detected before will still be a string token from shell-quote.
shell-quote misparses a command and drops a real path argument?	shell-quote has 47M weekly downloads and handles standard POSIX quoting. Edge cases (heredocs, complex expansions) are no worse than the regex tokenizer. The bare-slash guard and classifyTokenAsPathCandidate provide additional filtering layers.
First runtime dependency — supply chain risk?	shell-quote is zero-dependency, MIT, maintained by the shell-quote org. The @types/shell-quote package is DefinitelyTyped-sourced. Both are widely audited. Pin versions via lockfile.
shell-quote handles $VAR expansion by default?	parse(cmd) with no env argument replaces $VAR with empty string. This is acceptable — we don't want environment variables expanded for path extraction. If a command uses $HOME/foo, the path candidate will be /foo (or empty), not ~/foo. This is the same behavior as the regex tokenizer, which has no variable awareness.

Open Questions

• Follow-up: tree-sitter-bash for full AST parsing. Addressed by #74 — shell-quote has been replaced with web-tree-sitter + tree-sitter-bash, eliminating heredoc false positives and providing full AST-based path extraction.
• $VAR expansion: tree-sitter parses $HOME/foo as an expansion + word concatenation. classifyTokenAsPathCandidate does not expand variables, so $HOME/foo is not detected as an external path. This is the same pre-existing limitation as with shell-quote. Deferred — not a regression from current behavior.