my-pi/pi-permission-system/docs/plans/archive/0091-sed-regex-false-positive.md

issue, issue_title

issue	issue_title
91	Bash external-directory guard false-positive on sed regex containing absolute-path-like patterns

Fix sed/awk/grep false-positive in bash external-directory guard

Problem Statement

The bash external-directory guard extracts path candidates from ALL command arguments indiscriminately, then attempts to classify each token as path-or-not using character-level heuristics (classifyTokenAsPathCandidate). This is fundamentally flawed for commands like sed, awk, and grep, whose arguments mix regex patterns with file paths. A sed address pattern like /source: "tool",/{/origin:/!s/...} starts with / and passes the heuristic classifier, triggering a false-positive prompt for an "external directory" that is actually a regex pattern.

The root cause is not a missing heuristic — it is that the classifier is command-blind. The tree-sitter AST already gives us the command name, but we discard that context and classify each token independently. No amount of character heuristics can reliably distinguish /pattern/d (sed command) from /pattern/d (real path) without knowing the command.

How OpenCode handles this

OpenCode takes a strict allowlist approach: only extract path arguments from a known set of file-manipulating commands (cat, cp, mv, rm, mkdir, chmod, chown). All other commands — including sed, grep, awk — get zero external-directory scanning. This eliminates false positives entirely, but creates a false-negative gap: sed 's/foo/bar/' /etc/hosts does not trigger an external-directory prompt even though /etc/hosts is a real file argument.

Unified approach

Use command context from the tree-sitter AST to make better classification decisions rather than binary include/exclude. For known pattern-first commands (sed, awk, grep), identify and skip the script/pattern argument while still extracting the file arguments. For unknown commands, fall back to the current heuristic classification. Redirect targets remain universally extracted — they are syntactically unambiguous paths.

Goals

• Extract the command name from each tree-sitter command node and use it to guide argument classification.
• For known pattern-first commands (sed, awk, grep and variants), skip the inline script/pattern argument positionally and extract only the file arguments.
• Preserve path detection for file arguments in pattern-first commands (fewer false negatives than OpenCode's allowlist approach).
• Preserve the current heuristic fallback for commands not in the pattern-first set.
• Continue extracting redirect targets (> /path, < /path) for all commands.
• No config, schema, or policy changes.

Non-Goals

• Full POSIX/GNU option parser for every command.
• Combined flag handling (-ni, -ie) — deferred; regular flags that are not recognized are skipped without consuming the next argument.
• Long option handling (--regexp=PATTERN, --file=FILE) — deferred.
• PowerShell support — out of scope; this extension targets bash commands only.
• Changing any permission surface, default policy state, or merge precedence.
• Renaming the /permission-system slash command.

Background

Permission surface

external_directory (bash variant) — the special permission gate for bash commands referencing paths outside the working directory.

Existing modules

File	Role
src/external-directory.ts	extractExternalPathsFromBashCommand (entry point), collectPathCandidateTokens (AST walker), classifyTokenAsPathCandidate (heuristic classifier), tree-sitter parser init
src/handlers/tool-call.ts	Calls extractExternalPathsFromBashCommand for bash tool invocations
tests/bash-external-directory.test.ts	Test suite for extraction, classification, and formatting

Current flow

extractExternalPathsFromBashCommand(command, cwd)
  → tree-sitter parse → AST
  → collectPathCandidateTokens(root) → flat string[]
      walks ALL command nodes, extracts ALL arguments (command-blind)
      walks file_redirect nodes, extracts redirect targets
  → for each token: classifyTokenAsPathCandidate(token)
      heuristic reject: flags, env assignments, URLs, @scope, bare-slash, regex metachar
      heuristic accept: starts with /, starts with ~/, contains ..
  → resolve + isPathOutsideWorkingDirectory

False-positive reproducer from the issue

sed -i '' '/source: "tool",/{/origin:/!s/source: "tool",/source: "tool",\n      origin: "builtin",/;}' tests/tool-input-preview.test.ts

tree-sitter parses the single-quoted sed script as a raw_string argument. After quote stripping, the token is /source: "tool",/{/origin:/!s/...}. It starts with /, does not match REGEX_METACHAR_PATTERN, and is classified as a path candidate. The user sees a prompt about "external directory /source: "tool",/{...}".

Design Overview

Command name extraction

When visiting a command node in collectPathCandidateTokens, extract the command name from the command_name child node. Normalize it with basename to handle full-path invocations (/usr/bin/sed → sed). Look it up in a PATTERN_FIRST_COMMANDS map.

Pattern-first command config

interface PatternCommandConfig {
  /** Flags that consume the next argument as a non-path value (pattern, separator, etc.) */
  readonly argConsumingFlags: ReadonlySet<string>;
  /** Flags that consume the next argument as a file path */
  readonly fileConsumingFlags: ReadonlySet<string>;
  /**
   * Number of leading positional arguments that are patterns/scripts, not paths.
   * Default: 1 (covers sed, awk, grep, rg).
   * sd uses 2 (FIND and REPLACE_WITH are both non-path positionals).
   */
  readonly patternPositionals?: number;
}

const PATTERN_FIRST_COMMANDS: ReadonlyMap<string, PatternCommandConfig> = new Map([
  ["sed",   { argConsumingFlags: new Set(["-e", "-i"]),
              fileConsumingFlags: new Set(["-f"]) }],
  ["awk",   { argConsumingFlags: new Set(["-e", "-F", "-v"]),
              fileConsumingFlags: new Set(["-f"]) }],
  ["gawk",  { argConsumingFlags: new Set(["-e", "-F", "-v"]),
              fileConsumingFlags: new Set(["-f"]) }],
  ["nawk",  { argConsumingFlags: new Set(["-e", "-F", "-v"]),
              fileConsumingFlags: new Set(["-f"]) }],
  ["grep",  { argConsumingFlags: new Set(["-e", "-A", "-B", "-C", "-m"]),
              fileConsumingFlags: new Set(["-f"]) }],
  ["egrep", { argConsumingFlags: new Set(["-e", "-A", "-B", "-C", "-m"]),
              fileConsumingFlags: new Set(["-f"]) }],
  ["fgrep", { argConsumingFlags: new Set(["-e", "-A", "-B", "-C", "-m"]),
              fileConsumingFlags: new Set(["-f"]) }],
  ["rg",    { argConsumingFlags: new Set(["-e", "-A", "-B", "-C", "-m",
                                           "-g", "-t", "-T", "-j", "-M",
                                           "-r", "-E"]),
              fileConsumingFlags: new Set(["-f"]) }],
  ["sd",    { argConsumingFlags: new Set(["-n", "-f"]),
              fileConsumingFlags: new Set([]),
              patternPositionals: 2 }],
]);

Position-based argument skipping

For a command in PATTERN_FIRST_COMMANDS, process arguments with state tracking:

let hasExplicitScript = false   // set true when -e or -f is encountered
let positionalsSeen = 0
const patternPositionals = config.patternPositionals ?? 1
let nextArgAction: "skip" | "extract" | null = null

for each child node after command_name:
  if nextArgAction is "skip":
    nextArgAction = null; continue        // consumed by previous flag
  if nextArgAction is "extract":
    collect token; nextArgAction = null; continue  // file consumed by -f

  if child is a flag (word starting with "-", length > 1):
    if flag == "--":
      mark all remaining as positional; continue
    if flag in argConsumingFlags:
      nextArgAction = "skip"
      if flag == "-e" or flag == "-f":
        hasExplicitScript = true          // no inline script expected
    elif flag in fileConsumingFlags:
      nextArgAction = "extract"
      hasExplicitScript = true
    continue                              // regular flag, skip

  // positional argument
  if !hasExplicitScript && positionalsSeen < patternPositionals:
    positionalsSeen++
    continue                              // skip: this is an inline pattern/script

  collect token                           // file argument → path candidate

For commands NOT in PATTERN_FIRST_COMMANDS, the existing generic logic applies unchanged (extract all non-command-name arguments).

Redirect targets

Unchanged — file_redirect handling remains command-blind because redirect destinations are always filesystem paths regardless of the command.

classifyTokenAsPathCandidate

Unchanged — it remains as defense-in-depth for tokens that pass through the position-based filter. The existing heuristics (URL rejection, regex metachar rejection, bare-slash rejection, etc.) still apply to all collected tokens.

Worked examples

Issue reproducer

sed -i '' '/source: "tool",/{/origin:/!s/source: "tool",/source: "tool",\n      origin: "builtin",/;}' tests/tool-input-preview.test.ts

1. Command name: sed → in PATTERN_FIRST_COMMANDS.
2. -i → argConsumingFlag → nextArgAction = "skip".
3. '' → consumed by -i → skipped.
4. '/source: ...' → first positional, !hasExplicitScript && positionalsSeen < 1 → skipped as inline script.
5. tests/tool-input-preview.test.ts → second positional → collected as path candidate → relative path, within CWD → not flagged.
6. Result: no false-positive prompt. ✓

sed with external file argument

sed 's/foo/bar/g' /etc/hosts

1. Command name: sed.
2. 's/foo/bar/g' → first positional → skipped as inline script.
3. /etc/hosts → second positional → collected → classified as path → flagged as external.
4. Result: external-directory prompt for /etc/hosts. ✓

sed with -e flag

sed -e 's/foo/bar/' /etc/hosts

1. -e → argConsumingFlag, hasExplicitScript = true, nextArgAction = "skip".
2. 's/foo/bar/' → consumed by -e → skipped.
3. /etc/hosts → positional, hasExplicitScript is true → collected → flagged.
4. Result: prompt for /etc/hosts. ✓

grep with pattern and external file

grep '/etc/' /var/log/syslog

1. Command name: grep.
2. '/etc/' → first positional → skipped as pattern.
3. /var/log/syslog → second positional → collected → flagged.
4. Result: prompt for /var/log/syslog. ✓

Unknown command (fallback)

some-tool /etc/hosts

1. Command name: some-tool → not in PATTERN_FIRST_COMMANDS.
2. Falls through to existing generic extraction logic.
3. /etc/hosts → collected → classified → flagged.
4. Result: prompt for /etc/hosts. ✓ (no regression)

rg with pattern and external path

rg '/usr/local' /etc/profile.d/

1. Command name: rg → in PATTERN_FIRST_COMMANDS.
2. '/usr/local' → first positional → skipped as pattern.
3. /etc/profile.d/ → second positional → collected → flagged.
4. Result: prompt for /etc/profile.d/. ✓

sd with two pattern positionals

sd '/usr/local/bin' '/opt/bin' /etc/profile

1. Command name: sd → in PATTERN_FIRST_COMMANDS, patternPositionals = 2.
2. '/usr/local/bin' → first positional (positionalsSeen < 2) → skipped.
3. '/opt/bin' → second positional (positionalsSeen < 2) → skipped.
4. /etc/profile → third positional (positionalsSeen == 2) → collected → flagged.
5. Result: prompt for /etc/profile. ✓

Redirect target on sed

sed 's/foo/bar/' input.txt > /tmp/output.txt

1. Script and input.txt handled by position logic.
2. > /tmp/output.txt → file_redirect node → /tmp/output.txt collected → flagged.
3. Result: prompt for /tmp/output.txt. ✓

Known limitation: sed -i without extension (GNU sed)

sed -i 's/foo/bar/' /etc/hosts

GNU sed treats -i as a flag with no argument; 's/foo/bar/' is the inline script, /etc/hosts is the input. Our logic treats -i as arg-consuming, so 's/foo/bar/' is consumed as the -i extension, and /etc/hosts becomes the first positional — which is skipped as the inline script. This is a false negative: we miss /etc/hosts.

Mitigation:

• This invocation pattern (in-place edit of an external file with no backup) is uncommon.
• The bash permission gate still applies — if bash is set to ask, the user gets prompted for the command itself.
• A follow-up can refine -i handling by inspecting whether the consumed argument looks like a sed script (contains / delimiters) vs an extension suffix.

Module-Level Changes

File	Change
src/external-directory.ts	Add PatternCommandConfig interface and PATTERN_FIRST_COMMANDS map. Add extractCommandName helper (extracts command name from command_name child, applies basename). Refactor the command branch in collectPathCandidateTokens to dispatch to collectPatternCommandTokens for pattern-first commands vs existing generic logic for others. Add collectPatternCommandTokens implementing position-based skipping. Import basename from node:path (already imported).
tests/bash-external-directory.test.ts	Add describe("command-aware extraction") block with sub-describes for sed, grep, awk, rg, sd, unknown commands, and edge cases. Add the issue reproducer as a named test. Update any existing tests whose behavior changes (if any — the position-based skipping should only REDUCE the set of extracted tokens for pattern commands, never increase it for non-pattern commands).

No changes to schema, config, docs/architecture, or other source modules.

TDD Order

Step 1

Surface: extractExternalPathsFromBashCommand — sed command-aware extraction. Coverage: the exact reproducer from the issue, plus simple sed 'script' /external/file and sed 'script' internal-file. Suggested commit: test: add failing tests for sed pattern false-positive (#91)

Step 2

Surface: extractExternalPathsFromBashCommand — sed flag handling. Coverage: sed -e 'script' /external/file, sed -n 'script' /external/file, sed -f /script/file input, sed -i '' 'script' /external/file. Suggested commit: test: add failing tests for sed flag-aware extraction (#91)

Step 3

Surface: extractExternalPathsFromBashCommand — grep, awk, rg, and sd. Coverage: grep 'pattern' /external/file, grep -e 'pattern' /external/file, awk '{print}' /external/file, awk -F: '{print $1}' /external/file, rg '/pattern' /external/dir, rg -e '/pattern' /external/dir, sd '/find' '/replace' /external/file (two pattern positionals then file). Suggested commit: test: add failing tests for grep/awk/rg/sd pattern-first extraction (#91)

Step 4

Surface: collectPathCandidateTokens — implement command-aware extraction. Coverage: all tests from steps 1–3 go green. Run full suite to verify no regressions. Changes: add PATTERN_FIRST_COMMANDS, PatternCommandConfig, extractCommandName, collectPatternCommandTokens. Refactor command branch in collectPathCandidateTokens to dispatch based on command name. Suggested commit: feat: command-aware path extraction for pattern-first commands (#91)

Step 5

Surface: extractExternalPathsFromBashCommand — edge cases and defense-in-depth. Coverage: full-path command invocation (/usr/bin/sed 'script' /ext/file), -- end-of-flags, unknown commands still use generic extraction, redirect targets still extracted for pattern-first commands, pipeline with sed piped to cat (sed 'script' file | cat /external/file). Suggested commit: test: cover command-aware edge cases and fallback (#91)

Step 6

Surface: extractExternalPathsFromBashCommand — known limitation documentation. Coverage: add test.todo or comment documenting the sed -i 'script' /external/file (GNU sed no-extension) false negative. Suggested commit: test: document sed -i no-extension known limitation (#91)

Risks and Mitigations

Risk	Mitigation
Could this silently weaken a permission?	Position-based skipping only skips the inline script/pattern argument for known pattern-first commands. File arguments are still extracted. For unknown commands, behavior is unchanged. Redirect targets are still universally extracted. The only tokens we stop detecting are pattern arguments — which were false positives, not real paths.
PATTERN_FIRST_COMMANDS set is incomplete	The set covers the most common pattern-first commands (sed, awk, grep, rg, sd and variants). Uncommon commands with similar argument structure can be added incrementally. Commands NOT in the set fall through to the existing generic extraction — no regression.
Flag config is incomplete (combined flags, long options)	Unrecognized flags are treated as regular flags (no arg consumed). This may cause misidentification of argument positions in rare cases, but classifyTokenAsPathCandidate provides defense-in-depth. Combined flag and long option support can be added incrementally.
sed -i without extension (GNU sed) causes false negative	Documented as a known limitation. The bash permission gate still applies. A follow-up can refine -i handling.
Command name extraction fails (variable expansion, alias, subshell)	If the command name cannot be extracted (e.g., $CMD /etc/hosts), fall back to generic extraction. No regression.
Refactoring collectPathCandidateTokens breaks existing tests	The refactor only changes behavior for pattern-first commands. Generic extraction path is preserved as-is. Full test suite run confirms no regressions.

Open Questions

• Should -i for sed consume the next argument? Current plan says yes (handles BSD sed -i '' correctly). The trade-off is a false negative for GNU sed -i 'script' file. An alternative is to peek at the next argument's content — if it looks like a sed script (contains / delimiters and is longer than a typical extension), don't consume it. Defer this refinement unless test coverage reveals it matters in practice.
• Should we add perl, ruby, python, node to the command map? These interpreters' first positional argument is a script FILE (a real path), not inline code. They are not "pattern-first" — they are "script-first" and their arguments ARE paths. They should NOT be in PATTERN_FIRST_COMMANDS. Only commands whose first argument is an inline pattern/script (not a file reference) belong here.
• Should character-based defense-in-depth be added to classifyTokenAsPathCandidate? Adding script-indicator characters ({, }, !, ;) as a rejection heuristic would catch patterns that slip through position-based skipping. This is orthogonal and could be a follow-up.