my-pi/pi-chrome/test-suite/notes/scoring.md

Benchmark scoring

Unit challenges (manifest.json)

Score actual verdict against expected[mode], not raw PASS count.

Gate buckets:

• core: headline release gate. Exact expected verdict required.
• conditional: gate only when declared capability/prerequisite is present; otherwise report skipped/conditional.
• quality: adversarial humanization/fingerprint signal. Report trend and regressions, but do not block general release unless explicitly promoted.

Expected values:

• PASS / FAIL expected: exact match required inside active gate bucket.
• CONDITIONAL: exclude from core headline score unless environment prerequisite is declared present.
• SKIP / WARN: report separately.

Recommended flake policy:

• Run each non-destructive challenge up to 2 retries.
• Take best verdict only for known-flaky timing/scroll tests.
• Always keep all reasons/events in detailed report.

Difficulty weighting:

• L1 = 1
• L2 = 2
• L3 = 3

Report both unweighted and weighted score. Also report per-category score.

Long-horizon tasks (task-manifest.json)

Task score is deterministic grader PASS / not PASS. Record:

• action count
• wall time
• tools used
• observation mode (snapshot, screenshot, accessibility, evaluate)
• whether direct state mutation/evaluate was allowed
• final grader reason

Do not use LLM judge for hermetic tasks unless grader cannot express the target.