mirror of
https://bitbucket.org/siakitem/my-pi.git
synced 2026-08-28 08:35:57 +00:00
feat(chrome): hand snapshots to context mode
This commit is contained in:
@@ -0,0 +1,44 @@
|
||||
# Benchmark scoring
|
||||
|
||||
## Unit challenges (`manifest.json`)
|
||||
|
||||
Score actual verdict against `expected[mode]`, not raw PASS count.
|
||||
|
||||
Gate buckets:
|
||||
|
||||
- `core`: headline release gate. Exact expected verdict required.
|
||||
- `conditional`: gate only when declared capability/prerequisite is present; otherwise report skipped/conditional.
|
||||
- `quality`: adversarial humanization/fingerprint signal. Report trend and regressions, but do not block general release unless explicitly promoted.
|
||||
|
||||
Expected values:
|
||||
|
||||
- `PASS` / `FAIL` expected: exact match required inside active gate bucket.
|
||||
- `CONDITIONAL`: exclude from core headline score unless environment prerequisite is declared present.
|
||||
- `SKIP` / `WARN`: report separately.
|
||||
|
||||
Recommended flake policy:
|
||||
|
||||
- Run each non-destructive challenge up to 2 retries.
|
||||
- Take best verdict only for known-flaky timing/scroll tests.
|
||||
- Always keep all reasons/events in detailed report.
|
||||
|
||||
Difficulty weighting:
|
||||
|
||||
- L1 = 1
|
||||
- L2 = 2
|
||||
- L3 = 3
|
||||
|
||||
Report both unweighted and weighted score. Also report per-category score.
|
||||
|
||||
## Long-horizon tasks (`task-manifest.json`)
|
||||
|
||||
Task score is deterministic grader PASS / not PASS. Record:
|
||||
|
||||
- action count
|
||||
- wall time
|
||||
- tools used
|
||||
- observation mode (`snapshot`, screenshot, accessibility, evaluate)
|
||||
- whether direct state mutation/evaluate was allowed
|
||||
- final grader reason
|
||||
|
||||
Do not use LLM judge for hermetic tasks unless grader cannot express the target.
|
||||
Reference in New Issue
Block a user