feat(chrome): hand snapshots to context mode

This commit is contained in:
云服务部-叶林立
2026-08-28 14:45:37 +08:00
parent 221e978622
commit f3a2abe1e4
96 changed files with 12800 additions and 26 deletions
+44
View File
@@ -0,0 +1,44 @@
# Benchmark scoring
## Unit challenges (`manifest.json`)
Score actual verdict against `expected[mode]`, not raw PASS count.
Gate buckets:
- `core`: headline release gate. Exact expected verdict required.
- `conditional`: gate only when declared capability/prerequisite is present; otherwise report skipped/conditional.
- `quality`: adversarial humanization/fingerprint signal. Report trend and regressions, but do not block general release unless explicitly promoted.
Expected values:
- `PASS` / `FAIL` expected: exact match required inside active gate bucket.
- `CONDITIONAL`: exclude from core headline score unless environment prerequisite is declared present.
- `SKIP` / `WARN`: report separately.
Recommended flake policy:
- Run each non-destructive challenge up to 2 retries.
- Take best verdict only for known-flaky timing/scroll tests.
- Always keep all reasons/events in detailed report.
Difficulty weighting:
- L1 = 1
- L2 = 2
- L3 = 3
Report both unweighted and weighted score. Also report per-category score.
## Long-horizon tasks (`task-manifest.json`)
Task score is deterministic grader PASS / not PASS. Record:
- action count
- wall time
- tools used
- observation mode (`snapshot`, screenshot, accessibility, evaluate)
- whether direct state mutation/evaluate was allowed
- final grader reason
Do not use LLM judge for hermetic tasks unless grader cannot express the target.