The lab rule
A result is not an authorization. Experiments run on Bunsen, in transient service units, identity-free profiles, or detached worktrees. Herman remains untouched until Jon explicitly approves a candidate.
Every experiment records
- Hypothesis — what should change and why.
- Baseline — the pre-change number or failure mode.
- Intervention — the smallest isolated candidate.
- Result — measured runs, tests, latency, and process state.
- Falsifier — the observation that would overturn the conclusion.
- Promotion boundary — what still has to be true before Herman changes.
Evidence handling
Raw evidence stays on the lab VM and Herman’s evidence workspace. Public pages contain sanitized summaries and SHA-256 handles. Credential-bearing values, private configuration, internal addresses, and user history are not published.
A sensitive-name scanner checks values associated with TOKEN, KEY, SECRET, and PASSWORD keys. Scanner failures are treated as evidence failures, not formatting warnings.
Test interpretation
Targeted green tests prove the changed surface behaves as intended. They do not erase unrelated full-suite failures. A candidate remains held until the applicable repository test and lint gates are green in a clean promotion worktree.
Status vocabulary
- pass — the predeclared claim survived its falsifier.
- measured — useful evidence, but not a promotion candidate.
- candidate — worth human review; not installed.
- pass-with-observation — the safety claim passed while an adjacent workload criterion missed.
- hold-and-promote — a bundle contains both reviewable and blocked pieces.