SamCDay/rune

1 Fork 0

Files

T

Nick c3c95688f7 docs(fixtures): fixture pipeline workflow + seed fixture (#157 )

Seeds the pipeline with the REAL #192 CONVERGED case-mismatch payload: the
safety-net convergence step exported Python-cased True/False from
json.load while the steps compared env.CONVERGED == 'false' — both
conditionals unreachable (discharged on the release-safety-net branch,
public in PR #211's diff). Captured via the pipeline itself
(--class env-case-mismatch --source ci, 0 redactions — the payload is
already public and credential-free). The generated regression test asserts
the boolean-export checker still FLAGS the recorded payload
(checker_expectation: flags), so a recurrence of the class fails CI.

Manifest gains checker_expectation: capture records whether the class's
checker raises on the payload ('flags' — the raise IS the recurrence alarm)
or must stay quiet ('passes'); the generated tests enforce the recorded
expectation, so a checker that stops catching its recorded defect rots the
suite red.

Docs: bugfix-contract §10 gains the operational capture step + explicit
trivial-bug exemption list; README dev section points at the pipeline;
tests/fixtures/production/README.md documents the flow.

Henxels: capture_failure.py required in scripts/; production fixtures
governed (manifest.json + README.md required); payload.json and
manifest.json travel together (changed_with both directions, verified to
bite on a payload-only staged change).

2026-09-07 19:51:36 +01:00

6.0 KiB

Raw Permalink Blame History

Bug-fix contract

Binding on every agent, session, and pipeline stage in this repository. The henxel mirror of this contract lives in henxels.yaml ("bugs discharged red-first"); the enforcement hooks live in scripts/check_discharge_contract.py and .gitea/workflows/pipeline-gate.yml. Escapes from this contract are made by editing those files — visibly — never by silent deviation.

The contract

When a bug is found — by a gate, a reviewer, CI, a canary, production, or a human — the fix MUST follow this cycle before it may merge:

Red first. A failing test that reproduces the bug is written BEFORE the fix. The test must fail for the bug's cause, not for an unrelated reason. A fix without a red test is a symptom patch and is rejected at review.
Root cause, not symptom. The fix addresses the cause. If the reporter's diagnosis is wrong, the discharge says so and names the real cause.
Full pipeline re-entry. The fix is treated as a NEW SUBMISSION: full suite, pyright ratchet (not increased), ruff, canaries, henxels, and the spec + quality gates for anything beyond a mechanical minor. No "hotfix-skips-gates" path exists.
The regression test is permanent. Tests written for a discharged bug are frozen — they may be refactored, never weakened or deleted. The suite can only grow in assertion strength over time.
Recorded. The fix's PR (or plan tail) maps every finding → the red test → the commit. A discharge record with holes bounces at review.

Scope and proportionality

Gate findings (spec/quality): every BLOCKER/MAJOR gets the full cycle; MINORs get at minimum a red test OR an explicit reviewer-approved waiver recorded in the plan tail.
Production failures: the fix follows this contract AND, where the failing payload is preservable, #157's fixture pipeline (real failure → sanitised fixture → permanent regression test).
CI/workflow bugs: the red test is the workflow-contract assertion (parsed-YAML step/step-logic assertions, as in test_workflow_script_contract.py) — workflows are code; their bugs get tests too.
Trivial/no-test-class bugs (typos in comments, log strings): documented exemption list in this file. PR-only changes to those strings skip the red test.

Exemption list (trivial/no-test-class)

The red test is skipped only when the change is mechanical and carries no behavioural surface. Current exemptions:

typos in comments, docstrings, and documentation prose;
log-message wording with no parsing consumer (nothing asserts on the text);
README/wiki wording.

Anything touching executable behaviour — including workflow YAML steps, scripts, and configuration consumed at runtime — is NOT exempt.

Why (the failure class this kills)

LLM-authored changes share wrong assumptions between implementation, tests, and docs. A symptom patch shares the reporter's wrong assumption with the fix. The red-first test is the cheapest known breaker of both: it forces the fixer to state the bug's cause in executable form before claiming to have removed it.

Enforcement

scripts/check_discharge_contract.py: verifies a PR that touches files named in a Discharge: trailer block carries at least one test-hunk per finding id (structural check — PR description convention: Discharge: <id> -> <test file>).
Gate reviewers are instructed (dispatch templates, issue #151) to reject symptom patches: "what test failed red for this bug?" is a mandatory reviewer question for every fix PR.
henxels.yaml carries the tracked one-line mirror so the contract is part of the repo's auditable contract surface.

Soak-before-close for intermittent defects. A flaky, timing, or event-delivery bug cannot be proven fixed by one green run. The fix merges through the normal pipeline, but the issue CLOSES only on soak evidence: a stated criterion (e.g. "N consecutive green runs" or "48h without recurrence") plus the observations meeting it. The closing comment carries the soak record. Merge ≠ closure for this class.
Escape accountability. When a bug reaches production (escaped every gate), the discharge must answer: which gate should have caught this, and why didn't it? The answer becomes a gate improvement — a new contract assertion, a checker rule, or a documented gate limit. Gate hardening is a first-class output of bug fixing, not an afterthought.
Fresh-context root-cause verification. The red test proves the fixer understood a cause. For BLOCKER/MAJOR fixes, a fresh-context reviewer additionally asks: does this test fail on the pre-fix code for the right reason? (Same doctrine as the black-box test author: the person who wrote the fix does not certify its diagnosis.)
Déjà-vu detection. Before writing the red test, search the suite and closed issues for the same failure class. A second occurrence of a class means the first fix treated a symptom — the discharge must then cover the class, not the instance.
Fixture promotion. When the red test needs a complex payload, it goes through #157's sanitised-fixture pipeline (real failure → secrets stripped → tests/fixtures/production/) instead of a hand-crafted approximation. Real failures are the best regression tests. Operationally: after the fix merges, run the capture command from tests/fixtures/production/README.md (python3 scripts/capture_failure.py payload.txt --class <class> --source <issue|ci|manual>) on the ACTUAL payload and commit the fixture + manifest with the discharge record. The generated regression tests (tests/fixtures/production/test_production_fixtures.py) assert the payload's sha256 against the manifest (tamper-evidence) and that the class's checker still catches the recorded defect — a recurrence of the captured failure fails CI. Exemptions from fixture capture follow the trivial-bug rule below: a typo or a log string has no payload worth preserving.

6.0 KiB Raw Permalink Blame History