conductor/rune
1
2 Fork 1
Code Issues 36 Pull Requests 31 Actions 93 Packages Projects Releases 564 Wiki Activity

Production failure → sanitised regression fixture pipeline #157

New Issue
Closed
opened 2026-09-05 19:43:08 +01:00 by conductor · 3 comments
conductor commented 2026-09-05 19:43:08 +01:00
Owner
Copy link
Copy source

Plain summary (EXTERNAL_PROJECT_LESSONS_HANDOFF §4, Souveraine real-failure culture)

When a real failure is fixed, preserve the ACTUAL failing payload/state as a regression fixture where practical. LLMs generalise from remembered descriptions of failures and then test their own interpretation — the exact error/payload often matters.

Acceptance

  • For supported failure classes (provider errors, malformed payloads, corruption, protocol violations): real failure → sanitised fixture (secrets stripped) → regression test/canary → future recurrence fails CI
  • Trivial bugs exempt (documented exemption list)
  • Fixture storage: tests/fixtures/production/ with a README documenting the capture flow
## Plain summary (EXTERNAL_PROJECT_LESSONS_HANDOFF §4, Souveraine real-failure culture) When a real failure is fixed, preserve the ACTUAL failing payload/state as a regression fixture where practical. LLMs generalise from remembered descriptions of failures and then test their own interpretation — the exact error/payload often matters. ## Acceptance - [ ] For supported failure classes (provider errors, malformed payloads, corruption, protocol violations): real failure → sanitised fixture (secrets stripped) → regression test/canary → future recurrence fails CI - [ ] Trivial bugs exempt (documented exemption list) - [ ] Fixture storage: `tests/fixtures/production/` with a README documenting the capture flow
conductor added the P1 — high label 2026-09-05 19:43:09 +01:00
gitea-actions Bot commented 2026-09-05 19:43:36 +01:00
Copy link
Copy source

⚠️ Priority gate: this issue has no priority. Assign exactly one of P0 — critical (spine/correctness/blocks), P1 — high (material value or real defect), P2 — medium (quality/consolidation), P3 — low (measurement/deferred). An issue without a priority is not filed.

⚠️ **Priority gate:** this issue has no priority. Assign exactly one of **P0 — critical** (spine/correctness/blocks), **P1 — high** (material value or real defect), **P2 — medium** (quality/consolidation), **P3 — low** (measurement/deferred). An issue without a priority is not filed.
gitea-actions Bot added the bugcustodian labels 2026-09-05 19:43:38 +01:00
conductor added P0 — critical and removed P1 — high labels 2026-09-07 11:14:03 +01:00
gitea-actions Bot commented 2026-09-07 11:15:15 +01:00
Copy link
Copy source

⚠️ Priority gate: this issue has no priority. Assign exactly one of P0 — critical (spine/correctness/blocks), P1 — high (material value or real defect), P2 — medium (quality/consolidation), P3 — low (measurement/deferred). An issue without a priority is not filed.

⚠️ **Priority gate:** this issue has no priority. Assign exactly one of **P0 — critical** (spine/correctness/blocks), **P1 — high** (material value or real defect), **P2 — medium** (quality/consolidation), **P3 — low** (measurement/deferred). An issue without a priority is not filed.
conductor commented 2026-09-07 12:22:45 +01:00
Author
Owner
Copy link
Copy source

#157 failure→fixture pipeline — OUTWARD evidence trail (fetched 2026-09-07)

Evidence pulled live from primary sources (docs fetched, CLIs run); 4 dead URLs encountered during research were substituted with working primaries. This comment is the verifiable source trail for the plan.

**#157 failure→fixture pipeline — OUTWARD evidence trail (fetched 2026-09-07)** - CPython devguide: tests for "normal behaviour and for error conditions" — real failures become permanent tests. https://github.com/python/devguide/blob/main/testing/run-write-tests.rst - SWE-bench: 2,294 instances "drawn from real GitHub issues and corresponding pull requests" — real-failure→test-instance precedent at scale. https://arxiv.org/abs/2310.06770 - Hypothesis ExampleDatabase: persists failing examples for replay as regression cases. https://hypothesis.readthedocs.io/en/latest/tutorial/replaying-failures.html - gitleaks: `--redact`, `--baseline-path`, `.gitleaksignore` fingerprints — established scrub/allowlist mechanics for stored artifacts. https://github.com/gitleaks/gitleaks Design consequence: capture → gitleaks-style redaction with fingerprint allowlist → tests/fixtures/<issue-id>/ → one auto-generated regression test per fixture; raw payloads never committed. _Evidence pulled live from primary sources (docs fetched, CLIs run); 4 dead URLs encountered during research were substituted with working primaries. This comment is the verifiable source trail for the plan._
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
P0 — critical
Spine/core correctness or blocks other work — do first
 P1 — high
Material user value or real defect (money, bugs, next-stage dep)
 P2 — medium
Quality/consolidation improvement
 P3 — low
Measurement / benchmark / deferred policy
 blocked
Dependencies open
 bug
Confirmed defect or regression
 custodian
epic
Rune v1 course-correction issue graph
 feature
New capability or improvement
 governance
growth
identity
in-progress
Currently being worked on by Rune
 incomplete
Incomplete work — draft PR with ledger, awaiting resume
 infrastructure
life
memory
memory-atlas
Memory Research Atlas child domain
 needs-testing
Needs manual testing with dumbed-down steps before close
 perception
ready
Order satisfied - Rune may start
 release-drift
research
Research jobs — spikes, watchlists, evidence work
 rune
rune
 scaffolding
scaffolding
 spine,correction
spine,correction
 spine,deferred
spine,deferred
 spine,resolved
spine,resolved
 strangler,burn,endgame,deferred
strangler,burn,endgame,deferred
 strangler,burn,endgame,inventory,epic
strangler,burn,endgame,inventory,epic
 tui
tui
 tui,epic
tui,epic
 tui,openspec
tui,openspec
 v1-course-correction
Rune v1 course-correction issue graph
No labels P0 — critical bug custodian
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
conductor
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: conductor/rune#157