Failure auto-filing: subsystem failures become forge issues (closes #122) #209

Merged
conductor merged 22 commits from failure-auto-filing into master 2026-09-08 00:20:32 +01:00
Owner

Summary

  • Failure auto-filing (#122): subsystem failures become forge issues — one per failure signature (subsystem, error_class), marker-titled [auto-failure:...], dedup via server-side search, burst coalescing (50-burst → ≤2 writes), journal-backed spool + reconcile, T2 journal-only, RUNE_AUTOFILE_OFF kill switch. Dormant-by-construction when unwired (NullAdapter).
  • Both gate reviews discharged red-first (12 findings, 13 commits): the nonexistent /issues/search BLOCKER (real endpoint: paginated /issues?q=), append-only consumption sidecar (canonical journal never rewritten), reconcile grouping (no post-outage floods), secret redaction in excerpts, tier gate on replay, single-flight dedup, bounded queue with drop-and-persist, typed AutoFiler protocol.

Verification

  • Suite 680+84 green; pyright ratchet 118e/414w (unchanged); ruff clean; henxels exit 0.
  • Kill tests: T2 → zero client calls (through real call site); forge-down → spool, never raises into the failure path.
## Summary - Failure auto-filing (#122): subsystem failures become forge issues — one per failure signature `(subsystem, error_class)`, marker-titled `[auto-failure:...]`, dedup via server-side search, burst coalescing (50-burst → ≤2 writes), journal-backed spool + reconcile, T2 journal-only, `RUNE_AUTOFILE_OFF` kill switch. Dormant-by-construction when unwired (NullAdapter). - Both gate reviews discharged red-first (12 findings, 13 commits): the nonexistent `/issues/search` BLOCKER (real endpoint: paginated `/issues?q=`), append-only consumption sidecar (canonical journal never rewritten), reconcile grouping (no post-outage floods), secret redaction in excerpts, tier gate on replay, single-flight dedup, bounded queue with drop-and-persist, typed AutoFiler protocol. ## Verification - Suite 680+84 green; pyright ratchet 118e/414w (unchanged); ruff clean; henxels exit 0. - Kill tests: T2 → zero client calls (through real call site); forge-down → spool, never raises into the failure path.
conductor added 18 commits 2026-09-07 17:49:42 +01:00
Gitea/Forgejo has no repo-scoped /issues/search path — every dedup
search 404'd on a real forge, so everything spooled and nothing ever
filed. Dedup now queries GET /issues?state=open&type=issues&q=<marker>
(paginated, server-side q) with a client-side marker contains-check on
title/body. FakeForge keeps mirroring the server contract (q matches
titles only) instead of the client's filter; docs updated.
JsonlSpool.mark_consumed's read-modify-write on the shared canonical
journal raced concurrent DurableEventStore.appends (silently destroyed
between read and rewrite) and a crash mid-rewrite poisoned the store.
Consumption is now an append-only <journal>.consumed sidecar (atomic
O(1) appends, no O(N) rewrite); unconsumed() = spool lines minus
sidecar ids. Canonical journal stays byte-identical.
Per-line replay after an outage comment-flooded the issue. Reconcile
now groups unconsumed spool lines by signature and issues ONE write per
signature with the aggregated count ('+N occurrences while offline
(last: ...)'); all lines of a confirmed-written group are marked
consumed together.
excerpt() fed raw str(error)[:200] into persistent forge issues and
spool lines — broker exceptions can embed credentialed URLs/DSNs. A
redaction pass now strips query-string credential params, URL userinfo
(scheme://user:pass@host DSNs), Authorization/bearer headers, and
assignment forms, keeping the error class and message shape visible.
Applied inside excerpt() so every create/comment/spool path is covered.
reconcile called _write directly with the record's tier, bypassing the
gate — a crafted tier=2 spool line replayed to the forge. The gate is
now a module-level predicate (tier_gated) shared by submit, file, the
worker batch (defense in depth) and reconcile; T2 spool lines resolve
journal-only on replay (consumed, never written, reported as skipped).
The reconcile thread and the worker racing on the same signature could
each pass the search-then-create window and double-file. _write now
guards the write with an in-flight set (existing lock): a racing caller
returns False and its spool line stays spooled for a deduped replay.
Docstring notes the documented limitation: cross-PROCESS dedup remains
best-effort TOCTOU between search and create.
The red test called file() (synchronous inline write) instead of
submit(); the worker-vs-reconcile race the finding describes needs the
queued worker path. Also fixed the race double to return numbered
search results (real forge shape) so the dedup replay can comment.
queue.Queue() was unbounded — a forge outage could balloon failure-path
memory. The queue is now capped at 1000; an overflow occurrence is
dropped from the queue but spooled to the journal (nothing is lost) and
the drop is logged. put_nowait keeps the failure path non-blocking.
A 10KB single-token error string made a 10KB dedup marker (issue titles
and q= params unbounded). error_class_of now caps at 64 chars for both
exception names and string tokens. Also replaces the vacuous
forge_created_not_called() assertion (unconditional True) with real
fake-state assertions (zero client calls, failed=1, line left spooled).
subsystem_health.set_auto_filer/record_failure carried filer: Any | None.
A minimal AutoFiler Protocol (callable with (subsystem, error, tier),
matching FailureAutoFiler.__call__ and the test doubles) now carries the
contract; _reconcile_guarded's filer param in rune_operator is typed to
FailureAutoFiler.
Follow-up to the AutoFiler protocol change: _reconcile_guarded carried
filer: Any; it now annotates the FailureAutoFiler (TYPE_CHECKING import,
string annotation — no future import in this module). The startup
reconcile thread is explicitly documented as the maintenance cadence
(finding 11's deviation record).
Findings 11/12: the operator startup reconcile thread (rune_operator.py:1986)
IS the only reconcile cadence — no recurring timer exists in the codebase;
recorded as deviation 8 per the deviations contract. The O(N^2)
mark_consumed note is resolved by the finding-2 sidecar design. Full
finding -> commit -> red-evidence table appended to the Completion Record.
Merge remote-tracking branch 'gitea/master' into failure-auto-filing
Auto-label / priority-gate (pull_request) Skipped
Auto-label / auto-label (pull_request) Successful in 1s
CI Pipeline / lint (push) Successful in 23s
CI Pipeline / unit-tests (push) Successful in 30s
CI Pipeline / canary (push) Successful in 30s
CI Pipeline / best-practices (push) Successful in 2s
CI Pipeline / documentation (push) Successful in 1s
CI Pipeline / type-check (push) Successful in 21s
CI Pipeline / gitea-offline (push) Successful in 14s
CI Pipeline / required-checks (push) Skipped
CI Pipeline / lint (pull_request) Successful in 22s
CI Pipeline / unit-tests (pull_request) Successful in 30s
CI Pipeline / canary (pull_request) Successful in 26s
CI Pipeline / best-practices (pull_request) Successful in 2s
CI Pipeline / documentation (pull_request) Successful in 2s
CI Pipeline / type-check (pull_request) Successful in 22s
CI Pipeline / gitea-offline (pull_request) Successful in 13s
CI Pipeline / required-checks (pull_request) Successful in 1s
6946dbc0ba
conductor added 1 commit 2026-09-07 21:01:23 +01:00
Merge remote-tracking branch 'gitea/master' into failure-auto-filing
CI Pipeline / lint (pull_request) Successful in 24s
CI Pipeline / unit-tests (pull_request) Successful in 32s
CI Pipeline / canary (pull_request) Successful in 25s
CI Pipeline / best-practices (pull_request) Successful in 2s
CI Pipeline / documentation (pull_request) Successful in 2s
CI Pipeline / type-check (pull_request) Successful in 22s
CI Pipeline / gitea-offline (pull_request) Successful in 14s
CI Pipeline / required-checks (pull_request) Successful in 0s
auto-version.yml (push) Invalid workflow file
CI Pipeline / best-practices (push) Successful in 5s
CI Pipeline / documentation (push) Successful in 2s
CI Pipeline / lint (push) Successful in 40s
CI Pipeline / canary (push) Failing after 33s
CI Pipeline / unit-tests (push) Failing after 43s
CI Pipeline / gitea-offline (push) Successful in 22s
CI Pipeline / type-check (push) Successful in 36s
CI Pipeline / required-checks (push) Skipped
0a5e76e8c6
conductor added 1 commit 2026-09-07 22:34:32 +01:00
merge master (behind-base update for robot drain)
CI Pipeline / best-practices (push) Successful in 3s
CI Pipeline / documentation (push) Successful in 2s
CI Pipeline / lint (push) Successful in 24s
CI Pipeline / canary (push) Successful in 27s
CI Pipeline / type-check (push) Successful in 23s
CI Pipeline / gitea-offline (push) Successful in 13s
CI Pipeline / unit-tests (push) Successful in 34s
CI Pipeline / required-checks (push) Skipped
CI Pipeline / best-practices (pull_request) Successful in 3s
CI Pipeline / documentation (pull_request) Successful in 2s
CI Pipeline / lint (pull_request) Successful in 23s
CI Pipeline / canary (pull_request) Successful in 24s
CI Pipeline / unit-tests (pull_request) Successful in 33s
CI Pipeline / type-check (pull_request) Successful in 22s
CI Pipeline / gitea-offline (pull_request) Successful in 14s
CI Pipeline / required-checks (pull_request) Successful in 0s
b65bd50af1
conductor added 1 commit 2026-09-07 22:46:23 +01:00
merge master (behind-base update, round 2)
CI Pipeline / best-practices (push) Successful in 2s
CI Pipeline / documentation (push) Successful in 2s
CI Pipeline / lint (push) Successful in 25s
CI Pipeline / canary (push) Successful in 25s
CI Pipeline / type-check (push) Successful in 23s
CI Pipeline / unit-tests (push) Successful in 33s
CI Pipeline / gitea-offline (push) Successful in 14s
CI Pipeline / required-checks (push) Skipped
CI Pipeline / best-practices (pull_request) Successful in 2s
CI Pipeline / documentation (pull_request) Successful in 2s
CI Pipeline / lint (pull_request) Successful in 24s
CI Pipeline / canary (pull_request) Successful in 28s
CI Pipeline / type-check (pull_request) Successful in 22s
CI Pipeline / unit-tests (pull_request) Successful in 34s
CI Pipeline / gitea-offline (pull_request) Successful in 14s
CI Pipeline / required-checks (pull_request) Successful in 1s
c2f5c23063
conductor added 1 commit 2026-09-08 00:17:27 +01:00
merge master (behind-base)
CI Pipeline / best-practices (pull_request) Successful in 2s
CI Pipeline / lint (pull_request) Successful in 27s
CI Pipeline / documentation (pull_request) Successful in 3s
CI Pipeline / unit-tests (pull_request) Successful in 35s
CI Pipeline / canary (pull_request) Successful in 29s
CI Pipeline / gitea-offline (pull_request) Successful in 14s
CI Pipeline / type-check (pull_request) Successful in 24s
CI Pipeline / best-practices (push) Successful in 2s
CI Pipeline / documentation (push) Successful in 2s
CI Pipeline / lint (push) Successful in 26s
CI Pipeline / canary (push) Successful in 24s
CI Pipeline / unit-tests (push) Successful in 34s
CI Pipeline / gitea-offline (push) Successful in 14s
CI Pipeline / type-check (push) Successful in 22s
CI Pipeline / required-checks (push) Skipped
CI Pipeline / required-checks (pull_request) Successful in 1s
Spec gate (headless) / spec-gate (pull_request) Successful in 7s
Auto-version and Release / release (pull_request) Failing after 15m27s
65add381ea
Author
Owner

🤖 Headless spec gate (non-blocking, #204 first slice)

Spec-gate agent produced no text output — check runner logs.

Deterministic floors still govern the merge

This comment is advisory. Required checks (lint, tests, canaries, type-check, coverage, dependency lock, impact gate) remain the merge authority per docs/adversarial-review.md proportionality.

🤖 **Headless spec gate (non-blocking, #204 first slice)** Spec-gate agent produced no text output — check runner logs. <details><summary>Deterministic floors still govern the merge</summary> This comment is advisory. Required checks (lint, tests, canaries, type-check, coverage, dependency lock, impact gate) remain the merge authority per docs/adversarial-review.md proportionality.</details>
conductor merged commit 05f0f22dc1 into master 2026-09-08 00:20:32 +01:00
Sign in to join this conversation.