rune/docs/failure-semantics.md

Failure Semantics (P0-06)

Every except on the authoritative path must re-raise, return a structured failure, or emit. A silent pass is a defect. This document defines the three failure tiers, the observability core (src/rune/subsystem_health.py), and how to extend them.

The three tiers

Tier	Meaning	Behaviour on failure	User-visible?
T0	Authoritative-path failure (operator execution, tool loop, model transport)	Loud: canonical event + structured failure (success=False, error text). The operator's task-transition path already returns OperatorResponse(success=False, error=...) — that is the T0 contract.	Yes — error bubble, never silent
T1	Context-contribution failure (memory selector, warm router, capability broker, consolidation, …)	Degrade: the turn proceeds without the contribution, but the failure is journaled and surfaced once per subsystem per session.	Yes — one ⚠ <subsystem> unavailable message (anti-spam), then journal-only
T2	Ambient/background failure (cron, event-bus emits, report side-effects)	Journal only; the turn is unaffected.	No — visible via /health

The core: rune.subsystem_health

from rune.subsystem_health import record_failure

try:
    context = memory_store.cached_recall(db, query)
except Exception as exc:  # P0-06: observable degrade (T1)
    if record_failure("memory_selector", exc, tier=1):
        ch.add_message("system", f"⚠ memory_selector unavailable: {exc}")

record_failure(subsystem, error, tier, registry=None, context="") does three things:

1. Logs via logging.getLogger("rune.subsystem_health").
2. Emits a canonical event through rune.event_bus (tier 0/1 → task.failed / capability.failed, tier 2 → storage.degraded; payload carries subsystem, tier, truncated error).
3. Records the failure in the process-wide SubsystemHealthRegistry (failures / last_error / tier per subsystem).

It returns True when the caller should ALSO surface the failure to the user (first T0/T1 occurrence for that (tier, subsystem) pair — anti-spam). T2 never returns True. Events always fire (the journal is cheap); user messages fire once.

The operator process has no chat channel, so its T1 conversions append the subsystem name to a degraded list on OperatorResponse; the envelope carries it to the TUI, which renders ⚠ <subsystem> unavailable (operator path) lines.

Logging destination

configure_logging(log_dir=None) attaches a rotating file handler (~/.rune/logs/rune.log, 5 MB × 3) to the rune logger tree. Before P0-06 the loggers existed but had no handler — logs went nowhere. RuneApp.__init__ calls it once per process (guarded, so a read-only home never blocks startup). Tests call configure_logging(log_dir=tmp_path) with an explicit directory.

The /health command

/health in the TUI renders the registry snapshot — per subsystem: failures count, tier, last error — plus the event-journal total. Zero failures → "all subsystems healthy". Use it after any suspected breakage: if a subsystem degraded, it is in the ledger even when the turn looked normal.

Adding a new subsystem

1. Pick a name (kebab/snake, matches the module: memory_selector, warm_router, capability_broker, witness, consolidation, persona, abstraction, graph_memory, context_pressure, drift_check, goals, time_context, cron, provider_registry, model_routing, …).
2. Classify the call site: does the turn legitimately proceed without it? Yes → T1 (surface once). It is background/side-effect → T2 (journal only). It is the answer itself → T0 (re-raise or structured failure; do not swallow).
3. Convert the handler with the pattern above. The if record_failure(...) guard is mandatory for T1 — it is what makes the anti-spam contract work.
4. Check /health after forcing a failure; the subsystem must appear in the ledger and the event must land in the journal.

Failure auto-filing (#122)

T0/T1 failures additionally become ONE forge issue per failure signature (subsystem, error_class) via src/rune/failure_auto_filer.py — the forge's own watch system pings you; repeats comment with a rising count instead of flooding the board. T2 failures (cron noise) NEVER file.

How it works:

• Signature — (subsystem, error_class) where error_class is the exception class name (or first token of the error string). Never the full message: variable payloads would flood.
• Dedup — before filing, a repo-scoped marker search (ForgeClient.search_issues, GET /issues?state=open&type=issues&q=<marker> — Gitea/Forgejo has no repo-scoped /issues/search path) looks for the marker [auto-failure:<subsystem>:<error_class>] in titles and bodies. Found → add_comment with a rising count; not found → create_issue. The list endpoint is paginated, so older failure issues are still seen.
• Burst coalescing — an in-memory per-signature counter batches repeats into one comment per 10 occurrences or 10 minutes (whichever first). A 3am crash-loop writes one comment, not 200.
• Spool + reconcile — any client exception (forge unreachable/auth/ 5xx) appends a failure_auto_file_spool JSON line to the existing event journal. reconcile() (startup + recovery) replays them; dedup re-applies, filed lines are marked consumed.
• Threading — the failure path only enqueues; one daemon worker drains the queue and does all network work. Failure-path latency is untouched, and nothing ever raises into the caller's except block.
• Tier gate — tier in (0, 1) file/comment; tier >= 2 journal only. Enforced in ONE place: the filer.

Wiring: the composition root (rune_operator.get_operator) creates one FailureAutoFiler via create_forge_client() (unconfigured → NullAdapter → dormant by construction) and wires it through subsystem_health.set_auto_filer; record_failure(..., filer=...) accepts an explicit override. RUNE_AUTOFILE_OFF=1 disables entirely (fail-safe off switch).

Scope note

P0-06 converted the ~30 critical handlers on the authoritative path (surveyed in plans/2026-09-04-p0-06-ledger.md — the worst was one except: pass wrapping the four-subsystem parallel block in rune_tui._run_warm_router; it is now split so each subsystem reports independently). The remaining ~390 repo-wide silent handlers are a follow-up sweep.