# Failure Semantics (P0-06)

Every `except` on the authoritative path must **re-raise, return a structured
failure, or emit**. A silent `pass` is a defect. This document defines the
three failure tiers, the observability core (`src/rune/subsystem_health.py`),
and how to extend them.

## The three tiers

| Tier | Meaning | Behaviour on failure | User-visible? |
|------|---------|----------------------|---------------|
| **T0** | Authoritative-path failure (operator execution, tool loop, model transport) | Loud: canonical event + structured failure (`success=False`, error text). The operator's task-transition path already returns `OperatorResponse(success=False, error=...)` — that is the T0 contract. | Yes — error bubble, never silent |
| **T1** | Context-contribution failure (memory selector, warm router, capability broker, consolidation, …) | Degrade: the turn proceeds without the contribution, but the failure is journaled and surfaced once per subsystem per session. | Yes — one `⚠ <subsystem> unavailable` message (anti-spam), then journal-only |
| **T2** | Ambient/background failure (cron, event-bus emits, report side-effects) | Journal only; the turn is unaffected. | No — visible via `/health` |

## The core: `rune.subsystem_health`

```python
from rune.subsystem_health import record_failure

try:
    context = memory_store.cached_recall(db, query)
except Exception as exc:  # P0-06: observable degrade (T1)
    if record_failure("memory_selector", exc, tier=1):
        ch.add_message("system", f"⚠ memory_selector unavailable: {exc}")
```

`record_failure(subsystem, error, tier, registry=None, context="")` does three
things:

1. **Logs** via `logging.getLogger("rune.subsystem_health")`.
2. **Emits a canonical event** through `rune.event_bus` (tier 0/1 →
   `task.failed` / `capability.failed`, tier 2 → `storage.degraded`; payload
   carries subsystem, tier, truncated error).
3. **Records the failure** in the process-wide `SubsystemHealthRegistry`
   (failures / last_error / tier per subsystem).

It returns `True` when the caller should ALSO surface the failure to the user
(first T0/T1 occurrence for that `(tier, subsystem)` pair — anti-spam). T2
never returns `True`. Events always fire (the journal is cheap); user messages
fire once.

The operator process has no chat channel, so its T1 conversions append the
subsystem name to a `degraded` list on `OperatorResponse`; the envelope carries
it to the TUI, which renders `⚠ <subsystem> unavailable (operator path)` lines.

## Logging destination

`configure_logging(log_dir=None)` attaches a rotating file handler
(`~/.rune/logs/rune.log`, 5 MB × 3) to the `rune` logger tree. Before P0-06 the
loggers existed but had **no handler** — logs went nowhere. `RuneApp.__init__`
calls it once per process (guarded, so a read-only home never blocks startup).
Tests call `configure_logging(log_dir=tmp_path)` with an explicit directory.

## The `/health` command

`/health` in the TUI renders the registry snapshot — per subsystem: failures
count, tier, last error — plus the event-journal total. Zero failures →
"all subsystems healthy". Use it after any suspected breakage: if a subsystem
degraded, it is in the ledger even when the turn looked normal.

## Adding a new subsystem

1. Pick a name (kebab/snake, matches the module: `memory_selector`,
   `warm_router`, `capability_broker`, `witness`, `consolidation`, `persona`,
   `abstraction`, `graph_memory`, `context_pressure`, `drift_check`, `goals`,
   `time_context`, `cron`, `provider_registry`, `model_routing`, …).
2. Classify the call site: does the turn legitimately proceed without it?
   Yes → T1 (surface once). It is background/side-effect → T2 (journal only).
   It is the answer itself → T0 (re-raise or structured failure; do not
   swallow).
3. Convert the handler with the pattern above. The `if record_failure(...)`
   guard is mandatory for T1 — it is what makes the anti-spam contract work.
4. Check `/health` after forcing a failure; the subsystem must appear in the
   ledger and the event must land in the journal.

## Failure auto-filing (#122)

T0/T1 failures additionally become ONE forge issue per failure signature
`(subsystem, error_class)` via `src/rune/failure_auto_filer.py` — the
forge's own watch system pings you; repeats comment with a rising count
instead of flooding the board. T2 failures (cron noise) NEVER file.

How it works:

- **Signature** — `(subsystem, error_class)` where `error_class` is the
  exception class name (or first token of the error string). Never the
  full message: variable payloads would flood.
- **Dedup** — before filing, a repo-scoped marker search
  (`ForgeClient.search_issues`, `GET /issues?state=open&type=issues&q=<marker>`
  — Gitea/Forgejo has no repo-scoped `/issues/search` path) looks for
  the marker `[auto-failure:<subsystem>:<error_class>]` in titles and
  bodies. Found → `add_comment` with a rising count; not found →
  `create_issue`. The list endpoint is paginated, so older failure
  issues are still seen.
- **Burst coalescing** — an in-memory per-signature counter batches
  repeats into one comment per 10 occurrences or 10 minutes (whichever
  first). A 3am crash-loop writes one comment, not 200.
- **Spool + reconcile** — any client exception (forge unreachable/auth/
  5xx) appends a `failure_auto_file_spool` JSON line to the existing
  event journal. `reconcile()` (startup + recovery) replays them; dedup
  re-applies, filed lines are marked consumed.
- **Threading** — the failure path only enqueues; one daemon worker
  drains the queue and does all network work. Failure-path latency is
  untouched, and nothing ever raises into the caller's except block.
- **Tier gate** — `tier in (0, 1)` file/comment; `tier >= 2` journal
  only. Enforced in ONE place: the filer.

Wiring: the composition root (`rune_operator.get_operator`) creates one
`FailureAutoFiler` via `create_forge_client()` (unconfigured →
`NullAdapter` → dormant by construction) and wires it through
`subsystem_health.set_auto_filer`; `record_failure(..., filer=...)`
accepts an explicit override. `RUNE_AUTOFILE_OFF=1` disables entirely
(fail-safe off switch).

## Scope note

P0-06 converted the ~30 critical handlers on the authoritative path (surveyed
in plans/2026-09-04-p0-06-ledger.md — the worst was one `except: pass`
wrapping the four-subsystem parallel block in `rune_tui._run_warm_router`; it
is now split so each subsystem reports independently). The remaining ~390
repo-wide silent handlers are a follow-up sweep.