store/scaffold/memory_t0_external_report.md

@@ -0,0 +1,128 @@
+# Rune Memory T0 — As-Is Memory Inventory Report
+
+*Prepared for external review. Self-contained: no prior context assumed.*
+
+---
+
+## 1. What this is
+
+An archaeology report on the memory system of **Rune** — a self-hosted AI coding assistant / personal agent running on Linux (repo: private Gitea, deployed generations under `/mnt/workdrive/rune-code`, canonical worktree `/mnt/workdrive/issue161-harness`).
+
+Rune just completed a long Scaffolding V1 programme (tranches T21.4/T21.5, accepted 2026-09-17 with 17/17 gates, deployed release v0.83.3). The first post-V1 phase is **Memory**. Its first tranche, **Memory T0**, was deliberately read-mostly: establish what memory machinery *actually exists today* before designing anything new.
+
+Central rule of T0: never collapse `EXISTS IN SOURCE ≠ WIRED ≠ RUNTIME REACHABLE ≠ ACTUALLY USED ≠ CANONICAL AUTHORITY`.
+
+**Verdict: MEMORY_T0_COMPLETE** — 17/17 gates PASS, `can_accept() = true`, merged as PR #333. Artifacts: `store/scaffold/memory_t0_gates.json` (gate object) and `store/scaffold/memory_t0_asis_map.json` (central map).
+
+---
+
+## 2. What Rune actually has today (the short answer)
+
+One canonical experiential store, one derived knowledge file, several dormant subsystems, one harness-native duplicate, and a set of adjacent non-memory stores that are correctly separated.
+
+### The canonical store
+
+`~/.config/opencode/memory/store/memory.db` — 456.8 MB SQLite with **two retrieval engines side by side**:
+
+- **Dense**: sqlite-vec, 256-dim Matryoshka-truncated embeddings via local Ollama (`nomic-embed-text`), content-hash cached.
+- **Lexical**: FTS5/BM25.
+
+`recall()` fuses both with reciprocal-rank fusion, applies a **0.42 relevance floor** (below it: abstain rather than inject distractors), supports query expansion via a lexicon, type targeting, and QPP-based abstention.
+
+**Contents**: 899 entries (812 active / 57 superseded / 25 inactive / 5 archived), 118,210 chunks, 5,732 documents, 1,149 associative links, 337 use-history rows. Warm neurons are stored as `kind='neuron'` rows (195) in the same DB — a previous file-based system (`.pasture` markdown files) was migrated and retired.
+
+**Supersession is real and receipted.** A dedupe policy gate sits on writes: similar new content either touches the existing row (`duplicate`), or — if longer/stronger — **supersedes** it (old row marked `status='superseded'`, `valid_until=now`, embedding nulled; history retained). Receipt ledger: 1,465 decisions (41 created / 1,306 duplicates / 118 superseded). Every row carries `source`, `method`, `confidence`, `temperature`, timestamps.
+
+### Derived knowledge
+
+`~/.rune/consolidation/knowledge.json` — 7,965 consolidated items (source-attributed: 7,207 from the store, 583 from abstraction, 165 from neurons, 10 from graph), actively growing (675 new in September). Read back into context via one of the selector's six sources. Whole-file rewrite per consolidation; rebuildable.
+
+### Dormant subsystems (exist, wired for read, no live writer)
+
+- **graph.sqlite** — a well-designed bi-temporal concept knowledge graph (typed nodes, `valid_from`/`valid_until` + `learned_at` edge semantics, supersession columns). 281 edges, **272 superseded**, last write 2026-08-26. The write path is a CLI; nothing in the live runtime feeds it. The read path (selector `graphs` source) queries a graph that hasn't been fed in three weeks.
+- **shards** — 131 topic-partition DB files containing **10 rows total** (one-row topic placeholders); typed queries bypass shards by policy. Effectively retired-in-place.
+- **temporal.db** — 3 facts, no importer in the repo's `src/rune` at all (harness-script-era relic).
+- **warm/memory.db** — 0 bytes; dead path.
+
+### Adjacent systems (correctly *not* memory)
+
+Task state (durable task repo + orchestration), Spine/Project Scaffold/Project Registry (structural truth), capability/behaviour ledgers (governance), the W/R event-wait bridge (task continuation), `behavior.db` (corrections/habits — borderline, near-empty). T0 verified no experiential facts live in structural stores and no structural facts in memory stores.
+
+### The duplicate-authority surface (the big architectural question)
+
+The **MiMoCode harness itself** (the VSCode-like agent harness Rune runs inside) maintains its own memory: `mimocode.db` (1.2 GB — 651 sessions, 98,692 messages, 773,595 parts, plus its own FTS) and `~/.local/share/mimocode/memory` (4.3 MB of markdown). Two systems remember the same operational life. T0 flagged this as the primary authority-boundary question for future design — nothing was reconciled.
+
+A second, smaller drift risk: the repo module `src/rune/memory_store.py` and the harness copy `~/.config/opencode/scripts/memory_store.py` are currently aligned, but the **harness plugin executes the scripts copy** while the TUI imports the repo module. Same code today, two deployment paths.
+
+---
+
+## 3. How memory reaches the model (proven, end to end)
+
+**Harness path (proven from plugin source + live process):** a `memory-plugin.ts` runs per turn:
+1. **Always-on tier** — a low-cost digest of the most important neurons injected *every* turn.
+2. **Keyword tier** — `warm_router.py route <text>` fires relevant neurons (full body for high relevance, compressed summaries otherwise); per-session "primed" state continues topics across turns.
+3. **Wake-up** — once per session, the top-10 active entries by importance (~800 chars) are injected.
+4. **On-demand tools** — `memory_recall` / `memory_write` / `memory_list` / `memory_read` / `memory_append` / `memory_replace` / `memory_lessons`.
+5. On session idle, a sleep sub-agent is launched (once per hour guard).
+
+**TUI/operator path (proven from source):** `rune_operator.send_task` → `ContextBroker.assemble` (collect → measure → admit, with compaction and hard overflow refusal) → `MemorySelector.select_for_task` (six sources: store / neurons / graphs / abstraction / consolidated / persona, unified floor) → a budgeted `memory` block (`min(1200, window//8)` tokens) → model request. Zero candidates = recorded abstention, never filler.
+
+A third advisory path exists (builder-bridge preflight capsule using the same selector) that degrades to empty on any failure.
+
+**The T21.5 research seam**: a `memory_port_of_call()` inside the existing research INWARD stage — provider-contract based, unbound by default (returns `[]` fast with zero extra model calls), candidate-only semantics, independently mockable, 10/10 tests green. Verified as safe to preserve as-is.
+
+---
+
+## 4. Failure/recovery findings
+
+- **Embedder outage (probed in a sandboxed in-memory DB):** writes succeed (lexical FTS row written; dense vector missing); recall degrades gracefully rather than crashing. The system fails soft.
+- **Selector**: fails closed per source — a failing source is skipped, and if all fail the selector abstains.
+- **Rebuildable**: FTS indexes, embeddings, knowledge.json, cache. **Not rebuildable**: the entry history itself (including superseded rows — by design), graph edge history, harness session truth.
+- **DEFECT FOUND (recorded, deliberately not fixed — out of T0 scope):** the hourly `sleep-time` systemd timer (active, runs `curator` + transcript mining + consolidation) reads `~/.local/share/opencode/opencode.db` — a **stale database from a defunct harness, 30.7 days old** — instead of the live `~/.local/share/mimocode/mimocode.db`. Every run logs "No sessions found". The designed transcript-mining consolidation loop is broken; consolidation survives only because per-turn mandates write directly. This is the first concrete fix candidate for the Memory build phase.
+
+---
+
+## 5. Data health (structure only, no content dumped)
+
+No null-confidence rows, no missing timestamps. Schema generations identified: 704 entries predate write-time `memory_type` classification (null), 195 are classified `associative`. Shards/`warm_state.json`/temporal are residue. 121 of 131 shard files are empty. The oldest live data is 2026-08-12; the newest is the day of the audit. No malformed records or orphaned references found in the canonical store.
+
+---
+
+## 6. Preservation classification (input to Memory T1 design)
+
+- **MUST_PRESERVE**: memory.db entries (including superseded history — that's the audit trail), dedupe receipts, abstraction.db facts, graph edge history, harness session truth.
+- **LIKELY_PRESERVE**: knowledge.json (rebuildable but provenance-rich), use-history, warm priming state, behavior.db.
+- **REBUILDABLE**: cache, FTS, embeddings, knowledge.json, shards.
+- **LEGACY_BUT_UNIQUE**: temporal.db's 3 facts (inspect before retiring); .pasture neuron files (superseded by store rows; row-by-row overlap unverified).
+
+---
+
+## 7. Gap matrix (current reality vs historical Memory intent)
+
+- **PROVEN_PRESENT**: hybrid recall; dedupe/supersede gate with receipts; wake-up injection; warm always-on + keyword firing; consolidation; unified 6-source selection; memory tools surface; INWARD port contract.
+- **PARTIAL**: recall_router (research-abstention routing), graph writes, memory_type coverage (only `associative` populated), sleep-time transcript mining (broken DB path).
+- **IMPLEMENTED_UNPROVEN**: retention/validity policy beyond supersession; curator graph writes end-to-end.
+- **DORMANT**: shards, temporal.db, warm/memory.db, .pasture.
+- **ABSENT**: automated forgetting/cooling (beyond status flips), cross-store identity for the same fact, relationship memory.
+- **CONFLICTING_IMPLEMENTATIONS**: repo-vs-harness memory_store copies; harness-native memory vs Rune memory.
+- **UNKNOWN**: none remaining (T0 exit criterion).
+
+---
+
+## 8. What was *not* done (by design)
+
+No future architecture designed. No data migrated, repaired, or deleted. No runtime guard or behavior changed. Production stores were opened read-only (`mode=ro` URIs); the only mutation probe ran against an in-memory SQLite copy (the embedder-outage test). Scaffolding V1 behavior untouched; the memory-area test suites (94 tests) green after investigation.
+
+---
+
+## 9. Recommended next phase (Memory T1 — design, not build)
+
+1. Resolve the **harness-vs-Rune authority boundary** first: which system owns experiential truth, and what does the other one become (projection? source? separate scope?).
+2. Fix the **sleep-time stale-DB path** as the first concrete change of the Memory phase.
+3. Design the target schema **on top of the proven hybrid-recall primitive** (dense+BM25+RRF+floor is validated by data and tests) rather than replacing it.
+4. Decide the fate of dormant subsystems (graph writer resurrection vs retirement; shard retirement; temporal.db inspection) with the preservation register in hand.
+5. Close the repo-vs-harness code-copy drift risk.
+
+---
+
+*Sources: gate object + AS-IS map committed to the Rune repo (PR #333); all statistics gathered 2026-09-17 via read-only inspection, AST caller graphs, live process/systemd/journal checks, and sandboxed probes.*