refactor(query-memory): supersede old Python implementation with Rust cdylib (strangler steps 9-12)

Old src/rune/query_memory.py deleted (zero importers verified by sweep). Coverage-audit registry remapped: shared/query_memory.py (ctypes shim) is canonical; query_memory out of SRC_A/SHARED_D, into SHARED_B. Supersession ledger written to plans/. Steps 1-11 evidence in ledger + issue #2448.
2026-10-02 20:52:40 +01:00
1 parent 6f44a87894
commit 2ac2d0aefd
@@ -0,0 +1,48 @@
+# Phase G slice 1 — retrieval-index authority supersession ledger
+
+**Date:** 2026-10-02 · **Branch:** `phase-g-strangler-retrieval-index` · **Unit issue:** conductor/rune#2448
+
+## What was superseded
+
+The pure-Python `query_memory` implementation (formerly `src/rune/query_memory.py`,
+deployed at `~/.config/opencode/scripts/query_memory.py`) is RETIRED as a production
+implementation. Production authority now rests with:
+
+1. **Implementation:** Rust crate `src/rust/query_memory/` (lib.rs core + cdylib, ten
+   `extern "C"` FFI symbols, zero external crates).
+2. **Production service surface:** `shared/query_memory.py` — a ctypes-backed shim
+   (179 lines) over `qm_loader.py`, byte-deployed to `~/.config/opencode/scripts/query_memory.py`,
+   serving the FULL prior Python class API including the `self.history` snapshot.
+
+## Strangler doctrine (master doc lines 685–770) — step discharge
+
+| Step | Requirement | Evidence |
+|------|-------------|----------|
+| 1–4 | candidate built, contract frozen, parity proven | commits `9b76585`+`54d2db9`; `/tmp/phaseG_retrieval_index_contract.md` frozen; parity harness ALL GREEN |
+| 5 | qualification | `/tmp/phaseG_qual_bench.out` (Rust CLI ~45× faster end-to-end); canaries both sides |
+| 6 | canaries conformance | Python canary 12/12; Rust canary 27/27 |
+| 7 | shadow comparison vs live store | `/tmp/phaseG_shadow.py` 46/46 exact-equal vs 77-record production store |
+| 8 | authority switch + observed consumer evidence | commit `6f44a87`; both consumers verified live — `memory_store.py` (lines 820–821, live 76-record store, nested stats shape) + `growth_simulator.py` (record_query through Rust core); #2448 comment id 33795 |
+| 9 | runtime evidence post-switch | post-switch verification wall ALL GREEN: Python canary 12/12, FFI smoke 41/41, Rust canary 27/27, parity harness ALL GREEN |
+| 10 | old production path unreachable | ALL import paths (`import query_memory`) now resolve to the deployed shim; `shared/graph_shard_bridge.py` tolerant import serves the shim; the orphan `src/rune/query_memory.py` had ZERO importers (verified by sweep incl. path-string references) |
+| 11 | old implementation removed or reference-only | `src/rune/query_memory.py` DELETED (this commit); the only stale reference (`scripts/structural_coverage_audit.py` registry mapping shared/ as a mirror of src/rune) corrected — query_memory removed from `SRC_A`/`SHARED_D`, added to `SHARED_B` (canonical, ops-harness owned) |
+| 12 | supersession ledger | THIS FILE |
+
+Deletion discipline (language-governance rule ⑥) satisfied: old code removed only after
+the new implementation was PROVEN — wall green, shadow 46/46, both production consumers
+observed live on the Rust core before deletion.
+
+## Consumer map at supersession
+
+- `~/.config/opencode/scripts/memory_store.py` — in-process, `adaptive_exploration_rate` (line 820).
+- `~/.config/opencode/scripts/growth_simulator.py` — in-process, `record_query`.
+- `shared/graph_shard_bridge.py` — tolerant import (try/except), follows the switch.
+- `src/rune/tool_executor.py` — NOT a consumer (nominal grep hit only; goes through `memory_store.cached_recall`).
+
+## Verification commands (reproducible)
+
+- `/home/nick/.venvs/memory/bin/python3 src/rune/canary_query_memory.py` → 12/12
+- `/home/nick/.venvs/memory/bin/python3 src/rust/query_memory/canary_rust.py` → 27/27
+- `/home/nick/.venvs/memory/bin/python3 src/rune/ffi_query_memory_smoke.py` → 41/41
+- `/home/nick/.venvs/memory/bin/python3 /tmp/phaseG_parity.py` → ALL GREEN
+- `/home/nick/.venvs/memory/bin/python3 scripts/structural_coverage_audit.py` → exit 0, coverage 100%
@@ -50,7 +50,7 @@ SRC_A = {
    "memory_store": "memory", "memory_selector": "memory",
    "memory_consolidation": "memory", "memory_env": "memory",
    "shard_manager": "memory", "tier_manager": "memory",
-    "graph_memory": "memory", "query_memory": "memory",
+    "graph_memory": "memory",
    "curator": "memory", "sleep_time": "memory",
    "subconscious_pass": "memory", "warm_neuron_store": "memory",
    "warm_router": "memory", "abstraction": "memory",
@@ -173,7 +173,6 @@ SHARED_D = {
    "drift_core": "src/rune/drift_core.py",
    "drift_observability": "src/rune/drift_observability.py",
    "gitea_auth": "scripts/gitea_auth.py",
-    "query_memory": "src/rune/query_memory.py",
    "shard_manager": "src/rune/shard_manager.py",
    "subconscious_pass": "src/rune/subconscious_pass.py",
    "tier_manager": "src/rune/tier_manager.py",
@@ -182,7 +181,8 @@ SHARED_B_OWNER = "ops-harness"
 SHARED_B = {"auto_detect", "autonomy_loop", "build_orchestrator",
            "build_pipeline_policy", "cognitive_loop", "custodian",
            "dream_cycle", "gitea_pipeline", "gitea_provision",
-            "graph_shard_bridge", "growth_simulator", "umbrella_update"}
+            "graph_shard_bridge", "growth_simulator", "query_memory",
+            "umbrella_update"}


 def main() -> int:
@@ -1,257 +0,0 @@
-#!/usr/bin/env python3
-"""query_memory.py — Query memory system for adaptive retrieval.
-
-The system remembers what kinds of queries it gets, what retrieval
-strategy worked, and adapts its gate logic accordingly.
-
-This prevents dead weight — memories get tested based on usage
-patterns, not just random exploration.
-
-Research basis:
- IGMiRAG (arXiv:2602.07525): Intuition-guided retrieval
- Task-Adaptive Retrieval (arXiv:2604.07863): Learning from interaction history
- MemCon (arXiv:2607.13591): Learned policy for consolidation/forgetting
- Agent-Native Memory (arXiv:2606.24775): Workload-adaptive routing
-
-Run under ~/.venvs/memory/bin/python3.
-"""
-
-import json
-import os
-import sys
-from collections import Counter, defaultdict
-from datetime import datetime, timezone
-from typing import Dict, List, Optional, Tuple
-
-QUERY_MEMORY_DIR = os.path.expanduser("~/.config/opencode/memory/query_memory")
-QUERY_HISTORY_FILE = os.path.join(QUERY_MEMORY_DIR, "history.jsonl")
-TOPIC_STATS_FILE = os.path.join(QUERY_MEMORY_DIR, "topic_stats.json")
-
-# Topic keywords for classification
-TOPIC_KEYWORDS = {
-    "guitar": ["guitar", "music", "teaching", "course", "lesson", "chord", "scale"],
-    "coding": ["code", "python", "script", "function", "debug", "build", "deploy"],
-    "memory": ["memory", "recall", "remember", "forget", "consolidat", "neuron"],
-    "persona": ["persona", "identity", "voice", "constitution", "character"],
-    "project": ["project", "task", "goal", "deadline", "milestone", "progress"],
-    "ai": ["ai", "model", "llm", "agent", "neural", "training", "inference"],
-    "personal": ["nick", "nottingham", "coffee", "coconut", "dog", "ttrpg"],
-    "system": ["system", "config", "install", "setup", "troubleshoot", "error"],
-    "research": ["research", "paper", "study", "analysis", "survey", "evidence"],
-    "creative": ["write", "design", "create", "build", "make", "art", "music"],
-}
-
-
-class QueryMemory:
-    """Query memory system for adaptive retrieval."""
-
-    def __init__(self):
-        os.makedirs(QUERY_MEMORY_DIR, exist_ok=True)
-        self.history: List[dict] = self._load_history()
-        self.topic_stats: Dict[str, dict] = self._load_topic_stats()
-
-    def _load_history(self) -> List[dict]:
-        """Load query history from disk."""
-        if not os.path.exists(QUERY_HISTORY_FILE):
-            return []
-        entries = []
-        with open(QUERY_HISTORY_FILE) as f:
-            for line in f:
-                try:
-                    entries.append(json.loads(line.strip()))
-                except json.JSONDecodeError:
-                    continue
-        return entries
-
-    def _save_history(self, entry: dict):
-        """Append a query entry to history."""
-        with open(QUERY_HISTORY_FILE, "a") as f:
-            f.write(json.dumps(entry) + "\n")
-
-    def _load_topic_stats(self) -> Dict[str, dict]:
-        """Load topic statistics from disk."""
-        if os.path.exists(TOPIC_STATS_FILE):
-            try:
-                with open(TOPIC_STATS_FILE) as f:
-                    return json.load(f)
-            except json.JSONDecodeError:
-                return {}
-        return {}
-
-    def _save_topic_stats(self):
-        """Save topic statistics to disk."""
-        with open(TOPIC_STATS_FILE, "w") as f:
-            json.dump(self.topic_stats, f, indent=2)
-
-    def classify_query(self, query: str) -> str:
-        """Classify a query into a topic.
-
-        Returns the topic with the most keyword matches.
-        Falls back to "general" if no strong match.
-        """
-        query_lower = query.lower()
-        scores = {}
-        for topic, keywords in TOPIC_KEYWORDS.items():
-            score = sum(1 for kw in keywords if kw in query_lower)
-            if score > 0:
-                scores[topic] = score
-
-        if not scores:
-            return "general"
-
-        return max(scores, key=scores.get)
-
-    def record_query(self, query: str, strategy: str, success: bool):
-        """Record a query and its outcome.
-
-        Args:
-            query: The query text
-            strategy: "consolidated" or "full_retrieval"
-            success: Whether the retrieval produced useful results
-        """
-        topic = self.classify_query(query)
-        entry = {
-            "query": query[:100],
-            "topic": topic,
-            "strategy": strategy,
-            "success": success,
-            "timestamp": datetime.now(timezone.utc).isoformat(),
-        }
-        self.history.append(entry)
-        self._save_history(entry)
-
-        # Update topic stats
-        if topic not in self.topic_stats:
-            self.topic_stats[topic] = {
-                "total_queries": 0,
-                "successful_queries": 0,
-                "consolidated_hits": 0,
-                "full_retrieval_hits": 0,
-                "exploration_rate": 0.30,  # default
-            }
-
-        stats = self.topic_stats[topic]
-        stats["total_queries"] += 1
-        if success:
-            stats["successful_queries"] += 1
-        if strategy == "consolidated":
-            stats["consolidated_hits"] += 1
-        else:
-            stats["full_retrieval_hits"] += 1
-
-        # Adaptive exploration rate: decrease for well-tested topics
-        if stats["total_queries"] >= 10:
-            success_rate = stats["successful_queries"] / stats["total_queries"]
-            if success_rate > 0.8:
-                stats["exploration_rate"] = max(0.10, stats["exploration_rate"] - 0.05)
-            elif success_rate < 0.5:
-                stats["exploration_rate"] = min(0.50, stats["exploration_rate"] + 0.05)
-
-        self._save_topic_stats()
-
-    def get_topic_stats(self) -> Dict[str, dict]:
-        """Get topic statistics."""
-        return self.topic_stats
-
-    def adaptive_exploration_rate(self, query: str) -> float:
-        """Get the exploration rate for a query based on its topic.
-
-        Well-tested topics have lower exploration rates.
-        Untested topics have higher exploration rates.
-        """
-        topic = self.classify_query(query)
-        if topic in self.topic_stats:
-            return self.topic_stats[topic].get("exploration_rate", 0.30)
-        return 0.50  # untested topics get high exploration
-
-    def get_dead_weight_topics(self) -> List[str]:
-        """Get topics with low proven uses (dead weight)."""
-        dead = []
-        for topic, stats in self.topic_stats.items():
-            if stats["total_queries"] < 5:
-                dead.append(topic)
-        return dead
-
-    def get_health_report(self) -> dict:
-        """Get a health report of the query memory system."""
-        total_queries = len(self.history)
-        topics = len(self.topic_stats)
-        dead_topics = len(self.get_dead_weight_topics())
-
-        return {
-            "total_queries": total_queries,
-            "topics_tracked": topics,
-            "dead_weight_topics": dead_topics,
-            "topic_stats": self.topic_stats,
-        }
-
-
-# ═══════════════════════════════════════════════════════════════════════════════
-# CLI
-# ═══════════════════════════════════════════════════════════════════════════════
-
-
-def main():
-    import argparse
-
-    parser = argparse.ArgumentParser(description="Query memory system")
-    sub = parser.add_subparsers(dest="cmd")
-
-    # record <query> --strategy <s> --success
-    p_rec = sub.add_parser("record", help="Record a query")
-    p_rec.add_argument("query", nargs="+")
-    p_rec.add_argument("--strategy", default="consolidated")
-    p_rec.add_argument("--success", action="store_true")
-
-    # stats
-    sub.add_parser("stats", help="Show topic statistics")
-
-    # rate <query>
-    p_rate = sub.add_parser("rate", help="Get exploration rate for a query")
-    p_rate.add_argument("query", nargs="+")
-
-    # health
-    sub.add_parser("health", help="Show health report")
-
-    args = parser.parse_args()
-
-    if not args.cmd:
-        parser.print_help()
-        sys.exit(1)
-
-    qm = QueryMemory()
-
-    if args.cmd == "record":
-        query = " ".join(args.query)
-        qm.record_query(query, args.strategy, args.success)
-        print(f"Recorded: {query[:50]} → {args.strategy} ({'success' if args.success else 'fail'})")
-
-    elif args.cmd == "stats":
-        stats = qm.get_topic_stats()
-        if stats:
-            print("Topic Statistics:")
-            for topic, s in stats.items():
-                print(
-                    f"  {topic}: {s['total_queries']} queries, {s['successful_queries']} success, rate={s['exploration_rate']:.2f}"
-                )
-        else:
-            print("No topic statistics yet.")
-
-    elif args.cmd == "rate":
-        query = " ".join(args.query)
-        rate = qm.adaptive_exploration_rate(query)
-        topic = qm.classify_query(query)
-        print(f"Query: {query[:50]}")
-        print(f"Topic: {topic}")
-        print(f"Exploration rate: {rate:.2f}")
-
-    elif args.cmd == "health":
-        report = qm.get_health_report()
-        print("Query Memory Health:")
-        print(f"  Total queries: {report['total_queries']}")
-        print(f"  Topics tracked: {report['topics_tracked']}")
-        print(f"  Dead weight topics: {report['dead_weight_topics']}")
-
-
-if __name__ == "__main__":
-    main()
@@ -1,6 +1,6 @@
 {
 "a_src_unrepresented": [],
- "architecturally_significant_unknown": 0,
+ "architecturally_significant_unknown": 16,
 "classification": {
  "scripts/__init__.py": {
   "class": "B",
@@ -12,6 +12,21 @@
   "note": "",
   "owner": "subsystem:governance"
  },
+  "scripts/await_intent.py": {
+   "class": "E",
+   "note": "await_intent.py \u2014 ASYNC CORRECTIVE PATCH: intent normalization at the",
+   "owner": ""
+  },
+  "scripts/await_intent_cli.py": {
+   "class": "E",
+   "note": "await_intent_cli.py \u2014 CLI surface for the intent normalization layer.",
+   "owner": ""
+  },
+  "scripts/bulk_issue_guard.py": {
+   "class": "E",
+   "note": "Bulk-issue guard: ceiling + discipline wrapper for mass Gitea issue posts.",
+   "owner": ""
+  },
  "scripts/capture_failure.py": {
   "class": "A",
   "note": "",
@@ -42,6 +57,11 @@
   "note": "",
   "owner": "subsystem:governance"
  },
+  "scripts/completion_file.py": {
+   "class": "E",
+   "note": "completion_file.py \u2014 REAL_ASYNC_BRIDGE: durable completion-file delivery",
+   "owner": ""
+  },
  "scripts/construction_guards.py": {
   "class": "A",
   "note": "",
@@ -97,6 +117,16 @@
   "note": "",
   "owner": "subsystem:governance"
  },
+  "scripts/install_mimo_tools.py": {
+   "class": "E",
+   "note": "install_mimo_tools.py \u2014 canonical MiMo tool installer (TTY-repair tranche).",
+   "owner": ""
+  },
+  "scripts/install_wr_daemon.py": {
+   "class": "E",
+   "note": "install_wr_daemon.py \u2014 REAL_ASYNC_BRIDGE \u00a76: daemon governance.",
+   "owner": ""
+  },
  "scripts/lint_ratchet.py": {
   "class": "A",
   "note": "",
@@ -132,6 +162,21 @@
   "note": "",
   "owner": "subsystem:governance"
  },
+  "scripts/r8_fault.py": {
+   "class": "E",
+   "note": "r8_fault.py \u2014 REAL_ASYNC_BRIDGE R8: operation-scoped lost-event fault",
+   "owner": ""
+  },
+  "scripts/rebase_inventory.py": {
+   "class": "E",
+   "note": "Tranche R1 Wave 0 \u2014 whole-project inventory generator.",
+   "owner": ""
+  },
+  "scripts/rebase_issue_factory.py": {
+   "class": "E",
+   "note": "Tranche R1 Wave 0 \u2014 per-function issue factory.",
+   "owner": ""
+  },
  "scripts/release_history_cli.py": {
   "class": "B",
   "note": "",
@@ -162,6 +207,11 @@
   "note": "",
   "owner": "subsystem:governance"
  },
+  "scripts/research_memory_port.py": {
+   "class": "E",
+   "note": "research_memory_port.py \u2014 T21.5 \u00a710-12: the future-facing Memory port",
+   "owner": ""
+  },
  "scripts/research_prompt_compiler.py": {
   "class": "A",
   "note": "",
@@ -217,6 +267,11 @@
   "note": "",
   "owner": "subsystem:governance"
  },
+  "scripts/start_wr_daemon.py": {
+   "class": "E",
+   "note": "start_wr_daemon.py \u2014 T21.5 \u00a77: THE canonical daemon restart launcher.",
+   "owner": ""
+  },
  "scripts/structural_coverage_audit.py": {
   "class": "A",
   "note": "",
@@ -247,11 +302,26 @@
   "note": "",
   "owner": "subsystem:governance"
  },
+  "scripts/v1_parity_manifest.py": {
+   "class": "E",
+   "note": "v1_parity_manifest.py \u2014 T21.5 \u00a713: historical Scaffolding V1 parity.",
+   "owner": ""
+  },
  "scripts/version_convergence.py": {
   "class": "A",
   "note": "",
   "owner": "subsystem:governance"
  },
+  "scripts/wr_root_reconcile.py": {
+   "class": "E",
+   "note": "wr_root_reconcile.py \u2014 T21.5 \u00a77: governed split-brain reconciliation.",
+   "owner": ""
+  },
+  "scripts/wr_root_resolver.py": {
+   "class": "E",
+   "note": "wr_root_resolver.py \u2014 T21.5 \u00a77: THE ONE wait/resume root authority.",
+   "owner": ""
+  },
  "shared/audit_ledger.py": {
   "class": "D",
   "note": "harness-facing variant; canonical copy supersedes",
@@ -342,10 +412,15 @@
   "note": "",
   "owner": "ops-harness"
  },
+  "shared/native_units.py": {
+   "class": "E",
+   "note": "Thin adapter for native (compiled) units \u2014 R1 \u00a75 interop contract, Wave 0.",
+   "owner": ""
+  },
  "shared/query_memory.py": {
-   "class": "D",
-   "note": "harness-facing variant; canonical copy supersedes",
-   "owner": "src/rune/query_memory.py"
+   "class": "B",
+   "note": "",
+   "owner": "ops-harness"
  },
  "shared/shard_manager.py": {
   "class": "D",
@@ -742,6 +817,11 @@
   "note": "",
   "owner": "subsystem:governance"
  },
+  "src/rune/ffi_query_memory_smoke.py": {
+   "class": "E",
+   "note": "FFI smoke: prove the deployed consumer shape works against the Rust cdylib.",
+   "owner": ""
+  },
  "src/rune/forge.py": {
   "class": "A",
   "note": "",
@@ -942,11 +1022,6 @@
   "note": "",
   "owner": "subsystem:inference"
  },
-  "src/rune/query_memory.py": {
-   "class": "A",
-   "note": "",
-   "owner": "subsystem:memory"
-  },
  "src/rune/recall_router.py": {
   "class": "A",
   "note": "",
@@ -1189,11 +1264,11 @@
  }
 },
 "counts": {
-  "A": 157,
-  "B": 41,
+  "A": 156,
+  "B": 42,
  "C": 28,
-  "D": 11,
-  "E": 0
+  "D": 10,
+  "E": 16
 },
 "critical_areas": {
  "adapt_routing": "modeled (adaptive_router A + subsystem:inference membership)",
@@ -1226,14 +1301,31 @@
 "critical_interface_gaps": [],
 "critical_runtime_path_gaps": [],
 "critical_state_owner_gaps": [],
- "generated_at": "2026-09-17T00:27:25Z",
- "implementation_details": 41,
- "implementation_details_owned": 41,
+ "generated_at": "2026-10-02T19:51:41Z",
+ "implementation_details": 42,
+ "implementation_details_owned": 42,
 "implementation_ownership_percent": 100.0,
 "meaningful_coverage_percent": 100.0,
- "meaningful_units_represented": 113,
- "meaningful_units_src_rune": 113,
- "total_source_modules": 237,
+ "meaningful_units_represented": 112,
+ "meaningful_units_src_rune": 112,
+ "total_source_modules": 252,
 "tranche": "T21.4",
- "unknown_detail": []
+ "unknown_detail": [
+  "src/rune/ffi_query_memory_smoke.py :: FFI smoke: prove the deployed consumer shape works against the Rust cdylib.",
+  "scripts/await_intent.py :: await_intent.py \u2014 ASYNC CORRECTIVE PATCH: intent normalization at the",
+  "scripts/await_intent_cli.py :: await_intent_cli.py \u2014 CLI surface for the intent normalization layer.",
+  "scripts/bulk_issue_guard.py :: Bulk-issue guard: ceiling + discipline wrapper for mass Gitea issue posts.",
+  "scripts/completion_file.py :: completion_file.py \u2014 REAL_ASYNC_BRIDGE: durable completion-file delivery",
+  "scripts/install_mimo_tools.py :: install_mimo_tools.py \u2014 canonical MiMo tool installer (TTY-repair tranche).",
+  "scripts/install_wr_daemon.py :: install_wr_daemon.py \u2014 REAL_ASYNC_BRIDGE \u00a76: daemon governance.",
+  "scripts/r8_fault.py :: r8_fault.py \u2014 REAL_ASYNC_BRIDGE R8: operation-scoped lost-event fault",
+  "scripts/rebase_inventory.py :: Tranche R1 Wave 0 \u2014 whole-project inventory generator.",
+  "scripts/rebase_issue_factory.py :: Tranche R1 Wave 0 \u2014 per-function issue factory.",
+  "scripts/research_memory_port.py :: research_memory_port.py \u2014 T21.5 \u00a710-12: the future-facing Memory port",
+  "scripts/start_wr_daemon.py :: start_wr_daemon.py \u2014 T21.5 \u00a77: THE canonical daemon restart launcher.",
+  "scripts/v1_parity_manifest.py :: v1_parity_manifest.py \u2014 T21.5 \u00a713: historical Scaffolding V1 parity.",
+  "scripts/wr_root_reconcile.py :: wr_root_reconcile.py \u2014 T21.5 \u00a77: governed split-brain reconciliation.",
+  "scripts/wr_root_resolver.py :: wr_root_resolver.py \u2014 T21.5 \u00a77: THE ONE wait/resume root authority.",
+  "shared/native_units.py :: Thin adapter for native (compiled) units \u2014 R1 \u00a75 interop contract, Wave 0."
+ ]
 }