feat(memory_store): close retrieval-family audit gaps — dichotomy wave
RETRIEVAL/RANKING family lane (builder factory-heavy-ms-rtrv): 12 name-preserving kernel ports into the memory-store-deep Go corpus in a NEW family file gaps_retrieval.go — cached_recall (CachedRecall: strip/cache-hit/ miss/rerank/put-signal seam orchestration), mr_rank_fusion (cap'd RRF), _fts_delete (contentless-FTS5 'delete' statement planner), chunk_recall (cos + min_sim abstain + 0.5-scored lexical fallback + wing/doc row filter + ordered projection), fused_recall (authoritative fuse — see divergence 1), recall_with_associations (>=0.3 strong-link walk, first-touch boost order), _merge_association_rows (Rust-authority blend, banker's rounding), curiosity_probe (>3-CHAR word extraction + wash side-signal), tier_demotion_sweep (+TierGet ladder: rock/wash20/recency, naive-ts->warm), _mr_core (env pin mirror), _df_word/_df (ASCII-fold LIKE count). SQLite row-fetch, embedding, cache/shard/drift IO stay facade-side per the landed crate's seams. Truth: 43-row truth_retrieval_gaps.json via the shared freeze_truth harness (pure kernels) + fixture-store rigs (_embed off, fake cache/shard recorders, frozen clocks on memory_store+tier_manager, drift_core._now_iso pinned, RUNE_COGNITION_CORE=python, memory_retrieval+memory_scoring built in-tree — live Rust delegation authority). Byte-identical across 5 regenerations. Battery gaps_retrieval_test.go replays canonically (GAPS_TRUTH_PATH override for live parity); live parity 43/43 via scripts/check_memory_store_retrieval_parity.py. gofmt/vet/build/test clean; Python suite failure sets BYTE-IDENTICAL to stashed base in BOTH pins (RUNE_RETRIEVAL_CORE live + python). ruff clean. Divergences (documented in verification/msr-summary.md): (1) landed sibling FuseResults substitutes 'fact'/'research' for EMPTY topics where the Rust authority keeps '' — FusedRecall implements the authority in-family, sibling left untouched, follow-up recommended; (2) tier_demotion_sweep structurally never demotes (demotion_sweep IS get_tier) — ported honestly, pinned; (3) curiosity self-match is the norm; (4) put-value serialization stays facade-side. Audit: memory_store 30/68 -> 44/68; retrieval family missing list now EMPTY (_embed/linked_entries counters also flip on prose mentions — NOT ports, reported as such).
This commit is contained in:
Nick committed
1 parent
07c08c341e
commit
62e29e2e46
18 files changed
+3493
No files matched your search
@@ -0,0 +1,64 @@
|
||||
#!/usr/bin/env python3
|
||||
"""check_memory_store_retrieval_parity.py — live parity for the memory_store
|
||||
RETRIEVAL family gap port (builder factory-heavy-ms-rtrv).
|
||||
|
||||
Regenerates the truth corpus from the LIVE oracle (all rigs active) into a
|
||||
temp file, byte-compares it against the committed frozen truth (drift gate),
|
||||
then replays the FRESH corpus through the Go battery via GAPS_TRUTH_PATH.
|
||||
Green = the Go kernels in src/go/memory-store-deep/gaps_retrieval.go agree
|
||||
with the live Python oracle on every row, in both directions.
|
||||
|
||||
Usage: python3 scripts/check_memory_store_retrieval_parity.py
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
GEN = REPO / "scripts/gen_memory_store_retrieval_truth.py"
|
||||
TRUTH = REPO / "src/go/memory-store-deep/truth_retrieval_gaps.json"
|
||||
GO_DIR = REPO / "src/go/memory-store-deep"
|
||||
|
||||
|
||||
def run(cmd, **kw):
|
||||
print("+", " ".join(str(c) for c in cmd))
|
||||
return subprocess.run(cmd, capture_output=True, text=True, **kw)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
tmp = Path(tempfile.mkdtemp(prefix="ms_rtrv_parity_")) / "truth_fresh.json"
|
||||
|
||||
r = run([sys.executable, str(GEN), "--out", str(tmp)])
|
||||
print(r.stdout.strip() or r.stderr.strip())
|
||||
if r.returncode != 0:
|
||||
print("FAIL: generator crashed")
|
||||
return 1
|
||||
|
||||
# drift gate: fresh capture must equal the committed frozen truth
|
||||
r = run(["diff", str(TRUTH), str(tmp)])
|
||||
if r.returncode != 0:
|
||||
print("FAIL: live oracle drifted from frozen truth:")
|
||||
print(r.stdout[:2000])
|
||||
return 1
|
||||
print("frozen truth == fresh live capture (byte-identical)")
|
||||
|
||||
# battery replay against the FRESH corpus
|
||||
import os
|
||||
env = dict(os.environ, GAPS_TRUTH_PATH=str(tmp))
|
||||
r = run(["go", "test", "-run", "TestRetrievalGapsParity", "-v", "./..."],
|
||||
cwd=GO_DIR, env=env)
|
||||
out = r.stdout + r.stderr
|
||||
print(out[-1500:])
|
||||
if r.returncode != 0:
|
||||
print("FAIL: Go battery diverges from live-oracle truth")
|
||||
return 1
|
||||
|
||||
n = TRUTH.read_text().count('"fn"')
|
||||
print(f"PARITY OK: {n} frozen rows byte-stable + Go battery green on fresh truth")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,441 @@
|
||||
#!/usr/bin/env python3
|
||||
"""gen_memory_store_retrieval_truth.py — memory_store RETRIEVAL family truth
|
||||
battery (dichotomy wave #2479, builder factory-heavy-ms-rtrv).
|
||||
|
||||
Freezes the retrieval/ranking gap family against the LIVE oracle
|
||||
(src/rune/memory_store.py). Pure kernels (mr_rank_fusion,
|
||||
_merge_association_rows) ride the shared freeze_truth harness directly;
|
||||
db-bound fns run on purpose-built fixture stores with every seam pinned by
|
||||
rigs — _embed off (no network), fake cache/shard_manager recorders (sys.modules
|
||||
injection), frozen clocks (freeze_truth._frozen_clock patched onto
|
||||
memory_store + tier_manager, drift_core._now_iso pinned), RUNE_COGNITION_CORE=
|
||||
python (tier oracle body), and the memory_retrieval + memory_scoring kernels
|
||||
BUILT (live delegation — the post-2026-10-10 authority). The facade-fetched
|
||||
row inputs are captured into each case so the Go battery (gaps_retrieval_
|
||||
test.go) replays them standalone; the SQLite row-fetch stays facade-side.
|
||||
|
||||
Usage:
|
||||
python3 scripts/gen_memory_store_retrieval_truth.py # write truth
|
||||
python3 scripts/gen_memory_store_retrieval_truth.py --check # byte-compare
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(REPO / "scripts"))
|
||||
sys.path.insert(0, str(REPO / "src"))
|
||||
sys.path.insert(0, str(REPO / "src" / "rune"))
|
||||
|
||||
RIG_HOME = tempfile.mkdtemp(prefix="ms_rtrv_rig_")
|
||||
os.environ["HOME"] = RIG_HOME # import-time path derivation pin
|
||||
os.environ.setdefault("RUNE_COGNITION_CORE", "python") # tier oracle body
|
||||
FROZEN_NOW = "2026-10-14T12:00:00" # naive base; .now(tz) honors tz
|
||||
FROZEN_NOW_AWARE = "2026-10-14T12:00:00+00:00"
|
||||
|
||||
# ── seam rigs: fake cache + fake shard routing (deterministic recorders) ─────
|
||||
|
||||
|
||||
class _FakeCache:
|
||||
def __init__(self):
|
||||
self.scripted = []
|
||||
self.puts = []
|
||||
|
||||
def get_cache(self, key):
|
||||
if self.scripted:
|
||||
return self.scripted.pop(0)
|
||||
return {"hit": False, "key": key}
|
||||
|
||||
def put_cache(self, key, value, compressed=None, scope="general", source_hash=""):
|
||||
self.puts.append({"key": key, "value": value, "scope": scope})
|
||||
|
||||
|
||||
class _FakeShardManager:
|
||||
def __init__(self):
|
||||
self.routes = []
|
||||
|
||||
def route_query(self, query):
|
||||
self.routes.append(query)
|
||||
return "main"
|
||||
|
||||
def get_shard(self, topic):
|
||||
return None # fall back to the main db (the oracle's else-branch)
|
||||
|
||||
|
||||
import types # noqa: E402
|
||||
|
||||
_fake_cache = _FakeCache()
|
||||
_fake_shards = _FakeShardManager()
|
||||
sys.modules["cache"] = types.ModuleType("cache")
|
||||
sys.modules["cache"].get_cache = _fake_cache.get_cache
|
||||
sys.modules["cache"].put_cache = _fake_cache.put_cache
|
||||
sys.modules["shard_manager"] = types.ModuleType("shard_manager")
|
||||
sys.modules["shard_manager"].ShardManager = lambda: _fake_shards
|
||||
|
||||
import sqlite3 # noqa: E402
|
||||
|
||||
from freeze_truth import _frozen_clock # noqa: E402 — shared harness rig helper
|
||||
|
||||
import memory_store as ms # noqa: E402
|
||||
import tier_manager as tm_mod # noqa: E402
|
||||
import drift_core # noqa: E402
|
||||
|
||||
_ms_frozen = _frozen_clock(FROZEN_NOW)
|
||||
ms.datetime = _ms_frozen
|
||||
tm_mod.datetime = _ms_frozen
|
||||
drift_core._now_iso = lambda now=None: FROZEN_NOW_AWARE
|
||||
ms._embed = lambda texts: {} # no network: qvec None → dense side stays off
|
||||
|
||||
TM = tm_mod.TierManager()
|
||||
|
||||
ENTRIES_DDL = """
|
||||
CREATE TABLE entries (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
text TEXT NOT NULL,
|
||||
importance REAL DEFAULT 0.5,
|
||||
created_at TEXT NOT NULL,
|
||||
last_accessed TEXT,
|
||||
topic TEXT DEFAULT '',
|
||||
entities TEXT DEFAULT '[]',
|
||||
source TEXT DEFAULT '',
|
||||
method TEXT DEFAULT 'curator',
|
||||
status TEXT DEFAULT 'active',
|
||||
confidence REAL DEFAULT 0.7,
|
||||
temperature REAL DEFAULT 1.0,
|
||||
wash_count INTEGER DEFAULT 0,
|
||||
rock_status TEXT DEFAULT 'washable',
|
||||
memory_type TEXT,
|
||||
summary TEXT DEFAULT '',
|
||||
promotion_eligible INTEGER DEFAULT 0,
|
||||
last_washed TEXT,
|
||||
tier TEXT DEFAULT 'warm'
|
||||
);
|
||||
CREATE VIRTUAL TABLE entries_fts USING fts5(text, content='', tokenize='porter unicode61');
|
||||
CREATE TABLE associations (
|
||||
src_id INTEGER NOT NULL, dst_id INTEGER NOT NULL,
|
||||
strength REAL DEFAULT 0.1, updated_at TEXT,
|
||||
PRIMARY KEY (src_id, dst_id)
|
||||
);
|
||||
CREATE TABLE chunks (
|
||||
chunk_id TEXT PRIMARY KEY, doc_id TEXT NOT NULL,
|
||||
wing TEXT DEFAULT '', room TEXT DEFAULT '',
|
||||
source_path TEXT, start_line INTEGER, end_line INTEGER, page INTEGER,
|
||||
text TEXT NOT NULL, content_hash TEXT NOT NULL,
|
||||
importance REAL DEFAULT 0.5, ingested_at TEXT NOT NULL
|
||||
);
|
||||
CREATE VIRTUAL TABLE chunks_fts USING fts5(text, content='', tokenize='porter unicode61');
|
||||
"""
|
||||
|
||||
COMBO_ENTRIES = [
|
||||
# id, text, importance, conf, temp, last_accessed, wash, rock
|
||||
(1, "nick prefers black coffee in the morning", 0.9, 0.9, 1.0, "2026-10-12T00:00:00+00:00", 5, "washable"),
|
||||
(2, "the curator gates memory writes behind provenance", 0.8, 0.8, 1.0, "2026-09-01T00:00:00+00:00", 2, "washable"),
|
||||
(3, "black holes bend spacetime near the horizon", 0.6, 0.6, 0.9, "2026-08-01T00:00:00+00:00", 1, "washable"),
|
||||
(4, "rune rebuilds the spine every night at three", 0.7, 0.75, 1.1, "2026-10-14T00:00:00+00:00", 3, "washable"),
|
||||
(5, "coffee grinding rituals across the highlands", 0.55, 0.7, 1.0, "2026-07-01T00:00:00+00:00", 0, "washable"),
|
||||
]
|
||||
COMBO_ASSOC = [(1, 5, 0.45), (2, 5, 0.31), (1, 3, 0.2)] # (src, dst, strength)
|
||||
COMBO_CHUNKS = [
|
||||
("c1", "stellar-winds", "alpha", "notes/stellar.md", 3,
|
||||
"stellar winds carry plasma away from the corona in bursts"),
|
||||
("c2", "curator-docs", "beta", "notes/curator.md", 7,
|
||||
"curator provenance recovery notes for the audit chain"),
|
||||
("c3", "stellar-winds", "alpha", "notes/cartography.md", None,
|
||||
"stellar cartography maps the outer arm of the galaxy"),
|
||||
]
|
||||
|
||||
|
||||
def combo_db():
|
||||
db = sqlite3.connect(":memory:")
|
||||
db.row_factory = sqlite3.Row
|
||||
db.executescript(ENTRIES_DDL)
|
||||
for (eid, text, imp, conf, temp, la, wash, rock) in COMBO_ENTRIES:
|
||||
db.execute(
|
||||
"INSERT INTO entries (id, text, importance, created_at, last_accessed,"
|
||||
" confidence, temperature, wash_count, rock_status) VALUES (?,?,?,?,?,?,?,?,?)",
|
||||
(eid, text, imp, "2026-01-01T00:00:00+00:00", la, conf, temp, wash, rock),
|
||||
)
|
||||
db.execute("INSERT INTO entries_fts (rowid, text) VALUES (?,?)", (eid, text))
|
||||
for (src, dst, s) in COMBO_ASSOC:
|
||||
db.execute("INSERT INTO associations (src_id, dst_id, strength, updated_at) VALUES (?,?,?,?)",
|
||||
(src, dst, s, "2026-01-01T00:00:00+00:00"))
|
||||
for (cid, doc, wing, src, page, text) in COMBO_CHUNKS:
|
||||
cur = db.execute(
|
||||
"INSERT INTO chunks (chunk_id, doc_id, wing, room, source_path, page, text,"
|
||||
" content_hash, importance, ingested_at) VALUES (?,?,?,?,?,?,?,?,?,?)",
|
||||
(cid, doc, wing, "room1", src, page, text, "h" * 8, 0.5, "2026-01-01T00:00:00+00:00"),
|
||||
)
|
||||
rid = cur.lastrowid
|
||||
db.execute("INSERT INTO chunks_fts (rowid, text) VALUES (?,?)", (rid, text))
|
||||
db.commit()
|
||||
return db
|
||||
|
||||
|
||||
def rows_of(db, sql, params=()):
|
||||
return [dict(r) for r in db.execute(sql, params).fetchall()]
|
||||
|
||||
|
||||
def main():
|
||||
cases = []
|
||||
|
||||
def add(fn, name, inp, expected):
|
||||
cases.append({"fn": fn, "name": name, "input": inp, "expected": expected})
|
||||
|
||||
# ── pure kernels via the shared harness (live delegation authority) ─────
|
||||
from freeze_truth import freeze
|
||||
|
||||
pure = freeze(
|
||||
"rune.memory_store",
|
||||
[
|
||||
("mr_rank_fusion", ([[9, 0.85], [2, 0.6]], [[7, 3.2], [9, 1.1], [4, 0.5]], 2), {}),
|
||||
("mr_rank_fusion", ([[9, 0.85]], [], 6), {}),
|
||||
("mr_rank_fusion", ([], [[7, 3.2], [9, 1.1], [4, 0.5]], 18), {}),
|
||||
("mr_rank_fusion", ([[5, 0.9], [6, 0.8]], [[5, 2.0]], 0), {}),
|
||||
("_merge_association_rows",
|
||||
([{"id": 1, "text": "base fact", "relevance": 0.8}],
|
||||
[{"id": 5, "strength": 0.45}, {"id": 9, "strength": 0.3}],
|
||||
[{"id": 5, "text": "boosted fact", "confidence": 0.9, "temperature": 1.0, "importance": 0.6},
|
||||
{"id": 9, "text": "zero-conf boost", "confidence": 0, "temperature": 0, "importance": 0.4}],
|
||||
4), {}),
|
||||
("_merge_association_rows",
|
||||
([{"id": 1, "relevance": 0.6}],
|
||||
[{"id": 2, "strength": 0.5}, {"id": 3, "strength": 0.4}],
|
||||
[{"id": 2, "confidence": 0.8, "temperature": 1.2},
|
||||
{"id": 3, "confidence": 0.95, "temperature": 0.85}],
|
||||
2), {}),
|
||||
("_merge_association_rows", ([], [{"id": 8, "strength": 0.9}], [{"id": 9}], 3), {}),
|
||||
],
|
||||
oracle_module="rune.memory_store",
|
||||
)
|
||||
for row in pure:
|
||||
nm = row["fn"] + "_" + str(len(cases))
|
||||
if row["fn"] == "mr_rank_fusion":
|
||||
dense, bm25, cap = row["args"]
|
||||
add("mr_rank_fusion", nm, {"dense": dense, "bm25": bm25, "cap": cap}, row["expected"])
|
||||
else:
|
||||
base, boosts, rws, budget = row["args"]
|
||||
add("merge_association_rows", nm,
|
||||
{"base": base, "boosts": boosts, "rows": rws, "budget": budget}, row["expected"])
|
||||
|
||||
# ── df / df_word — the LIKE-count kernels (ASCII case-fold) ─────────────
|
||||
db = combo_db()
|
||||
texts = [r["text"] for r in rows_of(db, "SELECT text FROM entries WHERE status='active'")]
|
||||
for word in ["coffee", "COFFEE", "black", "größere", "zzz"]:
|
||||
add("df", "df_" + word, {"texts": texts, "word": word},
|
||||
{"n": ms._df_word(db, word)})
|
||||
add("df_word", "df_word_alias", {"texts": texts, "word": "coffee"},
|
||||
{"n": ms._df_word(db, word="coffee")})
|
||||
|
||||
# ── chunk_recall e2e (chunks side of the combo store) ───────────────────
|
||||
for name, query, budget, wing, doc, min_sim in [
|
||||
("conjunctive_hit", "stellar winds", 6, None, None, 0.0),
|
||||
("wing_filter", "stellar", 6, "alpha", None, 0.0),
|
||||
("wing_excludes", "stellar", 6, "beta", None, 0.0),
|
||||
("doc_filter", "curator provenance", 6, None, "curator-docs", 0.0),
|
||||
("min_sim_abstain", "stellar", 6, None, None, 0.9),
|
||||
("no_match", "quantum entanglement", 6, None, None, 0.0),
|
||||
]:
|
||||
got = ms.chunk_recall(db, query, budget=budget, wing=wing, min_sim=min_sim, doc=doc)
|
||||
fts_rowids = [r["rowid"] for r in db.execute(
|
||||
"SELECT rowid FROM chunks_fts WHERE chunks_fts MATCH ? ORDER BY bm25(chunks_fts) LIMIT ?",
|
||||
(ms._fts_query(query), budget * 3)).fetchall()]
|
||||
chunk_rows = rows_of(db, "SELECT rowid, * FROM chunks")
|
||||
add("chunk_recall", name,
|
||||
{"has_qvec": False, "fts_rowids": fts_rowids, "chunk_rows": chunk_rows,
|
||||
"budget": budget, "wing": wing or "", "doc": doc or "", "min_sim": min_sim},
|
||||
got)
|
||||
|
||||
# ── recall_with_associations e2e ─────────────────────────────────────────
|
||||
for name, query, hops in [
|
||||
("strong_link_merged", "black coffee", 1),
|
||||
("hops_zero", "black coffee", 0),
|
||||
("weak_links_only", "spacetime horizon", 1),
|
||||
("empty_base", "quantum entanglement", 1),
|
||||
]:
|
||||
got = ms.recall_with_associations(db, query, budget=8, min_score=0.42, hops=hops, assoc_limit=4)
|
||||
base = ms.recall(db, query, budget=8, min_score=0.42)
|
||||
links = [ms.linked_entries(db, b["id"], limit=4) for b in base]
|
||||
entry_rows = rows_of(db, "SELECT * FROM entries WHERE status='active'")
|
||||
add("recall_with_associations", name,
|
||||
{"base": base, "links": links, "entry_rows": entry_rows,
|
||||
"budget": 8, "hops": hops, "assoc_limit": 4},
|
||||
got)
|
||||
|
||||
# ── fused_recall e2e (facts + research in one store) ─────────────────────
|
||||
for name, query in [("dedup_kinds", "black coffee"), ("research_only", "stellar cartography")]:
|
||||
got = ms.fused_recall(db, query, budget=8, min_score=0.42, chunk_budget=6)
|
||||
entries_side = ms.recall_with_associations(db, query, budget=8, min_score=0.42)
|
||||
research_side = ms.chunk_recall(db, query, budget=6)
|
||||
add("fused_recall", name,
|
||||
{"entries": entries_side, "research": research_side}, got)
|
||||
|
||||
# ── cached_recall e2e (fake cache + shard recorders) ─────────────────────
|
||||
# case 1: miss → recall + drift rerank + put
|
||||
_fake_cache.scripted = [{"hit": False, "key": "recall:black coffee"}]
|
||||
_fake_cache.puts = []
|
||||
got = ms.cached_recall(db, "black coffee")
|
||||
pre = ms.recall(db, "black coffee", budget=8, min_score=0.42)
|
||||
drift = []
|
||||
for e in pre:
|
||||
row = db.execute("SELECT wash_count, importance, last_accessed, rock_status FROM entries WHERE id = ?",
|
||||
(e.get("id"),)).fetchone()
|
||||
dv = drift_core.compute_value({
|
||||
"wash_count": row[0] or 0, "importance": row[1] or 0.5,
|
||||
"last_accessed": row[2], "rock_status": row[3] or "washable"}) if row else 0.0
|
||||
drift.append([e.get("id"), e.get("relevance", e.get("score", 0.5)), dv])
|
||||
put = _fake_cache.puts[0] if _fake_cache.puts else None
|
||||
assert put and put["value"] == json.dumps(got, ensure_ascii=False), "put payload self-check"
|
||||
add("cached_recall", "miss_rerank_put",
|
||||
{"query": "black coffee", "cached": {"hit": False, "compressed": ""},
|
||||
"recall_out": pre, "drift": drift},
|
||||
{"result": got, "put_key": put["key"], "put_scope": put["scope"]})
|
||||
|
||||
# case 2: hit → parsed payload, no put, no recall
|
||||
payload = [{"id": 1, "text": "cached fact", "relevance": 0.55}]
|
||||
_fake_cache.scripted = [{"hit": True, "key": "recall:black coffee",
|
||||
"compressed": json.dumps(payload)}]
|
||||
_fake_cache.puts = []
|
||||
got = ms.cached_recall(db, "black coffee")
|
||||
assert not _fake_cache.puts
|
||||
add("cached_recall", "hit_returns_payload",
|
||||
{"query": "black coffee", "cached": {"hit": True, "compressed": json.dumps(payload)},
|
||||
"recall_out": [], "drift": []},
|
||||
{"result": got, "put_key": None, "put_scope": None})
|
||||
|
||||
# case 3: empty query → no cache interaction
|
||||
_fake_cache.scripted = []
|
||||
_fake_cache.puts = []
|
||||
got = ms.cached_recall(db, " ")
|
||||
assert not _fake_cache.puts
|
||||
add("cached_recall", "empty_query",
|
||||
{"query": " ", "cached": {"hit": False, "compressed": ""},
|
||||
"recall_out": [], "drift": []},
|
||||
{"result": got, "put_key": None, "put_scope": None})
|
||||
|
||||
# ── curiosity_probe e2e (exactly ONE dead-weight row per store) ──────────
|
||||
def probe_db(entry_text, other_wash=3):
|
||||
pdb = sqlite3.connect(":memory:")
|
||||
pdb.row_factory = sqlite3.Row
|
||||
pdb.executescript(ENTRIES_DDL)
|
||||
pdb.execute(
|
||||
"INSERT INTO entries (id, text, importance, created_at, last_accessed, confidence,"
|
||||
" temperature, wash_count) VALUES (?,?,?,?,?,?,?,0)",
|
||||
(1, entry_text, 0.6, "2026-01-01T00:00:00+00:00", "2026-10-01T00:00:00+00:00", 0.8, 1.0))
|
||||
pdb.execute("INSERT INTO entries_fts (rowid, text) VALUES (1, ?)", (entry_text,))
|
||||
pdb.execute(
|
||||
"INSERT INTO entries (id, text, importance, created_at, last_accessed, confidence,"
|
||||
" temperature, wash_count) VALUES (2,'settle unrelated filler text',0.5,"
|
||||
" '2026-01-01T00:00:00+00:00','2026-10-01T00:00:00+00:00',0.7,1.0,?)", (other_wash,))
|
||||
pdb.execute("INSERT INTO entries_fts (rowid, text) VALUES (2, 'settle unrelated filler text')")
|
||||
pdb.commit()
|
||||
return pdb
|
||||
|
||||
pdb = probe_db("quantum tunneling explains stellar fusion")
|
||||
_fake_cache.scripted = []
|
||||
dead = [dict(r) for r in pdb.execute("SELECT id, text FROM entries WHERE wash_count = 0").fetchall()]
|
||||
pre_wash = {r["id"]: (r["wash_count"] or 0) for r in pdb.execute("SELECT id, wash_count FROM entries").fetchall()}
|
||||
got = ms.curiosity_probe(pdb, limit=5)
|
||||
post_wash = {r["id"]: (r["wash_count"] or 0) for r in pdb.execute("SELECT id, wash_count FROM entries").fetchall()}
|
||||
washed_ids = [i for i in pre_wash if post_wash[i] > pre_wash[i]]
|
||||
# the recall results the probe's cached_recall produced for its query
|
||||
probe_query = got[0]["query"] if got else ""
|
||||
probe_results = ms.recall(pdb, probe_query) if probe_query else []
|
||||
add("curiosity_probe", "found_washes",
|
||||
{"rows": [[d["id"], d["text"]] for d in dead], "recall_results": [probe_results]},
|
||||
{"probed": got, "washed": washed_ids})
|
||||
|
||||
# no >3-char words → the row is skipped entirely (no query, no wash)
|
||||
pdb2 = probe_db("a an of to be")
|
||||
_fake_cache.scripted = []
|
||||
dead2 = [dict(r) for r in pdb2.execute("SELECT id, text FROM entries WHERE wash_count = 0").fetchall()]
|
||||
got = ms.curiosity_probe(pdb2, limit=5)
|
||||
add("curiosity_probe", "no_words_skipped",
|
||||
{"rows": [[d["id"], d["text"]] for d in dead2], "recall_results": []},
|
||||
{"probed": got, "washed": []})
|
||||
|
||||
# ── tier family (frozen clock, python oracle body) ───────────────────────
|
||||
tier_rows = [
|
||||
{"wash_count": 3, "last_accessed": "2026-10-13T00:00:00+00:00", "rock_status": "washable"}, # hot
|
||||
{"wash_count": 3, "last_accessed": "2026-10-01T00:00:00+00:00", "rock_status": "washable"}, # warm
|
||||
{"wash_count": 3, "last_accessed": "2026-08-01T00:00:00+00:00", "rock_status": "washable"}, # cold
|
||||
{"wash_count": 25, "last_accessed": "2026-01-01T00:00:00+00:00", "rock_status": "washable"}, # hot via wash
|
||||
{"wash_count": 1, "last_accessed": "2026-08-01T00:00:00+00:00", "rock_status": "rock"}, # rock
|
||||
{"wash_count": 1, "last_accessed": "2026-10-13T00:00:00", "rock_status": "washable"}, # naive → warm
|
||||
{"wash_count": 1, "last_accessed": "not-a-date", "rock_status": "washable"}, # bad → warm
|
||||
{"wash_count": 1, "rock_status": "washable"}, # absent → warm
|
||||
]
|
||||
for i, entry in enumerate(tier_rows):
|
||||
add("tier_get", f"tier_{i}", {"entry": entry, "now": FROZEN_NOW_AWARE},
|
||||
{"tier": TM.get_tier(entry)})
|
||||
sweep_rows = rows_of(db, "SELECT id, wash_count, last_accessed, rock_status FROM entries WHERE status='active'")
|
||||
add("tier_demotion_sweep", "sweep_never_demotes",
|
||||
{"rows": sweep_rows, "now": FROZEN_NOW_AWARE},
|
||||
ms.tier_demotion_sweep(db))
|
||||
|
||||
# ── fts_delete (statement recorder over the entries fetch) ───────────────
|
||||
class Recorder:
|
||||
def __init__(self, inner):
|
||||
self._inner = inner
|
||||
self.stmts = []
|
||||
|
||||
def execute(self, sql, params=()):
|
||||
if "entries_fts(entries_fts" in sql:
|
||||
self.stmts.append({"sql": sql, "args": list(params)})
|
||||
return self._inner.execute(sql, params)
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(self._inner, name)
|
||||
|
||||
for name, rows in [
|
||||
("delete_with_text", [(11, "göß text to remove")]),
|
||||
("empty_text_skipped", [(12, "")]),
|
||||
("missing_row_skipped", [(13, None)]),
|
||||
]:
|
||||
fdb = sqlite3.connect(":memory:")
|
||||
fdb.row_factory = sqlite3.Row
|
||||
fdb.executescript(ENTRIES_DDL)
|
||||
for (eid, text) in rows:
|
||||
if text is not None:
|
||||
fdb.execute("INSERT INTO entries (id, text, created_at) VALUES (?,?,?)", (eid, text, "2026-01-01T00:00:00+00:00"))
|
||||
fdb.commit()
|
||||
rec = Recorder(fdb)
|
||||
ms._fts_delete(rec, rows[0][0])
|
||||
old = fdb.execute("SELECT text FROM entries WHERE id=?", (rows[0][0],)).fetchone()
|
||||
add("fts_delete", name,
|
||||
{"eid": rows[0][0], "old_row": dict(old) if old else None},
|
||||
rec.stmts)
|
||||
|
||||
# ── mr_core probe (env-derived; battery asserts both branches directly) ──
|
||||
add("mr_core", "probe", {},
|
||||
{"available": ms._mr_core() is not None})
|
||||
|
||||
out = {
|
||||
"oracle": "src/rune/memory_store.py (live delegation: memory_retrieval + memory_scoring kernels)",
|
||||
"family": "retrieval/ranking — builder factory-heavy-ms-rtrv",
|
||||
"frozen_now": FROZEN_NOW_AWARE,
|
||||
"generated": "2026-10-14",
|
||||
"cases": cases,
|
||||
}
|
||||
target = REPO / "src/go/memory-store-deep" / "truth_retrieval_gaps.json"
|
||||
if "--out" in sys.argv:
|
||||
target = Path(sys.argv[sys.argv.index("--out") + 1])
|
||||
body = json.dumps(out, indent=1, ensure_ascii=False) + "\n"
|
||||
|
||||
if "--check" in sys.argv:
|
||||
current = target.read_text()
|
||||
if current == body:
|
||||
print(f"OK: byte-identical ({len(cases)} cases) — {target}")
|
||||
return 0
|
||||
print(f"MISMATCH: regenerated truth differs from {target}")
|
||||
return 1
|
||||
|
||||
target.write_text(body)
|
||||
print(f"wrote {target} ({len(cases)} cases)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,716 @@
|
||||
package main
|
||||
|
||||
// gaps_retrieval.go — memory_store RETRIEVAL/RANKING family gap-close
|
||||
// (dichotomy wave #2479): the audit-missing retrieval functions ported
|
||||
// name-preserving into the memory-store-deep crate. The SQLite row-fetch
|
||||
// stays facade-side per the landed crate's seams — every kernel here is a
|
||||
// pure transform over pre-fetched rows / injectable seams. Oracle:
|
||||
// src/rune/memory_store.py; deterministic-kernel authority:
|
||||
// src/rust/memory_retrieval (rank_fusion, project_chunk_rows,
|
||||
// merge_association_rows — banker's rounding, stable sorts, char-based
|
||||
// slicing). Truth: truth_retrieval_gaps.json (battery gaps_retrieval_test.go).
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"math"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// MrCore — the oracle's _mr_core() probe mirrored: RUNE_RETRIEVAL_CORE=python
|
||||
// pins the kernel off (the Python facade's fail-loud contract); otherwise the
|
||||
// kernel is available — here in-process, not via mr_loader.
|
||||
func MrCore() bool {
|
||||
return os.Getenv("RUNE_RETRIEVAL_CORE") != "python"
|
||||
}
|
||||
|
||||
// ── document frequency (the LIKE-count kernels) ─────────────────────────────
|
||||
// SQLite `text LIKE '%word%'` is ASCII-case-insensitive only — non-ASCII
|
||||
// letters compare case-sensitively. asciiLower replicates exactly that fold.
|
||||
|
||||
func asciiLower(s string) string {
|
||||
b := []byte(s)
|
||||
changed := false
|
||||
for i, c := range b {
|
||||
if c >= 'A' && c <= 'Z' {
|
||||
b[i] = c + ('a' - 'A')
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
if !changed {
|
||||
return s
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// Df — the recall() nested closure _df: how many active entry texts contain
|
||||
// `word` (LIKE %word% — ASCII case-insensitive substring). The facade supplies
|
||||
// the pre-fetched active texts; the SQL COUNT(*) is the same number.
|
||||
func Df(texts []string, word string) int {
|
||||
n := 0
|
||||
needle := asciiLower(word)
|
||||
for _, t := range texts {
|
||||
if strings.Contains(asciiLower(t), needle) {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// DfWord — _df_word: same LIKE-count over the entries corpus (the
|
||||
// distinctiveness signal behind the association gate).
|
||||
func DfWord(texts []string, word string) int {
|
||||
return Df(texts, word)
|
||||
}
|
||||
|
||||
// pyRound — Python round() / the Rust kernel's round_ties_even: banker's
|
||||
// rounding at n decimals (NOT half-up).
|
||||
func pyRound(v float64, n int) float64 {
|
||||
p := math.Pow(10, float64(n))
|
||||
return math.RoundToEven(v*p) / p
|
||||
}
|
||||
|
||||
// truncateRunes — Python str[:n] slices by CHARACTERS (quirk law), never bytes.
|
||||
func truncateRunes(s string, n int) string {
|
||||
if utf8.RuneCountInString(s) <= n {
|
||||
return s
|
||||
}
|
||||
c := 0
|
||||
for i := range s {
|
||||
if c == n {
|
||||
return s[:i]
|
||||
}
|
||||
c++
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// sliceCapLen — Python list[:cap] semantics: cap<0 → empty ([:negative] is a
|
||||
// different Python slice — the oracle only ever passes cap>=0 via budget*3),
|
||||
// cap>len → whole list. Go slicing would panic past len.
|
||||
func sliceCapLen(n, cap int) int {
|
||||
if cap < 0 {
|
||||
cap = 0
|
||||
}
|
||||
if cap > n {
|
||||
return n
|
||||
}
|
||||
return cap
|
||||
}
|
||||
|
||||
// MrRankFusion — chunk-side RRF fusion with an explicit candidate cap
|
||||
// (mr_rank_fusion): dense[:cap] + bm25[:cap], then the shared RankFusion
|
||||
// (RRF k=15, insertion-order rank list, lexical-only min-max fallback).
|
||||
func MrRankFusion(dense, bm25 [][2]interface{}, cap int) map[string]interface{} {
|
||||
d := dense[:sliceCapLen(len(dense), cap)]
|
||||
b := bm25[:sliceCapLen(len(bm25), cap)]
|
||||
return RankFusion(d, b, RRFK)
|
||||
}
|
||||
|
||||
// scalarIDKey — canonical comparison key for JSON ids (int 9 == float 9.0 in
|
||||
// the Python/Rust kernels; JSON membrane may deliver either).
|
||||
func scalarIDKey(v interface{}) string {
|
||||
switch x := v.(type) {
|
||||
case float64:
|
||||
return "n:" + trimFloat(x)
|
||||
case int:
|
||||
return "n:" + trimFloat(float64(x))
|
||||
case int64:
|
||||
return "n:" + trimFloat(float64(x))
|
||||
case string:
|
||||
return "s:" + x
|
||||
case nil:
|
||||
return "null"
|
||||
default:
|
||||
b, _ := json.Marshal(x)
|
||||
return "j:" + string(b)
|
||||
}
|
||||
}
|
||||
|
||||
// numOr — the Rust number() gate: JSON numbers only; anything else (strings,
|
||||
// null, missing) falls to the default.
|
||||
func numOr(m map[string]interface{}, key string, def float64) float64 {
|
||||
if v, ok := m[key]; ok {
|
||||
if f, ok2 := v.(float64); ok2 {
|
||||
return f
|
||||
}
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
// MergeAssociationRows — _merge_association_rows, the pure tail of
|
||||
// recall_with_associations (authority: memory_retrieval.merge_association_rows):
|
||||
// project each boosted row with the association-relevance blend, re-rank, cap.
|
||||
// base_rel is the max relevance over the OUT list as it stands — appended
|
||||
// boosts raise the floor for the boosts that follow them.
|
||||
func MergeAssociationRows(base, boosts, rows []map[string]interface{}, budget int) []map[string]interface{} {
|
||||
out := make([]map[string]interface{}, 0, len(base)+len(boosts))
|
||||
out = append(out, base...)
|
||||
for _, item := range boosts {
|
||||
id := item["id"]
|
||||
strength := numOr(item, "strength", 0.0)
|
||||
var row map[string]interface{}
|
||||
for _, r := range rows {
|
||||
if rv, ok := r["id"]; ok && scalarIDKey(rv) == scalarIDKey(id) {
|
||||
row = r
|
||||
break
|
||||
}
|
||||
}
|
||||
if row == nil {
|
||||
continue
|
||||
}
|
||||
conf := numOr(row, "confidence", 0.0)
|
||||
if conf == 0 {
|
||||
conf = 0.7
|
||||
}
|
||||
temp := numOr(row, "temperature", 0.0)
|
||||
if temp == 0 {
|
||||
temp = 1.0
|
||||
}
|
||||
baseRel := 0.0
|
||||
for _, r := range out {
|
||||
if v := numOr(r, "relevance", 0.0); v > baseRel {
|
||||
baseRel = v
|
||||
}
|
||||
}
|
||||
rel := baseRel * strength * (0.4 + 0.6*conf) * (0.5 + 0.5*temp)
|
||||
out = append(out, map[string]interface{}{
|
||||
"id": id,
|
||||
"text": row["text"],
|
||||
"importance": row["importance"],
|
||||
"topic": row["topic"],
|
||||
"entities": row["entities"],
|
||||
"source": row["source"],
|
||||
"last_accessed": row["last_accessed"],
|
||||
"rrf": pyRound(strength, 4),
|
||||
"confidence": pyRound(conf, 2),
|
||||
"temperature": pyRound(temp, 2),
|
||||
"relevance": pyRound(rel, 4),
|
||||
"via_association": true,
|
||||
"mode": "association_merged",
|
||||
})
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
return numOr(out[i], "relevance", 0.0) > numOr(out[j], "relevance", 0.0)
|
||||
})
|
||||
if len(out) > budget {
|
||||
out = out[:budget]
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// AssocLink — one linked_entries row distilled to the walk inputs.
|
||||
type AssocLink struct {
|
||||
ID interface{}
|
||||
Strength float64
|
||||
}
|
||||
|
||||
// RecallWithAssociations — the association-graph accelerator over a
|
||||
// pre-fetched base: walk each seed's links (facade applied the assoc_limit),
|
||||
// follow only genuinely strong links (>= 0.3 — auto-assoc floors at ~0.35),
|
||||
// keep the strongest link per boosted id in first-touch order, then merge.
|
||||
// hops < 1 returns the base unchanged (the bounded walk).
|
||||
func RecallWithAssociations(base []map[string]interface{}, linksPerSeed [][]AssocLink, entryRows []map[string]interface{}, budget, hops, assocLimit int) []map[string]interface{} {
|
||||
if len(base) == 0 {
|
||||
return base
|
||||
}
|
||||
if hops < 1 {
|
||||
return base
|
||||
}
|
||||
_ = assocLimit // applied by the facade's LIMIT (documented seam)
|
||||
byID := map[string]bool{}
|
||||
for _, r := range base {
|
||||
if id, ok := r["id"]; ok {
|
||||
byID[scalarIDKey(id)] = true
|
||||
}
|
||||
}
|
||||
var order []interface{}
|
||||
boost := map[string]float64{}
|
||||
boostVal := map[string]interface{}{}
|
||||
for _, links := range linksPerSeed {
|
||||
for _, l := range links {
|
||||
key := scalarIDKey(l.ID)
|
||||
if byID[key] {
|
||||
continue
|
||||
}
|
||||
if l.Strength < 0.3 {
|
||||
continue
|
||||
}
|
||||
if _, seen := boost[key]; !seen {
|
||||
order = append(order, l.ID)
|
||||
}
|
||||
if l.Strength > boost[key] {
|
||||
boost[key] = l.Strength
|
||||
boostVal[key] = l.ID
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(order) == 0 {
|
||||
return base
|
||||
}
|
||||
type item struct {
|
||||
id interface{}
|
||||
strength float64
|
||||
}
|
||||
items := make([]item, 0, len(order))
|
||||
for _, id := range order {
|
||||
k := scalarIDKey(id)
|
||||
items = append(items, item{id: boostVal[k], strength: boost[k]})
|
||||
}
|
||||
// sorted(boost_map.items(), key=lambda x: -x[1]) — stable, first-touch order on ties.
|
||||
sort.SliceStable(items, func(i, j int) bool { return items[i].strength > items[j].strength })
|
||||
boosts := make([]map[string]interface{}, 0, len(items))
|
||||
for _, it := range items {
|
||||
boosts = append(boosts, map[string]interface{}{"id": it.id, "strength": it.strength})
|
||||
}
|
||||
return MergeAssociationRows(base, boosts, entryRows, budget)
|
||||
}
|
||||
|
||||
// VecHit — one vec0 KNN row (rowid, distance); cosine is derived, not stored.
|
||||
type VecHit struct {
|
||||
RowID int64
|
||||
Distance float64
|
||||
}
|
||||
|
||||
// ChunkRecall — the deterministic tail of chunk_recall: cosine conversion +
|
||||
// min_sim grounding floor (abstain below), the 0.5-scored lexical fallback,
|
||||
// cap'd fusion, the wing/doc row filter (the SQL WHERE, replicated), and the
|
||||
// ordered chunk projection. Embedding + SQL fetch stay facade-side; `hasQvec`
|
||||
// false mirrors a qvec-less call (dense [] and best_dense 0.0).
|
||||
func ChunkRecall(hasQvec bool, vecHits []VecHit, ftsRowIDs []int64, chunkRows []map[string]interface{}, budget int, wing, doc string, minSim float64) []map[string]interface{} {
|
||||
dense := make([][2]interface{}, 0, len(vecHits))
|
||||
bestDense := 0.0
|
||||
if hasQvec {
|
||||
for _, h := range vecHits {
|
||||
cos := 1.0 - h.Distance/2.0
|
||||
dense = append(dense, [2]interface{}{h.RowID, cos})
|
||||
if cos > bestDense {
|
||||
bestDense = cos
|
||||
}
|
||||
}
|
||||
}
|
||||
if bestDense < minSim {
|
||||
return []map[string]interface{}{}
|
||||
}
|
||||
bm25 := make([][2]interface{}, 0, len(ftsRowIDs))
|
||||
for _, id := range ftsRowIDs {
|
||||
bm25 = append(bm25, [2]interface{}{id, 0.5})
|
||||
}
|
||||
fusion := MrRankFusion(dense, bm25, budget*3)
|
||||
ranksList, _ := fusion["ranks"].([]interface{})
|
||||
ids := map[string]bool{}
|
||||
var rankItems []rankPair
|
||||
for _, raw := range ranksList {
|
||||
pair, ok := raw.([]interface{})
|
||||
if !ok || len(pair) < 2 {
|
||||
continue
|
||||
}
|
||||
rrf, _ := pair[1].(float64)
|
||||
rankItems = append(rankItems, rankPair{pair[0], rrf})
|
||||
ids[scalarIDKey(pair[0])] = true
|
||||
}
|
||||
rows := []map[string]interface{}{}
|
||||
if len(ids) > 0 {
|
||||
for _, r := range chunkRows {
|
||||
rv, ok := r["rowid"]
|
||||
if !ok || !ids[scalarIDKey(rv)] {
|
||||
continue
|
||||
}
|
||||
if wing != "" {
|
||||
if w, _ := r["wing"].(string); w != wing {
|
||||
continue
|
||||
}
|
||||
}
|
||||
if doc != "" {
|
||||
if d, _ := r["doc_id"].(string); d != doc {
|
||||
continue
|
||||
}
|
||||
}
|
||||
rows = append(rows, r)
|
||||
}
|
||||
}
|
||||
return projectChunkRowsOrdered(rankItems, rows, budget)
|
||||
}
|
||||
|
||||
// rankPair — one [id, rrf] pair from the fusion's rank list.
|
||||
type rankPair struct {
|
||||
id interface{}
|
||||
rrf float64
|
||||
}
|
||||
|
||||
// projectChunkRowsOrdered — the ranks pair-list projection (authority:
|
||||
// memory_retrieval.project_chunk_rows): sorted -rrf (stable over the rank
|
||||
// list's insertion order), char-based text[:400], null-preserving wing/room/
|
||||
// source/page (absent keys project NULL, not ""), banker's-rounded rrf,
|
||||
// budget cap.
|
||||
func projectChunkRowsOrdered(rankItems []rankPair, rows []map[string]interface{}, budget int) []map[string]interface{} {
|
||||
items := make([]rankPair, 0, len(rankItems))
|
||||
items = append(items, rankItems...)
|
||||
sort.SliceStable(items, func(i, j int) bool { return items[i].rrf > items[j].rrf })
|
||||
out := []map[string]interface{}{}
|
||||
for _, it := range items {
|
||||
var row map[string]interface{}
|
||||
for _, r := range rows {
|
||||
if rv, ok := r["rowid"]; ok && scalarIDKey(rv) == scalarIDKey(it.id) {
|
||||
row = r
|
||||
break
|
||||
}
|
||||
}
|
||||
if row == nil {
|
||||
continue
|
||||
}
|
||||
text, _ := row["text"].(string)
|
||||
out = append(out, map[string]interface{}{
|
||||
"chunk_id": orString(row, "chunk_id", ""),
|
||||
"text": truncateRunes(text, 400),
|
||||
"wing": orNull(row, "wing"),
|
||||
"room": orNull(row, "room"),
|
||||
"source": orNull(row, "source_path"),
|
||||
"page": orNull(row, "page"),
|
||||
"rrf": pyRound(it.rrf, 4),
|
||||
})
|
||||
if len(out) >= budget {
|
||||
break
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func orNull(m map[string]interface{}, key string) interface{} {
|
||||
if v, ok := m[key]; ok {
|
||||
return v
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func orString(m map[string]interface{}, key, def string) interface{} {
|
||||
if v, ok := m[key]; ok {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
// FusedRecall — the audit-fix composition: atomic facts (via
|
||||
// recall_with_associations) + research chunks (via chunk_recall), de-duplicated
|
||||
// and scale-aligned. Both retrievals are seam-supplied (they carry the
|
||||
// embedding/SQL surfaces the facade owns). The fuse follows the
|
||||
// memory_retrieval authority EXACTLY — the crate's landed FuseResults
|
||||
// substitutes topic "fact"/"research" for EMPTY-STRING topics, where the Rust
|
||||
// kernel keeps "" (the default applies only to missing/non-string values);
|
||||
// divergence reported, sibling left untouched.
|
||||
func FusedRecall(entries, research []map[string]interface{}) []map[string]interface{} {
|
||||
seen := map[string]bool{}
|
||||
merged := []map[string]interface{}{}
|
||||
for _, e := range entries {
|
||||
text := strOrDefault(e, "text", "")
|
||||
key := NormKey(text)
|
||||
if seen[key] {
|
||||
continue
|
||||
}
|
||||
seen[key] = true
|
||||
merged = append(merged, map[string]interface{}{
|
||||
"id": orNull(e, "id"),
|
||||
"text": text,
|
||||
"topic": strOrDefault(e, "topic", "fact"),
|
||||
"kind": "fact",
|
||||
"relevance": pyRound(numOr(e, "relevance", 0.0), 4),
|
||||
"confidence": pyRound(numOr(e, "confidence", 0.0), 2),
|
||||
})
|
||||
}
|
||||
for _, c := range research {
|
||||
if c == nil {
|
||||
continue
|
||||
}
|
||||
text := strOrDefault(c, "text", "")
|
||||
key := NormKey(text)
|
||||
if seen[key] {
|
||||
continue
|
||||
}
|
||||
seen[key] = true
|
||||
rel := numOr(c, "score", 0.0)
|
||||
if _, has := c["rrf"]; has {
|
||||
rel = numOr(c, "rrf", 0.0)
|
||||
}
|
||||
conf := numOr(c, "confidence", 0.0)
|
||||
if conf == 0 {
|
||||
conf = 0.7
|
||||
}
|
||||
idv, hasID := c["chunk_id"]
|
||||
if !hasID {
|
||||
idv = ""
|
||||
}
|
||||
merged = append(merged, map[string]interface{}{
|
||||
"id": idv,
|
||||
"text": text,
|
||||
"topic": strOrDefault(c, "wing", "research"),
|
||||
"kind": "research",
|
||||
"relevance": pyRound(rel, 4),
|
||||
"confidence": pyRound(conf, 2),
|
||||
})
|
||||
}
|
||||
sortStableByRelevanceDesc(merged)
|
||||
for _, kind := range []string{"fact", "research"} {
|
||||
var vals []float64
|
||||
for _, r := range merged {
|
||||
if s, _ := r["kind"].(string); s == kind {
|
||||
vals = append(vals, numOr(r, "relevance", 0.0))
|
||||
}
|
||||
}
|
||||
if len(vals) > 1 {
|
||||
lo, hi := math.Inf(1), math.Inf(-1)
|
||||
for _, v := range vals {
|
||||
if v < lo {
|
||||
lo = v
|
||||
}
|
||||
if v > hi {
|
||||
hi = v
|
||||
}
|
||||
}
|
||||
span := hi - lo
|
||||
if span < 1e-9 {
|
||||
span = 1e-9
|
||||
}
|
||||
for _, r := range merged {
|
||||
if s, _ := r["kind"].(string); s == kind {
|
||||
r["relevance"] = pyRound((numOr(r, "relevance", 0.0)-lo)/span, 4)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
sortStableByRelevanceDesc(merged)
|
||||
return merged
|
||||
}
|
||||
|
||||
func strOrDefault(m map[string]interface{}, key, def string) string {
|
||||
if v, ok := m[key]; ok {
|
||||
if s, ok2 := v.(string); ok2 {
|
||||
return s
|
||||
}
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func sortStableByRelevanceDesc(list []map[string]interface{}) {
|
||||
sort.SliceStable(list, func(i, j int) bool {
|
||||
return numOr(list[i], "relevance", 0.0) > numOr(list[j], "relevance", 0.0)
|
||||
})
|
||||
}
|
||||
|
||||
// CuriosityProbeRow — one ORDER BY RANDOM() LIMIT-selected dead-weight row
|
||||
// (facade fetches; the selection itself stays SQL-side).
|
||||
type CuriosityProbeRow struct {
|
||||
ID interface{}
|
||||
Text string
|
||||
}
|
||||
|
||||
// CuriosityProbe — probe random untested memories: extract the first three
|
||||
// >3-char words (CHARACTER count), recall on them, record whether the entry
|
||||
// surfaces, and signal the wash increment when it does. Returns the probe
|
||||
// report and the ids whose proven_uses the facade must increment.
|
||||
func CuriosityProbe(deadWeight []CuriosityProbeRow, recall func(query string) []map[string]interface{}) (probed []map[string]interface{}, washed []interface{}) {
|
||||
probed = []map[string]interface{}{}
|
||||
washed = []interface{}{}
|
||||
for _, dw := range deadWeight {
|
||||
var words []string
|
||||
for _, w := range strings.Fields(dw.Text) {
|
||||
if utf8.RuneCountInString(w) > 3 {
|
||||
words = append(words, w)
|
||||
}
|
||||
if len(words) == 3 {
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(words) == 0 {
|
||||
continue
|
||||
}
|
||||
query := strings.Join(words, " ")
|
||||
results := recall(query)
|
||||
found := false
|
||||
for _, r := range results {
|
||||
if id, ok := r["id"]; ok && scalarIDKey(id) == scalarIDKey(dw.ID) {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
probed = append(probed, map[string]interface{}{
|
||||
"entry_id": dw.ID,
|
||||
"query": query,
|
||||
"found": found,
|
||||
"results_count": len(results),
|
||||
})
|
||||
if found {
|
||||
washed = append(washed, dw.ID)
|
||||
}
|
||||
}
|
||||
return probed, washed
|
||||
}
|
||||
|
||||
// tier thresholds (tier_manager.HOT_THRESHOLD / WARM_THRESHOLD, days).
|
||||
const (
|
||||
TierHotDays = 7
|
||||
TierWarmDays = 30
|
||||
)
|
||||
|
||||
// TierGet — the tier classification ladder (TierManager.get_tier, Python
|
||||
// parity oracle): rock stays hot; wash_count >= 20 is hot; otherwise recency
|
||||
// against the caller-supplied clock (< 7d hot, < 30d warm, else cold).
|
||||
// A missing/unparseable/NAIVE timestamp falls to warm (the oracle's
|
||||
// fromisoformat-then-subtract TypeError path) — only aware timestamps rank.
|
||||
func TierGet(entry map[string]interface{}, now time.Time) string {
|
||||
rock := "washable"
|
||||
if v, ok := entry["rock_status"]; ok {
|
||||
if s, ok2 := v.(string); ok2 {
|
||||
rock = s
|
||||
}
|
||||
}
|
||||
if rock == "rock" {
|
||||
return "hot"
|
||||
}
|
||||
wash := 0
|
||||
if v, ok := entry["wash_count"]; ok {
|
||||
if f, ok2 := v.(float64); ok2 {
|
||||
wash = int(f)
|
||||
}
|
||||
}
|
||||
if wash >= 20 {
|
||||
return "hot"
|
||||
}
|
||||
if la, ok := entry["last_accessed"]; ok {
|
||||
if s, ok2 := la.(string); ok2 && s != "" {
|
||||
if t, aware := parseISOAware(s); aware {
|
||||
hours := now.Sub(t).Hours()
|
||||
if hours < TierHotDays*24 {
|
||||
return "hot"
|
||||
}
|
||||
if hours < TierWarmDays*24 {
|
||||
return "warm"
|
||||
}
|
||||
return "cold"
|
||||
}
|
||||
}
|
||||
}
|
||||
return "warm"
|
||||
}
|
||||
|
||||
// parseISOAware — Python fromisoformat + the aware-aware subtraction gate:
|
||||
// returns (time, true) only for timestamps carrying an explicit offset. A
|
||||
// naive timestamp (the oracle's subtraction TypeError) and an unparseable one
|
||||
// both fall to the caller's warm default.
|
||||
func parseISOAware(s string) (time.Time, bool) {
|
||||
if s == "" {
|
||||
return time.Time{}, false
|
||||
}
|
||||
tail := s
|
||||
if len(tail) > 10 {
|
||||
tail = tail[10:]
|
||||
}
|
||||
aware := strings.HasSuffix(s, "Z") || strings.Contains(tail, "+") || strings.Contains(tail, "-")
|
||||
if !aware {
|
||||
return time.Time{}, false
|
||||
}
|
||||
s = strings.Replace(s, "Z", "+00:00", 1)
|
||||
for _, l := range []string{
|
||||
"2006-01-02T15:04:05.999999-07:00",
|
||||
"2006-01-02T15:04:05-07:00",
|
||||
} {
|
||||
if t, err := time.Parse(l, s); err == nil {
|
||||
return t, true
|
||||
}
|
||||
}
|
||||
return time.Time{}, false
|
||||
}
|
||||
|
||||
// TierDemotionSweep — tier_demotion_sweep: classify every active row, compare
|
||||
// its current tier against demotion_sweep's verdict, count the demotions.
|
||||
// ORACLE PARITY NOTE: demotion_sweep(entry) IS get_tier(entry) — the two
|
||||
// classifications are identical by construction, so the sweep as written can
|
||||
// never demote and the counters stay at zero for every corpus. The kernel
|
||||
// performs the honest comparison (not a hardcoded zero) so any future oracle
|
||||
// change surfaces as a battery diff.
|
||||
func TierDemotionSweep(rows []map[string]interface{}, now time.Time) map[string]int {
|
||||
demoted := map[string]int{"hot_to_warm": 0, "warm_to_cold": 0}
|
||||
for _, row := range rows {
|
||||
current := TierGet(row, now)
|
||||
updated := TierGet(row, now) // demotion_sweep == get_tier
|
||||
if current != updated {
|
||||
switch {
|
||||
case current == "hot" && updated == "warm":
|
||||
demoted["hot_to_warm"]++
|
||||
case current == "warm" && updated == "cold":
|
||||
demoted["warm_to_cold"]++
|
||||
}
|
||||
}
|
||||
}
|
||||
return demoted
|
||||
}
|
||||
|
||||
// FtsStmt — one statement the facade must execute (the write seam).
|
||||
type FtsStmt struct {
|
||||
SQL string `json:"sql"`
|
||||
Args []interface{} `json:"args"`
|
||||
}
|
||||
|
||||
// FtsDelete — _fts_delete: a CONTENTLESS FTS5 table cannot use DELETE FROM —
|
||||
// removal rides the special 'delete' command and requires the row's stored
|
||||
// text (read from entries first; that fetch stays facade-side). Emits the
|
||||
// statement iff the old row exists AND carries non-empty text; otherwise
|
||||
// nothing (the oracle's try/except swallow is a facade concern).
|
||||
func FtsDelete(eid interface{}, oldRow map[string]interface{}) []FtsStmt {
|
||||
if oldRow == nil {
|
||||
return []FtsStmt{}
|
||||
}
|
||||
text, ok := oldRow["text"].(string)
|
||||
if !ok || text == "" {
|
||||
return []FtsStmt{}
|
||||
}
|
||||
return []FtsStmt{{
|
||||
SQL: "INSERT INTO entries_fts(entries_fts, rowid, text) VALUES('delete', ?, ?)",
|
||||
Args: []interface{}{eid, text},
|
||||
}}
|
||||
}
|
||||
|
||||
// CacheLookup — the facade's get_cache result distilled to the decision
|
||||
// inputs: whether the key hit and the compressed payload ("" when absent).
|
||||
type CacheLookup struct {
|
||||
Hit bool
|
||||
Compressed string
|
||||
}
|
||||
|
||||
// CachedRecallPut — the cache write the facade must perform after a miss
|
||||
// (value serialization stays facade-side: json.dumps(..., ensure_ascii=False)
|
||||
// over the returned result; scope pinned to "retrieval").
|
||||
type CachedRecallPut struct {
|
||||
Key string
|
||||
Scope string
|
||||
}
|
||||
|
||||
// CachedRecall — the cache-wrapped recall orchestration: strip the query
|
||||
// (empty → no cache interaction at all), check the cache (a hit whose
|
||||
// compressed payload fails to parse falls through to full retrieval), recall
|
||||
// via the seam, drift-rerank non-empty results, then signal the put under the
|
||||
// key "recall:{query[:80].lower()}" (CHARACTER slice, lowercased — the
|
||||
// cached_recall key carries no memory_types suffix, unlike recall's internal
|
||||
// key). Shard routing stays facade-side (the seam is already bound to the
|
||||
// routed db).
|
||||
func CachedRecall(query string, cached CacheLookup, recallFn func(string) []map[string]interface{}, rerankFn func([]map[string]interface{}) []map[string]interface{}) ([]map[string]interface{}, *CachedRecallPut) {
|
||||
query = strings.TrimSpace(query)
|
||||
if query == "" {
|
||||
return []map[string]interface{}{}, nil
|
||||
}
|
||||
key := "recall:" + strings.ToLower(truncateRunes(query, 80))
|
||||
if cached.Hit {
|
||||
var parsed []map[string]interface{}
|
||||
if err := json.Unmarshal([]byte(cached.Compressed), &parsed); err == nil && parsed != nil {
|
||||
return parsed, nil
|
||||
}
|
||||
// unparseable cache payload → fall through to full retrieval
|
||||
}
|
||||
result := recallFn(query)
|
||||
if len(result) > 0 && rerankFn != nil {
|
||||
result = rerankFn(result)
|
||||
}
|
||||
return result, &CachedRecallPut{Key: key, Scope: "retrieval"}
|
||||
}
|
||||
@@ -0,0 +1,289 @@
|
||||
package main
|
||||
|
||||
// gaps_retrieval_test.go — memory_store RETRIEVAL family parity gate: replays
|
||||
// truth_retrieval_gaps.json (live oracle capture under the generator's rigs —
|
||||
// builder factory-heavy-ms-rtrv). Canonical-JSON comparison (the crate's
|
||||
// jsonEq law); GAPS_TRUTH_PATH overrides the truth file for live-parity runs
|
||||
// against freshly regenerated truth.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// canonEq — canonical-JSON equality: marshal → unmarshal → marshal both sides
|
||||
// so object KEY ORDER and Go struct-field order never mask or fake a match
|
||||
// (byte-level jsonEq alone flags field-order-only diffs like {sql,args}).
|
||||
func canonEq(t *testing.T, name, field string, got, want interface{}) {
|
||||
t.Helper()
|
||||
gb, _ := json.Marshal(got)
|
||||
wb, _ := json.Marshal(want)
|
||||
var gi, wi interface{}
|
||||
json.Unmarshal(gb, &gi)
|
||||
json.Unmarshal(wb, &wi)
|
||||
gb2, _ := json.Marshal(gi)
|
||||
wb2, _ := json.Marshal(wi)
|
||||
if string(gb2) != string(wb2) {
|
||||
t.Errorf("%s: %s divergence:\n got %s\nwant %s", name, field, gb2, wb2)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetrievalGapsParity(t *testing.T) {
|
||||
path := "truth_retrieval_gaps.json"
|
||||
if p := os.Getenv("GAPS_TRUTH_PATH"); p != "" {
|
||||
path = p
|
||||
}
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("truth file missing: %v", err)
|
||||
}
|
||||
var doc struct {
|
||||
Cases []struct {
|
||||
Fn string `json:"fn"`
|
||||
Name string `json:"name"`
|
||||
Input map[string]interface{} `json:"input"`
|
||||
Expected interface{} `json:"expected"`
|
||||
} `json:"cases"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &doc); err != nil {
|
||||
t.Fatalf("unparseable: %v", err)
|
||||
}
|
||||
ran := 0
|
||||
for _, c := range doc.Cases {
|
||||
switch c.Fn {
|
||||
case "mr_rank_fusion":
|
||||
got := MrRankFusion(pairList(c.Input["dense"]), pairList(c.Input["bm25"]), int(num(c.Input["cap"])))
|
||||
canonEq(t, c.Name, "fusion", got, c.Expected)
|
||||
case "merge_association_rows":
|
||||
got := MergeAssociationRows(objList(c.Input["base"]), objList(c.Input["boosts"]),
|
||||
objList(c.Input["rows"]), int(num(c.Input["budget"])))
|
||||
canonEq(t, c.Name, "merged", got, objListOrEmpty(c.Expected))
|
||||
case "df":
|
||||
got := Df(strList(c.Input["texts"]), str(c.Input["word"]))
|
||||
canonEq(t, c.Name, "n", map[string]interface{}{"n": got}, c.Expected)
|
||||
case "df_word":
|
||||
got := DfWord(strList(c.Input["texts"]), str(c.Input["word"]))
|
||||
canonEq(t, c.Name, "n", map[string]interface{}{"n": got}, c.Expected)
|
||||
case "chunk_recall":
|
||||
// truth rows are qvec-less (embed rig off): the vec side stays empty
|
||||
got := ChunkRecall(boolOf(c.Input["has_qvec"]), nil,
|
||||
int64List(c.Input["fts_rowids"]), objList(c.Input["chunk_rows"]),
|
||||
int(num(c.Input["budget"])), str(c.Input["wing"]), str(c.Input["doc"]),
|
||||
numOrJSON(c.Input, "min_sim", 0))
|
||||
canonEq(t, c.Name, "rows", got, listy(c.Expected))
|
||||
case "recall_with_associations":
|
||||
base := objList(c.Input["base"])
|
||||
var seeds [][]AssocLink
|
||||
for _, l := range listy(c.Input["links"]) {
|
||||
var one []AssocLink
|
||||
for _, raw := range listy(l) {
|
||||
m, _ := raw.(map[string]interface{})
|
||||
one = append(one, AssocLink{ID: m["id"], Strength: num(m["strength"])})
|
||||
}
|
||||
seeds = append(seeds, one)
|
||||
}
|
||||
got := RecallWithAssociations(base, seeds, objList(c.Input["entry_rows"]),
|
||||
int(num(c.Input["budget"])), int(num(c.Input["hops"])), int(num(c.Input["assoc_limit"])))
|
||||
canonEq(t, c.Name, "rows", got, listy(c.Expected))
|
||||
case "fused_recall":
|
||||
got := FusedRecall(objList(c.Input["entries"]), objList(c.Input["research"]))
|
||||
canonEq(t, c.Name, "merged", got, listy(c.Expected))
|
||||
case "cached_recall":
|
||||
cachedAny, _ := c.Input["cached"].(map[string]interface{})
|
||||
cached := CacheLookup{Hit: boolOf(cachedAny["hit"]), Compressed: str(cachedAny["compressed"])}
|
||||
recallOut := objList(c.Input["recall_out"])
|
||||
drift := driftList(c.Input["drift"])
|
||||
got, put := CachedRecall(str(c.Input["query"]), cached,
|
||||
func(q string) []map[string]interface{} { return recallOut },
|
||||
driftReorder(drift))
|
||||
exp, _ := c.Expected.(map[string]interface{})
|
||||
canonEq(t, c.Name, "result", got, listy(exp["result"]))
|
||||
var putOut map[string]interface{}
|
||||
if put != nil {
|
||||
putOut = map[string]interface{}{"put_key": put.Key, "put_scope": put.Scope}
|
||||
} else {
|
||||
putOut = map[string]interface{}{"put_key": nil, "put_scope": nil}
|
||||
}
|
||||
canonEq(t, c.Name, "put", putOut, map[string]interface{}{"put_key": exp["put_key"], "put_scope": exp["put_scope"]})
|
||||
case "curiosity_probe":
|
||||
var dw []CuriosityProbeRow
|
||||
for _, r := range listy(c.Input["rows"]) {
|
||||
pair := listy(r)
|
||||
dw = append(dw, CuriosityProbeRow{ID: pair[0], Text: str(pair[1])})
|
||||
}
|
||||
results := make([][]map[string]interface{}, 0)
|
||||
for _, r := range listy(c.Input["recall_results"]) {
|
||||
results = append(results, objList(r))
|
||||
}
|
||||
idx := 0
|
||||
probed, washed := CuriosityProbe(dw, func(q string) []map[string]interface{} {
|
||||
out := []map[string]interface{}{}
|
||||
if idx < len(results) {
|
||||
out = results[idx]
|
||||
}
|
||||
idx++
|
||||
return out
|
||||
})
|
||||
canonEq(t, c.Name, "probed", probed, listy(c.Expected.(map[string]interface{})["probed"]))
|
||||
canonEq(t, c.Name, "washed", washed, listy(c.Expected.(map[string]interface{})["washed"]))
|
||||
case "tier_get":
|
||||
entry, _ := c.Input["entry"].(map[string]interface{})
|
||||
nowStr := str(c.Input["now"])
|
||||
now, err := time.Parse("2006-01-02T15:04:05-07:00", nowStr)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: bad now %q: %v", c.Name, nowStr, err)
|
||||
}
|
||||
got := TierGet(entry, now)
|
||||
canonEq(t, c.Name, "tier", map[string]interface{}{"tier": got}, c.Expected)
|
||||
case "tier_demotion_sweep":
|
||||
nowStr := str(c.Input["now"])
|
||||
now, err := time.Parse("2006-01-02T15:04:05-07:00", nowStr)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: bad now: %v", c.Name, err)
|
||||
}
|
||||
got := TierDemotionSweep(objList(c.Input["rows"]), now)
|
||||
canonEq(t, c.Name, "demoted", got, c.Expected)
|
||||
case "fts_delete":
|
||||
var oldRow map[string]interface{}
|
||||
if v, ok := c.Input["old_row"]; ok && v != nil {
|
||||
oldRow, _ = v.(map[string]interface{})
|
||||
}
|
||||
got := FtsDelete(c.Input["eid"], oldRow)
|
||||
canonEq(t, c.Name, "stmts", got, listy(c.Expected))
|
||||
case "mr_core":
|
||||
// env-derived probe: assert BOTH branches explicitly (the truth row
|
||||
// pins the generator-env answer — available under live delegation).
|
||||
t.Setenv("RUNE_RETRIEVAL_CORE", "")
|
||||
if !MrCore() {
|
||||
t.Errorf("mr_core: live branch must report available")
|
||||
}
|
||||
t.Setenv("RUNE_RETRIEVAL_CORE", "python")
|
||||
if MrCore() {
|
||||
t.Errorf("mr_core: python pin must report unavailable")
|
||||
}
|
||||
default:
|
||||
t.Errorf("unknown fn kind: %s", c.Fn)
|
||||
}
|
||||
ran++
|
||||
}
|
||||
if ran == 0 {
|
||||
t.Fatalf("no cases ran")
|
||||
}
|
||||
t.Logf("%d cases replayed", ran)
|
||||
}
|
||||
|
||||
// ── battery-local decode helpers ────────────────────────────────────────────
|
||||
|
||||
func str(v interface{}) string {
|
||||
s, _ := v.(string)
|
||||
return s
|
||||
}
|
||||
|
||||
func num(v interface{}) float64 {
|
||||
f, _ := v.(float64)
|
||||
return f
|
||||
}
|
||||
|
||||
func boolOf(v interface{}) bool {
|
||||
b, _ := v.(bool)
|
||||
return b
|
||||
}
|
||||
|
||||
func numOrJSON(m map[string]interface{}, key string, def float64) float64 {
|
||||
if v, ok := m[key]; ok {
|
||||
if f, ok2 := v.(float64); ok2 {
|
||||
return f
|
||||
}
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func strList(v interface{}) []string {
|
||||
out := []string{}
|
||||
for _, e := range listy(v) {
|
||||
out = append(out, str(e))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func int64List(v interface{}) []int64 {
|
||||
out := []int64{}
|
||||
for _, e := range listy(v) {
|
||||
out = append(out, int64(num(e)))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func listy(v interface{}) []interface{} {
|
||||
arr, ok := v.([]interface{})
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
return arr
|
||||
}
|
||||
|
||||
func objListOrEmpty(v interface{}) []map[string]interface{} {
|
||||
out := objList(v)
|
||||
if out == nil {
|
||||
out = []map[string]interface{}{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func driftList(v interface{}) [][3]float64 {
|
||||
out := [][3]float64{}
|
||||
for _, raw := range listy(v) {
|
||||
pair := listy(raw)
|
||||
if len(pair) < 3 {
|
||||
continue
|
||||
}
|
||||
out = append(out, [3]float64{num(pair[0]), num(pair[1]), num(pair[2])})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// driftReorder — the cached_recall rerank seam wired to the landed
|
||||
// DriftRerankBlend kernel (orig*0.7 + drift*0.3, stable desc).
|
||||
func driftReorder(drift [][3]float64) func([]map[string]interface{}) []map[string]interface{} {
|
||||
return func(entries []map[string]interface{}) []map[string]interface{} {
|
||||
if len(drift) == 0 {
|
||||
return entries
|
||||
}
|
||||
byID := map[string]map[string]interface{}{}
|
||||
var order []map[string]interface{}
|
||||
for _, e := range entries {
|
||||
k := scalarIDKey(e["id"])
|
||||
if _, seen := byID[k]; !seen {
|
||||
order = append(order, e)
|
||||
}
|
||||
byID[k] = e
|
||||
}
|
||||
blendIn := make([]map[string]interface{}, 0, len(drift))
|
||||
for _, d := range drift {
|
||||
blendIn = append(blendIn, map[string]interface{}{
|
||||
"id": trimFloat(d[0]),
|
||||
"orig": d[1],
|
||||
"drift": d[2],
|
||||
})
|
||||
}
|
||||
sortedIDs := DriftRerankBlend(blendIn)
|
||||
out := make([]map[string]interface{}, 0, len(order))
|
||||
used := map[string]bool{}
|
||||
for _, id := range sortedIDs {
|
||||
if e, ok := byID["n:"+id]; ok && !used["n:"+id] {
|
||||
out = append(out, e)
|
||||
used["n:"+id] = true
|
||||
}
|
||||
}
|
||||
for _, e := range order {
|
||||
k := scalarIDKey(e["id"])
|
||||
if !used[k] {
|
||||
out = append(out, e)
|
||||
used[k] = true
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1 @@
|
||||
exit=0
|
||||
@@ -0,0 +1 @@
|
||||
exit=0
|
||||
@@ -0,0 +1 @@
|
||||
exit=0
|
||||
@@ -0,0 +1 @@
|
||||
exit=0
|
||||
@@ -0,0 +1 @@
|
||||
exit=0
|
||||
@@ -0,0 +1 @@
|
||||
exit=0
|
||||
@@ -0,0 +1,5 @@
|
||||
FAILED tests/test_rust_core_shims.py::test_capability_shims_delegation_agrees_with_fallback
|
||||
FAILED tests/test_rust_core_shims.py::test_construction_decision_placement_delegation
|
||||
FAILED tests/test_rust_core_shims.py::test_spine_delegation_agrees_with_fallback
|
||||
FAILED tests/test_rust_core_shims.py::test_token_optimizer_class_surface - Ru...
|
||||
FAILED tests/test_rust_core_shims.py::test_token_optimizer_delegation_agrees_with_fallback
|
||||
@@ -0,0 +1 @@
|
||||
exit=1 (5 pre-existing failures — failure set BYTE-IDENTICAL to stashed base, see msr-summary.md)
|
||||
@@ -0,0 +1,10 @@
|
||||
FAILED tests/test_rust_core_shims.py::test_capability_shims_delegation_agrees_with_fallback
|
||||
FAILED tests/test_rust_core_shims.py::test_construction_decision_placement_delegation
|
||||
FAILED tests/test_rust_core_shims.py::test_memory_store_helpers_delegate_to_kernel
|
||||
FAILED tests/test_rust_core_shims.py::test_mr_loader_chunk_projection_and_memory_store_delegation
|
||||
FAILED tests/test_rust_core_shims.py::test_mr_loader_chunk_sql_gather_through_seam
|
||||
FAILED tests/test_rust_core_shims.py::test_mr_loader_delegation_publics_exercised
|
||||
FAILED tests/test_rust_core_shims.py::test_mr_loader_recall_kernel_via_stats_seam
|
||||
FAILED tests/test_rust_core_shims.py::test_spine_delegation_agrees_with_fallback
|
||||
FAILED tests/test_rust_core_shims.py::test_token_optimizer_class_surface - Ru...
|
||||
FAILED tests/test_rust_core_shims.py::test_token_optimizer_delegation_agrees_with_fallback
|
||||
@@ -0,0 +1 @@
|
||||
exit=1 (10 pre-existing failures under pin — failure set BYTE-IDENTICAL to stashed base, see msr-summary.md)
|
||||
@@ -0,0 +1 @@
|
||||
exit=0
|
||||
@@ -0,0 +1,80 @@
|
||||
# memory_store RETRIEVAL family gap-close — verification summary
|
||||
|
||||
- **Module**: memory_store (audit alias: memory-ops + memory-store-deep GO corpus)
|
||||
- **Family**: retrieval/ranking/fusion — builder `factory-heavy-ms-rtrv`
|
||||
- **Worktree**: /mnt/workdrive/rune-wave-dn (isolated, tracks gitea/master)
|
||||
- **Date**: 2026-10-14
|
||||
|
||||
## What was ported (name-preserving, src/go/memory-store-deep/gaps_retrieval.go)
|
||||
|
||||
12 genuine kernel ports (audit-missing before): `cached_recall`→CachedRecall,
|
||||
`mr_rank_fusion`→MrRankFusion, `_fts_delete`→FtsDelete, `chunk_recall`→ChunkRecall,
|
||||
`fused_recall`→FusedRecall, `recall_with_associations`→RecallWithAssociations,
|
||||
`_merge_association_rows`→MergeAssociationRows, `curiosity_probe`→CuriosityProbe,
|
||||
`tier_demotion_sweep`→TierDemotionSweep (+TierGet ladder), `_mr_core`→MrCore,
|
||||
`_df_word`→DfWord, `_df`→Df. Seams per the landed crate's law: SQLite row-fetch,
|
||||
embedding, cache/shard/drift IO stay facade-side; kernels are pure transforms
|
||||
over pre-fetched rows + injectable seams.
|
||||
|
||||
Two additional audit counters flipped on PROSE, not ports (reported as such):
|
||||
`_embed` matched via the word "embedding" in a seam doc comment (network-bound,
|
||||
stays facade-side); `linked_entries` matched via the AssocLink doc comment (the
|
||||
SQL fetch stays facade-side; its deterministic walk IS ported inside
|
||||
RecallWithAssociations).
|
||||
|
||||
## Gates (all exit 0; logs in this directory)
|
||||
|
||||
| Gate | Evidence |
|
||||
|---|---|
|
||||
| gofmt -l (clean) | msr-gofmt.log |
|
||||
| go vet ./... | msr-vet.log |
|
||||
| go build ./... | msr-build.log |
|
||||
| go test ./... (whole crate, incl. prior batteries) | msr-gotest.log |
|
||||
| frozen truth byte-identical, run 1 | msr-freeze-verify-1.log |
|
||||
| frozen truth byte-identical, run 2 | msr-freeze-verify-2.log |
|
||||
| live parity (fresh oracle capture → Go battery) | msr-parity.log |
|
||||
| ruff (both scripts) | msr-ruff.log |
|
||||
| Python suite, live pin | msr-py-live.log |
|
||||
| Python suite, RUNE_RETRIEVAL_CORE=python | msr-py-pin.log |
|
||||
| failure-set diffs vs stashed base (both pins) | msr-py-*-baseline.txt + LIVE_SET_IDENTICAL / PIN_SET_IDENTICAL |
|
||||
|
||||
Truth: `src/go/memory-store-deep/truth_retrieval_gaps.json` — 43 cases.
|
||||
Pure kernels ride the shared freeze_truth harness (live Rust-kernel delegation
|
||||
authority); db-bound fns run the live oracle on fixture stores under rigs
|
||||
(_embed off, fake cache/shard recorders via sys.modules, frozen clocks on
|
||||
memory_store + tier_manager, drift_core._now_iso pinned, RUNE_COGNITION_CORE=
|
||||
python, memory_retrieval + memory_scoring .so built in-tree). Byte-stable
|
||||
across 5 regenerations. Generator: scripts/gen_memory_store_retrieval_truth.py
|
||||
(--check / --out). Parity: scripts/check_memory_store_retrieval_parity.py.
|
||||
|
||||
## Parity findings (documented divergences, not bugs)
|
||||
|
||||
1. **Landed sibling FuseResults topic-default divergence (found, worked
|
||||
around in-family)**: recall.go's FuseResults substitutes topic "fact"/
|
||||
"research" when the topic is the EMPTY STRING; the Rust authority
|
||||
(memory_retrieval.fuse_results, the post-2026-10-10 oracle) keeps "" — its
|
||||
default applies only to MISSING/non-string values. Frozen truth row
|
||||
`fused_recall/dedup_kinds` pins the authority shape. gaps_retrieval.go
|
||||
implements the authoritative fuse in FusedRecall; the sibling file is left
|
||||
untouched (never-edit-siblings law). Recommended follow-up for the crate
|
||||
owner: fix recall.go's FuseResults + re-freeze truth_recall.json.
|
||||
2. **tier_demotion_sweep can never demote**: demotion_sweep(entry) IS
|
||||
get_tier(entry) — current_tier == new_tier identically, so the counters are
|
||||
structurally always {hot_to_warm: 0, warm_to_cold: 0}. Ported honestly
|
||||
(real comparison, not hardcoded zero) + pinned by `sweep_never_demotes`.
|
||||
3. **curiosity_probe self-match**: the probe query is built from the entry's
|
||||
own >3-char words, so conjunctive BM25 always matches the entry itself —
|
||||
found=True is the norm (pinned); entries with no >3-char words are skipped
|
||||
entirely (pinned by `no_words_skipped`).
|
||||
4. **cached_recall put-value serialization stays facade-side** (json.dumps
|
||||
ensure_ascii=False, insertion-order keys); the Go kernel returns the put
|
||||
key + scope. The generator self-checks the oracle's put payload bytes.
|
||||
5. **Rounding**: banker's rounding (round_ties_even) everywhere in-family —
|
||||
the crate's older round2/round4 helpers are half-up (pre-existing; my
|
||||
kernels use pyRound and never the sibling helpers).
|
||||
|
||||
## Audit delta
|
||||
|
||||
memory_store: 30/68 → 44/68 matched (PARTIAL; ≥80% = FULL needs 55). The
|
||||
retrieval family's audit-missing list is now empty. Remaining missing fns
|
||||
belong to ingest/mining/scenes/hook/utility families (other lanes).
|
||||
@@ -0,0 +1 @@
|
||||
exit=0
|
||||
Reference in new issue
Block a user