conductor/rune

2 Fork 1

feat(#161): retrieval-quality benchmark harness — literal synthesis e584bbf895b3 #278

Closed

conductor wants to merge 5 commits from issue161-harness into master

pull from: issue161-harness

Author	SHA1	Message	Date
Nick	c12c155dd8	chore(#161 ): re-trigger CI — runner restart cleared wedged fetch loop CI Pipeline / lint (push) Canceled after 0s Details CI Pipeline / unit-tests (push) Canceled after 0s Details CI Pipeline / canary (push) Canceled after 0s Details CI Pipeline / best-practices (push) Canceled after 0s Details CI Pipeline / documentation (push) Canceled after 0s Details CI Pipeline / type-check (push) Canceled after 0s Details CI Pipeline / gitea-offline (push) Canceled after 0s Details CI Pipeline / lint (pull_request) Canceled after 0s Details CI Pipeline / unit-tests (pull_request) Canceled after 0s Details CI Pipeline / canary (pull_request) Canceled after 0s Details CI Pipeline / best-practices (pull_request) Canceled after 0s Details CI Pipeline / documentation (pull_request) Canceled after 0s Details CI Pipeline / type-check (pull_request) Canceled after 0s Details CI Pipeline / gitea-offline (pull_request) Canceled after 0s Details Auto-version and Release / release (pull_request) Canceled after 0s Details CI Pipeline / required-checks (push) Canceled after 0s Details CI Pipeline / required-checks (pull_request) Canceled after 0s Details	2026-09-09 02:18:52 +01:00
Nick	c01d33f6e8	feat(#161 ): retrieval-quality benchmark harness — literal synthesis e584bbf895b3 implementation CI Pipeline / lint (push) Canceled after 0s Details CI Pipeline / unit-tests (push) Canceled after 0s Details CI Pipeline / canary (push) Canceled after 0s Details CI Pipeline / best-practices (push) Canceled after 0s Details CI Pipeline / documentation (push) Canceled after 0s Details CI Pipeline / type-check (push) Canceled after 0s Details CI Pipeline / gitea-offline (push) Canceled after 0s Details Auto-label / priority-gate (pull_request) Canceled after 0s Details Auto-label / auto-label (pull_request) Canceled after 0s Details CI Pipeline / lint (pull_request) Canceled after 0s Details CI Pipeline / unit-tests (pull_request) Canceled after 0s Details CI Pipeline / canary (pull_request) Canceled after 0s Details CI Pipeline / best-practices (pull_request) Canceled after 0s Details CI Pipeline / documentation (pull_request) Canceled after 0s Details CI Pipeline / type-check (pull_request) Canceled after 0s Details CI Pipeline / gitea-offline (pull_request) Canceled after 0s Details CI Pipeline / required-checks (push) Canceled after 0s Details CI Pipeline / required-checks (pull_request) Canceled after 0s Details Implemented from the accepted synthesis of research-161-r1 (QUALIFY: PROCEED, 13/13 readiness checks). The adversarial constraints are structural: M1 (INV-13): memory_store.recall() result entries carry 'mode' (lexical-only \| dense-only \| hybrid) — observability ONLY, never read by store internals (structural test enforces). The silent-degradation hazard (A1) is now visible at every call site. M8 (INV-5): recall() probe caps parameterized (probe_cap, default RECALL_BUDGET2 — byte-identical default behavior, proven by test). M6 (INV-9): assert_isolated_store() — RUNE_ scratch roots only; rejects memory_env.store_db() and shard dirs before any write. M5 (INV-8): harness backends are memory_selector registry callables — no second adapter protocol; KNOWN RANKING BIAS disclosed per row (M4). M10 (INV-12): one run-record schema (schema_version 1); fingerprint() reused verbatim from orchestration/benchmark.py. M2 (INV-3): neutral-metadata path (frozen clock, confidence/temperature neutralized) with live-vs-neutral nDCG delta as first-class output. M4 (INV-10): backend-local PR curves at stated operating points — never a single thresholded cross-backend number. M7 (INV-11): abstention-aware headline (coverage@budget) + risk-coverage curve; abstention scored as designed behavior. FT1 (INV-1): run_harness() REFUSES to run with unreachable embedding service — no silent lexical-only benchmarking. Fail-loud, zero labeled results. M9 (INV-7): evidence-citation gate — no E20-E22 numeric claims in source. Also vendors memory_selector.py (P1-03, #26) verbatim from branch tip `70c2007` — the harness consumes the real registry contract; deleted when #26 merges (provenance in that commit message). 20 harness tests (all FT conversions + isolation + registry contract + schema sharing). Full suite: 529 passed (henxels env failures pre-existing, identical on master). Ruff clean. Part of RP10/#273: first real-issue implementation through the corrected Research-to-Build pipeline.	2026-09-09 02:00:20 +01:00
Nick	edc20b9e42	feat(#161 ): vendor memory_selector (P1-03, #26 ) onto master line The synthesis (e584bbf895b3) consumes the memory_selector registry as the ONLY backend abstraction (M5 — no new adapter protocol). memory_selector.py exists on the p1-03-memory-selection branch (#26, 66 selector tests) but is not yet merged to master. Vendoring the module verbatim here so the harness can register against the real contract; when #26 merges, the vendored copy is deleted in favor of the branch module (same file, same contract). NOT a fork of the contract — the file is byte-identical to branch tip `70c2007`. Harness work proceeds; merge order is #26 first or this PR subsumes the module (recorded for RP10 evidence).	2026-09-09 01:50:20 +01:00
Nick	a3ae72d353	fix(p1-07): dual-mode lazy import at the new runtime_objects failure site (#30 , follows #168 )	2026-09-05 23:50:29 +01:00
Nick	dc99f1fac3	feat(p1-07): work pane + first-class runtime objects — MemoryQuery, Observation, VerificationResult, Outcome (#30 )	2026-09-05 23:45:06 +01:00