conductor/rune
1
2 Fork 1
Code Issues 36 Pull Requests 32 Actions 100 Packages Projects Releases 564 Wiki Activity

feat(#161): retrieval-quality benchmark harness — literal synthesis e584bbf895b3 #278

Closed
conductor wants to merge 5 commits from issue161-harness into master
pull from: issue161-harness
merge into: conductor:master
Conversation 1 Commits 5 Files Changed 12
+2560 -9
conductor commented 2026-09-09 02:00:58 +01:00
Owner
Copy link
Copy source

RP10: #161 implementation — literal synthesis handoff (research-161-r1)

Literal handoff (§15): the accepted synthesis artifact e584bbf895b3 (with carried adversarial constraints — 10 mitigations, 7 falsification tests, 6 survivors, rejected-candidate reasons — embedded as provenance) is the verbatim implementation input. Branch issue161-harness implements it.

Implementation:

  • src/rune/memory_store.py — M1 mode field (observability-only, INV-13 structural test), M8 probe_cap parameterization (byte-identical defaults, proven)
  • src/rune/orchestration/retrieval_benchmark.py — NEW: the harness (FT1 fail-loud, isolation assertions, registry-contract backends, PR curves at operating points, abstention-aware metrics, stage profiler, shared run-record schema, fingerprint() reuse, evidence-citation gate)
  • tests/test_retrieval_benchmark.py — 20 tests: every FT conversion proven as a deterministic test
  • src/rune/memory_selector.py — vendored verbatim from p1-03-memory-selection tip 70c2007 (the real registry contract; deleted when #26 merges)

Verification: 529 tests pass (henxels env failures pre-existing on master, identical count); ruff clean.

Synthesis constraints honored: no new adapter protocol (M5), no silent mode degradation (FT1 fail-loud), no thresholded cross-backend numbers (M4), no E20-E22 claims (M9), one schema (M10).

Acceptance criteria mapping: AC-1..AC-12 from the synthesis are covered by the 20 harness tests + 529-suite green.

Closes #161 (the harness was the issue's demanded deliverable: "No migration decision without this harness").
Part of RP10/#273 — first real-issue implementation through the corrected Research-to-Build pipeline.

## RP10: #161 implementation — literal synthesis handoff (research-161-r1) **Literal handoff (§15):** the accepted synthesis artifact e584bbf895b3 (with carried adversarial constraints — 10 mitigations, 7 falsification tests, 6 survivors, rejected-candidate reasons — embedded as provenance) is the verbatim implementation input. Branch `issue161-harness` implements it. **Implementation:** - `src/rune/memory_store.py` — M1 mode field (observability-only, INV-13 structural test), M8 probe_cap parameterization (byte-identical defaults, proven) - `src/rune/orchestration/retrieval_benchmark.py` — NEW: the harness (FT1 fail-loud, isolation assertions, registry-contract backends, PR curves at operating points, abstention-aware metrics, stage profiler, shared run-record schema, fingerprint() reuse, evidence-citation gate) - `tests/test_retrieval_benchmark.py` — 20 tests: every FT conversion proven as a deterministic test - `src/rune/memory_selector.py` — vendored verbatim from p1-03-memory-selection tip 70c2007 (the real registry contract; deleted when #26 merges) **Verification:** 529 tests pass (henxels env failures pre-existing on master, identical count); ruff clean. **Synthesis constraints honored:** no new adapter protocol (M5), no silent mode degradation (FT1 fail-loud), no thresholded cross-backend numbers (M4), no E20-E22 claims (M9), one schema (M10). **Acceptance criteria mapping:** AC-1..AC-12 from the synthesis are covered by the 20 harness tests + 529-suite green. Closes #161 (the harness was the issue's demanded deliverable: "No migration decision without this harness"). Part of RP10/#273 — first real-issue implementation through the corrected Research-to-Build pipeline.
conductor added 4 commits 2026-09-09 02:00:59 +01:00
feat(p1-07): work pane + first-class runtime objects — MemoryQuery, Observation, VerificationResult, Outcome (#30) dc99f1fac3
fix(p1-07): dual-mode lazy import at the new runtime_objects failure site (#30, follows #168) a3ae72d353
feat(#161): vendor memory_selector (P1-03, #26) onto master line edc20b9e42 
The synthesis (e584bbf895b3) consumes the memory_selector registry as the
ONLY backend abstraction (M5 — no new adapter protocol). memory_selector.py
exists on the p1-03-memory-selection branch (#26, 66 selector tests) but is
not yet merged to master. Vendoring the module verbatim here so the harness
can register against the real contract; when #26 merges, the vendored copy
is deleted in favor of the branch module (same file, same contract).

NOT a fork of the contract — the file is byte-identical to branch tip
70c2007. Harness work proceeds; merge order is #26 first or this PR
subsumes the module (recorded for RP10 evidence).
feat(#161): retrieval-quality benchmark harness — literal synthesis e584bbf895b3 implementation
CI Pipeline / lint (push) Canceled after 0s
Details
CI Pipeline / unit-tests (push) Canceled after 0s
Details
CI Pipeline / canary (push) Canceled after 0s
Details
CI Pipeline / best-practices (push) Canceled after 0s
Details
CI Pipeline / documentation (push) Canceled after 0s
Details
CI Pipeline / type-check (push) Canceled after 0s
Details
CI Pipeline / gitea-offline (push) Canceled after 0s
Details
Auto-label / priority-gate (pull_request) Canceled after 0s
Details
Auto-label / auto-label (pull_request) Canceled after 0s
Details
CI Pipeline / lint (pull_request) Canceled after 0s
Details
CI Pipeline / unit-tests (pull_request) Canceled after 0s
Details
CI Pipeline / canary (pull_request) Canceled after 0s
Details
CI Pipeline / best-practices (pull_request) Canceled after 0s
Details
CI Pipeline / documentation (pull_request) Canceled after 0s
Details
CI Pipeline / type-check (pull_request) Canceled after 0s
Details
CI Pipeline / gitea-offline (pull_request) Canceled after 0s
Details
CI Pipeline / required-checks (push) Canceled after 0s
Details
CI Pipeline / required-checks (pull_request) Canceled after 0s
Details
c01d33f6e8 
Implemented from the accepted synthesis of research-161-r1 (QUALIFY: PROCEED,
13/13 readiness checks). The adversarial constraints are structural:

M1 (INV-13): memory_store.recall() result entries carry 'mode'
  (lexical-only | dense-only | hybrid) — observability ONLY, never read by
  store internals (structural test enforces). The silent-degradation hazard
  (A1) is now visible at every call site.
M8 (INV-5): recall() probe caps parameterized (probe_cap, default
  RECALL_BUDGET*2 — byte-identical default behavior, proven by test).
M6 (INV-9): assert_isolated_store() — RUNE_* scratch roots only; rejects
  memory_env.store_db() and shard dirs before any write.
M5 (INV-8): harness backends are memory_selector registry callables —
  no second adapter protocol; KNOWN RANKING BIAS disclosed per row (M4).
M10 (INV-12): one run-record schema (schema_version 1); fingerprint()
  reused verbatim from orchestration/benchmark.py.
M2 (INV-3): neutral-metadata path (frozen clock, confidence/temperature
  neutralized) with live-vs-neutral nDCG delta as first-class output.
M4 (INV-10): backend-local PR curves at stated operating points — never a
  single thresholded cross-backend number.
M7 (INV-11): abstention-aware headline (coverage@budget) + risk-coverage
  curve; abstention scored as designed behavior.
FT1 (INV-1): run_harness() REFUSES to run with unreachable embedding service
  — no silent lexical-only benchmarking. Fail-loud, zero labeled results.
M9 (INV-7): evidence-citation gate — no E20-E22 numeric claims in source.

Also vendors memory_selector.py (P1-03, #26) verbatim from branch tip
70c2007 — the harness consumes the real registry contract; deleted when
#26 merges (provenance in that commit message).

20 harness tests (all FT conversions + isolation + registry contract +
schema sharing). Full suite: 529 passed (henxels env failures pre-existing,
identical on master). Ruff clean.

Part of RP10/#273: first real-issue implementation through the corrected
Research-to-Build pipeline.
conductor added 1 commit 2026-09-09 02:18:57 +01:00
chore(#161): re-trigger CI — runner restart cleared wedged fetch loop
CI Pipeline / lint (push) Canceled after 0s
Details
CI Pipeline / unit-tests (push) Canceled after 0s
Details
CI Pipeline / canary (push) Canceled after 0s
Details
CI Pipeline / best-practices (push) Canceled after 0s
Details
CI Pipeline / documentation (push) Canceled after 0s
Details
CI Pipeline / type-check (push) Canceled after 0s
Details
CI Pipeline / gitea-offline (push) Canceled after 0s
Details
CI Pipeline / lint (pull_request) Canceled after 0s
Details
CI Pipeline / unit-tests (pull_request) Canceled after 0s
Details
CI Pipeline / canary (pull_request) Canceled after 0s
Details
CI Pipeline / best-practices (pull_request) Canceled after 0s
Details
CI Pipeline / documentation (pull_request) Canceled after 0s
Details
CI Pipeline / type-check (pull_request) Canceled after 0s
Details
CI Pipeline / gitea-offline (pull_request) Canceled after 0s
Details
Auto-version and Release / release (pull_request) Canceled after 0s
Details
CI Pipeline / required-checks (push) Canceled after 0s
Details
CI Pipeline / required-checks (pull_request) Canceled after 0s
Details
c12c155dd8
conductor commented 2026-09-09 02:49:01 +01:00
Author
Owner
Copy link
Copy source

CLOSED — branch provenance defect (RP10 correction 2026-09-09).

Root cause: issue161-harness was created from the research-stage checkout's STALE local master ref (0feeed6, pre-P1-07), carrying two unrelated P1-07 commits (dc99f1f, a3ae72d) as ancestors and a redundant vendored memory_selector conflicting with master's P1-03 line.

Repair: replaced by PR for issue161-harness-v2 — fresh from current master (c7a9a91), merge-base == master tip, exactly 3 files changed (memory_store M1+M8 re-applied; harness + tests carried verbatim). Implementation evidence preserved; no architectural conflicts silently resolved; the required dependency (memory_selector) is identified as already-on-master rather than imported.

This closure is a branch-hygiene fix, not an implementation rejection.

**CLOSED — branch provenance defect (RP10 correction 2026-09-09).** Root cause: `issue161-harness` was created from the research-stage checkout's STALE local master ref (0feeed6, pre-P1-07), carrying two unrelated P1-07 commits (dc99f1f, a3ae72d) as ancestors and a redundant vendored memory_selector conflicting with master's P1-03 line. Repair: replaced by **PR for `issue161-harness-v2`** — fresh from current master (c7a9a91), merge-base == master tip, exactly 3 files changed (memory_store M1+M8 re-applied; harness + tests carried verbatim). Implementation evidence preserved; no architectural conflicts silently resolved; the required dependency (memory_selector) is identified as already-on-master rather than imported. This closure is a branch-hygiene fix, not an implementation rejection.
conductor closed this pull request 2026-09-09 02:49:03 +01:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
P0 — critical
Spine/core correctness or blocks other work — do first
 P1 — high
Material user value or real defect (money, bugs, next-stage dep)
 P2 — medium
Quality/consolidation improvement
 P3 — low
Measurement / benchmark / deferred policy
 blocked
Dependencies open
 bug
Confirmed defect or regression
 custodian
epic
Rune v1 course-correction issue graph
 feature
New capability or improvement
 governance
growth
identity
in-progress
Currently being worked on by Rune
 incomplete
Incomplete work — draft PR with ledger, awaiting resume
 infrastructure
life
memory
memory-atlas
Memory Research Atlas child domain
 needs-testing
Needs manual testing with dumbed-down steps before close
 perception
ready
Order satisfied - Rune may start
 release-drift
research
Research jobs — spikes, watchlists, evidence work
 rune
rune
 scaffolding
scaffolding
 spine,correction
spine,correction
 spine,deferred
spine,deferred
 spine,resolved
spine,resolved
 strangler,burn,endgame,deferred
strangler,burn,endgame,deferred
 strangler,burn,endgame,inventory,epic
strangler,burn,endgame,inventory,epic
 tui
tui
 tui,epic
tui,epic
 tui,openspec
tui,openspec
 v1-course-correction
Rune v1 course-correction issue graph
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
conductor
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: conductor/rune#278