conductor/rune
1
2 Fork 1
Code Issues 36 Pull Requests 30 Actions 69 Packages Projects Releases 564 Wiki Activity

[P3-01] Build RuneBench v0.1 #44

New Issue
Closed
opened 2026-08-28 20:54:52 +01:00 by conductor · 3 comments
conductor commented 2026-08-28 20:54:52 +01:00
Owner
Copy link
Copy source

Plain summary

Builds RuneBench v0.1: controlled tasks (continuity, cross-session memory, coding, tool selection, preference learning, project continuation, error recovery, novel transfer) run under four conditions — fresh model, fresh Rune, experienced Rune, and experienced with memory/skills disabled — then repeated on another foundation model.

Example

Growth claims now refer to measured deltas between conditions A/B/C/D, not vibes.

Technical details

Parent epic: #16

Manifest ID: P3-01

Priority: P3 but required before synthetic-growth claims

Create controlled tasks covering:

  • conversation continuity;
  • cross-session memory;
  • coding;
  • tool selection;
  • preference learning;
  • project continuation;
  • error recovery;
  • novel transfer.

Run conditions:

A. naked/fresh model baseline
B. fresh Rune
C. experienced Rune
D. experienced Rune with memory/skills disabled or shuffled

Then repeat with another foundation model.

Metrics:

  • completion rate;
  • tool errors;
  • steps;
  • owner interventions;
  • false recall;
  • repeated mistakes;
  • token cost;
  • novel transfer;
  • latency.

Acceptance criteria:

  • reproducible benchmark dataset and runner;
  • growth claims refer to measured deltas.

Resolved dependency references

  • Depends on: None.
  • Blocks: #43 (P2-09), #45 (P3-02), #46 (P3-03), #47 (P3-04)

Relevant source files

Not specified in the handoff. P0-01 must identify authoritative paths before implementation; do not guess by adding another path.

Do not do

  • Do not begin this issue before its dependencies are complete.
  • Do not create a parallel authority path; replace, extend, or remain dormant.
  • Do not add unrelated cognitive subsystems or additive feature growth.
  • Do not treat module/schema existence, background activity, or model self-report as acceptance evidence.

Source directive

Preserved from /home/nick/Desktop/rune_v1_course_correction_handoff.md, Section 14. The handoff is authoritative if this issue summary is ambiguous.

Pipeline requirements

  • TDD: frozen/regression tests written first, red → green
  • ruff check and ruff format --check clean
  • Spec-compliance review gate: PASS
  • Code-quality review gate: Approved
  • Full orchestration suite regression green
  • Canary passes (where applicable)
  • Closed with a properly formatted closing comment + commit/PR references
## Plain summary Builds RuneBench v0.1: controlled tasks (continuity, cross-session memory, coding, tool selection, preference learning, project continuation, error recovery, novel transfer) run under four conditions — fresh model, fresh Rune, experienced Rune, and experienced with memory/skills disabled — then repeated on another foundation model. ## Example Growth claims now refer to measured deltas between conditions A/B/C/D, not vibes. ## Technical details Parent epic: #16 Manifest ID: **P3-01** **Priority:** P3 but required before synthetic-growth claims Create controlled tasks covering: - conversation continuity; - cross-session memory; - coding; - tool selection; - preference learning; - project continuation; - error recovery; - novel transfer. Run conditions: ```text A. naked/fresh model baseline B. fresh Rune C. experienced Rune D. experienced Rune with memory/skills disabled or shuffled ``` Then repeat with another foundation model. **Metrics:** - completion rate; - tool errors; - steps; - owner interventions; - false recall; - repeated mistakes; - token cost; - novel transfer; - latency. **Acceptance criteria:** - reproducible benchmark dataset and runner; - growth claims refer to measured deltas. --- ## Resolved dependency references - Depends on: None. - Blocks: #43 (P2-09), #45 (P3-02), #46 (P3-03), #47 (P3-04) ## Relevant source files Not specified in the handoff. P0-01 must identify authoritative paths before implementation; do not guess by adding another path. ## Do not do - Do not begin this issue before its dependencies are complete. - Do not create a parallel authority path; replace, extend, or remain dormant. - Do not add unrelated cognitive subsystems or additive feature growth. - Do not treat module/schema existence, background activity, or model self-report as acceptance evidence. ## Source directive Preserved from `/home/nick/Desktop/rune_v1_course_correction_handoff.md`, Section 14. The handoff is authoritative if this issue summary is ambiguous. ## Pipeline requirements - [ ] TDD: frozen/regression tests written first, red → green - [ ] `ruff check` and `ruff format --check` clean - [ ] Spec-compliance review gate: PASS - [ ] Code-quality review gate: Approved - [ ] Full orchestration suite regression green - [ ] Canary passes (where applicable) - [ ] Closed with a properly formatted closing comment + commit/PR references
conductor added the P3 — lowv1-course-correction labels 2026-08-28 20:54:52 +01:00
gitea-actions Bot added the featuregrowthmemory labels 2026-08-28 20:55:57 +01:00
gitea-actions Bot added the custodian label 2026-09-03 22:54:39 +01:00
conductor commented 2026-09-05 19:22:32 +01:00
Author
Owner
Copy link
Copy source

Handoff addendum (RUNE_VIBE_CODE_INTEGRITY_HANDOFF §5.8) — two evaluation classes

Extend this issue: RuneBench needs development-visible tests (routine development) AND holdout / novel evaluation (cases the implementation process cannot optimise against).

Holdout is particularly required for: abstraction transfer; self-model consistency; cross-model experienced-state transfer; skill reuse; correction handling; temporal supersession; outcome-conditioned memory utility.

Bar: benchmark gains on known fixtures alone do not prove developmental intelligence.

## Handoff addendum (RUNE_VIBE_CODE_INTEGRITY_HANDOFF §5.8) — two evaluation classes Extend this issue: RuneBench needs **development-visible** tests (routine development) AND **holdout / novel evaluation** (cases the implementation process cannot optimise against). Holdout is particularly required for: abstraction transfer; self-model consistency; cross-model experienced-state transfer; skill reuse; correction handling; temporal supersession; outcome-conditioned memory utility. **Bar:** benchmark gains on known fixtures alone do not prove developmental intelligence.
conductor commented 2026-09-05 20:06:32 +01:00
Author
Owner
Copy link
Copy source

Frontier addendum (ARCHITECTURE_FRONTIER_HANDOFF §14 — RuneBench strengthening)

Prior addenda stand (development-visible + holdout classes). Net-new requirements:

  • Pure causal-path mode: no hidden history injection, no fallback, no harness oracle may satisfy a benchmark outside the mechanism being measured. Every result must answer: "which live Rune mechanism caused this result?"
  • Ablation matrix (minimum): warm neurons off/on; graph off/on; adaptive concepts off/on; self-model off/on; abstraction off/on; skill layer off/on; sqlite-vec vs Qdrant; graph backend A/B; single-vector vs late-interaction retrieval.
  • Cross-model developmental test: fresh Rune + model A vs experienced Rune + model A vs fresh Rune + model B vs experienced Rune + model B. If experienced Rune retains measurable advantage after replacing the foundation model, the persistent substrate is demonstrably doing meaningful work.
  • External benchmarks: LongMemEval/LoCoMo-style recall, temporal questions, multi-hop, adversarial/contradiction cases, token-cost accounting. Never claim SOTA from self-designed RuneBench alone.
  • Reporting: per-query retrieval/activation provenance, paired A/B comparisons, ablations, token cost, latency, storage cost, bootstrap confidence intervals, model/provider, exact code/config version.
## Frontier addendum (ARCHITECTURE_FRONTIER_HANDOFF §14 — RuneBench strengthening) Prior addenda stand (development-visible + holdout classes). **Net-new requirements:** - **Pure causal-path mode**: no hidden history injection, no fallback, no harness oracle may satisfy a benchmark outside the mechanism being measured. Every result must answer: "which live Rune mechanism caused this result?" - **Ablation matrix (minimum)**: warm neurons off/on; graph off/on; adaptive concepts off/on; self-model off/on; abstraction off/on; skill layer off/on; sqlite-vec vs Qdrant; graph backend A/B; single-vector vs late-interaction retrieval. - **Cross-model developmental test**: fresh Rune + model A vs experienced Rune + model A vs fresh Rune + model B vs experienced Rune + model B. If experienced Rune retains measurable advantage after replacing the foundation model, the persistent substrate is demonstrably doing meaningful work. - **External benchmarks**: LongMemEval/LoCoMo-style recall, temporal questions, multi-hop, adversarial/contradiction cases, token-cost accounting. Never claim SOTA from self-designed RuneBench alone. - **Reporting**: per-query retrieval/activation provenance, paired A/B comparisons, ablations, token cost, latency, storage cost, bootstrap confidence intervals, model/provider, exact code/config version.
conductor commented 2026-10-08 12:28:08 +01:00
Author
Owner
Copy link
Copy source

CLOSED PROGRAMMATICALLY — commit 8e5e8ca (contract BUILDER-2026-10-08T11:25:21Z-1497c5). scripts/runebench.py: 8 task families × 4 conditions with offline fixtures (CI gate) + live Ollama mode; deltas C−B and C−D. #45/#46/#47 unblocked. 6/6 green.

CLOSED PROGRAMMATICALLY — commit 8e5e8ca (contract BUILDER-2026-10-08T11:25:21Z-1497c5). scripts/runebench.py: 8 task families × 4 conditions with offline fixtures (CI gate) + live Ollama mode; deltas C−B and C−D. #45/#46/#47 unblocked. 6/6 green.
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
P0 — critical
Spine/core correctness or blocks other work — do first
 P1 — high
Material user value or real defect (money, bugs, next-stage dep)
 P2 — medium
Quality/consolidation improvement
 P3 — low
Measurement / benchmark / deferred policy
 blocked
Dependencies open
 bug
Confirmed defect or regression
 custodian
epic
Rune v1 course-correction issue graph
 feature
New capability or improvement
 governance
growth
identity
in-progress
Currently being worked on by Rune
 incomplete
Incomplete work — draft PR with ledger, awaiting resume
 infrastructure
life
memory
memory-atlas
Memory Research Atlas child domain
 needs-testing
Needs manual testing with dumbed-down steps before close
 perception
ready
Order satisfied - Rune may start
 release-drift
research
Research jobs — spikes, watchlists, evidence work
 rune
rune
 scaffolding
scaffolding
 spine,correction
spine,correction
 spine,deferred
spine,deferred
 spine,resolved
spine,resolved
 strangler,burn,endgame,deferred
strangler,burn,endgame,deferred
 strangler,burn,endgame,inventory,epic
strangler,burn,endgame,inventory,epic
 tui
tui
 tui,epic
tui,epic
 tui,openspec
tui,openspec
 v1-course-correction
Rune v1 course-correction issue graph
No labels P3 — low custodian feature growth memory v1-course-correction
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
conductor
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: conductor/rune#44