conductor/rune
1
2 Fork 1
Code Issues 36 Pull Requests 30 Actions 45 Packages Projects Releases 564 Wiki Activity

Memory backend abstraction + comparative benchmark harness (SQLite FTS5/sqlite-vec vs Qdrant vs graph backends) #161

New Issue
Closed
opened 2026-09-05 20:05:22 +01:00 by conductor · 4 comments
conductor commented 2026-09-05 20:05:22 +01:00
Owner
Copy link
Copy source

Plain summary (ARCHITECTURE_FRONTIER_HANDOFF §4 — storage priority)

Before any production storage migration, create ONE canonical retrieval/graph contract so storage engines can be compared without changing cognitive semantics. Freeze deep sqlite-vec/custom-graph optimization unless needed for correctness or as a benchmark baseline.

Candidates

  • Baseline: SQLite FTS5 + sqlite-vec + existing graph implementation
  • Associative/vector: Qdrant (primary candidate); Weaviate/Vespa/Milvus as benchmark references only
  • Structural graph: Neo4j; FalkorDB; LadybugDB (experimental embedded — Kuzu's active successor lineage; do NOT adopt KuzuDB, archived Oct 2025)

Acceptance

Same corpus/query set runs against multiple backends, reporting: Recall@k, Precision@k, MRR/NDCG, abstention quality, temporal-query correctness, multi-hop correctness, p50/p95 latency, ingest/update latency, index rebuild time, RAM, disk, operational complexity, failure/recovery behaviour. No migration decision without this harness.

Architecture principle

Canonical durable state (events/evidence/episodes/outcomes) is the ONLY authority; Qdrant and the graph backend are REBUILDABLE projections. If a projector is buggy, fix and rebuild — unique autobiographical truth must not live only in a derived index.

## Plain summary (ARCHITECTURE_FRONTIER_HANDOFF §4 — storage priority) Before any production storage migration, create ONE canonical retrieval/graph contract so storage engines can be compared without changing cognitive semantics. **Freeze deep sqlite-vec/custom-graph optimization** unless needed for correctness or as a benchmark baseline. ## Candidates - **Baseline**: SQLite FTS5 + sqlite-vec + existing graph implementation - **Associative/vector**: Qdrant (primary candidate); Weaviate/Vespa/Milvus as benchmark references only - **Structural graph**: Neo4j; FalkorDB; LadybugDB (experimental embedded — Kuzu's active successor lineage; **do NOT adopt KuzuDB, archived Oct 2025**) ## Acceptance Same corpus/query set runs against multiple backends, reporting: Recall@k, Precision@k, MRR/NDCG, abstention quality, temporal-query correctness, multi-hop correctness, p50/p95 latency, ingest/update latency, index rebuild time, RAM, disk, operational complexity, failure/recovery behaviour. **No migration decision without this harness.** ## Architecture principle Canonical durable state (events/evidence/episodes/outcomes) is the ONLY authority; Qdrant and the graph backend are REBUILDABLE projections. If a projector is buggy, fix and rebuild — unique autobiographical truth must not live only in a derived index.
conductor added the P1 — high label 2026-09-05 20:05:23 +01:00
gitea-actions Bot commented 2026-09-05 20:12:55 +01:00
Copy link
Copy source

⚠️ Priority gate: this issue has no priority. Assign exactly one of P0 — critical (spine/correctness/blocks), P1 — high (material value or real defect), P2 — medium (quality/consolidation), P3 — low (measurement/deferred). An issue without a priority is not filed.

⚠️ **Priority gate:** this issue has no priority. Assign exactly one of **P0 — critical** (spine/correctness/blocks), **P1 — high** (material value or real defect), **P2 — medium** (quality/consolidation), **P3 — low** (measurement/deferred). An issue without a priority is not filed.
gitea-actions Bot added the buginfrastructurememory labels 2026-09-05 20:12:57 +01:00
conductor commented 2026-09-06 04:47:02 +01:00
Author
Owner
Copy link
Copy source

Addendum — canonical-state authority contract (from memory prior-art epic #181; D-equivalent)

#161 is the authoritative comparative harness. It also owns this contract, so no duplicate issue was created.

  • Canonical durable state owns: stable IDs, source/provenance, raw evidence, episodes/events, interpretations, outcomes, temporal validity, supersession/history.
  • Graph/vector stores (Neo4j, Qdrant, FalkorDB, ...) are derived, rebuildable projections.
  • Projection loss must be recoverable by rebuilding from canonical state (acceptance: delete + rebuild a projection, reproduce equivalent retrieval/graph output).
  • Changing vendors must never destroy historical state; projection-specific IDs must never silently become canonical identity.
  • Writes from a projection back into canonical state require explicit governed semantics.
  • No two stores may independently claim authority over the same fact lifecycle.

Depends on: canonical experience admission (#153) + provenance eventually; no migration before this contract is satisfied by the harness. (See epic #181.)

## Addendum — canonical-state authority contract (from memory prior-art epic #181; D-equivalent) #161 is the authoritative comparative harness. It also owns this contract, so no duplicate issue was created. - Canonical durable state owns: stable IDs, source/provenance, raw evidence, episodes/events, interpretations, outcomes, temporal validity, supersession/history. - Graph/vector stores (Neo4j, Qdrant, FalkorDB, ...) are **derived, rebuildable projections**. - Projection loss must be recoverable by rebuilding from canonical state (acceptance: delete + rebuild a projection, reproduce equivalent retrieval/graph output). - Changing vendors must never destroy historical state; projection-specific IDs must never silently become canonical identity. - Writes from a projection back into canonical state require explicit governed semantics. - No two stores may independently claim authority over the same fact lifecycle. Depends on: canonical experience admission (#153) + provenance eventually; no migration before this contract is satisfied by the harness. (See epic #181.)
conductor commented 2026-09-06 04:47:03 +01:00
Author
Owner
Copy link
Copy source

Addendum — bake-off scope extension (from memory prior-art epic #181; E-equivalent)

Add to the comparative benchmark (same corpus, same query set):

  • correct recall; stale-fact error rate; temporal queries; current-vs-historical fact distinction; multi-hop relational retrieval; contradiction/supersession handling; latency; storage/ops complexity; context usefulness; downstream task / Wish-Box success; rebuildability; failure behaviour.

Candidate configurations (architecture-driven, not brand-driven):

  • A. current Rune retrieval; B. graph-centric candidate; C. vector-centric candidate; D. graph + vector.

Selection rule (verbatim):

Adopt the simplest architecture that measurably wins. A second database is not justified merely because it is technically impressive.

#162 (Qdrant) and #163 (graph backends) are the component spikes feeding this bake-off. (See epic #181.)

## Addendum — bake-off scope extension (from memory prior-art epic #181; E-equivalent) Add to the comparative benchmark (same corpus, same query set): - correct recall; stale-fact error rate; temporal queries; current-vs-historical fact distinction; multi-hop relational retrieval; contradiction/supersession handling; latency; storage/ops complexity; context usefulness; downstream task / Wish-Box success; rebuildability; failure behaviour. Candidate configurations (architecture-driven, not brand-driven): - A. current Rune retrieval; B. graph-centric candidate; C. vector-centric candidate; D. graph + vector. Selection rule (verbatim): > Adopt the simplest architecture that measurably wins. A second database is not justified merely because it is technically impressive. #162 (Qdrant) and #163 (graph backends) are the component spikes feeding this bake-off. (See epic #181.)
conductor commented 2026-09-09 18:50:20 +01:00
Author
Owner
Copy link
Copy source

OUTWARD — ACCEPTED

  • artifact hash: bc57c4600880
  • runner: UNKNOWN (UNKNOWN)
  • model: UNKNOWN_NOT_EXPOSED
  • queries: 4
  • sources discovered: 16
  • sources actually read (machine receipt): 4
  • blocked: 0 | failed fetch: 0
  • evidence claims: 22 | contradictions: 4

SOURCE LEDGER

ID source/domain type status E-claims contradictions caveat
O1 www.dreaming.press comparison site NOT_SELECTED 0 0 backs ZERO accepted claims
O2 qdrant.tech vendor official (Qdrant benchmark portal) READ 6 0
O3 www.libhunt.com aggregator NOT_SELECTED 0 0 backs ZERO accepted claims
O4 github.com GitHub (independent benchmark harness) READ 6 0
O5 www.firecrawl.dev vendor blog NOT_SELECTED 0 0 backs ZERO accepted claims
O6 encore.dev vendor blog NOT_SELECTED 0 0 backs ZERO accepted claims
O7 www.reddit.com forum NOT_SELECTED 0 0 backs ZERO accepted claims
O8 github.com GitHub (benchmark suite: Qdrant/Milvus/Weaviate/Chroma/Pinecone/SQLite) NOT_SELECTED 0 0 backs ZERO accepted claims
O9 www.inductivee.com blog (agency benchmarks) NOT_SELECTED 0 0 backs ZERO accepted claims
O10 github.com GitHub (Mem0 evaluation suite) NOT_SELECTED 0 0 backs ZERO accepted claims
O11 agentmemorybenchmark.ai leaderboard site NOT_SELECTED 0 0 backs ZERO accepted claims
O12 github.com GitHub (reproducible memory-eval harness, MemoryAdapter interface) READ 5 0
O13 kranthib.github.io blog NOT_SELECTED 0 0 backs ZERO accepted claims
O14 arxiv.org academic (EvoMemBench) NOT_SELECTED 0 0 backs ZERO accepted claims
O15 memorypapers.org academic explainer (arXiv 2608.15008; Mem0-hosted site) READ 5 0
O16 hindsight-benchmarks.vercel.app leaderboard site NOT_SELECTED 0 0 backs ZERO accepted claims

Claim↔source links for this legacy artifact are derived from claim nesting (pre-#282 format); future artifacts carry explicit supports_source_ids/contradicts_source_ids.

<!-- source-ledger hash=bc57c4600880 --> ## OUTWARD — ACCEPTED - artifact hash: `bc57c4600880` - runner: UNKNOWN (UNKNOWN) - model: UNKNOWN_NOT_EXPOSED - queries: 4 - sources discovered: 16 - sources actually read (machine receipt): 4 - blocked: 0 | failed fetch: 0 - evidence claims: 22 | contradictions: 4 ## SOURCE LEDGER | ID | source/domain | type | status | E-claims | contradictions | caveat | |---|---|---|---|---|---|---| | O1 | [www.dreaming.press](https://www.dreaming.press/compare/sqlite-vec-vs-qdrant) | comparison site | NOT_SELECTED | 0 | 0 | backs ZERO accepted claims | | O2 | [qdrant.tech](https://qdrant.tech/benchmarks/) | vendor official (Qdrant benchmark portal) | READ | 6 | 0 | | | O3 | [www.libhunt.com](https://www.libhunt.com/compare-sqlite-vss-vs-qdrant) | aggregator | NOT_SELECTED | 0 | 0 | backs ZERO accepted claims | | O4 | [github.com](https://github.com/mycman/sqlite-vec-benchmark) | GitHub (independent benchmark harness) | READ | 6 | 0 | | | O5 | [www.firecrawl.dev](https://www.firecrawl.dev/blog/best-vector-databases) | vendor blog | NOT_SELECTED | 0 | 0 | backs ZERO accepted claims | | O6 | [encore.dev](https://encore.dev/articles/best-vector-databases) | vendor blog | NOT_SELECTED | 0 | 0 | backs ZERO accepted claims | | O7 | [www.reddit.com](https://www.reddit.com/r/vectordatabase/comments/170j6zd/) | forum | NOT_SELECTED | 0 | 0 | backs ZERO accepted claims | | O8 | [github.com](https://github.com/scriptstar/vector-db-benchmark) | GitHub (benchmark suite: Qdrant/Milvus/Weaviate/Chroma/Pinecone/SQLite) | NOT_SELECTED | 0 | 0 | backs ZERO accepted claims | | O9 | [www.inductivee.com](https://www.inductivee.com/blog/vector-database-performance-benchmarks-2025) | blog (agency benchmarks) | NOT_SELECTED | 0 | 0 | backs ZERO accepted claims | | O10 | [github.com](https://github.com/mem0ai/memory-benchmarks) | GitHub (Mem0 evaluation suite) | NOT_SELECTED | 0 | 0 | backs ZERO accepted claims | | O11 | [agentmemorybenchmark.ai](https://agentmemorybenchmark.ai/) | leaderboard site | NOT_SELECTED | 0 | 0 | backs ZERO accepted claims | | O12 | [github.com](https://github.com/Evanyuan-builder/memory-core-eval) | GitHub (reproducible memory-eval harness, MemoryAdapter interface) | READ | 5 | 0 | | | O13 | [kranthib.github.io](https://kranthib.github.io/tech-pulse/frameworks-n-platforms/agent-memory-benchmarks.html) | blog | NOT_SELECTED | 0 | 0 | backs ZERO accepted claims | | O14 | [arxiv.org](https://arxiv.org/abs/2605.18421) | academic (EvoMemBench) | NOT_SELECTED | 0 | 0 | backs ZERO accepted claims | | O15 | [memorypapers.org](https://memorypapers.org/papers/harness-the-memory-a-holistic-evaluation-of-memory-substrates-in-memory-agents) | academic explainer (arXiv 2608.15008; Mem0-hosted site) | READ | 5 | 0 | | | O16 | [hindsight-benchmarks.vercel.app](https://hindsight-benchmarks.vercel.app/) | leaderboard site | NOT_SELECTED | 0 | 0 | backs ZERO accepted claims | *Claim↔source links for this legacy artifact are derived from claim nesting (pre-#282 format); future artifacts carry explicit supports_source_ids/contradicts_source_ids.*
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
P0 — critical
Spine/core correctness or blocks other work — do first
 P1 — high
Material user value or real defect (money, bugs, next-stage dep)
 P2 — medium
Quality/consolidation improvement
 P3 — low
Measurement / benchmark / deferred policy
 blocked
Dependencies open
 bug
Confirmed defect or regression
 custodian
epic
Rune v1 course-correction issue graph
 feature
New capability or improvement
 governance
growth
identity
in-progress
Currently being worked on by Rune
 incomplete
Incomplete work — draft PR with ledger, awaiting resume
 infrastructure
life
memory
memory-atlas
Memory Research Atlas child domain
 needs-testing
Needs manual testing with dumbed-down steps before close
 perception
ready
Order satisfied - Rune may start
 release-drift
research
Research jobs — spikes, watchlists, evidence work
 rune
rune
 scaffolding
scaffolding
 spine,correction
spine,correction
 spine,deferred
spine,deferred
 spine,resolved
spine,resolved
 strangler,burn,endgame,deferred
strangler,burn,endgame,deferred
 strangler,burn,endgame,inventory,epic
strangler,burn,endgame,inventory,epic
 tui
tui
 tui,epic
tui,epic
 tui,openspec
tui,openspec
 v1-course-correction
Rune v1 course-correction issue graph
No labels P1 — high bug infrastructure memory
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
conductor
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: conductor/rune#161