feat(core): convert residual operator lifecycle onto the Go host (#2618) #2639

Open
conductor wants to merge 3 commits from feat/2618-operator-residual into master
pull from: feat/2618-operator-residual
119 changed files with 23566 additions and 1454 deletions

No files matched your search

+8 -6
View File
@@ -1,6 +1,6 @@
# Rune TUI — Product Requirements Document (PRD)
**Last Updated**: 2026-09-07
**Last Updated**: 2026-10-26
**Version**: v0.45.1
**Status**: Active Development
@@ -13,20 +13,22 @@ Rune is a stateful coding assistant with a Textual-based TUI. It connects to loc
---
<!-- BEGIN GENERATED: state — edits here are overwritten by scripts/gen_prd_state.py -->
**Version**: v0.54.0 · **HEAD**: 3e0b748 · **Regenerated**: 2026-09-07
**Version**: v0.359.0 · **HEAD**: 79ff61b3 · **Regenerated**: 2026-10-26
### Governed TUI surfaces (docs/tui-registry.md)
| `rune_tui.py` | The main Textual application — chat pane, work pane, status bar (version label + refresh), footer, dialogs | The known giant (file-size debt tracked in henxels); composition root for all UI |
| `tui_validator.py` | Import-allowlist validation for TUI code | Drift defect tracked in #139 (allowlist vs `__future__`/relative imports) |
| `tui_backlog.py` | Not a surface — the durable TUI Upgrade Backlog ledger (outstanding projection/control work per backend feature) | SCAFFOLD-TRANCHE-007; ledger only, never structural authority |
| `tui_transport.py` | Not a surface — the transport seam for the TUI's operator calls (in-process default / /v1 wire opt-in) | H4 slice 1 (contract BUILDER-2026-10-04T10:32:00Z-cli); RUNE_TUI_TRANSPORT selects; coverage: canary_tui_transport.py |
`rune_tui.py` is 4075 lines — the known giant, file-size debt tracked in henxels; decomposition target #125.
`rune_tui.py` is 4180 lines — the known giant, file-size debt tracked in henxels; decomposition target #125.
### TUI-adjacent subsystems (impact map)
- orchestration umbrella: 38 modules (operator, TUI presentation, task lifecycle)
- perception umbrella: 9 modules (context assembly, capsule)
- capability umbrella: 7 modules (tool authority)
- orchestration umbrella: 44 modules (operator, TUI presentation, task lifecycle)
- perception umbrella: 44 modules (context assembly, capsule)
- capability umbrella: 11 modules (tool authority)
<!-- END GENERATED: state -->
## Architecture
+1
View File
@@ -169,6 +169,7 @@
"lexicon_evolution",
"memory_consolidation",
"memory_env",
"memory_native",
"memory_selector",
"memory_store",
"memory_types",
+76
View File
@@ -0,0 +1,76 @@
# Governed model-visible context contract (#2618)
Contract ID: `ADMIT-2618-governed-context`.
## Authority and turn flow
The Go `runecore.ContextAssembler` is the only turn-seam orchestrator. `Engine.TurnStream` resolves the provider and persists the incoming user record, collects that channel's bounded history, and calls `ContextAssembler.Assemble` before any model call. The model receives exactly `AssembledContext.Messages`; there is no raw-history alternative when assembly is absent or fails.
`NativeContextAssembler` owns only collection and I/O orchestration:
1. it queries the current user text through the existing `MemoryStore` interface;
2. the sole implementation is the canonical `memory-store-deep` Go/Rust store from `docs/memory-store-contract.md`;
3. it sends channel history, the active user message, and the unranked recall rows to the `attention_capsule` Rust process over typed NDJSON;
4. it returns the admitted messages and joins the Go, Rust, and canonical-store traces.
The assembler never opens SQLite, computes relevance, constructs a second recall query, or implements admission policy. The existing Rust `memory_selector` crate projects store rows and performs relevance selection. The `attention_capsule` crate then owns token estimation, global/memory budget admission, deterministic capsule rendering, and selection status. The Go `context-broker` compatibility binary delegates admission/render decisions to the same Rust authority.
## Typed Go interface
```go
type ContextAssembler interface {
Assemble(context.Context, ContextRequest) (AssembledContext, error)
}
type ContextRequest struct {
Channel string
Query string
Model string
DBPath string
History []ChatMessage
System string
}
type AssembledContext struct {
Messages []ChatMessage
Trace map[string]any
}
```
`DBPath` may be empty; the canonical store's paths authority then applies its documented resolution order. A normal turn recalls at most 24 already-gated candidates with minimum score 0.42, and Rust selects at most four under the memory sub-budget. `RUNE_MEMORY_OFF` suppresses recall before collection and produces an explicit memory abstention, not an error.
`ContextError` carries `Code`, `Op`, `Message`, and an optional wrapped `Cause`. Canonical `MemoryError` values propagate unchanged. `TurnResult.ContextTrace` and the serving envelope's `rune.context_trace` expose the actual assembly receipt.
## Decision outcomes
- `ADMITTED`: the capsule is valid and at least one memory row was selected when used as `memory_status`.
- `ABSTAINED`: a valid query produced no selected memory (no rows, no match, below the canonical store's floor, empty projected text, or an explicit memory-off gate). This is successful and the rest of the capsule remains model-visible.
- `UNKNOWN`: the required active query/user evidence is blank, so relevance cannot be decided. The native `assemble` operation returns `CONTEXT_UNKNOWN`; it never relabels this as an empty recall or invents context.
A capsule whose memory abstains still has top-level `status=ADMITTED` when history/user admission succeeded. `memory_status`, `memory_abstained`, candidate/selected counts, and drop records make the distinction observable.
## Budget and rendering
Rust derives the default input budget from the supplied model window using the established broker formula, unless an explicit positive budget is supplied. Memory uses the established `min(1200, window/8)` sub-budget bounded by the global budget. Mandatory system/user blocks overflow loudly, including collective overflow. Evictable history/memory blocks admit by priority and stable collection order. Rendering is system (including selected memory), then history, then the active user message.
## Failure and trace contract
- `NATIVE_UNAVAILABLE`: required Rust or canonical native executable/library is missing or cannot be launched; HTTP chat answers 503 `native_unavailable` and the model is not called.
- `PROTOCOL_ERROR`: malformed/non-object native envelope or missing value shape; never reinterpreted as an empty capsule.
- `CONTEXT_UNKNOWN`: required selection evidence is absent.
- `CONTEXT_OVERFLOW`: mandatory content cannot fit the explicit budget.
- Canonical `STORE_ERROR` and other `MemoryError` codes propagate without reclassification as success.
Successful traces include `trace.context=go-context`, nested `native.context=rust-attention-capsule`, and the canonical memory trace. Native kernel names and admission/memory counts are evidence, not decorative constants inferred by the caller.
## Python compatibility and retirement
`AttentionCapsule` and `ContextBroker` retain their existing public shapes. Their default decision paths delegate to Rust; `NativeContextError` preserves operation and stable failure code. The catch-all fallback to a Python decision body is removed. Explicit `RUNE_AC_CORE=python` / `RUNE_COGNITION_CORE=python` pins remain solely for the independent parity oracle and legacy characterization runs.
Python contributor callbacks remain a compatibility collection seam because arbitrary Python callables cannot cross the native process boundary. Their admission and capsule decisions still delegate natively. This shim and the remaining Python collection bodies are marked for a later retirement tranche; this package does not delete Python authority or change the canonical memory-store retirement boundary.
## Serving proof and remaining boundary
Acceptance requires a real `Engine.TurnStream` turn in which relevant canonical recall text reaches the recording model client, the result/serving trace names `go-context` and the native kernel, and a `python`/`python3` oracle sentinel remains untouched. Native-absence and malformed-envelope tests must prove that neither the model nor a Python body produces a fallback answer.
Warm-neuron routing, multi-source `warm_router` orchestration, and the broader recall package are outside this tranche. They must compose this same assembler/store boundary rather than add another memory authority.
+62
View File
@@ -0,0 +1,62 @@
# Canonical memory-store contract (#2618)
Contract ID: `ADMIT-2618-canonical-memory-store`.
## Authority and why
The **Python serving behavior is the semantic oracle**: its 19-verb CLI, additive schema, calibrated recall/abstention, receipt shapes, and audit-chain behavior define compatibility. It is not a fourth implementation and this package does not invent a new store.
The sole native SQLite/store orchestrator is **`src/go/memory-store-deep`**. It already contains the broadest measured schema, lifecycle, mining, association, and recovery coverage. **`src/go/memory-ops` is only a compatibility launcher** to that binary; it no longer owns a database or a second set of store semantics. The older `search/get/wash/supersede` spellings are aliases, not canonical verbs.
Deterministic retrieval, scoring, row transforms, and policy belong to the existing Rust **`memory_retrieval`** and **`warm_neuron_store`** kernels. Go calls them through typed JSON/NDJSON process envelopes; no cgo is added. Python `memory_store`, `memory_env`, and `warm_neuron_store` remain compatibility facades during this tranche. Their default serving reads delegate to the native store; legacy connection-level APIs remain available for other consumers and for the independent parity oracle.
Path resolution belongs to the existing **paths authority**. `memory_env` contains no independent environment/path policy.
## Database location
Resolution order, highest precedence first:
1. `RUNE_STORE_DB`;
2. legacy alias `MEMORY_STORE_DB` (kept for API compatibility);
3. `<RUNE_MEMORY_DIR>/store/memory.db`;
4. `<RUNE_HOME>/memory/store/memory.db`;
5. `<paths.rune_home()>/memory/store/memory.db` (the paths default is `~/.rune`).
The old `.rune/mempalace/memories.db` and `mempalace/memories.db` conventions are non-canonical. The compatibility launcher and the canonical binary resolve the same path through the same paths package. Store creation makes the parent directory and bootstraps/migrates the schema; warm-neuron reads are read-only.
## Schema and writes
One logical schema covers `entries` (including metadata, classification, neuron, lifecycle, and supersession fields), `working`, `audit_log`, `associations`, `scenes`, `meta`, `documents`, `chunks`, `dedupe_receipts`, FTS5 indexes, and 256-dimensional `vec0` vector tables. Existing databases migrate additively. The upgraded pure-Go SQLite driver includes `sqlite-vec`, so Python-created `vec0` stores and older rowid/JSON fixtures are both readable without cgo. Dense candidate calculation and gating are Rust operations. A database whose vector representation cannot be read produces a typed error when dense recall needs it—never a silent lexical or Python fallback. `RUNE_MEMORY_EMBED=off` is an explicit deterministic/offline setting (used by hermetic tests), not an error fallback.
Entry insertion, update, and deletion are transactional. `add` validates the same text and memory-type rules, maintains FTS/vector/metadata state, and emits the Python receipt. `--no-dedupe` emits `{"added":true,"dedupe":"bypassed"}`. Default add uses the deterministic dedupe policy and persists a decision receipt (`created`, `duplicate`, `superseded`, or `rejected`). Update maintains FTS/vector state and keeps constitution entries read-only. Delete is explicit; working-set replacement, association, scene, mining, and reindex operations retain their Python-facing result shapes.
## Verbs and typed interface
Canonical CLI verbs are exactly the Python surface:
`add`, `recall`, `update`, `delete`, `working`, `audit`, `verify`, `stats`, `mine`, `chunk-recall`, `wake`, `export`, `associate`, `linked`, `scene`, `hierarchy`, `query-core`, `reindex`, `hook`.
The process interface is `memory-store-deep eval`: one JSON object per NDJSON line, `{"op":...,"arg":...}`, answered by `{"ok":true,"value":...,"trace":...}` or `{"ok":false,"error":{"code","message"},"trace":...}`. Runtime/serving operations are `stats`, `recall`, `read_query`, `warm_list`, `warm_has`, `warm_digests`, and `warm_slugs`; the complete 19-verb CLI covers the remaining store operations. The trace names the Go store and each Rust kernel actually called. This is the interface used by the Python compatibility facade and the Go core.
## Recall and warm-neuron semantics
Recall trims empty queries to abstention. Otherwise Go performs SQLite/embedding I/O and Rust owns conjunctive FTS query construction, dense candidate gating, RRF (`k=15`), metadata/recency scoring, mode labels (`lexical-only`, `dense-only`, `hybrid`, and association merge), budget caps, minimum-score abstention, association merge, chunk projection, and final fact/research fusion. Unclassified (`NULL`) rows remain eligible for typed queries. No match or a below-floor result is a successful empty list, not an error and not a fallback injection.
Warm-neuron serving performs read-only SQL in Go and sends the raw rows through the Rust kernel. Slug/text/importance/trigger coalescing, digests, slug sorting, and matching therefore have one authority. Store or kernel failure is an error; `[]`, `false`, and `{}` are valid only for successful empty/no-match results.
## Audit chain
`audit` appends `sha256(prev|claim|verdict|evidence)[:24]`, seeded by `GENESIS` and linked to the previous hash. `verify` recomputes every link and every payload, returns the Python `{"ok":...,"msg":...}` receipt, and exits nonzero on a broken/tampered chain. Resume reconciliation reads `verify` before any re-application; a duplicate audit append is observable as a chain-length increase.
## Failure and exit contract
- `0`: successful operation, including empty recall, no warm match, and empty stats.
- `1`: validly reached store/policy operation failed (`STORE_ERROR`, `POLICY_REJECTED`, `AUDIT_INVALID`, or invalid input reported by the CLI).
- `2`: usage, unknown verb, or malformed protocol (`USAGE`, `PROTOCOL_ERROR`).
- `127`: required native engine/kernel unavailable (`NATIVE_UNAVAILABLE`).
The Python facade raises `NativeMemoryStoreError` with a stable `code`, operation, and cause. It never converts native failure into Python defaults. The compatibility launcher returns the canonical process's exit code.
## Retirement boundary
This package does not delete Python authority and does not claim row 24 `PROVEN`: same-change Python removal is reserved for the final retirement tranche. The independent Python CLI remains the parity oracle while the default serving APIs prove native execution by trace and by fail-loud absence tests.
+11 -10
View File
@@ -42,21 +42,21 @@ the contract surface saying it may.
- `henxels.yaml` + `henxels_checks.py` — the auditable contract surface
<!-- BEGIN GENERATED: state — edits here are overwritten by scripts/gen_prd_state.py -->
**Version**: v0.54.0 · **HEAD**: 3e0b748 · **Regenerated**: 2026-09-07
**Version**: v0.359.0 · **HEAD**: 79ff61b3 · **Regenerated**: 2026-10-26
### Impact map (config/umbrellas.json — enforced complete by henxels)
| Umbrella | Modules | Canaries | Depends on |
|---|---|---|---|
| memory | 14 | canary_query_memory, canary_memory_synthesis, canary_full_system | governance |
| governance | 8 | canary_full_system, canary_schema_migration | — |
| identity | 9 | canary_full_system | governance |
| growth | 9 | canary_full_system | memory, governance |
| perception | 9 | canary_context_broker, canary_full_system | memory |
| capability | 7 | canary_p1_01, canary_p0_05, canary_native | memory |
| inference | 10 | canary_p1_06, canary_p1_02, canary_resource_authority | capability |
| orchestration | 38 | canary_p1_07, canary_orchestration_foundation, canary_conversation_state, canary_gitea_pipeline, canary_tui_drift, canary_cognitive_strata, canary_surface_parity | perception, capability, inference, identity, growth |
| **total** | **104 modules, 1:1 mapped** | | |
| capability | 11 | canary_native, canary_native, canary_p0_05, canary_p0_05, canary_p1_01, canary_p1_01, canary_rune_entry, canary_rune_serve | memory |
| governance | 11 | canary_audit_ledger, canary_full_system, canary_full_system, canary_schema_migration, canary_schema_migration, canary_spine_meta | — |
| growth | 20 | canary_full_system, canary_full_system | governance, memory |
| identity | 9 | canary_full_system, canary_full_system | governance |
| inference | 19 | canary_p1_02, canary_p1_02, canary_p1_06, canary_p1_06, canary_resource_authority, canary_resource_authority | capability |
| memory | 32 | canary_full_system, canary_full_system, canary_memory_synthesis, canary_memory_synthesis, canary_query_memory, canary_query_memory | governance |
| orchestration | 44 | canary_cognitive_strata, canary_cognitive_strata, canary_conversation_state, canary_conversation_state, canary_gitea_pipeline, canary_gitea_pipeline, canary_orchestration_foundation, canary_orchestration_foundation, canary_p1_07, canary_p1_07, canary_surface_parity, canary_surface_parity, canary_tui_drift, canary_tui_drift, canary_tui_transport | capability, growth, identity, inference, perception |
| perception | 44 | canary_context_broker, canary_context_broker, canary_full_system, canary_full_system | memory |
| **total** | **190 modules, 1:1 mapped** | | |
### Gates (merge-blocking, in enforcement order)
@@ -80,6 +80,7 @@ the contract surface saying it may.
- `workflows_are_governed`
- `tui_panels_are_governed`
- `new_rune_modules_have_canary_or_test_coverage`
- `dichotomy_language_verdict`
<!-- END GENERATED: state -->
---
+115
View File
@@ -0,0 +1,115 @@
# Governed Model-Visible Context Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use compose:subagent (recommended) or compose:execute to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make every Go-core turn send a budget-admitted history + canonical-memory capsule selected by Rust, with native traces and no silent Python or empty-context fallback.
**Architecture:** Go collects channel history and calls the existing `MemoryStore`; it does not rank rows or open a second database. The `attention_capsule` Rust process calls the existing `memory_selector` crate for row projection/selection, then owns token measurement, admission, and capsule rendering over a typed NDJSON boundary. Python keeps its public API as a delegating compatibility shim; only an explicit `RUNE_*_CORE=python` oracle pin may run retired Python decision bodies.
**Tech Stack:** Go 1.27, Rust 2021 + serde/serde_json, Python 3/pytest, NDJSON subprocess envelopes, existing canonical `memory-store-deep eval` interface.
## Global Constraints
- Work only on `feat/2618-context`; build on `f4b2b13b` and PR#2628's canonical memory store.
- Contract: `ADMIT-2618-governed-context`; every commit names it and maps findings to permanent tests.
- Go owns collection, subprocess/SQLite/model I/O, and turn orchestration; Rust owns scoring, selection, admission, budgets, and deterministic rendering.
- No new cgo in `rune-core`; no Python decision-body fallback and no raw-history fallback when a native engine is absent.
- Empty/below-floor recall is successful `ABSTAINED`; absent required evidence is `UNKNOWN`; failures carry stable typed codes.
- Existing Python consumers and frozen APIs remain supported; Python authority is marked for later retirement, not deleted.
---
### Task 1: Rust context decision and process contract
**Covers:** S1 budget/admission, S2 native broker decisions, S3 deterministic capsule selection and abstention
**Files:**
- Create: `src/rust/attention_capsule/src/context.rs`
- Create: `src/rust/attention_capsule/src/main.rs`
- Modify: `src/rust/attention_capsule/src/lib.rs`, `src/rust/attention_capsule/Cargo.toml`
- Test: `src/rust/attention_capsule/src/context.rs` unit tests and process integration tests
**Interfaces:**
- Consumes: `memory_selector::{store_project, select_core}` and canonical recall row dictionaries.
- Produces: `assemble_context(Value) -> Result<Value, ContextError>`; NDJSON `{"op":"assemble","arg":...}` answers `{"ok":true,"value":...,"trace":{"context":"rust-attention-capsule",...}}` or typed `error.code`.
- [ ] **Step 1: Write failing Rust tests** for priority/memory-budget admission, collective mandatory overflow, relevance-floor abstention, empty-query `UNKNOWN`, deterministic message order, and typed malformed-input errors.
- [ ] **Step 2: Run `cargo test` in `src/rust/attention_capsule`** and confirm failure because `assemble_context` and the process boundary do not exist.
- [ ] **Step 3: Implement the minimal pure Rust decision** using `memory_selector::store_project` + `select_core`; do not duplicate scoring or SQLite behavior. The assembled value carries full admitted blocks, messages, token totals, memory-selection counts, and status.
- [ ] **Step 4: Implement one-line NDJSON evaluation** with `CONTEXT_OVERFLOW`, `CONTEXT_UNKNOWN`, and `PROTOCOL_ERROR`; successful no-match responses remain `ok:true` with `status:"ABSTAINED"`.
- [ ] **Step 5: Rerun cargo tests and clippy**; confirm all pre-existing frozen capsule parity rows remain green.
### Task 2: Native Go broker and typed turn assembler
**Covers:** S1 turn seam, S2 Go collection + canonical store, S4 serving proof
**Files:**
- Modify: `src/go/context-broker/broker.go`, `src/go/context-broker/main.go`
- Create: `src/go/rune-core/internal/runecore/context_assembler.go`
- Create: `src/go/rune-core/internal/runecore/context_assembler_test.go`
- Modify: `src/go/rune-core/internal/runecore/engine.go`, `main.go`, `server.go`
**Interfaces:**
- Consumes: `MemoryStore.Recall(ctx, dbPath, query, budget, minScore, types)` and Rust `attention_capsule` NDJSON.
- Produces: `ContextAssembler.Assemble(ctx, ContextRequest) (AssembledContext, error)`; `AssembledContext{Messages []ChatMessage, Trace map[string]any}`; `ContextError{Code, Op, Message, Cause}`.
- [ ] **Step 1: Add failing Go tests** asserting the assembler sends raw recall rows to Rust, returns admitted model messages, joins `trace.context="go-context"` with store/kernel traces, and preserves typed native/store failures.
- [ ] **Step 2: Add a failing real-engine test** with a recording `ContextAssembler` and `fakeModel`: `TurnStream` must invoke assembly after channel history collection and the model must receive exactly the assembled messages.
- [ ] **Step 3: Run focused Go tests and record RED** (`ContextAssembler` undefined / raw history still sent).
- [ ] **Step 4: Implement Go collection and process orchestration**; remove broker admission policy from Go by delegating `admit`/`render` evaluation to Rust while retaining legacy pure helpers only for frozen oracle comparison.
- [ ] **Step 5: Compose one canonical memory store in `main.go`** shared by `/v1/memory/recall` and the engine assembler; native construction is mandatory and missing binaries stop the turn before model invocation.
- [ ] **Step 6: Add serving trace metadata and typed HTTP error mapping**, then rerun all `rune-core` and `context-broker` tests.
### Task 3: Python compatibility shims without fallback
**Covers:** S3 native attention capsule, S5 backward compatibility
**Files:**
- Modify: `src/rune/attention_capsule.py`, `src/rune/ac_loader.py`, `src/rune/context_broker.py`
- Create: `tests/test_context_native.py`
**Interfaces:**
- Consumes: existing Rust `ac_loader` API and native broker evaluation.
- Produces: `NativeContextError(code, operation, cause)`; unchanged `AttentionCapsule`, `ContextBroker.assemble`, `ContributionBlock`, and `MemorySelector` public shapes.
- [ ] **Step 1: Write failing Python no-oracle tests** patching every Python decision body to raise if entered; default capsule rendering/creation and broker admission must still succeed through native code.
- [ ] **Step 2: Write failing absence/malformed-response tests** requiring `NativeContextError.code` (`NATIVE_UNAVAILABLE` / `PROTOCOL_ERROR`) rather than a Python result.
- [ ] **Step 3: Run the focused pytest file and confirm RED** from the current catch-all fallback.
- [ ] **Step 4: Replace implicit fallback with thin delegation**, retaining `RUNE_AC_CORE=python` and `RUNE_COGNITION_CORE=python` solely as explicit independent-oracle pins. Add retirement notes, not Python-authority deletion.
- [ ] **Step 5: Run context, attention, and memory-selection suites** and fix only divergences proven by a failing characterization test.
### Task 4: Acceptance evidence and delivery
**Covers:** S4 no-Python serving proof, all acceptance checks, delivery
**Files:**
- Create: `docs/context-assembly-contract.md`
- Modify: `docs/dichotomy-rebase-honest-audit.md`, `verification/retirement-map.json`, this plan tail
- [ ] **Step 1: Run real serving-path proof** through `httptest` + `Engine.TurnStream` + native decision process + canonical store protocol double; assert memory text reaches `fakeModel`, native traces reach the response, and Python oracle sentinel files are absent.
- [ ] **Step 2: Run focused Rust/Go/Python suites, race tests, ruff, and dichotomy audit**; record commands and exact counts in the plan tail.
- [ ] **Step 3: Review the complete diff and read every substantive changed file**; map D1–D6 findings to tests and commits.
- [ ] **Step 4: Commit with `Contract: ADMIT-2618-governed-context`**, push `feat/2618-context`, and open an unmerged PR with `Relates to #2618`, the PR#2628 dependency, per-item evidence, and remaining warm_router/recall boundaries.
## Plan tail / discharge record
- D1 (no governed turn seam): `src/go/rune-core/internal/runecore/context_assembler_test.go` went RED at compile because `ContextRequest`, `AssembledContext`, and `ContextAssembler` did not exist.
- D2 (Go-owned admission): `src/rust/attention_capsule/src/context.rs` went RED at compile because `assemble_context` did not exist; the tests require Rust selection, budget, abstention, UNKNOWN, and overflow outcomes.
- D3 (silent Python capsule fallback): `tests/test_context_native.py` went RED 8/9 — six native-operation failures returned Python bodies, a missing library returned Python messages, and the broker trace had no native decision. The explicit Python-pin compatibility control passed.
- D4 (standalone broker still decided in Go): `TestEvalAdmissionUsesNativeRust` and `TestEvalBudgetAndMissingNativeDoNotUseGoFallback` went RED on local Go results and no native error. Live admission/render/budget now cross to Rust; `TestEvalNativeRejectsMalformedDecisionValue` separately went RED on an empty admission before protocol typing.
- D5 (compatibility boundary divergences): the first live native parity run was 770/803. Generic admission order, deferred collective overflow, and active-user capsule rendering each received a RED Rust regression; the fixed boundary is 803/803. Final model assembly still rejects collective overflow and appends the active user.
- D6 (serving failure/evidence typing): canonical `MemoryError` initially collapsed to HTTP `internal_error`; the new serving test went RED before `native_unavailable` / `store_error` mapping. A success envelope without native provenance also passed before the new RED test; it is now `PROTOCOL_ERROR`.
- D7 (real native turn): `RUNE_CONTEXT_PROOF=1 go test ./internal/runecore -run TestRealNativeServingPathWithCanonicalStore -v` passes against a real seeded canonical SQLite store and release Rust binary. The model double receives the recalled fact; traces name `go-context`, `rust-attention-capsule`, and `go-memory-store`; `python` / `python3` sentinels remain absent.
## Observed verification
- Rust: `attention_capsule` 15/15 and `memory_selector` 9/9 tests green; clippy `-D warnings` clean in both crates.
- Go: `go test -race ./...` and `go vet ./...` green in `src/go/context-broker` and `src/go/rune-core`.
- Python package matrix: 248 passed, 24 skipped across every `test_*context*.py` / `test_*memory*.py` file plus the capsule-authority suite. Native shim battery: 39 passed, 1 skipped. Full `ruff check src tests`: green.
- Compatibility/parity: `scripts/check_context_broker_parity.py` 803/803; `canary_context_broker.py`, `canary_p1_02.py`, and `canary_p1_03.py` all print PASSED.
- Native proof: real canonical-store/Rust serving test and scripted no-Python-oracle test both green. Absence, malformed envelope/value, UNKNOWN, ABSTAINED, overflow, and canonical failure codes have focused tests.
- Dichotomy: `python3 scripts/dichotomy_audit.py` exits 0; target rows are `context_broker` 23/24 FULL, `attention_capsule` 7/7 FULL, and `memory_selector` 13/13 FULL. The unrelated stale-report regeneration was discarded; no global completion claim is made.
- Repository checks: `henxels check` green. `scripts/type_check_ratchet.py` could not analyze because neither a `pyright` executable nor the Python module is installed. `scripts/lint_ratchet.py` is RED on pre-existing repository-wide BLE001 139>47 and S110 197>31 debt; changed files add no such handlers, and full ruff is green. These are not claimed as package failures or package greens.
- Additional consumer sweep: 181 tests passed before prerequisite builds, and the native shim suite then passed 39/39 runnable cases. The two remaining unrelated failures are the existing `test_sleight_chaos.py::test_fault[f_stale_index]` recall assertion and `test_p0_04_pipeline_removal.py::test_response_failure_still_carries_error_without_emitting_twice` source-text assertion; neither call path contains this package's changed code.
Construction close and Python-authority removal are not claimed. `warm_router`, multi-source warm recall, and the final compatibility-shim retirement remain for their own tranches.
+60
View File
@@ -0,0 +1,60 @@
# #2618 — one canonical memory store
Contract: `ADMIT-2618-canonical-memory-store`
Construction contract: `BUILDER-2026-10-26T01:59:46Z-c5e4b2-mimo-2618` (approved → building)
Scope supplement: `BUILDER-2026-10-26T04:10:18Z-6c3814-mimo-2618` (approved → building; umbrella/parity/facade/verification scripts)
## Baseline (before edits)
- Worktree `feat/2618-memory-store` at `79ff61b361e4c2f9e6a57978e7d00017f60a938d`, clean.
- `go test -run TestMemoryOps -v` in `src/go/parity-harness`: PASS; the frozen pin measures the live side-by-side as RED.
- `go test ./...`: PASS in `memory-ops` (no tests) and `memory-store-deep`; `cargo test`: 4/4 PASS in `warm_neuron_store`.
- Initial Python collection required the existing `src/go/paths/paths-bin` build prerequisite.
- Python memory baseline after that prerequisite: 117 passed, 26 skipped, 53 failed. Failures are dominated by unbuilt retired-kernel dependencies and `memory_store` recursion when `memory_retrieval` is unavailable. These are recorded baseline defects, not claimed as conversion evidence.
- Full parity baseline also has an unrelated pre-existing `model-routing` RED test. Package verification uses the memory-specific parity tests plus the full run to check for changed failures.
## Canonical contract
`docs/memory-store-contract.md` selects the Python behavior as semantic oracle, `memory-store-deep` as sole Go SQLite orchestrator, the existing Rust kernels as deterministic authorities, and paths as the sole location authority. `memory-ops` becomes a launcher only. No Python authority is deleted in this package.
## Red-first divergence discharge
| Finding | Red characterization required before implementation | Closure evidence |
|---|---|---|
| D1 incompatible verbs | Native must execute the 19-verb contract; the parity side-by-side expectation changes from recorded RED to GREEN | Go CLI tests + `TestMemoryOpsRowProvesNativeParity` |
| D2 three DB locations | Path tests require `RUNE_STORE_DB` / memory-dir/store/memory.db through paths | paths resolver tests + native scope trace |
| D3 missing schema/audit | Fresh-store connect must create the full logical schema; audit append/verify/tamper tests | Go schema and audit tests + parity recovery observable |
| D4 silent-zero and rc=0 failures | Missing tables and failed writes must return typed errors/nonzero; native/Rust absence must be 127/typed Python error | Go CLI failure tests + Python no-oracle tests |
| D5 raw insert / no dedupe receipt | Default add must exercise deterministic policy and persist receipts; bypass shape must match Python | Rust policy tests + Go add tests + parity add receipts |
| D6 LIKE recall | Native recall must call Rust FTS/fusion/scoring/final projection and preserve abstention/type filtering | Rust boundary tests + native integration tests + parity recall receipt |
| D7 Python/Rust opt-in fallback | Default `storage_backend`, selector recall, and warm serving must call native; missing binary/kernel raises | Python trace/fail-loud tests |
| D8 warm direct SQLite/default-on-error | Warm APIs must use Go read-only rows + Rust transforms and propagate typed errors | Go warm integration + Rust row battery + Python failure tests |
| D9 outdated SQLite | Pin modernc SQLite to a release carrying SQLite >=3.51.3 and record the actual `sqlite_version()` | Go version test + `go.mod/go.sum` |
## Implementation slices
1. Add failing paths, native protocol, warm-serving, core-composition, and parity tests; run and record the expected RED causes.
2. Unify path resolution and add the typed Go/Rust NDJSON boundaries.
3. Make `memory-store-deep` canonical: schema, verbs, recall, dedupe, audit, failures, aliases, and SQLite pin. Reduce `memory-ops` to a launcher.
4. Delegate Python path-based stats, selector recall, and warm-neuron serving through the native facade while retaining connection-level compatibility APIs and the independent Python CLI oracle.
5. Add an optional typed memory interface to `rune-core`, expose native recall through the core, and wire it in `main.go`.
6. Build native prerequisites, rerun focused suites, then broaden to memory Python suites, both Go store modules, paths, core, parity harness, and both Rust kernels. Read every substantive changed file before attesting.
## Delivery and remaining boundary
Commit with the named contract and `Discharge: D1..D9 -> <tests>` mapping, push `feat/2618-memory-store`, and open an unmerged PR. Row 24 remains PARTIAL solely because same-change Python removal belongs to the final retirement tranche. The PR must distinguish pre-existing unrelated failures from package evidence and identify any dense legacy-vec migration or context-package follow-up honestly.
## Plan tail / discharge record
- D1 verbs: `TestMemoryOpsCanonicalParityGreen` failed RED with the original receipt/scope/exit differences; the 19-verb matrix and two parity successor pins are GREEN.
- D2 paths: resolver precedence tests and `TestConnectUsesCanonicalPathAndBootstrapsSchema` failed before the paths resolver/canonical connect; they are GREEN and native parity scopes name only `memory/store/memory.db`.
- D3 schema/audit: fresh-store schema, audit-tamper, and missing-table tests prove the full vec0/FTS schema and loud audit behavior; both recovery breakpoints pass.
- D4 failures: short-add/unknown-verb/missing-kernel tests prove exit 1/2/127; Python and core tests prove typed errors are preserved without fallback.
- D5 write policy: Rust create/duplicate/supersede tests failed before `dedupe_decision`; exact and embedding-paraphrase integrations now pass, including receipt `prior_id` quirks and two persisted receipts.
- D6 recall: the original LIKE implementation failed the Rust-crossing and green-parity tests. Dense `vec0`, hybrid RRF, typed filtering, chunk projection, strong-association merge, and abstention now pass; a real Python-created vec0 store proves compatibility.
- D7 serving authority: native-by-default stats/selector tests failed before delegation. The facade now has real Python→Go trace evidence and native-absence/malformed-envelope tests; the old recursive retrieval probe raises `NativeMemoryStoreError`.
- D8 warm neurons: public warm APIs failed before native delegation. Go read-only rows now cross the warm Rust CLI; frozen replay is 35/35 and live oracle/Rust parity is 76/76 with four detected negative controls.
- D9 SQLite: the baseline driver measured 3.41.2 and failed the >=3.51.3 test. modernc.org/sqlite v1.61.0 (SQLite 3.54.1, sqlite-vec 0.1.9) passes the version and native-vec tests.
- Review discharges: metadata-flag normalization, FTS delete, non-object protocol envelopes, core error classification, and the `SetMaxOpenConns(1)` association deadlock each have focused regression coverage.
Final verification commands and results are summarized in the PR. Construction close is not claimed in this package; Python removal and final contract retirement remain for the last tranche.
+193
View File
@@ -0,0 +1,193 @@
# Residual Operator Lifecycle Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use compose:subagent (recommended) or compose:execute to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Convert the residual operator's serving-four capabilities—intake/classification, lifecycle, events, and records—onto `rune-core` with durable Go repositories, deterministic Rust decisions, exact resume/idempotency behavior, and a thin fail-loud Python compatibility client.
**Architecture:** `rune-core` composes one `NativeOperator` around the existing conversation `Store`, one JSONL event store, and one atomic task checkpoint. The `operator_lifecycle` Rust process owns intake/follow-up/family decisions, legal task transitions, record validation, and payload hashes; the existing `adaptive-router` binary remains a client dependency for route classification and research-dispatch payloads. Go owns all files, subprocesses, HTTP, and model orchestration, with no new cgo.
**Tech Stack:** Go 1.27 stdlib, Rust 2021 + serde/serde_json/sha2, NDJSON process envelopes, Python 3 stdlib HTTP client + pytest.
## Global Constraints
- Work only on `feat/2618-operator-residual` at the PR#2632 `eaa5c907` base.
- Contract: `ADMIT-2618-operator-residual`; the commit names it and maps findings to permanent tests.
- Serving four only: `rune_operator`, `orchestration/atomic`, `conversation_state`, `event_store`, and the three records those modules consume. Do not create services for the other orchestration modules.
- Construction execution and research execution stay outside this tranche. Route/research dispatch uses the existing adaptive-router client; no construction coordinator or research pipeline is reimplemented.
- Python authority is not deleted. `RUNE_OPERATOR_CORE=python` is an explicit oracle pin; the default Python operator is a native HTTP compatibility client with no fallback dispatch.
- Every production behavior starts with a failing test whose expected failure is observed. Existing Python oracle suites stay green under their explicit pin.
## Spec anchors
- **S1:** deterministic intake, follow-up identity, family classification, and route-client handoff.
- **S2:** canonical lifecycle transitions, nonterminal resume, atomic completed-turn receipts, and idempotent signals.
- **S3:** durable event append/replay, atomic files, task-scoped conversation records, and Rust record semantics.
- **S4:** real Go-host turn with native traces and no Python operator oracle.
- **S5:** default thin Python client, legacy pytest/orchestration parity, acceptance gates, and delivery.
---
### Task 1: Rust decision and process contract
**Covers:** S1, S2, S3
**Files:**
- Create: `src/rust/operator_lifecycle/Cargo.toml`
- Create: `src/rust/operator_lifecycle/src/lib.rs`
- Create: `src/rust/operator_lifecycle/src/main.rs`
**Interfaces:**
- Consumes: JSON task/message/event records plus caller-supplied ids and timestamps.
- Produces: `intake`, `transition`, `validate_record`, and `payload_hash` operations behind `operator_lifecycle eval`; success is `{"ok":true,"value":...,"trace":{"operator":"rust-operator-lifecycle"}}` and failure carries a stable `error.code`.
- [ ] **Step 1: Write failing Rust tests** for every legal/illegal edge, `research_required -> running`, nonterminal self-edge rules, follow-up amend versus fresh create/parent lineage, task-family precedence, schema-preserving records, and Python-compatible payload hashes.
```rust
#[test]
fn research_required_resumes_without_becoming_terminal() {
let task = task("research_required");
let decision = transition(&task, "running", "", NOW).unwrap();
assert_eq!(decision.record.status, "running");
assert!(decision.open);
}
```
- [ ] **Step 2: Run `cargo test` in `src/rust/operator_lifecycle`** and confirm RED because the crate and decision functions do not exist.
- [ ] **Step 3: Implement minimal pure decisions** with no filesystem, clock, entropy, network, or environment reads. Use the exact eight-status vocabulary and the Python edge map; duplicate completed-turn signals are a Go repository concern, not a new transition edge.
- [ ] **Step 4: Add the NDJSON process shell** with typed `ILLEGAL_TRANSITION`, `UNKNOWN_STATUS`, `INVALID_RECORD`, and `PROTOCOL_ERROR` outcomes and mandatory native provenance.
- [ ] **Step 5: Rerun cargo tests and clippy `-D warnings`**.
### Task 2: Go durable repositories and atomicity
**Covers:** S2, S3
**Files:**
- Create: `src/go/rune-core/internal/runecore/operator_records.go`
- Create: `src/go/rune-core/internal/runecore/operator_native.go`
- Create: `src/go/rune-core/internal/runecore/operator_atomic.go`
- Create: `src/go/rune-core/internal/runecore/operator_events.go`
- Create: `src/go/rune-core/internal/runecore/operator_tasks.go`
- Create: `src/go/rune-core/internal/runecore/operator_conversation.go`
- Test: corresponding `operator_*_test.go` files
**Interfaces:**
- Consumes: the Rust decision interface and existing `Store`.
- Produces: `EventStore.Append/Flush/Read/Replay`, `AtomicStore.Read/Write`, `TaskStateRepository.Intake/Transition/Resume/CompleteTurn`, and `ConversationRepository.Append/Window/Get`.
```go
type OperatorDecisions interface {
Intake(context.Context, IntakeDecisionRequest) (IntakeDecision, error)
Transition(context.Context, TaskStateRecord, string, string, string) (TransitionDecision, error)
PayloadHash(context.Context, string, any) (string, error)
}
```
- [ ] **Step 1: Add failing Go tests** for monotonic offsets, replay after reopen, duplicate event IDs, corrupt JSONL, failed-writer old-value preservation, concurrent readers seeing only complete old/new checkpoints, task-scoped message IDs, and event-before-checkpoint ordering.
- [ ] **Step 2: Run focused Go tests and record RED** from missing types and the absent lifecycle transition on the engine.
- [ ] **Step 3: Implement the real file repositories** using `CreateTemp + Sync + Close + Rename`, per-store mutexes, fsync'd JSONL appends, strict canonical-envelope validation, and Rust payload hashes. Reuse `TailStore`; do not open a second conversation database.
- [ ] **Step 4: Add failing resume tests** for `research_required` and `awaiting_approval`, duplicate `CompleteTurn` receipts after reopen, and a helper process killed after its durable acknowledgement. Require fresh opens to reconstruct identical task status, active pointer, route ref, and event count.
- [ ] **Step 5: Implement checkpointed receipts/resume**, rerun normal and `-race` tests, and retain the kill-and-resume proof as a permanent test.
### Task 3: Native operator composition and serving path
**Covers:** S1, S2, S3, S4
**Files:**
- Create: `src/go/rune-core/internal/runecore/operator.go`
- Create: `src/go/rune-core/internal/runecore/operator_serving_test.go`
- Modify: `src/go/rune-core/internal/runecore/engine.go`
- Modify: `src/go/rune-core/internal/runecore/server.go`
- Modify: `src/go/rune-core/main.go`
**Interfaces:**
- Produces: `Operator.Start/Running/CompleteTurn/Fail/Append/Snapshot`, `ProcessRouteClient.Classify/Dispatch`, `TurnResult.Task/Route/OperatorTrace`, `GET /v1/tasks`, and real `/v1/status` counters.
```go
type StartResult struct {
Task TaskStateRecord
Family string
Route RouteDecision
Research *ResearchDisposition
Duplicate *TurnReceipt
Trace map[string]any
}
```
- [ ] **Step 1: Write failing engine/server tests** requiring task IDs on both messages, created→running→completed events, real status/list-task reads, researched disposition without a model call, and duplicate signal IDs causing no second append/model call.
- [ ] **Step 2: Run the tests RED**; the pre-change engine has no operator and reports zero counters.
- [ ] **Step 3: Compose `NativeOperator` in `NewEngine`/production main**, keeping a typed `SetOperator` seam for hermetic legacy fixtures. Any nil/native failure is loud; there is no ungoverned raw-turn fallback.
- [ ] **Step 4: Implement route-client handoff and lifecycle error terminalization**. Route errors, context errors, provider errors, and persistence failures never leave an owned task running. Research-required and duplicate receipts are successful HTTP responses carrying `success:false` + `RESEARCH_REQUIRED` where applicable, not fabricated model answers.
- [ ] **Step 5: Prove a real HTTP turn** with `httptest`, actual Rust/Go route processes, canonical `TailStore`, native event/checkpoint files, and a fake model only at the provider boundary. Assert `go-operator`, `rust-operator-lifecycle`, and `go-event-store` traces, and assert `python`/`python3` sentinel scripts were never invoked.
### Task 4: Python compatibility client and oracle parity
**Covers:** S5
**Files:**
- Modify: `src/rune/rune_operator.py`
- Modify: `src/rune/orchestration/atomic.py`
- Modify: `src/rune/orchestration/conversation_state.py`
- Modify: `src/rune/orchestration/event_store.py`
- Modify: `src/rune/orchestration/records.py`
- Modify: `tests/conftest.py`
- Create: `tests/test_operator_native.py`
**Interfaces:**
- Default: `RuneOperator.send_task/get_status/conversation_window/list_tasks/list_models/resolve_pending_approval` calls the same Go `/v1` surface.
- Explicit oracle: `RUNE_OPERATOR_CORE=python` retains the existing bodies for parity and final retirement only.
- [ ] **Step 1: Write failing no-oracle tests** with a scripted HTTP core. Patch `_send_task_unmediated` and Python repository constructors to raise; default `send_task` must return the Go `rune` envelope and native trace without entering either body.
```python
def test_default_operator_uses_go_core_without_python_dispatch(monkeypatch):
monkeypatch.setenv("RUNE_OPERATOR_CORE", "go")
monkeypatch.setattr(rune_operator.RuneOperator, "_send_task_unmediated", _forbidden)
result = rune_operator.RuneOperator(core_url=core.url).send_task("hello", signal_id="sig-1")
assert result["operator_trace"]["operator"] == "go-operator"
```
- [ ] **Step 2: Add missing/malformed/native-error tests** and run RED against the current independent dispatch.
- [ ] **Step 3: Implement the thin stdlib client**, preserving stable `NativeOperatorError.code` and never falling back. Add native branches to the existing read/approval APIs; mark legacy repository/dispatch bodies for the final retirement tranche without deleting them.
- [ ] **Step 4: Pin the pre-existing pytest harness explicitly** to `RUNE_OPERATOR_CORE=python`; native tests override to `go` in their own bodies. Run operator, task-lifecycle, conversation, event, atomic, records, and full orchestration suites.
### Task 5: Evidence, review, and delivery
**Covers:** S4, S5
**Files:**
- Create: `verification/operator-lifecycle-serving.json`
- Modify: this plan tail
- [ ] **Step 1: Run focused red/green evidence, all `tests/test_orchestration_*.py`, operator/task suites, Go normal/race/vet, Rust test/clippy, ruff, henxels, and `python3 scripts/dichotomy_audit.py`**.
- [ ] **Step 2: Record the actual serving trace and exact test counts** in the verification JSON and plan tail; distinguish any pre-existing failures instead of hiding them.
- [ ] **Step 3: Review the complete diff and substantive files**, check staged paths against the admitted scope, run `git diff --cached --check`, and verify no binary/store-cache artifact is staged.
- [ ] **Step 4: Commit once** as `feat(core): convert residual operator lifecycle onto the Go host (#2618)` with `Contract`, `Construction-Contract`, and per-finding `Discharge` trailers; push the designated branch and open an unmerged PR whose body contains `Relates to #2618`, per-capability evidence, and remaining provider/background-memory boundaries.
## Plan tail / discharge record
Red and green observations are appended only after the commands actually run. The initial baseline is clean at `eaa5c907`; `go test ./...` in `src/go/rune-core` passed. The focused Python baseline required the declared `go-paths` helper and then produced 83 passed, 5 subtests passed, and one pre-existing native-prerequisite failure in `test_valid_local_final_uses_runtime_with_context_and_records_success` (missing capability/memory-selector/context-discovery engines, not an operator-lifecycle assertion).
Scope supplement: `ADMIT-2618-operator-oracle-canaries` (approved → building) covers the five direct scripted oracle canaries and the explicit `core="python"` constructor pin.
- D1 (deterministic decisions absent): the Rust decision battery went RED with 31 missing-type/function compile errors. Four process tests then went RED on missing `evaluate`. The finished crate has 14 decision + 4 process tests green, clippy `-D warnings`, fmt, and a real NDJSON `research_required -> running` call.
- D2 (Go persistence absent): atomic/event/native tests went RED on missing `EventEnvelope`, `OpenEventStore`, `OpenAtomicStore`, and `NewNativeOperatorDecisions`. Real file tests now cover failed writers, concurrent old/new readers, batch replay, corruption, duplicate IDs, and typed process failures.
- D3 (task/conversation/resume absent): task tests went RED on missing `TaskStateRepository`, `OpenTaskStateRepository`, `CorruptStateCheckpointError`, and `NewConversationRepository`. Fresh opens now recover task/route/receipt state, and helper processes are killed after durable acknowledgement for both nonterminal states before fresh-process resume.
- D4 (serving lifecycle absent): route/engine/server tests went RED on missing `RouteDecision`, `RouteClient`, `NativeOperator`, `StartRequest`, and `OperatorSnapshot`. Real turns now attach task IDs to both messages, persist route/event/task records, expose real counters/list tasks, skip models for research dispositions, and replay duplicate signals without new effects.
- D5 (independent Python dispatch): five no-oracle tests went RED because `core_url` did not exist; the explicit oracle control passed. Six tests now prove the default thin client, all read APIs, typed native failures, and the explicit Python pin.
- D6 (provider-refusal regression): the first full Go run had one RED, `TestEngineRefusesUnroutableProviderRecord`, because normal-turn user persistence moved before provider refusal. The sequence was corrected to resolve normal-turn providers before append while research remains model-independent; that test and the full suite are green.
- D7 (review: physical replay offsets): `TestEventStoreReopenCountsBlankPhysicalLines` went RED at offset 2 instead of 3 because open recovery counted parsed rows rather than physical lines. The replay reader now carries physical offsets; the regression and full suite pass.
- D8 (direct oracle canaries): `canary_p1_01` reached the live core and failed `no_provider` before an explicit pin existed. `core="python"` now selects the legacy body deliberately in p1_01/02/03/06/07. p1_01/02/03/07 pass; p1_06 still fails earlier in untouched provider routing (`SimpleNamespace.provider` missing), which is disclosed provider-stack work, not hidden or counted green.
## Observed verification
- `go test ./... -count=1` and `go test -race ./... -count=1` pass in `src/go/rune-core`; `go vet ./...` is clean.
- `cargo test`, `cargo clippy --all-targets -- -D warnings`, and `cargo fmt --check` pass in `src/rust/operator_lifecycle` (18 tests).
- The full orchestration/operator matrix passes: 307 tests + 5 subtests across every `tests/test_orchestration_*.py`, operator tools/probe, task lifecycle/state, conversation compaction, and native client tests. Building the declared native prerequisites removed the baseline-only missing-engine failure.
- `canary_conversation_state`, `canary_orchestration_foundation`, and all 35 `canary_rune_serve` checks pass. Explicit-oracle p1_01/02/03/07 canaries pass; p1_06 is the disclosed pre-existing provider-routing failure above.
- `RUNE_CONTEXT_PROOF=1 go test ./internal/runecore -run TestRealNativeServingPathWithCanonicalStore -v` passes: PR#2632 canonical recall still reaches the model on top of this lifecycle.
- `TestRealNativeServingTurnDoesNotInvokePythonOperator` passes through real HTTP composition with native route/event files and an absent Python sentinel. The captured trace is in `verification/operator-lifecycle-serving.json`.
- `ruff check src tests` and changed-file Ruff checks pass. `henxels check` passes. `python3 scripts/dichotomy_audit.py` exits 0.
- `python3 scripts/scaffolding_coverage.py` names 33 pre-existing ungoverned Go/Rust modules; `operator_lifecycle` is not among them and is covered by this contract. This unrelated existing set is not claimed green.
- `python3 scripts/type_check_ratchet.py` cannot analyze because `python3 -m pyright` is not installed and returns no JSON. `python3 scripts/lint_ratchet.py` remains RED on pre-existing BLE001 139>47 and S110 197>31 debt; full Ruff is green and the changed files add no such handlers.
No Python authority is deleted, no construction/research executor is reimplemented, and construction closure is not claimed. Provider registry/transport/profile work and the curator/consolidation/drift/tier/sleep/continuity background-memory pipeline remain separate packages.
@@ -0,0 +1,85 @@
"""Admit the operator oracle-canary scope supplement for #2618."""
import json
import os
import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
from rune import construction_activation as cact # noqa: E402
from rune import construction_admission as cadm # noqa: E402
from rune import construction_mode as cmode # noqa: E402
from rune import scaffold as sc # noqa: E402
ROOT = os.path.join(os.path.dirname(__file__), "..")
OBJECTIVE = (
"Keep the five direct scripted operator canaries green after the default "
"RuneOperator becomes a Go client. Select the retained Python dispatch "
"explicitly with core='python' in those oracle canaries; add no fallback "
"and change no production routing. Covers canary_p1_01/02/03/06/07."
)
AFFECTED = [
"plans/2618-operator-residual.md",
"scripts/admit_wave_operator_oracle_canaries.py",
"src/rune/canary_p1_01.py",
"src/rune/canary_p1_02.py",
"src/rune/canary_p1_03.py",
"src/rune/canary_p1_06.py",
"src/rune/canary_p1_07.py",
"src/rune/rune_operator.py",
"tests/test_operator_native.py",
]
def main() -> int:
store = os.path.join(ROOT, "store", "scaffold")
scaffold = sc.Scaffold(store, "rune")
mode = cmode.classify_mode(
OBJECTIVE,
target_project="rune",
project_exists=True,
affected_surface=AFFECTED,
)
need = {
"need_id": "need-2618-operator-oracle-canaries",
"objective": OBJECTIVE,
"origin": "worker",
"project": "rune",
"task_ref": "conductor/rune#2618",
"affected_surface": AFFECTED,
"evidence_refs": ["canary_p1_01 reached the live Go core and failed no_provider before the explicit oracle pin"],
}
activation = cact.activate(mode, need=need, scaffold=scaffold, builder="mimo-2618-operator-residual")
envelope = cadm.admit(
activation,
objective=OBJECTIVE,
builder="mimo-2618-operator-residual",
affected_surface=AFFECTED,
evidence_refs=need["evidence_refs"],
origin="worker",
task_ref="conductor/rune#2618",
expected_new_entities=[],
expected_runtime_effects=[
"direct scripted canaries select the independent Python oracle explicitly and never depend on a live core",
"the production default remains the Go client with no Python dispatch fallback",
],
verification_obligations=[
"python3 src/rune/canary_p1_01.py",
"python3 src/rune/canary_p1_02.py",
"python3 src/rune/canary_p1_03.py",
"python3 src/rune/canary_p1_06.py",
"python3 src/rune/canary_p1_07.py",
"pytest tests/test_operator_native.py",
],
regression_obligations=["no implicit Python fallback and no Python authority deletion in this tranche"],
approval_ref="user:conductor/rune#2618 residual operator conversion dispatch",
contract_id="ADMIT-2618-operator-oracle-canaries",
project_store=store,
project_id="rune",
)
print(json.dumps(envelope, indent=2, default=str)[:2200])
return 0 if envelope.get("success") else 1
if __name__ == "__main__":
raise SystemExit(main())
+134
View File
@@ -0,0 +1,134 @@
"""Admit the bounded residual-operator conversion for conductor/rune#2618."""
import json
import os
import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
from rune import construction_activation as cact # noqa: E402
from rune import construction_admission as cadm # noqa: E402
from rune import construction_mode as cmode # noqa: E402
from rune import scaffold as sc # noqa: E402
ROOT = os.path.join(os.path.dirname(__file__), "..")
OBJECTIVE = (
"Convert the residual operator serving four capability-by-capability onto "
"rune-core: deterministic task intake/family classification and route-client "
"handoff, canonical task lifecycle and nonterminal resume, durable event "
"append/replay, atomic task checkpoints, task-scoped conversation records, "
"and Rust record/transition semantics. Prove duplicate-signal idempotency, "
"kill-and-resume parity, and a real no-Python-operator serving turn. Keep "
"the Python operator as a thin fail-loud compatibility client and retain "
"its old bodies only behind an explicit oracle pin for the final retirement "
"tranche. Exclude construction/research execution, provider-stack conversion, "
"background memory, and new services for shared orchestration modules."
)
AFFECTED = [
"plans/2618-operator-residual.md",
"scripts/admit_wave_operator_residual.py",
"src/go/rune-core/",
"src/rust/operator_lifecycle/",
"src/rune/rune_operator.py",
"src/rune/orchestration/atomic.py",
"src/rune/orchestration/conversation_state.py",
"src/rune/orchestration/event_store.py",
"src/rune/orchestration/records.py",
"tests/conftest.py",
"tests/test_operator_native.py",
"verification/operator-lifecycle-serving.json",
]
EVIDENCE = [
"conductor/rune#2618 residual-operator dispatch",
"docs/language-dichotomy-ledger.md (Go reactive orchestration; Rust complex deterministic decisions)",
"plans/total-rebase-inventory.md (rune_operator is the residual Go authority)",
"remainder inventory T54: only atomic/conversation_state/event_store/records have direct operator consumers",
"PR#2632 eaa5c907 governed context assembly is the live turn seam; do not duplicate it",
"operator-deep explicitly excludes SQLite/model infrastructure and is not the serving host",
]
VERIFICATION = [
"cargo test and clippy -D warnings in src/rust/operator_lifecycle",
"go test ./..., go test -race ./..., and go vet ./... in src/go/rune-core",
"real file atomicity and event replay tests, including corrupt-input and duplicate-id cases",
"kill-and-resume tests for research_required and awaiting_approval with exact fresh-process state",
"duplicate completed-turn signals add no events/messages and make no second model call",
"real HTTP serving turn names go-operator, rust-operator-lifecycle, and go-event-store; Python sentinels stay absent",
"operator/task-lifecycle and complete orchestration pytest suites remain green under the explicit Python oracle pin",
"default Python no-oracle client tests prove no independent dispatch and typed native failures",
"ruff, henxels check, and python3 scripts/dichotomy_audit.py",
]
def main() -> int:
store = os.path.join(ROOT, "store", "scaffold")
scaffold = sc.Scaffold(store, "rune")
mode = cmode.classify_mode(
OBJECTIVE,
target_project="rune",
project_exists=True,
affected_surface=AFFECTED,
)
print("classified mode:", mode.mode)
need = {
"need_id": "need-2618-operator-residual",
"objective": OBJECTIVE,
"origin": "worker",
"project": "rune",
"task_ref": "conductor/rune#2618",
"affected_surface": AFFECTED,
"evidence_refs": EVIDENCE,
}
activation = cact.activate(
mode,
need=need,
scaffold=scaffold,
builder="mimo-2618-operator-residual",
)
print("activated:", activation.get("activated"), activation.get("mode"))
envelope = cadm.admit(
activation,
objective=OBJECTIVE,
builder="mimo-2618-operator-residual",
affected_surface=AFFECTED,
evidence_refs=EVIDENCE,
origin="worker",
task_ref="conductor/rune#2618",
expected_new_entities=[
"src/rust/operator_lifecycle/src/lib.rs",
"src/rust/operator_lifecycle/src/main.rs",
"src/go/rune-core/internal/runecore/operator.go",
"src/go/rune-core/internal/runecore/operator_tasks.go",
"src/go/rune-core/internal/runecore/operator_events.go",
"src/go/rune-core/internal/runecore/operator_atomic.go",
"src/go/rune-core/internal/runecore/operator_conversation.go",
"src/go/rune-core/internal/runecore/operator_native.go",
"src/go/rune-core/internal/runecore/operator_records.go",
"tests/test_operator_native.py",
"verification/operator-lifecycle-serving.json",
],
expected_runtime_effects=[
"every composed Go turn creates or amends a canonical task and persists task-linked user/assistant messages",
"normal turns transition created to running to completed; errors terminalize rather than stranding running tasks",
"research dispositions remain nonterminal and resume to running; awaiting approval resumes by the same legal edge",
"completed-turn signal receipts and the final transition share one atomic checkpoint; retries are idempotent after restart",
"status and task listing derive from canonical task records rather than zero placeholders or the legacy cache",
"the default Python operator calls the Go host and never enters its independent dispatch body",
],
verification_obligations=VERIFICATION,
regression_obligations=[
"PR#2632 governed context assembly and the canonical memory store remain composed and unchanged in authority",
"construction and research execution remain client boundaries owned by their epics",
"the other 41 orchestration modules gain no new services or authorities",
"Python authority remains available only as an explicit independent oracle in this tranche",
],
approval_ref="user:conductor/rune#2618 residual operator conversion dispatch",
contract_id="ADMIT-2618-operator-residual",
project_store=store,
project_id="rune",
)
print(json.dumps(envelope, indent=2, default=str)[:3000])
return 0 if envelope.get("success") else 1
if __name__ == "__main__":
raise SystemExit(main())
+4 -4
View File
@@ -83,13 +83,13 @@ def sweep(perturb=False):
seed_db(path, rows)
captured = oracle_rows(path)
want = with_store(path)(oracle.list_neurons)()
want = with_store(path)(oracle._list_neurons_python)()
if perturb and name == "single-full":
want = [{**want[0], "slug": "PERTURBED"}]
compare(f"list_neurons[{name}]",
ffi(lib.wns_list_neurons, captured), want)
want = with_store(path)(oracle.neuron_slugs)()
want = with_store(path)(oracle._neuron_slugs_python)()
if perturb and name == "order-by-slug":
want = want[::-1]
compare(f"neuron_slugs[{name}]",
@@ -98,12 +98,12 @@ def sweep(perturb=False):
from gen_warm_neuron_store_truth import digest_rows
drows = digest_rows(path)
if drows and all(r["slug"] for r in drows):
want = with_store(path)(oracle.neuron_digests)()
want = with_store(path)(oracle._neuron_digests_python)()
got = ffi(lib.wns_neuron_digests, drows)
compare(f"neuron_digests[{name}]", got, want)
for pname, term in HAS_PROBES:
want = with_store(path)(oracle.neuron_has)(term)
want = with_store(path)(oracle._neuron_has_python)(term)
if perturb and name == "single-full" and pname == "ascii-case-text":
want = not want
got = ffi(lib.wns_neuron_has, {"rows": captured, "term": term})
+1
View File
@@ -154,6 +154,7 @@ def module_to_keys(module_stem: str) -> list[str]:
# without this alias the audit mislabels it NOT TOUCHED.
"curator": ["curator-core"],
"memory_store": ["memory-ops", "memory-store-deep"],
"memory_native": ["memory-store-deep"],
"warm_router": ["warm-router", "operator-support"],
"tool_executor": ["tool-executor", "local-tool-runtime", "tool-protocol", "tool-protocol-rw"],
"local_tool_runtime": ["local-tool-runtime", "operator"],
+4 -4
View File
@@ -174,22 +174,22 @@ def build_cases():
inp = {"rows": copy.deepcopy(captured)}
groups["list_neurons"].append({
"name": name, "input": inp,
"expected": with_store(path)(oracle.list_neurons)()})
"expected": with_store(path)(oracle._list_neurons_python)()})
groups["neuron_slugs"].append({
"name": name, "input": inp,
"expected": with_store(path)(oracle.neuron_slugs)()})
"expected": with_store(path)(oracle._neuron_slugs_python)()})
if all(r["slug"] for r in digest_rows(path)) or not digest_rows(path):
groups["neuron_digests"].append({
"name": name,
"input": {"rows": digest_rows(path)},
"expected": with_store(path)(oracle.neuron_digests)()})
"expected": with_store(path)(oracle._neuron_digests_python)()})
probes = []
for pname, term in HAS_PROBES:
probes.append({
"name": pname, "term": term,
"expected": with_store(path)(oracle.neuron_has)(term),
"expected": with_store(path)(oracle._neuron_has_python)(term),
})
groups["neuron_has"].append({
"name": name, "input": inp, "probes": probes})
+4 -4
View File
@@ -1,9 +1,9 @@
package main
// broker.go — the ContextBroker orchestration core (admission, collection,
// rendering) from src/rune/context_broker.py, gap-close dichotomy wave.
// The Python class stays the runtime authority; this is the deterministic
// kernel with the contributor callbacks and failure recorder injectable.
// broker.go — Go collection/orchestration for the ContextBroker. Live eval
// admission, rendering, and budget decisions cross to attention_capsule's
// Rust authority in main.go. The pure helpers below remain only for frozen
// oracle replay and typed Go contributor compatibility.
import (
"errors"
+179 -69
View File
@@ -1,8 +1,8 @@
// Package main — context-broker: the GO port of src/rune/context_broker.py's
// budget/estimator kernels (Oracle-Battery Parity Pattern, #2479 GO batch 38;
// gap-close dichotomy wave — factory-gap-ctxbroker).
// The ContextBroker itself stays Python (it's the orchestration authority);
// the budget math is the deterministic core. The gap-close ports the missing
// Package main — context-broker: Go collection and native-process
// orchestration for src/rune/context_broker.py. Live admission, rendering,
// and budget decisions delegate to the attention_capsule Rust authority;
// pure Go helpers remain for frozen-oracle parity and injectable collection.
// The gap-close ports the missing
// functions name-preserving: the _py_* parity-floor oracles, the discovered-
// window registry, the rolling-summary store, _through_id, log_calibration,
// and the compact_history/summarize_history state machines with IO behind
@@ -10,11 +10,16 @@
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/json"
"fmt"
"math"
"os"
"os/exec"
"path/filepath"
"runtime"
"sort"
"strconv"
"strings"
@@ -683,10 +688,167 @@ func init() {
out := evalOne(payload)
blob, _ := json.Marshal(out)
fmt.Println(string(blob))
if object, ok := out.(map[string]interface{}); ok {
code, _ := object["error"].(string)
switch code {
case "NATIVE_UNAVAILABLE":
os.Exit(127)
case "PROTOCOL_ERROR":
os.Exit(2)
case "CONTEXT_UNKNOWN", "CONTEXT_DECISION_ERROR":
os.Exit(1)
}
}
os.Exit(0)
}
}
type nativeDecisionError struct {
code string
message string
}
func (e *nativeDecisionError) Error() string { return e.code + ": " + e.message }
func attentionCapsuleBinary() string {
if binary := os.Getenv("RUNE_ATTENTION_CAPSULE_BIN"); binary != "" {
return binary
}
_, file, _, ok := runtime.Caller(0)
if !ok {
return ""
}
dir := filepath.Dir(file)
for i := 0; i < 3; i++ {
dir = filepath.Dir(dir)
}
return filepath.Join(dir, "src", "rust", "attention_capsule", "target", "release", "attention_capsule")
}
func normalizeNativeNumbers(value interface{}) interface{} {
switch item := value.(type) {
case json.Number:
if integer, err := item.Int64(); err == nil {
return int(integer)
}
floating, _ := item.Float64()
return floating
case map[string]interface{}:
for key, child := range item {
item[key] = normalizeNativeNumbers(child)
}
case []interface{}:
for index, child := range item {
item[index] = normalizeNativeNumbers(child)
}
}
return value
}
func nativeContextCall(operation string, arg map[string]interface{}) (interface{}, map[string]interface{}, error) {
binary := attentionCapsuleBinary()
if binary == "" {
return nil, nil, &nativeDecisionError{code: "NATIVE_UNAVAILABLE", message: "attention-capsule native binary not found"}
}
info, err := os.Stat(binary)
if err != nil || info.IsDir() {
return nil, nil, &nativeDecisionError{code: "NATIVE_UNAVAILABLE", message: "attention-capsule native binary not found: " + binary}
}
payload, _ := json.Marshal(map[string]interface{}{"op": operation, "arg": arg})
cmd := exec.CommandContext(context.Background(), binary, "eval")
cmd.Stdin = bytes.NewReader(append(payload, '\n'))
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
runErr := cmd.Run()
var envelope struct {
OK bool `json:"ok"`
Value json.RawMessage `json:"value"`
Trace map[string]interface{} `json:"trace"`
Error *struct {
Code string `json:"code"`
Message string `json:"message"`
} `json:"error"`
}
if err := json.Unmarshal(stdout.Bytes(), &envelope); err != nil {
code := "PROTOCOL_ERROR"
if runErr != nil && stdout.Len() == 0 {
code = "NATIVE_UNAVAILABLE"
}
return nil, nil, &nativeDecisionError{code: code, message: fmt.Sprintf("invalid native envelope: %v (%s)", err, strings.TrimSpace(stderr.String()))}
}
if !envelope.OK {
code, message := "CONTEXT_DECISION_ERROR", "native context decision failed"
if envelope.Error != nil {
code, message = envelope.Error.Code, envelope.Error.Message
}
return nil, envelope.Trace, &nativeDecisionError{code: code, message: message}
}
if runErr != nil {
return nil, envelope.Trace, &nativeDecisionError{code: "PROTOCOL_ERROR", message: runErr.Error()}
}
decoder := json.NewDecoder(bytes.NewReader(envelope.Value))
decoder.UseNumber()
var value interface{}
if err := decoder.Decode(&value); err != nil {
return nil, envelope.Trace, &nativeDecisionError{code: "PROTOCOL_ERROR", message: "native value is not JSON"}
}
return normalizeNativeNumbers(value), envelope.Trace, nil
}
func nativeEvalResult(operation string, request map[string]interface{}, value interface{}, trace map[string]interface{}, err error) interface{} {
if err != nil {
code, detail := "PROTOCOL_ERROR", err.Error()
if native, ok := err.(*nativeDecisionError); ok {
code, detail = native.code, native.message
}
legacyCode := code
if code == "CONTEXT_OVERFLOW" {
legacyCode = "ContextOverflowError"
}
return map[string]interface{}{"error": legacyCode, "detail": detail}
}
if operation == "admit" || operation == "render" {
source, _ := value.(map[string]interface{})
projected := map[string]interface{}{}
keys := []string{"token_count", "messages", "system_preamble", "user_content", "blocks"}
if operation == "admit" {
keys = []string{"admitted", "blocks", "admitted_total", "memory_budget", "memory_used"}
dropped := []string{}
if rows, ok := source["dropped"].([]interface{}); ok {
for _, row := range rows {
if record, ok := row.(map[string]interface{}); ok {
name, _ := record["name"].(string)
dropped = append(dropped, name)
}
}
}
projected["dropped"] = dropped
}
for _, key := range keys {
projected[key] = source[key]
}
value = projected
}
if traced, _ := request["trace"].(bool); traced {
if object, ok := value.(map[string]interface{}); ok {
object["native_context"] = trace["context"]
object["native_operation"] = trace["operation"]
}
}
return value
}
func evalNative(operation string, request map[string]interface{}, arg map[string]interface{}) interface{} {
value, trace, err := nativeContextCall(operation, arg)
if err == nil && (operation == "admit" || operation == "render") {
if _, ok := value.(map[string]interface{}); !ok {
err = &nativeDecisionError{code: "PROTOCOL_ERROR", message: operation + " native value must be an object"}
}
}
return nativeEvalResult(operation, request, value, trace, err)
}
func evalOne(p map[string]interface{}) interface{} {
op, _ := p["op"].(string)
switch op {
@@ -697,9 +859,9 @@ func evalOne(p map[string]interface{}) interface{} {
}
return PyDefaultEstimator(nil)
case "budget":
return PyDefaultBudget(int(p["window"].(float64)))
return evalNative("budget", p, map[string]interface{}{"window": p["window"]})
case "membud":
return PyDefaultMemoryBudget(int(p["window"].(float64)))
return evalNative("memory_budget", p, map[string]interface{}{"window": p["window"]})
case "memoff":
v, _ := p["value"].(string)
return PyMemoryDisabledValue(v)
@@ -713,71 +875,19 @@ func evalOne(p map[string]interface{}) interface{} {
return ThroughID(msgs, int(p["through"].(float64)))
case "admit":
rawEntries, _ := p["entries"].([]interface{})
entries := make([]AdmissionEntry, 0, len(rawEntries))
for _, re := range rawEntries {
e, _ := re.(map[string]interface{})
b := &ContributionBlock{}
b.Name, _ = e["name"].(string)
b.Priority = int(e["priority"].(float64))
b.Tokens = int(e["tokens"].(float64))
b.Source, _ = e["source"].(string)
b.Evictable, _ = e["evictable"].(bool)
entries = append(entries, AdmissionEntry{Name: b.Name, Block: b})
}
budget := int(p["budget"].(float64))
memBudget := int(p["memory_budget"].(float64))
res, err := Admit(entries, budget, memBudget)
if err != nil {
return map[string]interface{}{"error": "ContextOverflowError", "detail": err.Error()}
}
blocksOut := make([]map[string]interface{}, 0, len(res.Blocks))
for _, r := range res.Blocks {
blocksOut = append(blocksOut, map[string]interface{}{
"name": r.Name, "priority": r.Priority, "tokens": r.Tokens,
"source": r.Source, "evictable": r.Evictable,
"admitted": r.Admitted, "reason": r.Reason,
})
}
dropped := make([]string, 0, len(res.Dropped))
for _, d := range res.Dropped {
dropped = append(dropped, d.Name)
}
return map[string]interface{}{
"admitted": blockNames(res.Admitted), "blocks": blocksOut,
"admitted_total": res.AdmittedTotal, "dropped": dropped,
"memory_budget": res.MemoryBudget, "memory_used": res.MemoryUsed,
for _, raw := range rawEntries {
if entry, ok := raw.(map[string]interface{}); ok {
if _, exists := entry["content"]; !exists {
entry["content"] = ""
}
}
}
return evalNative("admit", p, map[string]interface{}{
"entries": rawEntries, "budget": p["budget"], "memory_budget": p["memory_budget"],
})
case "render":
rawBlocks, _ := p["blocks"].([]interface{})
admitted := make([]*ContributionBlock, 0, len(rawBlocks))
for _, rb := range rawBlocks {
b, _ := rb.(map[string]interface{})
blk := &ContributionBlock{}
blk.Name, _ = b["name"].(string)
blk.Priority = int(b["priority"].(float64))
blk.Tokens = int(b["tokens"].(float64))
blk.Source, _ = b["source"].(string)
switch c := b["content"].(type) {
case string:
blk.Content = c
case []interface{}:
msgList := make([]Message, 0, len(c))
for _, rm := range c {
m, _ := rm.(map[string]interface{})
msgList = append(msgList, m)
}
blk.Content = msgList
}
admitted = append(admitted, blk)
}
capsule := Render(admitted)
return map[string]interface{}{
"token_count": capsule.TokenCount,
"messages": capsule.Messages,
"system_preamble": capsule.SystemPreamble,
"user_content": capsule.UserContent,
"blocks": blockNames(capsule.Blocks),
}
return evalNative("render", p, map[string]interface{}{"blocks": rawBlocks})
}
return nil
}
+61
View File
@@ -5,6 +5,8 @@ package main
import (
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"sync"
@@ -93,6 +95,65 @@ func TestRollingStoreRaceFreeAndSnapshot(t *testing.T) {
}
}
func TestEvalAdmissionUsesNativeRust(t *testing.T) {
dir := t.TempDir()
capture := filepath.Join(dir, "request.json")
binary := filepath.Join(dir, "attention-capsule")
response := `{"ok":true,"value":{"admitted":["user"],"blocks":[],"admitted_total":1,"dropped":[],"memory_budget":10,"memory_used":0},"trace":{"context":"rust-attention-capsule","operation":"admit"}}`
script := "#!/bin/sh\ncat >\"" + capture + "\"\nprintf '%s\\n' '" + response + "'\n"
if err := os.WriteFile(binary, []byte(script), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("RUNE_ATTENTION_CAPSULE_BIN", binary)
got := evalOne(map[string]interface{}{
"op": "admit", "trace": true, "budget": float64(10), "memory_budget": float64(10),
"entries": []interface{}{map[string]interface{}{"name": "user", "priority": float64(100), "tokens": float64(1), "source": "test", "evictable": false}},
})
result, ok := got.(map[string]interface{})
if !ok || result["native_context"] != "rust-attention-capsule" {
t.Fatalf("admission did not use Rust: %v", got)
}
raw, err := os.ReadFile(capture)
if err != nil || !strings.Contains(string(raw), `"op":"admit"`) {
t.Fatalf("native admission request=%s err=%v", raw, err)
}
}
func TestEvalBudgetAndMissingNativeDoNotUseGoFallback(t *testing.T) {
dir := t.TempDir()
binary := filepath.Join(dir, "attention-capsule")
response := `{"ok":true,"value":123,"trace":{"context":"rust-attention-capsule","operation":"budget"}}`
if err := os.WriteFile(binary, []byte("#!/bin/sh\ncat >/dev/null\nprintf '%s\\n' '"+response+"'\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("RUNE_ATTENTION_CAPSULE_BIN", binary)
if got := evalOne(map[string]interface{}{"op": "budget", "window": float64(8192)}); got != 123 {
t.Fatalf("budget bypassed Rust: %v", got)
}
t.Setenv("RUNE_ATTENTION_CAPSULE_BIN", filepath.Join(dir, "missing"))
got := evalOne(map[string]interface{}{"op": "admit", "budget": float64(10), "memory_budget": float64(10), "entries": []interface{}{}})
result, ok := got.(map[string]interface{})
if !ok || result["error"] != "NATIVE_UNAVAILABLE" {
t.Fatalf("missing native silently used Go admission: %v", got)
}
}
func TestEvalNativeRejectsMalformedDecisionValue(t *testing.T) {
dir := t.TempDir()
binary := filepath.Join(dir, "attention-capsule")
response := `{"ok":true,"value":[],"trace":{"context":"rust-attention-capsule","operation":"admit"}}`
if err := os.WriteFile(binary, []byte("#!/bin/sh\ncat >/dev/null\nprintf '%s\\n' '"+response+"'\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("RUNE_ATTENTION_CAPSULE_BIN", binary)
got := evalOne(map[string]interface{}{"op": "admit", "budget": float64(10), "memory_budget": float64(10), "entries": []interface{}{}})
result, ok := got.(map[string]interface{})
if !ok || result["error"] != "PROTOCOL_ERROR" {
t.Fatalf("malformed native value became an empty admission: %v", got)
}
}
func TestBlockSourceRegistryCompileTime(t *testing.T) {
var src BlockSource = BlockSourceFunc{
name: "tools",
-23
View File
@@ -1,26 +1,3 @@
module rune/memory-ops
go 1.21
require modernc.org/sqlite v1.28.0
require (
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/mattn/go-isatty v0.0.16 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
golang.org/x/mod v0.3.0 // indirect
golang.org/x/sys v0.9.0 // indirect
golang.org/x/tools v0.0.0-20201124115921-2c860bdd6e78 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
lukechampine.com/uint128 v1.2.0 // indirect
modernc.org/cc/v3 v3.40.0 // indirect
modernc.org/ccgo/v3 v3.16.13 // indirect
modernc.org/libc v1.29.0 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.7.2 // indirect
modernc.org/opt v0.1.3 // indirect
modernc.org/strutil v1.1.3 // indirect
modernc.org/token v1.0.1 // indirect
)
-73
View File
@@ -1,73 +0,0 @@
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/pprof v0.0.0-20221118152302-e6195bd50e26 h1:Xim43kblpZXfIBQsbuBVKCudVG457BR2GZFIz3uw3hQ=
github.com/google/pprof v0.0.0-20221118152302-e6195bd50e26/go.mod h1:dDKJzRmX4S37WGHujM7tX//fmj1uioxKzKxz3lo4HJo=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 h1:Z9n2FFNUXsshfwJMBgNA0RU6/i7WVaAegv3PtuIHPMs=
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51/go.mod h1:CzGEWj7cYgsdH8dAjBGEr58BoE7ScuLd+fwFZ44+/x8=
github.com/mattn/go-isatty v0.0.16 h1:bq3VjFmv/sOjHtdEhmkEV4x1AJtvUvOJ2PFAZ5+peKQ=
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/mattn/go-sqlite3 v1.14.16 h1:yOQRA0RpS5PFz/oikGwBEqvAWhWg5ufRz4ETLjwpU1Y=
github.com/mattn/go-sqlite3 v1.14.16/go.mod h1:2eHXhiwb8IkHr+BDWZGa96P6+rkvnG63S2DGjv9HUNg=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.9.0 h1:KS/R3tvhPqvJvwcKfnBHJwwthS11LRhmM5D59eEXa0s=
golang.org/x/sys v0.9.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20201124115921-2c860bdd6e78 h1:M8tBwCtWD/cZV9DZpFYRUgaymAYAr+aIUTWzDaM3uPs=
golang.org/x/tools v0.0.0-20201124115921-2c860bdd6e78/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
lukechampine.com/uint128 v1.2.0 h1:mBi/5l91vocEN8otkC5bDLhi2KdCticRiwbdB0O+rjI=
lukechampine.com/uint128 v1.2.0/go.mod h1:c4eWIwlEGaxC/+H1VguhU4PHXNWDCDMUlWdIWl2j1gk=
modernc.org/cc/v3 v3.40.0 h1:P3g79IUS/93SYhtoeaHW+kRCIrYaxJ27MFPv+7kaTOw=
modernc.org/cc/v3 v3.40.0/go.mod h1:/bTg4dnWkSXowUO6ssQKnOV0yMVxDYNIsIrzqTFDGH0=
modernc.org/ccgo/v3 v3.16.13 h1:Mkgdzl46i5F/CNR/Kj80Ri59hC8TKAhZrYSaqvkwzUw=
modernc.org/ccgo/v3 v3.16.13/go.mod h1:2Quk+5YgpImhPjv2Qsob1DnZ/4som1lJTodubIcoUkY=
modernc.org/ccorpus v1.11.6 h1:J16RXiiqiCgua6+ZvQot4yUuUy8zxgqbqEEUuGPlISk=
modernc.org/ccorpus v1.11.6/go.mod h1:2gEUTrWqdpH2pXsmTM1ZkjeSrUWDpjMu2T6m29L/ErQ=
modernc.org/httpfs v1.0.6 h1:AAgIpFZRXuYnkjftxTAZwMIiwEqAfk8aVB2/oA6nAeM=
modernc.org/httpfs v1.0.6/go.mod h1:7dosgurJGp0sPaRanU53W4xZYKh14wfzX420oZADeHM=
modernc.org/libc v1.29.0 h1:tTFRFq69YKCF2QyGNuRUQxKBm1uZZLubf6Cjh/pVHXs=
modernc.org/libc v1.29.0/go.mod h1:DaG/4Q3LRRdqpiLyP0C2m1B8ZMGkQ+cCgOIjEtQlYhQ=
modernc.org/mathutil v1.6.0 h1:fRe9+AmYlaej+64JsEEhoWuAYBkOtQiMEU7n/XgfYi4=
modernc.org/mathutil v1.6.0/go.mod h1:Ui5Q9q1TR2gFm0AQRqQUaBWFLAhQpCwNcuhBOSedWPo=
modernc.org/memory v1.7.2 h1:Klh90S215mmH8c9gO98QxQFsY+W451E8AnzjoE2ee1E=
modernc.org/memory v1.7.2/go.mod h1:NO4NVCQy0N7ln+T9ngWqOQfi7ley4vpwvARR+Hjw95E=
modernc.org/opt v0.1.3 h1:3XOZf2yznlhC+ibLltsDGzABUGVx8J6pnFMS3E4dcq4=
modernc.org/opt v0.1.3/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0=
modernc.org/sqlite v1.28.0 h1:Zx+LyDDmXczNnEQdvPuEfcFVA2ZPyaD7UCZDjef3BHQ=
modernc.org/sqlite v1.28.0/go.mod h1:Qxpazz0zH8Z1xCFyi5GSL3FzbtZ3fvbjmywNogldEW0=
modernc.org/strutil v1.1.3 h1:fNMm+oJklMGYfU9Ylcywl0CO5O6nTfaowNsh2wpPjzY=
modernc.org/strutil v1.1.3/go.mod h1:MEHNA7PdEnEwLvspRMtWTNnp2nnyvMfkimT1NKNAGbw=
modernc.org/tcl v1.15.2 h1:C4ybAYCGJw968e+Me18oW55kD/FexcHbqH2xak1ROSY=
modernc.org/tcl v1.15.2/go.mod h1:3+k/ZaEbKrC8ePv8zJWPtBSW0V7Gg9g8rkmhI1Kfs3c=
modernc.org/token v1.0.1 h1:A3qvTqOwexpfZZeyI0FeGPDlSWX5pjZu9hF4lU+EKWg=
modernc.org/token v1.0.1/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
modernc.org/z v1.7.3 h1:zDJf6iHjrnB+WRD88stbXokugjyc0/pB91ri1gO6LZY=
modernc.org/z v1.7.3/go.mod h1:Ipv4tsdxZRbQyLq9Q1M6gdbkxYzdlrciF2Hi/lS7nWE=
+46 -207
View File
@@ -1,222 +1,61 @@
// memory-ops — Go SQLite memory operations (R2 W5c).
// Replaces the orchestration layer of src/rune/memory_store.py.
// Uses modernc.org/sqlite (pure Go, no cgo).
//
// Usage:
//
// memory-ops add <text> [--importance <f>] [--topic <t>] [--source <s>]
// memory-ops search <query> [--limit N]
// memory-ops get <id>
// memory-ops wash <id>
// memory-ops supersede <id> <new-text>
// memory-ops stats
// memory-ops is the compatibility launcher for the ONE canonical store.
// SQLite orchestration lives in src/go/memory-store-deep; this process owns
// no schema, retrieval policy, or fallback implementation.
package main
import (
"database/sql"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"time"
_ "modernc.org/sqlite"
"os/exec"
"path/filepath"
"runtime"
)
type Result struct {
Success bool `json:"success"`
Data interface{} `json:"data,omitempty"`
Error string `json:"error,omitempty"`
}
func dbPath() string {
home := os.Getenv("RUNE_MEMORY_DIR")
if home == "" {
home = os.Getenv("HOME")
func canonicalBinary() string {
if value := os.Getenv("RUNE_MEMORY_STORE_BIN"); value != "" {
return value
}
return home + "/.rune/mempalace/memories.db"
}
func connect() (*sql.DB, error) {
db, err := sql.Open("sqlite", dbPath())
if err != nil {
return nil, err
_, file, _, ok := runtime.Caller(0)
if ok {
root := filepath.Dir(filepath.Dir(filepath.Dir(filepath.Dir(file))))
candidate := filepath.Join(root, "src", "go", "memory-store-deep", "memory-store-deep")
if info, err := os.Stat(candidate); err == nil && !info.IsDir() {
return candidate
}
}
db.Exec("PRAGMA journal_mode=WAL")
db.Exec("PRAGMA synchronous=NORMAL")
return db, nil
path, _ := exec.LookPath("memory-store-deep")
return path
}
func nowISO() string { return time.Now().UTC().Format("2006-01-02T15:04:05.000000Z") }
func runCompatibility(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
binary := canonicalBinary()
if binary == "" {
_ = json.NewEncoder(stderr).Encode(map[string]interface{}{
"ok": false,
"error": map[string]string{
"code": "NATIVE_UNAVAILABLE", "message": "canonical memory-store-deep binary not found",
},
})
return 127
}
cmd := exec.Command(binary, args...)
cmd.Stdin = stdin
cmd.Stdout = stdout
cmd.Stderr = stderr
if err := cmd.Run(); err != nil {
var exit *exec.ExitError
if errors.As(err, &exit) {
return exit.ExitCode()
}
fmt.Fprintf(stderr, "NATIVE_UNAVAILABLE: canonical memory store failed to launch: %v\n", err)
return 127
}
return 0
}
func main() {
if len(os.Args) < 2 {
fmt.Fprintln(os.Stderr, "usage: memory-ops <add|search|get|wash|supersede|stats> [args...]")
os.Exit(1)
}
db, err := connect()
if err != nil {
out, _ := json.Marshal(Result{Success: false, Error: err.Error()})
fmt.Println(string(out))
os.Exit(1)
}
defer db.Close()
cmd := os.Args[1]
args := os.Args[2:]
switch cmd {
case "add":
text := ""
importance := 0.5
topic := ""
source := ""
for i := 0; i < len(args); i++ {
switch args[i] {
case "--importance":
if i+1 < len(args) {
fmt.Sscanf(args[i+1], "%f", &importance)
i++
}
case "--topic":
if i+1 < len(args) {
topic = args[i+1]
i++
}
case "--source":
if i+1 < len(args) {
source = args[i+1]
i++
}
default:
if text == "" {
text = args[i]
}
}
}
if len(text) < 8 {
out, _ := json.Marshal(Result{Success: false, Error: "text too short (min 8 chars)"})
fmt.Println(string(out))
return
}
now := nowISO()
var entryID int64
err = db.QueryRow(
`INSERT INTO entries (text, importance, created_at, last_accessed, topic, source, method, status, confidence, temperature, always_on, priority, tier)
VALUES (?,?,?,?,?,?,?,'active',0.7,1.0,0,5,'warm') RETURNING id`,
text, importance, now, now, topic, source, "go-operator").Scan(&entryID)
if err != nil {
out, _ := json.Marshal(Result{Success: false, Error: err.Error()})
fmt.Println(string(out))
return
}
out, _ := json.Marshal(Result{Success: true, Data: map[string]interface{}{
"id": entryID, "text": text, "importance": importance, "topic": topic}})
fmt.Println(string(out))
case "search":
query := ""
limit := 5
for i := 0; i < len(args); i++ {
switch args[i] {
case "--limit":
if i+1 < len(args) {
fmt.Sscanf(args[i+1], "%d", &limit)
i++
}
default:
if query == "" {
query = args[i]
}
}
}
rows, err := db.Query(
`SELECT id, text, importance, topic, created_at FROM entries
WHERE status='active' AND text LIKE ? ORDER BY importance DESC LIMIT ?`,
"%"+query+"%", limit)
if err != nil {
out, _ := json.Marshal(Result{Success: false, Error: err.Error()})
fmt.Println(string(out))
return
}
defer rows.Close()
var results []map[string]interface{}
for rows.Next() {
var id int64
var text, topic, createdAt string
var importance float64
rows.Scan(&id, &text, &importance, &topic, &createdAt)
results = append(results, map[string]interface{}{
"id": id, "text": text, "importance": importance,
"topic": topic, "created_at": createdAt})
}
out, _ := json.Marshal(Result{Success: true, Data: results})
fmt.Println(string(out))
case "get":
if len(args) < 1 {
fmt.Fprintln(os.Stderr, "usage: memory-ops get <id>")
os.Exit(1)
}
var id int64
fmt.Sscanf(args[0], "%d", &id)
var text, topic, createdAt string
var importance float64
err := db.QueryRow(
"SELECT text, importance, topic, created_at FROM entries WHERE id=? AND status='active'", id).
Scan(&text, &importance, &topic, &createdAt)
if err == sql.ErrNoRows {
out, _ := json.Marshal(Result{Success: false, Error: "not found"})
fmt.Println(string(out))
return
}
out, _ := json.Marshal(Result{Success: true, Data: map[string]interface{}{
"id": id, "text": text, "importance": importance, "topic": topic, "created_at": createdAt}})
fmt.Println(string(out))
case "wash":
if len(args) < 1 {
fmt.Fprintln(os.Stderr, "usage: memory-ops wash <id>")
os.Exit(1)
}
var id int64
fmt.Sscanf(args[0], "%d", &id)
db.Exec("UPDATE entries SET wash_count = wash_count + 1, last_accessed = ? WHERE id = ?", nowISO(), id)
out, _ := json.Marshal(Result{Success: true, Data: map[string]interface{}{"washed": id}})
fmt.Println(string(out))
case "supersede":
if len(args) < 2 {
fmt.Fprintln(os.Stderr, "usage: memory-ops supersede <id> <new-text>")
os.Exit(1)
}
var oldID int64
fmt.Sscanf(args[0], "%d", &oldID)
newText := args[1]
now := nowISO()
var newID int64
err = db.QueryRow(
`INSERT INTO entries (text, importance, created_at, last_accessed, topic, source, method, status, superseded_by, confidence, temperature, always_on, priority, tier)
SELECT ?, importance, ?, ?, topic, source, 'go-operator', 'active', ?, confidence, temperature, always_on, priority, tier FROM entries WHERE id = ? RETURNING id`,
newText, now, now, oldID, oldID).Scan(&newID)
if err != nil {
out, _ := json.Marshal(Result{Success: false, Error: err.Error()})
fmt.Println(string(out))
return
}
db.Exec("UPDATE entries SET status='superseded' WHERE id=?", oldID)
out, _ := json.Marshal(Result{Success: true, Data: map[string]interface{}{
"new_id": newID, "old_id": oldID, "text": newText}})
fmt.Println(string(out))
case "stats":
var total, active, superseded int
db.QueryRow("SELECT COUNT(*), SUM(CASE WHEN status='active' THEN 1 ELSE 0 END), SUM(CASE WHEN status='superseded' THEN 1 ELSE 0 END) FROM entries").Scan(&total, &active, &superseded)
out, _ := json.Marshal(Result{Success: true, Data: map[string]interface{}{
"total": total, "active": active, "superseded": superseded}})
fmt.Println(string(out))
default:
fmt.Fprintf(os.Stderr, "unknown command: %s\n", cmd)
os.Exit(1)
}
os.Exit(runCompatibility(os.Args[1:], os.Stdin, os.Stdout, os.Stderr))
}
+40
View File
@@ -0,0 +1,40 @@
package main
import (
"bytes"
"os"
"path/filepath"
"strings"
"testing"
)
func TestCompatibilityLauncherForwardsToCanonicalStore(t *testing.T) {
dir := t.TempDir()
canonical := filepath.Join(dir, "canonical-store")
argsTrace := filepath.Join(dir, "args")
script := "#!/bin/sh\nprintf '%s' \"$*\" >\"" + argsTrace + "\"\ncat >/dev/null\nprintf '%s\n' '{\"ok\":true,\"value\":{\"active_entries\":1}}'\nexit 0\n"
if err := os.WriteFile(canonical, []byte(script), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("RUNE_MEMORY_STORE_BIN", canonical)
var stdout, stderr bytes.Buffer
code := runCompatibility([]string{"stats", "--json"}, strings.NewReader(""), &stdout, &stderr)
if code != 0 || !strings.Contains(stdout.String(), `"active_entries":1`) {
t.Fatalf("launcher code=%d stdout=%s stderr=%s", code, stdout.String(), stderr.String())
}
raw, err := os.ReadFile(argsTrace)
if err != nil || string(raw) != "stats --json" {
t.Fatalf("forwarded args = %q (%v)", raw, err)
}
}
func TestCompatibilityLauncherMissingCanonicalFailsLoud(t *testing.T) {
t.Setenv("RUNE_MEMORY_STORE_BIN", filepath.Join(t.TempDir(), "missing"))
var stdout, stderr bytes.Buffer
if code := runCompatibility([]string{"stats"}, strings.NewReader(""), &stdout, &stderr); code != 127 {
t.Fatalf("missing canonical exit = %d, want 127", code)
}
if !strings.Contains(stderr.String(), "NATIVE_UNAVAILABLE") {
t.Fatalf("missing typed error: %s", stderr.String())
}
}
+421
View File
@@ -0,0 +1,421 @@
package main
import (
"database/sql"
"encoding/json"
"fmt"
"io"
"os"
"strconv"
"strings"
)
type commandOutcome struct {
value any
kernels []string
exitCode int
raw bool
}
type cliOptions struct {
command string
positional []string
values map[string]interface{}
}
var booleanCLIFlags = map[string]bool{
"no-dedupe": true, "json": true, "dry-run": true, "version": true,
}
var canonicalVerbs = map[string]bool{
"add": true, "recall": true, "update": true, "delete": true,
"working": true, "audit": true, "verify": true, "stats": true,
"mine": true, "chunk-recall": true, "wake": true, "export": true,
"associate": true, "linked": true, "scene": true, "hierarchy": true,
"query-core": true, "reindex": true, "hook": true,
}
func parseCLIOptions(args []string) cliOptions {
opts := cliOptions{command: args[0], values: map[string]interface{}{}}
for i := 1; i < len(args); i++ {
arg := args[i]
if !strings.HasPrefix(arg, "--") {
opts.positional = append(opts.positional, arg)
continue
}
key := strings.TrimPrefix(arg, "--")
if booleanCLIFlags[key] || i+1 >= len(args) || strings.HasPrefix(args[i+1], "--") {
opts.values[key] = true
continue
}
i++
opts.values[key] = args[i]
}
return opts
}
func (o cliOptions) arguments() map[string]interface{} {
args := map[string]interface{}{}
for key, value := range o.values {
args[key] = value
}
for _, key := range []string{"importance", "confidence", "temperature", "min-score", "min-sim"} {
if raw, ok := args[key].(string); ok {
if parsed, err := strconv.ParseFloat(raw, 64); err == nil {
args[key] = parsed
}
}
}
for _, key := range []string{"priority", "always-on", "budget", "probe-cap", "limit"} {
if raw, ok := args[key].(string); ok {
if parsed, err := strconv.Atoi(raw); err == nil {
args[key] = float64(parsed)
}
}
}
for alias, canonical := range map[string]string{
"memory-type": "memory_type", "memory-types": "memory_types", "always-on": "always_on",
"min-score": "min_score", "probe-cap": "probe_cap",
} {
if value, ok := args[alias]; ok {
args[canonical] = value
}
}
return args
}
func optionID(opts cliOptions, index int) int64 {
if index >= len(opts.positional) {
return 0
}
id, _ := strconv.ParseInt(opts.positional[index], 10, 64)
return id
}
func runCanonical(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
if len(args) == 0 {
writeCanonicalError(stderr, &StoreError{Code: "USAGE", Message: "memory-store-deep requires a verb or eval"})
return 2
}
if args[0] == "eval" {
return runCanonicalEval(stdin, stdout)
}
opts := parseCLIOptions(args)
if !canonicalVerbs[opts.command] && !map[string]bool{"search": true, "get": true, "wash": true, "supersede": true, "invalidate": true}[opts.command] {
writeCanonicalError(stderr, &StoreError{Code: "USAGE", Op: opts.command, Message: "unknown memory-store verb"})
return 2
}
db, err := connect()
if err != nil {
writeCanonicalError(stderr, err)
return exitForStoreError(err)
}
defer db.Close()
outcome, err := executeCanonicalCommand(db, opts)
if err != nil {
writeCanonicalError(stderr, err)
return exitForStoreError(err)
}
if outcome.raw {
fmt.Fprintln(stdout, outcome.value)
} else {
writeCanonicalJSON(stdout, outcome.value)
}
return outcome.exitCode
}
func executeCanonicalCommand(db *sql.DB, opts cliOptions) (commandOutcome, error) {
args := opts.arguments()
switch opts.command {
case "add":
text := strings.Join(opts.positional, " ")
args["text"] = text
if opts.values["no-dedupe"] == true {
result := addEntry(db, args)
if !result.Success {
return commandOutcome{value: map[string]interface{}{"added": false, "dedupe": "bypassed", "error": result.Error}, exitCode: 1}, nil
}
return commandOutcome{value: map[string]interface{}{"added": true, "dedupe": "bypassed"}}, nil
}
return nativeDedupeAdd(db, args)
case "recall", "search":
args["query"] = strings.Join(opts.positional, " ")
entries, kernels, err := nativeRecallWithAssociations(db, args)
if err != nil {
return commandOutcome{}, err
}
research, chunkKernels, err := nativeChunkRecallEntries(db, args)
kernels = append(kernels, chunkKernels...)
if err != nil {
return commandOutcome{}, err
}
fused, err := nativeFuseResults(entries, research, retrievalBin())
if err != nil {
return commandOutcome{}, err
}
kernels = append(kernels, "memory_retrieval.fuse_results")
return commandOutcome{value: fused, kernels: kernels}, nil
case "update":
args["id"] = float64(optionID(opts, 0))
if text, ok := opts.values["text"].(string); ok {
args["text"] = text
} else if len(opts.positional) > 1 {
args["text"] = strings.Join(opts.positional[1:], " ")
}
var topic sql.NullString
if err := db.QueryRow("SELECT topic FROM entries WHERE id=?", optionID(opts, 0)).Scan(&topic); err != nil && err != sql.ErrNoRows {
return commandOutcome{}, err
}
if topic.Valid && topic.String == "constitution" {
return commandOutcome{value: map[string]interface{}{"updated": false, "reason": "constitution is read_only — only constitution-add.py may write it"}}, nil
}
result := updateEntry(db, args)
if !result.Success {
return commandOutcome{}, &StoreError{Code: "STORE_ERROR", Op: "update", Message: result.Error}
}
return commandOutcome{value: map[string]interface{}{"updated": result.Data}}, nil
case "delete":
if err := nativeDeleteEntry(db, optionID(opts, 0)); err != nil {
return commandOutcome{}, err
}
return commandOutcome{value: map[string]interface{}{"deleted": true}}, nil
case "working":
if len(opts.positional) == 0 {
return commandOutcome{}, &StoreError{Code: "USAGE", Op: "working", Message: "working requires a task id"}
}
args["task_id"] = opts.positional[0]
if len(opts.positional) > 1 {
args["text"] = strings.Join(opts.positional[1:], " ")
result := workingSet(db, args)
if !result.Success {
return commandOutcome{}, &StoreError{Code: "STORE_ERROR", Op: "working", Message: result.Error}
}
return commandOutcome{value: map[string]interface{}{"working_set": true}}, nil
}
result := workingGet(db, args)
if !result.Success {
return commandOutcome{}, &StoreError{Code: "STORE_ERROR", Op: "working", Message: result.Error}
}
return commandOutcome{value: result.Data}, nil
case "audit":
parts := append(append([]string{}, opts.positional...), "", "", "")
hash, err := nativeAuditAppend(db, parts[0], parts[1], parts[2])
if err != nil {
return commandOutcome{}, err
}
return commandOutcome{value: map[string]interface{}{"hash": hash}}, nil
case "verify":
receipt, err := nativeAuditVerify(db)
if err != nil {
return commandOutcome{}, err
}
code := 0
if receipt["ok"] != true {
code = 1
}
return commandOutcome{value: receipt, exitCode: code}, nil
case "stats":
stats, err := nativeStats(db)
return commandOutcome{value: stats}, err
case "associate":
a, b := optionID(opts, 0), optionID(opts, 1)
associate(db, a, b, 0.1)
return commandOutcome{value: map[string]interface{}{"associated": []int64{a, b}}}, nil
case "linked":
return commandOutcome{value: linkedEntries(db, optionID(opts, 0), 5)}, nil
case "scene":
if len(opts.positional) > 0 && opts.positional[0] == "list" {
return commandOutcome{value: listScenes(db)}, nil
}
if len(opts.positional) >= 2 {
ids := []int64{}
for _, raw := range opts.positional[1:] {
id, _ := strconv.ParseInt(raw, 10, 64)
ids = append(ids, id)
}
id := makeScene(db, opts.positional[0], ids, "")
return commandOutcome{value: map[string]interface{}{"scene_id": id, "members": len(ids)}}, nil
}
return commandOutcome{value: nil}, nil
case "hierarchy":
return commandOutcome{value: hierarchy(db, 3)}, nil
case "query-core":
rows, err := db.Query("SELECT id, text, topic, priority, confidence FROM entries WHERE always_on=1 AND status='active' ORDER BY priority ASC")
if err != nil {
return commandOutcome{}, err
}
defer rows.Close()
out := []map[string]interface{}{}
for rows.Next() {
var id, priority int64
var text, topic string
var confidence float64
if err := rows.Scan(&id, &text, &topic, &priority, &confidence); err != nil {
return commandOutcome{}, err
}
if topic == "" {
topic = "memory"
}
out = append(out, map[string]interface{}{"id": id, "text": text, "topic": topic, "priority": priority, "confidence": confidence})
}
return commandOutcome{value: out}, rows.Err()
case "wake":
return commandOutcome{value: wakeUp(db, 800), raw: true}, nil
case "export":
rows, err := db.Query("SELECT chunk_id, wing, room, source_path, text FROM chunks")
if err != nil {
return commandOutcome{}, err
}
defer rows.Close()
out := []map[string]interface{}{}
for rows.Next() {
var id, wing, room, source, text string
if err := rows.Scan(&id, &wing, &room, &source, &text); err != nil {
return commandOutcome{}, err
}
out = append(out, map[string]interface{}{"chunk_id": id, "wing": wing, "room": room, "source": source, "text": text})
}
return commandOutcome{value: out}, rows.Err()
case "reindex":
return commandOutcome{value: ReindexEntries(db, opts.values["dry-run"] == true)}, nil
case "mine":
if len(opts.positional) == 0 {
return commandOutcome{}, &StoreError{Code: "USAGE", Op: "mine", Message: "mine needs a file or directory"}
}
target := opts.positional[0]
info, err := os.Stat(target)
if err != nil {
return commandOutcome{}, err
}
wing, _ := opts.values["wing"].(string)
room, _ := opts.values["room"].(string)
if info.IsDir() {
return commandOutcome{value: mineDirectory(db, target, wing, room, 0)}, nil
}
added, message := mineFile(db, target, wing, room, 1500)
return commandOutcome{value: []interface{}{added, message}}, nil
case "chunk-recall":
args["query"] = strings.Join(opts.positional, " ")
args["min_sim"] = opts.values["min-sim"]
rows, kernels, err := nativeChunkRecallEntries(db, args)
return commandOutcome{value: rows, kernels: kernels}, err
case "hook":
event, _ := opts.values["event"].(string)
if event == "" {
event = "unknown"
}
var data interface{}
if raw, ok := opts.values["data"].(string); ok {
_ = json.Unmarshal([]byte(raw), &data)
}
return commandOutcome{value: handleHook(event, data)}, nil
case "wash":
return commandOutcome{value: incrementWash(db, optionID(opts, 0))}, nil
case "get":
id := optionID(opts, 0)
var text, topic string
if err := db.QueryRow("SELECT text, topic FROM entries WHERE id=? AND status='active'", id).Scan(&text, &topic); err != nil {
return commandOutcome{}, err
}
return commandOutcome{value: map[string]interface{}{"id": id, "text": text, "topic": topic}}, nil
case "supersede":
args["id"] = float64(optionID(opts, 0))
args["text"] = strings.Join(opts.positional[1:], " ")
result := supersedeEntry(db, args)
if !result.Success {
return commandOutcome{}, &StoreError{Code: "STORE_ERROR", Op: "supersede", Message: result.Error}
}
return commandOutcome{value: result.Data}, nil
case "invalidate":
id := optionID(opts, 0)
if _, err := db.Exec("UPDATE entries SET status='inactive' WHERE id=?", id); err != nil {
return commandOutcome{}, err
}
return commandOutcome{value: map[string]interface{}{"invalidated": id}}, nil
}
return commandOutcome{}, &StoreError{Code: "USAGE", Op: opts.command, Message: "unknown memory-store verb"}
}
func runCanonicalEval(stdin io.Reader, stdout io.Writer) int {
decoder := json.NewDecoder(stdin)
for {
var request struct {
Op string `json:"op"`
Arg map[string]interface{} `json:"arg"`
}
if err := decoder.Decode(&request); err != nil {
if err == io.EOF {
return 0
}
writeCanonicalError(stdout, &StoreError{Code: "PROTOCOL_ERROR", Message: err.Error()})
return 2
}
outcome, err := executeCanonicalEval(request.Op, request.Arg)
if err != nil {
writeCanonicalError(stdout, err)
return exitForStoreError(err)
}
writeCanonicalJSON(stdout, map[string]interface{}{
"ok": true, "value": outcome.value,
"trace": map[string]interface{}{"store": "go-memory-store", "kernels": outcome.kernels},
})
}
}
func executeCanonicalEval(op string, arg map[string]interface{}) (commandOutcome, error) {
path, _ := arg["db_path"].(string)
if path == "" {
path = dbPath()
}
if op == "read_query" {
query, _ := arg["sql"].(string)
params, _ := arg["params"].([]interface{})
rows, err := nativeReadQuery(path, query, params)
return commandOutcome{value: rows}, err
}
if strings.HasPrefix(op, "warm_") {
return nativeWarmOperation(path, op, arg)
}
db, err := connectPath(path)
if err != nil {
return commandOutcome{}, err
}
defer db.Close()
switch op {
case "stats":
value, err := nativeStats(db)
return commandOutcome{value: value}, err
case "recall":
rows, kernels, err := nativeRecallEntries(db, arg)
return commandOutcome{value: rows, kernels: kernels}, err
}
return commandOutcome{}, &StoreError{Code: "USAGE", Op: op, Message: "unknown eval operation"}
}
func writeCanonicalJSON(writer io.Writer, value any) {
encoded, err := json.Marshal(value)
if err != nil {
writeCanonicalError(writer, err)
return
}
fmt.Fprintln(writer, string(encoded))
}
func writeCanonicalError(writer io.Writer, err error) {
code := storeErrorCode(err)
writeCanonicalJSON(writer, map[string]interface{}{
"ok": false,
"error": map[string]interface{}{"code": code, "message": err.Error()},
})
}
func exitForStoreError(err error) int {
switch storeErrorCode(err) {
case "NATIVE_UNAVAILABLE":
return 127
case "USAGE", "PROTOCOL_ERROR":
return 2
default:
return 1
}
}
@@ -0,0 +1,324 @@
package main
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
)
func cliForTest(t *testing.T, args ...string) (int, map[string]interface{}, string) {
t.Helper()
var stdout, stderr bytes.Buffer
code := runCanonical(args, strings.NewReader(""), &stdout, &stderr)
var value map[string]interface{}
if strings.HasPrefix(strings.TrimSpace(stdout.String()), "{") {
if err := json.Unmarshal(stdout.Bytes(), &value); err != nil {
t.Fatalf("CLI %v non-JSON stdout %q: %v", args, stdout.String(), err)
}
}
return code, value, stdout.String()
}
func TestCanonicalCLIReceiptsAndAudit(t *testing.T) {
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "store", "memory.db"))
for _, add := range [][]string{
{"add", "the desk phone number is 555-0142", "--importance", "0.8", "--topic", "office", "--no-dedupe"},
{"add", "the desk fax number is 555-0143", "--importance", "0.6", "--topic", "office", "--no-dedupe"},
} {
code, receipt, _ := cliForTest(t, add...)
if code != 0 || receipt["added"] != true || receipt["dedupe"] != "bypassed" {
t.Fatalf("add receipt code=%d body=%v", code, receipt)
}
}
code, _, raw := cliForTest(t, "recall", "phone")
if code != 0 {
t.Fatalf("recall exit %d", code)
}
var recall []map[string]interface{}
if err := json.Unmarshal([]byte(raw), &recall); err != nil {
t.Fatal(err)
}
if len(recall) != 1 || recall[0]["id"] != float64(1) || recall[0]["kind"] != "fact" || recall[0]["relevance"] != 4.1 {
t.Fatalf("native recall receipt = %v", recall)
}
code, receipt, _ := cliForTest(t, "update", "1", "--text", "the desk phone number is 555-0199")
if code != 0 || receipt["updated"] != true {
t.Fatalf("update = %d %v", code, receipt)
}
code, receipt, _ = cliForTest(t, "audit", "desk-phone", "verified", "parity-run")
if code != 0 || receipt["hash"] != "0954c9ca5b41c7ed02da8105" {
t.Fatalf("audit = %d %v", code, receipt)
}
code, receipt, _ = cliForTest(t, "verify")
if code != 0 || receipt["ok"] != true || receipt["msg"] != "chain intact (1 entries)" {
t.Fatalf("verify = %d %v", code, receipt)
}
code, receipt, _ = cliForTest(t, "stats")
if code != 0 || receipt["active_entries"] != float64(2) || receipt["audit"] != float64(1) {
t.Fatalf("stats = %d %v", code, receipt)
}
}
func TestCanonicalNineteenVerbSurface(t *testing.T) {
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "memory.db"))
t.Setenv("RUNE_MEMORY_EMBED", "off")
for _, text := range []string{"the desk phone number is 555-0142", "the desk fax number is 555-0143"} {
if code, _, _ := cliForTest(t, "add", text, "--topic", "office", "--no-dedupe"); code != 0 {
t.Fatalf("seed add %q exit %d", text, code)
}
}
source := filepath.Join(t.TempDir(), "notes.md")
if err := os.WriteFile(source, []byte("a sufficiently long research note about desk phones"), 0o644); err != nil {
t.Fatal(err)
}
cases := [][]string{
{"recall", "phone"},
{"update", "1", "--text", "the desk phone number is 555-0199"},
{"working", "task-1", "current durable state"},
{"working", "task-1"},
{"audit", "desk-phone", "verified", "matrix"},
{"verify"},
{"stats"},
{"mine", source},
{"chunk-recall", "research", "--min-sim", "0"},
{"wake"},
{"export"},
{"associate", "1", "2"},
{"linked", "1"},
{"scene", "office", "1", "2"},
{"scene", "list"},
{"hierarchy"},
{"query-core"},
{"reindex", "--dry-run"},
{"hook", "--event", "boot"},
{"delete", "2"},
}
for _, args := range cases {
code, _, raw := cliForTest(t, args...)
if code != 0 {
t.Errorf("verb %s exited %d: %s", args[0], code, raw)
}
}
if len(canonicalVerbs) != 19 {
t.Fatalf("canonical verb registry has %d verbs, want 19", len(canonicalVerbs))
}
}
func TestCanonicalAddMetadataFlagsReachSchema(t *testing.T) {
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "memory.db"))
t.Setenv("RUNE_MEMORY_EMBED", "off")
code, _, _ := cliForTest(t, "add", "a self classified durable fact", "--memory-type", "self", "--always-on", "1", "--no-dedupe")
if code != 0 {
t.Fatal("metadata add failed")
}
db, err := connect()
if err != nil {
t.Fatal(err)
}
defer db.Close()
var memoryType string
var alwaysOn int
if err := db.QueryRow("SELECT memory_type, always_on FROM entries WHERE id=1").Scan(&memoryType, &alwaysOn); err != nil {
t.Fatal(err)
}
if memoryType != "self" || alwaysOn != 1 {
t.Fatalf("metadata flags stored memory_type=%q always_on=%d", memoryType, alwaysOn)
}
}
func TestCanonicalDeleteMaintainsFTS(t *testing.T) {
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "memory.db"))
t.Setenv("RUNE_MEMORY_EMBED", "off")
if code, _, _ := cliForTest(t, "add", "a durable fact to delete", "--no-dedupe"); code != 0 {
t.Fatal("seed add failed")
}
if code, _, _ := cliForTest(t, "delete", "1"); code != 0 {
t.Fatal("delete failed")
}
db, err := connect()
if err != nil {
t.Fatal(err)
}
defer db.Close()
var remaining int
if err := db.QueryRow("SELECT COUNT(*) FROM entries_fts WHERE rowid=1").Scan(&remaining); err != nil {
t.Fatal(err)
}
if remaining != 0 {
t.Fatalf("delete left %d stale FTS rows", remaining)
}
}
func TestCanonicalRecallIncludesStrongAssociations(t *testing.T) {
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "memory.db"))
t.Setenv("RUNE_MEMORY_EMBED", "off")
for _, text := range []string{"the desk phone number is 555-0142", "the office emergency contact is 555-0100"} {
if code, _, _ := cliForTest(t, "add", text, "--topic", "office", "--no-dedupe"); code != 0 {
t.Fatal("seed add failed")
}
}
db, err := connect()
if err != nil {
t.Fatal(err)
}
associate(db, 1, 2, 0.5)
db.Close()
code, _, raw := cliForTest(t, "recall", "phone")
if code != 0 {
t.Fatalf("association recall exit %d: %s", code, raw)
}
var rows []map[string]interface{}
if err := json.Unmarshal([]byte(raw), &rows); err != nil {
t.Fatal(err)
}
if len(rows) != 2 {
t.Fatalf("strong association was not merged: %v", rows)
}
}
func TestCanonicalCLIFailLoudExitCodes(t *testing.T) {
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "memory.db"))
if code, _, _ := cliForTest(t, "add", "tiny", "--no-dedupe"); code != 1 {
t.Fatalf("short add exit = %d, want 1", code)
}
if code, _, _ := cliForTest(t, "not-a-verb"); code != 2 {
t.Fatalf("unknown verb exit = %d, want 2", code)
}
t.Setenv("RUNE_RETRIEVAL_BIN", filepath.Join(t.TempDir(), "missing-rust"))
if code, _, _ := cliForTest(t, "recall", "phone"); code != 127 {
t.Fatalf("missing Rust recall exit = %d, want 127", code)
}
}
func TestAuditTamperAndReadErrorsAreNotSilentZero(t *testing.T) {
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "memory.db"))
if code, _, _ := cliForTest(t, "audit", "claim", "verified", "evidence"); code != 0 {
t.Fatal("seed audit failed")
}
db, err := connect()
if err != nil {
t.Fatal(err)
}
if _, err := db.Exec("UPDATE audit_log SET hash='tampered' WHERE id=1"); err != nil {
t.Fatal(err)
}
db.Close()
code, receipt, _ := cliForTest(t, "verify")
if code != 1 || receipt["ok"] != false || !strings.Contains(receipt["msg"].(string), "tampered") {
t.Fatalf("tampered verify = %d %v", code, receipt)
}
var stdout, stderr bytes.Buffer
request := `{"op":"read_query","arg":{"sql":"SELECT * FROM missing_table","params":[]}}` + "\n"
if code := runCanonical([]string{"eval"}, strings.NewReader(request), &stdout, &stderr); code != 1 {
t.Fatalf("missing table read exit=%d, want 1 (%s)", code, stdout.String())
}
if !strings.Contains(stdout.String(), "STORE_ERROR") || strings.Contains(stdout.String(), `"value":0`) {
t.Fatalf("missing table was a silent zero: %s", stdout.String())
}
}
func TestDefaultAddDedupeUsesRustPolicy(t *testing.T) {
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "memory.db"))
realBin := retrievalBin()
if realBin == "" {
t.Skip("memory_retrieval binary not built")
}
dir := t.TempDir()
trace := filepath.Join(dir, "dedupe-request.json")
wrapper := filepath.Join(dir, "memory_retrieval")
script := fmt.Sprintf("#!/bin/sh\ncat >%q\ncat %q | %q\n", trace, trace, realBin)
if err := os.WriteFile(wrapper, []byte(script), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("RUNE_RETRIEVAL_BIN", wrapper)
args := []string{"add", "the same durable fact", "--topic", "office"}
code, first, _ := cliForTest(t, args...)
if code != 0 || first["decision"] != "created" {
t.Fatalf("first dedupe add = %d %v", code, first)
}
code, second, _ := cliForTest(t, args...)
if code != 0 || second["decision"] != "duplicate" || second["id"] != first["id"] {
t.Fatalf("duplicate add = %d %v (first %v)", code, second, first)
}
if first["prior_id"] != nil || second["prior_id"] != nil {
t.Fatalf("created/duplicate prior_id must stay null: first=%v second=%v", first["prior_id"], second["prior_id"])
}
raw, err := os.ReadFile(trace)
if err != nil || !strings.Contains(string(raw), "dedupe_decision") {
t.Fatalf("dedupe policy did not cross Rust: %s (%v)", raw, err)
}
db, err := connect()
if err != nil {
t.Fatal(err)
}
defer db.Close()
var active, receipts int
if err := db.QueryRow("SELECT COUNT(*) FROM entries WHERE status='active'").Scan(&active); err != nil {
t.Fatal(err)
}
if err := db.QueryRow("SELECT COUNT(*) FROM dedupe_receipts").Scan(&receipts); err != nil {
t.Fatal(err)
}
if active != 1 || receipts != 2 {
t.Fatalf("active=%d receipts=%d, want 1 and 2", active, receipts)
}
}
func TestDedupeUsesNativeEmbeddingsForParaphrases(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
fmt.Fprintf(w, `{"embeddings":[%s]}`, canonicalVectorJSON())
}))
defer server.Close()
t.Setenv("OLLAMA_URL", server.URL)
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "memory.db"))
if code, first, _ := cliForTest(t, "add", "unrelated lexical tokens"); code != 0 || first["decision"] != "created" {
t.Fatalf("first add = %d %v", code, first)
}
code, second, _ := cliForTest(t, "add", "entirely new vocabulary")
if code != 0 || second["decision"] != "duplicate" || second["id"] != float64(1) {
t.Fatalf("embedding dedupe = %d %v", code, second)
}
}
func TestEvalWarmRowsUseRustAndMissingKernelFails(t *testing.T) {
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "memory.db"))
db, err := connect()
if err != nil {
t.Fatal(err)
}
now := nowIso()
if _, err := db.Exec(`INSERT INTO entries
(text, slug, triggers, importance, kind, status, created_at, last_accessed)
VALUES (?,?,?,?,?,?,?,?)`, " deployment stuff ", "deploy", "Deploy, build", 0.8, "neuron", "active", now, now); err != nil {
t.Fatal(err)
}
db.Close()
request := `{"op":"warm_list","arg":{}}` + "\n"
var stdout, stderr bytes.Buffer
code := runCanonical([]string{"eval"}, strings.NewReader(request), &stdout, &stderr)
if code != 0 {
t.Fatalf("warm eval exit=%d stderr=%s stdout=%s", code, stderr.String(), stdout.String())
}
var envelope map[string]interface{}
if err := json.Unmarshal(stdout.Bytes(), &envelope); err != nil {
t.Fatal(err)
}
rows := envelope["value"].([]interface{})
row := rows[0].(map[string]interface{})
if row["slug"] != "deploy" || row["text"] != "deployment stuff" {
t.Fatalf("warm row = %v", row)
}
t.Setenv("RUNE_WARM_NEURON_BIN", filepath.Join(t.TempDir(), "missing-warm-rust"))
stdout.Reset()
stderr.Reset()
if code := runCanonical([]string{"eval"}, strings.NewReader(request), &stdout, &stderr); code != 127 {
t.Fatalf("missing warm Rust exit=%d, want 127 (%s)", code, stdout.String())
}
}
+160
View File
@@ -0,0 +1,160 @@
package main
import (
"crypto/sha256"
"database/sql"
"encoding/hex"
"fmt"
"strings"
)
func nativeStats(db *sql.DB) (map[string]interface{}, error) {
out := map[string]interface{}{"top_topics": map[string]int64{}}
for query, key := range map[string]string{
"SELECT COUNT(*) FROM entries WHERE status='active'": "active_entries",
"SELECT COUNT(*) FROM working": "working",
"SELECT COUNT(*) FROM audit_log": "audit",
} {
var count int64
if err := db.QueryRow(query).Scan(&count); err != nil {
return nil, err
}
out[key] = count
}
rows, err := db.Query("SELECT topic, COUNT(*) AS n FROM entries WHERE status='active' GROUP BY topic ORDER BY n DESC LIMIT 6")
if err != nil {
return nil, err
}
defer rows.Close()
topics := map[string]int64{}
for rows.Next() {
var topic sql.NullString
var count int64
if err := rows.Scan(&topic, &count); err != nil {
return nil, err
}
if topic.Valid && topic.String != "" {
topics[topic.String] = count
}
}
if err := rows.Err(); err != nil {
return nil, err
}
out["top_topics"] = topics
return out, nil
}
func nativeAuditAppend(db *sql.DB, claim, verdict, evidence string) (string, error) {
prev := "GENESIS"
var last sql.NullString
if err := db.QueryRow("SELECT hash FROM audit_log ORDER BY id DESC LIMIT 1").Scan(&last); err != nil && err != sql.ErrNoRows {
return "", err
}
if last.Valid {
prev = last.String
}
sum := sha256.Sum256([]byte(fmt.Sprintf("%s|%s|%s|%s", prev, claim, verdict, evidence)))
hash := hex.EncodeToString(sum[:])[:24]
if _, err := db.Exec(
"INSERT INTO audit_log (prev_hash, hash, claim, verdict, evidence, created_at) VALUES (?,?,?,?,?,?)",
prev, hash, claim, verdict, evidence, nowIso(),
); err != nil {
return "", err
}
return hash, nil
}
func nativeAuditVerify(db *sql.DB) (map[string]interface{}, error) {
rows, err := db.Query("SELECT id, prev_hash, hash, claim, verdict, evidence FROM audit_log ORDER BY id")
if err != nil {
return nil, err
}
defer rows.Close()
prev := "GENESIS"
count := 0
for rows.Next() {
var id int64
var previous, hash string
var claim, verdict, evidence sql.NullString
if err := rows.Scan(&id, &previous, &hash, &claim, &verdict, &evidence); err != nil {
return nil, err
}
if previous != prev {
return map[string]interface{}{"ok": false, "msg": fmt.Sprintf("chain break at id %d", id)}, nil
}
sum := sha256.Sum256([]byte(fmt.Sprintf("%s|%s|%s|%s", prev, claim.String, verdict.String, evidence.String)))
recomputed := hex.EncodeToString(sum[:])[:24]
if recomputed != hash {
return map[string]interface{}{"ok": false, "msg": fmt.Sprintf("hash mismatch at id %d (tampered)", id)}, nil
}
prev = hash
count++
}
if err := rows.Err(); err != nil {
return nil, err
}
return map[string]interface{}{"ok": true, "msg": fmt.Sprintf("chain intact (%d entries)", count)}, nil
}
func nativeDeleteEntry(db *sql.DB, id int64) error {
tx, err := db.Begin()
if err != nil {
return err
}
defer tx.Rollback()
var oldText sql.NullString
if err := tx.QueryRow("SELECT text FROM entries WHERE id=?", id).Scan(&oldText); err != nil && err != sql.ErrNoRows {
return err
}
if oldText.Valid && oldText.String != "" {
if _, err := tx.Exec("INSERT INTO entries_fts(entries_fts, rowid, text) VALUES('delete', ?, ?)", id, oldText.String); err != nil {
return err
}
}
if _, err := tx.Exec("DELETE FROM entries WHERE id=?", id); err != nil {
return err
}
if _, err := tx.Exec("DELETE FROM entries_vec WHERE rowid=?", id); err != nil {
return err
}
return tx.Commit()
}
func nativeReadQuery(dbPath, query string, params []interface{}) ([][]interface{}, error) {
trimmed := strings.TrimSpace(strings.ToUpper(query))
if !strings.HasPrefix(trimmed, "SELECT") && !strings.HasPrefix(trimmed, "WITH") && !strings.HasPrefix(trimmed, "EXPLAIN") {
return nil, &StoreError{Code: "POLICY_REJECTED", Op: "read_query", Message: "only read-only SELECT/WITH/EXPLAIN queries are accepted"}
}
db, err := sql.Open("sqlite", "file:"+dbPath+"?mode=ro")
if err != nil {
return nil, err
}
defer db.Close()
rows, err := db.Query(query, params...)
if err != nil {
return nil, err
}
defer rows.Close()
columns, err := rows.Columns()
if err != nil {
return nil, err
}
out := [][]interface{}{}
for rows.Next() {
values := make([]interface{}, len(columns))
pointers := make([]interface{}, len(columns))
for i := range values {
pointers[i] = &values[i]
}
if err := rows.Scan(pointers...); err != nil {
return nil, err
}
for i, value := range values {
if bytes, ok := value.([]byte); ok {
values[i] = string(bytes)
}
}
out = append(out, values)
}
return out, rows.Err()
}
+242
View File
@@ -0,0 +1,242 @@
package main
import (
"fmt"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"testing"
)
func TestConnectUsesCanonicalPathAndBootstrapsSchema(t *testing.T) {
dbPath := filepath.Join(t.TempDir(), "store", "memory.db")
t.Setenv("RUNE_STORE_DB", dbPath)
db, err := connect()
if err != nil {
t.Fatalf("connect fresh canonical store: %v", err)
}
defer db.Close()
if _, err := os.Stat(dbPath); err != nil {
t.Fatalf("canonical database was not created: %v", err)
}
for _, table := range []string{
"entries", "working", "audit_log", "associations", "scenes", "meta",
"documents", "chunks", "dedupe_receipts", "entries_vec", "chunks_vec",
} {
var name string
if err := db.QueryRow("SELECT name FROM sqlite_master WHERE type IN ('table','view') AND name=?", table).Scan(&name); err != nil {
t.Errorf("canonical schema missing %s: %v", table, err)
}
}
for _, table := range []string{"entries_vec", "chunks_vec"} {
var definition string
if err := db.QueryRow("SELECT sql FROM sqlite_master WHERE name=?", table).Scan(&definition); err != nil {
t.Fatal(err)
}
if !strings.Contains(definition, "vec0") || !strings.Contains(definition, "float[256]") {
t.Errorf("%s is not the canonical 256-dimensional vec0 schema: %s", table, definition)
}
}
}
func TestSQLiteVersionMeetsWALResetPin(t *testing.T) {
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "memory.db"))
db, err := connect()
if err != nil {
t.Fatal(err)
}
defer db.Close()
var version string
if err := db.QueryRow("SELECT sqlite_version()").Scan(&version); err != nil {
t.Fatal(err)
}
parts := strings.Split(version, ".")
if len(parts) < 3 {
t.Fatalf("unrecognized sqlite version %q", version)
}
major, _ := strconv.Atoi(parts[0])
minor, _ := strconv.Atoi(parts[1])
patch, _ := strconv.Atoi(parts[2])
if major < 3 || (major == 3 && minor < 51) || (major == 3 && minor == 51 && patch < 3) {
t.Fatalf("sqlite %s is below the D40 >=3.51.3 pin", version)
}
}
func TestEmbedCanBeExplicitlyDisabled(t *testing.T) {
calls := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
fmt.Fprint(w, `{"embeddings":[[1.0,0.0]]}`)
}))
defer server.Close()
t.Setenv("OLLAMA_URL", server.URL)
t.Setenv("RUNE_MEMORY_EMBED", "off")
if got := Embed([]string{"unique-disable-probe"}); len(got) != 0 || calls != 0 {
t.Fatalf("explicit embedding disable leaked a native call: got=%v calls=%d", got, calls)
}
}
func TestNativeDenseReadsPythonVec0(t *testing.T) {
python, err := exec.LookPath("python3")
if err != nil {
t.Skip("python3 is needed only to create the compatibility fixture")
}
dbPath := filepath.Join(t.TempDir(), "python-vec.db")
fixture := `
import sqlite3, sqlite_vec, sys
db = sqlite3.connect(sys.argv[1])
db.enable_load_extension(True)
sqlite_vec.load(db)
db.executescript("""
CREATE TABLE entries (id INTEGER PRIMARY KEY, text TEXT, importance REAL DEFAULT 0.5, status TEXT, created_at TEXT, last_accessed TEXT, topic TEXT DEFAULT '', entities TEXT DEFAULT '[]', source TEXT DEFAULT '', method TEXT DEFAULT 'curator', valid_from TEXT, valid_until TEXT);
CREATE VIRTUAL TABLE entries_vec USING vec0(embedding float[2]);
CREATE VIRTUAL TABLE entries_fts USING fts5(text, content='', tokenize='porter unicode61');
""")
db.execute("INSERT INTO entries (id, text, status, created_at, last_accessed) VALUES (1, 'the desk phone number', 'active', '2026-01-01T00:00:00+00:00', '2026-01-01T00:00:00+00:00')")
db.execute("INSERT INTO entries_vec (rowid, embedding) VALUES (1, ?)", (json.dumps([1.0, 0.0]),))
db.execute("INSERT INTO entries_fts (rowid, text) VALUES (1, 'the desk phone number')")
db.commit()
`
fixture = "import json\n" + fixture
if output, err := exec.Command(python, "-c", fixture, dbPath).CombinedOutput(); err != nil {
t.Skipf("sqlite_vec fixture unavailable: %v (%s)", err, output)
}
t.Setenv("RUNE_STORE_DB", dbPath)
embedServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
fmt.Fprint(w, `{"embeddings":[[1.0,0.0]]}`)
}))
defer embedServer.Close()
t.Setenv("OLLAMA_URL", embedServer.URL)
embedCache.Lock()
embedCache.m = map[string][]float64{}
embedCache.Unlock()
db, err := connect()
if err != nil {
t.Fatal(err)
}
defer db.Close()
rows, _, err := nativeRecallEntries(db, map[string]interface{}{"query": "phone", "budget": 8.0, "min_score": 0.42})
if err != nil {
t.Fatalf("native vec0 read failed: %v", err)
}
if len(rows) != 1 || rows[0]["mode"] != "hybrid" {
t.Fatalf("python vec0 native recall = %v", rows)
}
}
func canonicalVectorJSON() string {
values := make([]string, 256)
for i := range values {
values[i] = "0.0"
}
values[0] = "1.0"
return "[" + strings.Join(values, ",") + "]"
}
func TestNativeRecallUsesRustDenseCandidates(t *testing.T) {
embedCache.Lock()
savedCache := embedCache.m
embedCache.m = map[string][]float64{}
embedCache.Unlock()
t.Cleanup(func() {
embedCache.Lock()
embedCache.m = savedCache
embedCache.Unlock()
})
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "memory.db"))
embedServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
fmt.Fprintf(w, `{"embeddings":[%s]}`, canonicalVectorJSON())
}))
defer embedServer.Close()
t.Setenv("OLLAMA_URL", embedServer.URL)
db, err := connect()
if err != nil {
t.Fatal(err)
}
defer db.Close()
now := nowIso()
result, err := db.Exec(`INSERT INTO entries
(text, importance, created_at, last_accessed, topic, entities, source, method, status, confidence, temperature)
VALUES (?,?,?,?,?,?,?,?,?,?,?)`,
"the desk phone number is 555-0142", 0.8, now, now, "office", "[]", "test", "curator", "active", 0.7, 1.0)
if err != nil {
t.Fatal(err)
}
id, _ := result.LastInsertId()
if _, err := db.Exec("INSERT INTO entries_fts (rowid, text) VALUES (?,?)", id, "the desk phone number is 555-0142"); err != nil {
t.Fatal(err)
}
if _, err := db.Exec("INSERT INTO entries_vec (rowid, embedding) VALUES (?,?)", id, canonicalVectorJSON()); err != nil {
t.Fatal(err)
}
realBin := retrievalBin()
if realBin == "" {
t.Skip("memory_retrieval binary not built")
}
dir := t.TempDir()
log := filepath.Join(dir, "requests.ndjson")
wrapper := filepath.Join(dir, "memory_retrieval")
script := fmt.Sprintf("#!/bin/sh\ntmp=$(mktemp)\ncat >$tmp\ncat $tmp >>%q\nprintf '\\n' >>%q\ncat $tmp | %q\nrm -f $tmp\n", log, log, realBin)
if err := os.WriteFile(wrapper, []byte(script), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("RUNE_RETRIEVAL_BIN", wrapper)
rows, kernels, err := nativeRecallEntries(db, map[string]interface{}{"query": "phone", "budget": 8.0, "min_score": 0.42})
if err != nil {
t.Fatal(err)
}
if len(rows) != 1 || rows[0]["mode"] != "hybrid" {
t.Fatalf("dense recall rows = %v (kernels %v)", rows, kernels)
}
raw, err := os.ReadFile(log)
if err != nil || !strings.Contains(string(raw), "dense_candidates") {
t.Fatalf("dense policy did not cross Rust: %s (%v)", raw, err)
}
}
func TestRecallCrossesRustBoundary(t *testing.T) {
t.Setenv("RUNE_STORE_DB", filepath.Join(t.TempDir(), "memory.db"))
db, err := connect()
if err != nil {
t.Fatal(err)
}
defer db.Close()
if err := migrateIOFamily(db); err != nil {
t.Fatalf("schema migration: %v", err)
}
now := nowIso()
res, err := db.Exec(`INSERT INTO entries
(text, importance, created_at, last_accessed, topic, entities, source, method, status, confidence, temperature)
VALUES (?,?,?,?,?,?,?,?,?,?,?)`,
"the desk phone number is 555-0142", 0.8, now, now, "office", "[]", "test", "curator", "active", 0.7, 1.0)
if err != nil {
t.Fatal(err)
}
id, _ := res.LastInsertId()
if _, err := db.Exec("INSERT INTO entries_fts (rowid, text) VALUES (?,?)", id, "the desk phone number is 555-0142"); err != nil {
t.Fatal(err)
}
trace := filepath.Join(t.TempDir(), "rust-request.json")
stub := filepath.Join(t.TempDir(), "memory_retrieval")
script := "#!/bin/sh\ncat >\"" + trace + "\"\nprintf '%s\n' '{\"ok\":true,\"value\":[]}'\n"
if err := os.WriteFile(stub, []byte(script), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("RUNE_RETRIEVAL_BIN", stub)
_ = recallEntries(db, map[string]interface{}{"query": "phone", "budget": 8.0, "min_score": 0.42})
raw, err := os.ReadFile(trace)
if err != nil {
t.Fatalf("recall never called the Rust retrieval kernel: %v", err)
}
if !strings.Contains(string(raw), "fts_query_conjunctive") {
t.Fatalf("first Rust request = %s, want FTS policy", raw)
}
}
+108
View File
@@ -0,0 +1,108 @@
package main
import (
"database/sql"
"fmt"
"strings"
)
func nativeChunkRecallEntries(db *sql.DB, args map[string]interface{}) ([]map[string]interface{}, []string, error) {
query := strings.TrimSpace(getString(args, "query"))
if query == "" {
return []map[string]interface{}{}, nil, nil
}
budget := getInt(args, "budget", 6)
minSim := getFloat(args, "min_sim", 0.0)
dense, denseKernels, err := nativeDenseCandidates(db, "chunks_vec", query, budget*3, false)
if err != nil {
return nil, denseKernels, err
}
bestDense := 0.0
for _, pair := range dense {
if score, ok := pair[1].(float64); ok && score > bestDense {
bestDense = score
}
}
if bestDense < minSim {
return []map[string]interface{}{}, denseKernels, nil
}
bin := retrievalBin()
kernels := append([]string{}, denseKernels...)
call := func(op string, arg any) (any, error) {
value, err := rustCoreCall(bin, op, arg)
if err != nil {
return nil, rustCallFailure(op, err)
}
kernels = append(kernels, "memory_retrieval."+op)
return value, nil
}
ftsValue, err := call("fts_query_conjunctive", map[string]interface{}{"query": query})
if err != nil {
return nil, kernels, err
}
ftsQuery, ok := ftsValue.(string)
if !ok {
return nil, kernels, &StoreError{Code: "PROTOCOL_ERROR", Op: "chunk-recall", Message: "Rust FTS result is not a string"}
}
rows, err := db.Query("SELECT rowid FROM chunks_fts WHERE chunks_fts MATCH ? ORDER BY bm25(chunks_fts) LIMIT ?", ftsQuery, budget*3)
if err != nil {
return nil, kernels, err
}
bm25 := [][2]interface{}{}
for rows.Next() {
var id int64
if err := rows.Scan(&id); err != nil {
rows.Close()
return nil, kernels, err
}
bm25 = append(bm25, [2]interface{}{id, 0.5})
}
if err := rows.Close(); err != nil {
return nil, kernels, err
}
fusionValue, err := call("rank_fusion", map[string]interface{}{"dense": dense, "bm25": bm25, "rrf_k": RRFK})
if err != nil {
return nil, kernels, err
}
fusion, ok := fusionValue.(map[string]interface{})
if !ok {
return nil, kernels, &StoreError{Code: "PROTOCOL_ERROR", Op: "chunk-recall", Message: "Rust fusion is not an object"}
}
ranks, _ := fusion["ranks"].([]interface{})
chunkRows, err := fetchChunkRows(db, rankInputIDs(fusion), getString(args, "wing"), getString(args, "doc"))
if err != nil {
return nil, kernels, err
}
projected, err := call("project_chunk_rows", map[string]interface{}{"ranks": ranks, "rows": chunkRows, "budget": budget})
if err != nil {
return nil, kernels, err
}
out, err := mapRows(projected)
return out, kernels, err
}
func fetchChunkRows(db *sql.DB, ids []interface{}, wing, doc string) ([]map[string]interface{}, error) {
if len(ids) == 0 {
return []map[string]interface{}{}, nil
}
placeholders := make([]string, len(ids))
params := append([]interface{}{}, ids...)
for i := range placeholders {
placeholders[i] = "?"
}
query := fmt.Sprintf("SELECT rowid, * FROM chunks WHERE rowid IN (%s)", strings.Join(placeholders, ","))
if wing != "" {
query += " AND wing=?"
params = append(params, wing)
}
if doc != "" {
query += " AND doc_id=?"
params = append(params, doc)
}
rows, err := db.Query(query, params...)
if err != nil {
return nil, err
}
defer rows.Close()
return rowsToMaps(rows), rows.Err()
}
+142
View File
@@ -0,0 +1,142 @@
package main
import (
"crypto/sha256"
"database/sql"
"encoding/hex"
"fmt"
)
func nativeDedupeAdd(db *sql.DB, args map[string]interface{}) (commandOutcome, error) {
text := getString(args, "text")
topic := getString(args, "topic")
candidates, err := dedupeCandidates(db, topic)
if err != nil {
return commandOutcome{}, err
}
bin := retrievalBin()
value, err := rustCoreCall(bin, "dedupe_decision", map[string]interface{}{
"text": text, "topic": topic, "confidence": args["confidence"], "candidates": candidates,
"new_embedding": Embed([]string{text})[text],
})
if err != nil {
return commandOutcome{}, rustCallFailure("dedupe_decision", err)
}
decision, ok := value.(map[string]interface{})
if !ok {
return commandOutcome{}, &StoreError{Code: "PROTOCOL_ERROR", Op: "dedupe_decision", Message: "Rust decision is not an object"}
}
kernels := []string{"memory_retrieval.dedupe_decision"}
action, _ := decision["action"].(string)
if action != "created" && action != "duplicate" && action != "superseded" && action != "rejected" {
return commandOutcome{}, &StoreError{Code: "PROTOCOL_ERROR", Op: "dedupe_decision", Message: "unknown Rust decision action: " + action}
}
priorID := decision["prior_id"]
reason, _ := decision["reason"].(string)
if action == "rejected" {
return commandOutcome{value: map[string]interface{}{"added": false, "error": reason}, kernels: kernels, exitCode: 1}, nil
}
entryID := priorID
receiptPriorID := priorID
outputPriorID := priorID
if action == "created" || action == "duplicate" {
outputPriorID = nil
}
if action == "duplicate" {
receiptPriorID = nil
}
if action == "duplicate" {
if _, err := db.Exec("UPDATE entries SET last_accessed=? WHERE id=?", nowIso(), priorID); err != nil {
return commandOutcome{}, err
}
} else {
if action == "superseded" {
if _, err := db.Exec("UPDATE entries SET status='superseded', valid_until=? WHERE id=?", nowIso(), priorID); err != nil {
return commandOutcome{}, err
}
if _, err := db.Exec("UPDATE entries_vec SET embedding=NULL WHERE rowid=?", priorID); err != nil {
return commandOutcome{}, err
}
}
args["method"] = "dedupe_policy"
result := addEntry(db, args)
if !result.Success {
return commandOutcome{value: map[string]interface{}{"added": false, "error": result.Error}, kernels: kernels, exitCode: 1}, nil
}
data, _ := result.Data.(map[string]interface{})
entryID = data["id"]
if action == "created" {
receiptPriorID = entryID
}
}
if err := persistDedupeReceipt(db, action, text, topic, getString(args, "source"), reason, receiptPriorID); err != nil {
return commandOutcome{}, err
}
return commandOutcome{value: map[string]interface{}{
"added": true, "decision": action, "id": entryID, "prior_id": outputPriorID, "reason": reason,
}, kernels: kernels}, nil
}
func dedupeCandidates(db *sql.DB, topic string) ([]map[string]interface{}, error) {
query := "SELECT e.id, e.text, e.topic, e.confidence, e.valid_until, v.embedding FROM entries e " +
"LEFT JOIN entries_vec v ON e.rowid=v.rowid WHERE e.status='active'"
params := []interface{}{}
if topic != "" {
query += " AND e.topic=?"
params = append(params, topic)
}
query += " ORDER BY e.last_accessed DESC LIMIT 60"
rows, err := db.Query(query, params...)
if err != nil {
return nil, err
}
defer rows.Close()
out := []map[string]interface{}{}
for rows.Next() {
var id int64
var text, candidateTopic string
var confidence sql.NullFloat64
var validUntil sql.NullString
var raw interface{}
if err := rows.Scan(&id, &text, &candidateTopic, &confidence, &validUntil, &raw); err != nil {
return nil, err
}
embedding, err := decodeNativeEmbedding(raw)
if err != nil {
return nil, err
}
row := map[string]interface{}{"id": id, "text": text, "topic": candidateTopic, "embedding": embedding}
if confidence.Valid {
row["confidence"] = confidence.Float64
} else {
row["confidence"] = nil
}
if validUntil.Valid {
row["valid_until"] = validUntil.String
} else {
row["valid_until"] = nil
}
out = append(out, row)
}
return out, rows.Err()
}
func persistDedupeReceipt(db *sql.DB, decision, text, topic, evidence, reason string, priorID interface{}) error {
ts := nowIso()
if len([]rune(text)) > 200 {
text = string([]rune(text)[:200])
}
if len([]rune(evidence)) > 200 {
evidence = string([]rune(evidence)[:200])
}
if len([]rune(reason)) > 200 {
reason = string([]rune(reason)[:200])
}
sum := sha256.Sum256([]byte(fmt.Sprintf("%s|%s|%s|%s", ts, decision, topic, text)))
hash := hex.EncodeToString(sum[:])[:16]
_, err := db.Exec(
"INSERT INTO dedupe_receipts (ts, decision, topic, text, prior_id, evidence, reason, hash) VALUES (?,?,?,?,?,?,?,?)",
ts, decision, topic, text, priorID, evidence, reason, hash,
)
return err
}
+122
View File
@@ -0,0 +1,122 @@
package main
import (
"database/sql"
"encoding/binary"
"encoding/json"
"fmt"
"math"
"strings"
)
func nativeDenseCandidates(db *sql.DB, table, query string, probeCap int, gate bool) ([][2]interface{}, []string, error) {
queryKey := "search_query: " + query
queryEmbedding := Embed([]string{queryKey})[queryKey]
if len(queryEmbedding) == 0 {
return [][2]interface{}{}, nil, nil
}
var vectorQuery, corpusQuery string
switch table {
case "entries_vec":
vectorQuery = "SELECT e.id, e.text, v.embedding FROM entries_vec v JOIN entries e ON e.rowid=v.rowid WHERE e.status='active'"
corpusQuery = "SELECT text FROM entries WHERE status='active'"
case "chunks_vec":
vectorQuery = "SELECT c.rowid, c.text, v.embedding FROM chunks_vec v JOIN chunks c ON c.rowid=v.rowid"
default:
return nil, nil, &StoreError{Code: "POLICY_REJECTED", Op: "dense_candidates", Message: "unknown vector table"}
}
rows, err := db.Query(vectorQuery)
if err != nil {
return nil, nil, &StoreError{Code: "NATIVE_UNAVAILABLE", Op: "dense_candidates", Message: "native vector rows are unreadable: " + err.Error(), Cause: err}
}
vectorRows := []map[string]interface{}{}
for rows.Next() {
var id int64
var text string
var raw interface{}
if err := rows.Scan(&id, &text, &raw); err != nil {
rows.Close()
return nil, nil, err
}
embedding, err := decodeNativeEmbedding(raw)
if err != nil {
rows.Close()
return nil, nil, err
}
if len(embedding) > 0 {
vectorRows = append(vectorRows, map[string]interface{}{"id": id, "text": text, "embedding": embedding})
}
}
if err := rows.Close(); err != nil {
return nil, nil, err
}
corpus := []string{}
if gate {
corpusRows, err := db.Query(corpusQuery)
if err != nil {
return nil, nil, err
}
for corpusRows.Next() {
var text string
if err := corpusRows.Scan(&text); err != nil {
corpusRows.Close()
return nil, nil, err
}
corpus = append(corpus, text)
}
if err := corpusRows.Close(); err != nil {
return nil, nil, err
}
}
value, err := rustCoreCall(retrievalBin(), "dense_candidates", map[string]interface{}{
"query": query, "query_embedding": queryEmbedding, "rows": vectorRows,
"corpus": corpus, "probe_cap": probeCap, "gate": gate,
})
if err != nil {
return nil, nil, rustCallFailure("dense_candidates", err)
}
items, ok := value.([]interface{})
if !ok {
return nil, nil, &StoreError{Code: "PROTOCOL_ERROR", Op: "dense_candidates", Message: fmt.Sprintf("expected candidate array, got %T", value)}
}
out := make([][2]interface{}, 0, len(items))
for _, item := range items {
pair, ok := item.([]interface{})
if !ok || len(pair) != 2 {
return nil, nil, &StoreError{Code: "PROTOCOL_ERROR", Op: "dense_candidates", Message: "invalid dense candidate pair"}
}
out = append(out, [2]interface{}{pair[0], pair[1]})
}
return out, []string{"memory_retrieval.dense_candidates"}, nil
}
func decodeNativeEmbedding(raw interface{}) ([]float64, error) {
var data []byte
switch value := raw.(type) {
case nil:
return nil, nil
case string:
data = []byte(value)
case []byte:
data = value
default:
return nil, fmt.Errorf("unsupported embedding storage type %T", raw)
}
trimmed := strings.TrimSpace(string(data))
if strings.HasPrefix(trimmed, "[") {
var embedding []float64
if err := json.Unmarshal([]byte(trimmed), &embedding); err != nil {
return nil, err
}
return embedding, nil
}
if len(data)%4 != 0 {
return nil, fmt.Errorf("unsupported embedding payload of %d bytes", len(data))
}
embedding := make([]float64, len(data)/4)
for i := range embedding {
bits := binary.LittleEndian.Uint32(data[i*4 : i*4+4])
embedding[i] = float64(math.Float32frombits(bits))
}
return embedding, nil
}
+12 -4
View File
@@ -120,6 +120,9 @@ var embedCache = struct {
// try/except around the curl call.
func Embed(texts []string) map[string][]float64 {
out := map[string][]float64{}
if os.Getenv("RUNE_MEMORY_EMBED") == "off" {
return out
}
if len(texts) == 0 {
return out
}
@@ -450,8 +453,8 @@ func runLexiconReconcile() {
}
// migrateIOFamily — the IO-family schema (idempotent): chunks, documents,
// associations, scenes, meta, FTS5 contentless indexes, and the vec tables as
// PLAIN tables (no vec0 in pure-Go sqlite — see the file header). The entries
// associations, scenes, meta, FTS5 contentless indexes, and native vec0 tables
// (the modernc sqlite/vec module is linked by sqlite_vec.go). The entries
// / working / audit_log tables belong to connect() and are created there when
// absent so family fns work on any handle.
func migrateIOFamily(db *sql.DB) error {
@@ -485,7 +488,7 @@ func migrateIOFamily(db *sql.DB) error {
memory_type TEXT
)`,
`CREATE VIRTUAL TABLE IF NOT EXISTS entries_fts USING fts5(text, content='', tokenize='porter unicode61')`,
`CREATE TABLE IF NOT EXISTS entries_vec (rowid INTEGER PRIMARY KEY, embedding TEXT)`,
`CREATE VIRTUAL TABLE IF NOT EXISTS entries_vec USING vec0(embedding float[256])`,
`CREATE TABLE IF NOT EXISTS chunks (
chunk_id TEXT PRIMARY KEY,
doc_id TEXT NOT NULL,
@@ -501,7 +504,7 @@ func migrateIOFamily(db *sql.DB) error {
ingested_at TEXT NOT NULL
)`,
`CREATE VIRTUAL TABLE IF NOT EXISTS chunks_fts USING fts5(text, content='', tokenize='porter unicode61')`,
`CREATE TABLE IF NOT EXISTS chunks_vec (rowid INTEGER PRIMARY KEY, embedding TEXT)`,
`CREATE VIRTUAL TABLE IF NOT EXISTS chunks_vec USING vec0(embedding float[256])`,
`CREATE TABLE IF NOT EXISTS documents (
path TEXT PRIMARY KEY,
mtime REAL,
@@ -544,6 +547,11 @@ func migrateIOFamily(db *sql.DB) error {
key TEXT PRIMARY KEY,
value TEXT NOT NULL
)`,
`CREATE TABLE IF NOT EXISTS dedupe_receipts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
ts TEXT, decision TEXT, topic TEXT, text TEXT,
prior_id INTEGER, evidence TEXT, reason TEXT, hash TEXT
)`,
}
for _, s := range stmts {
if _, err := db.Exec(s); err != nil {
+16 -20
View File
@@ -1,26 +1,22 @@
module rune/memory-store-deep
go 1.21
require modernc.org/sqlite v1.28.0
go 1.26.0
require (
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/mattn/go-isatty v0.0.16 // indirect
modernc.org/sqlite v1.61.0
rune/paths v0.0.0
)
replace rune/paths => ../paths
require (
github.com/dustin/go-humanize v1.1.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/mattn/go-isatty v0.0.24 // indirect
github.com/ncruces/go-strftime v1.1.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
golang.org/x/mod v0.3.0 // indirect
golang.org/x/sys v0.9.0 // indirect
golang.org/x/tools v0.0.0-20201124115921-2c860bdd6e78 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
lukechampine.com/uint128 v1.2.0 // indirect
modernc.org/cc/v3 v3.40.0 // indirect
modernc.org/ccgo/v3 v3.16.13 // indirect
modernc.org/libc v1.29.0 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.7.2 // indirect
modernc.org/opt v0.1.3 // indirect
modernc.org/strutil v1.1.3 // indirect
modernc.org/token v1.0.1 // indirect
golang.org/x/sys v0.49.0 // indirect
modernc.org/libc v1.79.0 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.12.1 // indirect
)
+48 -71
View File
@@ -1,73 +1,50 @@
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/pprof v0.0.0-20221118152302-e6195bd50e26 h1:Xim43kblpZXfIBQsbuBVKCudVG457BR2GZFIz3uw3hQ=
github.com/google/pprof v0.0.0-20221118152302-e6195bd50e26/go.mod h1:dDKJzRmX4S37WGHujM7tX//fmj1uioxKzKxz3lo4HJo=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 h1:Z9n2FFNUXsshfwJMBgNA0RU6/i7WVaAegv3PtuIHPMs=
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51/go.mod h1:CzGEWj7cYgsdH8dAjBGEr58BoE7ScuLd+fwFZ44+/x8=
github.com/mattn/go-isatty v0.0.16 h1:bq3VjFmv/sOjHtdEhmkEV4x1AJtvUvOJ2PFAZ5+peKQ=
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/mattn/go-sqlite3 v1.14.16 h1:yOQRA0RpS5PFz/oikGwBEqvAWhWg5ufRz4ETLjwpU1Y=
github.com/mattn/go-sqlite3 v1.14.16/go.mod h1:2eHXhiwb8IkHr+BDWZGa96P6+rkvnG63S2DGjv9HUNg=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/dustin/go-humanize v1.1.0 h1:dbKTrvD0klcbBV/h4AWJdMuZogJACoMlvWIWZ5b2xWg=
github.com/dustin/go-humanize v1.1.0/go.mod h1:hc1CvRkJMsgxqjmjMQF3QNRAZBwY8AXBAzKYoSX9sFI=
github.com/google/pprof v0.0.0-20261016150608-1facf8f91b50 h1:PlNWJRxu5APSV0pWqfIsZ0qt9OOGHSWfZQ4zvx+WgWs=
github.com/google/pprof v0.0.0-20261016150608-1facf8f91b50/go.mod h1:yUo1AMMpsPIDFGuwDK8nLcf8+vvAd5l2O6JDZ6hpwrM=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
github.com/ncruces/go-strftime v1.1.0 h1:vtPmPWMkb539RGV9WpWT9EwJGvrOCKvMNI4obWL47Hk=
github.com/ncruces/go-strftime v1.1.0/go.mod h1:yVaq7iwCN09vHRrKBrrflXT0M9tTMG2KmZ9Fj7xeG/g=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.9.0 h1:KS/R3tvhPqvJvwcKfnBHJwwthS11LRhmM5D59eEXa0s=
golang.org/x/sys v0.9.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20201124115921-2c860bdd6e78 h1:M8tBwCtWD/cZV9DZpFYRUgaymAYAr+aIUTWzDaM3uPs=
golang.org/x/tools v0.0.0-20201124115921-2c860bdd6e78/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
lukechampine.com/uint128 v1.2.0 h1:mBi/5l91vocEN8otkC5bDLhi2KdCticRiwbdB0O+rjI=
lukechampine.com/uint128 v1.2.0/go.mod h1:c4eWIwlEGaxC/+H1VguhU4PHXNWDCDMUlWdIWl2j1gk=
modernc.org/cc/v3 v3.40.0 h1:P3g79IUS/93SYhtoeaHW+kRCIrYaxJ27MFPv+7kaTOw=
modernc.org/cc/v3 v3.40.0/go.mod h1:/bTg4dnWkSXowUO6ssQKnOV0yMVxDYNIsIrzqTFDGH0=
modernc.org/ccgo/v3 v3.16.13 h1:Mkgdzl46i5F/CNR/Kj80Ri59hC8TKAhZrYSaqvkwzUw=
modernc.org/ccgo/v3 v3.16.13/go.mod h1:2Quk+5YgpImhPjv2Qsob1DnZ/4som1lJTodubIcoUkY=
modernc.org/ccorpus v1.11.6 h1:J16RXiiqiCgua6+ZvQot4yUuUy8zxgqbqEEUuGPlISk=
modernc.org/ccorpus v1.11.6/go.mod h1:2gEUTrWqdpH2pXsmTM1ZkjeSrUWDpjMu2T6m29L/ErQ=
modernc.org/httpfs v1.0.6 h1:AAgIpFZRXuYnkjftxTAZwMIiwEqAfk8aVB2/oA6nAeM=
modernc.org/httpfs v1.0.6/go.mod h1:7dosgurJGp0sPaRanU53W4xZYKh14wfzX420oZADeHM=
modernc.org/libc v1.29.0 h1:tTFRFq69YKCF2QyGNuRUQxKBm1uZZLubf6Cjh/pVHXs=
modernc.org/libc v1.29.0/go.mod h1:DaG/4Q3LRRdqpiLyP0C2m1B8ZMGkQ+cCgOIjEtQlYhQ=
modernc.org/mathutil v1.6.0 h1:fRe9+AmYlaej+64JsEEhoWuAYBkOtQiMEU7n/XgfYi4=
modernc.org/mathutil v1.6.0/go.mod h1:Ui5Q9q1TR2gFm0AQRqQUaBWFLAhQpCwNcuhBOSedWPo=
modernc.org/memory v1.7.2 h1:Klh90S215mmH8c9gO98QxQFsY+W451E8AnzjoE2ee1E=
modernc.org/memory v1.7.2/go.mod h1:NO4NVCQy0N7ln+T9ngWqOQfi7ley4vpwvARR+Hjw95E=
modernc.org/opt v0.1.3 h1:3XOZf2yznlhC+ibLltsDGzABUGVx8J6pnFMS3E4dcq4=
modernc.org/opt v0.1.3/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0=
modernc.org/sqlite v1.28.0 h1:Zx+LyDDmXczNnEQdvPuEfcFVA2ZPyaD7UCZDjef3BHQ=
modernc.org/sqlite v1.28.0/go.mod h1:Qxpazz0zH8Z1xCFyi5GSL3FzbtZ3fvbjmywNogldEW0=
modernc.org/strutil v1.1.3 h1:fNMm+oJklMGYfU9Ylcywl0CO5O6nTfaowNsh2wpPjzY=
modernc.org/strutil v1.1.3/go.mod h1:MEHNA7PdEnEwLvspRMtWTNnp2nnyvMfkimT1NKNAGbw=
modernc.org/tcl v1.15.2 h1:C4ybAYCGJw968e+Me18oW55kD/FexcHbqH2xak1ROSY=
modernc.org/tcl v1.15.2/go.mod h1:3+k/ZaEbKrC8ePv8zJWPtBSW0V7Gg9g8rkmhI1Kfs3c=
modernc.org/token v1.0.1 h1:A3qvTqOwexpfZZeyI0FeGPDlSWX5pjZu9hF4lU+EKWg=
modernc.org/token v1.0.1/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
modernc.org/z v1.7.3 h1:zDJf6iHjrnB+WRD88stbXokugjyc0/pB91ri1gO6LZY=
modernc.org/z v1.7.3/go.mod h1:Ipv4tsdxZRbQyLq9Q1M6gdbkxYzdlrciF2Hi/lS7nWE=
golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg=
golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs=
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/tools v0.52.0 h1:ryxg4U63oaUYk7uOLTF37n43f4qgtGvqV4Ioit1BaEs=
golang.org/x/tools v0.52.0/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs=
modernc.org/cc/v4 v4.30.3 h1:aDq8gIX1X7+VFgO/JYYji6pLWL2H4tv3g8WatT3KCys=
modernc.org/cc/v4 v4.30.3/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
modernc.org/ccgo/v4 v4.37.0 h1:BAuJCUtIHIxFIYaFhqXIw5YaoMBlNCUlFovejHfMip4=
modernc.org/ccgo/v4 v4.37.0/go.mod h1:/8627VhwWtAbqUXmPORxFQU5SgLrAx2jl9seLw9GBhY=
modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM=
modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU=
modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
modernc.org/gc/v3 v3.2.0 h1:/Y3LGDMQaoYf55PebP1qsZ5FDGgAZyG1RYKEHeQzh1o=
modernc.org/gc/v3 v3.2.0/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
modernc.org/libc v1.79.0 h1:9Dvvv++bAMSkSuBzuOFAatbTV5OIoq6Y6FohmD3e7q8=
modernc.org/libc v1.79.0/go.mod h1:uT3kR61NsbeKJSLqq83GJwSnZMVLlSjyAz4DfTaBy1M=
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
modernc.org/memory v1.12.1 h1:nFMiWrpStgZczNl6XI9GnIk/rWhYIyHGUaR04pGbp9g=
modernc.org/memory v1.12.1/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg=
modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
modernc.org/sqlite v1.61.0 h1:sQi63oLQ9uquAUA4CzOisH35Q3pEszkPBuHM0tFL8CY=
modernc.org/sqlite v1.61.0/go.mod h1:f4gr5tbpx2umklaT/QVtbi/50hNdM+4ng+2zMo8KXJ0=
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
+150 -223
View File
@@ -21,6 +21,7 @@ import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"time"
@@ -28,12 +29,14 @@ import (
"encoding/hex"
"math"
_ "modernc.org/sqlite"
"rune/paths/resolver"
)
type Result struct {
Success bool `json:"success"`
Data interface{} `json:"data,omitempty"`
Error string `json:"error,omitempty"`
Code string `json:"code,omitempty"`
}
type MemoryEntry struct {
@@ -60,67 +63,91 @@ var MEMORY_TYPES = map[string]bool{
}
func dbPath() string {
home := os.Getenv("RUNE_MEMORY_DIR")
if home == "" {
home = os.Getenv("HOME") + "/.rune"
}
os.MkdirAll(home+"/mempalace", 0755)
return home + "/mempalace/memories.db"
return resolver.StoreDB()
}
func connect() (*sql.DB, error) {
db, err := sql.Open("sqlite", dbPath())
return connectPath(dbPath())
}
func connectPath(path string) (*sql.DB, error) {
if path == "" {
return nil, fmt.Errorf("canonical memory store path is empty")
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return nil, err
}
db, err := sql.Open("sqlite", path)
if err != nil {
return nil, err
}
db.Exec("PRAGMA journal_mode=WAL")
db.Exec("PRAGMA synchronous=NORMAL")
// Create table if not exists (parity with Python schema)
db.Exec(`CREATE TABLE IF NOT EXISTS entries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
text TEXT NOT NULL,
importance REAL DEFAULT 0.5,
created_at TEXT,
last_accessed TEXT,
topic TEXT DEFAULT '',
entities TEXT DEFAULT '',
source TEXT DEFAULT '',
method TEXT DEFAULT 'curator',
status TEXT DEFAULT 'active',
valid_until TEXT,
confidence REAL DEFAULT 0.7,
temperature REAL DEFAULT 1.0,
always_on INTEGER DEFAULT 0,
priority INTEGER DEFAULT 5,
tier TEXT DEFAULT 'warm',
memory_type TEXT,
wash_count INTEGER DEFAULT 0,
superseded_by INTEGER,
rock_status TEXT DEFAULT 'washable'
)`)
db.Exec(`CREATE INDEX IF NOT EXISTS idx_entries_status ON entries(status)`)
db.Exec(`CREATE INDEX IF NOT EXISTS idx_entries_topic ON entries(topic)`)
// Batch-1 columns (parity with src/rune/memory_store.py)
db.Exec(`ALTER TABLE entries ADD COLUMN promotion_eligible INTEGER DEFAULT 0`)
db.Exec(`ALTER TABLE entries ADD COLUMN last_washed TEXT`)
db.Exec(`CREATE TABLE IF NOT EXISTS working (
id INTEGER PRIMARY KEY AUTOINCREMENT,
task_id TEXT NOT NULL,
text TEXT NOT NULL,
created_at TEXT
)`)
db.Exec(`CREATE TABLE IF NOT EXISTS audit_log (
id INTEGER PRIMARY KEY AUTOINCREMENT,
prev_hash TEXT NOT NULL,
hash TEXT NOT NULL,
claim TEXT NOT NULL,
verdict TEXT NOT NULL,
evidence TEXT DEFAULT '',
created_at TEXT
)`)
if _, err := db.Exec("PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL; PRAGMA foreign_keys=ON"); err != nil {
db.Close()
return nil, err
}
if err := migrateIOFamily(db); err != nil {
db.Close()
return nil, err
}
if err := migrateEntryColumns(db); err != nil {
db.Close()
return nil, err
}
return db, nil
}
func migrateEntryColumns(db *sql.DB) error {
rows, err := db.Query("PRAGMA table_info(entries)")
if err != nil {
return err
}
cols := map[string]bool{}
for rows.Next() {
var cid, notNull, pk int
var name, typ string
var dflt any
if err := rows.Scan(&cid, &name, &typ, &notNull, &dflt, &pk); err != nil {
rows.Close()
return err
}
cols[name] = true
}
if err := rows.Close(); err != nil {
return err
}
for col, ddl := range map[string]string{
"confidence": "REAL DEFAULT 0.7", "temperature": "REAL DEFAULT 1.0",
"always_on": "INTEGER DEFAULT 0", "priority": "INTEGER DEFAULT 5",
"triggers": "TEXT DEFAULT ''", "kind": "TEXT DEFAULT 'fact'",
"slug": "TEXT DEFAULT ''", "summary": "TEXT DEFAULT ''",
"wash_count": "INTEGER DEFAULT 0", "last_washed": "TEXT",
"promotion_eligible": "INTEGER DEFAULT 0", "rock_status": "TEXT DEFAULT 'washable'",
"tier": "TEXT DEFAULT 'warm'", "memory_type": "TEXT", "superseded_by": "INTEGER",
} {
if !cols[col] {
if _, err := db.Exec("ALTER TABLE entries ADD COLUMN " + col + " " + ddl); err != nil {
return err
}
}
}
for _, index := range []string{
"CREATE INDEX IF NOT EXISTS idx_entries_status_kind ON entries(status, kind)",
"CREATE INDEX IF NOT EXISTS idx_entries_status_topic ON entries(status, topic)",
"CREATE INDEX IF NOT EXISTS idx_entries_slug ON entries(slug)",
"CREATE INDEX IF NOT EXISTS idx_entries_status ON entries(status)",
"CREATE INDEX IF NOT EXISTS idx_entries_wash_count ON entries(wash_count)",
"CREATE INDEX IF NOT EXISTS idx_entries_rock_status ON entries(rock_status)",
"CREATE INDEX IF NOT EXISTS idx_associations_src ON associations(src_id, strength)",
"CREATE INDEX IF NOT EXISTS idx_associations_dst ON associations(dst_id, strength)",
} {
if _, err := db.Exec(index); err != nil {
return err
}
}
ensureMemoryTypeColumn(db)
return nil
}
func nowISO() string {
return time.Now().UTC().Format("2006-01-02T15:04:05.000000Z")
}
@@ -154,19 +181,40 @@ func addEntry(db *sql.DB, args map[string]interface{}) Result {
return Result{Success: false, Error: fmt.Sprintf("invalid memory_type %q; expected one of %v", memoryType, valid)}
}
now := nowISO()
now := nowIso()
tier := "warm"
vec := Embed([]string{text})[text]
tx, err := db.Begin()
if err != nil {
return Result{Success: false, Error: err.Error()}
}
defer tx.Rollback()
var entryID int64
err := db.QueryRow(
`INSERT INTO entries (text, importance, created_at, last_accessed, topic, source, method, status, confidence, temperature, always_on, priority, tier, memory_type)
VALUES (?,?,?,?,?,?,?,'active',?,?,?,?,?,?) RETURNING id`,
text, importance, now, now, topic, source, method,
err = tx.QueryRow(
`INSERT INTO entries (text, importance, created_at, last_accessed, topic, entities, source, method, status, valid_from, valid_until, confidence, temperature, always_on, priority, tier, memory_type)
VALUES (?,?,?,?,?,?,?,?,'active',?,?,?,?,?,?,?,?) RETURNING id`,
text, importance, now, now, topic, "[]", source, method, now, args["valid_until"],
confidence, temperature, alwaysOn, priority, tier, nullIfEmpty(memoryType),
).Scan(&entryID)
if err != nil {
return Result{Success: false, Error: err.Error()}
}
if len(vec) > 0 {
encoded, _ := json.Marshal(vec)
if _, err := tx.Exec("INSERT INTO entries_vec (rowid, embedding) VALUES (?,?)", entryID, string(encoded)); err != nil {
return Result{Success: false, Error: err.Error()}
}
}
if _, err := tx.Exec("INSERT INTO entries_fts (rowid, text) VALUES (?,?)", entryID, text); err != nil {
return Result{Success: false, Error: err.Error()}
}
if _, err := tx.Exec("UPDATE entries SET summary=? WHERE id=?", aaakSummary(text, topic, source), entryID); err != nil {
return Result{Success: false, Error: err.Error()}
}
if err := tx.Commit(); err != nil {
return Result{Success: false, Error: err.Error()}
}
autoAssociate(db, entryID, text, topic, source)
return Result{Success: true, Data: map[string]interface{}{
"id": entryID, "text": text, "importance": importance,
@@ -177,50 +225,11 @@ func addEntry(db *sql.DB, args map[string]interface{}) Result {
// ── recall: budget + min_score + type filtering ──
func recallEntries(db *sql.DB, args map[string]interface{}) Result {
query := getString(args, "query")
budget := getInt(args, "budget", 5)
minScore := getFloat(args, "min_score", 0.3)
_ = getString(args, "memory_types") // type filtering pending full port
if query == "" {
return Result{Success: false, Error: "query required"}
}
// Simple LIKE-based recall (the Go version doesn't have embeddings yet —
// that's a Wave 6 port). The important thing is the interface parity.
rows, err := db.Query(
`SELECT id, text, importance, topic, created_at, confidence, wash_count FROM entries
WHERE status='active' AND text LIKE ?
ORDER BY importance DESC LIMIT ?`,
"%"+query+"%", budget)
rows, _, err := nativeRecallEntries(db, args)
if err != nil {
return Result{Success: false, Error: err.Error()}
return resultFromError(err)
}
defer rows.Close()
var results []map[string]interface{}
for rows.Next() {
var id int64
var text, topic, createdAt string
var importance, confidence float64
var washCount int
rows.Scan(&id, &text, &importance, &topic, &createdAt, &confidence, &washCount)
if importance < minScore {
continue
}
results = append(results, map[string]interface{}{
"id": id, "text": text, "importance": importance,
"topic": topic, "created_at": createdAt,
"confidence": confidence, "wash_count": washCount,
})
}
// Parity: if nothing clears the score floor, return empty (ABSTAIN)
if len(results) == 0 {
return Result{Success: true, Data: []interface{}{}}
}
return Result{Success: true, Data: results}
return Result{Success: true, Data: rows}
}
// ── supersede: mark old as superseded, create new entry ──
@@ -341,9 +350,10 @@ func updateEntry(db *sql.DB, args map[string]interface{}) Result {
eid := getInt(args, "id", 0)
var sets []string
var params []interface{}
if text, ok := args["text"]; ok {
newText, hasText := args["text"].(string)
if hasText {
sets = append(sets, "text=?")
params = append(params, text)
params = append(params, newText)
}
if _, ok := args["importance"]; ok {
sets = append(sets, "importance=?")
@@ -356,11 +366,45 @@ func updateEntry(db *sql.DB, args map[string]interface{}) Result {
if len(sets) == 0 {
return Result{Success: true, Data: false}
}
tx, err := db.Begin()
if err != nil {
return Result{Success: false, Error: err.Error()}
}
defer tx.Rollback()
oldText := ""
if hasText {
if err := tx.QueryRow("SELECT text FROM entries WHERE id=?", eid).Scan(&oldText); err != nil {
return Result{Success: false, Error: err.Error()}
}
}
sets = append(sets, "last_accessed=?")
params = append(params, nowISO())
params = append(params, nowIso())
params = append(params, eid)
db.Exec("UPDATE entries SET "+strings.Join(sets, ", ")+" WHERE id=?", params...)
db.Exec("COMMIT")
if _, err := tx.Exec("UPDATE entries SET "+strings.Join(sets, ", ")+" WHERE id=?", params...); err != nil {
return Result{Success: false, Error: err.Error()}
}
if hasText {
if oldText != "" {
if _, err := tx.Exec("INSERT INTO entries_fts(entries_fts, rowid, text) VALUES('delete', ?, ?)", eid, oldText); err != nil {
return Result{Success: false, Error: err.Error()}
}
}
if _, err := tx.Exec("INSERT INTO entries_fts (rowid, text) VALUES (?,?)", eid, newText); err != nil {
return Result{Success: false, Error: err.Error()}
}
if _, err := tx.Exec("DELETE FROM entries_vec WHERE rowid=?", eid); err != nil {
return Result{Success: false, Error: err.Error()}
}
if vec := Embed([]string{newText})[newText]; len(vec) > 0 {
encoded, _ := json.Marshal(vec)
if _, err := tx.Exec("INSERT INTO entries_vec (rowid, embedding) VALUES (?,?)", eid, string(encoded)); err != nil {
return Result{Success: false, Error: err.Error()}
}
}
}
if err := tx.Commit(); err != nil {
return Result{Success: false, Error: err.Error()}
}
return Result{Success: true, Data: true}
}
@@ -591,122 +635,5 @@ func nullIfEmpty(s string) interface{} {
// ── CLI ──
func main() {
if len(os.Args) < 2 {
fmt.Fprintln(os.Stderr, "usage: memory-store-deep <add|search|get|wash|supersede|invalidate|stats> [--key value ...]")
os.Exit(1)
}
db, err := connect()
if err != nil {
out, _ := json.Marshal(Result{Success: false, Error: err.Error()})
fmt.Println(string(out))
os.Exit(1)
}
defer db.Close()
cmd := os.Args[1]
rest := os.Args[2:]
// Parse args into a map
args := make(map[string]interface{})
var positional []string
for i := 0; i < len(rest); i++ {
if strings.HasPrefix(rest[i], "--") {
key := strings.TrimPrefix(rest[i], "--")
if i+1 < len(rest) && !strings.HasPrefix(rest[i+1], "--") {
args[key] = rest[i+1]
i++
} else {
args[key] = true
}
} else {
positional = append(positional, rest[i])
}
}
if len(positional) > 0 {
if _, ok := args["text"]; !ok {
args["text"] = positional[0]
}
if _, ok := args["query"]; !ok && cmd == "search" {
args["query"] = positional[0]
}
if _, ok := args["id"]; !ok && (cmd == "get" || cmd == "wash" || cmd == "invalidate") && len(positional) > 0 {
var id float64
fmt.Sscanf(positional[0], "%f", &id)
args["id"] = id
}
if len(positional) > 1 && cmd == "supersede" {
var id float64
fmt.Sscanf(positional[0], "%f", &id)
args["id"] = id
args["text"] = positional[1]
}
}
var result Result
switch cmd {
case "add":
result = addEntry(db, args)
case "search":
result = recallEntries(db, args)
case "supersede":
result = supersedeEntry(db, args)
case "wash":
result = washEntry(db, args)
case "stats":
result = getStats(db)
case "get":
id := getInt(args, "id", 0)
var text, topic, createdAt string
var importance float64
err := db.QueryRow("SELECT text, importance, topic, created_at FROM entries WHERE id=? AND status='active'", id).
Scan(&text, &importance, &topic, &createdAt)
if err == sql.ErrNoRows {
result = Result{Success: false, Error: "not found"}
} else if err != nil {
result = Result{Success: false, Error: err.Error()}
} else {
result = Result{Success: true, Data: map[string]interface{}{
"id": id, "text": text, "importance": importance, "topic": topic, "created_at": createdAt}}
}
case "increment-wash":
result = incrementWash(db, int64(getInt(args, "id", 0)))
case "wash-score":
result = getWashScore(db, int64(getInt(args, "id", 0)))
case "promote-rock":
result = promoteToRock(db, int64(getInt(args, "id", 0)))
case "rock-entries":
result = getRockEntries(db)
case "washable-entries":
result = getWashableEntries(db, getInt(args, "limit", 100))
case "update":
result = updateEntry(db, args)
case "delete":
result = deleteEntry(db, args)
case "touch":
result = touchEntry(db, args)
case "working-set":
result = workingSet(db, args)
case "working-get":
result = workingGet(db, args)
case "working-clear":
result = workingClear(db, args)
case "audit-append":
result = auditAppend(db, args)
case "audit-verify":
result = auditVerify(db)
case "health":
result = memoryHealthCheck(db, getInt(args, "days", 7))
case "invalidate":
id := getInt(args, "id", 0)
db.Exec("UPDATE entries SET status='inactive' WHERE id=?", id)
result = Result{Success: true, Data: map[string]interface{}{"invalidated": id}}
default:
result = Result{Success: false, Error: fmt.Sprintf("unknown command: %s", cmd)}
}
out, _ := json.MarshalIndent(result, "", " ")
fmt.Println(string(out))
if !result.Success {
os.Exit(1)
}
os.Exit(runCanonical(os.Args[1:], os.Stdin, os.Stdout, os.Stderr))
}
+225
View File
@@ -0,0 +1,225 @@
package main
import (
"database/sql"
"fmt"
"sort"
"strings"
)
func stringList(value any) []string {
switch items := value.(type) {
case []string:
return items
case []interface{}:
out := make([]string, 0, len(items))
for _, item := range items {
if text, ok := item.(string); ok {
out = append(out, text)
}
}
return out
case string:
if items != "" {
return strings.Split(items, ",")
}
}
return nil
}
func nativeRecallEntries(db *sql.DB, args map[string]interface{}) ([]map[string]interface{}, []string, error) {
query := strings.TrimSpace(getString(args, "query"))
if query == "" {
return []map[string]interface{}{}, nil, nil
}
budget := getInt(args, "budget", RecallBudget)
minScore := getFloat(args, "min_score", RecallMinScore)
probeCap := getInt(args, "probe_cap", RecallBudget*2)
memoryTypes := stringList(args["memory_types"])
bin := retrievalBin()
kernels := []string{}
call := func(op string, arg any) (any, error) {
value, err := rustCoreCall(bin, op, arg)
if err != nil {
return nil, rustCallFailure(op, err)
}
kernels = append(kernels, "memory_retrieval."+op)
return value, nil
}
ftsValue, err := call("fts_query_conjunctive", map[string]interface{}{"query": query})
if err != nil {
return nil, kernels, err
}
ftsQuery, ok := ftsValue.(string)
if !ok {
return nil, kernels, &StoreError{Code: "PROTOCOL_ERROR", Op: "fts_query_conjunctive", Message: "Rust FTS result is not a string"}
}
dense, denseKernels, err := nativeDenseCandidates(db, "entries_vec", query, probeCap, true)
kernels = append(kernels, denseKernels...)
if err != nil {
return nil, kernels, err
}
bm25 := [][2]interface{}{}
rows, err := db.Query(
"SELECT rowid, bm25(entries_fts) AS score FROM entries_fts WHERE entries_fts MATCH ? ORDER BY bm25(entries_fts) LIMIT ?",
ftsQuery, probeCap,
)
if err != nil {
return nil, kernels, err
}
for rows.Next() {
var id int64
var score float64
if err := rows.Scan(&id, &score); err != nil {
rows.Close()
return nil, kernels, err
}
bm25 = append(bm25, [2]interface{}{id, score})
}
if err := rows.Close(); err != nil {
return nil, kernels, err
}
fusionValue, err := call("rank_fusion", map[string]interface{}{"dense": dense, "bm25": bm25, "rrf_k": RRFK})
if err != nil {
return nil, kernels, err
}
fusion, ok := fusionValue.(map[string]interface{})
if !ok {
return nil, kernels, &StoreError{Code: "PROTOCOL_ERROR", Op: "rank_fusion", Message: "Rust fusion result is not an object"}
}
ids := rankInputIDs(fusion)
entryRows, err := fetchRecallRows(db, ids, memoryTypes)
if err != nil {
return nil, kernels, err
}
scoredValue, err := call("score_recall_rows", map[string]interface{}{
"fusion": fusion, "rows": entryRows, "dense": dense, "bm25": bm25,
"budget": budget, "min_score": minScore, "now": nowIso(), "half_life": RecencyHalfLife,
})
if err != nil {
return nil, kernels, err
}
scored, err := mapRows(scoredValue)
if err != nil {
return nil, kernels, &StoreError{Code: "PROTOCOL_ERROR", Op: "score_recall_rows", Message: err.Error(), Cause: err}
}
return scored, kernels, nil
}
func nativeRecallWithAssociations(db *sql.DB, args map[string]interface{}) ([]map[string]interface{}, []string, error) {
base, kernels, err := nativeRecallEntries(db, args)
if err != nil || len(base) == 0 || getInt(args, "hops", 1) < 1 {
return base, kernels, err
}
budget := getInt(args, "budget", RecallBudget)
byID := map[string]bool{}
for _, row := range base {
byID[scalarIDKey(row["id"])] = true
}
strengths := map[string]float64{}
idsByKey := map[string]interface{}{}
order := []string{}
for _, seed := range base {
for _, linked := range linkedEntries(db, toInt64(seed["id"]), 4) {
key := scalarIDKey(linked["id"])
strength, _ := linked["strength"].(float64)
if byID[key] || strength < 0.3 {
continue
}
if _, seen := strengths[key]; !seen {
order = append(order, key)
}
if strength > strengths[key] {
strengths[key] = strength
idsByKey[key] = linked["id"]
}
}
}
if len(order) == 0 {
return base, kernels, nil
}
sort.SliceStable(order, func(i, j int) bool { return strengths[order[i]] > strengths[order[j]] })
boosts := make([]map[string]interface{}, 0, len(order))
ids := make([]interface{}, 0, len(order))
for _, key := range order {
id := idsByKey[key]
ids = append(ids, id)
boosts = append(boosts, map[string]interface{}{"id": id, "strength": strengths[key]})
}
rows, err := fetchRecallRows(db, ids, nil)
if err != nil {
return nil, kernels, err
}
value, err := rustCoreCall(retrievalBin(), "merge_association_rows", map[string]interface{}{
"base": base, "boosts": boosts, "rows": rows, "budget": budget,
})
if err != nil {
return nil, kernels, rustCallFailure("merge_association_rows", err)
}
merged, err := mapRows(value)
return merged, append(kernels, "memory_retrieval.merge_association_rows"), err
}
func rankInputIDs(fusion map[string]interface{}) []interface{} {
ranks, _ := fusion["ranks"].([]interface{})
ids := make([]interface{}, 0, len(ranks))
for _, raw := range ranks {
pair, ok := raw.([]interface{})
if ok && len(pair) == 2 {
ids = append(ids, pair[0])
}
}
return ids
}
func fetchRecallRows(db *sql.DB, ids []interface{}, memoryTypes []string) ([]map[string]interface{}, error) {
if len(ids) == 0 {
return []map[string]interface{}{}, nil
}
placeholders := make([]string, len(ids))
params := append([]interface{}{}, ids...)
for i := range placeholders {
placeholders[i] = "?"
}
query := fmt.Sprintf("SELECT * FROM entries WHERE id IN (%s) AND status='active'", strings.Join(placeholders, ","))
if len(memoryTypes) > 0 {
typePlaceholders := make([]string, len(memoryTypes))
for i, memoryType := range memoryTypes {
typePlaceholders[i] = "?"
params = append(params, memoryType)
}
query += fmt.Sprintf(" AND (memory_type IN (%s) OR memory_type IS NULL)", strings.Join(typePlaceholders, ","))
}
rows, err := db.Query(query, params...)
if err != nil {
return nil, err
}
defer rows.Close()
return rowsToMaps(rows), rows.Err()
}
func mapRows(value any) ([]map[string]interface{}, error) {
items, ok := value.([]interface{})
if !ok {
return nil, fmt.Errorf("expected row array, got %T", value)
}
out := make([]map[string]interface{}, 0, len(items))
for _, item := range items {
row, ok := item.(map[string]interface{})
if !ok {
return nil, fmt.Errorf("expected row object, got %T", item)
}
out = append(out, row)
}
return out, nil
}
func nativeFuseResults(entries, research []map[string]interface{}, bin string) ([]map[string]interface{}, error) {
value, err := rustCoreCall(bin, "fuse_results", map[string]interface{}{"entries": entries, "research": research})
if err != nil {
return nil, rustCallFailure("fuse_results", err)
}
return mapRows(value)
}
+118
View File
@@ -0,0 +1,118 @@
package main
import (
"bytes"
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"runtime"
)
type StoreError struct {
Code string
Op string
Message string
Cause error
}
func (e *StoreError) Error() string {
if e.Op == "" {
return e.Code + ": " + e.Message
}
return e.Code + " (" + e.Op + "): " + e.Message
}
func (e *StoreError) Unwrap() error { return e.Cause }
func storeErrorCode(err error) string {
if typed, ok := err.(*StoreError); ok {
return typed.Code
}
return "STORE_ERROR"
}
func resultFromError(err error) Result {
return Result{Success: false, Error: err.Error(), Code: storeErrorCode(err)}
}
func repoRoot() string {
_, file, _, ok := runtime.Caller(0)
if !ok {
return ""
}
return filepath.Dir(filepath.Dir(filepath.Dir(filepath.Dir(file))))
}
func rustBinary(envName, crate, name string) string {
if value := os.Getenv(envName); value != "" {
return value
}
if root := repoRoot(); root != "" {
candidate := filepath.Join(root, "src", "rust", crate, "target", "release", name)
if info, err := os.Stat(candidate); err == nil && !info.IsDir() {
return candidate
}
}
path, _ := exec.LookPath(name)
return path
}
func retrievalBin() string {
if value := os.Getenv("MR_RUST_BIN"); value != "" {
return value
}
return rustBinary("RUNE_RETRIEVAL_BIN", "memory_retrieval", "memory_retrieval")
}
type rustEnvelope struct {
OK bool `json:"ok"`
Value json.RawMessage `json:"value"`
Error *struct {
Code string `json:"code"`
Message string `json:"message"`
} `json:"error"`
}
func rustCoreCall(bin, op string, arg any) (any, error) {
if bin == "" {
return nil, &StoreError{Code: "NATIVE_UNAVAILABLE", Op: op, Message: "Rust kernel binary not found"}
}
payload, err := json.Marshal(map[string]any{"op": op, "arg": arg})
if err != nil {
return nil, err
}
cmd := exec.Command(bin)
cmd.Stdin = bytes.NewReader(append(payload, '\n'))
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
code := "STORE_ERROR"
if os.IsNotExist(err) {
code = "NATIVE_UNAVAILABLE"
}
return nil, &StoreError{Code: code, Op: op, Message: stderr.String() + err.Error(), Cause: err}
}
var envelope rustEnvelope
if err := json.Unmarshal(stdout.Bytes(), &envelope); err != nil {
return nil, &StoreError{Code: "PROTOCOL_ERROR", Op: op, Message: "non-JSON Rust response: " + stdout.String(), Cause: err}
}
if !envelope.OK {
code, message := "STORE_ERROR", "Rust operation failed"
if envelope.Error != nil {
code, message = envelope.Error.Code, envelope.Error.Message
}
return nil, &StoreError{Code: code, Op: op, Message: message}
}
var value any
if err := json.Unmarshal(envelope.Value, &value); err != nil {
return nil, &StoreError{Code: "PROTOCOL_ERROR", Op: op, Message: err.Error(), Cause: err}
}
return value, nil
}
func rustCallFailure(op string, err error) error {
return &StoreError{Code: storeErrorCode(err), Op: op, Message: fmt.Sprint(err), Cause: err}
}
+5
View File
@@ -0,0 +1,5 @@
package main
// The canonical SQLite build includes sqlite-vec natively (pure Go, no cgo).
// Registration makes Python-created vec0 stores readable by the same driver.
import _ "modernc.org/sqlite/vec"
+62
View File
@@ -0,0 +1,62 @@
package main
import (
"database/sql"
"os"
)
func warmNeuronBin() string {
if value := getenvFirst("WNS_RUST_BIN", "RUNE_WARM_NEURON_BIN"); value != "" {
return value
}
return rustBinary("RUNE_WARM_NEURON_BIN", "warm_neuron_store", "warm_neuron_store")
}
func getenvFirst(names ...string) string {
for _, name := range names {
if value := os.Getenv(name); value != "" {
return value
}
}
return ""
}
func nativeWarmOperation(path, op string, arg map[string]interface{}) (commandOutcome, error) {
db, err := sql.Open("sqlite", "file:"+path+"?mode=ro")
if err != nil {
return commandOutcome{}, err
}
defer db.Close()
query := "SELECT text, slug, triggers, importance FROM entries WHERE kind='neuron' AND status='active' ORDER BY slug"
kernelOp := ""
switch op {
case "warm_list":
kernelOp = "list_neurons"
case "warm_slugs":
kernelOp = "neuron_slugs"
case "warm_has":
kernelOp = "neuron_has"
case "warm_digests":
kernelOp = "neuron_digests"
query = "SELECT slug, text FROM entries WHERE kind='neuron' AND status='active'"
default:
return commandOutcome{}, &StoreError{Code: "USAGE", Op: op, Message: "unknown warm operation"}
}
rows, err := db.Query(query)
if err != nil {
return commandOutcome{}, err
}
rawRows := rowsToMaps(rows)
if err := rows.Close(); err != nil {
return commandOutcome{}, err
}
payload := interface{}(rawRows)
if op == "warm_has" {
payload = map[string]interface{}{"rows": rawRows, "term": arg["term"]}
}
value, err := rustCoreCall(warmNeuronBin(), kernelOp, payload)
if err != nil {
return commandOutcome{}, rustCallFailure(kernelOp, err)
}
return commandOutcome{value: value, kernels: []string{"warm_neuron_store." + kernelOp}}, nil
}
+15 -58
View File
@@ -3,11 +3,9 @@
// Oracle: python3 -m rune.memory_store (src/rune/memory_store.py).
// Native: src/go/memory-ops (modernc.org/sqlite vendored there).
//
// Registered RED-first on 2026-10-18: the first recorded run measured
// the two engines apart — verb vocabulary, store path, schema
// bootstrap, write-path semantics, recall semantics, provenance
// ledger — and the packet records the exact divergences; the row parks
// with them named rather than forcing a green verdict.
// Registered RED-first on 2026-10-18; closed natively by #2618 except
// same-change Python removal (the final retirement tranche). The packet
// carries both the baseline divergences and their measured closure evidence.
package main
import (
@@ -69,11 +67,11 @@ func memoryOpsSpec() CapSpec {
Row: 24,
Scenario: memoryOpsScenario,
OraclePathDisplay: "src/rune/memory_store.py (python3 -m rune.memory_store; still live — pre-removal)",
NativePathDisplay: "src/go/memory-ops (attested binary; modernc.org/sqlite vendored)",
NativePathDisplay: "src/go/memory-store-deep (canonical attested binary; memory-ops is a compatibility launcher)",
OracleModule: "src/rune/memory_store.py",
OracleArgv: []string{"-m", "rune.memory_store"},
PythonDeadArgv: []string{"-m", "rune.memory_store", "stats"},
NativeModule: "src/go/memory-ops",
NativeModule: "src/go/memory-store-deep",
NativeBin: "memory-ops-native",
Normalize: NormalizeReceipt,
Effects: auditChainEntries,
@@ -81,7 +79,7 @@ func memoryOpsSpec() CapSpec {
IrreversibleStepID: "mem.audit/claim",
KeyScopeStepID: "mem.add/fact1",
ReceiptOriginStepID: "mem.recall/phone",
Ordering: OrderingOracle,
Ordering: OrderingCross,
EffectsLabel: "audit chain",
LivePathsNote: "this capability HAS a live model path — the oracle's write path " +
"embeds via Ollama nomic-embed-text (/api/embed, live during the recorded " +
@@ -95,56 +93,15 @@ func memoryOpsSpec() CapSpec {
}
var memoryOpsFindings = []string{
"side-by-side measured RED on the first recorded run (2026-10-18): " +
"the engines do not share a verb vocabulary — the oracle CLI exposes " +
"add/recall/update/delete/working/audit/verify/stats/mine/chunk-recall/" +
"wake/export/associate/linked/scene/hierarchy/query-core/reindex/hook " +
"(19 verbs); the native path implements add/search/get/wash/supersede/stats — " +
"recall↔search are not even name-aligned, and update/audit/verify/recall " +
"have no native verb (native exits rc=1 'unknown command' with empty stdout)",
"store path divergence (a key_scopes diff on every write step): the oracle " +
"keeps <RUNE_MEMORY_DIR>/store/memory.db (+ cache.db); the native path writes " +
"<RUNE_MEMORY_DIR>/.rune/mempalace/memories.db — two engines on two " +
"databases; the sibling src/go/memory-store-deep uses a THIRD convention " +
"(<RUNE_MEMORY_DIR>/mempalace/memories.db), so the two Go ports disagree " +
"with each other as well",
"schema bootstrap: the oracle creates the full schema on connect; the native " +
"path creates nothing — on a fresh workspace add fails ('unable to open " +
"database file' — parent dirs absent; with dirs pre-made, 'no such table: " +
"entries') and stats answers success:true with zeros over a store that does " +
"not exist (silent-zero class, rc=0)",
"exit-code honesty: the native path reports failures with exit code 0 (the " +
"failed add above, the silent-zero stats); the oracle exits 1 on a failed add",
"write-path semantics: the oracle add is dedupe-gated with decision receipts — " +
"and the gate module dedupe_policy is ABSENT in this checkout, so the " +
"oracle's DEFAULT add path fails ('No module named dedupe_policy', rc=1); " +
"only the --no-dedupe bypass works (live oracle defect recorded here); the " +
"bypass path embeds via Ollama nomic-embed-text, FTS-indexes, writes an AAAK " +
"summary and auto-associates; the native add is a raw INSERT with method " +
"'go-operator' and hardcoded confidence/temperature/tier",
"recall semantics: the oracle is fused dense (sqlite-vec 256-d) + BM25 (FTS5) " +
"ranked by RRF with budget/min-score abstention; the native search is SQL " +
"LIKE ORDER BY importance",
"provenance: the oracle keeps a hash-chained audit ledger (audit/verify verbs — " +
"verify's receipt carries the chain length, this row's recovery observable); " +
"the native path has no audit leg at all",
"recovery leg RED: with the native unable to execute the oracle verbs, " +
"cross-engine resume receipts cannot be equal; the recovery-ordering mutant " +
"therefore runs both legs on the ORACLE engine for this row (the reference " +
"resume adds no audit rows; re-appending the audit claim adds exactly one — " +
"the counter's teeth are proven on the engine that can run the surface)",
"key-scope mutant trips on the pre-existing store-path divergence (both the " +
"injected escape and the divergence surface as key_scopes diffs); once the " +
"store path converges the mutant measures the escape itself",
"vendored SQLite pin: the module vendors modernc.org/sqlite v1.28.0 (an early " +
"SQLite ~3.44-era build) — BELOW the D40 pin law (SQLite ≥3.51.3, the " +
"WAL-reset bug class); the gap-close build must bump the vendored version " +
"with the row",
"row parked: native path landed and native-alone smoke green on a schema-prepared " +
"store (all six native verbs coherent), but parity vs the oracle is RED — the " +
"row holds PARTIAL with these named gaps; PROVEN requires the gap-close build " +
"(verb alignment or an agreed retirement contract, store-path unification, " +
"schema bootstrap, dedupe/audit legs) plus the full machine green",
"BASELINE D1 (2026-10-18): native add/search/get/wash/supersede/stats did not match the oracle's 19 verbs. CLOSED #2618: the canonical memory-store-deep binary passes the 19-verb live matrix and the 7-step cross-engine trace",
"BASELINE D2: three DB conventions (<memory>/store/memory.db, .rune/mempalace/memories.db, mempalace/memories.db). CLOSED #2618: paths owns resolution, both engines touch memory/store/memory.db, and memory-ops is only a launcher",
"BASELINE D3: fresh native stores had no schema or audit leg and stats silently returned zeros. CLOSED #2618: connect bootstraps/migrates the canonical vec0/FTS/schema; audit append/verify/tamper and missing-table tests prove loud behavior",
"BASELINE D4: native write failures returned rc=0. CLOSED #2618: operation failure=1, usage/protocol=2, required native engine unavailable=127, with typed Python/core errors",
"BASELINE D5: native add was a raw INSERT with no dedupe receipts. CLOSED #2618: Rust dedupe_decision owns create/duplicate/supersede/reject policy, Go persists dedupe_receipts, and the no-dedupe receipt matches the oracle",
"BASELINE D6: native search was SQL LIKE ORDER BY importance. CLOSED #2618: native vec0+FTS5 rows cross Rust dense_candidates/rank_fusion/score_recall_rows/fuse_results with abstention, typed filtering, chunk projection, and strong-association merge tests",
"BASELINE recovery/key-scope RED: the native path could not run cross-engine resume. CLOSED #2618: both recovery breakpoints match and the recovery-ordering mutant runs in cross-engine form; key-scope measures only the injected escape",
"BASELINE SQLite pin: modernc.org/sqlite v1.28.0 carried SQLite 3.41.2, below D40 >=3.51.3. CLOSED #2618: v1.61.0 carries SQLite 3.54.1 and sqlite-vec v0.1.9; the version test and real Python-vec0 compatibility test pass",
"REMAINING: same-change Python removal is deliberately deferred to the final retirement tranche. Verdict is GREEN and row state is PARTIAL, not PROVEN; Python serving facades prove native calls by trace and fail-loud absence tests",
}
// register the row without touching the sibling wave's map literal.
+46 -85
View File
@@ -1,16 +1,7 @@
// FROZEN TEST (red-first — the row was REGISTERED red: the first
// recorded run measured the two engines apart and the packet parked
// the row PARTIAL with the divergences named; this pin freezes that
// honest state).
//
// Retirement row 24 — memory-ops, the store/recall/consolidation CLI
// surface (oracle src/rune/memory_store.py's CLI vs native
// src/go/memory-ops). The machine must keep measuring the gap exactly:
// verdict RED while the engines diverge, row state PARTIAL with the
// missing legs named, mutants RED (the battery has teeth even on a red
// row), and the native digest attested. When the gap-close build lands
// and parity goes GREEN, this pin flips red — forcing the retirement
// map update and the pin's own rewrite in the same change.
// FROZEN TEST (red-first): row 24 was registered RED with named
// divergences. The #2618 gap-close turned that pin red as designed; this
// permanent successor freezes the GREEN canonical contract and the one
// deliberately deferred leg (same-change Python removal).
package main
import (
@@ -20,85 +11,56 @@ import (
"testing"
)
func TestMemoryOpsRowParksHonest(t *testing.T) {
func TestMemoryOpsCanonicalParityGreen(t *testing.T) {
packet, _ := runCapabilityMachine(t, "memory-ops")
if packet.Verdict != "GREEN" {
t.Fatalf("canonical memory store must close the recorded divergences: verdict=%s detail=%s", packet.Verdict, packet.SideBySide.Detail)
}
if !packet.SideBySide.OK || !packet.FailureRecovery.OK || !packet.Mutants.AllRed {
t.Fatalf("parity, recovery, and mutant legs must all hold: %+v", packet)
}
for _, missing := range packet.RowStateMissing {
if !strings.Contains(missing, "same-change Python removal") {
t.Fatalf("unexpected remaining canonical-store gap: %v", packet.RowStateMissing)
}
}
}
func TestMemoryOpsRowRetainsOnlyRemovalGap(t *testing.T) {
packet, packetPath := runCapabilityMachine(t, "memory-ops")
if packet.Verdict != "RED" {
t.Fatalf("verdict = %s, want RED — parity vs the oracle has not been "+
"built yet (verb vocabulary, store path, schema bootstrap, dedupe/"+
"audit legs); if this went GREEN the gap-close landed: update the "+
"retirement map and this pin in the same change\nside-by-side: %+v",
packet.Verdict, packet.SideBySide)
if packet.Verdict != "GREEN" || packet.RowState != "PARTIAL" {
t.Fatalf("verdict=%s row=%s, want GREEN/PARTIAL", packet.Verdict, packet.RowState)
}
if packet.RowState != "PARTIAL" {
t.Fatalf("row state = %s, want PARTIAL (honest parking)", packet.RowState)
if len(packet.RowStateMissing) != 1 || !strings.Contains(packet.RowStateMissing[0], "Python removal") {
t.Fatalf("only the deferred Python-removal leg may remain: %v", packet.RowStateMissing)
}
missing := strings.Join(packet.RowStateMissing, ";")
for _, leg := range []string{"side-by-side", "recovery", "Python removal"} {
if !strings.Contains(missing, leg) {
t.Fatalf("missing leg %q not named: %v", leg, packet.RowStateMissing)
if packet.Attestation == nil || len(packet.Attestation.SHA256) != 64 || packet.Attestation.Binary != "src/go/memory-store-deep" {
t.Fatalf("canonical native digest/path not attested: %+v", packet.Attestation)
}
if len(packet.OracleTrace.Steps) != 7 || len(packet.NativeTrace.Steps) != 7 {
t.Fatalf("mainline lengths = %d/%d, want 7/7", len(packet.OracleTrace.Steps), len(packet.NativeTrace.Steps))
}
for i, oracle := range packet.OracleTrace.Steps {
native := packet.NativeTrace.Steps[i]
if oracle.ExitCode != native.ExitCode || oracle.Receipt != native.Receipt {
t.Fatalf("step %s still diverges: oracle=%d/%s native=%d/%s", oracle.ActionID, oracle.ExitCode, oracle.Receipt, native.ExitCode, native.Receipt)
}
}
// the measured divergences are the packet's substance: the store-path
// key-scopes split and the receipt/exit-code splits must be recorded.
detail := packet.SideBySide.Detail
if !strings.Contains(detail, "key_scopes") {
t.Fatalf("the store-path key-scopes divergence is not recorded: %s", detail)
}
if !strings.Contains(detail, "receipts[") && !strings.Contains(detail, "exit_codes[") {
t.Fatalf("neither receipt nor exit-code divergence recorded: %s", detail)
}
var sawRCSplit, sawReceiptSplit bool
for i, o := range packet.OracleTrace.Steps {
n := packet.NativeTrace.Steps[i]
if o.ExitCode != n.ExitCode {
sawRCSplit = true
if strings.HasPrefix(oracle.ActionID, "mem.add") {
joined := strings.Join(native.Scope.Touched, " ")
if !strings.Contains(joined, "memory/store/memory.db") || strings.Contains(joined, "mempalace") {
t.Fatalf("non-canonical native write scope: %v", native.Scope.Touched)
}
}
if o.Receipt != n.Receipt {
sawReceiptSplit = true
}
}
if !sawRCSplit || !sawReceiptSplit {
t.Fatalf("expected measured rc and receipt splits (recall/update/audit/"+
"verify are not native verbs): rc=%v receipts=%v", sawRCSplit, sawReceiptSplit)
}
// the battery keeps its teeth on a red row: all four mutants RED —
// including the ordering mutant in its same-engine form (the native
// path cannot run this capability's verbs yet).
if !packet.Mutants.AllRed {
t.Fatalf("mutants leg lost teeth on the red row: %+v", packet.Mutants.Results)
}
var ordering MutantResult
for _, m := range packet.Mutants.Results {
if m.Name == "recovery-ordering" {
ordering = m
for _, mutant := range packet.Mutants.Results {
if mutant.Name == "recovery-ordering" {
ordering = mutant
}
}
if !ordering.TrippedRED || !strings.Contains(ordering.Violation, "same-engine form") {
t.Fatalf("ordering mutant must run its same-engine form and name it: %+v", ordering)
if !ordering.TrippedRED || strings.Contains(ordering.Violation, "same-engine form") {
t.Fatalf("ordering mutant did not regain cross-engine teeth: %+v", ordering)
}
// the oracle really ran: live receipts, live write scope, the audit
// chain observable present.
if len(packet.OracleTrace.Steps) != 7 {
t.Fatalf("oracle mainline = %d steps, want 7", len(packet.OracleTrace.Steps))
}
found := false
for _, s := range packet.OracleTrace.Steps {
if s.ActionID == "mem.verify" && strings.Contains(s.Receipt, "chain intact") {
found = true
}
}
if !found {
t.Fatal("the audit-chain observable (verify receipt) is absent from the oracle trace")
}
if packet.Attestation == nil || len(packet.Attestation.SHA256) != 64 {
t.Fatal("native digest not attested")
}
// on-disk identity.
b, err := os.ReadFile(packetPath)
if err != nil {
t.Fatalf("packet not written: %v", err)
@@ -107,8 +69,7 @@ func TestMemoryOpsRowParksHonest(t *testing.T) {
if err := json.Unmarshal(b, &onDisk); err != nil {
t.Fatalf("packet not valid json: %v", err)
}
if onDisk.Capability != "memory-ops" || onDisk.Row != 24 || onDisk.Verdict != "RED" {
t.Fatalf("packet identity wrong: %s row %d verdict %s",
onDisk.Capability, onDisk.Row, onDisk.Verdict)
if onDisk.Capability != "memory-ops" || onDisk.Row != 24 || onDisk.Verdict != "GREEN" {
t.Fatalf("packet identity wrong: %s row %d verdict %s", onDisk.Capability, onDisk.Row, onDisk.Verdict)
}
}
+2 -3
View File
@@ -120,9 +120,8 @@ func memoryOpsScenario() Scenario {
Description: "facts stored, recalled, corrected; provenance appended + verified; store state read",
Mainline: []Step{add1, add2, recall1, update1, audit1, verify1, stats1},
Recovery: RecoveryPlan{
RestartModel: "cold-process-per-step; durable state = the SQLite store " +
"(oracle: <memory>/store/memory.db + cache.db; native: its own " +
"<memory>/.rune/mempalace/memories.db) — every step after a crash " +
RestartModel: "cold-process-per-step; durable state = the canonical SQLite store " +
"<memory>/store/memory.db on both engines — every step after a crash " +
"is a fresh process resuming from durable state",
Breakpoints: []BreakpointPlan{
{
+8
View File
@@ -38,6 +38,14 @@ func pathsEvalOp(name string, op map[string]any) (any, string) {
return DeployRoot(), ""
case "rune_home":
return RuneHome(), ""
case "memory_dir":
return MemoryDir(), ""
case "store_db":
return StoreDB(), ""
case "scripts_dir":
return ScriptsDir(), ""
case "config_dir":
return ConfigDir(), ""
case "harness_scripts":
return HarnessScripts(), ""
case "wait_resume_dir":
+8
View File
@@ -0,0 +1,8 @@
package main
import "rune/paths/resolver"
func MemoryDir() string { return resolver.MemoryDir() }
func StoreDB() string { return resolver.StoreDB() }
func ScriptsDir() string { return resolver.ScriptsDir(RepoRootPath) }
func ConfigDir() string { return resolver.ConfigDir() }
+3 -4
View File
@@ -9,6 +9,8 @@ package main
import (
"os"
"testing"
"rune/paths/resolver"
)
// envOrDefault — the single env-override branch: LookupEnv (set and
@@ -29,10 +31,7 @@ func DeployRoot() string {
// RuneHome — the operator home (env override, ~/.rune default).
// os.path.expanduser("~/.rune") == $HOME + "/.rune" when HOME is set.
func RuneHome() string {
if v, ok := os.LookupEnv("RUNE_HOME"); ok {
return v
}
return os.Getenv("HOME") + "/.rune"
return resolver.RuneHome()
}
// HarnessScripts — harness scripts dir (env HARNESS_SCRIPTS override).
+75
View File
@@ -0,0 +1,75 @@
// Package resolver owns memory-related environment and path resolution.
package resolver
import (
"os"
"path/filepath"
"strings"
)
func env(key string) string {
if value, ok := os.LookupEnv(key); ok && value != "" {
return expand(value)
}
return ""
}
func expand(value string) string {
if value == "" {
return value
}
value = os.ExpandEnv(value)
if value == "~" {
if home, err := os.UserHomeDir(); err == nil {
return home
}
}
if strings.HasPrefix(value, "~/") {
if home, err := os.UserHomeDir(); err == nil {
return filepath.Join(home, strings.TrimPrefix(value, "~/"))
}
}
return value
}
func RuneHome() string {
if value := env("RUNE_HOME"); value != "" {
return value
}
home, err := os.UserHomeDir()
if err != nil {
return ""
}
return filepath.Join(home, ".rune")
}
func MemoryDir() string {
if value := env("RUNE_MEMORY_DIR"); value != "" {
return value
}
return filepath.Join(RuneHome(), "memory")
}
func StoreDB() string {
if value := env("RUNE_STORE_DB"); value != "" {
return value
}
if value := env("MEMORY_STORE_DB"); value != "" {
return value
}
return filepath.Join(MemoryDir(), "store", "memory.db")
}
func ScriptsDir(repoRoot string) string {
if value := env("RUNE_SCRIPTS_DIR"); value != "" {
return value
}
return filepath.Join(repoRoot, "src", "rune")
}
func ConfigDir() string {
if value := env("RUNE_CONFIG_DIR"); value != "" {
return value
}
return filepath.Join(RuneHome(), "config")
}
+43
View File
@@ -0,0 +1,43 @@
package resolver
import (
"path/filepath"
"testing"
)
func TestCanonicalMemoryPathPrecedence(t *testing.T) {
home := t.TempDir()
t.Setenv("HOME", home)
t.Setenv("RUNE_HOME", filepath.Join(home, "rune-home"))
t.Setenv("RUNE_MEMORY_DIR", filepath.Join(home, "memory-root"))
t.Setenv("RUNE_STORE_DB", filepath.Join(home, "explicit.db"))
if got, want := StoreDB(), filepath.Join(home, "explicit.db"); got != want {
t.Fatalf("StoreDB explicit = %q, want %q", got, want)
}
t.Setenv("RUNE_STORE_DB", "")
t.Setenv("MEMORY_STORE_DB", filepath.Join(home, "legacy.db"))
if got, want := StoreDB(), filepath.Join(home, "legacy.db"); got != want {
t.Fatalf("StoreDB legacy alias = %q, want %q", got, want)
}
t.Setenv("MEMORY_STORE_DB", "")
if got, want := StoreDB(), filepath.Join(home, "memory-root", "store", "memory.db"); got != want {
t.Fatalf("StoreDB memory root = %q, want %q", got, want)
}
t.Setenv("RUNE_MEMORY_DIR", "")
if got, want := StoreDB(), filepath.Join(home, "rune-home", "memory", "store", "memory.db"); got != want {
t.Fatalf("StoreDB rune home = %q, want %q", got, want)
}
}
func TestMemoryEnvExpansionMatchesExistingAPI(t *testing.T) {
home := t.TempDir()
t.Setenv("HOME", home)
t.Setenv("RUNE_MEMORY_DIR", "~/memory-$USER")
t.Setenv("USER", "nick")
if got, want := MemoryDir(), filepath.Join(home, "memory-nick"); got != want {
t.Fatalf("MemoryDir expansion = %q, want %q", got, want)
}
}
@@ -51,6 +51,7 @@ func newCloudTestSetup(t *testing.T, provider, model string) (*Engine, *fakeMode
})
fm := &fakeModel{replies: map[string]string{"": "LOCAL-MISROUTE"}}
eng := NewRoutedEngine(st, fm, filepath.Join(dir, "home"), router)
eng.SetContextAssembler(passthroughContextAssembler{})
return eng, fm, d
}
@@ -185,7 +186,9 @@ func TestServerCloudTypedEnvelopes(t *testing.T) {
d.ts.URL+`/v1","api_key_ref":"provider/`+provider+`","enabled":true}}}`)
router := NewProviderRouterAt(reg, func(string) (string, error) { return "k", nil })
fm := &fakeModel{}
srv := NewServer(NewRoutedEngine(st, fm, home, router), fm, home)
eng := NewRoutedEngine(st, fm, home, router)
eng.SetContextAssembler(passthroughContextAssembler{})
srv := NewServer(eng, fm, home)
ts := httptest.NewServer(srv.Handler())
t.Cleanup(ts.Close)
return ts, fm
@@ -364,6 +364,7 @@ func TestEngineCloudToolTurnEndToEnd(t *testing.T) {
}
router := NewProviderRouterAt(reg, func(string) (string, error) { return "vault-key", nil })
eng := NewRoutedEngine(st, &fakeModel{replies: map[string]string{"": "LOCAL-MISROUTE"}}, home, router)
eng.SetContextAssembler(passthroughContextAssembler{})
res, err := eng.Turn(context.Background(), TurnRequest{Channel: "chat", UserText: "read the notes"})
if err != nil {
t.Fatalf("cloud tool turn: %v", err)
@@ -0,0 +1,257 @@
package runecore
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
)
// ContextRequest is the Go-collected input to one turn's context decision.
type ContextRequest struct {
Channel string
Query string
Model string
DBPath string
History []ChatMessage
System string
}
// AssembledContext is the exact model payload plus its native provenance.
type AssembledContext struct {
Messages []ChatMessage
Trace map[string]any
}
// ContextAssembler is the mandatory governed-context seam on every turn.
type ContextAssembler interface {
Assemble(context.Context, ContextRequest) (AssembledContext, error)
}
// ContextError preserves a stable native context failure code and operation.
type ContextError struct {
Code string
Op string
Message string
Cause error
}
func (e *ContextError) Error() string {
if e.Op == "" {
return e.Code + ": " + e.Message
}
return e.Code + ": " + e.Op + ": " + e.Message
}
func (e *ContextError) Unwrap() error { return e.Cause }
// NativeContextAssembler collects Go history/recall and delegates every
// selection, admission, budget, and rendering decision to Rust.
type NativeContextAssembler struct {
Memory MemoryStore
Binary string
WindowTokens int
BudgetTokens int
MemoryBudgetTokens int
MemoryItemBudget int
MinScore float64
MemoryDisabled bool
}
func NewNativeContextAssembler(memory MemoryStore, binary string) *NativeContextAssembler {
if binary == "" {
binary = os.Getenv("RUNE_ATTENTION_CAPSULE_BIN")
}
if binary == "" {
_, file, _, ok := runtime.Caller(0)
if ok {
dir := filepath.Dir(file)
for i := 0; i < 5; i++ {
dir = filepath.Dir(dir)
}
binary = filepath.Join(dir, "src", "rust", "attention_capsule", "target", "release", "attention_capsule")
}
}
return &NativeContextAssembler{
Memory: memory, Binary: binary, WindowTokens: 8192,
MemoryItemBudget: 4, MinScore: 0.42,
}
}
func memoryOff(raw string) bool {
switch strings.ToLower(strings.TrimSpace(raw)) {
case "", "0", "false", "no", "off":
return false
default:
return true
}
}
func contextMessageWire(message ChatMessage) map[string]any {
wire := map[string]any{"role": message.Role, "content": message.Content}
if message.Thinking != nil {
wire["thinking"] = *message.Thinking
}
if len(message.ToolCalls) > 0 {
wire["tool_calls"] = message.ToolCalls
}
if message.ToolName != "" {
wire["tool_name"] = message.ToolName
}
return wire
}
func (a *NativeContextAssembler) Assemble(ctx context.Context, req ContextRequest) (AssembledContext, error) {
if strings.TrimSpace(req.Query) == "" {
return AssembledContext{}, &ContextError{Code: "CONTEXT_UNKNOWN", Op: "assemble", Message: "active query/user evidence is blank"}
}
memoryRows := []map[string]any{}
memoryTrace := map[string]any{}
disabled := a.MemoryDisabled || memoryOff(os.Getenv("RUNE_MEMORY_OFF"))
if !disabled {
if a.Memory == nil {
return AssembledContext{}, &ContextError{Code: "NATIVE_UNAVAILABLE", Op: "recall", Message: "canonical memory store is not composed"}
}
result, err := a.Memory.Recall(ctx, req.DBPath, req.Query, 24, a.MinScore, nil)
if err != nil {
return AssembledContext{}, err
}
memoryRows = result.Rows
memoryTrace = result.Trace
}
history := req.History
if len(history) > 0 {
last := history[len(history)-1]
if last.Role == "user" && last.Content == req.Query {
history = history[:len(history)-1]
}
}
historyWire := make([]map[string]any, 0, len(history))
for _, message := range history {
historyWire = append(historyWire, contextMessageWire(message))
}
window := a.WindowTokens
if window <= 0 {
window = 8192
}
itemBudget := a.MemoryItemBudget
if itemBudget <= 0 {
itemBudget = 4
}
minScore := a.MinScore
if minScore == 0 {
minScore = 0.42
}
arg := map[string]any{
"query": req.Query, "user": req.Query, "system": req.System,
"history": historyWire, "memory_rows": memoryRows,
"memory_disabled": disabled, "window": window,
"memory_item_budget": itemBudget, "min_score": minScore,
}
if a.BudgetTokens != 0 {
arg["budget"] = a.BudgetTokens
}
if a.MemoryBudgetTokens != 0 {
arg["memory_budget"] = a.MemoryBudgetTokens
}
envelope, err := runContextDecision(ctx, a.Binary, "assemble", arg)
if err != nil {
return AssembledContext{}, err
}
var value map[string]any
if err := json.Unmarshal(envelope.Value, &value); err != nil {
return AssembledContext{}, &ContextError{Code: "PROTOCOL_ERROR", Op: "assemble", Message: "native decision value must be an object", Cause: err}
}
rawMessages, ok := value["messages"]
if !ok {
return AssembledContext{}, &ContextError{Code: "PROTOCOL_ERROR", Op: "assemble", Message: "native decision omitted messages"}
}
encoded, err := json.Marshal(rawMessages)
if err != nil {
return AssembledContext{}, &ContextError{Code: "PROTOCOL_ERROR", Op: "assemble", Message: "native messages could not be decoded", Cause: err}
}
var messages []ChatMessage
if err := json.Unmarshal(encoded, &messages); err != nil {
return AssembledContext{}, &ContextError{Code: "PROTOCOL_ERROR", Op: "assemble", Message: "native messages have an invalid shape", Cause: err}
}
if len(messages) == 0 {
return AssembledContext{}, &ContextError{Code: "PROTOCOL_ERROR", Op: "assemble", Message: "native decision returned an empty model payload"}
}
trace := map[string]any{
"context": "go-context",
"native": envelope.Trace,
"memory": memoryTrace,
}
for _, key := range []string{
"status", "memory_status", "memory_abstained", "memory_reason",
"memory_candidates", "memory_selected", "memory_admitted", "memory_tokens",
"budget", "memory_budget", "memory_used", "admitted_total", "admitted", "dropped",
} {
if item, exists := value[key]; exists {
trace[key] = item
}
}
return AssembledContext{Messages: messages, Trace: trace}, nil
}
type contextEnvelope struct {
OK bool `json:"ok"`
Value json.RawMessage `json:"value"`
Trace map[string]any `json:"trace"`
Error *struct {
Code string `json:"code"`
Message string `json:"message"`
} `json:"error"`
}
func runContextDecision(ctx context.Context, binary, operation string, arg map[string]any) (*contextEnvelope, error) {
if binary == "" {
return nil, &ContextError{Code: "NATIVE_UNAVAILABLE", Op: operation, Message: "attention-capsule native binary not found"}
}
info, err := os.Stat(binary)
if err != nil || info.IsDir() {
return nil, &ContextError{Code: "NATIVE_UNAVAILABLE", Op: operation, Message: "attention-capsule native binary not found: " + binary, Cause: err}
}
payload, err := json.Marshal(map[string]any{"op": operation, "arg": arg})
if err != nil {
return nil, &ContextError{Code: "PROTOCOL_ERROR", Op: operation, Message: "native request could not be encoded", Cause: err}
}
cmd := exec.CommandContext(ctx, binary, "eval")
cmd.Stdin = bytes.NewReader(append(payload, '\n'))
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
runErr := cmd.Run()
var envelope contextEnvelope
if err := json.Unmarshal(stdout.Bytes(), &envelope); err != nil {
code := "PROTOCOL_ERROR"
message := "attention-capsule returned invalid NDJSON"
if runErr != nil && stdout.Len() == 0 {
code = "NATIVE_UNAVAILABLE"
message = fmt.Sprintf("attention-capsule call failed: %v (%s)", runErr, strings.TrimSpace(stderr.String()))
}
return nil, &ContextError{Code: code, Op: operation, Message: message, Cause: err}
}
if !envelope.OK {
code, message := "CONTEXT_DECISION_ERROR", "attention-capsule decision failed"
if envelope.Error != nil {
code, message = envelope.Error.Code, envelope.Error.Message
}
return nil, &ContextError{Code: code, Op: operation, Message: message, Cause: runErr}
}
if runErr != nil {
return nil, &ContextError{Code: "PROTOCOL_ERROR", Op: operation, Message: fmt.Sprintf("attention-capsule returned success with exit failure: %v", runErr), Cause: runErr}
}
if len(envelope.Value) == 0 || string(envelope.Value) == "null" {
return nil, &ContextError{Code: "PROTOCOL_ERROR", Op: operation, Message: "attention-capsule omitted its value"}
}
if contextName, ok := envelope.Trace["context"].(string); !ok || contextName != "rust-attention-capsule" {
return nil, &ContextError{Code: "PROTOCOL_ERROR", Op: operation, Message: "attention-capsule omitted its native trace"}
}
return &envelope, nil
}
@@ -0,0 +1,305 @@
package runecore
import (
"context"
"encoding/json"
"errors"
"net/http"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
type contextAssemblerFunc func(context.Context, ContextRequest) (AssembledContext, error)
func (fn contextAssemblerFunc) Assemble(ctx context.Context, req ContextRequest) (AssembledContext, error) {
return fn(ctx, req)
}
type passthroughContextAssembler struct{}
func (passthroughContextAssembler) Assemble(_ context.Context, req ContextRequest) (AssembledContext, error) {
return AssembledContext{
Messages: append([]ChatMessage(nil), req.History...),
Trace: map[string]any{"context": "go-context-test"},
}, nil
}
func nativeDecisionScript(t *testing.T, response string) (string, string) {
t.Helper()
dir := t.TempDir()
capture := filepath.Join(dir, "request.json")
binary := filepath.Join(dir, "attention-capsule")
script := "#!/bin/sh\ncat >\"" + capture + "\"\nprintf '%s\\n' '" + response + "'\n"
if err := os.WriteFile(binary, []byte(script), 0o755); err != nil {
t.Fatal(err)
}
return binary, capture
}
func TestNativeContextAssemblerUsesCanonicalRecallAndRustDecision(t *testing.T) {
response := `{"ok":true,"value":{"status":"ADMITTED","memory_status":"ADMITTED","memory_abstained":false,"messages":[{"role":"system","content":"native fact"},{"role":"user","content":"question"}],"budget":100,"admitted_total":2},"trace":{"context":"rust-attention-capsule","selection":"memory_selector.select_core"}}`
binary, capture := nativeDecisionScript(t, response)
var recallQuery string
memory := memoryStoreFunc(func(_ context.Context, dbPath, query string, budget int, minScore float64, types []string) (MemoryRecallResult, error) {
recallQuery = query
if dbPath != "/tmp/canonical.db" || budget != 24 || minScore != 0.42 {
t.Fatalf("recall arguments db=%q budget=%d floor=%v", dbPath, budget, minScore)
}
return MemoryRecallResult{
Rows: []map[string]any{{"id": 7, "text": "native fact", "relevance": 0.9}},
Trace: map[string]any{"store": "go-memory-store"},
}, nil
})
assembler := NewNativeContextAssembler(memory, binary)
got, err := assembler.Assemble(context.Background(), ContextRequest{
Channel: "chat", Query: "question", Model: "model", DBPath: "/tmp/canonical.db",
History: []ChatMessage{{Role: "user", Content: "question"}},
})
if err != nil {
t.Fatal(err)
}
if recallQuery != "question" || len(got.Messages) != 2 || got.Messages[0].Content != "native fact" {
t.Fatalf("assembled context = %+v, recall query %q", got, recallQuery)
}
if got.Trace["context"] != "go-context" {
t.Fatalf("assembler trace = %v", got.Trace)
}
nativeTrace := got.Trace["native"].(map[string]any)
if nativeTrace["context"] != "rust-attention-capsule" || got.Trace["memory"].(map[string]any)["store"] != "go-memory-store" {
t.Fatalf("native/memory trace = %v", got.Trace)
}
raw, err := os.ReadFile(capture)
if err != nil {
t.Fatal(err)
}
var request map[string]any
if err := json.Unmarshal(raw, &request); err != nil {
t.Fatal(err)
}
if request["op"] != "assemble" {
t.Fatalf("native request = %v", request)
}
arg := request["arg"].(map[string]any)
rows := arg["memory_rows"].([]any)
if rows[0].(map[string]any)["text"] != "native fact" {
t.Fatalf("Rust did not receive canonical raw rows: %v", arg)
}
}
func TestNativeContextAssemblerFailuresAreTyped(t *testing.T) {
memory := memoryStoreFunc(func(context.Context, string, string, int, float64, []string) (MemoryRecallResult, error) {
return MemoryRecallResult{Rows: []map[string]any{}, Trace: map[string]any{}}, nil
})
assembler := NewNativeContextAssembler(memory, filepath.Join(t.TempDir(), "missing"))
_, err := assembler.Assemble(context.Background(), ContextRequest{Query: "question", History: []ChatMessage{{Role: "user", Content: "question"}}})
var contextErr *ContextError
if !errors.As(err, &contextErr) || contextErr.Code != "NATIVE_UNAVAILABLE" {
t.Fatalf("missing native error = %v", err)
}
binary, _ := nativeDecisionScript(t, "not-json")
assembler = NewNativeContextAssembler(memory, binary)
_, err = assembler.Assemble(context.Background(), ContextRequest{Query: "question", History: []ChatMessage{{Role: "user", Content: "question"}}})
if !errors.As(err, &contextErr) || contextErr.Code != "PROTOCOL_ERROR" {
t.Fatalf("malformed native error = %v", err)
}
binary, _ = nativeDecisionScript(t, `{"ok":true,"value":{"status":"ADMITTED","messages":[{"role":"user","content":"question"}]},"trace":null}`)
assembler = NewNativeContextAssembler(memory, binary)
_, err = assembler.Assemble(context.Background(), ContextRequest{Query: "question", History: []ChatMessage{{Role: "user", Content: "question"}}})
if !errors.As(err, &contextErr) || contextErr.Code != "PROTOCOL_ERROR" {
t.Fatalf("missing native trace error = %v", err)
}
}
func TestNativeContextAssemblerPreservesCanonicalStoreFailure(t *testing.T) {
memory := memoryStoreFunc(func(context.Context, string, string, int, float64, []string) (MemoryRecallResult, error) {
return MemoryRecallResult{}, &MemoryError{Code: "STORE_ERROR", Message: "injected"}
})
assembler := NewNativeContextAssembler(memory, filepath.Join(t.TempDir(), "unused"))
_, err := assembler.Assemble(context.Background(), ContextRequest{Query: "question"})
var memoryErr *MemoryError
if !errors.As(err, &memoryErr) || memoryErr.Code != "STORE_ERROR" {
t.Fatalf("store failure = %v", err)
}
}
func TestEngineTurnStreamInvokesAssemblerAndSendsItsExactMessages(t *testing.T) {
fm := &fakeModel{replies: map[string]string{"": "reply"}}
eng, _ := newTestEngine(t, fm)
called := 0
admitted := []ChatMessage{{Role: "system", Content: "native capsule"}, {Role: "user", Content: "question"}}
eng.SetContextAssembler(contextAssemblerFunc(func(_ context.Context, req ContextRequest) (AssembledContext, error) {
called++
if req.Channel != "chat" || req.Query != "question" || len(req.History) != 1 {
t.Fatalf("assembly request = %+v", req)
}
return AssembledContext{Messages: admitted, Trace: map[string]any{"context": "go-context"}}, nil
}))
res, err := eng.TurnStream(context.Background(), TurnRequest{Channel: "chat", UserText: "question"}, nil)
if err != nil {
t.Fatal(err)
}
if called != 1 || res.ContextTrace["context"] != "go-context" {
t.Fatalf("assembler calls=%d result=%+v", called, res)
}
calls := fm.recorded()
if len(calls) != 1 || len(calls[0].Messages) != 2 || calls[0].Messages[0].Content != "native capsule" {
t.Fatalf("model received %+v", calls)
}
}
func TestEngineNativeContextFailureNeverCallsModel(t *testing.T) {
fm := &fakeModel{replies: map[string]string{"": "reply"}}
eng, _ := newTestEngine(t, fm)
eng.SetContextAssembler(contextAssemblerFunc(func(context.Context, ContextRequest) (AssembledContext, error) {
return AssembledContext{}, &ContextError{Code: "NATIVE_UNAVAILABLE", Op: "assemble", Message: "missing"}
}))
_, err := eng.Turn(context.Background(), TurnRequest{Channel: "chat", UserText: "question"})
var contextErr *ContextError
if !errors.As(err, &contextErr) || contextErr.Code != "NATIVE_UNAVAILABLE" {
t.Fatalf("turn error = %v", err)
}
if len(fm.recorded()) != 0 {
t.Fatal("model was called after native context failure")
}
}
func TestServingPathExposesContextTraceAndMapsNativeFailure(t *testing.T) {
srv, ts := newTestServer(t, nil, "")
srv.Engine().SetContextAssembler(contextAssemblerFunc(func(_ context.Context, req ContextRequest) (AssembledContext, error) {
return AssembledContext{Messages: req.History, Trace: map[string]any{"context": "go-context"}}, nil
}))
response, body := postJSON(t, ts.URL+"/v1/chat/completions", map[string]any{
"channel": "chat", "messages": []map[string]string{{"role": "user", "content": "question"}},
})
if response.StatusCode != http.StatusOK {
t.Fatalf("serving status=%d body=%v", response.StatusCode, body)
}
runeBlock := body["rune"].(map[string]any)
if runeBlock["context_trace"].(map[string]any)["context"] != "go-context" {
t.Fatalf("serving trace = %v", runeBlock)
}
srv.Engine().SetContextAssembler(contextAssemblerFunc(func(context.Context, ContextRequest) (AssembledContext, error) {
return AssembledContext{}, &ContextError{Code: "NATIVE_UNAVAILABLE", Op: "assemble", Message: "missing"}
}))
response, body = postJSON(t, ts.URL+"/v1/chat/completions", map[string]any{
"channel": "chat", "messages": []map[string]string{{"role": "user", "content": "again"}},
})
if response.StatusCode != http.StatusServiceUnavailable || body["error"].(map[string]any)["code"] != "native_unavailable" {
t.Fatalf("native failure status=%d body=%v", response.StatusCode, body)
}
}
func TestRealNativeServingPathWithCanonicalStore(t *testing.T) {
if os.Getenv("RUNE_CONTEXT_PROOF") != "1" {
t.Skip("set RUNE_CONTEXT_PROOF=1 after building release native engines")
}
memoryBinary := NewNativeMemoryStore("").Binary
rustBinary := NewNativeContextAssembler(nil, "").Binary
for _, binary := range []string{memoryBinary, rustBinary} {
if info, err := os.Stat(binary); err != nil || info.IsDir() {
t.Fatalf("required real native engine %q: %v", binary, err)
}
}
dir := t.TempDir()
dbPath := filepath.Join(dir, "memory.db")
t.Setenv("RUNE_STORE_DB", dbPath)
t.Setenv("RUNE_MEMORY_EMBED", "off")
t.Setenv("RUNE_MEMORY_OFF", "0")
seed := exec.Command(memoryBinary, "add", "Kubernetes native context fact", "--no-dedupe")
seed.Env = os.Environ()
if output, err := seed.CombinedOutput(); err != nil {
t.Fatalf("seed canonical store: %v (%s)", err, output)
}
fm := &fakeModel{replies: map[string]string{"": "reply"}}
eng, _ := newTestEngine(t, fm)
memory := NewNativeMemoryStore(memoryBinary)
eng.SetContextAssembler(NewNativeContextAssembler(memory, rustBinary))
marker := filepath.Join(dir, "python-used")
for _, name := range []string{"python", "python3"} {
script := "#!/bin/sh\nprintf invoked >\"" + marker + "\"\nexit 99\n"
if err := os.WriteFile(filepath.Join(dir, name), []byte(script), 0o755); err != nil {
t.Fatal(err)
}
}
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
res, err := eng.TurnStream(context.Background(), TurnRequest{Channel: "chat", UserText: "Kubernetes native context fact"}, nil)
if err != nil {
t.Fatal(err)
}
if _, err := os.Stat(marker); !os.IsNotExist(err) {
t.Fatal("real native turn invoked a Python context oracle")
}
calls := fm.recorded()
if len(calls) != 1 || !strings.Contains(calls[0].Messages[0].Content, "Kubernetes native context fact") {
t.Fatalf("real canonical memory did not reach model: %+v", calls)
}
if res.ContextTrace["context"] != "go-context" || res.ContextTrace["memory_status"] != "ADMITTED" {
t.Fatalf("real native trace = %v", res.ContextTrace)
}
if res.ContextTrace["native"].(map[string]any)["context"] != "rust-attention-capsule" ||
res.ContextTrace["memory"].(map[string]any)["store"] != "go-memory-store" {
t.Fatalf("real nested traces = %v", res.ContextTrace)
}
}
func TestServingPathPreservesCanonicalMemoryFailureCodes(t *testing.T) {
srv, ts := newTestServer(t, nil, "")
request := map[string]any{"channel": "chat", "messages": []map[string]string{{"role": "user", "content": "question"}}}
for _, test := range []struct {
code string
wantStatus int
wantCode string
}{
{code: "NATIVE_UNAVAILABLE", wantStatus: http.StatusServiceUnavailable, wantCode: "native_unavailable"},
{code: "STORE_ERROR", wantStatus: http.StatusInternalServerError, wantCode: "store_error"},
} {
srv.Engine().SetContextAssembler(NewNativeContextAssembler(memoryStoreFunc(func(context.Context, string, string, int, float64, []string) (MemoryRecallResult, error) {
return MemoryRecallResult{}, &MemoryError{Code: test.code, Message: "injected"}
}), ""))
response, body := postJSON(t, ts.URL+"/v1/chat/completions", request)
if response.StatusCode != test.wantStatus || body["error"].(map[string]any)["code"] != test.wantCode {
t.Fatalf("canonical %s status=%d body=%v", test.code, response.StatusCode, body)
}
}
}
func TestNativeTurnDoesNotInvokePythonContextOracle(t *testing.T) {
response := `{"ok":true,"value":{"status":"ADMITTED","memory_status":"ADMITTED","messages":[{"role":"system","content":"native fact"},{"role":"user","content":"question"}],"budget":100,"admitted_total":2},"trace":{"context":"rust-attention-capsule"}}`
binary, _ := nativeDecisionScript(t, response)
fm := &fakeModel{replies: map[string]string{"": "reply"}}
eng, _ := newTestEngine(t, fm)
memory := memoryStoreFunc(func(context.Context, string, string, int, float64, []string) (MemoryRecallResult, error) {
return MemoryRecallResult{Rows: []map[string]any{{"text": "native fact", "relevance": 0.9}}, Trace: map[string]any{"store": "go-memory-store"}}, nil
})
eng.SetContextAssembler(NewNativeContextAssembler(memory, binary))
dir := t.TempDir()
marker := filepath.Join(dir, "python-used")
for _, name := range []string{"python", "python3"} {
script := "#!/bin/sh\nprintf invoked >\"" + marker + "\"\nexit 99\n"
if err := os.WriteFile(filepath.Join(dir, name), []byte(script), 0o755); err != nil {
t.Fatal(err)
}
}
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
res, err := eng.Turn(context.Background(), TurnRequest{Channel: "chat", UserText: "question"})
if err != nil {
t.Fatal(err)
}
if _, err := os.Stat(marker); !os.IsNotExist(err) {
t.Fatal("Python context oracle was invoked")
}
if res.ContextTrace["context"] != "go-context" || !strings.Contains(fm.recorded()[0].Messages[0].Content, "native fact") {
t.Fatalf("native serving proof result=%+v calls=%+v", res, fm.recorded())
}
}
+157 -87
View File
@@ -122,6 +122,7 @@ type TurnRequest struct {
Channel string
UserText string
Model string
SignalID string
}
// TurnResult is the turn's receipt: both persisted message ids, the
@@ -134,6 +135,12 @@ type TurnResult struct {
Thinking []string
ContextUsed int
ContextLimit int
ContextTrace map[string]any
Task *TaskStateRecord
TaskResult string
Route string
Error string
OperatorTrace map[string]any
}
// servedByLocalBinding answers whether a record's provider spelling
@@ -152,10 +159,12 @@ func servedByLocalBinding(provider string) bool {
// cloud clients (D42: cloud is first-class; a routerless engine keeps
// the typed unrouted refusal).
type Engine struct {
store Store
client ModelClient
home string
router ClientRouter
store Store
client ModelClient
home string
router ClientRouter
contextAssembler ContextAssembler
operator Operator
flightsMu sync.Mutex
inFlight map[string]bool // channel -> a turn is running
@@ -177,9 +186,22 @@ type Engine struct {
// ErrProviderUnrouted. home is the rune home directory holding
// provider_selection.json (the TUI's first-run law record).
func NewEngine(store Store, client ModelClient, home string) *Engine {
return &Engine{store: store, client: client, home: home, inFlight: map[string]bool{}, governor: NewGovernor(home)}
e := &Engine{store: store, client: client, home: home, inFlight: map[string]bool{}, governor: NewGovernor(home)}
e.contextAssembler = NewNativeContextAssembler(NewNativeMemoryStore(""), "")
e.operator = NewNativeOperator(home, store, "", "")
return e
}
// SetContextAssembler composes the mandatory turn-context authority. A nil
// assembler is a typed native-unavailable failure at TurnStream, never a
// raw-history fallback.
func (e *Engine) SetContextAssembler(assembler ContextAssembler) {
e.contextAssembler = assembler
}
// SetOperator composes the mandatory task/conversation/event lifecycle.
func (e *Engine) SetOperator(operator Operator) { e.operator = operator }
// Governor exposes the engine's governor (the server surface reads it
// for /v1/status; tests seed it directly).
func (e *Engine) Governor() *Governor { return e.governor }
@@ -295,57 +317,89 @@ func (e *Engine) TurnStream(ctx context.Context, req TurnRequest, sink StreamSin
return TurnResult{}, ErrChannelBusy
}
defer e.release(req.Channel)
if e.operator == nil {
return TurnResult{}, &OperatorError{Code: "NATIVE_UNAVAILABLE", Op: "intake", Message: "operator lifecycle is not composed"}
}
var model *string
if req.Model != "" {
model = &req.Model
}
start, err := e.operator.Start(ctx, StartRequest{
Description: req.UserText, Channel: req.Channel, Model: model, SignalID: req.SignalID,
})
if err != nil {
return e.failTurn(ctx, start, TurnResult{Route: start.Route.Route, OperatorTrace: start.Trace}, "route_error", err)
}
if start.Duplicate != nil {
result := turnResultFromReceipt(*start.Duplicate)
result.Task = &start.Task
return result, nil
}
// Model resolution (fresh record read every turn — the operator
// switches models while the core runs): the RECORD's provider owns
// the ROUTE (D42: no local default — a cloud-naming record serves
// through the router's cloud client, and an unroutable spelling is
// a typed refusal, never a silent local misroute), the explicit
// model names a model ON the route when the caller passes one.
model := req.Model
taskID := start.Task.TaskID
userMsg := Message{
MsgID: newMsgID(), Ts: utcNow(), Role: "user", Content: req.UserText,
ChannelID: req.Channel, TaskID: &taskID, TokenEstimate: tokenEstimate(req.UserText), SchemaVersion: SnapshotSchemaVersion,
}
if start.Research != nil {
if err := e.operator.Append(ctx, userMsg); err != nil {
return e.failTurn(ctx, start, TurnResult{UserMsgID: userMsg.MsgID, Route: start.Route.Route}, "durable_save_error", err)
}
text := "Research required — dispatch recorded for task " + taskID + ". The canonical research executor is the next step."
result := TurnResult{
UserMsgID: userMsg.MsgID, AssistantText: text, TaskResult: "RESEARCH_REQUIRED",
Route: start.Route.Route, Error: "RESEARCH_REQUIRED", OperatorTrace: start.Trace,
}
return e.finishTurn(ctx, start, result, "research_required", "research_dispatch")
}
modelName := req.Model
rec, recErr := LoadProviderSelection(e.home)
client := e.client
if recErr == nil && !servedByLocalBinding(rec.Provider) {
if e.router == nil {
return TurnResult{}, fmt.Errorf("%w: the record names %q; this core serves it only with a router attached (ollama/local records serve the local binding)", ErrProviderUnrouted, rec.Provider)
err := fmt.Errorf("%w: the record names %q; this core serves it only with a router attached (ollama/local records serve the local binding)", ErrProviderUnrouted, rec.Provider)
return e.failTurn(ctx, start, TurnResult{Route: start.Route.Route}, "model_selection_error", err)
}
routed, err := e.router.ClientFor(ctx, rec.Provider)
if err != nil {
return TurnResult{}, err
return e.failTurn(ctx, start, TurnResult{Route: start.Route.Route}, "model_selection_error", err)
}
client = routed
}
if model == "" {
if modelName == "" {
if recErr != nil {
return TurnResult{}, ErrNoProvider
return e.failTurn(ctx, start, TurnResult{Route: start.Route.Route}, "model_selection_error", ErrNoProvider)
}
model = rec.Model
modelName = rec.Model
}
userMsg := Message{
MsgID: newMsgID(),
Ts: utcNow(),
Role: "user",
Content: req.UserText,
ChannelID: req.Channel,
TokenEstimate: tokenEstimate(req.UserText),
SchemaVersion: SnapshotSchemaVersion,
if err := e.operator.Append(ctx, userMsg); err != nil {
return e.failTurn(ctx, start, TurnResult{UserMsgID: userMsg.MsgID, Model: modelName, Route: start.Route.Route}, "durable_save_error", err)
}
if err := e.store.Append(userMsg); err != nil {
return TurnResult{}, err
if _, err := e.operator.Running(ctx, taskID, modelName); err != nil {
return e.failTurn(ctx, start, TurnResult{UserMsgID: userMsg.MsgID, Model: modelName, Route: start.Route.Route}, "durable_save_error", err)
}
history := e.store.Window(req.Channel, ContextMessages)
payload := make([]ChatMessage, 0, len(history))
for _, m := range history {
cm := ChatMessage{Role: m.Role, Content: m.Content}
if m.Role == "assistant" && m.Thinking != nil {
// write the thinking field back onto the replayed assistant
// message (the provider's parsed shape, mirrored)
cm.Thinking = m.Thinking
collected := make([]ChatMessage, 0, len(history))
for _, message := range history {
wire := ChatMessage{Role: message.Role, Content: message.Content}
if message.Role == "assistant" && message.Thinking != nil {
wire.Thinking = message.Thinking
}
payload = append(payload, cm)
collected = append(collected, wire)
}
if e.contextAssembler == nil {
err := &ContextError{Code: "NATIVE_UNAVAILABLE", Op: "assemble", Message: "context assembler is not composed"}
return e.failTurn(ctx, start, TurnResult{UserMsgID: userMsg.MsgID, Model: modelName, Route: start.Route.Route}, "context_error", err)
}
assembled, err := e.contextAssembler.Assemble(ctx, ContextRequest{
Channel: req.Channel, Query: req.UserText, Model: modelName, History: collected,
})
if err != nil {
return e.failTurn(ctx, start, TurnResult{UserMsgID: userMsg.MsgID, Model: modelName, Route: start.Route.Route}, "context_error", err)
}
payload := assembled.Messages
provider := "local"
if recErr == nil {
@@ -354,80 +408,96 @@ func (e *Engine) TurnStream(ctx context.Context, req TurnRequest, sink StreamSin
if servedByLocalBinding(provider) {
provider = "local"
}
start := time.Now()
reply, err := e.runTurn(ctx, client, model, req.Channel, payload, sink)
started := time.Now()
reply, err := e.runTurn(ctx, client, modelName, req.Channel, payload, sink)
if err != nil {
// the user message stays persisted: the user DID send it (the
// Python oracle's behavior); the failure is honest, not silent
e.governor.RecordError(provider, err)
return TurnResult{UserMsgID: userMsg.MsgID, Model: model}, err
return e.failTurn(ctx, start, TurnResult{UserMsgID: userMsg.MsgID, Model: modelName, Route: start.Route.Route}, "execution_error", err)
}
// D56 throughput telemetry: the governor learns from the real
// turn — the provider's own generated-token accounting when the
// wire carries it, the assistant text's token estimate otherwise
// (an honest estimate, never presented as a wire measurement).
outTokens := reply.OutputTokens
if outTokens <= 0 {
outTokens = tokenEstimate(reply.Content)
}
e.governor.RecordTurn(provider, model, outTokens, time.Since(start), 1)
e.governor.RecordTurn(provider, modelName, outTokens, time.Since(started), 1)
assistant := Message{
MsgID: newMsgID(),
Ts: utcNow(),
Role: "assistant",
Content: reply.Content,
ChannelID: req.Channel,
TokenEstimate: tokenEstimate(reply.Content),
SchemaVersion: SnapshotSchemaVersion,
MsgID: newMsgID(), Ts: utcNow(), Role: "assistant", Content: reply.Content,
ChannelID: req.Channel, TaskID: &taskID, TokenEstimate: tokenEstimate(reply.Content), SchemaVersion: SnapshotSchemaVersion,
}
if len(reply.Thinking) > 0 {
th := strings.Join(reply.Thinking, "\n\n")
assistant.Thinking = &th
thinking := strings.Join(reply.Thinking, "\n\n")
assistant.Thinking = &thinking
}
if err := e.store.Append(assistant); err != nil {
return TurnResult{UserMsgID: userMsg.MsgID, Model: model}, err
if err := e.operator.Append(ctx, assistant); err != nil {
return e.failTurn(ctx, start, TurnResult{UserMsgID: userMsg.MsgID, Model: modelName, Route: start.Route.Route}, "durable_save_error", err)
}
result := TurnResult{
UserMsgID: userMsg.MsgID, AssistantMsgID: assistant.MsgID, AssistantText: reply.Content,
TaskResult: reply.Content, Model: modelName, Thinking: reply.Thinking,
ContextUsed: reply.ContextUsed, ContextLimit: reply.ContextLimit,
ContextTrace: assembled.Trace, Route: start.Route.Route, OperatorTrace: start.Trace,
}
// D57 model-refusal authority: the model's capability claims are
// checked against the runtime's ACTUAL state. A false "I lack
// tools/access" claim is never passed to the owner as truth — the
// engine composes a correction turn, retries once with it on the
// wire, persists the corrected answer, and journals the event.
correction := e.checkRefusal(req.Channel, reply.Content)
if correction != "" {
wireTools := ToolsetFor(modeForDispatch(req.Channel, e.home))
retryMsgs := append(payload, ChatMessage{Role: "assistant", Content: reply.Content})
retryMsgs = append(retryMsgs, ChatMessage{Role: "user", Content: correction})
if retry, err := modelOnce(ctx, client, model, retryMsgs, wireTools, nil); err == nil {
if retry, err := modelOnce(ctx, client, modelName, retryMsgs, wireTools, nil); err == nil {
followUp := Message{
MsgID: newMsgID(),
Ts: utcNow(),
Role: "assistant",
Content: retry.Content,
ChannelID: req.Channel,
TokenEstimate: tokenEstimate(retry.Content),
SchemaVersion: SnapshotSchemaVersion,
MsgID: newMsgID(), Ts: utcNow(), Role: "assistant", Content: retry.Content,
ChannelID: req.Channel, TaskID: &taskID, TokenEstimate: tokenEstimate(retry.Content), SchemaVersion: SnapshotSchemaVersion,
}
if appendErr := e.store.Append(followUp); appendErr == nil {
return TurnResult{
UserMsgID: userMsg.MsgID,
AssistantMsgID: followUp.MsgID,
AssistantText: retry.Content,
Model: model,
ContextUsed: reply.ContextUsed,
ContextLimit: reply.ContextLimit,
}, nil
if appendErr := e.operator.Append(ctx, followUp); appendErr == nil {
result.AssistantMsgID = followUp.MsgID
result.AssistantText = retry.Content
result.TaskResult = retry.Content
}
}
}
return e.finishTurn(ctx, start, result, "completed", "response:"+taskID)
}
func (e *Engine) finishTurn(ctx context.Context, start StartResult, result TurnResult, status, outcomeRef string) (TurnResult, error) {
receipt := TurnReceipt{
SignalID: start.SignalID, TaskID: start.Task.TaskID, UserMsgID: result.UserMsgID, AssistantMsgID: result.AssistantMsgID,
AssistantText: result.AssistantText, TaskResult: result.TaskResult, Model: result.Model,
Route: result.Route, Error: result.Error, CompletedAt: utcNow(), Thinking: result.Thinking,
ContextUsed: result.ContextUsed, ContextLimit: result.ContextLimit,
ContextTrace: result.ContextTrace, OperatorTrace: start.Trace,
}
task, err := e.operator.CompleteTurn(ctx, start.Task.TaskID, status, outcomeRef, receipt)
result.Task = &task
result.OperatorTrace = start.Trace
return result, err
}
func (e *Engine) failTurn(ctx context.Context, start StartResult, result TurnResult, outcomeRef string, cause error) (TurnResult, error) {
result.Route = start.Route.Route
result.OperatorTrace = start.Trace
result.Error = cause.Error()
result.TaskResult = cause.Error()
if start.Task.TaskID == "" {
return result, cause
}
receipt := TurnReceipt{
SignalID: start.SignalID, TaskID: start.Task.TaskID, UserMsgID: result.UserMsgID, TaskResult: result.TaskResult,
Model: result.Model, Route: result.Route, Error: result.Error, CompletedAt: utcNow(), OperatorTrace: start.Trace,
}
task, err := e.operator.CompleteTurn(ctx, start.Task.TaskID, "failed", outcomeRef, receipt)
result.Task = &task
if err != nil {
return result, errors.Join(cause, err)
}
return result, cause
}
func turnResultFromReceipt(receipt TurnReceipt) TurnResult {
return TurnResult{
UserMsgID: userMsg.MsgID,
AssistantMsgID: assistant.MsgID,
AssistantText: reply.Content,
Model: model,
Thinking: reply.Thinking,
ContextUsed: reply.ContextUsed,
ContextLimit: reply.ContextLimit,
}, nil
UserMsgID: receipt.UserMsgID, AssistantMsgID: receipt.AssistantMsgID,
AssistantText: receipt.AssistantText, TaskResult: receipt.TaskResult,
Model: receipt.Model, Route: receipt.Route, Error: receipt.Error,
Thinking: receipt.Thinking, ContextUsed: receipt.ContextUsed, ContextLimit: receipt.ContextLimit,
ContextTrace: receipt.ContextTrace, OperatorTrace: receipt.OperatorTrace,
}
}
@@ -91,7 +91,9 @@ func newTestEngine(t *testing.T, model ModelClient) (*Engine, *TailStore) {
if err := writeTestProvider(home, "local", "test-model-27b"); err != nil {
t.Fatal(err)
}
return NewEngine(st, model, home), st
eng := NewEngine(st, model, home)
eng.SetContextAssembler(passthroughContextAssembler{})
return eng, st
}
func writeTestProvider(home, provider, model string) error {
@@ -372,8 +374,9 @@ func TestEngineServesLocalRecordSpellings(t *testing.T) {
if err := writeTestProvider(home, provider, "legacy-model"); err != nil {
t.Fatal(err)
}
if _, err := NewEngine(st, fm, home).Turn(context.Background(),
TurnRequest{Channel: "chat", UserText: "x"}); err != nil {
eng := NewEngine(st, fm, home)
eng.SetContextAssembler(passthroughContextAssembler{})
if _, err := eng.Turn(context.Background(), TurnRequest{Channel: "chat", UserText: "x"}); err != nil {
t.Fatalf("legacy spelling %q must stay served: %v", provider, err)
}
if got := fm.recorded()[0].Model; got != "legacy-model" {
@@ -0,0 +1,106 @@
package runecore
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"runtime"
)
type MemoryRecallResult struct {
Rows []map[string]any
Trace map[string]any
}
type MemoryStore interface {
Recall(ctx context.Context, dbPath, query string, budget int, minScore float64, types []string) (MemoryRecallResult, error)
}
type MemoryError struct {
Code string
Message string
Cause error
}
func (e *MemoryError) Error() string { return e.Code + ": " + e.Message }
func (e *MemoryError) Unwrap() error { return e.Cause }
type NativeMemoryStore struct {
Binary string
}
func NewNativeMemoryStore(binary string) *NativeMemoryStore {
if binary == "" {
binary = os.Getenv("RUNE_MEMORY_STORE_BIN")
}
if binary == "" {
_, file, _, ok := runtime.Caller(0)
if ok {
dir := filepath.Dir(file)
for i := 0; i < 5; i++ {
dir = filepath.Dir(dir)
}
binary = filepath.Join(dir, "src", "go", "memory-store-deep", "memory-store-deep")
}
}
return &NativeMemoryStore{Binary: binary}
}
func (s *NativeMemoryStore) Recall(ctx context.Context, dbPath, query string, budget int, minScore float64, types []string) (MemoryRecallResult, error) {
if s.Binary == "" {
return MemoryRecallResult{}, &MemoryError{Code: "NATIVE_UNAVAILABLE", Message: "canonical memory-store binary not found"}
}
if info, err := os.Stat(s.Binary); err != nil || info.IsDir() {
return MemoryRecallResult{}, &MemoryError{Code: "NATIVE_UNAVAILABLE", Message: "canonical memory-store binary not found: " + s.Binary, Cause: err}
}
payload, err := json.Marshal(map[string]any{
"op": "recall",
"arg": map[string]any{
"db_path": dbPath, "query": query, "budget": budget,
"min_score": minScore, "memory_types": types,
},
})
if err != nil {
return MemoryRecallResult{}, err
}
cmd := exec.CommandContext(ctx, s.Binary, "eval")
cmd.Stdin = bytes.NewReader(append(payload, '\n'))
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
runErr := cmd.Run()
var envelope struct {
OK bool `json:"ok"`
Value []map[string]any `json:"value"`
Trace map[string]any `json:"trace"`
Error *struct {
Code string `json:"code"`
Message string `json:"message"`
} `json:"error"`
}
if err := json.Unmarshal(stdout.Bytes(), &envelope); err != nil {
return MemoryRecallResult{}, &MemoryError{Code: "PROTOCOL_ERROR", Message: "canonical memory-store returned non-JSON", Cause: err}
}
if !envelope.OK {
code, message := "STORE_ERROR", "canonical memory-store operation failed"
if envelope.Error != nil {
code, message = envelope.Error.Code, envelope.Error.Message
}
return MemoryRecallResult{}, &MemoryError{Code: code, Message: message}
}
if runErr != nil {
code := "STORE_ERROR"
if exit, ok := runErr.(*exec.ExitError); ok && exit.ExitCode() == 127 {
code = "NATIVE_UNAVAILABLE"
}
return MemoryRecallResult{}, &MemoryError{Code: code, Message: fmt.Sprintf("canonical memory-store call failed: %v (%s)", runErr, stderr.String()), Cause: runErr}
}
if envelope.Value == nil {
envelope.Value = []map[string]any{}
}
return MemoryRecallResult{Rows: envelope.Value, Trace: envelope.Trace}, nil
}
@@ -0,0 +1,85 @@
package runecore
import (
"context"
"encoding/json"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
)
func TestNativeMemoryStoreRecallProtocol(t *testing.T) {
dir := t.TempDir()
trace := filepath.Join(dir, "request.json")
binary := filepath.Join(dir, "memory-store")
script := "#!/bin/sh\ncat >\"" + trace + "\"\nprintf '%s\n' '{\"ok\":true,\"value\":[{\"id\":7,\"text\":\"native fact\",\"relevance\":4.1}],\"trace\":{\"store\":\"go-memory-store\",\"kernels\":[\"memory_retrieval.score_recall_rows\"]}}'\n"
if err := os.WriteFile(binary, []byte(script), 0o755); err != nil {
t.Fatal(err)
}
store := NewNativeMemoryStore(binary)
result, err := store.Recall(context.Background(), "/tmp/memory.db", "native", 4, 0.42, []string{"self"})
if err != nil {
t.Fatal(err)
}
if len(result.Rows) != 1 || result.Rows[0]["text"] != "native fact" || result.Trace["store"] != "go-memory-store" {
t.Fatalf("native result = %+v", result)
}
raw, err := os.ReadFile(trace)
if err != nil {
t.Fatal(err)
}
var request map[string]interface{}
if err := json.Unmarshal(raw, &request); err != nil {
t.Fatal(err)
}
if request["op"] != "recall" {
t.Fatalf("request = %v", request)
}
}
func TestServerMemoryRecallUsesComposedNativeStore(t *testing.T) {
srv, ts := newTestServer(t, nil, "")
srv.SetMemoryStore(memoryStoreFunc(func(ctx context.Context, dbPath, query string, budget int, minScore float64, types []string) (MemoryRecallResult, error) {
return MemoryRecallResult{
Rows: []map[string]any{{"id": 7, "text": "native fact", "relevance": 4.1}},
Trace: map[string]any{"store": "go-memory-store"},
}, nil
}))
response, body := postJSON(t, ts.URL+"/v1/memory/recall", map[string]any{"query": "native", "db_path": "/tmp/memory.db"})
if response.StatusCode != http.StatusOK || body["core"] != "go" {
t.Fatalf("memory recall status=%d body=%v", response.StatusCode, body)
}
trace := body["trace"].(map[string]any)
if trace["store"] != "go-memory-store" {
t.Fatalf("serving trace = %v", trace)
}
}
func TestNativeMemoryStorePreservesStoreError(t *testing.T) {
binary := filepath.Join(t.TempDir(), "memory-store")
script := "#!/bin/sh\ncat >/dev/null\nprintf '%s\n' '{\"ok\":false,\"error\":{\"code\":\"STORE_ERROR\",\"message\":\"injected store failure\"}}'\nexit 1\n"
if err := os.WriteFile(binary, []byte(script), 0o755); err != nil {
t.Fatal(err)
}
store := NewNativeMemoryStore(binary)
_, err := store.Recall(context.Background(), "/tmp/memory.db", "native", 4, 0.42, nil)
if err == nil || !strings.Contains(err.Error(), "STORE_ERROR") || strings.Contains(err.Error(), "NATIVE_UNAVAILABLE") {
t.Fatalf("native store error was misclassified: %v", err)
}
}
func TestNativeMemoryStoreMissingBinaryIsTyped(t *testing.T) {
store := NewNativeMemoryStore(filepath.Join(t.TempDir(), "missing"))
_, err := store.Recall(context.Background(), "/tmp/memory.db", "native", 4, 0.42, nil)
if err == nil || !strings.Contains(err.Error(), "NATIVE_UNAVAILABLE") {
t.Fatalf("missing native error = %v", err)
}
}
type memoryStoreFunc func(context.Context, string, string, int, float64, []string) (MemoryRecallResult, error)
func (fn memoryStoreFunc) Recall(ctx context.Context, dbPath, query string, budget int, minScore float64, types []string) (MemoryRecallResult, error) {
return fn(ctx, dbPath, query, budget, minScore, types)
}
@@ -0,0 +1,154 @@
package runecore
import (
"context"
"path/filepath"
)
type StartRequest struct {
Description string
Channel string
Model *string
SignalID string
}
type ResearchDisposition struct {
Dispatch map[string]any
}
type StartResult struct {
Task TaskStateRecord
SignalID string
Family string
Route RouteDecision
Research *ResearchDisposition
Duplicate *TurnReceipt
Trace map[string]any
}
type OperatorTaskView struct {
Record TaskStateRecord
Receipt *TurnReceipt
}
type OperatorSnapshot struct {
Tasks []OperatorTaskView
Active *TaskStateRecord
}
type Operator interface {
Start(context.Context, StartRequest) (StartResult, error)
Running(context.Context, string, string) (TaskStateRecord, error)
CompleteTurn(context.Context, string, string, string, TurnReceipt) (TaskStateRecord, error)
Append(context.Context, Message) error
Snapshot(string) OperatorSnapshot
}
type NativeOperator struct {
home string
tasks *TaskStateRepository
conversation *ConversationRepository
router RouteClient
initErr error
}
func NewNativeOperator(home string, store Store, lifecycleBinary, routerBinary string) *NativeOperator {
return NewNativeOperatorWithClients(home, store, NewNativeOperatorDecisions(lifecycleBinary), NewProcessRouteClient(routerBinary))
}
func NewNativeOperatorWithClients(home string, store Store, decisions OperatorDecisions, router RouteClient) *NativeOperator {
operator := &NativeOperator{home: home, router: router}
events, err := OpenEventStore(filepath.Join(home, "events.jsonl"), 8, decisions)
if err != nil {
operator.initErr = err
return operator
}
hot, err := OpenAtomicStore(filepath.Join(home, "hot"))
if err != nil {
operator.initErr = err
return operator
}
operator.tasks, err = OpenTaskStateRepository(events, hot, decisions)
if err != nil {
operator.initErr = err
return operator
}
operator.conversation = NewConversationRepository(store, events, decisions)
return operator
}
func (o *NativeOperator) Start(ctx context.Context, request StartRequest) (StartResult, error) {
if o.initErr != nil {
return StartResult{}, o.initErr
}
if request.SignalID != "" {
if receipt, found := o.tasks.TurnReceipt(request.SignalID); found {
task, _ := o.tasks.Get(receipt.TaskID)
return StartResult{Task: task, Duplicate: &receipt, Trace: receipt.OperatorTrace}, nil
}
}
intake, err := o.tasks.Intake(ctx, request.Description, request.Channel, request.Model)
if err != nil {
return StartResult{}, err
}
result := StartResult{
Task: intake.Record, SignalID: request.SignalID, Family: intake.Family,
Trace: map[string]any{
"operator": "go-operator", "native": "rust-operator-lifecycle", "events": "go-event-store",
"family": intake.Family, "intake_action": intake.Action,
},
}
result.Route, err = o.router.Classify(ctx, request.Description, intake.Record.TaskID)
if err != nil {
return result, err
}
result.Trace["route"] = result.Route.Route
var dispatch map[string]any
if result.Route.Route == "RESEARCHED_CHANGE" {
dispatch, err = o.router.Dispatch(ctx, intake.Record.TaskID, request.Description, result.Route)
if err != nil {
return result, err
}
result.Research = &ResearchDisposition{Dispatch: dispatch}
}
if result.Route.DecidedAt != "" {
if err := o.router.Record(ctx, filepath.Join(o.home, "routes", "route.jsonl"), result.Route, dispatch); err != nil {
return result, err
}
result.Task, err = o.tasks.SetRouteRef(ctx, intake.Record.TaskID, "route:"+result.Route.DecidedAt)
if err != nil {
return result, err
}
}
return result, nil
}
func (o *NativeOperator) Running(ctx context.Context, taskID, _ string) (TaskStateRecord, error) {
return o.tasks.Transition(ctx, taskID, "running", "")
}
func (o *NativeOperator) CompleteTurn(ctx context.Context, taskID, status, outcomeRef string, receipt TurnReceipt) (TaskStateRecord, error) {
return o.tasks.CompleteTurn(ctx, taskID, status, outcomeRef, receipt)
}
func (o *NativeOperator) Append(ctx context.Context, message Message) error {
return o.conversation.Append(ctx, message)
}
func (o *NativeOperator) Snapshot(channel string) OperatorSnapshot {
snapshot := OperatorSnapshot{}
if o.tasks == nil {
return snapshot
}
for _, record := range o.tasks.ListAll() {
view := OperatorTaskView{Record: record}
if receipt, found := o.tasks.ReceiptForTask(record.TaskID); found {
view.Receipt = &receipt
}
snapshot.Tasks = append(snapshot.Tasks, view)
}
if active, found := o.tasks.Active(channel); found {
snapshot.Active = &active
}
return snapshot
}
@@ -0,0 +1,86 @@
package runecore
import (
"fmt"
"io"
"os"
"path/filepath"
"strings"
"sync"
)
type AtomicStore struct {
root string
mu sync.Mutex
}
func OpenAtomicStore(root string) (*AtomicStore, error) {
if err := os.MkdirAll(root, 0o755); err != nil {
return nil, fmt.Errorf("atomic store root: %w", err)
}
return &AtomicStore{root: root}, nil
}
func (s *AtomicStore) Root() string { return s.root }
func atomicName(name string) error {
if name == "" || name == "." || name == ".." || filepath.Base(name) != name || strings.ContainsAny(name, `/\`) {
return fmt.Errorf("atomic store invalid record name %q", name)
}
return nil
}
func (s *AtomicStore) Read(name string) (string, bool, error) {
if err := atomicName(name); err != nil {
return "", false, err
}
raw, err := os.ReadFile(filepath.Join(s.root, name))
if os.IsNotExist(err) {
return "", false, nil
}
if err != nil {
return "", false, err
}
return string(raw), true, nil
}
func (s *AtomicStore) Write(name, content string) error {
return s.WriteFunc(name, func(w io.Writer) error {
_, err := io.WriteString(w, content)
return err
})
}
func (s *AtomicStore) WriteFunc(name string, writer func(io.Writer) error) error {
if err := atomicName(name); err != nil {
return err
}
s.mu.Lock()
defer s.mu.Unlock()
file, err := os.CreateTemp(s.root, name+".*.tmp")
if err != nil {
return err
}
tmp := file.Name()
defer os.Remove(tmp)
if err := writer(file); err != nil {
_ = file.Close()
return err
}
if err := file.Sync(); err != nil {
_ = file.Close()
return err
}
if err := file.Close(); err != nil {
return err
}
if err := os.Rename(tmp, filepath.Join(s.root, name)); err != nil {
return err
}
dir, err := os.Open(s.root)
if err != nil {
return err
}
defer dir.Close()
return dir.Sync()
}
@@ -0,0 +1,112 @@
package runecore
import (
"errors"
"io"
"os"
"path/filepath"
"strings"
"sync"
"testing"
)
func TestAtomicStoreRoundTripAndMissing(t *testing.T) {
store, err := OpenAtomicStore(filepath.Join(t.TempDir(), "state"))
if err != nil {
t.Fatal(err)
}
if _, found, err := store.Read("missing.json"); err != nil || found {
t.Fatalf("missing read found=%v err=%v", found, err)
}
if err := store.Write("cursor.json", `{"offset":7}`); err != nil {
t.Fatal(err)
}
got, found, err := store.Read("cursor.json")
if err != nil || !found || got != `{"offset":7}` {
t.Fatalf("round trip got=%q found=%v err=%v", got, found, err)
}
entries, err := os.ReadDir(store.Root())
if err != nil || len(entries) != 1 {
t.Fatalf("temporary files leaked: entries=%v err=%v", entries, err)
}
}
func TestAtomicStoreFailedWriterPreservesOriginalAndCleansTemp(t *testing.T) {
store, err := OpenAtomicStore(t.TempDir())
if err != nil {
t.Fatal(err)
}
if err := store.Write("state.json", "original"); err != nil {
t.Fatal(err)
}
boom := errors.New("boom")
err = store.WriteFunc("state.json", func(w io.Writer) error {
if _, writeErr := w.Write([]byte("corrupt")); writeErr != nil {
return writeErr
}
return boom
})
if !errors.Is(err, boom) {
t.Fatalf("writer error = %v", err)
}
got, _, _ := store.Read("state.json")
if got != "original" {
t.Fatalf("original changed to %q", got)
}
entries, _ := os.ReadDir(store.Root())
if len(entries) != 1 || entries[0].Name() != "state.json" {
t.Fatalf("temporary files leaked: %v", entries)
}
}
func TestAtomicStoreConcurrentReadersNeverObserveTornCheckpoint(t *testing.T) {
store, err := OpenAtomicStore(t.TempDir())
if err != nil {
t.Fatal(err)
}
oldValue := `{"version":"` + strings.Repeat("a", 128*1024) + `"}`
newValue := `{"version":"` + strings.Repeat("b", 128*1024) + `"}`
if err := store.Write("state.json", oldValue); err != nil {
t.Fatal(err)
}
stop := make(chan struct{})
failures := make(chan string, 1)
var readers sync.WaitGroup
for range 4 {
readers.Add(1)
go func() {
defer readers.Done()
for {
select {
case <-stop:
return
default:
}
got, found, readErr := store.Read("state.json")
if readErr != nil || !found || (got != oldValue && got != newValue) {
select {
case failures <- got:
default:
}
return
}
}
}()
}
for i := range 30 {
value := oldValue
if i%2 == 1 {
value = newValue
}
if err := store.Write("state.json", value); err != nil {
t.Fatal(err)
}
}
close(stop)
readers.Wait()
select {
case failure := <-failures:
t.Fatalf("reader observed torn checkpoint of length %d", len(failure))
default:
}
}
@@ -0,0 +1,57 @@
package runecore
import (
"context"
"encoding/json"
)
type ConversationRepository struct {
store Store
events *EventStore
decisions OperatorDecisions
}
func NewConversationRepository(store Store, events *EventStore, decisions OperatorDecisions) *ConversationRepository {
return &ConversationRepository{store: store, events: events, decisions: decisions}
}
func (r *ConversationRepository) Append(ctx context.Context, message Message) error {
normalized, err := r.decisions.Validate(ctx, "message", message)
if err != nil {
return err
}
if err := json.Unmarshal(normalized, &message); err != nil {
return &OperatorError{Code: "PROTOCOL_ERROR", Op: "conversation.append", Message: "normalized message has an invalid shape", Cause: err}
}
if _, found := r.store.Get(message.MsgID); found {
return nil
}
hash, err := r.decisions.PayloadHash(ctx, "message", message)
if err != nil {
return err
}
_, err = r.events.Append(ctx, EventEnvelope{
SchemaVersion: SnapshotSchemaVersion,
EventType: "conversation.message",
PartitionKey: "channel:" + message.ChannelID,
EventID: "evt_" + message.MsgID,
ActorID: "conversation_state_repository",
PayloadHash: hash,
Timestamp: utcNow(),
})
if err != nil {
return err
}
if err := r.events.Flush(); err != nil {
return err
}
return r.store.Append(message)
}
func (r *ConversationRepository) Window(channel string, n int) []Message {
return r.store.Window(channel, n)
}
func (r *ConversationRepository) Get(msgID string) (Message, bool) {
return r.store.Get(msgID)
}
@@ -0,0 +1,43 @@
package runecore
import (
"context"
"path/filepath"
"testing"
)
func TestConversationRepositoryPersistsTaskScopedRecordsAndEvents(t *testing.T) {
root := t.TempDir()
decisions := NewNativeOperatorDecisions(operatorLifecycleTestBinary(t))
events, err := OpenEventStore(filepath.Join(root, "events.jsonl"), 8, decisions)
if err != nil {
t.Fatal(err)
}
store := openTestStore(t, filepath.Join(root, "conversations"))
conversation := NewConversationRepository(store, events, decisions)
taskID := "task_1"
first := Message{MsgID: "msg_user", Ts: "now", Role: "user", Content: "hello", ChannelID: "task", TaskID: &taskID, SchemaVersion: 1}
thinking := "reason"
second := Message{MsgID: "msg_assistant", Ts: "later", Role: "assistant", Content: "reply", ChannelID: "task", TaskID: &taskID, Thinking: &thinking, SchemaVersion: 1}
for _, message := range []Message{first, second, first} {
if err := conversation.Append(context.Background(), message); err != nil {
t.Fatal(err)
}
}
window := conversation.Window("task", 8)
if len(window) != 2 || window[0].TaskID == nil || *window[0].TaskID != taskID {
t.Fatalf("window=%+v", window)
}
replay, err := events.Replay()
if err != nil || len(replay) != 2 {
t.Fatalf("events=%+v err=%v", replay, err)
}
for _, event := range replay {
if event.EventType != "conversation.message" || event.PartitionKey != "channel:task" || len(event.PayloadHash) != len("sha256:")+64 {
t.Fatalf("event=%+v", event)
}
}
if got, found := conversation.Get(first.MsgID); !found || got.Content != first.Content {
t.Fatalf("exact read got=%+v found=%v", got, found)
}
}
@@ -0,0 +1,218 @@
package runecore
import (
"bufio"
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"sync"
)
type UnmigratedDialectError struct{ Message string }
func (e *UnmigratedDialectError) Error() string { return e.Message }
type CorruptEventStoreError struct {
Offset int
Message string
Cause error
}
func (e *CorruptEventStoreError) Error() string {
return fmt.Sprintf("corrupt event at offset %d: %s", e.Offset, e.Message)
}
func (e *CorruptEventStoreError) Unwrap() error { return e.Cause }
type EventOffset struct {
Offset int
Event EventEnvelope
}
type knownEvent struct {
offset int
event EventEnvelope
}
type EventStore struct {
path string
batchSize int
decisions OperatorDecisions
mu sync.Mutex
buffer []EventEnvelope
offset int
known map[string]knownEvent
}
func OpenEventStore(path string, batchSize int, decisions OperatorDecisions) (*EventStore, error) {
if batchSize <= 0 {
batchSize = 1
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return nil, err
}
s := &EventStore{path: path, batchSize: batchSize, decisions: decisions, known: map[string]knownEvent{}}
rows, physicalLines, err := readEventRows(path, 1)
if err != nil {
return nil, err
}
s.offset = physicalLines
for _, row := range rows {
s.known[row.Event.EventID] = knownEvent{offset: row.Offset, event: row.Event}
}
return s, nil
}
func decodeEvent(raw []byte, offset int) (EventEnvelope, error) {
var fields map[string]json.RawMessage
if err := json.Unmarshal(raw, &fields); err != nil {
return EventEnvelope{}, &CorruptEventStoreError{Offset: offset, Message: err.Error(), Cause: err}
}
if _, ok := fields["event_id"]; !ok {
return EventEnvelope{}, &UnmigratedDialectError{Message: "old-dialect event rejected — canonical event_id is required"}
}
for _, key := range []string{"payload", "source", "channel", "authority_class", "task_id"} {
if _, ok := fields[key]; ok {
return EventEnvelope{}, &UnmigratedDialectError{Message: "old-dialect event rejected (legacy source/channel/payload fields present)"}
}
}
decoder := json.NewDecoder(bytes.NewReader(raw))
decoder.DisallowUnknownFields()
var event EventEnvelope
if err := decoder.Decode(&event); err != nil {
return EventEnvelope{}, &UnmigratedDialectError{Message: "not a canonical event envelope: " + err.Error()}
}
if event.SchemaVersion < 1 {
return EventEnvelope{}, &UnmigratedDialectError{Message: "old-dialect event rejected (schema_version < 1)"}
}
if event.EventID == "" {
return EventEnvelope{}, &UnmigratedDialectError{Message: "canonical event requires a stable event_id"}
}
return event, nil
}
func readEventRows(path string, fromOffset int) ([]EventOffset, int, error) {
file, err := os.Open(path)
if os.IsNotExist(err) {
return []EventOffset{}, 0, nil
}
if err != nil {
return nil, 0, err
}
defer file.Close()
rows := []EventOffset{}
scanner := bufio.NewScanner(file)
scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024)
offset := 0
for scanner.Scan() {
offset++
line := bytes.TrimSpace(scanner.Bytes())
if len(line) == 0 || offset < fromOffset {
continue
}
event, err := decodeEvent(append([]byte(nil), line...), offset)
if err != nil {
return nil, 0, err
}
rows = append(rows, EventOffset{Offset: offset, Event: event})
}
if err := scanner.Err(); err != nil {
return nil, 0, err
}
return rows, offset, nil
}
func (s *EventStore) Append(ctx context.Context, event EventEnvelope) (int, error) {
s.mu.Lock()
defer s.mu.Unlock()
if prior, ok := s.known[event.EventID]; ok {
if !reflect.DeepEqual(prior.event, event) {
return 0, fmt.Errorf("event id %q reused with different content", event.EventID)
}
return prior.offset, nil
}
normalized, err := s.decisions.Validate(ctx, "event", event)
if err != nil {
return 0, err
}
if err := json.Unmarshal(normalized, &event); err != nil {
return 0, &OperatorError{Code: "PROTOCOL_ERROR", Op: "append", Message: "normalized event has an invalid shape", Cause: err}
}
s.buffer = append(s.buffer, event)
offset := s.offset + len(s.buffer)
s.known[event.EventID] = knownEvent{offset: offset, event: event}
if len(s.buffer) >= s.batchSize {
if err := s.flushLocked(); err != nil {
return 0, err
}
}
return offset, nil
}
func (s *EventStore) Flush() error {
s.mu.Lock()
defer s.mu.Unlock()
return s.flushLocked()
}
func (s *EventStore) flushLocked() error {
if len(s.buffer) == 0 {
return nil
}
file, err := os.OpenFile(s.path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
if err != nil {
return err
}
for _, event := range s.buffer {
raw, err := json.Marshal(event)
if err != nil {
_ = file.Close()
return err
}
if _, err := file.Write(append(raw, '\n')); err != nil {
_ = file.Close()
return err
}
}
if err := file.Sync(); err != nil {
_ = file.Close()
return err
}
if err := file.Close(); err != nil {
return err
}
s.offset += len(s.buffer)
s.buffer = nil
return nil
}
func (s *EventStore) LastOffset() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.offset + len(s.buffer)
}
func (s *EventStore) Read(fromOffset int) ([]EventOffset, error) {
s.mu.Lock()
defer s.mu.Unlock()
if fromOffset < 1 {
fromOffset = 1
}
rows, _, err := readEventRows(s.path, fromOffset)
return rows, err
}
func (s *EventStore) Replay() ([]EventEnvelope, error) {
rows, err := s.Read(1)
if err != nil {
return nil, err
}
events := make([]EventEnvelope, 0, len(rows))
for _, row := range rows {
events = append(events, row.Event)
}
return events, nil
}
@@ -0,0 +1,144 @@
package runecore
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
)
func testEvent(id, kind string) EventEnvelope {
return EventEnvelope{
SchemaVersion: SnapshotSchemaVersion,
EventID: id,
EventType: kind,
PartitionKey: "task:task_1",
ActorID: "task_state_repository",
PayloadHash: "sha256:abc",
Timestamp: "now",
}
}
func TestEventStoreOffsetsReplayAndDuplicateIdentity(t *testing.T) {
decisions := NewNativeOperatorDecisions(operatorLifecycleTestBinary(t))
path := filepath.Join(t.TempDir(), "events.jsonl")
store, err := OpenEventStore(path, 8, decisions)
if err != nil {
t.Fatal(err)
}
first := testEvent("evt_1", "task_state.created")
second := testEvent("evt_2", "conversation.message")
if offset, err := store.Append(context.Background(), first); err != nil || offset != 1 {
t.Fatalf("first offset=%d err=%v", offset, err)
}
if offset, err := store.Append(context.Background(), second); err != nil || offset != 2 {
t.Fatalf("second offset=%d err=%v", offset, err)
}
if offset, err := store.Append(context.Background(), first); err != nil || offset != 1 {
t.Fatalf("duplicate offset=%d err=%v", offset, err)
}
if err := store.Flush(); err != nil {
t.Fatal(err)
}
if got := store.LastOffset(); got != 2 {
t.Fatalf("last offset=%d", got)
}
reopened, err := OpenEventStore(path, 8, decisions)
if err != nil {
t.Fatal(err)
}
rows, err := reopened.Read(2)
if err != nil || len(rows) != 1 || rows[0].Offset != 2 || rows[0].Event.EventType != second.EventType {
t.Fatalf("read from offset rows=%+v err=%v", rows, err)
}
replay, err := reopened.Replay()
if err != nil || len(replay) != 2 || replay[0].EventID != first.EventID || replay[1].EventID != second.EventID {
t.Fatalf("replay=%+v err=%v", replay, err)
}
}
func TestEventStoreReopenCountsBlankPhysicalLines(t *testing.T) {
decisions := NewNativeOperatorDecisions(operatorLifecycleTestBinary(t))
path := filepath.Join(t.TempDir(), "events.jsonl")
store, err := OpenEventStore(path, 8, decisions)
if err != nil {
t.Fatal(err)
}
if _, err := store.Append(context.Background(), testEvent("evt_1", "a")); err != nil {
t.Fatal(err)
}
if err := store.Flush(); err != nil {
t.Fatal(err)
}
file, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644)
if err != nil {
t.Fatal(err)
}
if _, err := file.Write([]byte("\n")); err != nil {
t.Fatal(err)
}
if err := file.Close(); err != nil {
t.Fatal(err)
}
reopened, err := OpenEventStore(path, 8, decisions)
if err != nil {
t.Fatal(err)
}
offset, err := reopened.Append(context.Background(), testEvent("evt_2", "b"))
if err != nil || offset != 3 {
t.Fatalf("append after blank line offset=%d want=3 err=%v", offset, err)
}
}
func TestEventStoreBatchFlushAndCorruptionAreObservable(t *testing.T) {
decisions := NewNativeOperatorDecisions(operatorLifecycleTestBinary(t))
path := filepath.Join(t.TempDir(), "events.jsonl")
store, err := OpenEventStore(path, 2, decisions)
if err != nil {
t.Fatal(err)
}
if _, err := store.Append(context.Background(), testEvent("evt_1", "a")); err != nil {
t.Fatal(err)
}
if _, err := store.Append(context.Background(), testEvent("evt_2", "b")); err != nil {
t.Fatal(err)
}
if err := store.Flush(); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, append(mustReadFile(t, path), []byte("{not json\n")...), 0o644); err != nil {
t.Fatal(err)
}
_, err = store.Read(1)
var corrupt *CorruptEventStoreError
if !errors.As(err, &corrupt) {
t.Fatalf("corrupt read error = %v", err)
}
}
func TestEventStoreRejectsLegacyBodyWithoutWriting(t *testing.T) {
decisions := NewNativeOperatorDecisions(operatorLifecycleTestBinary(t))
path := filepath.Join(t.TempDir(), "events.jsonl")
legacy := []byte("{\"event_id\":\"evt_old\",\"event_type\":\"old\",\"partition_key\":\"p\",\"payload\":{\"x\":true}}\n")
if err := os.WriteFile(path, legacy, 0o644); err != nil {
t.Fatal(err)
}
_, err := OpenEventStore(path, 8, decisions)
var dialect *UnmigratedDialectError
if !errors.As(err, &dialect) {
t.Fatalf("legacy open error = %v", err)
}
if got := string(mustReadFile(t, path)); got != string(legacy) {
t.Fatalf("legacy input changed to %q", got)
}
}
func mustReadFile(t *testing.T, path string) []byte {
t.Helper()
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
return raw
}
@@ -0,0 +1,161 @@
package runecore
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
)
type OperatorError struct {
Code string
Op string
Message string
Cause error
}
func (e *OperatorError) Error() string {
if e.Op == "" {
return e.Code + ": " + e.Message
}
return e.Code + ": " + e.Op + ": " + e.Message
}
func (e *OperatorError) Unwrap() error { return e.Cause }
type OperatorDecisions interface {
Intake(context.Context, IntakeDecisionRequest) (IntakeDecision, error)
Transition(context.Context, TaskStateRecord, string, string, string) (TransitionDecision, error)
Validate(context.Context, string, any) (json.RawMessage, error)
PayloadHash(context.Context, string, any) (string, error)
}
type NativeOperatorDecisions struct {
Binary string
}
func NewNativeOperatorDecisions(binary string) *NativeOperatorDecisions {
if binary == "" {
binary = os.Getenv("RUNE_OPERATOR_LIFECYCLE_BIN")
}
if binary == "" {
_, file, _, ok := runtime.Caller(0)
if ok {
dir := filepath.Dir(file)
for range 5 {
dir = filepath.Dir(dir)
}
binary = filepath.Join(dir, "src", "rust", "operator_lifecycle", "target", "release", "operator_lifecycle")
}
}
return &NativeOperatorDecisions{Binary: binary}
}
type operatorDecisionEnvelope struct {
OK bool `json:"ok"`
Value json.RawMessage `json:"value"`
Trace map[string]any `json:"trace"`
Error *struct {
Code string `json:"code"`
Message string `json:"message"`
} `json:"error"`
}
func (n *NativeOperatorDecisions) eval(ctx context.Context, operation string, argument any) (*operatorDecisionEnvelope, error) {
if n.Binary == "" {
return nil, &OperatorError{Code: "NATIVE_UNAVAILABLE", Op: operation, Message: "operator-lifecycle native binary not found"}
}
info, err := os.Stat(n.Binary)
if err != nil || info.IsDir() {
return nil, &OperatorError{Code: "NATIVE_UNAVAILABLE", Op: operation, Message: "operator-lifecycle native binary not found: " + n.Binary, Cause: err}
}
payload, err := json.Marshal(map[string]any{"op": operation, "arg": argument})
if err != nil {
return nil, &OperatorError{Code: "PROTOCOL_ERROR", Op: operation, Message: "native request could not be encoded", Cause: err}
}
cmd := exec.CommandContext(ctx, n.Binary, "eval")
cmd.Stdin = bytes.NewReader(append(payload, '\n'))
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
runErr := cmd.Run()
var envelope operatorDecisionEnvelope
if err := json.Unmarshal(stdout.Bytes(), &envelope); err != nil {
code := "PROTOCOL_ERROR"
message := "operator-lifecycle returned invalid NDJSON"
if runErr != nil && stdout.Len() == 0 {
code = "NATIVE_UNAVAILABLE"
message = fmt.Sprintf("operator-lifecycle call failed: %v (%s)", runErr, strings.TrimSpace(stderr.String()))
}
return nil, &OperatorError{Code: code, Op: operation, Message: message, Cause: err}
}
if !envelope.OK {
code, message := "OPERATOR_DECISION_ERROR", "operator-lifecycle decision failed"
if envelope.Error != nil {
code, message = envelope.Error.Code, envelope.Error.Message
}
return nil, &OperatorError{Code: code, Op: operation, Message: message, Cause: runErr}
}
if runErr != nil {
return nil, &OperatorError{Code: "PROTOCOL_ERROR", Op: operation, Message: fmt.Sprintf("operator-lifecycle returned success with exit failure: %v", runErr), Cause: runErr}
}
if len(envelope.Value) == 0 || string(envelope.Value) == "null" {
return nil, &OperatorError{Code: "PROTOCOL_ERROR", Op: operation, Message: "operator-lifecycle omitted its value"}
}
if name, ok := envelope.Trace["operator"].(string); !ok || name != "rust-operator-lifecycle" {
return nil, &OperatorError{Code: "PROTOCOL_ERROR", Op: operation, Message: "operator-lifecycle omitted its native trace"}
}
return &envelope, nil
}
func (n *NativeOperatorDecisions) Intake(ctx context.Context, request IntakeDecisionRequest) (IntakeDecision, error) {
envelope, err := n.eval(ctx, "intake", request)
if err != nil {
return IntakeDecision{}, err
}
var decision IntakeDecision
if err := json.Unmarshal(envelope.Value, &decision); err != nil {
return IntakeDecision{}, &OperatorError{Code: "PROTOCOL_ERROR", Op: "intake", Message: "native intake has an invalid shape", Cause: err}
}
return decision, nil
}
func (n *NativeOperatorDecisions) Transition(ctx context.Context, record TaskStateRecord, status, outcomeRef, now string) (TransitionDecision, error) {
envelope, err := n.eval(ctx, "transition", map[string]any{
"record": record, "new_status": status, "outcome_ref": outcomeRef, "now": now,
})
if err != nil {
return TransitionDecision{}, err
}
var decision TransitionDecision
if err := json.Unmarshal(envelope.Value, &decision); err != nil {
return TransitionDecision{}, &OperatorError{Code: "PROTOCOL_ERROR", Op: "transition", Message: "native transition has an invalid shape", Cause: err}
}
return decision, nil
}
func (n *NativeOperatorDecisions) Validate(ctx context.Context, kind string, record any) (json.RawMessage, error) {
envelope, err := n.eval(ctx, "validate_record", map[string]any{"kind": kind, "record": record})
if err != nil {
return nil, err
}
return envelope.Value, nil
}
func (n *NativeOperatorDecisions) PayloadHash(ctx context.Context, kind string, record any) (string, error) {
envelope, err := n.eval(ctx, "payload_hash", map[string]any{"kind": kind, "record": record})
if err != nil {
return "", err
}
var value struct {
Hash string `json:"payload_hash"`
}
if err := json.Unmarshal(envelope.Value, &value); err != nil || !strings.HasPrefix(value.Hash, "sha256:") {
return "", &OperatorError{Code: "PROTOCOL_ERROR", Op: "payload_hash", Message: "native record hash has an invalid shape", Cause: err}
}
return value.Hash, nil
}
@@ -0,0 +1,77 @@
package runecore
import (
"context"
"errors"
"os"
"os/exec"
"path/filepath"
"sync"
"testing"
)
func buildOperatorLifecycle(binary string) error {
crate := filepath.Clean(filepath.Join(filepath.Dir(binary), "..", ".."))
cmd := exec.Command("cargo", "build", "--release", "--manifest-path", filepath.Join(crate, "Cargo.toml"))
if output, err := cmd.CombinedOutput(); err != nil {
return &OperatorError{Code: "NATIVE_UNAVAILABLE", Op: "build", Message: string(output), Cause: err}
}
return nil
}
var operatorLifecycleBuild sync.Once
var operatorLifecycleBuildErr error
func operatorLifecycleTestBinary(t *testing.T) string {
t.Helper()
operatorLifecycleBuild.Do(func() {
binary := NewNativeOperatorDecisions("").Binary
if _, err := os.Stat(binary); err == nil {
return
}
operatorLifecycleBuildErr = buildOperatorLifecycle(binary)
})
if operatorLifecycleBuildErr != nil {
t.Fatal(operatorLifecycleBuildErr)
}
return NewNativeOperatorDecisions("").Binary
}
func TestNativeOperatorDecisionsCrossRealRustProcess(t *testing.T) {
decisions := NewNativeOperatorDecisions(operatorLifecycleTestBinary(t))
model := "model-a"
intake, err := decisions.Intake(context.Background(), IntakeDecisionRequest{
Description: "fix the parser", ChannelID: "task", Model: &model, TaskID: "task_1", Now: "now",
})
if err != nil || intake.Action != "created" || intake.Family != "code" || intake.Record.TaskID != "task_1" {
t.Fatalf("intake=%+v err=%v", intake, err)
}
intake.Record.Status = "research_required"
transition, err := decisions.Transition(context.Background(), intake.Record, "running", "", "later")
if err != nil || !transition.Open || transition.Record.Status != "running" {
t.Fatalf("transition=%+v err=%v", transition, err)
}
hash, err := decisions.PayloadHash(context.Background(), "task", transition.Record)
if err != nil || len(hash) != len("sha256:")+64 {
t.Fatalf("hash=%q err=%v", hash, err)
}
}
func TestNativeOperatorDecisionsFailuresAreTyped(t *testing.T) {
missing := NewNativeOperatorDecisions(filepath.Join(t.TempDir(), "missing"))
_, err := missing.Intake(context.Background(), IntakeDecisionRequest{})
var operatorErr *OperatorError
if !errors.As(err, &operatorErr) || operatorErr.Code != "NATIVE_UNAVAILABLE" {
t.Fatalf("missing native error = %v", err)
}
binary := filepath.Join(t.TempDir(), "operator-lifecycle")
if err := os.WriteFile(binary, []byte("#!/bin/sh\ncat >/dev/null\nprintf 'not-json\\n'\n"), 0o755); err != nil {
t.Fatal(err)
}
malformed := NewNativeOperatorDecisions(binary)
_, err = malformed.Intake(context.Background(), IntakeDecisionRequest{})
if !errors.As(err, &operatorErr) || operatorErr.Code != "PROTOCOL_ERROR" {
t.Fatalf("malformed native error = %v", err)
}
}
@@ -0,0 +1,71 @@
package runecore
// TaskStateRecord is the canonical task identity shared with the Python wire shape.
type TaskStateRecord struct {
SchemaVersion int `json:"schema_version"`
TaskID string `json:"task_id"`
Title string `json:"title"`
Status string `json:"status"`
ChannelID string `json:"channel_id"`
ParentTaskID *string `json:"parent_task_id"`
AssignedModel *string `json:"assigned_model"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
OutcomeRef string `json:"outcome_ref"`
RouteRef string `json:"route_ref"`
}
// EventEnvelope is the body-free durable event marker.
type EventEnvelope struct {
SchemaVersion int `json:"schema_version"`
EventType string `json:"event_type"`
PartitionKey string `json:"partition_key"`
EventID string `json:"event_id"`
ParentID *string `json:"parent_id"`
ActorID string `json:"actor_id"`
DeviceID string `json:"device_id"`
Authorization string `json:"authorization"`
PayloadHash string `json:"payload_hash"`
Timestamp string `json:"timestamp"`
}
type IntakeDecisionRequest struct {
Description string `json:"description"`
ChannelID string `json:"channel_id"`
Model *string `json:"model"`
Active *TaskStateRecord `json:"active"`
Latest *TaskStateRecord `json:"latest"`
TaskID string `json:"task_id"`
Now string `json:"now"`
}
type IntakeDecision struct {
Action string `json:"action"`
Family string `json:"family"`
Record TaskStateRecord `json:"record"`
}
type TransitionDecision struct {
Open bool `json:"open"`
Record TaskStateRecord `json:"record"`
}
// TurnReceipt is persisted with the final transition to make a repeated
// caller-supplied signal a read-only replay rather than another model turn.
type TurnReceipt struct {
SignalID string `json:"signal_id"`
TaskID string `json:"task_id"`
UserMsgID string `json:"user_msg_id"`
AssistantMsgID string `json:"assistant_msg_id"`
AssistantText string `json:"assistant_text"`
TaskResult string `json:"task_result"`
Model string `json:"model"`
Route string `json:"route"`
Error string `json:"error"`
CompletedAt string `json:"completed_at"`
Thinking []string `json:"thinking,omitempty"`
ContextUsed int `json:"context_used"`
ContextLimit int `json:"context_limit"`
ContextTrace map[string]any `json:"context_trace,omitempty"`
OperatorTrace map[string]any `json:"operator_trace,omitempty"`
}
@@ -0,0 +1,126 @@
package runecore
import (
"bufio"
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
func TestTaskStateKillAndResume(t *testing.T) {
if os.Getenv("GO_OPERATOR_RESUME_HELPER") == "1" {
runOperatorResumeHelper()
return
}
for _, status := range []string{"research_required", "awaiting_approval"} {
t.Run(status, func(t *testing.T) {
root := t.TempDir()
taskID := runOperatorHelperUntilAck(t, root, "hold", status, "", "task_1")
repo, events, _ := openTaskRepositoryForTest(t, root)
got, found := repo.Get(taskID)
if !found || got.Status != status {
t.Fatalf("after kill task=%+v found=%v", got, found)
}
active, found := repo.Active("task")
if !found || active.TaskID != taskID {
t.Fatalf("after kill active=%+v found=%v", active, found)
}
runOperatorHelperUntilAck(t, root, "resume", status, taskID, "running")
fresh, freshEvents, _ := openTaskRepositoryForTest(t, root)
got, found = fresh.Get(taskID)
if !found || got.Status != "running" {
t.Fatalf("after resume kill task=%+v found=%v", got, found)
}
before := freshEvents.LastOffset()
if _, err := fresh.Resume(context.Background(), taskID, "resume-1"); err != nil {
t.Fatal(err)
}
if after := freshEvents.LastOffset(); after != before {
t.Fatalf("duplicate resume after fresh process appended: before=%d after=%d", before, after)
}
if events.LastOffset() != 2 {
t.Fatalf("initial durable events=%d, want create + %s", events.LastOffset(), status)
}
})
}
}
func runOperatorHelperUntilAck(t *testing.T, root, mode, status, taskID, wantAck string) string {
t.Helper()
cmd := exec.Command(os.Args[0], "-test.run=TestTaskStateKillAndResume", "--")
cmd.Env = append(os.Environ(),
"GO_OPERATOR_RESUME_HELPER=1",
"GO_OPERATOR_RESUME_ROOT="+root,
"GO_OPERATOR_RESUME_MODE="+mode,
"GO_OPERATOR_RESUME_STATUS="+status,
"GO_OPERATOR_RESUME_TASK="+taskID,
"RUNE_OPERATOR_LIFECYCLE_BIN="+operatorLifecycleTestBinary(t),
)
stdout, err := cmd.StdoutPipe()
if err != nil {
t.Fatal(err)
}
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
t.Fatal(err)
}
line, err := bufio.NewReader(stdout).ReadString('\n')
if err != nil {
t.Fatal(err)
}
parts := strings.Fields(strings.TrimSpace(line))
if len(parts) != 2 || parts[0] != "ack" || (wantAck != "" && parts[1] != wantAck && mode == "resume") {
t.Fatalf("helper ack=%q", line)
}
if mode == "hold" {
taskID = parts[1]
}
if err := cmd.Process.Kill(); err != nil {
t.Fatal(err)
}
if err := cmd.Wait(); err == nil {
t.Fatal("helper exited before kill")
}
return taskID
}
func runOperatorResumeHelper() {
root := os.Getenv("GO_OPERATOR_RESUME_ROOT")
mode := os.Getenv("GO_OPERATOR_RESUME_MODE")
status := os.Getenv("GO_OPERATOR_RESUME_STATUS")
taskID := os.Getenv("GO_OPERATOR_RESUME_TASK")
decisions := NewNativeOperatorDecisions("")
events, err := OpenEventStore(filepath.Join(root, "events.jsonl"), 8, decisions)
if err != nil {
panic(err)
}
hot, err := OpenAtomicStore(filepath.Join(root, "hot"))
if err != nil {
panic(err)
}
repo, err := OpenTaskStateRepository(events, hot, decisions)
if err != nil {
panic(err)
}
if mode == "hold" {
intake, err := repo.Intake(context.Background(), "durable task", "task", nil)
if err != nil {
panic(err)
}
taskID = intake.Record.TaskID
if _, err := repo.Transition(context.Background(), taskID, status, "dispatch"); err != nil {
panic(err)
}
fmt.Printf("ack %s\n", taskID)
} else {
if _, err := repo.Resume(context.Background(), taskID, "resume-1"); err != nil {
panic(err)
}
fmt.Printf("ack %s\n", "running")
}
select {}
}
@@ -0,0 +1,177 @@
package runecore
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
)
type SufficiencyDecision struct {
LocalAuthority string `json:"local_authority"`
ExternalDependency string `json:"external_dependency"`
VersionFreshnessRisk string `json:"version_freshness_risk"`
Verifiability string `json:"verifiability"`
ScopeNovelty string `json:"scope_novelty"`
Provenance map[string]string `json:"provenance"`
}
type RouteDecision struct {
Route string `json:"route"`
Reasons []string `json:"reasons"`
RiskFlags []string `json:"risk_flags"`
UncertaintyFlags []string `json:"uncertainty_flags"`
Sufficiency *SufficiencyDecision `json:"sufficiency,omitempty"`
Unknowns []string `json:"unknowns,omitempty"`
Confidence string `json:"confidence"`
DecidedBy string `json:"decided_by"`
DecidedAt string `json:"decided_at"`
TaskDescription string `json:"task_description"`
BaselineSHA string `json:"baseline_sha"`
Floor bool `json:"floor"`
TaskClass string `json:"-"`
}
type RouteClient interface {
Classify(context.Context, string, string) (RouteDecision, error)
Dispatch(context.Context, string, string, RouteDecision) (map[string]any, error)
Record(context.Context, string, RouteDecision, map[string]any) error
}
type ProcessRouteClient struct {
Binary string
}
func NewProcessRouteClient(binary string) *ProcessRouteClient {
if binary == "" {
binary = os.Getenv("RUNE_ADAPTIVE_ROUTER_BIN")
}
if binary == "" {
_, file, _, ok := runtime.Caller(0)
if ok {
dir := filepath.Dir(file)
for range 5 {
dir = filepath.Dir(dir)
}
binary = filepath.Join(dir, "src", "go", "adaptive-router", "adaptive-router")
}
}
return &ProcessRouteClient{Binary: binary}
}
func (c *ProcessRouteClient) call(ctx context.Context, request map[string]any) (map[string]any, error) {
op, _ := request["op"].(string)
info, err := os.Stat(c.Binary)
if c.Binary == "" || err != nil || info.IsDir() {
return nil, &OperatorError{Code: "NATIVE_UNAVAILABLE", Op: op, Message: "adaptive-router native binary not found: " + c.Binary, Cause: err}
}
payload, err := json.Marshal(request)
if err != nil {
return nil, &OperatorError{Code: "PROTOCOL_ERROR", Op: op, Message: "route request could not be encoded", Cause: err}
}
cmd := exec.CommandContext(ctx, c.Binary)
cmd.Stdin = bytes.NewReader(payload)
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
runErr := cmd.Run()
var response map[string]any
if err := json.Unmarshal(stdout.Bytes(), &response); err != nil {
code := "PROTOCOL_ERROR"
if runErr != nil && stdout.Len() == 0 {
code = "NATIVE_UNAVAILABLE"
}
return nil, &OperatorError{Code: code, Op: op, Message: fmt.Sprintf("adaptive-router returned invalid JSON: %v (%s)", err, strings.TrimSpace(stderr.String())), Cause: runErr}
}
if message, ok := response["error"]; ok {
return nil, &OperatorError{Code: "ROUTE_ERROR", Op: op, Message: fmt.Sprint(message), Cause: runErr}
}
if runErr != nil {
return nil, &OperatorError{Code: "ROUTE_ERROR", Op: op, Message: runErr.Error(), Cause: runErr}
}
return response, nil
}
func (c *ProcessRouteClient) Classify(ctx context.Context, description, _ string) (RouteDecision, error) {
derive, err := c.call(ctx, map[string]any{"op": "derive", "title": truncateRunes(description, 200), "body": description, "ambiguous": false})
if err != nil {
return RouteDecision{}, err
}
sufficiency := &SufficiencyDecision{
LocalAuthority: stringValue(derive["local_authority"]),
ExternalDependency: stringValue(derive["external_dependency"]),
VersionFreshnessRisk: stringValue(derive["version_freshness_risk"]),
Verifiability: stringValue(derive["verifiability"]),
ScopeNovelty: stringValue(derive["scope_novelty"]),
Provenance: map[string]string{"task_class_language": stringValue(derive["task_class"])},
}
classify, err := c.call(ctx, map[string]any{
"op": "classify", "body": description,
"has_local_repro": sufficiency.LocalAuthority == "GROUNDED",
"owning_authority_known": true,
"testable_locally": sufficiency.Verifiability == "TESTABLE",
"narrow_and_reversible": sufficiency.ScopeNovelty == "KNOWN_PATTERN",
"sufficiency": sufficiency,
"pipeline_sha": "operator",
})
if err != nil {
return RouteDecision{}, err
}
decision := RouteDecision{
Route: stringValue(classify["route"]),
Reasons: stringSlice(classify["reasons"]),
RiskFlags: stringSlice(classify["risk"]),
UncertaintyFlags: stringSlice(classify["uncertainty"]),
Sufficiency: sufficiency,
Unknowns: stringSlice(derive["unknowns"]),
Confidence: stringValue(classify["confidence"]),
DecidedBy: "deterministic",
DecidedAt: utcNow(),
TaskDescription: stringValue(classify["task_description"]),
BaselineSHA: "operator",
Floor: classify["floor"] == true,
TaskClass: stringValue(derive["task_class"]),
}
if decision.Route == "" {
return RouteDecision{}, &OperatorError{Code: "PROTOCOL_ERROR", Op: "classify", Message: "adaptive-router omitted route"}
}
return decision, nil
}
func (c *ProcessRouteClient) Dispatch(ctx context.Context, taskID, description string, decision RouteDecision) (map[string]any, error) {
return c.call(ctx, map[string]any{"op": "dispatch", "task_id": taskID, "description": description, "decision": decision})
}
func (c *ProcessRouteClient) Record(ctx context.Context, path string, decision RouteDecision, dispatch map[string]any) error {
_, err := c.call(ctx, map[string]any{"op": "record_route_decision", "path": path, "decision": decision, "dispatch": dispatch})
return err
}
func stringValue(value any) string {
text, _ := value.(string)
return text
}
func stringSlice(value any) []string {
items, _ := value.([]any)
out := make([]string, 0, len(items))
for _, item := range items {
if text, ok := item.(string); ok {
out = append(out, text)
}
}
return out
}
func truncateRunes(text string, limit int) string {
runes := []rune(text)
if len(runes) <= limit {
return text
}
return string(runes[:limit])
}
@@ -0,0 +1,96 @@
package runecore
import (
"context"
"errors"
"os"
"os/exec"
"path/filepath"
"sync"
"testing"
)
var adaptiveRouterBuild sync.Once
var adaptiveRouterBuildErr error
func buildAdaptiveRouter(binary string) error {
crate := filepath.Clean(filepath.Join(filepath.Dir(binary), "..", "..", "rust", "route_risk_flags"))
buildRust := exec.Command("cargo", "build", "--release", "--manifest-path", filepath.Join(crate, "Cargo.toml"))
if output, err := buildRust.CombinedOutput(); err != nil {
return &OperatorError{Code: "NATIVE_UNAVAILABLE", Op: "build-router-kernel", Message: string(output), Cause: err}
}
cmd := exec.Command("go", "build", "-o", binary, ".")
cmd.Dir = filepath.Dir(binary)
if output, err := cmd.CombinedOutput(); err != nil {
return &OperatorError{Code: "NATIVE_UNAVAILABLE", Op: "build-router", Message: string(output), Cause: err}
}
return nil
}
func adaptiveRouterTestBinary(t *testing.T) string {
t.Helper()
adaptiveRouterBuild.Do(func() {
binary := NewProcessRouteClient("").Binary
if _, err := os.Stat(binary); err == nil {
return
}
adaptiveRouterBuildErr = buildAdaptiveRouter(binary)
})
if adaptiveRouterBuildErr != nil {
t.Fatal(adaptiveRouterBuildErr)
}
return NewProcessRouteClient("").Binary
}
func TestProcessRouteClientUsesExistingRouteAndDispatchServices(t *testing.T) {
client := NewProcessRouteClient(adaptiveRouterTestBinary(t))
local, err := client.Classify(context.Background(), "fix the failing test in tests/test_x.py", "task_1")
if err != nil || local.Route != "LOCAL_FIX" || local.TaskClass != "defect" {
t.Fatalf("local route=%+v err=%v", local, err)
}
research, err := client.Classify(context.Background(), "adopt a replacement external service through architecture research", "task_2")
if err != nil || research.Route != "RESEARCHED_CHANGE" {
t.Fatalf("research route=%+v err=%v", research, err)
}
dispatch, err := client.Dispatch(context.Background(), "task_2", "adopt a replacement external service through architecture research", research)
if err != nil || len(dispatch) == 0 {
t.Fatalf("dispatch=%+v err=%v", dispatch, err)
}
}
func TestProcessRouteClientMissingBinaryIsTyped(t *testing.T) {
client := NewProcessRouteClient(filepath.Join(t.TempDir(), "missing"))
_, err := client.Classify(context.Background(), "hello", "task_1")
var operatorErr *OperatorError
if !errors.As(err, &operatorErr) || operatorErr.Code != "NATIVE_UNAVAILABLE" {
t.Fatalf("missing router error = %v", err)
}
}
type staticRouteClient struct {
routes []string
calls int
dispatchCalls int
}
func (c *staticRouteClient) Classify(_ context.Context, _, _ string) (RouteDecision, error) {
route := "STANDARD_CHANGE"
if len(c.routes) > 0 {
index := c.calls
if index >= len(c.routes) {
index = len(c.routes) - 1
}
route = c.routes[index]
}
c.calls++
return RouteDecision{Route: route, Reasons: []string{"static client"}, DecidedAt: "now", TaskClass: "neutral"}, nil
}
func (c *staticRouteClient) Dispatch(_ context.Context, taskID, description string, decision RouteDecision) (map[string]any, error) {
c.dispatchCalls++
return map[string]any{"task_id": taskID, "description": description, "route": decision.Route}, nil
}
func (c *staticRouteClient) Record(context.Context, string, RouteDecision, map[string]any) error {
return nil
}
@@ -0,0 +1,167 @@
package runecore
import (
"context"
"errors"
"net/http/httptest"
"os"
"path/filepath"
"testing"
)
func newNativeServingEngine(t *testing.T, model ModelClient, router RouteClient) (*Engine, *TailStore, *NativeOperator, string) {
t.Helper()
root := t.TempDir()
home := filepath.Join(root, "home")
if err := writeTestProvider(home, "local", "test-model-27b"); err != nil {
t.Fatal(err)
}
store := openTestStore(t, filepath.Join(root, "conversations"))
operator := NewNativeOperatorWithClients(home, store, NewNativeOperatorDecisions(operatorLifecycleTestBinary(t)), router)
engine := NewEngine(store, model, home)
engine.SetOperator(operator)
engine.SetContextAssembler(passthroughContextAssembler{})
return engine, store, operator, home
}
func TestNativeTurnPersistsLifecycleAndReplaysDuplicateSignal(t *testing.T) {
model := &fakeModel{replies: map[string]string{"": "reply"}}
engine, store, operator, home := newNativeServingEngine(t, model, &staticRouteClient{})
result, err := engine.Turn(context.Background(), TurnRequest{Channel: "task", UserText: "hello", SignalID: "sig-1"})
if err != nil {
t.Fatal(err)
}
if result.Task == nil || result.Task.Status != "completed" || result.Route != "STANDARD_CHANGE" {
t.Fatalf("turn result=%+v", result)
}
if result.OperatorTrace["operator"] != "go-operator" || result.OperatorTrace["native"] != "rust-operator-lifecycle" || result.OperatorTrace["events"] != "go-event-store" {
t.Fatalf("operator trace=%v", result.OperatorTrace)
}
window := store.Window("task", 8)
if len(window) != 2 || *window[0].TaskID != result.Task.TaskID || *window[1].TaskID != result.Task.TaskID {
t.Fatalf("task-scoped window=%+v", window)
}
decisions := NewNativeOperatorDecisions(operatorLifecycleTestBinary(t))
events, err := OpenEventStore(filepath.Join(home, "events.jsonl"), 8, decisions)
if err != nil {
t.Fatal(err)
}
before := events.LastOffset()
duplicate, err := engine.Turn(context.Background(), TurnRequest{Channel: "task", UserText: "hello", SignalID: "sig-1"})
if err != nil {
t.Fatal(err)
}
if duplicate.UserMsgID != result.UserMsgID || duplicate.AssistantMsgID != result.AssistantMsgID || duplicate.AssistantText != result.AssistantText {
t.Fatalf("duplicate result=%+v want=%+v", duplicate, result)
}
if len(model.recorded()) != 1 || len(store.Window("task", 8)) != 2 || events.LastOffset() != before {
t.Fatalf("duplicate had side effects calls=%d window=%d events=%d/%d", len(model.recorded()), len(store.Window("task", 8)), events.LastOffset(), before)
}
receipt, found := operator.tasks.TurnReceipt("sig-1")
if !found || receipt.TaskID != result.Task.TaskID {
t.Fatalf("receipt=%+v found=%v", receipt, found)
}
}
func TestResearchDispositionSkipsModelAndResumesSameTask(t *testing.T) {
model := &fakeModel{replies: map[string]string{"": "resumed reply"}}
router := &staticRouteClient{routes: []string{"RESEARCHED_CHANGE", "STANDARD_CHANGE"}}
engine, store, operator, _ := newNativeServingEngine(t, model, router)
research, err := engine.Turn(context.Background(), TurnRequest{Channel: "task", UserText: "research this", SignalID: "research-1"})
if err != nil {
t.Fatal(err)
}
if research.Error != "RESEARCH_REQUIRED" || research.Task == nil || research.Task.Status != "research_required" || len(model.recorded()) != 0 || router.dispatchCalls != 1 {
t.Fatalf("research result=%+v calls=%d dispatch=%d", research, len(model.recorded()), router.dispatchCalls)
}
if len(store.Window("task", 8)) != 1 {
t.Fatal("research disposition invented an assistant model answer")
}
if _, err := operator.tasks.Resume(context.Background(), research.Task.TaskID, "resume-1"); err != nil {
t.Fatal(err)
}
resumed, err := engine.Turn(context.Background(), TurnRequest{Channel: "task", UserText: "yes continue", SignalID: "resume-turn-1"})
if err != nil {
t.Fatal(err)
}
if resumed.Task == nil || resumed.Task.TaskID != research.Task.TaskID || resumed.Task.Status != "completed" || len(model.recorded()) != 1 {
t.Fatalf("resumed result=%+v calls=%d", resumed, len(model.recorded()))
}
}
func TestServerServesCanonicalStatusTasksAndOperatorTrace(t *testing.T) {
model := &fakeModel{replies: map[string]string{"": "reply"}}
engine, _, _, home := newNativeServingEngine(t, model, &staticRouteClient{})
server := httptest.NewServer(NewServer(engine, model, home).Handler())
t.Cleanup(server.Close)
response, body := postJSON(t, server.URL+"/v1/chat/completions", map[string]any{
"channel": "task", "signal_id": "sig-http-1", "messages": []map[string]string{{"role": "user", "content": "hello"}},
})
if response.StatusCode != 200 {
t.Fatalf("chat status=%d body=%v", response.StatusCode, body)
}
runeBlock := body["rune"].(map[string]any)
if runeBlock["success"] != true || runeBlock["task"].(map[string]any)["status"] != "completed" {
t.Fatalf("rune block=%v", runeBlock)
}
trace := runeBlock["operator_trace"].(map[string]any)
if trace["operator"] != "go-operator" || trace["native"] != "rust-operator-lifecycle" {
t.Fatalf("serving trace=%v", trace)
}
_, status := getJSON(t, server.URL+"/v1/status")
if status["tasks_completed"] != float64(1) || status["tasks_running"] != float64(0) || status["current_task"] != nil {
t.Fatalf("status=%v", status)
}
_, tasks := getJSON(t, server.URL+"/v1/tasks")
rows := tasks["tasks"].([]any)
if len(rows) != 1 || rows[0].(map[string]any)["result"] != "reply" || rows[0].(map[string]any)["description"] != "hello" {
t.Fatalf("tasks=%v", tasks)
}
}
func TestProviderFailureTerminalizesCanonicalTask(t *testing.T) {
model := &fakeModel{err: errors.New("provider down")}
engine, _, _, _ := newNativeServingEngine(t, model, &staticRouteClient{})
result, err := engine.Turn(context.Background(), TurnRequest{Channel: "task", UserText: "hello", SignalID: "failure-1"})
if err == nil || result.Task == nil || result.Task.Status != "failed" {
t.Fatalf("failure result=%+v err=%v", result, err)
}
}
func TestRealNativeServingTurnDoesNotInvokePythonOperator(t *testing.T) {
model := &fakeModel{replies: map[string]string{"": "reply"}}
router := NewProcessRouteClient(adaptiveRouterTestBinary(t))
engine, _, _, home := newNativeServingEngine(t, model, router)
server := httptest.NewServer(NewServer(engine, model, home).Handler())
t.Cleanup(server.Close)
dir := t.TempDir()
marker := filepath.Join(dir, "python-used")
for _, name := range []string{"python", "python3"} {
script := "#!/bin/sh\nprintf invoked >\"" + marker + "\"\nexit 99\n"
if err := os.WriteFile(filepath.Join(dir, name), []byte(script), 0o755); err != nil {
t.Fatal(err)
}
}
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
response, body := postJSON(t, server.URL+"/v1/chat/completions", map[string]any{
"channel": "task", "signal_id": "real-native-1", "messages": []map[string]string{{"role": "user", "content": "hello native operator"}},
})
if response.StatusCode != 200 {
t.Fatalf("native turn status=%d body=%v", response.StatusCode, body)
}
if _, err := os.Stat(marker); !os.IsNotExist(err) {
t.Fatal("real Go turn invoked a Python operator oracle")
}
runeBlock := body["rune"].(map[string]any)
trace := runeBlock["operator_trace"].(map[string]any)
if trace["operator"] != "go-operator" || trace["native"] != "rust-operator-lifecycle" || trace["events"] != "go-event-store" {
t.Fatalf("real native trace=%v", trace)
}
t.Logf("native serving proof: task=%v operator_trace=%v python_sentinel=absent", runeBlock["task"], trace)
if _, err := os.Stat(filepath.Join(home, "events.jsonl")); err != nil {
t.Fatalf("native event log missing: %v", err)
}
if _, err := os.Stat(filepath.Join(home, "routes", "route.jsonl")); err != nil {
t.Fatalf("route authority record missing: %v", err)
}
}
@@ -0,0 +1,352 @@
package runecore
import (
"context"
"encoding/json"
"fmt"
"io"
"sort"
"strings"
"sync"
)
const taskCheckpointName = "task_state.json"
type CorruptStateCheckpointError struct {
Message string
Cause error
}
func (e *CorruptStateCheckpointError) Error() string { return e.Message }
func (e *CorruptStateCheckpointError) Unwrap() error { return e.Cause }
type taskCheckpoint struct {
SchemaVersion int `json:"schema_version"`
Tasks []TaskStateRecord `json:"tasks"`
Active map[string]string `json:"active"`
TurnReceipts map[string]TurnReceipt `json:"turn_receipts"`
}
type TaskStateRepository struct {
events *EventStore
hot *AtomicStore
decisions OperatorDecisions
mu sync.Mutex
tasks map[string]TaskStateRecord
active map[string]string
order []string
receipts map[string]TurnReceipt
}
func OpenTaskStateRepository(events *EventStore, hot *AtomicStore, decisions OperatorDecisions) (*TaskStateRepository, error) {
repo := &TaskStateRepository{
events: events, hot: hot, decisions: decisions,
tasks: map[string]TaskStateRecord{}, active: map[string]string{}, receipts: map[string]TurnReceipt{},
}
blob, found, err := hot.Read(taskCheckpointName)
if err != nil {
return nil, err
}
if !found || strings.TrimSpace(blob) == "" {
return repo, nil
}
var checkpoint taskCheckpoint
decoder := json.NewDecoder(strings.NewReader(blob))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&checkpoint); err != nil {
return nil, &CorruptStateCheckpointError{Message: "task-state checkpoint is corrupt: " + err.Error(), Cause: err}
}
if err := decoder.Decode(&struct{}{}); err != io.EOF {
return nil, &CorruptStateCheckpointError{Message: "task-state checkpoint has trailing data"}
}
for _, record := range checkpoint.Tasks {
repo.tasks[record.TaskID] = record
repo.order = append(repo.order, record.TaskID)
}
for channel, taskID := range checkpoint.Active {
record, ok := repo.tasks[taskID]
if ok && record.ChannelID == channel && openTaskStatus(record.Status) {
repo.active[channel] = taskID
}
}
for signal, receipt := range checkpoint.TurnReceipts {
repo.receipts[signal] = receipt
}
return repo, nil
}
func openTaskStatus(status string) bool {
switch status {
case "created", "running", "awaiting_approval", "research_required":
return true
default:
return false
}
}
func newTaskID() string { return "task_" + hex12() }
func newOperatorEventID() string { return "evt_" + hex12() }
func (r *TaskStateRepository) checkpointLocked() error {
checkpoint := taskCheckpoint{
SchemaVersion: SnapshotSchemaVersion,
Tasks: make([]TaskStateRecord, 0, len(r.tasks)),
Active: r.active,
TurnReceipts: r.receipts,
}
for _, taskID := range r.order {
if record, ok := r.tasks[taskID]; ok {
checkpoint.Tasks = append(checkpoint.Tasks, record)
}
}
raw, err := json.Marshal(checkpoint)
if err != nil {
return err
}
return r.hot.Write(taskCheckpointName, string(raw))
}
func (r *TaskStateRepository) logLocked(ctx context.Context, eventType string, record TaskStateRecord) error {
hash, err := r.decisions.PayloadHash(ctx, "task", record)
if err != nil {
return err
}
_, err = r.events.Append(ctx, EventEnvelope{
SchemaVersion: SnapshotSchemaVersion,
EventType: eventType,
PartitionKey: "task:" + record.TaskID,
EventID: newOperatorEventID(),
ActorID: "task_state_repository",
PayloadHash: hash,
Timestamp: utcNow(),
})
if err != nil {
return err
}
if err := r.events.Flush(); err != nil {
return err
}
return r.checkpointLocked()
}
func (r *TaskStateRepository) activeLocked(channel string) (TaskStateRecord, bool) {
if taskID := r.active[channel]; taskID != "" {
record, ok := r.tasks[taskID]
if ok && openTaskStatus(record.Status) {
return record, true
}
}
var candidates []TaskStateRecord
for _, record := range r.tasks {
if record.ChannelID == channel && openTaskStatus(record.Status) {
candidates = append(candidates, record)
}
}
if len(candidates) == 0 {
delete(r.active, channel)
return TaskStateRecord{}, false
}
sort.SliceStable(candidates, func(i, j int) bool {
if candidates[i].UpdatedAt == candidates[j].UpdatedAt {
return candidates[i].TaskID < candidates[j].TaskID
}
return candidates[i].UpdatedAt < candidates[j].UpdatedAt
})
record := candidates[len(candidates)-1]
r.active[channel] = record.TaskID
return record, true
}
func (r *TaskStateRepository) latestLocked(channel string) (TaskStateRecord, bool) {
var candidates []TaskStateRecord
for _, record := range r.tasks {
if record.ChannelID == channel {
candidates = append(candidates, record)
}
}
if len(candidates) == 0 {
return TaskStateRecord{}, false
}
sort.SliceStable(candidates, func(i, j int) bool {
if candidates[i].CreatedAt == candidates[j].CreatedAt {
return candidates[i].TaskID < candidates[j].TaskID
}
return candidates[i].CreatedAt < candidates[j].CreatedAt
})
return candidates[len(candidates)-1], true
}
func (r *TaskStateRepository) Intake(ctx context.Context, description, channel string, model *string) (IntakeDecision, error) {
r.mu.Lock()
defer r.mu.Unlock()
active, hasActive := r.activeLocked(channel)
latest, hasLatest := r.latestLocked(channel)
request := IntakeDecisionRequest{
Description: description, ChannelID: channel, Model: model, TaskID: newTaskID(), Now: utcNow(),
}
if hasActive {
request.Active = &active
}
if hasLatest {
request.Latest = &latest
}
decision, err := r.decisions.Intake(ctx, request)
if err != nil {
return IntakeDecision{}, err
}
record := decision.Record
if _, exists := r.tasks[record.TaskID]; !exists {
r.order = append(r.order, record.TaskID)
}
r.tasks[record.TaskID] = record
r.active[channel] = record.TaskID
eventType := "task_state.created"
if decision.Action == "amended" {
eventType = "task_state.amended"
}
if err := r.logLocked(ctx, eventType, record); err != nil {
return IntakeDecision{}, err
}
return decision, nil
}
func (r *TaskStateRepository) Transition(ctx context.Context, taskID, status, outcomeRef string) (TaskStateRecord, error) {
return r.completeTransition(ctx, taskID, status, outcomeRef, nil)
}
func (r *TaskStateRepository) completeTransition(ctx context.Context, taskID, status, outcomeRef string, receipt *TurnReceipt) (TaskStateRecord, error) {
r.mu.Lock()
defer r.mu.Unlock()
if receipt != nil && receipt.SignalID != "" {
if _, exists := r.receipts[receipt.SignalID]; exists {
record, ok := r.tasks[taskID]
if !ok {
return TaskStateRecord{}, fmt.Errorf("unknown task_id %q", taskID)
}
return record, nil
}
}
record, ok := r.tasks[taskID]
if !ok {
return TaskStateRecord{}, fmt.Errorf("unknown task_id %q", taskID)
}
decision, err := r.decisions.Transition(ctx, record, status, outcomeRef, utcNow())
if err != nil {
return TaskStateRecord{}, err
}
record = decision.Record
r.tasks[taskID] = record
if !decision.Open && r.active[record.ChannelID] == taskID {
delete(r.active, record.ChannelID)
}
if receipt != nil && receipt.SignalID != "" {
receipt.TaskID = taskID
r.receipts[receipt.SignalID] = *receipt
}
if err := r.logLocked(ctx, "task_state.transition", record); err != nil {
return TaskStateRecord{}, err
}
return record, nil
}
func (r *TaskStateRepository) CompleteTurn(ctx context.Context, taskID, status, outcomeRef string, receipt TurnReceipt) (TaskStateRecord, error) {
return r.completeTransition(ctx, taskID, status, outcomeRef, &receipt)
}
func (r *TaskStateRepository) Resume(ctx context.Context, taskID, signalID string) (TaskStateRecord, error) {
return r.CompleteTurn(ctx, taskID, "running", "", TurnReceipt{SignalID: signalID, TaskID: taskID})
}
func (r *TaskStateRepository) SetRouteRef(ctx context.Context, taskID, routeRef string) (TaskStateRecord, error) {
r.mu.Lock()
defer r.mu.Unlock()
record, ok := r.tasks[taskID]
if !ok {
return TaskStateRecord{}, fmt.Errorf("unknown task_id %q", taskID)
}
record.RouteRef = routeRef
record.UpdatedAt = utcNow()
r.tasks[taskID] = record
if err := r.logLocked(ctx, "task_state.route_ref", record); err != nil {
return TaskStateRecord{}, err
}
return record, nil
}
func (r *TaskStateRepository) Get(taskID string) (TaskStateRecord, bool) {
r.mu.Lock()
defer r.mu.Unlock()
record, ok := r.tasks[taskID]
return record, ok
}
func (r *TaskStateRepository) Active(channel string) (TaskStateRecord, bool) {
r.mu.Lock()
defer r.mu.Unlock()
return r.activeLocked(channel)
}
func (r *TaskStateRepository) ListAll() []TaskStateRecord {
r.mu.Lock()
defer r.mu.Unlock()
out := make([]TaskStateRecord, 0, len(r.tasks))
for _, taskID := range r.order {
if record, ok := r.tasks[taskID]; ok {
out = append(out, record)
}
}
return out
}
func (r *TaskStateRepository) ListOpen(channel string) []TaskStateRecord {
r.mu.Lock()
defer r.mu.Unlock()
out := []TaskStateRecord{}
for _, record := range r.tasks {
if record.ChannelID == channel && openTaskStatus(record.Status) {
out = append(out, record)
}
}
sort.SliceStable(out, func(i, j int) bool { return out[i].UpdatedAt < out[j].UpdatedAt })
return out
}
func (r *TaskStateRepository) TurnReceipt(signalID string) (TurnReceipt, bool) {
r.mu.Lock()
defer r.mu.Unlock()
receipt, ok := r.receipts[signalID]
return receipt, ok
}
func (r *TaskStateRepository) ReceiptForTask(taskID string) (TurnReceipt, bool) {
r.mu.Lock()
defer r.mu.Unlock()
var latest TurnReceipt
found := false
for _, receipt := range r.receipts {
if receipt.TaskID == taskID && (!found || receipt.CompletedAt > latest.CompletedAt) {
latest = receipt
found = true
}
}
return latest, found
}
func (r *TaskStateRepository) ImportRecord(ctx context.Context, record TaskStateRecord) (bool, error) {
r.mu.Lock()
defer r.mu.Unlock()
if _, exists := r.tasks[record.TaskID]; exists {
return false, nil
}
r.tasks[record.TaskID] = record
r.order = append(r.order, record.TaskID)
if openTaskStatus(record.Status) {
if _, exists := r.active[record.ChannelID]; !exists {
r.active[record.ChannelID] = record.TaskID
}
}
if err := r.logLocked(ctx, "task_state.imported", record); err != nil {
return false, err
}
return true, nil
}
@@ -0,0 +1,113 @@
package runecore
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
)
func openTaskRepositoryForTest(t *testing.T, root string) (*TaskStateRepository, *EventStore, *TailStore) {
t.Helper()
decisions := NewNativeOperatorDecisions(operatorLifecycleTestBinary(t))
events, err := OpenEventStore(filepath.Join(root, "events.jsonl"), 8, decisions)
if err != nil {
t.Fatal(err)
}
hot, err := OpenAtomicStore(filepath.Join(root, "hot"))
if err != nil {
t.Fatal(err)
}
repo, err := OpenTaskStateRepository(events, hot, decisions)
if err != nil {
t.Fatal(err)
}
store, err := OpenTailStore(filepath.Join(root, "conversations"))
if err != nil {
t.Fatal(err)
}
return repo, events, store
}
func TestTaskRepositoryIntakeTransitionAndFreshOpenParity(t *testing.T) {
root := t.TempDir()
repo, events, _ := openTaskRepositoryForTest(t, root)
model := "model-a"
created, err := repo.Intake(context.Background(), "fix the parser", "task", &model)
if err != nil || created.Action != "created" || created.Family != "code" {
t.Fatalf("intake=%+v err=%v", created, err)
}
taskID := created.Record.TaskID
if _, err := repo.Transition(context.Background(), taskID, "research_required", "research_dispatch"); err != nil {
t.Fatal(err)
}
if _, err := repo.SetRouteRef(context.Background(), taskID, "route:now"); err != nil {
t.Fatal(err)
}
reopened, _, _ := openTaskRepositoryForTest(t, root)
got, found := reopened.Get(taskID)
if !found || got.Status != "research_required" || got.RouteRef != "route:now" || got.OutcomeRef != "research_dispatch" {
t.Fatalf("fresh task=%+v found=%v", got, found)
}
active, found := reopened.Active("task")
if !found || active.TaskID != taskID {
t.Fatalf("active=%+v found=%v", active, found)
}
resumed, err := reopened.Resume(context.Background(), taskID, "resume-1")
if err != nil || resumed.Status != "running" {
t.Fatalf("resumed=%+v err=%v", resumed, err)
}
before := events.LastOffset()
if _, err := reopened.Resume(context.Background(), taskID, "resume-1"); err != nil {
t.Fatal(err)
}
if after := events.LastOffset(); after != before {
t.Fatalf("duplicate resume appended events: before=%d after=%d", before, after)
}
fresh, _, _ := openTaskRepositoryForTest(t, root)
got, _ = fresh.Get(taskID)
if got.Status != "running" || got.UpdatedAt != resumed.UpdatedAt {
t.Fatalf("fresh resume mismatch got=%+v want=%+v", got, resumed)
}
}
func TestTaskRepositoryRejectsTerminalTransition(t *testing.T) {
repo, _, _ := openTaskRepositoryForTest(t, t.TempDir())
created, err := repo.Intake(context.Background(), "hello", "chat", nil)
if err != nil {
t.Fatal(err)
}
if _, err := repo.Transition(context.Background(), created.Record.TaskID, "completed", "response:1"); err != nil {
t.Fatal(err)
}
_, err = repo.Transition(context.Background(), created.Record.TaskID, "running", "")
var operatorErr *OperatorError
if !errors.As(err, &operatorErr) || operatorErr.Code != "ILLEGAL_TRANSITION" {
t.Fatalf("terminal transition error = %v", err)
}
}
func TestTaskCheckpointCorruptionIsLoud(t *testing.T) {
root := t.TempDir()
if err := os.MkdirAll(filepath.Join(root, "hot"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(root, "hot", "task_state.json"), []byte("{not-json"), 0o644); err != nil {
t.Fatal(err)
}
decisions := NewNativeOperatorDecisions(operatorLifecycleTestBinary(t))
events, err := OpenEventStore(filepath.Join(root, "events.jsonl"), 8, decisions)
if err != nil {
t.Fatal(err)
}
hot, err := OpenAtomicStore(filepath.Join(root, "hot"))
if err != nil {
t.Fatal(err)
}
_, err = OpenTaskStateRepository(events, hot, decisions)
var corrupt *CorruptStateCheckpointError
if !errors.As(err, &corrupt) {
t.Fatalf("corrupt checkpoint error = %v", err)
}
}
+200 -31
View File
@@ -13,7 +13,9 @@ import (
"errors"
"io"
"net/http"
"path/filepath"
"strconv"
"strings"
"sync"
"time"
)
@@ -23,6 +25,7 @@ type Server struct {
engine *Engine
client ModelClient // /v1/models data source (Tags)
home string // provider record (active model)
memory MemoryStore
mu sync.Mutex
ctx struct {
used, limit int // D44 context readout — the last turn's accounting
@@ -35,6 +38,8 @@ func NewServer(engine *Engine, client ModelClient, home string) *Server {
return &Server{engine: engine, client: client, home: home}
}
func (s *Server) SetMemoryStore(store MemoryStore) { s.memory = store }
// Handler mounts every route.
func (s *Server) Handler() http.Handler {
mux := http.NewServeMux()
@@ -49,8 +54,10 @@ func (s *Server) Handler() http.Handler {
}
handle("/v1/models", s.handleModels)
handle("/v1/status", s.handleStatus)
handle("/v1/tasks", s.handleTasks)
handle("/v1/conversation", s.handleConversation)
handle("/v1/health", s.handleHealth)
handle("/v1/memory/recall", s.handleMemoryRecall)
handle("/v1/approvals/resolve", s.handleApprovalsResolve)
handle("/v1/chat/completions", s.handleChatCompletions)
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
@@ -108,14 +115,32 @@ func (s *Server) handleStatus(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
ctxUsed, ctxLimit := s.ctx.used, s.ctx.limit
s.mu.Unlock()
var current any
pending, running, completed := 0, 0, 0
if s.engine.operator != nil {
snapshot := s.engine.operator.Snapshot("task")
if snapshot.Active != nil {
current = snapshot.Active.TaskID
}
for _, view := range snapshot.Tasks {
switch view.Record.Status {
case "created", "pending":
pending++
case "running":
running++
case "completed":
completed++
}
}
}
writeJSON(w, http.StatusOK, map[string]any{
"identity": "Rune",
"role": "Operator",
"current_task": nil,
"current_task": current,
"active_model": active,
"tasks_pending": 0,
"tasks_running": 0,
"tasks_completed": 0,
"tasks_pending": pending,
"tasks_running": running,
"tasks_completed": completed,
// D44 context readout: the last turn's real accounting
// (used = tokens sent, limit = the model's budget). Absent
// until the first turn — the bar never guesses.
@@ -139,6 +164,76 @@ func (s *Server) handleStatus(w http.ResponseWriter, r *http.Request) {
// health poll through it).
func (s *Server) Engine() *Engine { return s.engine }
func legacyTaskEnvelope(view OperatorTaskView) map[string]any {
record := view.Record
var result, completedAt, assignedModel any
route := ""
if record.AssignedModel != nil {
assignedModel = *record.AssignedModel
}
if view.Receipt != nil {
result = view.Receipt.TaskResult
if view.Receipt.Model != "" {
assignedModel = view.Receipt.Model
}
route = view.Receipt.Route
}
if !openTaskStatus(record.Status) {
completedAt = record.UpdatedAt
}
return map[string]any{
"task_id": record.TaskID, "description": record.Title, "status": record.Status,
"assigned_model": assignedModel, "result": result, "created_at": record.CreatedAt,
"completed_at": completedAt, "route": route, "route_instruction": "",
}
}
func legacyTurnTask(result TurnResult) any {
if result.Task == nil {
return nil
}
receipt := TurnReceipt{
TaskResult: result.TaskResult, Model: result.Model, Route: result.Route,
}
return legacyTaskEnvelope(OperatorTaskView{Record: *result.Task, Receipt: &receipt})
}
func turnRuneBlock(result TurnResult, turnErr error, channel string) map[string]any {
errorText := result.Error
if turnErr != nil {
errorText = clipStr(turnErr.Error(), 300)
}
return map[string]any{
"success": turnErr == nil && result.Error == "",
"user_text": result.AssistantText,
"response": result.AssistantText,
"model": result.Model,
"error": errorText,
"channel": channel,
"thinking": result.Thinking,
"messages": map[string]string{"user": result.UserMsgID, "assistant": result.AssistantMsgID},
"core": "go",
"task": legacyTurnTask(result),
"route": result.Route,
"context_trace": result.ContextTrace,
"operator_trace": result.OperatorTrace,
}
}
func (s *Server) handleTasks(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
errorEnvelope(w, http.StatusMethodNotAllowed, "method_not_allowed", "task listing requires GET")
return
}
rows := []map[string]any{}
if s.engine.operator != nil {
for _, view := range s.engine.operator.Snapshot("task").Tasks {
rows = append(rows, legacyTaskEnvelope(view))
}
}
writeJSON(w, http.StatusOK, map[string]any{"tasks": rows})
}
// handleHealth: the core's own liveness + identity.
func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
var model string
@@ -178,6 +273,54 @@ func (s *Server) handleConversation(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, map[string]any{"messages": msgs})
}
func (s *Server) handleMemoryRecall(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
errorEnvelope(w, http.StatusMethodNotAllowed, "method_not_allowed", "memory recall requires POST")
return
}
if s.memory == nil {
errorEnvelope(w, http.StatusServiceUnavailable, "native_unavailable", "canonical memory store is not composed")
return
}
var request struct {
DBPath string `json:"db_path"`
Query string `json:"query"`
Budget int `json:"budget"`
MinScore *float64 `json:"min_score"`
MemoryTypes []string `json:"memory_types"`
}
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&request); err != nil {
errorEnvelope(w, http.StatusBadRequest, "bad_request", "invalid JSON body: "+err.Error())
return
}
if request.Budget == 0 {
request.Budget = 8
}
minScore := 0.42
if request.MinScore != nil {
minScore = *request.MinScore
}
if request.DBPath == "" {
request.DBPath = filepath.Join(s.home, "memory", "store", "memory.db")
}
result, err := s.memory.Recall(r.Context(), request.DBPath, request.Query, request.Budget, minScore, request.MemoryTypes)
if err != nil {
code := "store_error"
status := http.StatusInternalServerError
if memoryErr, ok := err.(*MemoryError); ok {
code = memoryErr.Code
if memoryErr.Code == "NATIVE_UNAVAILABLE" {
status = http.StatusServiceUnavailable
}
}
errorEnvelope(w, status, code, err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]any{
"object": "memory.recall", "data": result.Rows, "core": "go", "trace": result.Trace,
})
}
// handleApprovalsResolve: the wave-1 core has no approval surface —
// the honest no-pending answer (the Python core's shape when nothing
// was pending).
@@ -196,6 +339,7 @@ type completionRequest struct {
Channel string `json:"channel"`
Messages []completionMsgIn `json:"messages"`
Stream bool `json:"stream"`
SignalID string `json:"signal_id"`
}
type completionMsgIn struct {
@@ -234,18 +378,21 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) {
}
if req.Stream {
s.streamCompletion(w, r, req.Model, channel, userText)
s.streamCompletion(w, r, req.Model, channel, userText, req.SignalID)
return
}
res, err := s.engine.Turn(r.Context(), TurnRequest{
Channel: channel, UserText: userText, Model: req.Model,
Channel: channel, UserText: userText, Model: req.Model, SignalID: req.SignalID,
})
if err == nil && res.ContextLimit > 0 {
s.setContext(res.ContextUsed, res.ContextLimit)
}
if err != nil {
var pe *ProviderError
var ce *ContextError
var me *MemoryError
var oe *OperatorError
switch {
case errors.Is(err, ErrChannelBusy):
errorEnvelope(w, http.StatusConflict, "channel_busy", err.Error())
@@ -255,6 +402,26 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) {
errorEnvelope(w, http.StatusServiceUnavailable, "provider_unrouted", clipStr(err.Error(), 300))
case errors.Is(err, ErrProviderKeyMissing):
errorEnvelope(w, http.StatusServiceUnavailable, "provider_key_missing", clipStr(err.Error(), 300))
case errors.As(err, &me) && me.Code == "NATIVE_UNAVAILABLE":
errorEnvelope(w, http.StatusServiceUnavailable, "native_unavailable", clipStr(err.Error(), 300))
case errors.As(err, &me):
errorEnvelope(w, http.StatusInternalServerError, strings.ToLower(me.Code), clipStr(err.Error(), 300))
case errors.As(err, &ce) && ce.Code == "NATIVE_UNAVAILABLE":
errorEnvelope(w, http.StatusServiceUnavailable, "native_unavailable", clipStr(err.Error(), 300))
case errors.As(err, &ce) && (ce.Code == "CONTEXT_UNKNOWN" || ce.Code == "CONTEXT_OVERFLOW"):
errorEnvelope(w, http.StatusBadRequest, strings.ToLower(ce.Code), clipStr(err.Error(), 300))
case errors.As(err, &ce) && ce.Code == "PROTOCOL_ERROR":
errorEnvelope(w, http.StatusBadGateway, "context_protocol_error", clipStr(err.Error(), 300))
case errors.As(err, &ce):
errorEnvelope(w, http.StatusInternalServerError, "context_error", clipStr(err.Error(), 300))
case errors.As(err, &oe) && oe.Code == "NATIVE_UNAVAILABLE":
errorEnvelope(w, http.StatusServiceUnavailable, "native_unavailable", clipStr(err.Error(), 300))
case errors.As(err, &oe) && oe.Code == "PROTOCOL_ERROR":
errorEnvelope(w, http.StatusBadGateway, "operator_protocol_error", clipStr(err.Error(), 300))
case errors.As(err, &oe) && (oe.Code == "ILLEGAL_TRANSITION" || oe.Code == "UNKNOWN_STATUS"):
errorEnvelope(w, http.StatusConflict, "lifecycle_error", clipStr(err.Error(), 300))
case errors.As(err, &oe):
errorEnvelope(w, http.StatusInternalServerError, "operator_error", clipStr(err.Error(), 300))
case errors.As(err, &pe):
errorEnvelope(w, http.StatusBadGateway, "provider_error", clipStr(err.Error(), 300))
default:
@@ -264,16 +431,7 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) {
}
assistantText := res.AssistantText
runeBlk := map[string]any{
"success": true,
"user_text": assistantText,
"model": res.Model,
"error": "",
"channel": channel,
"thinking": res.Thinking,
"messages": map[string]string{"user": res.UserMsgID, "assistant": res.AssistantMsgID},
"core": "go",
}
runeBlk := turnRuneBlock(res, nil, channel)
id := newCompletionID()
created := time.Now().UTC().Unix()
writeJSON(w, http.StatusOK, map[string]any{
@@ -300,7 +458,7 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) {
// finish_reason and the rune block. An error before the first byte
// answers the honest error envelope; an error mid-stream answers an
// error terminal chunk (success=false — the stream shape stays valid).
func (s *Server) streamCompletion(w http.ResponseWriter, r *http.Request, model, channel, userText string) {
func (s *Server) streamCompletion(w http.ResponseWriter, r *http.Request, model, channel, userText, signalID string) {
id := newCompletionID()
created := time.Now().UTC().Unix()
header := false
@@ -344,26 +502,17 @@ func (s *Server) streamCompletion(w http.ResponseWriter, r *http.Request, model,
writeChunk(chunkFor(d.Text))
}
res, err := s.engine.TurnStream(r.Context(), TurnRequest{
Channel: channel, UserText: userText, Model: model,
Channel: channel, UserText: userText, Model: model, SignalID: signalID,
}, sink)
if res.ContextLimit > 0 {
s.setContext(res.ContextUsed, res.ContextLimit)
}
runeBlk := map[string]any{
"success": err == nil,
"user_text": res.AssistantText,
"model": res.Model,
"error": "",
"channel": channel,
"thinking": res.Thinking,
"messages": map[string]string{"user": res.UserMsgID, "assistant": res.AssistantMsgID},
"core": "go",
}
if err != nil {
runeBlk["error"] = clipStr(err.Error(), 300)
}
runeBlk := turnRuneBlock(res, err, channel)
if err != nil && !header {
var pe *ProviderError
var ce *ContextError
var me *MemoryError
var oe *OperatorError
switch {
case errors.Is(err, ErrChannelBusy):
errorEnvelope(w, http.StatusConflict, "channel_busy", err.Error())
@@ -373,6 +522,26 @@ func (s *Server) streamCompletion(w http.ResponseWriter, r *http.Request, model,
errorEnvelope(w, http.StatusServiceUnavailable, "provider_unrouted", clipStr(err.Error(), 300))
case errors.Is(err, ErrProviderKeyMissing):
errorEnvelope(w, http.StatusServiceUnavailable, "provider_key_missing", clipStr(err.Error(), 300))
case errors.As(err, &me) && me.Code == "NATIVE_UNAVAILABLE":
errorEnvelope(w, http.StatusServiceUnavailable, "native_unavailable", clipStr(err.Error(), 300))
case errors.As(err, &me):
errorEnvelope(w, http.StatusInternalServerError, strings.ToLower(me.Code), clipStr(err.Error(), 300))
case errors.As(err, &ce) && ce.Code == "NATIVE_UNAVAILABLE":
errorEnvelope(w, http.StatusServiceUnavailable, "native_unavailable", clipStr(err.Error(), 300))
case errors.As(err, &ce) && (ce.Code == "CONTEXT_UNKNOWN" || ce.Code == "CONTEXT_OVERFLOW"):
errorEnvelope(w, http.StatusBadRequest, strings.ToLower(ce.Code), clipStr(err.Error(), 300))
case errors.As(err, &ce) && ce.Code == "PROTOCOL_ERROR":
errorEnvelope(w, http.StatusBadGateway, "context_protocol_error", clipStr(err.Error(), 300))
case errors.As(err, &ce):
errorEnvelope(w, http.StatusInternalServerError, "context_error", clipStr(err.Error(), 300))
case errors.As(err, &oe) && oe.Code == "NATIVE_UNAVAILABLE":
errorEnvelope(w, http.StatusServiceUnavailable, "native_unavailable", clipStr(err.Error(), 300))
case errors.As(err, &oe) && oe.Code == "PROTOCOL_ERROR":
errorEnvelope(w, http.StatusBadGateway, "operator_protocol_error", clipStr(err.Error(), 300))
case errors.As(err, &oe) && (oe.Code == "ILLEGAL_TRANSITION" || oe.Code == "UNKNOWN_STATUS"):
errorEnvelope(w, http.StatusConflict, "lifecycle_error", clipStr(err.Error(), 300))
case errors.As(err, &oe):
errorEnvelope(w, http.StatusInternalServerError, "operator_error", clipStr(err.Error(), 300))
case errors.As(err, &pe):
errorEnvelope(w, http.StatusBadGateway, "provider_error", clipStr(err.Error(), 300))
default:
@@ -34,7 +34,9 @@ func newTestServer(t *testing.T, fm *fakeModel, home string) (*Server, *httptest
t.Fatal(err)
}
}
srv := NewServer(NewEngine(st, fm, home), fm, home)
eng := NewEngine(st, fm, home)
eng.SetContextAssembler(passthroughContextAssembler{})
srv := NewServer(eng, fm, home)
ts := httptest.NewServer(srv.Handler())
t.Cleanup(ts.Close)
return srv, ts
+6 -1
View File
@@ -35,6 +35,7 @@ endpoints (the TUI/serve-door contract, served directly):
GET /v1/status identity/active_model (supervision gate)
GET /v1/conversation ?channel=X&n=N window over the channel's store
GET /v1/health liveness + identity
POST /v1/memory/recall canonical Go/Rust memory recall (typed NDJSON store boundary)
POST /v1/chat/completions one channel turn (channel in body; stream=true → NDJSON)
POST /v1/approvals/resolve honest no-pending answer (wave 1)
`
@@ -113,7 +114,11 @@ func main() {
// provider spellings (standardized table + landed registry + vault
// key read at call time); the local binding keeps this client.
router := runecore.NewProviderRouter(runecore.SecretToolReader())
srv := runecore.NewServer(runecore.NewRoutedEngine(st, client, home, router), client, home)
memory := runecore.NewNativeMemoryStore("")
engine := runecore.NewRoutedEngine(st, client, home, router)
engine.SetContextAssembler(runecore.NewNativeContextAssembler(memory, ""))
srv := runecore.NewServer(engine, client, home)
srv.SetMemoryStore(memory)
// D56 connection health (#2581): poll the registry's providers'
// live state so the governor routes only to providers actually
+46 -10
View File
@@ -36,7 +36,17 @@ _lib = None
class AttentionCapsuleCoreUnavailable(FileNotFoundError):
"""The Rust attention core is not built. Callers fall back to Python."""
"""The Rust attention core is not built."""
class NativeContextError(RuntimeError):
"""Stable native context failure; never converted into a Python default."""
def __init__(self, message, *, code="NATIVE_UNAVAILABLE", operation=None, cause=None):
super().__init__(message)
self.code = code
self.operation = operation
self.cause = cause
def available() -> bool:
@@ -51,7 +61,8 @@ def _load() -> ctypes.CDLL:
raise AttentionCapsuleCoreUnavailable(str(_LIB_PATH))
lib = ctypes.CDLL(str(_LIB_PATH))
for sym in ("ac_to_dict", "ac_to_prompt", "ac_to_messages",
"ac_capture_messages", "ac_estimate_tokens", "ac_create_capsule"):
"ac_capture_messages", "ac_estimate_tokens", "ac_create_capsule",
"ac_admit"):
getattr(lib, sym).argtypes = [ctypes.c_void_p]
getattr(lib, sym).restype = ctypes.c_void_p
# free comes from capability_scoring (same process) if loaded; else our own malloc path
@@ -68,19 +79,37 @@ def _load() -> ctypes.CDLL:
def _call(sym: str, payload):
lib = _load()
data = json.dumps(payload).encode()
buf = ctypes.create_string_buffer(data)
ptr = getattr(lib, sym)(ctypes.cast(buf, ctypes.c_void_p))
try:
lib = _load()
data = json.dumps(payload).encode()
buf = ctypes.create_string_buffer(data)
ptr = getattr(lib, sym)(ctypes.cast(buf, ctypes.c_void_p))
except Exception as exc:
raise NativeContextError(
f"native attention operation {sym} unavailable: {exc}",
code="NATIVE_UNAVAILABLE", operation=sym, cause=exc,
) from exc
if not ptr:
raise RuntimeError(f"{sym} returned null")
out = ctypes.string_at(ptr).decode()
raise NativeContextError(f"{sym} returned null", code="PROTOCOL_ERROR", operation=sym)
raw = ctypes.string_at(ptr)
free = getattr(lib, "_free", None)
if free is not None:
free(ptr)
doc = json.loads(out)
try:
doc = json.loads(raw.decode())
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
raise NativeContextError(
f"{sym} returned non-JSON", code="PROTOCOL_ERROR", operation=sym, cause=exc,
) from exc
if not isinstance(doc, dict):
raise NativeContextError(f"{sym} returned a non-object envelope", code="PROTOCOL_ERROR", operation=sym)
if not doc.get("ok"):
raise RuntimeError(doc.get("err", f"{sym} failed"))
error = doc.get("error") if isinstance(doc.get("error"), dict) else {}
code = error.get("code") or "CONTEXT_DECISION_ERROR"
message = error.get("message") or doc.get("err") or f"{sym} failed"
raise NativeContextError(message, code=code, operation=sym)
if "value" not in doc:
raise NativeContextError(f"{sym} omitted value", code="PROTOCOL_ERROR", operation=sym)
return doc["value"]
@@ -138,3 +167,10 @@ def create_capsule(task: str, context: str = "", max_tokens: int = 600) -> dict:
return _call("ac_create_capsule", {
"task": task, "context": context, "max_tokens": max_tokens,
})
def admit(entries: list, budget: int, memory_budget: int) -> dict:
return _call("ac_admit", {
"entries": entries, "budget": budget, "memory_budget": memory_budget,
"enforce_collective": True,
})
+31 -63
View File
@@ -6,6 +6,10 @@ Target: 300-600 tokens typical, ~800 tokens hard max.
The capsule only includes what's relevant to the current task,
not the entire constitution every time.
Compatibility shim (retire with the Python-authority tranche): native Rust
decisions are mandatory by default. RUNE_AC_CORE=python is solely the explicit
independent-oracle pin, never an implicit failure fallback.
Usage:
attention_capsule.py create <task> [--context <context>]
attention_capsule.py test
@@ -49,15 +53,9 @@ class AttentionCapsule:
blocks: List["ContributionBlock"] = field(default_factory=list)
def to_dict(self):
# Dichotomy wave (#2479): delegate to the Rust kernel when built
# (Python fallback — strangler law; RUNE_AC_CORE=python pins it).
if os.environ.get("RUNE_AC_CORE", "auto") != "python":
try:
from rune import ac_loader
if ac_loader.available():
return ac_loader.to_dict(self)
except Exception:
pass # fallback below; never crash the caller
from rune import ac_loader
return ac_loader.to_dict(self)
return {
"active_objective": self.active_objective,
"current_phase": self.current_phase,
@@ -79,15 +77,9 @@ class AttentionCapsule:
blocks render as history messages, and the block named "user"
renders as the final user message. Re-derived on every call.
"""
# Dichotomy wave (#2479): delegate to the Rust kernel when built
# (Python fallback — strangler law; RUNE_AC_CORE=python pins it).
if os.environ.get("RUNE_AC_CORE", "auto") != "python":
try:
from rune import ac_loader
if ac_loader.available():
return ac_loader.to_messages(self, system_preamble)
except Exception:
pass # fallback below; never crash the caller
from rune import ac_loader
return ac_loader.to_messages(self, system_preamble)
system_parts = [system_preamble] if system_preamble else []
history: List[dict] = []
user_content = ""
@@ -119,16 +111,10 @@ class AttentionCapsule:
The list is deep-copied so later caller-side mutation can never
diverge the logged payload from what was sent.
"""
# Dichotomy wave (#2479): delegate to the Rust kernel when built
# (Python fallback — strangler law; RUNE_AC_CORE=python pins it).
if os.environ.get("RUNE_AC_CORE", "auto") != "python":
try:
from rune import ac_loader
if ac_loader.available():
self.sent_messages = ac_loader.capture_messages(messages)
return self.sent_messages
except Exception:
pass # fallback below; never crash the caller
from rune import ac_loader
self.sent_messages = ac_loader.capture_messages(messages)
return self.sent_messages
self.sent_messages = json.loads(json.dumps(list(messages)))
return self.sent_messages
@@ -145,15 +131,9 @@ class AttentionCapsule:
DeprecationWarning,
stacklevel=2,
)
# Dichotomy wave (#2479): delegate to the Rust kernel when built
# (Python fallback — strangler law; RUNE_AC_CORE=python pins it).
if os.environ.get("RUNE_AC_CORE", "auto") != "python":
try:
from rune import ac_loader
if ac_loader.available():
return ac_loader.to_prompt(self)
except Exception:
pass # fallback below; never crash the caller
from rune import ac_loader
return ac_loader.to_prompt(self)
parts = []
if self.active_objective:
@@ -193,15 +173,9 @@ class AttentionCapsule:
def estimate_tokens(text: str) -> int:
"""Estimate token count (rough: 1 token ≈ 4 chars)."""
# Dichotomy wave (#2479): delegate to the Rust kernel when built
# (Python fallback — strangler law; RUNE_AC_CORE=python pins it).
if os.environ.get("RUNE_AC_CORE", "auto") != "python":
try:
from rune import ac_loader
if ac_loader.available():
return ac_loader.estimate_tokens(text)
except Exception:
pass # fallback below; never crash the caller
from rune import ac_loader
return ac_loader.estimate_tokens(text)
return len(text) // 4
@@ -222,29 +196,23 @@ def create_capsule(task: str, context: str = "", max_tokens: int = 600) -> Atten
DeprecationWarning,
stacklevel=2,
)
# Dichotomy wave (#2479): delegate to the Rust kernel when built
# (Python fallback — strangler law; RUNE_AC_CORE=python pins it).
if os.environ.get("RUNE_AC_CORE", "auto") != "python":
try:
from rune import ac_loader
if ac_loader.available():
from rune.context_broker import ContributionBlock
out = ac_loader.create_capsule(task, context, max_tokens)
capsule = AttentionCapsule(
active_objective=out["active_objective"],
current_phase=out["current_phase"],
relevant_memories=out["relevant_memories"],
active_commitments=out["active_commitments"],
important_constraints=out["important_constraints"],
current_blocker=out["current_blocker"],
recent_correction=out["recent_correction"],
critical_assumptions=out["critical_assumptions"],
token_count=out["token_count"],
)
capsule.blocks = [ContributionBlock(**b) for b in out["blocks"]]
return capsule
except Exception:
pass # fallback below; never crash the caller
from rune import ac_loader
from rune.context_broker import ContributionBlock
out = ac_loader.create_capsule(task, context, max_tokens)
capsule = AttentionCapsule(
active_objective=out["active_objective"],
current_phase=out["current_phase"],
relevant_memories=out["relevant_memories"],
active_commitments=out["active_commitments"],
important_constraints=out["important_constraints"],
current_blocker=out["current_blocker"],
recent_correction=out["recent_correction"],
critical_assumptions=out["critical_assumptions"],
token_count=out["token_count"],
)
capsule.blocks = [ContributionBlock(**b) for b in out["blocks"]]
return capsule
capsule = AttentionCapsule()
# Extract objective from task
+1 -1
View File
@@ -177,7 +177,7 @@ def check_escape_hatch_round_trip() -> int:
def check_scripted_operator_wiring() -> int:
"""Real RuneOperator feeds the selection to the runtime + trace v3."""
storage = Path(_canary_home) / "operator.json"
op = rune_operator.RuneOperator(str(storage))
op = rune_operator.RuneOperator(str(storage), core="python")
trace = {
"model_id": "qwen3:14b",
"window": 40960,
+1 -1
View File
@@ -91,7 +91,7 @@ class _ScriptedRuntime:
def _scripted_operator_run(description: str, tier_params: dict):
"""Real RuneOperator + scripted broker/transport → (envelope, runtime, capsule)."""
storage = Path(_canary_home) / "operator.json"
op = rune_operator.RuneOperator(str(storage))
op = rune_operator.RuneOperator(str(storage), core="python")
capsule = AttentionCapsule()
capsule.blocks = [
ContributionBlock("persona", 90, "I am Rune.", len("I am Rune.") // 4, "persona", evictable=False),
+3 -3
View File
@@ -101,7 +101,7 @@ def check_trivial_task_abstains() -> int:
from rune.context_broker import MEMORY_BLOCK_NAMES
from rune.rune_operator import RuneOperator
op = RuneOperator(str(Path(_canary_home) / "operator.json"))
op = RuneOperator(str(Path(_canary_home) / "operator.json"), core="python")
task = SimpleNamespace(task_id="canary-t1", description="what time is it", assigned_model="")
broker = op._context_broker()
session: dict = {}
@@ -132,7 +132,7 @@ def check_importance_fallback_dead() -> int:
from rune.context_broker import MEMORY_BLOCK_NAMES
from rune.rune_operator import RuneOperator
op = RuneOperator(str(Path(_canary_home) / "operator-irrelevant.json"))
op = RuneOperator(str(Path(_canary_home) / "operator-irrelevant.json"), core="python")
broker = op._context_broker()
irrelevant = SimpleNamespace(task_id="canary-t2", description="kubernetes pods restart", assigned_model="")
session: dict = {}
@@ -160,7 +160,7 @@ def check_taskstate_supremacy() -> int:
from rune.orchestration.records import TaskStateRecord
from rune.rune_operator import OPERATOR_CHANNEL, RuneOperator
op = RuneOperator(str(Path(_canary_home) / "operator-supremacy.json"))
op = RuneOperator(str(Path(_canary_home) / "operator-supremacy.json"), core="python")
# Seed a completed past task through the CANONICAL ledger.
past = TaskStateRecord(
title="deploy the kubernetes cluster to production",
+1 -1
View File
@@ -120,7 +120,7 @@ class _ScriptedTransport(OllamaChatTransport):
def _scripted_operator_run():
"""Real RuneOperator + scripted router/broker/transport → send_task."""
storage = Path(_canary_home) / "operator.json"
op = rune_operator.RuneOperator(str(storage))
op = rune_operator.RuneOperator(str(storage), core="python")
decision = SimpleNamespace(
model="qwen3:14b",
+1 -1
View File
@@ -178,7 +178,7 @@ def check_round_trip() -> int:
def _scripted_operator_run():
"""Real RuneOperator + scripted broker/transport → send_task envelope."""
storage = Path(_canary_home) / "operator.json"
op = rune_operator.RuneOperator(str(storage))
op = rune_operator.RuneOperator(str(storage), core="python")
trace = _broker_trace()
capsule = SimpleNamespace(
token_count=trace["admitted_total"],
+62 -17
View File
@@ -3,7 +3,9 @@
All injected context flows through this module. Contributors register,
budgets resolve from the model's real context length, admission is
deterministic (priority, then recency), every block is measured, and a
trace records what was admitted, dropped, and why.
trace records what was admitted, dropped, and why. This is a compatibility
shim for Python contributor callbacks; native Rust budget/admission decisions
are mandatory by default and the Python bodies are explicit-oracle-only.
Order of assembly (fixed): system -> tools -> memory/notes -> history
-> user. Contributors emit typed blocks with priority; admission sorts
@@ -130,17 +132,26 @@ def _cc_core():
return None
def _require_budget_core(operation: str):
core = _cc_core()
if core is None:
from rune.ac_loader import NativeContextError
raise NativeContextError(
f"native budget core unavailable for {operation}",
code="NATIVE_UNAVAILABLE", operation=operation,
)
return core
def default_budget(window: int) -> int:
"""Input budget = window - max(2048, window//4) - window//20.
Reserves room for the completion (at least 2048 or a quarter of the
window) plus a 5% safety margin for serving overhead.
R2 W3.1: delegates to the Rust kernel when built.
"""
core = _cc_core()
if core is not None:
return core.default_budget(window)
if os.environ.get("RUNE_COGNITION_CORE", "auto") != "python":
return _require_budget_core("default_budget").default_budget(window)
return _py_default_budget(window)
@@ -161,11 +172,10 @@ def default_memory_budget(window: int | None) -> int:
MEMORY_BLOCK_NAMES block may never exceed this, regardless of the
global input budget. An unknown window falls back to the cap only.
R2 W3.1: delegates to the Rust kernel when built.
Native Rust is mandatory unless the independent oracle is explicitly pinned.
"""
core = _cc_core()
if core is not None:
return core.default_memory_budget(int(window) if window else 0)
if os.environ.get("RUNE_COGNITION_CORE", "auto") != "python":
return _require_budget_core("default_memory_budget").default_memory_budget(int(window) if window else 0)
return _py_default_memory_budget(window)
@@ -654,6 +664,38 @@ class ContextBroker:
"""
if memory_budget is None:
memory_budget = default_memory_budget(None)
native_pinned = (
os.environ.get("RUNE_COGNITION_CORE", "auto") == "python"
or os.environ.get("RUNE_AC_CORE", "auto") == "python"
)
if not native_pinned:
from rune import ac_loader
entries = [block.to_dict() for _name, block, _bucket in collected]
try:
decision = ac_loader.admit(entries, budget, memory_budget)
indices = decision["admitted_indices"]
admitted = [collected[index][1] for index in indices]
native_trace = decision["trace"]
if native_trace.get("context") != "rust-attention-capsule":
raise ValueError("unexpected native trace")
except ac_loader.NativeContextError as exc:
if exc.code == "CONTEXT_OVERFLOW":
raise ContextOverflowError(str(exc)) from exc
raise
except (AttributeError, KeyError, IndexError, TypeError, ValueError) as exc:
raise ac_loader.NativeContextError(
f"invalid native admission receipt: {exc}",
code="PROTOCOL_ERROR", operation="ac_admit", cause=exc,
) from exc
return admitted, {
"blocks": decision["blocks"],
"admitted_total": decision["admitted_total"],
"dropped": decision["dropped"],
"memory_budget": decision["memory_budget"],
"memory_used": decision["memory_used"],
"native_decision": native_trace,
}
recency = {id(block): index for index, (_name, block, _b) in enumerate(collected)}
def sort_key(entry):
@@ -784,13 +826,14 @@ class ContextBroker:
# also a hard error — several mandatory blocks that individually fit
# must not silently exceed the budget (that would reintroduce the
# provider-side truncation the broker exists to prevent).
mandatory = [b for b in admitted if not b.evictable]
mandatory_total = sum(b.tokens for b in mandatory)
if mandatory_total > budget:
raise ContextOverflowError(
f"non-evictable blocks total {mandatory_total} tokens > budget {budget} "
f"({[b.name for b in mandatory]}) — refusing to silently truncate"
)
if admission.get("native_decision") is None:
mandatory = [b for b in admitted if not b.evictable]
mandatory_total = sum(b.tokens for b in mandatory)
if mandatory_total > budget:
raise ContextOverflowError(
f"non-evictable blocks total {mandatory_total} tokens > budget {budget} "
f"({[b.name for b in mandatory]}) — refusing to silently truncate"
)
memory_tokens = sum(block.tokens for block in admitted if block.name in MEMORY_BLOCK_NAMES)
trace = {
@@ -807,6 +850,8 @@ class ContextBroker:
"compaction": compaction_events,
"elapsed_ms": round((time.monotonic() - started) * 1000, 3),
}
if admission.get("native_decision") is not None:
trace["native_decision"] = admission["native_decision"]
capsule = self._render(admitted)
with self._lock:
+18 -23
View File
@@ -1,24 +1,19 @@
#!/usr/bin/env python3
"""memory_env.py — portable path resolution for the memory system.
"""Compatibility facade for the central paths authority (#2618).
Every path in the package is derived from a small set of roots that can be
overridden via environment variables, so the same code runs on ANY machine:
RUNE_HOME Rune home directory (default /mnt/workdrive/rune)
RUNE_MEMORY_DIR memory data root (default <home>/memory)
RUNE_SCRIPTS_DIR where the scripts live (default <home>/src/rune)
RUNE_STORE_DB explicit store path (default <memory>/store/memory.db)
Rules:
- Never hardcode a user path; always go through these helpers.
- Scripts invoked as `python <script>` will use sys.executable for child
processes (same venv), which is portable by construction.
Path policy and defaults live in ``paths`` and its Go engine. This module
keeps the established memory API without maintaining a second resolver.
"""
import os
import shutil
import sys
try:
from . import paths
except ImportError:
import paths
def expand(p):
"""expanduser + expandvars, for env values that may contain ~ or $HOME."""
@@ -28,28 +23,28 @@ def expand(p):
def rune_home():
"""Return the Rune home directory."""
return expand(os.environ.get("RUNE_HOME", "/mnt/workdrive/rune"))
"""Return the Rune home directory from the paths authority."""
return paths.rune_home()
def memory_dir():
"""Return the memory data directory."""
return expand(os.environ.get("RUNE_MEMORY_DIR", os.path.join(rune_home(), "memory")))
"""Return the memory data directory from the paths authority."""
return paths.memory_dir()
def scripts_dir():
"""Return the scripts directory."""
return expand(os.environ.get("RUNE_SCRIPTS_DIR", os.path.dirname(os.path.abspath(__file__))))
"""Return the scripts directory from the paths authority."""
return paths.scripts_dir()
def store_db():
"""Return the path to the memory store database."""
return expand(os.environ.get("RUNE_STORE_DB", os.path.join(memory_dir(), "store", "memory.db")))
"""Return the memory store database from the paths authority."""
return paths.store_db()
def config_dir():
"""Return the config directory."""
return expand(os.environ.get("RUNE_CONFIG_DIR", os.path.join(rune_home(), "config")))
"""Return the config directory from the paths authority."""
return paths.config_dir()
def python_bin():
+90
View File
@@ -0,0 +1,90 @@
"""Typed, fail-loud client for the canonical Go/Rust memory store."""
from __future__ import annotations
import json
import os
from pathlib import Path
import subprocess
import threading
class NativeMemoryStoreError(RuntimeError):
def __init__(self, message, *, code="NATIVE_UNAVAILABLE", operation=None, cause=None):
super().__init__(message)
self.code = code
self.operation = operation
self.cause = cause
_local = threading.local()
def _binary():
override = os.environ.get("RUNE_MEMORY_STORE_BIN")
if override:
return override
root = Path(__file__).resolve().parents[2]
return str(root / "src" / "go" / "memory-store-deep" / "memory-store-deep")
def native_call(operation, argument=None, *, db_path=None):
arg = dict(argument or {})
if db_path is not None:
arg["db_path"] = db_path
binary = _binary()
if not os.path.isfile(binary) or not os.access(binary, os.X_OK):
raise NativeMemoryStoreError(
f"canonical memory-store native binary missing or not executable: {binary}",
operation=operation,
)
request = json.dumps({"op": operation, "arg": arg}, ensure_ascii=False) + "\n"
try:
proc = subprocess.run(
[binary, "eval"],
input=request,
capture_output=True,
text=True,
timeout=120,
)
except (OSError, subprocess.TimeoutExpired) as exc:
raise NativeMemoryStoreError(
f"canonical memory-store native call failed to launch: {exc}",
operation=operation,
cause=exc,
) from exc
lines = [line for line in proc.stdout.splitlines() if line.strip()]
if len(lines) != 1:
raise NativeMemoryStoreError(
f"canonical memory-store returned invalid NDJSON (exit {proc.returncode}): "
f"{proc.stdout.strip() or proc.stderr.strip()}",
code="PROTOCOL_ERROR",
operation=operation,
)
try:
envelope = json.loads(lines[0])
except json.JSONDecodeError as exc:
raise NativeMemoryStoreError(
f"canonical memory-store returned non-JSON: {lines[0]}",
code="PROTOCOL_ERROR",
operation=operation,
cause=exc,
) from exc
if not isinstance(envelope, dict):
raise NativeMemoryStoreError(
"canonical memory-store response must be a JSON object",
code="PROTOCOL_ERROR",
operation=operation,
)
trace = envelope.get("trace")
_local.trace = dict(trace) if isinstance(trace, dict) else {}
if proc.returncode != 0 or not envelope.get("ok"):
error = envelope.get("error") if isinstance(envelope.get("error"), dict) else {}
code = error.get("code") or ("NATIVE_UNAVAILABLE" if proc.returncode == 127 else "STORE_ERROR")
message = error.get("message") or proc.stderr.strip() or f"native exit {proc.returncode}"
raise NativeMemoryStoreError(message, code=code, operation=operation)
return envelope.get("value")
def native_last_trace():
return dict(getattr(_local, "trace", {}))
+59 -39
View File
@@ -25,8 +25,11 @@ Policy (issue #26):
selector itself is unavailable the contributor abstains — never fail open
to "inject something important".
Degrade-not-crash throughout: sources are pluggable and isolated; one
exploding source never takes down assembly.
Source I/O remains pluggable and isolated; one exploding source never takes
down assembly. The native Rust selection core itself is required by default:
its absence/failure is typed and never falls back to Python ranking. The
RUNE_MSE_CORE=python body is the explicit independent oracle and a retirement
candidate, not an implicit serving fallback.
"""
from __future__ import annotations
@@ -50,18 +53,27 @@ except ImportError: # direct script execution
LOG = logging.getLogger("rune.memory_selector")
def _mse_core():
"""The Rust selection-core probe (strangler law). RUNE_MSE_CORE=python
pins the Python fallback — the pin is honored inside mse_loader.core so
every consumer gets it. Returns the loader MODULE (never the raw CDLL:
dlsym on the cdylib handle would resolve short names like `select`
against libc instead of the mse_* exports)."""
try:
from rune import mse_loader
def _mse_core(operation="mse_select"):
"""Required Rust selection core; Python requires the explicit oracle pin."""
import os
from rune.ac_loader import NativeContextError
from rune import mse_loader
return mse_loader if mse_loader.core() is not None else None
except Exception: # noqa: BLE001 — kernel failure → oracle below
if os.environ.get("RUNE_MSE_CORE", "auto") == "python":
return None
try:
core = mse_loader.core()
except Exception as exc:
raise NativeContextError(
f"native memory-selection core unavailable: {exc}",
code="NATIVE_UNAVAILABLE", operation=operation, cause=exc,
) from exc
if core is None:
raise NativeContextError(
"native memory-selection core unavailable",
code="NATIVE_UNAVAILABLE", operation=operation,
)
return mse_loader
# Candidate cap BEFORE budgeting (plan DD 5: the selector caps its own
# candidate list before returning).
@@ -148,7 +160,7 @@ class MemorySelector:
owned by the Rust core when built (mse_add_source — frozen truth:
append vs position-preserving replace); the facade applies the
kernel's order as authoritative while carrying the caller's fn."""
_core = _mse_core()
_core = _mse_core("mse_add_source")
if _core is not None:
names = [str(n) for n in _core.add_source(list(self._sources), str(name))]
merged: dict[str, Callable] = {}
@@ -195,7 +207,17 @@ class MemorySelector:
_core = _mse_core()
if _core is not None:
out = _core.select(candidates, budget, self.min_score, types_queried)
try:
out = _core.select(candidates, budget, self.min_score, types_queried)
except Exception as exc:
from rune.ac_loader import NativeContextError
if isinstance(exc, NativeContextError):
raise
raise NativeContextError(
f"native memory selection failed: {exc}",
code="CONTEXT_DECISION_ERROR", operation="mse_select", cause=exc,
) from exc
blocks = [
ContributionBlock(
b["name"],
@@ -325,26 +347,24 @@ def _store_recall(
carries no memory_type column, so routing through it would break typed
recall and the preserve-stores/isolation invariants.
"""
db = memory_store.connect(db_path)
try:
results = memory_store.recall(db, query, budget=budget, min_score=min_score, memory_types=types or None)
_core = _mse_core()
persona_types = ["self", "personality"]
if _core is not None:
# The lane gate delegates to the Rust core (mse_persona_wanted):
# the persona lane runs unless the caller itself asked for
# self/personality types — wanted==[] means the lane must run.
persona_lane = not _core.persona_wanted(list(types or []))
else:
persona_lane = not (set(types or []) & set(persona_types))
if persona_lane:
# The persona lane: always executed for the untyped production
# caller (and any caller that did not itself ask persona types).
results = results + memory_store.recall(
db, query, budget=budget, min_score=min_score, memory_types=persona_types
)
finally:
db.close()
results = memory_store.native_recall(
db_path, query, budget=budget, min_score=min_score, memory_types=types or None
)
_core = _mse_core("mse_store_project")
persona_types = ["self", "personality"]
if _core is not None:
# The lane gate delegates to the Rust core (mse_persona_wanted):
# the persona lane runs unless the caller itself asked for
# self/personality types — wanted==[] means the lane must run.
persona_lane = not _core.persona_wanted(list(types or []))
else:
persona_lane = not (set(types or []) & set(persona_types))
if persona_lane:
# The persona lane: always executed for the untyped production
# caller (and any caller that did not itself ask persona types).
results = results + memory_store.native_recall(
db_path, query, budget=budget, min_score=min_score, memory_types=persona_types
)
if _core is not None:
return _core.store_project(results)
out: list[dict] = []
@@ -410,7 +430,7 @@ def _neurons_recall(query: str, types: list[str] | None, budget: int, min_score:
import rune.warm_neuron_store as warm_neuron_store # type: ignore[no-redef]
neurons = warm_neuron_store.list_neurons()
_core = _mse_core()
_core = _mse_core("mse_neurons_decision")
if _core is not None:
return _core.neurons_decision(neurons, query, budget, min_score)
task_words = {w.lower().strip(".,;:!?()[]{}\"'") for w in query.split()}
@@ -460,7 +480,7 @@ def _graphs_recall(query: str, types: list[str] | None, budget: int, min_score:
lines = [f"{f.get('subject')} {f.get('predicate', '-')} {f.get('object')}" for f in facts]
text = "; ".join(lines) if lines else str(entity)
texts.append(text)
_core = _mse_core()
_core = _mse_core("mse_graphs_decision")
if _core is not None:
return _core.graphs_decision(texts, query, len(task_words), budget, min_score)
for text in texts:
@@ -499,7 +519,7 @@ def _abstraction_recall(query: str, types: list[str] | None, budget: int, min_sc
params,
).fetchall()
texts = [str(row[0]) for row in rows]
_core = _mse_core()
_core = _mse_core("mse_abstraction_decision")
if _core is not None:
return _core.abstraction_decision(texts, query, budget, min_score)
for text in texts:
@@ -539,7 +559,7 @@ def _consolidated_recall(query: str, types: list[str] | None, budget: int, min_s
if not content:
continue
out.append(content)
_core = _mse_core()
_core = _mse_core("mse_consolidated_decision")
if _core is not None:
return _core.consolidated_decision(out, query, budget, min_score)
contents = out
@@ -572,7 +592,7 @@ def _persona_recall(
PINNED store (``db_path``) — never the module-default store that a bare
``memory_store.connect()`` would open.
"""
_core = _mse_core()
_core = _mse_core("mse_persona_wanted")
if _core is not None:
wanted_list = _core.persona_wanted(types)
else:
+58 -28
View File
@@ -44,6 +44,11 @@ if _SD not in sys.path:
sys.path.insert(0, _SD)
import memory_env
try:
from . import memory_native
except ImportError:
import memory_native
# DEP06: memory writes mediate through the ONE effect registry (store
# boundary). Dual import: this module is also launched script-style.
try:
@@ -838,7 +843,9 @@ def _py_score_recall_rows(fusion, rows, dense, bm25, budget, min_score, now, rec
src/rust/memory_retrieval (parity proven against the frozen oracle in
frozen_score_recall.json). Delegation-only stub; the public delegator
raises if the kernel is absent."""
return _score_recall_rows(fusion, rows, dense, bm25, budget, min_score, now)
return _required_mr_core().score_recall_rows(
fusion, rows, dense, bm25, budget, min_score, now, recency_half_life
)
def _drift_rerank(db, entries):
"""Drift-aware re-ranking: apply multi-factor value and FAMA scoring.
@@ -888,16 +895,27 @@ def _drift_rerank(db, entries):
def _mr_core():
"""Rust memory_retrieval core when built; Python fallback remains
load-bearing during the strangler. RUNE_RETRIEVAL_CORE=python pins it."""
"""Required Rust retrieval core; an explicit python pin is test-only."""
if os.environ.get("RUNE_RETRIEVAL_CORE", "auto") == "python":
return None
try:
from rune import mr_loader
except Exception as exc:
raise memory_native.NativeMemoryStoreError(
f"memory_retrieval loader failed: {exc}", operation="_mr_core", cause=exc
) from exc
if not mr_loader.available():
raise memory_native.NativeMemoryStoreError(
f"memory_retrieval core not built: {mr_loader._LIB}", operation="_mr_core"
)
return mr_loader
return mr_loader if mr_loader.available() else None
except Exception: # noqa: BLE001 — loader probe failure means Python fallback
return None
def _required_mr_core():
core = _mr_core()
if core is None:
raise RuntimeError("memory_retrieval Python oracle is retired; unpin RUNE_RETRIEVAL_CORE")
return core
def _map_reduce(entries: list, chunk_size: int = 10) -> list:
@@ -912,7 +930,7 @@ def _py_map_reduce(entries, chunk_size=10):
src/rust/memory_retrieval (parity proven against the frozen oracle in
frozen_score_recall.json). Delegation-only stub; the public delegator
raises if the kernel is absent."""
return _map_reduce(entries, chunk_size)
return _required_mr_core().map_reduce(entries, chunk_size)
def curiosity_probe(db, limit: int = 5) -> list:
"""Curiosity mechanism: probe random untested memories.
@@ -1117,7 +1135,7 @@ def _py_position_aware(top):
src/rust/memory_retrieval (parity proven against the frozen oracle in
frozen_score_recall.json). Delegation-only stub; the public delegator
raises if the kernel is absent."""
return _position_aware(top)
return _required_mr_core().position_aware(top)
_STOPWORDS = {
"the",
"and",
@@ -1186,7 +1204,7 @@ def _py_fts_query_conjunctive(query):
src/rust/memory_retrieval (parity proven against the frozen oracle in
frozen_score_recall.json). Delegation-only stub; the public delegator
raises if the kernel is absent."""
return _fts_query_conjunctive(query)
return _required_mr_core().fts_query_conjunctive(query)
def _fts_query(query):
"""Backward-compat alias for the conjunctive builder."""
return _fts_query_conjunctive(query)
@@ -1217,7 +1235,8 @@ def _py_project_chunk_rows(ranks, rows, budget):
src/rust/memory_retrieval (parity proven against the frozen oracle in
frozen_score_recall.json). Delegation-only stub; the public delegator
raises if the kernel is absent."""
return _project_chunk_rows(ranks, rows, budget)
pairs = ranks if isinstance(ranks, list) else [list(r) for r in ranks.items()]
return _required_mr_core().project_chunk_rows(pairs, rows, budget)
def _rank_fusion(dense, bm25):
"""Pure dense/BM25 fusion and mode selection (R2 W2.1).
@@ -1235,7 +1254,7 @@ def _py_rank_fusion(dense, bm25, rrf_k=RRF_K):
src/rust/memory_retrieval (parity proven against the frozen oracle in
frozen_score_recall.json). Delegation-only stub; the public delegator
raises if the kernel is absent."""
return _rank_fusion(dense, bm25)
return _required_mr_core().rank_fusion(dense, bm25, rrf_k)
def update_entry(db, eid, text=None, importance=None, valid_until=None):
sets, params = [], []
if text is not None:
@@ -1434,7 +1453,7 @@ def _py_chunk_text(text, size=1500, overlap=150):
src/rust/memory_retrieval (parity proven against the frozen oracle in
frozen_score_recall.json). Delegation-only stub; the public delegator
raises if the kernel is absent."""
return _chunk_text(text, size, overlap)
return _required_mr_core().chunk_text(text, size, overlap)
def add_chunk(
db, text, wing="", room="", source_path="", doc_id="", start_line=None, end_line=None, importance=0.5, page=None
):
@@ -1865,24 +1884,35 @@ class _RustReadBackend:
return [tuple(r) for r in mems_loader.read_query(db_path, sql, list(params))]
def storage_backend():
"""The active read backend. Default sqlite; RUNE_MEMORY_STORAGE=rust
opts into the crate when built (falls back loudly-free to sqlite)."""
if os.environ.get("RUNE_MEMORY_STORAGE", "auto") == "rust":
try:
from rune import mems_loader
class _NativeBackend:
"""Default serving reads through the canonical Go/Rust store."""
if mems_loader.available():
return _RustReadBackend()
except Exception: # noqa: BLE001, S110 — backend probe failure means the default serves (fallback, not silence)
pass
return _SQLiteBackend()
name = "go-memory-store"
def read_query(self, db_path: str, sql: str, params=()) -> list:
rows = memory_native.native_call("read_query", {"sql": sql, "params": list(params)}, db_path=db_path)
return [tuple(row) for row in rows]
def storage_backend():
"""The canonical native backend; native failure is never a Python fallback."""
return _NativeBackend()
def native_recall(db_path, query, budget=RECALL_BUDGET, min_score=RECALL_MIN_SCORE, memory_types=None):
"""Compatibility facade for selector-serving recall through the native store."""
return memory_native.native_call(
"recall",
{"query": query, "budget": budget, "min_score": min_score, "memory_types": memory_types},
db_path=db_path,
)
def stats_for_path(db_path: str) -> dict:
"""stats() through the storage backend seam — the opt-in consumer the
parity battery and future Rust callers use. Same result dict shape."""
"""stats() through the canonical native serving path."""
backend = storage_backend()
if isinstance(backend, _NativeBackend):
return memory_native.native_call("stats", db_path=db_path)
e = backend.read_query(db_path, "SELECT COUNT(*) FROM entries WHERE status='active'")[0][0]
w = backend.read_query(db_path, "SELECT COUNT(*) FROM working")[0][0]
a = backend.read_query(db_path, "SELECT COUNT(*) FROM audit_log")[0][0]
@@ -1986,7 +2016,7 @@ def _py_merge_association_rows(base, boost_items, rows, budget):
src/rust/memory_retrieval (parity proven against the frozen oracle in
frozen_score_recall.json). Delegation-only stub; the public delegator
raises if the kernel is absent."""
return _merge_association_rows(base, boost_items, rows, budget)
return _required_mr_core().merge_association_rows(base, boost_items, rows, budget)
def fused_recall(db, query, budget=RECALL_BUDGET, min_score=RECALL_MIN_SCORE, chunk_budget=6, wings=None):
"""FUSED RECALL (audit fix 2026-08-16): search BOTH the atomic entries
table AND the research corpus (chunks) in one call.
@@ -2018,7 +2048,7 @@ def _py_fuse_results(entries, research):
src/rust/memory_retrieval (parity proven against the frozen oracle in
frozen_score_recall.json). Delegation-only stub; the public delegator
raises if the kernel is absent."""
return _fuse_results(entries, research)
return _required_mr_core().fuse_results(entries, research)
def _norm_key(text):
core = _mr_core()
if core is not None:
@@ -2031,7 +2061,7 @@ def _py_norm_key(text):
src/rust/memory_retrieval (parity proven against the frozen oracle in
frozen_score_recall.json). Delegation-only stub; the public delegator
raises if the kernel is absent."""
return _norm_key(text)
return _required_mr_core().norm_key(text)
def make_scene(db, name, member_ids, summary=""):
"""Consolidate related entries into a themed scene (EverMemOS-style):
episodic traces -> semantic scene. Members keep their own identity; the
+3
View File
@@ -1,5 +1,8 @@
"""Atomic hot-state writes (S8).
The serving authority is the Go rune-core AtomicStore. This Python body is
retained for oracle/shared composition until the final retirement tranche.
Hot cursors, queues, and ownership transfers are written via
temp-file + fsync + rename: a crash or writer failure never leaves a
corrupt target, and temp files never leak. Commits, not locks, are the
@@ -1,5 +1,9 @@
"""Canonical ConversationState and TaskState repositories (#18 / P0-02).
The serving authorities are the Go rune-core repositories with Rust
transition/record decisions. These Python bodies remain for explicit oracle
and shared-composition compatibility until the final retirement tranche.
Two repositories form the single conversation/task authority:
- **TaskStateRepository** — canonical task identity. One task persists
+3
View File
@@ -1,5 +1,8 @@
"""Durable append-only event store (S8/S9).
The serving authority is the Go rune-core EventStore. This Python body is
retained for oracle/shared composition until the final retirement tranche.
Envelopes are committed to a durable append-only JSONL store before any
fan-out. Offsets are 1-based line numbers. Writes are batched; flush()
durably commits buffered appends. Corruption is a typed error, never a
+4
View File
@@ -1,5 +1,9 @@
"""Canonical orchestration records (spec S9).
Serving task/message/event record semantics live in the Rust
operator_lifecycle process behind Go repositories. These Python records
remain compatibility shapes until the final retirement tranche.
Every record carries a stable ID and schema_version. Records are the only
shape allowed across component boundaries; dicts are for serialization.
"""
+28
View File
@@ -116,6 +116,34 @@ def _rune_home_python() -> str:
raise RuntimeError("escape hatch retired for this row - the Go engine is the authority")
def memory_dir() -> str:
"""Canonical memory data root from the Go paths authority."""
if _go_engine_enabled():
return _go_eval([{"op": "memory_dir"}])[0]
raise RuntimeError("escape hatch retired for this row - the Go engine is the authority")
def store_db() -> str:
"""Canonical memory SQLite path from the Go paths authority."""
if _go_engine_enabled():
return _go_eval([{"op": "store_db"}])[0]
raise RuntimeError("escape hatch retired for this row - the Go engine is the authority")
def scripts_dir() -> str:
"""Canonical Rune scripts directory from the Go paths authority."""
if _go_engine_enabled():
return _go_eval([{"op": "scripts_dir"}])[0]
raise RuntimeError("escape hatch retired for this row - the Go engine is the authority")
def config_dir() -> str:
"""Canonical Rune config directory from the Go paths authority."""
if _go_engine_enabled():
return _go_eval([{"op": "config_dir"}])[0]
raise RuntimeError("escape hatch retired for this row - the Go engine is the authority")
def harness_scripts() -> str:
"""Fail-loud Go delegation (see module header)."""
if _go_engine_enabled():
+105 -14
View File
@@ -20,6 +20,8 @@ import re
import sys
import time
import uuid
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime, timezone
from dataclasses import asdict, dataclass, field
@@ -248,6 +250,16 @@ _SAFETY_RE = re.compile("|".join(_SAFETY_NOISE), re.IGNORECASE)
OPERATOR_CHANNEL = "task"
class NativeOperatorError(RuntimeError):
"""A typed Go-host failure; never a trigger for Python dispatch."""
def __init__(self, code: str, message: str, operation: str = "", cause: BaseException | None = None):
self.code = code
self.operation = operation
self.cause = cause
super().__init__(f"{code}: {operation + ': ' if operation else ''}{message}")
def _bounded(text: object, limit: int = 300) -> str:
"""Collapse text to one line bounded to ``limit`` chars (P1-07 #30)."""
return " ".join(str(text or "").split())[:limit]
@@ -482,17 +494,76 @@ class RuneState:
class RuneOperator:
"""Rune — the persistent operator that runs the TUI."""
"""Go-client by default; the Python body is the final-retirement oracle."""
def __init__(self, storage_path: str | None = None):
def __init__(self, storage_path: str | None = None, core_url: str | None = None, core: str | None = None):
self.storage_path = storage_path or os.path.expanduser("~/.rune/operator.json")
self._core_url = (core_url or os.environ.get("RUNE_OPERATOR_CORE_URL") or "http://127.0.0.1:8376").rstrip("/")
core_mode = core or os.environ.get("RUNE_OPERATOR_CORE")
if core_mode is None:
core_mode = "python" if os.environ.get("RUNE_CORE") == "python" else "go"
if core_mode not in ("go", "python"):
raise ValueError("RUNE_OPERATOR_CORE must be 'go' or 'python'")
self._native = core_mode == "go"
self.state = RuneState()
self._broker = None # ContextBroker, built lazily (#19)
self._continuity = None # TopicContinuity (K15), built lazily
self._last_trace: Optional[Dict[str, Any]] = None
self._events_store = None
self._hot_store = None
self._task_state = None
self._conversation = None
self._migration_report = None
if self._native:
return
self._load()
self._init_conversation_state()
def _native_request(self, method: str, path: str, body: dict | None = None) -> dict:
data = json.dumps(body).encode("utf-8") if body is not None else None
request = urllib.request.Request(
self._core_url + path,
data=data,
method=method,
headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(request, timeout=300) as response:
payload = json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as exc:
try:
detail = json.loads(exc.read().decode("utf-8"))
error = detail.get("error", {})
code = error.get("code", "OPERATOR_ERROR")
message = error.get("message", str(exc))
except (ValueError, UnicodeDecodeError):
code, message = "OPERATOR_ERROR", str(exc)
raise NativeOperatorError(code, message, path, exc) from exc
except (urllib.error.URLError, TimeoutError, OSError) as exc:
raise NativeOperatorError("NATIVE_UNAVAILABLE", str(exc), path, exc) from exc
except (ValueError, UnicodeDecodeError) as exc:
raise NativeOperatorError("PROTOCOL_ERROR", "Go host returned invalid JSON", path, exc) from exc
if not isinstance(payload, dict):
raise NativeOperatorError("PROTOCOL_ERROR", "Go host returned a non-object envelope", path)
return payload
def _native_rune(self, payload: dict) -> Dict[str, Any]:
result = payload.get("rune")
if not isinstance(result, dict):
raise NativeOperatorError("PROTOCOL_ERROR", "Go completion omitted its rune envelope")
return result
def _native_send_task(self, description: str, model: str | None, channel: str, signal_id: str | None) -> Dict[str, Any]:
body: Dict[str, Any] = {
"messages": [{"role": "user", "content": description}],
"channel": channel,
}
if model is not None:
body["model"] = model
if signal_id is not None:
body["signal_id"] = signal_id
return self._native_rune(self._native_request("POST", "/v1/chat/completions", body))
def _routing_profiles_path(self):
"""Measured-routing-profile store path under RUNE_HOME/state."""
try:
@@ -807,6 +878,8 @@ class RuneOperator:
"""Get Rune's current status. T17 17F: task counters derive from
the CANONICAL TaskStateRepository — operator.json's task list is
a legacy cache, never a second authority."""
if self._native:
return self._native_request("GET", "/v1/status")
counters = self._task_counters()
return {
"identity": self.state.identity,
@@ -1017,15 +1090,11 @@ class RuneOperator:
.strftime("%Y-%m-%dT%H:%M:%SZ")})
s.save(doc)
def send_task(self, description: str, model: str | None = None, channel: str = OPERATOR_CHANNEL) -> Dict[str, Any]:
"""Send a task to a model (mediated: one operator-boundary effect
per dispatch through the DEP06 effect registry).
Rune decides what to do and which model to use. Task identity is
canonical (#18): if a task is already active on the channel, a
follow-up AMENDS it (same task_id) instead of minting a new one;
a fresh task records its parent for lineage.
"""
def send_task(self, description: str, model: str | None = None, channel: str = OPERATOR_CHANNEL,
signal_id: str | None = None) -> Dict[str, Any]:
"""Dispatch through Go by default; Python remains an explicit oracle."""
if self._native:
return self._native_send_task(description, model, channel, signal_id)
_mref = _em_begin(_EM_BOUNDARY, "task_dispatch",
(description or "")[:80])
try:
@@ -1712,6 +1781,8 @@ class RuneOperator:
def resolve_pending_approval(self, approved: bool) -> Dict[str, Any]:
"""Resolve and resume the single persisted operator approval."""
if self._native:
return self._native_request("POST", "/v1/approvals/resolve", {"approved": bool(approved)})
task = next(
(item for item in self.state.tasks if item.task_id == self.state.pending_task_id),
None,
@@ -1907,6 +1978,13 @@ class RuneOperator:
def conversation_window(self, channel: str = OPERATOR_CHANNEL, n: int = 8) -> List[Dict[str, Any]]:
"""Recent canonical conversation messages for a channel (#18)."""
if self._native:
query = urllib.parse.urlencode({"channel": channel, "n": n})
payload = self._native_request("GET", "/v1/conversation?" + query)
messages = payload.get("messages")
if not isinstance(messages, list):
raise NativeOperatorError("PROTOCOL_ERROR", "Go conversation omitted messages")
return messages
return [m.to_dict() for m in self._conversation.window(channel, n=n)]
@staticmethod
@@ -2779,6 +2857,12 @@ I have tools to read, write, edit, search, lint, test, format, and commit code.
def list_models(self) -> List[Dict[str, Any]]:
"""List available models."""
if self._native:
payload = self._native_request("GET", "/v1/models")
rows = payload.get("data")
if not isinstance(rows, list):
raise NativeOperatorError("PROTOCOL_ERROR", "Go model listing omitted data")
return [{"name": row.get("id", ""), "size_gb": float(row.get("size_gb", 0.0))} for row in rows]
try:
req = urllib.request.Request("http://localhost:11434/api/tags")
with urllib.request.urlopen(req, timeout=5) as resp:
@@ -2798,6 +2882,12 @@ I have tools to read, write, edit, search, lint, test, format, and commit code.
def list_tasks(self) -> List[Dict[str, Any]]:
"""List all tasks."""
if self._native:
payload = self._native_request("GET", "/v1/tasks")
tasks = payload.get("tasks")
if not isinstance(tasks, list):
raise NativeOperatorError("PROTOCOL_ERROR", "Go task listing omitted tasks")
return tasks
return [t.to_dict() for t in self.state.tasks]
@@ -2887,9 +2977,10 @@ def get_status() -> Dict[str, Any]:
return get_operator().get_status()
def send_task(description: str, model: str | None = None, channel: str = OPERATOR_CHANNEL) -> Dict[str, Any]:
"""Send a task to a model."""
return get_operator().send_task(description, model, channel)
def send_task(description: str, model: str | None = None, channel: str = OPERATOR_CHANNEL,
signal_id: str | None = None) -> Dict[str, Any]:
"""Send a task through the same Go host as the compatibility client."""
return get_operator().send_task(description, model, channel, signal_id)
def get_conversation_window(channel: str = OPERATOR_CHANNEL, n: int = 8) -> List[Dict[str, Any]]:
+26 -2
View File
@@ -18,6 +18,11 @@ import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import memory_env
try:
from . import memory_native
except ImportError:
import memory_native
STORE_DB = memory_env.store_db()
@@ -34,7 +39,12 @@ def _db():
def list_neurons():
"""All active warm neurons from the store (kind='neuron')."""
"""All active warm neurons through the canonical Go/Rust serving path."""
return memory_native.native_call("warm_list", db_path=STORE_DB)
def _list_neurons_python():
"""Retained oracle implementation for parity and later retirement."""
out = []
try:
db = _db()
@@ -64,6 +74,11 @@ def list_neurons():
def neuron_has(term):
"""True if any active neuron's body or triggers mention `term`."""
return memory_native.native_call("warm_has", {"term": term}, db_path=STORE_DB)
def _neuron_has_python(term):
"""Retained oracle implementation for parity and later retirement."""
term = (term or "").lower()
if not term:
return False
@@ -84,7 +99,12 @@ def neuron_has(term):
def neuron_digests():
"""{slug: body} for all active neurons (what gets fired into context)."""
"""{slug: body} for all active neurons through the native serving path."""
return memory_native.native_call("warm_digests", db_path=STORE_DB)
def _neuron_digests_python():
"""Retained oracle implementation for parity and later retirement."""
# Direct SQL SELECT instead of fetching all columns (audit fix)
try:
db = _db()
@@ -101,6 +121,10 @@ def neuron_slugs():
return sorted(n["slug"] for n in list_neurons())
def _neuron_slugs_python():
return sorted(n["slug"] for n in _list_neurons_python())
# ---------------------------------------------------------------------------
# Hook handler — called by the microkernel when an event fires.
# ---------------------------------------------------------------------------
+8
View File
@@ -6,6 +6,7 @@ version = 4
name = "attention_capsule"
version = "0.1.0"
dependencies = [
"memory_selector",
"serde",
"serde_json",
]
@@ -22,6 +23,13 @@ version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
[[package]]
name = "memory_selector"
version = "0.1.0"
dependencies = [
"serde_json",
]
[[package]]
name = "proc-macro2"
version = "1.0.107"
+1
View File
@@ -9,3 +9,4 @@ crate-type = ["rlib", "cdylib", "staticlib"]
[dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
memory_selector = { path = "../memory_selector" }
+481
View File
@@ -0,0 +1,481 @@
//! Deterministic model-visible context selection and admission.
use crate::estimate_tokens;
use memory_selector::{select_core, store_project};
use serde_json::{json, Value};
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ContextError {
pub code: &'static str,
pub operation: &'static str,
pub message: String,
}
impl ContextError {
fn new(code: &'static str, operation: &'static str, message: impl Into<String>) -> Self {
Self { code, operation, message: message.into() }
}
fn protocol(operation: &'static str, message: impl Into<String>) -> Self {
Self::new("PROTOCOL_ERROR", operation, message)
}
}
#[derive(Debug, Clone)]
struct Block {
index: usize,
name: String,
priority: i64,
content: Value,
tokens: i64,
source: String,
evictable: bool,
}
impl Block {
fn wire(&self) -> Value {
json!({
"name": self.name,
"priority": self.priority,
"content": self.content,
"tokens": self.tokens,
"source": self.source,
"evictable": self.evictable,
})
}
fn trace(&self, admitted: bool, reason: String) -> Value {
json!({
"name": self.name,
"priority": self.priority,
"tokens": self.tokens,
"source": self.source,
"evictable": self.evictable,
"admitted": admitted,
"reason": reason,
})
}
}
#[derive(Debug)]
struct Admission {
blocks: Vec<Block>,
trace: Vec<Value>,
dropped: Vec<Value>,
total: i64,
memory_used: i64,
}
#[derive(Debug)]
struct Rendered {
messages: Vec<Value>,
system_preamble: String,
user_content: String,
token_count: i64,
capsule_messages: Vec<Value>,
ordered: Vec<Block>,
}
pub fn default_budget(window: i64) -> i64 {
let reserve = (window / 4).max(2048);
(window - reserve - window / 20).max(1)
}
pub fn default_memory_budget(window: i64) -> i64 {
if window <= 0 { 1200 } else { (window / 8).min(1200) }
}
fn content_tokens(content: &Value, operation: &'static str) -> Result<i64, ContextError> {
match content {
Value::String(text) => Ok(estimate_tokens(text)),
Value::Array(messages) => messages.iter().try_fold(0, |total, message| {
let object = message.as_object().ok_or_else(|| {
ContextError::protocol(operation, "history entries must be objects")
})?;
let content = object.get("content").and_then(Value::as_str).ok_or_else(|| {
ContextError::protocol(operation, "history entries require string content")
})?;
Ok(total + estimate_tokens(content))
}),
_ => Err(ContextError::protocol(operation, "block content must be a string or message list")),
}
}
fn parse_blocks(input: &[Value], measured: bool, operation: &'static str) -> Result<Vec<Block>, ContextError> {
input.iter().enumerate().map(|(index, entry)| {
let object = entry.as_object().ok_or_else(|| ContextError::protocol(operation, "entries must be objects"))?;
let content = object.get("content").cloned().unwrap_or(Value::Null);
let measured_tokens = content_tokens(&content, operation)?;
let tokens = if measured {
object.get("tokens").and_then(Value::as_i64).unwrap_or(measured_tokens)
} else {
measured_tokens
};
if tokens < 0 {
return Err(ContextError::protocol(operation, "block tokens must be non-negative"));
}
Ok(Block {
index,
name: object.get("name").and_then(Value::as_str).unwrap_or("").to_string(),
priority: object.get("priority").and_then(Value::as_i64).unwrap_or(0),
content,
tokens,
source: object.get("source").and_then(Value::as_str).unwrap_or("").to_string(),
evictable: object.get("evictable").and_then(Value::as_bool).unwrap_or(true),
})
}).collect()
}
fn admit(mut candidates: Vec<Block>, budget: i64, memory_budget: i64, enforce_collective: bool, operation: &'static str) -> Result<Admission, ContextError> {
candidates.sort_by_key(|block| (block.evictable, -block.priority, block.index));
let mut admitted = Vec::new();
let mut trace = Vec::new();
let mut dropped = Vec::new();
let mut used = 0;
let mut memory_used = 0;
for block in candidates {
if !block.evictable {
if block.tokens > budget {
return Err(ContextError::new("CONTEXT_OVERFLOW", operation, format!(
"mandatory block '{}' needs {} tokens but the input budget is {} — refusing to silently truncate",
block.name, block.tokens, budget,
)));
}
if enforce_collective && used + block.tokens > budget {
return Err(ContextError::new("CONTEXT_OVERFLOW", operation, format!(
"non-evictable blocks total {} tokens > budget {} — refusing to silently truncate",
used + block.tokens, budget,
)));
}
used += block.tokens;
trace.push(block.trace(true, String::new()));
admitted.push(block);
continue;
}
let memory = matches!(block.name.as_str(), "memory" | "persona_memories" | "consolidated");
let mut reason = String::new();
if memory && used + block.tokens <= budget && memory_used + block.tokens > memory_budget {
reason = format!(
"memory budget: {}+{}>{} (memory sub-budget exhausted; lowest-priority memory dropped)",
memory_used, block.tokens, memory_budget,
);
} else if used + block.tokens > budget {
reason = format!("budget: {}+{}>{} (lowest-priority evictable dropped)", used, block.tokens, budget);
}
if reason.is_empty() {
used += block.tokens;
if memory { memory_used += block.tokens; }
trace.push(block.trace(true, String::new()));
admitted.push(block);
} else {
let record = block.trace(false, reason);
dropped.push(record.clone());
trace.push(record);
}
}
Ok(Admission { blocks: admitted, trace, dropped, total: used, memory_used })
}
fn render(mut admitted: Vec<Block>) -> Rendered {
admitted.sort_by_key(|block| (-block.priority, block.index));
let token_count = admitted.iter().map(|block| block.tokens).sum();
let mut system = Vec::new();
let mut history = Vec::new();
let mut user = None;
for block in &admitted {
match &block.content {
Value::String(text) if block.name == "user" => user = Some(text.clone()),
Value::String(text) if !text.is_empty() => system.push(text.clone()),
Value::Array(messages) => history.extend(messages.iter().cloned()),
_ => {}
}
}
let system_preamble = system.join("\n");
let mut capsule_messages = Vec::new();
if !system_preamble.is_empty() {
capsule_messages.push(json!({"role": "system", "content": system_preamble}));
}
capsule_messages.extend(history);
let mut messages = capsule_messages.clone();
let user_content = user.unwrap_or_default();
if !user_content.is_empty() {
messages.push(json!({"role": "user", "content": user_content}));
}
Rendered { messages, system_preamble, user_content, token_count, capsule_messages, ordered: admitted }
}
fn budget_arg(input: &Value, window: i64) -> i64 {
input.get("budget").and_then(Value::as_i64).unwrap_or_else(|| default_budget(window))
}
fn admission_value(result: Admission, budget: i64, memory_budget: i64, operation: &'static str) -> Value {
let admitted_indices: Vec<i64> = result.blocks.iter().map(|block| block.index as i64).collect();
let admitted_names: Vec<String> = result.blocks.iter().map(|block| block.name.clone()).collect();
let admitted_blocks: Vec<Value> = result.blocks.iter().map(Block::wire).collect();
let rendered = render(result.blocks);
json!({
"status": "ADMITTED",
"budget": budget,
"memory_budget": memory_budget,
"memory_used": result.memory_used,
"admitted_total": result.total,
"admitted": admitted_names,
"admitted_indices": admitted_indices,
"admitted_blocks": admitted_blocks,
"blocks": result.trace,
"dropped": result.dropped,
"messages": rendered.messages,
"system_preamble": rendered.system_preamble,
"user_content": rendered.user_content,
"token_count": rendered.token_count,
"trace": {"context": "rust-attention-capsule", "operation": operation},
})
}
pub fn admit_context(input: &Value) -> Result<Value, ContextError> {
let entries = input.get("entries").and_then(Value::as_array).ok_or_else(|| {
ContextError::protocol("admit", "entries must be a list")
})?;
let blocks = parse_blocks(entries, true, "admit")?;
let budget = input.get("budget").and_then(Value::as_i64).unwrap_or(0);
let memory_budget = input.get("memory_budget").and_then(Value::as_i64).unwrap_or_else(|| default_memory_budget(0));
let enforce_collective = input.get("enforce_collective").and_then(Value::as_bool).unwrap_or(false);
let result = admit(blocks, budget, memory_budget, enforce_collective, "admit")?;
Ok(admission_value(result, budget, memory_budget, "admit"))
}
pub fn render_context(input: &Value) -> Result<Value, ContextError> {
let entries = input.get("blocks").and_then(Value::as_array).ok_or_else(|| {
ContextError::protocol("render", "blocks must be a list")
})?;
let blocks = parse_blocks(entries, true, "render")?;
let rendered = render(blocks);
let order: Vec<i64> = rendered.ordered.iter().map(|block| block.index as i64).collect();
let names: Vec<String> = rendered.ordered.iter().map(|block| block.name.clone()).collect();
Ok(json!({
"messages": rendered.capsule_messages,
"system_preamble": rendered.system_preamble,
"user_content": rendered.user_content,
"token_count": rendered.token_count,
"order_indices": order,
"blocks": names,
"trace": {"context": "rust-attention-capsule", "operation": "render"},
}))
}
pub fn assemble_context(input: &Value) -> Result<Value, ContextError> {
let query = input.get("query").and_then(Value::as_str).unwrap_or("");
let user = input.get("user").and_then(Value::as_str).unwrap_or(query);
if query.trim().is_empty() || user.trim().is_empty() {
return Err(ContextError::new("CONTEXT_UNKNOWN", "assemble", "active query/user evidence is blank; relevance is UNKNOWN"));
}
let window = input.get("window").and_then(Value::as_i64).unwrap_or(8192);
let budget = budget_arg(input, window);
let memory_budget = input.get("memory_budget").and_then(Value::as_i64)
.unwrap_or_else(|| default_memory_budget(window)).min(budget);
let disabled = input.get("memory_disabled").and_then(Value::as_bool).unwrap_or(false);
let empty = Vec::new();
let rows = if disabled { &empty } else { input.get("memory_rows").and_then(Value::as_array).unwrap_or(&empty) };
let projected = store_project(rows);
let candidates = projected.as_array().ok_or_else(|| ContextError::protocol("assemble", "memory projection was not a list"))?;
let item_budget = input.get("memory_item_budget").and_then(Value::as_i64).unwrap_or(4);
let min_score = input.get("min_score").and_then(Value::as_f64).unwrap_or(0.42);
let selection = select_core(candidates, item_budget, min_score, &[]);
let selected = selection["blocks"].as_array().cloned().unwrap_or_default();
let mut entries = Vec::new();
if let Some(system) = input.get("system").and_then(Value::as_str).filter(|text| !text.is_empty()) {
entries.push(json!({"name": "system", "priority": 100, "content": system, "source": "runecore.system", "evictable": false}));
}
if !selected.is_empty() {
let lines: Vec<String> = selected.iter().filter_map(|block| block["content"].as_str().map(str::to_string)).collect();
if !lines.is_empty() {
entries.push(json!({
"name": "memory", "priority": 70,
"content": format!("## Context\n{}", lines.join("\n")),
"source": "memory_selector.select_for_task", "evictable": true,
}));
}
}
let history = input.get("history").cloned().unwrap_or_else(|| json!([]));
if history.as_array().is_some_and(|messages| !messages.is_empty()) {
entries.push(json!({"name": "history", "priority": 30, "content": history, "source": "runecore.channel.history", "evictable": true}));
}
entries.push(json!({"name": "user", "priority": 100, "content": user, "source": "runecore.user", "evictable": false}));
let blocks = parse_blocks(&entries, false, "assemble")?;
let result = admit(blocks, budget, memory_budget, true, "assemble")?;
let memory_admitted = result.blocks.iter().filter(|block| block.name == "memory").count();
let selected_count = selection["selected_count"].as_i64().unwrap_or(0);
let memory_status = if selected_count > 0 && memory_admitted > 0 { "ADMITTED" } else { "ABSTAINED" };
let memory_reason = if disabled {
"disabled"
} else if selected_count == 0 {
"no relevant memory"
} else if memory_admitted == 0 {
"budget"
} else {
"selected"
};
let mut value = admission_value(result, budget, memory_budget, "assemble");
value["memory_status"] = json!(memory_status);
value["memory_abstained"] = json!(memory_status == "ABSTAINED");
value["memory_reason"] = json!(memory_reason);
value["memory_candidates"] = selection["candidates_evaluated"].clone();
value["memory_selected"] = selection["selected_count"].clone();
value["memory_admitted"] = json!(memory_admitted);
value["memory_tokens"] = value["memory_used"].clone();
value["trace"]["selection"] = json!("memory_selector.select_core");
value["trace"]["admission"] = json!("attention_capsule.admit");
Ok(value)
}
#[cfg(test)]
mod tests {
use super::assemble_context;
use serde_json::{json, Value};
fn request(memory_rows: Vec<Value>) -> Value {
json!({
"query": "native context question",
"system": "identity",
"history": [{"role": "assistant", "content": "prior answer"}],
"user": "native context question",
"memory_rows": memory_rows,
"window": 8192,
"memory_budget": 64,
"memory_item_budget": 4,
"min_score": 0.42
})
}
#[test]
fn selects_memory_and_renders_history_then_user() {
let out = assemble_context(&request(vec![json!({
"id": 7,
"text": "native remembered fact",
"relevance": 0.9
})]))
.unwrap();
assert_eq!(out["status"], "ADMITTED");
assert_eq!(out["memory_status"], "ADMITTED");
assert_eq!(out["memory_abstained"], false);
assert_eq!(out["memory_selected"], 1);
assert_eq!(out["messages"][0]["role"], "system");
assert!(out["messages"][0]["content"]
.as_str()
.unwrap()
.contains("native remembered fact"));
assert_eq!(out["messages"][1]["content"], "prior answer");
assert_eq!(out["messages"][2], json!({
"role": "user",
"content": "native context question"
}));
assert!(out["admitted_total"].as_i64().unwrap() <= out["budget"].as_i64().unwrap());
assert_eq!(out["trace"]["context"], "rust-attention-capsule");
}
#[test]
fn no_relevant_memory_is_successful_abstention() {
let out = assemble_context(&request(Vec::new())).unwrap();
assert_eq!(out["status"], "ADMITTED");
assert_eq!(out["memory_status"], "ABSTAINED");
assert_eq!(out["memory_abstained"], true);
assert_eq!(out["memory_selected"], 0);
assert!(!out["messages"][0]["content"]
.as_str()
.unwrap()
.contains("## Context"));
assert_eq!(out["messages"].as_array().unwrap().len(), 3);
}
#[test]
fn blank_query_is_unknown_not_empty_context() {
let mut input = request(Vec::new());
input["query"] = json!(" ");
input["user"] = json!(" ");
let error = assemble_context(&input).unwrap_err();
assert_eq!(error.code, "CONTEXT_UNKNOWN");
assert_eq!(error.operation, "assemble");
}
#[test]
fn collective_mandatory_overflow_is_typed() {
let mut input = request(Vec::new());
input["system"] = json!("s".repeat(80));
input["user"] = json!("u".repeat(80));
input["query"] = json!("u".repeat(80));
input["budget"] = json!(30);
let error = assemble_context(&input).unwrap_err();
assert_eq!(error.code, "CONTEXT_OVERFLOW");
assert!(error.message.contains("non-evictable"));
}
#[test]
fn generic_admission_keeps_admission_order_not_render_order() {
let out = super::admit_context(&json!({
"entries": [
{"name": "history", "priority": 100, "content": "h", "tokens": 1, "evictable": true},
{"name": "system", "priority": 1, "content": "s", "tokens": 1, "evictable": false}
],
"budget": 10,
"memory_budget": 10
}))
.unwrap();
assert_eq!(out["admitted"], json!(["system", "history"]));
assert_eq!(out["admitted_indices"], json!([1, 0]));
}
#[test]
fn generic_admission_defers_collective_overflow_to_assembly() {
let out = super::admit_context(&json!({
"entries": [
{"name": "system", "priority": 100, "content": "s", "tokens": 20, "evictable": false},
{"name": "user", "priority": 100, "content": "u", "tokens": 20, "evictable": false}
],
"budget": 30,
"memory_budget": 30
}))
.unwrap();
assert_eq!(out["admitted_total"], 40);
assert_eq!(out["admitted"], json!(["system", "user"]));
}
#[test]
fn generic_render_keeps_active_user_out_of_capsule_messages() {
let out = super::render_context(&json!({
"blocks": [
{"name": "history", "priority": 30, "content": [{"role": "user", "content": "earlier"}], "tokens": 1},
{"name": "user", "priority": 100, "content": "active", "tokens": 1, "evictable": false}
]
}))
.unwrap();
assert_eq!(out["messages"], json!([{"role": "user", "content": "earlier"}]));
assert_eq!(out["user_content"], "active");
}
#[test]
fn memory_subbudget_is_enforced_after_selection() {
let out = assemble_context(&request(vec![
json!({"id": 1, "text": "short fact", "relevance": 0.9}),
json!({"id": 2, "text": "x".repeat(400), "relevance": 0.8}),
]))
.unwrap();
assert_eq!(out["memory_selected"], 2);
assert!(out["memory_used"].as_i64().unwrap() <= 64);
assert!(out["dropped"].as_array().unwrap().iter().any(|row| {
row["name"] == "memory" && row["reason"].as_str().unwrap_or("").contains("memory budget")
}));
}
}
+28
View File
@@ -32,6 +32,8 @@ use serde::{Deserialize, Serialize};
use serde_json::{json, Value};
use std::collections::HashSet;
pub mod context;
const REQUIREMENT_PATTERNS: &[&str] = &[
"need to", "want to", "please", "fix", "build", "create", "update", "change",
];
@@ -453,6 +455,19 @@ fn envelope(r: Result<Value, String>) -> *mut c_char {
.unwrap_or(std::ptr::null_mut())
}
fn context_envelope(r: Result<Value, context::ContextError>) -> *mut c_char {
let value = match r {
Ok(value) => json!({"ok": true, "value": value}),
Err(error) => json!({
"ok": false,
"error": {"code": error.code, "message": error.message},
}),
};
CString::new(serde_json::to_string(&value).unwrap_or_default())
.map(|c| c.into_raw())
.unwrap_or(std::ptr::null_mut())
}
fn parse_capsule(v: &Value) -> Result<AttentionCapsule, String> {
serde_json::from_value(v.clone()).map_err(|e| e.to_string())
}
@@ -518,6 +533,19 @@ pub extern "C" fn ac_create_capsule(arg: *const c_char) -> *mut c_char {
envelope(r)
}
/// ac_admit({entries, budget, memory_budget}) -> governed admission receipt
#[no_mangle]
pub extern "C" fn ac_admit(arg: *const c_char) -> *mut c_char {
let result = read_arg(arg)
.map_err(|message| context::ContextError {
code: "PROTOCOL_ERROR",
operation: "admit",
message,
})
.and_then(|value| context::admit_context(&value));
context_envelope(result)
}
#[path = "branch_battery.rs"]
#[cfg(test)]
mod branch_battery;
+74
View File
@@ -0,0 +1,74 @@
//! Typed NDJSON process boundary for model-visible context decisions.
use attention_capsule::context::{
admit_context, assemble_context, default_budget, default_memory_budget, render_context, ContextError,
};
use serde_json::{json, Value};
use std::io::Read;
fn failure(error: ContextError) -> Value {
json!({
"ok": false,
"error": {"code": error.code, "message": error.message},
"trace": {"context": "rust-attention-capsule", "operation": error.operation},
})
}
fn evaluate(request: &Value) -> Result<Value, ContextError> {
let operation = request.get("op").and_then(Value::as_str).unwrap_or("");
let argument = request.get("arg").cloned().unwrap_or_else(|| json!({}));
match operation {
"assemble" => assemble_context(&argument),
"admit" => admit_context(&argument),
"render" => render_context(&argument),
"budget" => Ok(json!(default_budget(argument.get("window").and_then(Value::as_i64).unwrap_or(0)))),
"memory_budget" => Ok(json!(default_memory_budget(argument.get("window").and_then(Value::as_i64).unwrap_or(0)))),
_ => Err(ContextError {
code: "PROTOCOL_ERROR",
operation: "eval",
message: format!("unknown context operation: {operation}"),
}),
}
}
fn main() {
if std::env::args().nth(1).as_deref() != Some("eval") {
eprintln!("usage: attention_capsule eval < request.ndjson");
std::process::exit(2);
}
let mut input = String::new();
if let Err(error) = std::io::stdin().read_to_string(&mut input) {
println!("{}", failure(ContextError {
code: "PROTOCOL_ERROR",
operation: "eval",
message: error.to_string(),
}));
std::process::exit(2);
}
let request: Value = match serde_json::from_str(&input) {
Ok(value) => value,
Err(error) => {
println!("{}", failure(ContextError {
code: "PROTOCOL_ERROR",
operation: "eval",
message: error.to_string(),
}));
std::process::exit(2);
}
};
let operation = request.get("op").and_then(Value::as_str).unwrap_or("eval").to_string();
match evaluate(&request) {
Ok(value) => {
let trace = value.get("trace").cloned().unwrap_or_else(|| json!({
"context": "rust-attention-capsule",
"operation": operation,
}));
println!("{}", json!({"ok": true, "value": value, "trace": trace}));
}
Err(error) => {
let protocol = error.code == "PROTOCOL_ERROR";
println!("{}", failure(error));
std::process::exit(if protocol { 2 } else { 1 });
}
}
}
+151 -147
View File
@@ -54,155 +54,159 @@ fn envelope(r: Result<pyjson::Json, String>) -> *mut c_char {
.unwrap_or(std::ptr::null_mut())
}
pub fn evaluate(a: pyjson::Json) -> Result<pyjson::Json, String> {
let op = s(get(&a, "op"));
let p = get(&a, "arg").unwrap_or(&pyjson::Json::Null);
match op {
"map_reduce" => {
let e = match get(p, "entries") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
Ok(pyjson::Json::Arr(retrieval::map_reduce(
&e,
num(get(p, "chunk_size"), 10),
)))
}
"position_aware" => {
let e = match get(p, "entries") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
Ok(pyjson::Json::Arr(retrieval::position_aware(&e)))
}
"fts_query_conjunctive" => Ok(pyjson::Json::Str(retrieval::fts_query_conjunctive(s(get(
p, "query",
))))),
"chunk_text" => Ok(pyjson::Json::Arr(
retrieval::chunk_text(
s(get(p, "text")),
num(get(p, "size"), 1500),
num(get(p, "overlap"), 150),
)
.into_iter()
.map(pyjson::Json::Str)
.collect(),
)),
"norm_key" => {
let text = match get(p, "text") {
Some(pyjson::Json::Str(x)) => Some(x.as_str()),
_ => None,
};
Ok(pyjson::Json::Str(retrieval::norm_key(text)))
}
"score_of" => {
let item = get(p, "item").cloned().unwrap_or(pyjson::Json::Null);
Ok(pyjson::Json::Num(retrieval::score_of(&item)))
}
"lex_overlap_score" => Ok(pyjson::Json::Num(retrieval::lex_overlap_score(
s(get(p, "text")),
s(get(p, "query")),
))),
"dense_candidates" => Ok(retrieval::dense_candidates(p)),
"dedupe_decision" => Ok(retrieval::dedupe_decision(p)),
"fuse_results" => {
let e = match get(p, "entries") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let r = match get(p, "research") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
Ok(pyjson::Json::Arr(retrieval::fuse_results(&e, &r)))
}
"merge_association_rows" => {
let base = match get(p, "base") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let boosts = match get(p, "boosts") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let rows = match get(p, "rows") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
Ok(pyjson::Json::Arr(retrieval::merge_association_rows(
&base,
&boosts,
&rows,
num(get(p, "budget"), 10),
)))
}
"project_chunk_rows" => {
let ranks = match get(p, "ranks") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let rows = match get(p, "rows") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
Ok(pyjson::Json::Arr(retrieval::project_chunk_rows(
&ranks,
&rows,
num(get(p, "budget"), 6),
)))
}
"rank_fusion" => {
let d = match get(p, "dense") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let b = match get(p, "bm25") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let k = match get(p, "rrf_k") {
Some(pyjson::Json::Num(n)) => *n,
_ => 60.0,
};
Ok(retrieval::rank_fusion(&d, &b, k))
}
"score_recall_rows" => {
let fusion = get(p, "fusion").cloned().unwrap_or(pyjson::Json::Null);
let rows = match get(p, "rows") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let d = match get(p, "dense") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let b = match get(p, "bm25") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let min = match get(p, "min_score") {
Some(pyjson::Json::Num(n)) => *n,
_ => 0.42,
};
let half = match get(p, "half_life") {
Some(pyjson::Json::Num(n)) => *n,
_ => 30.0,
};
let now = s(get(p, "now"));
Ok(pyjson::Json::Arr(retrieval::score_recall_rows(
&fusion,
&rows,
&d,
&b,
num(get(p, "budget"), 10),
min,
now,
half,
)))
}
_ => Err(format!("unknown op: {:?}", op)),
}
}
#[no_mangle]
pub extern "C" fn mr_call(arg: *const c_char) -> *mut c_char {
let r = read_arg(arg).and_then(|a| {
let op = s(get(&a, "op"));
let p = get(&a, "arg").unwrap_or(&pyjson::Json::Null);
match op {
"map_reduce" => {
let e = match get(p, "entries") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
Ok(pyjson::Json::Arr(retrieval::map_reduce(
&e,
num(get(p, "chunk_size"), 10),
)))
}
"position_aware" => {
let e = match get(p, "entries") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
Ok(pyjson::Json::Arr(retrieval::position_aware(&e)))
}
"fts_query_conjunctive" => Ok(pyjson::Json::Str(retrieval::fts_query_conjunctive(s(
get(p, "query"),
)))),
"chunk_text" => Ok(pyjson::Json::Arr(
retrieval::chunk_text(
s(get(p, "text")),
num(get(p, "size"), 1500),
num(get(p, "overlap"), 150),
)
.into_iter()
.map(pyjson::Json::Str)
.collect(),
)),
"norm_key" => {
let text = match get(p, "text") {
Some(pyjson::Json::Str(x)) => Some(x.as_str()),
_ => None,
};
Ok(pyjson::Json::Str(retrieval::norm_key(text)))
}
"score_of" => {
let item = get(p, "item").cloned().unwrap_or(pyjson::Json::Null);
Ok(pyjson::Json::Num(retrieval::score_of(&item)))
}
"lex_overlap_score" => Ok(pyjson::Json::Num(retrieval::lex_overlap_score(
s(get(p, "text")),
s(get(p, "query")),
))),
"fuse_results" => {
let e = match get(p, "entries") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let r = match get(p, "research") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
Ok(pyjson::Json::Arr(retrieval::fuse_results(&e, &r)))
}
"merge_association_rows" => {
let base = match get(p, "base") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let boosts = match get(p, "boosts") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let rows = match get(p, "rows") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
Ok(pyjson::Json::Arr(retrieval::merge_association_rows(
&base,
&boosts,
&rows,
num(get(p, "budget"), 10),
)))
}
"project_chunk_rows" => {
let ranks = match get(p, "ranks") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let rows = match get(p, "rows") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
Ok(pyjson::Json::Arr(retrieval::project_chunk_rows(
&ranks,
&rows,
num(get(p, "budget"), 6),
)))
}
"rank_fusion" => {
let d = match get(p, "dense") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let b = match get(p, "bm25") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let k = match get(p, "rrf_k") {
Some(pyjson::Json::Num(n)) => *n,
_ => 60.0,
};
Ok(retrieval::rank_fusion(&d, &b, k))
}
"score_recall_rows" => {
let fusion = get(p, "fusion").cloned().unwrap_or(pyjson::Json::Null);
let rows = match get(p, "rows") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let d = match get(p, "dense") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let b = match get(p, "bm25") {
Some(pyjson::Json::Arr(v)) => v.clone(),
_ => vec![],
};
let min = match get(p, "min_score") {
Some(pyjson::Json::Num(n)) => *n,
_ => 0.42,
};
let half = match get(p, "half_life") {
Some(pyjson::Json::Num(n)) => *n,
_ => 30.0,
};
let now = s(get(p, "now"));
Ok(pyjson::Json::Arr(retrieval::score_recall_rows(
&fusion,
&rows,
&d,
&b,
num(get(p, "budget"), 10),
min,
now,
half,
)))
}
_ => Err(format!("unknown op: {:?}", op)),
}
});
let r = read_arg(arg).and_then(evaluate);
envelope(r)
}
+27 -25
View File
@@ -1,30 +1,32 @@
//! JSON stdin CLI for debugging the memory_retrieval kernel.
use memory_retrieval::{pyjson, retrieval};
//! Typed JSON/NDJSON process boundary for the memory_retrieval kernel.
use memory_retrieval::{evaluate, pyjson};
use std::io::Read;
fn get<'a>(v: &'a pyjson::Json, k: &str) -> Option<&'a pyjson::Json> {
match v {
pyjson::Json::Obj(p) => p.iter().find(|(x, _)| x == k).map(|(_, v)| v),
_ => None,
fn envelope(result: Result<pyjson::Json, String>) -> pyjson::Json {
match result {
Ok(value) => pyjson::Json::Obj(vec![
("ok".into(), pyjson::Json::Bool(true)),
("value".into(), value),
]),
Err(message) => pyjson::Json::Obj(vec![
("ok".into(), pyjson::Json::Bool(false)),
(
"error".into(),
pyjson::Json::Obj(vec![
("code".into(), pyjson::Json::Str("STORE_ERROR".into())),
("message".into(), pyjson::Json::Str(message)),
]),
),
]),
}
}
fn main() {
let mut x = String::new();
std::io::stdin().read_to_string(&mut x).unwrap();
let a = pyjson::parse(&x).unwrap();
let op = match get(&a, "op") {
Some(pyjson::Json::Str(s)) => s.as_str(),
_ => "",
};
let arg = get(&a, "arg").unwrap_or(&pyjson::Json::Null);
let v = match op {
"fts_query_conjunctive" => {
let q = match get(arg, "query") {
Some(pyjson::Json::Str(s)) => s,
_ => "",
};
pyjson::Json::Str(retrieval::fts_query_conjunctive(q))
}
_ => pyjson::Json::Str(format!("unsupported CLI op: {}", op)),
};
println!("{}", pyjson::dumps(&v));
let mut input = String::new();
if let Err(error) = std::io::stdin().read_to_string(&mut input) {
println!("{}", pyjson::dumps(&envelope(Err(error.to_string()))));
return;
}
let result = pyjson::parse(&input).and_then(evaluate);
println!("{}", pyjson::dumps(&envelope(result)));
}
+336
View File
@@ -284,6 +284,342 @@ pub fn rank_fusion(dense: &[Json], bm25: &[Json], rrf_k: f64) -> Json {
/// Pure transform extracted from fused_recall: project facts/research,
/// de-duplicate on norm_key, min-max normalize within each kind, stable-sort.
fn json_floats(value: Option<&Json>) -> Vec<f64> {
match value {
Some(Json::Arr(items)) => items
.iter()
.filter_map(|item| match item {
Json::Num(number) => Some(*number),
_ => None,
})
.collect(),
_ => vec![],
}
}
fn cosine_similarity(left: &[f64], right: &[f64]) -> Option<f64> {
if left.is_empty() || left.len() != right.len() {
return None;
}
let mut dot = 0.0;
let mut left_norm = 0.0;
let mut right_norm = 0.0;
for (a, b) in left.iter().zip(right) {
dot += a * b;
left_norm += a * a;
right_norm += b * b;
}
if left_norm == 0.0 || right_norm == 0.0 {
return None;
}
Some(dot / (left_norm.sqrt() * right_norm.sqrt()))
}
pub fn dense_candidates(arg: &Json) -> Json {
let query = text(get(arg, "query"), "").to_lowercase();
let query_embedding = json_floats(get(arg, "query_embedding"));
let rows = match get(arg, "rows") {
Some(Json::Arr(rows)) => rows.as_slice(),
_ => &[],
};
let probe_cap = number(get(arg, "probe_cap"), 16.0).max(0.0) as usize;
let mut candidates: Vec<(Json, String, f64)> = rows
.iter()
.filter_map(|row| {
let id = get(row, "id").cloned()?;
let text = text(get(row, "text"), "");
let embedding = json_floats(get(row, "embedding"));
let cosine = cosine_similarity(&query_embedding, &embedding)?;
Some((id, text, cosine))
})
.collect();
candidates.sort_by(|a, b| b.2.partial_cmp(&a.2).unwrap_or(Ordering::Equal));
candidates.truncate(probe_cap);
if matches!(get(arg, "gate"), Some(Json::Bool(false))) {
return Json::Arr(
candidates
.into_iter()
.map(|(id, _, cosine)| Json::Arr(vec![id, Json::Num(cosine)]))
.collect(),
);
}
let query_content: Vec<String> = query
.split_whitespace()
.filter(|word| word.chars().count() > 3 && !STOPWORDS.contains(word))
.map(|word| word.to_string())
.collect();
let corpus: Vec<String> = match get(arg, "corpus") {
Some(Json::Arr(rows)) => rows
.iter()
.map(|row| text(Some(row), "").to_lowercase())
.collect(),
_ => vec![],
};
let total = corpus.len().max(1) as f64;
let filtered = candidates
.into_iter()
.filter(|(_, entry_text, cosine)| {
let entry_text = entry_text.to_lowercase();
let distinctive = query_content.iter().any(|word| {
if !entry_text.contains(word) {
return false;
}
let df = corpus.iter().filter(|text| text.contains(word)).count() as f64;
df / total < 0.05
});
distinctive || query_content.is_empty() || *cosine >= 0.80
})
.map(|(id, _, cosine)| Json::Arr(vec![id, Json::Num(cosine)]))
.collect();
Json::Arr(filtered)
}
fn dedupe_norm(text: &str) -> String {
let stripped = text
.trim()
.trim_start_matches(|c: char| c.is_whitespace() || matches!(c, '-' | '*' | '#' | '>'));
let lowered = stripped.to_lowercase();
let cleaned: String = lowered
.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == ' ' {
c
} else {
' '
}
})
.collect();
cleaned.split_whitespace().collect::<Vec<_>>().join(" ")
}
fn dedupe_similarity(a: &str, b: &str) -> f64 {
use std::collections::HashSet;
let left: HashSet<String> = dedupe_norm(a)
.split_whitespace()
.map(|s| s.to_owned())
.collect();
let right: HashSet<String> = dedupe_norm(b)
.split_whitespace()
.map(|s| s.to_owned())
.collect();
if left.is_empty() || right.is_empty() {
return 0.0;
}
let intersection = left.iter().filter(|word| right.contains(*word)).count();
let union = left.len() + right.len() - intersection;
if union == 0 {
0.0
} else {
intersection as f64 / union as f64
}
}
fn dedupe_outcome(action: &str, prior_id: Json, reason: String, score: f64) -> Json {
Json::Obj(vec![
("action".into(), Json::Str(action.into())),
("prior_id".into(), prior_id),
("reason".into(), Json::Str(reason)),
("score".into(), Json::Num(score)),
])
}
pub fn dedupe_decision(arg: &Json) -> Json {
let new_text = text(get(arg, "text"), "");
let trimmed = new_text.trim();
if trimmed.is_empty() {
return dedupe_outcome("rejected", Json::Null, "empty text".into(), 0.0);
}
let candidates: &[Json] = match get(arg, "candidates") {
Some(Json::Arr(rows)) => rows,
_ => &[],
};
let confidence = number(get(arg, "confidence"), 0.0);
let new_embedding = json_floats(get(arg, "new_embedding"));
let mut best: Option<(&Json, f64)> = None;
for candidate in candidates {
let old_text = text(get(candidate, "text"), "");
let old_embedding = json_floats(get(candidate, "embedding"));
let similarity = if old_text.trim().to_lowercase() == trimmed.to_lowercase() {
1.0
} else if let Some(cosine) = cosine_similarity(&new_embedding, &old_embedding) {
cosine
} else {
dedupe_similarity(trimmed, &old_text)
};
if best
.as_ref()
.map(|(_, score)| similarity > *score)
.unwrap_or(true)
{
best = Some((candidate, similarity));
}
}
let Some((existing, similarity)) = best else {
return dedupe_outcome(
"created",
Json::Null,
"first record for this data point".into(),
0.0,
);
};
if similarity < 0.72 {
return dedupe_outcome(
"created",
Json::Null,
"first record for this data point".into(),
similarity,
);
}
let prior_id = get(existing, "id").cloned().unwrap_or(Json::Null);
let old_text = text(get(existing, "text"), "");
let old_confidence = number(get(existing, "confidence"), 0.0);
let new_longer = trimmed.chars().count() as isize - old_text.chars().count() as isize >= 20;
let new_stronger = confidence > old_confidence + 0.15;
let id_display = match &prior_id {
Json::Num(id) => format!("{}", id),
_ => "?".into(),
};
if !new_longer && !new_stronger {
return dedupe_outcome(
"duplicate",
prior_id,
format!("same data point (sim {similarity:.2}); no new info — touched existing #{id_display}"),
similarity,
);
}
let delta = trimmed.chars().count() as isize - old_text.chars().count() as isize;
dedupe_outcome(
"superseded",
prior_id,
format!(
"new info replaces #{id_display} (sim {similarity:.2}, {delta:+} chars, conf {confidence} vs {old_confidence})"
),
similarity,
)
}
#[cfg(test)]
mod canonical_policy_tests {
use super::*;
fn candidate(id: f64, text: &str, confidence: f64) -> Json {
Json::Obj(vec![
("id".into(), Json::Num(id)),
("text".into(), Json::Str(text.into())),
("confidence".into(), Json::Num(confidence)),
])
}
#[test]
fn dense_candidates_compute_cosine_and_apply_gate() {
let result = dense_candidates(&Json::Obj(vec![
("query".into(), Json::Str("phone".into())),
(
"query_embedding".into(),
Json::Arr(vec![Json::Num(1.0), Json::Num(0.0)]),
),
(
"rows".into(),
Json::Arr(vec![
Json::Obj(vec![
("id".into(), Json::Num(1.0)),
("text".into(), Json::Str("desk phone".into())),
(
"embedding".into(),
Json::Arr(vec![Json::Num(1.0), Json::Num(0.0)]),
),
]),
Json::Obj(vec![
("id".into(), Json::Num(2.0)),
("text".into(), Json::Str("unrelated".into())),
(
"embedding".into(),
Json::Arr(vec![Json::Num(0.0), Json::Num(1.0)]),
),
]),
]),
),
(
"corpus".into(),
Json::Arr(vec![Json::Str("desk phone".into())]),
),
("probe_cap".into(), Json::Num(8.0)),
("gate".into(), Json::Bool(true)),
]));
let Json::Arr(rows) = result else {
panic!("dense result must be an array")
};
assert_eq!(rows.len(), 1);
assert_eq!(rows[0], Json::Arr(vec![Json::Num(1.0), Json::Num(1.0)]));
}
#[test]
fn dedupe_uses_embeddings_when_available() {
let result = dedupe_decision(&Json::Obj(vec![
("text".into(), Json::Str("entirely new vocabulary".into())),
(
"new_embedding".into(),
Json::Arr(vec![Json::Num(1.0), Json::Num(0.0)]),
),
(
"candidates".into(),
Json::Arr(vec![Json::Obj(vec![
("id".into(), Json::Num(9.0)),
("text".into(), Json::Str("unrelated lexical tokens".into())),
("confidence".into(), Json::Num(0.7)),
(
"embedding".into(),
Json::Arr(vec![Json::Num(1.0), Json::Num(0.0)]),
),
])]),
),
]));
assert_eq!(get(&result, "action"), Some(&Json::Str("duplicate".into())));
assert_eq!(get(&result, "prior_id"), Some(&Json::Num(9.0)));
}
#[test]
fn dedupe_decision_created_duplicate_and_superseded() {
let created = dedupe_decision(&Json::Obj(vec![
("text".into(), Json::Str("a durable new fact".into())),
("candidates".into(), Json::Arr(vec![])),
]));
assert_eq!(get(&created, "action"), Some(&Json::Str("created".into())));
let duplicate = dedupe_decision(&Json::Obj(vec![
("text".into(), Json::Str("The same durable fact".into())),
(
"candidates".into(),
Json::Arr(vec![candidate(7.0, "the same durable fact", 0.7)]),
),
]));
assert_eq!(
get(&duplicate, "action"),
Some(&Json::Str("duplicate".into()))
);
assert_eq!(get(&duplicate, "prior_id"), Some(&Json::Num(7.0)));
let superseded = dedupe_decision(&Json::Obj(vec![
(
"text".into(),
Json::Str(
"the same durable fact the same durable fact the same durable fact".into(),
),
),
(
"candidates".into(),
Json::Arr(vec![candidate(7.0, "the same durable fact", 0.7)]),
),
("confidence".into(), Json::Num(0.95)),
]));
assert_eq!(
get(&superseded, "action"),
Some(&Json::Str("superseded".into()))
);
assert_eq!(get(&superseded, "prior_id"), Some(&Json::Num(7.0)));
}
}
pub fn fuse_results(entries: &[Json], research: &[Json]) -> Vec<Json> {
let mut seen: Vec<String> = vec![];
let mut merged: Vec<Json> = vec![];
+191
View File
@@ -0,0 +1,191 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "block-buffer"
version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
dependencies = [
"generic-array",
]
[[package]]
name = "cfg-if"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
[[package]]
name = "cpufeatures"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
dependencies = [
"libc",
]
[[package]]
name = "crypto-common"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
dependencies = [
"generic-array",
"typenum",
]
[[package]]
name = "digest"
version = "0.10.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
"block-buffer",
"crypto-common",
]
[[package]]
name = "generic-array"
version = "0.14.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
dependencies = [
"typenum",
"version_check",
]
[[package]]
name = "itoa"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "libc"
version = "0.2.191"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "65cd87c9a6d60536580b316ad985726104730a5151c99845f2490428ef91240a"
[[package]]
name = "memchr"
version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
[[package]]
name = "operator_lifecycle"
version = "0.1.0"
dependencies = [
"serde",
"serde_json",
"sha2",
]
[[package]]
name = "proc-macro2"
version = "1.0.107"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.47"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
dependencies = [
"proc-macro2",
]
[[package]]
name = "serde"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
dependencies = [
"serde_core",
"serde_derive",
]
[[package]]
name = "serde_core"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "serde_json"
version = "1.0.154"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6"
dependencies = [
"itoa",
"memchr",
"serde",
"serde_core",
"zmij",
]
[[package]]
name = "sha2"
version = "0.10.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
"cfg-if",
"cpufeatures",
"digest",
]
[[package]]
name = "syn"
version = "3.0.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d78c8dee4c7bf0e14673097256fed6142ce9d3b85a408189d07482442145823b"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "typenum"
version = "1.20.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
[[package]]
name = "unicode-ident"
version = "1.0.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2c754d6c33795a1c324727428e5a7dedb5b06195f9890bdbcba760d3e246563"
[[package]]
name = "version_check"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
[[package]]
name = "zmij"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
+12
View File
@@ -0,0 +1,12 @@
[package]
name = "operator_lifecycle"
version = "0.1.0"
edition = "2021"
[lib]
crate-type = ["rlib"]
[dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
sha2 = "0.10"
+666
View File
@@ -0,0 +1,666 @@
use serde::{Deserialize, Serialize};
use serde_json::Value;
use sha2::{Digest, Sha256};
use std::collections::BTreeMap;
pub const TASK_STATUSES: [&str; 8] = [
"created",
"running",
"awaiting_approval",
"research_required",
"completed",
"failed",
"cancelled",
"superseded",
];
#[derive(Debug, Clone, PartialEq)]
pub struct OperatorError {
pub code: String,
pub operation: String,
pub message: String,
}
impl OperatorError {
pub fn new(code: &str, operation: &str, message: impl Into<String>) -> Self {
Self {
code: code.into(),
operation: operation.into(),
message: message.into(),
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(default, deny_unknown_fields)]
pub struct TaskStateRecord {
pub schema_version: i32,
pub task_id: String,
pub title: String,
pub status: String,
pub channel_id: String,
pub parent_task_id: Option<String>,
pub assigned_model: Option<String>,
pub created_at: String,
pub updated_at: String,
pub outcome_ref: String,
pub route_ref: String,
}
impl Default for TaskStateRecord {
fn default() -> Self {
Self {
schema_version: 1,
task_id: String::new(),
title: String::new(),
status: "created".into(),
channel_id: String::new(),
parent_task_id: None,
assigned_model: None,
created_at: String::new(),
updated_at: String::new(),
outcome_ref: String::new(),
route_ref: String::new(),
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(default, deny_unknown_fields)]
pub struct ConversationMessageRecord {
pub schema_version: i32,
pub msg_id: String,
pub ts: String,
pub role: String,
pub content: String,
pub channel_id: String,
pub task_id: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub thinking: Option<String>,
pub token_estimate: i64,
}
impl Default for ConversationMessageRecord {
fn default() -> Self {
Self {
schema_version: 1,
msg_id: String::new(),
ts: String::new(),
role: "user".into(),
content: String::new(),
channel_id: String::new(),
task_id: None,
thinking: None,
token_estimate: 0,
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(default, deny_unknown_fields)]
pub struct EventEnvelope {
pub schema_version: i32,
pub event_type: String,
pub partition_key: String,
pub event_id: String,
pub parent_id: Option<String>,
pub actor_id: String,
pub device_id: String,
pub authorization: String,
pub payload_hash: String,
pub timestamp: String,
}
impl Default for EventEnvelope {
fn default() -> Self {
Self {
schema_version: 1,
event_type: String::new(),
partition_key: String::new(),
event_id: String::new(),
parent_id: None,
actor_id: String::new(),
device_id: String::new(),
authorization: String::new(),
payload_hash: String::new(),
timestamp: String::new(),
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct IntakeRequest {
pub description: String,
pub channel_id: String,
pub model: Option<String>,
pub active: Option<TaskStateRecord>,
pub latest: Option<TaskStateRecord>,
pub task_id: String,
pub now: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct IntakeDecision {
pub action: String,
pub family: String,
pub record: TaskStateRecord,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TransitionDecision {
pub open: bool,
pub record: TaskStateRecord,
}
pub fn infer_task_family(description: &str) -> &'static str {
let lower = description.to_lowercase();
for (family, words) in [
(
"code",
&["fix", "code", "debug", "test", "refactor", "implement"][..],
),
(
"research",
&["design", "architecture", "research", "analyze"][..],
),
(
"chat",
&["hello", "hi", "thanks", "status", "summarize"][..],
),
] {
if words.iter().any(|word| lower.contains(word)) {
return family;
}
}
"general"
}
pub fn is_follow_up(description: &str) -> bool {
let lower = description.trim().to_lowercase();
const PREFIXES: [&str; 17] = [
"yes",
"yeah",
"yep",
"yup",
"ok",
"okay",
"sure",
"please do",
"go ahead",
"continue",
"do it",
"proceed",
"sounds good",
"approved",
"confirm",
"that works",
"lgtm",
];
PREFIXES.iter().any(|prefix| {
lower.strip_prefix(prefix).is_some_and(|rest| {
rest.chars()
.next()
.is_none_or(|c| !(c.is_alphanumeric() || c == '_'))
})
})
}
pub fn intake(request: &IntakeRequest) -> Result<IntakeDecision, OperatorError> {
let family = infer_task_family(&request.description).to_string();
if let Some(mut record) = request
.active
.clone()
.filter(|record| record.status == "running" || is_follow_up(&request.description))
{
record.title = request.description.clone();
record.updated_at = request.now.clone();
return Ok(IntakeDecision {
action: "amended".into(),
family,
record,
});
}
let record = TaskStateRecord {
task_id: request.task_id.clone(),
title: request.description.clone(),
status: "created".into(),
channel_id: request.channel_id.clone(),
parent_task_id: request.latest.as_ref().map(|latest| latest.task_id.clone()),
assigned_model: request.model.clone().filter(|model| !model.is_empty()),
created_at: request.now.clone(),
updated_at: request.now.clone(),
..TaskStateRecord::default()
};
Ok(IntakeDecision {
action: "created".into(),
family,
record,
})
}
pub fn transition(
original: &TaskStateRecord,
new_status: &str,
outcome_ref: &str,
now: &str,
) -> Result<TransitionDecision, OperatorError> {
if !TASK_STATUSES.contains(&new_status) {
return Err(OperatorError::new(
"UNKNOWN_STATUS",
"transition",
format!("unknown status {new_status:?}; expected one of {TASK_STATUSES:?}"),
));
}
let legal = match original.status.as_str() {
"created" => matches!(
new_status,
"created"
| "running"
| "awaiting_approval"
| "research_required"
| "completed"
| "failed"
| "cancelled"
| "superseded"
),
"running" => matches!(
new_status,
"running"
| "awaiting_approval"
| "research_required"
| "completed"
| "failed"
| "cancelled"
| "superseded"
),
"research_required" => matches!(
new_status,
"running" | "completed" | "failed" | "cancelled" | "superseded"
),
"awaiting_approval" => matches!(
new_status,
"awaiting_approval" | "running" | "completed" | "failed" | "cancelled" | "superseded"
),
"completed" | "failed" | "cancelled" | "superseded" => false,
_ => {
return Err(OperatorError::new(
"INVALID_RECORD",
"transition",
format!("unknown current status {:?}", original.status),
))
}
};
if !legal {
return Err(OperatorError::new(
"ILLEGAL_TRANSITION",
"transition",
format!(
"illegal transition {:?} -> {new_status:?} for {}",
original.status, original.task_id
),
));
}
let mut record = original.clone();
record.status = new_status.into();
if !outcome_ref.is_empty() {
record.outcome_ref = outcome_ref.into();
}
record.updated_at = now.into();
let open = matches!(
new_status,
"created" | "running" | "awaiting_approval" | "research_required"
);
Ok(TransitionDecision { open, record })
}
pub fn validate_record(kind: &str, value: &Value) -> Result<Value, OperatorError> {
let invalid = |detail: String| OperatorError::new("INVALID_RECORD", "validate_record", detail);
match kind {
"task" => {
let mut record: TaskStateRecord =
serde_json::from_value(value.clone()).map_err(|e| invalid(e.to_string()))?;
record.schema_version = 1;
serde_json::to_value(record).map_err(|e| invalid(e.to_string()))
}
"message" => {
let mut record: ConversationMessageRecord =
serde_json::from_value(value.clone()).map_err(|e| invalid(e.to_string()))?;
record.schema_version = 1;
serde_json::to_value(record).map_err(|e| invalid(e.to_string()))
}
"event" => {
if value.get("event_id").and_then(Value::as_str).is_none() {
return Err(invalid("canonical event requires event_id".into()));
}
if value
.get("schema_version")
.and_then(Value::as_i64)
.is_some_and(|version| version < 1)
{
return Err(invalid("schema_version must be >= 1".into()));
}
if ["payload", "source", "channel", "authority_class", "task_id"]
.iter()
.any(|key| value.get(key).is_some())
{
return Err(invalid("legacy event body fields are not canonical".into()));
}
let mut record: EventEnvelope =
serde_json::from_value(value.clone()).map_err(|e| invalid(e.to_string()))?;
record.schema_version = 1;
serde_json::to_value(record).map_err(|e| invalid(e.to_string()))
}
_ => Err(invalid(format!("unknown record kind {kind:?}"))),
}
}
fn write_string(text: &str, out: &mut String) {
out.push('"');
for c in text.chars() {
let cp = c as u32;
match c {
'"' => out.push_str("\\\""),
'\\' => out.push_str("\\\\"),
'\u{08}' => out.push_str("\\b"),
'\t' => out.push_str("\\t"),
'\n' => out.push_str("\\n"),
'\u{0c}' => out.push_str("\\f"),
'\r' => out.push_str("\\r"),
_ if !(0x20..=0x7e).contains(&cp) => {
if cp <= 0xffff {
out.push_str(&format!("\\u{cp:04x}"));
} else {
let offset = cp - 0x10000;
out.push_str(&format!(
"\\u{:04x}\\u{:04x}",
0xd800 + (offset >> 10),
0xdc00 + (offset & 0x3ff)
));
}
}
_ => out.push(c),
}
}
out.push('"');
}
fn write_canonical(value: &Value, out: &mut String) {
match value {
Value::Null => out.push_str("null"),
Value::Bool(value) => out.push_str(if *value { "true" } else { "false" }),
Value::Number(value) => out.push_str(&value.to_string()),
Value::String(value) => write_string(value, out),
Value::Array(values) => {
out.push('[');
for (index, value) in values.iter().enumerate() {
if index > 0 {
out.push(',');
}
write_canonical(value, out);
}
out.push(']');
}
Value::Object(values) => {
let sorted: BTreeMap<_, _> = values.iter().collect();
out.push('{');
for (index, (key, value)) in sorted.into_iter().enumerate() {
if index > 0 {
out.push(',');
}
write_string(key, out);
out.push(':');
write_canonical(value, out);
}
out.push('}');
}
}
}
// Python's payload_hash uses sorted, compact, ensure_ascii JSON, not serde's wire rendering.
pub fn canonical_json(value: &Value) -> String {
let mut out = String::new();
write_canonical(value, &mut out);
out
}
pub fn payload_hash(value: &Value) -> String {
let digest = Sha256::digest(canonical_json(value).as_bytes());
let hex = digest
.iter()
.map(|byte| format!("{byte:02x}"))
.collect::<String>();
format!("sha256:{hex}")
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
const NOW: &str = "2026-10-26T00:00:00+00:00";
fn task(status: &str) -> TaskStateRecord {
TaskStateRecord {
schema_version: 1,
task_id: "task_abc".into(),
title: "original".into(),
status: status.into(),
channel_id: "task".into(),
parent_task_id: None,
assigned_model: Some("model-a".into()),
created_at: "created".into(),
updated_at: "before".into(),
outcome_ref: String::new(),
route_ref: String::new(),
}
}
#[test]
fn research_required_resumes_without_becoming_terminal() {
let decision = transition(&task("research_required"), "running", "", NOW).unwrap();
assert_eq!(decision.record.status, "running");
assert_eq!(decision.record.updated_at, NOW);
assert!(decision.open);
}
#[test]
fn awaiting_approval_resumes_and_reasserts_idempotently_at_decision_layer() {
let resumed = transition(&task("awaiting_approval"), "running", "", NOW).unwrap();
assert_eq!(resumed.record.status, "running");
assert!(resumed.open);
let waiting = transition(&task("awaiting_approval"), "awaiting_approval", "", NOW).unwrap();
assert_eq!(waiting.record.status, "awaiting_approval");
}
#[test]
fn nonterminal_self_edges_match_the_python_spine() {
for status in ["created", "running", "awaiting_approval"] {
assert!(
transition(&task(status), status, "", NOW).is_ok(),
"{status}"
);
}
let error =
transition(&task("research_required"), "research_required", "", NOW).unwrap_err();
assert_eq!(error.code, "ILLEGAL_TRANSITION");
}
#[test]
fn terminal_states_reject_every_outgoing_edge() {
for status in ["completed", "failed", "cancelled", "superseded"] {
let error = transition(&task(status), "running", "", NOW).unwrap_err();
assert_eq!(error.code, "ILLEGAL_TRANSITION", "{status}");
}
}
#[test]
fn unknown_status_is_rejected_before_mutation() {
let original = task("created");
let error = transition(&original, "bogus", "", NOW).unwrap_err();
assert_eq!(error.code, "UNKNOWN_STATUS");
assert_eq!(original.status, "created");
}
#[test]
fn outcome_ref_is_replaced_only_when_supplied() {
let mut original = task("running");
original.outcome_ref = "prior".into();
let kept = transition(&original, "completed", "", NOW).unwrap();
assert_eq!(kept.record.outcome_ref, "prior");
let replaced = transition(&original, "failed", "execution_error", NOW).unwrap();
assert_eq!(replaced.record.outcome_ref, "execution_error");
}
#[test]
fn follow_up_amends_active_identity_and_preserves_model() {
let active = task("awaiting_approval");
let latest = active.clone();
let decision = intake(&IntakeRequest {
description: "yes continue with that".into(),
channel_id: "task".into(),
model: Some("model-b".into()),
active: Some(active),
latest: Some(latest),
task_id: "task_new".into(),
now: NOW.into(),
})
.unwrap();
assert_eq!(decision.action, "amended");
assert_eq!(decision.record.task_id, "task_abc");
assert_eq!(decision.record.title, "yes continue with that");
assert_eq!(decision.record.assigned_model.as_deref(), Some("model-a"));
}
#[test]
fn running_active_task_amends_even_without_follow_up_marker() {
let decision = intake(&IntakeRequest {
description: "new detail".into(),
channel_id: "task".into(),
model: None,
active: Some(task("running")),
latest: Some(task("running")),
task_id: "task_new".into(),
now: NOW.into(),
})
.unwrap();
assert_eq!(decision.action, "amended");
assert_eq!(decision.record.task_id, "task_abc");
}
#[test]
fn fresh_intake_preserves_parent_lineage_and_assigns_model() {
let mut latest = task("completed");
latest.task_id = "task_parent".into();
let decision = intake(&IntakeRequest {
description: "implement the parser".into(),
channel_id: "task".into(),
model: Some("model-b".into()),
active: None,
latest: Some(latest),
task_id: "task_new".into(),
now: NOW.into(),
})
.unwrap();
assert_eq!(decision.action, "created");
assert_eq!(decision.record.task_id, "task_new");
assert_eq!(
decision.record.parent_task_id.as_deref(),
Some("task_parent")
);
assert_eq!(decision.record.assigned_model.as_deref(), Some("model-b"));
assert_eq!(decision.family, "code");
}
#[test]
fn task_family_uses_python_keyword_precedence() {
assert_eq!(infer_task_family("fix and research this"), "code");
assert_eq!(infer_task_family("design the architecture"), "research");
assert_eq!(infer_task_family("hello and thanks"), "chat");
assert_eq!(infer_task_family("unclassified"), "general");
}
#[test]
fn follow_up_vocabulary_requires_a_word_boundary() {
for text in [
"yes",
"Please do it",
"go ahead",
"approved, thanks",
"lgtm",
] {
assert!(is_follow_up(text), "{text}");
}
for text in ["yesterday", "okayish", "continueplease", "write tests"] {
assert!(!is_follow_up(text), "{text}");
}
}
#[test]
fn payload_hash_matches_python_sorted_compact_ensure_ascii_json() {
let mut record = task("created");
record.title = "Fix ✓".into();
record.created_at = "now".into();
record.updated_at = "now".into();
record.assigned_model = Some("m".into());
let value = serde_json::to_value(record).unwrap();
let canonical = canonical_json(&value);
assert_eq!(
canonical,
r#"{"assigned_model":"m","channel_id":"task","created_at":"now","outcome_ref":"","parent_task_id":null,"route_ref":"","schema_version":1,"status":"created","task_id":"task_abc","title":"Fix \u2713","updated_at":"now"}"#
);
assert_eq!(
payload_hash(&value),
"sha256:c9606353b2ec8281111ab910f5f5bb2ae961895eaeb27d125ee583e008c7529c"
);
}
#[test]
fn serving_records_round_trip_with_schema_and_tuple_free_shapes() {
let task_value = serde_json::to_value(task("created")).unwrap();
let normalized = validate_record("task", &task_value).unwrap();
assert_eq!(normalized["schema_version"], 1);
assert_eq!(normalized["parent_task_id"], json!(null));
let message = ConversationMessageRecord {
schema_version: 1,
msg_id: "msg_abc".into(),
ts: NOW.into(),
role: "user".into(),
content: "hello".into(),
channel_id: "task".into(),
task_id: Some("task_abc".into()),
token_estimate: 1,
thinking: Some("reason".into()),
};
let normalized =
validate_record("message", &serde_json::to_value(message).unwrap()).unwrap();
assert_eq!(normalized["task_id"], "task_abc");
assert_eq!(normalized["thinking"], "reason");
let event = EventEnvelope {
schema_version: 1,
event_type: "task_state.created".into(),
partition_key: "task:task_abc".into(),
event_id: "evt_abc".into(),
parent_id: None,
actor_id: "task_state_repository".into(),
device_id: String::new(),
authorization: String::new(),
payload_hash: "sha256:abc".into(),
timestamp: NOW.into(),
};
let normalized = validate_record("event", &serde_json::to_value(event).unwrap()).unwrap();
assert!(normalized.get("payload").is_none());
assert_eq!(normalized["parent_id"], json!(null));
}
#[test]
fn legacy_event_body_fields_are_not_canonical_records() {
let error = validate_record(
"event",
&json!({"event_id":"evt_old","event_type":"x","partition_key":"p","payload":{"secret":true}}),
)
.unwrap_err();
assert_eq!(error.code, "INVALID_RECORD");
}
}
Loaded 100 of 119 files, more files were not shown because too many files have changed in this diff. Show more