feat(token_optimizer): close PARTIAL gaps — dichotomy wave
Name-preserving counterparts for the 9 audit-gap fns (16/25 -> 25/25 FULL): Entry::to_dict, State::_to_state/_apply_state, _run_kernel (op dispatch), _compress_context (tc delegation mirror), _write_snapshot_doc (file-doc conversion promoted out of the battery), get_optimizer (construction decision + FFI op), _tc_core/_to_core pin contracts (frozen rows + battery assertions). _tc_core/_to_core/get_optimizer I/O halves documented as facade seams (memory_selector pattern). New truth: 19-case gap freeze via the shared harness (rig scripts/to_freeze_seams) — byte-identical across gen runs and --verify. Pins the _load partial-adoption quirk (bad entry mid-loop: partial cache in file document order, stats skipped) + CacheEntry kwargs semantics (optional use_count/hit_count, unknown key TypeError); serde_json preserve_order for document-order iteration. Battery 7/7; live parity 60/60 (FFI construction rows + 40 dual-mode fuzz trials + module-API/singleton E2E); clippy -D warnings clean; module tests 5/5 both pins. Evidence: verification/to-*.
This commit is contained in:
Nick committed
1 parent
e481184fff
commit
19c062e15c
13 files changed
+1586
-93
No files matched your search
@@ -2,14 +2,14 @@
|
||||
|
||||
_Regenerate with `python3 scripts/dichotomy_audit.py --write`. A module is FULL only when 80%+ of its Python functions have name-preserving counterparts in the Go/Rust implementation — directories without the functions are facades and count as NOT TOUCHED._
|
||||
|
||||
**FULL completion: 58%** (111/192 runtime modules at FULL)
|
||||
**FULL completion: 58%** (112/192 runtime modules at FULL)
|
||||
|
||||
Function-level detail: 72% (1644/2287 runtime functions matched) — secondary; includes in-flight work
|
||||
Function-level detail: 72% (1653/2287 runtime functions matched) — secondary; includes in-flight work
|
||||
|
||||
| Status | Modules | Lines | Meaning |
|
||||
|---|---|---|---|
|
||||
| FULL | 111 | 46509 | 80%+ of functions ported |
|
||||
| PARTIAL | 42 | 17236 | 20–80% — real logic exists, gaps named below |
|
||||
| FULL | 112 | 47023 | 80%+ of functions ported |
|
||||
| PARTIAL | 41 | 16722 | 20–80% — real logic exists, gaps named below |
|
||||
| NOT TOUCHED | 39 | 11471 | no name-preserving implementation |
|
||||
| Total runtime | 192 | 75216 | |
|
||||
|
||||
@@ -65,7 +65,6 @@ Function-level detail: 72% (1644/2287 runtime functions matched) — secondary;
|
||||
| memory_store | 2333 | 30/68 | _ctx_budget, now_iso, ensure_memory_type_column, _embed, _temp_json, _entities_to_json |
|
||||
| warm_router | 1518 | 17/33 | _ensure_use_history_table, record_outcome, record_neuron_fire, base_level_activation, _db, _aaak |
|
||||
| tool_executor | 1836 | 33/47 | _execute_tool_now, serialize_prepared_tool, deserialize_prepared_tool, confirm_write, _resolve_execution_cwd, _run_command |
|
||||
| token_optimizer | 514 | 16/25 | get_optimizer, _tc_core, _to_core, to_dict, _to_state, _apply_state |
|
||||
| time_awareness | 402 | 12/20 | _clock_args, _ta_core, get_time_awareness, to_dict, _to_state, _apply_state |
|
||||
| intent_continuity | 284 | 6/13 | _now, _raise_core_err, to_dict, from_dict, save, _ledger_state |
|
||||
| audit_ledger | 212 | 2/8 | _compute_hash, get_entries, log_promotion, log_gate_result, log_drift_alert, main |
|
||||
@@ -144,6 +143,7 @@ Function-level detail: 72% (1644/2287 runtime functions matched) — secondary;
|
||||
| build_mode | 521 | 33/33 | GO |
|
||||
| growth_delta | 520 | 17/18 | GO |
|
||||
| event_wait | 518 | 14/14 | GO |
|
||||
| token_optimizer | 514 | 25/25 | RUST |
|
||||
| cache | 492 | 17/17 | GO |
|
||||
| claim_audit | 484 | 14/14 | RUST |
|
||||
| model_connect | 471 | 11/11 | GO |
|
||||
|
||||
@@ -0,0 +1,320 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Live parity: token_optimizer Rust kernel vs Python oracle (dichotomy wave,
|
||||
close-PARTIAL gap pass). Layers:
|
||||
|
||||
1. FROZEN — the gap truth corpus (src/rust/token_optimizer/frozen_truth.json,
|
||||
generated by scripts/gen_token_optimizer_truth.py under the
|
||||
scripts/to_freeze_seams rig against the LIVE oracle with both core
|
||||
pins python): the get_optimizer construction rows replayed through the
|
||||
ctypes FFI op (envelope state vs the frozen oracle observation,
|
||||
last_save_time excluded — the facade stamps the monotonic clock after
|
||||
_load); the pin contracts asserted in BOTH directions.
|
||||
2. FUZZ — 40 seeded optimizer instances (seed 2479), each running a
|
||||
random op script (static/dynamic/optimize/status/stats/save_if_dirty/
|
||||
flush/mark_dirty) with a PATCHED module clock (frozen datetime +
|
||||
counter monotonic): the full observation tape (per-op result + state +
|
||||
storage file) must be IDENTICAL with RUNE_TOKEN_OPTIMIZER_CORE=python
|
||||
(oracle floor) and with the Rust kernel live through the facade's
|
||||
_run_kernel delegation.
|
||||
3. LIVE — dual-mode E2E on the gap fns: the real module-level
|
||||
get_optimizer()/optimize_prompt() API, the 8 storage-file construction
|
||||
variants, and the _write_snapshot conversion, both pins.
|
||||
|
||||
Exit 0 only at 100% parity on the gap-closing functions.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import shutil
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
os.environ["RUNE_TOKEN_OPTIMIZER_CORE"] = "python" # oracle floor to start
|
||||
PARITY_HOME = "/tmp/to_parity"
|
||||
|
||||
sys.path.insert(0, str(REPO / "src"))
|
||||
|
||||
from rune import to_loader # noqa: E402
|
||||
from rune import token_optimizer as to # noqa: E402
|
||||
|
||||
if not to_loader.available():
|
||||
sys.exit("FATAL: Rust core not built — cargo build --release in src/rust/token_optimizer")
|
||||
|
||||
TRUTH = json.loads(
|
||||
(REPO / "src" / "rust" / "token_optimizer" / "frozen_truth.json").read_text())
|
||||
|
||||
fails: list[str] = []
|
||||
counts = {"frozen": 0, "fuzz": 0, "live": 0}
|
||||
|
||||
|
||||
def fail(layer, name, detail):
|
||||
fails.append(f"[{layer}] {name}: {detail}")
|
||||
print(f"FAIL[{layer}] {name}: {detail}"[:400])
|
||||
|
||||
|
||||
def j(x):
|
||||
return json.dumps(x, sort_keys=True, ensure_ascii=True)
|
||||
|
||||
|
||||
def check(layer, name, want, got):
|
||||
counts[layer] += 1
|
||||
if j(want) != j(got):
|
||||
fail(layer, name, f"want={j(want)[:300]} got={j(got)[:300]}")
|
||||
|
||||
|
||||
def pin(mode):
|
||||
if mode == "python":
|
||||
os.environ["RUNE_TOKEN_OPTIMIZER_CORE"] = "python"
|
||||
else:
|
||||
os.environ.pop("RUNE_TOKEN_OPTIMIZER_CORE", None)
|
||||
|
||||
|
||||
# ── layer 1: FROZEN (construction rows through the FFI) ────────────────────
|
||||
|
||||
for row in TRUTH:
|
||||
if row["fn"] != "to_get_optimizer_state":
|
||||
continue
|
||||
file_state, seed = row["args"]
|
||||
exp = row["expected"]
|
||||
env = to_loader.run_op(
|
||||
"get_optimizer", {},
|
||||
{"storage_text": None if file_state == "absent" else seed})
|
||||
got_state = env["value"]
|
||||
want_state = {
|
||||
"cache": exp["cache"],
|
||||
"stats": exp["stats"],
|
||||
"dirty": exp["dirty"],
|
||||
"changes_since_save": exp["changes_since_save"],
|
||||
}
|
||||
got_cmp = {k: got_state[k] for k in want_state}
|
||||
check("frozen", f"ffi_get_optimizer:{file_state}", want_state, got_cmp)
|
||||
|
||||
# pin contracts, both directions (probes live in the facade seam).
|
||||
pin("python")
|
||||
check("frozen", "pin_contract:to_core_python_floor", None, to._to_core())
|
||||
pin("rust")
|
||||
check("frozen", "pin_contract:to_core_reachable", True, to._to_core() is not None)
|
||||
pin("python")
|
||||
|
||||
# ── the patched clock (both pins see identical time) ───────────────────────
|
||||
|
||||
FROZEN_NOW = datetime(2026, 10, 14, 6, 0, 0, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
class _FrozenDatetime(datetime):
|
||||
@classmethod
|
||||
def now(cls, tz=None):
|
||||
return FROZEN_NOW if tz is not None else FROZEN_NOW.replace(tzinfo=None)
|
||||
|
||||
|
||||
class _Clock:
|
||||
def __init__(self):
|
||||
self.t = 1000.0
|
||||
|
||||
def monotonic(self):
|
||||
return self.t
|
||||
|
||||
def advance(self, dt):
|
||||
self.t += dt
|
||||
|
||||
|
||||
def reset_scratch():
|
||||
if os.path.isdir(PARITY_HOME):
|
||||
shutil.rmtree(PARITY_HOME)
|
||||
os.makedirs(PARITY_HOME, exist_ok=True)
|
||||
|
||||
|
||||
STORAGE = os.path.join(PARITY_HOME, "token_optimizer.json")
|
||||
PERSONA = "P" * 80 # deterministic long persona (bypasses persona_definition)
|
||||
|
||||
TEXTS = [
|
||||
"fix the login bug",
|
||||
"kubernetes pods restart on failure",
|
||||
" spaced out text with runs ",
|
||||
"multi\nline\ncontext\nwith\nnewlines",
|
||||
"",
|
||||
"unicode ✓ snowman ☃ and dashes — oh my",
|
||||
]
|
||||
CONTEXTS = ["", "recent work on the parser; recent work on the parser;",
|
||||
"noise noise noise noise noise noise noise noise noise noise",
|
||||
"one two three four five six seven eight nine ten"]
|
||||
|
||||
|
||||
def drive(script, seed_file=None):
|
||||
"""Run one op script through the facade; return the observation tape."""
|
||||
reset_scratch()
|
||||
if seed_file is not None:
|
||||
with open(STORAGE, "w") as f:
|
||||
f.write(seed_file)
|
||||
clock = _Clock()
|
||||
to.time = clock # the facade's `import time` binding — module-local
|
||||
to.datetime = _FrozenDatetime
|
||||
to._optimizer = None
|
||||
opt = to.TokenOptimizer(storage_path=STORAGE)
|
||||
opt._load_persona_definition = lambda: PERSONA
|
||||
tape = []
|
||||
|
||||
def state_obs():
|
||||
return {
|
||||
"cache": [e.to_dict() for e in opt.cache.values()],
|
||||
"stats": dict(opt.stats),
|
||||
"dirty": opt._dirty,
|
||||
"changes": opt._changes_since_save,
|
||||
}
|
||||
|
||||
def file_obs():
|
||||
if not os.path.exists(STORAGE):
|
||||
return None
|
||||
with open(STORAGE) as f:
|
||||
return f.read()
|
||||
|
||||
for kind, arg in script:
|
||||
clock.advance(0.5)
|
||||
if kind == "dt":
|
||||
clock.advance(arg) # jump the clock (debounce boundary probes)
|
||||
elif kind == "get_static_cache":
|
||||
tape.append({"r": opt.get_static_cache(), "s": state_obs()})
|
||||
elif kind == "get_dynamic_cache":
|
||||
tape.append({"r": opt.get_dynamic_cache(arg), "s": state_obs()})
|
||||
elif kind == "optimize_prompt":
|
||||
tape.append({"r": opt.optimize_prompt(arg[0], arg[1]), "s": state_obs()})
|
||||
elif kind == "get_status":
|
||||
tape.append({"r": opt.get_status(), "s": state_obs()})
|
||||
elif kind == "get_cache_stats":
|
||||
tape.append({"r": opt.get_cache_stats(), "s": state_obs()})
|
||||
elif kind == "save_if_dirty":
|
||||
opt._save_if_dirty()
|
||||
tape.append({"r": None, "s": state_obs(), "f": file_obs()})
|
||||
elif kind == "flush":
|
||||
opt.flush()
|
||||
tape.append({"r": None, "s": state_obs(), "f": file_obs()})
|
||||
elif kind == "mark_dirty":
|
||||
opt._mark_dirty()
|
||||
tape.append({"r": None, "s": state_obs()})
|
||||
final = {"tape": tape, "file": file_obs(), "status": opt.get_status()}
|
||||
to.time, to.datetime = __import__("time"), datetime
|
||||
return final
|
||||
|
||||
|
||||
# ── layer 2: FUZZ (dual-mode full state machine) ──────────────────────────
|
||||
|
||||
rng = random.Random(2479)
|
||||
SEEDS = [None,
|
||||
'{"cache": {"static": {"key": "static", "content": "seed persona",'
|
||||
' "tokens": 3, "last_used": "2026-10-14T00:00:00+00:00",'
|
||||
' "use_count": 9, "hit_count": 4}}, "stats": {"total_calls": 2,'
|
||||
' "total_tokens_saved": 10, "cache_hits": 1, "cache_misses": 1}}']
|
||||
for trial in range(40):
|
||||
script = []
|
||||
for _ in range(rng.randrange(4, 9)):
|
||||
roll = rng.random()
|
||||
if roll < 0.10:
|
||||
script.append(("dt", rng.choice([0.0, 10.0, 31.0, 65.0])))
|
||||
elif roll < 0.30:
|
||||
script.append(("get_static_cache", None))
|
||||
elif roll < 0.50:
|
||||
script.append(("get_dynamic_cache", rng.choice(CONTEXTS)))
|
||||
elif roll < 0.70:
|
||||
script.append(("optimize_prompt", (rng.choice(TEXTS), rng.choice(CONTEXTS))))
|
||||
elif roll < 0.78:
|
||||
script.append(("get_status", None))
|
||||
elif roll < 0.86:
|
||||
script.append(("get_cache_stats", None))
|
||||
elif roll < 0.92:
|
||||
script.append(("save_if_dirty", None))
|
||||
elif roll < 0.97:
|
||||
script.append(("flush", None))
|
||||
else:
|
||||
script.append(("mark_dirty", None))
|
||||
seed_file = rng.choice(SEEDS)
|
||||
|
||||
pin("python")
|
||||
want = drive(script, seed_file)
|
||||
pin("rust")
|
||||
got = drive(script, seed_file)
|
||||
check("fuzz", f"trial{trial}", want, got)
|
||||
if fails:
|
||||
break
|
||||
pin("python")
|
||||
|
||||
# ── layer 3: LIVE (dual-mode E2E on the gap fns) ───────────────────────────
|
||||
|
||||
|
||||
def dual(label, call):
|
||||
pin("python")
|
||||
want = call()
|
||||
pin("rust")
|
||||
got = call()
|
||||
check("live", label, want, got)
|
||||
|
||||
|
||||
# module-level API through the REAL global singleton (get_optimizer path).
|
||||
def module_api():
|
||||
reset_scratch()
|
||||
to._optimizer = None
|
||||
clock = _Clock()
|
||||
to.time, to.datetime = clock, _FrozenDatetime
|
||||
os.environ["RUNE_MEMORY_DIR"] = PARITY_HOME
|
||||
try:
|
||||
r1 = to.optimize_prompt("hello world", "some context here")
|
||||
opt = to.get_optimizer()
|
||||
r2 = {"status": opt.get_status(), "stats": opt.get_cache_stats(),
|
||||
"singleton": to.get_optimizer() is opt}
|
||||
return {"r1": r1, "r2": r2}
|
||||
finally:
|
||||
to.time, to.datetime = __import__("time"), datetime
|
||||
to._optimizer = None
|
||||
|
||||
|
||||
dual("module_api:get_optimizer_singleton", module_api)
|
||||
|
||||
# the 8 construction variants through the REAL get_optimizer (both pins).
|
||||
CONS = [r for r in TRUTH if r["fn"] == "to_get_optimizer_state"]
|
||||
for row in CONS:
|
||||
file_state, seed = row["args"]
|
||||
|
||||
def construct(file_state=file_state, seed=seed):
|
||||
reset_scratch()
|
||||
if file_state != "absent":
|
||||
with open(STORAGE, "w") as f:
|
||||
f.write(seed)
|
||||
to._optimizer = None
|
||||
os.environ["RUNE_MEMORY_DIR"] = PARITY_HOME
|
||||
try:
|
||||
opt = to.get_optimizer()
|
||||
second = to.get_optimizer()
|
||||
out = {
|
||||
"cache": [e.to_dict() for e in opt.cache.values()],
|
||||
"stats": dict(opt.stats),
|
||||
"dirty": opt._dirty,
|
||||
"changes": opt._changes_since_save,
|
||||
"singleton_reused": second is opt,
|
||||
}
|
||||
return out
|
||||
finally:
|
||||
to._optimizer = None
|
||||
|
||||
dual(f"construct:{file_state}", construct)
|
||||
|
||||
# _write_snapshot conversion through the REAL facade method (both pins).
|
||||
SNAP = [r for r in TRUTH if r["fn"] == "to_write_snapshot_doc"][0]["args"][0]
|
||||
|
||||
|
||||
def snap_write():
|
||||
reset_scratch()
|
||||
path = os.path.join(PARITY_HOME, "snap.json")
|
||||
opt = to.TokenOptimizer.__new__(to.TokenOptimizer)
|
||||
opt.storage_path = path
|
||||
to.TokenOptimizer._write_snapshot(opt, SNAP)
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
dual("_write_snapshot:doc", snap_write)
|
||||
|
||||
print(f"parity: {sum(counts.values())} checks "
|
||||
f"(frozen {counts['frozen']}, fuzz {counts['fuzz']}, live {counts['live']}) "
|
||||
f"— {len(fails)} failures")
|
||||
sys.exit(1 if fails else 0)
|
||||
@@ -0,0 +1,165 @@
|
||||
#!/usr/bin/env python3
|
||||
"""gen_token_optimizer_truth.py — factory wave GAP truth for token_optimizer.
|
||||
|
||||
Freezes the oracle behavior of the 9 audit-gap functions (dichotomy wave
|
||||
#2479 gap close: get_optimizer/_tc_core/_to_core/to_dict/_to_state/
|
||||
_apply_state/_write_snapshot/_run_kernel/_compress_context) through the
|
||||
SHARED harness (scripts/freeze_truth.py — one law).
|
||||
|
||||
Runs under the freeze rig (scripts/to_freeze_seams/sitecustomize.py —
|
||||
PYTHONPATH-imported by BOTH this generator and --verify): both core pins
|
||||
python (oracle floor), hermetic FIXED-path memory dir, JSON-safe seam fns
|
||||
driving the REAL facade methods on __new__-built receivers.
|
||||
|
||||
Run (from the repo root):
|
||||
PYTHONHASHSEED=0 PYTHONPATH=scripts/to_freeze_seams:src \
|
||||
python3 scripts/gen_token_optimizer_truth.py
|
||||
|
||||
Verify (byte-identical, same rig):
|
||||
PYTHONHASHSEED=0 PYTHONPATH=scripts/to_freeze_seams:src \
|
||||
python3 scripts/freeze_truth.py --verify \
|
||||
src/rust/token_optimizer/frozen_truth.json \
|
||||
--module rune.token_optimizer
|
||||
|
||||
Case map (every gap branch):
|
||||
* _tc_core / _to_core — the pin contracts (RUNE_TOKEN_CORE /
|
||||
RUNE_TOKEN_OPTIMIZER_CORE=python -> None, the
|
||||
python floor; asserted by the battery as
|
||||
contract rows, replayed both directions by
|
||||
live parity).
|
||||
* TokenOptimizer._run_kernel — under the pin the delegation loop is a
|
||||
no-op: None for every op (the facade floor).
|
||||
* CacheEntry.to_dict — the six-field persisted projection (issue #770
|
||||
shape: content REQUIRED).
|
||||
* to_to_state — the kernel transfer form: cache as an ORDERED
|
||||
ARRAY (insertion order), stats, dirty,
|
||||
changes, monotonic stamp; populated + empty.
|
||||
* to_apply_state — adoption of a transfer doc: cache rebuilt as
|
||||
key->CacheEntry in array order + full attr
|
||||
set; populated + empty.
|
||||
* to_get_optimizer_state — construction E2E per storage-file variant:
|
||||
absent (defaults), seeded (cache+stats load,
|
||||
file key order preserved), stats_only (empty
|
||||
cache, stats adopted), corrupt (blanket-except
|
||||
-> defaults), partial_bad (a bad entry raises
|
||||
mid-loop: the PARTIAL cache survives, stats
|
||||
stays default — the blanket-except recovery
|
||||
#770), optional_fields (use_count/hit_count
|
||||
default 0, stats adopted), extra_field
|
||||
(unknown kwarg = TypeError mid-loop, document
|
||||
order decides what survives), notdict
|
||||
(non-object top level -> defaults); singleton
|
||||
identity on the second call.
|
||||
* to_write_snapshot_doc — the facade's file conversion: transfer array
|
||||
-> on-disk object doc; populated + empty with
|
||||
missing stats (null passthrough).
|
||||
|
||||
NOT a row: _compress_context — the oracle fn is a pure delegation hop to
|
||||
the token_counting kernel (its math is frozen in src/rust/token_counting
|
||||
and exercised verbatim through the get_dynamic_cache/optimize_prompt rows
|
||||
of frozen_to_truth.json's s1-s8 scenarios); the Rust mirror
|
||||
_compress_context sits on that same frozen path.
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(REPO / "scripts"))
|
||||
|
||||
from freeze_truth import freeze, write # noqa: E402
|
||||
|
||||
import rune.token_optimizer as oracle # noqa: E402
|
||||
|
||||
E_STATIC = {
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2,
|
||||
}
|
||||
E_DYN = {
|
||||
"key": "0123456789abcdef",
|
||||
"content": "recent: pods restarted",
|
||||
"tokens": 5,
|
||||
"last_used": "2026-10-14T00:00:01+00:00",
|
||||
"use_count": 1,
|
||||
"hit_count": 0,
|
||||
}
|
||||
STATS = {
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1,
|
||||
}
|
||||
|
||||
SEEDED = '{"cache": {"static": ' + str(E_STATIC).replace("'", '"') + \
|
||||
', "0123456789abcdef": ' + str(E_DYN).replace("'", '"') + \
|
||||
'}, "stats": ' + str(STATS).replace("'", '"') + '}'
|
||||
STATS_ONLY = '{"stats": {"total_calls": 9, "total_tokens_saved": 7, "cache_hits": 4, "cache_misses": 4}}'
|
||||
PARTIAL_BAD = ('{"cache": {"static": ' + str(E_STATIC).replace("'", '"')
|
||||
+ ', "bad": {"key": "bad"}}, "stats": ' + str(STATS).replace("'", '"') + '}')
|
||||
E_REQ = {"key": "req", "content": "required only entry", "tokens": 4,
|
||||
"last_used": "2026-10-14T00:00:02+00:00"}
|
||||
OPTIONAL_FIELDS = ('{"cache": {"req": ' + str(E_REQ).replace("'", '"')
|
||||
+ '}, "stats": ' + str(STATS).replace("'", '"') + '}')
|
||||
EXTRA_FIELD = ('{"cache": {"static": ' + str(E_STATIC).replace("'", '"')
|
||||
+ ', "zz_bad": {"key": "zz_bad", "content": "x", "tokens": 1,'
|
||||
+ ' "last_used": "t", "bogus": true}}, "stats": '
|
||||
+ str(STATS).replace("'", '"') + '}')
|
||||
CORRUPT = "NOT JSON {{{"
|
||||
NOTDICT = "[1, 2, 3]"
|
||||
|
||||
TRANSFER = {
|
||||
"cache": [E_STATIC, E_DYN],
|
||||
"stats": STATS,
|
||||
"dirty": True,
|
||||
"changes_since_save": 7,
|
||||
"last_save_time": 99.25,
|
||||
}
|
||||
|
||||
cases = [
|
||||
# pin contracts — the delegation probes under the python pins.
|
||||
("_tc_core", (), {}),
|
||||
("_to_core", (), {}),
|
||||
# the delegation loop under the pin: no-op for any op.
|
||||
{"fn": "TokenOptimizer._run_kernel", "recv": {"cls": "TokenOptimizer"},
|
||||
"args": ["optimize_prompt", {}], "kwargs": {}},
|
||||
# CacheEntry.to_dict — the persisted projection.
|
||||
{"fn": "CacheEntry.to_dict",
|
||||
"recv": {"cls": "CacheEntry", **E_STATIC}, "args": [], "kwargs": {}},
|
||||
# _to_state: populated (ordered array) + empty.
|
||||
("to_to_state", ([E_STATIC, E_DYN], STATS, True, 7, 99.25), {}),
|
||||
("to_to_state", ([], {"total_calls": 0, "total_tokens_saved": 0,
|
||||
"cache_hits": 0, "cache_misses": 0}, False, 0, 0.0), {}),
|
||||
# _apply_state: populated + empty adoption.
|
||||
("to_apply_state", (TRANSFER,), {}),
|
||||
("to_apply_state", ({"cache": [], "stats": STATS, "dirty": False,
|
||||
"changes_since_save": 0, "last_save_time": 1.5},), {}),
|
||||
# _apply_state with a required-only entry: use_count/hit_count default 0.
|
||||
("to_apply_state", ({"cache": [E_REQ], "stats": STATS, "dirty": True,
|
||||
"changes_since_save": 2, "last_save_time": 3.25},), {}),
|
||||
# get_optimizer E2E per storage-file variant.
|
||||
("to_get_optimizer_state", ("absent", ""), {}),
|
||||
("to_get_optimizer_state", ("seeded", SEEDED), {}),
|
||||
("to_get_optimizer_state", ("stats_only", STATS_ONLY), {}),
|
||||
("to_get_optimizer_state", ("corrupt", CORRUPT), {}),
|
||||
("to_get_optimizer_state", ("partial_bad", PARTIAL_BAD), {}),
|
||||
("to_get_optimizer_state", ("optional_fields", OPTIONAL_FIELDS), {}),
|
||||
("to_get_optimizer_state", ("extra_field", EXTRA_FIELD), {}),
|
||||
("to_get_optimizer_state", ("notdict", NOTDICT), {}),
|
||||
# _write_snapshot: the array -> file object doc conversion. (Both keys
|
||||
# are required by the oracle — snap["cache"]/snap["stats"] KeyError on
|
||||
# a malformed snapshot; production snapshots always come from the
|
||||
# kernel's snapshot_doc, so no malformed row is freezable.)
|
||||
("to_write_snapshot_doc", ({"cache": [E_STATIC, E_DYN], "stats": STATS},), {}),
|
||||
("to_write_snapshot_doc", ({"cache": [], "stats": {}},), {}),
|
||||
]
|
||||
|
||||
assert not [c for c in cases if isinstance(c, tuple) and c[0] == "missing"]
|
||||
truth = freeze("rune.token_optimizer", cases, oracle_module="rune.token_optimizer")
|
||||
dest = REPO / "src" / "rust" / "token_optimizer" / "frozen_truth.json"
|
||||
write(truth, dest)
|
||||
print(f"wrote {len(truth)} cases -> {dest}")
|
||||
print("seams bound:", sorted(n for n in dir(oracle) if n.startswith("to_")))
|
||||
@@ -0,0 +1,125 @@
|
||||
"""sitecustomize — the token_optimizer freeze rig (dichotomy wave gap close).
|
||||
|
||||
Auto-imported via PYTHONPATH=scripts/to_freeze_seams:src by BOTH the truth
|
||||
generator and `freeze_truth.py --verify`, so the --verify law (replay
|
||||
against the pristine oracle) still holds: the seams below only make the
|
||||
oracle's already-existing behavior reachable with JSON-safe shapes.
|
||||
|
||||
* RUNE_TOKEN_OPTIMIZER_CORE=python + RUNE_TOKEN_CORE=python pinned (the
|
||||
oracle floor; freeze_truth's _pin_oracle_cores only reaches the
|
||||
TOKEN_OPTIMIZER gate — the tc gate is another module's pin).
|
||||
* RUNE_MEMORY_DIR pinned to a hermetic FIXED-path scratch
|
||||
(/tmp/to_gap_freeze — wiped at import; every case rewrites its own
|
||||
storage file, so no state leaks between cases).
|
||||
* Module seams bound onto rune.token_optimizer (marker shapes that drive
|
||||
the REAL facade fns and return JSON-safe observations):
|
||||
- to_get_optimizer_state(file_state, seed_text) — resets the global
|
||||
singleton, lays down the storage-file variant, calls the REAL
|
||||
get_optimizer(), returns the post-_load state + the singleton
|
||||
identity check. _last_save_time is EXCLUDED (monotonic clock —
|
||||
quirk law: no clock-tainted expected values).
|
||||
- to_to_state(cache_list, stats, dirty, changes, last_save) — drives
|
||||
the REAL TokenOptimizer._to_state on a __new__-built receiver.
|
||||
- to_apply_state(state_doc) — drives the REAL
|
||||
TokenOptimizer._apply_state, returns the post-call attrs.
|
||||
- to_write_snapshot_doc(snap) — drives the REAL
|
||||
TokenOptimizer._write_snapshot into the scratch and reads the file
|
||||
back (the conversion half is the kernel's _write_snapshot_doc).
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(os.environ.get("TO_FREEZE_REPO", "")) or Path.cwd()
|
||||
|
||||
os.environ["RUNE_TOKEN_OPTIMIZER_CORE"] = "python"
|
||||
os.environ["RUNE_TOKEN_CORE"] = "python"
|
||||
os.environ["RUNE_MEMORY_DIR"] = "/tmp/to_gap_freeze"
|
||||
|
||||
SCRATCH = os.environ["RUNE_MEMORY_DIR"]
|
||||
if os.path.isdir(SCRATCH):
|
||||
shutil.rmtree(SCRATCH)
|
||||
os.makedirs(SCRATCH, exist_ok=True)
|
||||
|
||||
if str(REPO / "src") not in sys.path:
|
||||
sys.path.insert(0, str(REPO / "src"))
|
||||
|
||||
import rune.token_optimizer as _to # noqa: E402
|
||||
|
||||
STORAGE = os.path.join(SCRATCH, "token_optimizer.json")
|
||||
SNAP_PATH = os.path.join(SCRATCH, "snap_probe.json")
|
||||
|
||||
|
||||
def _bare_optimizer(**attrs):
|
||||
opt = _to.TokenOptimizer.__new__(_to.TokenOptimizer)
|
||||
for k, v in attrs.items():
|
||||
setattr(opt, k, v)
|
||||
return opt
|
||||
|
||||
|
||||
def _state_view(opt):
|
||||
return {
|
||||
"cache": [e.to_dict() for e in opt.cache.values()],
|
||||
"stats": dict(opt.stats),
|
||||
"dirty": opt._dirty,
|
||||
"changes_since_save": opt._changes_since_save,
|
||||
}
|
||||
|
||||
|
||||
def _to_get_optimizer_state(file_state, seed_text):
|
||||
_to._optimizer = None
|
||||
if file_state == "absent":
|
||||
if os.path.exists(STORAGE):
|
||||
os.remove(STORAGE)
|
||||
else:
|
||||
with open(STORAGE, "w") as f:
|
||||
f.write(seed_text)
|
||||
try:
|
||||
opt = _to.get_optimizer()
|
||||
second = _to.get_optimizer()
|
||||
return {
|
||||
"storage_tail": os.path.basename(opt.storage_path),
|
||||
**_state_view(opt),
|
||||
"singleton_reused": second is opt,
|
||||
}
|
||||
finally:
|
||||
_to._optimizer = None
|
||||
if os.path.exists(STORAGE):
|
||||
os.remove(STORAGE)
|
||||
|
||||
|
||||
def _to_to_state(cache_list, stats, dirty, changes, last_save):
|
||||
opt = _bare_optimizer(
|
||||
cache={e["key"]: _to.CacheEntry(**e) for e in cache_list},
|
||||
stats=dict(stats),
|
||||
_dirty=dirty,
|
||||
_changes_since_save=changes,
|
||||
_last_save_time=last_save,
|
||||
)
|
||||
return _to.TokenOptimizer._to_state(opt)
|
||||
|
||||
|
||||
def _to_apply_state(state_doc):
|
||||
opt = _bare_optimizer(
|
||||
cache={}, stats={}, _dirty=False, _changes_since_save=0, _last_save_time=0.0
|
||||
)
|
||||
_to.TokenOptimizer._apply_state(opt, state_doc)
|
||||
return {**_state_view(opt), "last_save_time": opt._last_save_time}
|
||||
|
||||
|
||||
def _to_write_snapshot_doc(snap):
|
||||
opt = _bare_optimizer(storage_path=SNAP_PATH)
|
||||
_to.TokenOptimizer._write_snapshot(opt, snap)
|
||||
with open(SNAP_PATH) as f:
|
||||
doc = json.load(f)
|
||||
os.remove(SNAP_PATH)
|
||||
return doc
|
||||
|
||||
|
||||
_to.to_get_optimizer_state = _to_get_optimizer_state
|
||||
_to.to_to_state = _to_to_state
|
||||
_to.to_apply_state = _to_apply_state
|
||||
_to.to_write_snapshot_doc = _to_write_snapshot_doc
|
||||
Generated
+30
@@ -2,6 +2,34 @@
|
||||
# It is not intended for manual editing.
|
||||
version = 4
|
||||
|
||||
[[package]]
|
||||
name = "equivalent"
|
||||
version = "1.0.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "00d174d5400e5e8fd687ad1049e2f578285fa914201b1af7e8b112a4546bd826"
|
||||
|
||||
[[package]]
|
||||
name = "foldhash"
|
||||
version = "0.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb"
|
||||
|
||||
[[package]]
|
||||
name = "hashbrown"
|
||||
version = "0.17.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
|
||||
|
||||
[[package]]
|
||||
name = "indexmap"
|
||||
version = "2.14.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "cc4e190f5d26ca7051642629da2c52fc03bde85a03197c99408dcd291734c855"
|
||||
dependencies = [
|
||||
"equivalent",
|
||||
"hashbrown",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "itoa"
|
||||
version = "1.0.18"
|
||||
@@ -73,6 +101,8 @@ version = "1.0.154"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6"
|
||||
dependencies = [
|
||||
"foldhash",
|
||||
"indexmap",
|
||||
"itoa",
|
||||
"memchr",
|
||||
"serde",
|
||||
|
||||
@@ -8,6 +8,6 @@ description = "FULL parity port of src/rune/token_optimizer.py cache/state machi
|
||||
crate-type = ["rlib", "cdylib", "staticlib"]
|
||||
|
||||
[dependencies]
|
||||
serde_json = "1"
|
||||
serde_json = { version = "1", features = ["preserve_order"] }
|
||||
md5 = "0.7"
|
||||
token_counting = { path = "../token_counting" }
|
||||
@@ -0,0 +1,572 @@
|
||||
[
|
||||
{
|
||||
"fn": "_tc_core",
|
||||
"args": [],
|
||||
"kwargs": {},
|
||||
"expected": null
|
||||
},
|
||||
{
|
||||
"fn": "_to_core",
|
||||
"args": [],
|
||||
"kwargs": {},
|
||||
"expected": null
|
||||
},
|
||||
{
|
||||
"fn": "TokenOptimizer._run_kernel",
|
||||
"recv": {
|
||||
"cls": "TokenOptimizer"
|
||||
},
|
||||
"args": [
|
||||
"optimize_prompt",
|
||||
{}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"result": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "CacheEntry.to_dict",
|
||||
"recv": {
|
||||
"cls": "CacheEntry",
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2
|
||||
},
|
||||
"args": [],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"result": {
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2
|
||||
},
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_to_state",
|
||||
"args": [
|
||||
[
|
||||
{
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2
|
||||
},
|
||||
{
|
||||
"key": "0123456789abcdef",
|
||||
"content": "recent: pods restarted",
|
||||
"tokens": 5,
|
||||
"last_used": "2026-10-14T00:00:01+00:00",
|
||||
"use_count": 1,
|
||||
"hit_count": 0
|
||||
}
|
||||
],
|
||||
{
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1
|
||||
},
|
||||
true,
|
||||
7,
|
||||
99.25
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"cache": [
|
||||
{
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2
|
||||
},
|
||||
{
|
||||
"key": "0123456789abcdef",
|
||||
"content": "recent: pods restarted",
|
||||
"tokens": 5,
|
||||
"last_used": "2026-10-14T00:00:01+00:00",
|
||||
"use_count": 1,
|
||||
"hit_count": 0
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1
|
||||
},
|
||||
"dirty": true,
|
||||
"changes_since_save": 7,
|
||||
"last_save_time": 99.25
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_to_state",
|
||||
"args": [
|
||||
[],
|
||||
{
|
||||
"total_calls": 0,
|
||||
"total_tokens_saved": 0,
|
||||
"cache_hits": 0,
|
||||
"cache_misses": 0
|
||||
},
|
||||
false,
|
||||
0,
|
||||
0.0
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"cache": [],
|
||||
"stats": {
|
||||
"total_calls": 0,
|
||||
"total_tokens_saved": 0,
|
||||
"cache_hits": 0,
|
||||
"cache_misses": 0
|
||||
},
|
||||
"dirty": false,
|
||||
"changes_since_save": 0,
|
||||
"last_save_time": 0.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_apply_state",
|
||||
"args": [
|
||||
{
|
||||
"cache": [
|
||||
{
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2
|
||||
},
|
||||
{
|
||||
"key": "0123456789abcdef",
|
||||
"content": "recent: pods restarted",
|
||||
"tokens": 5,
|
||||
"last_used": "2026-10-14T00:00:01+00:00",
|
||||
"use_count": 1,
|
||||
"hit_count": 0
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1
|
||||
},
|
||||
"dirty": true,
|
||||
"changes_since_save": 7,
|
||||
"last_save_time": 99.25
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"cache": [
|
||||
{
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2
|
||||
},
|
||||
{
|
||||
"key": "0123456789abcdef",
|
||||
"content": "recent: pods restarted",
|
||||
"tokens": 5,
|
||||
"last_used": "2026-10-14T00:00:01+00:00",
|
||||
"use_count": 1,
|
||||
"hit_count": 0
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1
|
||||
},
|
||||
"dirty": true,
|
||||
"changes_since_save": 7,
|
||||
"last_save_time": 99.25
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_apply_state",
|
||||
"args": [
|
||||
{
|
||||
"cache": [],
|
||||
"stats": {
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1
|
||||
},
|
||||
"dirty": false,
|
||||
"changes_since_save": 0,
|
||||
"last_save_time": 1.5
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"cache": [],
|
||||
"stats": {
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1
|
||||
},
|
||||
"dirty": false,
|
||||
"changes_since_save": 0,
|
||||
"last_save_time": 1.5
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_apply_state",
|
||||
"args": [
|
||||
{
|
||||
"cache": [
|
||||
{
|
||||
"key": "req",
|
||||
"content": "required only entry",
|
||||
"tokens": 4,
|
||||
"last_used": "2026-10-14T00:00:02+00:00"
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1
|
||||
},
|
||||
"dirty": true,
|
||||
"changes_since_save": 2,
|
||||
"last_save_time": 3.25
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"cache": [
|
||||
{
|
||||
"key": "req",
|
||||
"content": "required only entry",
|
||||
"tokens": 4,
|
||||
"last_used": "2026-10-14T00:00:02+00:00",
|
||||
"use_count": 0,
|
||||
"hit_count": 0
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1
|
||||
},
|
||||
"dirty": true,
|
||||
"changes_since_save": 2,
|
||||
"last_save_time": 3.25
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_get_optimizer_state",
|
||||
"args": [
|
||||
"absent",
|
||||
""
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"storage_tail": "token_optimizer.json",
|
||||
"cache": [],
|
||||
"stats": {
|
||||
"total_calls": 0,
|
||||
"total_tokens_saved": 0,
|
||||
"cache_hits": 0,
|
||||
"cache_misses": 0
|
||||
},
|
||||
"dirty": false,
|
||||
"changes_since_save": 0,
|
||||
"singleton_reused": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_get_optimizer_state",
|
||||
"args": [
|
||||
"seeded",
|
||||
"{\"cache\": {\"static\": {\"key\": \"static\", \"content\": \"You are Rune. A stateful coding assistant.\", \"tokens\": 11, \"last_used\": \"2026-10-14T00:00:00+00:00\", \"use_count\": 3, \"hit_count\": 2}, \"0123456789abcdef\": {\"key\": \"0123456789abcdef\", \"content\": \"recent: pods restarted\", \"tokens\": 5, \"last_used\": \"2026-10-14T00:00:01+00:00\", \"use_count\": 1, \"hit_count\": 0}}, \"stats\": {\"total_calls\": 5, \"total_tokens_saved\": 42, \"cache_hits\": 2, \"cache_misses\": 1}}"
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"storage_tail": "token_optimizer.json",
|
||||
"cache": [
|
||||
{
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2
|
||||
},
|
||||
{
|
||||
"key": "0123456789abcdef",
|
||||
"content": "recent: pods restarted",
|
||||
"tokens": 5,
|
||||
"last_used": "2026-10-14T00:00:01+00:00",
|
||||
"use_count": 1,
|
||||
"hit_count": 0
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1
|
||||
},
|
||||
"dirty": false,
|
||||
"changes_since_save": 0,
|
||||
"singleton_reused": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_get_optimizer_state",
|
||||
"args": [
|
||||
"stats_only",
|
||||
"{\"stats\": {\"total_calls\": 9, \"total_tokens_saved\": 7, \"cache_hits\": 4, \"cache_misses\": 4}}"
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"storage_tail": "token_optimizer.json",
|
||||
"cache": [],
|
||||
"stats": {
|
||||
"total_calls": 9,
|
||||
"total_tokens_saved": 7,
|
||||
"cache_hits": 4,
|
||||
"cache_misses": 4
|
||||
},
|
||||
"dirty": false,
|
||||
"changes_since_save": 0,
|
||||
"singleton_reused": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_get_optimizer_state",
|
||||
"args": [
|
||||
"corrupt",
|
||||
"NOT JSON {{{"
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"storage_tail": "token_optimizer.json",
|
||||
"cache": [],
|
||||
"stats": {
|
||||
"total_calls": 0,
|
||||
"total_tokens_saved": 0,
|
||||
"cache_hits": 0,
|
||||
"cache_misses": 0
|
||||
},
|
||||
"dirty": false,
|
||||
"changes_since_save": 0,
|
||||
"singleton_reused": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_get_optimizer_state",
|
||||
"args": [
|
||||
"partial_bad",
|
||||
"{\"cache\": {\"static\": {\"key\": \"static\", \"content\": \"You are Rune. A stateful coding assistant.\", \"tokens\": 11, \"last_used\": \"2026-10-14T00:00:00+00:00\", \"use_count\": 3, \"hit_count\": 2}, \"bad\": {\"key\": \"bad\"}}, \"stats\": {\"total_calls\": 5, \"total_tokens_saved\": 42, \"cache_hits\": 2, \"cache_misses\": 1}}"
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"storage_tail": "token_optimizer.json",
|
||||
"cache": [
|
||||
{
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"total_calls": 0,
|
||||
"total_tokens_saved": 0,
|
||||
"cache_hits": 0,
|
||||
"cache_misses": 0
|
||||
},
|
||||
"dirty": false,
|
||||
"changes_since_save": 0,
|
||||
"singleton_reused": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_get_optimizer_state",
|
||||
"args": [
|
||||
"optional_fields",
|
||||
"{\"cache\": {\"req\": {\"key\": \"req\", \"content\": \"required only entry\", \"tokens\": 4, \"last_used\": \"2026-10-14T00:00:02+00:00\"}}, \"stats\": {\"total_calls\": 5, \"total_tokens_saved\": 42, \"cache_hits\": 2, \"cache_misses\": 1}}"
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"storage_tail": "token_optimizer.json",
|
||||
"cache": [
|
||||
{
|
||||
"key": "req",
|
||||
"content": "required only entry",
|
||||
"tokens": 4,
|
||||
"last_used": "2026-10-14T00:00:02+00:00",
|
||||
"use_count": 0,
|
||||
"hit_count": 0
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1
|
||||
},
|
||||
"dirty": false,
|
||||
"changes_since_save": 0,
|
||||
"singleton_reused": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_get_optimizer_state",
|
||||
"args": [
|
||||
"extra_field",
|
||||
"{\"cache\": {\"static\": {\"key\": \"static\", \"content\": \"You are Rune. A stateful coding assistant.\", \"tokens\": 11, \"last_used\": \"2026-10-14T00:00:00+00:00\", \"use_count\": 3, \"hit_count\": 2}, \"zz_bad\": {\"key\": \"zz_bad\", \"content\": \"x\", \"tokens\": 1, \"last_used\": \"t\", \"bogus\": true}}, \"stats\": {\"total_calls\": 5, \"total_tokens_saved\": 42, \"cache_hits\": 2, \"cache_misses\": 1}}"
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"storage_tail": "token_optimizer.json",
|
||||
"cache": [
|
||||
{
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"total_calls": 0,
|
||||
"total_tokens_saved": 0,
|
||||
"cache_hits": 0,
|
||||
"cache_misses": 0
|
||||
},
|
||||
"dirty": false,
|
||||
"changes_since_save": 0,
|
||||
"singleton_reused": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_get_optimizer_state",
|
||||
"args": [
|
||||
"notdict",
|
||||
"[1, 2, 3]"
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"storage_tail": "token_optimizer.json",
|
||||
"cache": [],
|
||||
"stats": {
|
||||
"total_calls": 0,
|
||||
"total_tokens_saved": 0,
|
||||
"cache_hits": 0,
|
||||
"cache_misses": 0
|
||||
},
|
||||
"dirty": false,
|
||||
"changes_since_save": 0,
|
||||
"singleton_reused": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_write_snapshot_doc",
|
||||
"args": [
|
||||
{
|
||||
"cache": [
|
||||
{
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2
|
||||
},
|
||||
{
|
||||
"key": "0123456789abcdef",
|
||||
"content": "recent: pods restarted",
|
||||
"tokens": 5,
|
||||
"last_used": "2026-10-14T00:00:01+00:00",
|
||||
"use_count": 1,
|
||||
"hit_count": 0
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"cache": {
|
||||
"static": {
|
||||
"key": "static",
|
||||
"content": "You are Rune. A stateful coding assistant.",
|
||||
"tokens": 11,
|
||||
"last_used": "2026-10-14T00:00:00+00:00",
|
||||
"use_count": 3,
|
||||
"hit_count": 2
|
||||
},
|
||||
"0123456789abcdef": {
|
||||
"key": "0123456789abcdef",
|
||||
"content": "recent: pods restarted",
|
||||
"tokens": 5,
|
||||
"last_used": "2026-10-14T00:00:01+00:00",
|
||||
"use_count": 1,
|
||||
"hit_count": 0
|
||||
}
|
||||
},
|
||||
"stats": {
|
||||
"total_calls": 5,
|
||||
"total_tokens_saved": 42,
|
||||
"cache_hits": 2,
|
||||
"cache_misses": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "to_write_snapshot_doc",
|
||||
"args": [
|
||||
{
|
||||
"cache": [],
|
||||
"stats": {}
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"cache": {},
|
||||
"stats": {}
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -10,7 +10,7 @@
|
||||
|
||||
use serde_json::{json, Map, Value};
|
||||
|
||||
use super::{entry_to_value, load_from_str, run_op, stats_to_value, State};
|
||||
use super::{get_optimizer, stats_to_value, State, Entry, _run_kernel, _write_snapshot_doc};
|
||||
|
||||
const TRUTH: &str = include_str!("frozen_to_truth.json");
|
||||
|
||||
@@ -35,7 +35,7 @@ fn masked_entry(v: &Value) -> Value {
|
||||
|
||||
fn state_obs(st: &State, file: &FileModel) -> Value {
|
||||
json!({
|
||||
"cache": st.cache.iter().map(|e| masked_entry(&entry_to_value(e))).collect::<Vec<_>>(),
|
||||
"cache": st.cache.iter().map(|e| masked_entry(&e.to_dict())).collect::<Vec<_>>(),
|
||||
"stats": stats_to_value(&st.stats),
|
||||
"dirty": st.dirty,
|
||||
"changes_since_save": st.changes_since_save,
|
||||
@@ -62,17 +62,6 @@ fn file_obs(file: &FileModel) -> Value {
|
||||
}
|
||||
}
|
||||
|
||||
/// Snapshot (transfer form: cache array) → on-disk form (cache object).
|
||||
fn file_doc_from_snapshot(s: &Value) -> Value {
|
||||
let mut cache = Map::new();
|
||||
for e in s.get("cache").and_then(Value::as_array).into_iter().flatten() {
|
||||
if let Some(k) = e.get("key").and_then(Value::as_str) {
|
||||
cache.insert(k.to_string(), e.clone());
|
||||
}
|
||||
}
|
||||
json!({"cache": cache, "stats": s.get("stats").cloned().unwrap_or(Value::Null)})
|
||||
}
|
||||
|
||||
/// State from the transfer form (cache as ordered array) — the on-disk dict
|
||||
/// preserves insertion order, so reloads must come from the ordered snapshot.
|
||||
fn state_from_transfer(s: &Value) -> State {
|
||||
@@ -92,7 +81,7 @@ fn state_from_transfer(s: &Value) -> State {
|
||||
|
||||
fn transfer_from_state(st: &State) -> Value {
|
||||
json!({
|
||||
"cache": st.cache.iter().map(entry_to_value).collect::<Vec<_>>(),
|
||||
"cache": st.cache.iter().map(Entry::to_dict).collect::<Vec<_>>(),
|
||||
"stats": stats_to_value(&st.stats),
|
||||
})
|
||||
}
|
||||
@@ -122,7 +111,7 @@ fn step(st: &mut State, op: &Value, now: f64, file: &mut FileModel, persona: &mu
|
||||
} else {
|
||||
match op.get("seed_file").and_then(Value::as_str) {
|
||||
Some(s) => {
|
||||
*st = load_from_str(s);
|
||||
*st = get_optimizer(Some(s));
|
||||
match serde_json::from_str::<Value>(s) {
|
||||
Ok(v) => {
|
||||
*file = FileModel::Doc(v.clone());
|
||||
@@ -150,7 +139,7 @@ fn step(st: &mut State, op: &Value, now: f64, file: &mut FileModel, persona: &mu
|
||||
st.last_save_time = now - elapsed;
|
||||
// the oracle returns None; the save flag is consumed by the
|
||||
// facade (the file obs below carries the effect)
|
||||
let _ = run_op(st, "save_if_dirty", &json!({}), "", now, &mut snap)
|
||||
let _ = _run_kernel(st, "save_if_dirty", &json!({}), "", now, &mut snap)
|
||||
.expect("save_if_dirty op failed");
|
||||
Value::Null
|
||||
}
|
||||
@@ -176,7 +165,7 @@ fn step(st: &mut State, op: &Value, now: f64, file: &mut FileModel, persona: &mu
|
||||
}),
|
||||
_ => json!({}),
|
||||
};
|
||||
let mut r = run_op(st, other, &args, "NOW", now, &mut snap).expect("op failed");
|
||||
let mut r = _run_kernel(st, other, &args, "NOW", now, &mut snap).expect("op failed");
|
||||
// the truth generator masks last_used inside get_cache_stats results
|
||||
if other == "get_cache_stats" {
|
||||
for e in r.as_array_mut().expect("stats array") {
|
||||
@@ -203,11 +192,11 @@ fn step(st: &mut State, op: &Value, now: f64, file: &mut FileModel, persona: &mu
|
||||
let doc = if kind == "flush" {
|
||||
let mut cache = Map::new();
|
||||
for e in &st.cache {
|
||||
cache.insert(e.key.clone(), entry_to_value(e));
|
||||
cache.insert(e.key.clone(), e.to_dict());
|
||||
}
|
||||
json!({"cache": cache, "stats": stats_to_value(&st.stats)})
|
||||
} else {
|
||||
file_doc_from_snapshot(snap.as_ref().unwrap())
|
||||
_write_snapshot_doc(snap.as_ref().unwrap())
|
||||
};
|
||||
*file = FileModel::Doc(doc);
|
||||
*saved = Some(if kind == "flush" {
|
||||
@@ -250,3 +239,145 @@ fn replay_frozen_truth_verbatim() {
|
||||
}
|
||||
assert!(checked >= 60, "battery shrank: {checked} observations replayed");
|
||||
}
|
||||
|
||||
// ── gap truth (dichotomy wave close-PARTIAL pass, 2026-10-14) ──────────────
|
||||
//
|
||||
// The gap freeze (scripts/gen_token_optimizer_truth.py under the
|
||||
// scripts/to_freeze_seams rig) — embedded via include_str! so the file and
|
||||
// the battery can never drift. Rows: the two probe pin contracts
|
||||
// (_tc_core/_to_core -> None -> python floor), the _run_kernel pin no-op,
|
||||
// CacheEntry.to_dict, the _to_state/_apply_state transfer shapes, the
|
||||
// get_optimizer construction E2E per storage-file variant (including the
|
||||
// blanket-except PARTIAL-adoption quirk), and the _write_snapshot file-doc
|
||||
// conversion. The singleton identity and file I/O halves are facade-side
|
||||
// by the port law — replayed end-to-end by
|
||||
// scripts/check_token_optimizer_parity.py's dual-mode LIVE layer.
|
||||
|
||||
const GAP_TRUTH: &str = include_str!("../frozen_truth.json");
|
||||
|
||||
#[cfg(test)]
|
||||
mod gap_battery {
|
||||
use super::*;
|
||||
use serde_json::json;
|
||||
|
||||
fn rows() -> Vec<Value> {
|
||||
serde_json::from_str(GAP_TRUTH).expect("gap truth parses")
|
||||
}
|
||||
|
||||
fn entry_of(v: &Value) -> Entry {
|
||||
Entry {
|
||||
key: v["key"].as_str().unwrap().to_string(),
|
||||
content: v["content"].as_str().unwrap().to_string(),
|
||||
tokens: v["tokens"].as_i64().unwrap(),
|
||||
last_used: v["last_used"].as_str().unwrap().to_string(),
|
||||
use_count: v.get("use_count").and_then(Value::as_i64).unwrap_or(0),
|
||||
hit_count: v.get("hit_count").and_then(Value::as_i64).unwrap_or(0),
|
||||
}
|
||||
}
|
||||
|
||||
fn state_obs_full(st: &State) -> Value {
|
||||
json!({
|
||||
"cache": st.cache.iter().map(Entry::to_dict).collect::<Vec<_>>(),
|
||||
"stats": stats_to_value(&st.stats),
|
||||
"dirty": st.dirty,
|
||||
"changes_since_save": st.changes_since_save,
|
||||
"last_save_time": st.last_save_time,
|
||||
})
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pin_contract_rows_freeze_none() {
|
||||
// RUNE_TOKEN_CORE / RUNE_TOKEN_OPTIMIZER_CORE=python -> the probes
|
||||
// return None (the python floor) and the delegation loop is a
|
||||
// no-op. The kernel cannot execute the probes (facade loader
|
||||
// seams); the frozen rows keep a facade that ignored a pin failing
|
||||
// live parity instead of silently passing.
|
||||
let rs = rows();
|
||||
for fn_name in ["_tc_core", "_to_core"] {
|
||||
let row = rs
|
||||
.iter()
|
||||
.find(|r| r["fn"].as_str() == Some(fn_name))
|
||||
.unwrap_or_else(|| panic!("{fn_name} pin row missing"));
|
||||
assert!(row["expected"].is_null(), "{fn_name} must freeze None");
|
||||
}
|
||||
let rk = rs
|
||||
.iter()
|
||||
.find(|r| r["fn"].as_str() == Some("TokenOptimizer._run_kernel"))
|
||||
.expect("_run_kernel pin row missing");
|
||||
assert!(rk["expected"]["result"].is_null(), "_run_kernel under the pin is a no-op");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn entry_to_dict_matches_frozen_oracle() {
|
||||
let mut replayed = 0;
|
||||
for row in rows().iter().filter(|r| r["fn"].as_str() == Some("CacheEntry.to_dict")) {
|
||||
let e = entry_of(&row["recv"]);
|
||||
assert_eq!(e.to_dict(), row["expected"]["result"], "to_dict diverged");
|
||||
replayed += 1;
|
||||
}
|
||||
assert_eq!(replayed, 1, "expected the 1 to_dict gap row");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn to_state_matches_frozen_oracle() {
|
||||
let mut replayed = 0;
|
||||
for row in rows().iter().filter(|r| r["fn"].as_str() == Some("to_to_state")) {
|
||||
let st = State {
|
||||
cache: row["args"][0].as_array().unwrap().iter().map(entry_of).collect(),
|
||||
stats: super::super::stats_from_value(&row["args"][1]),
|
||||
dirty: row["args"][2].as_bool().unwrap(),
|
||||
changes_since_save: row["args"][3].as_i64().unwrap(),
|
||||
last_save_time: row["args"][4].as_f64().unwrap(),
|
||||
};
|
||||
assert_eq!(st._to_state(), row["expected"], "_to_state diverged");
|
||||
replayed += 1;
|
||||
}
|
||||
assert_eq!(replayed, 2, "expected the 2 _to_state gap rows");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn apply_state_matches_frozen_oracle() {
|
||||
let mut replayed = 0;
|
||||
for row in rows().iter().filter(|r| r["fn"].as_str() == Some("to_apply_state")) {
|
||||
let st = State::_apply_state(&row["args"][0]).expect("transfer doc must apply");
|
||||
assert_eq!(state_obs_full(&st), row["expected"], "_apply_state diverged");
|
||||
replayed += 1;
|
||||
}
|
||||
assert_eq!(replayed, 3, "expected the 3 _apply_state gap rows");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn get_optimizer_construction_matches_frozen_oracle() {
|
||||
let mut replayed = 0;
|
||||
for row in rows().iter().filter(|r| r["fn"].as_str() == Some("to_get_optimizer_state")) {
|
||||
let file_state = row["args"][0].as_str().unwrap();
|
||||
let seed = row["args"][1].as_str().unwrap();
|
||||
let st = if file_state == "absent" {
|
||||
get_optimizer(None)
|
||||
} else {
|
||||
get_optimizer(Some(seed))
|
||||
};
|
||||
let got = json!({
|
||||
"storage_tail": "token_optimizer.json",
|
||||
"cache": st.cache.iter().map(Entry::to_dict).collect::<Vec<_>>(),
|
||||
"stats": stats_to_value(&st.stats),
|
||||
"dirty": st.dirty,
|
||||
"changes_since_save": st.changes_since_save,
|
||||
"singleton_reused": true,
|
||||
});
|
||||
assert_eq!(got, row["expected"], "get_optimizer({file_state}) diverged");
|
||||
replayed += 1;
|
||||
}
|
||||
assert_eq!(replayed, 8, "expected the 8 construction gap rows");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn write_snapshot_doc_matches_frozen_oracle() {
|
||||
let mut replayed = 0;
|
||||
for row in rows().iter().filter(|r| r["fn"].as_str() == Some("to_write_snapshot_doc")) {
|
||||
assert_eq!(_write_snapshot_doc(&row["args"][0]), row["expected"], "snapshot doc diverged");
|
||||
replayed += 1;
|
||||
}
|
||||
assert_eq!(replayed, 2, "expected the 2 _write_snapshot gap rows");
|
||||
}
|
||||
}
|
||||
@@ -27,15 +27,39 @@
|
||||
//! borrowed `Args<'a>` over the wire doc, the md5 dynamic key hexes only
|
||||
//! the first 8 digest bytes (no intermediate string), and the state parse
|
||||
//! is all-or-nothing via `collect::<Option<Vec<_>>>`.
|
||||
//!
|
||||
//! Facade seams (src/rune/token_optimizer.py — the I/O side, gap-close
|
||||
//! 2026-10-14; truth in frozen_truth.json via scripts/to_freeze_seams):
|
||||
//! * `_tc_core` / `_to_core` — the delegation probes. The RUNE_TOKEN_CORE
|
||||
//! and RUNE_TOKEN_OPTIMIZER_CORE=python pins are honored in the facade
|
||||
//! (env read per call); the gap truth freezes the pin contract (probe →
|
||||
//! None → python floor) and the battery asserts the rows, so a facade
|
||||
//! that ignored a pin fails live parity, not silently.
|
||||
//! * `get_optimizer` — the process-global singleton, the
|
||||
//! `memory_env.memory_dir()` default-path resolution and the storage
|
||||
//! file read stay facade; the CONSTRUCTION decision (the post-`_load`
|
||||
//! state with the blanket-except discard) is the kernel `get_optimizer`
|
||||
//! below, driven by the battery's scenario `new` op and the frozen E2E
|
||||
//! rows.
|
||||
//! * `_write_snapshot` — makedirs/tmp-file/`os.replace` I/O stays facade;
|
||||
//! the file-doc conversion for the bytes it writes is the kernel
|
||||
//! `_write_snapshot_doc`.
|
||||
//! * `_run_kernel` — the facade loop (envelope unwrap, snapshot write,
|
||||
//! `_apply_state` adoption); the op dispatch it drives is the kernel
|
||||
//! `_run_kernel` below, reached through the `to_run` FFI.
|
||||
//! * `_compress_context` — mirrored 1:1 as the delegation hop to the
|
||||
//! token_counting kernel (the SAME crate the oracle's tc_loader
|
||||
//! resolves); the compression math has no second implementation on
|
||||
//! either side.
|
||||
|
||||
use serde_json::{json, Value};
|
||||
use serde_json::{json, Map, Value};
|
||||
use token_counting::counting;
|
||||
|
||||
const SAVE_CHANGE_THRESHOLD: i64 = 10;
|
||||
const SAVE_DEBOUNCE_SECS: f64 = 30.0;
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
struct Entry {
|
||||
pub struct Entry {
|
||||
key: String,
|
||||
content: String,
|
||||
tokens: i64,
|
||||
@@ -53,7 +77,7 @@ struct Stats {
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
struct State {
|
||||
pub struct State {
|
||||
cache: Vec<Entry>,
|
||||
stats: Stats,
|
||||
dirty: bool,
|
||||
@@ -75,27 +99,43 @@ impl Default for State {
|
||||
|
||||
// ── state (de)serialization ─────────────────────────────────────────────
|
||||
|
||||
/// `CacheEntry(**fields)` kwargs semantics: key/content/tokens/last_used
|
||||
/// required, use_count/hit_count default to 0, and an unknown key is a
|
||||
/// TypeError (bad entry). JSON-typed values only — Python's dynamic typing
|
||||
/// can adopt e.g. a string `tokens` from a hand-written storage file; the
|
||||
/// typed kernel rejects that class (documented divergence, dynamic-typing
|
||||
/// family, same as the bool/int quirk).
|
||||
fn entry_from_value(v: &Value) -> Option<Entry> {
|
||||
let o = v.as_object()?;
|
||||
if o
|
||||
.keys()
|
||||
.any(|k| !matches!(k.as_str(), "key" | "content" | "tokens" | "last_used" | "use_count" | "hit_count"))
|
||||
{
|
||||
return None;
|
||||
}
|
||||
Some(Entry {
|
||||
key: o.get("key")?.as_str()?.to_string(),
|
||||
content: o.get("content")?.as_str()?.to_string(),
|
||||
tokens: o.get("tokens")?.as_i64()?,
|
||||
last_used: o.get("last_used")?.as_str()?.to_string(),
|
||||
use_count: o.get("use_count")?.as_i64()?,
|
||||
hit_count: o.get("hit_count")?.as_i64()?,
|
||||
use_count: o.get("use_count").and_then(Value::as_i64).unwrap_or(0),
|
||||
hit_count: o.get("hit_count").and_then(Value::as_i64).unwrap_or(0),
|
||||
})
|
||||
}
|
||||
|
||||
fn entry_to_value(e: &Entry) -> Value {
|
||||
json!({
|
||||
"key": e.key,
|
||||
"content": e.content,
|
||||
"tokens": e.tokens,
|
||||
"last_used": e.last_used,
|
||||
"use_count": e.use_count,
|
||||
"hit_count": e.hit_count,
|
||||
})
|
||||
impl Entry {
|
||||
/// `CacheEntry.to_dict` — the persisted/transfer projection (issue #770:
|
||||
/// `content` is required; omitting it made the JSON unloadable).
|
||||
fn to_dict(&self) -> Value {
|
||||
json!({
|
||||
"key": self.key,
|
||||
"content": self.content,
|
||||
"tokens": self.tokens,
|
||||
"last_used": self.last_used,
|
||||
"use_count": self.use_count,
|
||||
"hit_count": self.hit_count,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn stats_from_value(v: &Value) -> Stats {
|
||||
@@ -120,73 +160,78 @@ fn stats_to_value(s: &Stats) -> Value {
|
||||
})
|
||||
}
|
||||
|
||||
/// Facade state doc: cache as an ORDERED ARRAY (insertion order is load-
|
||||
/// bearing for `get_cache_stats`).
|
||||
fn parse_state(v: &Value) -> Result<State, String> {
|
||||
let o = v.as_object().ok_or("state must be an object")?;
|
||||
let mut cache = Vec::new();
|
||||
if let Some(arr) = o.get("cache").and_then(Value::as_array) {
|
||||
// all-or-nothing: one bad entry discards the whole state (the
|
||||
// oracle's blanket-except _load semantics)
|
||||
cache = arr
|
||||
.iter()
|
||||
.map(entry_from_value)
|
||||
.collect::<Option<Vec<_>>>()
|
||||
.ok_or("bad cache entry")?;
|
||||
impl State {
|
||||
/// `TokenOptimizer._apply_state` — rebuild the live state from the
|
||||
/// facade's transfer doc (cache as an ORDERED ARRAY; insertion order is
|
||||
/// load-bearing for `get_cache_stats`). All-or-nothing: one bad entry
|
||||
/// rejects the whole doc.
|
||||
fn _apply_state(v: &Value) -> Result<State, String> {
|
||||
let o = v.as_object().ok_or("state must be an object")?;
|
||||
let mut cache = Vec::new();
|
||||
if let Some(arr) = o.get("cache").and_then(Value::as_array) {
|
||||
// all-or-nothing: one bad entry discards the whole state (the
|
||||
// oracle's blanket-except _load semantics)
|
||||
cache = arr
|
||||
.iter()
|
||||
.map(entry_from_value)
|
||||
.collect::<Option<Vec<_>>>()
|
||||
.ok_or("bad cache entry")?;
|
||||
}
|
||||
Ok(State {
|
||||
cache,
|
||||
stats: o.get("stats").map(stats_from_value).unwrap_or_default(),
|
||||
dirty: o.get("dirty").and_then(Value::as_bool).unwrap_or(false),
|
||||
changes_since_save: o
|
||||
.get("changes_since_save")
|
||||
.and_then(Value::as_i64)
|
||||
.unwrap_or(0),
|
||||
last_save_time: o.get("last_save_time").and_then(Value::as_f64).unwrap_or(0.0),
|
||||
})
|
||||
}
|
||||
Ok(State {
|
||||
cache,
|
||||
stats: o.get("stats").map(stats_from_value).unwrap_or_default(),
|
||||
dirty: o.get("dirty").and_then(Value::as_bool).unwrap_or(false),
|
||||
changes_since_save: o
|
||||
.get("changes_since_save")
|
||||
.and_then(Value::as_i64)
|
||||
.unwrap_or(0),
|
||||
last_save_time: o.get("last_save_time").and_then(Value::as_f64).unwrap_or(0.0),
|
||||
})
|
||||
}
|
||||
|
||||
fn state_to_value(st: &State) -> Value {
|
||||
json!({
|
||||
"cache": st.cache.iter().map(entry_to_value).collect::<Vec<_>>(),
|
||||
"stats": stats_to_value(&st.stats),
|
||||
"dirty": st.dirty,
|
||||
"changes_since_save": st.changes_since_save,
|
||||
"last_save_time": st.last_save_time,
|
||||
})
|
||||
/// `TokenOptimizer._to_state` — the kernel transfer form; cache as an
|
||||
/// ORDERED ARRAY (insertion order is load-bearing for get_cache_stats).
|
||||
fn _to_state(&self) -> Value {
|
||||
json!({
|
||||
"cache": self.cache.iter().map(Entry::to_dict).collect::<Vec<_>>(),
|
||||
"stats": stats_to_value(&self.stats),
|
||||
"dirty": self.dirty,
|
||||
"changes_since_save": self.changes_since_save,
|
||||
"last_save_time": self.last_save_time,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Replicates the oracle's `_load` semantics over the FILE format (cache as
|
||||
/// a JSON object): one bad entry, a non-object cache, or corrupt JSON
|
||||
/// discards the ENTIRE persisted state — the blanket `except` recovery that
|
||||
/// issue #770 pinned. A missing stats block keeps the defaults.
|
||||
/// Battery-only: the production facade keeps the real `_load` (I/O side).
|
||||
#[cfg(test)]
|
||||
/// a JSON object, DOCUMENT order — preserve_order keeps the parser
|
||||
/// faithful to Python's dict order): corrupt JSON or a top-level non-object
|
||||
/// discards everything (blanket `except`, issue #770); inside the loop a
|
||||
/// bad entry raises mid-iteration and the blanket `except` adopts the
|
||||
/// PARTIAL cache built so far — the stats assignment (the next line) never
|
||||
/// runs, so stats keeps the constructor defaults. `.items()` on a
|
||||
/// non-object cache raises before any adoption. A missing stats block
|
||||
/// keeps the defaults.
|
||||
fn load_from_value(data: &Value) -> State {
|
||||
let mut st = State::default();
|
||||
let mut cache = Vec::new();
|
||||
match data.get("cache") {
|
||||
None => {}
|
||||
Some(Value::Object(m)) => {
|
||||
for (k, v) in m {
|
||||
let mut e = match entry_from_value(v) {
|
||||
Some(e) => e,
|
||||
None => return State::default(),
|
||||
let Some(mut e) = entry_from_value(v) else {
|
||||
return st; // CacheEntry(**fields) raised: partial cache, default stats
|
||||
};
|
||||
e.key = k.clone();
|
||||
cache.push(e);
|
||||
st.cache.push(e);
|
||||
}
|
||||
}
|
||||
Some(_) => return State::default(), // .items() on a non-dict raises
|
||||
Some(_) => {} // .items() on a non-dict raises: empty cache, default stats
|
||||
}
|
||||
st.cache = cache;
|
||||
if let Some(sv) = data.get("stats") {
|
||||
st.stats = stats_from_value(sv);
|
||||
}
|
||||
st
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn load_from_str(s: &str) -> State {
|
||||
match serde_json::from_str(s) {
|
||||
Ok(v) => load_from_value(&v),
|
||||
@@ -194,6 +239,19 @@ fn load_from_str(s: &str) -> State {
|
||||
}
|
||||
}
|
||||
|
||||
/// `get_optimizer` — the singleton's CONSTRUCTION decision: the state a
|
||||
/// fresh `TokenOptimizer` holds after `_load` over the facade-read storage
|
||||
/// file. `None` (absent file) → defaults; `Some(text)` parses with the
|
||||
/// oracle's blanket-except semantics (corrupt JSON → defaults; a bad
|
||||
/// cache entry mid-loop → the PARTIAL cache + default stats). The
|
||||
/// process-global singleton, the `memory_env.memory_dir()` default-path
|
||||
/// resolution and
|
||||
/// the file read stay facade (I/O law); the gap truth freezes the E2E and
|
||||
/// the battery replays the construction rows.
|
||||
pub fn get_optimizer(storage_text: Option<&str>) -> State {
|
||||
storage_text.map_or_else(State::default, load_from_str)
|
||||
}
|
||||
|
||||
// ── cache mechanics ─────────────────────────────────────────────────────
|
||||
|
||||
fn find(st: &State, key: &str) -> Option<usize> {
|
||||
@@ -239,11 +297,25 @@ fn maybe_save(st: &mut State, now: f64, snap: &mut Option<Value>) -> bool {
|
||||
/// facade turns into a dict: cache as an ordered array.
|
||||
fn snapshot_doc(st: &State) -> Value {
|
||||
json!({
|
||||
"cache": st.cache.iter().map(entry_to_value).collect::<Vec<_>>(),
|
||||
"cache": st.cache.iter().map(Entry::to_dict).collect::<Vec<_>>(),
|
||||
"stats": stats_to_value(&st.stats),
|
||||
})
|
||||
}
|
||||
|
||||
/// The pure half of the facade's `_write_snapshot`: the transfer-form
|
||||
/// snapshot (cache as an ORDERED ARRAY) → the on-disk file doc (cache as a
|
||||
/// JSON object keyed by entry key). The makedirs/tmp-file/`os.replace`
|
||||
/// write stays facade (I/O law).
|
||||
fn _write_snapshot_doc(snap: &Value) -> Value {
|
||||
let mut cache = Map::new();
|
||||
for e in snap.get("cache").and_then(Value::as_array).into_iter().flatten() {
|
||||
if let Some(k) = e.get("key").and_then(Value::as_str) {
|
||||
cache.insert(k.to_string(), e.clone());
|
||||
}
|
||||
}
|
||||
json!({"cache": cache, "stats": snap.get("stats").cloned().unwrap_or(Value::Null)})
|
||||
}
|
||||
|
||||
fn dyn_key(context: &str) -> String {
|
||||
// hex of the FIRST 8 md5 bytes only (16 hex chars, exactly the oracle's
|
||||
// hexdigest()[:16]) — no 32-char intermediate allocation.
|
||||
@@ -258,6 +330,14 @@ fn dyn_key(context: &str) -> String {
|
||||
|
||||
// ── oracle methods (pure cores) ─────────────────────────────────────────
|
||||
|
||||
/// `TokenOptimizer._compress_context` — the oracle's delegation hop to the
|
||||
/// token_counting kernel (the SAME crate the oracle's tc_loader resolves),
|
||||
/// mirrored 1:1 so the compression math has no second implementation on
|
||||
/// this side either.
|
||||
fn _compress_context(context: &str) -> String {
|
||||
counting::compress_context(context)
|
||||
}
|
||||
|
||||
/// `TokenOptimizer.get_static_cache` — hit branch increments the entry's
|
||||
/// hit_count and the stats counter; miss branch caches the (facade-resolved)
|
||||
/// persona text with the oracle's use_count=1/hit_count=0 stamps.
|
||||
@@ -292,7 +372,7 @@ fn get_static_cache(st: &mut State, persona: &str, now_iso: &str, now: f64, snap
|
||||
/// (the oracle has no hit branch here; repeats reset use/hit counts) and
|
||||
/// never touches the stats hit/miss counters. Faithful, including the quirk.
|
||||
fn get_dynamic_cache(st: &mut State, context: &str, now_iso: &str, now: f64, snap: &mut Option<Value>) -> String {
|
||||
let compressed = counting::compress_context(context);
|
||||
let compressed = _compress_context(context);
|
||||
let tokens = counting::estimate_tokens(&compressed);
|
||||
upsert(
|
||||
st,
|
||||
@@ -389,7 +469,7 @@ fn get_status(st: &State) -> Value {
|
||||
}
|
||||
|
||||
fn get_cache_stats(st: &State) -> Value {
|
||||
Value::Array(st.cache.iter().map(entry_to_value).collect())
|
||||
Value::Array(st.cache.iter().map(Entry::to_dict).collect())
|
||||
}
|
||||
|
||||
// ── dispatch ────────────────────────────────────────────────────────────
|
||||
@@ -401,6 +481,7 @@ fn get_cache_stats(st: &State) -> Value {
|
||||
/// until it is handled.
|
||||
#[derive(Clone, Copy, PartialEq, Eq)]
|
||||
enum Op {
|
||||
GetOptimizer,
|
||||
GetStaticCache,
|
||||
GetDynamicCache,
|
||||
OptimizePrompt,
|
||||
@@ -413,6 +494,7 @@ enum Op {
|
||||
impl Op {
|
||||
fn parse(s: &str) -> Result<Op, String> {
|
||||
match s {
|
||||
"get_optimizer" => Ok(Op::GetOptimizer),
|
||||
"get_static_cache" => Ok(Op::GetStaticCache),
|
||||
"get_dynamic_cache" => Ok(Op::GetDynamicCache),
|
||||
"optimize_prompt" => Ok(Op::OptimizePrompt),
|
||||
@@ -444,9 +526,11 @@ impl<'a> Args<'a> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Runs one op against the state. Returns the op VALUE; `snap` receives the
|
||||
/// file content to persist when the oracle would have written mid-call.
|
||||
fn run_op(
|
||||
/// `TokenOptimizer._run_kernel` — the op dispatch the facade's delegation
|
||||
/// loop drives (through the `to_run` FFI). Returns the op VALUE; `snap`
|
||||
/// receives the file content to persist when the oracle would have written
|
||||
/// mid-call.
|
||||
fn _run_kernel(
|
||||
st: &mut State,
|
||||
op: &str,
|
||||
args: &Value,
|
||||
@@ -457,6 +541,11 @@ fn run_op(
|
||||
let op = Op::parse(op)?;
|
||||
let a = Args::from_value(args);
|
||||
Ok(match op {
|
||||
Op::GetOptimizer => {
|
||||
let text = args.get("storage_text").and_then(Value::as_str);
|
||||
*st = get_optimizer(text);
|
||||
st._to_state()
|
||||
}
|
||||
Op::GetStaticCache => Value::String(get_static_cache(st, a.persona, now_iso, now, snap)),
|
||||
Op::GetDynamicCache => Value::String(get_dynamic_cache(st, a.context, now_iso, now, snap)),
|
||||
Op::OptimizePrompt => optimize_prompt(st, a.text, a.context, a.persona, now_iso, now, snap),
|
||||
@@ -488,7 +577,7 @@ fn envelope(r: Result<(Value, State, Option<Value>), String>) -> *mut c_char {
|
||||
Ok((value, st, snap)) => json!({
|
||||
"ok": true,
|
||||
"value": value,
|
||||
"state": state_to_value(&st),
|
||||
"state": st._to_state(),
|
||||
"save": snap.is_some(),
|
||||
"snapshot": snap.unwrap_or(Value::Null),
|
||||
}),
|
||||
@@ -507,13 +596,13 @@ fn envelope(r: Result<(Value, State, Option<Value>), String>) -> *mut c_char {
|
||||
#[no_mangle]
|
||||
pub extern "C" fn to_run(arg: *const c_char) -> *mut c_char {
|
||||
envelope(read_arg(arg).and_then(|a| {
|
||||
let mut st = parse_state(a.get("state").ok_or("missing state")?)?;
|
||||
let mut st = State::_apply_state(a.get("state").ok_or("missing state")?)?;
|
||||
let args = a.get("args").cloned().unwrap_or_else(|| json!({}));
|
||||
let now_iso = a.get("now_iso").and_then(Value::as_str).unwrap_or("");
|
||||
let now = a.get("now").and_then(Value::as_f64).unwrap_or(0.0);
|
||||
let op = a.get("op").and_then(Value::as_str).ok_or("missing op")?;
|
||||
let mut snap = None;
|
||||
let value = run_op(&mut st, op, &args, now_iso, now, &mut snap)?;
|
||||
let value = _run_kernel(&mut st, op, &args, now_iso, now, &mut snap)?;
|
||||
Ok((value, st, snap))
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1,58 @@
|
||||
# token_optimizer gap-close verification summary (dichotomy wave, worktree rune-wave-cg)
|
||||
|
||||
Module: token_optimizer — oracle /mnt/workdrive/rune/src/rune/token_optimizer.py (514 lines, 25 fns).
|
||||
Gap closed: 16/25 PARTIAL → **25/25 FULL** (get_optimizer, _tc_core, _to_core, to_dict,
|
||||
_to_state, _apply_state, _write_snapshot, _run_kernel, _compress_context).
|
||||
|
||||
Note: the assigned worktree was removed externally mid-session before any edit;
|
||||
recreated at the same path from fresh gitea/master (6208acdc) — zero work lost.
|
||||
|
||||
## Evidence (this checkout, all cross-checked via Read after capture)
|
||||
|
||||
| Gate | Command | Result | Evidence |
|
||||
|---|---|---|---|
|
||||
| Battery | `cargo test` (crate) | 7/7 (s1-s8 replay + 6 gap tests) | verification/to-battery.{log,exit} |
|
||||
| Truth freeze | `PYTHONHASHSEED=0 PYTHONPATH=scripts/to_freeze_seams:src python3 scripts/gen_token_optimizer_truth.py` | 19 cases; stable across 2 gen runs (diff clean) | verification/to_freeze_gen2.log |
|
||||
| Byte-identity | same env + `scripts/freeze_truth.py --verify src/rust/token_optimizer/frozen_truth.json --module rune.token_optimizer` | OK ×2 | verification/to-freeze-verify.{log,exit}, to_freeze_verify2.log |
|
||||
| Live parity | `python3 scripts/check_token_optimizer_parity.py` | **60/60** (frozen 10: 8 FFI construction rows + 2 pin contracts; fuzz 40: dual-mode full state machine, seed 2479; live 10: module API singleton + 8 construction variants + _write_snapshot) | verification/to-parity.{log,exit} |
|
||||
| Clippy | `cargo clippy --all-targets -- -D warnings` | clean | verification/to_clippy.log |
|
||||
| Build | `cargo build` + `cargo build --release` (both crates) | ok | verification/to_cargo_build.log |
|
||||
| Python live pin | `PYTHONPATH=src python3 -m pytest tests/test_token_optimizer_cache.py tests/test_token_optimizer_port.py` | 5/5 | verification/to_pytest_live.log |
|
||||
| Python pinned | same + `RUNE_TOKEN_OPTIMIZER_CORE=python` | 5/5 | verification/to_pytest_pinned.log |
|
||||
| Ruff | `ruff check` on the 3 new py files | clean | (inline, exit 0) |
|
||||
| Audit | `python3 scripts/dichotomy_audit.py --write` | token_optimizer 25/25 FULL | verification/to_audit.log |
|
||||
|
||||
Pre-existing, NOT caused by this diff: tests/test_rust_core_shims.py has 7 failures in
|
||||
this worktree for OTHER modules (drift/capability/spine/warm/construction_decision/
|
||||
memory_store/mr_loader loaders — their .so builds absent in the fresh worktree). Proven
|
||||
identical on the stashed clean base (7 failed, 15 passed, 18 skipped); the diff touches
|
||||
no Python source. token_optimizer-relevant shims passed.
|
||||
|
||||
## Divergences (documented, not bugs)
|
||||
|
||||
1. **Dynamic-typing class** (same family as the bool/int quirk): a hand-written storage
|
||||
file entry with e.g. a string `tokens` is adopted by Python's untyped
|
||||
`CacheEntry(**fields)`; the typed kernel rejects it as a bad entry (→ partial
|
||||
adoption). Kernel-typed rows freeze the contract branches; crate header documents.
|
||||
Evidence: src/rust/token_optimizer/src/lib.rs entry_from_value doc.
|
||||
2. **`_write_snapshot` malformed input**: the oracle raises KeyError on a snapshot
|
||||
missing cache/stats (unreachable — snapshots always come from the kernel's
|
||||
snapshot_doc); the kernel conversion is tolerant. Not freezable (oracle crash);
|
||||
documented in scripts/gen_token_optimizer_truth.py case map.
|
||||
|
||||
## Quirk discovered and pinned (NEW)
|
||||
|
||||
Oracle `_load` blanket-except does NOT roll back: a bad cache entry mid-loop leaves the
|
||||
PARTIAL cache adopted (entries before the bad one, in file document order) and skips the
|
||||
stats assignment. Frozen as the partial_bad/extra_field rows; kernel load_from_value
|
||||
mirrors (serde_json preserve_order enabled for document-order iteration — token_counting
|
||||
is zero-dep so no feature unification risk).
|
||||
|
||||
## Rig/seams
|
||||
|
||||
- scripts/to_freeze_seams/sitecustomize.py — pins both cores python, hermetic
|
||||
/tmp/to_gap_freeze, JSON-safe seams driving the REAL facade fns.
|
||||
- scripts/gen_token_optimizer_truth.py — 19-case gap truth via the shared harness.
|
||||
- scripts/check_token_optimizer_parity.py — frozen/FFI + fuzz + dual-mode live layers.
|
||||
|
||||
SHAs: captured post-push in the completion report (rebase rewrites).
|
||||
Reference in new issue
Block a user