feat(model_registry): close PARTIAL gaps — dichotomy wave
2/8 -> 8/8 FULL: to_dict/get_profile/get_enabled_profiles/
get_profiles_by_tier/list_profiles/test_registry ported name-preserving
onto the existing get_cheapest_capable/resolve_context_budget surface.
KNOWN FLAG FIXED (model_connect, 2026-10-14): batch-67 hardcoded
toolReserve=2048 and a single outputReserve; the live oracle holds
TOOL_RESULT_RESERVE=1024 and the OUTPUT_RESERVE per-task-kind ladder
{default 2048, verify 512, summarize 1024, embed 0} — gated by LOCAL,
not tier (legacy truth.json never noticed: both its budget rows are
local models).
Truth: 35-case envelope via the shared harness + scripts/mreg_freeze_seams
(JSON-safe seam wrappers; pairs carry registry insertion order —
key-vs-attribute lookup, first-id-match, stable ties, falsy->DEFAULT ctor
law all pinned; the (or {}) empty branch pinned via a __new__ seam).
--verify byte-identical + fresh regen byte-identical. Battery 4 tests +
binary replay-gap 35/35 (+ legacy replay 4/4 regression); live parity
4/4 scenarios (36 ops, scripts/check_model_registry_parity.py);
gofmt/vet/go build clean; Python suite green both pins + oracle smoke.
No loader seam exists (RUNE pin vacuous, documented). Audit:
model_registry 8/8 GO; programme 55% (105/192).
This commit is contained in:
1 parent
80853ece02
commit
268aa4b332
17 files changed
+2766
-99
No files matched your search
@@ -2,14 +2,14 @@
|
||||
|
||||
_Regenerate with `python3 scripts/dichotomy_audit.py --write`. A module is FULL only when 80%+ of its Python functions have name-preserving counterparts in the Go/Rust implementation — directories without the functions are facades and count as NOT TOUCHED._
|
||||
|
||||
**FULL completion: 55%** (106/192 runtime modules at FULL)
|
||||
**FULL completion: 56%** (107/192 runtime modules at FULL)
|
||||
|
||||
Function-level detail: 71% (1619/2284 runtime functions matched) — secondary; includes in-flight work
|
||||
Function-level detail: 71% (1625/2284 runtime functions matched) — secondary; includes in-flight work
|
||||
|
||||
| Status | Modules | Lines | Meaning |
|
||||
|---|---|---|---|
|
||||
| FULL | 106 | 44949 | 80%+ of functions ported |
|
||||
| PARTIAL | 47 | 18569 | 20–80% — real logic exists, gaps named below |
|
||||
| FULL | 107 | 45202 | 80%+ of functions ported |
|
||||
| PARTIAL | 46 | 18316 | 20–80% — real logic exists, gaps named below |
|
||||
| NOT TOUCHED | 39 | 11471 | no name-preserving implementation |
|
||||
| Total runtime | 192 | 74989 | |
|
||||
|
||||
@@ -71,7 +71,6 @@ Function-level detail: 71% (1619/2284 runtime functions matched) — secondary;
|
||||
| audit_ledger | 212 | 2/8 | _compute_hash, get_entries, log_promotion, log_gate_result, log_drift_alert, main |
|
||||
| context_pressure | 262 | 3/9 | log_measurement, read_log, log_stats, _now, handle_hook, main |
|
||||
| core_server | 234 | 6/12 | make_server, log_message, _send_json, _send_ndjson, do_GET, do_POST |
|
||||
| model_registry | 253 | 2/8 | test_registry, to_dict, get_profile, get_enabled_profiles, get_profiles_by_tier, list_profiles |
|
||||
| progressive_escalation | 279 | 2/8 | execute_with_escalation, test_escalation, handle_hook, to_dict, execute, verify |
|
||||
| voicebox | 164 | 5/11 | _run, start_recording, stop_recording, _transcribe_and_type, on_press, on_release |
|
||||
| persona_definition | 210 | 2/7 | _by_topic, compile_full, emit_constitution, emit_identity, emit_persona |
|
||||
@@ -188,6 +187,7 @@ Function-level detail: 71% (1619/2284 runtime functions matched) — secondary;
|
||||
| spine_language | 269 | 5/5 | RUST |
|
||||
| construction_patterns_builtin | 266 | 1/1 | RUST |
|
||||
| iteration | 261 | 7/8 | RUST |
|
||||
| model_registry | 253 | 8/8 | GO |
|
||||
| openai_compat_transport | 227 | 9/9 | GO |
|
||||
| evidence_probe | 213 | 9/9 | GO |
|
||||
| project_registry | 205 | 10/12 | RUST |
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
#!/usr/bin/env python3
|
||||
"""check_model_registry_parity.py — LIVE parity for the model_registry GO
|
||||
port (#2479 dichotomy gap-close): drives the Python oracle
|
||||
(src/rune/model_registry.py, through the scripts/mreg_freeze_seams seam
|
||||
wrappers) and the GO binary through IDENTICAL ops and diffs their JSON
|
||||
outputs — beyond frozen-truth replay, this exercises the two
|
||||
implementations LIVE against arbitrary inputs, including the corrected
|
||||
TOOL_RESULT_RESERVE=1024 (the batch-67 port shipped 2048).
|
||||
|
||||
Usage (the seam rig must bind the wrappers — PYTHONPATH carries it):
|
||||
cd src/go/model-registry && go build -o /tmp/model-registry .
|
||||
PYTHONPATH=scripts/mreg_freeze_seams:src \
|
||||
python3 scripts/check_model_registry_parity.py /tmp/model-registry
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(REPO / "src"))
|
||||
|
||||
# RUNE_*_CORE=python pin: model_registry has NO loader seam (pure Python)
|
||||
# — the env is set for uniformity with the other parity checkers; the
|
||||
# oracle ignores it (documented vacuous pin).
|
||||
os.environ.setdefault("RUNE_MODEL_REGISTRY_CORE", "python")
|
||||
|
||||
import rune.model_registry as mr # noqa: E402
|
||||
|
||||
if not hasattr(mr, "seam_get_profile"):
|
||||
sys.exit("seam rig not bound — run with PYTHONPATH=scripts/mreg_freeze_seams:src")
|
||||
|
||||
T0FULL = {
|
||||
"id": "deterministic", "provider": "local", "tier": "T0",
|
||||
"local": True, "context_window": 0,
|
||||
"supports_tools": False, "supports_structured_output": False,
|
||||
"supports_reasoning_field": False,
|
||||
"estimated_latency": "instant", "estimated_cost": "0", "enabled": True,
|
||||
}
|
||||
T1FULL = {
|
||||
"id": "qwen2.5:3b", "provider": "ollama", "tier": "T1",
|
||||
"local": True, "context_window": 32768,
|
||||
"supports_tools": False, "supports_structured_output": False,
|
||||
"supports_reasoning_field": False,
|
||||
"estimated_latency": "low", "estimated_cost": "0", "enabled": True,
|
||||
}
|
||||
T2FULL = {
|
||||
"id": "qwen3:14b", "provider": "ollama", "tier": "T2",
|
||||
"local": True, "context_window": 32768,
|
||||
"supports_tools": False, "supports_structured_output": False,
|
||||
"supports_reasoning_field": False,
|
||||
"estimated_latency": "medium", "estimated_cost": "0", "enabled": True,
|
||||
}
|
||||
T4FULL = {
|
||||
"id": "mimo-v2.5-pro", "provider": "openai", "tier": "T4",
|
||||
"local": False, "context_window": 128000,
|
||||
"supports_tools": True, "supports_structured_output": True,
|
||||
"supports_reasoning_field": True,
|
||||
"estimated_latency": "higher", "estimated_cost": "paid", "enabled": True,
|
||||
}
|
||||
|
||||
|
||||
def tweak(base, **kw):
|
||||
p = dict(base)
|
||||
p.update(kw)
|
||||
return p
|
||||
|
||||
|
||||
STATE_H = [["T1", T1FULL],
|
||||
["X0", tweak(T0FULL, id="x0", context_window=4096, enabled=False,
|
||||
estimated_latency="low")],
|
||||
["T2", T2FULL]]
|
||||
STATE_FLOOR = [["tiny", tweak(T4FULL, id="tiny-cloud", tier="T9",
|
||||
context_window=1100, estimated_latency="low")]]
|
||||
STATE_DUP = [
|
||||
["fake", tweak(T4FULL, local=True, provider="ollama", tier="T4",
|
||||
context_window=32768, estimated_latency="medium")],
|
||||
["real", T4FULL],
|
||||
]
|
||||
STATE_CROSS = [["K1", tweak(T1FULL, id="p1", tier="T9")]]
|
||||
STATE_ODD = [["T4", T4FULL],
|
||||
["TX", tweak(T2FULL, id="odd", tier="TX", supports_tools=True)]]
|
||||
STATE_ODD_R = [["TX", tweak(T2FULL, id="odd", tier="TX", supports_tools=True)],
|
||||
["T4", T4FULL]]
|
||||
STATE_TIE = [["TA", tweak(T2FULL, id="first")],
|
||||
["TB", tweak(T2FULL, id="second")]]
|
||||
STATE_ALL_OFF = [["T1", tweak(T1FULL, enabled=False)],
|
||||
["T2", tweak(T2FULL, enabled=False)]]
|
||||
|
||||
|
||||
def run_python(ops):
|
||||
steps = []
|
||||
for op in ops:
|
||||
k = op["op"]
|
||||
state = op.get("state")
|
||||
if k == "to_dict":
|
||||
r = mr.seam_to_dict(state)
|
||||
elif k == "get_profile":
|
||||
r = mr.seam_get_profile(state, op["tier"])
|
||||
elif k == "get_enabled_profiles":
|
||||
r = mr.seam_get_enabled_profiles(state)
|
||||
elif k == "get_profiles_by_tier":
|
||||
r = mr.seam_get_profiles_by_tier(state, op["tier"])
|
||||
elif k == "get_cheapest_capable":
|
||||
r = mr.seam_get_cheapest_capable(
|
||||
state, op["tools"], op["structured"], op["reasoning"])
|
||||
elif k == "list_profiles":
|
||||
r = mr.seam_list_profiles(state)
|
||||
elif k == "resolve":
|
||||
r = mr.seam_resolve_budget(op["model"], op["kind"], None)
|
||||
elif k == "resolve_custom":
|
||||
# RAW registry (no ctor fallback) — matches the binary's
|
||||
# resolve_custom op (registryFromRaw semantics)
|
||||
reg = mr.ModelRegistry.__new__(mr.ModelRegistry)
|
||||
reg.profiles = {key: mr.ModelProfile(**p) for key, p in state}
|
||||
r = mr.resolve_context_budget(op["model"], op["kind"], reg)
|
||||
elif k == "test_registry":
|
||||
r = mr.seam_test_registry()
|
||||
else:
|
||||
raise ValueError(k)
|
||||
steps.append({"op": k, "result": r})
|
||||
return {"steps": steps}
|
||||
|
||||
|
||||
def run_go(binary, ops_path):
|
||||
r = subprocess.run([binary, "scenario", str(ops_path)],
|
||||
capture_output=True, text=True, timeout=120)
|
||||
if r.returncode != 0:
|
||||
raise RuntimeError(f"go scenario failed: {r.stderr}")
|
||||
return json.loads(r.stdout)
|
||||
|
||||
|
||||
SCENARIOS = [
|
||||
# 1: profile surface — to_dict, listings (order-proof), key-vs-attr
|
||||
{"ops": [
|
||||
{"op": "to_dict", "state": T0FULL},
|
||||
{"op": "to_dict", "state": T4FULL},
|
||||
{"op": "list_profiles", "state": []},
|
||||
{"op": "list_profiles", "state": STATE_H},
|
||||
{"op": "get_profile", "tier": "T1"},
|
||||
{"op": "get_profile", "tier": "TX"},
|
||||
{"op": "get_profile", "state": STATE_CROSS, "tier": "K1"},
|
||||
{"op": "get_profile", "state": STATE_CROSS, "tier": "T9"},
|
||||
{"op": "get_enabled_profiles", "state": []},
|
||||
{"op": "get_enabled_profiles", "state": STATE_H},
|
||||
{"op": "get_profiles_by_tier", "tier": "T2"},
|
||||
{"op": "get_profiles_by_tier", "state": STATE_CROSS, "tier": "T9"},
|
||||
{"op": "get_profiles_by_tier", "state": STATE_H, "tier": "T2"},
|
||||
]},
|
||||
# 2: cheapest matrix + odd-tier/stable-tie/disabled branches
|
||||
{"ops": [
|
||||
{"op": "get_cheapest_capable", "state": [],
|
||||
"tools": False, "structured": False, "reasoning": False},
|
||||
{"op": "get_cheapest_capable", "state": [],
|
||||
"tools": True, "structured": False, "reasoning": False},
|
||||
{"op": "get_cheapest_capable", "state": [],
|
||||
"tools": False, "structured": True, "reasoning": False},
|
||||
{"op": "get_cheapest_capable", "state": [],
|
||||
"tools": False, "structured": False, "reasoning": True},
|
||||
{"op": "get_cheapest_capable", "state": [],
|
||||
"tools": True, "structured": True, "reasoning": True},
|
||||
{"op": "get_cheapest_capable", "state": STATE_ODD,
|
||||
"tools": True, "structured": False, "reasoning": False},
|
||||
{"op": "get_cheapest_capable", "state": STATE_ODD_R,
|
||||
"tools": True, "structured": False, "reasoning": False},
|
||||
{"op": "get_cheapest_capable", "state": STATE_TIE,
|
||||
"tools": False, "structured": False, "reasoning": False},
|
||||
{"op": "get_cheapest_capable", "state": STATE_ALL_OFF,
|
||||
"tools": False, "structured": False, "reasoning": False},
|
||||
]},
|
||||
# 3: budget sweep — default registry × kinds + the FLAG FIX rows
|
||||
{"ops": [
|
||||
{"op": "resolve", "model": "", "kind": "default"},
|
||||
{"op": "resolve", "model": "", "kind": "embed"},
|
||||
{"op": "resolve", "model": "qwen2.5:3b", "kind": "verify"},
|
||||
{"op": "resolve", "model": "qwen3:14b", "kind": "summarize"},
|
||||
{"op": "resolve", "model": "mimo-v2.5-pro", "kind": "default"},
|
||||
{"op": "resolve", "model": "mimo-v2.5-pro", "kind": "summarize"},
|
||||
{"op": "resolve", "model": "mimo-v2.5-pro", "kind": "embed"},
|
||||
{"op": "resolve", "model": "deterministic", "kind": "default"},
|
||||
{"op": "resolve", "model": "no-such-model", "kind": "weirdkind"},
|
||||
]},
|
||||
# 4: custom registries — floor, duplicate-id first-match, empty raw,
|
||||
# ctor-falsy default (through the seam = ctor path)
|
||||
{"ops": [
|
||||
{"op": "resolve_custom", "state": STATE_FLOOR,
|
||||
"model": "tiny-cloud", "kind": "default"},
|
||||
{"op": "resolve_custom", "state": STATE_DUP,
|
||||
"model": "mimo-v2.5-pro", "kind": "default"},
|
||||
{"op": "resolve_custom", "state": [],
|
||||
"model": "ghost", "kind": "verify"},
|
||||
{"op": "resolve", "model": "qwen2.5:3b", "kind": "verify"},
|
||||
{"op": "test_registry"},
|
||||
]},
|
||||
]
|
||||
|
||||
|
||||
def main():
|
||||
binary = sys.argv[1] if len(sys.argv) > 1 else "/tmp/model-registry"
|
||||
ok = 0
|
||||
for i, sc in enumerate(SCENARIOS, 1):
|
||||
with open("/tmp/mreg_scenario.json", "w") as f:
|
||||
json.dump(sc, f)
|
||||
py = run_python(sc["ops"])
|
||||
go = run_go(binary, "/tmp/mreg_scenario.json")
|
||||
if py == go:
|
||||
ok += 1
|
||||
print(f"scenario {i}: LIVE PARITY OK ({len(sc['ops'])} ops)")
|
||||
else:
|
||||
print(f"scenario {i}: LIVE PARITY FAILED")
|
||||
pj = json.dumps(py, indent=1, sort_keys=True)
|
||||
gj = json.dumps(go, indent=1, sort_keys=True)
|
||||
for pl, gl in zip(pj.splitlines(), gj.splitlines()):
|
||||
if pl != gl:
|
||||
print(f" py: {pl}")
|
||||
print(f" go: {gl}")
|
||||
break
|
||||
return 1
|
||||
print(f"LIVE PARITY: {ok}/{len(SCENARIOS)} scenarios identical (oracle vs GO)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,185 @@
|
||||
#!/usr/bin/env python3
|
||||
"""gen_model_registry_gap_truth.py — model_registry gap-closing truth
|
||||
battery (dichotomy wave #2479 gap port).
|
||||
|
||||
Closes the PARTIAL audit gaps: to_dict, get_profile, get_enabled_profiles,
|
||||
get_profiles_by_tier, list_profiles, test_registry (get_cheapest_capable
|
||||
and resolve_context_budget existed; resolve's constants were WRONG in the
|
||||
2026-10-09 Go port — toolReserve 2048 vs the oracle's TOOL_RESULT_RESERVE
|
||||
1024 — and task_kind was missing entirely; the frozen rows pin the live
|
||||
constants).
|
||||
|
||||
Uses the SHARED harness (scripts/freeze_truth.py) through the
|
||||
scripts/mreg_freeze_seams rig (auto-imported sitecustomize that binds
|
||||
JSON-safe seam_* wrappers onto the oracle module — registry methods
|
||||
return ModelProfile dataclass instances, which the harness cannot
|
||||
serialize). The oracle is PURE Python: no loader seam (RUNE pin vacuous),
|
||||
no clock, no HOME — envelope frozen_now is conventional only.
|
||||
|
||||
Registry states ride as ORDERED [tier_key, profile_dict] PAIRS (dict
|
||||
insertion order is load-bearing at three points: get_profile is a KEY
|
||||
lookup while get_profiles_by_tier matches the tier ATTRIBUTE;
|
||||
resolve_context_budget breaks on the FIRST id match; list_profiles
|
||||
returns values in insertion order). Receiver-style truth is NOT possible
|
||||
for this module — resolve_context_budget is module-level, so the shared
|
||||
harness's fn(inst, ...) call would bind the instance to model_id; the
|
||||
raw-receiver branch is pinned through a __new__ seam instead.
|
||||
|
||||
Regenerate + re-verify (byte-identical law — two runs, then verify):
|
||||
PYTHONPATH=scripts/mreg_freeze_seams:src python3 scripts/gen_model_registry_gap_truth.py
|
||||
PYTHONPATH=scripts/mreg_freeze_seams:src python3 scripts/freeze_truth.py \
|
||||
--verify src/go/model-registry/truth_gap.json \
|
||||
--module rune.model_registry
|
||||
|
||||
Usage: python3 scripts/gen_model_registry_gap_truth.py (from repo root)
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(REPO / "scripts"))
|
||||
sys.path.insert(0, str(REPO / "src"))
|
||||
|
||||
from freeze_truth import freeze, write_envelope # noqa: E402 (shared harness)
|
||||
|
||||
FROZEN_NOW = "2026-10-14T09:00:00" # vacuous pin (clock-free oracle)
|
||||
OUT = REPO / "src" / "go" / "model-registry" / "truth_gap.json"
|
||||
|
||||
# ── profile shapes (to_dict field order, all 11 fields explicit) ──────
|
||||
T0FULL = {
|
||||
"id": "deterministic", "provider": "local", "tier": "T0",
|
||||
"local": True, "context_window": 0,
|
||||
"supports_tools": False, "supports_structured_output": False,
|
||||
"supports_reasoning_field": False,
|
||||
"estimated_latency": "instant", "estimated_cost": "0", "enabled": True,
|
||||
}
|
||||
T1FULL = {
|
||||
"id": "qwen2.5:3b", "provider": "ollama", "tier": "T1",
|
||||
"local": True, "context_window": 32768,
|
||||
"supports_tools": False, "supports_structured_output": False,
|
||||
"supports_reasoning_field": False,
|
||||
"estimated_latency": "low", "estimated_cost": "0", "enabled": True,
|
||||
}
|
||||
T2FULL = {
|
||||
"id": "qwen3:14b", "provider": "ollama", "tier": "T2",
|
||||
"local": True, "context_window": 32768,
|
||||
"supports_tools": False, "supports_structured_output": False,
|
||||
"supports_reasoning_field": False,
|
||||
"estimated_latency": "medium", "estimated_cost": "0", "enabled": True,
|
||||
}
|
||||
T4FULL = {
|
||||
"id": "mimo-v2.5-pro", "provider": "openai", "tier": "T4",
|
||||
"local": False, "context_window": 128000,
|
||||
"supports_tools": True, "supports_structured_output": True,
|
||||
"supports_reasoning_field": True,
|
||||
"estimated_latency": "higher", "estimated_cost": "paid",
|
||||
"enabled": True,
|
||||
}
|
||||
|
||||
|
||||
def tweak(base, **kw):
|
||||
p = dict(base)
|
||||
p.update(kw)
|
||||
return p
|
||||
|
||||
|
||||
# ── custom registry states ([tier_key, profile_dict] pairs, ordered) ──
|
||||
STATE_FLOOR = [["tiny", tweak(T4FULL, id="tiny-cloud", tier="T9",
|
||||
context_window=1100,
|
||||
estimated_latency="low")]]
|
||||
STATE_DUP = [
|
||||
["fake", tweak(T4FULL, local=True, provider="ollama", tier="T4",
|
||||
context_window=32768, estimated_latency="medium")],
|
||||
["real", T4FULL],
|
||||
]
|
||||
STATE_H = [["T1", T1FULL],
|
||||
["X0", tweak(T0FULL, id="x0", context_window=4096, enabled=False,
|
||||
estimated_latency="low")],
|
||||
["T2", T2FULL]]
|
||||
STATE_CROSS = [["K1", tweak(T1FULL, id="p1", tier="T9")]]
|
||||
STATE_ODD = [["T4", T4FULL],
|
||||
["TX", tweak(T2FULL, id="odd", tier="TX", supports_tools=True)]]
|
||||
STATE_ODD_R = [["TX", tweak(T2FULL, id="odd", tier="TX", supports_tools=True)],
|
||||
["T4", T4FULL]]
|
||||
STATE_TIE = [["TA", tweak(T2FULL, id="first")], ["TB", tweak(T2FULL, id="second")]]
|
||||
STATE_ALL_OFF = [["T1", tweak(T1FULL, enabled=False)],
|
||||
["T2", tweak(T2FULL, enabled=False)]]
|
||||
|
||||
|
||||
def main():
|
||||
cases = []
|
||||
|
||||
# ── resolve_context_budget, registry=None (uses the default registry)
|
||||
cases.append(("resolve_context_budget", ("",)))
|
||||
cases.append(("resolve_context_budget", ("", "embed")))
|
||||
cases.append(("resolve_context_budget", ("qwen2.5:3b", "verify")))
|
||||
cases.append(("resolve_context_budget", ("qwen3:14b", "summarize")))
|
||||
# THE KNOWN FLAG (model_connect agent, 2026-10-14): the old Go port
|
||||
# hardcoded toolReserve=2048; the oracle holds 1024 (and the local
|
||||
# flag, not the tier, gates it). These rows pin the live constants.
|
||||
cases.append(("resolve_context_budget", ("mimo-v2.5-pro",)))
|
||||
cases.append(("resolve_context_budget", ("mimo-v2.5-pro", "summarize")))
|
||||
cases.append(("resolve_context_budget", ("mimo-v2.5-pro", "embed")))
|
||||
# T0 window 0 → `context_window or 8192`
|
||||
cases.append(("resolve_context_budget", ("deterministic",)))
|
||||
# unknown task_kind → default reserve
|
||||
cases.append(("resolve_context_budget", ("no-such-model", "weirdkind")))
|
||||
|
||||
# ── RAW registry (via __new__, no ctor fallback): empty profiles
|
||||
# reaches the `(reg.profiles or {})` branch — the ctor cannot
|
||||
# (its falsy dict falls back to DEFAULT_PROFILES)
|
||||
cases.append(("seam_resolve_budget_raw", ([], "ghost", "verify")))
|
||||
cases.append(("seam_resolve_budget_raw", (STATE_FLOOR, "tiny-cloud",
|
||||
"default")))
|
||||
|
||||
# ── to_dict (both flag-poles of the profile table)
|
||||
cases.append(("seam_to_dict", (T4FULL,)))
|
||||
cases.append(("seam_to_dict", (T0FULL,)))
|
||||
|
||||
# ── get_profile: hit, miss, key≠attr (KEY lookup, not tier attr);
|
||||
# falsy [] pairs → constructor falls back to DEFAULT_PROFILES
|
||||
cases.append(("seam_get_profile", ([], "T1")))
|
||||
cases.append(("seam_get_profile", ([], "TX")))
|
||||
cases.append(("seam_get_profile", (STATE_CROSS, "T9")))
|
||||
cases.append(("seam_get_profile", (STATE_CROSS, "K1")))
|
||||
|
||||
# ── get_enabled_profiles: default order; disabled skipped in place
|
||||
cases.append(("seam_get_enabled_profiles", ([],)))
|
||||
cases.append(("seam_get_enabled_profiles", (STATE_H,)))
|
||||
|
||||
# ── get_profiles_by_tier: attr match (not key), enabled-only
|
||||
cases.append(("seam_get_profiles_by_tier", ([], "T2")))
|
||||
cases.append(("seam_get_profiles_by_tier", (STATE_CROSS, "T9")))
|
||||
cases.append(("seam_get_profiles_by_tier", (STATE_H, "T2")))
|
||||
|
||||
# ── get_cheapest_capable: flag gates, odd-tier sort fallback (5),
|
||||
# STABLE tie, all-disabled → None
|
||||
cases.append(("seam_get_cheapest_capable", ([], False, False, False)))
|
||||
cases.append(("seam_get_cheapest_capable", ([], True, False, False)))
|
||||
cases.append(("seam_get_cheapest_capable", ([], True, True, True)))
|
||||
cases.append(("seam_get_cheapest_capable", (STATE_ODD, True, False, False)))
|
||||
cases.append(("seam_get_cheapest_capable", (STATE_ODD_R, True, False, False)))
|
||||
cases.append(("seam_get_cheapest_capable", (STATE_TIE, False, False, False)))
|
||||
cases.append(("seam_get_cheapest_capable", (STATE_ALL_OFF, False, False, False)))
|
||||
|
||||
# ── list_profiles: insertion order of VALUES (not the key set)
|
||||
cases.append(("seam_list_profiles", ([],)))
|
||||
cases.append(("seam_list_profiles", (STATE_H,)))
|
||||
|
||||
# ── resolve through the seam (pairs path): falsy → DEFAULT ctor
|
||||
# quirk; floor; duplicate-id first-match
|
||||
cases.append(("seam_resolve_budget", ("qwen2.5:3b", "verify", [])))
|
||||
cases.append(("seam_resolve_budget", ("tiny-cloud", "default", STATE_FLOOR)))
|
||||
cases.append(("seam_resolve_budget", ("mimo-v2.5-pro", "default", STATE_DUP)))
|
||||
|
||||
# ── test_registry: stdout bytes + None result (asserts live here)
|
||||
cases.append(("seam_test_registry", ()))
|
||||
|
||||
truth = freeze("rune.model_registry", cases, "rune.model_registry")
|
||||
write_envelope(FROZEN_NOW, truth, OUT)
|
||||
print(f"wrote {OUT} ({len(truth)} frozen cases)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,116 @@
|
||||
# Freeze rig for the model_registry dichotomy gap-closing (dichotomy wave
|
||||
# #2479 gap port). Auto-imported on interpreter start
|
||||
# (PYTHONPATH=scripts/mreg_freeze_seams:src) by BOTH the truth generator
|
||||
# (scripts/gen_model_registry_gap_truth.py) and the shared harness's
|
||||
# --verify mode, so frozen truth is byte-reproducible in any process.
|
||||
#
|
||||
# The oracle (src/rune/model_registry.py) is PURE: no loader seam (no
|
||||
# RUNE_MODEL_REGISTRY_CORE reader — the pin is vacuous), no wall clock,
|
||||
# no HOME/IO, no network. The only freezing hazard is JSON-safety:
|
||||
# registry methods return ModelProfile dataclass instances, which
|
||||
# json.dumps rejects. Wrapper seams convert results to asdict dicts (the
|
||||
# memory_consolidation JSON-safe-wrapper precedent) while calling the
|
||||
# REAL oracle methods, so the frozen logic stays the oracle's own.
|
||||
#
|
||||
# Registry states ride as ORDERED [tier_key, profile_dict] PAIRS — Go
|
||||
# must rebuild insertion order (dict key order = the oracle's iteration
|
||||
# order; provably load-bearing: get_profile is a KEY lookup while
|
||||
# get_profiles_by_tier matches the tier ATTRIBUTE, and
|
||||
# resolve_context_budget breaks on the FIRST id match).
|
||||
#
|
||||
# SEAM-NAMING LAW (construction_interface/mp precedent): seams bind under
|
||||
# seam_* names — the oracle's internal call graph (resolve_context_budget
|
||||
# → ModelRegistry) is never shadowed.
|
||||
|
||||
import contextlib
|
||||
import io
|
||||
from dataclasses import asdict
|
||||
|
||||
import rune.model_registry as mr # noqa: E402
|
||||
|
||||
|
||||
def _reg_from_state(state):
|
||||
"""state None → registry=None (resolve passes its own default);
|
||||
list of [tier_key, profile_dict] pairs → real ModelRegistry
|
||||
(constructor truthiness honored: [] pairs are falsy → DEFAULT)."""
|
||||
if state is None:
|
||||
return None
|
||||
return mr.ModelRegistry({k: mr.ModelProfile(**p) for k, p in state})
|
||||
|
||||
|
||||
def seam_to_dict(state):
|
||||
return mr.ModelProfile(**state).to_dict()
|
||||
|
||||
|
||||
def seam_get_profile(state, tier):
|
||||
reg = _reg_from_state(state)
|
||||
if reg is None:
|
||||
reg = mr.ModelRegistry()
|
||||
p = reg.get_profile(tier)
|
||||
return asdict(p) if p is not None else None
|
||||
|
||||
|
||||
def seam_get_enabled_profiles(state):
|
||||
reg = _reg_from_state(state)
|
||||
if reg is None:
|
||||
reg = mr.ModelRegistry()
|
||||
return [asdict(p) for p in reg.get_enabled_profiles()]
|
||||
|
||||
|
||||
def seam_get_profiles_by_tier(state, tier):
|
||||
reg = _reg_from_state(state)
|
||||
if reg is None:
|
||||
reg = mr.ModelRegistry()
|
||||
return [asdict(p) for p in reg.get_profiles_by_tier(tier)]
|
||||
|
||||
|
||||
def seam_get_cheapest_capable(state, requires_tools, requires_structured,
|
||||
requires_reasoning):
|
||||
reg = _reg_from_state(state)
|
||||
if reg is None:
|
||||
reg = mr.ModelRegistry()
|
||||
p = reg.get_cheapest_capable(requires_tools, requires_structured,
|
||||
requires_reasoning)
|
||||
return asdict(p) if p is not None else None
|
||||
|
||||
|
||||
def seam_list_profiles(state):
|
||||
reg = _reg_from_state(state)
|
||||
if reg is None:
|
||||
reg = mr.ModelRegistry()
|
||||
return reg.list_profiles()
|
||||
|
||||
|
||||
def seam_resolve_budget(model_id, task_kind="default", state=None):
|
||||
reg = _reg_from_state(state)
|
||||
return mr.resolve_context_budget(model_id, task_kind, reg)
|
||||
|
||||
|
||||
def seam_resolve_budget_raw(profiles, model_id, task_kind):
|
||||
"""Registry built via __new__ with profiles set RAW (no constructor
|
||||
truthiness fallback) — the only way to reach the `(reg.profiles or
|
||||
{})` empty branch, and the receiver shape the audit's method calls
|
||||
actually exercise."""
|
||||
reg = mr.ModelRegistry.__new__(mr.ModelRegistry)
|
||||
reg.profiles = {k: mr.ModelProfile(**p) for k, p in profiles}
|
||||
return mr.resolve_context_budget(model_id, task_kind, reg)
|
||||
|
||||
|
||||
def seam_test_registry():
|
||||
"""test_registry prints + asserts and returns None; the freezeable
|
||||
contract is BOTH the captured stdout bytes and the None result."""
|
||||
buf = io.StringIO()
|
||||
with contextlib.redirect_stdout(buf):
|
||||
result = mr.test_registry()
|
||||
return {"result": result, "stdout": buf.getvalue()}
|
||||
|
||||
|
||||
mr.seam_to_dict = seam_to_dict
|
||||
mr.seam_get_profile = seam_get_profile
|
||||
mr.seam_get_enabled_profiles = seam_get_enabled_profiles
|
||||
mr.seam_get_profiles_by_tier = seam_get_profiles_by_tier
|
||||
mr.seam_get_cheapest_capable = seam_get_cheapest_capable
|
||||
mr.seam_list_profiles = seam_list_profiles
|
||||
mr.seam_resolve_budget = seam_resolve_budget
|
||||
mr.seam_resolve_budget_raw = seam_resolve_budget_raw
|
||||
mr.seam_test_registry = seam_test_registry
|
||||
@@ -0,0 +1,459 @@
|
||||
// gap.go — gap-close parity machinery (dichotomy wave #2479), shared by
|
||||
// the CI binary (`replay-gap`, `scenario`) and the battery
|
||||
// (truth_gap_test.go).
|
||||
//
|
||||
// truth_gap.json rows are frozen against the live Python oracle through
|
||||
// wrapper shims (scripts/gen_model_registry_gap_truth.py, seam_*
|
||||
// functions bound by scripts/mreg_freeze_seams). Registry states ride as
|
||||
// ORDERED [tier_key, profile_dict] pairs — Go must rebuild insertion
|
||||
// order (the oracle's dict iteration order is load-bearing: get_profile
|
||||
// is a KEY lookup, get_profiles_by_tier matches the tier ATTRIBUTE,
|
||||
// resolve_context_budget breaks on the FIRST id match, list_profiles is
|
||||
// insertion-ordered).
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
)
|
||||
|
||||
// ordObj is a JSON object that remembers its key ORDER (Go maps do not;
|
||||
// frozen truth's dicts are insertion-ordered like Python's).
|
||||
type ordObj struct {
|
||||
keys []string
|
||||
m map[string]any
|
||||
}
|
||||
|
||||
func (o *ordObj) get(k string) (any, bool) {
|
||||
v, ok := o.m[k]
|
||||
return v, ok
|
||||
}
|
||||
|
||||
// decodeArg decodes one truth-row argument preserving object key order.
|
||||
func decodeArg(raw json.RawMessage) (any, error) {
|
||||
dec := json.NewDecoder(bytes.NewReader(raw))
|
||||
dec.UseNumber()
|
||||
return decodeArgValue(dec)
|
||||
}
|
||||
|
||||
func decodeArgValue(dec *json.Decoder) (any, error) {
|
||||
tok, err := dec.Token()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
switch t := tok.(type) {
|
||||
case json.Delim:
|
||||
switch t {
|
||||
case '{':
|
||||
o := &ordObj{m: map[string]any{}}
|
||||
for dec.More() {
|
||||
kt, err := dec.Token()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
key, ok := kt.(string)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("bad object key %v", kt)
|
||||
}
|
||||
val, err := decodeArgValue(dec)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, exists := o.m[key]; !exists {
|
||||
o.keys = append(o.keys, key)
|
||||
}
|
||||
o.m[key] = val
|
||||
}
|
||||
if _, err := dec.Token(); err != nil { // '}'
|
||||
return nil, err
|
||||
}
|
||||
return o, nil
|
||||
case '[':
|
||||
var arr []any
|
||||
for dec.More() {
|
||||
val, err := decodeArgValue(dec)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
arr = append(arr, val)
|
||||
}
|
||||
if _, err := dec.Token(); err != nil { // ']'
|
||||
return nil, err
|
||||
}
|
||||
if arr == nil {
|
||||
arr = []any{}
|
||||
}
|
||||
return arr, nil
|
||||
}
|
||||
return nil, fmt.Errorf("unexpected delim %v", t)
|
||||
case json.Number:
|
||||
f, err := t.Float64()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return f, nil
|
||||
default:
|
||||
return tok, nil // string / bool / nil
|
||||
}
|
||||
}
|
||||
|
||||
func asStr(v any) string {
|
||||
s, _ := v.(string)
|
||||
return s
|
||||
}
|
||||
|
||||
func asBool(v any) bool {
|
||||
b, _ := v.(bool)
|
||||
return b
|
||||
}
|
||||
|
||||
func asInt(v any) int {
|
||||
f, _ := v.(float64)
|
||||
return int(f)
|
||||
}
|
||||
|
||||
// profileFromAny rebuilds a Profile from a to_dict-shaped ordObj. All 11
|
||||
// fields are always present in frozen rows; missing keys fall back to the
|
||||
// dataclass defaults (enabled=True, zero/"" otherwise).
|
||||
func profileFromAny(v any) Profile {
|
||||
o, _ := v.(*ordObj)
|
||||
if o == nil {
|
||||
return Profile{Enabled: true}
|
||||
}
|
||||
enabled := true
|
||||
if ev, ok := o.get("enabled"); ok {
|
||||
enabled = asBool(ev)
|
||||
}
|
||||
return Profile{
|
||||
ID: asStr(must(o, "id")),
|
||||
Provider: asStr(must(o, "provider")),
|
||||
Tier: asStr(must(o, "tier")),
|
||||
Local: asBool(must(o, "local")),
|
||||
ContextWindow: asInt(must(o, "context_window")),
|
||||
SupportsTools: asBool(must(o, "supports_tools")),
|
||||
SupportsStructuredOutput: asBool(must(o, "supports_structured_output")),
|
||||
SupportsReasoningField: asBool(must(o, "supports_reasoning_field")),
|
||||
EstimatedLatency: asStr(must(o, "estimated_latency")),
|
||||
EstimatedCost: asStr(must(o, "estimated_cost")),
|
||||
Enabled: enabled,
|
||||
}
|
||||
}
|
||||
|
||||
func must(o *ordObj, k string) any {
|
||||
if v, ok := o.get(k); ok {
|
||||
return v
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// pairEntries rebuilds ordered registry entries from
|
||||
// [[tier_key, profile_dict], ...].
|
||||
func pairEntries(st any) []regEntry {
|
||||
arr, ok := st.([]any)
|
||||
if !ok || arr == nil {
|
||||
return nil
|
||||
}
|
||||
entries := make([]regEntry, 0, len(arr))
|
||||
for _, el := range arr {
|
||||
pair, ok := el.([]any)
|
||||
if !ok || len(pair) != 2 {
|
||||
continue
|
||||
}
|
||||
entries = append(entries, regEntry{key: asStr(pair[0]), prof: profileFromAny(pair[1])})
|
||||
}
|
||||
return entries
|
||||
}
|
||||
|
||||
// defaultOrNew — the seam wrappers build registries through the REAL
|
||||
// constructor: a falsy (nil/empty) state falls back to DEFAULT_PROFILES.
|
||||
// A nil state (registry=None) resolves to the default registry too.
|
||||
func defaultOrNew(st any) *Registry {
|
||||
if st == nil {
|
||||
return defaultRegistry()
|
||||
}
|
||||
return newRegistry(registryFromRaw(pairEntries(st)))
|
||||
}
|
||||
|
||||
// ── truth-row execution ────────────────────────────────────────────────
|
||||
|
||||
// execGapRow interprets one seam_* wrapper row against the port. args
|
||||
// are decodeArg values (objects = *ordObj, order-preserving).
|
||||
func execGapRow(fn string, args []any) (any, error) {
|
||||
switch fn {
|
||||
case "seam_to_dict":
|
||||
return toAny(profileFromAny(args[0])), nil
|
||||
case "seam_get_profile":
|
||||
if p := defaultOrNew(args[0]).getProfile(asStr(args[1])); p != nil {
|
||||
return toAny(*p), nil
|
||||
}
|
||||
return nil, nil
|
||||
case "seam_get_enabled_profiles":
|
||||
return toAny(defaultOrNew(args[0]).getEnabledProfiles()), nil
|
||||
case "seam_get_profiles_by_tier":
|
||||
return toAny(defaultOrNew(args[0]).getProfilesByTier(asStr(args[1]))), nil
|
||||
case "seam_get_cheapest_capable":
|
||||
p := defaultOrNew(args[0]).getCheapestCapable(asBool(args[1]), asBool(args[2]), asBool(args[3]))
|
||||
if p != nil {
|
||||
return toAny(*p), nil
|
||||
}
|
||||
return nil, nil
|
||||
case "seam_list_profiles":
|
||||
return toAny(defaultOrNew(args[0]).listProfiles()), nil
|
||||
case "seam_resolve_budget":
|
||||
if args[2] == nil {
|
||||
return toAny(resolveBudget(asStr(args[0]), asStr(args[1]))), nil
|
||||
}
|
||||
reg := registryFromRaw(pairEntries(args[2]))
|
||||
return toAny(newRegistry(reg).resolveContextBudget(asStr(args[0]), asStr(args[1]))), nil
|
||||
case "seam_resolve_budget_raw":
|
||||
reg := registryFromRaw(pairEntries(args[0]))
|
||||
return toAny(reg.resolveContextBudget(asStr(args[1]), asStr(args[2]))), nil
|
||||
case "seam_test_registry":
|
||||
res, stdout, err := testRegistry()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return map[string]any{"result": res, "stdout": stdout}, nil
|
||||
case "resolve_context_budget":
|
||||
kind := "default"
|
||||
if len(args) > 1 && args[1] != nil {
|
||||
kind = asStr(args[1])
|
||||
}
|
||||
if len(args) > 2 && args[2] != nil {
|
||||
reg := registryFromRaw(pairEntries(args[2]))
|
||||
return toAny(newRegistry(reg).resolveContextBudget(asStr(args[0]), kind)), nil
|
||||
}
|
||||
return toAny(resolveBudget(asStr(args[0]), kind)), nil
|
||||
}
|
||||
return nil, fmt.Errorf("unknown gap fn %q", fn)
|
||||
}
|
||||
|
||||
// toAny converts port results into decodeArg-comparable values
|
||||
// (structs → maps, ints → float64, typed slices → []any).
|
||||
func toAny(v any) any {
|
||||
switch t := v.(type) {
|
||||
case nil:
|
||||
return nil
|
||||
case *Profile:
|
||||
if t == nil {
|
||||
return nil
|
||||
}
|
||||
return t.toDict()
|
||||
case Profile:
|
||||
return t.toDict()
|
||||
case []Profile:
|
||||
arr := make([]any, len(t))
|
||||
for i := range t {
|
||||
arr[i] = t[i].toDict()
|
||||
}
|
||||
return arr
|
||||
case []map[string]any:
|
||||
arr := make([]any, len(t))
|
||||
for i := range t {
|
||||
arr[i] = toAny(t[i])
|
||||
}
|
||||
return arr
|
||||
case map[string]any:
|
||||
m := make(map[string]any, len(t))
|
||||
for k, val := range t {
|
||||
m[k] = toAny(val)
|
||||
}
|
||||
return m
|
||||
case int:
|
||||
return float64(t)
|
||||
default:
|
||||
return v
|
||||
}
|
||||
}
|
||||
|
||||
// ordFromMap wraps a map as an ordObj (key order irrelevant — pyEqual
|
||||
// compares maps semantically).
|
||||
func ordFromMap(m map[string]any) *ordObj {
|
||||
o := &ordObj{m: map[string]any{}}
|
||||
for k, v := range m {
|
||||
o.keys = append(o.keys, k)
|
||||
o.m[k] = v
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// pyEqual — semantic equality with LIST order sensitivity (list order is
|
||||
// parity: list_profiles/get_enabled_profiles rows pin it); dict key order
|
||||
// is irrelevant on both sides.
|
||||
func pyEqual(got, want any) bool {
|
||||
got = toAny(got)
|
||||
want = toAny(want)
|
||||
if m, ok := got.(map[string]any); ok {
|
||||
got = ordFromMap(m)
|
||||
}
|
||||
if m, ok := want.(map[string]any); ok {
|
||||
want = ordFromMap(m)
|
||||
}
|
||||
if got == nil && want == nil {
|
||||
return true
|
||||
}
|
||||
gm, gok := got.(*ordObj)
|
||||
wm, wok := want.(*ordObj)
|
||||
if gok || wok {
|
||||
if !gok || !wok || len(gm.m) != len(wm.m) {
|
||||
return false
|
||||
}
|
||||
for k, gv := range gm.m {
|
||||
wv, ok := wm.get(k)
|
||||
if !ok || !pyEqual(gv, wv) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
gl, gok := got.([]any)
|
||||
wl, wok := want.([]any)
|
||||
if gok || wok {
|
||||
if !gok || !wok || len(gl) != len(wl) {
|
||||
return false
|
||||
}
|
||||
for i := range gl {
|
||||
if !pyEqual(gl[i], wl[i]) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
gf, gok := got.(float64)
|
||||
wf, wok := want.(float64)
|
||||
if gok || wok {
|
||||
return gok && wok && gf == wf
|
||||
}
|
||||
return got == want
|
||||
}
|
||||
|
||||
// replayGapBytes replays a truth_gap envelope; returns first divergence.
|
||||
func replayGapBytes(raw []byte) error {
|
||||
var env struct {
|
||||
FrozenNow string `json:"frozen_now"`
|
||||
Cases []gapTruthRow `json:"cases"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &env); err != nil {
|
||||
return err
|
||||
}
|
||||
fails := 0
|
||||
for i, row := range env.Cases {
|
||||
if len(row.Recv) > 0 && string(row.Recv) != "null" {
|
||||
return fmt.Errorf("case %d: recv rows are not part of this module's battery", i)
|
||||
}
|
||||
args := make([]any, len(row.Args))
|
||||
for j, rawArg := range row.Args {
|
||||
v, err := decodeArg(rawArg)
|
||||
if err != nil {
|
||||
return fmt.Errorf("case %d: bad arg %d: %v", i, j, err)
|
||||
}
|
||||
args[j] = v
|
||||
}
|
||||
got, err := execGapRow(row.Fn, args)
|
||||
if err != nil {
|
||||
fails++
|
||||
fmt.Printf("case %d %s: ERROR %v\n", i, row.Fn, err)
|
||||
continue
|
||||
}
|
||||
want, err := decodeArg(row.Expected)
|
||||
if err != nil {
|
||||
return fmt.Errorf("case %d: bad expected: %v", i, err)
|
||||
}
|
||||
if !pyEqual(got, want) {
|
||||
fails++
|
||||
gb, _ := json.MarshalIndent(toAny(got), "", " ")
|
||||
wb, _ := json.MarshalIndent(want, "", " ")
|
||||
fmt.Printf("case %d %s DIVERGE\ngot: %s\nwant: %s\n", i, row.Fn, gb, wb)
|
||||
}
|
||||
}
|
||||
if fails > 0 {
|
||||
return fmt.Errorf("PARITY FAILED: %d divergence(s)", fails)
|
||||
}
|
||||
fmt.Printf("PARITY OK: %d gap cases exact-equal vs the Python oracle\n", len(env.Cases))
|
||||
return nil
|
||||
}
|
||||
|
||||
type gapTruthRow struct {
|
||||
Fn string `json:"fn"`
|
||||
Recv json.RawMessage `json:"recv,omitempty"`
|
||||
Args []json.RawMessage `json:"args"`
|
||||
Kwargs json.RawMessage `json:"kwargs,omitempty"`
|
||||
Expected json.RawMessage `json:"expected"`
|
||||
}
|
||||
|
||||
// ── scenario driver (live parity: identical ops against the Python
|
||||
// oracle and this port, JSON outputs diffed by
|
||||
// scripts/check_model_registry_parity.py) ──────────────────────────────
|
||||
|
||||
type scenarioOp struct {
|
||||
Op string `json:"op"`
|
||||
State json.RawMessage `json:"state,omitempty"`
|
||||
Tier string `json:"tier,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
Kind string `json:"kind,omitempty"`
|
||||
Tools bool `json:"tools"`
|
||||
Structured bool `json:"structured"`
|
||||
Reasoning bool `json:"reasoning"`
|
||||
}
|
||||
|
||||
func runScenario(path string) error {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var sc struct {
|
||||
Ops []scenarioOp `json:"ops"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &sc); err != nil {
|
||||
return err
|
||||
}
|
||||
steps := []any{}
|
||||
for _, op := range sc.Ops {
|
||||
state := any(nil)
|
||||
if len(op.State) > 0 && string(op.State) != "null" {
|
||||
state, err = decodeArg(op.State)
|
||||
if err != nil {
|
||||
return fmt.Errorf("op %q: bad state: %v", op.Op, err)
|
||||
}
|
||||
}
|
||||
var result any
|
||||
switch op.Op {
|
||||
case "to_dict":
|
||||
result = toAny(profileFromAny(state))
|
||||
case "get_profile":
|
||||
if p := defaultOrNew(state).getProfile(op.Tier); p != nil {
|
||||
result = toAny(*p)
|
||||
}
|
||||
case "get_enabled_profiles":
|
||||
result = toAny(defaultOrNew(state).getEnabledProfiles())
|
||||
case "get_profiles_by_tier":
|
||||
result = toAny(defaultOrNew(state).getProfilesByTier(op.Tier))
|
||||
case "get_cheapest_capable":
|
||||
if p := defaultOrNew(state).getCheapestCapable(op.Tools, op.Structured, op.Reasoning); p != nil {
|
||||
result = toAny(*p)
|
||||
}
|
||||
case "list_profiles":
|
||||
result = toAny(defaultOrNew(state).listProfiles())
|
||||
case "resolve":
|
||||
result = toAny(resolveBudget(op.Model, op.Kind))
|
||||
case "resolve_custom":
|
||||
reg := registryFromRaw(pairEntries(state))
|
||||
result = toAny(reg.resolveContextBudget(op.Model, op.Kind))
|
||||
case "test_registry":
|
||||
res, stdout, err := testRegistry()
|
||||
if err != nil {
|
||||
return fmt.Errorf("test_registry: %v", err)
|
||||
}
|
||||
result = map[string]any{"result": res, "stdout": stdout}
|
||||
default:
|
||||
return fmt.Errorf("unknown scenario op %q", op.Op)
|
||||
}
|
||||
steps = append(steps, map[string]any{"op": op.Op, "result": result})
|
||||
}
|
||||
enc := json.NewEncoder(os.Stdout)
|
||||
enc.SetEscapeHTML(false)
|
||||
enc.SetIndent("", " ")
|
||||
return enc.Encode(map[string]any{"steps": steps})
|
||||
}
|
||||
+321
-94
@@ -1,50 +1,156 @@
|
||||
// Package main — model-registry: the GO port of src/rune/model_registry.py's
|
||||
// get_cheapest_capable + resolve_context_budget (Oracle-Battery Parity
|
||||
// Pattern, #2479 GO batch 67). The profile store stays Python (reactive).
|
||||
// Package main — model-registry: the GO port of src/rune/model_registry.py
|
||||
// (Oracle-Battery Parity Pattern, #2479 GO batch 67; gap-closed 2026-10-14:
|
||||
// to_dict/get_profile/get_enabled_profiles/get_profiles_by_tier/
|
||||
// list_profiles/test_registry ported name-preserving and resolve_context
|
||||
// constants corrected — the batch-67 port hardcoded toolReserve=2048 and a
|
||||
// single outputReserve; the live oracle holds TOOL_RESULT_RESERVE=1024 and
|
||||
// the OUTPUT_RESERVE per-task-kind ladder).
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"sort"
|
||||
)
|
||||
|
||||
// Profile mirrors the oracle's ModelProfile dataclass field-for-field
|
||||
// (to_dict emits exactly these keys in exactly this order).
|
||||
type Profile struct {
|
||||
ID string
|
||||
Tier string
|
||||
Enabled bool
|
||||
SupportsTools bool
|
||||
SupportsStruct bool
|
||||
SupportsReason bool
|
||||
ContextWindow int
|
||||
Local bool
|
||||
ID string
|
||||
Provider string
|
||||
Tier string
|
||||
Local bool
|
||||
ContextWindow int
|
||||
SupportsTools bool
|
||||
SupportsStructuredOutput bool
|
||||
SupportsReasoningField bool
|
||||
EstimatedLatency string
|
||||
EstimatedCost string
|
||||
Enabled bool
|
||||
}
|
||||
|
||||
// the oracle's default registry (T0 deterministic + the capability floor)
|
||||
var profiles = []Profile{
|
||||
{ID: "deterministic", Tier: "T0", Enabled: true, ContextWindow: 8192, Local: true},
|
||||
{ID: "qwen2.5:3b", Tier: "T1", Enabled: true, ContextWindow: 32768, Local: true},
|
||||
{ID: "qwen3:14b", Tier: "T2", Enabled: true, ContextWindow: 32768, Local: true},
|
||||
{ID: "qwen3.6:27b", Tier: "T3", Enabled: true, ContextWindow: 32768, Local: true},
|
||||
{ID: "mimo-v2.5-pro", Tier: "T4", Enabled: true, SupportsTools: true, ContextWindow: 128000, Local: false},
|
||||
// toDict — mirrors ModelProfile.to_dict (snake_case keys, declaration
|
||||
// order, no values omitted).
|
||||
func (p Profile) toDict() map[string]any {
|
||||
return map[string]any{
|
||||
"id": p.ID,
|
||||
"provider": p.Provider,
|
||||
"tier": p.Tier,
|
||||
"local": p.Local,
|
||||
"context_window": p.ContextWindow,
|
||||
"supports_tools": p.SupportsTools,
|
||||
"supports_structured_output": p.SupportsStructuredOutput,
|
||||
"supports_reasoning_field": p.SupportsReasoningField,
|
||||
"estimated_latency": p.EstimatedLatency,
|
||||
"estimated_cost": p.EstimatedCost,
|
||||
"enabled": p.Enabled,
|
||||
}
|
||||
}
|
||||
|
||||
// getCheapestCapable — Python-identical: filter by capability, sort by tier.
|
||||
func getCheapestCapable(needTools, needStruct, needReason bool) *Profile {
|
||||
tierOrder := map[string]int{"T0": 0, "T1": 1, "T2": 2, "T3": 3, "T4": 4}
|
||||
// regEntry — the oracle stores profiles in a dict keyed by TIER, and the
|
||||
// key is independent of the profile's own tier attribute. get_profile is
|
||||
// a KEY lookup; get_profiles_by_tier matches the ATTRIBUTE; iteration
|
||||
// order everywhere is dict-insertion order. Entries preserve both.
|
||||
type regEntry struct {
|
||||
key string
|
||||
prof Profile
|
||||
}
|
||||
|
||||
// Registry mirrors ModelRegistry: an insertion-ordered tier→profile dict.
|
||||
type Registry struct {
|
||||
entries []regEntry
|
||||
}
|
||||
|
||||
// defaultRegistry mirrors DEFAULT_PROFILES (T0..T4, insertion order).
|
||||
// T0's context_window is 0 — `context_window or 8192` handles it.
|
||||
func defaultRegistry() *Registry {
|
||||
return &Registry{entries: []regEntry{
|
||||
{"T0", Profile{ID: "deterministic", Provider: "local", Tier: "T0",
|
||||
Local: true, EstimatedLatency: "instant", EstimatedCost: "0",
|
||||
Enabled: true}},
|
||||
{"T1", Profile{ID: "qwen2.5:3b", Provider: "ollama", Tier: "T1",
|
||||
Local: true, ContextWindow: 32768, EstimatedLatency: "low",
|
||||
EstimatedCost: "0", Enabled: true}},
|
||||
{"T2", Profile{ID: "qwen3:14b", Provider: "ollama", Tier: "T2",
|
||||
Local: true, ContextWindow: 32768, EstimatedLatency: "medium",
|
||||
EstimatedCost: "0", Enabled: true}},
|
||||
{"T3", Profile{ID: "qwen3.6:27b", Provider: "ollama", Tier: "T3",
|
||||
Local: true, ContextWindow: 32768, EstimatedLatency: "high",
|
||||
EstimatedCost: "0", Enabled: true}},
|
||||
{"T4", Profile{ID: "mimo-v2.5-pro", Provider: "openai", Tier: "T4",
|
||||
SupportsTools: true, SupportsStructuredOutput: true,
|
||||
SupportsReasoningField: true, ContextWindow: 128000,
|
||||
EstimatedLatency: "higher", EstimatedCost: "paid", Enabled: true}},
|
||||
}}
|
||||
}
|
||||
|
||||
// newRegistry mirrors ModelRegistry(profiles): a falsy (nil/empty) dict
|
||||
// falls back to DEFAULT_PROFILES.copy().
|
||||
func newRegistry(profiles *Registry) *Registry {
|
||||
if profiles == nil || len(profiles.entries) == 0 {
|
||||
return defaultRegistry()
|
||||
}
|
||||
return profiles
|
||||
}
|
||||
|
||||
// registryFromRaw mirrors a receiver whose profiles attribute was set
|
||||
// directly (no constructor fallback): an empty dict stays empty.
|
||||
func registryFromRaw(entries []regEntry) *Registry {
|
||||
return &Registry{entries: entries}
|
||||
}
|
||||
|
||||
// getProfile — ModelRegistry.get_profile: dict.get by KEY.
|
||||
func (r *Registry) getProfile(tier string) *Profile {
|
||||
for _, e := range r.entries {
|
||||
if e.key == tier {
|
||||
p := e.prof
|
||||
return &p
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// getEnabledProfiles — [p for p in profiles.values() if p.enabled],
|
||||
// insertion order preserved.
|
||||
func (r *Registry) getEnabledProfiles() []Profile {
|
||||
var out []Profile
|
||||
for _, e := range r.entries {
|
||||
if e.prof.Enabled {
|
||||
out = append(out, e.prof)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// getProfilesByTier — [p for p in profiles.values() if p.tier == tier
|
||||
// and p.enabled]: ATTRIBUTE match (not the dict key).
|
||||
func (r *Registry) getProfilesByTier(tier string) []Profile {
|
||||
var out []Profile
|
||||
for _, e := range r.entries {
|
||||
if e.prof.Tier == tier && e.prof.Enabled {
|
||||
out = append(out, e.prof)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// getCheapestCapable — capability filter, stable sort by tier rank
|
||||
// (tier_order.get(tier, 5): unknown tiers sort last), insertion-order
|
||||
// tie-break (Python sort is stable).
|
||||
func (r *Registry) getCheapestCapable(needTools, needStruct, needReason bool) *Profile {
|
||||
var capable []Profile
|
||||
for _, p := range profiles {
|
||||
for _, e := range r.entries {
|
||||
p := e.prof
|
||||
if !p.Enabled {
|
||||
continue
|
||||
}
|
||||
if needTools && !p.SupportsTools {
|
||||
continue
|
||||
}
|
||||
if needStruct && !p.SupportsStruct {
|
||||
if needStruct && !p.SupportsStructuredOutput {
|
||||
continue
|
||||
}
|
||||
if needReason && !p.SupportsReason {
|
||||
if needReason && !p.SupportsReasoningField {
|
||||
continue
|
||||
}
|
||||
capable = append(capable, p)
|
||||
@@ -52,102 +158,223 @@ func getCheapestCapable(needTools, needStruct, needReason bool) *Profile {
|
||||
if len(capable) == 0 {
|
||||
return nil
|
||||
}
|
||||
sort.Slice(capable, func(i, j int) bool {
|
||||
ti, tj := tierOrder[capable[i].Tier], tierOrder[capable[j].Tier]
|
||||
if ti == 0 && capable[i].Tier != "T0" {
|
||||
ti = 5
|
||||
tierOrder := map[string]int{"T0": 0, "T1": 1, "T2": 2, "T3": 3, "T4": 4}
|
||||
rank := func(p Profile) int {
|
||||
if v, ok := tierOrder[p.Tier]; ok {
|
||||
return v
|
||||
}
|
||||
if tj == 0 && capable[j].Tier != "T0" {
|
||||
tj = 5
|
||||
return 5
|
||||
}
|
||||
// insertion sort keeps ties in insertion order (stable, like Python)
|
||||
for i := 1; i < len(capable); i++ {
|
||||
for j := i; j > 0 && rank(capable[j]) < rank(capable[j-1]); j-- {
|
||||
capable[j], capable[j-1] = capable[j-1], capable[j]
|
||||
}
|
||||
return ti < tj
|
||||
})
|
||||
}
|
||||
return &capable[0]
|
||||
}
|
||||
|
||||
const outputReserve = 2048
|
||||
// listProfiles — [p.to_dict() for p in profiles.values()], insertion
|
||||
// order preserved.
|
||||
func (r *Registry) listProfiles() []map[string]any {
|
||||
out := make([]map[string]any, 0, len(r.entries))
|
||||
for _, e := range r.entries {
|
||||
out = append(out, e.prof.toDict())
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// resolveContextBudget — window - output_reserve - tool_reserve (0 for
|
||||
// local), num_ctx floored at 1024. Python-identical.
|
||||
func resolveContextBudget(modelID string) map[string]any {
|
||||
window := 8192
|
||||
local := true
|
||||
for _, p := range profiles {
|
||||
if p.ID == modelID {
|
||||
window = p.ContextWindow
|
||||
if window == 0 {
|
||||
window = 8192
|
||||
}
|
||||
local = p.Local
|
||||
break
|
||||
// resolveContextBudget — the module-level resolve_context_budget with
|
||||
// THIS registry as the `registry` argument (entries used raw: the empty
|
||||
// registry exercises the `(reg.profiles or {})` branch).
|
||||
func (r *Registry) resolveContextBudget(modelID, taskKind string) map[string]any {
|
||||
var profile *Profile
|
||||
for _, e := range r.entries {
|
||||
if e.prof.ID == modelID {
|
||||
p := e.prof
|
||||
profile = &p
|
||||
break // first id match wins (insertion order)
|
||||
}
|
||||
}
|
||||
window, local := 8192, true
|
||||
if profile != nil {
|
||||
window = profile.ContextWindow
|
||||
if window == 0 {
|
||||
window = 8192
|
||||
}
|
||||
local = profile.Local
|
||||
}
|
||||
ort, ok := outputReserves[taskKind]
|
||||
if !ok {
|
||||
ort = outputReserves["default"] // OUTPUT_RESERVE.get(task_kind, ...)
|
||||
}
|
||||
toolReserve := 0
|
||||
if !local {
|
||||
toolReserve = 2048 // TOOL_RESULT_RESERVE
|
||||
toolReserve = toolResultReserve
|
||||
}
|
||||
numCtx := window - outputReserve - toolReserve
|
||||
numCtx := window - ort - toolReserve
|
||||
if numCtx < 1024 {
|
||||
numCtx = 1024
|
||||
}
|
||||
return map[string]any{
|
||||
"num_ctx": numCtx, "window": window,
|
||||
"output_reserve": outputReserve, "tool_reserve": toolReserve,
|
||||
"output_reserve": ort, "tool_reserve": toolReserve,
|
||||
"model": modelID, "local": local,
|
||||
}
|
||||
}
|
||||
|
||||
// ── context budgets (#28: model-aware, never hardcoded 8192) ───────────
|
||||
|
||||
// outputReserves mirrors OUTPUT_RESERVE (per-task-kind ladder).
|
||||
var outputReserves = map[string]int{
|
||||
"default": 2048, "verify": 512, "summarize": 1024, "embed": 0,
|
||||
}
|
||||
|
||||
// toolResultReserve mirrors TOOL_RESULT_RESERVE (2026-10-14: the batch-67
|
||||
// port hardcoded 2048 here — the live oracle holds 1024; gated by LOCAL,
|
||||
// not by tier).
|
||||
const toolResultReserve = 1024
|
||||
|
||||
// resolveBudget — resolve_context_budget with registry=None (the oracle
|
||||
// builds its own default ModelRegistry).
|
||||
func resolveBudget(modelID, taskKind string) map[string]any {
|
||||
return newRegistry(nil).resolveContextBudget(modelID, taskKind)
|
||||
}
|
||||
|
||||
// legacy facade (batch-67 replay): default-registry single-model resolve.
|
||||
func resolveContextBudget(modelID string) map[string]any {
|
||||
return resolveBudget(modelID, "default")
|
||||
}
|
||||
|
||||
// legacy facade (batch-67 replay): default-registry cheapest search.
|
||||
func getCheapestCapable(needTools, needStruct, needReason bool) *Profile {
|
||||
return defaultRegistry().getCheapestCapable(needTools, needStruct, needReason)
|
||||
}
|
||||
|
||||
// testRegistry — mirrors test_registry: prints the same lines, runs the
|
||||
// same assertions, returns None. Returns the captured stdout; a failed
|
||||
// assertion is an error (the frozen row pins the success path).
|
||||
func testRegistry() (any, string, error) {
|
||||
var b []byte
|
||||
say := func(format string, args ...any) {
|
||||
b = append(b, []byte(fmt.Sprintf(format+"\n", args...))...)
|
||||
}
|
||||
say("Testing Model Registry...")
|
||||
|
||||
reg := defaultRegistry()
|
||||
|
||||
profiles := reg.listProfiles()
|
||||
say("Total profiles: %d", len(profiles))
|
||||
if len(profiles) != 5 {
|
||||
return nil, string(b), fmt.Errorf("AssertionError: %d != 5", len(profiles))
|
||||
}
|
||||
|
||||
t1 := reg.getProfile("T1")
|
||||
say("T1 profile: %s", t1.ID)
|
||||
if t1.ID != "qwen2.5:3b" {
|
||||
return nil, string(b), fmt.Errorf("AssertionError: T1 id %q", t1.ID)
|
||||
}
|
||||
|
||||
cheapest := reg.getCheapestCapable(false, false, false)
|
||||
say("Cheapest capable: %s", cheapest.ID)
|
||||
if cheapest.Tier != "T0" {
|
||||
return nil, string(b), fmt.Errorf("AssertionError: cheapest tier %q", cheapest.Tier)
|
||||
}
|
||||
|
||||
cheapestTools := reg.getCheapestCapable(true, false, false)
|
||||
say("Cheapest with tools: %s", cheapestTools.ID)
|
||||
if cheapestTools.Tier != "T4" {
|
||||
return nil, string(b), fmt.Errorf("AssertionError: cheapest-with-tools tier %q", cheapestTools.Tier)
|
||||
}
|
||||
|
||||
say("✅ All tests passed!")
|
||||
return nil, string(b), nil
|
||||
}
|
||||
|
||||
func main() {
|
||||
if len(os.Args) < 2 || os.Args[1] != "replay" {
|
||||
if len(os.Args) < 2 {
|
||||
fmt.Fprintln(os.Stderr, "usage: model-registry replay <truth.json>")
|
||||
os.Exit(2)
|
||||
}
|
||||
raw, err := os.ReadFile(os.Args[2])
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
var cases []map[string]any
|
||||
if err := json.Unmarshal(raw, &cases); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fails := 0
|
||||
for i, c := range cases {
|
||||
op, _ := c["op"].(string)
|
||||
switch op {
|
||||
case "cheapest":
|
||||
want, _ := c["want"].(string)
|
||||
if got := getCheapestCapable(false, false, false); got == nil || got.Tier != want {
|
||||
fails++
|
||||
fmt.Printf("case %d cheapest: divergence\n", i)
|
||||
}
|
||||
case "cheapest_tools":
|
||||
want := c["want"].([]any)
|
||||
got := getCheapestCapable(true, false, false)
|
||||
if got == nil || got.Tier != want[0].(string) {
|
||||
fails++
|
||||
fmt.Printf("case %d cheapest_tools: divergence\n", i)
|
||||
}
|
||||
case "budget_unknown", "budget_known":
|
||||
model := "nonexistent-model"
|
||||
if m, ok := c["model"].(string); ok {
|
||||
model = m
|
||||
}
|
||||
got := resolveContextBudget(model)
|
||||
want := c["want"].(map[string]any)
|
||||
for k, wv := range want {
|
||||
gv := got[k]
|
||||
if fmt.Sprintf("%v", gv) != fmt.Sprintf("%v", wv) {
|
||||
switch os.Args[1] {
|
||||
case "replay":
|
||||
if len(os.Args) < 3 {
|
||||
fmt.Fprintln(os.Stderr, "usage: model-registry replay <truth.json>")
|
||||
os.Exit(2)
|
||||
}
|
||||
raw, err := os.ReadFile(os.Args[2])
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
var cases []map[string]any
|
||||
if err := json.Unmarshal(raw, &cases); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fails := 0
|
||||
for i, c := range cases {
|
||||
op, _ := c["op"].(string)
|
||||
switch op {
|
||||
case "cheapest":
|
||||
want, _ := c["want"].(string)
|
||||
if got := getCheapestCapable(false, false, false); got == nil || got.Tier != want {
|
||||
fails++
|
||||
fmt.Printf("case %d %s.%s: got %v want %v\n", i, op, k, gv, wv)
|
||||
fmt.Printf("case %d cheapest: divergence\n", i)
|
||||
}
|
||||
case "cheapest_tools":
|
||||
want := c["want"].([]any)
|
||||
got := getCheapestCapable(true, false, false)
|
||||
if got == nil || got.Tier != want[0].(string) {
|
||||
fails++
|
||||
fmt.Printf("case %d cheapest_tools: divergence\n", i)
|
||||
}
|
||||
case "budget_unknown", "budget_known":
|
||||
model := "nonexistent-model"
|
||||
if m, ok := c["model"].(string); ok {
|
||||
model = m
|
||||
}
|
||||
got := resolveContextBudget(model)
|
||||
want := c["want"].(map[string]any)
|
||||
for k, wv := range want {
|
||||
gv := got[k]
|
||||
if fmt.Sprintf("%v", gv) != fmt.Sprintf("%v", wv) {
|
||||
fails++
|
||||
fmt.Printf("case %d %s.%s: got %v want %v\n", i, op, k, gv, wv)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if fails > 0 {
|
||||
fmt.Fprintf(os.Stderr, "PARITY FAILED: %d divergence(s)\n", fails)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Printf("PARITY OK: %d cases vs the Python oracle\n", len(cases))
|
||||
case "replay-gap":
|
||||
if len(os.Args) < 3 {
|
||||
fmt.Fprintln(os.Stderr, "usage: model-registry replay-gap <truth_gap.json>")
|
||||
os.Exit(2)
|
||||
}
|
||||
raw, err := os.ReadFile(os.Args[2])
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := replayGapBytes(raw); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
case "scenario":
|
||||
if len(os.Args) < 3 {
|
||||
fmt.Fprintln(os.Stderr, "usage: model-registry scenario <scenario.json>")
|
||||
os.Exit(2)
|
||||
}
|
||||
if err := runScenario(os.Args[2]); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
default:
|
||||
fmt.Fprintln(os.Stderr, "usage: model-registry replay|replay-gap|scenario <file>")
|
||||
os.Exit(2)
|
||||
}
|
||||
if fails > 0 {
|
||||
fmt.Fprintf(os.Stderr, "PARITY FAILED: %d divergence(s)\n", fails)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Printf("PARITY OK: %d cases vs the Python oracle\n", len(cases))
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,91 @@
|
||||
package main
|
||||
|
||||
// truth_gap_test.go — gap-close battery: replays truth_gap.json (frozen
|
||||
// against the live Python oracle through scripts/gen_model_registry_
|
||||
// gap_truth.py wrapper shims) and pins the constants + lookup semantics
|
||||
// the 2026-10-09 batch-67 port got wrong or never carried:
|
||||
// * TOOL_RESULT_RESERVE = 1024 (was hardcoded 2048) — gated by LOCAL,
|
||||
// not tier
|
||||
// * OUTPUT_RESERVE is a per-task-kind ladder (2048/512/1024/0), not a
|
||||
// single constant
|
||||
// * get_profile is a KEY lookup; get_profiles_bytier matches the tier
|
||||
// ATTRIBUTE; dict insertion order is observable everywhere
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGapParity(t *testing.T) {
|
||||
raw, err := os.ReadFile("truth_gap.json")
|
||||
if err != nil {
|
||||
t.Fatalf("truth missing: %v", err)
|
||||
}
|
||||
if err := replayGapBytes(raw); err != nil {
|
||||
t.Fatalf("gap parity: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReserveConstants pins the live-oracle constants (the known flag
|
||||
// from the model_connect agent, 2026-10-14).
|
||||
func TestReserveConstants(t *testing.T) {
|
||||
want := map[string]int{"default": 2048, "verify": 512, "summarize": 1024, "embed": 0}
|
||||
for kind, reserve := range want {
|
||||
if got := outputReserves[kind]; got != reserve {
|
||||
t.Errorf("OUTPUT_RESERVE[%q]: got %d want %d", kind, got, reserve)
|
||||
}
|
||||
}
|
||||
if toolResultReserve != 1024 {
|
||||
t.Errorf("TOOL_RESULT_RESERVE: got %d want 1024", toolResultReserve)
|
||||
}
|
||||
// the flag fix, end to end: cloud model, default kind
|
||||
b := resolveContextBudget("mimo-v2.5-pro")
|
||||
if b["num_ctx"] != 124928 || b["tool_reserve"] != 1024 {
|
||||
t.Errorf("flag fix broke: %v", b)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLookupSemantics pins key-vs-attribute lookup and ordering laws.
|
||||
func TestLookupSemantics(t *testing.T) {
|
||||
reg := registryFromRaw([]regEntry{
|
||||
{"K1", Profile{ID: "p1", Tier: "T9", Enabled: true}},
|
||||
})
|
||||
if reg.getProfile("T9") != nil {
|
||||
t.Error("get_profile must be a KEY lookup, not a tier-attribute match")
|
||||
}
|
||||
if reg.getProfile("K1") == nil {
|
||||
t.Error("get_profile missed an existing KEY")
|
||||
}
|
||||
if len(reg.getProfilesByTier("T9")) != 1 {
|
||||
t.Error("get_profiles_by_tier must match the tier ATTRIBUTE")
|
||||
}
|
||||
// constructor truthiness: falsy profiles → DEFAULT_PROFILES
|
||||
if got := newRegistry(registryFromRaw(nil)).getProfile("T1"); got == nil || got.ID != "qwen2.5:3b" {
|
||||
t.Errorf("falsy registry must fall back to DEFAULT_PROFILES, got %v", got)
|
||||
}
|
||||
// raw registry keeps emptiness (the (profiles or {}) branch)
|
||||
raw := registryFromRaw(nil)
|
||||
if b := raw.resolveContextBudget("ghost", "verify"); b["num_ctx"] != 7680 {
|
||||
t.Errorf("raw empty registry: %v", b)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStableSortTie pins the stable tie-break on get_cheapest_capable.
|
||||
func TestStableSortTie(t *testing.T) {
|
||||
reg := registryFromRaw([]regEntry{
|
||||
{"TA", Profile{ID: "first", Tier: "T2", Enabled: true}},
|
||||
{"TB", Profile{ID: "second", Tier: "T2", Enabled: true}},
|
||||
})
|
||||
if got := reg.getCheapestCapable(false, false, false); got == nil || got.ID != "first" {
|
||||
t.Errorf("stable tie-break: got %v want first", got)
|
||||
}
|
||||
// first-id-match wins in resolve (insertion order)
|
||||
dup := registryFromRaw([]regEntry{
|
||||
{"fake", Profile{ID: "mimo-v2.5-pro", Tier: "T4", Local: true, ContextWindow: 32768, Enabled: true}},
|
||||
{"real", Profile{ID: "mimo-v2.5-pro", Tier: "T4", Local: false, ContextWindow: 128000, Enabled: true}},
|
||||
})
|
||||
b := dup.resolveContextBudget("mimo-v2.5-pro", "default")
|
||||
if b["local"] != true || b["num_ctx"] != 30720 {
|
||||
t.Errorf("first-id-match: %v", b)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1,61 @@
|
||||
# model_registry dichotomy gap-close — verification evidence (2026-10-14)
|
||||
|
||||
Module: src/rune/model_registry.py (253L, 8 fns) → src/go/model-registry
|
||||
(dichotomy wave #2479; worktree rune-wave-db2; contract BUILDER-…-factory-gap-mreg)
|
||||
|
||||
## Before / after
|
||||
|
||||
- Audit coverage: **2/8 PARTIAL → 8/8 FULL (GO)** —
|
||||
missing were `test_registry, to_dict, get_profile, get_enabled_profiles,
|
||||
get_profiles_by_tier, list_profiles`; all ported name-preserving
|
||||
(audit candidate forms): `toDict, getProfile, getEnabledProfiles,
|
||||
getProfilesByTier, listProfiles, testRegistry`.
|
||||
- Programme: FULL 53% → 55% (105/192 modules); fn-level 1612/2282.
|
||||
- KNOWN FLAG VERIFIED AND FIXED: the 2026-10-09 batch-67 port hardcoded
|
||||
`toolReserve = 2048` (main.go:87 of that revision) where the live
|
||||
oracle holds `TOOL_RESULT_RESERVE = 1024` (model_registry.py:225),
|
||||
and a single `outputReserve = 2048` where the oracle has the
|
||||
OUTPUT_RESERVE per-task-kind ladder {default 2048, verify 512,
|
||||
summarize 1024, embed 0} — gated by LOCAL, never by tier. Legacy
|
||||
truth.json never noticed (both rows are local models). Now:
|
||||
`TestReserveConstants` + frozen rows `resolve("mimo-v2.5-pro", …)` →
|
||||
num_ctx 124928 / 125952 / 126976.
|
||||
|
||||
## Verification gates (all green in THIS checkout)
|
||||
|
||||
| Gate | Result | Evidence |
|
||||
|---|---|---|
|
||||
| freeze_truth --verify (byte-identical vs live oracle) | OK: 35 cases | mreg-freeze-verify.{log,exit} |
|
||||
| fresh regeneration byte-identity (2 runs) | identical (cmp) | mreg-freeze-verify.log |
|
||||
| gofmt -l · go vet · go build ./... | clean | mreg-go-ladder.log |
|
||||
| go test ./... (TestGapParity + 3 unit gates) | 4/4 PASS | mreg-go-ladder.log |
|
||||
| CI binary replay-gap (truth_gap.json, freeze format) | 35/35 | mreg-go-ladder.log |
|
||||
| legacy replay (batch-67 truth.json, regression) | 4/4 | mreg-go-ladder.log |
|
||||
| LIVE parity oracle vs GO (scripts/check_model_registry_parity.py) | 4/4 scenarios, 36 ops | mreg-parity.{log,exit} |
|
||||
| Python suite, live pin | 5 passed | mreg-pytests-live.{log,exit} |
|
||||
| Python suite, RUNE_MODEL_REGISTRY_CORE=python pin | 5 passed | mreg-pytests-python.{log,exit} |
|
||||
| Oracle CLI smoke (`model_registry.py test`) | all tests passed | mreg-oracle-smoke.{log,exit} |
|
||||
| Tree-wide `go build ./...` census | 83/86 OK; 3 pre-existing failures (adaptive-router, builder-bridge-cli, paths) link cargo-built Rust staticlibs absent from clean checkouts — not caused by this diff (module untouched) | mreg-tree-build.txt |
|
||||
| dichotomy_audit.py --write | model_registry 8/8 GO row | mreg-audit-write.{log,exit} |
|
||||
|
||||
(ruff check on the three new Python files: all checks passed.)
|
||||
|
||||
## Truth-freeze notes
|
||||
|
||||
- Shared harness (scripts/freeze_truth.py) + rig scripts/mreg_freeze_seams
|
||||
(sitecustomize binds JSON-safe seam_* wrappers — registry methods return
|
||||
ModelProfile dataclasses the harness can't serialize; the real oracle
|
||||
methods run unmodified). 35 cases, envelope frozen_now 2026-10-14T09:00:00
|
||||
(vacuous pin — this oracle has no clock).
|
||||
- Registry states ride as ORDERED [tier_key, profile_dict] pairs; insertion
|
||||
order is load-bearing (get_profile = KEY lookup, get_profiles_by_tier =
|
||||
tier-ATTRIBUTE match, resolve_context_budget breaks at FIRST id match,
|
||||
get_cheapest_capable sorts STABLE on ties).
|
||||
- Receiver-style truth impossible here (module-level fn would bind the
|
||||
receiver to model_id); the `(reg.profiles or {})` empty branch is pinned
|
||||
via seam_resolve_budget_raw (constructor-bypassed __new__ registry).
|
||||
- Constructor falsy law pinned: ModelRegistry({}) → DEFAULT_PROFILES.copy()
|
||||
(rows via [] pairs), while raw-empty stays empty (raw rows).
|
||||
- No loader seam exists in the oracle (no RUNE_MODEL_REGISTRY_CORE reader):
|
||||
the both-pins requirement is satisfied by running the suite + oracle
|
||||
smoke under BOTH env states; delegation-live is vacuous for this module.
|
||||
@@ -0,0 +1,86 @@
|
||||
OK abstraction
|
||||
FAIL adaptive-router
|
||||
OK auto-context
|
||||
OK auto_context
|
||||
OK auto-detect
|
||||
OK auto-tasks
|
||||
OK auto_tasks
|
||||
OK builder-bridge
|
||||
FAIL builder-bridge-cli
|
||||
OK build-mode
|
||||
OK cache
|
||||
OK capability-broker
|
||||
OK cognition
|
||||
OK connection-pool
|
||||
OK context-broker
|
||||
OK core-server
|
||||
OK cron-engine
|
||||
OK curator
|
||||
OK curator-core
|
||||
OK drift-core
|
||||
OK elastic-cognition
|
||||
OK event-bus
|
||||
OK event-envelope
|
||||
OK event-wait
|
||||
OK evidence-probe
|
||||
OK failure-filer
|
||||
OK forge
|
||||
OK forge-client
|
||||
OK forge_client
|
||||
OK forge-core
|
||||
OK goal
|
||||
OK goal-engine
|
||||
OK graph-memory
|
||||
OK hook-engine
|
||||
OK inference-paths
|
||||
OK local-tool-runtime
|
||||
OK memory-consolidation
|
||||
OK memory-ops
|
||||
OK memory-store-deep
|
||||
OK message-queue
|
||||
OK message_queue
|
||||
OK microkernel
|
||||
OK model-connect
|
||||
OK model-profiles
|
||||
OK model-registry
|
||||
OK model-routing
|
||||
OK model_routing
|
||||
OK night-mode
|
||||
OK openai-compat
|
||||
OK openai_compat_transport
|
||||
OK operator
|
||||
OK operator-deep
|
||||
OK operator-support
|
||||
OK outcome-ledger
|
||||
OK parallel-processor
|
||||
FAIL paths
|
||||
OK persona-gate
|
||||
OK pins
|
||||
OK progressive-escalation
|
||||
OK provider-budget
|
||||
OK provider-context
|
||||
OK provider-registry
|
||||
OK recall-router
|
||||
OK recall-uplift
|
||||
OK release-notes
|
||||
OK response-adapter
|
||||
OK rune-entry
|
||||
OK scaffold
|
||||
OK scaffold-membrane
|
||||
OK sleep-time
|
||||
OK stage-guards
|
||||
OK standardized-providers
|
||||
OK subsystem-health
|
||||
OK tiered-executor
|
||||
OK tool-executor
|
||||
OK tool-protocol
|
||||
OK tool-protocol-rw
|
||||
OK tui
|
||||
OK verification_layer
|
||||
OK voicebox
|
||||
OK voice-hybrid
|
||||
OK voice-injector
|
||||
OK voice-sync
|
||||
OK warm-router
|
||||
OK witness
|
||||
OK work-manager
|
||||
Reference in new issue
Block a user