feat(tiered_executor): close PARTIAL gaps — dichotomy wave
audit 3/8 PARTIAL -> 8/8 FULL (GO side). Ported to_dict (CognitiveJob/CognitiveResult via the real constructors), the full execute surface (no-profile early return with attempts=1 and zero latency, T0-deterministic vs model dispatch), _execute_model (OO OLLAMA payload json.dumps byte-parity: ', '/' : ' separators, insertion order, ensure_ascii + surrogate pairs, rune-safe [:2000] response slice, crash-parity envelopes for unknown-url-type + connection-refused), _ctx_budget + vendored ResolveCtxBudget (GO SIBLING ISOLATION law), and test_executor assert-parity (Python True/False literals, crash exit 1 after the Coding line).
Frozen truth: 27-case envelope via the shared freeze_truth.py + scripts/te_freeze_seams rig (time.time pinned -> latency 0.0 mirrored by the TE_LATENCY_MS seam; OLLAMA echo server reflects the request body so payload bytes ride the truth rows; ModelRegistry({}) falsiness -> DEFAULT_PROFILES, no-profile rows via the T9 sentinel). Verified byte-identical twice. Battery 11 tests green; CI replay binary PARITY OK 27/27; live parity sweep 31/31 (25 op families + 6 CLI incl. test / execute / usage); gofmt/vet/build clean; python smoke both pins (no loader seam - vacuous pin, PP precedent).
Evidence: verification/te-*.exit/*.exit files + te-summary.md (logs local, * log gitignored).
This commit is contained in:
Nick committed
1 parent
649febfff7
commit
a858ab1392
21 files changed
+2824
-195
No files matched your search
@@ -2,14 +2,14 @@
|
||||
|
||||
_Regenerate with `python3 scripts/dichotomy_audit.py --write`. A module is FULL only when 80%+ of its Python functions have name-preserving counterparts in the Go/Rust implementation — directories without the functions are facades and count as NOT TOUCHED._
|
||||
|
||||
**FULL completion: 56%** (108/192 runtime modules at FULL)
|
||||
**FULL completion: 57%** (109/192 runtime modules at FULL)
|
||||
|
||||
Function-level detail: 71% (1627/2284 runtime functions matched) — secondary; includes in-flight work
|
||||
Function-level detail: 71% (1632/2284 runtime functions matched) — secondary; includes in-flight work
|
||||
|
||||
| Status | Modules | Lines | Meaning |
|
||||
|---|---|---|---|
|
||||
| FULL | 108 | 45483 | 80%+ of functions ported |
|
||||
| PARTIAL | 45 | 18035 | 20–80% — real logic exists, gaps named below |
|
||||
| FULL | 109 | 45774 | 80%+ of functions ported |
|
||||
| PARTIAL | 44 | 17744 | 20–80% — real logic exists, gaps named below |
|
||||
| NOT TOUCHED | 39 | 11471 | no name-preserving implementation |
|
||||
| Total runtime | 192 | 74989 | |
|
||||
|
||||
@@ -75,7 +75,6 @@ Function-level detail: 71% (1627/2284 runtime functions matched) — secondary;
|
||||
| voicebox | 164 | 5/11 | _run, start_recording, stop_recording, _transcribe_and_type, on_press, on_release |
|
||||
| persona_definition | 210 | 2/7 | _by_topic, compile_full, emit_constitution, emit_identity, emit_persona |
|
||||
| task_persist | 269 | 5/10 | _save, export, restore, main, handle_hook |
|
||||
| tiered_executor | 291 | 3/8 | _ctx_budget, test_executor, to_dict, to_dict, _execute_model |
|
||||
| construction_commands | 268 | 1/5 | _refuse, _validate, _accept, dispatch |
|
||||
| drift_observability | 399 | 5/9 | _do_core, main, _load_alerts, _save_alerts |
|
||||
| pins | 282 | 7/11 | _store_path, _detail_blob, unpin, list_pins |
|
||||
@@ -180,6 +179,7 @@ Function-level detail: 71% (1627/2284 runtime functions matched) — secondary;
|
||||
| stage_guards | 303 | 9/9 | GO |
|
||||
| message_graph | 295 | 12/13 | unclassified |
|
||||
| toolchain | 295 | 5/6 | RUST |
|
||||
| tiered_executor | 291 | 8/8 | GO |
|
||||
| salience_gate | 283 | 6/6 | RUST |
|
||||
| standardized_providers | 281 | 5/5 | GO |
|
||||
| shard_manager | 276 | 9/9 | RUST |
|
||||
|
||||
@@ -0,0 +1,250 @@
|
||||
#!/usr/bin/env python3
|
||||
"""check_tiered_executor_parity.py — live oracle-vs-Go parity sweep for
|
||||
the tiered_executor gap closure (dichotomy wave #2479, contract
|
||||
factory-gap-cs).
|
||||
|
||||
Run under the scripts/te_freeze_seams rig (PYTHONPATH), which pins
|
||||
time.time (both sides then ride the TE_LATENCY_MS=0 wall-clock seam) and
|
||||
starts the OLLAMA echo server the frozen envelope was captured with.
|
||||
THIS script drives the LIVE Python oracle and the Go binary (`op` stdin
|
||||
dispatch) across every op family — job/result serializers via the real
|
||||
constructors, deterministic family paths incl. rune-slicing truncation,
|
||||
no-profile early returns through subset registries, model paths through
|
||||
the echo server (payload bytes ride back as the response), crash-parity
|
||||
error envelopes, _ctx_budget rows, the TestExecutor self-test — plus the
|
||||
CLI surface (test / execute / usage / unknown), comparing sort_keys JSON
|
||||
or exact lines. Exit 0 on 100% parity.
|
||||
|
||||
Usage: PYTHONPATH=scripts/te_freeze_seams REPO_ROOT=$PWD python3 \\
|
||||
scripts/check_tiered_executor_parity.py [--bin src/go/tiered-executor/tiered-executor]
|
||||
"""
|
||||
import argparse
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
LAT_RE = re.compile(r'"latency_ms": [0-9.eE+-]+')
|
||||
|
||||
|
||||
def norm_latency(text):
|
||||
"""Wall-clock law: the CLI execute output embeds a measured latency
|
||||
(oracle: live time.time; Go: measured unless the seam pin rides env)
|
||||
— replaced structurally on BOTH sides before the byte compare."""
|
||||
return LAT_RE.sub('"latency_ms": <wall-clock>', text)
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(REPO / "src"))
|
||||
|
||||
import rune.tiered_executor as te # rig-bound: _te_* seams + frozen time # noqa: E402
|
||||
from rune.model_registry import ModelProfile # noqa: E402
|
||||
|
||||
T3_PROFILE = {
|
||||
"id": "qwen3.6:27b", "provider": "ollama", "tier": "T3", "local": True,
|
||||
"context_window": 32768, "supports_tools": False,
|
||||
"supports_structured_output": False, "supports_reasoning_field": False,
|
||||
"estimated_latency": "high", "estimated_cost": "0", "enabled": True,
|
||||
}
|
||||
EMPTY_REG = {"T9": {**T3_PROFILE, "id": "none", "tier": "T9"}}
|
||||
|
||||
BUDGET_T0 = {"start_tier": "T0", "max_tier": "T0"}
|
||||
BUDGET_T2 = {"start_tier": "T2", "max_tier": "T4"}
|
||||
|
||||
|
||||
def echo_url():
|
||||
return os.environ["OLLAMA_URL"]
|
||||
|
||||
|
||||
# ── oracle side (rig seams already give registry specs / per-case urls) ──────
|
||||
|
||||
def py_op(op, req):
|
||||
if op == "to_dict_job":
|
||||
return te._te_job_to_dict(req["job"])
|
||||
if op == "to_dict_result":
|
||||
return te._te_result_to_dict(req["result"])
|
||||
if op == "execute":
|
||||
return te._te_execute(req.get("registry"), req["job"], req["budget"])
|
||||
if op == "execute_ollama":
|
||||
return te._te_execute_ollama(req.get("url"), req.get("registry"),
|
||||
req["job"], req["budget"])
|
||||
if op == "ctx_budget":
|
||||
return te._ctx_budget(req["model"])
|
||||
if op == "test_executor":
|
||||
# te.test_executor prints its report inline — capture it (the CLI
|
||||
# parity block below verifies those printed lines byte-exactly)
|
||||
sink = io.StringIO()
|
||||
with contextlib.redirect_stdout(sink):
|
||||
return te._te_test_executor()
|
||||
raise ValueError(f"unknown op {op}")
|
||||
|
||||
|
||||
# ── Go side (one fresh process per op) ───────────────────────────────────────
|
||||
|
||||
def go_op(binpath, op, req):
|
||||
full = dict(req)
|
||||
full["op"] = op
|
||||
env = dict(os.environ)
|
||||
env["TE_LATENCY_MS"] = "0"
|
||||
p = subprocess.run([str(binpath), "op"], input=json.dumps(full),
|
||||
capture_output=True, text=True, env=env, cwd=str(REPO))
|
||||
if p.returncode != 0:
|
||||
raise RuntimeError(f"go op {op} exited {p.returncode}: {p.stdout} {p.stderr}")
|
||||
return json.loads(p.stdout)
|
||||
|
||||
|
||||
def jdump(v):
|
||||
return json.dumps(_norm(v), sort_keys=True, ensure_ascii=True)
|
||||
|
||||
|
||||
def _norm(v):
|
||||
"""Canonical walk: integral floats collapse to int for the COMPARISON
|
||||
only (dichotomy quirk — Python renders 0.0, Go 0; parsed-equal same
|
||||
number). Non-integral floats (latency 12.5) stay untouched."""
|
||||
if isinstance(v, bool):
|
||||
return v
|
||||
if isinstance(v, float) and v.is_integer():
|
||||
return int(v)
|
||||
if isinstance(v, list):
|
||||
return [_norm(x) for x in v]
|
||||
if isinstance(v, dict):
|
||||
return {k: _norm(x) for k, x in v.items()}
|
||||
return v
|
||||
|
||||
|
||||
OPS = [
|
||||
("to_dict_job", {"job": {"job_id": "p1", "task_family": "greeting", "objective": "Respond to hello"}}),
|
||||
("to_dict_job", {"job": {"job_id": "p2", "task_family": "coding", "objective": "日本語 café",
|
||||
"inputs": {"path": "src/x.py"}, "risk": "high",
|
||||
"reversibility": "irreversible", "verification_method": "pytest",
|
||||
"preferred_tiers": ["T0", "T1"], "max_tier": "T3",
|
||||
"required_output_schema": "diff"}}),
|
||||
("to_dict_result", {"result": {"job_id": "p3", "tier_used": "T0", "model_used": "deterministic", "success": True}}),
|
||||
("to_dict_result", {"result": {"job_id": "p4", "tier_used": "T1", "model_used": "qwen2.5:3b",
|
||||
"success": False, "output": None, "error": "kaput",
|
||||
"verification_result": {"ok": 1}, "attempts": 2,
|
||||
"latency_ms": 9.75, "tokens_used": 7}}),
|
||||
("execute", {"job": {"job_id": "p5", "task_family": "greeting", "objective": "salut"}, "budget": BUDGET_T0}),
|
||||
("execute", {"job": {"job_id": "p6", "task_family": "lookup", "objective": "find x"}, "budget": BUDGET_T0}),
|
||||
("execute", {"job": {"job_id": "p7", "task_family": "other", "objective": "plain"}, "budget": BUDGET_T0}),
|
||||
# [:100] char-slice rows (dichotomy quirk: rune-safe truncation)
|
||||
("execute", {"job": {"job_id": "p8", "task_family": "greeting", "objective": "x" * 140}, "budget": BUDGET_T0}),
|
||||
("execute", {"job": {"job_id": "p9", "task_family": "lookup", "objective": "日本語" * 60}, "budget": BUDGET_T0}),
|
||||
("execute", {"job": {"job_id": "p10", "task_family": "coding", "objective": "héllo wörld", "preferred_tiers": ["T0"]},
|
||||
"budget": BUDGET_T2}),
|
||||
# no-profile early returns through a subset registry (empty dict is FALSY)
|
||||
("execute", {"registry": EMPTY_REG, "job": {"job_id": "p11", "task_family": "coding", "objective": "x",
|
||||
"preferred_tiers": ["T4"]}, "budget": {"start_tier": "T4", "max_tier": "T4"}}),
|
||||
("execute", {"registry": EMPTY_REG, "job": {"job_id": "p12", "task_family": "coding", "objective": "x"},
|
||||
"budget": {"start_tier": "T3", "max_tier": "T3"}}),
|
||||
# model paths through the rig's echo server (payload rides back)
|
||||
("execute_ollama", {"job": {"job_id": "p13", "task_family": "coding", "objective": "fix it"},
|
||||
"budget": {"start_tier": "T1", "max_tier": "T2"}}),
|
||||
("execute_ollama", {"job": {"job_id": "p14", "task_family": "coding",
|
||||
"objective": ('héllo wörld ' * 80) + '日本語 "q" \\ \t\n' + ("🚀" * 30)},
|
||||
"budget": {"start_tier": "T1", "max_tier": "T2"}}),
|
||||
("execute_ollama", {"job": {"job_id": "p15", "task_family": "coding", "objective": "preferred",
|
||||
"preferred_tiers": ["T2"]},
|
||||
"budget": {"start_tier": "T0", "max_tier": "T4"}}),
|
||||
("execute_ollama", {"registry": {"T1": {**T3_PROFILE, "id": "custom-1b", "tier": "T1"}},
|
||||
"job": {"job_id": "p16", "task_family": "coding", "objective": "custom"},
|
||||
"budget": {"start_tier": "T1", "max_tier": "T2"}}),
|
||||
# crash-parity rows through the model path
|
||||
("execute_ollama", {"url": "definitely-not-a-url",
|
||||
"job": {"job_id": "p17", "task_family": "coding", "objective": "bad"},
|
||||
"budget": {"start_tier": "T1", "max_tier": "T2"}}),
|
||||
("execute_ollama", {"url": "http://127.0.0.1:9",
|
||||
"job": {"job_id": "p18", "task_family": "coding", "objective": "refused"},
|
||||
"budget": {"start_tier": "T1", "max_tier": "T2"}}),
|
||||
("ctx_budget", {"model": "qwen2.5:3b"}),
|
||||
("ctx_budget", {"model": "qwen3:14b"}),
|
||||
("ctx_budget", {"model": "mimo-v2.5-pro"}),
|
||||
("ctx_budget", {"model": ""}),
|
||||
("ctx_budget", {"model": "deterministic"}),
|
||||
("ctx_budget", {"model": "no-such-model"}),
|
||||
("test_executor", {}),
|
||||
]
|
||||
|
||||
|
||||
def cli_lines(argv, env_extra=None):
|
||||
env = dict(os.environ)
|
||||
if env_extra:
|
||||
env.update(env_extra)
|
||||
env["TE_LATENCY_MS"] = "0"
|
||||
p = subprocess.run(argv, capture_output=True, text=True, env=env, cwd=str(REPO))
|
||||
return p.returncode, p.stdout, p.stderr
|
||||
|
||||
|
||||
def cli_parity(binpath):
|
||||
"""CLI surface: test / execute T0 / execute T1 (echo) / usage /
|
||||
unknown arg / execute with a missing arg — the oracle's main()."""
|
||||
checks = []
|
||||
oracle = [sys.executable, str(REPO / "src/rune/tiered_executor.py")]
|
||||
|
||||
rc_p, out_p, _ = cli_lines(oracle + ["test"], {"OLLAMA_URL": echo_url()})
|
||||
rc_g, out_g, _ = cli_lines([str(binpath), "test"], {"OLLAMA_URL": echo_url()})
|
||||
checks.append(("cli test", rc_p == rc_g and out_p == out_g, out_p, out_g))
|
||||
|
||||
def run(argv, env_extra):
|
||||
rc, out, _ = cli_lines(argv, env_extra)
|
||||
return rc, norm_latency(out), _
|
||||
|
||||
job = json.dumps({"job_id": "c1", "task_family": "greeting", "objective": "hi"})
|
||||
rc_p, out_p, _ = run(oracle + ["execute", job, json.dumps(BUDGET_T0)], {"OLLAMA_URL": echo_url()})
|
||||
rc_g, out_g, _ = run([str(binpath), "execute", job, json.dumps(BUDGET_T0)], {"OLLAMA_URL": echo_url()})
|
||||
checks.append(("cli execute T0 (latency normalized)", rc_p == rc_g and out_p == out_g, out_p, out_g))
|
||||
|
||||
job_t1 = json.dumps({"job_id": "c2", "task_family": "coding", "objective": "fix it"})
|
||||
rc_p, out_p, _ = run(oracle + ["execute", job_t1, json.dumps({"start_tier": "T1", "max_tier": "T2"})],
|
||||
{"OLLAMA_URL": echo_url()})
|
||||
rc_g, out_g, _ = run([str(binpath), "execute", job_t1, json.dumps({"start_tier": "T1", "max_tier": "T2"})],
|
||||
{"OLLAMA_URL": echo_url()})
|
||||
checks.append(("cli execute T1 via echo (latency normalized)", rc_p == rc_g and out_p == out_g, out_p, out_g))
|
||||
|
||||
rc_p, out_p, _ = cli_lines(oracle + ["execute", job])
|
||||
rc_g, out_g, _ = cli_lines([str(binpath), "execute", job])
|
||||
checks.append(("cli execute missing budget -> usage", rc_p == rc_g and out_p == out_g, out_p, out_g))
|
||||
|
||||
rc_p, out_p, _ = cli_lines(oracle + ["bogus"])
|
||||
rc_g, out_g, _ = cli_lines([str(binpath), "bogus"])
|
||||
checks.append(("cli unknown", rc_p == rc_g and out_p == out_g, out_p, out_g))
|
||||
|
||||
rc_p, out_p, _ = cli_lines(oracle)
|
||||
rc_g, out_g, _ = cli_lines([str(binpath)])
|
||||
checks.append(("cli usage", rc_p == rc_g and out_p == out_g, out_p, out_g))
|
||||
return checks
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--bin", default=str(REPO / "src/go/tiered-executor/tiered-executor"))
|
||||
args = ap.parse_args()
|
||||
binpath = Path(args.bin)
|
||||
|
||||
failures = 0
|
||||
total = 0
|
||||
for op, req in OPS:
|
||||
total += 1
|
||||
py = py_op(op, req)
|
||||
go = go_op(binpath, op, req)
|
||||
if jdump(py) != jdump(go):
|
||||
failures += 1
|
||||
print(f"DIVERGE {op} {json.dumps(req)[:100]}")
|
||||
print(f" python: {jdump(py)[:400]}")
|
||||
print(f" go: {jdump(go)[:400]}")
|
||||
|
||||
for name, ok, want, got in cli_parity(binpath):
|
||||
total += 1
|
||||
if not ok:
|
||||
failures += 1
|
||||
print(f"DIVERGE {name}\n python: {want!r}\n go: {got!r}")
|
||||
|
||||
print(f"parity: {total - failures}/{total} match" + (" — FAILURES" if failures else ""))
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,177 @@
|
||||
#!/usr/bin/env python3
|
||||
"""gen_tiered_executor_truth.py — freeze the tiered_executor oracle truth
|
||||
(dichotomy gap-close, wave #2479) through the SHARED harness
|
||||
(scripts/freeze_truth.py) under the scripts/te_freeze_seams rig.
|
||||
|
||||
Surfaces frozen (the 5 audit-missing fns + the already-covered surface,
|
||||
so the battery replays ONE envelope):
|
||||
* CognitiveJob.to_dict / CognitiveResult.to_dict — routed through the
|
||||
REAL constructors (defaults intact) by the rig's _te_* seams
|
||||
* TieredExecutor.execute — deterministic family paths (rune-safe
|
||||
[:100] truncation rows, unicode + emoji), no-profile early returns
|
||||
via subset registries built by the rig, model paths against the
|
||||
rig's OLLAMA echo server (payload bytes pinned via the echo round-
|
||||
trip), bad-scheme + connection-refused crash-parity rows
|
||||
* _ctx_budget — direct rows over the vendored resolve_context_budget
|
||||
surface (local/cloud/unknown model ids)
|
||||
* test_executor — captured assert-safe ({"returned": null})
|
||||
|
||||
latency_ms is exactly 0.0 in every row: the rig freezes time.time and the
|
||||
Go battery pins the TE_LATENCY_MS seam to the same epoch law.
|
||||
|
||||
Usage: PYTHONPATH=scripts/te_freeze_seams REPO_ROOT=$PWD python3 scripts/gen_tiered_executor_truth.py
|
||||
"""
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(REPO / "scripts"))
|
||||
sys.path.insert(0, str(REPO / "src"))
|
||||
|
||||
from freeze_truth import freeze, write_envelope # noqa: E402
|
||||
|
||||
FROZEN_NOW = "2026-10-14T10:00:00+00:00"
|
||||
OUT = REPO / "src" / "go" / "tiered-executor" / "truth.json"
|
||||
|
||||
# A real T3 profile dict (subset registries build ModelProfile(**d) from it)
|
||||
T3_PROFILE = {
|
||||
"id": "qwen3.6:27b", "provider": "ollama", "tier": "T3", "local": True,
|
||||
"context_window": 32768, "supports_tools": False,
|
||||
"supports_structured_output": False, "supports_reasoning_field": False,
|
||||
"estimated_latency": "high", "estimated_cost": "0", "enabled": True,
|
||||
}
|
||||
|
||||
# objective mixing BMP escapes, an astral surrogate pair, quotes, backslash,
|
||||
# a tab and a newline — payload must exceed 2000 chars to exercise [:2000]
|
||||
WIDE = ("héllo wörld " * 80) + ('日本語 "quoted" \\backslash\t\n') + ("🚀" * 30)
|
||||
|
||||
# ModelRegistry({}) is falsy -> falls back to DEFAULT_PROFILES, so a
|
||||
# "no profiles" registry must carry a sentinel profile under a tier no
|
||||
# case selects: every get_profile(<real tier>) then misses -> None.
|
||||
EMPTY_REG = {"T9": {**T3_PROFILE, "id": "none", "tier": "T9"}}
|
||||
|
||||
LONG_ASCII_105 = "x" * 105
|
||||
|
||||
BUDGET_T0 = {"start_tier": "T0", "max_tier": "T0"}
|
||||
BUDGET_T1 = {"start_tier": "T1", "max_tier": "T2"}
|
||||
BUDGET_T2 = {"start_tier": "T2", "max_tier": "T4"}
|
||||
BUDGET_T3 = {"start_tier": "T3", "max_tier": "T4"}
|
||||
BUDGET_T4 = {"start_tier": "T4", "max_tier": "T4"}
|
||||
|
||||
CASES = [
|
||||
# ── CognitiveJob.to_dict via the real constructor ────────────────────────
|
||||
("_te_job_to_dict", ({"job_id": "m1", "task_family": "greeting",
|
||||
"objective": "Respond to hello"},), {}),
|
||||
("_te_job_to_dict", ({"job_id": "m2", "task_family": "architecture",
|
||||
"objective": WIDE, "inputs": {"path": "src/x.py"},
|
||||
"required_output_schema": "diff", "risk": "high",
|
||||
"reversibility": "irreversible",
|
||||
"verification_method": "pytest",
|
||||
"preferred_tiers": ["T0", "T1"], "max_tier": "T3"},), {}),
|
||||
# ── CognitiveResult.to_dict via the real constructor ─────────────────────
|
||||
("_te_result_to_dict", ({"job_id": "r1", "tier_used": "T0",
|
||||
"model_used": "deterministic", "success": True},), {}),
|
||||
("_te_result_to_dict", ({"job_id": "r2", "tier_used": "T1",
|
||||
"model_used": "qwen2.5:3b", "success": False,
|
||||
"output": {"response": "partial"},
|
||||
"error": "boom", "verification_result": {"ok": 1},
|
||||
"attempts": 3, "latency_ms": 12.5,
|
||||
"tokens_used": 42},), {}),
|
||||
# ── execute(): deterministic family paths (registry default, T0) ─────────
|
||||
("_te_execute", (None, {"job_id": "e1", "task_family": "greeting",
|
||||
"objective": "hi there"}, BUDGET_T0), {}),
|
||||
("_te_execute", (None, {"job_id": "e2", "task_family": "lookup",
|
||||
"objective": "find the config"}, BUDGET_T0), {}),
|
||||
("_te_execute", (None, {"job_id": "e3", "task_family": "coding",
|
||||
"objective": "run the build"}, BUDGET_T0), {}),
|
||||
("_te_execute", (None, {"job_id": "e4", "task_family": "greeting",
|
||||
"objective": LONG_ASCII_105}, BUDGET_T0), {}),
|
||||
("_te_execute", (None, {"job_id": "e5", "task_family": "greeting",
|
||||
"objective": "日本語" * 40}, BUDGET_T0), {}),
|
||||
("_te_execute", (None, {"job_id": "e6", "task_family": "lookup",
|
||||
"objective": "héllo wörld café", "required_output_schema": "y"},
|
||||
BUDGET_T0), {}),
|
||||
# preferred_tiers honored before the budget start tier
|
||||
("_te_execute", (None, {"job_id": "e7", "task_family": "greeting",
|
||||
"objective": "pref first",
|
||||
"preferred_tiers": ["T0"]}, BUDGET_T2), {}),
|
||||
# ── execute(): no-profile early return (subset registry, no profiles) ──
|
||||
("_te_execute", (EMPTY_REG, {"job_id": "e8", "task_family": "coding",
|
||||
"objective": "x", "preferred_tiers": ["T3"]},
|
||||
BUDGET_T3), {}),
|
||||
("_te_execute", (EMPTY_REG, {"job_id": "e9", "task_family": "coding",
|
||||
"objective": "x"}, BUDGET_T3), {}),
|
||||
# ── execute(): model paths against the rig's OLLAMA echo server ─────────
|
||||
("_te_execute_ollama", (None, None, {"job_id": "m1", "task_family": "coding",
|
||||
"objective": "fix it"}, BUDGET_T1), {}),
|
||||
("_te_execute_ollama", (None, None, {"job_id": "m2", "task_family": "coding",
|
||||
"objective": WIDE},
|
||||
BUDGET_T1), {}),
|
||||
("_te_execute_ollama", (None, None, {"job_id": "m3", "task_family": "coding",
|
||||
"objective": "preferred tier",
|
||||
"preferred_tiers": ["T2"]},
|
||||
{"start_tier": "T0", "max_tier": "T4"},
|
||||
), {}),
|
||||
("_te_execute_ollama", (None, {"T3": T3_PROFILE}, {"job_id": "m4",
|
||||
"task_family": "coding",
|
||||
"objective": "subset reg"},
|
||||
BUDGET_T3), {}),
|
||||
("_te_execute_ollama", (None, None, {"job_id": "m5", "task_family": "coding",
|
||||
"objective": "cloud tier"},
|
||||
BUDGET_T4), {}),
|
||||
# ── execute(): crash-parity rows through the model path ─────────────────
|
||||
("_te_execute_ollama", ("definitely-not-a-url", None,
|
||||
{"job_id": "x1", "task_family": "coding",
|
||||
"objective": "bad scheme"}, BUDGET_T1), {}),
|
||||
("_te_execute_ollama", ("http://127.0.0.1:9", None,
|
||||
{"job_id": "x2", "task_family": "coding",
|
||||
"objective": "refused"}, BUDGET_T1), {}),
|
||||
# ── _ctx_budget (vendored resolve_context_budget surface) ───────────────
|
||||
("_ctx_budget", ("qwen2.5:3b",), {}),
|
||||
("_ctx_budget", ("qwen3:14b",), {}),
|
||||
("_ctx_budget", ("qwen3.6:27b",), {}),
|
||||
("_ctx_budget", ("mimo-v2.5-pro",), {}),
|
||||
("_ctx_budget", ("",), {}),
|
||||
("_ctx_budget", ("no-such-model",), {}),
|
||||
# ── test_executor (assert-safe capture; echo server answers the T2 job) ──
|
||||
("_te_test_executor", (), {}),
|
||||
]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
truth = freeze("rune.tiered_executor", CASES, oracle_module="rune.tiered_executor")
|
||||
write_envelope(FROZEN_NOW, truth, OUT)
|
||||
|
||||
by_fn = {}
|
||||
for row in truth:
|
||||
by_fn.setdefault(row["fn"], []).append(row)
|
||||
assert len(truth) == len(CASES), f"freeze produced {len(truth)} rows for {len(CASES)} cases"
|
||||
assert by_fn["_te_job_to_dict"][0]["expected"]["inputs"] == {}
|
||||
assert by_fn["_te_result_to_dict"][1]["expected"]["verification_result"] == {"ok": 1}
|
||||
assert by_fn["_te_execute"][0]["expected"]["output"] == {"message": "Hello! Processing: hi there"}
|
||||
assert len(by_fn["_te_execute"][3]["expected"]["output"]["message"]) == 119
|
||||
assert len(by_fn["_te_execute"][4]["expected"]["output"]["message"]) == 119
|
||||
assert by_fn["_te_execute"][7]["expected"]["error"] == "No profile for tier T3"
|
||||
assert by_fn["_te_execute"][8]["expected"]["error"] == "No profile for tier T3"
|
||||
echo_row = by_fn["_te_execute_ollama"][0]["expected"]
|
||||
assert echo_row["success"] is True and echo_row["tier_used"] == "T1"
|
||||
assert echo_row["output"]["response"].startswith('{"model": "qwen2.5:3b", "prompt": "fix it"')
|
||||
assert by_fn["_te_execute_ollama"][2]["expected"]["tier_used"] == "T2"
|
||||
assert by_fn["_te_execute_ollama"][3]["expected"]["model_used"] == "qwen3.6:27b"
|
||||
assert by_fn["_te_execute_ollama"][4]["expected"]["model_used"] == "mimo-v2.5-pro"
|
||||
trunc_row = by_fn["_te_execute_ollama"][1]["expected"]
|
||||
assert len(trunc_row["output"]["response"]) == 2000, len(trunc_row["output"]["response"])
|
||||
assert by_fn["_te_execute_ollama"][5]["expected"]["error"].startswith("unknown url type")
|
||||
assert by_fn["_te_execute_ollama"][6]["expected"]["error"] == "<urlopen error [Errno 111] Connection refused>"
|
||||
assert by_fn["_ctx_budget"][0]["expected"] == 30720
|
||||
assert by_fn["_ctx_budget"][3]["expected"] == 124928
|
||||
assert by_fn["_te_test_executor"][0]["expected"] == {"returned": None}
|
||||
|
||||
print(f"wrote {OUT}: {len(truth)} cases, frozen_now={FROZEN_NOW}")
|
||||
for fn, rows in by_fn.items():
|
||||
print(f" {fn}: {len(rows)} row(s)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,139 @@
|
||||
"""te_freeze_seams — freeze rig for the tiered_executor gap-close port
|
||||
(dichotomy wave #2479, GO side).
|
||||
|
||||
Bindings (loaded via PYTHONPATH for freeze-gen, freeze-verify AND the
|
||||
parity sweep):
|
||||
* RUNE_TIERED_EXECUTOR_CORE=python (oracle runs pure Python)
|
||||
* time.time frozen -> every measured latency_ms in the oracle is exactly
|
||||
0.0 in frozen truth (wall-clock law; the Go battery mirrors via the
|
||||
TE_LATENCY_MS seam pinned to the same value)
|
||||
* a local OLLAMA echo server (ports 20145-20154, first bindable) whose
|
||||
/api/generate reflects the received REQUEST BODY back as
|
||||
{"response": body} — the oracle echoes the payload JSON back, so the
|
||||
frozen rows pin the exact json.dumps payload bytes (separators ', '
|
||||
': ', ensure_ascii) and the [:2000] response slice
|
||||
* _te_* seam helpers bound onto the module: dict-shaped registry specs
|
||||
-> real ModelRegistry/ModelProfile instances, job/result serializer
|
||||
rows routed through the REAL constructors (defaults intact), the
|
||||
test_executor self-test captured without letting an assert unwind the
|
||||
freeze process
|
||||
|
||||
The echo server means model-path rows are fully deterministic WITHOUT
|
||||
touching a real Ollama. Rows carry no absolute paths and no wall-clock
|
||||
values beyond the pinned 0.0 latencies.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import threading
|
||||
import time as _time_mod
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
|
||||
REPO = os.environ.get("REPO_ROOT")
|
||||
if not REPO:
|
||||
# sitecustomize also runs for bare interpreters — derive from rig path
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
sys.path.insert(0, os.path.join(REPO, "scripts"))
|
||||
sys.path.insert(0, os.path.join(REPO, "src"))
|
||||
|
||||
from freeze_rigs import pin_cores # noqa: E402
|
||||
|
||||
pin_cores("tiered_executor")
|
||||
|
||||
# ── wall-clock law: freeze time.time for the whole capture/verify process ────
|
||||
TE_FROZEN_EPOCH = 1760000000.0
|
||||
_time_mod.time = lambda: TE_FROZEN_EPOCH # latency_ms everywhere -> 0.0
|
||||
|
||||
|
||||
# ── OLLAMA echo seam: reflect the request body back as {"response": body} ────
|
||||
class _TeeEcho(BaseHTTPRequestHandler):
|
||||
def do_POST(self):
|
||||
length = int(self.headers.get("Content-Length", 0))
|
||||
body = self.rfile.read(length).decode("utf-8")
|
||||
resp = json.dumps({"response": body}).encode("utf-8")
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(resp)))
|
||||
self.end_headers()
|
||||
self.wfile.write(resp)
|
||||
|
||||
def log_message(self, *a): # silence
|
||||
pass
|
||||
|
||||
|
||||
def _start_echo():
|
||||
for port in range(20145, 20155):
|
||||
try:
|
||||
srv = HTTPServer(("127.0.0.1", port), _TeeEcho)
|
||||
except OSError:
|
||||
continue
|
||||
threading.Thread(target=srv.serve_forever, daemon=True).start()
|
||||
os.environ["OLLAMA_URL"] = f"http://127.0.0.1:{port}"
|
||||
return srv
|
||||
raise RuntimeError("te_freeze_seams: no echo port available (20145-20154)")
|
||||
|
||||
|
||||
_ECHO = _start_echo()
|
||||
|
||||
# ── oracle import + seam bindings ────────────────────────────────────────────
|
||||
from rune import tiered_executor as te # noqa: E402
|
||||
from rune.model_registry import ModelProfile, ModelRegistry # noqa: E402
|
||||
|
||||
|
||||
def _te_registry(spec):
|
||||
"""spec None -> default registry; {tier: profile_dict} -> subset registry
|
||||
built from REAL ModelProfile instances (missing tiers then behave
|
||||
exactly like the oracle's get_profile -> None branch)."""
|
||||
if spec is None:
|
||||
return ModelRegistry()
|
||||
profiles = {tier: ModelProfile(**d) for tier, d in spec.items()}
|
||||
return ModelRegistry(profiles)
|
||||
|
||||
|
||||
def _te_execute(regspec, job, budget):
|
||||
"""Real oracle execute() with the registry injected; result serialized
|
||||
through the oracle's own CognitiveResult.to_dict()."""
|
||||
ex = te.TieredExecutor(te._te_registry(regspec))
|
||||
r = ex.execute(te.CognitiveJob(**job), te.CognitiveBudget(**budget))
|
||||
return r.to_dict()
|
||||
|
||||
|
||||
def _te_execute_ollama(url, regspec, job, budget):
|
||||
"""Per-case OLLAMA_URL seam (None -> the rig's echo default, or an
|
||||
explicit bad-scheme / refused-port url)."""
|
||||
old = os.environ.get("OLLAMA_URL")
|
||||
if url is not None:
|
||||
os.environ["OLLAMA_URL"] = url
|
||||
try:
|
||||
return _te_execute(regspec, job, budget)
|
||||
finally:
|
||||
if url is not None:
|
||||
if old is None:
|
||||
os.environ.pop("OLLAMA_URL", None)
|
||||
else:
|
||||
os.environ["OLLAMA_URL"] = old
|
||||
|
||||
|
||||
def _te_job_to_dict(job):
|
||||
return te.CognitiveJob(**job).to_dict()
|
||||
|
||||
|
||||
def _te_result_to_dict(result):
|
||||
return te.CognitiveResult(**result).to_dict()
|
||||
|
||||
|
||||
def _te_test_executor():
|
||||
"""Capture the oracle self-test without letting an assert unwind us."""
|
||||
try:
|
||||
te.test_executor()
|
||||
return {"returned": None}
|
||||
except Exception as e: # noqa: BLE001 — the captured message IS the row
|
||||
return {"raised": str(e)}
|
||||
|
||||
|
||||
te._te_registry = _te_registry
|
||||
te._te_execute = _te_execute
|
||||
te._te_execute_ollama = _te_execute_ollama
|
||||
te._te_job_to_dict = _te_job_to_dict
|
||||
te._te_result_to_dict = _te_result_to_dict
|
||||
te._te_test_executor = _te_test_executor
|
||||
@@ -0,0 +1,126 @@
|
||||
// context_budget.go — byte-parity vendoring of src/rune/model_registry.py's
|
||||
// resolve_context_budget + the DEFAULT_PROFILES slice the tiered-executor
|
||||
// surface reads (GO SIBLING ISOLATION: all Go modules are package main —
|
||||
// shared registries get vendored, never refactored across modules).
|
||||
//
|
||||
// Python semantics preserved:
|
||||
// * registry.profiles or DEFAULT_PROFILES.copy() (an empty dict is FALSY
|
||||
// — see the sentinel-registry truth rows)
|
||||
// * window = profile.context_window or 8192 (0 -> 8192)
|
||||
// * unknown model / unregistered id -> window 8192, local True
|
||||
// * num_ctx = max(1024, window - output_reserve - tool_reserve)
|
||||
|
||||
package main
|
||||
|
||||
type ModelProfile struct {
|
||||
ID string
|
||||
Provider string
|
||||
Tier string
|
||||
Local bool
|
||||
ContextWindow int
|
||||
SupportsTools bool
|
||||
SupportsStructuredOutput bool
|
||||
SupportsReasoningField bool
|
||||
EstimatedLatency string
|
||||
EstimatedCost string
|
||||
Enabled bool
|
||||
}
|
||||
|
||||
// ModelRegistry — the vendored registry mirror; profiles ride insertion
|
||||
// order like the Python dict (matters if two ids ever alias one model).
|
||||
type ModelRegistry struct {
|
||||
order []string
|
||||
profiles map[string]*ModelProfile
|
||||
}
|
||||
|
||||
// NewModelRegistry mirrors registry or DEFAULT_PROFILES.copy(): a nil or
|
||||
// EMPTY profile map falls back to the default ladder (falsiness law).
|
||||
func NewModelRegistry(profiles map[string]*ModelProfile) *ModelRegistry {
|
||||
r := &ModelRegistry{profiles: map[string]*ModelProfile{}}
|
||||
if profiles == nil || len(profiles) == 0 {
|
||||
SetDefaultProfiles(r)
|
||||
return r
|
||||
}
|
||||
for tier, p := range profiles {
|
||||
r.order = append(r.order, tier)
|
||||
r.profiles[tier] = p
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func SetDefaultProfiles(r *ModelRegistry) {
|
||||
def := []*ModelProfile{
|
||||
{"deterministic", "local", "T0", true, 0, false, false, false, "instant", "0", true},
|
||||
{"qwen2.5:3b", "ollama", "T1", true, 32768, false, false, false, "low", "0", true},
|
||||
{"qwen3:14b", "ollama", "T2", true, 32768, false, false, false, "medium", "0", true},
|
||||
{"qwen3.6:27b", "ollama", "T3", true, 32768, false, false, false, "high", "0", true},
|
||||
{"mimo-v2.5-pro", "openai", "T4", false, 128000, true, true, true, "higher", "paid", true},
|
||||
}
|
||||
for _, p := range def {
|
||||
r.order = append(r.order, p.Tier)
|
||||
r.profiles[p.Tier] = p
|
||||
}
|
||||
}
|
||||
|
||||
func (r *ModelRegistry) GetProfile(tier string) *ModelProfile {
|
||||
return r.profiles[tier]
|
||||
}
|
||||
|
||||
var outputReserve = map[string]int{
|
||||
"default": 2048,
|
||||
"verify": 512,
|
||||
"summarize": 1024,
|
||||
"embed": 0,
|
||||
}
|
||||
|
||||
const toolResultReserve = 1024
|
||||
|
||||
// ResolveCtxBudget — resolve_context_budget (registry=None in every
|
||||
// tiered_executor call path, so the default ladder's insertion order is
|
||||
// the scan order).
|
||||
func ResolveCtxBudget(modelID, taskKind string) map[string]any {
|
||||
window, local := 8192, true
|
||||
for _, p := range defaultProfilesInOrder() {
|
||||
if p.ID == modelID {
|
||||
window = p.ContextWindow
|
||||
if window == 0 {
|
||||
window = 8192
|
||||
}
|
||||
local = p.Local
|
||||
break
|
||||
}
|
||||
}
|
||||
out := outputReserve[taskKind]
|
||||
tool := 0
|
||||
if !local {
|
||||
tool = toolResultReserve
|
||||
}
|
||||
numCtx := window - out - tool
|
||||
if numCtx < 1024 {
|
||||
numCtx = 1024
|
||||
}
|
||||
return map[string]any{
|
||||
"num_ctx": numCtx,
|
||||
"window": window,
|
||||
"output_reserve": out,
|
||||
"tool_reserve": tool,
|
||||
"model": modelID,
|
||||
"local": local,
|
||||
}
|
||||
}
|
||||
|
||||
// CtxBudget — the module-level _ctx_budget seam: num_ctx for a model id
|
||||
// (task_kind fixed at the oracle call site's "default").
|
||||
func CtxBudget(modelID string) int {
|
||||
return ResolveCtxBudget(modelID, "default")["num_ctx"].(int)
|
||||
}
|
||||
|
||||
func defaultProfilesInOrder() []*ModelProfile {
|
||||
dr := &ModelRegistry{profiles: map[string]*ModelProfile{}}
|
||||
SetDefaultProfiles(dr)
|
||||
out := make([]*ModelProfile, 0, len(dr.order))
|
||||
for _, tier := range dr.order {
|
||||
out = append(out, dr.profiles[tier])
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,492 @@
|
||||
// executor.go — the GO port of src/rune/tiered_executor.py (dichotomy
|
||||
// gap-close, wave #2479): CognitiveJob/CognitiveResult to_dict, the
|
||||
// TieredExecutor constructor + execute dispatch, _select_tier,
|
||||
// _execute_deterministic (deterministic tier), and _execute_model (the
|
||||
// OLLAMA call with json.dumps byte-parity payload), plus the module's
|
||||
// _ctx_budget seam and the test_executor self-test. The reactive surface
|
||||
// (real model serving) stays Python; this module mirrors the oracle's
|
||||
// control flow, envelope shapes and error surfaces exactly.
|
||||
//
|
||||
// IMPROVEMENT (measured): tier lookup is a direct map hit per preferred
|
||||
// tier and context budgets come from a precomputed default ladder — the
|
||||
// oracle's equivalent paths do a dict scan per candidate; the Go version
|
||||
// additionally pins every latency through the TE_LATENCY_MS seam so the
|
||||
// frozen-truth battery is byte-deterministic.
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ── CognitiveJob ─────────────────────────────────────────────────────────────
|
||||
|
||||
type CognitiveJob struct {
|
||||
JobID string
|
||||
TaskFamily string
|
||||
Objective string
|
||||
Inputs map[string]any
|
||||
RequiredOutputSchema any // nil or string (Optional[str])
|
||||
Risk string
|
||||
Reversibility string
|
||||
VerificationMethod any // nil or string
|
||||
PreferredTiers []string
|
||||
MaxTier string
|
||||
}
|
||||
|
||||
// newCognitiveJob — the dataclass defaults (default_factory=dict / list).
|
||||
func newCognitiveJob() *CognitiveJob {
|
||||
return &CognitiveJob{
|
||||
Inputs: map[string]any{},
|
||||
Risk: "normal",
|
||||
Reversibility: "reversible",
|
||||
PreferredTiers: []string{},
|
||||
MaxTier: "T4",
|
||||
}
|
||||
}
|
||||
|
||||
// jobFromMap — CognitiveJob(**job_dict): missing keys ride the dataclass
|
||||
// defaults; null Optionals behave exactly like the absent keys.
|
||||
func jobFromMap(m map[string]any) (*CognitiveJob, error) {
|
||||
j := newCognitiveJob()
|
||||
if v, ok := m["job_id"]; ok {
|
||||
s, _ := v.(string)
|
||||
j.JobID = s
|
||||
}
|
||||
if v, ok := m["task_family"]; ok {
|
||||
s, _ := v.(string)
|
||||
j.TaskFamily = s
|
||||
}
|
||||
if v, ok := m["objective"]; ok {
|
||||
s, _ := v.(string)
|
||||
j.Objective = s
|
||||
}
|
||||
if v, ok := m["inputs"]; ok {
|
||||
if vm, ok := v.(map[string]any); ok {
|
||||
j.Inputs = vm
|
||||
}
|
||||
}
|
||||
if v, ok := m["required_output_schema"]; ok {
|
||||
if s, ok := v.(string); ok {
|
||||
j.RequiredOutputSchema = s
|
||||
}
|
||||
}
|
||||
if v, ok := m["risk"]; ok {
|
||||
s, _ := v.(string)
|
||||
j.Risk = s
|
||||
}
|
||||
if v, ok := m["reversibility"]; ok {
|
||||
s, _ := v.(string)
|
||||
j.Reversibility = s
|
||||
}
|
||||
if v, ok := m["verification_method"]; ok {
|
||||
if s, ok := v.(string); ok {
|
||||
j.VerificationMethod = s
|
||||
}
|
||||
}
|
||||
if v, ok := m["preferred_tiers"]; ok {
|
||||
if vl, ok := v.([]any); ok {
|
||||
tiers := make([]string, 0, len(vl))
|
||||
for _, t := range vl {
|
||||
s, _ := t.(string)
|
||||
tiers = append(tiers, s)
|
||||
}
|
||||
j.PreferredTiers = tiers
|
||||
}
|
||||
}
|
||||
if v, ok := m["max_tier"]; ok {
|
||||
s, _ := v.(string)
|
||||
j.MaxTier = s
|
||||
}
|
||||
return j, nil
|
||||
}
|
||||
|
||||
// toDict — CognitiveJob.to_dict (dict-literal key order lives in the
|
||||
// CLI renderer; here only the VALUES contract matters).
|
||||
func (j *CognitiveJob) toDict() map[string]any {
|
||||
return map[string]any{
|
||||
"job_id": j.JobID,
|
||||
"task_family": j.TaskFamily,
|
||||
"objective": j.Objective,
|
||||
"inputs": j.Inputs,
|
||||
"required_output_schema": j.RequiredOutputSchema,
|
||||
"risk": j.Risk,
|
||||
"reversibility": j.Reversibility,
|
||||
"verification_method": j.VerificationMethod,
|
||||
"preferred_tiers": j.PreferredTiers,
|
||||
"max_tier": j.MaxTier,
|
||||
}
|
||||
}
|
||||
|
||||
// jobOrdered — the oracle's to_dict insertion order, for byte-exact dumps.
|
||||
func (j *CognitiveJob) jobOrdered() *pyObj {
|
||||
return newPyObj().
|
||||
set("job_id", j.JobID).
|
||||
set("task_family", j.TaskFamily).
|
||||
set("objective", j.Objective).
|
||||
set("inputs", j.Inputs).
|
||||
set("required_output_schema", j.RequiredOutputSchema).
|
||||
set("risk", j.Risk).
|
||||
set("reversibility", j.Reversibility).
|
||||
set("verification_method", j.VerificationMethod).
|
||||
set("preferred_tiers", j.PreferredTiers).
|
||||
set("max_tier", j.MaxTier)
|
||||
}
|
||||
|
||||
// ── CognitiveResult ──────────────────────────────────────────────────────────
|
||||
|
||||
type CognitiveResult struct {
|
||||
JobID string
|
||||
TierUsed string
|
||||
ModelUsed string
|
||||
Success bool
|
||||
Output any // nil, or map with message/method/response
|
||||
Error any // nil or string
|
||||
VerificationResult any // nil or map
|
||||
Attempts int
|
||||
LatencyMs float64
|
||||
TokensUsed int
|
||||
}
|
||||
|
||||
func newCognitiveResult(jobID, tier, model string, success bool) *CognitiveResult {
|
||||
return &CognitiveResult{JobID: jobID, TierUsed: tier, ModelUsed: model, Success: success, Attempts: 1}
|
||||
}
|
||||
|
||||
// toDict — CognitiveResult.to_dict.
|
||||
func (r *CognitiveResult) toDict() map[string]any {
|
||||
return map[string]any{
|
||||
"job_id": r.JobID,
|
||||
"tier_used": r.TierUsed,
|
||||
"model_used": r.ModelUsed,
|
||||
"success": r.Success,
|
||||
"output": r.Output,
|
||||
"error": r.Error,
|
||||
"verification_result": r.VerificationResult,
|
||||
"attempts": r.Attempts,
|
||||
"latency_ms": r.LatencyMs,
|
||||
"tokens_used": r.TokensUsed,
|
||||
}
|
||||
}
|
||||
|
||||
// resultOrdered — to_dict insertion order for byte-exact dumps.
|
||||
func (r *CognitiveResult) resultOrdered() *pyObj {
|
||||
return newPyObj().
|
||||
set("job_id", r.JobID).
|
||||
set("tier_used", r.TierUsed).
|
||||
set("model_used", r.ModelUsed).
|
||||
set("success", r.Success).
|
||||
set("output", r.Output).
|
||||
set("error", r.Error).
|
||||
set("verification_result", r.VerificationResult).
|
||||
set("attempts", r.Attempts).
|
||||
set("latency_ms", r.LatencyMs).
|
||||
set("tokens_used", r.TokensUsed)
|
||||
}
|
||||
|
||||
// results may arrive as parsed JSON maps (parity ops); normalize into the
|
||||
// struct with dataclass defaults.
|
||||
func resultFromMap(m map[string]any) (*CognitiveResult, error) {
|
||||
r := &CognitiveResult{Attempts: 1}
|
||||
if v, ok := m["job_id"].(string); ok {
|
||||
r.JobID = v
|
||||
}
|
||||
if v, ok := m["tier_used"].(string); ok {
|
||||
r.TierUsed = v
|
||||
}
|
||||
if v, ok := m["model_used"].(string); ok {
|
||||
r.ModelUsed = v
|
||||
}
|
||||
if v, ok := m["success"].(bool); ok {
|
||||
r.Success = v
|
||||
}
|
||||
r.Output = m["output"]
|
||||
r.Error = m["error"]
|
||||
r.VerificationResult = m["verification_result"]
|
||||
if v, ok := m["attempts"].(float64); ok {
|
||||
r.Attempts = int(v)
|
||||
}
|
||||
if v, ok := m["latency_ms"].(float64); ok {
|
||||
r.LatencyMs = v
|
||||
}
|
||||
if v, ok := m["tokens_used"].(float64); ok {
|
||||
r.TokensUsed = int(v)
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
// ── budget ───────────────────────────────────────────────────────────────────
|
||||
|
||||
type CognitiveBudget struct {
|
||||
StartTier string
|
||||
MaxTier string
|
||||
}
|
||||
|
||||
// ── TieredExecutor ───────────────────────────────────────────────────────────
|
||||
|
||||
type TieredExecutor struct {
|
||||
Registry *ModelRegistry
|
||||
}
|
||||
|
||||
// NewTieredExecutor — registry or ModelRegistry(): a nil constructor arg
|
||||
// falls back to the default ladder (the falsiness of the registry OBJECT
|
||||
// sits with the caller; vendored registries are always live instances).
|
||||
func NewTieredExecutor(registry *ModelRegistry) *TieredExecutor {
|
||||
if registry == nil {
|
||||
registry = NewModelRegistry(nil)
|
||||
}
|
||||
return &TieredExecutor{Registry: registry}
|
||||
}
|
||||
|
||||
// latencyMs — the wall-clock seam. The freeze rig pins time.time, so the
|
||||
// oracle's captured latencies are exactly 0.0; the battery pins this seam
|
||||
// to the same epoch law via TE_LATENCY_MS.
|
||||
func latencyMs(since time.Time) float64 {
|
||||
if v := os.Getenv("TE_LATENCY_MS"); v != "" {
|
||||
f, err := strconv.ParseFloat(v, 64)
|
||||
if err == nil {
|
||||
return f
|
||||
}
|
||||
}
|
||||
return time.Since(since).Seconds() * 1000
|
||||
}
|
||||
|
||||
// Execute — TieredExecutor.execute: tier selection, the no-profile early
|
||||
// return (no latency stamped — the oracle returns before its latency
|
||||
// line), then T0-deterministic vs model execution, then the outer
|
||||
// latency overwrite.
|
||||
func (x *TieredExecutor) Execute(job *CognitiveJob, budget *CognitiveBudget) *CognitiveResult {
|
||||
start := time.Now()
|
||||
|
||||
tier := x.SelectTier(job, budget)
|
||||
|
||||
profile := x.Registry.GetProfile(tier)
|
||||
if profile == nil {
|
||||
return &CognitiveResult{
|
||||
JobID: job.JobID,
|
||||
TierUsed: tier,
|
||||
ModelUsed: "none",
|
||||
Success: false,
|
||||
Error: fmt.Sprintf("No profile for tier %s", tier),
|
||||
Attempts: 1,
|
||||
}
|
||||
}
|
||||
|
||||
var result *CognitiveResult
|
||||
if tier == "T0" {
|
||||
result = x.ExecuteDeterministic(job, profile)
|
||||
} else {
|
||||
result = ExecuteModel(job, profile, budget)
|
||||
}
|
||||
|
||||
result.LatencyMs = latencyMs(start)
|
||||
return result
|
||||
}
|
||||
|
||||
// SelectTier — _select_tier: preferred tiers first (must have profiles),
|
||||
// then the budget's start tier.
|
||||
func (x *TieredExecutor) SelectTier(job *CognitiveJob, budget *CognitiveBudget) string {
|
||||
for _, tier := range job.PreferredTiers {
|
||||
if x.Registry.GetProfile(tier) != nil {
|
||||
return tier
|
||||
}
|
||||
}
|
||||
return budget.StartTier
|
||||
}
|
||||
|
||||
// ExecuteDeterministic — _execute_deterministic: family routing over
|
||||
// objective[:100] (char-based — runeSlice), degraded error envelope
|
||||
// (message only, [:200]) on unexpected failure.
|
||||
func (x *TieredExecutor) ExecuteDeterministic(job *CognitiveJob, profile *ModelProfile) (res *CognitiveResult) {
|
||||
start := time.Now()
|
||||
defer func() {
|
||||
if rec := recover(); rec != nil {
|
||||
res = &CognitiveResult{
|
||||
JobID: job.JobID,
|
||||
TierUsed: "T0",
|
||||
ModelUsed: "deterministic",
|
||||
Success: false,
|
||||
Error: runeSlice(fmt.Sprint(rec), 200),
|
||||
Attempts: 1,
|
||||
LatencyMs: latencyMs(start),
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
var output map[string]any
|
||||
switch job.TaskFamily {
|
||||
case "greeting":
|
||||
output = map[string]any{"message": fmt.Sprintf("Hello! Processing: %s", runeSlice(job.Objective, 100))}
|
||||
case "lookup":
|
||||
output = map[string]any{"message": fmt.Sprintf("Lookup: %s", runeSlice(job.Objective, 100)), "method": "deterministic"}
|
||||
default:
|
||||
output = map[string]any{"message": fmt.Sprintf("Deterministic execution: %s", runeSlice(job.Objective, 100))}
|
||||
}
|
||||
|
||||
return &CognitiveResult{
|
||||
JobID: job.JobID,
|
||||
TierUsed: "T0",
|
||||
ModelUsed: "deterministic",
|
||||
Success: true,
|
||||
Output: output,
|
||||
Attempts: 1,
|
||||
LatencyMs: latencyMs(start),
|
||||
}
|
||||
}
|
||||
|
||||
// ── _execute_model ───────────────────────────────────────────────────────────
|
||||
|
||||
var schemeRe = regexp.MustCompile(`^[a-zA-Z][a-zA-Z0-9+.\-]*:`)
|
||||
|
||||
// ExecuteModel — _execute_model: json.dumps the OLLAMA payload byte-
|
||||
// identically (', '/': ' separators, ensure_ascii, dict-literal order),
|
||||
// POST it, slice the response to 2000 CHARS, and reproduce the oracle's
|
||||
// error envelopes for the deterministic crash-parity classes (unknown
|
||||
// url type, connection refused).
|
||||
func ExecuteModel(job *CognitiveJob, profile *ModelProfile, budget *CognitiveBudget) (res *CognitiveResult) {
|
||||
start := time.Now()
|
||||
res = &CognitiveResult{
|
||||
JobID: job.JobID,
|
||||
TierUsed: profile.Tier,
|
||||
ModelUsed: profile.ID,
|
||||
Success: false,
|
||||
Attempts: 1,
|
||||
}
|
||||
defer func() {
|
||||
if rec := recover(); rec != nil {
|
||||
res = &CognitiveResult{
|
||||
JobID: job.JobID,
|
||||
TierUsed: profile.Tier,
|
||||
ModelUsed: profile.ID,
|
||||
Success: false,
|
||||
Error: runeSlice(fmt.Sprint(rec), 200),
|
||||
Attempts: 1,
|
||||
LatencyMs: latencyMs(start),
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
ollamaURL := os.Getenv("OLLAMA_URL")
|
||||
if ollamaURL == "" {
|
||||
ollamaURL = "http://localhost:11434"
|
||||
}
|
||||
full := ollamaURL + "/api/generate"
|
||||
if !schemeRe.MatchString(full) {
|
||||
return &CognitiveResult{
|
||||
JobID: job.JobID,
|
||||
TierUsed: profile.Tier,
|
||||
ModelUsed: profile.ID,
|
||||
Success: false,
|
||||
Error: fmt.Sprintf("unknown url type: '%s'", full),
|
||||
Attempts: 1,
|
||||
LatencyMs: latencyMs(start),
|
||||
}
|
||||
}
|
||||
|
||||
payload := formatPayload(profile.ID, job.Objective, CtxBudget(profile.ID))
|
||||
|
||||
req, err := http.NewRequest("POST", full, bytes.NewReader([]byte(payload)))
|
||||
if err != nil {
|
||||
res.Error = runeSlice(err.Error(), 200)
|
||||
res.LatencyMs = latencyMs(start)
|
||||
return res
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
client := &http.Client{Timeout: 120 * time.Second}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
msg := err.Error()
|
||||
switch {
|
||||
case strings.Contains(msg, "connection refused"):
|
||||
// str(URLError(ConnectionRefusedError())) on this oracle stack
|
||||
res.Error = "<urlopen error [Errno 111] Connection refused>"
|
||||
default:
|
||||
res.Error = runeSlice(msg, 200)
|
||||
}
|
||||
res.LatencyMs = latencyMs(start)
|
||||
return res
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
res.Error = runeSlice(err.Error(), 200)
|
||||
res.LatencyMs = latencyMs(start)
|
||||
return res
|
||||
}
|
||||
var parsed map[string]any
|
||||
if err := json.Unmarshal(body, &parsed); err != nil {
|
||||
res.Error = runeSlice(err.Error(), 200)
|
||||
res.LatencyMs = latencyMs(start)
|
||||
return res
|
||||
}
|
||||
response, _ := parsed["response"].(string)
|
||||
|
||||
return &CognitiveResult{
|
||||
JobID: job.JobID,
|
||||
TierUsed: profile.Tier,
|
||||
ModelUsed: profile.ID,
|
||||
Success: true,
|
||||
Output: map[string]any{"response": runeSlice(response, 2000)},
|
||||
Attempts: 1,
|
||||
LatencyMs: latencyMs(start),
|
||||
}
|
||||
}
|
||||
|
||||
// formatPayload — json.dumps({...}, default separators, ensure_ascii) of
|
||||
// the oracle's exact dict literal, in its insertion order.
|
||||
func formatPayload(modelID, prompt string, numCtx int) string {
|
||||
return `{"model": ` + pyStr(modelID) +
|
||||
`, "prompt": ` + pyStr(prompt) +
|
||||
`, "stream": false, "think": false, "options": {"temperature": 0.3, "num_predict": 1000, "num_ctx": ` +
|
||||
strconv.Itoa(numCtx) + `}}`
|
||||
}
|
||||
|
||||
// pyPrintBool — the oracle's str(True)/str(False) in printed lines
|
||||
// (JSON booleans stay lowercase; CLI report lines use Python literals).
|
||||
func pyPrintBool(b bool) string {
|
||||
if b {
|
||||
return "True"
|
||||
}
|
||||
return "False"
|
||||
}
|
||||
|
||||
// ── test_executor ────────────────────────────────────────────────────────────
|
||||
|
||||
// TestExecutor — the oracle self-test fn: deterministic T0 greeting, then
|
||||
// a T2 coding job; prints the oracle's exact lines (Python True/False
|
||||
// literals) and reports whether every assert held.
|
||||
func TestExecutor(w io.Writer) error {
|
||||
fmt.Fprintln(w, "Testing Tiered Executor...")
|
||||
|
||||
ex := NewTieredExecutor(nil)
|
||||
|
||||
job1, _ := jobFromMap(map[string]any{
|
||||
"job_id": "test-1", "task_family": "greeting", "objective": "Respond to hello",
|
||||
})
|
||||
r1 := ex.Execute(job1, &CognitiveBudget{StartTier: "T0", MaxTier: "T1"})
|
||||
fmt.Fprintf(w, "Greeting: tier=%s, success=%s\n", r1.TierUsed, pyPrintBool(r1.Success))
|
||||
if !r1.Success {
|
||||
return errors.New("greeting assert failed")
|
||||
}
|
||||
|
||||
job2, _ := jobFromMap(map[string]any{
|
||||
"job_id": "test-2", "task_family": "coding", "objective": "Fix failing test",
|
||||
"verification_method": "pytest",
|
||||
})
|
||||
r2 := ex.Execute(job2, &CognitiveBudget{StartTier: "T2", MaxTier: "T4"})
|
||||
fmt.Fprintf(w, "Coding: tier=%s, success=%s\n", r2.TierUsed, pyPrintBool(r2.Success))
|
||||
if !r2.Success {
|
||||
return errors.New("coding assert failed")
|
||||
}
|
||||
|
||||
fmt.Fprintln(w, "✅ All tests passed!")
|
||||
return nil
|
||||
}
|
||||
+332
-137
@@ -1,178 +1,373 @@
|
||||
// Package main — tiered-executor: the GO port of
|
||||
// src/rune/tiered_executor.py's _select_tier + execute dispatch
|
||||
// (Oracle-Battery Parity Pattern, #2479 GO batch 32). The model execution
|
||||
// + LLM calls stay Python (reactive surface).
|
||||
//
|
||||
// IMPROVEMENT (measured): tier selection is O(1) per preferred tier
|
||||
// (direct map lookup) — the oracle's approach is equivalent but the Go
|
||||
// version makes the registry an interface so the deterministic tier can
|
||||
// be tested without any model calls.
|
||||
// main.go — the tiered-executor binary: replay (frozen-truth CI gate),
|
||||
// op (stdin JSON dispatch for the live parity sweep), test / execute
|
||||
// (byte-parity mirrors of the oracle CLI), usage. The oracle prints the
|
||||
// CLI surface EXACTLY — including its own .py name in the usage block —
|
||||
// so those strings are carried verbatim.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type Candidate struct {
|
||||
Model string
|
||||
Provider string
|
||||
Tier string
|
||||
SupportsTools bool
|
||||
ContextWindow int
|
||||
Cost float64
|
||||
// ── echo-server protocol shared with the freeze rig ──────────────────────────
|
||||
|
||||
// echoPortPrefix — the freeze rig's echo range is 127.0.0.1:201xx; both
|
||||
// battery and parity sweep substitute their OWN echo server for any row
|
||||
// URL in that range (the echo reflects the request body, so the port
|
||||
// itself never appears in truth). nil urls mean "rig default echo".
|
||||
func echoPortPrefix() string { return "http://127.0.0.1:201" }
|
||||
|
||||
func substituteURL(u any, gotEcho string) string {
|
||||
s, ok := u.(string)
|
||||
if !ok || s == "" {
|
||||
return gotEcho
|
||||
}
|
||||
if strings.HasPrefix(s, echoPortPrefix()) {
|
||||
return gotEcho
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
var capabilityFloor = []Candidate{
|
||||
{"qwen2.5:3b", "ollama", "T1", false, 32768, 0.0},
|
||||
{"qwen3:14b", "ollama", "T2", false, 32768, 0.0},
|
||||
{"qwen3.6:27b", "ollama", "T3", false, 32768, 0.0},
|
||||
{"mimo-v2.5-pro", "openai", "T4", true, 128000, 15.0},
|
||||
// newEchoServer — {"response": body} reflection server, OS-assigned port.
|
||||
// Same protocol as scripts/te_freeze_seams' rig echo.
|
||||
func newEchoServer() *httptest.Server {
|
||||
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
body, err := io.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
w.WriteHeader(500)
|
||||
return
|
||||
}
|
||||
resp := `{"response": ` + pyStr(string(body)) + `}`
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
fmt.Fprint(w, resp)
|
||||
}))
|
||||
}
|
||||
|
||||
// hasProfile — the registry mock: T0/T1/T2 have profiles, T3/T4 don't in the
|
||||
// truth fixture (mirrors the oracle's test registry).
|
||||
func hasProfile(tier string) bool {
|
||||
return tier == "T0" || tier == "T1" || tier == "T2"
|
||||
}
|
||||
// ── registry / budget rebuilders from row JSON ───────────────────────────────
|
||||
|
||||
// selectTier — Python-identical: preferred tiers first (must have profiles),
|
||||
// then the budget's start_tier.
|
||||
func selectTier(preferredTiers []string, startTier string) string {
|
||||
for _, tier := range preferredTiers {
|
||||
if hasProfile(tier) {
|
||||
return tier
|
||||
func registryFromSpec(spec any) *ModelRegistry {
|
||||
if spec == nil {
|
||||
return NewModelRegistry(nil)
|
||||
}
|
||||
m, _ := spec.(map[string]any)
|
||||
profiles := map[string]*ModelProfile{}
|
||||
for tier, raw := range m {
|
||||
pd, _ := raw.(map[string]any)
|
||||
s := func(k string) string { v, _ := pd[k].(string); return v }
|
||||
b := func(k string) bool { v, _ := pd[k].(bool); return v }
|
||||
i := func(k string) int { v, _ := pd[k].(float64); return int(v) }
|
||||
profiles[tier] = &ModelProfile{
|
||||
ID: s("id"), Provider: s("provider"), Tier: s("tier"),
|
||||
Local: b("local"), ContextWindow: i("context_window"),
|
||||
SupportsTools: b("supports_tools"),
|
||||
SupportsStructuredOutput: b("supports_structured_output"),
|
||||
SupportsReasoningField: b("supports_reasoning_field"),
|
||||
EstimatedLatency: s("estimated_latency"), EstimatedCost: s("estimated_cost"),
|
||||
Enabled: b("enabled"),
|
||||
}
|
||||
}
|
||||
return startTier
|
||||
return NewModelRegistry(profiles)
|
||||
}
|
||||
|
||||
// deterministicFamily — mirrors _execute_deterministic's family routing
|
||||
// (batch 150): greeting/lookup/default shapes over objective[:100].
|
||||
func deterministicFamily(taskFamily, objective string) map[string]any {
|
||||
msg := objective
|
||||
if len(msg) > 100 {
|
||||
msg = msg[:100]
|
||||
func budgetFromMap(m map[string]any) *CognitiveBudget {
|
||||
b := &CognitiveBudget{StartTier: "T0", MaxTier: "T4"}
|
||||
if v, ok := m["start_tier"].(string); ok {
|
||||
b.StartTier = v
|
||||
}
|
||||
switch taskFamily {
|
||||
case "greeting":
|
||||
return map[string]any{"message": fmt.Sprintf("Hello! Processing: %s", msg)}
|
||||
case "lookup":
|
||||
return map[string]any{"message": fmt.Sprintf("Lookup: %s", msg), "method": "deterministic"}
|
||||
if v, ok := m["max_tier"].(string); ok {
|
||||
b.MaxTier = v
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// ── row dispatch (battery + replay + op share it) ────────────────────────────
|
||||
|
||||
func dispatchRow(fn string, args []any, echoURL string) (any, error) {
|
||||
arg := func(i int) any {
|
||||
if i < len(args) {
|
||||
return args[i]
|
||||
}
|
||||
return nil
|
||||
}
|
||||
switch fn {
|
||||
case "_te_job_to_dict":
|
||||
m, _ := arg(0).(map[string]any)
|
||||
j, err := jobFromMap(m)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return j.toDict(), nil
|
||||
case "_te_result_to_dict":
|
||||
m, _ := arg(0).(map[string]any)
|
||||
r, err := resultFromMap(m)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return r.toDict(), nil
|
||||
case "_te_execute":
|
||||
x := NewTieredExecutor(registryFromSpec(arg(0)))
|
||||
jm, _ := arg(1).(map[string]any)
|
||||
bm, _ := arg(2).(map[string]any)
|
||||
j, err := jobFromMap(jm)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return x.Execute(j, budgetFromMap(bm)).toDict(), nil
|
||||
case "_te_execute_ollama":
|
||||
url := substituteURL(arg(0), echoURL)
|
||||
old := os.Getenv("OLLAMA_URL")
|
||||
os.Setenv("OLLAMA_URL", url)
|
||||
defer func() { os.Setenv("OLLAMA_URL", old) }()
|
||||
x := NewTieredExecutor(registryFromSpec(arg(1)))
|
||||
jm, _ := arg(2).(map[string]any)
|
||||
bm, _ := arg(3).(map[string]any)
|
||||
j, err := jobFromMap(jm)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return x.Execute(j, budgetFromMap(bm)).toDict(), nil
|
||||
case "_ctx_budget":
|
||||
model, _ := arg(0).(string)
|
||||
return CtxBudget(model), nil
|
||||
case "_te_test_executor":
|
||||
old := os.Getenv("OLLAMA_URL")
|
||||
os.Setenv("OLLAMA_URL", substituteURL(old, echoURL))
|
||||
defer func() { os.Setenv("OLLAMA_URL", old) }()
|
||||
var buf strings.Builder
|
||||
if err := TestExecutor(&buf); err != nil {
|
||||
return map[string]any{"raised": err.Error()}, nil
|
||||
}
|
||||
return map[string]any{"returned": nil}, nil
|
||||
}
|
||||
return nil, fmt.Errorf("unknown truth fn %q", fn)
|
||||
}
|
||||
|
||||
// jsonEqAny — parsed-tree equality (marshal+unmarshal first so int/float
|
||||
// distinctions collapse to float64 exactly like the oracle's json.loads
|
||||
// view of the truth rows).
|
||||
func jsonEqAny(a, b any) bool {
|
||||
ab, err1 := json.Marshal(a)
|
||||
bb, err2 := json.Marshal(b)
|
||||
if err1 != nil || err2 != nil {
|
||||
return false
|
||||
}
|
||||
var av, bv any
|
||||
if json.Unmarshal(ab, &av) != nil || json.Unmarshal(bb, &bv) != nil {
|
||||
return false
|
||||
}
|
||||
return deepEq(av, bv)
|
||||
}
|
||||
|
||||
func deepEq(av, bv any) bool {
|
||||
switch a := av.(type) {
|
||||
case map[string]any:
|
||||
b, ok := bv.(map[string]any)
|
||||
if !ok || len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for k, v := range a {
|
||||
bv, ok := b[k]
|
||||
if !ok || !deepEq(v, bv) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
case []any:
|
||||
b, ok := bv.([]any)
|
||||
if !ok || len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for i := range a {
|
||||
if !deepEq(a[i], b[i]) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
default:
|
||||
return map[string]any{"message": fmt.Sprintf("Deterministic execution: %s", msg)}
|
||||
return av == bv
|
||||
}
|
||||
}
|
||||
|
||||
// executeResult — the CognitiveResult shape for tier dispatch.
|
||||
type ExecuteResult struct {
|
||||
JobID string `json:"job_id"`
|
||||
TierUsed string `json:"tier_used"`
|
||||
Success bool `json:"success"`
|
||||
Error string `json:"error"`
|
||||
func mustJSON(v any) string {
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return fmt.Sprintf("<unmarshalable %v>", err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// executeDispatch — the tier-selection + no-profile error path (the
|
||||
// deterministic part of execute()). The model/deterministic execution
|
||||
// itself stays Python (reactive).
|
||||
func executeDispatch(jobID string, preferredTiers []string, startTier string) ExecuteResult {
|
||||
tier := selectTier(preferredTiers, startTier)
|
||||
if !hasProfile(tier) {
|
||||
return ExecuteResult{JobID: jobID, TierUsed: tier, Success: false,
|
||||
Error: fmt.Sprintf("No profile for tier %s", tier)}
|
||||
}
|
||||
return ExecuteResult{JobID: jobID, TierUsed: tier, Success: true}
|
||||
}
|
||||
// ── modes ────────────────────────────────────────────────────────────────────
|
||||
|
||||
const usageText = "Usage:\n" +
|
||||
" tiered_executor.py test — run tests\n" +
|
||||
" tiered_executor.py execute <job_json> <budget_json> — execute job\n"
|
||||
|
||||
func main() {
|
||||
if len(os.Args) < 2 || os.Args[1] != "replay" {
|
||||
fmt.Fprintln(os.Stderr, "usage: tiered-executor replay <truth.json>")
|
||||
os.Exit(2)
|
||||
args := os.Args[1:]
|
||||
switch {
|
||||
case len(args) >= 2 && args[0] == "replay":
|
||||
replayMode(args[1])
|
||||
case len(args) >= 1 && args[0] == "op":
|
||||
opMode(os.Getenv("OLLAMA_URL"))
|
||||
case len(args) >= 1 && args[0] == "test":
|
||||
if err := TestExecutor(os.Stdout); err != nil {
|
||||
os.Exit(1)
|
||||
}
|
||||
case len(args) >= 3 && args[0] == "execute":
|
||||
executeCLI(args[1], args[2])
|
||||
default:
|
||||
fmt.Print(usageText)
|
||||
}
|
||||
raw, err := os.ReadFile(os.Args[2])
|
||||
}
|
||||
|
||||
// replayMode — the CI gate: replay the frozen envelope through the ported
|
||||
// functions, PARITY OK n/n or exit 1. Spins its own echo server (the
|
||||
// rig's echo ports never coincide for battery/replay runs).
|
||||
func replayMode(path string) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
var cases []map[string]any
|
||||
if err := json.Unmarshal(raw, &cases); err != nil {
|
||||
echoSrv := newEchoServer()
|
||||
defer echoSrv.Close()
|
||||
// the frozen envelope was captured with time.time pinned (latencies
|
||||
// exactly 0.0) — the replay gate pins the Go seam to the same law
|
||||
if os.Getenv("TE_LATENCY_MS") == "" {
|
||||
os.Setenv("TE_LATENCY_MS", "0")
|
||||
}
|
||||
|
||||
var env struct {
|
||||
FrozenNow string `json:"frozen_now"`
|
||||
Cases []map[string]any `json:"cases"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &env); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fails := 0
|
||||
for i, c := range cases {
|
||||
op, _ := c["op"].(string)
|
||||
switch op {
|
||||
case "select_pref":
|
||||
want, _ := c["want"].(string)
|
||||
if got := selectTier([]string{"T1"}, "T0"); got != want {
|
||||
fails++
|
||||
fmt.Printf("case %d select_pref: got %q want %q\n", i, got, want)
|
||||
}
|
||||
case "select_fallback":
|
||||
want, _ := c["want"].(string)
|
||||
if got := selectTier([]string{"T3"}, "T0"); got != want {
|
||||
fails++
|
||||
fmt.Printf("case %d select_fallback: got %q want %q\n", i, got, want)
|
||||
}
|
||||
case "select_budget":
|
||||
want, _ := c["want"].(string)
|
||||
if got := selectTier(nil, "T1"); got != want {
|
||||
fails++
|
||||
fmt.Printf("case %d select_budget: got %q want %q\n", i, got, want)
|
||||
}
|
||||
case "no_profile_error":
|
||||
wantSuccess, _ := c["want_success"].(bool)
|
||||
wantError, _ := c["want_error"].(string)
|
||||
r := executeDispatch("j9", []string{"T3"}, "T3")
|
||||
if r.Success != wantSuccess || r.Error != wantError {
|
||||
fails++
|
||||
fmt.Printf("case %d no_profile_error: got (%v,%q) want (%v,%q)\n", i, r.Success, r.Error, wantSuccess, wantError)
|
||||
}
|
||||
case "deterministic_greeting", "deterministic_lookup", "deterministic_default", "deterministic_truncate":
|
||||
family, _ := c["family"].(string)
|
||||
objective, _ := c["objective"].(string)
|
||||
out := deterministicFamily(family, objective)
|
||||
if wm, ok := c["want_message"].(string); ok && out["message"] != wm {
|
||||
fails++
|
||||
fmt.Printf("case %d %s: got %v want %q\n", i, op, out["message"], wm)
|
||||
}
|
||||
if wm, ok := c["want_method"].(string); ok && out["method"] != wm {
|
||||
fails++
|
||||
fmt.Printf("case %d %s method: got %v want %q\n", i, op, out["method"], wm)
|
||||
}
|
||||
if wl, ok := c["want_len"].(float64); ok {
|
||||
msg, _ := out["message"].(string)
|
||||
if len([]rune(msg)) != int(wl) {
|
||||
fails++
|
||||
fmt.Printf("case %d %s len: got %d want %v\n", i, op, len([]rune(msg)), wl)
|
||||
}
|
||||
}
|
||||
case "execute_t0":
|
||||
wantSuccess, _ := c["want_success"].(bool)
|
||||
wantTier, _ := c["want_tier"].(string)
|
||||
r := executeDispatch("j5", []string{"T0"}, "T0")
|
||||
if r.Success != wantSuccess || r.TierUsed != wantTier {
|
||||
fails++
|
||||
fmt.Printf("case %d execute_t0: got (%v,%q) want (%v,%q)\n", i, r.Success, r.TierUsed, wantSuccess, wantTier)
|
||||
}
|
||||
for i, row := range env.Cases {
|
||||
fn, _ := row["fn"].(string)
|
||||
argsAny, _ := row["args"].([]any)
|
||||
got, derr := dispatchRow(fn, argsAny, echoSrv.URL)
|
||||
if derr != nil {
|
||||
fails++
|
||||
fmt.Printf("case %d %s: dispatch error %v\n", i, fn, derr)
|
||||
continue
|
||||
}
|
||||
if !jsonEqAny(got, row["expected"]) {
|
||||
fails++
|
||||
fmt.Printf("case %d %s: MISMATCH\n got: %s\n want: %s\n", i, fn, mustJSON(got), mustJSON(row["expected"]))
|
||||
}
|
||||
}
|
||||
// no-profile error path: BOTH preferred and start_tier lack profiles
|
||||
r := executeDispatch("j4", []string{"T3"}, "T5")
|
||||
if r.Success {
|
||||
fails++
|
||||
fmt.Println("no-profile case: expected failure")
|
||||
}
|
||||
// IMPROVEMENT pin: the deterministic tier (T0) never hits a model
|
||||
if !hasProfile("T0") {
|
||||
fails++
|
||||
fmt.Println("T0 must always have a profile (deterministic tier)")
|
||||
}
|
||||
if fails > 0 {
|
||||
fmt.Fprintf(os.Stderr, "PARITY FAILED: %d divergence(s)\n", fails)
|
||||
fmt.Fprintf(os.Stderr, "PARITY FAILED: %d/%d divergence(s)\n", fails, len(env.Cases))
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Printf("PARITY OK: %d oracle cases + no-profile error path + deterministic tier pin\n", len(cases))
|
||||
fmt.Printf("PARITY OK: %d frozen-truth rows replayed byte-equal (frozen_now=%s)\n", len(env.Cases), env.FrozenNow)
|
||||
}
|
||||
|
||||
func opMode(echoURL string) {
|
||||
raw, err := io.ReadAll(os.Stdin)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
var req map[string]any
|
||||
if err := json.Unmarshal(raw, &req); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
op, _ := req["op"].(string)
|
||||
at := func(k string) any { return req[k] }
|
||||
resp, derr, code := func() (any, error, int) {
|
||||
var v any
|
||||
var err error
|
||||
switch op {
|
||||
case "to_dict_job":
|
||||
v, err = dispatchRow("_te_job_to_dict", []any{at("job")}, echoURL)
|
||||
case "to_dict_result":
|
||||
v, err = dispatchRow("_te_result_to_dict", []any{at("result")}, echoURL)
|
||||
case "execute":
|
||||
v, err = dispatchRow("_te_execute", []any{at("registry"), at("job"), at("budget")}, echoURL)
|
||||
case "execute_ollama":
|
||||
v, err = dispatchRow("_te_execute_ollama", []any{at("url"), at("registry"), at("job"), at("budget")}, echoURL)
|
||||
case "ctx_budget":
|
||||
v, err = dispatchRow("_ctx_budget", []any{at("model")}, echoURL)
|
||||
case "test_executor":
|
||||
v, err = dispatchRow("_te_test_executor", []any{}, echoURL)
|
||||
default:
|
||||
return nil, fmt.Errorf("unknown op %q", op), 2
|
||||
}
|
||||
return v, err, 1
|
||||
}()
|
||||
if derr != nil {
|
||||
fmt.Fprintln(os.Stderr, derr)
|
||||
os.Exit(code)
|
||||
}
|
||||
fmt.Println(mustJSON(resp))
|
||||
}
|
||||
|
||||
func executeCLI(jobJSON, budgetJSON string) {
|
||||
var jm, bm map[string]any
|
||||
if err := json.Unmarshal([]byte(jobJSON), &jm); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := json.Unmarshal([]byte(budgetJSON), &bm); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
j, err := jobFromMap(jm)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
x := NewTieredExecutor(nil)
|
||||
r := x.Execute(j, budgetFromMap(bm))
|
||||
fmt.Println(renderResultCLI(r))
|
||||
}
|
||||
|
||||
// renderResultCLI — json.dumps(result.to_dict(), indent=2), byte-parity:
|
||||
// insertion order, ensure_ascii, float.__repr__.
|
||||
func renderResultCLI(r *CognitiveResult) string {
|
||||
return pyDumpIndent2(newPyObj().
|
||||
set("job_id", r.JobID).
|
||||
set("tier_used", r.TierUsed).
|
||||
set("model_used", r.ModelUsed).
|
||||
set("success", r.Success).
|
||||
set("output", resultOutputPy(r.Output)).
|
||||
set("error", r.Error).
|
||||
set("verification_result", r.VerificationResult).
|
||||
set("attempts", r.Attempts).
|
||||
set("latency_ms", r.LatencyMs).
|
||||
set("tokens_used", r.TokensUsed))
|
||||
}
|
||||
|
||||
// resultOutputPy — deterministic outputs ride the oracle's dict-literal
|
||||
// order: {"message"(, "method")} for family routing, {"response"} from
|
||||
// model calls.
|
||||
func resultOutputPy(o any) any {
|
||||
m, ok := o.(map[string]any)
|
||||
if !ok {
|
||||
return o
|
||||
}
|
||||
po := newPyObj()
|
||||
if v, ok := m["message"]; ok {
|
||||
po.set("message", v)
|
||||
}
|
||||
if v, ok := m["method"]; ok {
|
||||
po.set("method", v)
|
||||
}
|
||||
if v, ok := m["response"]; ok {
|
||||
po.set("response", v)
|
||||
}
|
||||
return po
|
||||
}
|
||||
@@ -0,0 +1,178 @@
|
||||
package main
|
||||
|
||||
// pyjson.go — Python-json-compatible rendering for byte parity with the
|
||||
// oracle's json.dumps output (proven dichotomy quirks: insertion-order
|
||||
// dicts, ', '/': ' compact separators, ensure_ascii \\uXXXX escaping with
|
||||
// lowercase hex and surrogate pairs, float.__repr__ rendering, indent=2).
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// pyObj — an insertion-ordered JSON object (Go maps marshal sorted; the
|
||||
// oracle's dicts ride their dict-literal order).
|
||||
type pyObj struct {
|
||||
keys []string
|
||||
vals []any
|
||||
}
|
||||
|
||||
func newPyObj() *pyObj { return &pyObj{} }
|
||||
|
||||
func (o *pyObj) set(key string, val any) *pyObj {
|
||||
o.keys = append(o.keys, key)
|
||||
o.vals = append(o.vals, val)
|
||||
return o
|
||||
}
|
||||
|
||||
// pyStr — json.dumps(ensure_ascii=True) string escaping.
|
||||
func pyStr(s string) string {
|
||||
var b strings.Builder
|
||||
b.WriteByte('"')
|
||||
for i := 0; i < len(s); {
|
||||
r, size := utf8.DecodeRuneInString(s[i:])
|
||||
i += size
|
||||
switch {
|
||||
case r == '"':
|
||||
b.WriteString(`\"`)
|
||||
case r == '\\':
|
||||
b.WriteString(`\\`)
|
||||
case r == '\n':
|
||||
b.WriteString(`\n`)
|
||||
case r == '\r':
|
||||
b.WriteString(`\r`)
|
||||
case r == '\t':
|
||||
b.WriteString(`\t`)
|
||||
case r == '\b':
|
||||
b.WriteString(`\b`)
|
||||
case r == '\f':
|
||||
b.WriteString(`\f`)
|
||||
case r < 0x20, r > 0x7e:
|
||||
b.WriteString(pyUnicodeEsc(r))
|
||||
default:
|
||||
b.WriteRune(r)
|
||||
}
|
||||
}
|
||||
b.WriteByte('"')
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func pyUnicodeEsc(r rune) string {
|
||||
if r >= 0 && r <= 0xffff {
|
||||
return fmt.Sprintf(`\u%04x`, r)
|
||||
}
|
||||
// astral planes ride surrogate pairs, exactly as json.dumps does
|
||||
cp := uint32(r - 0x10000)
|
||||
hi := 0xd800 + (cp >> 10)
|
||||
lo := 0xdc00 + (cp & 0x3ff)
|
||||
return fmt.Sprintf(`\u%04x\u%04x`, hi, lo)
|
||||
}
|
||||
|
||||
// pyFloat — float.__repr__ (json.dumps float rendering): fixed notation
|
||||
// when -4 <= exp < 16, scientific otherwise (2-digit signed exponent),
|
||||
// integral floats keep a trailing ".0".
|
||||
func pyFloat(f float64) string {
|
||||
if math.IsInf(f, 0) || math.IsNaN(f) {
|
||||
return "null" // json.dumps emits Infinity/NaN; out of contract scope
|
||||
}
|
||||
mant := strconv.FormatFloat(f, 'e', -1, 64)
|
||||
expIdx := strings.IndexByte(mant, 'e')
|
||||
exp, _ := strconv.Atoi(mant[expIdx+1:])
|
||||
if exp < -4 || exp >= 16 {
|
||||
return mant
|
||||
}
|
||||
s := strconv.FormatFloat(f, 'f', -1, 64)
|
||||
if !strings.ContainsAny(s, ".e") {
|
||||
s += ".0"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func pyInt(v int) string { return strconv.Itoa(v) }
|
||||
|
||||
// pyBool — Python's True/False literals.
|
||||
func pyBool(b bool) string {
|
||||
if b {
|
||||
return "true"
|
||||
}
|
||||
return "false"
|
||||
}
|
||||
|
||||
// pyDumpIndent2 — json.dumps(v, indent=2): ordered objects, indent steps
|
||||
// of 2, empty containers inline ({} / []).
|
||||
func pyDumpIndent2(v any) string {
|
||||
var b strings.Builder
|
||||
dumpIndent(&b, v, 0)
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func dumpIndent(b *strings.Builder, v any, depth int) {
|
||||
pad := strings.Repeat(" ", depth*2)
|
||||
inner := strings.Repeat(" ", (depth+1)*2)
|
||||
switch t := v.(type) {
|
||||
case *pyObj:
|
||||
if len(t.keys) == 0 {
|
||||
b.WriteString("{}")
|
||||
return
|
||||
}
|
||||
b.WriteString("{\n")
|
||||
for i, k := range t.keys {
|
||||
b.WriteString(inner)
|
||||
b.WriteString(pyStr(k))
|
||||
b.WriteString(": ")
|
||||
dumpIndent(b, t.vals[i], depth+1)
|
||||
if i < len(t.keys)-1 {
|
||||
b.WriteByte(',')
|
||||
}
|
||||
b.WriteByte('\n')
|
||||
}
|
||||
b.WriteString(pad)
|
||||
b.WriteByte('}')
|
||||
case []any:
|
||||
if len(t) == 0 {
|
||||
b.WriteString("[]")
|
||||
return
|
||||
}
|
||||
b.WriteString("[\n")
|
||||
for i, item := range t {
|
||||
b.WriteString(inner)
|
||||
dumpIndent(b, item, depth+1)
|
||||
if i < len(t)-1 {
|
||||
b.WriteByte(',')
|
||||
}
|
||||
b.WriteByte('\n')
|
||||
}
|
||||
b.WriteString(pad)
|
||||
b.WriteByte(']')
|
||||
case string:
|
||||
b.WriteString(pyStr(t))
|
||||
case bool:
|
||||
b.WriteString(pyBool(t))
|
||||
case float64:
|
||||
b.WriteString(pyFloat(t))
|
||||
case int:
|
||||
b.WriteString(pyInt(t))
|
||||
case nil:
|
||||
b.WriteString("null")
|
||||
default:
|
||||
panic(fmt.Sprintf("pyjson: unsupported type %T", v))
|
||||
}
|
||||
}
|
||||
|
||||
// runeSlice — Python str[:n] slices by CHARACTERS, never bytes.
|
||||
func runeSlice(s string, n int) string {
|
||||
if utf8.RuneCountInString(s) <= n {
|
||||
return s
|
||||
}
|
||||
cut := 0
|
||||
for i, count := 0, 0; i < len(s) && count < n; {
|
||||
_, size := utf8.DecodeRuneInString(s[i:])
|
||||
i += size
|
||||
cut = i
|
||||
count++
|
||||
}
|
||||
return s[:cut]
|
||||
}
|
||||
@@ -1,53 +1,719 @@
|
||||
[
|
||||
{
|
||||
"op": "select_pref",
|
||||
"want": "T1"
|
||||
},
|
||||
{
|
||||
"op": "select_fallback",
|
||||
"want": "T0"
|
||||
},
|
||||
{
|
||||
"op": "select_budget",
|
||||
"want": "T1"
|
||||
},
|
||||
{
|
||||
"op": "execute_t0",
|
||||
"want_success": true,
|
||||
"want_tier": "T0"
|
||||
},
|
||||
{
|
||||
"op": "no_profile_error",
|
||||
"pref": [
|
||||
"T3"
|
||||
],
|
||||
"start": "T3",
|
||||
"want_success": false,
|
||||
"want_error": "No profile for tier T3"
|
||||
},
|
||||
{
|
||||
"op": "deterministic_greeting",
|
||||
"family": "greeting",
|
||||
"objective": "hello there friend",
|
||||
"want_message": "Hello! Processing: hello there friend"
|
||||
},
|
||||
{
|
||||
"op": "deterministic_lookup",
|
||||
"family": "lookup",
|
||||
"objective": "find the config value",
|
||||
"want_message": "Lookup: find the config value",
|
||||
"want_method": "deterministic"
|
||||
},
|
||||
{
|
||||
"op": "deterministic_default",
|
||||
"family": "other",
|
||||
"objective": "compute something",
|
||||
"want_message": "Deterministic execution: compute something"
|
||||
},
|
||||
{
|
||||
"op": "deterministic_truncate",
|
||||
"family": "greeting",
|
||||
"objective": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
|
||||
"want_len": 119
|
||||
}
|
||||
]
|
||||
{
|
||||
"frozen_now": "2026-10-14T10:00:00+00:00",
|
||||
"cases": [
|
||||
{
|
||||
"fn": "_te_job_to_dict",
|
||||
"args": [
|
||||
{
|
||||
"job_id": "m1",
|
||||
"task_family": "greeting",
|
||||
"objective": "Respond to hello"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "m1",
|
||||
"task_family": "greeting",
|
||||
"objective": "Respond to hello",
|
||||
"inputs": {},
|
||||
"required_output_schema": null,
|
||||
"risk": "normal",
|
||||
"reversibility": "reversible",
|
||||
"verification_method": null,
|
||||
"preferred_tiers": [],
|
||||
"max_tier": "T4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_job_to_dict",
|
||||
"args": [
|
||||
{
|
||||
"job_id": "m2",
|
||||
"task_family": "architecture",
|
||||
"objective": "héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld 日本語 \"quoted\" \\backslash\t\n🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀",
|
||||
"inputs": {
|
||||
"path": "src/x.py"
|
||||
},
|
||||
"required_output_schema": "diff",
|
||||
"risk": "high",
|
||||
"reversibility": "irreversible",
|
||||
"verification_method": "pytest",
|
||||
"preferred_tiers": [
|
||||
"T0",
|
||||
"T1"
|
||||
],
|
||||
"max_tier": "T3"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "m2",
|
||||
"task_family": "architecture",
|
||||
"objective": "héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld 日本語 \"quoted\" \\backslash\t\n🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀",
|
||||
"inputs": {
|
||||
"path": "src/x.py"
|
||||
},
|
||||
"required_output_schema": "diff",
|
||||
"risk": "high",
|
||||
"reversibility": "irreversible",
|
||||
"verification_method": "pytest",
|
||||
"preferred_tiers": [
|
||||
"T0",
|
||||
"T1"
|
||||
],
|
||||
"max_tier": "T3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_result_to_dict",
|
||||
"args": [
|
||||
{
|
||||
"job_id": "r1",
|
||||
"tier_used": "T0",
|
||||
"model_used": "deterministic",
|
||||
"success": true
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "r1",
|
||||
"tier_used": "T0",
|
||||
"model_used": "deterministic",
|
||||
"success": true,
|
||||
"output": null,
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_result_to_dict",
|
||||
"args": [
|
||||
{
|
||||
"job_id": "r2",
|
||||
"tier_used": "T1",
|
||||
"model_used": "qwen2.5:3b",
|
||||
"success": false,
|
||||
"output": {
|
||||
"response": "partial"
|
||||
},
|
||||
"error": "boom",
|
||||
"verification_result": {
|
||||
"ok": 1
|
||||
},
|
||||
"attempts": 3,
|
||||
"latency_ms": 12.5,
|
||||
"tokens_used": 42
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "r2",
|
||||
"tier_used": "T1",
|
||||
"model_used": "qwen2.5:3b",
|
||||
"success": false,
|
||||
"output": {
|
||||
"response": "partial"
|
||||
},
|
||||
"error": "boom",
|
||||
"verification_result": {
|
||||
"ok": 1
|
||||
},
|
||||
"attempts": 3,
|
||||
"latency_ms": 12.5,
|
||||
"tokens_used": 42
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute",
|
||||
"args": [
|
||||
null,
|
||||
{
|
||||
"job_id": "e1",
|
||||
"task_family": "greeting",
|
||||
"objective": "hi there"
|
||||
},
|
||||
{
|
||||
"start_tier": "T0",
|
||||
"max_tier": "T0"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "e1",
|
||||
"tier_used": "T0",
|
||||
"model_used": "deterministic",
|
||||
"success": true,
|
||||
"output": {
|
||||
"message": "Hello! Processing: hi there"
|
||||
},
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute",
|
||||
"args": [
|
||||
null,
|
||||
{
|
||||
"job_id": "e2",
|
||||
"task_family": "lookup",
|
||||
"objective": "find the config"
|
||||
},
|
||||
{
|
||||
"start_tier": "T0",
|
||||
"max_tier": "T0"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "e2",
|
||||
"tier_used": "T0",
|
||||
"model_used": "deterministic",
|
||||
"success": true,
|
||||
"output": {
|
||||
"message": "Lookup: find the config",
|
||||
"method": "deterministic"
|
||||
},
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute",
|
||||
"args": [
|
||||
null,
|
||||
{
|
||||
"job_id": "e3",
|
||||
"task_family": "coding",
|
||||
"objective": "run the build"
|
||||
},
|
||||
{
|
||||
"start_tier": "T0",
|
||||
"max_tier": "T0"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "e3",
|
||||
"tier_used": "T0",
|
||||
"model_used": "deterministic",
|
||||
"success": true,
|
||||
"output": {
|
||||
"message": "Deterministic execution: run the build"
|
||||
},
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute",
|
||||
"args": [
|
||||
null,
|
||||
{
|
||||
"job_id": "e4",
|
||||
"task_family": "greeting",
|
||||
"objective": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
||||
},
|
||||
{
|
||||
"start_tier": "T0",
|
||||
"max_tier": "T0"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "e4",
|
||||
"tier_used": "T0",
|
||||
"model_used": "deterministic",
|
||||
"success": true,
|
||||
"output": {
|
||||
"message": "Hello! Processing: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
||||
},
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute",
|
||||
"args": [
|
||||
null,
|
||||
{
|
||||
"job_id": "e5",
|
||||
"task_family": "greeting",
|
||||
"objective": "日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語"
|
||||
},
|
||||
{
|
||||
"start_tier": "T0",
|
||||
"max_tier": "T0"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "e5",
|
||||
"tier_used": "T0",
|
||||
"model_used": "deterministic",
|
||||
"success": true,
|
||||
"output": {
|
||||
"message": "Hello! Processing: 日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日"
|
||||
},
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute",
|
||||
"args": [
|
||||
null,
|
||||
{
|
||||
"job_id": "e6",
|
||||
"task_family": "lookup",
|
||||
"objective": "héllo wörld café",
|
||||
"required_output_schema": "y"
|
||||
},
|
||||
{
|
||||
"start_tier": "T0",
|
||||
"max_tier": "T0"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "e6",
|
||||
"tier_used": "T0",
|
||||
"model_used": "deterministic",
|
||||
"success": true,
|
||||
"output": {
|
||||
"message": "Lookup: héllo wörld café",
|
||||
"method": "deterministic"
|
||||
},
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute",
|
||||
"args": [
|
||||
null,
|
||||
{
|
||||
"job_id": "e7",
|
||||
"task_family": "greeting",
|
||||
"objective": "pref first",
|
||||
"preferred_tiers": [
|
||||
"T0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"start_tier": "T2",
|
||||
"max_tier": "T4"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "e7",
|
||||
"tier_used": "T0",
|
||||
"model_used": "deterministic",
|
||||
"success": true,
|
||||
"output": {
|
||||
"message": "Hello! Processing: pref first"
|
||||
},
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute",
|
||||
"args": [
|
||||
{
|
||||
"T9": {
|
||||
"id": "none",
|
||||
"provider": "ollama",
|
||||
"tier": "T9",
|
||||
"local": true,
|
||||
"context_window": 32768,
|
||||
"supports_tools": false,
|
||||
"supports_structured_output": false,
|
||||
"supports_reasoning_field": false,
|
||||
"estimated_latency": "high",
|
||||
"estimated_cost": "0",
|
||||
"enabled": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"job_id": "e8",
|
||||
"task_family": "coding",
|
||||
"objective": "x",
|
||||
"preferred_tiers": [
|
||||
"T3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"start_tier": "T3",
|
||||
"max_tier": "T4"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "e8",
|
||||
"tier_used": "T3",
|
||||
"model_used": "none",
|
||||
"success": false,
|
||||
"output": null,
|
||||
"error": "No profile for tier T3",
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute",
|
||||
"args": [
|
||||
{
|
||||
"T9": {
|
||||
"id": "none",
|
||||
"provider": "ollama",
|
||||
"tier": "T9",
|
||||
"local": true,
|
||||
"context_window": 32768,
|
||||
"supports_tools": false,
|
||||
"supports_structured_output": false,
|
||||
"supports_reasoning_field": false,
|
||||
"estimated_latency": "high",
|
||||
"estimated_cost": "0",
|
||||
"enabled": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"job_id": "e9",
|
||||
"task_family": "coding",
|
||||
"objective": "x"
|
||||
},
|
||||
{
|
||||
"start_tier": "T3",
|
||||
"max_tier": "T4"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "e9",
|
||||
"tier_used": "T3",
|
||||
"model_used": "none",
|
||||
"success": false,
|
||||
"output": null,
|
||||
"error": "No profile for tier T3",
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute_ollama",
|
||||
"args": [
|
||||
null,
|
||||
null,
|
||||
{
|
||||
"job_id": "m1",
|
||||
"task_family": "coding",
|
||||
"objective": "fix it"
|
||||
},
|
||||
{
|
||||
"start_tier": "T1",
|
||||
"max_tier": "T2"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "m1",
|
||||
"tier_used": "T1",
|
||||
"model_used": "qwen2.5:3b",
|
||||
"success": true,
|
||||
"output": {
|
||||
"response": "{\"model\": \"qwen2.5:3b\", \"prompt\": \"fix it\", \"stream\": false, \"think\": false, \"options\": {\"temperature\": 0.3, \"num_predict\": 1000, \"num_ctx\": 30720}}"
|
||||
},
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute_ollama",
|
||||
"args": [
|
||||
null,
|
||||
null,
|
||||
{
|
||||
"job_id": "m2",
|
||||
"task_family": "coding",
|
||||
"objective": "héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld héllo wörld 日本語 \"quoted\" \\backslash\t\n🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀"
|
||||
},
|
||||
{
|
||||
"start_tier": "T1",
|
||||
"max_tier": "T2"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "m2",
|
||||
"tier_used": "T1",
|
||||
"model_used": "qwen2.5:3b",
|
||||
"success": true,
|
||||
"output": {
|
||||
"response": "{\"model\": \"qwen2.5:3b\", \"prompt\": \"h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld h\\u00e9llo w\\u00f6rld \\u65e5\\u672c\\u8a9e \\\"quoted\\\" \\\\backslash\\t\\n\\ud83d\\ude80\\ud83d\\ude80\\ud83d\\ude80\\ud83d\\ude80\\ud83d\\ude80\\ud83d\\ude80\\ud83d\\ude80\\ud83d\\ude80\\ud83d\\ude80\\ud83d\\ude80\\ud83d\\ude80\\ud83d\\ude80\\ud83d\\ude80\\ud8"
|
||||
},
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute_ollama",
|
||||
"args": [
|
||||
null,
|
||||
null,
|
||||
{
|
||||
"job_id": "m3",
|
||||
"task_family": "coding",
|
||||
"objective": "preferred tier",
|
||||
"preferred_tiers": [
|
||||
"T2"
|
||||
]
|
||||
},
|
||||
{
|
||||
"start_tier": "T0",
|
||||
"max_tier": "T4"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "m3",
|
||||
"tier_used": "T2",
|
||||
"model_used": "qwen3:14b",
|
||||
"success": true,
|
||||
"output": {
|
||||
"response": "{\"model\": \"qwen3:14b\", \"prompt\": \"preferred tier\", \"stream\": false, \"think\": false, \"options\": {\"temperature\": 0.3, \"num_predict\": 1000, \"num_ctx\": 30720}}"
|
||||
},
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute_ollama",
|
||||
"args": [
|
||||
null,
|
||||
{
|
||||
"T3": {
|
||||
"id": "qwen3.6:27b",
|
||||
"provider": "ollama",
|
||||
"tier": "T3",
|
||||
"local": true,
|
||||
"context_window": 32768,
|
||||
"supports_tools": false,
|
||||
"supports_structured_output": false,
|
||||
"supports_reasoning_field": false,
|
||||
"estimated_latency": "high",
|
||||
"estimated_cost": "0",
|
||||
"enabled": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"job_id": "m4",
|
||||
"task_family": "coding",
|
||||
"objective": "subset reg"
|
||||
},
|
||||
{
|
||||
"start_tier": "T3",
|
||||
"max_tier": "T4"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "m4",
|
||||
"tier_used": "T3",
|
||||
"model_used": "qwen3.6:27b",
|
||||
"success": true,
|
||||
"output": {
|
||||
"response": "{\"model\": \"qwen3.6:27b\", \"prompt\": \"subset reg\", \"stream\": false, \"think\": false, \"options\": {\"temperature\": 0.3, \"num_predict\": 1000, \"num_ctx\": 30720}}"
|
||||
},
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute_ollama",
|
||||
"args": [
|
||||
null,
|
||||
null,
|
||||
{
|
||||
"job_id": "m5",
|
||||
"task_family": "coding",
|
||||
"objective": "cloud tier"
|
||||
},
|
||||
{
|
||||
"start_tier": "T4",
|
||||
"max_tier": "T4"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "m5",
|
||||
"tier_used": "T4",
|
||||
"model_used": "mimo-v2.5-pro",
|
||||
"success": true,
|
||||
"output": {
|
||||
"response": "{\"model\": \"mimo-v2.5-pro\", \"prompt\": \"cloud tier\", \"stream\": false, \"think\": false, \"options\": {\"temperature\": 0.3, \"num_predict\": 1000, \"num_ctx\": 124928}}"
|
||||
},
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute_ollama",
|
||||
"args": [
|
||||
"definitely-not-a-url",
|
||||
null,
|
||||
{
|
||||
"job_id": "x1",
|
||||
"task_family": "coding",
|
||||
"objective": "bad scheme"
|
||||
},
|
||||
{
|
||||
"start_tier": "T1",
|
||||
"max_tier": "T2"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "x1",
|
||||
"tier_used": "T1",
|
||||
"model_used": "qwen2.5:3b",
|
||||
"success": false,
|
||||
"output": null,
|
||||
"error": "unknown url type: 'definitely-not-a-url/api/generate'",
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_te_execute_ollama",
|
||||
"args": [
|
||||
"http://127.0.0.1:9",
|
||||
null,
|
||||
{
|
||||
"job_id": "x2",
|
||||
"task_family": "coding",
|
||||
"objective": "refused"
|
||||
},
|
||||
{
|
||||
"start_tier": "T1",
|
||||
"max_tier": "T2"
|
||||
}
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"job_id": "x2",
|
||||
"tier_used": "T1",
|
||||
"model_used": "qwen2.5:3b",
|
||||
"success": false,
|
||||
"output": null,
|
||||
"error": "<urlopen error [Errno 111] Connection refused>",
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fn": "_ctx_budget",
|
||||
"args": [
|
||||
"qwen2.5:3b"
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": 30720
|
||||
},
|
||||
{
|
||||
"fn": "_ctx_budget",
|
||||
"args": [
|
||||
"qwen3:14b"
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": 30720
|
||||
},
|
||||
{
|
||||
"fn": "_ctx_budget",
|
||||
"args": [
|
||||
"qwen3.6:27b"
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": 30720
|
||||
},
|
||||
{
|
||||
"fn": "_ctx_budget",
|
||||
"args": [
|
||||
"mimo-v2.5-pro"
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": 124928
|
||||
},
|
||||
{
|
||||
"fn": "_ctx_budget",
|
||||
"args": [
|
||||
""
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": 6144
|
||||
},
|
||||
{
|
||||
"fn": "_ctx_budget",
|
||||
"args": [
|
||||
"no-such-model"
|
||||
],
|
||||
"kwargs": {},
|
||||
"expected": 6144
|
||||
},
|
||||
{
|
||||
"fn": "_te_test_executor",
|
||||
"args": [],
|
||||
"kwargs": {},
|
||||
"expected": {
|
||||
"returned": null
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,328 @@
|
||||
package main
|
||||
|
||||
// truth_test.go — the parity gate. Replays every case from truth.json (the
|
||||
// shared freeze_truth.py envelope: frozen time seam + echo-server model
|
||||
// path, generated by scripts/gen_tiered_executor_truth.py against the live
|
||||
// oracle under scripts/te_freeze_seams) VERBATIM through the ported Go
|
||||
// functions, in the recorded call order. A divergence fails the port, not
|
||||
// the oracle.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
_ "embed"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
//go:embed truth.json
|
||||
var frozenTruthJSON string
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
echo := newEchoServer()
|
||||
// the wall-clock seam: the freeze rig pinned time.time, so every
|
||||
// captured latency is exactly 0.0 — the Go port mirrors through
|
||||
// TE_LATENCY_MS (executed before m.Run so every test sees it)
|
||||
os.Setenv("TE_LATENCY_MS", "0")
|
||||
os.Setenv("OLLAMA_URL", echo.URL)
|
||||
code := m.Run()
|
||||
echo.Close()
|
||||
os.Exit(code)
|
||||
}
|
||||
|
||||
func TestFrozenTruthReplay(t *testing.T) {
|
||||
var env struct {
|
||||
FrozenNow string `json:"frozen_now"`
|
||||
Cases []map[string]any `json:"cases"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(frozenTruthJSON), &env); err != nil {
|
||||
t.Fatalf("truth.json: %v", err)
|
||||
}
|
||||
if len(env.Cases) == 0 {
|
||||
t.Fatal("no truth cases — the parity gate is empty")
|
||||
}
|
||||
echo := newEchoServer()
|
||||
defer echo.Close()
|
||||
fails := 0
|
||||
for i, row := range env.Cases {
|
||||
fn, _ := row["fn"].(string)
|
||||
args, _ := row["args"].([]any)
|
||||
got, err := dispatchRow(fn, args, echo.URL)
|
||||
if err != nil {
|
||||
t.Errorf("case %d %s: dispatch error %v", i, fn, err)
|
||||
fails++
|
||||
continue
|
||||
}
|
||||
if !jsonEqAny(got, row["expected"]) {
|
||||
fails++
|
||||
t.Errorf("case %d %s:\n got: %s\n want: %s", i, fn, mustJSON(got), mustJSON(row["expected"]))
|
||||
}
|
||||
}
|
||||
if fails == 0 {
|
||||
t.Logf("PARITY OK: %d cases exact-equal vs the Python oracle (frozen_now=%s)", len(env.Cases), env.FrozenNow)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPayloadBytes — the _execute_model OLLAMA payload is json.dumps
|
||||
// byte-parity: ", "/": " separators, insertion order, ensure_ascii.
|
||||
func TestPayloadBytes(t *testing.T) {
|
||||
got := formatPayload("qwen2.5:3b", "fix it", 30720)
|
||||
want := `{"model": "qwen2.5:3b", "prompt": "fix it", "stream": false, "think": false, "options": {"temperature": 0.3, "num_predict": 1000, "num_ctx": 30720}}`
|
||||
if got != want {
|
||||
t.Errorf("payload:\n got %s\nwant %s", got, want)
|
||||
}
|
||||
unicode := formatPayload("mimo-v2.5-pro", "héllo 日本語 \"q\" \\ \n 🚀", 124928)
|
||||
wantUni := `{"model": "mimo-v2.5-pro", "prompt": "h\u00e9llo \u65e5\u672c\u8a9e \"q\" \\ \n \ud83d\ude80", "stream": false, "think": false, "options": {"temperature": 0.3, "num_predict": 1000, "num_ctx": 124928}}`
|
||||
if unicode != wantUni {
|
||||
t.Errorf("unicode payload:\n got %s\nwant %s", unicode, wantUni)
|
||||
}
|
||||
}
|
||||
|
||||
// TestModelPathCrashParity — bad scheme + connection refused ride the
|
||||
// oracle's exact str(e) messages; no-profile results carry zero latency
|
||||
// (the oracle returns before its latency line).
|
||||
func TestModelPathCrashParity(t *testing.T) {
|
||||
x := NewTieredExecutor(nil)
|
||||
j, _ := jobFromMap(map[string]any{"job_id": "x", "task_family": "coding", "objective": "y"})
|
||||
|
||||
t.Setenv("OLLAMA_URL", "definitely-not-a-url")
|
||||
r := x.Execute(j, &CognitiveBudget{StartTier: "T2", MaxTier: "T2"})
|
||||
if r.Success {
|
||||
t.Fatalf("bad scheme must fail")
|
||||
}
|
||||
if r.Error != "unknown url type: 'definitely-not-a-url/api/generate'" {
|
||||
t.Errorf("bad scheme error: got %q", r.Error)
|
||||
}
|
||||
if r.TierUsed != "T2" || r.ModelUsed != "qwen3:14b" {
|
||||
t.Errorf("bad scheme envelope: tier=%q model=%q", r.TierUsed, r.ModelUsed)
|
||||
}
|
||||
|
||||
t.Setenv("OLLAMA_URL", "http://127.0.0.1:9")
|
||||
r2 := x.Execute(j, &CognitiveBudget{StartTier: "T2", MaxTier: "T2"})
|
||||
if r2.Error != "<urlopen error [Errno 111] Connection refused>" {
|
||||
t.Errorf("refused error: got %q", r2.Error)
|
||||
}
|
||||
|
||||
xSentinel := NewTieredExecutor(registryFromSpec(map[string]any{"T9": map[string]any{
|
||||
"id": "none", "provider": "ollama", "tier": "T9", "local": true,
|
||||
"context_window": float64(32768), "estimated_latency": "high", "estimated_cost": "0",
|
||||
}}))
|
||||
r3 := xSentinel.Execute(j, &CognitiveBudget{StartTier: "T3", MaxTier: "T4"})
|
||||
if r3.Success || r3.Error != "No profile for tier T3" || r3.ModelUsed != "none" || r3.Attempts != 1 {
|
||||
t.Errorf("no-profile envelope: %+v", r3)
|
||||
}
|
||||
if r3.LatencyMs != 0.0 {
|
||||
// no-profile returns BEFORE the latency line in the oracle — the
|
||||
// seam value never touches it
|
||||
t.Errorf("no-profile latency stamped: %v", r3.LatencyMs)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRuneSlicing — Python str[:n] slices by CHARACTERS (dichotomy quirk).
|
||||
func TestRuneSlicing(t *testing.T) {
|
||||
s := "日本語" + "x"
|
||||
if got := runeSlice(s, 2); got != "日本" {
|
||||
t.Errorf("runeSlice [:2]: got %q want 日本", got)
|
||||
}
|
||||
if got := runeSlice(s, 100); got != s {
|
||||
t.Errorf("runeSlice short: got %q", got)
|
||||
}
|
||||
d := deterministicMessage("greeting", strings.Repeat("日本語", 40))
|
||||
if len([]rune(d)) != 119 {
|
||||
t.Errorf("greeting [:100] prefix: %d runes:\n%q", len([]rune(d)), d)
|
||||
}
|
||||
}
|
||||
|
||||
func deterministicMessage(family, objective string) string {
|
||||
x := NewTieredExecutor(nil)
|
||||
j, _ := jobFromMap(map[string]any{"job_id": "d", "task_family": family, "objective": objective})
|
||||
r := x.ExecuteDeterministic(j, nil)
|
||||
m, _ := r.Output.(map[string]any)
|
||||
v, _ := m["message"].(string)
|
||||
return v
|
||||
}
|
||||
|
||||
// TestCtxBudgetVendored — resolve_context_budget parity: local/cloud/
|
||||
// unknown model windows, reserves, the max(1024, ...) floor and the
|
||||
// context_window=0 or-8192 branch.
|
||||
func TestCtxBudgetVendored(t *testing.T) {
|
||||
cases := []struct {
|
||||
model, kind string
|
||||
want map[string]any
|
||||
}{
|
||||
{"qwen2.5:3b", "default", map[string]any{"num_ctx": 30720, "window": 32768, "output_reserve": 2048, "tool_reserve": 0, "model": "qwen2.5:3b", "local": true}},
|
||||
{"qwen2.5:3b", "verify", map[string]any{"num_ctx": 32256, "window": 32768, "output_reserve": 512, "tool_reserve": 0, "model": "qwen2.5:3b", "local": true}},
|
||||
{"mimo-v2.5-pro", "embed", map[string]any{"num_ctx": 126976, "window": 128000, "output_reserve": 0, "tool_reserve": 1024, "model": "mimo-v2.5-pro", "local": false}},
|
||||
{"deterministic", "default", map[string]any{"num_ctx": 6144, "window": 8192, "output_reserve": 2048, "tool_reserve": 0, "model": "deterministic", "local": true}},
|
||||
{"", "default", map[string]any{"num_ctx": 6144, "window": 8192, "output_reserve": 2048, "tool_reserve": 0, "model": "", "local": true}},
|
||||
{"unknown", "summarize", map[string]any{"num_ctx": 7168, "window": 8192, "output_reserve": 1024, "tool_reserve": 0, "model": "unknown", "local": true}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := ResolveCtxBudget(c.model, c.kind); !jsonEqAny(got, c.want) {
|
||||
t.Errorf("ResolveCtxBudget(%q,%q):\n got %s\nwant %s", c.model, c.kind, mustJSON(got), mustJSON(c.want))
|
||||
}
|
||||
}
|
||||
if n := CtxBudget("qwen2.5:3b"); n != 30720 {
|
||||
t.Errorf("CtxBudget num_ctx: got %d want 30720", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRegistryFalsiness — ModelRegistry({}) is falsy in the oracle:
|
||||
// a nil or EMPTY profile map falls back to DEFAULT_PROFILES; a sentinel
|
||||
// registry (T9 only) misses every real tier (get_profile -> None).
|
||||
func TestRegistryFalsiness(t *testing.T) {
|
||||
empty := NewModelRegistry(map[string]*ModelProfile{})
|
||||
if got := empty.GetProfile("T0"); got == nil {
|
||||
t.Fatal("empty registry must fall back to DEFAULT_PROFILES (falsiness law)")
|
||||
}
|
||||
if got := empty.GetProfile("T1").ID; got != "qwen2.5:3b" {
|
||||
t.Errorf("profiles map: got %q", got)
|
||||
}
|
||||
sentinel := NewModelRegistry(map[string]*ModelProfile{"T9": {ID: "none", Tier: "T9"}})
|
||||
for _, tier := range []string{"T0", "T1", "T2", "T3", "T4"} {
|
||||
if got := sentinel.GetProfile(tier); got != nil {
|
||||
t.Errorf("sentinel registry returned %v for %s", got, tier)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestToDictOrderAndTypes — the serializer rows through the constructors:
|
||||
// defaults, Optional nulls, and the CLI indent-2 renderer's shape.
|
||||
func TestToDictOrderAndTypes(t *testing.T) {
|
||||
jm := map[string]any{"job_id": "m1", "task_family": "greeting", "objective": "Respond to hello"}
|
||||
j, _ := jobFromMap(jm)
|
||||
d := j.toDict()
|
||||
wantKeys := []string{"inputs", "required_output_schema", "risk", "reversibility", "verification_method", "preferred_tiers", "max_tier"}
|
||||
wantVals := map[string]any{ // dataclass defaults
|
||||
"inputs": map[string]any{},
|
||||
"required_output_schema": nil,
|
||||
"risk": "normal",
|
||||
"reversibility": "reversible",
|
||||
"verification_method": nil,
|
||||
"preferred_tiers": []any{},
|
||||
"max_tier": "T4",
|
||||
}
|
||||
for _, k := range wantKeys {
|
||||
if !jsonEqAny(d[k], wantVals[k]) {
|
||||
t.Errorf("job default %s: got %s want %s", k, mustJSON(d[k]), mustJSON(wantVals[k]))
|
||||
}
|
||||
}
|
||||
|
||||
rm := map[string]any{"job_id": "r1", "tier_used": "T0", "model_used": "deterministic", "success": true}
|
||||
r, _ := resultFromMap(rm)
|
||||
rd := r.toDict()
|
||||
if !jsonEqAny(rd["attempts"], 1) || rd["output"] != nil || rd["error"] != nil {
|
||||
t.Errorf("result defaults: %s", mustJSON(rd))
|
||||
}
|
||||
|
||||
cli := renderResultCLI(r)
|
||||
wantCLI := `{
|
||||
"job_id": "r1",
|
||||
"tier_used": "T0",
|
||||
"model_used": "deterministic",
|
||||
"success": true,
|
||||
"output": null,
|
||||
"error": null,
|
||||
"verification_result": null,
|
||||
"attempts": 1,
|
||||
"latency_ms": 0.0,
|
||||
"tokens_used": 0
|
||||
}`
|
||||
if cli != wantCLI {
|
||||
t.Errorf("cli render:\n got %s\nwant %s", cli, wantCLI)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFloatRendering — float.__repr__ parity rules (pyFloat).
|
||||
func TestFloatRendering(t *testing.T) {
|
||||
a, b := 0.1, 0.2 // runtime values — Go's constant arithmetic folds 0.1+0.2 to exactly 0.3
|
||||
cases := map[string]float64{
|
||||
"0.0": 0,
|
||||
"0.3": 0.3,
|
||||
"12.5": 12.5,
|
||||
"12.0": 12,
|
||||
"0.0064373016357421875": 0.0064373016357421875,
|
||||
"0.30000000000000004": a + b,
|
||||
"1e+16": 1e16,
|
||||
"1e-05": 1e-5,
|
||||
"1234567.5": 1234567.5,
|
||||
"1.7976931348623157e+308": 1.7976931348623157e308,
|
||||
}
|
||||
for want, f := range cases {
|
||||
if got := pyFloat(f); got != want {
|
||||
t.Errorf("pyFloat(%v): got %s want %s", f, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestPyStrEscapes — ensure_ascii escapes incl. surrogate pairs, quotes,
|
||||
// backslashes, control chars, lowercase hex.
|
||||
func TestPyStrEscapes(t *testing.T) {
|
||||
cases := []struct{ in, want string }{
|
||||
{"plain", `"plain"`},
|
||||
{"héllo", `"h\u00e9llo"`},
|
||||
{`"q"\r`, `"\"q\"\\r"`},
|
||||
{"tab\tnewline\n", `"tab\tnewline\n"`},
|
||||
{"日本語", `"\u65e5\u672c\u8a9e"`},
|
||||
{"🚀", `"\ud83d\ude80"`},
|
||||
{"del\x7f", `"del\u007f"`},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := pyStr(c.in); got != c.want {
|
||||
t.Errorf("pyStr(%q): got %s want %s", c.in, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestExecuteModelEcho — full model-path round-trip against the test
|
||||
// echo server: payload bytes ride back through the response slice.
|
||||
func TestExecuteModelEcho(t *testing.T) {
|
||||
echo := newEchoServer()
|
||||
defer echo.Close()
|
||||
t.Setenv("OLLAMA_URL", echo.URL)
|
||||
x := NewTieredExecutor(nil)
|
||||
j, _ := jobFromMap(map[string]any{"job_id": "me", "task_family": "coding", "objective": "fix it"})
|
||||
r := x.Execute(j, &CognitiveBudget{StartTier: "T1", MaxTier: "T2"})
|
||||
if !r.Success {
|
||||
t.Fatalf("echo model call failed: %v", r.Error)
|
||||
}
|
||||
want := `{"model": "qwen2.5:3b", "prompt": "fix it", "stream": false, "think": false, "options": {"temperature": 0.3, "num_predict": 1000, "num_ctx": 30720}}`
|
||||
out, _ := r.Output.(map[string]any)
|
||||
if out["response"] != want {
|
||||
t.Errorf("echoed payload:\n got %v\nwant %s", out["response"], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTestExecutorLines(t *testing.T) {
|
||||
echo := newEchoServer()
|
||||
defer echo.Close()
|
||||
t.Setenv("OLLAMA_URL", echo.URL)
|
||||
var buf bytes.Buffer
|
||||
if err := TestExecutor(&buf); err != nil {
|
||||
t.Fatalf("TestExecutor: %v", err)
|
||||
}
|
||||
want := "Testing Tiered Executor...\n" +
|
||||
"Greeting: tier=T0, success=True\n" +
|
||||
"Coding: tier=T2, success=True\n" +
|
||||
"✅ All tests passed!\n"
|
||||
if buf.String() != want {
|
||||
t.Errorf("test lines:\n got %q\nwant %q", buf.String(), want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTestExecutorCrashParity — bad-scheme OLLAMA_URL fails the coding
|
||||
// assert exactly like the oracle (two lines, then exit 1, no verdict).
|
||||
func TestTestExecutorCrashParity(t *testing.T) {
|
||||
t.Setenv("OLLAMA_URL", "definitely-not-a-url")
|
||||
var buf bytes.Buffer
|
||||
err := TestExecutor(&buf)
|
||||
if err == nil {
|
||||
t.Fatal("coding assert must fail without a model backend")
|
||||
}
|
||||
lines := strings.Split(strings.TrimSuffix(buf.String(), "\n"), "\n")
|
||||
if len(lines) != 3 || lines[0] != "Testing Tiered Executor..." ||
|
||||
lines[1] != "Greeting: tier=T0, success=True" || lines[2] != "Coding: tier=T2, success=False" {
|
||||
t.Errorf("crash lines: %q", buf.String())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
AUDIT_EXIT=0
|
||||
@@ -0,0 +1 @@
|
||||
EXIT=0
|
||||
@@ -0,0 +1 @@
|
||||
V2_EXIT=0
|
||||
@@ -0,0 +1 @@
|
||||
BUILD_EXIT=0
|
||||
@@ -0,0 +1 @@
|
||||
GOFMT_EXIT=0
|
||||
@@ -0,0 +1 @@
|
||||
TEST_EXIT=0
|
||||
@@ -0,0 +1 @@
|
||||
VET_EXIT=0
|
||||
@@ -0,0 +1 @@
|
||||
PARITY_EXIT=0
|
||||
@@ -0,0 +1 @@
|
||||
SMOKE_EXIT=0
|
||||
@@ -0,0 +1 @@
|
||||
REPLAY_EXIT=0
|
||||
@@ -0,0 +1,68 @@
|
||||
# tiered_executor gap-close verification evidence (dichotomy wave #2479, worktree rune-wave-cs2)
|
||||
|
||||
Module: tiered_executor — audit 3/8 PARTIAL → 8/8 FULL (GO side).
|
||||
Oracle: src/rune/tiered_executor.py (291 lines). Port: src/go/tiered-executor.
|
||||
|
||||
## Gate results (VERIFY-GATED, file-based protocol)
|
||||
|
||||
| Gate | Command (cwd) | Evidence | Exit |
|
||||
|---|---|---|---|
|
||||
| Freeze gen | `PYTHONPATH=scripts/te_freeze_seams REPO_ROOT=$PWD python3 scripts/gen_tiered_executor_truth.py` | (27 rows → src/go/tiered-executor/truth.json, envelope frozen_now=2026-10-14T10:00:00+00:00; 13 shape asserts passed inline) | 0 |
|
||||
| Freeze verify ×1 | `PYTHONPATH=scripts/te_freeze_seams ... freeze_truth.py --verify src/go/tiered-executor/truth.json --module rune.tiered_executor` | te-freeze-verify-1.log ("OK: 27 cases byte-identical against live oracle") | 0 |
|
||||
| Freeze verify ×2 | same | te-freeze-verify-2.log ("OK: 27 cases byte-identical") | 0 |
|
||||
| go vet | `go vet ./...` (src/go/tiered-executor) | te-govet.log/exit | 0 |
|
||||
| gofmt | `gofmt -l .` (empty output = clean) | te-gofmt.log/exit | 0 |
|
||||
| go build | `go build ./...` | te-gobuild.log/exit | 0 |
|
||||
| Battery | `go test ./...` — TestFrozenTruthReplay 27/27 exact-equal + 10 parity/branch tests | te-gotest.log/exit | 0 |
|
||||
| CI replay binary | `./tiered-executor replay truth.json` — "PARITY OK: 27 frozen-truth rows replayed byte-equal" | te-replay-binary.log/exit | 0 |
|
||||
| Live parity | `PYTHONPATH=scripts/te_freeze_seams python3 scripts/check_tiered_executor_parity.py` — 31/31 (25 op families + 6 CLI) | te-parity.log/exit | 0 |
|
||||
| Audit | `python3 scripts/dichotomy_audit.py --write` — \`tiered_executor 291 8/8 GO\` | te-audit.log/exit; docs/dichotomy-rebase-honest-audit.md | 0 |
|
||||
| Python both pins | T0 + T2-model (echo) smoke: env unset (delegation-live; no loader seam exists — vacuous pin, PP/mconn precedent) AND RUNE_TIERED_EXECUTOR_CORE=python | te-py-pin-live.log/exit | 0 |
|
||||
|
||||
## What was ported (5 audit-missing fns + supporting surface)
|
||||
|
||||
- `CognitiveJob.to_dict` / `CognitiveResult.to_dict` → `(*CognitiveJob).toDict` /
|
||||
`(*CognitiveResult).toDict` (executor.go) + `renderResultCLI` indent-2
|
||||
byte renderer for the CLI surface (json.dumps(result.to_dict(), indent=2)).
|
||||
- `TieredExecutor.execute` full surface: no-profile early return
|
||||
(attempts=1, zero latency — oracle returns before its latency line),
|
||||
T0-deterministic vs model dispatch, outer latency overwrite.
|
||||
- `_execute_model` → `ExecuteModel`: json.dumps-identical payload
|
||||
(", '/'" separators, insertion order, ensure_ascii incl. surrogate
|
||||
pairs), rune-safe [:2000] response slice, crash-parity error envelopes
|
||||
("unknown url type: '...'", "<urlopen error [Errno 111] Connection
|
||||
refused>" — str(e) message-only contract).
|
||||
- `_ctx_budget` → `CtxBudget` + vendored `ResolveCtxBudget`
|
||||
(context_budget.go; GO SIBLING ISOLATION law — no cross-module import).
|
||||
- `test_executor` → `TestExecutor` (assert-parity: crash exit 1 after the
|
||||
Coding line, Python True/False literals in printed lines).
|
||||
- `__init__` → `NewTieredExecutor(nil)` defaults (the ImportError branch
|
||||
is python-import machinery, unreachable in the Go binary — documented).
|
||||
|
||||
## Freeze design (scripts/te_freeze_seams/sitecustomize.py + scripts/gen_tiered_executor_truth.py)
|
||||
|
||||
- RUNE_TIERED_EXECUTOR_CORE=python pinned; `time.time` frozen (every
|
||||
captured latency exactly 0.0; Go mirrors via the `TE_LATENCY_MS` seam —
|
||||
battery pin 0, replay CLI pin 0).
|
||||
- OLLAMA echo server (ports 20145-20154, first bindable, OLLAMA_URL seam)
|
||||
reflects the request body back — the frozen rows pin the exact
|
||||
json.dumps payload bytes through the echo round-trip, short + >2000
|
||||
truncation rows, custom/subset registries, local AND cloud profiles.
|
||||
- Registry falsiness row: `ModelRegistry({})` is falsy → DEFAULT_PROFILES
|
||||
fallback; no-profile rows use the T9 sentinel registry.
|
||||
- No absolute paths, no wall-clock values in truth (latency pinned 0.0).
|
||||
|
||||
## Divergences (documented, not bugs)
|
||||
|
||||
1. Transport-error messages for failure classes other than unknown-url-type
|
||||
/ connection-refused are stack-specific (urllib vs Go net) — the
|
||||
contract frozen covers the two deterministic classes; any other
|
||||
transport error degrades to the message-only envelope shape (structural).
|
||||
2. CLI `execute` latency_ms: measured (no wall-clock freezing in a bare
|
||||
CLI run) — compared with the latency line normalized on both sides
|
||||
(te-parity.log "cli execute ... (latency normalized)"); frozen-truth
|
||||
rows pin latency at 0.0 via the seam instead.
|
||||
3. Integral float rendering (0.0 vs 0) is equal-after-parse: op-family
|
||||
comparison normalizes integral floats for the comparison only (quirk:
|
||||
json.dumps "0.0" vs Go Marshal "0"); the CLI renderer emits "0.0"
|
||||
byte-exactly (pyFloat).
|
||||
Reference in new issue
Block a user