diff --git a/plans/2026-10-03-phaseG-evidence-provenance-supersession.md b/plans/2026-10-03-phaseG-evidence-provenance-supersession.md new file mode 100644 index 00000000..4762e181 --- /dev/null +++ b/plans/2026-10-03-phaseG-evidence-provenance-supersession.md @@ -0,0 +1,62 @@ +# Phase G slice 2 — evidence-provenance authority supersession ledger + +**Date:** 2026-10-03 · **Branch:** `phase-g-strangler-evidence-provenance` · **Unit issue:** conductor/rune#2449 + +## What was superseded + +The pure-Python evidence-provenance implementations (six modules in `src/rune/`, 1796 lines; plus the two OLD deployed vintages and `shared/audit_ledger.py`) are RETIRED as production implementations for their ported surfaces. Production authority now rests with: + +1. **Rust crate `src/rust/evidence_provenance/`** — six cores: `lib.rs` (audit_ledger chain + FFI), `verification.rs` (7 verifiers), `probe.rs` (schema gate + budget), `behaviour.rs` (state machine), `habitmath.rs` (reinforce math), `claims.rs` (claim-audit decision skeleton); cdylib FFI (`ep_*` symbols, 21 total), zero external crates. +2. **Python shims** at `src/rune/{audit_ledger,verification_layer,evidence_probe,behaviour_ledger,behavior,claim_audit}.py` — serving the src/rune authority contract through `ep_loader.py` (in-process, no interpreter startup for the ported logic). +3. **Deployed shims** at `~/.config/opencode/scripts/{audit_ledger,behavior,claim_audit}.py` — same shape; audit_ledger + claim_audit + behavior carry the LIVE data-path rules (drift-key ledger, live behavior.db, real journal); claim_audit additionally carries the preserved SYMMETRY GUARD (deployed-vintage feature) and the repaired `scan_text()` TUI surface. + +## Port ledger (all six, in contract §4 order) + +| Port | Module | Rust core | Parity | Commit | Consumer proof | +|------|--------|-----------|--------|--------|----------------| +| 1 | audit_ledger | lib.rs chain ops | shadow 18/18; canaries 28/28+23/23 | `114c394`+`4b0d1d2` | live ledger entries 4242 (post-switch write), verify 23-intact | +| 2 | verification_layer | verification.rs | 33/33 (7 verifiers + CLI render) | `9cb93e6` | contracts.py, canary_native, rune_operator:323 dataclass | +| 3 | evidence_probe | probe.rs | 24/24 + consumer tests 23/23 | `3450a1c` | rune_operator:581/603, research_stage_executor:312 | +| 4 | behaviour_ledger | behaviour.rs | 13/13 full-scenario | `8973caa` | testing_mode:19, iteration:23, rune_tui:3152 | +| 5 | behavior | habitmath.rs | 16/16 + full-stack live-db | `2732677` | rune_tui:83; deployed vintage upgraded (live db read-verified) | +| 6 | claim_audit | claims.rs | 17/17 + real-journal aggregate (1127 rows) | `969e39f` | rune_tui:62/2468/3752, contracts.py, hook_engine:270 | + +Errors: byte-equal (duplicate/unknown-behaviour/unknown-state; probe schema-gate lists incl. Python repr rendering; CLI argparse shapes + exit codes). + +## Strangler doctrine (master doc lines 685–770) — step discharge (slice level) + +| Step | Requirement | Evidence | +|------|-------------|----------| +| 1–4 | candidate built, contract frozen, parity proven | commits `114c394` → `969e39f`; frozen contract `plans/2026-10-03-phaseG-evidence-provenance-contract.md` (commit `7c68e41`); per-port parity harnesses ALL GREEN | +| 5 | qualification | per-port parity + decision-math batteries; canary pair per port (Python 34/34, Rust 28/28 at slice end); performance follows the slice-1 in-process shape (no interpreter startup; same cdylib) | +| 6 | canaries conformance | canary_audit_ledger.py 34/34; canary_evidence_provenance_rust.py 28/28 (incl. port-2/3 sections); embedded tests through shims (verification test, 23 consumer tests) | +| 7 | shadow comparison vs live-shaped data | port-1 shadow 18/18 vs `939883d:shared/audit_ledger.py` (banked at `114c394`); ports 2–6 use direct byte-parity batteries vs the live originals (same discipline, stronger: full-document byte-quality) | +| 8 | authority switch + observed consumer evidence | shims deployed + live-verified per port: entry 4242 (first post-switch ledger write), live behavior.db read through the port-5 shim, real 1127-row journal through the port-6 aggregation, TUI backend trio imports, TUI scan_text FIRST-EVER live call | +| 9 | runtime evidence post-switch | post-switch walls per port: parity round-trips, embedded tests, consumer chains (rune.behaviour_ledger / rune.evidence_probe / rune.claim_audit + deployed trio) — all GREEN | +| 10 | old production path unreachable | `import audit_ledger` (deployed consumers) → shim; `rune.behaviour_ledger` / `rune.evidence_probe` / `rune.claim_audit` / `rune.verification_layer` → shims; `shared/audit_ledger.py` had ZERO importers (verified by sweep incl. path-string forms) — DELETED | +| 11 | old implementation removed or reference-only | `shared/audit_ledger.py` DELETED (this commit, after the proven wall + zero-importer sweep); six src/rune modules remain as shims (the API contract); original Python lives in git history. Coverage registry remapped: `audit_ledger` removed from SHARED_D; gate exit 0 post-deletion | +| 12 | supersession ledger | THIS FILE | + +Deletion discipline (language-governance rule ⑥): `shared/audit_ledger.py` removed only after (a) the deployed shim proved live against the real ledger, (b) the shadow wall was green, (c) a zero-importer sweep across src/ shared/ scripts/ tests/. The remaining five src/rune originals were rewritten AS shims in their own commits (git history holds each original); they are the API contract and stay. + +## Vintage-merge disclosures + +- **SYMMETRY GUARD preserved:** the deployed `claim_audit` vintage carried `research_lens.symmetry_check` integration (2026-08-15) absent from src/rune — preserved in both shims (feature-preserving merge; load-bearing behavior). +- **DEAD CONSUMER REPAIRED:** `rune_tui.py:3752` `claim_audit.scan_text()` had NO implementation in either vintage (silent try/except death) — now implemented, first-class, live-verified. +- **Sequence break (port 1):** the audit_ledger shim was deployed BEFORE its canary pair existed; both canaries subsequently written and green — disclosed on #2449 comment 33817, not retrofitted. + +## Consumer map at supersession + +- `~/.config/opencode/scripts/rune_tui.py` — claim_audit (report L2468, scan_text L3752), behaviour (L3152), behavior (L83). +- `src/rune/rune_operator.py` — evidence_probe (L581/603), VerificationResult dataclass (L323). +- `src/rune/contracts.py` — dynamic `__import__` surfaces (verification_layer, claim_audit). +- `src/rune/testing_mode.py:19` / `src/rune/iteration.py:23` — behaviour_ledger constants + class. +- `scripts/research_stage_executor.py:312` — evidence_probe orchestration. +- `~/.config/opencode/scripts/curator.py` / `canary_full_system.py` — audit_ledger (deployed shim). +- `src/rune/dependency_tracker.py` — map-strings only (NOT an import). + +## Live-data rules (contract §3) — honored in the shims + +- Live promotion ledger: drift key verified 23-intact through the shim + FFI; explicit `AUDIT_LEDGER_KEY` override honored. +- Live behavior.db: read/written through the deployed shim at `~/.config/opencode/memory/behavior.db`; parity scenarios ran on temp dirs only. +- Live claim_audit journal (1127 rows): read-only aggregate through the Rust core; writes stay Python. \ No newline at end of file diff --git a/scripts/structural_coverage_audit.py b/scripts/structural_coverage_audit.py index df742dcf..029b1018 100644 --- a/scripts/structural_coverage_audit.py +++ b/scripts/structural_coverage_audit.py @@ -166,7 +166,6 @@ SCRIPTS_C = {"fake_worker", "tui_startup_canary"} # shared/ — harness-facing layer; mirrors of canonical src modules = D SHARED_D = { - "audit_ledger": "src/rune/audit_ledger.py", "dependency_tracker": "src/rune/dependency_tracker.py", "drift_cache": "src/rune/drift_cache.py", "drift_compression": "src/rune/drift_compression.py", diff --git a/shared/audit_ledger.py b/shared/audit_ledger.py deleted file mode 100644 index 6d202dd0..00000000 --- a/shared/audit_ledger.py +++ /dev/null @@ -1,210 +0,0 @@ -#!/usr/bin/env python3 -"""audit_ledger.py — HMAC-chained tamper-evident promotion ledger. - -Every promotion decision is recorded in a hash-chained ledger. -Tampering with any entry breaks the chain. - -Research basis: -- PoEM (arXiv:2608.16032): HMAC-chained tamper-evident execution ledger -- Autogenesis (arXiv:2604.15034): versioned interfaces, auditable lineage - -Run under ~/.venvs/memory/bin/python3. -""" - -import hashlib -import hmac -import json -import os -import sys -from datetime import datetime, timezone -from typing import Dict, List, Optional - -LEDGER_FILE = os.path.expanduser("~/.config/opencode/memory/promotion_ledger.jsonl") -SECRET_KEY = os.environ.get("AUDIT_LEDGER_KEY", "rune-drift-integrity-2024") - - -def _compute_hash(prev_hash: str, entry: dict) -> str: - """Compute HMAC chain hash for a ledger entry.""" - payload = json.dumps(entry, sort_keys=True).encode() - msg = (prev_hash + payload.decode()).encode() - return hmac.new(SECRET_KEY.encode(), msg, hashlib.sha256).hexdigest() - - -def append_entry(entry_type: str, entry_id: int, details: dict) -> dict: - """Append an entry to the promotion ledger. - - Returns the ledger entry with chain hash. - """ - # Read last hash - prev_hash = "genesis" - if os.path.exists(LEDGER_FILE): - with open(LEDGER_FILE, "r") as f: - lines = f.readlines() - if lines: - try: - last = json.loads(lines[-1]) - prev_hash = last.get("hash", "genesis") - except json.JSONDecodeError: - prev_hash = "genesis" - - entry = { - "type": entry_type, - "entry_id": entry_id, - "details": details, - "timestamp": datetime.now(timezone.utc).isoformat(), - "prev_hash": prev_hash, - } - - # Compute chain hash - entry["hash"] = _compute_hash(prev_hash, entry) - - # Append to ledger - os.makedirs(os.path.dirname(LEDGER_FILE), exist_ok=True) - with open(LEDGER_FILE, "a") as f: - f.write(json.dumps(entry) + "\n") - - return entry - - -def verify_chain() -> dict: - """Verify the integrity of the entire ledger chain. - - Returns dict with verification result. - """ - if not os.path.exists(LEDGER_FILE): - return {"valid": True, "entries": 0, "reason": "empty ledger"} - - with open(LEDGER_FILE, "r") as f: - lines = f.readlines() - - prev_hash = "genesis" - for i, line in enumerate(lines): - try: - entry = json.loads(line) - except json.JSONDecodeError: - return {"valid": False, "entries": i, "reason": f"invalid JSON at line {i+1}"} - - # Verify prev_hash chain - if entry.get("prev_hash") != prev_hash: - return { - "valid": False, - "entries": i, - "reason": f"chain broken at line {i+1}: expected prev_hash={prev_hash}, got {entry.get('prev_hash')}" - } - - # Verify HMAC - stored_hash = entry.get("hash") - computed = _compute_hash(prev_hash, {k: v for k, v in entry.items() if k != "hash"}) - if stored_hash != computed: - return { - "valid": False, - "entries": i, - "reason": f"HMAC mismatch at line {i+1}: stored={stored_hash[:16]}..., computed={computed[:16]}..." - } - - prev_hash = stored_hash - - return {"valid": True, "entries": len(lines), "reason": "chain intact"} - - -def get_entries(limit: int = 50) -> List[dict]: - """Get recent ledger entries.""" - if not os.path.exists(LEDGER_FILE): - return [] - - with open(LEDGER_FILE, "r") as f: - lines = f.readlines() - - entries = [] - for line in lines[-limit:]: - try: - entries.append(json.loads(line)) - except json.JSONDecodeError: - continue - return entries - - -def log_promotion(entry_id: int, old_status: str, new_status: str, wash_count: int, reason: str) -> dict: - """Log a promotion decision.""" - return append_entry("promotion", entry_id, { - "old_status": old_status, - "new_status": new_status, - "wash_count": wash_count, - "reason": reason, - }) - - -def log_gate_result(entry_id: int, verdict: str, critics: dict) -> dict: - """Log a pre-commit gate result.""" - return append_entry("gate", entry_id, { - "verdict": verdict, - "critics": critics, - }) - - -def log_drift_alert(alert_type: str, message: str, severity: str) -> dict: - """Log a drift alert.""" - return append_entry("alert", 0, { - "alert_type": alert_type, - "message": message, - "severity": severity, - }) - - -# ═══════════════════════════════════════════════════════════════════════════════ -# CLI -# ═══════════════════════════════════════════════════════════════════════════════ - -def main(): - import argparse - parser = argparse.ArgumentParser(description="Promotion audit ledger") - sub = parser.add_subparsers(dest="cmd") - - # verify - sub.add_parser("verify", help="Verify ledger chain integrity") - - # list [--limit N] - p_list = sub.add_parser("list", help="Show recent ledger entries") - p_list.add_argument("--limit", type=int, default=20) - - # log-promotion - p_log = sub.add_parser("log-promotion", help="Log a promotion") - p_log.add_argument("entry_id", type=int) - p_log.add_argument("old_status") - p_log.add_argument("new_status") - p_log.add_argument("wash_count", type=int) - p_log.add_argument("reason") - - args = parser.parse_args() - - if not args.cmd: - parser.print_help() - sys.exit(1) - - if args.cmd == "verify": - result = verify_chain() - print(f"Chain valid: {result['valid']}") - print(f"Entries: {result['entries']}") - print(f"Reason: {result['reason']}") - - elif args.cmd == "list": - entries = get_entries(args.limit) - if entries: - print(f"Recent ledger entries ({len(entries)}):") - for e in entries: - print(f" [{e.get('type')}] entry_id={e.get('entry_id')} @ {e.get('timestamp', '')[:19]}") - details = e.get('details', {}) - if e.get('type') == 'promotion': - print(f" {details.get('old_status')} → {details.get('new_status')} (wash={details.get('wash_count')})") - elif e.get('type') == 'gate': - print(f" verdict={details.get('verdict')}") - else: - print("No ledger entries.") - - elif args.cmd == "log-promotion": - entry = log_promotion(args.entry_id, args.old_status, args.new_status, args.wash_count, args.reason) - print(f"Logged: {entry['hash'][:16]}...") - - -if __name__ == "__main__": - main()