P0-07: durable inference traces (closes #23) #146

Merged
conductor merged 6 commits from p0-07-inference-trace into master 2026-09-05 19:18:32 +01:00
Owner

Plain summary

PR for #23 (P0-07, the last P0 of the course-correction epic): durable per-inference traces. Every operator run now records ONE trace answering both "why did this call cost this much?" and "what exactly did the local model see?" — provider/model, discovered context window, budget, estimated + served prompt tokens, output tokens, model time, wall latency, per-block contributions, dropped blocks, tools and memories exposed, task linkage, cost basis.

Technical details

  • Spec gate: PASS 14/15 (general-181; the 15th was a process catch — deviation tail now recorded). Quality gate: NOT approved → both Importants discharged in a720600: I1 rotation chain (guard-on-source + full range; .2/.3 now actually created — empirically verified) and I2 run-level delta-folding accumulator keyed by transport id (escalated runs report total spend across tier attempts; discharge test with 4-tier refusal escalation proves trace == per-transport reality).
  • Tests: suite 501 → 513 passed + 84 subtests (11 P0-07 frozen + 1 I2 discharge; E2E deselected in CI, runs live locally). Canary P0-07 INFERENCE TRACE CANARY PASSED registered; henxels exit 0.
  • Commits: 2a00174 → 785c033 → 8ff9ee1 → a720600. Follow-up: #145 (approval-resume, compaction, side-path tracing).
## Plain summary PR for #23 (P0-07, the last P0 of the course-correction epic): durable per-inference traces. Every operator run now records ONE trace answering both "why did this call cost this much?" and "what exactly did the local model see?" — provider/model, discovered context window, budget, estimated + served prompt tokens, output tokens, model time, wall latency, per-block contributions, dropped blocks, tools and memories exposed, task linkage, cost basis. ## Technical details - **Spec gate**: PASS 14/15 (general-181; the 15th was a process catch — deviation tail now recorded). **Quality gate**: NOT approved → both Importants discharged in `a720600`: I1 rotation chain (guard-on-source + full range; .2/.3 now actually created — empirically verified) and I2 run-level delta-folding accumulator keyed by transport id (escalated runs report total spend across tier attempts; discharge test with 4-tier refusal escalation proves trace == per-transport reality). - **Tests**: suite 501 → **513 passed + 84 subtests** (11 P0-07 frozen + 1 I2 discharge; E2E deselected in CI, runs live locally). Canary `P0-07 INFERENCE TRACE CANARY PASSED` registered; henxels exit 0. - **Commits**: 2a00174 → 785c033 → 8ff9ee1 → a720600. Follow-up: #145 (approval-resume, compaction, side-path tracing).
conductor added 4 commits 2026-09-05 18:19:08 +01:00
- OllamaChatTransport: additive usage accumulators (usage_calls,
  usage_prompt_tokens, usage_output_tokens, usage_total_duration_ns)
  summed from the /api/chat response envelope; last_prompt_eval_count
  behavior unchanged (calibration pin). provider = 'ollama' class attr.
- new rune.inference_trace: frozen InferenceTrace dataclass (schema v1),
  record_trace JSONL store under <rune_home>/state/inference_traces.jsonl
  with 10MB x 3 rotation, price_for (empty map -> None / local-unpriced).
- RuneOperator._execute_on_model: one trace per RUN on the final response
  (run_started covers the whole tier-escalation attempt), task_id +
  channel from TaskState, built FROM the broker trace + transport
  counters — never re-measures. Degrade-not-crash: trace failure records
  a tier=2 'inference_trace' failure and never fails the turn.
- canary_p0_07 (registered in ci.yml same commit): scripted trace ->
  record -> read back -> every issue-listed field asserted; rotation
  probe; no live model.
- docs/inference-trace.md: schema v1, field sources, cost basis,
  cloud/local comparison, consumers #25/#28/#29.
- v1_authority_map: inference_trace entry under context_construction.

Tests: 11 new frozen tests (tests/test_p0_07_inference_trace.py),
red-first; full suite 513 passed + 84 subtests (baseline 501).
fix(p0-07): quality discharges — full 3-deep rotation chain, run-level usage sums across tier escalations (#23)
Auto-label / priority-gate (pull_request) Skipped
Auto-label / auto-label (pull_request) Successful in 1s
CI Pipeline / lint (pull_request) Failing after 1m21s
CI Pipeline / unit-tests (pull_request) Successful in 2m11s
CI Pipeline / canary (pull_request) Successful in 1m31s
CI Pipeline / best-practices (pull_request) Successful in 5s
CI Pipeline / documentation (pull_request) Successful in 5s
CI Pipeline / type-check (pull_request) Failing after 2m23s
CI Pipeline / gitea-offline (pull_request) Successful in 39s
CI Pipeline / required-checks (pull_request) Skipped
CI Pipeline / lint (push) Successful in 1m56s
CI Pipeline / unit-tests (push) Successful in 2m6s
CI Pipeline / canary (push) Successful in 1m32s
CI Pipeline / best-practices (push) Successful in 5s
CI Pipeline / documentation (push) Successful in 5s
CI Pipeline / type-check (push) Failing after 2m5s
CI Pipeline / gitea-offline (push) Successful in 42s
CI Pipeline / required-checks (push) Skipped
a720600465
- I1: rotation guarded on the source (newer) with the full BACKUPS range —
  .2/.3 now actually created (empirically verified with a tiny-limit probe).
- I2: run-level delta-folding accumulator keyed by transport id — escalated
  runs report total spend across all tier attempts instead of the final
  attempt's counters. Discharge test with 4-tier refusal escalation.
- I3: untraced-path census (approval-resume, compaction, 13 raw-urllib
  sites) recorded as follow-up; primary operator path fully traced.
gitea-actions Bot added the growth label 2026-09-05 18:19:14 +01:00
conductor added 1 commit 2026-09-05 18:22:03 +01:00
style(p0-07): ruff-format test file (CI format gate on added files)
CI Pipeline / lint (push) Successful in 1m49s
CI Pipeline / unit-tests (push) Successful in 2m11s
CI Pipeline / canary (push) Successful in 1m35s
CI Pipeline / best-practices (push) Successful in 4s
CI Pipeline / documentation (push) Successful in 5s
CI Pipeline / type-check (push) Failing after 2m5s
CI Pipeline / gitea-offline (push) Successful in 38s
CI Pipeline / required-checks (push) Skipped
CI Pipeline / lint (pull_request) Successful in 1m52s
CI Pipeline / unit-tests (pull_request) Successful in 2m10s
CI Pipeline / canary (pull_request) Successful in 1m30s
CI Pipeline / best-practices (pull_request) Successful in 4s
CI Pipeline / documentation (pull_request) Successful in 5s
CI Pipeline / type-check (pull_request) Failing after 2m12s
CI Pipeline / gitea-offline (pull_request) Successful in 40s
CI Pipeline / required-checks (pull_request) Successful in 0s
39b395e543
conductor added 1 commit 2026-09-05 19:00:47 +01:00
fix(p0-07): type-assert canary trace fields — clears 18 pyright Any-flow errors
CI Pipeline / lint (push) Successful in 1m53s
CI Pipeline / unit-tests (push) Successful in 2m4s
CI Pipeline / canary (push) Successful in 1m38s
CI Pipeline / best-practices (push) Successful in 5s
CI Pipeline / documentation (push) Successful in 5s
CI Pipeline / type-check (push) Successful in 2m9s
CI Pipeline / gitea-offline (push) Successful in 40s
CI Pipeline / required-checks (push) Skipped
CI Pipeline / lint (pull_request) Successful in 1m50s
CI Pipeline / unit-tests (pull_request) Successful in 2m9s
CI Pipeline / canary (pull_request) Successful in 1m30s
CI Pipeline / best-practices (pull_request) Successful in 4s
CI Pipeline / documentation (pull_request) Successful in 5s
CI Pipeline / type-check (pull_request) Successful in 2m9s
CI Pipeline / gitea-offline (pull_request) Successful in 38s
CI Pipeline / required-checks (pull_request) Successful in 0s
Auto-version and Release / release (pull_request) Failing after 4s
bf7f4a9bca
The canary passed broker_trace dict values (Any) straight into the typed
InferenceTrace constructor. int()/float() asserts make the flow explicit;
pyright: 18 errors -> 0 on this file.
conductor merged commit 0a0c95266a into master 2026-09-05 19:18:32 +01:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: conductor/rune#146