docs(RP13): throughput/quality metrics for real runs #161 + #163 #283

Merged
conductor merged 3 commits from rp13-metrics into master 2026-09-09 14:01:58 +01:00
Owner

RP13: throughput/quality comparison (closes #276)

Report: docs/RP13_THROUGHPUT_REPORT.json on branch rp13-metrics.

Measured (two real runs, post-cleanup canonical path):

Metric #161 #163
Stages executed 7 (+3 CI-contract fix loops) 10 (incl. INNOVATION + REFINE round)
MiMo tool calls 101 103
External queries 4 5
Sources discovered / read w/ receipts 16 / 4 45 / 10
Contradictions / limitations captured 4 / 8 4 / 8
Adversarial attacks → survivors → rejected 16 → 6 → 1 13 → 4 → 1
Mitigations / kill-tests carried to synthesis 10 / 7 7 / 6
Refinement loops 0 1 (REFINE_SYNTHESIS, targeted)
Implementation rework 0 (merged first try) n/a (design-stage)
Wall-clock research chain ~75 min (git window 00:16→01:31, incl. live CI-defect fixes) per-stage elapsed recorded (OUTWARD→QUALIFY ≈ 28 min cognition)

Old-process baseline (recorded RSC0 evidence): foreground slept 300s×N while child MiMo was productive; receipts were model-claimed; snippets counted as evidence; no durable state across restarts (issues #245/#254/#255 each restarted from scratch).

Quality outcomes (what the pipeline PREVENTED):

  • #161: MemoryAdapter duplicate-authority rejected before coding; silent non-hybrid benchmark validation prevented (FT1); cache-pollution hazard neutralized (A4)
  • #163: external graph DB adoption blocked (3 CONFLICTS_WITH_RUNE); false bi-temporal premise corrected from code before synthesis; INN-1 falsified and corrected in synthesis v2

Cognition vs orchestration: cognition = recorded stage elapsed (101/103 tool calls; QUALIFY 310s on #161; APPLICABILITY 300s, INNOVATION 422s, SYNTHESIS 677s on #163). Orchestration/wait (runner queue, CI) excluded from pipeline cost.

KEEP / TUNE / REMOVE:

  • KEEP: unified OUTWARD w/ receipts (evidence chain held under adversarial audit), adversarial disposition contract (caught false premises on both runs), narrowed QUALIFY (caught 3 real gaps without reopening research), placement gate
  • TUNE: per-stage elapsed persistence (reconciliation zeroes earlier stages — record at advance() time); OUTWARD file-channel should be the PRIMARY documented channel (stdout parse is the fallback); prose-only stdout still costs one gate-retry
  • REMOVE: nothing further — RP11 retirement complete; no remaining competing authority

Honest data note: elapsed_s zeroed for earlier stages on #161 due to reconciliation overwrites — timing persistence at advance() is the fix (TUNE).

## RP13: throughput/quality comparison (closes #276) Report: `docs/RP13_THROUGHPUT_REPORT.json` on branch `rp13-metrics`. **Measured (two real runs, post-cleanup canonical path):** | Metric | #161 | #163 | |---|---|---| | Stages executed | 7 (+3 CI-contract fix loops) | 10 (incl. INNOVATION + REFINE round) | | MiMo tool calls | 101 | 103 | | External queries | 4 | 5 | | Sources discovered / read w/ receipts | 16 / 4 | 45 / 10 | | Contradictions / limitations captured | 4 / 8 | 4 / 8 | | Adversarial attacks → survivors → rejected | 16 → 6 → 1 | 13 → 4 → 1 | | Mitigations / kill-tests carried to synthesis | 10 / 7 | 7 / 6 | | Refinement loops | 0 | 1 (REFINE_SYNTHESIS, targeted) | | Implementation rework | 0 (merged first try) | n/a (design-stage) | | Wall-clock research chain | ~75 min (git window 00:16→01:31, incl. live CI-defect fixes) | per-stage elapsed recorded (OUTWARD→QUALIFY ≈ 28 min cognition) | **Old-process baseline** (recorded RSC0 evidence): foreground slept 300s×N while child MiMo was productive; receipts were model-claimed; snippets counted as evidence; no durable state across restarts (issues #245/#254/#255 each restarted from scratch). **Quality outcomes (what the pipeline PREVENTED):** - #161: MemoryAdapter duplicate-authority rejected before coding; silent non-hybrid benchmark validation prevented (FT1); cache-pollution hazard neutralized (A4) - #163: external graph DB adoption blocked (3 CONFLICTS_WITH_RUNE); false bi-temporal premise corrected from code before synthesis; INN-1 falsified and corrected in synthesis v2 **Cognition vs orchestration:** cognition = recorded stage elapsed (101/103 tool calls; QUALIFY 310s on #161; APPLICABILITY 300s, INNOVATION 422s, SYNTHESIS 677s on #163). Orchestration/wait (runner queue, CI) excluded from pipeline cost. **KEEP / TUNE / REMOVE:** - KEEP: unified OUTWARD w/ receipts (evidence chain held under adversarial audit), adversarial disposition contract (caught false premises on both runs), narrowed QUALIFY (caught 3 real gaps without reopening research), placement gate - TUNE: per-stage elapsed persistence (reconciliation zeroes earlier stages — record at advance() time); OUTWARD file-channel should be the PRIMARY documented channel (stdout parse is the fallback); prose-only stdout still costs one gate-retry - REMOVE: nothing further — RP11 retirement complete; no remaining competing authority Honest data note: elapsed_s zeroed for earlier stages on #161 due to reconciliation overwrites — timing persistence at advance() is the fix (TUNE).
conductor added 1 commit 2026-09-09 13:53:25 +01:00
docs(RP13): throughput/quality metrics for #161 + #163 real runs (closes #276)
CI Pipeline / best-practices (push) Successful in 4s
CI Pipeline / documentation (push) Successful in 3s
CI Pipeline / canary (push) Successful in 31s
CI Pipeline / lint (push) Failing after 37s
Auto-label / priority-gate (pull_request) Skipped
CI Pipeline / type-check (push) Successful in 28s
Auto-label / auto-label (pull_request) Successful in 3s
CI Pipeline / gitea-offline (push) Successful in 15s
CI Pipeline / unit-tests (push) Successful in 54s
CI Pipeline / best-practices (pull_request) Successful in 3s
CI Pipeline / documentation (pull_request) Successful in 2s
CI Pipeline / canary (pull_request) Successful in 29s
CI Pipeline / lint (pull_request) Failing after 41s
CI Pipeline / type-check (pull_request) Successful in 28s
Spec gate (headless) / spec-gate (pull_request) Successful in 7s
CI Pipeline / required-checks (push) Successful in 1s
CI Pipeline / gitea-offline (pull_request) Successful in 15s
CI Pipeline / unit-tests (pull_request) Successful in 55s
CI Pipeline / required-checks (pull_request) Successful in 1s
d91f428844
Per-stage cognition metrics (elapsed where recorded, tool calls, queries,
sources read, failures), research quality (sources/contradictions/
limitations), adversarial outcomes (attacks/survivors/rejected/mitigations/
kill-tests/dispositions), old-process baseline (300s foreground sleeps,
model-claimed receipts, no read provenance), and quality outcomes
(architectural mistakes prevented per run, implementation ambiguity
reduction, rework loops).

Honest data note: stage elapsed_s only recorded where the last artifact
write set it (reconciliation zeroed earlier stages on #161) — per-stage
timing persistence is a TUNE recommendation, not a measurement blocker.
gitea-actions Bot added the bugcustodiangovernanceinfrastructurememory labels 2026-09-09 13:53:34 +01:00

🤖 Headless spec gate (non-blocking, #204 first slice)

Spec-gate agent produced no text output — check runner logs.

Deterministic floors still govern the merge

This comment is advisory. Required checks (lint, tests, canaries, type-check, coverage, dependency lock, impact gate) remain the merge authority per docs/adversarial-review.md proportionality.

🤖 **Headless spec gate (non-blocking, #204 first slice)** Spec-gate agent produced no text output — check runner logs. <details><summary>Deterministic floors still govern the merge</summary> This comment is advisory. Required checks (lint, tests, canaries, type-check, coverage, dependency lock, impact gate) remain the merge authority per docs/adversarial-review.md proportionality.</details>
conductor added 1 commit 2026-09-09 13:58:12 +01:00
fix(RP13): metrics report as YAML — top-level docs henxel requires md/yaml
CI Pipeline / best-practices (pull_request) Successful in 3s
CI Pipeline / documentation (pull_request) Successful in 3s
CI Pipeline / lint (pull_request) Successful in 34s
CI Pipeline / canary (pull_request) Successful in 32s
CI Pipeline / gitea-offline (pull_request) Successful in 14s
Spec gate (headless) / spec-gate (pull_request) Successful in 7s
CI Pipeline / unit-tests (pull_request) Successful in 51s
CI Pipeline / type-check (pull_request) Successful in 28s
CI Pipeline / required-checks (pull_request) Successful in 1s
CI Pipeline / best-practices (push) Successful in 2s
CI Pipeline / documentation (push) Successful in 1s
CI Pipeline / lint (push) Failing after 36s
CI Pipeline / canary (push) Failing after 35s
CI Pipeline / gitea-offline (push) Successful in 17s
CI Pipeline / type-check (push) Successful in 33s
CI Pipeline / unit-tests (push) Failing after 1m5s
CI Pipeline / required-checks (push) Successful in 6s
bfbc153ec2

🤖 Headless spec gate (non-blocking, #204 first slice)

Spec-gate agent produced no text output — check runner logs.

Deterministic floors still govern the merge

This comment is advisory. Required checks (lint, tests, canaries, type-check, coverage, dependency lock, impact gate) remain the merge authority per docs/adversarial-review.md proportionality.

🤖 **Headless spec gate (non-blocking, #204 first slice)** Spec-gate agent produced no text output — check runner logs. <details><summary>Deterministic floors still govern the merge</summary> This comment is advisory. Required checks (lint, tests, canaries, type-check, coverage, dependency lock, impact gate) remain the merge authority per docs/adversarial-review.md proportionality.</details>
conductor added 1 commit 2026-09-09 14:00:08 +01:00
Merge branch 'master' into rp13-metrics
CI Pipeline / best-practices (push) Successful in 4s
CI Pipeline / documentation (push) Successful in 2s
CI Pipeline / canary (push) Successful in 33s
CI Pipeline / lint (push) Successful in 37s
CI Pipeline / type-check (push) Successful in 28s
CI Pipeline / gitea-offline (push) Successful in 14s
CI Pipeline / unit-tests (push) Successful in 58s
CI Pipeline / best-practices (pull_request) Successful in 3s
CI Pipeline / documentation (pull_request) Successful in 3s
CI Pipeline / lint (pull_request) Successful in 34s
CI Pipeline / canary (pull_request) Successful in 31s
Spec gate (headless) / spec-gate (pull_request) Successful in 5s
CI Pipeline / gitea-offline (pull_request) Successful in 14s
CI Pipeline / required-checks (push) Successful in 1s
CI Pipeline / unit-tests (pull_request) Successful in 51s
CI Pipeline / type-check (pull_request) Successful in 28s
CI Pipeline / required-checks (pull_request) Successful in 1s
Auto-version and Release / release (pull_request) Successful in 1m2s
6b2a3c0743

🤖 Headless spec gate (non-blocking, #204 first slice)

Spec-gate agent produced no text output — check runner logs.

Deterministic floors still govern the merge

This comment is advisory. Required checks (lint, tests, canaries, type-check, coverage, dependency lock, impact gate) remain the merge authority per docs/adversarial-review.md proportionality.

🤖 **Headless spec gate (non-blocking, #204 first slice)** Spec-gate agent produced no text output — check runner logs. <details><summary>Deterministic floors still govern the merge</summary> This comment is advisory. Required checks (lint, tests, canaries, type-check, coverage, dependency lock, impact gate) remain the merge authority per docs/adversarial-review.md proportionality.</details>
conductor merged commit bd920fb497 into master 2026-09-09 14:01:58 +01:00
Sign in to join this conversation.
No Reviewers
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: conductor/rune#283