Nick
f9f83e0c35
- #129: dimensional review rubric folded into the dispatch template (correctness/design/test-quality/observability/security, forced-choice verdicts; missing dimension = incomplete review) - #130: calibration harness (scripts/reviewer_calibration.py) — 5 known defects from the ledger + 1 clean control, 6/6 caught (100%, CALIBRATED). The pagination pattern needed widening for dotted receivers (self._rows[:50]) — the harness caught its own blind spot. - #131: local-model first-pass reviewer — Gitea workflow (local-review.yml) + agent script (local_firstpass_review.py, Ollama + review-lessons + #129 rubric). Fail-open by design: Ollama unreachable → skip with a note; findings are advisory PR comments, never gates.
4.9 KiB
Reviewer Dispatch Template — includes review-lessons selection
This template is for controller use when dispatching spec-compliance or code-quality review agents. It incorporates the review-lessons file with tag-based selection.
Controller pre-selection step
Before dispatching, the controller:
- Identifies the files/subsystems in the diff (from the commit's name-status).
- Reads
docs/review-lessons.yamltag index. - Selects entries whose tags match the diff scope.
- Includes ONLY the selected entries + the tag index in the dispatch prompt (so the reviewer stays focused; the file can grow to 500+ entries without relevance decay).
Tag-to-scope mapping (for controller selection)
| Diff touches | Relevant tags |
|---|---|
| *_store.py, *_registry.py, persistence | corruption-handling, concurrency, transactional-integrity, type-hygiene, identity-collision |
| *_test.py, tests/ | test-vacuity, count-accuracy |
| plans/*.md | count-accuracy, plan-drift, api-contract, dead-import |
| lifecycle.py, *_engine.py, state machines | state-machine, transactional-integrity |
| *_health.py, observability, logging | error-masking, emit-self-failure |
| router.py, delegation, planner | identity-collision, api-contract, pagination-truncation |
Any file with except blocks |
error-masking, corruption-handling |
| Any function returning objects | aliasing |
| Any API that lists/filters | empty-input-edge, pagination-truncation |
Standard reviewer preamble (include in every review dispatch)
Read docs/review-lessons.yaml (tag index + selected entries below) before
reviewing. These are defect classes this project's review gates have caught
— check for them proactively. If you find a defect class NOT in the file,
include a structured `## New lessons` section in your findings report with:
id, class, severity, tags, detection, example (this commit), fix.
Auto-append convention
At issue close-time, the controller:
- Extracts any
## New lessonssections from the review findings. - Validates the entries (required fields present, example commit exists).
- Appends to
docs/review-lessons.yaml. - Commits with message:
docs(lessons): add [N] entries from [issue] review
The file grows as a byproduct of reviewing — no separate maintenance task.
Cross-model decorrelation (#128)
The reviewer model MUST differ from the implementer model. Self-preference bias (Panickssery et al., NeurIPS 2024; arXiv 2604.22891) shows 17/20 models systematically favor their own outputs. The mitigation: cross-model judging.
Rules:
- Implementation dispatch:
model: "Verboo Code/glm-5.3-flash"(or the current implementation model) - Review dispatches (spec + quality): use a DIFFERENT model family — e.g. a local Qwen via Ollama, or a different provider
- The reviewer model doesn't need to be BETTER, just DIFFERENT
- Log the model used per review in the findings (for calibration tracking)
Adversarial procedure (mandatory for P0/P1 — docs/adversarial-review.md)
Every spec-gate dispatch on a P0/P1 change includes, verbatim:
Mandatory question: "What incorrect implementation could still pass these tests?" Answer it BEFORE reading the implementation diff. Enumerate the wrong implementations the frozen tests would not catch. A review that cannot answer is incomplete and will bounce.
And the role-separation rules: the black-box test author (fresh context) has already written the spec tests; the reviewer checks the implementation against tests it did not write; vacuity spot-checks (one manual mutant per load-bearing branch) are BLOCKER-grade findings when tests survive them.
Dimensional review rubric (#129 — mandatory for P0/P1)
Holistic judgment ("is this good?") is replaced by per-dimension forced-choice scoring. Every review dispatch MUST score each dimension explicitly — a review missing any dimension is INCOMPLETE and bounces.
| Dimension | What to score | Verdicts |
|---|---|---|
| Correctness | Does the code do what the issue/spec requires? Verify call sites, not claims. | PASS / CONCERN / FAIL |
| Design | Does the shape fit the codebase (parity laws, authority maps, no second authorities)? | PASS / CONCERN / FAIL |
| Test quality | Do the tests prove the behavior (real-path, oracle-derived, would fail without the fix)? | PASS / CONCERN / FAIL |
| Observability | Are failures visible (logged, journaled, gated) rather than silent? | PASS / CONCERN / FAIL |
| Security | Approval boundaries, secret handling, injection surfaces — anything that widens trust? | PASS / CONCERN / FAIL |
Example (the required output shape):
Correctness: PASS (36 call sites verified).
Design: PASS with concern (registry singleton).
Test quality: CONCERN (no real-path test).
Observability: PASS.
Security: PASS.
Five explicit verdicts the controller acts on dimensionally — a CONCERN or FAIL in any dimension blocks the gate until discharged or explicitly accepted with a reason.