Nick
f9f83e0c35
local-review.yml (push) Invalid workflow file
Auto-version and Release / release (push) Failing after 9s
CI Pipeline / lint (push) Failing after 29s
CI Pipeline / go-surface (push) Failing after 18s
CI Pipeline / rust-surface (push) Successful in 4m15s
CI Pipeline / unit-tests (push) Successful in 5m29s
CI Pipeline / best-practices (push) Successful in 7s
CI Pipeline / documentation (push) Successful in 8s
CI Pipeline / canary (push) Successful in 1m6s
CI Pipeline / type-check (push) Failing after 35s
CI Pipeline / gitea-offline (push) Successful in 18s
deploy / deploy (push) Failing after 15m11s
CI Pipeline / required-checks (push) Failing after 1s
CI Pipeline / CI Pipeline / notify-event-bridge (push) Successful in 10s
- #129: dimensional review rubric folded into the dispatch template (correctness/design/test-quality/observability/security, forced-choice verdicts; missing dimension = incomplete review) - #130: calibration harness (scripts/reviewer_calibration.py) — 5 known defects from the ledger + 1 clean control, 6/6 caught (100%, CALIBRATED). The pagination pattern needed widening for dotted receivers (self._rows[:50]) — the harness caught its own blind spot. - #131: local-model first-pass reviewer — Gitea workflow (local-review.yml) + agent script (local_firstpass_review.py, Ollama + review-lessons + #129 rubric). Fail-open by design: Ollama unreachable → skip with a note; findings are advisory PR comments, never gates.
104 lines
4.9 KiB
Markdown
104 lines
4.9 KiB
Markdown
# Reviewer Dispatch Template — includes review-lessons selection
|
|
|
|
This template is for controller use when dispatching spec-compliance or
|
|
code-quality review agents. It incorporates the review-lessons file with
|
|
tag-based selection.
|
|
|
|
## Controller pre-selection step
|
|
|
|
Before dispatching, the controller:
|
|
1. Identifies the files/subsystems in the diff (from the commit's name-status).
|
|
2. Reads `docs/review-lessons.yaml` tag index.
|
|
3. Selects entries whose tags match the diff scope.
|
|
4. Includes ONLY the selected entries + the tag index in the dispatch prompt
|
|
(so the reviewer stays focused; the file can grow to 500+ entries without
|
|
relevance decay).
|
|
|
|
## Tag-to-scope mapping (for controller selection)
|
|
|
|
| Diff touches | Relevant tags |
|
|
|---|---|
|
|
| *_store.py, *_registry.py, persistence | corruption-handling, concurrency, transactional-integrity, type-hygiene, identity-collision |
|
|
| *_test.py, tests/ | test-vacuity, count-accuracy |
|
|
| plans/*.md | count-accuracy, plan-drift, api-contract, dead-import |
|
|
| lifecycle.py, *_engine.py, state machines | state-machine, transactional-integrity |
|
|
| *_health.py, observability, logging | error-masking, emit-self-failure |
|
|
| router.py, delegation, planner | identity-collision, api-contract, pagination-truncation |
|
|
| Any file with `except` blocks | error-masking, corruption-handling |
|
|
| Any function returning objects | aliasing |
|
|
| Any API that lists/filters | empty-input-edge, pagination-truncation |
|
|
|
|
## Standard reviewer preamble (include in every review dispatch)
|
|
|
|
```
|
|
Read docs/review-lessons.yaml (tag index + selected entries below) before
|
|
reviewing. These are defect classes this project's review gates have caught
|
|
— check for them proactively. If you find a defect class NOT in the file,
|
|
include a structured `## New lessons` section in your findings report with:
|
|
id, class, severity, tags, detection, example (this commit), fix.
|
|
```
|
|
|
|
## Auto-append convention
|
|
|
|
At issue close-time, the controller:
|
|
1. Extracts any `## New lessons` sections from the review findings.
|
|
2. Validates the entries (required fields present, example commit exists).
|
|
3. Appends to `docs/review-lessons.yaml`.
|
|
4. Commits with message: `docs(lessons): add [N] entries from [issue] review`
|
|
|
|
The file grows as a byproduct of reviewing — no separate maintenance task.
|
|
|
|
## Cross-model decorrelation (#128)
|
|
|
|
The reviewer model MUST differ from the implementer model. Self-preference
|
|
bias (Panickssery et al., NeurIPS 2024; arXiv 2604.22891) shows 17/20 models
|
|
systematically favor their own outputs. The mitigation: cross-model judging.
|
|
|
|
Rules:
|
|
- Implementation dispatch: `model: "Verboo Code/glm-5.3-flash"` (or the current
|
|
implementation model)
|
|
- Review dispatches (spec + quality): use a DIFFERENT model family — e.g.
|
|
a local Qwen via Ollama, or a different provider
|
|
- The reviewer model doesn't need to be BETTER, just DIFFERENT
|
|
- Log the model used per review in the findings (for calibration tracking)
|
|
|
|
## Adversarial procedure (mandatory for P0/P1 — docs/adversarial-review.md)
|
|
|
|
Every spec-gate dispatch on a P0/P1 change includes, verbatim:
|
|
|
|
> **Mandatory question:** "What incorrect implementation could still pass
|
|
> these tests?" Answer it BEFORE reading the implementation diff. Enumerate
|
|
> the wrong implementations the frozen tests would not catch. A review that
|
|
> cannot answer is incomplete and will bounce.
|
|
|
|
And the role-separation rules: the black-box test author (fresh context) has
|
|
already written the spec tests; the reviewer checks the implementation
|
|
against tests it did not write; vacuity spot-checks (one manual mutant per
|
|
load-bearing branch) are BLOCKER-grade findings when tests survive them.
|
|
|
|
## Dimensional review rubric (#129 — mandatory for P0/P1)
|
|
|
|
Holistic judgment ("is this good?") is replaced by per-dimension
|
|
forced-choice scoring. Every review dispatch MUST score each dimension
|
|
explicitly — a review missing any dimension is INCOMPLETE and bounces.
|
|
|
|
| Dimension | What to score | Verdicts |
|
|
|---|---|---|
|
|
| Correctness | Does the code do what the issue/spec requires? Verify call sites, not claims. | PASS / CONCERN / FAIL |
|
|
| Design | Does the shape fit the codebase (parity laws, authority maps, no second authorities)? | PASS / CONCERN / FAIL |
|
|
| Test quality | Do the tests prove the behavior (real-path, oracle-derived, would fail without the fix)? | PASS / CONCERN / FAIL |
|
|
| Observability | Are failures visible (logged, journaled, gated) rather than silent? | PASS / CONCERN / FAIL |
|
|
| Security | Approval boundaries, secret handling, injection surfaces — anything that widens trust? | PASS / CONCERN / FAIL |
|
|
|
|
Example (the required output shape):
|
|
```
|
|
Correctness: PASS (36 call sites verified).
|
|
Design: PASS with concern (registry singleton).
|
|
Test quality: CONCERN (no real-path test).
|
|
Observability: PASS.
|
|
Security: PASS.
|
|
```
|
|
Five explicit verdicts the controller acts on dimensionally — a CONCERN
|
|
or FAIL in any dimension blocks the gate until discharged or explicitly
|
|
accepted with a reason.
|