Compare commits

...
Author SHA1 Message Date
Nick 4b54451444 fix(event-driven): resolve short SHAs to full 40-char before authority read — 12-char prefix silently failed _fetch_runs equality (962e351 root cause)
CI Pipeline / best-practices (pull_request) Successful in 10s
Auto-label / auto-label (pull_request) Successful in 4s
Auto-label / priority-gate (pull_request) Skipped
CI Pipeline / documentation (pull_request) Successful in 5s
CI Pipeline / canary (pull_request) Successful in 36s
CI Pipeline / gitea-offline (pull_request) Successful in 17s
CI Pipeline / type-check (pull_request) Successful in 37s
Spec gate (headless) / spec-gate (pull_request) Successful in 19s
CI Pipeline / lint (pull_request) Successful in 51s
CI Pipeline / unit-tests (pull_request) Successful in 4m1s
CI Pipeline / required-checks (pull_request) Successful in 2s
CI Pipeline / CI Pipeline / notify-event-bridge (pull_request) Successful in 4s
deploy / deploy (push) Skipped
CI Pipeline / best-practices (push) Successful in 3s
CI Pipeline / documentation (push) Successful in 2s
CI Pipeline / lint (push) Failing after 38s
CI Pipeline / gitea-offline (push) Successful in 13s
CI Pipeline / canary (push) Failing after 43s
CI Pipeline / type-check (push) Successful in 40s
CI Pipeline / unit-tests (push) Failing after 4m32s
CI Pipeline / required-checks (push) Failing after 10s
CI Pipeline / CI Pipeline / notify-event-bridge (push) Successful in 8s
2026-09-18 19:14:23 +01:00
Nick fd9e1bffb8 chore(deps): apply regenerated lock (multidict 6.9.0)
CI Pipeline / lint (pull_request) Successful in 41s
CI Pipeline / canary (pull_request) Successful in 43s
CI Pipeline / best-practices (pull_request) Successful in 6s
CI Pipeline / documentation (pull_request) Successful in 6s
CI Pipeline / gitea-offline (pull_request) Successful in 16s
CI Pipeline / type-check (pull_request) Successful in 36s
CI Pipeline / lint (push) Successful in 44s
CI Pipeline / best-practices (push) Successful in 6s
CI Pipeline / documentation (push) Successful in 5s
CI Pipeline / canary (push) Successful in 39s
CI Pipeline / gitea-offline (push) Successful in 18s
CI Pipeline / type-check (push) Successful in 32s
deploy / deploy (push) Skipped
CI Pipeline / unit-tests (pull_request) Successful in 3m54s
Spec gate (headless) / spec-gate (pull_request) Successful in 15s
CI Pipeline / required-checks (pull_request) Successful in 3s
CI Pipeline / CI Pipeline / notify-event-bridge (pull_request) Successful in 12s
CI Pipeline / unit-tests (push) Successful in 3m54s
CI Pipeline / required-checks (push) Successful in 3s
CI Pipeline / CI Pipeline / notify-event-bridge (push) Successful in 9s
Auto-version and Release / release (pull_request) Failing after 1m11s
2026-09-18 17:18:30 +01:00
Nick 631426763e chore(deps): re-lock requirements-ci.txt (upstream multidict 6.9.0) — dep-lock gate
CI Pipeline / best-practices (push) Successful in 11s
CI Pipeline / lint (push) Failing after 34s
CI Pipeline / documentation (push) Successful in 6s
CI Pipeline / canary (push) Successful in 39s
CI Pipeline / best-practices (pull_request) Successful in 8s
CI Pipeline / documentation (pull_request) Successful in 15s
CI Pipeline / canary (pull_request) Successful in 43s
CI Pipeline / gitea-offline (pull_request) Successful in 18s
CI Pipeline / type-check (pull_request) Successful in 31s
CI Pipeline / type-check (push) Successful in 30s
CI Pipeline / unit-tests (push) Successful in 4m4s
CI Pipeline / unit-tests (pull_request) Successful in 4m25s
CI Pipeline / gitea-offline (push) Successful in 22s
deploy / deploy (push) Skipped
Spec gate (headless) / spec-gate (pull_request) Successful in 23s
CI Pipeline / required-checks (push) Failing after 2s
CI Pipeline / lint (pull_request) Failing after 34s
CI Pipeline / required-checks (pull_request) Failing after 3s
CI Pipeline / CI Pipeline / notify-event-bridge (push) Successful in 10s
CI Pipeline / CI Pipeline / notify-event-bridge (pull_request) Successful in 9s
2026-09-18 17:17:52 +01:00
Nick 09cd18ad54 fix(event-driven): orphaned-lease release + durable-event recheck on re-arm + paginated runs fetch (all pages) + behavioral deadline tests
CI Pipeline / best-practices (push) Successful in 9s
CI Pipeline / documentation (push) Successful in 6s
CI Pipeline / lint (push) Failing after 33s
Auto-label / priority-gate (pull_request) Skipped
CI Pipeline / canary (push) Successful in 35s
CI Pipeline / type-check (push) Successful in 28s
CI Pipeline / gitea-offline (push) Successful in 18s
Auto-label / auto-label (pull_request) Successful in 6s
CI Pipeline / best-practices (pull_request) Successful in 10s
CI Pipeline / documentation (pull_request) Successful in 9s
CI Pipeline / canary (pull_request) Successful in 37s
CI Pipeline / gitea-offline (pull_request) Successful in 18s
CI Pipeline / type-check (pull_request) Successful in 33s
deploy / deploy (push) Skipped
CI Pipeline / lint (pull_request) Failing after 45s
CI Pipeline / unit-tests (push) Successful in 3m56s
CI Pipeline / required-checks (push) Failing after 1s
CI Pipeline / CI Pipeline / notify-event-bridge (push) Successful in 5s
CI Pipeline / unit-tests (pull_request) Successful in 4m0s
CI Pipeline / required-checks (pull_request) Failing after 1s
CI Pipeline / CI Pipeline / notify-event-bridge (pull_request) Successful in 5s
ROOT CAUSE (Defect A, 9db239f + 51416aa deadline_pending->success):
ci_event_wait exited at deadline without releasing the daemon lease.
The orphaned lease blocked the daemon from writing resume tokens to
the FIFO for re-registered waits — so completion events that WERE
persisted (10:43:01 for 9db239f, 12:01:49 for 51416aa) could never be
delivered to a later waiter. Additionally: internal re-arms were
timer-blind (no durable-event recheck), and _fetch_runs used a single
50-run page (scheduled merge-robot floods push older SHAs past it).

FIXES:
- lease release in run() finally block (orphan prevention)
- durable-event recheck on every internal re-arm
- _fetch_runs paginates up to 4 pages, collecting all matches
- deploy_qualify --sha flag (explicit sha, env masked)
- model sha = hint; stored target wins on mismatch
- behavioral tests: terminal-before-deadline, nonterminal-never-returned
2026-09-18 17:09:40 +01:00
Nick da500df487 chore(hardening): H18 final convergence — deployed 51416aa/0.85.6 (duplicate-run quirk disclosed)
CI Pipeline / best-practices (push) Successful in 11s
Auto-version and Release / release (push) Successful in 24s
CI Pipeline / documentation (push) Successful in 4s
CI Pipeline / canary (push) Successful in 37s
CI Pipeline / lint (push) Successful in 49s
CI Pipeline / gitea-offline (push) Successful in 16s
CI Pipeline / type-check (push) Successful in 37s
CI Pipeline / unit-tests (push) Successful in 3m24s
CI Pipeline / required-checks (push) Successful in 1s
CI Pipeline / CI Pipeline / notify-event-bridge (push) Successful in 6s
deploy / deploy (push) Successful in 3m12s
2026-09-18 13:25:21 +01:00
Nick 51416aa95e chore: retrigger CI (duplicate-run quirk: 9281 failure + 9282 success for same sha; upstream Gitea run duplication)
Auto-version and Release / release (push) Successful in 28s
CI Pipeline / canary (push) Successful in 39s
CI Pipeline / lint (push) Successful in 43s
CI Pipeline / type-check (push) Successful in 28s
CI Pipeline / gitea-offline (push) Successful in 16s
CI Pipeline / best-practices (push) Successful in 5s
CI Pipeline / documentation (push) Successful in 11s
CI Pipeline / unit-tests (push) Successful in 3m24s
CI Pipeline / required-checks (push) Successful in 1s
CI Pipeline / CI Pipeline / notify-event-bridge (push) Successful in 5s
deploy / deploy (push) Successful in 3m19s
2026-09-18 12:57:52 +01:00
4 changed files with 127 additions and 5 deletions

No files matched your search

+2 -2
View File
@@ -1,5 +1,5 @@
# This file was autogenerated by uv via the following command:
# uv pip compile pyproject.toml -o /tmp/regen2.txt --extra dev --extra memory --python-version 3.11
# uv pip compile pyproject.toml -o /tmp/req3.txt --extra dev --extra memory --python-version 3.11
aiohappyeyeballs==2.7.1
# via aiohttp
aiohttp==3.14.3
@@ -56,7 +56,7 @@ mdit-py-plugins==0.6.1
# via textual
mdurl==0.1.2
# via markdown-it-py
multidict==6.8.0
multidict==6.9.0
# via
# aiohttp
# yarl
+41
View File
@@ -328,6 +328,20 @@ def run(*, sha: str, deadline_secs: int,
# python-level run() callers (tool normalization flow) must export it.
if sha and not os.environ.get("DEPLOY_SHA"):
os.environ["DEPLOY_SHA"] = sha
# H-PATCH: resolve short SHAs to full 40-char — _fetch_runs and
# deploy_qualify match on the FULL sha; a 12-char prefix would
# silently fail equality and park the waiter until deadline
# (the 962e351 root cause).
if sha and len(sha) < 40:
try:
full = subprocess.run(
["git", "-C", str(_SCRIPT_DIR.parent), "rev-parse", sha],
capture_output=True, text=True, timeout=10)
if full.returncode == 0:
sha = full.stdout.strip()
os.environ["DEPLOY_SHA"] = sha
except (OSError, subprocess.TimeoutExpired):
pass
t0 = time.time()
# §16Y #10 race fix: the FIFO reader exists BEFORE registration, so a
# wake arriving in the registration window can always be delivered.
@@ -434,6 +448,22 @@ def run(*, sha: str, deadline_secs: int,
rearm_count += 1
rearm_reason = "eof without token; durable record pending"
continue
# H-PATCH: re-check the durable event log on each re-arm —
# the completion event may have arrived while the waiter was
# between await_resume_event cycles (the 9db239f/51416aa root
# cause). This makes internal re-arms event-aware, not
# timer-blind.
prior_event = ew.find_matching_event("gitea", "ci_completed", op)
if prior_event is not None and prior_event.get("exit_code") is not None:
event_at = event_at or time.strftime(
"%Y-%m-%dT%H:%M:%SZ", time.gmtime())
resumed_at = time.strftime(
"%Y-%m-%dT%H:%M:%SZ", time.gmtime())
verdict = normalize_verdict(
authority_after_wake(op_type, sha))
if verdict != "pending":
mode = "event_recovered_from_durable_record"
break
# watchdog expiry: operation-scoped liveness decides re-arm
# vs truthful recovery — failure detector, never scheduler
if op_ci_alive(op_type, sha):
@@ -461,6 +491,17 @@ def run(*, sha: str, deadline_secs: int,
continue
finally:
os.close(fifo_fd)
# H-PATCH: release the daemon lease on exit. Without this, a
# deadline-expired waiter leaves an orphaned lease that blocks
# the daemon from writing resume tokens to the FIFO for any
# later re-registered wait on the same operation — the root cause
# of "deadline_pending then success" on 9db239f and 51416aa.
try:
ws = ew._load_waitstore()
ws.release_lease(f"controller-{op}", rec.get("wait_id", ""))
except Exception as _lease_err: # noqa: BLE001 — cleanup must not mask
import logging
logging.debug("lease release: %s", _lease_err)
if verdict == "pending":
if mode == "event":
mode = "deadline_pending"
+29 -3
View File
@@ -105,12 +105,12 @@
"mandatory": true,
"requirement": "git==release==deployed==runtime==healthy",
"status": "PASS",
"evidence": "git master 9db239f == release v0.85.5 == deployed current == runtime build identity (deployed 2026-09-18T10:44:46Z) == healthy; last-good 658ec63; tags v0.85.4/v0.85.5; harness tools + repo scripts in",
"evidence": "git master 51416aa == release v0.85.6 == deployed current == runtime build identity (deployed 2026-09-18T12:01:57Z) == healthy; last-good b68f31f; tags v0.85.5/v0.85.6. Full CI green on 51416aa (the b",
"blocker": "",
"waiver_reason": "",
"waived_by": "",
"declared_at": "2026-09-18T08:56:52Z",
"resolved_at": "2026-09-18T11:13:21Z"
"resolved_at": "2026-09-18T12:25:21Z"
},
{
"gate_id": "H1_OPERATION_ID_FIRST_CONTINUATION",
@@ -225,5 +225,31 @@
"ok": true,
"blockers": []
},
"verdict": "EVENT_DRIVEN_CONTINUATION_COMPLETE"
"verdict": "EVENT_DRIVEN_CONTINUATION_COMPLETE",
"supersession": {
"superseded_at": "2026-09-18T15:35:46Z",
"reason": "RAW EVIDENCE CONTRADICTS PRIOR PASS RECORDS: (A) H8 violated \u2014 master waits 9db239f (wall ~1807s, mode=deadline_pending, verdict=success) and 51416aa (wall ~1521s, same) discovered terminal success AT the hard deadline: DEADLINE_DRIVEN_TERMINAL_DISCOVERY_COUNT != 0. (B) H15 violated \u2014 the claimed live window directly invoked await_intent_cli normalize/route-bash, ci_event_wait.run(), and slept 120 between PR-check re-arms: not tool-only proofs. (C) H18 violated \u2014 convergence was shown for 51416aa/0.85.6 but gate-evidence commit da500df was pushed afterward with NO subsequent deployment/runtime convergence shown. Corrected state: EVENT_DRIVEN_CONTINUATION_RETEST_REQUIRED. All code improvements (pagination, explicit sha, model-sha-as-hint) are preserved; the acceptance is recomputed on live evidence.",
"prior_pass_gates_recomputed": [
"H6",
"H7",
"H8",
"H12",
"H13",
"H14",
"H15",
"H17",
"H18"
],
"gates_retained_valid": [
"H1",
"H2",
"H3",
"H4",
"H5",
"H9",
"H10",
"H11",
"H16"
]
}
}
+55
View File
@@ -12,6 +12,7 @@ from __future__ import annotations
import importlib.util
import json
import os
import sys
import threading
import time
@@ -214,3 +215,57 @@ def test_deadline_is_the_only_model_return_for_pending(wr_daemon,
wall = time.time() - t0
assert rc == 1 # deadline pending
assert wall >= 7.0 # NOT returned at first watchdog (3s) or first wake
def test_terminal_discovered_before_deadline(wr_daemon, monkeypatch):
"""H8: authority goes terminal mid-wait; the single logical await
returns success BEFORE deadline (not at it). Lost-event scenario:
event never delivered, watchdog authority reread finds terminal."""
mod = _load()
calls = {"n": 0}
import time as _t
start = _t.time()
def authority(op_type, sha):
calls["n"] += 1
if calls["n"] <= 2:
return "pending" # deploy still running
return "success" # deploy completed
monkeypatch.setattr(mod, "op_authority_verdict", authority)
monkeypatch.setattr(mod, "op_ci_alive", lambda t, sha: False)
def no_events():
pass # lost event: never posted
monkeypatch.setattr(mod, "ew", type("EW", (), {
"open_fifo_reader": staticmethod(lambda op: os.open(os.devnull, os.O_RDONLY)),
"register_process_wait": mod.ew.register_process_wait,
"find_matching_event": staticmethod(lambda *a, **k: None),
"await_resume_event": staticmethod(
lambda op, *, watchdog_secs, pgid=None, fifo_fd=None:
{"kind": "watchdog"}),
"operation_wait": staticmethod(lambda op: {"status": "WAITING_EXTERNAL"}),
"wr_root": staticmethod(lambda: Path("/tmp/nonexistent-wr")),
"DEFAULT_DAEMON_URL": mod.ew.DEFAULT_DAEMON_URL,
}))
rc = mod.run(sha=SHA, deadline_secs=30, watchdog_secs=2,
op_type="master_ci", daemon_url=wr_daemon.url())
wall = _t.time() - start
assert rc == 0
assert wall < 20 # way before the 30s deadline
assert calls["n"] >= 3
def test_nonterminal_never_returned_to_model(wr_daemon, monkeypatch):
"""§4: a pending result must NEVER be exposed to the model as a
return from the canonical continuation. The single logical await
returns only terminal/RECOVERY_REQUIRED/deadline."""
# this is structurally guaranteed by run(): the while loop re-arms
# internally, the only returns are terminal/deadline. Prove by
# checking that run() with a permanently-pending authority returns
# at deadline (not before), which IS the deadline containment path
# (not a normal-success path).
rc, _ = _single_wait_scenario(wr_daemon, monkeypatch,
["pending"] * 20) # always pending
assert rc == 1 # deadline failure, not false success