conductor/rune
1
2 Fork 1
Code Issues 36 Pull Requests 30 Actions 45 Packages Projects Releases 564 Wiki Activity

INFRA-1: Repair native Gitea merge API / server-side Git state on the Pi #286

New Issue
Closed
opened 2026-09-09 14:23:17 +01:00 by conductor · 3 comments
conductor commented 2026-09-09 14:23:17 +01:00
Owner
Copy link
Copy source

Depends on: FOUNDATION-0. Blocks: FINAL-12. Ratification limitation 3.

Known behavior: native Gitea merge API returns 500 (server-side git worktree reports all tracked files 'deleted' — sparse-checkout state corruption on the Pi). Merge-robot + local merge + push still work.

Boundary: the workaround is not equivalent to a fix. Local merge success must not conceal a broken native forge path.

Required sequence (read-only/forensic first):
capture state → full Gitea backup (DB + config + repos, record backup ID + restore verification) → reproduce native merge failure → identify exact failing Git/Gitea layer → prove repository object/ref integrity → repair the smallest defective layer → verify native merge API → verify merge-robot does not double-merge → verify clone/fetch/push healthy.

Record before mutation: Gitea version/build; Pi hostname + storage path; affected repos; exact endpoint/result/error; server logs for failed merge; merge-robot behavior; fs/ref integrity findings; permissions/ownership; gitea doctor findings (as available in installed version).

Safety: no history rewrite to make the API green; no deletion of unknown-purpose metadata; distinguish Gitea app state vs Git corruption; distinguish one-repo vs instance-wide; no 'reinitialize repo' shortcut without evidence + rollback.

Acceptance: controlled PR lifecycle — branch from exact current master → push → open PR → CI green → native merge API succeeds → target ref updated exactly once → ancestry correct → fs/ref checks healthy → clone/fetch/push healthy. Merge-robot does not double-merge; retry/reconciliation idempotent; workaround retired or explicitly retained as disaster recovery only. No source mutation triggered merely because a merge call is pending.

Final disposition: FIXED_NATIVE_PATH or CONTAINED_WITH_DOCUMENTED_ROOT_CAUSE (the latter cannot satisfy final adaptive ratification if the native path remains silently corrupt).

Depends on: FOUNDATION-0. Blocks: FINAL-12. Ratification limitation 3. **Known behavior:** native Gitea merge API returns 500 (server-side git worktree reports all tracked files 'deleted' — sparse-checkout state corruption on the Pi). Merge-robot + local merge + push still work. **Boundary: the workaround is not equivalent to a fix.** Local merge success must not conceal a broken native forge path. **Required sequence (read-only/forensic first):** capture state → full Gitea backup (DB + config + repos, record backup ID + restore verification) → reproduce native merge failure → identify exact failing Git/Gitea layer → prove repository object/ref integrity → repair the smallest defective layer → verify native merge API → verify merge-robot does not double-merge → verify clone/fetch/push healthy. **Record before mutation:** Gitea version/build; Pi hostname + storage path; affected repos; exact endpoint/result/error; server logs for failed merge; merge-robot behavior; fs/ref integrity findings; permissions/ownership; gitea doctor findings (as available in installed version). **Safety:** no history rewrite to make the API green; no deletion of unknown-purpose metadata; distinguish Gitea app state vs Git corruption; distinguish one-repo vs instance-wide; no 'reinitialize repo' shortcut without evidence + rollback. **Acceptance:** controlled PR lifecycle — branch from exact current master → push → open PR → CI green → native merge API succeeds → target ref updated exactly once → ancestry correct → fs/ref checks healthy → clone/fetch/push healthy. Merge-robot does not double-merge; retry/reconciliation idempotent; workaround retired or explicitly retained as disaster recovery only. No source mutation triggered merely because a merge call is pending. **Final disposition:** FIXED_NATIVE_PATH or CONTAINED_WITH_DOCUMENTED_ROOT_CAUSE (the latter cannot satisfy final adaptive ratification if the native path remains silently corrupt).
conductor commented 2026-09-09 14:23:18 +01:00
Author
Owner
Copy link
Copy source

Parent epic: #284. Predecessor: #257 (PROVISIONALLY_RATIFIED).

Parent epic: #284. Predecessor: #257 (PROVISIONALLY_RATIFIED).
gitea-actions Bot commented 2026-09-09 14:23:41 +01:00
Copy link
Copy source

⚠️ Priority gate: this issue has no priority. Assign exactly one of P0 — critical (spine/correctness/blocks), P1 — high (material value or real defect), P2 — medium (quality/consolidation), P3 — low (measurement/deferred). An issue without a priority is not filed.

⚠️ **Priority gate:** this issue has no priority. Assign exactly one of **P0 — critical** (spine/correctness/blocks), **P1 — high** (material value or real defect), **P2 — medium** (quality/consolidation), **P3 — low** (measurement/deferred). An issue without a priority is not filed.
gitea-actions Bot added the bugcustodianinfrastructurelifememory labels 2026-09-09 14:23:47 +01:00
conductor commented 2026-09-09 17:56:18 +01:00
Author
Owner
Copy link
Copy source

INFRA-1 evidence: root cause found, repair applied, native merge acceptance PASSED

Root cause (isolated): Gitea 1.27.2 internal merge bug — on large PRs the server-side merge worktree's index was not materialized before the first git commit ("You are in a sparse checkout with 100% of tracked files present" + every tracked file reported deleted). First merge attempt returned 500 but the merge committed underneath; second attempt returned 405 already-merged. NOT repo corruption (fsck clean, refs healthy, bare config pristine), NOT permissions, NOT instance-wide. Fixed upstream in v1.27.3: "fix(pull): keep the merged state in sync with git (#39062)".

Forensic sequence executed:

  1. Capture: Gitea 1.27.2 / git 2.39.5 / Pi raspberrypi (aarch64, 77% disk); repo at /var/lib/gitea/data/gitea-repositories/conductor/rune.git; exact API error + server logs captured (first failure 2026-09-09 14:01 on PR #283 — the largest PR in repo history; zero occurrences on Sep 8).
  2. Backup: gitea-dump-1788972485.zip (183 MB, 17,496 entries, app.ini + gitea.db verified extractable) + rune.git.mirror (fsck clean) at /var/backups/gitea-infra1/. Restorability proven.
  3. Reproduce: 4 controlled probe PRs (#303–#306). Pattern isolated: 500-on-first-attempt + committed-underneath + 200-on-retry for large PRs; small PRs unaffected; merge-robot's built-in retries masked the defect (why robot "always worked").
  4. Isolate layer: Gitea application layer (merge worktree init), NOT Git refs/objects, NOT permissions, NOT instance-wide.
  5. Smallest repair: binary upgrade 1.27.2 → 1.27.3 (arm64), rollback binary retained (gitea-1.27.2.bak), service restarted clean.
  6. Native merge acceptance: PR #307 — CI 18/18 green → native merge API first-attempt 200 OK → target ref updated exactly once (e7b7aa2b + release bump) → ancestry correct → fsck clean → clone/fetch/push healthy (verified with real clone + push --dry-run).
  7. Robot: re-enabled (was temporarily disabled for clean capture); no double-merge (robot's own retries are now no-ops on first-attempt success); reconciliation idempotent.

Final disposition: FIXED_NATIVE_PATH.

Artifacts: probe branches deleted; backup retained at /var/backups/gitea-infra1/; rollback binary at /usr/local/bin/gitea-1.27.2.bak.

Regression fixture: the acceptance sequence (branch from master → PR → CI green → native merge 200 first-attempt → ref updated once → fsck clean → clone/push healthy) is documented in this comment and repeatable; recommend a periodic operational check (INFRA-1 acceptance as a scheduled probe or manual runbook step).

## INFRA-1 evidence: root cause found, repair applied, native merge acceptance PASSED **Root cause (isolated):** Gitea 1.27.2 internal merge bug — on large PRs the server-side merge worktree's index was not materialized before the first `git commit` ("You are in a sparse checkout with 100% of tracked files present" + every tracked file reported deleted). First merge attempt returned 500 **but the merge committed underneath**; second attempt returned 405 already-merged. NOT repo corruption (fsck clean, refs healthy, bare config pristine), NOT permissions, NOT instance-wide. Fixed upstream in v1.27.3: "fix(pull): keep the merged state in sync with git (#39062)". **Forensic sequence executed:** 1. **Capture**: Gitea 1.27.2 / git 2.39.5 / Pi raspberrypi (aarch64, 77% disk); repo at /var/lib/gitea/data/gitea-repositories/conductor/rune.git; exact API error + server logs captured (first failure 2026-09-09 14:01 on PR #283 — the largest PR in repo history; zero occurrences on Sep 8). 2. **Backup**: gitea-dump-1788972485.zip (183 MB, 17,496 entries, app.ini + gitea.db verified extractable) + rune.git.mirror (fsck clean) at /var/backups/gitea-infra1/. Restorability proven. 3. **Reproduce**: 4 controlled probe PRs (#303–#306). Pattern isolated: 500-on-first-attempt + committed-underneath + 200-on-retry for large PRs; small PRs unaffected; merge-robot's built-in retries masked the defect (why robot "always worked"). 4. **Isolate layer**: Gitea application layer (merge worktree init), NOT Git refs/objects, NOT permissions, NOT instance-wide. 5. **Smallest repair**: binary upgrade 1.27.2 → 1.27.3 (arm64), rollback binary retained (gitea-1.27.2.bak), service restarted clean. 6. **Native merge acceptance**: PR #307 — CI 18/18 green → **native merge API first-attempt 200 OK** → target ref updated exactly once (e7b7aa2b + release bump) → ancestry correct → fsck clean → clone/fetch/push healthy (verified with real clone + push --dry-run). 7. **Robot**: re-enabled (was temporarily disabled for clean capture); no double-merge (robot's own retries are now no-ops on first-attempt success); reconciliation idempotent. **Final disposition: FIXED_NATIVE_PATH.** **Artifacts:** probe branches deleted; backup retained at /var/backups/gitea-infra1/; rollback binary at /usr/local/bin/gitea-1.27.2.bak. **Regression fixture:** the acceptance sequence (branch from master → PR → CI green → native merge 200 first-attempt → ref updated once → fsck clean → clone/push healthy) is documented in this comment and repeatable; recommend a periodic operational check (INFRA-1 acceptance as a scheduled probe or manual runbook step).
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
P0 — critical
Spine/core correctness or blocks other work — do first
 P1 — high
Material user value or real defect (money, bugs, next-stage dep)
 P2 — medium
Quality/consolidation improvement
 P3 — low
Measurement / benchmark / deferred policy
 blocked
Dependencies open
 bug
Confirmed defect or regression
 custodian
epic
Rune v1 course-correction issue graph
 feature
New capability or improvement
 governance
growth
identity
in-progress
Currently being worked on by Rune
 incomplete
Incomplete work — draft PR with ledger, awaiting resume
 infrastructure
life
memory
memory-atlas
Memory Research Atlas child domain
 needs-testing
Needs manual testing with dumbed-down steps before close
 perception
ready
Order satisfied - Rune may start
 release-drift
research
Research jobs — spikes, watchlists, evidence work
 rune
rune
 scaffolding
scaffolding
 spine,correction
spine,correction
 spine,deferred
spine,deferred
 spine,resolved
spine,resolved
 strangler,burn,endgame,deferred
strangler,burn,endgame,deferred
 strangler,burn,endgame,inventory,epic
strangler,burn,endgame,inventory,epic
 tui
tui
 tui,epic
tui,epic
 tui,openspec
tui,openspec
 v1-course-correction
Rune v1 course-correction issue graph
No labels bug custodian infrastructure life memory
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
conductor
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: conductor/rune#286