coilyco-flight-deck/ward

Fork 0

Fix 4 KB truncation in director PR mergeability reads #863

New issue

Closed

opened 2026-07-09 19:24:45 +00:00 by coilyco-ops · 4 comments

coilyco-ops commented

2026-07-09 19:24:45 +00:00

Member

Bug

A director merge sweep hit the same failure across every open ward PR on 2026-07-09:

ward agent director --harness codex: not merging coilyco-flight-deck/ward#862 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#862: unexpected end of JSON input
ward agent director --harness codex: not merging coilyco-flight-deck/ward#858 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#858: unexpected end of JSON input
ward agent director --harness codex: not merging coilyco-flight-deck/ward#857 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#857: unexpected end of JSON input
ward agent director --harness codex: not merging coilyco-flight-deck/ward#856 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#856: unexpected end of JSON input
ward agent director --harness codex: not merging coilyco-flight-deck/ward#854 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#854: unexpected end of JSON input
ward agent director --harness codex: not merging coilyco-flight-deck/ward#851 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#851: unexpected end of JSON input
ward agent director --harness codex: not merging coilyco-flight-deck/ward#850 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#850: unexpected end of JSON input
ward agent director --harness codex: not merging coilyco-flight-deck/ward#848 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#848: unexpected end of JSON input

The code path is forgejoClient.getPullRequest: it accepts a 2xx status, reads up to 4096 bytes, and immediately json.Unmarshals the body. If Forgejo, Caddy, or the network path transiently returns an empty 2xx body, the director parks every PR in the sweep with a low-signal JSON parse error. A manual probe a few minutes later returned normal JSON for PR #862 and #858, so this looks transient rather than a permanent PR data issue.

Desired behavior

Harden the PR detail read used by ward agent director merge:

Retry a small number of times when the response is 2xx but the body is empty or malformed JSON. Keep non-2xx behavior fail-loud.
If it still fails, report a useful error that names the HTTP status, byte count, and first response line/snippet rather than only unexpected end of JSON input.
Keep the merge gate fail-closed: unknown mergeability still skips the PR.
Add focused httptest coverage for empty-body-then-valid and persistent empty-body cases.
Run ward exec test.

Notes

This is separate from host AWS auth expiry. Expired AWS can break launch or token retrieval paths, but this specific symptom came after the PR list was readable and while parsing each PR detail body.

## Bug A director merge sweep hit the same failure across every open ward PR on 2026-07-09: ``` ward agent director --harness codex: not merging coilyco-flight-deck/ward#862 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#862: unexpected end of JSON input ward agent director --harness codex: not merging coilyco-flight-deck/ward#858 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#858: unexpected end of JSON input ward agent director --harness codex: not merging coilyco-flight-deck/ward#857 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#857: unexpected end of JSON input ward agent director --harness codex: not merging coilyco-flight-deck/ward#856 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#856: unexpected end of JSON input ward agent director --harness codex: not merging coilyco-flight-deck/ward#854 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#854: unexpected end of JSON input ward agent director --harness codex: not merging coilyco-flight-deck/ward#851 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#851: unexpected end of JSON input ward agent director --harness codex: not merging coilyco-flight-deck/ward#850 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#850: unexpected end of JSON input ward agent director --harness codex: not merging coilyco-flight-deck/ward#848 - could not read PR mergeability: forgejo: parse pull request coilyco-flight-deck/ward#848: unexpected end of JSON input ``` The code path is `forgejoClient.getPullRequest`: it accepts a 2xx status, reads up to 4096 bytes, and immediately `json.Unmarshal`s the body. If Forgejo, Caddy, or the network path transiently returns an empty 2xx body, the director parks every PR in the sweep with a low-signal JSON parse error. A manual probe a few minutes later returned normal JSON for PR #862 and #858, so this looks transient rather than a permanent PR data issue. ## Desired behavior Harden the PR detail read used by `ward agent director merge`: * Retry a small number of times when the response is 2xx but the body is empty or malformed JSON. Keep non-2xx behavior fail-loud. * If it still fails, report a useful error that names the HTTP status, byte count, and first response line/snippet rather than only `unexpected end of JSON input`. * Keep the merge gate fail-closed: unknown mergeability still skips the PR. * Add focused `httptest` coverage for empty-body-then-valid and persistent empty-body cases. * Run `ward exec test`. ## Notes This is separate from host AWS auth expiry. Expired AWS can break launch or token retrieval paths, but this specific symptom came after the PR list was readable and while parsing each PR detail body.

coilyco-ops added the

headless

labels

2026-07-09 19:24:45 +00:00

coilyco-ops commented

2026-07-09 19:24:53 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-863 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-09T19:24:51Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#863 · branch issue-863 · harness codex · workflow direct-to-main
Run: engineer-codex-ward-863 · ward v0.522.0 · dispatched 2026-07-09T19:24:51Z
Comment thread: 0 included in the pre-flight read, 0 stripped (ward's own automated comments).

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.522.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-863` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-09T19:24:51Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#863` · branch `issue-863` · harness `codex` · workflow `direct-to-main` - **Run:** `engineer-codex-ward-863` · ward `v0.522.0` · dispatched `2026-07-09T19:24:51Z` - **Comment thread:** 0 included in the pre-flight read, 0 stripped (ward's own automated comments). Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.522.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops commented

2026-07-09 19:25:57 +00:00

Author

Member

Root-cause correction from the director surface after filing:

This is deterministic truncation, not primarily an empty/transient Forgejo response. getPullRequest does:

data, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
...
json.Unmarshal(data, &raw)

A live PR detail response for #862 is about 7.8 KB. Reading only 4096 bytes cuts valid JSON mid-object, so json.Unmarshal returns unexpected end of JSON input for every open PR whose detail payload crosses that cap.

Please fix by reading the full body with a defensible maximum for diagnostics, or decode only the needed mergeable field without truncating the JSON stream. Keep the fail-closed merge gate and add a regression test with a PR detail body larger than 4096 bytes.

Root-cause correction from the director surface after filing: This is deterministic truncation, not primarily an empty/transient Forgejo response. `getPullRequest` does: ```go data, _ := io.ReadAll(io.LimitReader(resp.Body, 4096)) ... json.Unmarshal(data, &raw) ``` A live PR detail response for #862 is about 7.8 KB. Reading only 4096 bytes cuts valid JSON mid-object, so `json.Unmarshal` returns `unexpected end of JSON input` for every open PR whose detail payload crosses that cap. Please fix by reading the full body with a defensible maximum for diagnostics, or decode only the needed `mergeable` field without truncating the JSON stream. Keep the fail-closed merge gate and add a regression test with a PR detail body larger than 4096 bytes.

coilyco-ops changed title from ~~Harden director PR mergeability reads against empty Forgejo responses~~ to Fix 4 KB truncation in director PR mergeability reads

2026-07-09 19:26:02 +00:00

coilyco-ops commented

2026-07-09 19:26:23 +00:00

Author

Member

WARD-RESERVATION: held 🔒

reservation details

Holder: container engineer-codex-ward-863 on host kais-macbook-pro-2.local.

Reserved by ward agent --harness codex (reserved 2026-07-09T19:26:21Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (1h TTL). --force overrides.

run seed context — what this run is carrying (ward#609)

Resolved: coilyco-flight-deck/ward#863 · branch issue-863 · harness codex · workflow direct-to-main
Run: engineer-codex-ward-863 · ward v0.522.0 · dispatched 2026-07-09T19:26:21Z
Comment thread: 1 included in the pre-flight read, 1 stripped (ward's own automated comments).
- included: @coilyco-ops (2026-07-09T19:25:57Z)
- stripped: @coilyco-ops (2026-07-09T19:24:53Z)

Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.522.0).

— Codex, via ward agent

WARD-RESERVATION: held 🔒 <details><summary>reservation details</summary> Holder: container `engineer-codex-ward-863` on host `kais-macbook-pro-2.local`. Reserved by `ward agent --harness codex` (reserved 2026-07-09T19:26:21Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (1h TTL). `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh**. That is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494). <details><summary>run seed context — what this run is carrying (ward#609)</summary> - **Resolved:** `coilyco-flight-deck/ward#863` · branch `issue-863` · harness `codex` · workflow `direct-to-main` - **Run:** `engineer-codex-ward-863` · ward `v0.522.0` · dispatched `2026-07-09T19:26:21Z` - **Comment thread:** 1 included in the pre-flight read, 1 stripped (ward's own automated comments). - included: @coilyco-ops (2026-07-09T19:25:57Z) - stripped: @coilyco-ops (2026-07-09T19:24:53Z) Static container doctrine and seed boilerplate are identical every run and omitted here (they ride ward v0.522.0). </details> </details>  — Codex, via `ward agent`

coilyco-ops referenced this issue

2026-07-09 19:30:30 +00:00

Clean invalid and conflicted PRs out of the ward director merge queue #864

coilyco-ops referenced this issue from a commit

2026-07-09 19:30:50 +00:00

fix director PR mergeability reads

coilyco-ops closed this issue

2026-07-09 19:30:50 +00:00

coilyco-ops commented

2026-07-09 19:31:19 +00:00

Author

Member

WARD-OUTCOME: done

details

workflow: direct-to-main; review summary: review gate skipped by ~/.ward/config.yaml default

Implementation felt straightforward after tracing the 4 KB limit to getPullRequest and pinning retry plus snippet errors.
Confidence: high.
Surprises: none beyond the deterministic truncation and the existing fail-closed merge path.
Follow-ups: none.

WARD-OUTCOME: done <details><summary>details</summary> workflow: direct-to-main; review summary: review gate skipped by ~/.ward/config.yaml default Implementation felt straightforward after tracing the 4 KB limit to getPullRequest and pinning retry plus snippet errors. Confidence: high. Surprises: none beyond the deterministic truncation and the existing fail-closed merge path. Follow-ups: none. </details>