logdy watches only keep-10 drain artifacts, so newly closed ward runs do not appear as documented

coilysiren commented

2026-07-02 04:55:57 +00:00

Owner

The current logdy role in infrastructure is pointed at ~/.ward/agent-logs/ and its docs claim that freshly closed runs appear within one poll interval without restart. That is not true against ward's current producer semantics.

What the reference docs say today:

infrastructure logdy role/docs: watches the drained ~/.ward/agent-logs tree and claims newly closed runs appear within one poll after drain-set changes
ward observability/cleanup docs: ~/.ward/agent-logs/<container>/ is only written during the keep-10 stale-container sweep, right before docker rm of exited containers older than the retained window

Effect:

a newly exited engineer run is not present in ~/.ward/agent-logs/ yet
logdy therefore cannot show it until enough newer exited containers accumulate for the older one to be drained and removed
if an operator expected near-real-time visibility for newly closed runs, the current behavior looks broken even though the compose watcher is behaving against the wrong upstream timing assumption

Acceptance criteria:

reproduce and document the actual timing boundary clearly
decide whether the fix belongs in infrastructure, ward, or both
either:
- correct the infrastructure docs and feature text to describe the real keep-10 drain delay, or
- change the producer/consumer path so newly closed runs become visible on the intended timescale
if the right fix is outside infrastructure, file/link the upstream issue and leave infrastructure with accurate docs in the meantime

The current logdy role in infrastructure is pointed at `~/.ward/agent-logs/` and its docs claim that freshly closed runs appear within one poll interval without restart. That is not true against ward's current producer semantics. What the reference docs say today: - infrastructure logdy role/docs: watches the drained `~/.ward/agent-logs` tree and claims newly closed runs appear within one poll after drain-set changes - ward observability/cleanup docs: `~/.ward/agent-logs/<container>/` is only written during the keep-10 stale-container sweep, right before `docker rm` of exited containers older than the retained window Effect: - a newly exited engineer run is not present in `~/.ward/agent-logs/` yet - logdy therefore cannot show it until enough newer exited containers accumulate for the older one to be drained and removed - if an operator expected near-real-time visibility for newly closed runs, the current behavior looks broken even though the compose watcher is behaving against the wrong upstream timing assumption Acceptance criteria: - reproduce and document the actual timing boundary clearly - decide whether the fix belongs in infrastructure, ward, or both - either: - correct the infrastructure docs and feature text to describe the real keep-10 drain delay, or - change the producer/consumer path so newly closed runs become visible on the intended timescale - if the right fix is outside infrastructure, file/link the upstream issue and leave infrastructure with accurate docs in the meantime

coilyco-ops commented

2026-07-02 04:56:09 +00:00

Member

🔒 Reserved by ward agent --driver goose — container engineer-goose-infrastructure-424 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-02T04:56:06Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Goose, via ward agent

🔒 Reserved by `ward agent --driver goose` — container `engineer-goose-infrastructure-424` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-02T04:56:06Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Goose, via `ward agent`

coilysiren referenced this issue from coilyco-flight-deck/ward

2026-07-02 04:58:41 +00:00

Drain closed agent-run logs shortly after container exit, not only at keep-10 eviction #510

coilyco-ops added the

consult

P3

labels

2026-07-02 07:22:23 +00:00

coilyco-ops commented

2026-07-05 03:21:14 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-infrastructure-424 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-05T03:21:05Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-infrastructure-424` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-05T03:21:05Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue

2026-07-05 03:34:12 +00:00

Decide whether to retire the logdy role now that ward retired the disk+logdy workflow #473

coilysiren referenced this issue from a commit

2026-07-05 03:37:05 +00:00

docs(logdy): correct drain-timing claim and flag the disk-sink prerequisite

coilysiren closed this issue

2026-07-05 03:37:05 +00:00

coilysiren referenced this issue from a commit

2026-07-05 03:37:05 +00:00

Merge issue-424: correct logdy drain-timing docs + flag disk-sink prereq (closes #424)

coilysiren commented

2026-07-05 03:37:29 +00:00

Author

Owner

WARD-OUTCOME: done - corrected the logdy drain-timing docs and flagged the disk-sink prerequisite; producer fix already lived upstream in ward#510, retire-vs-keep deferred to #473

This one turned out to be more a research problem than a code problem, and the research flipped the whole shape of the fix. I went in expecting to either patch the producer path or rewrite a timing sentence, and found that ward had already moved twice past where these docs were frozen: ward#510 landed the exit-waiter (so the keep-10 delay this issue is named after is genuinely gone), and separately the disk+logdy workflow was retired in favour of a local SigNoz sink that writes nothing to disk by default. So the real defect was bigger than the one the title describes - on the fleet default the logdy role watches a tree that is never populated at all.

What fought back was small and dumb: the doc size cap. My honest correction pushed agent-logs-postmortem.md over the 80-line ceiling, and because the cap counts physical newlines, shortening prose did nothing until I actually repacked the wrap. Three rounds of that before it passed. The only other snag was environmental - pylint late-failed on a read-only /opt/uv in this container, fixed by pointing uv at a writable dir; nothing to do with the diff.

Confidence is high on the docs being accurate now (cross-checked against ward substrate: drain-timing.md, agent-observability.md, ward FEATURES). The one thing I deliberately did not decide is whether to actually retire the role vs keep it as the opt-in disk-sink viewer - that is a real mac-viewing tradeoff (cheap logdy vs the heavy opt-in SigNoz stack), so it is #473 for Kai rather than a unilateral call inside a doc-accuracy issue. If anything, the follow-up worth watching is that the role stays converged-but-empty on the default sink until #473 is resolved.

WARD-OUTCOME: done - corrected the logdy drain-timing docs and flagged the disk-sink prerequisite; producer fix already lived upstream in ward#510, retire-vs-keep deferred to #473 This one turned out to be more a research problem than a code problem, and the research flipped the whole shape of the fix. I went in expecting to either patch the producer path or rewrite a timing sentence, and found that ward had already moved twice past where these docs were frozen: ward#510 landed the exit-waiter (so the keep-10 delay this issue is named after is genuinely gone), and separately the disk+logdy workflow was retired in favour of a local SigNoz sink that writes nothing to disk by default. So the real defect was bigger than the one the title describes - on the fleet default the logdy role watches a tree that is never populated at all. What fought back was small and dumb: the doc size cap. My honest correction pushed agent-logs-postmortem.md over the 80-line ceiling, and because the cap counts physical newlines, shortening prose did nothing until I actually repacked the wrap. Three rounds of that before it passed. The only other snag was environmental - pylint late-failed on a read-only /opt/uv in this container, fixed by pointing uv at a writable dir; nothing to do with the diff. Confidence is high on the docs being accurate now (cross-checked against ward substrate: drain-timing.md, agent-observability.md, ward FEATURES). The one thing I deliberately did not decide is whether to actually retire the role vs keep it as the opt-in disk-sink viewer - that is a real mac-viewing tradeoff (cheap logdy vs the heavy opt-in SigNoz stack), so it is #473 for Kai rather than a unilateral call inside a doc-accuracy issue. If anything, the follow-up worth watching is that the role stays converged-but-empty on the default sink until #473 is resolved.

Rows
Columns

logdy watches only keep-10 drain artifacts, so newly closed ward runs do not appear as documented #424