coilyco-flight-deck/agentic-os

Watch

Issue-corpus renderer: git-mirrored discovery index of fj issues for warded containers #297

New issue

Closed

opened 2026-06-26 02:40:34 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-06-26 02:40:34 +00:00

Owner

Motivation

A critical bit of information lived in coilysiren/inbox and neither Kai nor the
agent could find it. It surfaced only after a native agent grepped two weeks of
session logs. The real gap is offline full-text grep over the issue/inbox
corpus. Today ward ops forgejo issue list-all gives title/label filtering and
one-issue-at-a-time API reads. It cannot do "grep every body and comment across
every repo for this phrase", which is exactly the operation that finally worked.

Design decision (settled with Kai)

Build a discovery index, not an offline source of truth. Render the issue
corpus to git-mirrored markdown, hydrated into containers via the existing
warm-cache / gitcache machinery (the same path that seeds /substrate/<name>
from /opt/substrate-seed, see docker/dev-base/Dockerfile:163-181). The agent
greps the corpus offline to locate an issue, then confirms current state live
via ward ops forgejo issue view <owner> <name> <N>. Grep finds the needle, the
API confirms the live state. This respects the anti-drift doctrine in AGENTS.md
(a point-in-time snapshot must not silently anchor a stale picture) and reuses
the hydrate path unchanged: issues become just another mirror input.

Scope chosen: all tracked repos, open AND closed issues. The needle could
just as easily live in a closed issue.

This issue = the renderer + its cron (authoring side)

Per the authoring-vs-rollout contract, the renderer is a tool authored here in
agentic-os and triggered by a cron, never hand-run. The container mount wiring is
a separate ward issue (filed as a sibling, blocked on this).

Spec

Source set - the warm-cache substrate repos
(docker/dev-base/substrate-image-repos.txt) plus coilysiren/inbox, open and
closed, bodies and all comments. Make the repo list a small config, not hardcoded.
Output - one markdown file per issue, path <owner>/<name>/<index>-<slug>.md,
in a dedicated git mirror repo.
Per-file header - issue number, repo, state: open|closed, rendered-at
(timestamp + source commit/issue updated_at), title, labels, author. Then the
body and every comment. The header must carry the index disclaimer: "discovery
index, confirm live state via ward ops forgejo issue view <owner> <name> <N>".
Privacy - coilysiren/inbox is private. The mirror repo MUST be private and
MUST NOT land in the public substrate image seed
(substrate-image-repos.txt is the public tier). It goes in the private
cache-tier, hydrated with creds. Run trufflehog over the rendered corpus before
push as the secret-scan backstop.
Token boundary - all forgejo I/O goes through ward ops forgejo (ward-kdl),
so the renderer holds no FORGEJO_TOKEN, same as scripts/goose-triage.py:41-46.
Cadence - hourly CI cron (a .forgejo/workflows/ job). Tight cadence bounds
staleness. Incremental where practical (use updated_at / since to re-render
only changed issues).

Out of scope (separate issues)

Mounting the corpus into containers via gitcache + mount-eligibility - ward
sibling issue, blocked on this.
A session-log search corpus (the other half of the motivation, the thing that
was actually grepped) - separate agentic-os sibling issue.

Design consult and capture from a warded architect session.

## Motivation A critical bit of information lived in `coilysiren/inbox` and neither Kai nor the agent could find it. It surfaced only after a native agent grepped two weeks of session logs. The real gap is **offline full-text grep over the issue/inbox corpus**. Today `ward ops forgejo issue list-all` gives title/label filtering and one-issue-at-a-time API reads. It cannot do "grep every body and comment across every repo for this phrase", which is exactly the operation that finally worked. ## Design decision (settled with Kai) Build a **discovery index**, not an offline source of truth. Render the issue corpus to git-mirrored markdown, hydrated into containers via the existing warm-cache / gitcache machinery (the same path that seeds `/substrate/<name>` from `/opt/substrate-seed`, see `docker/dev-base/Dockerfile:163-181`). The agent greps the corpus offline to **locate** an issue, then confirms current state live via `ward ops forgejo issue view <owner> <name> <N>`. Grep finds the needle, the API confirms the live state. This respects the anti-drift doctrine in AGENTS.md (a point-in-time snapshot must not silently anchor a stale picture) and reuses the hydrate path unchanged: issues become just another mirror input. Scope chosen: **all tracked repos, open AND closed** issues. The needle could just as easily live in a closed issue. ## This issue = the renderer + its cron (authoring side) Per the authoring-vs-rollout contract, the renderer is a tool authored here in agentic-os and triggered by a cron, never hand-run. The container mount wiring is a separate ward issue (filed as a sibling, blocked on this). ### Spec - **Source set** - the warm-cache substrate repos (`docker/dev-base/substrate-image-repos.txt`) plus `coilysiren/inbox`, open and closed, bodies and all comments. Make the repo list a small config, not hardcoded. - **Output** - one markdown file per issue, path `<owner>/<name>/<index>-<slug>.md`, in a dedicated git mirror repo. - **Per-file header** - issue number, repo, `state: open|closed`, `rendered-at` (timestamp + source commit/issue updated_at), title, labels, author. Then the body and every comment. The header must carry the index disclaimer: "discovery index, confirm live state via `ward ops forgejo issue view <owner> <name> <N>`". - **Privacy** - `coilysiren/inbox` is private. The mirror repo MUST be private and MUST NOT land in the public substrate image seed (`substrate-image-repos.txt` is the public tier). It goes in the private cache-tier, hydrated with creds. Run trufflehog over the rendered corpus before push as the secret-scan backstop. - **Token boundary** - all forgejo I/O goes through `ward ops forgejo` (ward-kdl), so the renderer holds no `FORGEJO_TOKEN`, same as `scripts/goose-triage.py:41-46`. - **Cadence** - hourly CI cron (a `.forgejo/workflows/` job). Tight cadence bounds staleness. Incremental where practical (use `updated_at` / `since` to re-render only changed issues). ### Out of scope (separate issues) - Mounting the corpus into containers via gitcache + mount-eligibility - **ward** sibling issue, blocked on this. - A session-log search corpus (the other half of the motivation, the thing that was actually grepped) - separate agentic-os sibling issue. Design consult and capture from a `warded architect` session.

coilysiren commented

2026-06-26 02:41:17 +00:00

Author

Owner

🔒 Reserved by ward agent --driver claude — container ward-agentic-os-issue-297-claude-8d9f0a47 on host 61f1003a88ec is carrying this issue (reserved 2026-06-26T02:41:16Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `ward-agentic-os-issue-297-claude-8d9f0a47` on host `61f1003a88ec` is carrying this issue (reserved 2026-06-26T02:41:16Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-06-26 02:52:35 +00:00

feat(issue-corpus): render Forgejo issues to a git-mirrored discovery index

coilysiren closed this issue

2026-06-26 02:52:35 +00:00

coilysiren commented

2026-06-26 02:53:19 +00:00

Author

Owner

WARD-OUTCOME: done - issue-corpus renderer + hourly cron landed on main (closes #297)

Retrospective from the agent that carried this:

The shape settled fast because the issue had already done the hard design work
with Kai, so this was mostly faithful execution. The renderer worked first try
against the live Forgejo API - ward ops forgejo issue list-all --state all plus
a per-issue issue-comment list gave everything the header needed, and the
incremental manifest skip dropped the second run of 296 agentic-os issues to zero
re-renders and zero comment fetches, which is exactly the cheap-listing-expensive-
comment split I was hoping for.

What fought back was small: the code-comments hook rejected a YAML comment tucked
under the env: block (a key-sorter would drift it), so I moved that note up into
the top header. Good discipline, caught at author time.

The honest soft spot is that this is the authoring half only. The renderer and
cron are real and tested (29 unit tests, live smoke run over the full agentic-os
backlog), but the cron cannot actually fire green until the rollout side lands in
infrastructure/ansible: ward + AWS/SSM on the Forgejo runner, the private
coilysiren/issue-corpus mirror repo, and the ISSUE_CORPUS_PUSH_TOKEN secret.
I made it no-op cleanly while the token is unset so it does not red-fail hourly in
the meantime. Confidence in the renderer logic is high; confidence that the first
scheduled run is green is lower, since the runner provisioning is out of this repo
and untested here.

Follow-ups worth filing: the container mount that hydrates this mirror into
/substrate is the blocked ward sibling and still needs writing; deleted issues
currently leave a stale file behind (no reaping); and coilysiren/coilysiren is
omitted until its Forgejo mirror lands (infrastructure#352).

WARD-OUTCOME: done - issue-corpus renderer + hourly cron landed on main (closes #297) Retrospective from the agent that carried this: The shape settled fast because the issue had already done the hard design work with Kai, so this was mostly faithful execution. The renderer worked first try against the live Forgejo API - `ward ops forgejo issue list-all --state all` plus a per-issue `issue-comment list` gave everything the header needed, and the incremental manifest skip dropped the second run of 296 agentic-os issues to zero re-renders and zero comment fetches, which is exactly the cheap-listing-expensive- comment split I was hoping for. What fought back was small: the code-comments hook rejected a YAML comment tucked under the `env:` block (a key-sorter would drift it), so I moved that note up into the top header. Good discipline, caught at author time. The honest soft spot is that this is the authoring half only. The renderer and cron are real and tested (29 unit tests, live smoke run over the full agentic-os backlog), but the cron cannot actually fire green until the rollout side lands in infrastructure/ansible: `ward` + AWS/SSM on the Forgejo runner, the private `coilysiren/issue-corpus` mirror repo, and the `ISSUE_CORPUS_PUSH_TOKEN` secret. I made it no-op cleanly while the token is unset so it does not red-fail hourly in the meantime. Confidence in the renderer logic is high; confidence that the first scheduled run is green is lower, since the runner provisioning is out of this repo and untested here. Follow-ups worth filing: the container mount that hydrates this mirror into `/substrate` is the blocked ward sibling and still needs writing; deleted issues currently leave a stale file behind (no reaping); and `coilysiren/coilysiren` is omitted until its Forgejo mirror lands (infrastructure#352).

coilysiren commented

2026-06-26 03:02:43 +00:00

Author

Owner

Design refinement (from the architect session, surfaced after dispatch):

Segment the rendered output by sensitivity. Emit two corpus subtrees (or two
mirror repos):

a public-tier corpus - the public substrate repos
(docker/dev-base/substrate-image-repos.txt), greppable by any agent, and
a private corpus - coilysiren/inbox plus any private repos, gated.

Why: the mount layer (ward#367) gates inbox content behind per-driver permission
segmentation. A clean public/private split lets the public issue corpus mount
broadly while inbox mounts only for the cleared driver tier. If a split is
expensive, whole-corpus-private is the safe default (it just over-gates the
public issues) and the split can be a follow-up - do not block this carry on it.

Either way the mirror stays private and never touches the public image seed.

Design refinement (from the architect session, surfaced after dispatch): **Segment the rendered output by sensitivity.** Emit two corpus subtrees (or two mirror repos): - a **public-tier corpus** - the public substrate repos (`docker/dev-base/substrate-image-repos.txt`), greppable by any agent, and - a **private corpus** - `coilysiren/inbox` plus any private repos, gated. Why: the mount layer (ward#367) gates inbox content behind per-driver permission segmentation. A clean public/private split lets the public issue corpus mount broadly while inbox mounts only for the cleared driver tier. If a split is expensive, **whole-corpus-private is the safe default** (it just over-gates the public issues) and the split can be a follow-up - do not block this carry on it. Either way the mirror stays private and never touches the public image seed.

coilysiren referenced this issue from coilyco-flight-deck/ward

2026-07-03 17:50:30 +00:00

Mount the issue-corpus into /substrate via coilysiren/inbox double-duty (consumer half of agentic-os#297) #575

coilysiren referenced this issue from coilyco-flight-deck/infrastructure

2026-07-03 18:11:12 +00:00

Activate issue-corpus: provision Forgejo runner + wire ISSUE_CORPUS_PUSH_TOKEN (write to coilysiren/inbox) - keystone for agentic-os#324 / ward#575 #453